Browse Source

Merge remote-tracking branch 'cauldron/upstream-master' into cauldron-cdh6.x

Change-Id: I7b220c1c4d280d7bca86146bcc86c087adbf9763
Romain Rigaux 6 years ago
parent
commit
d2593bdf4e
100 changed files with 977 additions and 350 deletions
  1. 29 14
      apps/impala/src/impala/api.py
  2. 23 2
      apps/impala/src/impala/conf.py
  3. 2 1
      apps/impala/src/impala/dbms.py
  4. 1 0
      apps/impala/src/impala/urls.py
  5. 13 7
      apps/jobbrowser/src/jobbrowser/api2.py
  6. 2 2
      apps/jobbrowser/src/jobbrowser/apis/base_api.py
  7. 13 6
      apps/jobbrowser/src/jobbrowser/apis/query_api.py
  8. 14 11
      apps/jobbrowser/src/jobbrowser/conf.py
  9. 0 0
      apps/jobbrowser/src/jobbrowser/static/jobbrowser/css/jobbrowser-embeddable.css
  10. 147 37
      apps/jobbrowser/src/jobbrowser/static/jobbrowser/js/impala_dagre.js
  11. 116 10
      apps/jobbrowser/src/jobbrowser/static/jobbrowser/less/jobbrowser-embeddable.less
  12. 78 27
      apps/jobbrowser/src/jobbrowser/templates/job_browser.mako
  13. 6 6
      apps/jobbrowser/src/jobbrowser/views.py
  14. 28 17
      apps/jobbrowser/src/jobbrowser/yarn_models.py
  15. 2 2
      apps/oozie/src/oozie/models2.py
  16. 1 1
      apps/oozie/src/oozie/views/dashboard.py
  17. 0 19
      desktop/core/ext-py/pam/PKG-INFO
  18. 0 123
      desktop/core/ext-py/pam/pam.py
  19. 0 32
      desktop/core/ext-py/pam/setup.py
  20. 21 0
      desktop/core/ext-py/python-pam-1.8.4/LICENSE
  21. 2 0
      desktop/core/ext-py/python-pam-1.8.4/MANIFEST.in
  22. 74 0
      desktop/core/ext-py/python-pam-1.8.4/PKG-INFO
  23. 47 0
      desktop/core/ext-py/python-pam-1.8.4/README.md
  24. 47 0
      desktop/core/ext-py/python-pam-1.8.4/README.rst
  25. 224 0
      desktop/core/ext-py/python-pam-1.8.4/pam.py
  26. 0 1
      desktop/core/ext-py/python-pam-1.8.4/setup.cfg
  27. 37 0
      desktop/core/ext-py/python-pam-1.8.4/setup.py
  28. 2 2
      desktop/core/src/desktop/api2.py
  29. 19 4
      desktop/core/src/desktop/conf.py
  30. 1 0
      desktop/core/src/desktop/lib/fsmanager.py
  31. 13 17
      desktop/core/src/desktop/models.py
  32. 1 1
      desktop/core/src/desktop/settings.py
  33. 0 0
      desktop/core/src/desktop/static/desktop/css/hue-embedded.css
  34. 0 0
      desktop/core/src/desktop/static/desktop/css/hue.css
  35. 0 0
      desktop/core/src/desktop/static/desktop/css/hue3-extra.css
  36. 0 1
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_abort_on_default_limit_exceeded.json
  37. 1 1
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_abort_on_error.json
  38. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_aggregate_functions.json
  39. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_aliases.json
  40. 1 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_allow_erasure_coded_files.json
  41. 1 1
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_allow_unsupported_formats.json
  42. 1 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_alter_database.json
  43. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_alter_table.json
  44. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_alter_view.json
  45. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_analytic_functions.json
  46. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_appx_count_distinct.json
  47. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_appx_median.json
  48. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_array.json
  49. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_avg.json
  50. 1 1
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_batch_size.json
  51. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_bigint.json
  52. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_bit_functions.json
  53. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_boolean.json
  54. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_buffer_pool_limit.json
  55. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_char.json
  56. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_comment.json
  57. 1 1
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_comments.json
  58. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_complex_types.json
  59. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_compression_codec.json
  60. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_compute_stats.json
  61. 1 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_compute_stats_min_sample_size.json
  62. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_conditional_functions.json
  63. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_conversion_functions.json
  64. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_count.json
  65. 1 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_create_data_source.json
  66. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_create_database.json
  67. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_create_function.json
  68. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_create_role.json
  69. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_create_table.json
  70. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_create_view.json
  71. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_custom_timezones.json
  72. 1 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_data_sources.json
  73. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_databases.json
  74. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_datatypes.json
  75. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_date.json
  76. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_datetime_functions.json
  77. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_ddl.json
  78. 1 1
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_debug_action.json
  79. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_decimal.json
  80. 1 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_decimal_v2.json
  81. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_default_join_distribution_mode.json
  82. 0 1
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_default_order_by_limit.json
  83. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_default_spillable_buffer_size.json
  84. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_delete.json
  85. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_describe.json
  86. 1 1
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_disable_codegen.json
  87. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_disable_codegen_rows_threshold.json
  88. 1 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_disable_outermost_topn.json
  89. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_disable_row_runtime_filtering.json
  90. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_disable_streaming_preaggregations.json
  91. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_disable_unsafe_spills.json
  92. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_distinct.json
  93. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_dml.json
  94. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_double.json
  95. 1 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_drop_data_source.json
  96. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_drop_database.json
  97. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_drop_function.json
  98. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_drop_role.json
  99. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_drop_stats.json
  100. 0 0
      desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_drop_table.json

+ 29 - 14
apps/impala/src/impala/api.py

@@ -25,27 +25,27 @@ import struct
 from django.utils.translation import ugettext as _
 from django.views.decorators.http import require_POST
 
-from desktop.lib.django_util import JsonResponse
-from desktop.models import Document2
-
 from beeswax.api import error_handler
-from beeswax.server.dbms import get_cluster_config
 from beeswax.models import Session
 from beeswax.server import dbms as beeswax_dbms
+from beeswax.server.dbms import get_cluster_config
 from beeswax.views import authorized_get_query_history
 
+from desktop.lib.django_util import JsonResponse
+from desktop.models import Document2
+
+from jobbrowser.apis.query_api import _get_api
 from impala import dbms
-from impala.dbms import _get_server_name
 from impala.server import get_api as get_impalad_api, _get_impala_server_url
 
-from libanalyze import analyze as analyzer
-from libanalyze import rules
+from libanalyze import analyze as analyzer, rules
 
 from notebook.models import make_notebook
 
 LOG = logging.getLogger(__name__)
 ANALYZER = rules.TopDownAnalysis() # We need to parse some files so save as global
 
+
 @require_POST
 @error_handler
 def invalidate(request):
@@ -130,17 +130,13 @@ def alanize(request):
   cluster = json.loads(request.POST.get('cluster', '{}'))
   query_id = json.loads(request.POST.get('query_id'))
 
-  application = _get_server_name(cluster)
-  query_server = dbms.get_query_server_config()
-  session = Session.objects.get_session(request.user, query_server['server_name'])
-  server_url = _get_impala_server_url(session)
+  api = _get_api(request.user, cluster=cluster)
 
   if query_id:
-    LOG.debug("Attempting to get Impala query profile at server_url %s for query ID: %s" % (server_url, query_id))
+    LOG.debug("Attempting to get Impala query profile for query ID: %s" % (query_id))
     doc = Document2.objects.get(id=query_id)
     snippets = doc.data_dict.get('snippets', [])
     secret = snippets[0]['result']['handle']['secret']
-    api = get_impalad_api(user=request.user, url=server_url)
     impala_query_id = "%x:%x" % struct.unpack(b"QQ", base64.decodestring(secret))
     api.kill(impala_query_id) # There are many statistics that are not present when the query is open. Close it first.
     query_profile = api.get_query_profile_encoded(impala_query_id)
@@ -158,10 +154,28 @@ def alanize(request):
     response['status'] = 0
   return JsonResponse(response)
 
+def alanize_metrics(request):
+  response = {'status': -1}
+  cluster = json.loads(request.POST.get('cluster', '{}'))
+  query_id = json.loads(request.POST.get('query_id'))
+
+  api = _get_api(request.user, cluster=cluster)
+
+  if query_id:
+    LOG.debug("Attempting to get Impala query profile for query ID: %s" % (query_id))
+    query_profile = api.get_query_profile_encoded(query_id)
+    profile = analyzer.analyze(analyzer.parse_data(query_profile))
+    ANALYZER.pre_process(profile)
+    metrics = analyzer.metrics(profile)
+    response['data'] = metrics
+    response['status'] = 0
+  return JsonResponse(response)
+
 @require_POST
 @error_handler
 def alanize_fix(request):
   response = {'status': -1}
+  cluster = json.loads(request.POST.get('cluster', '{}'))
   fix = json.loads(request.POST.get('fix'))
   start_time = json.loads(request.POST.get('start_time'), '-1')
   if fix['id'] == 0:
@@ -171,7 +185,8 @@ def alanize_fix(request):
       statement='compute stats %(data)s' % fix,
       status='ready',
       last_executed=start_time,
-      is_task=True
+      is_task=True,
+      compute=cluster
     )
     response['details'] = { 'task': notebook.execute(request, batch=True) }
     response['status'] = 0

+ 23 - 2
apps/impala/src/impala/conf.py

@@ -178,11 +178,32 @@ AUTH_PASSWORD_SCRIPT = Config(
   default=None
 )
 
+def get_daemon_config(key):
+  from metadata.conf import MANAGER
+  from metadata.manager_client import ManagerApi
+
+  if MANAGER.API_URL.get():
+    return ManagerApi().get_impalad_config(key=key, impalad_host=SERVER_HOST.get())
+
+  return None
+
+def get_daemon_api_username():
+  """
+    Try to get daemon_api_username from Cloudera Manager API
+  """
+  return get_daemon_config('webserver_htpassword_user')
+
+def get_daemon_api_password():
+  """
+    Try to get daemon_api_password from Cloudera Manager API
+  """
+  return get_daemon_config('webserver_htpassword_password')
+
 DAEMON_API_PASSWORD = Config(
   key="daemon_api_password",
   help=_t("Password for Impala Daemon when username/password authentication is enabled for the Impala Daemon UI."),
   private=True,
-  default=None
+  dynamic_default=get_daemon_api_password
 )
 
 DAEMON_API_PASSWORD_SCRIPT = Config(
@@ -197,7 +218,7 @@ DAEMON_API_USERNAME = Config(
   key="daemon_api_username",
   help=_t("Username for Impala Daemon when username/password authentication is enabled for the Impala Daemon UI."),
   private=True,
-  default=None
+  dynamic_default=get_daemon_api_username
 )
 
 DAEMON_API_PASSWORD = Config(

+ 2 - 1
apps/impala/src/impala/dbms.py

@@ -25,6 +25,7 @@ from beeswax.server import dbms
 from beeswax.server.dbms import HiveServer2Dbms, QueryServerException, QueryServerTimeoutException,\
   get_query_server_config as beeswax_query_server_config
 
+from desktop.conf import CLUSTER_ID
 from impala import conf
 
 
@@ -53,7 +54,7 @@ def get_query_server_config(cluster_config=None):
 
 
 def _get_server_name(cluster_config):
-  return 'impala' + ('-' + cluster_config.get('name') if cluster_config else '')
+  return 'impala' + ('-' + cluster_config.get('name') if cluster_config and cluster_config.get('name') != CLUSTER_ID.get() else '')
 
 
 class ImpalaDbms(HiveServer2Dbms):

+ 1 - 0
apps/impala/src/impala/urls.py

@@ -27,6 +27,7 @@ urlpatterns = [
   url(r'^api/query/(?P<query_history_id>\d+)/runtime_profile', impala_api.get_runtime_profile, name='get_runtime_profile'),
   url(r'^api/query/alanize$', impala_api.alanize, name='alanize'),
   url(r'^api/query/alanize/fix$', impala_api.alanize_fix, name='alanize_fix'),
+  url(r'^api/query/alanize/metrics', impala_api.alanize_metrics, name='alanize_metrics'),
 ]
 
 urlpatterns += beeswax_urls

+ 13 - 7
apps/jobbrowser/src/jobbrowser/api2.py

@@ -18,9 +18,9 @@
 import json
 import logging
 
-from django.utils.translation import ugettext as _
 from desktop.lib.i18n import smart_unicode
 from desktop.lib.django_util import JsonResponse
+from django.utils.translation import ugettext as _
 from desktop.views import serve_403_error
 
 from jobbrowser.apis.base_api import get_api
@@ -51,10 +51,11 @@ def api_error_handler(func):
 def jobs(request, interface=None):
   response = {'status': -1}
 
+  cluster = json.loads(request.POST.get('cluster', '{}'))
   interface = json.loads(request.POST.get('interface'))
   filters = dict([(key, value) for _filter in json.loads(request.POST.get('filters', '[]')) for key, value in _filter.items() if value])
 
-  jobs = get_api(request.user, interface).apps(filters)
+  jobs = get_api(request.user, interface, cluster=cluster).apps(filters)
 
   response['apps'] = jobs['apps']
   response['total'] = jobs.get('total')
@@ -67,10 +68,12 @@ def jobs(request, interface=None):
 def job(request, interface=None):
   response = {'status': -1}
 
+  cluster = json.loads(request.POST.get('cluster', '{}'))
   interface = json.loads(request.POST.get('interface'))
   app_id = json.loads(request.POST.get('app_id'))
 
-  response_app = get_api(request.user, interface).app(app_id)
+  response_app = get_api(request.user, interface, cluster=cluster).app(app_id)
+
   if response_app.get('status') == -1 and response_app.get('message'):
     response.update(response_app)
   else:
@@ -84,6 +87,7 @@ def job(request, interface=None):
 def action(request, interface=None, action=None):
   response = {'status': -1, 'message': ''}
 
+  cluster = json.loads(request.POST.get('cluster', '{}'))
   interface = json.loads(request.POST.get('interface'))
   app_ids = json.loads(request.POST.get('app_ids'))
   operation = json.loads(request.POST.get('operation'))
@@ -92,7 +96,7 @@ def action(request, interface=None, action=None):
     return serve_403_error(request)
 
   response['operation'] = operation
-  response.update(get_api(request.user, interface).action(app_ids, operation))
+  response.update(get_api(request.user, interface, cluster=cluster).action(app_ids, operation))
 
   return JsonResponse(response)
 
@@ -101,12 +105,13 @@ def action(request, interface=None, action=None):
 def logs(request):
   response = {'status': -1}
 
+  cluster = json.loads(request.POST.get('cluster', '{}'))
   interface = json.loads(request.POST.get('interface'))
   app_id = json.loads(request.POST.get('app_id'))
   app_type = json.loads(request.POST.get('type'))
   log_name = json.loads(request.POST.get('name'))
 
-  response['logs'] = get_api(request.user, interface).logs(app_id, app_type, log_name, json.loads(request.GET.get('is_embeddable', 'false').lower()))
+  response['logs'] = get_api(request.user, interface, cluster=cluster).logs(app_id, app_type, log_name, json.loads(request.GET.get('is_embeddable', 'false').lower()))
   response['status'] = 0
 
   return JsonResponse(response)
@@ -116,16 +121,17 @@ def logs(request):
 def profile(request):
   response = {'status': -1}
 
+  cluster = json.loads(request.POST.get('cluster', '{}'))
   interface = json.loads(request.POST.get('interface'))
   app_id = json.loads(request.POST.get('app_id'))
   app_type = json.loads(request.POST.get('app_type'))
   app_property = json.loads(request.POST.get('app_property'))
   app_filters = dict([(key, value) for _filter in json.loads(request.POST.get('app_filters', '[]')) for key, value in _filter.items() if value])
 
-  api = get_api(request.user, interface)
+  api = get_api(request.user, interface, cluster=cluster)
   api._set_request(request) # For YARN
 
   response[app_property] = api.profile(app_id, app_type, app_property, app_filters)
   response['status'] = 0
 
-  return JsonResponse(response)
+  return JsonResponse(response)

+ 2 - 2
apps/jobbrowser/src/jobbrowser/apis/base_api.py

@@ -28,7 +28,7 @@ from desktop.lib.exceptions_renderable import PopupException
 LOG = logging.getLogger(__name__)
 
 
-def get_api(user, interface):
+def get_api(user, interface, cluster=None):
   from jobbrowser.apis.bundle_api import BundleApi
   from jobbrowser.apis.data_eng_api import DataEngClusterApi, DataEngJobApi
   from jobbrowser.apis.clusters import ClusterApi
@@ -42,7 +42,7 @@ def get_api(user, interface):
   if interface == 'jobs':
     return JobApi(user)
   elif interface == 'queries':
-    return QueryApi(user)
+    return QueryApi(user, cluster=cluster)
   elif interface == 'workflows':
     return WorkflowApi(user)
   elif interface == 'schedules':

+ 13 - 6
apps/jobbrowser/src/jobbrowser/apis/query_api.py

@@ -31,21 +31,27 @@ LOG = logging.getLogger(__name__)
 try:
   from beeswax.models import Session
   from impala.server import get_api as get_impalad_api, _get_impala_server_url
+  from impala.dbms import _get_server_name
 except Exception, e:
   LOG.exception('Some application are not enabled: %s' % e)
 
-def _get_api(user):
-  session = Session.objects.get_session(user, application='impala')
-  server_url = _get_impala_server_url(session)
+
+def _get_api(user, cluster=None):
+  if cluster and cluster.get('type') == 'altus-dw':
+    server_url = 'http://impala-coordinator-%(name)s:25000' % cluster
+  else:
+    session = Session.objects.get_session(user, application=_get_server_name(cluster))
+    server_url = _get_impala_server_url(session)
   return get_impalad_api(user=user, url=server_url)
 
+
 class QueryApi(Api):
 
-  def __init__(self, user, impala_api=None):
+  def __init__(self, user, impala_api=None, cluster=None):
     if impala_api:
       self.api = impala_api
     else:
-      self.api = _get_api(user)
+      self.api = _get_api(user, cluster)
 
   def apps(self, filters):
     kwargs = {}
@@ -124,7 +130,8 @@ class QueryApi(Api):
         'profile': '',
         'plan': '',
         'backends': '',
-        'finstances': ''
+        'finstances': '',
+        'metrics': ''
       }
     })
 

+ 14 - 11
apps/jobbrowser/src/jobbrowser/conf.py

@@ -15,22 +15,25 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from desktop.lib.conf import Config, coerce_bool
 from django.utils.translation import ugettext_lazy as _
 
+from desktop.lib.conf import Config, coerce_bool
+
 
 SHARE_JOBS = Config(
   key='share_jobs',
   default=True,
   type=coerce_bool,
   help=_('Share submitted jobs information with all users. If set to false, '
-       'submitted jobs are visible only to the owner and administrators.'))
+       'submitted jobs are visible only to the owner and administrators.')
+)
 
 DISABLE_KILLING_JOBS = Config(
   key='disable_killing_jobs',
   default=False,
   type=coerce_bool,
-  help=_('Disable the job kill button for all users in the job browser.'))
+  help=_('Disable the job kill button for all users in the job browser.')
+)
 
 LOG_OFFSET = Config(
   key='log_offset',
@@ -40,10 +43,10 @@ LOG_OFFSET = Config(
 )
 
 ENABLE_V2 = Config(
-    key="enable_v2",
-    help=_("Show the version 2 of app which unifies all the past browsers into one."),
-    type=coerce_bool,
-    default=True
+  key="enable_v2",
+  help=_("Show the version 2 of app which unifies all the past browsers into one."),
+  type=coerce_bool,
+  default=True
 )
 
 MAX_JOB_FETCH = Config(
@@ -54,8 +57,8 @@ MAX_JOB_FETCH = Config(
 )
 
 ENABLE_QUERY_BROWSER = Config(
-    key="enable_query_browser",
-    help=_("Show the query section for listing and showing more troubleshooting information."),
-    type=coerce_bool,
-    default=True
+  key="enable_query_browser",
+  help=_("Show the query section for listing and showing more troubleshooting information."),
+  type=coerce_bool,
+  default=True
 )

File diff suppressed because it is too large
+ 0 - 0
apps/jobbrowser/src/jobbrowser/static/jobbrowser/css/jobbrowser-embeddable.css


+ 147 - 37
apps/jobbrowser/src/jobbrowser/static/jobbrowser/js/impala_dagre.js

@@ -12,19 +12,27 @@ function impalaDagre(id) {
   var g = new dagreD3.graphlib.Graph().setGraph({rankDir: "BT"});
   var svg = d3.select("#"+id + " svg");
   var inner = svg.select("g");
+  var states_by_name = { };
+  var colors = HueColors.cuiD3Scale();
   var _impalaDagree = {
+    _metrics: {},
     init: function (initialScale) {
-      _impalaDagree.scale = initialScale;
+      clearSelection();
       zoom.translate([((svg.attr("width") || $("#"+id).width()) - g.graph().width * initialScale) / 2, 20])
       .scale(initialScale)
       .event(svg);
     },
+    metrics: function(data) {
+      _impalaDagree._metrics = data;
+      renderGraph()
+    },
     update: function(plan) {
-      renderGraph(plan);
+      _impalaDagree._plan = plan;
+      renderGraph();
       _impalaDagree._width = $(svg[0]).width();
     },
     height: function(value) {
-      var scale = _impalaDagree.scale || 1;
+      var scale = zoom.scale() || 1;
       var height = value || 600;
       _impalaDagree._height = height;
       svg.attr('height', height);
@@ -45,51 +53,46 @@ function impalaDagre(id) {
     },
     select: function(id) {
       select(id);
-    },
+    }
   };
   createActions();
 
   function createActions () {
+    svg.on('click', function () {
+      hideDetail();
+      clearSelection();
+    });
     d3.select("#"+id)
       .style('position', 'relative')
     .append('div')
-      .style('position', 'absolute')
-      .style('right', '5px')
-      .style('bottom', '5px')
       .classed('buttons', true)
-    .selectAll('button').data([{ type: 'reset', svg: 'hi-crop-free', divider: true }, { type: 'plus', icon: 'fa-plus', divider: true }, { type: 'minus', icon: 'fa-minus' }])
+    .selectAll('button').data([{ type: 'reset', svg: 'hi-crop-free', divider: true }, { type: 'plus', font: 'fa-plus', divider: true }, { type: 'minus', font: 'fa-minus' }])
     .enter()
     .append(function (data) {
       var text = "";
-      if (data.svg) {
-        text += "<div><svg class='hi'><use xlink:href='#"+ data.svg +"'></use></svg>";
-        if (data.divider) {
-          text += "<div class='divider'></div>";
-        }
-        text += "</div>";
-        button = $()[0];
-      } else if (data.icon) {
-        text += "<div><div class='fa fa-fw valign-middle " + data.icon + "'></div>";
-        if (data.divider) {
-          text += "<div class='divider'></div></div>";
-        }
-        text += "</div>";
+      text += '<div>';
+      text += getIcon(data);
+      if (data.divider) {
+        text += '<div class="divider"></div>';
       }
+      text += '</div>';
       var button = $(text)[0];
       $(button).on('click', function () {
         _impalaDagree.action(data.type);
       });
       return button;
     });
+    d3.select("#"+id)
+    .append('div')
+      .classed('details', true);
   }
 
   // Set up zoom support
   var zoom = d3.behavior.zoom().on("zoom", function() {
     var e = d3.event,
-        scale = Math.min(Math.max(e.scale, Math.min(_impalaDagree._width / g.graph().width, _impalaDagree._height / g.graph().height)), 2),
+        scale = Math.min(Math.max(e.scale, Math.min(Math.min(_impalaDagree._width / g.graph().width, _impalaDagree._height / g.graph().height), 1)), 2),
         tx = Math.min(40, Math.max(e.translate[0], _impalaDagree._width - 40 - g.graph().width * scale)),
         ty = Math.min(40, Math.max(e.translate[1], _impalaDagree._height - 40 - g.graph().height * scale));
-    _impalaDagree.scale = scale;
     zoom.translate([tx, ty]);
     zoom.scale(scale);
     inner.attr("transform", "translate(" + [tx, ty] + ")" +
@@ -123,7 +126,7 @@ function impalaDagre(id) {
                   "is_broadcast": node["is_broadcast"],
                   "max_time_val": node["max_time_val"]});
     if (parent) {
-      var label_val = "" + node["output_card"].toLocaleString();
+      var label_val = "" + ko.bindingHandlers.simplesize.humanSize(parseInt(node["output_card"], 10));
       edges.push({ start: node["label"], end: parent,
                    style: { label: label_val }});
     }
@@ -132,7 +135,7 @@ function impalaDagre(id) {
     if (node["data_stream_target"]) {
       edges.push({ "start": node["label"],
                    "end": node["data_stream_target"],
-                   "style": { label: "" + node["output_card"].toLocaleString(),
+                   "style": { label: ko.bindingHandlers.simplesize.humanSize(parseInt(node["output_card"], 10)),
                               style: "stroke-dasharray: 5, 5;"}});
     }
     max_node_time = Math.max(node["max_time_val"], max_node_time)
@@ -150,8 +153,13 @@ function impalaDagre(id) {
     if (!key) {
       return;
     }
-    $("g.node").attr('class', 'node') // addClass doesn't work in svg on our version of jQuery
+    clearSelection();
     $("g.node:contains('" + key + "')").attr('class', 'node active');
+    showDetail(node);
+  }
+
+  function clearSelection() {
+    $("g.node").attr('class', 'node'); // addClass doesn't work in svg on our version of jQuery
   }
 
   function getKey(node) {
@@ -185,7 +193,114 @@ function impalaDagre(id) {
             .scale(scale).event);
   }
 
-  function renderGraph(plan) {
+  function getIcon(icon) {
+    var html = '';
+    if (icon && icon.svg) {
+      html += '<svg class="hi"><use xlink:href="#'+ icon.svg +'"></use></svg>'
+    } else if (icon && icon.font) {
+      html += "<div class='fa fa-fw valign-middle " + icon.font + "'></div>";
+    }
+    return html;
+  }
+
+  function getTimelineData(key) {
+    if (!_impalaDagree._metrics) {
+      return [];
+    }
+    var id = parseInt(key.split(':')[0], 10);
+    if (!_impalaDagree._metrics[id]) {
+      return [];
+    }
+    var times = _impalaDagree._metrics[id];
+    var timesKeys = Object.keys(times);
+    var timesKey;
+    for (var i = 0; i < timesKeys.length; i++) {
+      if (times[timesKeys[i]]['timeline'] && times[timesKeys[i]]['timeline']['Node Lifecycle Event Timeline']) {
+        timesKey = timesKeys[i];
+        break;
+      }
+    }
+    if (!timesKey) {
+      return [];
+    }
+    var time = times[timesKey]['timeline']['Node Lifecycle Event Timeline'];
+    return time.map(function (time, index, array) {
+      var startTime = index > 0 && array[index - 1].value || 0;
+      return { starting_time: startTime, ending_time : time.value, duration: time.value - startTime, color: colors[index % colors.length], name: time.name, unit: time.unit };
+    });
+  }
+
+  function renderTimeline(key) {
+    var datum = getTimelineData(key);
+    if (!datum.length) {
+      return '';
+    }
+    var end = _impalaDagree._metrics && _impalaDagree._metrics['max'] || 10;
+    var divider = end > 33554428 ? 1000000 : 1; // values are in NS, scaling to MS as max pixel value is 33554428px ~9h in MS
+    var html = '<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 ' + (end / divider) + ' 10" class="timeline" preserveAspectRatio="none">';
+    html += datum.map(function(time, index) {
+      return '<rect x="' + (time.starting_time / divider) + '" width="' + (time.duration / divider)  + '" height="10" style="fill:' + time.color  +'"></rect>';
+    }).join('');
+    html += '</svg>';
+    return html;
+  }
+
+  function renderTimelineLegend(key) {
+    var datum = getTimelineData(key);
+    if (!datum.length) {
+      return '';
+    }
+    return datum.map(function(time, index) {
+      return '<li><div class="legend-icon" style="background-color:' + time.color +' "></div><div class="metric-name">' + time.name + '</div> <div class="metric-value">' + ko.bindingHandlers.numberFormat.human(time.duration, time.unit) + '</div></li>';
+    }).join('');
+  }
+
+  function showDetail(id) {
+    var data;
+    if (_impalaDagree._metrics[id] && _impalaDagree._metrics[id]['averaged']['metrics']) {
+      data = _impalaDagree._metrics[id]['averaged']['metrics'];
+    } else if (_impalaDagree._metrics[id] && _impalaDagree._metrics[id]['metrics']) {
+      data = _impalaDagree._metrics[id]['metrics'][Object.keys(_impalaDagree._metrics[id]['metrics'])[0]];
+    }
+    d3.select('.query-plan').classed('open', true);
+    var details = d3.select('.query-plan .details');
+    var key = getKey(id);
+    details.html('<header class="metric-title">' + getIcon(states_by_name[key].icon) + '<h3>' + states_by_name[key].label+ '</h3></div>')
+    var detailsContent = details.append('div').classed('details-content', true);
+
+    var timeline = renderTimeline(key, '');
+    if (timeline) {
+      var timelineSection = detailsContent.append('div').classed('details-section', true);
+      var timelineTitle = timelineSection.append('header');
+      timelineTitle.append('svg').classed('hi', true).append('use').attr('xlink:href', '#hi-access-time');
+      timelineTitle.append('h4').text(window.HUE_I18n.profile.timeline);
+      timelineSection.node().appendChild($.parseXML(renderTimeline(key, '')).children[0]);
+
+      timelineSection.append('ol').classed('', true).html(renderTimelineLegend(key));
+      detailsContent.append('div').classed('divider', true);
+    }
+
+    var metricsSection = detailsContent.append('div').classed('details-section', true);
+
+    var metricsTitle = metricsSection.append('header');
+    metricsTitle.append('svg').classed('hi', true).append('use').attr('xlink:href', '#hi-bar-chart');
+    metricsTitle.append('h4').text(window.HUE_I18n.profile.metrics);
+
+    var metricsContent = metricsSection.append('ul').classed('metrics', true);
+
+    var metrics = metricsContent.selectAll('li')
+    .data(Object.keys(data).sort().map(function (key) { return data[key]; }));
+    metrics.exit().remove();
+    metrics.enter().append('li');
+    metrics.html(function (datum) { return '<div class="metric-name">' + datum.name + '</div> <div class="metric-value">' + ko.bindingHandlers.numberFormat.human(datum.value, datum.unit) + '</div>'; });
+  }
+
+  function hideDetail(id) {
+    d3.select('.query-plan').classed('open', false);
+  }
+
+  function renderGraph() {
+    var plan = _impalaDagree._plan;
     if (!plan || !plan.plan_nodes || !plan.plan_nodes.length) return;
     var states = [];
     var edges = [];
@@ -200,21 +315,16 @@ function impalaDagre(id) {
     });
 
     // Keep a map of names to states for use when processing edges.
-    var states_by_name = { };
     states.forEach(function(state) {
       // Build the label for the node from the name and the detail
-      var html = "";
-      if (state.icon && state.icon.svg) {
-        html += '<svg class="hi"><use xlink:href="#'+ state.icon.svg +'"></use></svg>'
-        //html += "<img src=\"" + icon.svg + "\"></img>";
-      } else if (state.icon && state.icon.font){
-        html += "<span class='fa fa-fw valign-middle " + state.icon.font + "'></span>";
-      }
+      var html = "<div onclick=\"event.stopPropagation(); huePubSub.publish('impala.node.select', " + parseInt(state.name.split(':')[0], 10) + ");\">"; // TODO: Remove Hue dependency
+      html += getIcon(state.icon)
       html += "<span class='name'>" + state.label + "</span><br/>";
       html += "<span class='metric'>" + state.max_time + "</span>";
       html += "<span class='detail'>" + state.detail + "</span><br/>";
-      html += "<span class='metric'>" + state.max_time + "</span>"
-      html += "<span class='id'>" + state.name + "</span>";;
+      html += "<span class='id'>" + state.name + "</span>";
+      html += renderTimeline(state.name);
+      html += "</div>";
 
       var style = state.style;
 

+ 116 - 10
apps/jobbrowser/src/jobbrowser/static/jobbrowser/less/jobbrowser-embeddable.less

@@ -91,6 +91,7 @@
 
   .query-plan {
     border: 1px solid @cui-gray-300;
+    overflow: hidden;
     .label,
     .badge {
       color: @cui-gray-800;
@@ -110,21 +111,29 @@
       overflow: hidden;
       text-overflow: ellipsis;
       width: calc(~"100% - 32px");
+      max-width: 185px;
       display: inline-block;
+      height: 14px;
     }
-    span.fa {
+    foreignObject {
+      .fa {
+        padding-top: 3px;
+        padding-right: 2px;
+        float: left;
+      }
+      .hi {
+        padding-right: 5px;
+        float: left;
+      }
+    }
+    .fa {
       color: @cui-gray-600;
-      float: left;
       font-size: 21px;
-      padding-top: 3px;
-      padding-right: 2px;
     }
-    svg .hi {
+    .hi {
       color: @cui-gray-600 !important;
-      float: left;
       width: 2.2em !important;
       height: 2.2em !important;
-      padding-right: 5px;
     }
     .buttons {
       .hi {
@@ -157,9 +166,11 @@
       }
     }
     .buttons {
+      position: absolute;
       background-color: white;
-      box-shadow: 0px 0px 2px 0px;
-      border: 1px solid @cui-gray-300;
+      box-shadow: 0px 0px 5px 0px;
+      right: 5px;
+      bottom: 5px;
       color: @cui-gray-600;
       div {
         line-height: 32px !important;
@@ -168,7 +179,7 @@
         height: 32px;
       }
     }
-    .button div:hover {
+    .buttons div:hover {
       color: @hue-primary-color-dark
     }
     .node rect {
@@ -189,6 +200,101 @@
     foreignObject > div {
       position: relative;
     }
+    .timeline {
+      width: 175px;
+      height: 10px;
+      border: 1px solid @cui-gray-300;
+      border-radius: 4px;
+    }
+    .details {
+      background-color: white;
+      box-shadow: 0px 0px 10px 0px;
+      color: @cui-gray-600;
+      position: absolute;
+      display: none;
+      top: 0px;
+      right: 0px;
+      width: 200px;
+      height: 100%;
+      flex-direction: column;
+      .divider {
+        background-color: @cui-gray-300;
+        width: calc(~"100% - 10px");
+        margin-left: 5px;
+        padding: 0px;
+        height: 1px;
+      }
+      h3 {
+        display: inline-block;
+        margin: 0px;
+      }
+      h4 {
+        margin: 0px;
+        display: inline-block;
+      }
+      ol {
+        list-style-type: none;
+        margin: 0px;
+      }
+      ul {
+        margin: 0px;
+      }
+      .details-content {
+        overflow: scroll;
+      }
+      .details-section {
+        padding: 4px;
+      }
+      .metric-title {
+        background-color: @hue-primary-color-light;
+        padding: 4px;
+        .fa {
+          padding-top: 0px;
+          line-height: 40px;
+          padding-right: 2px;
+        }
+        .hi {
+          padding-top: 2px;
+          padding-right: 5px;
+        }
+      }
+      .metrics {
+        overflow-y: scroll;
+        max-height: calc(~"100% - 40px");
+      }
+      .metric-name {
+        color: @cui-gray-800;
+        width: 107px;
+        overflow: hidden;
+        text-align: right;
+        display: inline-block;
+        white-space: nowrap;
+        vertical-align: middle;
+        font-weight: bold;
+        padding-left: 2px;
+      }
+      .metric-value {
+        color: @cui-gray-800;
+        overflow: hidden;
+        width: 60px;
+        vertical-align: middle;
+        display: inline-block;
+        white-space: nowrap;
+      }
+      .legend-icon {
+        width: 8px;
+        height: 8px;
+        border-radius: 5px;
+        border: 1px solid @cui-gray-300;
+        display: inline-block;
+      }
+    }
+  }
+  .query-plan.open .details {
+    display: flex;
+  }
+  .query-plan.open .buttons {
+    right: 208px;
   }
 
   div[data-jobType="queries"] pre {

+ 78 - 27
apps/jobbrowser/src/jobbrowser/templates/job_browser.mako

@@ -14,12 +14,13 @@
 ## See the License for the specific language governing permissions and
 ## limitations under the License.
 <%
-  from django.utils.translation import ugettext as _
+from django.utils.translation import ugettext as _
 
-  from desktop import conf
-  from desktop.views import commonheader, commonfooter, _ko
+from desktop.conf import CUSTOM, IS_K8S_ONLY, is_hue4
+from desktop.views import commonheader, commonfooter, _ko
+from metadata.conf import PROMETHEUS
 
-  from jobbrowser.conf import DISABLE_KILLING_JOBS, MAX_JOB_FETCH, ENABLE_QUERY_BROWSER
+from jobbrowser.conf import DISABLE_KILLING_JOBS, MAX_JOB_FETCH, ENABLE_QUERY_BROWSER
 %>
 
 <%
@@ -39,7 +40,7 @@ ${ commonheader("Job Browser", "jobbrowser", user, request) | n,unicode }
 <link rel="stylesheet" href="${ static('notebook/css/notebook.css') }">
 <link rel="stylesheet" href="${ static('notebook/css/notebook-layout.css') }">
 <style type="text/css">
-% if conf.CUSTOM.BANNER_TOP_HTML.get():
+% if CUSTOM.BANNER_TOP_HTML.get():
   .show-assist {
     top: 110px!important;
   }
@@ -85,6 +86,19 @@ ${ commonheader("Job Browser", "jobbrowser", user, request) | n,unicode }
 
 % if is_mini:
   <div class="mini-jb-context-selector">
+    <!-- ko component: {
+      name: 'hue-context-selector',
+      params: {
+        sourceType: 'impala',
+        compute: compute,
+        ##namespace: namespace,
+        ##availableDatabases: availableDatabases,
+        ##database: database,
+        hideDatabases: true
+      }
+    } --><!-- /ko -->
+
+    % if not IS_K8S_ONLY.get():
     <!-- ko component: {
       name: 'hue-context-selector',
       params: {
@@ -94,6 +108,7 @@ ${ commonheader("Job Browser", "jobbrowser", user, request) | n,unicode }
         hideLabels: true
       }
     } --><!-- /ko -->
+    % endif
   </div>
   <ul class="nav nav-pills">
     <!-- ko foreach: availableInterfaces -->
@@ -109,6 +124,12 @@ ${ commonheader("Job Browser", "jobbrowser", user, request) | n,unicode }
         <div class="nav-collapse">
           <ul class="nav">
             <li class="app-header">
+              % if IS_K8S_ONLY.get():
+                <a data-bind="click: function() { selectInterface('dataware2-clusters'); }">
+                  <i class="altus-icon altus-adb-cluster"></i>
+                  ${ _('Warehouses') }
+                </a>
+              % else:
               <!-- ko ifnot: $root.cluster() && $root.cluster()['type'].indexOf("altus-dw") !== -1 -->
               <a href="/${app_name}">
                 <img src="${ static('jobbrowser/art/icon_jobbrowser_48.png') }" class="app-icon" alt="${ _('Job browser icon') }"/>
@@ -132,6 +153,7 @@ ${ commonheader("Job Browser", "jobbrowser", user, request) | n,unicode }
                   > gke_gcp-eng-dsdw_us-west2-b_impala-demo
                 </span>
               <!-- /ko -->
+              % endif
             </li>
             <!-- ko foreach: availableInterfaces -->
               <li data-bind="css: {'active': $parent.interface() === interface}, visible: condition()">
@@ -898,7 +920,7 @@ ${ commonheader("Job Browser", "jobbrowser", user, request) | n,unicode }
             <tr>
               <th>${_('Assigned Container Id')}</th>
               <th>${_('Node Id')}</th>
-              <th>${_('appAttemptId')}</th>
+              <th>${_('Application Attempt Id')}</th>
               <th>${_('Start Time')}</th>
               <th>${_('Finish Time')}</th>
               <th>${_('Node Http Address')}</th>
@@ -910,7 +932,7 @@ ${ commonheader("Job Browser", "jobbrowser", user, request) | n,unicode }
               <tr class="pointer" data-bind="click: function() { $root.job().id(id); $root.job().fetchJob(); }">
                 <td data-bind="text: containerId"></td>
                 <td data-bind="text: nodeId"></td>
-                <td data-bind="text: appAttemptId"></td>
+                <td data-bind="text: id"></td>
                 <td data-bind="moment: {data: startTime, format: 'LLL'}"></td>
                 <td data-bind="moment: {data: finishedTime, format: 'LLL'}"></td>
                 <td data-bind="text: nodeHttpAddress"></td>
@@ -1315,16 +1337,20 @@ ${ commonheader("Job Browser", "jobbrowser", user, request) | n,unicode }
 
       <div class="acl-panel-content">
         <ul class="nav nav-tabs">
-          <li class="active"><a href="#servicesLoad" data-toggle="tab">Load</a></li>
-          <li><a href="#servicesPrivileges" data-toggle="tab">Privileges</a></li>
-          <li><a href="#servicesTroubleshooting" data-toggle="tab">Troubleshooting</a></li>
+          <li class="active"><a href="#servicesLoad" data-toggle="tab">${ _("Load") }</a></li>
+          <li><a href="#servicesPrivileges" data-toggle="tab">${ _("Privileges") }</a></li>
+          <li><a href="#servicesTroubleshooting" data-toggle="tab">${ _("Troubleshooting") }</a></li>
         </ul>
 
         <div class="tab-content">
           <div class="tab-pane active" id="servicesLoad">
             <div class="wxm-poc" style="clear: both;">
               <div style="float:left; margin-right: 10px; margin-bottom: 10px;">
+                % if PROMETHEUS.API_URL.get():
                 <!-- ko component: { name: 'performance-graph', params: { clusterName: name(), type: 'cpu' } } --><!-- /ko -->
+                % else:
+                  ${ _("Metrics are not setup") }
+                % endif
               </div>
             </div>
           </div>
@@ -1470,7 +1496,7 @@ ${ commonheader("Job Browser", "jobbrowser", user, request) | n,unicode }
     <div data-bind="css:{'span10': !$root.isMini(), 'span12 no-margin': $root.isMini() }">
       <ul class="nav nav-pills margin-top-20">
         <li>
-          <a href="#queries-page-plan${ SUFFIX }" data-bind="click: function(){ $('a[href=\'#queries-page-plan${ SUFFIX }\']').tab('show'); }, event: {'shown': function () { if (!properties.plan || !properties.plan().plan_json) { fetchProfile('plan'); } } }">
+          <a href="#queries-page-plan${ SUFFIX }" data-bind="click: function(){ $('a[href=\'#queries-page-plan${ SUFFIX }\']').tab('show'); }, event: {'shown': function () { if (!properties.plan || !properties.plan().plan_json) { fetchProfile('plan'); fetchMetrics(); } } }">
             ${ _('Plan') }</a>
         </li>
         <li>
@@ -1508,7 +1534,7 @@ ${ commonheader("Job Browser", "jobbrowser", user, request) | n,unicode }
       <div class="tab-content">
         <div class="tab-pane" id="queries-page-plan${ SUFFIX }" data-profile="plan">
           <div data-bind="visible:properties.plan && properties.plan().plan_json && properties.plan().plan_json.plan_nodes.length">
-            <div class="query-plan" id="queries-page-plan-graph${ SUFFIX }" data-bind="impalaDagre: {value: properties.plan && properties.plan().plan_json, height:$root.isMini() ? 535 : 600 }">
+            <div class="query-plan" id="queries-page-plan-graph${ SUFFIX }" data-bind="impalaDagre: {value: properties.plan && properties.plan().plan_json, metrics: properties.metrics && properties.metrics(), height:$root.isMini() ? 535 : 600 }">
               <svg style="width:100%;height:100%;" id="queries-page-plan-svg${ SUFFIX }">
                 <defs>
                   <filter id="dropshadow" height="130%">
@@ -2488,6 +2514,7 @@ ${ commonheader("Job Browser", "jobbrowser", user, request) | n,unicode }
         }
 
         return $.post("/jobbrowser/api/job/" + vm.interface(), {
+          cluster: ko.mapping.toJSON(vm.compute),
           app_id: ko.mapping.toJSON(self.id),
           interface: ko.mapping.toJSON(vm.interface)
         }, function (data) {
@@ -2643,6 +2670,7 @@ ${ commonheader("Job Browser", "jobbrowser", user, request) | n,unicode }
       self.fetchLogs = function (name) {
         name = name || 'default';
         $.post("/jobbrowser/api/job/logs?is_embeddable=${ str(is_embeddable).lower() }", {
+          cluster: ko.mapping.toJSON(vm.compute),
           app_id: ko.mapping.toJSON(self.id),
           interface: ko.mapping.toJSON(vm.interface),
           type: ko.mapping.toJSON(self.type),
@@ -2663,6 +2691,7 @@ ${ commonheader("Job Browser", "jobbrowser", user, request) | n,unicode }
 
       self.fetchProfile = function (name, callback) {
         $.post("/jobbrowser/api/job/profile", {
+          cluster: ko.mapping.toJSON(vm.compute),
           app_id: ko.mapping.toJSON(self.id),
           interface: ko.mapping.toJSON(vm.interface),
           app_type: ko.mapping.toJSON(self.type),
@@ -2680,8 +2709,18 @@ ${ commonheader("Job Browser", "jobbrowser", user, request) | n,unicode }
         });
       };
 
+      self.fetchMetrics = function (name, callback) {
+        ApiHelper.getInstance().fetchQueryExecutionStatistics({
+          queryId: self.id(),
+          cluster: vm.compute()
+        }).done(function(data) {
+          self.properties['metrics'](data);
+        });
+      };
+
       self.fetchStatus = function () {
         $.post("/jobbrowser/api/job", {
+          cluster: ko.mapping.toJSON(vm.compute),
           app_id: ko.mapping.toJSON(self.id),
           interface: ko.mapping.toJSON(self.mainType)
         }, function (data) {
@@ -2931,11 +2970,11 @@ ${ commonheader("Job Browser", "jobbrowser", user, request) | n,unicode }
       });
       self.paginationPage = ko.observable(1);
       self.paginationOffset = ko.observable(1); // Starting index
-      %if conf.is_hue4():
-      self.paginationResultPage = ko.observable(100);
-      %else:
-      self.paginationResultPage = ko.observable(50);
-      %endif
+      % if is_hue4():
+        self.paginationResultPage = ko.observable(100);
+      % else:
+        self.paginationResultPage = ko.observable(50);
+      % endif
       self.pagination = ko.computed(function() {
         return {
             'page': self.paginationPage(),
@@ -2982,6 +3021,7 @@ ${ commonheader("Job Browser", "jobbrowser", user, request) | n,unicode }
 
       self._fetchJobs = function (callback) {
         return $.post("/jobbrowser/api/jobs/" + vm.interface(), {
+          cluster: ko.mapping.toJSON(vm.compute),
           interface: ko.mapping.toJSON(vm.interface),
           filters: ko.mapping.toJSON(self.filters),
         }, function (data) {
@@ -3093,6 +3133,8 @@ ${ commonheader("Job Browser", "jobbrowser", user, request) | n,unicode }
           $.post("/metadata/api/analytic_db/create_cluster/", {
             "is_k8": vm.interface().indexOf('dataware2-clusters') != -1,
             "cluster_name": self.createClusterName(),
+            "cluster_hdfs_host": "hdfs-namenode",
+            "cluster_hdfs_port": 9820,
             "cdh_version": "CDH515",
             "public_key": "public_key",
             "instance_type": "m4.xlarge",
@@ -3184,6 +3226,10 @@ ${ commonheader("Job Browser", "jobbrowser", user, request) | n,unicode }
       self.isMini = ko.observable(false);
 
       self.cluster = ko.observable();
+      self.compute = ko.observable();
+      self.compute.subscribe(function () {
+        self.jobs.fetchJobs();
+      });
 
       self.availableInterfaces = ko.pureComputed(function () {
         var jobsInterfaceCondition = function () {
@@ -3294,22 +3340,24 @@ ${ commonheader("Job Browser", "jobbrowser", user, request) | n,unicode }
         interface = self.isValidInterface(interface);
         self.interface(interface);
         self.resetBreadcrumbs();
+
         % if not is_mini:
-        huePubSub.publish('graph.stop.refresh.view');
-        if (window.location.hash !== '#!' + interface) {
-          hueUtils.changeURL('#!' + interface);
-        }
+          huePubSub.publish('graph.stop.refresh.view');
+          if (window.location.hash !== '#!' + interface) {
+            hueUtils.changeURL('#!' + interface);
+          }
         % endif
         self.jobs.selectedJobs([]);
         self.job(null);
+
         if (interface === 'slas'){
           % if not is_mini:
-          self.loadSlaPage();
+            self.loadSlaPage();
           % endif
         }
         else if (interface === 'oozie-info') {
           % if not is_mini:
-          self.loadOozieInfoPage();
+            self.loadOozieInfoPage();
           % endif
         }
         else {
@@ -3388,9 +3436,9 @@ ${ commonheader("Job Browser", "jobbrowser", user, request) | n,unicode }
         }
         loaded = true;
         var h = window.location.hash;
-        %if not is_mini:
+        % if not is_mini:
         huePubSub.publish('graph.stop.refresh.view');
-        %endif
+        % endif
 
         h = h.indexOf('#!') === 0 ? h.substr(2) : '';
         switch (h) {
@@ -3496,9 +3544,12 @@ ${ commonheader("Job Browser", "jobbrowser", user, request) | n,unicode }
       }, 'jobbrowser');
 
       % if is_mini:
-        huePubSub.subscribe('mini.jb.navigate', function (interface) {
+        huePubSub.subscribe('mini.jb.navigate', function (options) {
+          if (options.compute) {
+            jobBrowserViewModel.compute(options.compute);
+          }
           $('#jobsPanel .nav-pills li').removeClass('active');
-          interface = jobBrowserViewModel.isValidInterface(interface);
+          interface = jobBrowserViewModel.isValidInterface(options.section);
           $('#jobsPanel .nav-pills li[data-interface="' + interface + '"]').addClass('active');
           jobBrowserViewModel.selectInterface(interface);
         });

+ 6 - 6
apps/jobbrowser/src/jobbrowser/views.py

@@ -22,27 +22,27 @@ import time
 import urllib2
 import urlparse
 
-from urllib import quote_plus
 from lxml import html
+from urllib import quote_plus
 
 from django.http import HttpResponseRedirect
 from django.utils.functional import wraps
 from django.utils.translation import ugettext as _
 from django.urls import reverse
 
-from desktop.log.access import access_log_level
-from desktop.lib.rest.http_client import RestException
-from desktop.lib.rest.resource import Resource
+from desktop.auth.backend import is_admin
 from desktop.lib.django_util import JsonResponse, render_json, render, copy_query_dict
-from desktop.lib.json_utils import JSONEncoderForHTML
 from desktop.lib.exceptions import MessageException
 from desktop.lib.exceptions_renderable import PopupException
+from desktop.lib.json_utils import JSONEncoderForHTML
+from desktop.lib.rest.http_client import RestException
+from desktop.lib.rest.resource import Resource
+from desktop.log.access import access_log_level
 from desktop.views import register_status_bar_view
 
 from hadoop import cluster
 from hadoop.yarn.clients import get_log_client
 from hadoop.yarn import resource_manager_api as resource_manager_api
-from desktop.auth.backend import is_admin
 
 
 LOG = logging.getLogger(__name__)

+ 28 - 17
apps/jobbrowser/src/jobbrowser/yarn_models.py

@@ -393,7 +393,7 @@ class OozieYarnJob(Job):
   def get_task(self, task_id):
     task = YarnTask(self)
     task.taskId = None
-    task.taskAttemptIds = [appAttempt['appAttemptId'] for appAttempt in self.job_attempts['jobAttempt']]
+    task.taskAttemptIds = [appAttempt['id'] for appAttempt in self.job_attempts['jobAttempt']]
     return task
 
   def filter_tasks(self, task_types=None, task_states=None, task_text=None):
@@ -403,8 +403,6 @@ class OozieYarnJob(Job):
   def job_attempts(self):
     if not hasattr(self, '_job_attempts'):
       attempts = self.api.appattempts(self.id)['appAttempts']['appAttempt']
-      for attempt in attempts:
-        attempt['id'] = attempt['appAttemptId']
       self._job_attempts = {
         'jobAttempt': attempts
       }
@@ -547,7 +545,7 @@ class Attempt:
     log_link = attempt['logsLink']
 
     # Generate actual task log link from logsLink url
-    if self.task.job.status in ('NEW', 'SUBMITTED', 'RUNNING') or self.type == 'Oozie Launcher':
+    if self.task.job.status in ('NEW', 'SUBMITTED', 'RUNNING'):
       logs_path = '/node/containerlogs/'
       node_url, tracking_path = log_link.split(logs_path)
       container_id, user = tracking_path.strip('/').split('/')
@@ -564,19 +562,32 @@ class Attempt:
         'user': user
       }
     else:  # Completed jobs
-      logs_path = '/jobhistory/logs/'
-      root_url, tracking_path = log_link.split(logs_path)
-      node_url, container_id, attempt_id, user = tracking_path.strip('/').split('/')
+      if '/node/containerlogs/' in log_link:
+        # Applications that use NodeManager API instead of JobHistory API, like new "Oozie Launcher",
+        # have `logsLink` URL pointed to NodeManager even for completed jobs
+        logs_path = '/node/containerlogs/'
 
-      # Replace log path tokens with actual attempt properties if available
-      if hasattr(self, 'nodeHttpAddress') and 'nodeId' in attempt:
-        node_url = '%s:%s' % (self.nodeHttpAddress.split(':')[0], attempt['nodeId'].split(':')[1])
-      container_id = self.assignedContainerId if hasattr(self, 'assignedContainerId') else container_id
-      attempt_id = self.attemptId if hasattr(self, 'attemptId') else attempt_id
+        yarn_cluster = cluster.get_cluster_conf_for_job_submission()
+        root_url = yarn_cluster.HISTORY_SERVER_API_URL.get()
+
+        tracking_path = log_link.split(logs_path)[1]
+        container_id, user = tracking_path.strip('/').split('/')
+
+        node_url = self.nodeId
+        attempt_id = self.id
+      else:
+        logs_path = '/jobhistory/logs/'
+        root_url, tracking_path = log_link.split(logs_path)
+        node_url, container_id, attempt_id, user = tracking_path.strip('/').split('/')
 
-      log_link = '%(root_url)s/%(logs_path)s/%(node)s/%(container)s/%(attempt)s/%(user)s' % {
+        # Replace log path tokens with actual attempt properties if available
+        if hasattr(self, 'nodeHttpAddress') and 'nodeId' in attempt:
+          node_url = '%s:%s' % (self.nodeHttpAddress.split(':')[0], attempt['nodeId'].split(':')[1])
+        container_id = self.assignedContainerId if hasattr(self, 'assignedContainerId') else container_id
+        attempt_id = self.attemptId if hasattr(self, 'attemptId') else attempt_id
+
+      log_link = '%(root_url)s/jobhistory/logs/%(node)s/%(container)s/%(attempt)s/%(user)s' % {
         'root_url': root_url,
-        'logs_path': logs_path.strip('/'),
         'node': node_url,
         'container': container_id,
         'attempt': attempt_id,
@@ -599,7 +610,7 @@ class Attempt:
 
       response = None
       try:
-        log_link = re.sub('job_[^/]+', self.id, log_link)
+        log_link = re.sub('job_[^/]+', str(self.id), log_link)
         root = Resource(get_log_client(log_link), urlparse.urlsplit(log_link)[2], urlencode=False)
         response = root.get(link, params=params)
         log = html.fromstring(response, parser=html.HTMLParser()).xpath('/html/body/table/tbody/tr/td[2]')[0].text_content()
@@ -627,9 +638,9 @@ class YarnOozieAttempt(Attempt):
     self._fixup()
 
   def _fixup(self):
-    setattr(self, 'diagnostics', self.diagnosticsInfo)
+    if not hasattr(self, 'diagnostics'):
+      self.diagnostics = ''
     setattr(self, 'type', 'Oozie Launcher')
-    setattr(self, 'id', self.appAttemptId)
 
 class Container:
 

+ 2 - 2
apps/oozie/src/oozie/models2.py

@@ -3515,12 +3515,12 @@ class Coordinator(Job):
     if self.data['properties']['document']:
       document = Document2.objects.get_by_uuid(user=self.document.owner, uuid=self.data['properties']['document'])
       wf_doc = WorkflowBuilder().create_workflow(document=document, user=self.document.owner, managed=True)
-      wf = Workflow(data=wf_doc.data)
+      wf = Workflow(data=wf_doc.data,user=self.document.owner)
       wf_doc.delete()
       return wf
     else:
       wf_doc = Document2.objects.get_by_uuid(user=self.document.owner, uuid=self.data['properties']['workflow'])
-      return Workflow(document=wf_doc)
+      return Workflow(document=wf_doc,user=self.document.owner)
 
   def get_absolute_url(self):
     return reverse('oozie:edit_coordinator') + '?coordinator=%s' % self.id

+ 1 - 1
apps/oozie/src/oozie/views/dashboard.py

@@ -1194,7 +1194,7 @@ def check_job_edition_permission(oozie_job, user):
 
 
 def has_job_edition_permission(oozie_job, user):
-  return is_admin(user) or oozie_job.user == user.username or (oozie_job.group and user.groups.filter(name=oozie_job.group).exists()) or (oozie_job.acl and user.username in oozie_job.acl.split(','))
+  return is_admin(user) or oozie_job.user == user.username or (oozie_job.group and user.groups.filter(name__in=oozie_job.group.split(",")).exists()) or (oozie_job.acl and user.username in oozie_job.acl.split(','))
 
 
 def has_dashboard_jobs_access(user):

+ 0 - 19
desktop/core/ext-py/pam/PKG-INFO

@@ -1,19 +0,0 @@
-Metadata-Version: 1.0
-Name: pam
-Version: 0.1.3
-Summary: PAM interface using ctypes
-Home-page: http://atlee.ca/software/pam
-Author: Chris AtLee
-Author-email: chris@atlee.ca
-License: MIT
-Download-URL: http://atlee.ca/software/pam/dist/0.1.3
-Description: An interface to the Pluggable Authentication Modules (PAM) library on linux, written in pure python (using ctypes)
-Platform: UNKNOWN
-Classifier: Development Status :: 3 - Alpha
-Classifier: Intended Audience :: Developers
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Operating System :: POSIX :: Linux
-Classifier: Operating System :: MacOS :: MacOS X
-Classifier: Programming Language :: Python
-Classifier: Topic :: Software Development :: Libraries :: Python Modules
-Classifier: Topic :: System :: Systems Administration :: Authentication/Directory

+ 0 - 123
desktop/core/ext-py/pam/pam.py

@@ -1,123 +0,0 @@
-# (c) 2007 Chris AtLee <chris@atlee.ca>
-# Licensed under the MIT license:
-# http://www.opensource.org/licenses/mit-license.php
-"""
-PAM module for python
-
-Provides an authenticate function that will allow the caller to authenticate
-a user against the Pluggable Authentication Modules (PAM) on the system.
-
-Implemented using ctypes, so no compilation is necessary.
-"""
-__all__ = ['authenticate']
-
-from ctypes import CDLL, POINTER, Structure, CFUNCTYPE, cast, pointer, sizeof
-from ctypes import c_void_p, c_uint, c_char_p, c_char, c_int
-from ctypes.util import find_library
-
-LIBPAM = CDLL(find_library("pam"))
-LIBC = CDLL(find_library("c"))
-
-CALLOC = LIBC.calloc
-CALLOC.restype = c_void_p
-CALLOC.argtypes = [c_uint, c_uint]
-
-STRDUP = LIBC.strdup
-STRDUP.argstypes = [c_char_p]
-STRDUP.restype = POINTER(c_char) # NOT c_char_p !!!!
-
-# Various constants
-PAM_PROMPT_ECHO_OFF = 1
-PAM_PROMPT_ECHO_ON = 2
-PAM_ERROR_MSG = 3
-PAM_TEXT_INFO = 4
-
-class PamHandle(Structure):
-    """wrapper class for pam_handle_t"""
-    _fields_ = [
-            ("handle", c_void_p)
-            ]
-
-    def __init__(self):
-        Structure.__init__(self)
-        self.handle = 0
-
-class PamMessage(Structure):
-    """wrapper class for pam_message structure"""
-    _fields_ = [
-            ("msg_style", c_int),
-            ("msg", c_char_p),
-            ]
-
-    def __repr__(self):
-        return "<PamMessage %i '%s'>" % (self.msg_style, self.msg)
-
-class PamResponse(Structure):
-    """wrapper class for pam_response structure"""
-    _fields_ = [
-            ("resp", c_char_p),
-            ("resp_retcode", c_int),
-            ]
-
-    def __repr__(self):
-        return "<PamResponse %i '%s'>" % (self.resp_retcode, self.resp)
-
-CONV_FUNC = CFUNCTYPE(c_int,
-        c_int, POINTER(POINTER(PamMessage)),
-               POINTER(POINTER(PamResponse)), c_void_p)
-
-class PamConv(Structure):
-    """wrapper class for pam_conv structure"""
-    _fields_ = [
-            ("conv", CONV_FUNC),
-            ("appdata_ptr", c_void_p)
-            ]
-
-PAM_START = LIBPAM.pam_start
-PAM_START.restype = c_int
-PAM_START.argtypes = [c_char_p, c_char_p, POINTER(PamConv),
-        POINTER(PamHandle)]
-
-PAM_AUTHENTICATE = LIBPAM.pam_authenticate
-PAM_AUTHENTICATE.restype = c_int
-PAM_AUTHENTICATE.argtypes = [PamHandle, c_int]
-
-def authenticate(username, password, service='login'):
-    """Returns True if the given username and password authenticate for the
-    given service.  Returns False otherwise
-    
-    ``username``: the username to authenticate
-    
-    ``password``: the password in plain text
-    
-    ``service``: the PAM service to authenticate against.
-                 Defaults to 'login'"""
-    @CONV_FUNC
-    def my_conv(n_messages, messages, p_response, app_data):
-        """Simple conversation function that responds to any
-        prompt where the echo is off with the supplied password"""
-        # Create an array of n_messages response objects
-        addr = CALLOC(n_messages, sizeof(PamResponse))
-        p_response[0] = cast(addr, POINTER(PamResponse))
-        for i in range(n_messages):
-            if messages[i].contents.msg_style == PAM_PROMPT_ECHO_OFF:
-                pw_copy = STRDUP(str(password))
-                p_response.contents[i].resp = cast(pw_copy, c_char_p)
-                p_response.contents[i].resp_retcode = 0
-        return 0
-
-    handle = PamHandle()
-    conv = PamConv(my_conv, 0)
-    retval = PAM_START(service, username, pointer(conv), pointer(handle))
-
-    if retval != 0:
-        # TODO: This is not an authentication error, something
-        # has gone wrong starting up PAM
-        return False
-
-    retval = PAM_AUTHENTICATE(handle, 0)
-    return retval == 0
-
-if __name__ == "__main__":
-    import getpass
-    print authenticate(getpass.getuser(), getpass.getpass())

+ 0 - 32
desktop/core/ext-py/pam/setup.py

@@ -1,32 +0,0 @@
-from setuptools import setup, find_packages
-import sys, os
-
-version = '0.1.3'
-
-setup(name='pam',
-      version=version,
-      description="PAM interface using ctypes",
-      long_description="""\
-An interface to the Pluggable Authentication Modules (PAM) library on linux, written in pure python (using ctypes)""",
-      classifiers=["Development Status :: 3 - Alpha",
-          "Intended Audience :: Developers",
-          "License :: OSI Approved :: MIT License",
-          "Operating System :: POSIX :: Linux",
-          "Operating System :: MacOS :: MacOS X",
-          "Programming Language :: Python",
-          "Topic :: Software Development :: Libraries :: Python Modules",
-          "Topic :: System :: Systems Administration :: Authentication/Directory"
-          ],
-      keywords='',
-      author='Chris AtLee',
-      author_email='chris@atlee.ca',
-      url='http://atlee.ca/software/pam',
-      download_url = "http://atlee.ca/software/pam/dist/%s" % version,
-      license='MIT',
-      py_modules=["pam"],
-      zip_safe=True,
-      install_requires=[],
-      entry_points="""
-      # -*- Entry points: -*-
-      """,
-      )

+ 21 - 0
desktop/core/ext-py/python-pam-1.8.4/LICENSE

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2014 David Ford
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

+ 2 - 0
desktop/core/ext-py/python-pam-1.8.4/MANIFEST.in

@@ -0,0 +1,2 @@
+include README.md
+include LICENSE

+ 74 - 0
desktop/core/ext-py/python-pam-1.8.4/PKG-INFO

@@ -0,0 +1,74 @@
+Metadata-Version: 1.2
+Name: python-pam
+Version: 1.8.4
+Summary: Python PAM module using ctypes, py3/py2
+Home-page: https://github.com/FirefighterBlu3/python-pam
+Author: David Ford
+Author-email: david@blue-labs.org
+Maintainer: David Ford
+Maintainer-email: david@blue-labs.org
+License: License :: OSI Approved :: MIT License
+Download-URL: https://github.com/FirefighterBlu3/python-pam
+Description: python-pam
+        ==========
+        
+        Python pam module supporting py3 (and py2)
+        
+        Commandline example:
+        
+        ```
+        [david@Scott python-pam]$ python pam.py
+        Username: david
+        Password: 
+        0 Success
+        
+        [david@Scott python-pam]$ python2 pam.py
+        Username: david
+        Password: 
+        0 Success
+        ```
+        
+        Inline examples:
+        ```
+        [david@Scott python-pam]$ python
+        Python 3.4.1 (default, May 19 2014, 17:23:49)
+        [GCC 4.9.0 20140507 (prerelease)] on linux
+        Type "help", "copyright", "credits" or "license" for more information.
+        >>> import pam
+        >>> p = pam.pam()
+        >>> p.authenticate('david', 'correctpassword')
+        True
+        >>> p.authenticate('david', 'badpassword')
+        False
+        >>> p.authenticate('david', 'correctpassword', service='login')
+        True
+        >>> p.authenticate('david', 'correctpassword', service='unknownservice')
+        False
+        >>> p.authenticate('david', 'correctpassword', service='login', resetcreds=True)
+        True
+        >>> p.authenticate('david', 'correctpassword', encoding='latin-1')
+        True
+        >>> print('{} {}'.format(p.code, p.reason))
+        0 Success
+        >>> p.authenticate('david', 'badpassword')
+        False
+        >>> print('{} {}'.format(p.code, p.reason))
+        7 Authentication failure
+        >>>
+        ```
+        
+Platform: i686
+Platform: x86_64
+Classifier: Development Status :: 6 - Mature
+Classifier: Environment :: Plugins
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Information Technology
+Classifier: Intended Audience :: System Administrators
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: POSIX
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 2
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Security
+Classifier: Topic :: System :: Systems Administration :: Authentication/Directory

+ 47 - 0
desktop/core/ext-py/python-pam-1.8.4/README.md

@@ -0,0 +1,47 @@
+python-pam
+==========
+
+Python pam module supporting py3 (and py2)
+
+Commandline example:
+
+```
+[david@Scott python-pam]$ python pam.py
+Username: david
+Password: 
+0 Success
+
+[david@Scott python-pam]$ python2 pam.py
+Username: david
+Password: 
+0 Success
+```
+
+Inline examples:
+```
+[david@Scott python-pam]$ python
+Python 3.4.1 (default, May 19 2014, 17:23:49)
+[GCC 4.9.0 20140507 (prerelease)] on linux
+Type "help", "copyright", "credits" or "license" for more information.
+>>> import pam
+>>> p = pam.pam()
+>>> p.authenticate('david', 'correctpassword')
+True
+>>> p.authenticate('david', 'badpassword')
+False
+>>> p.authenticate('david', 'correctpassword', service='login')
+True
+>>> p.authenticate('david', 'correctpassword', service='unknownservice')
+False
+>>> p.authenticate('david', 'correctpassword', service='login', resetcreds=True)
+True
+>>> p.authenticate('david', 'correctpassword', encoding='latin-1')
+True
+>>> print('{} {}'.format(p.code, p.reason))
+0 Success
+>>> p.authenticate('david', 'badpassword')
+False
+>>> print('{} {}'.format(p.code, p.reason))
+7 Authentication failure
+>>>
+```

+ 47 - 0
desktop/core/ext-py/python-pam-1.8.4/README.rst

@@ -0,0 +1,47 @@
+python-pam
+==========
+
+Python pam module supporting py3 (and py2)
+
+Commandline example:
+
+```
+[david@Scott python-pam]$ python pam.py
+Username: david
+Password: 
+0 Success
+
+[david@Scott python-pam]$ python2 pam.py
+Username: david
+Password: 
+0 Success
+```
+
+Inline examples:
+```
+[david@Scott python-pam]$ python
+Python 3.4.1 (default, May 19 2014, 17:23:49)
+[GCC 4.9.0 20140507 (prerelease)] on linux
+Type "help", "copyright", "credits" or "license" for more information.
+>>> import pam
+>>> p = pam.pam()
+>>> p.authenticate('david', 'correctpassword')
+True
+>>> p.authenticate('david', 'badpassword')
+False
+>>> p.authenticate('david', 'correctpassword', service='login')
+True
+>>> p.authenticate('david', 'correctpassword', service='unknownservice')
+False
+>>> p.authenticate('david', 'correctpassword', service='login', resetcreds=True)
+True
+>>> p.authenticate('david', 'correctpassword', encoding='latin-1')
+True
+>>> print('{} {}'.format(p.code, p.reason))
+0 Success
+>>> p.authenticate('david', 'badpassword')
+False
+>>> print('{} {}'.format(p.code, p.reason))
+7 Authentication failure
+>>>
+```

+ 224 - 0
desktop/core/ext-py/python-pam-1.8.4/pam.py

@@ -0,0 +1,224 @@
+# (c) 2007 Chris AtLee <chris@atlee.ca>
+# Licensed under the MIT license:
+# http://www.opensource.org/licenses/mit-license.php
+#
+# Original author: Chris AtLee
+#
+# Modified by David Ford, 2011-12-6
+# added py3 support and encoding
+# added pam_end
+# added pam_setcred to reset credentials after seeing Leon Walker's remarks
+# added byref as well
+# use readline to prestuff the getuser input
+
+'''
+PAM module for python
+
+Provides an authenticate function that will allow the caller to authenticate
+a user against the Pluggable Authentication Modules (PAM) on the system.
+
+Implemented using ctypes, so no compilation is necessary.
+'''
+
+__all__      = ['pam']
+__version__  = '1.8.4'
+__author__   = 'David Ford <david@blue-labs.org>'
+__released__ = '2018 June 15'
+
+import sys
+
+from ctypes import CDLL, POINTER, Structure, CFUNCTYPE, cast, byref, sizeof
+from ctypes import c_void_p, c_size_t, c_char_p, c_char, c_int
+from ctypes import memmove
+from ctypes.util import find_library
+
+class PamHandle(Structure):
+    """wrapper class for pam_handle_t pointer"""
+    _fields_ = [ ("handle", c_void_p) ]
+
+    def __init__(self):
+        Structure.__init__(self)
+        self.handle = 0
+
+class PamMessage(Structure):
+    """wrapper class for pam_message structure"""
+    _fields_ = [ ("msg_style", c_int), ("msg", c_char_p) ]
+
+    def __repr__(self):
+        return "<PamMessage %i '%s'>" % (self.msg_style, self.msg)
+
+class PamResponse(Structure):
+    """wrapper class for pam_response structure"""
+    _fields_ = [ ("resp", c_char_p), ("resp_retcode", c_int) ]
+
+    def __repr__(self):
+        return "<PamResponse %i '%s'>" % (self.resp_retcode, self.resp)
+
+conv_func = CFUNCTYPE(c_int, c_int, POINTER(POINTER(PamMessage)), POINTER(POINTER(PamResponse)), c_void_p)
+
+class PamConv(Structure):
+    """wrapper class for pam_conv structure"""
+    _fields_ = [ ("conv", conv_func), ("appdata_ptr", c_void_p) ]
+
+# Various constants
+PAM_PROMPT_ECHO_OFF       = 1
+PAM_PROMPT_ECHO_ON        = 2
+PAM_ERROR_MSG             = 3
+PAM_TEXT_INFO             = 4
+PAM_REINITIALIZE_CRED     = 8
+
+libc                      = CDLL(find_library("c"))
+libpam                    = CDLL(find_library("pam"))
+
+calloc                    = libc.calloc
+calloc.restype            = c_void_p
+calloc.argtypes           = [c_size_t, c_size_t]
+
+# bug #6 (@NIPE-SYSTEMS), some libpam versions don't include this function
+if hasattr(libpam, 'pam_end'):
+    pam_end                   = libpam.pam_end
+    pam_end.restype           = c_int
+    pam_end.argtypes          = [PamHandle, c_int]
+
+pam_start                 = libpam.pam_start
+pam_start.restype         = c_int
+pam_start.argtypes        = [c_char_p, c_char_p, POINTER(PamConv), POINTER(PamHandle)]
+
+pam_setcred               = libpam.pam_setcred
+pam_setcred.restype       = c_int
+pam_setcred.argtypes      = [PamHandle, c_int]
+
+pam_strerror              = libpam.pam_strerror
+pam_strerror.restype      = c_char_p
+pam_strerror.argtypes     = [PamHandle, c_int]
+
+pam_authenticate          = libpam.pam_authenticate
+pam_authenticate.restype  = c_int
+pam_authenticate.argtypes = [PamHandle, c_int]
+
+class pam():
+    code   = 0
+    reason = None
+
+    def __init__(self):
+        pass
+
+    def authenticate(self, username, password, service='login', encoding='utf-8', resetcreds=True):
+        """username and password authentication for the given service.
+
+           Returns True for success, or False for failure.
+
+           self.code (integer) and self.reason (string) are always stored and may
+           be referenced for the reason why authentication failed. 0/'Success' will
+           be stored for success.
+
+           Python3 expects bytes() for ctypes inputs.  This function will make
+           necessary conversions using the supplied encoding.
+
+        Inputs:
+          username: username to authenticate
+          password: password in plain text
+          service:  PAM service to authenticate against, defaults to 'login'
+
+        Returns:
+          success:  True
+          failure:  False
+        """
+
+        @conv_func
+        def my_conv(n_messages, messages, p_response, app_data):
+            """Simple conversation function that responds to any
+               prompt where the echo is off with the supplied password"""
+            # Create an array of n_messages response objects
+            addr = calloc(n_messages, sizeof(PamResponse))
+            response = cast(addr, POINTER(PamResponse))
+            p_response[0] = response
+            for i in range(n_messages):
+                if messages[i].contents.msg_style == PAM_PROMPT_ECHO_OFF:
+                    dst = calloc(len(password)+1, sizeof(c_char))
+                    memmove(dst, cpassword, len(password))
+                    response[i].resp = dst
+                    response[i].resp_retcode = 0
+            return 0
+
+        # python3 ctypes prefers bytes
+        if sys.version_info >= (3,):
+            if isinstance(username, str): username = username.encode(encoding)
+            if isinstance(password, str): password = password.encode(encoding)
+            if isinstance(service, str):  service  = service.encode(encoding)
+        else:
+            if isinstance(username, unicode):
+                username = username.encode(encoding)
+            if isinstance(password, unicode):
+                password = password.encode(encoding)
+            if isinstance(service, unicode):
+                service  = service.encode(encoding)
+
+        if b'\x00' in username or b'\x00' in password or b'\x00' in service:
+            self.code = 4  # PAM_SYSTEM_ERR in Linux-PAM
+            self.reason = 'strings may not contain NUL'
+            return False
+
+        # do this up front so we can safely throw an exception if there's
+        # anything wrong with it
+        cpassword = c_char_p(password)
+
+        handle = PamHandle()
+        conv   = PamConv(my_conv, 0)
+        retval = pam_start(service, username, byref(conv), byref(handle))
+
+        if retval != 0:
+            # This is not an authentication error, something has gone wrong starting up PAM
+            self.code   = retval
+            self.reason = "pam_start() failed"
+            return False
+
+        retval = pam_authenticate(handle, 0)
+        auth_success = retval == 0
+
+        if auth_success and resetcreds:
+            retval = pam_setcred(handle, PAM_REINITIALIZE_CRED);
+
+        # store information to inform the caller why we failed
+        self.code   = retval
+        self.reason = pam_strerror(handle, retval)
+        if sys.version_info >= (3,):
+            self.reason = self.reason.decode(encoding)
+
+        if hasattr(libpam, 'pam_end'):
+            pam_end(handle, retval)
+
+        return auth_success
+
+
+def authenticate(*vargs, **dargs):
+    """
+    Compatibility function for older versions of python-pam.
+    """
+    return pam().authenticate(*vargs, **dargs)
+
+
+if __name__ == "__main__":
+    import readline, getpass
+
+    def input_with_prefill(prompt, text):
+        def hook():
+            readline.insert_text(text)
+            readline.redisplay()
+        readline.set_pre_input_hook(hook)
+
+        if sys.version_info >= (3,):
+            result = input(prompt)
+        else:
+            result = raw_input(prompt)
+
+        readline.set_pre_input_hook()
+        return result
+
+    pam = pam()
+
+    username = input_with_prefill('Username: ', getpass.getuser())
+
+    # enter a valid username and an invalid/valid password, to verify both failure and success
+    pam.authenticate(username, getpass.getpass())
+    print('{} {}'.format(pam.code, pam.reason))

+ 0 - 1
desktop/core/ext-py/pam/setup.cfg → desktop/core/ext-py/python-pam-1.8.4/setup.cfg

@@ -1,5 +1,4 @@
 [egg_info]
 tag_build = 
 tag_date = 0
-tag_svn_revision = 0
 

+ 37 - 0
desktop/core/ext-py/python-pam-1.8.4/setup.py

@@ -0,0 +1,37 @@
+import os
+from setuptools import setup
+
+def read(fname):
+    return open(os.path.join(os.path.dirname(__file__), fname)).read()
+
+__sdesc = 'Python PAM module using ctypes, py3/py2'
+
+setup(name             = 'python-pam',
+      description      =  __sdesc,
+      long_description = read('README.md'),
+      py_modules       = ['pam'],
+      version          = '1.8.4',
+      author           = 'David Ford',
+      author_email     = 'david@blue-labs.org',
+      maintainer       = 'David Ford',
+      maintainer_email = 'david@blue-labs.org',
+      url              = 'https://github.com/FirefighterBlu3/python-pam',
+      download_url     = 'https://github.com/FirefighterBlu3/python-pam',
+      license          = 'License :: OSI Approved :: MIT License',
+      platforms        = ['i686','x86_64'],
+      classifiers      = [
+          'Development Status :: 6 - Mature',
+          'Environment :: Plugins',
+          'Intended Audience :: Developers',
+          'Intended Audience :: Information Technology',
+          'Intended Audience :: System Administrators',
+          'License :: OSI Approved :: MIT License',
+          'Operating System :: POSIX',
+          'Operating System :: POSIX :: Linux',
+          'Programming Language :: Python',
+          'Programming Language :: Python :: 2',
+          'Programming Language :: Python :: 3',
+          'Topic :: Security',
+          'Topic :: System :: Systems Administration :: Authentication/Directory',
+          ],
+      )

+ 2 - 2
desktop/core/src/desktop/api2.py

@@ -116,7 +116,7 @@ def get_context_namespaces(request, interface):
         sdx_namespaces = SdxApi(request.user).list_namespaces()
 
       # Adding "fake" namespace for cluster without one
-      sdx_namespaces.extend([_cluster for _cluster in adb_clusters if not _cluster.get('namespaceCrn') or (IS_K8S_ONLY.get() and _cluster['status'] != 'TERMINATING')])
+      sdx_namespaces.extend([_cluster for _cluster in adb_clusters if not _cluster.get('namespaceCrn') or (IS_K8S_ONLY.get() and 'TERMINAT' not in _cluster['status'])])
 
       namespaces.extend([{
           'id': namespace.get('crn', 'None'),
@@ -164,7 +164,7 @@ def get_context_computes(request, interface):
           'namespace': cluster.get('namespaceCrn', cluster.get('crn')),
           'compute_end_point': IS_K8S_ONLY.get() and '%(publicHost)s' % cluster['coordinatorEndpoint'] or '',
           'type': 'altus-dw'
-        } for cluster in dw_clusters if (cluster.get('status') == 'CREATED' and cluster.get('cdhVersion') >= 'CDH515') or (IS_K8S_ONLY.get() and cluster['status'] != 'TERMINATING')]
+        } for cluster in dw_clusters if (cluster.get('status') == 'CREATED' and cluster.get('cdhVersion') >= 'CDH515') or (IS_K8S_ONLY.get() and 'TERMINAT' not in cluster['status'])]
       )
 
     if interface == 'oozie' or interface == 'spark2':

+ 19 - 4
desktop/core/src/desktop/conf.py

@@ -1638,6 +1638,18 @@ def get_clusters(user):
       )
     )
 
+  if IS_K8S_ONLY.get():
+    clusters.append(
+      ('impala', {
+          'id': 'impala',
+          'name': 'impala',
+          'type': 'impala',
+          'interface': 'impala',
+          'server_host': 'impala'
+        }
+      )
+    )
+
   return OrderedDict(clusters)
 
 
@@ -1833,10 +1845,13 @@ def config_validator(user):
   res.extend(validate_database(user))
 
   # Validate if oozie email server is active
-  from oozie.views.editor2 import _is_oozie_mail_enabled
-
-  if not _is_oozie_mail_enabled(user):
-    res.append(('OOZIE_EMAIL_SERVER', unicode(_('Email notifications is disabled for Workflows and Jobs as SMTP server is localhost.'))))
+  try:
+    from oozie.views.editor2 import _is_oozie_mail_enabled
+  
+    if not _is_oozie_mail_enabled(user):
+      res.append(('OOZIE_EMAIL_SERVER', unicode(_('Email notifications is disabled for Workflows and Jobs as SMTP server is localhost.'))))
+  except Exception, e:
+    LOG.warn('Config check failed because Oozie app not installed %s' % e)
 
   from notebook.models import make_notebook
   from notebook.api import _save_notebook

+ 1 - 0
desktop/core/src/desktop/lib/fsmanager.py

@@ -40,6 +40,7 @@ if is_s3_enabled():
 if is_adls_enabled():
   FS_GETTERS['adl'] = azure.client.get_client
 
+
 def get_filesystem(name='default'):
   """
   Return the filesystem with the given name.

+ 13 - 17
desktop/core/src/desktop/models.py

@@ -22,11 +22,7 @@ import os
 import urllib
 import uuid
 
-try:
-  from collections import OrderedDict
-except ImportError:
-  from ordereddict import OrderedDict # Python 2.6
-
+from collections import OrderedDict
 from itertools import chain
 
 from django.contrib.auth import models as auth_models
@@ -1663,8 +1659,8 @@ class ClusterConfig():
 
     _interpreters = get_ordered_interpreters(self.user)
 
-    if self.cluster_type == ANALYTIC_DB:
-      _interpreters = [interpreter for interpreter in _interpreters if interpreter['type'] in ('impala', 'hive', 'spark2', 'pyspark', 'mapreduce')]
+    if ANALYTIC_DB in self.cluster_type:
+      _interpreters = [interpreter for interpreter in _interpreters if interpreter['type'] in ('impala')] #, 'hive', 'spark2', 'pyspark', 'mapreduce')]
 
     for interpreter in _interpreters:
       interpreters.append({
@@ -1677,7 +1673,7 @@ class ClusterConfig():
         'is_sql': interpreter['is_sql']
       })
 
-    if SHOW_NOTEBOOKS.get() and self.cluster_type != ANALYTIC_DB:
+    if SHOW_NOTEBOOKS.get() and ANALYTIC_DB not in self.cluster_type:
       try:
         first_non_sql_index = [interpreter['is_sql'] for interpreter in interpreters].index(False)
       except ValueError:
@@ -1709,7 +1705,7 @@ class ClusterConfig():
     interpreters = get_engines(self.user)
     _interpreters = []
 
-    if interpreters and self.cluster_type != ANALYTIC_DB:
+    if interpreters and ANALYTIC_DB not in self.cluster_type:
       if HAS_REPORT_ENABLED.get():
         _interpreters.append({
           'type': 'report',
@@ -1743,7 +1739,7 @@ class ClusterConfig():
   def _get_browser(self):
     interpreters = []
 
-    if 'filebrowser' in self.apps and self.cluster_type != ANALYTIC_DB:
+    if 'filebrowser' in self.apps and ANALYTIC_DB not in self.cluster_type:
       interpreters.append({
         'type': 'hdfs',
         'displayName': _('Files'),
@@ -1761,7 +1757,7 @@ class ClusterConfig():
         'page': '/filebrowser/view=S3A://'
       })
 
-    if is_adls_enabled() and has_adls_access(self.user) and self.cluster_type != ANALYTIC_DB:
+    if is_adls_enabled() and has_adls_access(self.user) and ANALYTIC_DB not in self.cluster_type:
       interpreters.append({
         'type': 'adls',
         'displayName': _('ADLS'),
@@ -1779,7 +1775,7 @@ class ClusterConfig():
         'page': '/metastore/tables'
       })
 
-    if 'search' in self.apps and self.cluster_type != ANALYTIC_DB:
+    if 'search' in self.apps and ANALYTIC_DB not in self.cluster_type:
       interpreters.append({
         'type': 'indexes',
         'displayName': _('Indexes'),
@@ -1791,7 +1787,7 @@ class ClusterConfig():
     if 'jobbrowser' in self.apps:
       from hadoop.cluster import get_default_yarncluster # Circular loop
 
-      title =  _('Jobs') if self.cluster_type != ANALYTIC_DB else _('Queries')
+      title =  _('Jobs') if ANALYTIC_DB not in self.cluster_type else _('Queries')
 
       if get_default_yarncluster():
         interpreters.append({
@@ -1802,7 +1798,7 @@ class ClusterConfig():
           'page': '/jobbrowser/'
         })
 
-    if has_kafka() and self.cluster_type != ANALYTIC_DB:
+    if has_kafka() and ANALYTIC_DB not in self.cluster_type:
       interpreters.append({
         'type': 'kafka',
         'displayName': _('Streams'),
@@ -1811,7 +1807,7 @@ class ClusterConfig():
         'page': '/kafka/'
       })
 
-    if 'hbase' in self.apps and self.cluster_type != ANALYTIC_DB:
+    if 'hbase' in self.apps and ANALYTIC_DB not in self.cluster_type:
       interpreters.append({
         'type': 'hbase',
         'displayName': _('HBase'),
@@ -1829,7 +1825,7 @@ class ClusterConfig():
         'page': '/security/hive'
       })
 
-    if 'sqoop' in self.apps and self.cluster_type != ANALYTIC_DB:
+    if 'sqoop' in self.apps and ANALYTIC_DB not in self.cluster_type:
       from sqoop.conf import IS_ENABLED
       if IS_ENABLED.get():
         interpreters.append({
@@ -1874,7 +1870,7 @@ class ClusterConfig():
       }
     ]
 
-    if 'oozie' in self.apps and not (self.user.has_hue_permission(action="disable_editor_access", app="oozie") and not is_admin(self.user)) and self.cluster_type != ANALYTIC_DB:
+    if 'oozie' in self.apps and not (self.user.has_hue_permission(action="disable_editor_access", app="oozie") and not is_admin(self.user)) and ANALYTIC_DB not in self.cluster_type:
       return {
           'name': 'oozie',
           'displayName': _('Scheduler'),

+ 1 - 1
desktop/core/src/desktop/settings.py

@@ -90,7 +90,7 @@ LANGUAGES = [
   ('ko', _('Korean')),
   ('pt', _('Portuguese')),
   ('pt_BR', _('Brazilian Portuguese')),
-  ('zh_CN', _('Simplified Chinese')),
+  ('zh-CN', _('Simplified Chinese')),
 ]
 
 SITE_ID = 1

File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/css/hue-embedded.css


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/css/hue.css


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/css/hue3-extra.css


+ 0 - 1
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_abort_on_default_limit_exceeded.json

@@ -1 +0,0 @@
-{"body":"<div><div><div class=\"hue-doc-title\">ABORT_ON_DEFAULT_LIMIT_EXCEEDED Query Option</div><div><p>\n        Now that the <span class=\"hue-doc-codeph\">ORDER BY</span> clause no longer requires an accompanying <span class=\"hue-doc-codeph\">LIMIT</span>\n        clause in Impala 1.4.0 and higher, this query option is deprecated and has no effect.\n      </p><p><b>Type:</b> Boolean; recognized values are 1 and 0, or <span class=\"hue-doc-codeph\">true</span> and <span class=\"hue-doc-codeph\">false</span>;\n        any other value interpreted as <span class=\"hue-doc-codeph\">false</span></p><p><b>Default:</b><span class=\"hue-doc-codeph\">false</span> (shown as 0 in output of <span class=\"hue-doc-codeph\">SET</span> statement)\n      </p></div></div></div>","title":"ABORT_ON_DEFAULT_LIMIT_EXCEEDED Query Option"}

+ 1 - 1
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_abort_on_error.json

@@ -1 +1 @@
-{"body":"<div><div><div class=\"hue-doc-title\">ABORT_ON_ERROR Query Option</div><div><p>\n      When this option is enabled, Impala cancels a query immediately when any of the nodes encounters an error,\n      rather than continuing and possibly returning incomplete results. This option is disabled by default, to help\n      gather maximum diagnostic information when an error occurs, for example, whether the same problem occurred on\n      all nodes or only a single node. Currently, the errors that Impala can skip over involve data corruption,\n      such as a column that contains a string value when expected to contain an integer value.\n    </p><p>\n      To control how much logging Impala does for non-fatal errors when <span class=\"hue-doc-codeph\">ABORT_ON_ERROR</span> is turned\n      off, use the <span class=\"hue-doc-codeph\">MAX_ERRORS</span> option.\n    </p><p><b>Type:</b> Boolean; recognized values are 1 and 0, or <span class=\"hue-doc-codeph\">true</span> and <span class=\"hue-doc-codeph\">false</span>;\n        any other value interpreted as <span class=\"hue-doc-codeph\">false</span></p><p><b>Default:</b><span class=\"hue-doc-codeph\">false</span> (shown as 0 in output of <span class=\"hue-doc-codeph\">SET</span> statement)\n      </p><p><b>Related information:</b></p><p><a class=\"hue-doc-internal-link\" href=\"javascript:void(0);\" data-doc-ref=\"topics/impala_max_errors.xml\" data-doc-anchor-id=\"max_errors\">MAX_ERRORS Query Option</a>,\n      <a class=\"hue-doc-external-link\" href=\"https://www.cloudera.com/documentation/enterprise/latest/topics/impala_logging.html#logging\" target=\"_blank\">Using Impala Logging</a></p></div></div></div>","title":"ABORT_ON_ERROR Query Option"}
+{"body":"<div><div id=\"abort_on_error\"><div class=\"hue-doc-title\">ABORT_ON_ERROR Query Option</div><div><p>\n      When this option is enabled, Impala cancels a query immediately when any of the nodes encounters an error,\n      rather than continuing and possibly returning incomplete results. This option is disabled by default, to help\n      gather maximum diagnostic information when an error occurs, for example, whether the same problem occurred on\n      all nodes or only a single node. Currently, the errors that Impala can skip over involve data corruption,\n      such as a column that contains a string value when expected to contain an integer value.\n    </p><p>\n      To control how much logging Impala does for non-fatal errors when <span class=\"hue-doc-codeph\">ABORT_ON_ERROR</span> is turned\n      off, use the <span class=\"hue-doc-codeph\">MAX_ERRORS</span> option.\n    </p><p id=\"type_boolean\"><b>Type:</b> Boolean; recognized values are 1 and 0, or <span class=\"hue-doc-codeph\">true</span> and <span class=\"hue-doc-codeph\">false</span>;\n        any other value interpreted as <span class=\"hue-doc-codeph\">false</span></p><p id=\"default_false_0\"><b>Default:</b><span class=\"hue-doc-codeph\">false</span> (shown as 0 in output of <span class=\"hue-doc-codeph\">SET</span> statement)\n      </p><p id=\"related_info\"><b>Related information:</b></p><p><a class=\"hue-doc-internal-link\" href=\"javascript:void(0);\" data-doc-ref=\"topics/impala_max_errors.xml\" data-doc-anchor-id=\"max_errors\">MAX_ERRORS Query Option</a>,\n      <a class=\"hue-doc-external-link\" href=\"https://www.cloudera.com/documentation/enterprise/latest/topics/impala_logging.html#logging\" target=\"_blank\">Using Impala Logging</a></p></div></div></div>","title":"ABORT_ON_ERROR Query Option"}

File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_aggregate_functions.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_aliases.json


+ 1 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_allow_erasure_coded_files.json

@@ -0,0 +1 @@
+{"body":"<div><div id=\"allow_erasure_coded_files\"><div class=\"hue-doc-title\">ALLOW_ERASURE_CODED_FILES Query Option</div><div><p>\n      Use the <span class=\"hue-doc-codeph\">ALLOW_ERASURE_CODED_FILES</span> query option to enable or disable the\n      support of erasure coded files in Impala. Until Impala is fully tested and certified with\n      erasure coded files, this query option is set to <span class=\"hue-doc-codeph\">FALSE</span> by default.\n    </p><p>\n      When the <span class=\"hue-doc-codeph\">ALLOW_ERASURE_CODED_FILES</span> query option is set to\n      <span class=\"hue-doc-codeph\">FALSE</span>, Impala returns an error when a query requires scanning an erasure\n      coded file.\n    </p><p><b>Type:</b><span class=\"hue-doc-codeph\">BOOLEAN</span></p><p><b>Default:</b><span class=\"hue-doc-codeph\">FALSE</span></p><p><b>Added in:</b>Impala 3.1</p></div></div></div>","title":"ALLOW_ERASURE_CODED_FILES Query Option"}

+ 1 - 1
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_allow_unsupported_formats.json

@@ -1 +1 @@
-{"body":"<div><div><div class=\"hue-doc-title\">ALLOW_UNSUPPORTED_FORMATS Query Option</div><div><p>\n      An obsolete query option from early work on support for file formats. Do not use. Might be removed in the\n      future.\n    </p><p><b>Type:</b> Boolean; recognized values are 1 and 0, or <span class=\"hue-doc-codeph\">true</span> and <span class=\"hue-doc-codeph\">false</span>;\n        any other value interpreted as <span class=\"hue-doc-codeph\">false</span></p><p><b>Default:</b><span class=\"hue-doc-codeph\">false</span> (shown as 0 in output of <span class=\"hue-doc-codeph\">SET</span> statement)\n      </p></div></div></div>","title":"ALLOW_UNSUPPORTED_FORMATS Query Option"}
+{"body":"<div><div id=\"allow_unsupported_formats\"><div class=\"hue-doc-title\">ALLOW_UNSUPPORTED_FORMATS Query Option</div><div><div class=\"hue-doc-note\">This query option was removed in Impala 3.1 and no\n      longer has any effect. Do not use.</div><p id=\"type_boolean\"><b>Type:</b> Boolean; recognized values are 1 and 0, or <span class=\"hue-doc-codeph\">true</span> and <span class=\"hue-doc-codeph\">false</span>;\n        any other value interpreted as <span class=\"hue-doc-codeph\">false</span></p><p id=\"default_false_0\"><b>Default:</b><span class=\"hue-doc-codeph\">false</span> (shown as 0 in output of <span class=\"hue-doc-codeph\">SET</span> statement)\n      </p></div></div></div>","title":"ALLOW_UNSUPPORTED_FORMATS Query Option"}

+ 1 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_alter_database.json

@@ -0,0 +1 @@
+{"body":"<div><div id=\"alter_database\"><div class=\"hue-doc-title\">ALTER DATABASE Statement</div><div><p>\n      The <span class=\"hue-doc-codeph\">ALTER DATABASE</span> statement changes the characteristics of a database.\n    </p><p>\n      Use the <span class=\"hue-doc-codeph\">SET OWNER</span> clause to transfer the ownership of the database from\n      the current owner to another user or a role.\n    </p><p>\n      The database owner is originally set to the user who creates the database. When object\n      ownership is enabled in Sentry, an owner of a database can have the <span class=\"hue-doc-codeph\">ALL</span>\n      with <span class=\"hue-doc-codeph\">GRANT</span> or <span class=\"hue-doc-codeph\">ALL</span> without <span class=\"hue-doc-codeph\">GRANT</span>\n      privilege. The term <span class=\"hue-doc-codeph\">OWNER</span> is used to differentiate between the\n      <span class=\"hue-doc-codeph\">ALL</span> privilege that is explicitly granted via the <span class=\"hue-doc-codeph\">GRANT</span>\n      statement and a privilege that is implicitly granted by the <span class=\"hue-doc-codeph\">CREATE\n      DATABASE</span> statement.\n    </p><p id=\"syntax_blurb\"><b>Syntax:</b></p><div class=\"hue-doc-codeblock\">ALTER DATABASE <span class=\"hue-doc-varname\">database_name</span> SET OWNER USER user_name;\nALTER DATABASE <span class=\"hue-doc-varname\">database_name</span> SET OWNER ROLE role_name;\n</div><p id=\"ddl_blurb\"><b>Statement type:</b> DDL\n      </p><p id=\"cancel_blurb_no\"><b>Cancellation:</b> Cannot be cancelled.\n      </p><p><b>Added in:</b>Impala 3.1</p></div></div></div>","title":"ALTER DATABASE Statement"}

File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_alter_table.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_alter_view.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_analytic_functions.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_appx_count_distinct.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_appx_median.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_array.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_avg.json


+ 1 - 1
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_batch_size.json

@@ -1 +1 @@
-{"body":"<div><div><div class=\"hue-doc-title\">BATCH_SIZE Query Option</div><div><p>\n      Number of rows evaluated at a time by SQL operators. Unspecified or a size of 0 uses a predefined default\n      size. Using a large number improves responsiveness, especially for scan operations, at the cost of a higher memory footprint.\n    </p><p>\n      This option is primarily for testing during Impala development, or for use under the direction of the appropriate support channel.\n    </p><p><b>Type:</b> numeric\n    </p><p><b>Default:</b> 0 (meaning the predefined default of 1024)\n    </p><p><b>Range:</b> 0-65536. The value of 0 still has the special meaning of <q>use the default</q>,\n      so the effective range is 1-65536. The maximum applies in Impala 2.11 and higher.\n    </p></div></div></div>","title":"BATCH_SIZE Query Option"}
+{"body":"<div><div id=\"batch_size\"><div class=\"hue-doc-title\">BATCH_SIZE Query Option</div><div><p>\n      Number of rows evaluated at a time by SQL operators. Unspecified or a size of 0 uses a predefined default\n      size. Using a large number improves responsiveness, especially for scan operations, at the cost of a higher memory footprint.\n    </p><p>\n      This option is primarily for testing during Impala development, or for use under the direction of the appropriate support channel.\n    </p><p><b>Type:</b> numeric\n    </p><p><b>Default:</b> 0 (meaning the predefined default of 1024)\n    </p><p><b>Range:</b> 0-65536. The value of 0 still has the special meaning of <q>use the default</q>,\n      so the effective range is 1-65536. The maximum applies in Impala 2.11 and higher.\n    </p></div></div></div>","title":"BATCH_SIZE Query Option"}

File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_bigint.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_bit_functions.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_boolean.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_buffer_pool_limit.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_char.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_comment.json


+ 1 - 1
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_comments.json

@@ -1 +1 @@
-{"body":"<div><div><div class=\"hue-doc-title\">Comments</div><div><p>\n      Impala supports the familiar styles of SQL comments:\n    </p><ul><li>\n        All text from a <span class=\"hue-doc-codeph\">--</span> sequence to the end of the line is considered a comment and ignored.\n        This type of comment can occur on a single line by itself, or after all or part of a statement.\n      </li><li>\n        All text from a <span class=\"hue-doc-codeph\">/*</span> sequence to the next <span class=\"hue-doc-codeph\">*/</span> sequence is considered a\n        comment and ignored. This type of comment can stretch over multiple lines. This type of comment can occur\n        on one or more lines by itself, in the middle of a statement, or before or after a statement.\n      </li></ul><p>\n      For example:\n    </p><div class=\"hue-doc-codeblock\">-- This line is a comment about a table.\ncreate table ...;\n\n/*\nThis is a multi-line comment about a query.\n*/\nselect ...;\n\nselect * from t /* This is an embedded comment about a query. */ where ...;\n\nselect * from t -- This is a trailing comment within a multi-line command.\nwhere ...;\n</div></div></div></div>","title":"Comments"}
+{"body":"<div><div id=\"comments\"><div class=\"hue-doc-title\">Comments</div><div><p>\n      Impala supports the familiar styles of SQL comments:\n    </p><ul><li>\n        All text from a <span class=\"hue-doc-codeph\">--</span> sequence to the end of the line is considered a comment and ignored.\n        This type of comment can occur on a single line by itself, or after all or part of a statement.\n      </li><li>\n        All text from a <span class=\"hue-doc-codeph\">/*</span> sequence to the next <span class=\"hue-doc-codeph\">*/</span> sequence is considered a\n        comment and ignored. This type of comment can stretch over multiple lines. This type of comment can occur\n        on one or more lines by itself, in the middle of a statement, or before or after a statement.\n      </li></ul><p>\n      For example:\n    </p><div class=\"hue-doc-codeblock\">-- This line is a comment about a table.\ncreate table ...;\n\n/*\nThis is a multi-line comment about a query.\n*/\nselect ...;\n\nselect * from t /* This is an embedded comment about a query. */ where ...;\n\nselect * from t -- This is a trailing comment within a multi-line command.\nwhere ...;\n</div></div></div></div>","title":"Comments"}

File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_complex_types.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_compression_codec.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_compute_stats.json


+ 1 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_compute_stats_min_sample_size.json

@@ -0,0 +1 @@
+{"body":"<div><div id=\"compute_stats_sample_min_sample_size\"><div class=\"hue-doc-title\">COMPUTE_STATS_MIN_SAMPLE_SIZE Query Option</div><div><p>The <span class=\"hue-doc-codeph\">COMPUTE_STATS_MIN_SAMPLE_SIZE</span> query option specifies\n      the minimum number of bytes that will be scanned in <span class=\"hue-doc-codeph\">COMPUTE STATS\n        TABLESAMPLE</span>, regardless of the user-supplied sampling percent.\n      This query option prevents sampling for very small tables where accurate\n      stats can be obtained cheaply without sampling because the minimum sample\n      size is required to get meaningful stats.</p><p id=\"type_integer\"><b>Type:</b> integer\n      </p><p><b>Default:</b> 1GB</p><p><b>Added in</b>: Impala 2.12</p><p id=\"usage_notes_blurb\"><b>Usage notes:</b></p></div></div></div>","title":"COMPUTE_STATS_MIN_SAMPLE_SIZE Query Option"}

File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_conditional_functions.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_conversion_functions.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_count.json


+ 1 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_create_data_source.json

@@ -0,0 +1 @@
+{"body":"<div><div style=\"display:none;\" id=\"create_data_source\"><div class=\"hue-doc-title\">CREATE DATA SOURCE Statement</div><div><p/><p id=\"ddl_blurb\"><b>Statement type:</b> DDL\n      </p><p id=\"sync_ddl_blurb\">\n        If you connect to different Impala nodes within an <span class=\"hue-doc-cmdname\">impala-shell</span> session for\n        load-balancing purposes, you can enable the <span class=\"hue-doc-codeph\">SYNC_DDL</span> query option to make each DDL\n        statement wait before returning, until the new or changed metadata has been received by all the Impala\n        nodes. See <a class=\"hue-doc-internal-link\" href=\"javascript:void(0);\" data-doc-ref=\"topics/impala_sync_ddl.xml\" data-doc-anchor-id=\"sync_ddl\">SYNC_DDL Query Option</a> for details.\n      </p><p id=\"usage_notes_blurb\"><b>Usage notes:</b></p><p id=\"example_blurb\"><b>Examples:</b></p><p id=\"cancel_blurb_no\"><b>Cancellation:</b> Cannot be cancelled.\n      </p><p id=\"related_info\"><b>Related information:</b></p></div></div></div>","title":"CREATE DATA SOURCE Statement"}

File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_create_database.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_create_function.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_create_role.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_create_table.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_create_view.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_custom_timezones.json


+ 1 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_data_sources.json

@@ -0,0 +1 @@
+{"body":"<div><div id=\"data_sources\"><div class=\"hue-doc-title\">Data Sources</div><div><p><a class=\"hue-doc-internal-link\" href=\"javascript:void(0);\" data-doc-ref=\"topics/impala_create_data_source.xml\" data-doc-anchor-id=\"create_data_source\">CREATE DATA SOURCE Statement</a><a class=\"hue-doc-internal-link\" href=\"javascript:void(0);\" data-doc-ref=\"topics/impala_drop_data_source.xml\" data-doc-anchor-id=\"drop_data_source\">DROP DATA SOURCE Statement</a><a class=\"hue-doc-internal-link\" href=\"javascript:void(0);\" data-doc-ref=\"topics/impala_create_table.xml\" data-doc-anchor-id=\"create_table\">CREATE TABLE Statement</a></p></div></div></div>","title":"Data Sources"}

File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_databases.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_datatypes.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_date.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_datetime_functions.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_ddl.json


+ 1 - 1
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_debug_action.json

@@ -1 +1 @@
-{"body":"<div><div><div class=\"hue-doc-title\">DEBUG_ACTION Query Option</div><div><p>\n      Introduces artificial problem conditions within queries. For internal debugging and troubleshooting.\n    </p><p><b>Type:</b><span class=\"hue-doc-codeph\">STRING</span></p><p><b>Default:</b> empty string\n    </p></div></div></div>","title":"DEBUG_ACTION Query Option"}
+{"body":"<div><div id=\"debug_action\"><div class=\"hue-doc-title\">DEBUG_ACTION Query Option</div><div><p>\n      Introduces artificial problem conditions within queries. For internal debugging and troubleshooting.\n    </p><p><b>Type:</b><span class=\"hue-doc-codeph\">STRING</span></p><p><b>Default:</b> empty string\n    </p></div></div></div>","title":"DEBUG_ACTION Query Option"}

File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_decimal.json


+ 1 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_decimal_v2.json

@@ -0,0 +1 @@
+{"body":"<div><div id=\"decimal_v2\"><div class=\"hue-doc-title\">DECIMAL_V2 Query Option</div><div><p>\n      A query option that changes behavior related to the <span class=\"hue-doc-codeph\">DECIMAL</span>\n      data type.\n    </p><div class=\"hue-doc-note\"><p>\n        This query option is currently unsupported.\n        Its precise behavior is currently undefined and might change\n        in the future.\n      </p></div><p id=\"type_boolean\"><b>Type:</b> Boolean; recognized values are 1 and 0, or <span class=\"hue-doc-codeph\">true</span> and <span class=\"hue-doc-codeph\">false</span>;\n        any other value interpreted as <span class=\"hue-doc-codeph\">false</span></p><p id=\"default_false_0\"><b>Default:</b><span class=\"hue-doc-codeph\">false</span> (shown as 0 in output of <span class=\"hue-doc-codeph\">SET</span> statement)\n      </p></div></div></div>","title":"DECIMAL_V2 Query Option"}

File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_default_join_distribution_mode.json


+ 0 - 1
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_default_order_by_limit.json

@@ -1 +0,0 @@
-{"body":"<div><div><div class=\"hue-doc-title\">DEFAULT_ORDER_BY_LIMIT Query Option</div><div><p>\n        Now that the <span class=\"hue-doc-codeph\">ORDER BY</span> clause no longer requires an accompanying <span class=\"hue-doc-codeph\">LIMIT</span>\n        clause in Impala 1.4.0 and higher, this query option is deprecated and has no effect.\n      </p><p>\n      Prior to Impala 1.4.0, Impala queries that use the <span class=\"hue-doc-codeph\"><a class=\"hue-doc-internal-link\" href=\"javascript:void(0);\" data-doc-ref=\"topics/impala_order_by.xml\" data-doc-anchor-id=\"order_by\">ORDER\n      BY</a></span> clause must also include a\n      <span class=\"hue-doc-codeph\"><a class=\"hue-doc-internal-link\" href=\"javascript:void(0);\" data-doc-ref=\"topics/impala_limit.xml\" data-doc-anchor-id=\"limit\">LIMIT</a></span> clause, to avoid accidentally producing\n      huge result sets that must be sorted. Sorting a huge result set is a memory-intensive operation. In Impala\n      1.4.0 and higher, Impala uses a temporary disk work area to perform the sort if that operation would\n      otherwise exceed the Impala memory limit on a particular host.\n    </p><p><b>Type: numeric</b></p><p><b>Default:</b> -1 (no default limit)\n    </p></div></div></div>","title":"DEFAULT_ORDER_BY_LIMIT Query Option"}

File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_default_spillable_buffer_size.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_delete.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_describe.json


+ 1 - 1
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_disable_codegen.json

@@ -1 +1 @@
-{"body":"<div><div><div class=\"hue-doc-title\">DISABLE_CODEGEN Query Option</div><div><p>\n      This is a debug option, intended for diagnosing and working around issues that cause crashes. If a query\n      fails with an <q>illegal instruction</q> or other hardware-specific message, try setting\n      <span class=\"hue-doc-codeph\">DISABLE_CODEGEN=true</span> and running the query again. If the query succeeds only when the\n      <span class=\"hue-doc-codeph\">DISABLE_CODEGEN</span> option is turned on, submit the problem to the appropriate support channel and include that\n      detail in the problem report. Do not otherwise run with this setting turned on, because it results in lower\n      overall performance.\n    </p><p>\n      Because the code generation phase adds a small amount of overhead for each query, you might turn on the\n      <span class=\"hue-doc-codeph\">DISABLE_CODEGEN</span> option to achieve maximum throughput when running many short-lived queries\n      against small tables.\n    </p><p><b>Type:</b> Boolean; recognized values are 1 and 0, or <span class=\"hue-doc-codeph\">true</span> and <span class=\"hue-doc-codeph\">false</span>;\n        any other value interpreted as <span class=\"hue-doc-codeph\">false</span></p><p><b>Default:</b><span class=\"hue-doc-codeph\">false</span> (shown as 0 in output of <span class=\"hue-doc-codeph\">SET</span> statement)\n      </p></div></div></div>","title":"DISABLE_CODEGEN Query Option"}
+{"body":"<div><div id=\"disable_codegen\"><div class=\"hue-doc-title\">DISABLE_CODEGEN Query Option</div><div><p>\n      This is a debug option, intended for diagnosing and working around issues that cause crashes. If a query\n      fails with an <q>illegal instruction</q> or other hardware-specific message, try setting\n      <span class=\"hue-doc-codeph\">DISABLE_CODEGEN=true</span> and running the query again. If the query succeeds only when the\n      <span class=\"hue-doc-codeph\">DISABLE_CODEGEN</span> option is turned on, submit the problem to the appropriate support channel and include that\n      detail in the problem report. Do not otherwise run with this setting turned on, because it results in lower\n      overall performance.\n    </p><p>\n      Because the code generation phase adds a small amount of overhead for each query, you might turn on the\n      <span class=\"hue-doc-codeph\">DISABLE_CODEGEN</span> option to achieve maximum throughput when running many short-lived queries\n      against small tables.\n    </p><p id=\"type_boolean\"><b>Type:</b> Boolean; recognized values are 1 and 0, or <span class=\"hue-doc-codeph\">true</span> and <span class=\"hue-doc-codeph\">false</span>;\n        any other value interpreted as <span class=\"hue-doc-codeph\">false</span></p><p id=\"default_false_0\"><b>Default:</b><span class=\"hue-doc-codeph\">false</span> (shown as 0 in output of <span class=\"hue-doc-codeph\">SET</span> statement)\n      </p></div></div></div>","title":"DISABLE_CODEGEN Query Option"}

File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_disable_codegen_rows_threshold.json


+ 1 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_disable_outermost_topn.json

@@ -0,0 +1 @@
+{"body":"<div><div id=\"disable_outermost_topn\"><div class=\"hue-doc-title\">DISABLE_OUTERMOST_TOPN Query Option</div><div><p/><p><b>Type:</b></p><p><b>Default:</b></p></div></div></div>","title":"DISABLE_OUTERMOST_TOPN Query Option"}

File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_disable_row_runtime_filtering.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_disable_streaming_preaggregations.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_disable_unsafe_spills.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_distinct.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_dml.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_double.json


+ 1 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_drop_data_source.json

@@ -0,0 +1 @@
+{"body":"<div><div style=\"display:none;\" id=\"drop_data_source\"><div class=\"hue-doc-title\">DROP DATA SOURCE Statement</div><div><p/><p id=\"syntax_blurb\"><b>Syntax:</b></p><p id=\"ddl_blurb\"><b>Statement type:</b> DDL\n      </p><p id=\"related_info\"><b>Related information:</b></p><p id=\"usage_notes_blurb\"><b>Usage notes:</b></p><p id=\"example_blurb\"><b>Examples:</b></p><p id=\"cancel_blurb_no\"><b>Cancellation:</b> Cannot be cancelled.\n      </p></div></div></div>","title":"DROP DATA SOURCE Statement"}

File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_drop_database.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_drop_function.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_drop_role.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_drop_stats.json


File diff suppressed because it is too large
+ 0 - 0
desktop/core/src/desktop/static/desktop/docs/impala/topics/impala_drop_table.json


Some files were not shown because too many files changed in this diff