瀏覽代碼

Importer fails with READ permission due to IMPALA-10272. (#2374)

The only way to over come this is to have the scratch dir and its underlying file to be of 777.

Exception seen when the load command fails:
QueryError: AnalysisException: Unable to LOAD DATA from hdfs://<HDFS_PATH>/.scratchdir/6749117b-f371-4bac-a2aa-b06cb45330e3/test_data.csv because Impala does not have READ permissions on this file

This fix was tested out at a customer end which worked.

PR: 2374
Mahesh Balakrishnan 4 年之前
父節點
當前提交
048b8b8607

+ 8 - 0
apps/impala/src/impala/conf.py

@@ -118,6 +118,14 @@ CONFIG_WHITELIST = Config(
   help=_t('A comma-separated list of white-listed Impala configuration properties that users are authorized to set.')
 )
 
+USER_SCRATCH_DIR_PERMISSION = Config(
+  key="user_scratch_dir_permission",
+  help=_t("Due to IMPALA-10272, the importer fails with READ permissions."
+          "Setting this to True, means setting the scratch directory and its file to 777 so the importer does not fail with permission issue."),
+  type=coerce_bool,
+  default=False
+)
+
 IMPALA_CONF_DIR = Config(
   key='impala_conf_dir',
   help=_t('Impala configuration directory, where impala_flags is located.'),

+ 4 - 0
desktop/conf.dist/hue.ini

@@ -1455,6 +1455,10 @@
   # Use Thrift over HTTP for the transport mode.
   ## use_thrift_http=false
 
+  # Due to IMPALA-10272, the importer fails with READ permissions.
+  # Setting this to True, means setting the scratch directory and its file to 777 so the importer does not fail with permission issue
+  ## user_scratch_dir_permission=false
+
   [[ssl]]
     # SSL communication enabled for this server.
     ## enabled=false

+ 4 - 0
desktop/conf/pseudo-distributed.ini.tmpl

@@ -1440,6 +1440,10 @@
   # Use Thrift over HTTP for the transport mode.
   ## use_thrift_http=false
 
+  # Due to IMPALA-10272, the importer fails with READ permissions.
+  # Setting this to True, means setting the scratch directory and its file to 777 so the importer does not fail with permission issue
+  ## user_scratch_dir_permission=false
+
   [[ssl]]
     # SSL communication enabled for this server.
     ## enabled=false

+ 3 - 0
desktop/libs/indexer/src/indexer/indexers/sql.py

@@ -32,6 +32,7 @@ from hadoop.fs.hadoopfs import Hdfs
 from notebook.connectors.base import get_interpreter
 from notebook.models import make_notebook
 from useradmin.models import User
+from impala.conf import USER_SCRATCH_DIR_PERMISSION
 
 from desktop.lib import django_mako
 from desktop.lib.exceptions_renderable import PopupException
@@ -169,6 +170,8 @@ class SQLIndexer(object):
         user_scratch_dir = self.fs.get_home_dir() + '/.scratchdir/%s' % str(uuid.uuid4()) # Make sure it's unique.
         self.fs.do_as_user(self.user, self.fs.mkdir, user_scratch_dir, 0o0777)
         self.fs.do_as_user(self.user, self.fs.rename, source['path'], user_scratch_dir)
+        if USER_SCRATCH_DIR_PERMISSION.get():
+          self.fs.do_as_user(self.user, self.fs.chmod, user_scratch_dir, 0o0777, True)
         source_path = user_scratch_dir + '/' + source['path'].split('/')[-1]
 
     if external_path.lower().startswith("abfs"): #this is to check if its using an ABFS path