浏览代码

HUE-5169 [spark] cannot spark submit jar or python

https://github.com/cloudera/hue/pull/441

cannot spark submit jar or python due to incorrect params to create livy session.
The Spark Livy create batch session api does not accept 'pyFiles' == null or 'files' == null params.
Li Jiahong 9 年之前
父节点
当前提交
232903e
共有 1 个文件被更改,包括 24 次插入13 次删除
  1. 24 13
      desktop/libs/notebook/src/notebook/connectors/spark_batch.py

+ 24 - 13
desktop/libs/notebook/src/notebook/connectors/spark_batch.py

@@ -33,19 +33,30 @@ class SparkBatchApi(Api):
 
   def execute(self, notebook, snippet):
     api = get_spark_api(self.user)
-
-    properties = {
-        'file': snippet['properties'].get('app_jar'),
-        'className': snippet['properties'].get('class'),
-        'args': snippet['properties'].get('arguments'),
-        'pyFiles': snippet['properties'].get('py_file'),
-        'files': snippet['properties'].get('files'),
-        # driverMemory
-        # driverCores
-        # executorMemory
-        # executorCores
-        # archives
-    }
+    if snippet['type'] == 'jar':
+        properties = {
+            'file': snippet['properties'].get('app_jar'),
+            'className': snippet['properties'].get('class'),
+            'args': snippet['properties'].get('arguments'),
+        }
+    elif snippet['type'] == 'py':
+        properties = {
+            'file': snippet['properties'].get('py_file'),
+            'args': snippet['properties'].get('argument'),
+        }
+    else:
+        properties = {
+            'file': snippet['properties'].get('app_jar'),
+            'className': snippet['properties'].get('class'),
+            'args': snippet['properties'].get('arguments'),
+            'pyFiles': snippet['properties'].get('py_file'),
+            'files': snippet['properties'].get('files'),
+            # driverMemory
+            # driverCores
+            # executorMemory
+            # executorCores
+            # archives
+        }
 
     response = api.submit_batch(properties)
     return {