Эх сурвалжийг харах

[jdbc] Add a new Kyuubi jdbc connector (#3586)

What changes were proposed in this pull request?

Create a new Kyuubi JDBC connector so that users may utilize Kyuubi to query Spark SQL and fill up Kyuubi JDBC in the hue.ini configuration file such as :

[[[kyuubi_jdbc]]]
name=Kyuubi_JDBC
interface=jdbc
options='{"url": "jdbc:hive2://xxxx:10009", "driver": "org.apache.kyuubi.jdbc.KyuubiHiveDriver", "user": "xxxx", "password": "xxxx"}'


Setting up Kyuubi + Spark SQL Test Environment:

1.Use the instructions at https://kyuubi.readthedocs.io/en/master/quick_start/ to install and launch Kyuubi.
2.Go to https://kyuubi.readthedocs.io/en/master/client/jdbc/kyuubi_jdbc.html to get the Kyuubi JDBC driver.
3.Update hue.ini with the Kyuubi JDBC interface.
4.Restart Hue after adding the Kyuubi JDBC driver path to your Java CLASSPATH.
5.Use the Hue Kyuubi JDBC Editor to run sparksql queries.

Testing Scenario:
1.Test running various SQL queries from Hue, this includes basic DDL, DML queries.

---------
Co-authored-by: Harsh Gupta <42064744+Harshg999@users.noreply.github.com>
chengduhaha 1 жил өмнө
parent
commit
94fddd7398

+ 3 - 0
desktop/libs/notebook/src/notebook/connectors/base.py

@@ -493,6 +493,9 @@ def get_api(request, snippet):
     elif interpreter['options'] and interpreter['options'].get('url', '').find('vertica') >= 0:
       from notebook.connectors.jdbc_vertica import JdbcApiVertica
       return JdbcApiVertica(request.user, interpreter=interpreter)
+    elif interpreter['options'] and interpreter['options'].get('driver', '').find('kyuubi') >= 0:
+      from notebook.connectors.jdbc_kyuubi import JdbcApiKyuubi
+      return JdbcApiKyuubi(request.user, interpreter=interpreter)
     else:
       from notebook.connectors.jdbc import JdbcApi
       return JdbcApi(request.user, interpreter=interpreter)

+ 45 - 0
desktop/libs/notebook/src/notebook/connectors/jdbc_kyuubi.py

@@ -0,0 +1,45 @@
+#!/usr/bin/env python
+# Licensed to Cloudera, Inc. under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  Cloudera, Inc. licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from librdbms.jdbc import query_and_fetch
+
+from notebook.connectors.jdbc import JdbcApi
+from notebook.connectors.jdbc import Assist
+
+class JdbcApiKyuubi(JdbcApi):
+
+  def _createAssist(self, db):
+    return KyuubiAssist(db)
+
+class KyuubiAssist(Assist):
+
+  def get_databases(self):
+    dbs, description = query_and_fetch(self.db, 'SHOW DATABASES')
+    return [db[0] and db[0].strip() for db in dbs]
+
+  def get_tables_full(self, database, table_names=[]):
+    tables, description = query_and_fetch(self.db, "SHOW TABLES IN %s" % database)
+    return [{"comment": '', "type": "Table", "name": table[1] and table[1].strip()} for table in tables]
+
+  def get_columns_full(self, database, table):
+    columns, description = query_and_fetch(self.db, "DESCRIBE %s.%s" % (database, table))
+    return [{"comment": col[2] and col[2].strip(), "type": col[1], "name": col[0] and col[0].strip()} for col in columns]
+
+  def get_sample_data(self, database, table, column=None):
+    column = column or '*'
+    return query_and_fetch(self.db, 'SELECT %s FROM %s.%s limit 100' % (column, database, table))
+