|
|
@@ -5,82 +5,64 @@ draft: false
|
|
|
weight: 3
|
|
|
---
|
|
|
|
|
|
-They provide integration with any SQL database or Job execution engine. Here is a list of the [existing connectors](https://github.com/cloudera/hue/tree/master/desktop/libs/notebook/src/notebook/connectors).
|
|
|
+Connectors provide pluggable integration to any external data service so that an admin can easily allow end users to interact with them.
|
|
|
|
|
|
-Connectors are pluggable and new engines can be added. Feel free to contact the [community](https://discourse.gethue.com/c/developer-sdk-api).
|
|
|
+* List of all the [existing connectors](/administrator/configuration/connectors/)
|
|
|
+* Check "Potential connectors" ideas in each section
|
|
|
+* Feel free to contact the [community](https://discourse.gethue.com/c/developer-sdk-api)
|
|
|
|
|
|
-## Querying
|
|
|
+## Databases
|
|
|
|
|
|
-### SQL
|
|
|
+### SqlAlchemy
|
|
|
|
|
|
-#### SqlAlchemy
|
|
|
+[SqlAlchemy](https://www.sqlalchemy.org) is the prefered way if the Hive API is not supported by the database. The core implementation is in [`sql_alchemy.py`](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/sql_alchemy.py) and relies on each repective SqlAlchemy dialect.
|
|
|
|
|
|
-[SqlAlchemy](https://www.sqlalchemy.org) is the prefered way if the HiveServer2 API is not supported by the database. The implementation is in [`sql_alchemy.py`](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/sql_alchemy.py) and is depends on the repective SqlAlchemy dialects.
|
|
|
+### Hive Interface
|
|
|
|
|
|
-#### Kafka SQL
|
|
|
+This [asynchronous API](https://github.com/cloudera/hue/tree/master/apps/beeswax) based on the Thrift API of Hive is very mature and powers an excellent integration of Apache Hive and Apache Impala.
|
|
|
|
|
|
-[Kafka connector](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/ksql.py).
|
|
|
-
|
|
|
-#### Solr SQL
|
|
|
-
|
|
|
-[Solr connector](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/solr.py).
|
|
|
-
|
|
|
-#### Custom
|
|
|
+### Custom
|
|
|
|
|
|
If the built-in HiveServer2 (Hive, Impala, Spark SQL), RDBMS (MySQL, PostgreSQL, Oracle, SQLite), and JDBC interfaces don’t meet your needs, you can implement your own connector to the notebook app:
|
|
|
|
|
|
-* List of the existing [Notebook Connectors](https://github.com/cloudera/hue/tree/master/desktop/libs/notebook/src/notebook/connectors)
|
|
|
-* Each connector API subclasses the [Base API](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/base.py) and must implement the methods defined within
|
|
|
-* Refer to the [JdbcApi](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/jdbc.py) or [RdbmsApi](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/rdbms.py) for representative examples
|
|
|
-
|
|
|
-#### JDBC
|
|
|
-
|
|
|
-With the JDBC proxy, query editor with any JDBC compatible database. View the [JDBC connector](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/jdbc.py).
|
|
|
-
|
|
|
-**Note** In the long term, SqlAlchemy is prefered as more "Python native".
|
|
|
-
|
|
|
-
|
|
|
-### Jobs
|
|
|
-
|
|
|
-#### Spark / Livy
|
|
|
-
|
|
|
-Based on the [Livy REST API](/administrator/configuration/connectors/#apache-spark).
|
|
|
+* List of [all connectors](https://github.com/cloudera/hue/tree/master/desktop/libs/notebook/src/notebook/connectors)
|
|
|
+* Each connector API subclasses the [Base API](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/base.py) and must implement the methods defined within. Refer to the [JDBC](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/jdbc.py) or [RdbmsApi](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/rdbms.py) for representative examples
|
|
|
|
|
|
-* [Notebook connector](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/spark_shell.py)
|
|
|
- * PySpark
|
|
|
- * Scala
|
|
|
- * Spark SQL
|
|
|
-* [Batch connector](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/spark_batch.py)
|
|
|
+* [Kafka SQL](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/ksql.py)
|
|
|
+* [Solr SQL](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/solr.py)
|
|
|
+* [JDBC](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/jdbc.py)
|
|
|
|
|
|
-#### Oozie
|
|
|
+The JDBC API relies on a small JDBC proxy running next to the Hue API. By default it won't be built without setting the `BUILD_DB_PROXY` flag, e.g.:
|
|
|
|
|
|
-MapReduce, Pig, Java, Shell, Sqoop, DistCp [Oozie connector](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/oozie_batch.py).
|
|
|
+ export BUILD_DB_PROXY=true make install
|
|
|
|
|
|
+**Note** In the long term, SqlAlchemy is prefered as more "Python native".
|
|
|
|
|
|
-## Job Browser
|
|
|
-
|
|
|
-The Job Browser is generic and can list any type of jobs, queries and provide bulk operations like kill, pause, delete... and access to logs and recommendations.
|
|
|
+### Potential connectors
|
|
|
|
|
|
-Here is its [API](https://github.com/cloudera/hue/tree/master/apps/jobbrowser/src/jobbrowser/apis).
|
|
|
+It is recommended to develop an SqlAlchemy connector if yours is not already [existing](/administrator/configuration/connectors/#databases).
|
|
|
|
|
|
-#### SQL Queries
|
|
|
+## Catalogs
|
|
|
|
|
|
-The API currently supports:
|
|
|
+The backends is pluggable by providing alternative [client interfaces](https://github.com/cloudera/hue/tree/master/desktop/libs/metadata/src/metadata/catalog):
|
|
|
|
|
|
-* [Apache Impala](https://github.com/cloudera/hue/blob/master/apps/jobbrowser/src/jobbrowser/apis/query_api.py)
|
|
|
-* [Apache Hive](https://github.com/cloudera/hue/blob/master/apps/jobbrowser/src/jobbrowser/apis/beeswax_query_api.py)
|
|
|
+* [Apache Atlas](https://atlas.apache.org/)
|
|
|
+* Cloudera Navigator
|
|
|
+* Dummy (skeleton for integrating new catalogs)
|
|
|
|
|
|
-#### Spark / Livy
|
|
|
+### Apache Atlas
|
|
|
|
|
|
-* [Livy API](https://github.com/cloudera/hue/blob/master/apps/jobbrowser/src/jobbrowser/apis/livy_api.py)
|
|
|
+* [Client API](desktop/libs/metadata/src/metadata/catalog/atlas_client.py)
|
|
|
|
|
|
-#### Oozie
|
|
|
+### Potential connectors
|
|
|
|
|
|
-* [Workflow API](https://github.com/cloudera/hue/blob/master/apps/jobbrowser/src/jobbrowser/apis/workflow_api.py)
|
|
|
-* [Coordinators API](https://github.com/cloudera/hue/blob/master/apps/jobbrowser/src/jobbrowser/apis/schedule_api.py)
|
|
|
-* [Bundles API](https://github.com/cloudera/hue/blob/master/apps/jobbrowser/src/jobbrowser/apis/bundle_api.py)
|
|
|
+* [Linkedin DataHub](https://github.com/linkedin/datahub)
|
|
|
+* [Lift Amundsen](https://github.com/lyft/amundsen)
|
|
|
+* AWS Glue
|
|
|
+* Google Cloud Data Catalog
|
|
|
+* Alation
|
|
|
|
|
|
-## File Browser
|
|
|
+## Storages
|
|
|
|
|
|
Various storage systems can be interacted with. The [`fsmanager.py`](https://github.com/cloudera/hue/blob/master/desktop/core/src/desktop/lib/fsmanager.py) is the main router to each API.
|
|
|
|
|
|
@@ -101,49 +83,31 @@ Various storage systems can be interacted with. The [`fsmanager.py`](https://git
|
|
|
|
|
|
### HBase / Key Value Stores
|
|
|
|
|
|
-With just a few changes in the [Python API](https://github.com/cloudera/hue/blob/master/apps/hbase/src/hbase/api.py),
|
|
|
-the HBase browser could be compatible with Apache Kudu or Google Big Table.
|
|
|
-
|
|
|
-## Dashboard
|
|
|
-
|
|
|
-[Dashboards](/user/querying/#dashboards) are generic and support Apache Solr and SQL:
|
|
|
-
|
|
|
-The API was influenced by Solr but is now generic:
|
|
|
-
|
|
|
-[Dashboard API](https://github.com/cloudera/hue/blob/master/desktop/libs/dashboard/src/dashboard/dashboard_api.py)
|
|
|
-
|
|
|
-### SQL
|
|
|
-
|
|
|
-[SQL API](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/dashboard_api.py)
|
|
|
-
|
|
|
-Implementations:
|
|
|
-
|
|
|
-* [Impala API](https://github.com/cloudera/hue/blob/master/apps/impala/src/impala/dashboard_api.py)
|
|
|
-* [Hive API](https://github.com/cloudera/hue/blob/master/apps/beeswax/src/beeswax/dashboard_api.py)
|
|
|
-
|
|
|
-### Apache Solr
|
|
|
-
|
|
|
-[Solr Dashboard API](https://github.com/cloudera/hue/blob/master/apps/search/src/search/dashboard_api.py)
|
|
|
+With just a few changes in the [Python API](https://github.com/cloudera/hue/blob/master/apps/hbase/src/hbase/api.py), the HBase browser could be compatible with Apache Kudu or Google Big Table.
|
|
|
|
|
|
-### Elastic Search
|
|
|
+### Potential connectors
|
|
|
|
|
|
-A connector similar to Solr or SQL Alchemy binding would need to be developed [HUE-7828](https://issues.cloudera.org/browse/HUE-7828).
|
|
|
+* Google Cloud Storage is currently a work in progress with [HUE-8978](https://issues.cloudera.org/browse/HUE-8978)
|
|
|
|
|
|
-## Data Catalog
|
|
|
+## Jobs
|
|
|
|
|
|
-The backends is pluggable by providing alternative [client interfaces](https://github.com/cloudera/hue/tree/master/desktop/libs/metadata/src/metadata/catalog):
|
|
|
-
|
|
|
-* Cloudera Navigator (default)
|
|
|
-* Dummy (skeleton for integrating new catalogs)
|
|
|
+### Apache Spark / Livy
|
|
|
|
|
|
-### Apache Atlas
|
|
|
+Based on the [Livy REST API](/administrator/configuration/connectors/#apache-spark).
|
|
|
|
|
|
-* [Client API](desktop/libs/metadata/src/metadata/catalog/atlas_client.py)
|
|
|
+* [Notebook connector](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/spark_shell.py)
|
|
|
+ * PySpark
|
|
|
+ * Scala
|
|
|
+ * Spark SQL
|
|
|
+* [Batch connector](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/spark_batch.py)
|
|
|
|
|
|
-## Scheduling
|
|
|
+### Schedulers
|
|
|
|
|
|
-### Oozie
|
|
|
+Currently only Apache Oozie is supported for your Datawarehouse, but the API is getting generic with [HUE-3797](https://issues.cloudera.org/browse/HUE-3797).
|
|
|
|
|
|
-Currently only Apache Oozie is supported for your Datawarehouse, but the API is getting generic with [HUE-3797](https://issues.cloudera.org/browse/HUE-3797) that is bringing Celery Beat integration.
|
|
|
+### Potential connectors
|
|
|
|
|
|
-* [API](https://github.com/cloudera/hue/blob/master/desktop/core/src/desktop/lib/scheduler/lib/beat.py)
|
|
|
+* Elastic Search: a connector similar to Solr for Searching [HUE-7828](https://issues.cloudera.org/browse/HUE-7828). SQL querying is already supported.
|
|
|
+* [Livy Browser API](https://github.com/cloudera/hue/blob/master/apps/jobbrowser/src/jobbrowser/apis/livy_api.py)
|
|
|
+* [Celery API](https://github.com/cloudera/hue/blob/master/desktop/core/src/desktop/lib/scheduler/lib/beat.py)
|
|
|
+* Apache Hive native support of query scheduling [HIVE-21884](https://issues.apache.org/jira/browse/HIVE-21884)
|