Pārlūkot izejas kodu

HUE-8888 [docs] Refactoring of the connector developer section

Romain 5 gadi atpakaļ
vecāks
revīzija
dbc78f2bef

+ 5 - 1
docs/ROADMAP.md

@@ -3,6 +3,10 @@
 High Level Roadmap
 ------------------
 
+* Connectors
+  * Better [connectors](https://docs.gethue.com/administrator/configuration/connectors/)
+  * Potential [connectors ideas](https://docs.gethue.com/developer/connectors/)
+
 * Editor
   * [ ] Editor v2 [HUE-8768](https://issues.cloudera.org/browse/HUE-8768)
   * [ ] Query Browser v2
@@ -21,7 +25,6 @@ High Level Roadmap
   * [ ] Configuration of connectors via UI [HUE-8758](https://issues.cloudera.org/browse/HUE-8758)
   * [ ] Multi clusters [HUE-8330](https://issues.cloudera.org/browse/HUE-8330)
   * [ ] Multi tenants [HUE-8530](https://issues.cloudera.org/browse/HUE-8530)
-  * [ ] Modernize CI/CD [HUE-8888](https://issues.cloudera.org/browse/HUE-8888)
   * [ ] SQL queries and requests tracing [HUE-8936](https://issues.cloudera.org/browse/HUE-8936)
   * [ ] Productionize metrics [HUE-9021](https://issues.cloudera.org/browse/HUE-9021)
   * [ ] Google Storage connector [HUE-8978](https://issues.cloudera.org/browse/HUE-8978)
@@ -51,6 +54,7 @@ Done
 * [x] Documentation refresh [HUE-8741](https://issues.cloudera.org/browse/HUE-8741)
 * [x] Frontend moved to Webpack [HUE-8687](https://issues.cloudera.org/browse/HUE-8687)
 * [x] SQL Alchemy full support [HUE-8740](https://issues.cloudera.org/browse/HUE-8740)
+* [x] Modernize CI/CD [HUE-8888](https://issues.cloudera.org/browse/HUE-8888)
 
 Links
 

+ 5 - 1
docs/docs-site/content/administrator/configuration/connectors/_index.md

@@ -5,11 +5,15 @@ draft: false
 weight: 2
 ---
 
+All of these connectors come out of the box.
+
+Looking at improving or adding a new one? Go check the **[connector API section](/developer/connectors/)**!
+
 ## Databases
 
 Hue connect to any database or warehouse via native connectors or SqlAlchemy.
 
-Read about building some [better autocompletes](/developer/parsers/) or extending the connectors with SQL Alchemy, JDBC or building your own [connectors](/developer/sdk).
+Read about [how to build your own parser](/developer/parsers/) if you are looking at better autocompletes for your own SQL dialects.
 
 ### Apache Impala
 

+ 2 - 2
docs/docs-site/content/developer/api/_index.md

@@ -52,9 +52,9 @@ Once the request is successful then capture headers and cookies for subsequent r
 
 ### Data Catalog
 
-The [metadata API](https://github.com/cloudera/hue/tree/master/desktop/libs/metadata) is powering [Search and Tagging here](/user/browsing/#data-catalogs).
+The [metadata API](https://github.com/cloudera/hue/tree/master/desktop/libs/metadata) is powering the external [Catalog integrations](/user/browsing/#data-catalogs).
 
-See the backends API in the [data catalog connector](/developer/connectors/#data-catalog) section.
+Additional catalogs can be integrated via some [connectors](/developer/connectors/#data-catalog).
 
 #### Searching for entities
 

+ 52 - 88
docs/docs-site/content/developer/connectors/_index.md

@@ -5,82 +5,64 @@ draft: false
 weight: 3
 ---
 
-They provide integration with any SQL database or Job execution engine. Here is a list of the [existing connectors](https://github.com/cloudera/hue/tree/master/desktop/libs/notebook/src/notebook/connectors).
+Connectors provide pluggable integration to any external data service so that an admin can easily allow end users to interact with them.
 
-Connectors are pluggable and new engines can be added. Feel free to contact the [community](https://discourse.gethue.com/c/developer-sdk-api).
+* List of all the [existing connectors](/administrator/configuration/connectors/)
+* Check "Potential connectors" ideas in each section
+* Feel free to contact the [community](https://discourse.gethue.com/c/developer-sdk-api)
 
-## Querying
+## Databases
 
-### SQL
+### SqlAlchemy
 
-#### SqlAlchemy
+[SqlAlchemy](https://www.sqlalchemy.org) is the prefered way if the Hive API is not supported by the database. The core implementation is in [`sql_alchemy.py`](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/sql_alchemy.py) and relies on each repective SqlAlchemy dialect.
 
-[SqlAlchemy](https://www.sqlalchemy.org) is the prefered way if the HiveServer2 API is not supported by the database. The implementation is in [`sql_alchemy.py`](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/sql_alchemy.py) and is depends on the repective SqlAlchemy dialects.
+### Hive Interface
 
-#### Kafka SQL
+This [asynchronous API](https://github.com/cloudera/hue/tree/master/apps/beeswax) based on the Thrift API of Hive is very mature and powers an excellent integration of Apache Hive and Apache Impala.
 
-[Kafka connector](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/ksql.py).
-
-#### Solr SQL
-
-[Solr connector](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/solr.py).
-
-#### Custom
+### Custom
 
 If the built-in HiveServer2 (Hive, Impala, Spark SQL), RDBMS (MySQL, PostgreSQL, Oracle, SQLite), and JDBC interfaces don’t meet your needs, you can implement your own connector to the notebook app:
 
-* List of the existing [Notebook Connectors](https://github.com/cloudera/hue/tree/master/desktop/libs/notebook/src/notebook/connectors)
-* Each connector API subclasses the [Base API](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/base.py) and must implement the methods defined within
-* Refer to the [JdbcApi](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/jdbc.py) or [RdbmsApi](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/rdbms.py) for representative examples
-
-#### JDBC
-
-With the JDBC proxy, query editor with any JDBC compatible database. View the [JDBC connector](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/jdbc.py).
-
-**Note** In the long term, SqlAlchemy is prefered as more "Python native".
-
-
-### Jobs
-
-#### Spark / Livy
-
-Based on the [Livy REST API](/administrator/configuration/connectors/#apache-spark).
+* List of [all connectors](https://github.com/cloudera/hue/tree/master/desktop/libs/notebook/src/notebook/connectors)
+* Each connector API subclasses the [Base API](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/base.py) and must implement the methods defined within. Refer to the [JDBC](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/jdbc.py) or [RdbmsApi](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/rdbms.py) for representative examples
 
-* [Notebook connector](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/spark_shell.py)
-  * PySpark
-  * Scala
-  * Spark SQL
-* [Batch connector](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/spark_batch.py)
+* [Kafka SQL](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/ksql.py)
+* [Solr SQL](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/solr.py)
+* [JDBC](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/jdbc.py)
 
-#### Oozie
+The JDBC API relies on a small JDBC proxy running next to the Hue API. By default it won't be built without setting the `BUILD_DB_PROXY` flag, e.g.:
 
-MapReduce, Pig, Java, Shell, Sqoop, DistCp [Oozie connector](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/oozie_batch.py).
+    export BUILD_DB_PROXY=true make install
 
+**Note** In the long term, SqlAlchemy is prefered as more "Python native".
 
-## Job Browser
-
-The Job Browser is generic and can list any type of jobs, queries and provide bulk operations like kill, pause, delete... and access to logs and recommendations.
+### Potential connectors
 
-Here is its [API](https://github.com/cloudera/hue/tree/master/apps/jobbrowser/src/jobbrowser/apis).
+It is recommended to develop an SqlAlchemy connector if yours is not already [existing](/administrator/configuration/connectors/#databases).
 
-#### SQL Queries
+## Catalogs
 
-The API currently supports:
+The backends is pluggable by providing alternative [client interfaces](https://github.com/cloudera/hue/tree/master/desktop/libs/metadata/src/metadata/catalog):
 
-* [Apache Impala](https://github.com/cloudera/hue/blob/master/apps/jobbrowser/src/jobbrowser/apis/query_api.py)
-* [Apache Hive](https://github.com/cloudera/hue/blob/master/apps/jobbrowser/src/jobbrowser/apis/beeswax_query_api.py)
+* [Apache Atlas](https://atlas.apache.org/)
+* Cloudera Navigator
+* Dummy (skeleton for integrating new catalogs)
 
-#### Spark / Livy
+### Apache Atlas
 
-* [Livy API](https://github.com/cloudera/hue/blob/master/apps/jobbrowser/src/jobbrowser/apis/livy_api.py)
+* [Client API](desktop/libs/metadata/src/metadata/catalog/atlas_client.py)
 
-#### Oozie
+### Potential connectors
 
-* [Workflow API](https://github.com/cloudera/hue/blob/master/apps/jobbrowser/src/jobbrowser/apis/workflow_api.py)
-* [Coordinators API](https://github.com/cloudera/hue/blob/master/apps/jobbrowser/src/jobbrowser/apis/schedule_api.py)
-* [Bundles API](https://github.com/cloudera/hue/blob/master/apps/jobbrowser/src/jobbrowser/apis/bundle_api.py)
+* [Linkedin DataHub](https://github.com/linkedin/datahub)
+* [Lift Amundsen](https://github.com/lyft/amundsen)
+* AWS Glue
+* Google Cloud Data Catalog
+* Alation
 
-## File Browser
+## Storages
 
 Various storage systems can be interacted with. The [`fsmanager.py`](https://github.com/cloudera/hue/blob/master/desktop/core/src/desktop/lib/fsmanager.py) is the main router to each API.
 
@@ -101,49 +83,31 @@ Various storage systems can be interacted with. The [`fsmanager.py`](https://git
 
 ### HBase / Key Value Stores
 
-With just a few changes in the [Python API](https://github.com/cloudera/hue/blob/master/apps/hbase/src/hbase/api.py),
-the HBase browser could be compatible with Apache Kudu or Google Big Table.
-
-## Dashboard
-
-[Dashboards](/user/querying/#dashboards) are generic and support Apache Solr and SQL:
-
-The API was influenced by Solr but is now generic:
-
-[Dashboard API](https://github.com/cloudera/hue/blob/master/desktop/libs/dashboard/src/dashboard/dashboard_api.py)
-
-### SQL
-
-[SQL API](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/dashboard_api.py)
-
-Implementations:
-
-* [Impala API](https://github.com/cloudera/hue/blob/master/apps/impala/src/impala/dashboard_api.py)
-* [Hive API](https://github.com/cloudera/hue/blob/master/apps/beeswax/src/beeswax/dashboard_api.py)
-
-### Apache Solr
-
-[Solr Dashboard API](https://github.com/cloudera/hue/blob/master/apps/search/src/search/dashboard_api.py)
+With just a few changes in the [Python API](https://github.com/cloudera/hue/blob/master/apps/hbase/src/hbase/api.py), the HBase browser could be compatible with Apache Kudu or Google Big Table.
 
-### Elastic Search
+### Potential connectors
 
-A connector similar to Solr or SQL Alchemy binding would need to be developed [HUE-7828](https://issues.cloudera.org/browse/HUE-7828).
+* Google Cloud Storage is currently a work in progress with [HUE-8978](https://issues.cloudera.org/browse/HUE-8978)
 
-## Data Catalog
+## Jobs
 
-The backends is pluggable by providing alternative [client interfaces](https://github.com/cloudera/hue/tree/master/desktop/libs/metadata/src/metadata/catalog):
-
-* Cloudera Navigator (default)
-* Dummy (skeleton for integrating new catalogs)
+### Apache Spark / Livy
 
-### Apache Atlas
+Based on the [Livy REST API](/administrator/configuration/connectors/#apache-spark).
 
-* [Client API](desktop/libs/metadata/src/metadata/catalog/atlas_client.py)
+* [Notebook connector](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/spark_shell.py)
+  * PySpark
+  * Scala
+  * Spark SQL
+* [Batch connector](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/spark_batch.py)
 
-## Scheduling
+### Schedulers
 
-### Oozie
+Currently only Apache Oozie is supported for your Datawarehouse, but the API is getting generic with [HUE-3797](https://issues.cloudera.org/browse/HUE-3797).
 
-Currently only Apache Oozie is supported for your Datawarehouse, but the API is getting generic with [HUE-3797](https://issues.cloudera.org/browse/HUE-3797) that is bringing Celery Beat integration.
+### Potential connectors
 
-* [API](https://github.com/cloudera/hue/blob/master/desktop/core/src/desktop/lib/scheduler/lib/beat.py)
+* Elastic Search: a connector similar to Solr for Searching [HUE-7828](https://issues.cloudera.org/browse/HUE-7828). SQL querying is already supported.
+* [Livy Browser API](https://github.com/cloudera/hue/blob/master/apps/jobbrowser/src/jobbrowser/apis/livy_api.py)
+* [Celery API](https://github.com/cloudera/hue/blob/master/desktop/core/src/desktop/lib/scheduler/lib/beat.py)
+* Apache Hive native support of query scheduling [HIVE-21884](https://issues.apache.org/jira/browse/HIVE-21884)

+ 4 - 6
docs/docs-site/content/user/browsing/_index.md

@@ -5,9 +5,9 @@ draft: false
 weight: 3
 ---
 
-Browsers power the Data Catalog. They let you easily search, glance and perform actions on data or jobs in Cloud or on premise clusters.
+Browsers power the Data Catalog. They let you easily search, glance, import datasets or jobs.
 
-The browsers can be "enriched" with [Search and Tagging](http://gethue.com/improved-sql-exploration-in-hue-4-3/) by metadata services.
+The browsers can be "enriched" with external [catalog/metadata services](/administrator/configuration/connectors/#storage/).
 
 ## Catalogs
 
@@ -15,7 +15,7 @@ The browsers can be "enriched" with [Search and Tagging](http://gethue.com/impro
 
 The Table Browser enables you to manage the databases, tables, and partitions of the metastore shared by the Hive and Impala. You can perform the following operations:
 
--   Search and display metadata like tags and additional description from [Catalog backends](/administrator/configuration/connectors/#storage/).
+-   Search and display metadata like tags and additional description
 
 -   Databases
     -   Select a database
@@ -32,12 +32,10 @@ The Table Browser enables you to manage the databases, tables, and partitions of
 
 ### Data Catalogs
 
-[Apache Atlas](https://atlas.apache.org/) is powering the Search and Commenting of tables, columns. New Catalogs can be integrated via [connectors](/developer/connectors/).
+Before typing any query to get insights, users need to find and explore the correct datasets.  It is accessible from the top bar of the interface and offers free text search of SQL tables, columns, tags and saved queries. This is particularly useful for quickly looking up a table among thousands or finding existing queries already analyzing a certain dataset.
 
 Existing tags, descriptions and indexed objects show up automatically, any additional tags you add appear back in metadata server, and the familiar metadata server search syntax is supported.
 
-Before typing any query to get insights, users need to find and explore the correct datasets. The Data Catalog search usability experience has been improved in each release since. It is accessible from the top bar of the interface and offers free text search of SQL tables, columns, tags and saved queries. This is particularly useful for quickly looking up a table among thousands or finding existing queries already analyzing a certain dataset.
-
 ![Data Catalog top search](https://cdn.gethue.com/uploads/2018/04/blog_top_search_.png)
 
 Searching all the available queries or data in the cluster