Quellcode durchsuchen

[docs] Adding current design docs

Romain vor 5 Jahren
Ursprung
Commit
4361be4923

+ 44 - 0
docs/designs/connectors.md

@@ -0,0 +1,44 @@
+# Connectors
+
+Goal: [HUE-8758](https://issues.cloudera.org/browse/HUE-8758)
+Configuration of external services via an interface and API instead of hue.ini.
+
+
+## Apps and Connectors
+
+In Hue 4, apps and dialects are combined into the same configuration.
+
+Desktop + apps and Connectors
+
+* Apps: editor, notebook, filebrowser, jobbrowser, metastore, importer, home
+* Connectors: impala, hive, hive-llap, mysql...
+* Connections: Impala Cluster1, MySql Analytics
+
+Each app is using instances of connectors which represent a connection. For example the Editor App
+can point to several MySql servers.
+
+## Permissions
+
+Before there was app level permissions, and one language (e.g. Hive) was associated to an app.
+
+Now the goal is to be able to add your own connector instance in a self service way without any Hue restart.
+The instance is restricted with a corresponding Permission object.
+
+* HuePermission --> ConnectorPermission / OrganizationConnectorPermission
+
+**Note**
+Out of scope but to keep in mind
+
+* AWS, Apache ranger permissions: HuePermission --> OrganizationAwsPermission, OrganizationRangerPermission ...
+* HuePermission (e.g. specific settings to assign to a group, like download limit, superuser group...)
+
+## Connectors
+
+### HMS
+
+Activate "Tables" app when there is at least one.
+Add "catalog" connectors. e.g. SQL connectors, HMS, Metadata catalogs...
+
+### Storages
+
+Multiple S3s configs, need "namespaces" or list of keys.

+ 204 - 0
docs/designs/multi-cluster.md

@@ -0,0 +1,204 @@
+# Multi Cluster Hue
+
+Goal: clusters of configurations of external services
+
+* Multi clusters [HUE-8330](https://issues.cloudera.org/browse/HUE-8330)
+
+## In the classic world
+
+###  Cluster with no Hive, just Impala
+
+Create a 'apps/hive', when this one is blacklisted, there is no Hive Editor
+'apps/beeswax' is seen like the HiveServer2 lib, and so should never be blacklisted (one day to move to libs). Impala, HMS also depends on it.
+Add HMS configuration in the 'apps/metastore'
+
+To work with CM to have them blacklist 'hive' instead of 'beeswax' now. To also have them provide the HMS info into [metastore].
+
+One trick is that the HMS should not be seen as an interpreter, but more like a namespace.
+
+## In the new world
+
+Multi tenant, multi cluster. In practice, 1 cluster means CM instance, 1 GCP Project, 1 Altus instance, 1 Azure resource (less ambiguity with "computes" this way).
+
+e.g.
+
+- CM instance with 1 base storage and 2 compute only clusters. Each compute cluster can consist of combos of at least 1 HS2 LLAP, 1 HS2 batch, 1 Impala services.
+- Altus SaaS instance, we have one Data warehouse service with 2 possible engines
+- GCP instance: we have one Project with one BigQuery and File storage
+- AWS instance: we have a general Glue, Athena, S3 and combinations of Redshifts. Even S3 and Athenas could be multiple with services. Each compute services is tights to the storages.
+
+hue.ini
+
+    [desktop]
+
+      [[tenants]]
+
+        [[[[extra]]]]
+
+        [[[clusters]]]
+
+          [[[[storages]]]]
+
+          [[[[interpreters]]]]
+
+          [[[[extra]]]]
+
+**[extra]** TBD stuff. Could be several sections too of global stuff, like Hue specific stuff, common service catalogs / NavOpts, schedulers...
+**note** We have default tenant and cluster to work with the new format. Work to do to par app configs in the new sub-sections instead of classic top sections like [search], [impala]... but this is the way to go. Will be done without braking the current classic way of writing in the full ini.
+
+Each instance of the sections:
+
+- tenants
+- clusters
+
+can also be edited via the UI interface [HUE-8758](https://issues.cloudera.org/browse/HUE-8769) and saved into 2 new tables. In practice either the ini or the UI should be used (or read only if config coming from ini). It is simpler to start with interpreters (similar to Superset, Periscope, e.g. https://superset.incubator.apache.org/installation.html#deeper-sqlalchemy-integration).
+
+
+## Mockups
+
+1 CM multi cluster from above
+3 interpreters (1 HS2 LLAP, 1 HS2 batch, 1 Impal), multi cluster dropdown hidden.
+Can browse Catalog via HMS or the 3 other interpreters
+Can browse the base HDFS.
+
+1 DP2 Hive & Impala
+2 interpreters (1 Hive, 1 Impala), multi cluster listing computes of each one.
+Can browse Catalog via HMS or any running computes (to get samples)
+Can browse each S3 of each namespace.
+
+The **concept** is to drive the flow via the metadata (top search, left assist...), where selecting a namespace automatically restricts the computes (and if possible even hides their concept). e.g. like we do when selecting Impala Editor, we auto switch the left assist to the Impala view of table. It would be the same for all the data/metadata (tables, files,...).
+
+
+    [desktop]
+
+    enabled_connectors = ['hive-tez', 'impala', 'hdfs'....]
+    desktop/connectors Python class? with fields to display in UI
+
+    [clusters]
+
+      [[cm-classic]]
+
+        # One section by connector API category: notebook, catalog, optimizer, browsers, scheduler, apps
+
+        [[[connectors]]   ## Classes
+
+          [[[[hive]]]]
+          class=query
+          interface=hiveserver2
+          type=hive-tez
+
+          # r/w pemrs to generate: DWX? create-cluster perm to pull?
+
+            [[[[hive-Tez]]]] ## Instances
+            server_host=...
+            # beeswax config?
+            [[[[hive-Tez2]]]]
+
+          [[[[[impala]]]]]
+          class=query
+          type=impala
+          interface=hiveserver
+
+          [[[[[solr]]]]]
+          type=query
+          interface=solr
+
+          [[[[[impala-3]]]]]
+
+            ## instance 3
+            ## instance 4
+
+        [[[hms]]]
+        type=browser
+        interface=hiveserver2
+
+
+          [[[[[[hdfs]]]]]]
+
+          [[[[[[yarn]]]]]]
+
+        [[[[[metadata]]]]]]
+
+            [[[[[[atlas]]]]]]
+
+      [[[[cm-mutli]]]]
+
+        [[[[[interpreters]]]]]   ## Classes
+
+          [[[[[[[hive-tez]]]]]]
+
+                                 ## Instances
+
+          [[[[[[hive-llap]]]]]]
+
+          [[[[[[impala-1]]]]]]
+
+          [[[[[[impala-2]]]]]]
+
+          [[[[[[impala-3]]]]]]
+
+            ## instance 3
+            ## instance 4
+
+        [[[[[browsers]]]]]]
+
+          [[[[[[hms]]]]]]
+
+          [[[[[[hdfs]]]]]]
+
+          [[[[[[yarn]]]]]]
+
+        [[[[[metadata]]]]]]
+
+            [[[[[[atlas]]]]]]
+
+      [[[snowball]]]
+
+        [[[[interpreters]]]]
+
+          [[[[[hive-llap]]]]]
+
+          [[[[[impala]]]]]
+
+          [[[[[druid]]]]]
+
+        [[[[browsers]]]]]
+
+          [[[[s3]]]]
+
+        [[[[[metadata]]]]]]
+
+            [[[[[[atlas]]]]]]
+
+            [[[[[[ranger]]]]]]
+
+
+      [[[aws]]]
+
+          [[[[interpreters]]]]
+
+            [[[[[redshift-1]]]]]
+
+            [[[[[redshift-2]]]]]
+
+            [[[[[athena]]]]]
+
+          [[[[browsers]]]]]
+
+            [[[[s3]]]]
+
+          [[[[metadata]]]]]
+
+              [[[[[glue]]]]]
+
+      [[[gcp]]]
+
+          [[[[interpreters]]]]
+
+            [[[[[bigquery-1]]]]]
+
+            [[[[[bigquery-2]]]]]
+
+          [[[[browsers]]]]]
+
+            [[[[gfs]]]]

+ 5 - 0
docs/designs/query_result_handling.md

@@ -0,0 +1,5 @@
+
+We could do auto limit 5000  at the result fetch and/or even SQL (optional). Then we could have warnings when we know we have more and the user wants to see/download, and he could extend the limit.
+
+- Caching service interface: e.g. https://docs.djangoproject.com/en/2.2/topics/cache/#filesystem-caching
+- Format: Parquet via Panda?

+ 43 - 0
docs/designs/scheduling.md

@@ -0,0 +1,43 @@
+
+# Scheduling
+
+Goal: allow easy task scheduling
+
+* Query scheduling
+  * Notification failure/success
+  * Results
+    * Attached
+    * Link
+  * Report
+* Directly from right assist
+
+## Job Browser
+
+* Manage schedules (Hue DB / ORM)
+* Manage scheduled tasks (Flower)
+
+## Beat
+
+* Configure persistence to not miss events
+* Task needs user / document id
+
+Reuse Coordinator panel. Save into query?
+
+
+## Tasks
+
+### Query
+
+Parameters
+
+* User Id
+* Query Id
+
+* All queries or one selected
+
+* Batch mode
+
+## API
+
+* Oozie
+* Beat

+ 34 - 0
docs/designs/sharing.md

@@ -0,0 +1,34 @@
+
+# Sharing
+
+* Fork?
+* Slack
+
+* Share query with query/result/chart to other tools
+* Export to Slack, Google spreadsheet
+
+* Copy paste result into email, wiki, slack
+
+
+## Share button
+
+Share button available all the time (except empty query?)
+
+People: perms documents attached to a saved document
+Public link:
+Gist: text document public
+
+* Saved query --> share as publick link or people
+* Query History --> share as public link
+* Query Selection --> share as gist link
+
+* Email notification when adding user or group
+
+## Document
+
+* Show on hover [i] icon, not click
+
+Stats
+
+* Home page query stats and schedules
+* Single query stats in Editor in saved query

+ 0 - 0
docs/designs/sql_alchemy.md