浏览代码

HUE-9277 [docs] Scope Livy Spark Sql Editor

Romain 5 年之前
父节点
当前提交
ae6711191b

+ 3 - 5
desktop/conf.dist/hue.ini

@@ -1382,19 +1382,17 @@
   # The Livy Server URL.
   ## livy_server_url=http://localhost:8998
 
-  # Configure Livy to start in local 'process' mode, or 'yarn' workers.
-  ## livy_server_session_kind=yarn
-
   # Whether Livy requires client to perform Kerberos authentication.
   ## security_enabled=false
 
   # Whether Livy requires client to use csrf protection.
   ## csrf_enabled=false
 
-  # Host of the Sql Server
+  # Host of the Spark Thrift Server
+  # https://spark.apache.org/docs/latest/sql-distributed-sql-engine.html
   ## sql_server_host=localhost
 
-  # Port of the Sql Server
+  # Port of the Spark Thrift Server
   ## sql_server_port=10000
 
   # Choose whether Hue should validate certificates received from the server.

+ 3 - 5
desktop/conf/pseudo-distributed.ini.tmpl

@@ -1367,19 +1367,17 @@
   # The Livy Server URL.
   ## livy_server_url=http://localhost:8998
 
-  # Configure Livy to start in local 'process' mode, or 'yarn' workers.
-  ## livy_server_session_kind=yarn
-
   # Whether Livy requires client to perform Kerberos authentication.
   ## security_enabled=false
 
   # Whether Livy requires client to use csrf protection.
   ## csrf_enabled=false
 
-  # Host of the Sql Server
+  # Host of the Spark Thrift Server
+  # https://spark.apache.org/docs/latest/sql-distributed-sql-engine.html
   ## sql_server_host=localhost
 
-  # Port of the Sql Server
+  # Port of the Spark Thrift Server
   ## sql_server_port=10000
 
   # Choose whether Hue should validate certificates received from the server.

+ 39 - 0
docs/designs/spark_sql_livy.md

@@ -0,0 +1,39 @@
+
+# SQL Editor for Spark SQL via Livy
+
+## Background
+
+* https://spark.apache.org/docs/latest/sql-programming-guide.html
+* https://docs.gethue.com/user/querying/#spark
+
+As detailed in the documentation, Spark SQL comes with different [connectors](https://docs.gethue.com/administrator/configuration/connectors/#apache-spark-sql). Here we will just show with Livy.
+
+[Apache Livy](https://livy.incubator.apache.org/) provides a bridge to a running Spark interpreter so that SQL, pyspark and scala snippets can be executed interactively.
+
+In the hue.ini configure the API url:
+
+    [spark]
+    # The Livy Server URL.
+    livy_server_url=http://localhost:8998
+
+And as always, make sure you have an interpreter configured:
+
+    [notebook]
+    [[interpreters]]
+    [[[sparksql]]]
+    name=Spark SQL
+    interface=livy
+
+And that's it, the editor will appear:
+
+![Hue Spark Sql Editor](https://cdn.gethue.com/uploads/2020/04/editor_spark_sql_livy.png)
+
+One advantage of using Hue is its [File Browser](https://docs.gethue.com/user/browsing/#data) for HDFS / S3 / Azure and full security.
+
+![Hue Phoenix Editor](https://cdn.gethue.com/uploads/2016/08/image2.png)
+
+### Future improvements
+
+* Database/table/column autocomplete is currently empty
+* SQL grammar autocomplete can [be extended](https://docs.gethue.com/developer/parsers/)
+* [SQL Scratchpad](https://docs.gethue.com/developer/api/#scratchpad) module to allow a mini SQL Editor popup is in progress

+ 34 - 14
docs/docs-site/content/administrator/configuration/connectors/_index.md

@@ -311,36 +311,56 @@ Alternative:
 
 ### Apache Spark SQL
 
-The dialect should be added to the Python system or Hue Python virtual environment:
+There are two ways to connect depending on your infrastructure:
+
+* [Thrift Server](https://spark.apache.org/docs/latest/sql-distributed-sql-engine.html)
+* [Apache Livy REST API](https://livy.apache.org/)
+
+#### Thrift Server
+
+Either via SqlAlchemy or HiveServer Thrift interface:
+
+With SqlAlchemy the dialect should be added to the Python system or Hue Python virtual environment:
 
     ./build/env/bin/pip install pyhive
 
 Then give Hue the information about the database source:
 
     [[[sparksql]]]
-       name = SparkSql
-       interface=sqlalchemy
-       options='{"url": "hive://user:password@host:10000/database"}'
+    name = Spark SQL
+    interface=sqlalchemy
+    options='{"url": "hive://user:password@host:10000/database"}'
+
+With the HiveServer Thrift (same as the one used by Hive and Impala so more robust depending on the use cases):
+
+    [spark]
+    # Host of the Spark Thrift Server
+    # https://spark.apache.org/docs/latest/sql-distributed-sql-engine.html
+    sql_server_host=localhost
 
-Alternatives:
+    # Port of the Spark Thrift Server
+    sql_server_port=10000
 
-Via [Apache Livy](https://livy.incubator.apache.org/):
+And make sure you have a `sparksql` interpreter configured:
 
     [[[sparksql]]]
-      name=SparkSql
-      interface=livy
+    name=Spark SQL
+    interface=hiveserver2
 
-    ...
+#### Apache Livy
+
+[Apache Livy](https://livy.incubator.apache.org/) provides a bridge to a running Spark interpreter so that SQL, pyspark and scala snippets can be executed interactively:
 
     [spark]
-      # The Livy Server URL.
-      livy_server_url=http://localhost:8998
+    # The Livy Server URL.
+    livy_server_url=http://localhost:8998
 
-Via native HiveServer2 API:
+And as always, make sure you have an interpreter configured:
 
     [[[sparksql]]]
-      name=SparkSql
-      interface=hiveserver2
+    name=Spark SQL
+    interface=livy
+
 
 ### ksqlDB
 

+ 1 - 1
docs/gethue/content/en/posts/2020-04-07-some-4-7-admin-improvements.md

@@ -74,4 +74,4 @@ The document sharing modal was restyled to be friendlier with proper full names
 Any feedback or question? Feel free to comment here or on the <a href="https://discourse.gethue.com/">Forum</a> and <a href="https://docs.gethue.com/quickstart/">quick start</a> SQL querying!
 
 
-Romain fromthe Hue Team
+Romain from the Hue Team

+ 87 - 0
docs/gethue/content/en/posts/2020-04-27-sql-editor-for-apache-spark-sql-with-livy.md

@@ -0,0 +1,87 @@
+---
+title: SQL Editor for Apache Spark SQL with Livy
+author: Romain
+type: post
+date: 2020-04-27T00:00:00+00:00
+url: /blog/quick-task-sql-editor-for-apache-spark-sql-with-livy/
+sf_thumbnail_type:
+  - none
+sf_thumbnail_link_type:
+  - link_to_post
+sf_detail_type:
+  - none
+sf_page_title:
+  - 1
+sf_page_title_style:
+  - standard
+sf_no_breadcrumbs:
+  - 1
+sf_page_title_bg:
+  - none
+sf_page_title_text_style:
+  - light
+sf_background_image_size:
+  - cover
+sf_social_sharing:
+  - 1
+sf_related_articles:
+  - 1
+sf_sidebar_config:
+  - left-sidebar
+sf_left_sidebar:
+  - Sidebar-2
+sf_right_sidebar:
+  - Sidebar-1
+sf_caption_position:
+  - caption-right
+sf_remove_promo_bar:
+  - 1
+ampforwp-amp-on-off:
+  - default
+categories:
+  - Querying
+  - Version 4.8
+
+---
+
+## Spark SQL
+
+[Spark SQL](https://spark.apache.org/docs/latest/sql-programming-guide.html) is convenient for embedding clean data querying logic within your Spark apps. Hue brings an Editor so that it is easier to develop your SQL snippets.
+
+As detailed in the documentation, Spark SQL comes with different [connectors](https://docs.gethue.com/administrator/configuration/connectors/#apache-spark-sql). Here we will just show with Livy.
+
+[Apache Livy](https://livy.incubator.apache.org/) provides a bridge to a running Spark interpreter so that SQL, pyspark and scala snippets can be executed interactively.
+
+In the [hue.ini](https://docs.gethue.com/administrator/configuration/) configure the API url:
+
+    [spark]
+    # The Livy Server URL.
+    livy_server_url=http://localhost:8998
+
+And as always, make sure you have an interpreter configured:
+
+    [notebook]
+    [[interpreters]]
+    [[[sparksql]]]
+    name=Spark SQL
+    interface=livy
+
+And that's it, the editor will appear:
+
+![Hue Spark Sql Editor](https://cdn.gethue.com/uploads/2020/04/editor_spark_sql_livy.png)
+
+One advantage of using Hue is its [File Browser](https://docs.gethue.com/user/browsing/#data) for HDFS / S3 / Azure and full security (Kerberos and even using the real user credentials via [Knox IdBroker](https://docs.cloudera.com/runtime/7.1.0/cdp-security-overview/topics/security_how_identity_federation_works_in_cdp.html) integration).
+
+![Hue Phoenix Editor](https://cdn.gethue.com/uploads/2016/08/image2.png)
+
+Here are some of the future improvements:
+
+* Database/table/column autocomplete is currently empty
+* SQL grammar autocomplete can [be extended](https://docs.gethue.com/developer/parsers/)
+* [SQL Scratchpad](https://docs.gethue.com/developer/api/#scratchpad) module to allow a mini SQL Editor popup is in progress
+
+
+Any feedback or question? Feel free to comment here or on the <a href="https://discourse.gethue.com/">Forum</a> and <a href="https://docs.gethue.com/quickstart/">quick start</a> SQL querying!
+
+
+Romain from the Hue Team

+ 10 - 8
docs/gethue/themes/stack-hue-theme/layouts/partials/home.en.html

@@ -176,14 +176,16 @@
           <div class="switchable__text">
             <h2>Connect to all the databases</h2>
             <p class="lead">
-              Pick one of the multiple interpreters for <a href="https://docs.gethue.com/administrator/configuration/connectors/#apache-hive">Apache Hive</a>,
-              <a href="https://docs.gethue.com/administrator/configuration/connectors/#apache-impala">Apache Impala</a>,
-              <a href="https://docs.gethue.com/administrator/configuration/connectors/#presto">Presto</a>
-              and all the others too: MySQL, SparkSQL, Oracle,
-              <a href="{{ .Site.Params.docsHost }}/sql-querying-apache-hbase-with-apache-phoenix/">Apache Phoenix</a>,
-              <a href="https://docs.gethue.com/administrator/configuration/connectors/#ksqlDB">ksqlDB</a>,
-              <a href="https://docs.gethue.com/administrator/configuration/connectors/#elastic-search">Elastic Search</a>,
-              <a href="{{ .Site.Params.docsHost }}/quick-task-how-to-query-apache-druid-analytic-database/)">Apache Druid</a>, PostgreSQL, Redshift, BigQuery...
+              Pick one of the multiple interpreters for <a href="{{ .Site.Params.docsHost }}/administrator/configuration/connectors/#apache-hive">Apache Hive</a>,
+              <a href="{{ .Site.Params.docsHost }}/administrator/configuration/connectors/#apache-impala">Apache Impala</a>,
+              <a href="{{ .Site.Params.docsHost }}/administrator/configuration/connectors/#presto">Presto</a>
+              and all the others too: MySQL,
+              <a href="/blog/quick-task-sql-editor-for-apache-spark-sql-with-livy/)">SparkSQL</a>,
+              Oracle,
+              <a href="/sql-querying-apache-hbase-with-apache-phoenix/">Apache Phoenix</a>,
+              <a href="{{ .Site.Params.docsHost }}/administrator/configuration/connectors/#ksqlDB">ksqlDB</a>,
+              <a href="{{ .Site.Params.docsHost }}/administrator/configuration/connectors/#elastic-search">Elastic Search</a>,
+              <a href="/quick-task-how-to-query-apache-druid-analytic-database/)">Apache Druid</a>, PostgreSQL, Redshift, BigQuery...
             </p>
             <a href="{{ .Site.Params.docsHost }}/administrator/configuration/connectors/"
               >Learn More »</a