spark_sql_livy.md 1.4 KB

SQL Editor for Spark SQL via Livy

Background

As detailed in the documentation, Spark SQL comes with different connectors. Here we will just show with Livy.

Apache Livy provides a bridge to a running Spark interpreter so that SQL, pyspark and scala snippets can be executed interactively.

In the hue.ini configure the API url:

[spark]
# The Livy Server URL.
livy_server_url=http://localhost:8998

And as always, make sure you have an interpreter configured:

[notebook]
[[interpreters]]
[[[sparksql]]]
name=Spark SQL
interface=livy

And that's it, the editor will appear:

Hue Spark Sql Editor

One advantage of using Hue is its File Browser for HDFS / S3 / Azure and full security.

Hue Phoenix Editor

Future improvements

  • Database/table/column autocomplete is currently empty
  • SQL grammar autocomplete can be extended
  • SQL Scratchpad module to allow a mini SQL Editor popup is in progress