---
title: 'Beta of new Notebook Application for Spark & SQL'
author: admin
type: post
date: 2015-04-23T13:25:34+00:00
url: /new-notebook-application-for-spark-sql/
sf_thumbnail_type:
- none
sf_thumbnail_link_type:
- link_to_post
sf_detail_type:
- none
sf_page_title:
- 1
sf_page_title_style:
- standard
sf_no_breadcrumbs:
- 1
sf_page_title_bg:
- none
sf_page_title_text_style:
- light
sf_background_image_size:
- cover
sf_social_sharing:
- 1
sf_sidebar_config:
- left-sidebar
sf_left_sidebar:
- Sidebar-2
sf_right_sidebar:
- Sidebar-1
sf_caption_position:
- caption-right
sf_remove_promo_bar:
- 1
categories:
- Querying
---
[Hue 3.8][1] brings a new way to directly submit Spark jobs from a Web UI.
Last year we released [Spark Igniter][2] to enable developers to submit Spark jobs through a Web Interface. While this approach worked, the UX left a lot to be desired. Programs had to implement an interface, be compiled beforehand and YARN support was lacking. We also wanted to add support for Python and Scala, focusing on delivering an interactive and iterative programming experience similar to using a REPL.
[
][3]
This is for this that we started developing a new [Spark REST Job Server][4] that could provide these missing functionalities. On top of it, we revamped the UI for providing a Python Notebook-like feeling.
Note that this new application is pretty new and labeled as ‘Beta’. This means we recommend trying it out and contributing, but its usage is not officially supported yet as the UX is going to evolve a lot!
This post describes the Web Application part. We are using [Spark 1.3][5] and [Hue master branch][6].
Based on a new:
* [Spark REST Job Server][4]
* Notebook Web UI
Supports:
* Scala
* Python
* Java
* SQL
* YARN
If the Spark app is not visible in the ‘Editor’ menu, you will need to unblacklist it from the [hue.ini][7]:
{{< highlight bash >}}[desktop]
app_blacklist=
{{< /highlight >}}
**Note:** To override a value in Cloudera Manager, you need to enter verbatim each mini section from below into the Hue [Safety Valve][8]: Hue Service → Configuration → Service-Wide → Advanced → Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini
On the same machine as Hue, go in the Hue home:
If using the package installed:
{{< highlight bash >}}cd /usr/lib/hue{{< /highlight >}}
**Recommended**
Use Livy Spark Job Server from the Hue master repository instead of CDH (it is currently much more advanced): [see build & start the latest Livy][4]
If not, use Cloudera Manager:
{{< highlight bash >}}cd /opt/cloudera/parcels/CDH/lib/
HUE_CONF_DIR=/var/run/cloudera-scm-agent/process/-hue-HUE_SERVER-#
echo $HUE_CONF_DIR
export HUE_CONF_DIR
{{< /highlight >}}