title: 'A new Spark Web UI: Spark App' author: admin type: post date: 2014-01-02T14:06:00+00:00 url: /a-new-spark-web-ui-spark-app/ tumblr_gethue_permalink:
Hi Spark Makers!
A Hue Spark application was recently created. It lets users execute and monitor Spark jobs directly from their browser from any machine, with interactivity.
The new application is using the Spark Job Server contributed by Ooyala at the last Spark Summit.
We hope to work with the community and have support for Python, Java, direct script submission without compiling/uploading and other improvements in the future!
As usual feel free to comment on the hue-user list or @gethue!
Currently only Scala jobs are supported and programs need to implement this trait and be packaged into a jar. Here is a WordCount example. To learn more about Spark Job Server, check its README.
If you are using Cloudera Manager, enable the Spark App by removing it from the blacklist by adding this in the Hue Safety Valve:
{{< highlight bash >}}
[desktop]
app_blacklist=
{{< /highlight >}}
We assume you have Spark 0.9.0, Scala 2.10. installed on your system. Make sure you have the good scala and sbt versions, e.g. for Ubuntu: https://gist.github.com/visenger/5496675
Currently on github on this branch:
{{< highlight bash >}}git clone https://github.com/ooyala/spark-jobserver.git
cd spark-jobserver
{{< /highlight >}}
Then type:
{{< highlight bash >}}sbt
re-start{{< /highlight >}}
If Hue and Spark Job Server are not on the same machine update the hue.ini property in desktop/conf/pseudo-distributed.ini:
{{< highlight bash >}}
[spark]
# URL of the Spark Job Server.
server_url=http://localhost:8090/{{< /highlight >}}
To point to your Spark Cluster
{{< highlight bash >}}vim ./job-server/src/main/resources/application.conf{{< /highlight >}}
Replace:
{{< highlight bash >}}master = "local4"{{< /highlight >}}
With the Spark Master URL (you can get it from the Spark Master UI: http://SPARK-HOST:18080/):
{{< highlight bash >}}master = "spark://localhost:7077"{{< /highlight >}}
Then follow this walk-through and create the example jar that is used in the video demo.