Hue SDK Documentation ===================== [TOC] Introduction and Overview ========================= Hue leverages the browser to provide users with an environment for exploring and analyzing data. Build on top of the Hue SDK to enable your application to interact efficiently with Hadoop and the other Hue services. By building on top of Hue SDK, you get, out of the box: + Configuration Management + Hadoop interoperability + Supervision of subprocesses + A collaborative UI + Basic user administration and authorization This document will orient you with the general structure of Hue and will walk you through adding a new application using the SDK. From 30,000 feet ----------------  Hue, as a "container" web application, sits in between your Hadoop installation and the browser. It hosts all the Hue Apps, including the built-in ones, and ones that you may write yourself. The Hue Server --------------  Hue is a web application built on the Django python web framework. Django, running on the WSGI container/web server (typically CherryPy), manages the url dispatch, executes application logic code, and puts together the views from their templates. Django uses a database (typically sqlite) to manage session data, and Hue applications can use it as well for their "models". (For example, the JobDesigner application stores job designs in the database.) In addition to the web server, some Hue applications run daemon processes "on the side". For example, Spark runs a daemon ("livy_server") that keeps track of the Spark shells of the user. Running a separate process for applications is the preferred way to manage long-running tasks that you may wish to co-exist with web page rendering. The web "views" typically communicate with these side daemons by using Thrift (e.g., for Hive query execution) or by exchanging state through the database. Interacting with Hadoop -----------------------  Hue provides some APIs for interacting with Hadoop. Most noticeably, there are python file-object-like APIs for interacting with HDFS. These APIs work by making REST API or Thrift calls the Hadoop daemons. The Hadoop administrator must enable these interfaces from Hadoop. On the Front-End ---------------- Hue provides a front-end framework based on [Bootstrap](http://twitter.github.com/bootstrap/) and [jQuery](http://jquery.com/). If you are used to the Hue 1.x front-end, this is a major difference. All application pages are full screen requests from the browser. The HTML generated by your application's template is directly rendered. You do not need to worry about interference from another application. And you have more freedom to customize the front-end behavior of your application. An Architectural View ---------------------  A Hue application may span three tiers: (1) the UI and user interaction in the client's browser, (2) the core application logic in the Hue web server, and (3) external services with which applications may interact. The absolute minimum that you must implement (besides boilerplate), is a "Django [view](https://docs.djangoproject.com/en/1.2/topics/http/views/)" function that processes the request and the associated template to render the response into HTML. Many apps will evolve to have a bit of custom JavaScript and CSS styles. Apps that need to talk to an external service will pull in the code necessary to talk to that service. File Layout ----------- The Hue "framework" is in ``desktop/core/`` and contains the Web components. ``desktop/libs/`` is the API for talking to various Hadoop services. The installable apps live in ``apps/``. Please place third-party dependencies in the app's ext-py/ directory. The typical directory structure for inside an application includes: ``` src/ for Python/Django code models.py urls.py views.py forms.py settings.py conf/ for configuration (``.ini``) files to be installed static/ for static HTML/js resources and help doc templates/ for data to be put through a template engine locales/ for localizations in multiple languages ``` For the URLs within your application, you should make your own ``urls.py`` which will be automatically rooted at ``/yourappname/`` in the global namespace. See ``apps/about/src/about/urls.py`` for an example. Pre-requisites ============== Software -------- Developing for the Hue SDK has similar requirements to running Hue itself. We require python (2.6 to 2.7), Django (1.4 included with our distribution), Hadoop (Apache Hadoop 1.2+), Java (Sun Java 1.7), and Browser (latest Chrome, Firefox or IE9+). Recommended Reading / Important Technologies -------------------------------------------- The following are core technologies used inside of Hue. * Python. Dive Into Python is one of several excellent books on python. * Django. Start with [The Django Tutorial](http://docs.djangoproject.com/en/1.2/intro/tutorial01/). * [Thrift](http://incubator.apache.org/thrift/) is used for communication between daemons. * [Mako](http://www.makotemplates.org/) is the preferred templating language. Fast-Guide to Creating a New Hue Application ============================================ Now that we have a high-level overview of what's going on, let's go ahead and create a new installation. Download, Unpack, Build Distro ------------------------------ The Hue SDK is available from [Github](http://github.com/cloudera/hue). Releases can be found on the [download page](https://github.com/cloudera/hue/downloads). Releases are missing a few dependencies that could not be included because of licencing issues. So if you prefer to have an environment ready from scratch, it is preferable to checkout a particular release tag instead. $ cd hue ## Build $ make apps ## Run $ build/env/bin/hue runserver ## Alternative run $ build/env/bin/hue supervisor ## Visit http://localhost:8000/ with your web browser. Run "create_desktop_app" to Set up a New Source Tree -------------------------------------------- $ ./build/env/bin/hue create_desktop_app calculator $ find calculator -type f calculator/setup.py # distutils setup file calculator/src/calculator/__init__.py # main src module calculator/src/calculator/forms.py calculator/src/calculator/models.py calculator/src/calculator/settings.py # app metadata setting calculator/src/calculator/urls.py # url mapping calculator/src/calculator/views.py # app business logic calculator/src/calculator/templates/index.mako calculator/src/calculator/templates/shared_components.mako # Static resources calculator/src/static/calculator/art/calculator.png # logo calculator/src/static/calculator/css/calculator.css calculator/src/static/calculator/js/calculator.js To download an app or browse dditional plugin apps available in the Hue app store: ## Visit http://gethue.com/app-store/
Customizing Views and Templates
-------------------------------
Now that your app has been installed, you'll want to customize it.
As you may have guessed, we're going to build a small calculator
application. Edit `calculator/src/calculator/templates/index.mako`
to include a simple form:
<%!from desktop.views import commonheader, commonfooter %>
<%namespace name="shared" file="shared_components.mako" />
${commonheader("Calculator", "calculator", user, "100px") | n,unicode}
## Main body
Integrate external Web applications in any language
===================================================
Use the [create_proxy_app command](http://gethue.com/integrate-external-web-applications-in-any-language)
A Look at Three Existing Apps
=============================

Help
----
The Help application is as minimal as they get. Take a look at it!
The core logic is in the "views.py" file. The central function
there takes `(app, path)` (which are mapped from the request URL
by the regular expression in `urls.py`). The view function
finds the data file that needs to be rendered, renders it through
the markdown module, if necessary, and then displays it through
a simple template.
You'll note that the "Help Index" is presented in a "split view".
No JavaScript was written to make this happen! Instead, the template
applied certain CSS classes to the relevant `div`'s, and JFrame
did the rest.
Proxy
-----
### Setup
You need to have Hue running:
$ ./build/env/bin/hue runserver
Then if you want to access localhost/50030/jobtracker.jsp you just do:
http://127.0.0.1:8000/proxy/localhost/50030/jobtracker.jsp
and the page will be displayed within Hue.
You can configure it in ``desktop/conf/pseudo-distributed.ini``
[proxy]
whitelist="(localhost|127\.0\.0\.1)50030|50070|50060|50075)",
#Comma-separated list of regular expressions, which match 'host:port' of requested proxy target.
blacklist=""
#Comma-separated list of regular expressions, which match any prefix of 'host:port/path' of requested proxy target.
# This does not support matching GET parameters.
### Usage
You can create a new app (or modify a current one for testing).
Then in order to display the proxied page in your app, you could add in the template of a view of the new app a
snippet of Javacript similar to this for loading the JobTracker page:
or alternatively get the page in the view (better solution) with the Hue
[REST API](https://github.com/cloudera/hue/tree/master/desktop/core/src/desktop/lib/rest). Example of use of
this API can be found in the [HDFS lib](https://github.com/cloudera/hue/blob/master/desktop/libs/hadoop/src/hadoop/fs/webhdfs.py).
If you need to browse through the proxied page, using an iframe might be a better solution.
Beeswax
-------
Beeswax is on the opposite end of the complexity scale from Help.
In addition to many views (in `views.py`), Beeswax uses
Django Forms for server-side form validation (the forms are in `forms.py`),
several features of the Mako templating engine (especially includes and
functions), a separate server (implemented in Java), and significant
JavaScript for user interaction.
Backend Development
===================
This section goes into greater detail on useful features within
the Hue environment.
User Management
---------------
Except for static content, `request.user` is always populated. It is a
standard Django `models.User` object. If you were to set a breakpoint at the
`index()` function in our calculator app, you will find:
>>> request.user
request.user gets populated. There's also a
middleware for Hue that makes sure that no pages are displayed unless the
user is authenticated.
MY_PROPERTY = Config(key='app_property', default='Beatles', help='blah')You can access its value by `MY_PROPERTY.get()`. * A `desktop.lib.conf.ConfigSection` object for `section_a`, such as:
SECTION_A = ConfigSection(key='section_a',
help='blah',
members=dict(
AWEIGHT=Config(key='a_weight', type=int, default=0),
AHEIGHT=Config(key='a_height', type=int, default=0)))
You can access the values by `SECTION_A.AWEIGHT.get()`.
* A `desktop.lib.conf.UnspecifiedConfigSection` object for `filesystems`, such as:
FS = UnspecifiedConfigSection(
key='filesystems',
each=ConfigSection(members=dict(
nn_host=Config(key='namenode_host', required=True))
An `UnspecifiedConfigSection` is useful when the children of the section are not known.
When Hue loads your application's configuration, it binds all sub-sections. You can
access the values by:
cluster1_val = FS['cluster_1'].nn_host.get()
all_clusters = FS.keys()
for cluster in all_clusters:
val = FS[cluster].nn_host.get()
Your Hue application can automatically detect configuration problems and alert
the admin. To take advantage of this feature, create a `config_validator`
function in your `conf.py`:
def config_validator(user):
"""
config_validator(user) -> [(config_variable, error_msg)] or None
Called by core check_config() view.
"""
res = [ ]
if not REQUIRED_PROPERTY.get():
res.append((REQUIRED_PROPERTY, "This variable must be set"))
if MY_INT_PROPERTY.get() < 0:
res.append((MY_INT_PROPERTY, "This must be a non-negative number"))
return res
help="..." argument to all configuration
related objects in your conf.py. The examples omit some for the
sake of space. But you and your application's users can view all the
configuration variables by doing:
$ build/env/bin/hue config_help
entry_points = {
'desktop.sdk.application': 'my_app = my_app',
'desktop.supervisor.specs': [ 'my_daemon = my_app:SUPERVISOR_SPEC' ] }
* In `src/my_app/__init__.py`, tell Hue what to run by adding:
SUPERVISOR_SPEC = dict(django_command="my_daemon")
* Then in `src/my_app/management/commands`, create `__init__.py` and `my_daemon.py`. Your
daemon program has only one requirement: it must define a class called `Command` that
extends `django.core.management.base.BaseCommand`. Please see `kt_renewer.py` for an example.
The next time Hue restarts, your `my_daemon` will start automatically.
If your daemon program dies (exits with a non-zero exit code), Hue will
restart it.
"Under the covers:" Threading. Hue, by default, runs CherryPy web server.
If Hue is configured (and it may be, in the future)
to use mod_wsgi under Apache httpd, then there would be multiple python
processes serving the backend. This means that your Django application
code should avoid depending on shared process state. Instead, place
the stored state in a database or run a separate server.
Walk-through of a Django View
-----------------------------

Django is an MVC framework, except that the controller is called a
"[view](https://docs.djangoproject.com/en/1.2/topics/http/views/)" and
the "view" is called a "template". For an application developer, the essential
flow to understand is how the "urls.py" file provides a mapping between URLs (expressed as a
regular expression, optionally with captured parameters) and view functions.
These view functions typically use their arguments (for example, the captured parameters) and
their request object (which has, for example, the POST and GET parameters) to
prepare dynamic content to be rendered using a template.
Templates: Django and Mako
--------------------------
In Hue, the typical pattern for rendering data through a template
is:
from desktop.lib.django_util import render
def view_function(request):
return render('view_function.mako', request, dict(greeting="hello"))
The `render()` function chooses a template engine (either Django or Mako) based on the
extension of the template file (".html" or ".mako"). Mako templates are more powerful,
in that they allow you to run arbitrary code blocks quite easily, and are more strict (some
would say finicky); Django templates are simpler, but are less expressive.
Django Models
-------------
[Django Models](http://docs.djangoproject.com/en/1.2/topics/db/models/#topics-db-models)
are Django's Object-Relational Mapping framework. If your application
needs to store data (history, for example), models are a good way to do it.
From an abstraction perspective, it's common to imagine external services
as "models". For example, the Job Browser treats the Hadoop JobTracker
as a "model", even though there's no database involved.
Accessing Hadoop
----------------
It is common for applications to need to access the underlying HDFS.
The `request.fs` object is a "file system" object that exposes
operations that manipulate HDFS. It is pre-configured to access
HDFS as the user that's currently logged in. Operations available
on `request.fs` are similar to the file operations typically
available in python. See `webhdfs.py` for details; the list
of functions available is as follows:
`chmod`,
`chown`,
`exists`,
`isdir`,
`isfile`,
`listdir` (and `listdir_stats`),
`mkdir`,
`open` (which exposes a file-like object with `read()`, `write()`, `seek()`, and `tell()` methods),
`remove`,
`rmdir`,
`rmtree`, and
`stats`.
Making Your Views Thread-safe
-----------------------------
Hue works in any WSGI-compliant container web server.
The current recommended deployment server is the built-in CherryPy server.
The CherryPy server, which is multi-threaded, is invoked by `runcpserver`
and is configured to start when Hue's `supervisor` script is used.
Meanwhile, `runserver` start a single-threaded
testing server.
Because multiple threads may be accessing your views
concurrently, your views should not use shared state.
An exception is that it is acceptable to initialize
some state when the module is first imported.
If you must use shared state, use Python's `threading.Lock`.
Note that any module initialization may happen multiple times.
Some WSGI containers (namely, Apache), will start multiple
Unix processes, each with multiple threads. So, while
you have to use locks to protect state within the process,
there still may be multiple copies of this state.
For persistent global state, it is common to place the state
in the database. If the state needs to be managed with application code,
a common pattern to push state into a "helper process". For example, in the Job Designer,
a helper process keeps track of the processes that have been launched. The Django views
themselves are stateless, but they talk to this stateful helper process for
updates. A similar approach is taken with updating metrics for
the Beeswax application.
Authentication Backends
-----------------------
Hue exposes a configuration flag ("auth") to configure
a custom authentication backend. See
See http://docs.djangoproject.com/en/dev/topics/auth/#writing-an-authentication-backend
for writing such a backend.
In addition to that, backends may support a `manages_passwords_externally()` method, returning
True or False, to tell the user manager application whether or not changing
passwords within Hue is possible.
Authorization
-------------
Applications may define permission sets for different actions. Administrators
can assign permissions to user groups in the UserAdmin application. To define
custom permission sets, modify your app's `settings.py` to create a list of
`(identifier, description)` tuples:
PERMISSION_ACTIONS = [
("delete", "Delete really important data"),
("email", "Send email to the entire company"),
("identifier", "Description of the permission")
]
Then you can use this decorator on your view functions to enforce permission:
@desktop.decorators.hue_permission_required("delete", "my_app_name")
def delete_financial_report(request):
...
Using and Installing Thrift
---------------------------
Right now, we check in the generated thrift code.
To generate the code, you'll need the thrift binary version 0.9.0.
Please download from http://thrift.apache.org/.
The modules using ``Thrift`` have some helper scripts like ``regenerate_thrift.sh``
for regenerating the code from the interfaces.
Profiling Hue Apps
------------------
Hue has a profiling system built in, which can be used to analyze server-side
performance of applications. To enable profiling::
$ build/env/bin/hue runprofileserver
Then, access the page that you want to profile. This will create files like
/tmp/useradmin.users.000072ms.2011-02-21T13:03:39.745851.prof. The format for
the file names is /tmp/./build/env/bin/hue collectstaticAdding Interactive Elements to Your UI -------------------------------------- Hue by default loads these JavaScript components: * jQuery * jQuery.dataTables * Bootstrap These are used by some Hue applications, but not loaded by default: * Knockout js (`desktop/core/static/ext/js/knockout-min.js`) * DataTables pagination using the Bootstrap style (`desktop/core/static/ext/js/datatables-paging-0.1.js`) * jQuery UI (`desktop/core/static/ext/js/jquery/plugins/jquery-ui-autocomplete-1.8.18.min.js`) These standard components have their own online documentation, which we will not repeat here. They let you write interactive behaviors with little or no JavaScript. ## Key Differences from Hue 1.x Here are the key differences between the Hue 1.x front-end SDK and the later versions. In Hue 2.0 and beyond: * Since each page view only loads one application, you can declare HTML elements by ID, you can declare and load your JavaScripts anywhere, and are in full control of all the UI interactions. * You can use standard HTML links to other applications. * You do not need to register your application with the front-end, or declare any dependencies using YAML. * The navigation bar is not pluggable in Hue 2.0. * The old "accordion" behavior can be replaced by [Bootstrap collapse](http://twitter.github.com/bootstrap/javascript.html#collapse). * The old "art buttons" pattern can be replaced by [Bootstrap buttons](http://twitter.github.com/bootstrap/base-css.html#buttons), and [button groups](http://twitter.github.com/bootstrap/components.html#buttonGroups). * The old "art inputs" pattern can be replaced by [Bootstrap form inputs](http://twitter.github.com/bootstrap/base-css.html#forms). * The old "autocomplete" behavior can be replaced by [jQuery autocomplete](http://jqueryui.com/demos/autocomplete/) or [Bootstrap typeahead](http://twitter.github.com/bootstrap/javascript.html#typeahead). * The old "collapser" behavior can be replaced by [Bootstrap collapse](http://twitter.github.com/bootstrap/javascript.html#collapse). * The old "context menu" behavior can be replaced by [Bootstrap button dropdowns](http://twitter.github.com/bootstrap/components.html#buttonDropdowns). * The old "fittext" behavior is no longer supported. * The old "flash message" behavior is no longer supported. * The old "html table" behavior can be replaced by [DataTables](http://datatables.net/). * The old "overtext" behavior can be replaced by [Bootstrap form placeholder](http://twitter.github.com/bootstrap/base-css.html#forms). * The old "popup" behavior can be replaced by [Bootstrap modals](http://twitter.github.com/bootstrap/javascript.html#modals). * The old "side-by-side select" pattern is no longer supported. * The old "splitview" layout is no longer supported. * The old "tabs" layout can be replaced by [Bootstrap tabs](http://twitter.github.com/bootstrap/javascript.html#tabs). * The old "tool tips" behavior can be replaced by [Bootstrap tooltips](http://twitter.github.com/bootstrap/javascript.html#tooltips). Including Other JavaScript Frameworks ------------------------------------- It is possible to include other JavaScript frameworks to do your development. Simply include them to your application's pages. MooTools, Dojo, YUI, etc are all fine. Including them represents an additional burden for your users to download, and they also make it harder for us to support you, but it is your call. Internationalization ==================== How to update all the messages and compile them:: $ make locales How to update and compile the messages of one app:: $ cd apps/beeswax $ make compile-locale How to create a new locale for an app:: $ cd $APP_ROOT/src/$APP_NAME/locale $ $HUE_ROOT/build/env/bin/pybabel init -D django -i en_US.pot -d . -l fr Debugging Tips and Tricks ========================= * Set `DESKTOP_DEBUG=1` as an environment variable if you want logs to go to stderr as well as to the respective log files. * Use runserver. If you want to set a CLI breakpoint, just insert `__import__("ipdb").set_trace()` into your code. * Django tends to restart its server whenever it notices a file changes. For certain things (like configuration changes), this is not sufficient. Restart the server whole-heartedly. * If you find yourself writing a lot of JavaScript, you'll want to disable the JavaScript caching that the server does. At startup Hue reads all your dependencies and JS files into memory to make things faster. You can disable this by executing the runserver command with an environment variable set. Hue will be a little slower, but your JS will always represent what's on the disk. Here's what that looks like: `$ DESKTOP_DEPENDER_DEBUG=1 build/env/bin/hue runserver` * We highly recommend developing with the [Firebug](http://getfirebug.com) debugging plugin for Firefox. With it enabled, you can use a utility called [dbug](http://www.clientcide.com/docs/Core/dbug) which wraps Firebug commands. This allows you to leave debug statements in your code and display them on demand. In particular, typing in `dbug.cookie()` in Firebug will set a cookie in your browser that will turn these statements on until you type that command again to toggle them off. You'll see some of our own debugging statements and you can add your own. In the future, entering this state may also provide access to additional debugging features. * When the dbug state is enabled in the browser, right clicking on elements is re-enabled which makes element inspection a little easier in Firebug.