# HUE SDK Documentation [TOC] ## Introduction and Overview HUE leverages the browser to provide users with an environment for exploring and analyzing data. Build on top of the HUE SDK to enable your application to interact efficiently with Hadoop and the other HUE services. By building on top of HUE SDK, you get, out of the box: + Configuration Management + Hadoop interoperability + Supervision of subprocesses + A collaborative co-habitative UI + A CSS class-based UI toolkit + Basic user administration This document will orient you with the general structure of HUE and will walk you through adding a new application using the SDK. NOTE: HUE began its life as "Cloudera Desktop," so you may find references to "Desktop" in a few places. ### From 30,000 feet  HUE, as a "container" web application, sits in between your Hadoop installation and the browser. It hosts all the HUE Apps, including the built-in ones, and ones that you may write yourself. ### The HUE Server  HUE is a web application built on the Django python web framework. Django, running on the WSGI container/web server (typically CherryPy), manages the url dispatch, executes application logic code, and puts together the views from their templates. Django uses a database (typically sqlite) to manage session data, and HUE applications can use it as well for their "models". (For example, the JobDesigner application stores job designs in the database.) In addition to the web server, some HUE applications run daemon processes "on the side". For example, Beeswax runs a daemon ("beeswax_server") that keeps track of query states. Running a separate process for applications is the preferred way to manage long-running tasks that you may wish to co-exist with web page rendering. The web "views" typically communicate with these side daemons by using Thrift (e.g., for job submission) or by exchanging state through the database (e.g., when a job completes). ### Interacting with Hadoop  HUE provides some APIs for interacting with Hadoop. Most noticeably, there are python file-object-like APIs for interacting with HDFS. These APIs work by making Thrift calls to "plug-ins" running inside the Hadoop daemons. The Hadoop administrator must enable these plug-ins. ### On the Front-End  HUE provides a front-end framework allowing you to co-locate many logical applications within the same windowing environment. The essential pattern is that individual apps handle their own HTTP requests. The front-end renders the responses of those requests in windows within the same web page. We have dubbed these windows "JFrames". ### An Architectural View  A HUE application may span three tiers: (1) the UI and user interaction in the client's browser, (2) the core application logic in the HUE web server, and (3) external services with which applications may interact. The absolute minimum that you must implement (besides boilerplate), is a "django view" function that processes the request and the associated template to render the response into HTML. Many apps will evolve to have a bit of custom JavaScript and CSS styles. Apps that need to talk to an external service will pull in the code necessary to talk to that service. ## Pre-requisites ### Software Developing for the HUE SDK has similar requirements to running HUE itself. We require python (2.4, 2.5, or 2.6), Django (1.1 included with our distribution), Hadoop (Cloudera's Distribution for Hadoop, at least `0.20.1+152`), Java (Sun Java 1.6), and Firefox (at least 3.0). ### Recommended Reading / Important Technologies The following are core technologies used inside of HUE. * Python. Dive Into Python is one of several excellent books on python. * Django. Start with [The Django Tutorial](http://docs.djangoproject.com/en/1.1/intro/tutorial01/). * [Thrift](http://incubator.apache.org/thrift/) is used for communication between daemons. * [Mako](http://www.makotemplates.org/) is the preferred templating language. * [MooTools](http://mootools.net/) is the Javascript framework. ## Fast-Guide to Creating a New HUE Application Now that we have a high-level overview of what's going on, let's go ahead and create a new installation. ### Crepo Development versions of HUE require "crepo", a tool to grab external repositories into your workspace. Crepo can be run out of its repository (simply point the CREPO environment variable to `crepo.py`) and is available at http://github.com/cloudera/crepo. ### Download, unpack, build distro The HUE SDK is available from http://github.com/cloudera/hue; releases are at http://archive.cloudera.com/cdh/3. $ cd hue # Build $ export HADOOP_HOME=... $ make apps # Run $ build/env/bin/hue runserver_plus $ build/env/bin/hue beeswax_server $ build/env/bin/hue jobsubd # Visit http://localhost:8000/ with your web browser.
### Run "create_app" to set up a new source tree $ ./build/env/bin/hue create_desktop_app calculator $ find calculator -type f calculator/setup.py # distutils setup file calculator/src/calculator/__init__.py # main src module calculator/src/calculator/forms.py calculator/src/calculator/models.py calculator/src/calculator/settings.py calculator/src/calculator/urls.py # url mapping calculator/src/calculator/views.py # app business logic calculator/src/calculator/windmilltests.py calculator/src/calculator/templates/index.mako # Static resources calculator/src/calculator/static/art/calculator.png # logo calculator/src/calculator/static/bootstrap.js calculator/src/calculator/static/css/calculator.css calculator/src/calculator/static/help/index.md # Help file calculator/src/calculator/static/js/package.yml # Declaration of JS files calculator/src/calculator/static/js/Source/Calculator/Calculator.js # JS entrypoint ### Install that app As you'll discover if you look at calculator's setup.py, HUE uses a distutils entrypoint to register applications. By installing the calculator package into HUE's python virtual environment, you'll install a new app. The "app_reg.py" tool manages the applications that are installed. Note that in the following example, the value after the "--install" option is the path to the root directory of the application you want to install. In this example, it is a relative path to "/Users/philip/src/hue/calculator". $ ./build/env/bin/python tools/app_reg/app_reg.py --install calculator === Installing app at calculator Updating registry with calculator (version 0.1) --- Making egg-info for calculator $ ./build/env/bin/python tools/app_reg/app_reg.py --list 2>&1 | grep calculator calculator 0.1 /Users/philip/src/hue/calculator
### Debugging Django
If you enter a number only in the first text box, you'll see
HUE noticed that an exception was thrown, and created a modal
dialog error for you.
If you access the calculator URL at `/calculator` in your browser,
and if you're using
`runserver_plus`, you'll get a handy debugging page. You can click on any
stack frame to get a debugging console:
Great! Now that we've added a single application, we're going to
delve further into the back-end.
## A Look at Two Existing Apps

### Help
The Help application is as minimal as they get. Take a look at it!
The core logic is in the "views.py" file. The central function
there takes `(app, path)` (which are mapped from the request URL
by the regular expression in `urls.py`). The view function
finds the data file that needs to be rendered, renders it through
the markdown module, if necessary, and then displays it through
a simple template.
You'll note that the "Help Index" is presented in a "split view".
No Javascript was written to make this happen! Instead, the template
applied certain CSS classes to the relevant `div`'s, and JFrame
did the rest.
### Beeswax
Beeswax is on the opposite end of the complexity scale from Help.
In addition to many views (in `views.py`), Beeswax uses
Django Forms for server-side form validation (the forms are in `forms.py`),
several features of the Mako templating engine (especially includes and
functions), a separate server (implemented in Java), and significant
JavaScript for user interaction.
## Backend Development
This section goes into greater detail on useful features within
the HUE environment.
### User management
Except for static content, `request.user` is always populated. It is a
standard Django `models.User` object. If you were to set a breakpoint at the
`index()` function in our calculator app, you will find:
>>> request.user
request.user
gets populated. There's also a middleware for HUE that makes sure that no pages
are displayed unless the user is authenticated.
MY_PROPERTY = Config(key='app_property', default='Beatles', help='blah')You can access its value by `MY_PROPERTY.get()`. * A `desktop.lib.conf.ConfigSection` object for `section_a`, such as:
SECTION_A = ConfigSection(key='section_a', help='blah', members=dict( AWEIGHT=Config(key='a_weight', type=int, default=0), AHEIGHT=Config(key='a_height', type=int, default=0)))You can access the values by `SECTION_A.AWEIGHT.get()`. * A `desktop.lib.conf.UnspecifiedConfigSection` object for `filesystems`, such as:
FS = UnspecifiedConfigSection( key='filesystems', each=ConfigSection(members=dict( nn_host=Config(key='namenode_host', required=True))An `UnspecifiedConfigSection` is useful when the children of the section are not known. When HUE loads your application's configuration, it binds all sub-sections. You can access the values by:
cluster1_val = FS['cluster_1'].nn_host.get() all_clusters = FS.keys() for cluster in all_clusters: val = FS[cluster].nn_host.get()Your HUE application can automatically detect configuration problems and alert the admin. To take advantage of this feature, create a `config_validator` function in your `conf.py`:
def config_validator(): """ config_validator() -> [(config_variable, error_msg)] or None Called by core check_config() view. """ res = [ ] if not REQUIRED_PROPERTY.get(): res.append((REQUIRED_PROPERTY, "This variable must be set")) if MY_INT_PROPERTY.get() < 0: res.append((MY_INT_PROPERTY, "This must be a non-negative number")) return res
help="..." argument to all configuration related objects in
your conf.py. The examples omit some for the sake of space. But you and your
application's users can view all the configuration variables by doing:
$ build/env/bin/hue config_help
entry_points = {
'desktop.sdk.application': 'my_app = my_app',
'desktop.supervisor.specs': [ 'my_daemon = my_app:SUPERVISOR_SPEC' ] }
* In `src/my_app/__init__.py`, tell HUE what to run by adding:
SUPERVISOR_SPEC = dict(django_command="my_daemon")* Then in `src/my_app/management/commands`, create `__init__.py` and `my_daemon.py`. Your daemon program has only one requirement: it must define a class called `Command` that extends `django.core.management.base.BaseCommand`. Please see `jobsubd.py` for an example. The next time HUE restarts, your `my_daemon` will start automatically. If your daemon program dies (exits with a non-zero exit code), HUE will restart it. "Under the covers:" Threading. HUE, by default, runs under a slightly modified CherryPy WSGI server. This server is multi-threaded, so you can use python threading support (such as it is). The "runserver_plus" version is single-threaded. If HUE is configured (and it may be, in the future) to use mod_wsgi under Apache httpd, then there would be multiple python processes serving the backend. This means that your Django application code should avoid depending on shared process state. Instead, place the stored state in a database or run a separate server. ### Walk-through of a Django view  Django is an MVC framework, except that the controller is called a "view" and the "view" is called a "template". For an application developer, the essential flow to understand is how the "urls.py" file provides a mapping between URLs (expressed as a regular expression, optionally with captured parameters) and view functions. These view functions typically use their arguments (for example, the captured parameters) and their request object (which has, for example, the POST and GET parameters) to prepare dynamic content to be rendered using a template. ### Templates: Django and Mako In HUE, the typical pattern for rendering data through a template is: from desktop.lib.django_util import render def view_function(request): return render('view_function.mako', request, dict(greeting="hello")) The `render()` function chooses a template engine (either Django or Mako) based on the extension of the template file (".html" or ".mako"). Mako templates are more powerful, in that they allow you to run arbitrary code blocks quite easily, and are more strict (some would say finicky); Django templates are simpler, but are less expressive. ### Django Models [Django Models](http://docs.djangoproject.com/en/1.1/topics/db/models/#topics-db-models) are Django's Object-Relational Mapping framework. If your application needs to store data (history, for example), models are a good way to do it. From an abstraction perspective, it's common to imagine external services as "models". For example, the Job Browser treats the Hadoop JobTracker as a "model", even though there's no database involved. ### Accessing Hadoop It is common for applications to need to access the underlying HDFS. The `request.fs` object is a "file system" object that exposes operations that manipulate HDFS. It is pre-configured to access HDFS as the user that's currently logged in. Operations available on `request.fs` are similar to the file operations typically available in python. See `hadoopfs.py` for details; the list of functions available is as follows: `chmod`, `chown, `exists`, `isdir`, `isfile`, `listdir` (and `listdir_stats`), `mkdir`, `open` (which exposes a file-like object with `read()`, `write()`, `seek()`, and `tell()` methods), `remove`, `rmdir`, `rmtree`, and `stats`. ### Making your views thread-safe HUE works in any WSGI-compliant container web server. The current recommended deployment server is the built-in CherryPy server. The CherryPy server, which is multi-threaded, is invoked by `runcpserver` and is configured to start when HUE's `supervisor` script is used. Meanwhile, `runserver` and `runserver_plus` start a single-threaded testing server. Because multiple threads may be accessing your views concurrently, your views should not use shared state. An exception is that it is acceptable to initialize some state when the module is first imported. If you must use shared state, use Python's `threading.Lock`. Note that any module initialization may happen multiple times. Some WSGI containers (namely, Apache), will start multiple Unix processes, each with multiple threads. So, while you have to use locks to protect state within the process, there still may be multiple copies of this state. For persistent global state, it is common to place the state in the database. If the state needs to be managed with application code, a common pattern to push state into a "helper process". For example, in the Job Designer, a helper process keeps track of the processes that have been launched. The Django views themselves are stateless, but they talk to this stateful helper process for updates. A similar approach is taken with updating metrics for the Beeswax application. ### Authentication Backends HUE exposes a configuration flag ("auth") to configure a custom authentication backend. See See http://docs.djangoproject.com/en/dev/topics/auth/#writing-an-authentication-backend for writing such a backend. In addition to that, backends may support a `manages_passwords_externally()` method, returning True or False, to tell the user manager application whether or not changing passwords within HUE is possible. ## Front-end Development Developing applications for HUE requires a minimal amount of CSS and JavaScript to use existing functionality. As covered above, creating an application for the HUE is a matter of creating a standard HTML application that is available when you visit it in your browser (i.e. http://localhost:8000/calculator/) but also in the HUE environment. Our goal is to make it easy for you to author Django applications and enhance their views with default styles and behaviors provided by our SDK. If you find you want to use a UI feature that we don't have in the SDK yet, there are clear pathways for you to author that functionality using JavaScript. As time goes on and our SDK matures, you'll find more and more UI patterns present for your use. ### Default CSS Styles HUE comes with a collection of default styles that you can use to make your apps and their UI components look like all the other apps. Your app doesn't have to use these styles, but if you do, it'll save you some time and make your app look at home in HUE. You can find the default styles inside the SDK in the file located at *desktop/core/static/css/shared.css*. ### Defining Styles For Your Application When you create your application it will provision a CSS file for you in the *static/css* directory. All your styles should go here (and any images you have should go in *static/art*). Your app's name will be a class that is assigned to the root of your app in the DOM. So if you created an app called "calculator" then every window you create for your app will have the class "calculator". Every style you define should be prefixed with this to prevent you from accidentally polluting the styles of any other app. Examples: /* the right way: */ .calculator p { /* all my paragraphs should have a margin of 8px */ margin: 8px; /* and a background from my art directory */ background: url(../art/paragraph.gif); } /* the wrong way: */ p { /* woops; we're styling all the paragraphs on the page, affecting every application! */ margin: 8px; background: url(../art/paragraph.gif); } #### Do Not Use IDs You should not use IDs in your HTML markup. This is because users can create more than one widow for your application. For instance, let's consider the File Browser. The user might launch the File Browser and view the contents of a directory where the output of a Job is being written. She may start another job and launch another File Browser to watch its output directory, too. If you use IDs in your applications you will have two elements on the page (one for each window) with the same ID. This is technically not legal HTML, but it's even worse if you start writing JavaScript and try and reference an element this way. **Always use classes.** Note that the JavaScript environment (which we cover later) always provides you with a pointer for the DOM element that represents the window your app instance is in. This allows you to select all the elements in a given CSS class limited to the instance's window to prevent you from accidentally selecting elements that might have the same CSS class in another window. #### Icons You should create an icon for your application that is a transparent png sized 55px by 55px. You'll probably want to put this in the header of your application to make it easy to recognize. We'll cover that when we get to the Jframe patterns and how to add a toolbar to your application. HUE includes a selection of around 1,500 small 16px by 16px icons that can be useful when making links that perform actions. These are open source icons that you can use in your own applications as you like. You can find these in *desktop/core/static/art/icons/* and *desktop/core/static/art/led-icons/* and you can style your elements to use them like this (in your css file): /* show an add icon next to the text of the link: */ .calculator a.add { background: url(/static/art/icons/add.png) no-repeat 1px 0px; display: block; height: 18px; padding-left: 22px; } /* or hide the link text and show only the icon: */ .calculator a.add { background: url(/static/art/icons/add.png) no-repeat 1px 0px; display: block; height: 16px; width: 16px text-indent: -200px; /* this pushes the text out of view, leaving only the icon */ overflow: hidden; } ### Registering your Application By default, all windows in HUE have a the same basic look and feel. If you don't do anything, you'll get a window with the history component (the forward, back, and refresh buttons and the history list / current location). You can style this window to be radically different if you choose, but doing so will take some JavaScript work on your part. Before we can change the style of our application's window, we need to change the way our application is invoked. By default, our bootstrap.js file looks like this: Hue.Desktop.register({ Calculator : { name : 'Calculator', css : '/calculator/static/css/calculator.css', require: [ /* put required js assets here example: */ 'JFrame/JFrame.Browser' ], launch: function(path, options){ // application launch code here // example code below: return new Calculator(path || '/calculator/', options); }, menu: { id: 'hue-calculator-menu', img: { // Replace this with a real icon! src: '/calculator/static/art/calculator.png' } } } }); Let's walk through this line by line. Hue.Desktop.register({ You can see we register our application with HUE. This does several things, but principally it tells the HUE environment that your app exists and how to invoke it as well as what dependencies it has. By default, you can see that we tell it about the following: Calculator : { name : 'calculator', The name is declared twice. The first time you declare it it's a property of the object we pass to the register method. The name here has to be a string with no spaces or punctuation (other than underscores). Typically it's a mixed case value, but it doesn't really matter as the user never sees it. It is a key that you'll use if you do a lot of customizing and debugging though, and it also is used as the CSS class your app is given (like ".calculator"). The second line though, "name: 'app_name'" defines the value that users see. You'll probably want to customize this. For example, the app key for the file browser is "FileBrowser" but the name defined for the users is "File Browser". This name shows up, for example, when the user mouses over the icon in the dock. css : '/calculator/static/css/calculator.css', This is pretty straightforward. Your application has its own dedicated css file that is included. If you don't have any styles you can comment this out. You cannot include more than one css file. require: [ /* put required js assets here example: */ 'JFrame/JFrame.Browser' ], Here's where things get interesting. By default, our application only requires the file JFrame/JFrame.Browser.js, which gives us all our default application functionality. There's really no reason to change this unless you want to add more JavaScript functionality; we'll cover how to do that in the next section. For now, let's leave this one alone and we'll revisit it later. launch: function(path, options){ // application launch code here // example code below: return new Calculator(path || '/calculator/', options); }, This defines the function that is executed when your application is launched. There are a few important requirements here: * Your application will always be passed a path and, maybe, an object with options (but not likely). * Your application will always return an object. In the default example it returns an instance of JFrame.Browser * If your application returns something else, it must be an object that inherits the properties of JFrame.Browser Finally, we have the menu icon: menu: { id: 'hue-calculator-menu', img: { // Replace this with a real icon! src: '/calculator/static/art/calculator.png' } } This object defines the id for the menu icon (one of the few places we use ids) and then properties of the image. 'src' is the url to the icon. It's actually not uncommon to have an application without adding it to the dock. ### Adding Interactive Elements to Your UI We'll talk a bit more about JavaScript in a bit, but one of the things the HUE SDK does for you is allow you to imbue your app with various UI behaviors by using pre-defined HTML markup. These patterns include things like: * Alert and confirmation messages * Client side form validation * Custom right-click context menus * Sortable tables * Tabbed interfaces * Tool tips * etc Adding these kinds of things to your application requires only that you use prescribed DOM structures with specific attributes. For example, let's say you wanted to have a sortable table in your response. All that you would need to do is write your template to output your table with the following HTML structure:
| ID | TimeZone | Name | GEO Latitude | GEO Longitude |
|---|---|---|---|---|
| 22 | New York City | America/New_York | 40.7255 | -73.9983 |
| 23 | San Francisco | America/Los_Angeles | 37.7587 | -122.433 |
note: this view will auto refresh in seconds
That element will be updated (emptied and filled) with the countdown. #### Putting Elements into the Navigation Bar When you provision your application we automatically set it up to have the history component in the header along with your app's icon. To accommodate these things, the header area of your application is given a relatively generous amount of space that you can use to show additional elements. You might want to use the space, for example, to add buttons to do things or to navigate. If you look around HUE itself you'll find a lot of examples. Here's the header of the File Browser application:  The icon and history components are there, but so are buttons to upload a file or create a folder. To get HTML into the navigation area of your application you simply give the container of that HTML a DIV with the CSS class `toolbar`. here's the relevant HTML of the File Browser: There are some other things going on here - the button styling is achieved by `data-filters="ArtButton"` the buttons themselves are styled with more CSS properties and even some custom values for the icon styles (the little images in the button next to the text). Let's gloss over these for now and just focus on the toolbar aspect itself. When JFrame retrieves your HTML from the server it looks in the response for any of these `toolbar` elements and when it finds them it moves them into the header. Every time JFrame loads a new page the header is emptied and filled anew. Thus if you want your icon to always be present, you need to have it in every response. In this case, we've also wrapped it in a link to go to the root view of the File Browser app, which makes it act something like a "home" button. **Important** Items in the Toolbar **must be positioned absolutely**. You can't push them around with margins or `top` and `left` values with the element positioned `relative`. If you don't use `position: absolute;` your element will overlay the close/minimize/maximize buttons and maybe even the history bar. You can have as many of these toolbar elements as you like in your response; they'll all get injected into the header. It's also worth noting that the size of the header - its height - is controlled in a JavaScript declaration using ART.Sheet. We'll cover how to change this, as well as how to style other things that are being rendered by JavaScript, when we touch on ART.Sheet below. #### Defining a "View" You can think of JFrame as being the portion of your application window between the header and the footer. In this image it's the white area with the folder list below the grey header and the grey footer:  If you were to wrap your HTML response in a DIV and give it a CSS class that was different for each view of your application with the purpose of styling it, you'd discover that you wouldn't be able to style the content that got moved into the header of the application window. This is because the JavaScript class that manages that window (JFrame.Browser) is the thing that actually moves content out of JFrame and into its header (a JFrame.Browser has a JFrame, but JFrame doesn't have any real knowledge that it's in a JFrame.Browser). So if you want to style your header or any of your other content depending on which view of your application is displayed you instead define that view by wrapping your response with a special DIV. This DIV is given the CSS class `view` and is a root level node (i.e. its direct parent is the BODY tag). You can give it additional classes if you want to, but these will only wrap the content area when it is displayed. In addition to the `view` CSS class, you assign this DIV an ID - one of the very few places where it's ok to use IDs. When such a response comes back, JFrame finds it and passes along the value assigned to the ID of this element as the "view" *and removes the ID attribute from the DOM element*. When JFrame renders your response, this ID will *not* be present. Instead, JFrame.Browser takes this ID and assigns it as a class name to the element that contains both the header and the body of your application. In this manner you can have styles that will affect both your header and body, despite the fact that these are split apart. Here's what that HTML might look like:I'm the section for Tab 1. Note that my UL has both a .tab_sections class but also a .jframe-clear; that's because the tabs are aligned to the right, so we need to clear that float.
(cropping out the rest of the text for space)
I'm the section for Tab 2. Notice how, since this section is much longer, I resize gracefully when I display and hide. Here's a paragraph or two of Lorem Ipsum just to make this section longer.
(cropping out the rest of the text for space)