Browse Source

HUE-8741 [doc] TOC re-organization of the SDK

Romain Rigaux 6 years ago
parent
commit
7ab3fde
2 changed files with 766 additions and 789 deletions
  1. 759 768
      docs/sdk/sdk.md
  2. 7 21
      package-lock.json

+ 759 - 768
docs/sdk/sdk.md

@@ -130,1054 +130,1045 @@ Here is an example on how the File Browser can list HDFS, S3 files and now [ADLS
 * [How to manage the Hue database with the shell](http://gethue.com/how-to-manage-the-hue-database-with-the-shell/).
 
 
-# New application
-
-Building a brand new application is more work but is ideal for creating a custom solution.
-
-## Introduction and Overview
 
-Hue leverages the browser to provide users with an environment for exploring
-and analyzing data.
+# Backend Development
 
-Build on top of the Hue SDK to enable your application to interact efficiently with
-Hadoop and the other Hue services.
 
-By building on top of Hue SDK, you get, out of the box:
+This section goes into greater detail on useful features within
+the Hue environment.
 
-+ Configuration Management
-+ Hadoop interoperability
-+ Supervision of subprocesses
-+ A collaborative UI
-+ Basic user administration and authorization
+## User Management
 
-This document will orient you with the general structure of Hue
-and will walk you through adding a new application using the SDK.
+Except for static content, `request.user` is always populated.  It is a
+standard Django `models.User` object.  If you were to set a breakpoint at the
+`index()` function in our calculator app, you will find:
 
+    >>> request.user
+    <User: test>
 
-### From 30,000 feet
+<div class="note">
+  "Under the covers:" Django uses a notion called
+  <a href="https://docs.djangoproject.com/en/1.2/topics/http/middleware/">middleware</a>
+  that's called in between the request coming in and the view being executed.
+  That's how <code>request.user</code> gets populated.  There's also a
+  middleware for Hue that makes sure that no pages are displayed unless the
+  user is authenticated.
+</div>
 
-![From up on high](from30kfeet.png)
+## Configuration
 
-Hue, as a "container" web application, sits in between your Hadoop installation
-and the browser.  It hosts all the Hue Apps, including the built-in ones, and
-ones that you may write yourself.
+### Configuration File
 
-### The Hue Server
+Hue uses a typed configuration system that reads configuration files (in an
+ini-style format).  By default, Hue loads all `*.ini` files in the `build/desktop/conf`
+directory.  The configuration files have the following format:
 
-![Web Back-end](webbackend.png)
+    # This is a comment
+    [ app_name ]          # Same as your app's name
+    app_property = "Pink Floyd"
 
-Hue is a web application built on the Django python web framework.
-Django, running on the WSGI container/web server (typically CherryPy), manages
-the url dispatch, executes application logic code, and puts together the views
-from their templates.  Django uses a database (typically sqlite)
-to manage session data, and Hue applications can use it as well
-for their "models".  (For example, the JobDesigner application stores
-job designs in the database.)
+    [[ section_a ]]         # The double brackets start a section under [ app_name ]
+    a_weight = 80         # that is useful for grouping
+    a_height = 180
 
-In addition to the web server, some Hue applications run
-daemon processes "on the side".  For example, Spark runs a daemon
-("livy_server") that keeps track of the Spark shells of the user. Running
-a separate process for applications is the preferred
-way to manage long-running tasks that you may wish
-to co-exist with web page rendering.  The web "views"
-typically communicate with these side daemons
-by using Thrift (e.g., for Hive query execution) or by exchanging state
-through the database.
+    [[ filesystems ]]       # Sections are also useful for making a list
+    [[[ cluster_1 ]]]       # All list members are sub-sections of the same type
+    namenode_host = localhost
+    # User may define more:
+    # [[[ cluster_2 ]]]
+    # namenode_host = 10.0.0.1
 
-### Interacting with Hadoop
 
-![Interacting with Hadoop](interactingwithhadoop.png)
+### Configuration Variables
 
-Hue provides some APIs for interacting with Hadoop.
-Most noticeably, there are python file-object-like APIs for
-interacting with HDFS.  These APIs work by making REST API or Thrift calls
-the Hadoop daemons. The Hadoop administrator must enable these interfaces from
-Hadoop.
+Your application's `conf.py` is special. It provides access to the configuration file (and even
+default configurations not specified in the file). Using the above example, your `conf.py` should
+define the following:
 
-### On the Front-End
+* A `desktop.lib.conf.Config` object for `app_property`, such as:
+<pre>
+  MY_PROPERTY = Config(key='app_property', default='Beatles', help='blah')
+</pre>
+  You can access its value by `MY_PROPERTY.get()`.
 
-Hue provides a front-end framework based on
-[Bootstrap](http://twitter.github.com/bootstrap/) and
-[Knockout js](http://knockoutjs.com/).
+* A `desktop.lib.conf.ConfigSection` object for `section_a`, such as:
+<pre>
+  SECTION_A = ConfigSection(key='section_a',
+        help='blah',
+        members=dict(
+          AWEIGHT=Config(key='a_weight', type=int, default=0),
+          AHEIGHT=Config(key='a_height', type=int, default=0)))
+</pre>
+  You can access the values by `SECTION_A.AWEIGHT.get()`.
 
+* A `desktop.lib.conf.UnspecifiedConfigSection` object for `filesystems`, such as:
+<pre>
+  FS = UnspecifiedConfigSection(
+      key='filesystems',
+      each=ConfigSection(members=dict(
+          nn_host=Config(key='namenode_host', required=True))
+</pre>
+  An `UnspecifiedConfigSection` is useful when the children of the section are not known.
+  When Hue loads your application's configuration, it binds all sub-sections. You can
+  access the values by:
+<pre>
+  cluster1_val = FS['cluster_1'].nn_host.get()
+  all_clusters = FS.keys()
+  for cluster in all_clusters:
+      val = FS[cluster].nn_host.get()
+</pre>
 
-### An Architectural View
+Your Hue application can automatically detect configuration problems and alert
+the admin. To take advantage of this feature, create a `config_validator`
+function in your `conf.py`:
 
-![Architecture](architecture.png)
+<pre>
+  def config_validator(user):
+    """
+    config_validator(user) -> [(config_variable, error_msg)] or None
+    Called by core check_config() view.
+    """
+    res = [ ]
+    if not REQUIRED_PROPERTY.get():
+      res.append((REQUIRED_PROPERTY, "This variable must be set"))
+    if MY_INT_PROPERTY.get() < 0:
+      res.append((MY_INT_PROPERTY, "This must be a non-negative number"))
+    return res
+</pre>
 
-A Hue application may span three tiers: (1) the UI
-and user interaction in the client's browser, (2) the
-core application logic in the Hue web
-server, and (3) external services with which applications
-may interact.
 
-The absolute minimum that you must implement (besides
-boilerplate), is a
-"Django [view](https://docs.djangoproject.com/en/1.11/#the-view-layer/)"
-function that processes the request and the associated template
-to render the response into HTML.
+<div class="note">
+  You should specify the <code>help="..."</code> argument to all configuration
+  related objects in your <code>conf.py</code>. The examples omit some for the
+  sake of space. But you and your application's users can view all the
+  configuration variables by doing:
+  <pre>
+    $ build/env/bin/hue config_help
+  </pre>
+</div>
 
-Many apps will evolve to have a bit of custom JavaScript and
-CSS styles. Apps that need to talk to an external service
-will pull in the code necessary to talk to that service.
 
-### File Layout
+### Running "Helper Processes"
 
-The Hue "framework" is in ``desktop/core/`` and contains the Web components.
-``desktop/libs/`` is the API for talking to various Hadoop services.
-The installable apps live in ``apps/``.  Please place third-party dependencies in the app's ext-py/
-directory.
+Some Hue applications need to run separate daemon processes on the side.
 
-The typical directory structure for inside an application includes:
-```
-  src/
-    for Python/Django code
-      models.py
-      urls.py
-      views.py
-      forms.py
-      settings.py
+Suppose your application needs a helper `my_daemon.py`. You need to register it by:
 
-  conf/
-    for configuration (``.ini``) files to be installed
+* In `setup.py`, add to `entry_points`:
+<pre>
+    entry_points = {
+      'desktop.sdk.application': 'my_app = my_app',
+      'desktop.supervisor.specs': [ 'my_daemon = my_app:SUPERVISOR_SPEC' ] }
+</pre>
 
-  static/
-    for static HTML/js resources and help doc
+* In `src/my_app/__init__.py`, tell Hue what to run by adding:
+<pre>
+    SUPERVISOR_SPEC = dict(django_command="my_daemon")
+</pre>
 
-  templates/
-    for data to be put through a template engine
+* Then in `src/my_app/management/commands`, create `__init__.py` and `my_daemon.py`. Your
+  daemon program has only one requirement: it must define a class called `Command` that
+  extends `django.core.management.base.BaseCommand`. Please see `kt_renewer.py` for an example.
 
-  locales/
-    for localizations in multiple languages
-```
+The next time Hue restarts, your `my_daemon` will start automatically.
+If your daemon program dies (exits with a non-zero exit code), Hue will
+restart it.
 
-For the URLs within your application, you should make your own ``urls.py``
-which will be automatically rooted at ``/yourappname/`` in the global
-namespace. See ``apps/about/src/about/urls.py`` for an example.
+"Under the covers:" Threading.  Hue, by default, runs CherryPy web server.
+If Hue is configured (and it may be, in the future)
+to use mod_wsgi under Apache httpd, then there would be multiple python
+processes serving the backend.  This means that your Django application
+code should avoid depending on shared process state.  Instead, place
+the stored state in a database or run a separate server.
 
+## Walk-through of a Django View
 
-## Pre-requisites
+![Django Flow](django_request.png)
 
+Django is an MVC framework, except that the controller is called a
+"[view](https://docs.djangoproject.com/en/1.11/#the-view-layer)" and
+the "view" is called a "template".  For an application developer, the essential
+flow to understand is how the "urls.py" file provides a mapping between URLs (expressed as a
+regular expression, optionally with captured parameters) and view functions.
+These view functions typically use their arguments (for example, the captured parameters) and
+their request object (which has, for example, the POST and GET parameters) to
+prepare dynamic content to be rendered using a template.
 
-### Dependencies
+## Templates: Django and Mako
 
-* The OS specific dependencies listed [here](http://cloudera.github.io/hue/latest/admin-manual/manual.html)
-* Python 2.7
-* Django (1.11 included with our distribution)
-* Hadoop (Apache Hadoop 2+)
-* Java (Java 1.8)
-* npm (6.4+)
+In Hue, the typical pattern for rendering data through a template
+is:
 
-### Recommended Reading / Important Technologies
+    from desktop.lib.django_util import render
 
-The following are core technologies used inside of Hue.
+    def view_function(request):
+      return render('view_function.mako', request, dict(greeting="hello"))
 
-* Python.  <a href="http://diveintopython.net/">Dive Into Python</a> is one of
-  several excellent books on python.
-* Django.  Start with [The Django Tutorial](https://docs.djangoproject.com/en/1.11/intro/).
-* [Thrift](http://incubator.apache.org/thrift/) is used for communication
-  between daemons.
-* [Mako](http://www.makotemplates.org/) is the preferred templating language.
+The `render()` function chooses a template engine (either Django or Mako) based on the
+extension of the template file (".html" or ".mako"). Mako templates are more powerful,
+in that they allow you to run arbitrary code blocks quite easily, and are more strict (some
+would say finicky); Django templates are simpler, but are less expressive.
 
-## Fast-Guide to Creating a New Hue Application
+## Django Models
 
-Now that we have a high-level overview of what's going on,
-let's go ahead and create a new installation.
+[Django Models](https://docs.djangoproject.com/en/1.11/#the-model-layer)
+are Django's Object-Relational Mapping framework. If your application
+needs to store data (history, for example), models are a good way to do it.
 
-### Download, Unpack, Build Distro
+From an abstraction perspective, it's common to imagine external services
+as "models".  For example, the Job Browser treats the Hadoop JobTracker
+as a "model", even though there's no database involved.
 
-The Hue SDK is available from [Github](http://github.com/cloudera/hue). Releases
-can be found on the [download page](http://gethue.com/category/release/).
-Releases are missing a few dependencies that could not be included because of
-licencing issues. So if you prefer to have an environment ready from scratch,
-it is preferable to checkout a particular release tag instead.
-
-    cd hue
-    ## Build
-    make apps
-    ## Run
-    build/env/bin/hue runserver
-    ## Alternative run
-    build/env/bin/hue supervisor
-    ## Visit http://localhost:8000/ with your web browser.
+## Accessing Hadoop
 
+It is common for applications to need to access the underlying HDFS.
+The `request.fs` object is a "file system" object that exposes
+operations that manipulate HDFS.  It is pre-configured to access
+HDFS as the user that's currently logged in.  Operations available
+on `request.fs` are similar to the file operations typically
+available in python.  See `webhdfs.py` for details; the list
+of functions available is as follows:
+`chmod`,
+`chown`,
+`exists`,
+`isdir`,
+`isfile`,
+`listdir` (and `listdir_stats`),
+`mkdir`,
+`open` (which exposes a file-like object with `read()`, `write()`, `seek()`, and `tell()` methods),
+`remove`,
+`rmdir`,
+`rmtree`, and
+`stats`.
 
-### Run "create_desktop_app" to Set up a New Source Tree
 
-    ./build/env/bin/hue create_desktop_app calculator
-    find calculator -type f
-    calculator/setup.py                                 # distutils setup file
-    calculator/src/calculator/__init__.py               # main src module
-    calculator/src/calculator/forms.py
-    calculator/src/calculator/models.py
-    calculator/src/calculator/settings.py               # app metadata setting
-    calculator/src/calculator/urls.py                   # url mapping
-    calculator/src/calculator/views.py                  # app business logic
-    calculator/src/calculator/templates/index.mako
-    calculator/src/calculator/templates/shared_components.mako
+## Making Your Views Thread-safe
 
-    # Static resources
-    calculator/src/static/calculator/art/calculator.png # logo
-    calculator/src/static/calculator/css/calculator.css
-    calculator/src/static/calculator/js/calculator.js
+Hue works in any WSGI-compliant container web server.
+The current recommended deployment server is the built-in CherryPy server.
+The CherryPy server, which is multi-threaded, is invoked by `runcpserver`
+and is configured to start when Hue's `supervisor` script is used.
+Meanwhile, `runserver` start a single-threaded
+testing server.
 
+Because multiple threads may be accessing your views
+concurrently, your views should not use shared state.
+An exception is that it is acceptable to initialize
+some state when the module is first imported.
+If you must use shared state, use Python's `threading.Lock`.
 
-<div class="note">
-  Some apps are blacklisted on certain versions of CDH (such as the 'Spark' app) due to
-  certain incompatibilities, which prevent them loading from in Hue.
-  Check the hue.ini 'app_blacklist' parameter for details.
-</div>
+Note that any module initialization may happen multiple times.
+Some WSGI containers (namely, Apache), will start multiple
+Unix processes, each with multiple threads. So, while
+you have to use locks to protect state within the process,
+there still may be multiple copies of this state.
 
-### Install SDK Application
+For persistent global state, it is common to place the state
+in the database or on the Browser local storage.
 
-As you'll discover if you look at calculator's <tt>setup.py</tt>,
-Hue uses a distutils <tt>entrypoint</tt> to
-register applications.  By installing the calculator
-package into Hue's python virtual environment,
-you'll install a new app.  The "app_reg.py" tool manages
-the applications that are installed. Note that in the following example, the value after the
-"--install" option is the path to the root directory of the application you want to install. In this
-example, it is a relative path to "/Users/philip/src/hue/calculator".
+## Authentication Backends
 
-        ./build/env/bin/python tools/app_reg/app_reg.py --install calculator --relative-paths
-        === Installing app at calculator
-        Updating registry with calculator (version 0.1)
-        --- Making egg-info for calculator
+Hue exposes a configuration flag ("auth") to configure
+a custom authentication backend.  See
+See http://docs.djangoproject.com/en/dev/topics/auth/#writing-an-authentication-backend
+for writing such a backend.
 
+In addition to that, backends may support a `manages_passwords_externally()` method, returning
+True or False, to tell the user manager application whether or not changing
+passwords within Hue is possible.
 
-<div class="note">
-  If you'd like to customize the build process, you can modify (or even complete
-  rewrite) your own `Makefile`, as long as it supports the set of required
-  targets. Please see `Makefile.sdk` for the required targets and their
-  semantics.
-</div>
+## Authorization
 
-Congrats, you've added a new app!
+Applications may define permission sets for different actions. Administrators
+can assign permissions to user groups in the UserAdmin application. To define
+custom permission sets, modify your app's `settings.py` to create a list of
+`(identifier, description)` tuples:
 
-<div class="note">
-  What was that all about?
-  <a href="http://pypi.python.org/pypi/virtualenv">virtualenv</a>
-  is a way to isolate python environments in your system, and isolate
-  incompatible versions of dependencies.  Hue uses the system python, and
-  that's about all.  It installs its own versions of dependencies.
+    PERMISSION_ACTIONS = [
+      ("delete", "Delete really important data"),
+      ("email", "Send email to the entire company"),
+      ("identifier", "Description of the permission")
+    ]
 
-  <a href="http://peak.telecommunity.com/DevCenter/PkgResources#entry-points">Entry Points</a>
-  are a way for packages to optionally hook up with other packages.
-</div>
+Then you can use this decorator on your view functions to enforce permission:
 
-You can now browse the new application.
+    @desktop.decorators.hue_permission_required("delete", "my_app_name")
+    def delete_financial_report(request):
+      ...
 
-    # If you haven't killed the old process, do so now.
-    build/env/bin/hue runserver
+## Using and Installing Thrift
 
-And then visit <a href="http://localhost:8000">http://localhost:8000/</a> to check it out!
-You should see the app in the left menu.
+Right now, we check in the generated thrift code.
+To generate the code, you'll need the thrift binary version 0.9.0.
+Please download from http://thrift.apache.org/.
 
+The modules using ``Thrift`` have some helper scripts like ``regenerate_thrift.sh``
+for regenerating the code from the interfaces.
 
-### Customizing Views and Templates
+## Profiling Hue Apps
 
-Now that your app has been installed, you'll want to customize it.
-As you may have guessed, we're going to build a small calculator
-application.  Edit `calculator/src/calculator/templates/index.mako`
-to include a simple form and a Knockout viewmodel:
+Hue has a profiling system built in, which can be used to analyze server-side
+performance of applications.  To enable profiling::
 
+    build/env/bin/hue runprofileserver
 
-    <%!from desktop.views import commonheader, commonfooter %>
-    <%namespace name="shared" file="shared_components.mako" />
+Then, access the page that you want to profile.  This will create files like
+/tmp/useradmin.users.000072ms.2011-02-21T13:03:39.745851.prof.  The format for
+the file names is /tmp/<app_module>.<page_url>.<time_taken>.<timestamp>.prof.
 
-    %if not is_embeddable:
-    ${commonheader("Calculator", "calculator", user, "100px") | n,unicode}
-    %endif
+Hue uses the hotshot profiling library for instrumentation.  The documentation
+for this library is located at: http://docs.python.org/library/hotshot.html.
 
-    ## Main body
-    <div class="container-fluid calculator-components">
-      <div class="row">
-        <div class="span6 offset3 margin-top-30 text-center">
-          <form class="form-inline">
-            <input type="text" class="input-mini margin-right-10" placeholder="A" data-bind="value: a">
-            <!-- ko foreach: operations -->
-            <label class="radio margin-left-5">
-              <input type="radio" name="op" data-bind="checkedValue: $data, checked: $parent.chosenOperation" /><span data-bind="text: $data"></span>
-            </label>
-            <!-- /ko -->
-            <input type="text" class="input-mini margin-left-10" placeholder="B" data-bind="value: b">
-            <button class="btn" data-bind="click: calculate">Calculate</button>
-          </form>
+You can use kcachegrind to view the profiled data graphically::
 
-          <h2 data-bind="visible: result() !== null">The result is <strong data-bind="text: result"></strong></h2>
-        </div>
-      </div>
-    </div>
+    $ hotshot2calltree /tmp/xyz.prof > /tmp/xyz.trace
+    $ kcachegrind /tmp/xyz.trace
 
-    <script>
-      (function() {
-        var CalculatorViewModel = function () {
-          var self = this;
+More generally, you can programmatically inspect a trace::
 
-          self.operations = ko.observableArray(['+', '-', '*', '/']);
+    #!/usr/bin/python
+    import hotshot.stats
+    import sys
 
-          self.a = ko.observable();
-          self.b = ko.observable();
-          self.chosenOperation = ko.observable('+');
-          self.result = ko.observable(null);
+    stats = hotshot.stats.load(sys.argv[1])
+    stats.sort_stats('cumulative', 'calls')
+    stats.print_stats(100)
 
-          self.calculate = function () {
-            var a = parseFloat(self.a());
-            var b = parseFloat(self.b());
-            var result = null;
-            switch (self.chosenOperation()) {
-              case '+':
-                result = a + b;
-                break;
-              case '-':
-                result = a - b;
-                break;
-              case '*':
-                result = a * b;
-                break;
-              case '/':
-                result = a / b;
-            }
-            self.result(result);
-          }
-        };
-        $(document).ready(function () {
-          ko.applyBindings(new CalculatorViewModel(), $('.calculator-components')[0]);
-        });
-      })();
-    </script>
+This script takes in a .prof file, and orders function calls by the cumulative
+time spent in that function, followed by the number of times the function was
+called, and then prints out the top 100 time-wasters.  For information on the
+other stats available, take a look at this website:
+http://docs.python.org/library/profile.html#pstats.Stats
 
-    %if not is_embeddable:
-    ${commonfooter(messages) | n,unicode}
-    %endif
 
-The template language here is <a href="http://www.makotemplates.org/docs/">Mako</a>,
-which is flexible and powerful.  If you use the "`.html`" extension, Hue
-will render your page using
-<a href="https://docs.djangoproject.com/en/1.11/#the-template-layer">Django templates</a>
-instead.
 
-Note that we use Knockout.js to do the heavy lifting of this app.
+## Django Models
 
-Let's edit `calculator/src/calculator/views.py` to simply render the page:
+Each app used to have its own model to store its data (e.g. a SQL query, a workflow). In Hue 3
+a unification of all the models happened and any app now uses a single Document2 model:
+``desktop/core/src/desktop/models.py``. This enables to avoid simply re-use document
+creation, sharing, saving etc...
 
-    #!/usr/bin/env python
+## REST
+Hue is Ajax based and has a REST API used by the browser to communicate (e.g. submit a query or workflow,
+list some S3 files, export a document...). Currently this API is private and subject to change but
+can be easily reused. You would need to GET ``/accounts/login`` to get the CSRF token
+and POST it back along ``username`` and ``password`` and reuse the ``sessionid`` cookie in next
+communication calls.
 
-    from desktop.lib.django_util import render
+** With Python Request **
 
-    def index(request):
-      return render('index.mako', request, {
-        'is_embeddable': request.GET.get('is_embeddable', False),
-      })
+Hue is based on the Django Web Framework. Django comes with user authentication system. Django uses sessions and middleware to hook the authentication system into request object. HUE uses stock auth form which uses “username” and “password” and “csrftoken” form variables to authenticate.
 
+In this code snippet, we will use well-known python “requests” library. we will first acquire “csrftoken” by GET “login_url”. We will create python dictionary of form data which contains “username”, “password” and “csrftoken” and the “next_url” and another python dictionary for header which contains the “Referer” url and empty python dictionary for the cookies. After POST request to “login_url” we will get status. Check the r.status_code. If r.status_code!=200 then you have problem in username and/or password.
 
-You can now go and try the calculator.
+Once the request is successful then capture headers and cookies for subsequent requests. Subsequent request.session calls can be made by providing cookies=session.cookies and headers=session.headers.
 
+<pre>
+import requests
 
-## Backend Development
+def login_djangosite():
+ next_url = "/"
+ login_url = "http://localhost:8888/accounts/login?next=/"
 
+ session = requests.Session()
+ r = session.get(login_url)
+ form_data = dict(username="[your hue username]",password="[your hue password]",
+                  csrfmiddlewaretoken=session.cookies['csrftoken'],next=next_url)
+ r = session.post(login_url, data=form_data, cookies=dict(), headers=dict(Referer=login_url))
 
-This section goes into greater detail on useful features within
-the Hue environment.
+ # check if request executed successfully?
+ print r.status_code
 
-### User Management
+ cookies = session.cookies
+ headers = session.headers
 
-Except for static content, `request.user` is always populated.  It is a
-standard Django `models.User` object.  If you were to set a breakpoint at the
-`index()` function in our calculator app, you will find:
+ r=session.get('http://localhost:8888/metastore/databases/default/metadata',
+ cookies=session.cookies, headers=session.headers)
+ print r.status_code
 
-    >>> request.user
-    <User: test>
+ # check metadata output
+ print r.text
+</pre>
+
+[Read more about it here](http://gethue.com/login-into-hue-using-the-python-request-library/).
 
 <div class="note">
-  "Under the covers:" Django uses a notion called
-  <a href="https://docs.djangoproject.com/en/1.2/topics/http/middleware/">middleware</a>
-  that's called in between the request coming in and the view being executed.
-  That's how <code>request.user</code> gets populated.  There's also a
-  middleware for Hue that makes sure that no pages are displayed unless the
-  user is authenticated.
+  http://issues.cloudera.org/browse/HUE-1450 is tracking a more official public API.
 </div>
 
-### Configuration
 
-#### Configuration File
+## Upgrade path
 
-Hue uses a typed configuration system that reads configuration files (in an
-ini-style format).  By default, Hue loads all `*.ini` files in the `build/desktop/conf`
-directory.  The configuration files have the following format:
+After upgrading the version of Hue, running these two commands will make sure the
+database has the correct tables and fields.
 
-    # This is a comment
-    [ app_name ]          # Same as your app's name
-    app_property = "Pink Floyd"
+    ./build/env/bin/hue syncdb
+    ./build/env/bin/hue migrate
 
-    [[ section_a ]]         # The double brackets start a section under [ app_name ]
-    a_weight = 80         # that is useful for grouping
-    a_height = 180
+# Front-end Development
 
-    [[ filesystems ]]       # Sections are also useful for making a list
-    [[[ cluster_1 ]]]       # All list members are sub-sections of the same type
-    namenode_host = localhost
-    # User may define more:
-    # [[[ cluster_2 ]]]
-    # namenode_host = 10.0.0.1
+Developing applications for Hue requires a minimal amount of CSS
+(and potentially JavaScript) to use existing functionality. As covered above,
+creating an application for the Hue is a matter of creating a standard HTML
+application.
 
+In a nutshell, front-end development in Hue is using
+[Bootstrap](http://twitter.github.com/bootstrap/) and
+[Knockout js](http://knockoutjs.com/) to layout your app and script the custom
+interactions.
 
-#### Configuration Variables
 
-Your application's `conf.py` is special. It provides access to the configuration file (and even
-default configurations not specified in the file). Using the above example, your `conf.py` should
-define the following:
+## CSS Styles
 
-* A `desktop.lib.conf.Config` object for `app_property`, such as:
-<pre>
-  MY_PROPERTY = Config(key='app_property', default='Beatles', help='blah')
-</pre>
-  You can access its value by `MY_PROPERTY.get()`.
+Hue uses [Bootstrap](http://twitter.github.com/bootstrap/) version 2.0 CSS
+styles and layouts. They are highly reusable and flexible. Your app doesn't
+have to use these styles, but if you do, it'll save you some time and make your
+app look at home in Hue.
 
-* A `desktop.lib.conf.ConfigSection` object for `section_a`, such as:
-<pre>
-  SECTION_A = ConfigSection(key='section_a',
-        help='blah',
-        members=dict(
-          AWEIGHT=Config(key='a_weight', type=int, default=0),
-          AHEIGHT=Config(key='a_height', type=int, default=0)))
-</pre>
-  You can access the values by `SECTION_A.AWEIGHT.get()`.
+On top of the standard Bootstrap styles, Hue defines a small set of custom
+styles in *desktop/core/static/css/jhue.css*.
 
-* A `desktop.lib.conf.UnspecifiedConfigSection` object for `filesystems`, such as:
-<pre>
-  FS = UnspecifiedConfigSection(
-      key='filesystems',
-      each=ConfigSection(members=dict(
-          nn_host=Config(key='namenode_host', required=True))
-</pre>
-  An `UnspecifiedConfigSection` is useful when the children of the section are not known.
-  When Hue loads your application's configuration, it binds all sub-sections. You can
-  access the values by:
-<pre>
-  cluster1_val = FS['cluster_1'].nn_host.get()
-  all_clusters = FS.keys()
-  for cluster in all_clusters:
-      val = FS[cluster].nn_host.get()
-</pre>
+## Defining Styles for Your Application
 
-Your Hue application can automatically detect configuration problems and alert
-the admin. To take advantage of this feature, create a `config_validator`
-function in your `conf.py`:
+When you create your application it will provision a CSS file for you in the
+*static/css* directory. For organization purposes, your styles should go here
+(and any images you have should go in *static/art*). Your app's name will be a
+class that is assigned to the root of your app in the DOM. So if you created an
+app called "calculator" then every window you create for your app will have the
+class "calculator".  Every style you define should be prefixed with this to
+prevent you from accidentally colliding with the framework style. Examples:
 
-<pre>
-  def config_validator(user):
-    """
-    config_validator(user) -> [(config_variable, error_msg)] or None
-    Called by core check_config() view.
-    """
-    res = [ ]
-    if not REQUIRED_PROPERTY.get():
-      res.append((REQUIRED_PROPERTY, "This variable must be set"))
-    if MY_INT_PROPERTY.get() < 0:
-      res.append((MY_INT_PROPERTY, "This must be a non-negative number"))
-    return res
-</pre>
+    /* the right way: */
+    .calculator p {
+      /* all my paragraphs should have a margin of 8px */
+      margin: 8px;
+      /* and a background from my art directory */
+      background: url(../art/paragraph.gif);
+    }
+    /* the wrong way: */
+    p {
+      /* woops; we're styling all the paragraphs on the page, affecting
+         the common header! */
+      margin: 8px;
+      background: url(../art/paragraph.gif);
+    }
 
+## Icons
+
+You should create an icon for your application that is a transparent png sized
+24px by 24px. Your `settings.py` file should point to your icon via the `ICON`
+variable. The `create_desktop_app` command creates a default icon for you.
 
 <div class="note">
-  You should specify the <code>help="..."</code> argument to all configuration
-  related objects in your <code>conf.py</code>. The examples omit some for the
-  sake of space. But you and your application's users can view all the
-  configuration variables by doing:
-  <pre>
-    $ build/env/bin/hue config_help
-  </pre>
+  If you do not define an application icon, your application will not show up
+  in the navigation bar.
 </div>
 
+Hue ships with Twitter Bootstrap and Font Awesome 3 (http://fortawesome.github.io/Font-Awesome/)
+so you have plenty of scalable icons to choose from. You can style your elements to use them
+like this (in your mako template):
 
-#### Running "Helper Processes"
+    <!-- show a trash icon in a link -->
+    <a href="#something"><i class="icon-trash"></i> Trash</a>
 
-Some Hue applications need to run separate daemon processes on the side.
+## Static files
 
-Suppose your application needs a helper `my_daemon.py`. You need to register it by:
+For better performances, Hue uses the Django staticfiles app. If in production mode, if you edit
+some static files, you would need to run this command or `make apps`. No actions are needed in
+development mode.
+```
+./build/env/bin/hue collectstatic
+```
 
-* In `setup.py`, add to `entry_points`:
-<pre>
-    entry_points = {
-      'desktop.sdk.application': 'my_app = my_app',
-      'desktop.supervisor.specs': [ 'my_daemon = my_app:SUPERVISOR_SPEC' ] }
-</pre>
+## Adding Interactive Elements to Your UI
 
-* In `src/my_app/__init__.py`, tell Hue what to run by adding:
-<pre>
-    SUPERVISOR_SPEC = dict(django_command="my_daemon")
-</pre>
+Hue by default loads these JavaScript components:
 
-* Then in `src/my_app/management/commands`, create `__init__.py` and `my_daemon.py`. Your
-  daemon program has only one requirement: it must define a class called `Command` that
-  extends `django.core.management.base.BaseCommand`. Please see `kt_renewer.py` for an example.
+* Ko js
+* jQuery
+* Bootstrap
 
-The next time Hue restarts, your `my_daemon` will start automatically.
-If your daemon program dies (exits with a non-zero exit code), Hue will
-restart it.
+These are used by some Hue applications, but not loaded by default:
 
-"Under the covers:" Threading.  Hue, by default, runs CherryPy web server.
-If Hue is configured (and it may be, in the future)
-to use mod_wsgi under Apache httpd, then there would be multiple python
-processes serving the backend.  This means that your Django application
-code should avoid depending on shared process state.  Instead, place
-the stored state in a database or run a separate server.
+* Knockout js (`desktop/core/static/ext/js/knockout-min.js`)
+* jQuery UI (`desktop/core/static/ext/js/jquery/plugins/jquery-ui-autocomplete-1.8.18.min.js`)
 
-<!-- "Wheel reinvention" Supervisor is following the Erlang model. -->
+These standard components have their own online documentation, which we will
+not repeat here. They let you write interactive behaviors with little or no
+JavaScript.
 
-### Walk-through of a Django View
 
-![Django Flow](django_request.png)
+## Debugging Tips and Tricks
 
-Django is an MVC framework, except that the controller is called a
-"[view](https://docs.djangoproject.com/en/1.11/#the-view-layer)" and
-the "view" is called a "template".  For an application developer, the essential
-flow to understand is how the "urls.py" file provides a mapping between URLs (expressed as a
-regular expression, optionally with captured parameters) and view functions.
-These view functions typically use their arguments (for example, the captured parameters) and
-their request object (which has, for example, the POST and GET parameters) to
-prepare dynamic content to be rendered using a template.
+* Set `DESKTOP_DEBUG=1` as an environment variable if you want logs to go to stderr
+  as well as to the respective log files.
+* Use runserver.  If you want to set a CLI breakpoint, just insert
+  `__import__("ipdb").set_trace()`
+  into your code.
+* Django tends to restart its server whenever it notices a file changes.  For
+  certain things (like configuration changes), this is not sufficient.  Restart
+  the server whole-heartedly.
+* We recommend developing with the Chrome console.
 
-#### Templates: Django and Mako
+## Building
 
-In Hue, the typical pattern for rendering data through a template
-is:
+### Documentation
 
-    from desktop.lib.django_util import render
+Building with
 
-    def view_function(request):
-      return render('view_function.mako', request, dict(greeting="hello"))
+    make docs
 
-The `render()` function chooses a template engine (either Django or Mako) based on the
-extension of the template file (".html" or ".mako"). Mako templates are more powerful,
-in that they allow you to run arbitrary code blocks quite easily, and are more strict (some
-would say finicky); Django templates are simpler, but are less expressive.
+### Javascript
 
-### Django Models
+The javascript files are currently being migrated to webpack bundles, during this process some files will live under src/desktop/static/ and some will live under src/dekstop/js
 
-[Django Models](https://docs.djangoproject.com/en/1.11/#the-model-layer)
-are Django's Object-Relational Mapping framework. If your application
-needs to store data (history, for example), models are a good way to do it.
+#### For changes to the files under src/desktop/js the following applies:
 
-From an abstraction perspective, it's common to imagine external services
-as "models".  For example, the Job Browser treats the Hadoop JobTracker
-as a "model", even though there's no database involved.
+First make sure all third-party dependencies defined in package.json are installed into node_modules/
 
-### Accessing Hadoop
+    npm install
 
-It is common for applications to need to access the underlying HDFS.
-The `request.fs` object is a "file system" object that exposes
-operations that manipulate HDFS.  It is pre-configured to access
-HDFS as the user that's currently logged in.  Operations available
-on `request.fs` are similar to the file operations typically
-available in python.  See `webhdfs.py` for details; the list
-of functions available is as follows:
-`chmod`,
-`chown`,
-`exists`,
-`isdir`,
-`isfile`,
-`listdir` (and `listdir_stats`),
-`mkdir`,
-`open` (which exposes a file-like object with `read()`, `write()`, `seek()`, and `tell()` methods),
-`remove`,
-`rmdir`,
-`rmtree`, and
-`stats`.
+Also run this after making changes to package.json, adding new third-party dependencies etc.
 
+To generate the js bundles run:
 
-### Making Your Views Thread-safe
+    npm run webpack
+    npm run webpack-workers
+    npm run webpack-login
 
-Hue works in any WSGI-compliant container web server.
-The current recommended deployment server is the built-in CherryPy server.
-The CherryPy server, which is multi-threaded, is invoked by `runcpserver`
-and is configured to start when Hue's `supervisor` script is used.
-Meanwhile, `runserver` start a single-threaded
-testing server.
+During development the bundles can be autogenerated when it detects changes to the .js files, for this run:
 
-Because multiple threads may be accessing your views
-concurrently, your views should not use shared state.
-An exception is that it is acceptable to initialize
-some state when the module is first imported.
-If you must use shared state, use Python's `threading.Lock`.
-
-Note that any module initialization may happen multiple times.
-Some WSGI containers (namely, Apache), will start multiple
-Unix processes, each with multiple threads. So, while
-you have to use locks to protect state within the process,
-there still may be multiple copies of this state.
+    npm run dev
 
-For persistent global state, it is common to place the state
-in the database or on the Browser local storage.
+Before sending a review with changes to the bundles run:
 
-## Authentication Backends
+    npm run lint-fix
 
-Hue exposes a configuration flag ("auth") to configure
-a custom authentication backend.  See
-See http://docs.djangoproject.com/en/dev/topics/auth/#writing-an-authentication-backend
-for writing such a backend.
+and possibly fix any issues it might report.
 
-In addition to that, backends may support a `manages_passwords_externally()` method, returning
-True or False, to tell the user manager application whether or not changing
-passwords within Hue is possible.
+### CSS / LESS
 
-### Authorization
+After changing the CSS in a .less file, rebuilding with:
 
-Applications may define permission sets for different actions. Administrators
-can assign permissions to user groups in the UserAdmin application. To define
-custom permission sets, modify your app's `settings.py` to create a list of
-`(identifier, description)` tuples:
+    make css
 
-    PERMISSION_ACTIONS = [
-      ("delete", "Delete really important data"),
-      ("email", "Send email to the entire company"),
-      ("identifier", "Description of the permission")
-    ]
+### SQL Autocomplete
 
-Then you can use this decorator on your view functions to enforce permission:
+Install a patched jison:
 
-    @desktop.decorators.hue_permission_required("delete", "my_app_name")
-    def delete_financial_report(request):
-      ...
+    git clone https://github.com/JohanAhlen/jison
+    cd jison
+    npm install -g .
 
-### Using and Installing Thrift
+Then run:
 
-Right now, we check in the generated thrift code.
-To generate the code, you'll need the thrift binary version 0.9.0.
-Please download from http://thrift.apache.org/.
+    make sql-all-parsers
 
-The modules using ``Thrift`` have some helper scripts like ``regenerate_thrift.sh``
-for regenerating the code from the interfaces.
+### Ace Editor
 
-### Profiling Hue Apps
+After modifying files under tools/ace-editor run the following to build ace.js
 
-Hue has a profiling system built in, which can be used to analyze server-side
-performance of applications.  To enable profiling::
+    npm install
+    make ace
 
-    build/env/bin/hue runprofileserver
+### Internationalization
 
-Then, access the page that you want to profile.  This will create files like
-/tmp/useradmin.users.000072ms.2011-02-21T13:03:39.745851.prof.  The format for
-the file names is /tmp/<app_module>.<page_url>.<time_taken>.<timestamp>.prof.
+How to update all the messages and compile them:
 
-Hue uses the hotshot profiling library for instrumentation.  The documentation
-for this library is located at: http://docs.python.org/library/hotshot.html.
+    make locales
 
-You can use kcachegrind to view the profiled data graphically::
+How to update and compile the messages of one app:
 
-    $ hotshot2calltree /tmp/xyz.prof > /tmp/xyz.trace
-    $ kcachegrind /tmp/xyz.trace
+    cd apps/beeswax
+    make compile-locale
 
-More generally, you can programmatically inspect a trace::
+How to create a new locale for an app:
 
-    #!/usr/bin/python
-    import hotshot.stats
-    import sys
+    cd $APP_ROOT/src/$APP_NAME/locale
+    $HUE_ROOT/build/env/bin/pybabel init -D django -i en_US.pot -d . -l fr
 
-    stats = hotshot.stats.load(sys.argv[1])
-    stats.sort_stats('cumulative', 'calls')
-    stats.print_stats(100)
+# API
 
-This script takes in a .prof file, and orders function calls by the cumulative
-time spent in that function, followed by the number of times the function was
-called, and then prints out the top 100 time-wasters.  For information on the
-other stats available, take a look at this website:
-http://docs.python.org/library/profile.html#pstats.Stats
+## Metadata Catalog
 
+The [metadata API](https://github.com/cloudera/hue/tree/master/desktop/libs/metadata) is powering [Search and Tagging here](http://gethue.com/improved-sql-exploration-in-hue-4-3/) and the [Query Assistant with Navigator Optimizer Integration](http://gethue.com/hue-4-sql-editor-improvements/).
 
+The backends is pluggable by providing alternative [client interfaces](https://github.com/cloudera/hue/tree/master/desktop/libs/metadata/catalog):
 
-## Django Models
+* navigator (default)
+* dummy
 
-Each app used to have its own model to store its data (e.g. a SQL query, a workflow). In Hue 3
-a unification of all the models happened and any app now uses a single Document2 model:
-``desktop/core/src/desktop/models.py``. This enables to avoid simply re-use document
-creation, sharing, saving etc...
+### Searching for entities
 
-## REST
-Hue is Ajax based and has a REST API used by the browser to communicate (e.g. submit a query or workflow,
-list some S3 files, export a document...). Currently this API is private and subject to change but
-can be easily reused. You would need to GET ``/accounts/login`` to get the CSRF token
-and POST it back along ``username`` and ``password`` and reuse the ``sessionid`` cookie in next
-communication calls.
+    $.post("/metadata/api/catalog/search_entities_interactive/", {
+        query_s: ko.mapping.toJSON("*sample"),
+        sources: ko.mapping.toJSON(["sql", "hdfs", "s3"]),
+        field_facets: ko.mapping.toJSON([]),
+        limit: 10
+    }, function(data) {
+        console.log(ko.mapping.toJSON(data));
+    });
 
-** With Python Request **
 
-Hue is based on the Django Web Framework. Django comes with user authentication system. Django uses sessions and middleware to hook the authentication system into request object. HUE uses stock auth form which uses “username” and “password” and “csrftoken” form variables to authenticate.
+### Searching for entities with the dummy backend
 
-In this code snippet, we will use well-known python “requests” library. we will first acquire “csrftoken” by GET “login_url”. We will create python dictionary of form data which contains “username”, “password” and “csrftoken” and the “next_url” and another python dictionary for header which contains the “Referer” url and empty python dictionary for the cookies. After POST request to “login_url” we will get status. Check the r.status_code. If r.status_code!=200 then you have problem in username and/or password.
+    $.post("/metadata/api/catalog/search_entities_interactive/", {
+        query_s: ko.mapping.toJSON("*sample"),
+        interface: "dummy"
+    }, function(data) {
+        console.log(ko.mapping.toJSON(data));
+    });
 
-Once the request is successful then capture headers and cookies for subsequent requests. Subsequent request.session calls can be made by providing cookies=session.cookies and headers=session.headers.
+### Finding an entity in order to get its id
 
-<pre>
-import requests
+    $.get("/metadata/api/navigator/find_entity", {
+        type: "table",
+        database: "default",
+        name: "sample_07",
+        interface: "dummy"
+    }, function(data) {
+        console.log(ko.mapping.toJSON(data));
+    });
 
-def login_djangosite():
- next_url = "/"
- login_url = "http://localhost:8888/accounts/login?next=/"
+### Adding/updating a comment with the dummy backend
 
- session = requests.Session()
- r = session.get(login_url)
- form_data = dict(username="[your hue username]",password="[your hue password]",
-                  csrfmiddlewaretoken=session.cookies['csrftoken'],next=next_url)
- r = session.post(login_url, data=form_data, cookies=dict(), headers=dict(Referer=login_url))
+    $.post("/metadata/api/catalog/update_properties/", {
+        id: "22",
+        properties: ko.mapping.toJSON({"description":"Adding a description"}),
+        interface: "dummy"
+    }, function(data) {
+        console.log(ko.mapping.toJSON(data));
+    });
 
- # check if request executed successfully?
- print r.status_code
+### Adding a tag with the dummy backend
 
- cookies = session.cookies
- headers = session.headers
+    $.post("/metadata/api/catalog/add_tags/", {
+      id: "22",
+      tags: ko.mapping.toJSON(["usage"]),
+      interface: "dummy"
+    }, function(data) {
+        console.log(ko.mapping.toJSON(data));
+    });
 
- r=session.get('http://localhost:8888/metastore/databases/default/metadata',
- cookies=session.cookies, headers=session.headers)
- print r.status_code
+### Deleting a key/value property
 
- # check metadata output
- print r.text
-</pre>
+    $.post("/metadata/api/catalog/delete_metadata_properties/", {
+       "id": "32",
+       "keys": ko.mapping.toJSON(["project", "steward"])
+    }, function(data) {
+       console.log(ko.mapping.toJSON(data));
+    });
 
-[Read more about it here](http://gethue.com/login-into-hue-using-the-python-request-library/).
+### Deleting a key/value property
 
-<div class="note">
-  http://issues.cloudera.org/browse/HUE-1450 is tracking a more official public API.
-</div>
+    $.post("/metadata/api/catalog/delete_metadata_properties/", {
+      "id": "32",
+      "keys": ko.mapping.toJSON(["project", "steward"])
+    }, function(data) {
+      console.log(ko.mapping.toJSON(data));
+    });
 
 
-## Upgrade path
+### Getting the model mapping of custom metadata
 
-After upgrading the version of Hue, running these two commands will make sure the
-database has the correct tables and fields.
+    $.get("/metadata/api/catalog/models/properties/mappings/", function(data) {
+      console.log(ko.mapping.toJSON(data));
+    });
 
-    ./build/env/bin/hue syncdb
-    ./build/env/bin/hue migrate
 
-## Front-end Development
+### Getting a namespace
 
-Developing applications for Hue requires a minimal amount of CSS
-(and potentially JavaScript) to use existing functionality. As covered above,
-creating an application for the Hue is a matter of creating a standard HTML
-application.
+    $.post("/metadata/api/catalog/namespace/", {
+      namespace: 'huecatalog'
+    }, function(data) {
+      console.log(ko.mapping.toJSON(data));
+    });
 
-In a nutshell, front-end development in Hue is using
-[Bootstrap](http://twitter.github.com/bootstrap/) and
-[Knockout js](http://knockoutjs.com/) to layout your app and script the custom
-interactions.
+### Creating a namespace
 
+    $.post("/metadata/api/catalog/namespace/create/", {
+      "namespace": "huecatalog",
+      "description": "my desc"
+    }, function(data) {
+      console.log(ko.mapping.toJSON(data));
+    });
 
-### CSS Styles
 
-Hue uses [Bootstrap](http://twitter.github.com/bootstrap/) version 2.0 CSS
-styles and layouts. They are highly reusable and flexible. Your app doesn't
-have to use these styles, but if you do, it'll save you some time and make your
-app look at home in Hue.
+### Creating a namespace property
 
-On top of the standard Bootstrap styles, Hue defines a small set of custom
-styles in *desktop/core/static/css/jhue.css*.
+    $.post("/metadata/api/catalog/namespace/property/create/", {
+      "namespace": "huecatalog",
+      "properties": ko.mapping.toJSON({
+        "name" : "relatedEntities2",
+        "displayName" : "Related objects",
+        "description" : "My desc",
+        "multiValued" : true,
+        "maxLength" : 50,
+        "pattern" : ".*",
+        "enumValues" : null,
+        "type" : "TEXT"
+      })
+    }, function(data) {
+      console.log(ko.mapping.toJSON(data));
+    });
 
-### Defining Styles for Your Application
 
-When you create your application it will provision a CSS file for you in the
-*static/css* directory. For organization purposes, your styles should go here
-(and any images you have should go in *static/art*). Your app's name will be a
-class that is assigned to the root of your app in the DOM. So if you created an
-app called "calculator" then every window you create for your app will have the
-class "calculator".  Every style you define should be prefixed with this to
-prevent you from accidentally colliding with the framework style. Examples:
+### Map a namespace property to a class entity
 
-    /* the right way: */
-    .calculator p {
-      /* all my paragraphs should have a margin of 8px */
-      margin: 8px;
-      /* and a background from my art directory */
-      background: url(../art/paragraph.gif);
-    }
-    /* the wrong way: */
-    p {
-      /* woops; we're styling all the paragraphs on the page, affecting
-         the common header! */
-      margin: 8px;
-      background: url(../art/paragraph.gif);
-    }
-
-### Icons
+    $.post("/metadata/api/catalog/namespace/property/map/", {
+      "class": "hv_view",
+      "properties": ko.mapping.toJSON([{
+          namespace: "huecatalog",
+          name: "relatedQueries"
+      }])
+    }, function(data) {
+      console.log(ko.mapping.toJSON(data));
+    });
 
-You should create an icon for your application that is a transparent png sized
-24px by 24px. Your `settings.py` file should point to your icon via the `ICON`
-variable. The `create_desktop_app` command creates a default icon for you.
+# New application
 
-<div class="note">
-  If you do not define an application icon, your application will not show up
-  in the navigation bar.
-</div>
+Building a brand new application is more work but is ideal for creating a custom solution.
 
-Hue ships with Twitter Bootstrap and Font Awesome 3 (http://fortawesome.github.io/Font-Awesome/)
-so you have plenty of scalable icons to choose from. You can style your elements to use them
-like this (in your mako template):
+## Introduction and Overview
 
-    <!-- show a trash icon in a link -->
-    <a href="#something"><i class="icon-trash"></i> Trash</a>
+Hue leverages the browser to provide users with an environment for exploring
+and analyzing data.
 
-### Static files
+Build on top of the Hue SDK to enable your application to interact efficiently with
+Hadoop and the other Hue services.
 
-For better performances, Hue uses the Django staticfiles app. If in production mode, if you edit
-some static files, you would need to run this command or `make apps`. No actions are needed in
-development mode.
-```
-./build/env/bin/hue collectstatic
-```
+By building on top of Hue SDK, you get, out of the box:
 
-### Adding Interactive Elements to Your UI
++ Configuration Management
++ Hadoop interoperability
++ Supervision of subprocesses
++ A collaborative UI
++ Basic user administration and authorization
 
-Hue by default loads these JavaScript components:
+This document will orient you with the general structure of Hue
+and will walk you through adding a new application using the SDK.
 
-* Ko js
-* jQuery
-* Bootstrap
 
-These are used by some Hue applications, but not loaded by default:
+### From 30,000 feet
 
-* Knockout js (`desktop/core/static/ext/js/knockout-min.js`)
-* jQuery UI (`desktop/core/static/ext/js/jquery/plugins/jquery-ui-autocomplete-1.8.18.min.js`)
+![From up on high](from30kfeet.png)
 
-These standard components have their own online documentation, which we will
-not repeat here. They let you write interactive behaviors with little or no
-JavaScript.
+Hue, as a "container" web application, sits in between your Hadoop installation
+and the browser.  It hosts all the Hue Apps, including the built-in ones, and
+ones that you may write yourself.
 
+### The Hue Server
 
-## Debugging Tips and Tricks
+![Web Back-end](webbackend.png)
 
-* Set `DESKTOP_DEBUG=1` as an environment variable if you want logs to go to stderr
-  as well as to the respective log files.
-* Use runserver.  If you want to set a CLI breakpoint, just insert
-  `__import__("ipdb").set_trace()`
-  into your code.
-* Django tends to restart its server whenever it notices a file changes.  For
-  certain things (like configuration changes), this is not sufficient.  Restart
-  the server whole-heartedly.
-* If you find yourself writing a lot of JavaScript, you'll want to disable the
-  JavaScript caching that the server does. At startup Hue reads all your
-  dependencies and JS files into memory to make things faster. You can disable
-  this by executing the runserver command with an environment variable
-  set. Hue will be a little slower, but your JS will always represent what's on
-  the disk. Here's what that looks like:
+Hue is a web application built on the Django python web framework.
+Django, running on the WSGI container/web server (typically CherryPy), manages
+the url dispatch, executes application logic code, and puts together the views
+from their templates.  Django uses a database (typically sqlite)
+to manage session data, and Hue applications can use it as well
+for their "models".  (For example, the JobDesigner application stores
+job designs in the database.)
 
-    `$ DESKTOP_DEPENDER_DEBUG=1 build/env/bin/hue runserver`
+In addition to the web server, some Hue applications run
+daemon processes "on the side".  For example, Spark runs a daemon
+("livy_server") that keeps track of the Spark shells of the user. Running
+a separate process for applications is the preferred
+way to manage long-running tasks that you may wish
+to co-exist with web page rendering.  The web "views"
+typically communicate with these side daemons
+by using Thrift (e.g., for Hive query execution) or by exchanging state
+through the database.
 
-* We recommend developing with the Chrome console.
+### Interacting with Hadoop
 
-## Building
+![Interacting with Hadoop](interactingwithhadoop.png)
 
-### Documentation
+Hue provides some APIs for interacting with Hadoop.
+Most noticeably, there are python file-object-like APIs for
+interacting with HDFS.  These APIs work by making REST API or Thrift calls
+the Hadoop daemons. The Hadoop administrator must enable these interfaces from
+Hadoop.
 
-Building with
+### On the Front-End
 
-    make docs
+Hue provides a front-end framework based on
+[Bootstrap](http://twitter.github.com/bootstrap/) and
+[Knockout js](http://knockoutjs.com/).
 
-### Javascript
 
-The javascript files are currently being migrated to webpack bundles, during this process some files will live under src/desktop/static/ and some will live under src/dekstop/js
+### An Architectural View
 
-#### For changes to the files under src/desktop/js the following applies:
+![Architecture](architecture.png)
 
-First make sure all third-party dependencies defined in package.json are installed into node_modules/
+A Hue application may span three tiers: (1) the UI
+and user interaction in the client's browser, (2) the
+core application logic in the Hue web
+server, and (3) external services with which applications
+may interact.
 
-    npm install
+The absolute minimum that you must implement (besides
+boilerplate), is a
+"Django [view](https://docs.djangoproject.com/en/1.11/#the-view-layer/)"
+function that processes the request and the associated template
+to render the response into HTML.
 
-Also run this after making changes to package.json, adding new third-party dependencies etc.
+Many apps will evolve to have a bit of custom JavaScript and
+CSS styles. Apps that need to talk to an external service
+will pull in the code necessary to talk to that service.
 
-To generate the js bundles run:
+### File Layout
 
-    npm run webpack
+The Hue "framework" is in ``desktop/core/`` and contains the Web components.
+``desktop/libs/`` is the API for talking to various Hadoop services.
+The installable apps live in ``apps/``.  Please place third-party dependencies in the app's ext-py/
+directory.
 
-During development the bundles can be autogenerated when it detects changes to the .js files, for this run:
+The typical directory structure for inside an application includes:
+```
+  src/
+    for Python/Django code
+      models.py
+      urls.py
+      views.py
+      forms.py
+      settings.py
 
-    npm run dev
+  conf/
+    for configuration (``.ini``) files to be installed
 
-Before sending a review with changes to the bundles run:
+  static/
+    for static HTML/js resources and help doc
 
-    npm run lint-fix
+  templates/
+    for data to be put through a template engine
 
-and possibly fix any issues it might report.
+  locales/
+    for localizations in multiple languages
+```
 
-### CSS / LESS
+For the URLs within your application, you should make your own ``urls.py``
+which will be automatically rooted at ``/yourappname/`` in the global
+namespace. See ``apps/about/src/about/urls.py`` for an example.
 
-After changing the CSS in a .less file, rebuilding with:
 
+## Pre-requisites
 
-    make css
 
-### SQL Autocomplete
+### Dependencies
 
-Install a patched jison:
+* The OS specific dependencies listed [here](http://cloudera.github.io/hue/latest/admin-manual/manual.html)
+* Python 2.7
+* Django (1.11 included with our distribution)
+* Hadoop (Apache Hadoop 2+)
+* Java (Java 1.8)
+* npm (6.4+)
 
-    git clone https://github.com/JohanAhlen/jison
-    cd jison
-    npm install -g .
+### Recommended Reading / Important Technologies
 
-Then run:
+The following are core technologies used inside of Hue.
 
-    make sql-all-parsers
+* Python.  <a href="http://diveintopython.net/">Dive Into Python</a> is one of
+  several excellent books on python.
+* Django.  Start with [The Django Tutorial](https://docs.djangoproject.com/en/1.11/intro/).
+* [Thrift](http://incubator.apache.org/thrift/) is used for communication
+  between daemons.
+* [Mako](http://www.makotemplates.org/) is the preferred templating language.
 
-### Ace Editor
+## Fast-Guide to Creating a New Hue Application
 
-After modifying files under tools/ace-editor run the following to build ace.js
+Now that we have a high-level overview of what's going on,
+let's go ahead and create a new installation.
 
-    npm install
-    make ace
+### Download, Unpack, Build Distro
 
-### Internationalization
+The Hue SDK is available from [Github](http://github.com/cloudera/hue). Releases
+can be found on the [download page](http://gethue.com/category/release/).
+Releases are missing a few dependencies that could not be included because of
+licencing issues. So if you prefer to have an environment ready from scratch,
+it is preferable to checkout a particular release tag instead.
 
-How to update all the messages and compile them:
+    cd hue
+    ## Build
+    make apps
+    ## Run
+    build/env/bin/hue runserver
+    ## Alternative run
+    build/env/bin/hue supervisor
+    ## Visit http://localhost:8000/ with your web browser.
 
-    make locales
 
-How to update and compile the messages of one app:
+### Run "create_desktop_app" to Set up a New Source Tree
 
-    cd apps/beeswax
-    make compile-locale
+    ./build/env/bin/hue create_desktop_app calculator
+    find calculator -type f
+    calculator/setup.py                                 # distutils setup file
+    calculator/src/calculator/__init__.py               # main src module
+    calculator/src/calculator/forms.py
+    calculator/src/calculator/models.py
+    calculator/src/calculator/settings.py               # app metadata setting
+    calculator/src/calculator/urls.py                   # url mapping
+    calculator/src/calculator/views.py                  # app business logic
+    calculator/src/calculator/templates/index.mako
+    calculator/src/calculator/templates/shared_components.mako
 
-How to create a new locale for an app:
+    # Static resources
+    calculator/src/static/calculator/art/calculator.png # logo
+    calculator/src/static/calculator/css/calculator.css
+    calculator/src/static/calculator/js/calculator.js
 
-    cd $APP_ROOT/src/$APP_NAME/locale
-    $HUE_ROOT/build/env/bin/pybabel init -D django -i en_US.pot -d . -l fr
 
-# API
+<div class="note">
+  Some apps are blacklisted on certain versions of CDH (such as the 'Spark' app) due to
+  certain incompatibilities, which prevent them loading from in Hue.
+  Check the hue.ini 'app_blacklist' parameter for details.
+</div>
 
-## Metadata Catalog
+### Install SDK Application
 
-The [metadata API](https://github.com/cloudera/hue/tree/master/desktop/libs/metadata) is powering [Search and Tagging here](http://gethue.com/improved-sql-exploration-in-hue-4-3/) and the [Query Assistant with Navigator Optimizer Integration](http://gethue.com/hue-4-sql-editor-improvements/).
+As you'll discover if you look at calculator's <tt>setup.py</tt>,
+Hue uses a distutils <tt>entrypoint</tt> to
+register applications.  By installing the calculator
+package into Hue's python virtual environment,
+you'll install a new app.  The "app_reg.py" tool manages
+the applications that are installed. Note that in the following example, the value after the
+"--install" option is the path to the root directory of the application you want to install. In this
+example, it is a relative path to "/Users/philip/src/hue/calculator".
 
-The backends is pluggable by providing alternative [client interfaces](https://github.com/cloudera/hue/tree/master/desktop/libs/metadata/catalog):
+        ./build/env/bin/python tools/app_reg/app_reg.py --install calculator --relative-paths
+        === Installing app at calculator
+        Updating registry with calculator (version 0.1)
+        --- Making egg-info for calculator
 
-* navigator (default)
-* dummy
 
-### Searching for entities
+<div class="note">
+  If you'd like to customize the build process, you can modify (or even complete
+  rewrite) your own `Makefile`, as long as it supports the set of required
+  targets. Please see `Makefile.sdk` for the required targets and their
+  semantics.
+</div>
 
-    $.post("/metadata/api/catalog/search_entities_interactive/", {
-        query_s: ko.mapping.toJSON("*sample"),
-        sources: ko.mapping.toJSON(["sql", "hdfs", "s3"]),
-        field_facets: ko.mapping.toJSON([]),
-        limit: 10
-    }, function(data) {
-        console.log(ko.mapping.toJSON(data));
-    });
+Congrats, you've added a new app!
 
+<div class="note">
+  What was that all about?
+  <a href="http://pypi.python.org/pypi/virtualenv">virtualenv</a>
+  is a way to isolate python environments in your system, and isolate
+  incompatible versions of dependencies.  Hue uses the system python, and
+  that's about all.  It installs its own versions of dependencies.
 
-### Searching for entities with the dummy backend
+  <a href="http://peak.telecommunity.com/DevCenter/PkgResources#entry-points">Entry Points</a>
+  are a way for packages to optionally hook up with other packages.
+</div>
 
-    $.post("/metadata/api/catalog/search_entities_interactive/", {
-        query_s: ko.mapping.toJSON("*sample"),
-        interface: "dummy"
-    }, function(data) {
-        console.log(ko.mapping.toJSON(data));
-    });
+You can now browse the new application.
 
-### Finding an entity in order to get its id
+    # If you haven't killed the old process, do so now.
+    build/env/bin/hue runserver
 
-    $.get("/metadata/api/navigator/find_entity", {
-        type: "table",
-        database: "default",
-        name: "sample_07",
-        interface: "dummy"
-    }, function(data) {
-        console.log(ko.mapping.toJSON(data));
-    });
+And then visit <a href="http://localhost:8000">http://localhost:8000/</a> to check it out!
+You should see the app in the left menu.
 
-### Adding/updating a comment with the dummy backend
 
-    $.post("/metadata/api/catalog/update_properties/", {
-        id: "22",
-        properties: ko.mapping.toJSON({"description":"Adding a description"}),
-        interface: "dummy"
-    }, function(data) {
-        console.log(ko.mapping.toJSON(data));
-    });
+### Customizing Views and Templates
 
-### Adding a tag with the dummy backend
+Now that your app has been installed, you'll want to customize it.
+As you may have guessed, we're going to build a small calculator
+application.  Edit `calculator/src/calculator/templates/index.mako`
+to include a simple form and a Knockout viewmodel:
 
-    $.post("/metadata/api/catalog/add_tags/", {
-      id: "22",
-      tags: ko.mapping.toJSON(["usage"]),
-      interface: "dummy"
-    }, function(data) {
-        console.log(ko.mapping.toJSON(data));
-    });
 
-### Deleting a key/value property
+    <%!from desktop.views import commonheader, commonfooter %>
+    <%namespace name="shared" file="shared_components.mako" />
 
-    $.post("/metadata/api/catalog/delete_metadata_properties/", {
-       "id": "32",
-       "keys": ko.mapping.toJSON(["project", "steward"])
-    }, function(data) {
-       console.log(ko.mapping.toJSON(data));
-    });
+    %if not is_embeddable:
+    ${commonheader("Calculator", "calculator", user, "100px") | n,unicode}
+    %endif
 
-### Deleting a key/value property
+    ## Main body
+    <div class="container-fluid calculator-components">
+      <div class="row">
+        <div class="span6 offset3 margin-top-30 text-center">
+          <form class="form-inline">
+            <input type="text" class="input-mini margin-right-10" placeholder="A" data-bind="value: a">
+            <!-- ko foreach: operations -->
+            <label class="radio margin-left-5">
+              <input type="radio" name="op" data-bind="checkedValue: $data, checked: $parent.chosenOperation" /><span data-bind="text: $data"></span>
+            </label>
+            <!-- /ko -->
+            <input type="text" class="input-mini margin-left-10" placeholder="B" data-bind="value: b">
+            <button class="btn" data-bind="click: calculate">Calculate</button>
+          </form>
 
-    $.post("/metadata/api/catalog/delete_metadata_properties/", {
-      "id": "32",
-      "keys": ko.mapping.toJSON(["project", "steward"])
-    }, function(data) {
-      console.log(ko.mapping.toJSON(data));
-    });
+          <h2 data-bind="visible: result() !== null">The result is <strong data-bind="text: result"></strong></h2>
+        </div>
+      </div>
+    </div>
 
+    <script>
+      (function() {
+        var CalculatorViewModel = function () {
+          var self = this;
 
-### Getting the model mapping of custom metadata
+          self.operations = ko.observableArray(['+', '-', '*', '/']);
 
-    $.get("/metadata/api/catalog/models/properties/mappings/", function(data) {
-      console.log(ko.mapping.toJSON(data));
-    });
+          self.a = ko.observable();
+          self.b = ko.observable();
+          self.chosenOperation = ko.observable('+');
+          self.result = ko.observable(null);
 
+          self.calculate = function () {
+            var a = parseFloat(self.a());
+            var b = parseFloat(self.b());
+            var result = null;
+            switch (self.chosenOperation()) {
+              case '+':
+                result = a + b;
+                break;
+              case '-':
+                result = a - b;
+                break;
+              case '*':
+                result = a * b;
+                break;
+              case '/':
+                result = a / b;
+            }
+            self.result(result);
+          }
+        };
+        $(document).ready(function () {
+          ko.applyBindings(new CalculatorViewModel(), $('.calculator-components')[0]);
+        });
+      })();
+    </script>
 
-### Getting a namespace
+    %if not is_embeddable:
+    ${commonfooter(messages) | n,unicode}
+    %endif
 
-    $.post("/metadata/api/catalog/namespace/", {
-      namespace: 'huecatalog'
-    }, function(data) {
-      console.log(ko.mapping.toJSON(data));
-    });
+The template language here is <a href="http://www.makotemplates.org/docs/">Mako</a>,
+which is flexible and powerful.  If you use the "`.html`" extension, Hue
+will render your page using
+<a href="https://docs.djangoproject.com/en/1.11/#the-template-layer">Django templates</a>
+instead.
 
-### Creating a namespace
+Note that we use Knockout.js to do the heavy lifting of this app.
 
-    $.post("/metadata/api/catalog/namespace/create/", {
-      "namespace": "huecatalog",
-      "description": "my desc"
-    }, function(data) {
-      console.log(ko.mapping.toJSON(data));
-    });
+Let's edit `calculator/src/calculator/views.py` to simply render the page:
 
+    #!/usr/bin/env python
 
-### Creating a namespace property
+    from desktop.lib.django_util import render
 
-    $.post("/metadata/api/catalog/namespace/property/create/", {
-      "namespace": "huecatalog",
-      "properties": ko.mapping.toJSON({
-        "name" : "relatedEntities2",
-        "displayName" : "Related objects",
-        "description" : "My desc",
-        "multiValued" : true,
-        "maxLength" : 50,
-        "pattern" : ".*",
-        "enumValues" : null,
-        "type" : "TEXT"
+    def index(request):
+      return render('index.mako', request, {
+        'is_embeddable': request.GET.get('is_embeddable', False),
       })
-    }, function(data) {
-      console.log(ko.mapping.toJSON(data));
-    });
 
 
-### Map a namespace property to a class entity
+You can now go and try the calculator.
 
-    $.post("/metadata/api/catalog/namespace/property/map/", {
-      "class": "hv_view",
-      "properties": ko.mapping.toJSON([{
-          namespace: "huecatalog",
-          name: "relatedQueries"
-      }])
-    }, function(data) {
-      console.log(ko.mapping.toJSON(data));
-    });
 
 # Testing
 

+ 7 - 21
package-lock.json

@@ -4628,14 +4628,12 @@
         "balanced-match": {
           "version": "1.0.0",
           "bundled": true,
-          "dev": true,
-          "optional": true
+          "dev": true
         },
         "brace-expansion": {
           "version": "1.1.11",
           "bundled": true,
           "dev": true,
-          "optional": true,
           "requires": {
             "balanced-match": "^1.0.0",
             "concat-map": "0.0.1"
@@ -4650,20 +4648,17 @@
         "code-point-at": {
           "version": "1.1.0",
           "bundled": true,
-          "dev": true,
-          "optional": true
+          "dev": true
         },
         "concat-map": {
           "version": "0.0.1",
           "bundled": true,
-          "dev": true,
-          "optional": true
+          "dev": true
         },
         "console-control-strings": {
           "version": "1.1.0",
           "bundled": true,
-          "dev": true,
-          "optional": true
+          "dev": true
         },
         "core-util-is": {
           "version": "1.0.2",
@@ -4780,8 +4775,7 @@
         "inherits": {
           "version": "2.0.3",
           "bundled": true,
-          "dev": true,
-          "optional": true
+          "dev": true
         },
         "ini": {
           "version": "1.3.5",
@@ -4793,7 +4787,6 @@
           "version": "1.0.0",
           "bundled": true,
           "dev": true,
-          "optional": true,
           "requires": {
             "number-is-nan": "^1.0.0"
           }
@@ -4808,7 +4801,6 @@
           "version": "3.0.4",
           "bundled": true,
           "dev": true,
-          "optional": true,
           "requires": {
             "brace-expansion": "^1.1.7"
           }
@@ -4816,14 +4808,12 @@
         "minimist": {
           "version": "0.0.8",
           "bundled": true,
-          "dev": true,
-          "optional": true
+          "dev": true
         },
         "minipass": {
           "version": "2.3.5",
           "bundled": true,
           "dev": true,
-          "optional": true,
           "requires": {
             "safe-buffer": "^5.1.2",
             "yallist": "^3.0.0"
@@ -4842,7 +4832,6 @@
           "version": "0.5.1",
           "bundled": true,
           "dev": true,
-          "optional": true,
           "requires": {
             "minimist": "0.0.8"
           }
@@ -4923,8 +4912,7 @@
         "number-is-nan": {
           "version": "1.0.1",
           "bundled": true,
-          "dev": true,
-          "optional": true
+          "dev": true
         },
         "object-assign": {
           "version": "4.1.1",
@@ -4936,7 +4924,6 @@
           "version": "1.4.0",
           "bundled": true,
           "dev": true,
-          "optional": true,
           "requires": {
             "wrappy": "1"
           }
@@ -5058,7 +5045,6 @@
           "version": "1.0.2",
           "bundled": true,
           "dev": true,
-          "optional": true,
           "requires": {
             "code-point-at": "^1.0.0",
             "is-fullwidth-code-point": "^1.0.0",