Browse Source

HUE-8756 [docs] Refactoring of the developer section

Romain Rigaux 6 years ago
parent
commit
792b687

+ 5 - 5
docs/docs-site/content/developer/_index.md

@@ -9,7 +9,7 @@ pre = "<b>3. </b>"
 Hue service integration is generic and lets you integrate with other analytics systems. Here is a list of the main APIs:
 
 * Connect or create SQL connectors to any database (Impala, MySQL, Presto...)
-* List any files from any filesytem (HDFS, S3, ADLS, GSF...)
+* List any files from any filesytem (HDFS, S3, ADLS...)
 * List any job or query list (YARN, SQL queries, Kubernetes...)
 * Integrate with a Data Catalog (Cloudera Navigator, Apache Atlas...)
 
@@ -18,10 +18,10 @@ In addition, whole new apps can also be created in order to provide end to end s
 Upcoming & Contribution ideas
 
 * How to add new SQL grammar and highlighting
-* Better Query Builder
-* Better Dashboard & Reporting
-* Framework to add new vizualizations
-* Framework to embed vizualizations
+* Improved Query Builder
+* Improved Dashboard & Reporting
+* Framework to add new/embed vizualizations
 * How to integrate smart query troubleshooting/recommendation systems (Cloudera Navigator Optimizer...)
 * Generic workflow system (Apache Oozie, Apache Airflow, Celery Tasks...)
+* ADLS gen2, Google FS integrations
 * Check the [Roadmap](https://github.com/cloudera/hue/blob/master/docs/ROADMAP.md)

+ 116 - 74
docs/docs-site/content/developer/api/_index.md

@@ -5,101 +5,62 @@ draft: false
 weight: 3
 ---
 
-Hue can be accessed directly via a Django Python Shell or by its REST API (recommended).
+Hue can be accessed directly via a Django Python Shell or by its REST API.
 
-## Python
-
-* [Hue API: Execute some builtin or shell commands](http://gethue.com/hue-api-execute-some-builtin-commands/).
-* [How to manage the Hue database with the shell](http://gethue.com/how-to-manage-the-hue-database-with-the-shell/).
-
-### How to count documents of a user
-
-On the command line:
-
-    ./build/env/bin/hue shell
-
-If using Cloudera Manager, as a `root` user launch the shell.
-
-Export the configuration directory:
-
-    export HUE_CONF_DIR="/var/run/cloudera-scm-agent/process/`ls -alrt /var/run/cloudera-scm-agent/process | grep HUE_SERVER | tail -1 | awk '{print $9}'`"
-    echo $HUE_CONF_DIR
-    > /var/run/cloudera-scm-agent/process/2061-hue-HUE_SERVER
-
-Get the process id:
 
-    lsof -i :8888|grep -m1 hue|awk '{ print $2 }'
-    > 14850
-
-In order to export all Hue's env variables:
-
-    for line in `strings /proc/$(lsof -i :8888|grep -m1 hue|awk '{ print $2 }')/environ|egrep -v "^HOME=|^TERM=|^PWD="`;do export $line;done
-
-And finally launch the shell by:
-
-    HUE_IGNORE_PASSWORD_SCRIPT_ERRORS=1 /opt/cloudera/parcels/CDH/lib/hue/build/env/bin/hue shell
-    > ALERT: This appears to be a CM Managed environment
-    > ALERT: HUE_CONF_DIR must be set when running hue commands in CM Managed environment
-    > ALERT: Please run 'hue <command> --cm-managed'
+## REST
 
-Then use the Python code to access a certain user information:
+REST APIs are not all public yet but this is work in progress in [HUE-1450](https://issues.cloudera.org/browse/HUE-1450).
 
-    Python 2.7.6 (default, Oct 26 2016, 20:30:19)
-    Type "copyright", "credits" or "license" for more information.
+Hue is Ajax based and has a REST API used by the browser to communicate (e.g. submit a query or workflow,
+list some S3 files, export a document...). Currently this API is private and subject to change but
+can be easily reused. You would need to GET */accounts/login* to get the CSRF token
+and POST it back along *username* and *password* and reuse the *sessionid* cookie in next
+communication calls.
 
-    IPython 5.2.0 -- An enhanced Interactive Python.
-    ?         -> Introduction and overview of IPython's features.
-    %quickref -> Quick reference.
-    help      -> Python's own help system.
-    object?   -> Details about 'object', use 'object??' for extra details.
+### Quickstart
 
-    from django.contrib.auth.models import User
-    from desktop.models import Document2
+Hue is based on the Django Web Framework. Django comes with user authentication system. Django uses sessions and middleware to hook the authentication system into request object. Hue uses stock auth form which uses *username* and *password* and *csrftoken* form variables to authenticate.
 
-    user = User.objects.get(username='demo')
-    Document2.objects.documents(user=user).count()
+In this code snippet, we will use well-known python *requests* library. We will first acquire *csrftoken* by GET *login_url* and then create a dictionary of form data which contains *username*, *password* and *csrftoken* and the *next_url* and another dictionary for header which contains the *Referer* url and an empty dictionary for the cookies. After the POST request to *login_url* we will check the reponse code, which should be *r.status_code == 200*.
 
-    In [8]: Document2.objects.documents(user=user).count()
-    Out[8]: 1167
+Once the request is successful then capture headers and cookies for subsequent requests. Subsequent *request.session* calls can be made by providing *cookies=session.cookies* and *headers=session.headers*.
 
-    In [10]: Document2.objects.documents(user=user, perms='own').count()
-    Out[10]: 1166
+    import requests
 
-    In [11]: Document2.objects.documents(user=user, perms='own', include_history=True).count()
-    Out[11]: 7125
+    def login_djangosite():
+    next_url = "/"
+    login_url = "http://localhost:8888/accounts/login?next=/"
 
-    In [12]: Document2.objects.documents(user=user, perms='own', include_history=True, include_trashed=True).count()
-    Out[12]: 7638
+    session = requests.Session()
+    r = session.get(login_url)
+    form_data = dict(username="[your hue username]",password="[your hue password]",
+                      csrfmiddlewaretoken=session.cookies['csrftoken'],next=next_url)
+    r = session.post(login_url, data=form_data, cookies=dict(), headers=dict(Referer=login_url))
 
-    In [13]: Document2.objects.documents(user=user, perms='own', include_history=True, include_trashed=True, include_managed=True).count()
-    Out[13]: 31408
+    # check if request executed successfully?
+    print r.status_code
 
-    Out[14]:
-    (85667L,
-    {u'desktop.Document': 18524L,
-      u'desktop.Document2': 31409L,
-      u'desktop.Document2Permission': 556L,
-      u'desktop.Document2Permission_groups': 277L,
-      u'desktop.Document2Permission_users': 0L,
-      u'desktop.Document2_dependencies': 15087L,
-      u'desktop.DocumentPermission': 1290L,
-      u'desktop.DocumentPermission_groups': 0L,
-      u'desktop.DocumentPermission_users': 0L,
-      u'desktop.Document_tags': 18524L})
+    cookies = session.cookies
+    headers = session.headers
 
+    r=session.get('http://localhost:8888/metastore/databases/default/metadata',
+    cookies=session.cookies, headers=session.headers)
+    print r.status_code
 
-## REST
+    # check metadata output
+    print r.text
 
-REST APIs are not all public yet but this is work in progress in [HUE-1450](https://issues.cloudera.org/browse/HUE-1450).
 
-### Metadata Catalog
+### Data Catalog
 
 The [metadata API](https://github.com/cloudera/hue/tree/master/desktop/libs/metadata) is powering [Search and Tagging here](http://gethue.com/improved-sql-exploration-in-hue-4-3/) and the [Query Assistant with Navigator Optimizer Integration](http://gethue.com/hue-4-sql-editor-improvements/).
 
 The backends is pluggable by providing alternative [client interfaces](https://github.com/cloudera/hue/tree/master/desktop/libs/metadata/catalog):
 
-* navigator (default)
-* dummy
+* Cloudera Navigator (default)
+* Apache Atlas ([HUE-8749](https://issues.cloudera.org/browse/HUE-8749))
+* Dummy (skeleton for integrating new catalogs)
 
 #### Searching for entities
 
@@ -226,4 +187,85 @@ The backends is pluggable by providing alternative [client interfaces](https://g
       }])
     }, function(data) {
       console.log(ko.mapping.toJSON(data));
-    });
+    });
+
+
+## Python
+
+* [Hue API: Execute some builtin or shell commands](http://gethue.com/hue-api-execute-some-builtin-commands/).
+* [How to manage the Hue database with the shell](http://gethue.com/how-to-manage-the-hue-database-with-the-shell/).
+
+### How to count documents of a user
+
+On the command line:
+
+    ./build/env/bin/hue shell
+
+If using Cloudera Manager, as a *root* user launch the shell.
+
+Export the configuration directory:
+
+    export HUE_CONF_DIR="/var/run/cloudera-scm-agent/process/`ls -alrt /var/run/cloudera-scm-agent/process | grep HUE_SERVER | tail -1 | awk '{print $9}'`"
+    echo $HUE_CONF_DIR
+    > /var/run/cloudera-scm-agent/process/2061-hue-HUE_SERVER
+
+Get the process id:
+
+    lsof -i :8888|grep -m1 hue|awk '{ print $2 }'
+    > 14850
+
+In order to export all Hue's env variables:
+
+    for line in `strings /proc/$(lsof -i :8888|grep -m1 hue|awk '{ print $2 }')/environ|egrep -v "^HOME=|^TERM=|^PWD="`;do export $line;done
+
+And finally launch the shell by:
+
+    HUE_IGNORE_PASSWORD_SCRIPT_ERRORS=1 /opt/cloudera/parcels/CDH/lib/hue/build/env/bin/hue shell
+    > ALERT: This appears to be a CM Managed environment
+    > ALERT: HUE_CONF_DIR must be set when running hue commands in CM Managed environment
+    > ALERT: Please run 'hue <command> --cm-managed'
+
+Then use the Python code to access a certain user information:
+
+    Python 2.7.6 (default, Oct 26 2016, 20:30:19)
+    Type "copyright", "credits" or "license" for more information.
+
+    IPython 5.2.0 -- An enhanced Interactive Python.
+    ?         -> Introduction and overview of IPython's features.
+    %quickref -> Quick reference.
+    help      -> Python's own help system.
+    object?   -> Details about 'object', use 'object??' for extra details.
+
+    from django.contrib.auth.models import User
+    from desktop.models import Document2
+
+    user = User.objects.get(username='demo')
+    Document2.objects.documents(user=user).count()
+
+    In [8]: Document2.objects.documents(user=user).count()
+    Out[8]: 1167
+
+    In [10]: Document2.objects.documents(user=user, perms='own').count()
+    Out[10]: 1166
+
+    In [11]: Document2.objects.documents(user=user, perms='own', include_history=True).count()
+    Out[11]: 7125
+
+    In [12]: Document2.objects.documents(user=user, perms='own', include_history=True, include_trashed=True).count()
+    Out[12]: 7638
+
+    In [13]: Document2.objects.documents(user=user, perms='own', include_history=True, include_trashed=True, include_managed=True).count()
+    Out[13]: 31408
+
+    Out[14]:
+    (85667L,
+    {u'desktop.Document': 18524L,
+      u'desktop.Document2': 31409L,
+      u'desktop.Document2Permission': 556L,
+      u'desktop.Document2Permission_groups': 277L,
+      u'desktop.Document2Permission_users': 0L,
+      u'desktop.Document2_dependencies': 15087L,
+      u'desktop.DocumentPermission': 1290L,
+      u'desktop.DocumentPermission_groups': 0L,
+      u'desktop.DocumentPermission_users': 0L,
+      u'desktop.Document_tags': 18524L})

+ 2 - 254
docs/docs-site/content/developer/application/_index.md

@@ -7,6 +7,8 @@ weight: 5
 
 Building a brand new application is more work but is ideal for creating a custom solution.
 
+**Note** It is now more recommended to integrate external services (e.g. integrate a new SQL Datatase with the Editor, add a new visualization...) to the core Hue APIs instead of building brand new application. This page gives good content in both cases. Feel free to contact the community for advice.
+
 ## Introduction and Overview
 
 Hue leverages the browser to provide users with an environment for exploring
@@ -26,125 +28,6 @@ By building on top of Hue SDK, you get, out of the box:
 This document will orient you with the general structure of Hue
 and will walk you through adding a new application using the SDK.
 
-
-### From 30,000 feet
-
-![From up on high]({{% param baseURL %}}images/from30kfeet.png)
-
-Hue, as a "container" web application, sits in between your Hadoop installation
-and the browser.  It hosts all the Hue Apps, including the built-in ones, and
-ones that you may write yourself.
-
-### The Hue Server
-
-![Web Back-end]({{% param baseURL %}}images/webbackend.png)
-
-Hue is a web application built on the Django python web framework.
-Django, running on the WSGI container/web server (typically CherryPy), manages
-the url dispatch, executes application logic code, and puts together the views
-from their templates.  Django uses a database (typically sqlite)
-to manage session data, and Hue applications can use it as well
-for their "models".  (For example, the saved Editor stores
-saved queries in the database.)
-
-In addition to the web server, some Hue applications run
-daemon processes "on the side". Some examples are the `Celery Task Server`, `Celery Beat`.
-
-### Interacting with Hadoop
-
-![Interacting with Hadoop]({{% param baseURL %}}images/interactingwithhadoop.png)
-
-Hue provides some APIs for interacting with Hadoop.
-Most noticeably, there are python file-object-like APIs for
-interacting with HDFS.  These APIs work by making REST API or Thrift calls
-the Hadoop daemons. The Hadoop administrator must enable these interfaces from
-Hadoop.
-
-### On the Front-End
-
-Hue provides a front-end framework based on
-[Bootstrap](http://twitter.github.com/bootstrap/) and
-[Knockout js](http://knockoutjs.com/).
-
-
-### An Architectural View
-
-![Architecture]({{% param baseURL %}}images/architecture.png)
-
-A Hue application may span three tiers: (1) the UI
-and user interaction in the client's browser, (2) the
-core application logic in the Hue web
-server, and (3) external services with which applications
-may interact.
-
-The absolute minimum that you must implement (besides
-boilerplate), is a
-"Django [view](https://docs.djangoproject.com/en/1.11/#the-view-layer/)"
-function that processes the request and the associated template
-to render the response into HTML.
-
-Many apps will evolve to have a bit of custom JavaScript and
-CSS styles. Apps that need to talk to an external service
-will pull in the code necessary to talk to that service.
-
-### File Layout
-
-The Hue "framework" is in ``desktop/core/`` and contains the Web components.
-``desktop/libs/`` is the API for talking to various Hadoop services.
-The installable apps live in ``apps/``.  Please place third-party dependencies in the app's ext-py/
-directory.
-
-The typical directory structure for inside an application includes:
-```
-  src/
-    for Python/Django code
-      models.py
-      urls.py
-      views.py
-      forms.py
-      settings.py
-
-  conf/
-    for configuration (``.ini``) files to be installed
-
-  static/
-    for static HTML/js resources and help doc
-
-  templates/
-    for data to be put through a template engine
-
-  locales/
-    for localizations in multiple languages
-```
-
-For the URLs within your application, you should make your own ``urls.py``
-which will be automatically rooted at ``/yourappname/`` in the global
-namespace. See ``apps/about/src/about/urls.py`` for an example.
-
-
-## Pre-requisites
-
-
-### Dependencies
-
-* The OS specific dependencies listed [here](http://cloudera.github.io/hue/latest/admin-manual/manual.html)
-* Python 2.7
-* Django (1.11 included with our distribution)
-* Hadoop (Apache Hadoop 2+)
-* Java (Java 1.8)
-* npm (6.4+)
-
-### Recommended Reading / Important Technologies
-
-The following are core technologies used inside of Hue.
-
-* Python.  <a href="http://diveintopython.net/">Dive Into Python</a> is one of
-  several excellent books on python.
-* Django.  Start with [The Django Tutorial](https://docs.djangoproject.com/en/1.11/intro/).
-* [Thrift](http://incubator.apache.org/thrift/) is used for communication
-  between daemons.
-* [Mako](http://www.makotemplates.org/) is the preferred templating language.
-
 ## Fast-Guide to Creating a New Hue Application
 
 Now that we have a high-level overview of what's going on,
@@ -338,138 +221,3 @@ Let's edit `calculator/src/calculator/views.py` to simply render the page:
 
 
 You can now go and try the calculator.
-
-
-# Testing
-
-## The short story
-
-Install the mini cluster (only once):
-
-    ./tools/jenkins/jenkins.sh slow
-
-Run all the tests:
-
-    build/env/bin/hue test all
-
-Or just some parts of the tests, e.g.:
-
-    build/env/bin/hue test specific impala
-    build/env/bin/hue test specific impala.tests:TestMockedImpala
-    build/env/bin/hue test specific impala.tests:TestMockedImpala.test_basic_flow
-
-Jasmine tests:
-
-    npm run test
-
-
-## Longer story
-
-The ``test`` management command prepares the arguments (test app names)
-and passes them to nose (django_nose.nose_runner). Nose will then magically
-find all the tests to run.
-
-Tests themselves should be named *_test.py.  These will be found
-as long as they're in packages covered by django.  You can use the
-unittest frameworks, or you can just name your method with
-the word "test" at a word boundary, and nose will find it.
-See apps/hello/src/hello/hello_test.py for an example.
-
-
-### Helpful command-line tricks
-
-To run tests that do not depend on Hadoop, use:
-
-    build/env/bin/hue test fast
-
-To run all tests, use:
-
-    build/env/bin/hue test all
-
-To run only tests of a particular app, use:
-
-    build/env/bin/hue test specific <app>
-
-E.g.
-  build/env/bin/hue test specific filebrowser
-
-To run a specific test, use:
-
-    build/env/bin/hue test specific <test_func>
-
-E.g.
-  build/env/bin/hue test specific useradmin.tests:test_user_admin
-
-Start up pdb on test failures:
-
-    build/env/bin/hue test <args> --pdb --pdb-failure -s
-
-Point to an Impalad and trigger the Impala tests:
-
-    build/env/bin/hue test impala impalad-01.gethue.com
-
-
-### Create and run the Jasmine tests
-
-Add them in a "spec" subfolder relative to the file under test and the filename of the test has to end with "Spec.js".
-
-    someFile.js              <- File under test
-    ├── spec/
-    │   ├── someFileSpec.js  <- File containing tests
-
-Run all the tests once with:
-
-    npm run test
-
-Optionally to use Karma and headless chrome for the tests you can run
-
-    npm run test-karma
-
-See ```desktop/core/src/desktop/js/spec/karma.config.js``` for various options
-
-
-### Special environment variables
-
-    DESKTOP_LOGLEVEL=<level>
-      level can be DEBUG, INFO, WARN, ERROR, or CRITICAL
-
-      When specified, the console logger is set to the given log level. A console
-      logger is created if one is not defined.
-
-    DESKTOP_DEBUG
-      A shorthand for DESKTOP_LOG_LEVEL=DEBUG. Also turns on output HTML
-      validation.
-
-    DESKTOP_PROFILE
-      Turn on Python profiling. The profile data is saved in a file. See the
-      console output for the location of the file.
-
-    DESKTOP_LOG_DIR=$dir
-      Specify the HUE log directory. Defaults to ./log.
-
-    DESKTOP_DB_CONFIG=$db engine:db name:test db name:username:password:host:port
-      Specify alternate DB connection parameters for HUE to use. Useful for
-      testing your changes against, for example, MySQL instead of sqlite. String
-      is a colon-delimited list.
-
-    TEST_IMPALAD_HOST=impalad-01.gethue.com
-      Point to an Impalad and trigger the Impala tests.
-
-
-### Writing tests that depend on Hadoop
-
-Use pseudo_hdfs4.py!  You should tag such tests with "requires_hadoop", as follows:
-
-    from nose.plugins.attrib import attr
-
-    @attr('requires_hadoop')
-    def your_test():
-      ...
-
-
-### Jenkins Configuration
-
-Because building Hadoop (for the tests that require it) is slow, we've
-separated the Jenkins builds into "fast" and "slow".  Both are run
-via scripts/jenkins.sh, which should be kept updated with the latest
-and greatest in build technologies.

+ 6 - 3
docs/docs-site/content/developer/browsers/_index.md

@@ -7,11 +7,14 @@ weight: 3
 
 ## Jobs
 
-Here is an example on how the Job Browser can list:
+Here are examples on how the Job Browser can list:
 
-* [Livy jobs and sessions](https://issues.cloudera.org/browse/HUE-6908)
 * [Impala queries](https://issues.cloudera.org/browse/HUE-7420)
+* [Livy jobs and sessions](https://issues.cloudera.org/browse/HUE-6908)
 
+By default, hooks to perform bulk operation like kill/pause/submit and retrieving details/logs are provided by the API.
 
 ## Files
-Here is an example on how the File Browser can list HDFS, S3 files and now [ADLS](https://issues.cloudera.org/browse/HUE-7248).
+Here is an example on how the File Browser can list HDFS, S3 files and [ADLS](https://issues.cloudera.org/browse/HUE-7248).
+
+**Note** Ceph can be used via the S3 browser.

+ 277 - 167
docs/docs-site/content/developer/development/_index.md

@@ -5,9 +5,19 @@ draft: false
 weight: 1
 ---
 
-
 This section goes into greater detail on how to build and reuse the components of Hue.
 
+## Dependencies
+
+* The OS specific dependencies are listed [here](/administrator/installation/dependencies/)
+* Python 2.7+ (Python 3 support tracked in [HUE-8737](https://issues.cloudera.org/browse/HUE-8737))
+* Django (1.11 already included in the distribution)
+* Java (Java 1.8) (should go away after [HUE-8740](https://issues.cloudera.org/browse/HUE-8740))
+* npm (6.4+)
+* [Mako](http://www.makotemplates.org/) is the templating language
+* [Bootstrap](http://twitter.github.com/bootstrap/)
+* [Knockout js](http://knockoutjs.com/)
+
 ## Building
 
 ### Documentation
@@ -115,26 +125,93 @@ How to create a new locale for an app:
     cd $APP_ROOT/src/$APP_NAME/locale
     $HUE_ROOT/build/env/bin/pybabel init -D django -i en_US.pot -d . -l fr
 
+## Webapp API
 
-## User Management
+### From 30,000 feet
 
-Except for static content, `request.user` is always populated.  It is a
-standard Django `models.User` object.  If you were to set a breakpoint at the
-`index()` function in our calculator app, you will find:
+![From up on high]({{% param baseURL %}}images/from30kfeet.png)
 
-    >>> request.user
-    <User: test>
+Hue, as a "container" web application, sits in between your Hadoop installation
+and the browser.  It hosts all the Hue Apps, including the built-in ones, and
+ones that you may write yourself.
 
-<div class="note">
-  "Under the covers:" Django uses a notion called
-  <a href="https://docs.djangoproject.com/en/1.2/topics/http/middleware/">middleware</a>
-  that's called in between the request coming in and the view being executed.
-  That's how <code>request.user</code> gets populated.  There's also a
-  middleware for Hue that makes sure that no pages are displayed unless the
-  user is authenticated.
-</div>
+### The Hue Server
+
+![Web Back-end]({{% param baseURL %}}images/webbackend.png)
+
+Hue is a web application built on the Django python web framework.
+Django, running on the WSGI container/web server (typically CherryPy), manages
+the url dispatch, executes application logic code, and puts together the views
+from their templates.  Django uses a database (typically sqlite)
+to manage session data, and Hue applications can use it as well
+for their "models".  (For example, the saved Editor stores
+saved queries in the database.)
+
+In addition to the web server, some Hue applications run
+daemon processes "on the side". Some examples are the `Celery Task Server`, `Celery Beat`.
 
-## Configuration
+### Interacting with external services
+
+![Interacting with Hadoop]({{% param baseURL %}}images/interactingwithhadoop.png)
+
+Hue provides some APIs for interacting with Hadoop.
+Most noticeably, there are python file-object-like APIs for
+interacting with HDFS.  These APIs work by making REST API or Thrift calls
+the Hadoop daemons.
+
+### An Architectural View
+
+![Architecture]({{% param baseURL %}}images/architecture.png)
+
+A Hue application may span three tiers: (1) the UI
+and user interaction in the client's browser, (2) the
+core application logic in the Hue web
+server, and (3) external services with which applications
+may interact.
+
+The absolute minimum that you must implement (besides
+boilerplate), is a
+"Django [view](https://docs.djangoproject.com/en/1.11/#the-view-layer/)"
+function that processes the request and the associated template
+to render the response into HTML.
+
+Many apps will evolve to have a bit of custom JavaScript and
+CSS styles. Apps that need to talk to an external service
+will pull in the code necessary to talk to that service.
+
+### File Layout
+
+The Hue "framework" is in ``desktop/core/`` and contains the Web components.
+``desktop/libs/`` is the API for talking to various Hadoop services.
+The installable apps live in ``apps/``.  Please place third-party dependencies in the app's ext-py/
+directory.
+
+The typical directory structure for inside an application includes:
+```
+  src/
+    for Python/Django code
+      models.py
+      urls.py
+      views.py
+      forms.py
+      settings.py
+
+  conf/
+    for configuration (``.ini``) files to be installed
+
+  static/
+    for static HTML/js resources and help doc
+
+  templates/
+    for data to be put through a template engine
+
+  locales/
+    for localizations in multiple languages
+```
+
+For the URLs within your application, you should make your own ``urls.py``
+which will be automatically rooted at ``/yourappname/`` in the global
+namespace. See ``apps/about/src/about/urls.py`` for an example.
 
 ### Configuration File
 
@@ -260,7 +337,37 @@ processes serving the backend.  This means that your Django application
 code should avoid depending on shared process state.  Instead, place
 the stored state in a database or run a separate server.
 
-## Walk-through of a Django View
+### Saving documents
+
+Each app used to have its own model to store its data (e.g. a saving a SQL query, query history...). All the models have been unified into a single Document2 model in the desktop app:
+
+``desktop/core/src/desktop/models.py``.
+
+The `Document2` model provides automatically creation, sharing and saving. It persists the document data into a json field, which limits the need ot database migrations and simplifies the interaction with the frontend.
+
+`Document2` is based on [Django Models](https://docs.djangoproject.com/en/1.11/#the-model-layer)
+are Django's Object-Relational Mapping framework.
+
+
+### Managing users
+
+Except for static content, `request.user` is always populated.  It is a
+standard Django `models.User` object.  If you were to set a breakpoint at the
+`index()` function in our calculator app, you will find:
+
+    >>> request.user
+    <User: test>
+
+<div class="note">
+  "Under the covers:" Django uses a notion called
+  <a href="https://docs.djangoproject.com/en/1.2/topics/http/middleware/">middleware</a>
+  that's called in between the request coming in and the view being executed.
+  That's how <code>request.user</code> gets populated.  There's also a
+  middleware for Hue that makes sure that no pages are displayed unless the
+  user is authenticated.
+</div>
+
+### Walk-through of a Django View
 
 ![Django Flow](django_request.png)
 
@@ -273,7 +380,7 @@ These view functions typically use their arguments (for example, the captured pa
 their request object (which has, for example, the POST and GET parameters) to
 prepare dynamic content to be rendered using a template.
 
-## Templates: Django and Mako
+### Templates: Django and Mako
 
 In Hue, the typical pattern for rendering data through a template
 is:
@@ -288,67 +395,11 @@ extension of the template file (".html" or ".mako"). Mako templates are more pow
 in that they allow you to run arbitrary code blocks quite easily, and are more strict (some
 would say finicky); Django templates are simpler, but are less expressive.
 
-## Django Models
-
-[Django Models](https://docs.djangoproject.com/en/1.11/#the-model-layer)
-are Django's Object-Relational Mapping framework. If your application
-needs to store data (history, for example), models are a good way to do it.
-
-From an abstraction perspective, it's common to imagine external services
-as "models".  For example, the Job Browser treats the Hadoop JobTracker
-as a "model", even though there's no database involved.
-
-## Accessing Hadoop
-
-It is common for applications to need to access the underlying HDFS.
-The `request.fs` object is a "file system" object that exposes
-operations that manipulate HDFS.  It is pre-configured to access
-HDFS as the user that's currently logged in.  Operations available
-on `request.fs` are similar to the file operations typically
-available in python.  See `webhdfs.py` for details; the list
-of functions available is as follows:
-`chmod`,
-`chown`,
-`exists`,
-`isdir`,
-`isfile`,
-`listdir` (and `listdir_stats`),
-`mkdir`,
-`open` (which exposes a file-like object with `read()`, `write()`, `seek()`, and `tell()` methods),
-`remove`,
-`rmdir`,
-`rmtree`, and
-`stats`.
-
-
-## Making Your Views Thread-safe
-
-Hue works in any WSGI-compliant container web server.
-The current recommended deployment server is the built-in CherryPy server.
-The CherryPy server, which is multi-threaded, is invoked by `runcpserver`
-and is configured to start when Hue's `supervisor` script is used.
-Meanwhile, `runserver` start a single-threaded
-testing server.
-
-Because multiple threads may be accessing your views
-concurrently, your views should not use shared state.
-An exception is that it is acceptable to initialize
-some state when the module is first imported.
-If you must use shared state, use Python's `threading.Lock`.
-
-Note that any module initialization may happen multiple times.
-Some WSGI containers (namely, Apache), will start multiple
-Unix processes, each with multiple threads. So, while
-you have to use locks to protect state within the process,
-there still may be multiple copies of this state.
-
-For persistent global state, it is common to place the state
-in the database or on the Browser local storage.
-
-## Authentication Backends
+
+### Authentication Backends
 
 Hue exposes a configuration flag ("auth") to configure
-a custom authentication backend.  See
+a custom authentication backend.
 See http://docs.djangoproject.com/en/dev/topics/auth/#writing-an-authentication-backend
 for writing such a backend.
 
@@ -356,7 +407,7 @@ In addition to that, backends may support a `manages_passwords_externally()` met
 True or False, to tell the user manager application whether or not changing
 passwords within Hue is possible.
 
-## Authorization
+### Authorization
 
 Applications may define permission sets for different actions. Administrators
 can assign permissions to user groups in the UserAdmin application. To define
@@ -375,7 +426,7 @@ Then you can use this decorator on your view functions to enforce permission:
     def delete_financial_report(request):
       ...
 
-## Using and Installing Thrift
+### Using and Installing Thrift
 
 Right now, we check in the generated thrift code.
 To generate the code, you'll need the thrift binary version 0.9.0.
@@ -384,7 +435,7 @@ Please download from http://thrift.apache.org/.
 The modules using ``Thrift`` have some helper scripts like ``regenerate_thrift.sh``
 for regenerating the code from the interfaces.
 
-## Profiling Hue Apps
+### Profiling Hue Apps
 
 Hue has a profiling system built in, which can be used to analyze server-side
 performance of applications.  To enable profiling::
@@ -420,64 +471,7 @@ other stats available, take a look at this website:
 http://docs.python.org/library/profile.html#pstats.Stats
 
 
-
-## Django Models
-
-Each app used to have its own model to store its data (e.g. a SQL query, a workflow). In Hue 3
-a unification of all the models happened and any app now uses a single Document2 model:
-``desktop/core/src/desktop/models.py``. This enables to avoid simply re-use document
-creation, sharing, saving etc...
-
-## REST
-Hue is Ajax based and has a REST API used by the browser to communicate (e.g. submit a query or workflow,
-list some S3 files, export a document...). Currently this API is private and subject to change but
-can be easily reused. You would need to GET ``/accounts/login`` to get the CSRF token
-and POST it back along ``username`` and ``password`` and reuse the ``sessionid`` cookie in next
-communication calls.
-
-** With Python Request **
-
-Hue is based on the Django Web Framework. Django comes with user authentication system. Django uses sessions and middleware to hook the authentication system into request object. HUE uses stock auth form which uses “username” and “password” and “csrftoken” form variables to authenticate.
-
-In this code snippet, we will use well-known python “requests” library. we will first acquire “csrftoken” by GET “login_url”. We will create python dictionary of form data which contains “username”, “password” and “csrftoken” and the “next_url” and another python dictionary for header which contains the “Referer” url and empty python dictionary for the cookies. After POST request to “login_url” we will get status. Check the r.status_code. If r.status_code!=200 then you have problem in username and/or password.
-
-Once the request is successful then capture headers and cookies for subsequent requests. Subsequent request.session calls can be made by providing cookies=session.cookies and headers=session.headers.
-
-<pre>
-import requests
-
-def login_djangosite():
- next_url = "/"
- login_url = "http://localhost:8888/accounts/login?next=/"
-
- session = requests.Session()
- r = session.get(login_url)
- form_data = dict(username="[your hue username]",password="[your hue password]",
-                  csrfmiddlewaretoken=session.cookies['csrftoken'],next=next_url)
- r = session.post(login_url, data=form_data, cookies=dict(), headers=dict(Referer=login_url))
-
- # check if request executed successfully?
- print r.status_code
-
- cookies = session.cookies
- headers = session.headers
-
- r=session.get('http://localhost:8888/metastore/databases/default/metadata',
- cookies=session.cookies, headers=session.headers)
- print r.status_code
-
- # check metadata output
- print r.text
-</pre>
-
-[Read more about it here](http://gethue.com/login-into-hue-using-the-python-request-library/).
-
-<div class="note">
-  http://issues.cloudera.org/browse/HUE-1450 is tracking a more official public API.
-</div>
-
-
-## Upgrade path
+### Upgrades
 
 After upgrading the version of Hue, running these two commands will make sure the
 database has the correct tables and fields.
@@ -485,12 +479,24 @@ database has the correct tables and fields.
     ./build/env/bin/hue syncdb
     ./build/env/bin/hue migrate
 
-# Front-end Development
+
+### Debugging Tips and Tricks
+
+* Set `DESKTOP_DEBUG=1` as an environment variable if you want logs to go to stderr
+  as well as to the respective log files.
+* Use runserver.  If you want to set a CLI breakpoint, just insert
+  `__import__("ipdb").set_trace()`
+  into your code.
+* Django tends to restart its server whenever it notices a file changes.  For
+  certain things (like configuration changes), this is not sufficient.  Restart
+  the server whole-heartedly.
+* We recommend developing with the Chrome console.
+
+
+## Web interface
 
 Developing applications for Hue requires a minimal amount of CSS
-(and potentially JavaScript) to use existing functionality. As covered above,
-creating an application for the Hue is a matter of creating a standard HTML
-application.
+(and potentially JavaScript) to use existing functionality.
 
 In a nutshell, front-end development in Hue is using
 [Bootstrap](http://twitter.github.com/bootstrap/) and
@@ -498,7 +504,7 @@ In a nutshell, front-end development in Hue is using
 interactions.
 
 
-## CSS Styles
+### CSS Styles
 
 Hue uses [Bootstrap](http://twitter.github.com/bootstrap/) version 2.0 CSS
 styles and layouts. They are highly reusable and flexible. Your app doesn't
@@ -508,7 +514,7 @@ app look at home in Hue.
 On top of the standard Bootstrap styles, Hue defines a small set of custom
 styles in *desktop/core/static/css/jhue.css*.
 
-## Defining Styles for Your Application
+### Defining Styles for Your Application
 
 When you create your application it will provision a CSS file for you in the
 *static/css* directory. For organization purposes, your styles should go here
@@ -533,7 +539,7 @@ prevent you from accidentally colliding with the framework style. Examples:
       background: url(../art/paragraph.gif);
     }
 
-## Icons
+### Icons
 
 You should create an icon for your application that is a transparent png sized
 24px by 24px. Your `settings.py` file should point to your icon via the `ICON`
@@ -551,7 +557,7 @@ like this (in your mako template):
     <!-- show a trash icon in a link -->
     <a href="#something"><i class="icon-trash"></i> Trash</a>
 
-## Static files
+### Static files
 
 For better performances, Hue uses the Django staticfiles app. If in production mode, if you edit
 some static files, you would need to run this command or `make apps`. No actions are needed in
@@ -560,32 +566,136 @@ development mode.
 ./build/env/bin/hue collectstatic
 ```
 
-## Adding Interactive Elements to Your UI
+## Testing
 
-Hue by default loads these JavaScript components:
+### The short story
 
-* Ko js
-* jQuery
-* Bootstrap
+Install the mini cluster (only once):
 
-These are used by some Hue applications, but not loaded by default:
+    ./tools/jenkins/jenkins.sh slow
 
-* Knockout js (`desktop/core/static/ext/js/knockout-min.js`)
-* jQuery UI (`desktop/core/static/ext/js/jquery/plugins/jquery-ui-autocomplete-1.8.18.min.js`)
+Run all the tests:
 
-These standard components have their own online documentation, which we will
-not repeat here. They let you write interactive behaviors with little or no
-JavaScript.
+    build/env/bin/hue test all
 
+Or just some parts of the tests, e.g.:
 
-## Debugging Tips and Tricks
+    build/env/bin/hue test specific impala
+    build/env/bin/hue test specific impala.tests:TestMockedImpala
+    build/env/bin/hue test specific impala.tests:TestMockedImpala.test_basic_flow
 
-* Set `DESKTOP_DEBUG=1` as an environment variable if you want logs to go to stderr
-  as well as to the respective log files.
-* Use runserver.  If you want to set a CLI breakpoint, just insert
-  `__import__("ipdb").set_trace()`
-  into your code.
-* Django tends to restart its server whenever it notices a file changes.  For
-  certain things (like configuration changes), this is not sufficient.  Restart
-  the server whole-heartedly.
-* We recommend developing with the Chrome console.
+Jasmine tests:
+
+    npm run test
+
+
+### Longer story
+
+The ``test`` management command prepares the arguments (test app names)
+and passes them to nose (django_nose.nose_runner). Nose will then magically
+find all the tests to run.
+
+Tests themselves should be named *_test.py.  These will be found
+as long as they're in packages covered by django.  You can use the
+unittest frameworks, or you can just name your method with
+the word "test" at a word boundary, and nose will find it.
+See apps/hello/src/hello/hello_test.py for an example.
+
+
+#### Helpful command-line tricks
+
+To run tests that do not depend on Hadoop, use:
+
+    build/env/bin/hue test fast
+
+To run all tests, use:
+
+    build/env/bin/hue test all
+
+To run only tests of a particular app, use:
+
+    build/env/bin/hue test specific <app>
+
+E.g.
+  build/env/bin/hue test specific filebrowser
+
+To run a specific test, use:
+
+    build/env/bin/hue test specific <test_func>
+
+E.g.
+  build/env/bin/hue test specific useradmin.tests:test_user_admin
+
+Start up pdb on test failures:
+
+    build/env/bin/hue test <args> --pdb --pdb-failure -s
+
+Point to an Impalad and trigger the Impala tests:
+
+    build/env/bin/hue test impala impalad-01.gethue.com
+
+
+#### Create and run the Jasmine tests
+
+Add them in a "spec" subfolder relative to the file under test and the filename of the test has to end with "Spec.js".
+
+    someFile.js              <- File under test
+    ├── spec/
+    │   ├── someFileSpec.js  <- File containing tests
+
+Run all the tests once with:
+
+    npm run test
+
+Optionally to use Karma and headless chrome for the tests you can run
+
+    npm run test-karma
+
+See ```desktop/core/src/desktop/js/spec/karma.config.js``` for various options
+
+
+#### Special environment variables
+
+    DESKTOP_LOGLEVEL=<level>
+      level can be DEBUG, INFO, WARN, ERROR, or CRITICAL
+
+      When specified, the console logger is set to the given log level. A console
+      logger is created if one is not defined.
+
+    DESKTOP_DEBUG
+      A shorthand for DESKTOP_LOG_LEVEL=DEBUG. Also turns on output HTML
+      validation.
+
+    DESKTOP_PROFILE
+      Turn on Python profiling. The profile data is saved in a file. See the
+      console output for the location of the file.
+
+    DESKTOP_LOG_DIR=$dir
+      Specify the HUE log directory. Defaults to ./log.
+
+    DESKTOP_DB_CONFIG=$db engine:db name:test db name:username:password:host:port
+      Specify alternate DB connection parameters for HUE to use. Useful for
+      testing your changes against, for example, MySQL instead of sqlite. String
+      is a colon-delimited list.
+
+    TEST_IMPALAD_HOST=impalad-01.gethue.com
+      Point to an Impalad and trigger the Impala tests.
+
+
+#### Writing tests that depend on Hadoop
+
+Use pseudo_hdfs4.py!  You should tag such tests with "requires_hadoop", as follows:
+
+    from nose.plugins.attrib import attr
+
+    @attr('requires_hadoop')
+    def your_test():
+      ...
+
+
+#### Jenkins Configuration
+
+Because building Hadoop (for the tests that require it) is slow, we've
+separated the Jenkins builds into "fast" and "slow".  Both are run
+via scripts/jenkins.sh, which should be kept updated with the latest
+and greatest in build technologies.