|
@@ -7,9 +7,12 @@ weight: 1
|
|
|
|
|
|
|
|
This section goes into greater detail on how to build and reuse the components of Hue.
|
|
This section goes into greater detail on how to build and reuse the components of Hue.
|
|
|
|
|
|
|
|
-## Dependencies
|
|
|
|
|
|
|
|
|
|
-* The OS specific dependencies are listed [here](/administrator/installation/dependencies/)
|
|
|
|
|
|
|
+## Building
|
|
|
|
|
+
|
|
|
|
|
+### Dependencies
|
|
|
|
|
+
|
|
|
|
|
+* The OS specific install instructions are listed in the [install guide](/administrator/installation/dependencies/)
|
|
|
* Python 2.7+ (Python 3 support tracked in [HUE-8737](https://issues.cloudera.org/browse/HUE-8737))
|
|
* Python 2.7+ (Python 3 support tracked in [HUE-8737](https://issues.cloudera.org/browse/HUE-8737))
|
|
|
* Django (1.11 already included in the distribution)
|
|
* Django (1.11 already included in the distribution)
|
|
|
* Java (Java 1.8) (should go away after [HUE-8740](https://issues.cloudera.org/browse/HUE-8740))
|
|
* Java (Java 1.8) (should go away after [HUE-8740](https://issues.cloudera.org/browse/HUE-8740))
|
|
@@ -18,7 +21,33 @@ This section goes into greater detail on how to build and reuse the components o
|
|
|
* [Bootstrap](http://twitter.github.com/bootstrap/)
|
|
* [Bootstrap](http://twitter.github.com/bootstrap/)
|
|
|
* [Knockout js](http://knockoutjs.com/)
|
|
* [Knockout js](http://knockoutjs.com/)
|
|
|
|
|
|
|
|
-## Building
|
|
|
|
|
|
|
+### Javascript
|
|
|
|
|
+
|
|
|
|
|
+The javascript files are currently being migrated to webpack bundles, during this process some files will live under src/desktop/static/ and some will live under src/dekstop/js
|
|
|
|
|
+
|
|
|
|
|
+For changes to the files under src/desktop/js the following applies:
|
|
|
|
|
+
|
|
|
|
|
+First make sure all third-party dependencies defined in package.json are installed into node_modules/
|
|
|
|
|
+
|
|
|
|
|
+ npm install
|
|
|
|
|
+
|
|
|
|
|
+Also run this after making changes to package.json, adding new third-party dependencies etc.
|
|
|
|
|
+
|
|
|
|
|
+To generate the js bundles run:
|
|
|
|
|
+
|
|
|
|
|
+ npm run webpack
|
|
|
|
|
+ npm run webpack-workers
|
|
|
|
|
+ npm run webpack-login
|
|
|
|
|
+
|
|
|
|
|
+During development the bundles can be autogenerated when it detects changes to the .js files, for this run:
|
|
|
|
|
+
|
|
|
|
|
+ npm run dev
|
|
|
|
|
+
|
|
|
|
|
+Before sending a review with changes to the bundles run:
|
|
|
|
|
+
|
|
|
|
|
+ npm run lint-fix
|
|
|
|
|
+
|
|
|
|
|
+and possibly fix any issues it might report.
|
|
|
|
|
|
|
|
### Documentation
|
|
### Documentation
|
|
|
|
|
|
|
@@ -56,34 +85,6 @@ Old documentation:
|
|
|
|
|
|
|
|
make docs
|
|
make docs
|
|
|
|
|
|
|
|
-### Javascript
|
|
|
|
|
-
|
|
|
|
|
-The javascript files are currently being migrated to webpack bundles, during this process some files will live under src/desktop/static/ and some will live under src/dekstop/js
|
|
|
|
|
-
|
|
|
|
|
-For changes to the files under src/desktop/js the following applies:
|
|
|
|
|
-
|
|
|
|
|
-First make sure all third-party dependencies defined in package.json are installed into node_modules/
|
|
|
|
|
-
|
|
|
|
|
- npm install
|
|
|
|
|
-
|
|
|
|
|
-Also run this after making changes to package.json, adding new third-party dependencies etc.
|
|
|
|
|
-
|
|
|
|
|
-To generate the js bundles run:
|
|
|
|
|
-
|
|
|
|
|
- npm run webpack
|
|
|
|
|
- npm run webpack-workers
|
|
|
|
|
- npm run webpack-login
|
|
|
|
|
-
|
|
|
|
|
-During development the bundles can be autogenerated when it detects changes to the .js files, for this run:
|
|
|
|
|
-
|
|
|
|
|
- npm run dev
|
|
|
|
|
-
|
|
|
|
|
-Before sending a review with changes to the bundles run:
|
|
|
|
|
-
|
|
|
|
|
- npm run lint-fix
|
|
|
|
|
-
|
|
|
|
|
-and possibly fix any issues it might report.
|
|
|
|
|
-
|
|
|
|
|
### CSS / LESS
|
|
### CSS / LESS
|
|
|
|
|
|
|
|
After changing the CSS in a .less file, rebuilding with:
|
|
After changing the CSS in a .less file, rebuilding with:
|
|
@@ -125,7 +126,26 @@ How to create a new locale for an app:
|
|
|
cd $APP_ROOT/src/$APP_NAME/locale
|
|
cd $APP_ROOT/src/$APP_NAME/locale
|
|
|
$HUE_ROOT/build/env/bin/pybabel init -D django -i en_US.pot -d . -l fr
|
|
$HUE_ROOT/build/env/bin/pybabel init -D django -i en_US.pot -d . -l fr
|
|
|
|
|
|
|
|
-## Webapp API
|
|
|
|
|
|
|
+### Dev environment
|
|
|
|
|
+
|
|
|
|
|
+#### Debugging with PyCharm
|
|
|
|
|
+
|
|
|
|
|
+First step is to configure pycharm to use the Hue virtual environment at ./build/env/env
|
|
|
|
|
+
|
|
|
|
|
+Second step is to configure the debug configuration
|
|
|
|
|
+
|
|
|
|
|
+
|
|
|
|
|
+#### Debugging with Eclipse
|
|
|
|
|
+
|
|
|
|
|
+First step is to configure Eclipse to use the Hue virtual environment at ./build/env/env
|
|
|
|
|
+
|
|
|
|
|
+Second step is to configure the debug configuration
|
|
|
|
|
+
|
|
|
|
|
+
|
|
|
|
|
+
|
|
|
|
|
+
|
|
|
|
|
+
|
|
|
|
|
+## API Server
|
|
|
|
|
|
|
|
### From 30,000 feet
|
|
### From 30,000 feet
|
|
|
|
|
|
|
@@ -154,9 +174,8 @@ daemon processes "on the side". Some examples are the `Celery Task Server`, `Cel
|
|
|
|
|
|
|
|

|
|

|
|
|
|
|
|
|
|
-Hue provides some APIs for interacting with Hadoop.
|
|
|
|
|
-Most noticeably, there are python file-object-like APIs for
|
|
|
|
|
-interacting with HDFS. These APIs work by making REST API or Thrift calls
|
|
|
|
|
|
|
+Hue provides some APIs for interacting with external services like Databases of File storages.
|
|
|
|
|
+These APIs work by making REST API or Thrift calls
|
|
|
the Hadoop daemons.
|
|
the Hadoop daemons.
|
|
|
|
|
|
|
|
### An Architectural View
|
|
### An Architectural View
|
|
@@ -213,7 +232,9 @@ For the URLs within your application, you should make your own ``urls.py``
|
|
|
which will be automatically rooted at ``/yourappname/`` in the global
|
|
which will be automatically rooted at ``/yourappname/`` in the global
|
|
|
namespace. See ``apps/about/src/about/urls.py`` for an example.
|
|
namespace. See ``apps/about/src/about/urls.py`` for an example.
|
|
|
|
|
|
|
|
-### Configuration File
|
|
|
|
|
|
|
+### Configurations
|
|
|
|
|
+
|
|
|
|
|
+#### File
|
|
|
|
|
|
|
|
Hue uses a typed configuration system that reads configuration files (in an
|
|
Hue uses a typed configuration system that reads configuration files (in an
|
|
|
ini-style format). By default, Hue loads all `*.ini` files in the `build/desktop/conf`
|
|
ini-style format). By default, Hue loads all `*.ini` files in the `build/desktop/conf`
|
|
@@ -235,62 +256,60 @@ directory. The configuration files have the following format:
|
|
|
# namenode_host = 10.0.0.1
|
|
# namenode_host = 10.0.0.1
|
|
|
|
|
|
|
|
|
|
|
|
|
-### Configuration Variables
|
|
|
|
|
|
|
+#### Variables
|
|
|
|
|
|
|
|
Your application's `conf.py` is special. It provides access to the configuration file (and even
|
|
Your application's `conf.py` is special. It provides access to the configuration file (and even
|
|
|
default configurations not specified in the file). Using the above example, your `conf.py` should
|
|
default configurations not specified in the file). Using the above example, your `conf.py` should
|
|
|
define the following:
|
|
define the following:
|
|
|
|
|
|
|
|
-* A `desktop.lib.conf.Config` object for `app_property`, such as:
|
|
|
|
|
-<pre>
|
|
|
|
|
- MY_PROPERTY = Config(key='app_property', default='Beatles', help='blah')
|
|
|
|
|
-</pre>
|
|
|
|
|
|
|
+A `desktop.lib.conf.Config` object for `app_property`, such as:
|
|
|
|
|
+
|
|
|
|
|
+ MY_PROPERTY = Config(key='app_property', default='Beatles', help='blah')
|
|
|
|
|
+
|
|
|
You can access its value by `MY_PROPERTY.get()`.
|
|
You can access its value by `MY_PROPERTY.get()`.
|
|
|
|
|
|
|
|
-* A `desktop.lib.conf.ConfigSection` object for `section_a`, such as:
|
|
|
|
|
-<pre>
|
|
|
|
|
- SECTION_A = ConfigSection(key='section_a',
|
|
|
|
|
- help='blah',
|
|
|
|
|
- members=dict(
|
|
|
|
|
- AWEIGHT=Config(key='a_weight', type=int, default=0),
|
|
|
|
|
- AHEIGHT=Config(key='a_height', type=int, default=0)))
|
|
|
|
|
-</pre>
|
|
|
|
|
|
|
+A `desktop.lib.conf.ConfigSection` object for `section_a`, such as:
|
|
|
|
|
+
|
|
|
|
|
+ SECTION_A = ConfigSection(key='section_a',
|
|
|
|
|
+ help='blah',
|
|
|
|
|
+ members=dict(
|
|
|
|
|
+ AWEIGHT=Config(key='a_weight', type=int, default=0),
|
|
|
|
|
+ AHEIGHT=Config(key='a_height', type=int, default=0)))
|
|
|
|
|
+
|
|
|
You can access the values by `SECTION_A.AWEIGHT.get()`.
|
|
You can access the values by `SECTION_A.AWEIGHT.get()`.
|
|
|
|
|
|
|
|
-* A `desktop.lib.conf.UnspecifiedConfigSection` object for `filesystems`, such as:
|
|
|
|
|
-<pre>
|
|
|
|
|
- FS = UnspecifiedConfigSection(
|
|
|
|
|
- key='filesystems',
|
|
|
|
|
- each=ConfigSection(members=dict(
|
|
|
|
|
- nn_host=Config(key='namenode_host', required=True))
|
|
|
|
|
-</pre>
|
|
|
|
|
- An `UnspecifiedConfigSection` is useful when the children of the section are not known.
|
|
|
|
|
- When Hue loads your application's configuration, it binds all sub-sections. You can
|
|
|
|
|
- access the values by:
|
|
|
|
|
-<pre>
|
|
|
|
|
- cluster1_val = FS['cluster_1'].nn_host.get()
|
|
|
|
|
- all_clusters = FS.keys()
|
|
|
|
|
- for cluster in all_clusters:
|
|
|
|
|
- val = FS[cluster].nn_host.get()
|
|
|
|
|
-</pre>
|
|
|
|
|
-
|
|
|
|
|
-Your Hue application can automatically detect configuration problems and alert
|
|
|
|
|
|
|
+A `desktop.lib.conf.UnspecifiedConfigSection` object for `filesystems`, such as:
|
|
|
|
|
+
|
|
|
|
|
+ FS = UnspecifiedConfigSection(
|
|
|
|
|
+ key='filesystems',
|
|
|
|
|
+ each=ConfigSection(members=dict(
|
|
|
|
|
+ nn_host=Config(key='namenode_host', required=True))
|
|
|
|
|
+
|
|
|
|
|
+An `UnspecifiedConfigSection` is useful when the children of the section are not known.
|
|
|
|
|
+When Hue loads your application's configuration, it binds all sub-sections. You can
|
|
|
|
|
+access the values by:
|
|
|
|
|
+
|
|
|
|
|
+ cluster1_val = FS['cluster_1'].nn_host.get()
|
|
|
|
|
+ all_clusters = FS.keys()
|
|
|
|
|
+ for cluster in all_clusters:
|
|
|
|
|
+ val = FS[cluster].nn_host.get()
|
|
|
|
|
+
|
|
|
|
|
+
|
|
|
|
|
+Application can automatically detect configuration problems and alert
|
|
|
the admin. To take advantage of this feature, create a `config_validator`
|
|
the admin. To take advantage of this feature, create a `config_validator`
|
|
|
function in your `conf.py`:
|
|
function in your `conf.py`:
|
|
|
|
|
|
|
|
-<pre>
|
|
|
|
|
- def config_validator(user):
|
|
|
|
|
- """
|
|
|
|
|
- config_validator(user) -> [(config_variable, error_msg)] or None
|
|
|
|
|
- Called by core check_config() view.
|
|
|
|
|
- """
|
|
|
|
|
- res = [ ]
|
|
|
|
|
- if not REQUIRED_PROPERTY.get():
|
|
|
|
|
- res.append((REQUIRED_PROPERTY, "This variable must be set"))
|
|
|
|
|
- if MY_INT_PROPERTY.get() < 0:
|
|
|
|
|
- res.append((MY_INT_PROPERTY, "This must be a non-negative number"))
|
|
|
|
|
- return res
|
|
|
|
|
-</pre>
|
|
|
|
|
|
|
+ def config_validator(user):
|
|
|
|
|
+ """
|
|
|
|
|
+ config_validator(user) -> [(config_variable, error_msg)] or None
|
|
|
|
|
+ Called by core check_config() view.
|
|
|
|
|
+ """
|
|
|
|
|
+ res = [ ]
|
|
|
|
|
+ if not REQUIRED_PROPERTY.get():
|
|
|
|
|
+ res.append((REQUIRED_PROPERTY, "This variable must be set"))
|
|
|
|
|
+ if MY_INT_PROPERTY.get() < 0:
|
|
|
|
|
+ res.append((MY_INT_PROPERTY, "This must be a non-negative number"))
|
|
|
|
|
+ return res
|
|
|
|
|
|
|
|
|
|
|
|
|
<div class="note">
|
|
<div class="note">
|
|
@@ -304,39 +323,6 @@ function in your `conf.py`:
|
|
|
</div>
|
|
</div>
|
|
|
|
|
|
|
|
|
|
|
|
|
-### Running "Helper Processes"
|
|
|
|
|
-
|
|
|
|
|
-Some Hue applications need to run separate daemon processes on the side.
|
|
|
|
|
-
|
|
|
|
|
-Suppose your application needs a helper `my_daemon.py`. You need to register it by:
|
|
|
|
|
-
|
|
|
|
|
-* In `setup.py`, add to `entry_points`:
|
|
|
|
|
-<pre>
|
|
|
|
|
- entry_points = {
|
|
|
|
|
- 'desktop.sdk.application': 'my_app = my_app',
|
|
|
|
|
- 'desktop.supervisor.specs': [ 'my_daemon = my_app:SUPERVISOR_SPEC' ] }
|
|
|
|
|
-</pre>
|
|
|
|
|
-
|
|
|
|
|
-* In `src/my_app/__init__.py`, tell Hue what to run by adding:
|
|
|
|
|
-<pre>
|
|
|
|
|
- SUPERVISOR_SPEC = dict(django_command="my_daemon")
|
|
|
|
|
-</pre>
|
|
|
|
|
-
|
|
|
|
|
-* Then in `src/my_app/management/commands`, create `__init__.py` and `my_daemon.py`. Your
|
|
|
|
|
- daemon program has only one requirement: it must define a class called `Command` that
|
|
|
|
|
- extends `django.core.management.base.BaseCommand`. Please see `kt_renewer.py` for an example.
|
|
|
|
|
-
|
|
|
|
|
-The next time Hue restarts, your `my_daemon` will start automatically.
|
|
|
|
|
-If your daemon program dies (exits with a non-zero exit code), Hue will
|
|
|
|
|
-restart it.
|
|
|
|
|
-
|
|
|
|
|
-"Under the covers:" Threading. Hue, by default, runs CherryPy web server.
|
|
|
|
|
-If Hue is configured (and it may be, in the future)
|
|
|
|
|
-to use mod_wsgi under Apache httpd, then there would be multiple python
|
|
|
|
|
-processes serving the backend. This means that your Django application
|
|
|
|
|
-code should avoid depending on shared process state. Instead, place
|
|
|
|
|
-the stored state in a database or run a separate server.
|
|
|
|
|
-
|
|
|
|
|
### Saving documents
|
|
### Saving documents
|
|
|
|
|
|
|
|
Each app used to have its own model to store its data (e.g. a saving a SQL query, query history...). All the models have been unified into a single Document2 model in the desktop app:
|
|
Each app used to have its own model to store its data (e.g. a saving a SQL query, query history...). All the models have been unified into a single Document2 model in the desktop app:
|
|
@@ -349,27 +335,10 @@ The `Document2` model provides automatically creation, sharing and saving. It pe
|
|
|
are Django's Object-Relational Mapping framework.
|
|
are Django's Object-Relational Mapping framework.
|
|
|
|
|
|
|
|
|
|
|
|
|
-### Managing users
|
|
|
|
|
-
|
|
|
|
|
-Except for static content, `request.user` is always populated. It is a
|
|
|
|
|
-standard Django `models.User` object. If you were to set a breakpoint at the
|
|
|
|
|
-`index()` function in our calculator app, you will find:
|
|
|
|
|
-
|
|
|
|
|
- >>> request.user
|
|
|
|
|
- <User: test>
|
|
|
|
|
-
|
|
|
|
|
-<div class="note">
|
|
|
|
|
- "Under the covers:" Django uses a notion called
|
|
|
|
|
- <a href="https://docs.djangoproject.com/en/1.2/topics/http/middleware/">middleware</a>
|
|
|
|
|
- that's called in between the request coming in and the view being executed.
|
|
|
|
|
- That's how <code>request.user</code> gets populated. There's also a
|
|
|
|
|
- middleware for Hue that makes sure that no pages are displayed unless the
|
|
|
|
|
- user is authenticated.
|
|
|
|
|
-</div>
|
|
|
|
|
-
|
|
|
|
|
### Walk-through of a Django View
|
|
### Walk-through of a Django View
|
|
|
|
|
|
|
|
-
|
|
|
|
|
|
|
+
|
|
|
|
|
+
|
|
|
|
|
|
|
|
Django is an MVC framework, except that the controller is called a
|
|
Django is an MVC framework, except that the controller is called a
|
|
|
"[view](https://docs.djangoproject.com/en/1.11/#the-view-layer)" and
|
|
"[view](https://docs.djangoproject.com/en/1.11/#the-view-layer)" and
|
|
@@ -435,10 +404,10 @@ Please download from http://thrift.apache.org/.
|
|
|
The modules using ``Thrift`` have some helper scripts like ``regenerate_thrift.sh``
|
|
The modules using ``Thrift`` have some helper scripts like ``regenerate_thrift.sh``
|
|
|
for regenerating the code from the interfaces.
|
|
for regenerating the code from the interfaces.
|
|
|
|
|
|
|
|
-### Profiling Hue Apps
|
|
|
|
|
|
|
+### Profiling
|
|
|
|
|
|
|
|
Hue has a profiling system built in, which can be used to analyze server-side
|
|
Hue has a profiling system built in, which can be used to analyze server-side
|
|
|
-performance of applications. To enable profiling::
|
|
|
|
|
|
|
+performance of applications. To enable profiling:
|
|
|
|
|
|
|
|
build/env/bin/hue runprofileserver
|
|
build/env/bin/hue runprofileserver
|
|
|
|
|
|
|
@@ -449,12 +418,12 @@ the file names is /tmp/<app_module>.<page_url>.<time_taken>.<timestamp>.prof.
|
|
|
Hue uses the hotshot profiling library for instrumentation. The documentation
|
|
Hue uses the hotshot profiling library for instrumentation. The documentation
|
|
|
for this library is located at: http://docs.python.org/library/hotshot.html.
|
|
for this library is located at: http://docs.python.org/library/hotshot.html.
|
|
|
|
|
|
|
|
-You can use kcachegrind to view the profiled data graphically::
|
|
|
|
|
|
|
+You can use kcachegrind to view the profiled data graphically:
|
|
|
|
|
|
|
|
- $ hotshot2calltree /tmp/xyz.prof > /tmp/xyz.trace
|
|
|
|
|
- $ kcachegrind /tmp/xyz.trace
|
|
|
|
|
|
|
+ hotshot2calltree /tmp/xyz.prof > /tmp/xyz.trace
|
|
|
|
|
+ kcachegrind /tmp/xyz.trace
|
|
|
|
|
|
|
|
-More generally, you can programmatically inspect a trace::
|
|
|
|
|
|
|
+More generally, you can programmatically inspect a trace:
|
|
|
|
|
|
|
|
#!/usr/bin/python
|
|
#!/usr/bin/python
|
|
|
import hotshot.stats
|
|
import hotshot.stats
|
|
@@ -503,6 +472,11 @@ In a nutshell, front-end development in Hue is using
|
|
|
[Knockout js](http://knockoutjs.com/) to layout your app and script the custom
|
|
[Knockout js](http://knockoutjs.com/) to layout your app and script the custom
|
|
|
interactions.
|
|
interactions.
|
|
|
|
|
|
|
|
|
|
+### Node.js
|
|
|
|
|
+
|
|
|
|
|
+During development the bundles can be autogenerated when it detects changes to the .js files, for this run:
|
|
|
|
|
+
|
|
|
|
|
+ npm run dev
|
|
|
|
|
|
|
|
### CSS Styles
|
|
### CSS Styles
|
|
|
|
|
|
|
@@ -514,7 +488,7 @@ app look at home in Hue.
|
|
|
On top of the standard Bootstrap styles, Hue defines a small set of custom
|
|
On top of the standard Bootstrap styles, Hue defines a small set of custom
|
|
|
styles in *desktop/core/static/css/jhue.css*.
|
|
styles in *desktop/core/static/css/jhue.css*.
|
|
|
|
|
|
|
|
-### Defining Styles for Your Application
|
|
|
|
|
|
|
+### Defining Styles
|
|
|
|
|
|
|
|
When you create your application it will provision a CSS file for you in the
|
|
When you create your application it will provision a CSS file for you in the
|
|
|
*static/css* directory. For organization purposes, your styles should go here
|
|
*static/css* directory. For organization purposes, your styles should go here
|
|
@@ -699,23 +673,3 @@ Because building Hadoop (for the tests that require it) is slow, we've
|
|
|
separated the Jenkins builds into "fast" and "slow". Both are run
|
|
separated the Jenkins builds into "fast" and "slow". Both are run
|
|
|
via scripts/jenkins.sh, which should be kept updated with the latest
|
|
via scripts/jenkins.sh, which should be kept updated with the latest
|
|
|
and greatest in build technologies.
|
|
and greatest in build technologies.
|
|
|
-
|
|
|
|
|
-#### Debugging with PyCharm
|
|
|
|
|
-
|
|
|
|
|
-First step is to configure pycharm to use the Hue virtual environment at ./build/env/env
|
|
|
|
|
-
|
|
|
|
|
-Second step is to configure the debug configuration
|
|
|
|
|
-
|
|
|
|
|
-
|
|
|
|
|
-#### Debugging with Eclipse
|
|
|
|
|
-
|
|
|
|
|
-First step is to configure Eclipse to use the Hue virtual environment at ./build/env/env
|
|
|
|
|
-
|
|
|
|
|
-Second step is to configure the debug configuration
|
|
|
|
|
-
|
|
|
|
|
-
|
|
|
|
|
-
|
|
|
|
|
-
|
|
|
|
|
-
|
|
|
|
|
-
|
|
|
|
|
-
|
|
|