Hue Manual
==========
== Introduction
Hue is a graphical user interface to operate and develop for Hadoop.
Hue "applications" are collected into a desktop
environment and delivered as a Web application, requiring no
additional installation for individual users'.
This document will you install and configure Hue. There
is a companion SDK document for information
on developing new Hue applications.
Hue currently requires Cloudera's Distribution of Hadoop,
version CDH2 or CDH3.
.Note:
* All commands that need to be run as +root+ have a +#+ command prompt.
* All commands that do not require +root+ have a +$+ command prompt.
== Installation Instructions
These instructions will guide you through installing Hue
on a multi-node cluster. You will need to update
some Hadoop configuration as well as install Hue.
IMPORTANT: You'll need to install the Hue Plug-ins
on _every_ machine that's running Hadoop daemons.
=== Install Hadoop First!
Hue requires an installation of the Cloudera Distribution
of Hadoop (version 2 or above). See link:http://archive.cloudera.com/
for instructions.
To use Hue, you must be running Cloudera's Distribution for Hadoop
with a version number of at least `0.20.1+133`. If you are not running at least this
version of Hadoop, please upgrade your cluster before proceeding.
=== Install Hue
Hue consists of a web service that runs on a special node in your cluster.
You should pick one node to run Hue. From this point on, we'll refer to
that node as the "Hue Server". This should be one of the nodes within your
cluster for optimal performance, though it can be a remote node as long as there are no
overly restrictive firewalls.
For small clusters of less than 10 nodes, it is fine to choose your existing master node
as the Hue Server.
You install Hue, begin by downloading the link:http://archive.cloudera.com/cdh/3/hue-0.9.tar.gz[tarball].
==== Installing Dependencies
Hue employs some Python modules which use native code and depend
on certain development libraries being on your system. To install from the
tarball you must have the following installed:
.Required Dependencies
[grid="rows"]
``~
Redhat,Debian
~~~~~~~~~~
libxslt-devel,libxslt-dev
libxml2-devel,libxml2-dev
sqlite-devel,libsqlite3-dev
python-devel,python2.4-dev(orpython2.5-dev)
gcc,gcc
python-setuptools,python-setuptools
~~~~~~~~~~
Additionally, you must have Cloudera's Distribution for Hadoop installed
and available. Specifically, your Hadoop version must be at least 0.20.1+133.
If you have a previous release, upgrade Hadoop before continuing with Hue.
==== Build
With `$HADOOP_HOME` and `$PREFIX` configured to the path of your Hadoop
installation and the path where you'd like to install Hue,
respectively, run:
----
$ HADOOP_HOME=/path/to/hadoop-0.20.1+152 PREFIX=/path/to/install/into make install
----
You may install Hue anywhere on your system - it does not need root access.
We recommend that you create a new user for Hue and either install in that
user's home directory or in a directory within /usr/local.
==== Install Hadoop Plugins
In order to communicate with Hadoop, Hue requires that you install and configure
a plugin JAR. This jar is desktop/libs/hadoop/java-lib/hue-plugins-0.9.jar.
Symlink this jar into your Hadoop lib directory (/usr/lib/hadoop-0.20/lib if you've installed
CDH via a Debian or RPM package):
----
$ cd /usr/lib/hadoop/lib
$ ln -s /usr/share/hue/desktop/libs/hadoop/java-lib/hue*jar .
# Restart Hadoop
----
NOTE: On a multi-node cluster, you will need to install the plug-ins on every
node. You do not need to install the entirety of Hue everywhere,
but the plug-in jars need to be available on every machine.
==== Restart Hadoop
Once you have made these changes in your Hadoop configuration, go ahead and
restart your Hadoop daemons.
==== Running Hue
To start Hue, you'll use `build/env/bin/supervisor`. This will start
several subprocesses, corresponding to different bits of Hue.
==== FAQ about Tarball Installation
.I moved my Hue installation from one directory to another and it no longer
functions correctly.
Due to the use of absolute paths by some python packages, you'll have to run a series of commands
if you relocate your Hue installation. From within the new location, run:
----
$ make apps
----
This should solve the problem.
.Why does "make install" compile all of these other pieces of software?
In order to ensure that Hue is stable on a variety of distributions and architectures,
it installs a Python "virtual environment" which includes its dependencies. This ensures that
the software can depend against specific versions of various Python libraries and not have to worry
about what might or might not be installed already on your particular system.
=== Configuring Hadoop for Hue
Hue requires that you install and configure some plugins in your
Hadoop installation.
When you installed the `hue-plugins` package above, your package manager
automatically added the required plugin jar
to your Hadoop installation's `lib` directory, making them available to Hadoop.
In order to enable the plugins, you'll need to make some small additions to your
configuration.
These configuration changes should be made on each node in your cluster by editing the files
in `/etc/hadoop-0.20/conf/`
==== `hdfs-site.xml`
You'll want to add the following configuration
options to hdfs-site.xml.
----
dfs.namenode.plugins
org.apache.hadoop.thriftfs.NamenodePlugin
Comma-separated list of namenode plug-ins to be activated.
dfs.datanode.plugins
org.apache.hadoop.thriftfs.DatanodePlugin
Comma-separated list of datanode plug-ins to be activated.
dfs.thrift.address
0.0.0.0:9090
-----
==== `mapred-site.xml`
Add the following to mapred-site.xml:
----
jobtracker.thrift.address
0.0.0.0:9290
mapred.jobtracker.plugins
org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin
Comma-separated list of jobtracker plug-ins to be activated.
----
==== `hadoop-metrics.properties`
To enable full monitoring in the Health application, the metrics
contexts must not be NullContext. You might configure `hadoop-metrics.properties`
like so:
----
# Exposes /metrics URL endpoint for metrics information.
dfs.class=org.apache.hadoop.metrics.spi.NoEmitMetricsContext
mapred.class=org.apache.hadoop.metrics.spi.NoEmitMetricsContext
jvm.class=org.apache.hadoop.metrics.spi.NoEmitMetricsContext
rpc.class=org.apache.hadoop.metrics.spi.NoEmitMetricsContext
----
=== Further Hadoop Configuration and Caveats
==== `HADOOP_CLASSPATH` Caveat
If you are setting `$HADOOP_CLASSPATH` in your `hadoop-env.sh`, be sure
to set it in such a way that user-specified options are preserved. For example,
----
# Good
HADOOP_CLASSPATH=:$HADOOP_CLASSPATH
# Bad
# HADOOP_CLASSPATH=
----
This will enable certain portions of Hue to add to
Hadoop's classpath using the environment variable.
==== `hadoop.tmp.dir`
If your users are likely to be submitting jobs both using Hue
and from the same machine, they will be doing so as the `hue`
user if they're using Hue and as the their own username
if they're using the command line. This yields to some contention
on the directory specified by `hadoop.tmp.dir`, which defaults
to `/tmp/hadoop-${user.name}`.
Specifically, `hadoop.tmp.dir` is used to unpack jars in `bin/hadoop jar`.
One work around to this is
to set `hadoop.tmp.dir` to `/tmp/hadoop-${user.name}-${hue.suffix}`:
----
hadoop.tmp.dir
/tmp/hadoop-${user.name}${hue.suffix}
----
Unfortunately, when the variable is unset, you'll end up
with directories named `/tmp/hadoop-user_name-${hue.suffix}` in
`/tmp`. The job submission daemon, however, will
=== Restart Your Hadoop Cluster
At this point you should restart all of the daemons in your cluster so that the plugins can be loaded.
You can confirm that the plugins are running correctly by tailing the daemon logs:
----
$ tail --lines=500 /var/log/hadoop-0.20/hadoop*namenode*.log | grep ThriftPlugin
2009-09-28 16:30:44,337 INFO org.apache.hadoop.thriftfs.ThriftPluginServer: Starting Thrift server
2009-09-28 16:30:44,419 INFO org.apache.hadoop.thriftfs.ThriftPluginServer: Thrift server listening on 0.0.0.0:9090
----
[TIP]
.Configuring Your Firewall for Hue
============================================================
Hue currently requires that the machines within your cluster can speak to each other freely over TCP.
The machines outside your cluster only need to be able to open TCP port 8088 on the Hue Server to interact with the system.
============================================================
== Configuring Hue
Hue ships with a default configuration that will work for
pseudo-distributed clusters. If you are running on a real cluster, you'll need
to make a few small changes to its configuration file. Here we go through the
key configuration options.
Edit `/etc/hue/hue.ini` in your favorite editor.
[TIP]
.Full Listing of Configuration Options
============================================================
In order to see a full listing of the available configuration options, you can run:
----
/usr/share/hue/build/env/bin/desktop config_help | less
----
This will outline the various sections and options in the configuration, as
well as provide help and information on the default values.
============================================================
[TIP]
.View Current Configuration Options
============================================================
You can also view the current configuration from within Hue, at:
----
http:///dump_config
----
============================================================
[TIP]
.Using Multiple Files to Store Your Configuration
============================================================
Hue will load and merge all of the files with extension `.ini`
located in the `/etc/hue/conf/` directory. Files that are alphabetically later
will take precedence.
============================================================
=== Webserver Configuration
Hue uses the CherryPy web server. You can change the IP address
and port the web server listens on, which is port 8088 on all configured IP
addresses. (Use `http_host` and `http_port`.)
=== Authentication
By default, the first user who logins to Hue may choose any
username and password, and becomes an administrator automatically. This
user may create other user and administrator accounts. User information is
stored in the Django backend, in the Django database.
The authentication system is pluggable; developers should
refer to the SDK documentation, if interested.
=== Configuring for SSL
You can configure Hue to serve over HTTPS. To do so, you'll need
to install "pyOpenSSL" within Desktop's context and configure your keys.
To install `pyOpenSSL`, from the root of your Desktop installation
(`/usr/share/hue` if you installed from packages, type)
execute:
----
$ ./build/env/bin/easy_install pyOpenSSL
----
Then, configure Hue to use your private
key by putting the following inside your
`/etc/hue/hue.ini`:
----
ssl_certificate=/path/to/certificate
ssl_private_key=/path/to/key
----
Ideally, you would have an appropriate key signed by a Certificate Authority.
If you're just testing, you can create a self-signed key using the `openssl`
command, that may be installed on your system:
----
# Create a key
$ openssl genrsa 1024 > host.key
# Create a self-signed certificate
$ openssl req -new -x509 -nodes -sha1 -key host.key > host.cert
----
[NOTE]
.Self-signed Certificates and File Uploads
============================================================
To upload files using the File Browser over HTTPS requires
using a proper SSL Certificate. Self-signed certificates
are known to not work.
============================================================
=== Pointing Hue to Your Master Nodes
If your Hadoop cluster is made up of multiple nodes, you should configure
Hue to point to the external hostnames of your Namenode and
JobTracker. To do so, simply change the `namenode_host` and `jobtracker_host`
lines in the configuration file. The inline comments in the existing file will
guide you.
== Starting Hue
Once your cluster is up and running with the plugins enabled, you can start Hue.
On your Hue Server, run:
----
# build/env/bin/supervisor
----
Congratulations! Your Hue installation is now up and running!
== Administering Hue
Now that you've installed and started Hue, you can feel free to skip ahead
to <>. Administrators may want to refer to this section
for more details about managing and operating a Hue installation.
=== Hue Processes
==== Process Hierarchy
Hue runs several processes under the hood, all managed by a script
called the `supervisor`. The supervisor is a watchdog process -- its only purpose
is to spawn and monitor other processes.
A stock Hue installation will spawn and monitor the following processes:
* `runcpserver` - a web server based on CherryPy that provides the core web functionality of Hue
* `jobsubd` - a daemon which handles submission of jobs to Hadoop
If you have installed other applications into your Hue instance, you may see other daemons running under
the supervisor as well.
We can see the supervised processes running in the output of `ps`:
------------------
[todd@monster01 ~]$ ps -f -u hue
UID PID PPID C STIME TTY TIME CMD
hue 7899 1 0 12:14 ? 00:00:00 /usr/share/hue/build/build/env/bin/python2.4 \
/usr/share/hue/desktop/build/env/bin/supervisor -p /var/run/hue/desktop/s
hue 7903 7899 1 12:14 ? 00:02:59 /usr/share/hue/desktop/build/env/bin/python2.4 \
/usr/share/hue/desktop/build/env/bin/desktop runcpserver
hue 7906 7899 0 12:14 ? 00:00:00 /usr/share/hue/desktop/build/env/bin/python2.4 \
/usr/share/hue/desktop/build/env/bin/desktop jobsubd
hue 7907 7899 0 12:14 ? 00:00:12 /usr/share/hue/desktop/build/env/bin/python2.4 \
/usr/share/hue/desktop/build/env/bin/desktop run_healthd
------------------
Note that the supervisor will automatically restart these processes should they fail for any reason.
If the processes fail repeatedly within a small time window, the supervisor itself will shut down.
==== Managing Hue Processes
Hue RPMs and Debian packages ship with an `init.d` script to manage the Hue processes.
You can start and stop the Hue Supervisor using this init script, for example:
----
# /etc/init.d/hue stop
----
If for some reason the init scripts are unable to stop the process, you can kill the daemon manually by locating and
killing the `supervisor` process as described above.
=== Hue Logging
The Hue logs are found in `/path/to/hue/logs` if
you've installed via a tarball. Inside the log directory you will find:
* `access.log`, which contains a log for all requests against the Hue web server.
* A `.log` file containing the logs for each of the processes described above.
* A `.out` file containing the stdout and stderr for each of the processes described above.
If users on your cluster experience issues, you can often find error messages in these log files.
If you are unable to start Hue from the init script, the
`supervisor.log` log file can often contain clues.
==== Viewing recent log messages through your browser
In addition to logging `INFO` level messages to the log directory, the Hue web server keeps a small buffer
of log messages at all levels in memory. You can view these logs by visiting `http://myserver:8088/logs`.
The `DEBUG` level messages present here can sometimes be helpful in troubleshooting issues.
=== The Hue Database
Hue requires a SQL database to store small amounts of data, including user account information
as well as history of job submissions and Hive queries. By default, Hue is configured to use the
embedded database SQLite for this purpose, and should require no configuration or management by the
administrator.
==== Inspecting the Hue Database
The default SQLite database used by Hue is located in `/usr/share/hue/desktop/desktop.db`.
You can inspect this database from the command line using the `sqlite3` program. For example:
---------
# sqlite3 /usr/share/hue/desktop/desktop.db
SQLite version 3.3.6
Enter ".help" for instructions
sqlite> select username from auth_user;
admin
test
sample
sqlite>
----------
We strongly advise you not to make any modifications to the database directly using SQLite, though this trick
may be handy for management or troubleshooting.
==== Backing up the Hue Database
To back up the Hue Database, you can simply copy the `desktop.db` file to another node. We recommend that
you back it up on a regular schedule, and also that you back it up before any upgrade to a new version of
Hue.
==== Configuring Hue to access another Database
Although SQLite is the default database type, some advanced users may prefer to have Hue access an
alternate database type. Please note that, if you elect to configure Hue to use an external database,
upgrades may require more manual steps in the future.
Here we provide instructions for MySQL, though Hue may also be made to work
with other common databases including PostgreSQL, Oracle, etc.
===== Configuring Hue to store data in MySQL
First, you must create a new database in MySQL and grant privileges to a Hue user to manage
this database.
----
mysql> create database hue;
Query OK, 1 row affected (0.01 sec)
mysql> grant all on hue.* to 'hue'@'localhost' identified by 'secretpassword';
Query OK, 0 rows affected (0.00 sec)
----
Next, shut down Hue if it is running, and edit `/etc/hue/hue.ini`. Directly below the
`[desktop]` line, add the following:
----
[[database]]
host=localhost
port=3306
engine=mysql
user=hue
password=secretpassword
name=hue
----
Next we need to install the python drivers for MySQL into Hue's environment:
----
# su - hue -s /bin/bash
$ /usr/share/hue/build/env/bin/easy_install MySQL-python
----
`easy_install` will access the Internet to find MySQL-python.
If you don't have network access, you may specify a path
to a tarball (e.g., `/tmp/MySQL-python-1.2.2.tar.gz`) instead.
Now, still as the hue user, we instruct Hue to create the necessary database tables:
----
$ /usr/share/hue/build/env/bin/desktop syncdb --noinput
$ /usr/share/hue/build/env/bin/desktop migrate
----
Now you are all set up and can start the Hue server as normal.
IMPORTANT: The process above will not migrate your existing data to MySQL.
To migrate, you can use `/usr/share/hue/build/env/bin/desktop dumpdata`
(as a first step in the above instructions, before reconfiguring the settings)
to dump the existing database to a text file, and load it back with the
`/usr/share/hue/build/env/bin/desktop loaddata` command (after
updating the configuration and creating the tables).
== Hue Applications
This section includes documentation specific to built-in Hue applications.
=== Beeswax, the Hive UI
==== Introduction
Beeswax is an application that lives within Hue and helps you use Hive to query your data.
==== Installation
Beeswax should already be installed as part of Hue.
==== Hive Configuration
Beeswax, the Hive interface in Hue, includes Hive 0.5. You do
not need an existing Hive installation.
Your Hive data is stored in HDFS, normally under under `/user/hive/warehouse`
(or any path you specify as `hive.metastore.warehouse.dir` in your
`hive-site.xml`). Please ensure that this location exists, and is writable by
the users whom you expect to be creating tables. `/tmp` (on the local file
system) should be world-writable, as Hive makes extensive use of it.
==== Configuration
===== No Existing Hive Installation
Please first familiarize yourself with the configuration options in
`hive-site.xml` (see
http://wiki.apache.org/hadoop/Hive/AdminManual/Configuration).
Having a `hive-site.xml` is optional but often useful, particularly on setting
up a http://wiki.apache.org/hadoop/Hive/AdminManual/MetastoreAdmin[metastore].
You may store the `hive-site.xml` in `/etc/hue/conf`, or instruct
Beeswax to locate it using the `hive_conf_dir` configuration variable. See
`/etc/hue/conf/hue-beeswax.ini`.
===== Existing Hive Installation
In `/etc/hue/conf/hue-beeswax.ini`, modify `hive_conf_dir` to point to the directory containing `hive-site.xml`.
[[usage]]
== Conclusion: Using Hue
After installation, you use Hue by simply navigating to `http://myserver:8088/`.
You'll be greeted with a login screen:
image:images/login.png[]
Launch applications on the top-right.
image:images/open-apps.png[,width=50%]
The Help application, visible in the screenshot above, guides
users through the various installed applications.
=== Supported Browsers
Hue is primarily tested on Firefox 3.5 and Firefox 3.6, on Windows, Mac, and Linux.
Google Chrome and Safari work as well.
=== Feedback
Hue is currently at version 0.9. We're excited
to receive your feedback and criticisms.
The best way to send feedback is to join our
https://groups.google.com/a/cloudera.org/group/hue-user[mailing list], and send us e-mail,
at mailto:hue-user@cloudera.org[hue-user@cloudera.org].
=== Reporting Bugs
If you find that something doesn't work, it'll often be helpful to include logs
from your server. These are available at the +/logs+ URL on Hue's webserver
(not part of the graphical Hue UI). Please download the logs as a zip (or cut
and paste the ones that look relevant) and send those with your bug reports.
image:images/logs.png[]