manual.txt 21 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624
  1. Hue Manual
  2. ==========
  3. == Introduction
  4. Hue is a graphical user interface to operate and develop for Hadoop.
  5. Hue "applications" are collected into a desktop
  6. environment and delivered as a Web application, requiring no
  7. additional installation for individual users'.
  8. This document will you install and configure Hue. There
  9. is a companion SDK document for information
  10. on developing new Hue applications.
  11. Hue currently requires Cloudera's Distribution of Hadoop,
  12. version CDH2 or CDH3.
  13. .Note:
  14. * All commands that need to be run as +root+ have a +#+ command prompt.
  15. * All commands that do not require +root+ have a +$+ command prompt.
  16. == Installation Instructions
  17. These instructions will guide you through installing Hue
  18. on a multi-node cluster. You will need to update
  19. some Hadoop configuration as well as install Hue.
  20. IMPORTANT: You'll need to install the Hue Plug-ins
  21. on _every_ machine that's running Hadoop daemons.
  22. === Install Hadoop First!
  23. Hue requires an installation of the Cloudera Distribution
  24. of Hadoop (version 2 or above). See link:http://archive.cloudera.com/
  25. for instructions.
  26. To use Hue, you must be running Cloudera's Distribution for Hadoop
  27. with a version number of at least `0.20.1+133`. If you are not running at least this
  28. version of Hadoop, please upgrade your cluster before proceeding.
  29. === Install Hue
  30. Hue consists of a web service that runs on a special node in your cluster.
  31. You should pick one node to run Hue. From this point on, we'll refer to
  32. that node as the "Hue Server". This should be one of the nodes within your
  33. cluster for optimal performance, though it can be a remote node as long as there are no
  34. overly restrictive firewalls.
  35. For small clusters of less than 10 nodes, it is fine to choose your existing master node
  36. as the Hue Server.
  37. You install Hue, begin by downloading the link:http://archive.cloudera.com/cdh/3/hue-0.9.tar.gz[tarball].
  38. ==== Installing Dependencies
  39. Hue employs some Python modules which use native code and depend
  40. on certain development libraries being on your system. To install from the
  41. tarball you must have the following installed:
  42. .Required Dependencies
  43. [grid="rows"]
  44. ``~
  45. Redhat,Debian
  46. ~~~~~~~~~~
  47. libxslt-devel,libxslt-dev
  48. libxml2-devel,libxml2-dev
  49. sqlite-devel,libsqlite3-dev
  50. python-devel,python2.4-dev(orpython2.5-dev)
  51. gcc,gcc
  52. python-setuptools,python-setuptools
  53. ~~~~~~~~~~
  54. Additionally, you must have Cloudera's Distribution for Hadoop installed
  55. and available. Specifically, your Hadoop version must be at least 0.20.1+133.
  56. If you have a previous release, upgrade Hadoop before continuing with Hue.
  57. ==== Build
  58. With `$HADOOP_HOME` and `$PREFIX` configured to the path of your Hadoop
  59. installation and the path where you'd like to install Hue,
  60. respectively, run:
  61. ----
  62. $ HADOOP_HOME=/path/to/hadoop-0.20.1+152 PREFIX=/path/to/install/into make install
  63. ----
  64. You may install Hue anywhere on your system - it does not need root access.
  65. We recommend that you create a new user for Hue and either install in that
  66. user's home directory or in a directory within /usr/local.
  67. ==== Install Hadoop Plugins
  68. In order to communicate with Hadoop, Hue requires that you install and configure
  69. a plugin JAR. This jar is desktop/libs/hadoop/java-lib/hue-plugins-0.9.jar.
  70. Symlink this jar into your Hadoop lib directory (/usr/lib/hadoop-0.20/lib if you've installed
  71. CDH via a Debian or RPM package):
  72. ----
  73. $ cd /usr/lib/hadoop/lib
  74. $ ln -s /usr/share/hue/desktop/libs/hadoop/java-lib/hue*jar .
  75. # Restart Hadoop
  76. ----
  77. NOTE: On a multi-node cluster, you will need to install the plug-ins on every
  78. node. You do not need to install the entirety of Hue everywhere,
  79. but the plug-in jars need to be available on every machine.
  80. ==== Restart Hadoop
  81. Once you have made these changes in your Hadoop configuration, go ahead and
  82. restart your Hadoop daemons.
  83. ==== Running Hue
  84. To start Hue, you'll use `build/env/bin/supervisor`. This will start
  85. several subprocesses, corresponding to different bits of Hue.
  86. ==== FAQ about Tarball Installation
  87. .I moved my Hue installation from one directory to another and it no longer
  88. functions correctly.
  89. Due to the use of absolute paths by some python packages, you'll have to run a series of commands
  90. if you relocate your Hue installation. From within the new location, run:
  91. ----
  92. $ make apps
  93. ----
  94. This should solve the problem.
  95. .Why does "make install" compile all of these other pieces of software?
  96. In order to ensure that Hue is stable on a variety of distributions and architectures,
  97. it installs a Python "virtual environment" which includes its dependencies. This ensures that
  98. the software can depend against specific versions of various Python libraries and not have to worry
  99. about what might or might not be installed already on your particular system.
  100. === Configuring Hadoop for Hue
  101. Hue requires that you install and configure some plugins in your
  102. Hadoop installation.
  103. When you installed the `hue-plugins` package above, your package manager
  104. automatically added the required plugin jar
  105. to your Hadoop installation's `lib` directory, making them available to Hadoop.
  106. In order to enable the plugins, you'll need to make some small additions to your
  107. configuration.
  108. These configuration changes should be made on each node in your cluster by editing the files
  109. in `/etc/hadoop-0.20/conf/`
  110. ==== `hdfs-site.xml`
  111. You'll want to add the following configuration
  112. options to hdfs-site.xml.
  113. ----
  114. <property>
  115. <name>dfs.namenode.plugins</name>
  116. <value>org.apache.hadoop.thriftfs.NamenodePlugin</value>
  117. <description>Comma-separated list of namenode plug-ins to be activated.
  118. </description>
  119. </property>
  120. <property>
  121. <name>dfs.datanode.plugins</name>
  122. <value>org.apache.hadoop.thriftfs.DatanodePlugin</value>
  123. <description>Comma-separated list of datanode plug-ins to be activated.
  124. </description>
  125. </property>
  126. <property>
  127. <name>dfs.thrift.address</name>
  128. <value>0.0.0.0:9090</value>
  129. </property>
  130. -----
  131. ==== `mapred-site.xml`
  132. Add the following to mapred-site.xml:
  133. ----
  134. <property>
  135. <name>jobtracker.thrift.address</name>
  136. <value>0.0.0.0:9290</value>
  137. </property>
  138. <property>
  139. <name>mapred.jobtracker.plugins</name>
  140. <value>org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin</value>
  141. <description>Comma-separated list of jobtracker plug-ins to be activated.
  142. </description>
  143. </property>
  144. ----
  145. ==== `hadoop-metrics.properties`
  146. To enable full monitoring in the Health application, the metrics
  147. contexts must not be NullContext. You might configure `hadoop-metrics.properties`
  148. like so:
  149. ----
  150. # Exposes /metrics URL endpoint for metrics information.
  151. dfs.class=org.apache.hadoop.metrics.spi.NoEmitMetricsContext
  152. mapred.class=org.apache.hadoop.metrics.spi.NoEmitMetricsContext
  153. jvm.class=org.apache.hadoop.metrics.spi.NoEmitMetricsContext
  154. rpc.class=org.apache.hadoop.metrics.spi.NoEmitMetricsContext
  155. ----
  156. === Further Hadoop Configuration and Caveats
  157. ==== `HADOOP_CLASSPATH` Caveat
  158. If you are setting `$HADOOP_CLASSPATH` in your `hadoop-env.sh`, be sure
  159. to set it in such a way that user-specified options are preserved. For example,
  160. ----
  161. # Good
  162. HADOOP_CLASSPATH=<your_additions>:$HADOOP_CLASSPATH
  163. # Bad
  164. # HADOOP_CLASSPATH=<your_additions>
  165. ----
  166. This will enable certain portions of Hue to add to
  167. Hadoop's classpath using the environment variable.
  168. ==== `hadoop.tmp.dir`
  169. If your users are likely to be submitting jobs both using Hue
  170. and from the same machine, they will be doing so as the `hue`
  171. user if they're using Hue and as the their own username
  172. if they're using the command line. This yields to some contention
  173. on the directory specified by `hadoop.tmp.dir`, which defaults
  174. to `/tmp/hadoop-${user.name}`.
  175. Specifically, `hadoop.tmp.dir` is used to unpack jars in `bin/hadoop jar`.
  176. One work around to this is
  177. to set `hadoop.tmp.dir` to `/tmp/hadoop-${user.name}-${hue.suffix}`:
  178. ----
  179. <property>
  180. <name>hadoop.tmp.dir</name>
  181. <value>/tmp/hadoop-${user.name}${hue.suffix}</value>
  182. </property>
  183. ----
  184. Unfortunately, when the variable is unset, you'll end up
  185. with directories named `/tmp/hadoop-user_name-${hue.suffix}` in
  186. `/tmp`. The job submission daemon, however, will
  187. === Restart Your Hadoop Cluster
  188. At this point you should restart all of the daemons in your cluster so that the plugins can be loaded.
  189. You can confirm that the plugins are running correctly by tailing the daemon logs:
  190. ----
  191. $ tail --lines=500 /var/log/hadoop-0.20/hadoop*namenode*.log | grep ThriftPlugin
  192. 2009-09-28 16:30:44,337 INFO org.apache.hadoop.thriftfs.ThriftPluginServer: Starting Thrift server
  193. 2009-09-28 16:30:44,419 INFO org.apache.hadoop.thriftfs.ThriftPluginServer: Thrift server listening on 0.0.0.0:9090
  194. ----
  195. [TIP]
  196. .Configuring Your Firewall for Hue
  197. ============================================================
  198. Hue currently requires that the machines within your cluster can speak to each other freely over TCP.
  199. The machines outside your cluster only need to be able to open TCP port 8088 on the Hue Server to interact with the system.
  200. ============================================================
  201. == Configuring Hue
  202. Hue ships with a default configuration that will work for
  203. pseudo-distributed clusters. If you are running on a real cluster, you'll need
  204. to make a few small changes to its configuration file. Here we go through the
  205. key configuration options.
  206. Edit `/etc/hue/hue.ini` in your favorite editor.
  207. [TIP]
  208. .Full Listing of Configuration Options
  209. ============================================================
  210. In order to see a full listing of the available configuration options, you can run:
  211. ----
  212. /usr/share/hue/build/env/bin/desktop config_help | less
  213. ----
  214. This will outline the various sections and options in the configuration, as
  215. well as provide help and information on the default values.
  216. ============================================================
  217. [TIP]
  218. .View Current Configuration Options
  219. ============================================================
  220. You can also view the current configuration from within Hue, at:
  221. ----
  222. http://<hue>/dump_config
  223. ----
  224. ============================================================
  225. [TIP]
  226. .Using Multiple Files to Store Your Configuration
  227. ============================================================
  228. Hue will load and merge all of the files with extension `.ini`
  229. located in the `/etc/hue/conf/` directory. Files that are alphabetically later
  230. will take precedence.
  231. ============================================================
  232. === Webserver Configuration
  233. Hue uses the CherryPy web server. You can change the IP address
  234. and port the web server listens on, which is port 8088 on all configured IP
  235. addresses. (Use `http_host` and `http_port`.)
  236. === Authentication
  237. By default, the first user who logins to Hue may choose any
  238. username and password, and becomes an administrator automatically. This
  239. user may create other user and administrator accounts. User information is
  240. stored in the Django backend, in the Django database.
  241. The authentication system is pluggable; developers should
  242. refer to the SDK documentation, if interested.
  243. === Configuring for SSL
  244. You can configure Hue to serve over HTTPS. To do so, you'll need
  245. to install "pyOpenSSL" within Desktop's context and configure your keys.
  246. To install `pyOpenSSL`, from the root of your Desktop installation
  247. (`/usr/share/hue` if you installed from packages, type)
  248. execute:
  249. ----
  250. $ ./build/env/bin/easy_install pyOpenSSL
  251. ----
  252. Then, configure Hue to use your private
  253. key by putting the following inside your
  254. `/etc/hue/hue.ini`:
  255. ----
  256. ssl_certificate=/path/to/certificate
  257. ssl_private_key=/path/to/key
  258. ----
  259. Ideally, you would have an appropriate key signed by a Certificate Authority.
  260. If you're just testing, you can create a self-signed key using the `openssl`
  261. command, that may be installed on your system:
  262. ----
  263. # Create a key
  264. $ openssl genrsa 1024 > host.key
  265. # Create a self-signed certificate
  266. $ openssl req -new -x509 -nodes -sha1 -key host.key > host.cert
  267. ----
  268. [NOTE]
  269. .Self-signed Certificates and File Uploads
  270. ============================================================
  271. To upload files using the File Browser over HTTPS requires
  272. using a proper SSL Certificate. Self-signed certificates
  273. are known to not work.
  274. ============================================================
  275. === Pointing Hue to Your Master Nodes
  276. If your Hadoop cluster is made up of multiple nodes, you should configure
  277. Hue to point to the external hostnames of your Namenode and
  278. JobTracker. To do so, simply change the `namenode_host` and `jobtracker_host`
  279. lines in the configuration file. The inline comments in the existing file will
  280. guide you.
  281. == Starting Hue
  282. Once your cluster is up and running with the plugins enabled, you can start Hue.
  283. On your Hue Server, run:
  284. ----
  285. # build/env/bin/supervisor
  286. ----
  287. Congratulations! Your Hue installation is now up and running!
  288. == Administering Hue
  289. Now that you've installed and started Hue, you can feel free to skip ahead
  290. to <<usage,Using Hue>>. Administrators may want to refer to this section
  291. for more details about managing and operating a Hue installation.
  292. === Hue Processes
  293. ==== Process Hierarchy
  294. Hue runs several processes under the hood, all managed by a script
  295. called the `supervisor`. The supervisor is a watchdog process -- its only purpose
  296. is to spawn and monitor other processes.
  297. A stock Hue installation will spawn and monitor the following processes:
  298. * `runcpserver` - a web server based on CherryPy that provides the core web functionality of Hue
  299. * `jobsubd` - a daemon which handles submission of jobs to Hadoop
  300. If you have installed other applications into your Hue instance, you may see other daemons running under
  301. the supervisor as well.
  302. We can see the supervised processes running in the output of `ps`:
  303. ------------------
  304. [todd@monster01 ~]$ ps -f -u hue
  305. UID PID PPID C STIME TTY TIME CMD
  306. hue 7899 1 0 12:14 ? 00:00:00 /usr/share/hue/build/build/env/bin/python2.4 \
  307. /usr/share/hue/desktop/build/env/bin/supervisor -p /var/run/hue/desktop/s
  308. hue 7903 7899 1 12:14 ? 00:02:59 /usr/share/hue/desktop/build/env/bin/python2.4 \
  309. /usr/share/hue/desktop/build/env/bin/desktop runcpserver
  310. hue 7906 7899 0 12:14 ? 00:00:00 /usr/share/hue/desktop/build/env/bin/python2.4 \
  311. /usr/share/hue/desktop/build/env/bin/desktop jobsubd
  312. hue 7907 7899 0 12:14 ? 00:00:12 /usr/share/hue/desktop/build/env/bin/python2.4 \
  313. /usr/share/hue/desktop/build/env/bin/desktop run_healthd
  314. ------------------
  315. Note that the supervisor will automatically restart these processes should they fail for any reason.
  316. If the processes fail repeatedly within a small time window, the supervisor itself will shut down.
  317. ==== Managing Hue Processes
  318. Hue RPMs and Debian packages ship with an `init.d` script to manage the Hue processes.
  319. You can start and stop the Hue Supervisor using this init script, for example:
  320. ----
  321. # /etc/init.d/hue stop
  322. ----
  323. If for some reason the init scripts are unable to stop the process, you can kill the daemon manually by locating and
  324. killing the `supervisor` process as described above.
  325. === Hue Logging
  326. The Hue logs are found in `/path/to/hue/logs` if
  327. you've installed via a tarball. Inside the log directory you will find:
  328. * `access.log`, which contains a log for all requests against the Hue web server.
  329. * A `.log` file containing the logs for each of the processes described above.
  330. * A `.out` file containing the stdout and stderr for each of the processes described above.
  331. If users on your cluster experience issues, you can often find error messages in these log files.
  332. If you are unable to start Hue from the init script, the
  333. `supervisor.log` log file can often contain clues.
  334. ==== Viewing recent log messages through your browser
  335. In addition to logging `INFO` level messages to the log directory, the Hue web server keeps a small buffer
  336. of log messages at all levels in memory. You can view these logs by visiting `http://myserver:8088/logs`.
  337. The `DEBUG` level messages present here can sometimes be helpful in troubleshooting issues.
  338. === The Hue Database
  339. Hue requires a SQL database to store small amounts of data, including user account information
  340. as well as history of job submissions and Hive queries. By default, Hue is configured to use the
  341. embedded database SQLite for this purpose, and should require no configuration or management by the
  342. administrator.
  343. ==== Inspecting the Hue Database
  344. The default SQLite database used by Hue is located in `/usr/share/hue/desktop/desktop.db`.
  345. You can inspect this database from the command line using the `sqlite3` program. For example:
  346. ---------
  347. # sqlite3 /usr/share/hue/desktop/desktop.db
  348. SQLite version 3.3.6
  349. Enter ".help" for instructions
  350. sqlite> select username from auth_user;
  351. admin
  352. test
  353. sample
  354. sqlite>
  355. ----------
  356. We strongly advise you not to make any modifications to the database directly using SQLite, though this trick
  357. may be handy for management or troubleshooting.
  358. ==== Backing up the Hue Database
  359. To back up the Hue Database, you can simply copy the `desktop.db` file to another node. We recommend that
  360. you back it up on a regular schedule, and also that you back it up before any upgrade to a new version of
  361. Hue.
  362. ==== Configuring Hue to access another Database
  363. Although SQLite is the default database type, some advanced users may prefer to have Hue access an
  364. alternate database type. Please note that, if you elect to configure Hue to use an external database,
  365. upgrades may require more manual steps in the future.
  366. Here we provide instructions for MySQL, though Hue may also be made to work
  367. with other common databases including PostgreSQL, Oracle, etc.
  368. ===== Configuring Hue to store data in MySQL
  369. First, you must create a new database in MySQL and grant privileges to a Hue user to manage
  370. this database.
  371. ----
  372. mysql> create database hue;
  373. Query OK, 1 row affected (0.01 sec)
  374. mysql> grant all on hue.* to 'hue'@'localhost' identified by 'secretpassword';
  375. Query OK, 0 rows affected (0.00 sec)
  376. ----
  377. Next, shut down Hue if it is running, and edit `/etc/hue/hue.ini`. Directly below the
  378. `[desktop]` line, add the following:
  379. ----
  380. [[database]]
  381. host=localhost
  382. port=3306
  383. engine=mysql
  384. user=hue
  385. password=secretpassword
  386. name=hue
  387. ----
  388. Next we need to install the python drivers for MySQL into Hue's environment:
  389. ----
  390. # su - hue -s /bin/bash
  391. $ /usr/share/hue/build/env/bin/easy_install MySQL-python
  392. ----
  393. `easy_install` will access the Internet to find MySQL-python.
  394. If you don't have network access, you may specify a path
  395. to a tarball (e.g., `/tmp/MySQL-python-1.2.2.tar.gz`) instead.
  396. Now, still as the hue user, we instruct Hue to create the necessary database tables:
  397. ----
  398. $ /usr/share/hue/build/env/bin/desktop syncdb --noinput
  399. $ /usr/share/hue/build/env/bin/desktop migrate
  400. ----
  401. Now you are all set up and can start the Hue server as normal.
  402. IMPORTANT: The process above will not migrate your existing data to MySQL.
  403. To migrate, you can use `/usr/share/hue/build/env/bin/desktop dumpdata`
  404. (as a first step in the above instructions, before reconfiguring the settings)
  405. to dump the existing database to a text file, and load it back with the
  406. `/usr/share/hue/build/env/bin/desktop loaddata` command (after
  407. updating the configuration and creating the tables).
  408. == Hue Applications
  409. This section includes documentation specific to built-in Hue applications.
  410. === Beeswax, the Hive UI
  411. ==== Introduction
  412. Beeswax is an application that lives within Hue and helps you use Hive to query your data.
  413. ==== Installation
  414. Beeswax should already be installed as part of Hue.
  415. ==== Hive Configuration
  416. Beeswax, the Hive interface in Hue, includes Hive 0.5. You do
  417. not need an existing Hive installation.
  418. Your Hive data is stored in HDFS, normally under under `/user/hive/warehouse`
  419. (or any path you specify as `hive.metastore.warehouse.dir` in your
  420. `hive-site.xml`). Please ensure that this location exists, and is writable by
  421. the users whom you expect to be creating tables. `/tmp` (on the local file
  422. system) should be world-writable, as Hive makes extensive use of it.
  423. ==== Configuration
  424. ===== No Existing Hive Installation
  425. Please first familiarize yourself with the configuration options in
  426. `hive-site.xml` (see
  427. http://wiki.apache.org/hadoop/Hive/AdminManual/Configuration).
  428. Having a `hive-site.xml` is optional but often useful, particularly on setting
  429. up a http://wiki.apache.org/hadoop/Hive/AdminManual/MetastoreAdmin[metastore].
  430. You may store the `hive-site.xml` in `/etc/hue/conf`, or instruct
  431. Beeswax to locate it using the `hive_conf_dir` configuration variable. See
  432. `/etc/hue/conf/hue-beeswax.ini`.
  433. ===== Existing Hive Installation
  434. In `/etc/hue/conf/hue-beeswax.ini`, modify `hive_conf_dir` to point to the directory containing `hive-site.xml`.
  435. [[usage]]
  436. == Conclusion: Using Hue
  437. After installation, you use Hue by simply navigating to `http://myserver:8088/`.
  438. You'll be greeted with a login screen:
  439. image:images/login.png[]
  440. Launch applications on the top-right.
  441. image:images/open-apps.png[,width=50%]
  442. The Help application, visible in the screenshot above, guides
  443. users through the various installed applications.
  444. === Supported Browsers
  445. Hue is primarily tested on Firefox 3.5 and Firefox 3.6, on Windows, Mac, and Linux.
  446. Google Chrome and Safari work as well.
  447. === Feedback
  448. Hue is currently at version 0.9. We're excited
  449. to receive your feedback and criticisms.
  450. The best way to send feedback is to join our
  451. https://groups.google.com/a/cloudera.org/group/hue-user[mailing list], and send us e-mail,
  452. at mailto:hue-user@cloudera.org[hue-user@cloudera.org].
  453. === Reporting Bugs
  454. If you find that something doesn't work, it'll often be helpful to include logs
  455. from your server. These are available at the +/logs+ URL on Hue's webserver
  456. (not part of the graphical Hue UI). Please download the logs as a zip (or cut
  457. and paste the ones that look relevant) and send those with your bug reports.
  458. image:images/logs.png[]