README.rst 5.9 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208
  1. Welcome to the repository for Hue
  2. =================================
  3. .. note::
  4. This is the development-oriented readme. If you want to write notes for
  5. end users, please put them in ``dist/README``.
  6. Hue is both a web UI for Hadoop and a framework to create interactive web
  7. applications. It features a FileBrowser for accessing HDFS, JobSub and
  8. JobBrowser applications for submitting and viewing MapReduce jobs, a Beeswax
  9. application for interacting with Hive. On top of that, the web frontend
  10. is mostly built from declarative widgets that require no JavaScript and are
  11. easy to learn.
  12. File Layout
  13. ===========
  14. The "core" stuff is in ``desktop/core/``, whereas installable apps live in
  15. ``apps/``. Please place third-party dependencies in the app's ext-py/
  16. directory.
  17. The typical directory structure for inside an application includes:
  18. src/
  19. for Python code
  20. conf/
  21. for configuration (``.ini``) files to be installed
  22. static/
  23. for static HTML and js resources
  24. templates/
  25. for data to be put through a template engine
  26. docs/
  27. for helpful notes
  28. The python code is structured simply as
  29. ``module/package.py``,
  30. where module may be "filebrowser" or "jobsub". Because it is unlikely that
  31. there are going to be huge conflicts, we're going without a deep nested
  32. hierarchy.
  33. URL Layout
  34. ==========
  35. ``core/src/desktop/urls.py`` contains the current layout for top-level URLs.
  36. For the URLs within your application, you should make your own ``urls.py``
  37. which will be automatically rooted at ``/yourappname/`` in the global
  38. namespace. See ``apps/hello/src/hello/urls.py`` for an example.
  39. Development Prerequisites
  40. ===========================
  41. You'll need these library development packages and tools installed on
  42. your system:
  43. Ubuntu:
  44. * ant
  45. * gcc
  46. * g++
  47. * libkrb5-dev
  48. * libmysqlclient-dev
  49. * libssl-dev
  50. * libsasl2-dev
  51. * libsasl2-modules-gssapi-mit
  52. * libsqlite3-dev
  53. * libtidy-0.99-0 (for unit tests only)
  54. * libxml2-dev
  55. * libxslt-dev
  56. * mvn (from ``maven2`` package or tarball)
  57. * openldap-dev / libldap2-dev
  58. * python-dev
  59. * python-simplejson
  60. * python-setuptools
  61. CentOS:
  62. * ant
  63. * asciidoc
  64. * cyrus-sasl-devel
  65. * cyrus-sasl-gssapi
  66. * gcc
  67. * gcc-c++
  68. * krb5-devel
  69. * libtidy (for unit tests only)
  70. * libxml2-devel
  71. * libxslt-devel
  72. * mvn (from ``maven2`` package or tarball)
  73. * mysql
  74. * mysql-devel
  75. * openldap-devel
  76. * python-devel
  77. * python-simplejson
  78. * sqlite-devel
  79. MacOS (mac port):
  80. * liblxml
  81. * libxml2
  82. * libxslt
  83. * mysql5-devel
  84. * simplejson (easy_install)
  85. * sqlite3
  86. Getting Started
  87. ===============
  88. To build and get the core server running::
  89. $ export HADOOP_HOME=<path-to-hadoop-home>
  90. $ git clone http://github.com/cloudera/hue.git
  91. $ cd hue
  92. $ make apps
  93. $ build/env/bin/hue runserver_plus
  94. To start the helper daemons::
  95. $ build/env/bin/hue beeswax_server
  96. $ build/env/bin/hue jobsubd
  97. Now Hue should be running on http://localhost:8000.
  98. FAQ
  99. ===
  100. 1: What does "Exception: no app!" mean?
  101. Your template has an error in it. Check for messages from the server that
  102. look like::
  103. INFO:root:Processing exception: Unclosed tag 'if'. Looking for one of: else, endif
  104. 2: What do I do if I get "There was an error launching ..."?
  105. Turn on debugging by issuing ``dbug.cookie()`` in a Firebug console.
  106. Django Conventions
  107. ==================
  108. If you need to name your urls
  109. (http://docs.djangoproject.com/en/dev/topics/http/urls/#naming-url-patterns)
  110. because there's ambiguity in the view, be sure to prefix the name
  111. with the application name. The url name namespace is global. So
  112. ``jobsub.list`` is fine, but ``list`` is not.
  113. Hue is using Django 1.2, which supports the notion of URL namespaces:
  114. http://docs.djangoproject.com/en/dev/topics/http/urls/#url-namespaces.
  115. We have yet to move over our URLs to this construct. Brownie points for the
  116. developer who takes this on.
  117. Using and Installing Thrift
  118. ===========================
  119. Right now, we check in the generated thrift code.
  120. To generate the code, you'll need the thrift binary version 0.7.0.
  121. Please download from http://thrift.apache.org/.
  122. When preparing ``.thrift`` files, you can use she-bangs to generate
  123. the python bindings like so::
  124. #!/usr/bin/env thrift -r --gen py:new_style -o ../../../
  125. Profiling Hue Apps
  126. ==================
  127. Hue has a profiling system built in, which can be used to analyze server-side
  128. performance of applications. To enable profiling::
  129. $ build/env/bin/hue runprofileserver
  130. Then, access the page that you want to profile. This will create files like
  131. /tmp/useradmin.users.000072ms.2011-02-21T13:03:39.745851.prof. The format for
  132. the file names is /tmp/<app_module>.<page_url>.<time_taken>.<timestamp>.prof.
  133. Hue uses the hotshot profiling library for instrumentation. The documentation
  134. for this library is located at: http://docs.python.org/library/hotshot.html.
  135. You can use kcachegrind to view the profiled data graphically::
  136. $ hotshot2calltree /tmp/xyz.prof > /tmp/xyz.trace
  137. $ kcachegrind /tmp/xyz.trace
  138. More generally, you can programmatically inspect a trace::
  139. #!/usr/bin/python
  140. import hotshot.stats
  141. import sys
  142. stats = hotshot.stats.load(sys.argv[1])
  143. stats.sort_stats('cumulative', 'calls')
  144. stats.print_stats(100)
  145. This script takes in a .prof file, and orders function calls by the cumulative
  146. time spent in that function, followed by the number of times the function was
  147. called, and then prints out the top 100 time-wasters. For information on the
  148. other stats available, take a look at this website:
  149. http://docs.python.org/library/profile.html#pstats.Stats
  150. Internationalization
  151. ====================
  152. How to update the messages:
  153. compile-locale rule in Makefile.sdk
  154. How to create a new locale for an app:
  155. LOCALE_ROOT = $APP_ROOT/src/$APP_NAME/locale
  156. ./build/env/bin/pybabel init -D django -i $LOCALE_ROOT/en_US.pot -d $LOCALE_ROOT -l fr