README.rst 6.1 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222
  1. Welcome to the repository for Hue
  2. =================================
  3. .. note::
  4. This is the development-oriented readme. If you want to write notes for
  5. end users, please put them in ``dist/README``.
  6. Hue is both a web UI for Hadoop and a framework to create interactive web
  7. applications. It features a FileBrowser for accessing HDFS, JobSub and
  8. JobBrowser applications for submitting and viewing MapReduce jobs, a Beeswax
  9. application for interacting with Hive. On top of that, the web frontend
  10. is mostly built from declarative widgets that require no JavaScript and are
  11. easy to learn.
  12. More documentation is available at http://cloudera.github.com/hue/.
  13. File Layout
  14. ===========
  15. The "core" stuff is in ``desktop/core/``, whereas installable apps live in
  16. ``apps/``. Please place third-party dependencies in the app's ext-py/
  17. directory.
  18. The typical directory structure for inside an application includes:
  19. src/
  20. for Python code
  21. conf/
  22. for configuration (``.ini``) files to be installed
  23. static/
  24. for static HTML and js resources
  25. templates/
  26. for data to be put through a template engine
  27. docs/
  28. for helpful notes
  29. The python code is structured simply as
  30. ``module/package.py``,
  31. where module may be "filebrowser" or "jobsub". Because it is unlikely that
  32. there are going to be huge conflicts, we're going without a deep nested
  33. hierarchy.
  34. URL Layout
  35. ==========
  36. ``core/src/desktop/urls.py`` contains the current layout for top-level URLs.
  37. For the URLs within your application, you should make your own ``urls.py``
  38. which will be automatically rooted at ``/yourappname/`` in the global
  39. namespace. See ``apps/about/src/about/urls.py`` for an example.
  40. Development Prerequisites
  41. ===========================
  42. You'll need these library development packages and tools installed on
  43. your system:
  44. Ubuntu:
  45. * ant
  46. * gcc
  47. * g++
  48. * libkrb5-dev
  49. * libmysqlclient-dev
  50. * libssl-dev
  51. * libsasl2-dev
  52. * libsasl2-modules-gssapi-mit
  53. * libsqlite3-dev
  54. * libtidy-0.99-0 (for unit tests only)
  55. * libxml2-dev
  56. * libxslt-dev
  57. * mvn (from ``maven2`` package or tarball)
  58. * openldap-dev / libldap2-dev
  59. * python-dev
  60. * python-simplejson
  61. * python-setuptools
  62. CentOS:
  63. * ant
  64. * asciidoc
  65. * cyrus-sasl-devel
  66. * cyrus-sasl-gssapi
  67. * gcc
  68. * gcc-c++
  69. * krb5-devel
  70. * libtidy (for unit tests only)
  71. * libxml2-devel
  72. * libxslt-devel
  73. * mvn (from ``maven2`` package or tarball)
  74. * mysql
  75. * mysql-devel
  76. * openldap-devel
  77. * python-devel
  78. * python-simplejson
  79. * sqlite-devel
  80. MacOS (mac port):
  81. * liblxml
  82. * libxml2
  83. * libxslt
  84. * mysql5-devel
  85. * simplejson (easy_install)
  86. * sqlite3
  87. Getting Started
  88. ===============
  89. To build and get the core server running::
  90. $ git clone http://github.com/cloudera/hue.git
  91. $ cd hue
  92. $ make apps
  93. $ build/env/bin/hue runserver
  94. If using the Beeswax application, start the daemon::
  95. $ build/env/bin/hue beeswax_server
  96. Now Hue should be running on http://localhost:8000.
  97. The configuration in development mode is ``desktop/conf/pseudo-distributed.ini``.
  98. Note: to start all the servers in one command (but lose the automatic reloading after source modification)::
  99. $ build/env/bin/supervisor
  100. To run the tests::
  101. $ build/env/bin/hue test all
  102. $ build/env/bin/hue test specific filebrowser
  103. Django Conventions
  104. ==================
  105. If you need to name your urls
  106. (http://docs.djangoproject.com/en/dev/topics/http/urls/#naming-url-patterns)
  107. because there's ambiguity in the view, be sure to prefix the name
  108. with the application name. The url name namespace is global. So
  109. ``jobsub.list`` is fine, but ``list`` is not.
  110. Hue is using Django 1.2, which supports the notion of URL namespaces:
  111. http://docs.djangoproject.com/en/dev/topics/http/urls/#url-namespaces.
  112. We have yet to move over our URLs to this construct. Brownie points for the
  113. developer who takes this on.
  114. Using and Installing Thrift
  115. ===========================
  116. Right now, we check in the generated thrift code.
  117. To generate the code, you'll need the thrift binary version 0.7.0.
  118. Please download from http://thrift.apache.org/.
  119. The modules using ``Thrift`` have some helper scripts like ``regenerate_thrift.sh``
  120. for regenerating the code from the interfaces.
  121. Profiling Hue Apps
  122. ==================
  123. Hue has a profiling system built in, which can be used to analyze server-side
  124. performance of applications. To enable profiling::
  125. $ build/env/bin/hue runprofileserver
  126. Then, access the page that you want to profile. This will create files like
  127. /tmp/useradmin.users.000072ms.2011-02-21T13:03:39.745851.prof. The format for
  128. the file names is /tmp/<app_module>.<page_url>.<time_taken>.<timestamp>.prof.
  129. Hue uses the hotshot profiling library for instrumentation. The documentation
  130. for this library is located at: http://docs.python.org/library/hotshot.html.
  131. You can use kcachegrind to view the profiled data graphically::
  132. $ hotshot2calltree /tmp/xyz.prof > /tmp/xyz.trace
  133. $ kcachegrind /tmp/xyz.trace
  134. More generally, you can programmatically inspect a trace::
  135. #!/usr/bin/python
  136. import hotshot.stats
  137. import sys
  138. stats = hotshot.stats.load(sys.argv[1])
  139. stats.sort_stats('cumulative', 'calls')
  140. stats.print_stats(100)
  141. This script takes in a .prof file, and orders function calls by the cumulative
  142. time spent in that function, followed by the number of times the function was
  143. called, and then prints out the top 100 time-wasters. For information on the
  144. other stats available, take a look at this website:
  145. http://docs.python.org/library/profile.html#pstats.Stats
  146. Internationalization
  147. ====================
  148. How to update all the messages and compile them::
  149. $ make locales
  150. How to update and compile the messages of one app::
  151. $ cd apps/beeswax
  152. $ make compile-locale
  153. How to create a new locale for an app::
  154. $ cd $APP_ROOT/src/$APP_NAME/locale
  155. $ $HUE_ROOT/build/env/bin/pybabel init -D django -i en_US.pot -d . -l fr
  156. License
  157. =======
  158. Apache License, Version 2.0
  159. http://www.apache.org/licenses/LICENSE-2.0