README.rst 17 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481
  1. Welcome to Livy, the REST Spark Server
  2. ======================================
  3. Livy is an open source REST interface for interacting with a remote Spark Shell
  4. running locally or from inside YARN.
  5. Prerequisites
  6. =============
  7. To build Livy, you will need:
  8. Debian/Ubuntu:
  9. * mvn (from ``maven`` package or maven3 tarball)
  10. * openjdk-7-jdk (or Oracle Java7 jdk)
  11. Redhat/CentOS:
  12. * mvn (from ``maven`` package or maven3 tarball)
  13. * java-1.7.0-openjdk (or Oracle Java7 jdk)
  14. Building Livy
  15. =============
  16. Livy is normally built by the `Hue Build System`_, it can also be built on it's
  17. own (aka without any other Hue dependency) with `Apache Maven`_. To build, run:
  18. .. code:: shell
  19. % cd $HUE_HOME/apps/spark/java
  20. % mvn -DskipTests clean package
  21. .. _Hue Build System: https://github.com/cloudera/hue/#getting-started
  22. .. _Apache Maven: http://maven.apache.org
  23. Running Tests
  24. =============
  25. In order to run the Livy Tests, first follow the instructions in `Building
  26. Livy`_. Then run:
  27. .. code:: shell
  28. % cd $HUE_HOME/apps/spark/java
  29. % mvn test
  30. Spark Example
  31. =============
  32. Now to see it in action by interacting with it in Python with the `Requests`_
  33. library. By default livy runs on port 8998 (which can be changed with the
  34. ``livy_server_port config`` option). We’ll start off with a Spark session that
  35. takes Scala code:
  36. .. code:: python
  37. >>> import json, pprint, requests, textwrap
  38. >>> host = 'http://localhost:8998'
  39. >>> data = {'lang': 'spark'}
  40. >>> r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
  41. >>> r.json()
  42. {u'state': u'starting', u'id': u'89beded3-04eb-4602-9a9e-6d77780f2572', u’kind’: u’spark’}
  43. Once the session has completed starting up, it transitions to the idle state:
  44. .. code:: python
  45. >>> session_url = host + r.headers['location']
  46. >>> r = requests.get(session_url, headers=headers)
  47. >>> r.json()
  48. {u'state': u'idle', u'id': u'89beded3-04eb-4602-9a9e-6d77780f2572', u’kind’: u’spark’}
  49. Now we can execute Scala by passing in a simple JSON command:
  50. .. code:: python
  51. >>> data = {'code': '1 + 1'}
  52. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  53. >>> r.json()
  54. {u'output': None, u'state': u'running', u'id': 0}
  55. If a statement takes longer than a few milliseconds to execute, Livy returns
  56. early and provides a URL that can be polled until it is complete:
  57. .. code:: python
  58. >>> statement_url = host + r.headers['location']
  59. >>> r = requests.get(statement_url, headers=headers)
  60. >>> pprint.pprint(r.json())
  61. [{u'id': 0,
  62. u'output': {u'data': {u'text/plain': u'res0: Int = 2'},
  63. u'execution_count': 0,
  64. u'status': u'ok'},
  65. u'state': u'available'}]
  66. That was a pretty simple example. More interesting is using Spark to estimate
  67. PI. This is from the `Spark Examples`_:
  68. .. code:: python
  69. >>> data = {
  70. ... 'code': textwrap.dedent("""\
  71. ... val NUM_SAMPLES = 100000;
  72. ... val count = sc.parallelize(1 to NUM_SAMPLES).map { i =>
  73. ... val x = Math.random();
  74. ... val y = Math.random();
  75. ... if (x*x + y*y < 1) 1 else 0
  76. ... }.reduce(_ + _);
  77. ... println(\"Pi is roughly \" + 4.0 * count / NUM_SAMPLES)
  78. ... """)
  79. ... }
  80. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  81. >>> pprint.pprint(r.json())
  82. {u'id': 1,
  83. u'output': {u'data': {u'text/plain': u'Pi is roughly 3.14004\nNUM_SAMPLES: Int = 100000\ncount: Int = 78501'},
  84. u'execution_count': 1,
  85. u'status': u'ok'},
  86. u'state': u'available'}
  87. Finally, lets close our session:
  88. .. code:: python
  89. >>> session_url = 'http://localhost:8998/sessions/73e30e74-3bf9-49ce-8dbd-5345fd5128d6'
  90. >>> requests.delete(session_url, headers=headers)
  91. <Response [204]>
  92. .. _Requests: http://docs.python-requests.org/en/latest/
  93. .. _Spark Examples: https://spark.apache.org/examples.html
  94. PySpark Example
  95. ===============
  96. pyspark has the exact same API, just with a different initial command:: python
  97. >>> data = {'lang': 'pyspark'}
  98. >>> r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
  99. >>> r.json()
  100. {u'id': u'73e30e74-3bf9-49ce-8dbd-5345fd5128d6', u'state': u'idle'}
  101. The PI example from before then can be run as:: python
  102. >>> data = {
  103. ... 'code': textwrap.dedent("""\
  104. ... import random
  105. ... NUM_SAMPLES = 100000
  106. ... def sample(p):
  107. ... x, y = random.random(), random.random()
  108. ... return 1 if x*x + y*y < 1 else 0
  109. ...
  110. ... count = sc.parallelize(xrange(0, NUM_SAMPLES)).map(sample) \
  111. ... .reduce(lambda a, b: a + b)
  112. ... print "Pi is roughly %f" % (4.0 * count / NUM_SAMPLES)
  113. ... """)
  114. ... }
  115. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  116. >>> pprint.pprint(r.json())
  117. {u'id': 12,
  118. u'output': {u'data': {u'text/plain': u'Pi is roughly 3.136000'},
  119. u'execution_count': 12,
  120. u'status': u'ok'},
  121. u'state': u'running'}
  122. REST API
  123. ========
  124. GET /batches
  125. ------------
  126. Return all the active batch jobs.
  127. Response Body
  128. ^^^^^^^^^^^^^
  129. +---------+---------------+------+
  130. | name | description | type |
  131. +=========+===============+======+
  132. | batches | `batch`_ list | list |
  133. +---------+---------------+------+
  134. POST /batches
  135. -------------
  136. Request Body
  137. ^^^^^^^^^^^^
  138. +----------------+--------------------------------------------------+-----------------+
  139. | name | description | type |
  140. +================+==================================================+=================+
  141. | file | archive holding the file | path (required) |
  142. +----------------+--------------------------------------------------+-----------------+
  143. | args | command line arguments | list of strings |
  144. +----------------+--------------------------------------------------+-----------------+
  145. | className | application's java/spark main class | string |
  146. +----------------+--------------------------------------------------+-----------------+
  147. | jars | files to be placed on the java classpath | list of paths |
  148. +----------------+--------------------------------------------------+-----------------+
  149. | pyFiles | files to be placed on the PYTHONPATH | list of paths |
  150. +----------------+--------------------------------------------------+-----------------+
  151. | files | files to be placed in executor working directory | list of paths |
  152. +----------------+--------------------------------------------------+-----------------+
  153. | driverMemory | memory for driver | string |
  154. +----------------+--------------------------------------------------+-----------------+
  155. | driverCores | number of cores used by driver | int |
  156. +----------------+--------------------------------------------------+-----------------+
  157. | executorMemory | memory for executor | string |
  158. +----------------+--------------------------------------------------+-----------------+
  159. | executorCores | number of cores used by executor | int |
  160. +----------------+--------------------------------------------------+-----------------+
  161. | archives | | list of paths |
  162. +----------------+--------------------------------------------------+-----------------+
  163. Response Body
  164. ^^^^^^^^^^^^^
  165. The created `Batch`_ object.
  166. GET /batches/{batchId}
  167. ----------------------
  168. Request Parameters
  169. ^^^^^^^^^^^^^^^^^^
  170. +------+-----------------------------+------+
  171. | name | description | type |
  172. +======+=============================+======+
  173. | from | offset | int |
  174. +------+-----------------------------+------+
  175. | size | amount of batches to return | int |
  176. +------+-----------------------------+------+
  177. Response Body
  178. ^^^^^^^^^^^^^
  179. +-------+-----------------------------+-----------------+
  180. | name | description | type |
  181. +=======+=============================+=================+
  182. | id | `batch`_ list | list |
  183. +-------+-----------------------------+-----------------+
  184. | state | The state of the batch | `batch`_ state |
  185. +-------+-----------------------------+-----------------+
  186. | lines | The output of the batch job | list of strings |
  187. +-------+-----------------------------+-----------------+
  188. DELETE /batches/{batchId}
  189. -------------------------
  190. Kill the `Batch`_ job.
  191. GET /sessions
  192. -------------
  193. Returns all the active interactive sessions.
  194. Response Body
  195. ^^^^^^^^^^^^^
  196. +----------+-----------------+------+
  197. | name | description | type |
  198. +==========+=================+======+
  199. | sessions | `session`_ list | list |
  200. +----------+-----------------+------+
  201. POST /sessions
  202. --------------
  203. Request Body
  204. ^^^^^^^^^^^^
  205. +------+--------------+----------------------------+
  206. | name | description | type |
  207. +======+==============+============================+
  208. | lang | session kind | `session kind`_ (required) |
  209. +------+--------------+----------------------------+
  210. Response Body
  211. ^^^^^^^^^^^^^
  212. The created `Session`_.
  213. GET /sessions/{sessionId}
  214. -------------------------
  215. Return the session information
  216. Response
  217. ^^^^^^^^
  218. The `Session`_.
  219. DELETE /sessions/{batchId}
  220. -------------------------
  221. Kill the `Session`_ job.
  222. GET /sessions/{sessionId}/statements
  223. ------------------------------------
  224. Return all the statements in a session.
  225. Response Body
  226. ^^^^^^^^^^^^^
  227. +------------+-------------------+------+
  228. | name | description | type |
  229. +============+===================+======+
  230. | statements | `statement`_ list | list |
  231. +------------+-------------------+------+
  232. POST /sessions/{sessionId}/statements
  233. -------------------------------------
  234. Execute a statement in a session.
  235. Request Body
  236. ^^^^^^^^^^^^
  237. +------+---------------------+--------+
  238. | name | description | type |
  239. +======+=====================+========+
  240. | code | The code to execute | string |
  241. +------+---------------------+--------+
  242. Response Body
  243. ^^^^^^^^^^^^^
  244. The `statement`_ object.
  245. REST Objects
  246. ============
  247. Batch
  248. -----
  249. +----------------+--------------------------------------------------+-----------------+
  250. | name | description | type |
  251. +================+==================================================+=================+
  252. | file | archive holding the file | path (required) |
  253. +----------------+--------------------------------------------------+-----------------+
  254. | args | command line arguments | list of strings |
  255. +----------------+--------------------------------------------------+-----------------+
  256. | className | application's java/spark main class | string |
  257. +----------------+--------------------------------------------------+-----------------+
  258. | jars | files to be placed on the java classpath | list of paths |
  259. +----------------+--------------------------------------------------+-----------------+
  260. | pyFiles | files to be placed on the PYTHONPATH | list of paths |
  261. +----------------+--------------------------------------------------+-----------------+
  262. | files | files to be placed in executor working directory | list of paths |
  263. +----------------+--------------------------------------------------+-----------------+
  264. | driverMemory | memory for driver | string |
  265. +----------------+--------------------------------------------------+-----------------+
  266. | driverCores | number of cores used by driver | int |
  267. +----------------+--------------------------------------------------+-----------------+
  268. | executorMemory | memory for executor | string |
  269. +----------------+--------------------------------------------------+-----------------+
  270. | executorCores | number of cores used by executor | int |
  271. +----------------+--------------------------------------------------+-----------------+
  272. | archives | | list of paths |
  273. +----------------+--------------------------------------------------+-----------------+
  274. Session
  275. -------
  276. Sessions represent an interactive shell.
  277. +-----------+-------------------------------+------------------+
  278. | name | description | type |
  279. +===========+===============================+==================+
  280. | id | The session id | string |
  281. +-----------+-------------------------------+------------------+
  282. | state | The state of the session | `session state`_ |
  283. +-----------+-------------------------------+------------------+
  284. | kind | The session kind | `session kind`_ |
  285. +-----------+-------------------------------+------------------+
  286. | proxyUser | The user running this session | optional string |
  287. +-----------+-------------------------------+------------------+
  288. Session State
  289. ^^^^^^^^^^^^^
  290. +-------------+----------------------------------+
  291. | name | description |
  292. +=============+==================================+
  293. | not_started | session has not been started |
  294. +-------------+----------------------------------+
  295. | starting | session is starting |
  296. +-------------+----------------------------------+
  297. | idle | session is waiting for input |
  298. +-------------+----------------------------------+
  299. | busy | session is executing a statement |
  300. +-------------+----------------------------------+
  301. | error | session errored out |
  302. +-------------+----------------------------------+
  303. | dead | session has exited |
  304. +-------------+----------------------------------+
  305. Session Kind
  306. ^^^^^^^^^^^^
  307. +---------+----------------------------------+
  308. | name | description |
  309. +=========+==================================+
  310. | spark | interactive scala/spark session |
  311. +---------+----------------------------------+
  312. | pyspark | interactive python/spark session |
  313. +---------+----------------------------------+
  314. Statement
  315. ---------
  316. Statements represent the result of an execution statement.
  317. +--------+----------------------+---------------------+
  318. | name | description | type |
  319. +========+======================+=====================+
  320. | id | The statement id | integer |
  321. +--------+----------------------+---------------------+
  322. | state | The execution state | `statement state`_ |
  323. +--------+----------------------+---------------------+
  324. | output | The execution output | `statement output`_ |
  325. +--------+----------------------+---------------------+
  326. Statement State
  327. ^^^^^^^^^^^^^^^
  328. +-----------+----------------------------------+
  329. | name | description |
  330. +===========+==================================+
  331. | running | Statement is currently executing |
  332. +-----------+----------------------------------+
  333. | available | Statement has a ready response |
  334. +-----------+----------------------------------+
  335. | error | Statement failed |
  336. +-----------+----------------------------------+
  337. Statement Output
  338. ^^^^^^^^^^^^^^^^
  339. +-----------------+-------------------+----------------------------------+
  340. | name | description | type |
  341. +=================+===================+==================================+
  342. | status | execution status | string |
  343. +-----------------+-------------------+----------------------------------+
  344. | execution_count | a monotomically | integer |
  345. | | increasing number | |
  346. +-----------------+-------------------+----------------------------------+
  347. | data | statement output | an object mapping a mime type to |
  348. | | | the result. If the mime type is |
  349. | | | ``application/json``, the value |
  350. | | | will be a JSON value |
  351. +-----------------+-------------------+----------------------------------+
  352. Community
  353. =========
  354. * User group: http://groups.google.com/a/cloudera.org/group/hue-user
  355. * Jira: https://issues.cloudera.org/browse/HUE
  356. * Reviews: https://review.cloudera.org/dashboard/?view=to-group&group=hue (repo 'hue-rw')
  357. License
  358. =======
  359. Apache License, Version 2.0
  360. http://www.apache.org/licenses/LICENSE-2.0