README.rst 17 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504
  1. Welcome to Livy, the REST Spark Server
  2. ======================================
  3. Livy is an open source REST interface (in **Beta**) for interacting with a remote Spark Shell
  4. running locally or from inside YARN.
  5. Interactive Spark Scala, Python and batch jar/py submission are supported.
  6. Livy is used for powering the Spark Notebook of Hue, which you can see the `implementation here`_.
  7. .. _implementation here: https://github.com/cloudera/hue/blob/master/apps/spark/src/spark/job_server_api.py
  8. Prerequisites
  9. =============
  10. To build Livy, you will need:
  11. Debian/Ubuntu:
  12. * mvn (from ``maven`` package or maven3 tarball)
  13. * openjdk-7-jdk (or Oracle Java7 jdk)
  14. Redhat/CentOS:
  15. * mvn (from ``maven`` package or maven3 tarball)
  16. * java-1.7.0-openjdk (or Oracle Java7 jdk)
  17. MacOS:
  18. * Xcode command line tools
  19. * Oracle's JDK 1.7+
  20. * Maven (Homebrew)
  21. Building Livy
  22. =============
  23. Livy is currently built by the `Hue Build System`_, it can also be built on it's
  24. own (aka without any other Hue dependency) with `Apache Maven`_. To build, run:
  25. .. code:: shell
  26. % cd $HUE_HOME/apps/spark/java
  27. % mvn -DskipTests clean package
  28. .. _Hue Build System: https://github.com/cloudera/hue/#getting-started
  29. .. _Apache Maven: http://maven.apache.org
  30. Running Tests
  31. =============
  32. In order to run the Livy Tests, first follow the instructions in `Building
  33. Livy`_. Then run:
  34. .. code:: shell
  35. % cd $HUE_HOME/apps/spark/java
  36. % mvn test
  37. Running Livy
  38. ============
  39. In order to run Livy:
  40. .. code:: shell
  41. % ????????????
  42. Spark Example
  43. =============
  44. Now to see it in action by interacting with it in Python with the `Requests`_
  45. library. By default livy runs on port 8998 (which can be changed with the
  46. ``livy_server_port config`` option). We’ll start off with a Spark session that takes Scala code:
  47. .. code:: python
  48. >>> import json, pprint, requests, textwrap
  49. >>> host = 'http://localhost:8998'
  50. >>> data = {'lang': 'spark'}
  51. >>> r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
  52. >>> r.json()
  53. {u'state': u'starting', u'id': u'89beded3-04eb-4602-9a9e-6d77780f2572', u’kind’: u’spark’}
  54. Once the session has completed starting up, it transitions to the idle state:
  55. .. code:: python
  56. >>> session_url = host + r.headers['location']
  57. >>> r = requests.get(session_url, headers=headers)
  58. >>> r.json()
  59. {u'state': u'idle', u'id': u'89beded3-04eb-4602-9a9e-6d77780f2572', u’kind’: u’spark’}
  60. Now we can execute Scala by passing in a simple JSON command:
  61. .. code:: python
  62. >>> data = {'code': '1 + 1'}
  63. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  64. >>> r.json()
  65. {u'output': None, u'state': u'running', u'id': 0}
  66. If a statement takes longer than a few milliseconds to execute, Livy returns
  67. early and provides a URL that can be polled until it is complete:
  68. .. code:: python
  69. >>> statement_url = host + r.headers['location']
  70. >>> r = requests.get(statement_url, headers=headers)
  71. >>> pprint.pprint(r.json())
  72. [{u'id': 0,
  73. u'output': {u'data': {u'text/plain': u'res0: Int = 2'},
  74. u'execution_count': 0,
  75. u'status': u'ok'},
  76. u'state': u'available'}]
  77. That was a pretty simple example. More interesting is using Spark to estimate
  78. PI. This is from the `Spark Examples`_:
  79. .. code:: python
  80. >>> data = {
  81. ... 'code': textwrap.dedent("""\
  82. ... val NUM_SAMPLES = 100000;
  83. ... val count = sc.parallelize(1 to NUM_SAMPLES).map { i =>
  84. ... val x = Math.random();
  85. ... val y = Math.random();
  86. ... if (x*x + y*y < 1) 1 else 0
  87. ... }.reduce(_ + _);
  88. ... println(\"Pi is roughly \" + 4.0 * count / NUM_SAMPLES)
  89. ... """)
  90. ... }
  91. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  92. >>> pprint.pprint(r.json())
  93. {u'id': 1,
  94. u'output': {u'data': {u'text/plain': u'Pi is roughly 3.14004\nNUM_SAMPLES: Int = 100000\ncount: Int = 78501'},
  95. u'execution_count': 1,
  96. u'status': u'ok'},
  97. u'state': u'available'}
  98. Finally, lets close our session:
  99. .. code:: python
  100. >>> session_url = 'http://localhost:8998/sessions/73e30e74-3bf9-49ce-8dbd-5345fd5128d6'
  101. >>> requests.delete(session_url, headers=headers)
  102. <Response [204]>
  103. .. _Requests: http://docs.python-requests.org/en/latest/
  104. .. _Spark Examples: https://spark.apache.org/examples.html
  105. PySpark Example
  106. ===============
  107. pyspark has the exact same API, just with a different initial command:
  108. .. code:: python
  109. >>> data = {'lang': 'pyspark'}
  110. >>> r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
  111. >>> r.json()
  112. {u'id': u'73e30e74-3bf9-49ce-8dbd-5345fd5128d6', u'state': u'idle'}
  113. The PI example from before then can be run as:
  114. .. code:: python
  115. >>> data = {
  116. ... 'code': textwrap.dedent("""\
  117. ... import random
  118. ... NUM_SAMPLES = 100000
  119. ... def sample(p):
  120. ... x, y = random.random(), random.random()
  121. ... return 1 if x*x + y*y < 1 else 0
  122. ...
  123. ... count = sc.parallelize(xrange(0, NUM_SAMPLES)).map(sample) \
  124. ... .reduce(lambda a, b: a + b)
  125. ... print "Pi is roughly %f" % (4.0 * count / NUM_SAMPLES)
  126. ... """)
  127. ... }
  128. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  129. >>> pprint.pprint(r.json())
  130. {u'id': 12,
  131. u'output': {u'data': {u'text/plain': u'Pi is roughly 3.136000'},
  132. u'execution_count': 12,
  133. u'status': u'ok'},
  134. u'state': u'running'}
  135. Community
  136. =========
  137. * User group: http://groups.google.com/a/cloudera.org/group/hue-user
  138. * Jira: https://issues.cloudera.org/browse/HUE-2588
  139. * Reviews: https://review.cloudera.org/dashboard/?view=to-group&group=hue (repo 'hue-rw')
  140. REST API
  141. ========
  142. GET /batches
  143. ------------
  144. Return all the active batch jobs.
  145. Response Body
  146. ^^^^^^^^^^^^^
  147. +---------+---------------+------+
  148. | name | description | type |
  149. +=========+===============+======+
  150. | batches | `batch`_ list | list |
  151. +---------+---------------+------+
  152. POST /batches
  153. -------------
  154. Request Body
  155. ^^^^^^^^^^^^
  156. +----------------+--------------------------------------------------+-----------------+
  157. | name | description | type |
  158. +================+==================================================+=================+
  159. | file | archive holding the file | path (required) |
  160. +----------------+--------------------------------------------------+-----------------+
  161. | args | command line arguments | list of strings |
  162. +----------------+--------------------------------------------------+-----------------+
  163. | className | application's java/spark main class | string |
  164. +----------------+--------------------------------------------------+-----------------+
  165. | jars | files to be placed on the java classpath | list of paths |
  166. +----------------+--------------------------------------------------+-----------------+
  167. | pyFiles | files to be placed on the PYTHONPATH | list of paths |
  168. +----------------+--------------------------------------------------+-----------------+
  169. | files | files to be placed in executor working directory | list of paths |
  170. +----------------+--------------------------------------------------+-----------------+
  171. | driverMemory | memory for driver | string |
  172. +----------------+--------------------------------------------------+-----------------+
  173. | driverCores | number of cores used by driver | int |
  174. +----------------+--------------------------------------------------+-----------------+
  175. | executorMemory | memory for executor | string |
  176. +----------------+--------------------------------------------------+-----------------+
  177. | executorCores | number of cores used by executor | int |
  178. +----------------+--------------------------------------------------+-----------------+
  179. | archives | | list of paths |
  180. +----------------+--------------------------------------------------+-----------------+
  181. Response Body
  182. ^^^^^^^^^^^^^
  183. The created `Batch`_ object.
  184. GET /batches/{batchId}
  185. ----------------------
  186. Request Parameters
  187. ^^^^^^^^^^^^^^^^^^
  188. +------+-----------------------------+------+
  189. | name | description | type |
  190. +======+=============================+======+
  191. | from | offset | int |
  192. +------+-----------------------------+------+
  193. | size | amount of batches to return | int |
  194. +------+-----------------------------+------+
  195. Response Body
  196. ^^^^^^^^^^^^^
  197. +-------+-----------------------------+-----------------+
  198. | name | description | type |
  199. +=======+=============================+=================+
  200. | id | `batch`_ list | list |
  201. +-------+-----------------------------+-----------------+
  202. | state | The state of the batch | `batch`_ state |
  203. +-------+-----------------------------+-----------------+
  204. | lines | The output of the batch job | list of strings |
  205. +-------+-----------------------------+-----------------+
  206. DELETE /batches/{batchId}
  207. -------------------------
  208. Kill the `Batch`_ job.
  209. GET /sessions
  210. -------------
  211. Returns all the active interactive sessions.
  212. Response Body
  213. ^^^^^^^^^^^^^
  214. +----------+-----------------+------+
  215. | name | description | type |
  216. +==========+=================+======+
  217. | sessions | `session`_ list | list |
  218. +----------+-----------------+------+
  219. POST /sessions
  220. --------------
  221. Request Body
  222. ^^^^^^^^^^^^
  223. +------+--------------+----------------------------+
  224. | name | description | type |
  225. +======+==============+============================+
  226. | lang | session kind | `session kind`_ (required) |
  227. +------+--------------+----------------------------+
  228. Response Body
  229. ^^^^^^^^^^^^^
  230. The created `Session`_.
  231. GET /sessions/{sessionId}
  232. -------------------------
  233. Return the session information
  234. Response
  235. ^^^^^^^^
  236. The `Session`_.
  237. DELETE /sessions/{batchId}
  238. -------------------------
  239. Kill the `Session`_ job.
  240. GET /sessions/{sessionId}/statements
  241. ------------------------------------
  242. Return all the statements in a session.
  243. Response Body
  244. ^^^^^^^^^^^^^
  245. +------------+-------------------+------+
  246. | name | description | type |
  247. +============+===================+======+
  248. | statements | `statement`_ list | list |
  249. +------------+-------------------+------+
  250. POST /sessions/{sessionId}/statements
  251. -------------------------------------
  252. Execute a statement in a session.
  253. Request Body
  254. ^^^^^^^^^^^^
  255. +------+---------------------+--------+
  256. | name | description | type |
  257. +======+=====================+========+
  258. | code | The code to execute | string |
  259. +------+---------------------+--------+
  260. Response Body
  261. ^^^^^^^^^^^^^
  262. The `statement`_ object.
  263. REST Objects
  264. ============
  265. Batch
  266. -----
  267. +----------------+--------------------------------------------------+-----------------+
  268. | name | description | type |
  269. +================+==================================================+=================+
  270. | file | archive holding the file | path (required) |
  271. +----------------+--------------------------------------------------+-----------------+
  272. | args | command line arguments | list of strings |
  273. +----------------+--------------------------------------------------+-----------------+
  274. | className | application's java/spark main class | string |
  275. +----------------+--------------------------------------------------+-----------------+
  276. | jars | files to be placed on the java classpath | list of paths |
  277. +----------------+--------------------------------------------------+-----------------+
  278. | pyFiles | files to be placed on the PYTHONPATH | list of paths |
  279. +----------------+--------------------------------------------------+-----------------+
  280. | files | files to be placed in executor working directory | list of paths |
  281. +----------------+--------------------------------------------------+-----------------+
  282. | driverMemory | memory for driver | string |
  283. +----------------+--------------------------------------------------+-----------------+
  284. | driverCores | number of cores used by driver | int |
  285. +----------------+--------------------------------------------------+-----------------+
  286. | executorMemory | memory for executor | string |
  287. +----------------+--------------------------------------------------+-----------------+
  288. | executorCores | number of cores used by executor | int |
  289. +----------------+--------------------------------------------------+-----------------+
  290. | archives | | list of paths |
  291. +----------------+--------------------------------------------------+-----------------+
  292. Session
  293. -------
  294. Sessions represent an interactive shell.
  295. +-----------+-------------------------------+------------------+
  296. | name | description | type |
  297. +===========+===============================+==================+
  298. | id | The session id | string |
  299. +-----------+-------------------------------+------------------+
  300. | state | The state of the session | `session state`_ |
  301. +-----------+-------------------------------+------------------+
  302. | kind | The session kind | `session kind`_ |
  303. +-----------+-------------------------------+------------------+
  304. | proxyUser | The user running this session | optional string |
  305. +-----------+-------------------------------+------------------+
  306. Session State
  307. ^^^^^^^^^^^^^
  308. +-------------+----------------------------------+
  309. | name | description |
  310. +=============+==================================+
  311. | not_started | session has not been started |
  312. +-------------+----------------------------------+
  313. | starting | session is starting |
  314. +-------------+----------------------------------+
  315. | idle | session is waiting for input |
  316. +-------------+----------------------------------+
  317. | busy | session is executing a statement |
  318. +-------------+----------------------------------+
  319. | error | session errored out |
  320. +-------------+----------------------------------+
  321. | dead | session has exited |
  322. +-------------+----------------------------------+
  323. Session Kind
  324. ^^^^^^^^^^^^
  325. +---------+----------------------------------+
  326. | name | description |
  327. +=========+==================================+
  328. | spark | interactive scala/spark session |
  329. +---------+----------------------------------+
  330. | pyspark | interactive python/spark session |
  331. +---------+----------------------------------+
  332. Statement
  333. ---------
  334. Statements represent the result of an execution statement.
  335. +--------+----------------------+---------------------+
  336. | name | description | type |
  337. +========+======================+=====================+
  338. | id | The statement id | integer |
  339. +--------+----------------------+---------------------+
  340. | state | The execution state | `statement state`_ |
  341. +--------+----------------------+---------------------+
  342. | output | The execution output | `statement output`_ |
  343. +--------+----------------------+---------------------+
  344. Statement State
  345. ^^^^^^^^^^^^^^^
  346. +-----------+----------------------------------+
  347. | name | description |
  348. +===========+==================================+
  349. | running | Statement is currently executing |
  350. +-----------+----------------------------------+
  351. | available | Statement has a ready response |
  352. +-----------+----------------------------------+
  353. | error | Statement failed |
  354. +-----------+----------------------------------+
  355. Statement Output
  356. ^^^^^^^^^^^^^^^^
  357. +-----------------+-------------------+----------------------------------+
  358. | name | description | type |
  359. +=================+===================+==================================+
  360. | status | execution status | string |
  361. +-----------------+-------------------+----------------------------------+
  362. | execution_count | a monotomically | integer |
  363. | | increasing number | |
  364. +-----------------+-------------------+----------------------------------+
  365. | data | statement output | an object mapping a mime type to |
  366. | | | the result. If the mime type is |
  367. | | | ``application/json``, the value |
  368. | | | will be a JSON value |
  369. +-----------------+-------------------+----------------------------------+
  370. License
  371. =======
  372. Apache License, Version 2.0
  373. http://www.apache.org/licenses/LICENSE-2.0