README.rst 17 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516
  1. Welcome to Livy, the REST Spark Server
  2. ======================================
  3. Livy is an open source REST interface (in **Beta**) for interacting with a
  4. remote Spark Shell running locally or from inside YARN.
  5. Interactive Spark Scala, Python and batch jar/py submission are supported.
  6. Livy is used for powering the Spark Notebook of Hue, which you can see the
  7. `implementation here`_.
  8. .. _implementation here: https://github.com/cloudera/hue/blob/master/apps/spark/src/spark/job_server_api.py
  9. Prerequisites
  10. =============
  11. To build Livy, you will need:
  12. Debian/Ubuntu:
  13. * mvn (from ``maven`` package or maven3 tarball)
  14. * openjdk-7-jdk (or Oracle Java7 jdk)
  15. Redhat/CentOS:
  16. * mvn (from ``maven`` package or maven3 tarball)
  17. * java-1.7.0-openjdk (or Oracle Java7 jdk)
  18. MacOS:
  19. * Xcode command line tools
  20. * Oracle's JDK 1.7+
  21. * Maven (Homebrew)
  22. Building Livy
  23. =============
  24. Livy is currently built by the `Hue Build System`_, it can also be built on
  25. it's own (aka without any other Hue dependency) with `Apache Maven`_. To build,
  26. run:
  27. .. code:: shell
  28. % cd $HUE_HOME/apps/spark/java
  29. % mvn -DskipTests clean package
  30. .. _Hue Build System: https://github.com/cloudera/hue/#getting-started
  31. .. _Apache Maven: http://maven.apache.org
  32. Running Tests
  33. =============
  34. In order to run the Livy Tests, first follow the instructions in `Building
  35. Livy`_. Then run:
  36. .. code:: shell
  37. % cd $HUE_HOME/apps/spark/java
  38. % mvn test
  39. Running Livy
  40. ============
  41. In order to run Livy with local sessions, start the server with:
  42. .. code:: shell
  43. % ./bin/livy-server
  44. Or with YARN sessions by running:
  45. .. code:: shell
  46. % env \
  47. LIVY_SERVER_JAVA_OPTS="-Dlivy.server.session.factory=yarn" \
  48. CLASSPATH=`hadoop classpath` \
  49. ./bin/livy-server
  50. Spark Example
  51. =============
  52. Now to see it in action by interacting with it in Python with the `Requests`_
  53. library. By default livy runs on port 8998 (which can be changed with the
  54. ``livy_server_port config`` option). We’ll start off with a Spark session that
  55. takes Scala code:
  56. .. code:: python
  57. >>> import json, pprint, requests, textwrap
  58. >>> host = 'http://localhost:8998'
  59. >>> data = {'lang': 'spark'}
  60. >>> r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
  61. >>> r.json()
  62. {u'state': u'starting', u'id': 0, u’kind’: u’spark’}
  63. Once the session has completed starting up, it transitions to the idle state:
  64. .. code:: python
  65. >>> session_url = host + r.headers['location']
  66. >>> r = requests.get(session_url, headers=headers)
  67. >>> r.json()
  68. {u'state': u'idle', u'id': 0, u’kind’: u’spark’}
  69. Now we can execute Scala by passing in a simple JSON command:
  70. .. code:: python
  71. >>> data = {'code': '1 + 1'}
  72. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  73. >>> r.json()
  74. {u'output': None, u'state': u'running', u'id': 0}
  75. If a statement takes longer than a few milliseconds to execute, Livy returns
  76. early and provides a URL that can be polled until it is complete:
  77. .. code:: python
  78. >>> statement_url = host + r.headers['location']
  79. >>> r = requests.get(statement_url, headers=headers)
  80. >>> pprint.pprint(r.json())
  81. [{u'id': 0,
  82. u'output': {u'data': {u'text/plain': u'res0: Int = 2'},
  83. u'execution_count': 0,
  84. u'status': u'ok'},
  85. u'state': u'available'}]
  86. That was a pretty simple example. More interesting is using Spark to estimate
  87. PI. This is from the `Spark Examples`_:
  88. .. code:: python
  89. >>> data = {
  90. ... 'code': textwrap.dedent("""\
  91. ... val NUM_SAMPLES = 100000;
  92. ... val count = sc.parallelize(1 to NUM_SAMPLES).map { i =>
  93. ... val x = Math.random();
  94. ... val y = Math.random();
  95. ... if (x*x + y*y < 1) 1 else 0
  96. ... }.reduce(_ + _);
  97. ... println(\"Pi is roughly \" + 4.0 * count / NUM_SAMPLES)
  98. ... """)
  99. ... }
  100. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  101. >>> pprint.pprint(r.json())
  102. {u'id': 1,
  103. u'output': {u'data': {u'text/plain': u'Pi is roughly 3.14004\nNUM_SAMPLES: Int = 100000\ncount: Int = 78501'},
  104. u'execution_count': 1,
  105. u'status': u'ok'},
  106. u'state': u'available'}
  107. Finally, lets close our session:
  108. .. code:: python
  109. >>> session_url = 'http://localhost:8998/sessions/0'
  110. >>> requests.delete(session_url, headers=headers)
  111. <Response [204]>
  112. .. _Requests: http://docs.python-requests.org/en/latest/
  113. .. _Spark Examples: https://spark.apache.org/examples.html
  114. PySpark Example
  115. ===============
  116. pyspark has the exact same API, just with a different initial command:
  117. .. code:: python
  118. >>> data = {'lang': 'pyspark'}
  119. >>> r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
  120. >>> r.json()
  121. {u'id': 1, u'state': u'idle'}
  122. The PI example from before then can be run as:
  123. .. code:: python
  124. >>> data = {
  125. ... 'code': textwrap.dedent("""\
  126. ... import random
  127. ... NUM_SAMPLES = 100000
  128. ... def sample(p):
  129. ... x, y = random.random(), random.random()
  130. ... return 1 if x*x + y*y < 1 else 0
  131. ...
  132. ... count = sc.parallelize(xrange(0, NUM_SAMPLES)).map(sample) \
  133. ... .reduce(lambda a, b: a + b)
  134. ... print "Pi is roughly %f" % (4.0 * count / NUM_SAMPLES)
  135. ... """)
  136. ... }
  137. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  138. >>> pprint.pprint(r.json())
  139. {u'id': 12,
  140. u'output': {u'data': {u'text/plain': u'Pi is roughly 3.136000'},
  141. u'execution_count': 12,
  142. u'status': u'ok'},
  143. u'state': u'running'}
  144. Community
  145. =========
  146. * User group: http://groups.google.com/a/cloudera.org/group/hue-user
  147. * Jira: https://issues.cloudera.org/browse/HUE-2588
  148. * Reviews: https://review.cloudera.org/dashboard/?view=to-group&group=hue (repo 'hue-rw')
  149. REST API
  150. ========
  151. GET /batches
  152. ------------
  153. Return all the active batch jobs.
  154. Response Body
  155. ^^^^^^^^^^^^^
  156. +---------+---------------+------+
  157. | name | description | type |
  158. +=========+===============+======+
  159. | batches | `batch`_ list | list |
  160. +---------+---------------+------+
  161. POST /batches
  162. -------------
  163. Request Body
  164. ^^^^^^^^^^^^
  165. +----------------+--------------------------------------------------+-----------------+
  166. | name | description | type |
  167. +================+==================================================+=================+
  168. | file | archive holding the file | path (required) |
  169. +----------------+--------------------------------------------------+-----------------+
  170. | args | command line arguments | list of strings |
  171. +----------------+--------------------------------------------------+-----------------+
  172. | className | application's java/spark main class | string |
  173. +----------------+--------------------------------------------------+-----------------+
  174. | jars | files to be placed on the java classpath | list of paths |
  175. +----------------+--------------------------------------------------+-----------------+
  176. | pyFiles | files to be placed on the PYTHONPATH | list of paths |
  177. +----------------+--------------------------------------------------+-----------------+
  178. | files | files to be placed in executor working directory | list of paths |
  179. +----------------+--------------------------------------------------+-----------------+
  180. | driverMemory | memory for driver | string |
  181. +----------------+--------------------------------------------------+-----------------+
  182. | driverCores | number of cores used by driver | int |
  183. +----------------+--------------------------------------------------+-----------------+
  184. | executorMemory | memory for executor | string |
  185. +----------------+--------------------------------------------------+-----------------+
  186. | executorCores | number of cores used by executor | int |
  187. +----------------+--------------------------------------------------+-----------------+
  188. | archives | | list of paths |
  189. +----------------+--------------------------------------------------+-----------------+
  190. Response Body
  191. ^^^^^^^^^^^^^
  192. The created `Batch`_ object.
  193. GET /batches/{batchId}
  194. ----------------------
  195. Request Parameters
  196. ^^^^^^^^^^^^^^^^^^
  197. +------+-----------------------------+------+
  198. | name | description | type |
  199. +======+=============================+======+
  200. | from | offset | int |
  201. +------+-----------------------------+------+
  202. | size | amount of batches to return | int |
  203. +------+-----------------------------+------+
  204. Response Body
  205. ^^^^^^^^^^^^^
  206. +-------+-----------------------------+-----------------+
  207. | name | description | type |
  208. +=======+=============================+=================+
  209. | id | `batch`_ list | list |
  210. +-------+-----------------------------+-----------------+
  211. | state | The state of the batch | `batch`_ state |
  212. +-------+-----------------------------+-----------------+
  213. | lines | The output of the batch job | list of strings |
  214. +-------+-----------------------------+-----------------+
  215. DELETE /batches/{batchId}
  216. -------------------------
  217. Kill the `Batch`_ job.
  218. GET /sessions
  219. -------------
  220. Returns all the active interactive sessions.
  221. Response Body
  222. ^^^^^^^^^^^^^
  223. +----------+-----------------+------+
  224. | name | description | type |
  225. +==========+=================+======+
  226. | sessions | `session`_ list | list |
  227. +----------+-----------------+------+
  228. POST /sessions
  229. --------------
  230. Request Body
  231. ^^^^^^^^^^^^
  232. +------+--------------+----------------------------+
  233. | name | description | type |
  234. +======+==============+============================+
  235. | lang | session kind | `session kind`_ (required) |
  236. +------+--------------+----------------------------+
  237. Response Body
  238. ^^^^^^^^^^^^^
  239. The created `Session`_.
  240. GET /sessions/{sessionId}
  241. -------------------------
  242. Return the session information
  243. Response
  244. ^^^^^^^^
  245. The `Session`_.
  246. DELETE /sessions/{batchId}
  247. -------------------------
  248. Kill the `Session`_ job.
  249. GET /sessions/{sessionId}/statements
  250. ------------------------------------
  251. Return all the statements in a session.
  252. Response Body
  253. ^^^^^^^^^^^^^
  254. +------------+-------------------+------+
  255. | name | description | type |
  256. +============+===================+======+
  257. | statements | `statement`_ list | list |
  258. +------------+-------------------+------+
  259. POST /sessions/{sessionId}/statements
  260. -------------------------------------
  261. Execute a statement in a session.
  262. Request Body
  263. ^^^^^^^^^^^^
  264. +------+---------------------+--------+
  265. | name | description | type |
  266. +======+=====================+========+
  267. | code | The code to execute | string |
  268. +------+---------------------+--------+
  269. Response Body
  270. ^^^^^^^^^^^^^
  271. The `statement`_ object.
  272. REST Objects
  273. ============
  274. Batch
  275. -----
  276. +----------------+--------------------------------------------------+-----------------+
  277. | name | description | type |
  278. +================+==================================================+=================+
  279. | file | archive holding the file | path (required) |
  280. +----------------+--------------------------------------------------+-----------------+
  281. | args | command line arguments | list of strings |
  282. +----------------+--------------------------------------------------+-----------------+
  283. | className | application's java/spark main class | string |
  284. +----------------+--------------------------------------------------+-----------------+
  285. | jars | files to be placed on the java classpath | list of paths |
  286. +----------------+--------------------------------------------------+-----------------+
  287. | pyFiles | files to be placed on the PYTHONPATH | list of paths |
  288. +----------------+--------------------------------------------------+-----------------+
  289. | files | files to be placed in executor working directory | list of paths |
  290. +----------------+--------------------------------------------------+-----------------+
  291. | driverMemory | memory for driver | string |
  292. +----------------+--------------------------------------------------+-----------------+
  293. | driverCores | number of cores used by driver | int |
  294. +----------------+--------------------------------------------------+-----------------+
  295. | executorMemory | memory for executor | string |
  296. +----------------+--------------------------------------------------+-----------------+
  297. | executorCores | number of cores used by executor | int |
  298. +----------------+--------------------------------------------------+-----------------+
  299. | archives | | list of paths |
  300. +----------------+--------------------------------------------------+-----------------+
  301. Session
  302. -------
  303. Sessions represent an interactive shell.
  304. +-----------+-------------------------------+------------------+
  305. | name | description | type |
  306. +===========+===============================+==================+
  307. | id | The session id | string |
  308. +-----------+-------------------------------+------------------+
  309. | state | The state of the session | `session state`_ |
  310. +-----------+-------------------------------+------------------+
  311. | kind | The session kind | `session kind`_ |
  312. +-----------+-------------------------------+------------------+
  313. | proxyUser | The user running this session | optional string |
  314. +-----------+-------------------------------+------------------+
  315. Session State
  316. ^^^^^^^^^^^^^
  317. +-------------+----------------------------------+
  318. | name | description |
  319. +=============+==================================+
  320. | not_started | session has not been started |
  321. +-------------+----------------------------------+
  322. | starting | session is starting |
  323. +-------------+----------------------------------+
  324. | idle | session is waiting for input |
  325. +-------------+----------------------------------+
  326. | busy | session is executing a statement |
  327. +-------------+----------------------------------+
  328. | error | session errored out |
  329. +-------------+----------------------------------+
  330. | dead | session has exited |
  331. +-------------+----------------------------------+
  332. Session Kind
  333. ^^^^^^^^^^^^
  334. +---------+----------------------------------+
  335. | name | description |
  336. +=========+==================================+
  337. | spark | interactive scala/spark session |
  338. +---------+----------------------------------+
  339. | pyspark | interactive python/spark session |
  340. +---------+----------------------------------+
  341. Statement
  342. ---------
  343. Statements represent the result of an execution statement.
  344. +--------+----------------------+---------------------+
  345. | name | description | type |
  346. +========+======================+=====================+
  347. | id | The statement id | integer |
  348. +--------+----------------------+---------------------+
  349. | state | The execution state | `statement state`_ |
  350. +--------+----------------------+---------------------+
  351. | output | The execution output | `statement output`_ |
  352. +--------+----------------------+---------------------+
  353. Statement State
  354. ^^^^^^^^^^^^^^^
  355. +-----------+----------------------------------+
  356. | name | description |
  357. +===========+==================================+
  358. | running | Statement is currently executing |
  359. +-----------+----------------------------------+
  360. | available | Statement has a ready response |
  361. +-----------+----------------------------------+
  362. | error | Statement failed |
  363. +-----------+----------------------------------+
  364. Statement Output
  365. ^^^^^^^^^^^^^^^^
  366. +-----------------+-------------------+----------------------------------+
  367. | name | description | type |
  368. +=================+===================+==================================+
  369. | status | execution status | string |
  370. +-----------------+-------------------+----------------------------------+
  371. | execution_count | a monotomically | integer |
  372. | | increasing number | |
  373. +-----------------+-------------------+----------------------------------+
  374. | data | statement output | an object mapping a mime type to |
  375. | | | the result. If the mime type is |
  376. | | | ``application/json``, the value |
  377. | | | will be a JSON value |
  378. +-----------------+-------------------+----------------------------------+
  379. License
  380. =======
  381. Apache License, Version 2.0
  382. http://www.apache.org/licenses/LICENSE-2.0