README.rst 18 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528
  1. Welcome to Livy, the REST Spark Server
  2. ======================================
  3. Livy is an open source REST interface (in **Beta**) for interacting with a
  4. remote Spark Shell running locally or from inside YARN.
  5. Interactive Spark Scala, Python and batch jar/py submission are supported.
  6. Livy is used for powering the `Spark Notebook`_ of `Hue 3.8`_, which you can see the
  7. `implementation here`_.
  8. .. _Spark Notebook: http://gethue.com/new-notebook-application-for-spark-sql/
  9. .. _Hue 3.8: http://gethue.com/hue-3-8-with-an-oozie-editor-revamp-better-performances-improved-spark-ui-is-out/
  10. .. _implementation here: https://github.com/cloudera/hue/blob/master/apps/spark/src/spark/job_server_api.py
  11. Prerequisites
  12. =============
  13. To build Livy, you will need:
  14. Debian/Ubuntu:
  15. * mvn (from ``maven`` package or maven3 tarball)
  16. * openjdk-7-jdk (or Oracle Java7 jdk)
  17. * spark 1.3 from (from `Apache Spark tarball`_)
  18. Redhat/CentOS:
  19. * mvn (from ``maven`` package or maven3 tarball)
  20. * java-1.7.0-openjdk (or Oracle Java7 jdk)
  21. * spark 1.3 (from `Apache Spark tarball`_)
  22. MacOS:
  23. * Xcode command line tools
  24. * Oracle's JDK 1.7+
  25. * Maven (Homebrew)
  26. * apache-spark (Homebrew)
  27. .. _Apache Spark Tarball: https://spark.apache.org/downloads.html
  28. Building Livy
  29. =============
  30. Livy is currently built by the `Hue Build System`_, it can also be built on
  31. it's own (aka without any other Hue dependency) with `Apache Maven`_. To build,
  32. run:
  33. .. code:: shell
  34. % cd $HUE_HOME/apps/spark/java
  35. % mvn -DskipTests clean package
  36. .. _Hue Build System: https://github.com/cloudera/hue/#getting-started
  37. .. _Apache Maven: http://maven.apache.org
  38. Running Tests
  39. =============
  40. In order to run the Livy Tests, first follow the instructions in `Building
  41. Livy`_. Then run:
  42. .. code:: shell
  43. % cd $HUE_HOME/apps/spark/java
  44. % mvn test
  45. Running Livy
  46. ============
  47. In order to run Livy with local sessions, start the server with:
  48. .. code:: shell
  49. % ./bin/livy-server
  50. Or with YARN sessions by running:
  51. .. code:: shell
  52. % env \
  53. LIVY_SERVER_JAVA_OPTS="-Dlivy.server.session.factory=yarn" \
  54. CLASSPATH=`hadoop classpath` \
  55. ./bin/livy-server
  56. Spark Example
  57. =============
  58. Now to see it in action by interacting with it in Python with the `Requests`_
  59. library. By default livy runs on port 8998 (which can be changed with the
  60. ``livy_server_port config`` option). We’ll start off with a Spark session that
  61. takes Scala code:
  62. .. code:: python
  63. >>> import json, pprint, requests, textwrap
  64. >>> host = 'http://localhost:8998'
  65. >>> data = {'lang': 'spark'}
  66. >>> r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
  67. >>> r.json()
  68. {u'state': u'starting', u'id': 0, u’kind’: u’spark’}
  69. Once the session has completed starting up, it transitions to the idle state:
  70. .. code:: python
  71. >>> session_url = host + r.headers['location']
  72. >>> r = requests.get(session_url, headers=headers)
  73. >>> r.json()
  74. {u'state': u'idle', u'id': 0, u’kind’: u’spark’}
  75. Now we can execute Scala by passing in a simple JSON command:
  76. .. code:: python
  77. >>> data = {'code': '1 + 1'}
  78. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  79. >>> r.json()
  80. {u'output': None, u'state': u'running', u'id': 0}
  81. If a statement takes longer than a few milliseconds to execute, Livy returns
  82. early and provides a URL that can be polled until it is complete:
  83. .. code:: python
  84. >>> statement_url = host + r.headers['location']
  85. >>> r = requests.get(statement_url, headers=headers)
  86. >>> pprint.pprint(r.json())
  87. [{u'id': 0,
  88. u'output': {u'data': {u'text/plain': u'res0: Int = 2'},
  89. u'execution_count': 0,
  90. u'status': u'ok'},
  91. u'state': u'available'}]
  92. That was a pretty simple example. More interesting is using Spark to estimate
  93. PI. This is from the `Spark Examples`_:
  94. .. code:: python
  95. >>> data = {
  96. ... 'code': textwrap.dedent("""\
  97. ... val NUM_SAMPLES = 100000;
  98. ... val count = sc.parallelize(1 to NUM_SAMPLES).map { i =>
  99. ... val x = Math.random();
  100. ... val y = Math.random();
  101. ... if (x*x + y*y < 1) 1 else 0
  102. ... }.reduce(_ + _);
  103. ... println(\"Pi is roughly \" + 4.0 * count / NUM_SAMPLES)
  104. ... """)
  105. ... }
  106. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  107. >>> pprint.pprint(r.json())
  108. {u'id': 1,
  109. u'output': {u'data': {u'text/plain': u'Pi is roughly 3.14004\nNUM_SAMPLES: Int = 100000\ncount: Int = 78501'},
  110. u'execution_count': 1,
  111. u'status': u'ok'},
  112. u'state': u'available'}
  113. Finally, lets close our session:
  114. .. code:: python
  115. >>> session_url = 'http://localhost:8998/sessions/0'
  116. >>> requests.delete(session_url, headers=headers)
  117. <Response [204]>
  118. .. _Requests: http://docs.python-requests.org/en/latest/
  119. .. _Spark Examples: https://spark.apache.org/examples.html
  120. PySpark Example
  121. ===============
  122. pyspark has the exact same API, just with a different initial command:
  123. .. code:: python
  124. >>> data = {'lang': 'pyspark'}
  125. >>> r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
  126. >>> r.json()
  127. {u'id': 1, u'state': u'idle'}
  128. The PI example from before then can be run as:
  129. .. code:: python
  130. >>> data = {
  131. ... 'code': textwrap.dedent("""\
  132. ... import random
  133. ... NUM_SAMPLES = 100000
  134. ... def sample(p):
  135. ... x, y = random.random(), random.random()
  136. ... return 1 if x*x + y*y < 1 else 0
  137. ...
  138. ... count = sc.parallelize(xrange(0, NUM_SAMPLES)).map(sample) \
  139. ... .reduce(lambda a, b: a + b)
  140. ... print "Pi is roughly %f" % (4.0 * count / NUM_SAMPLES)
  141. ... """)
  142. ... }
  143. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  144. >>> pprint.pprint(r.json())
  145. {u'id': 12,
  146. u'output': {u'data': {u'text/plain': u'Pi is roughly 3.136000'},
  147. u'execution_count': 12,
  148. u'status': u'ok'},
  149. u'state': u'running'}
  150. Community
  151. =========
  152. * User group: http://groups.google.com/a/cloudera.org/group/hue-user
  153. * Jira: https://issues.cloudera.org/browse/HUE-2588
  154. * Reviews: https://review.cloudera.org/dashboard/?view=to-group&group=hue (repo 'hue-rw')
  155. REST API
  156. ========
  157. GET /batches
  158. ------------
  159. Return all the active batch jobs.
  160. Response Body
  161. ^^^^^^^^^^^^^
  162. +---------+---------------+------+
  163. | name | description | type |
  164. +=========+===============+======+
  165. | batches | `batch`_ list | list |
  166. +---------+---------------+------+
  167. POST /batches
  168. -------------
  169. Request Body
  170. ^^^^^^^^^^^^
  171. +----------------+--------------------------------------------------+-----------------+
  172. | name | description | type |
  173. +================+==================================================+=================+
  174. | file | archive holding the file | path (required) |
  175. +----------------+--------------------------------------------------+-----------------+
  176. | args | command line arguments | list of strings |
  177. +----------------+--------------------------------------------------+-----------------+
  178. | className | application's java/spark main class | string |
  179. +----------------+--------------------------------------------------+-----------------+
  180. | jars | files to be placed on the java classpath | list of paths |
  181. +----------------+--------------------------------------------------+-----------------+
  182. | pyFiles | files to be placed on the PYTHONPATH | list of paths |
  183. +----------------+--------------------------------------------------+-----------------+
  184. | files | files to be placed in executor working directory | list of paths |
  185. +----------------+--------------------------------------------------+-----------------+
  186. | driverMemory | memory for driver | string |
  187. +----------------+--------------------------------------------------+-----------------+
  188. | driverCores | number of cores used by driver | int |
  189. +----------------+--------------------------------------------------+-----------------+
  190. | executorMemory | memory for executor | string |
  191. +----------------+--------------------------------------------------+-----------------+
  192. | executorCores | number of cores used by executor | int |
  193. +----------------+--------------------------------------------------+-----------------+
  194. | numExecutors | number of executor | int |
  195. +----------------+--------------------------------------------------+-----------------+
  196. | archives | | list of paths |
  197. +----------------+--------------------------------------------------+-----------------+
  198. Response Body
  199. ^^^^^^^^^^^^^
  200. The created `Batch`_ object.
  201. GET /batches/{batchId}
  202. ----------------------
  203. Request Parameters
  204. ^^^^^^^^^^^^^^^^^^
  205. +------+-----------------------------+------+
  206. | name | description | type |
  207. +======+=============================+======+
  208. | from | offset | int |
  209. +------+-----------------------------+------+
  210. | size | amount of batches to return | int |
  211. +------+-----------------------------+------+
  212. Response Body
  213. ^^^^^^^^^^^^^
  214. +-------+-----------------------------+-----------------+
  215. | name | description | type |
  216. +=======+=============================+=================+
  217. | id | `batch`_ list | list |
  218. +-------+-----------------------------+-----------------+
  219. | state | The state of the batch | `batch`_ state |
  220. +-------+-----------------------------+-----------------+
  221. | lines | The output of the batch job | list of strings |
  222. +-------+-----------------------------+-----------------+
  223. DELETE /batches/{batchId}
  224. -------------------------
  225. Kill the `Batch`_ job.
  226. GET /sessions
  227. -------------
  228. Returns all the active interactive sessions.
  229. Response Body
  230. ^^^^^^^^^^^^^
  231. +----------+-----------------+------+
  232. | name | description | type |
  233. +==========+=================+======+
  234. | sessions | `session`_ list | list |
  235. +----------+-----------------+------+
  236. POST /sessions
  237. --------------
  238. Request Body
  239. ^^^^^^^^^^^^
  240. +------+--------------+----------------------------+
  241. | name | description | type |
  242. +======+==============+============================+
  243. | lang | session kind | `session kind`_ (required) |
  244. +------+--------------+----------------------------+
  245. Response Body
  246. ^^^^^^^^^^^^^
  247. The created `Session`_.
  248. GET /sessions/{sessionId}
  249. -------------------------
  250. Return the session information
  251. Response
  252. ^^^^^^^^
  253. The `Session`_.
  254. DELETE /sessions/{batchId}
  255. -------------------------
  256. Kill the `Session`_ job.
  257. GET /sessions/{sessionId}/statements
  258. ------------------------------------
  259. Return all the statements in a session.
  260. Response Body
  261. ^^^^^^^^^^^^^
  262. +------------+-------------------+------+
  263. | name | description | type |
  264. +============+===================+======+
  265. | statements | `statement`_ list | list |
  266. +------------+-------------------+------+
  267. POST /sessions/{sessionId}/statements
  268. -------------------------------------
  269. Execute a statement in a session.
  270. Request Body
  271. ^^^^^^^^^^^^
  272. +------+---------------------+--------+
  273. | name | description | type |
  274. +======+=====================+========+
  275. | code | The code to execute | string |
  276. +------+---------------------+--------+
  277. Response Body
  278. ^^^^^^^^^^^^^
  279. The `statement`_ object.
  280. REST Objects
  281. ============
  282. Batch
  283. -----
  284. +----------------+--------------------------------------------------+-----------------+
  285. | name | description | type |
  286. +================+==================================================+=================+
  287. | file | archive holding the file | path (required) |
  288. +----------------+--------------------------------------------------+-----------------+
  289. | args | command line arguments | list of strings |
  290. +----------------+--------------------------------------------------+-----------------+
  291. | className | application's java/spark main class | string |
  292. +----------------+--------------------------------------------------+-----------------+
  293. | jars | files to be placed on the java classpath | list of paths |
  294. +----------------+--------------------------------------------------+-----------------+
  295. | pyFiles | files to be placed on the PYTHONPATH | list of paths |
  296. +----------------+--------------------------------------------------+-----------------+
  297. | files | files to be placed in executor working directory | list of paths |
  298. +----------------+--------------------------------------------------+-----------------+
  299. | driverMemory | memory for driver | string |
  300. +----------------+--------------------------------------------------+-----------------+
  301. | driverCores | number of cores used by driver | int |
  302. +----------------+--------------------------------------------------+-----------------+
  303. | executorMemory | memory for executor | string |
  304. +----------------+--------------------------------------------------+-----------------+
  305. | executorCores | number of cores used by executor | int |
  306. +----------------+--------------------------------------------------+-----------------+
  307. | numExecutors | number of executor | int |
  308. +----------------+--------------------------------------------------+-----------------+
  309. | archives | | list of paths |
  310. +----------------+--------------------------------------------------+-----------------+
  311. Session
  312. -------
  313. Sessions represent an interactive shell.
  314. +-----------+-------------------------------+------------------+
  315. | name | description | type |
  316. +===========+===============================+==================+
  317. | id | The session id | string |
  318. +-----------+-------------------------------+------------------+
  319. | state | The state of the session | `session state`_ |
  320. +-----------+-------------------------------+------------------+
  321. | kind | The session kind | `session kind`_ |
  322. +-----------+-------------------------------+------------------+
  323. | proxyUser | The user running this session | optional string |
  324. +-----------+-------------------------------+------------------+
  325. Session State
  326. ^^^^^^^^^^^^^
  327. +-------------+----------------------------------+
  328. | name | description |
  329. +=============+==================================+
  330. | not_started | session has not been started |
  331. +-------------+----------------------------------+
  332. | starting | session is starting |
  333. +-------------+----------------------------------+
  334. | idle | session is waiting for input |
  335. +-------------+----------------------------------+
  336. | busy | session is executing a statement |
  337. +-------------+----------------------------------+
  338. | error | session errored out |
  339. +-------------+----------------------------------+
  340. | dead | session has exited |
  341. +-------------+----------------------------------+
  342. Session Kind
  343. ^^^^^^^^^^^^
  344. +---------+----------------------------------+
  345. | name | description |
  346. +=========+==================================+
  347. | spark | interactive scala/spark session |
  348. +---------+----------------------------------+
  349. | pyspark | interactive python/spark session |
  350. +---------+----------------------------------+
  351. Statement
  352. ---------
  353. Statements represent the result of an execution statement.
  354. +--------+----------------------+---------------------+
  355. | name | description | type |
  356. +========+======================+=====================+
  357. | id | The statement id | integer |
  358. +--------+----------------------+---------------------+
  359. | state | The execution state | `statement state`_ |
  360. +--------+----------------------+---------------------+
  361. | output | The execution output | `statement output`_ |
  362. +--------+----------------------+---------------------+
  363. Statement State
  364. ^^^^^^^^^^^^^^^
  365. +-----------+----------------------------------+
  366. | name | description |
  367. +===========+==================================+
  368. | running | Statement is currently executing |
  369. +-----------+----------------------------------+
  370. | available | Statement has a ready response |
  371. +-----------+----------------------------------+
  372. | error | Statement failed |
  373. +-----------+----------------------------------+
  374. Statement Output
  375. ^^^^^^^^^^^^^^^^
  376. +-----------------+-------------------+----------------------------------+
  377. | name | description | type |
  378. +=================+===================+==================================+
  379. | status | execution status | string |
  380. +-----------------+-------------------+----------------------------------+
  381. | execution_count | a monotomically | integer |
  382. | | increasing number | |
  383. +-----------------+-------------------+----------------------------------+
  384. | data | statement output | an object mapping a mime type to |
  385. | | | the result. If the mime type is |
  386. | | | ``application/json``, the value |
  387. | | | will be a JSON value |
  388. +-----------------+-------------------+----------------------------------+
  389. License
  390. =======
  391. Apache License, Version 2.0
  392. http://www.apache.org/licenses/LICENSE-2.0