README.rst 24 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580
  1. Welcome to Livy, the REST Spark Server
  2. ======================================
  3. Livy is an open source REST interface (in **Beta**) for interacting with a
  4. remote Spark Shell running locally or from inside YARN.
  5. * Interactive Spark Scala, Python shells
  6. * Batch submissions in Scala, Java, Python
  7. * Multi users can submit to the same server with their own credentials in YARN mode (impersonation support)
  8. * Can be used for submitting jobs from anywhere with REST
  9. * Does not require any code change to your programs
  10. Livy is used for powering the `Spark Notebook`_ of `Hue 3.8`_, which you can see the
  11. `implementation here`_.
  12. .. _Spark Notebook: http://gethue.com/new-notebook-application-for-spark-sql/
  13. .. _Hue 3.8: http://gethue.com/hue-3-8-with-an-oozie-editor-revamp-better-performances-improved-spark-ui-is-out/
  14. .. _implementation here: https://github.com/cloudera/hue/blob/master/apps/spark/src/spark/job_server_api.py
  15. Prerequisites
  16. =============
  17. To build Livy, you will need:
  18. Debian/Ubuntu:
  19. * mvn (from ``maven`` package or maven3 tarball)
  20. * openjdk-7-jdk (or Oracle Java7 jdk)
  21. * spark 1.3 from (from `Apache Spark tarball`_)
  22. Redhat/CentOS:
  23. * mvn (from ``maven`` package or maven3 tarball)
  24. * java-1.7.0-openjdk (or Oracle Java7 jdk)
  25. * spark 1.3 (from `Apache Spark tarball`_)
  26. MacOS:
  27. * Xcode command line tools
  28. * Oracle's JDK 1.7+
  29. * Maven (Homebrew)
  30. * apache-spark (Homebrew)
  31. .. _Apache Spark Tarball: https://spark.apache.org/downloads.html
  32. Building Livy
  33. =============
  34. Livy is currently built by the `Hue Build System`_, it can also be built on
  35. it's own (aka without any other Hue dependency) with `Apache Maven`_. To build,
  36. run:
  37. .. code:: shell
  38. % cd $HUE_HOME/apps/spark/java
  39. % mvn -DskipTests clean package
  40. .. _Hue Build System: https://github.com/cloudera/hue/#getting-started
  41. .. _Apache Maven: http://maven.apache.org
  42. Running Tests
  43. =============
  44. In order to run the Livy Tests, first follow the instructions in `Building
  45. Livy`_. Then run:
  46. .. code:: shell
  47. % cd $HUE_HOME/apps/spark/java
  48. % export SPARK_HOME=/usr/lib/spark
  49. % mvn test
  50. Running Livy
  51. ============
  52. In order to run Livy with local sessions, start the server with:
  53. .. code:: shell
  54. % export SPARK_HOME=/usr/lib/spark
  55. % ./bin/livy-server
  56. Or with YARN sessions by running:
  57. .. code:: shell
  58. % env \
  59. LIVY_SERVER_JAVA_OPTS="-Dlivy.server.session.factory=yarn" \
  60. CLASSPATH=`hadoop classpath` \
  61. ./bin/livy-server
  62. Spark Example
  63. =============
  64. Now to see it in action by interacting with it in Python with the `Requests`_
  65. library. By default livy runs on port 8998 (which can be changed with the
  66. ``livy_server_port config`` option). We’ll start off with a Spark session that
  67. takes Scala code:
  68. .. code:: python
  69. >>> import json, pprint, requests, textwrap
  70. >>> host = 'http://localhost:8998'
  71. >>> data = {'kind': 'spark'}
  72. >>> r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
  73. >>> r.json()
  74. {u'state': u'starting', u'id': 0, u’kind’: u’spark’}
  75. Once the session has completed starting up, it transitions to the idle state:
  76. .. code:: python
  77. >>> session_url = host + r.headers['location']
  78. >>> r = requests.get(session_url, headers=headers)
  79. >>> r.json()
  80. {u'state': u'idle', u'id': 0, u’kind’: u’spark’}
  81. Now we can execute Scala by passing in a simple JSON command:
  82. .. code:: python
  83. >>> data = {'code': '1 + 1'}
  84. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  85. >>> r.json()
  86. {u'output': None, u'state': u'running', u'id': 0}
  87. If a statement takes longer than a few milliseconds to execute, Livy returns
  88. early and provides a URL that can be polled until it is complete:
  89. .. code:: python
  90. >>> statement_url = host + r.headers['location']
  91. >>> r = requests.get(statement_url, headers=headers)
  92. >>> pprint.pprint(r.json())
  93. [{u'id': 0,
  94. u'output': {u'data': {u'text/plain': u'res0: Int = 2'},
  95. u'execution_count': 0,
  96. u'status': u'ok'},
  97. u'state': u'available'}]
  98. That was a pretty simple example. More interesting is using Spark to estimate
  99. PI. This is from the `Spark Examples`_:
  100. .. code:: python
  101. >>> data = {
  102. ... 'code': textwrap.dedent("""\
  103. ... val NUM_SAMPLES = 100000;
  104. ... val count = sc.parallelize(1 to NUM_SAMPLES).map { i =>
  105. ... val x = Math.random();
  106. ... val y = Math.random();
  107. ... if (x*x + y*y < 1) 1 else 0
  108. ... }.reduce(_ + _);
  109. ... println(\"Pi is roughly \" + 4.0 * count / NUM_SAMPLES)
  110. ... """)
  111. ... }
  112. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  113. >>> pprint.pprint(r.json())
  114. {u'id': 1,
  115. u'output': {u'data': {u'text/plain': u'Pi is roughly 3.14004\nNUM_SAMPLES: Int = 100000\ncount: Int = 78501'},
  116. u'execution_count': 1,
  117. u'status': u'ok'},
  118. u'state': u'available'}
  119. Finally, lets close our session:
  120. .. code:: python
  121. >>> session_url = 'http://localhost:8998/sessions/0'
  122. >>> requests.delete(session_url, headers=headers)
  123. <Response [204]>
  124. .. _Requests: http://docs.python-requests.org/en/latest/
  125. .. _Spark Examples: https://spark.apache.org/examples.html
  126. PySpark Example
  127. ===============
  128. pyspark has the exact same API, just with a different initial command:
  129. .. code:: python
  130. >>> data = {'kind': 'pyspark'}
  131. >>> r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
  132. >>> r.json()
  133. {u'id': 1, u'state': u'idle'}
  134. The PI example from before then can be run as:
  135. .. code:: python
  136. >>> data = {
  137. ... 'code': textwrap.dedent("""\
  138. ... import random
  139. ... NUM_SAMPLES = 100000
  140. ... def sample(p):
  141. ... x, y = random.random(), random.random()
  142. ... return 1 if x*x + y*y < 1 else 0
  143. ...
  144. ... count = sc.parallelize(xrange(0, NUM_SAMPLES)).map(sample) \
  145. ... .reduce(lambda a, b: a + b)
  146. ... print "Pi is roughly %f" % (4.0 * count / NUM_SAMPLES)
  147. ... """)
  148. ... }
  149. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  150. >>> pprint.pprint(r.json())
  151. {u'id': 12,
  152. u'output': {u'data': {u'text/plain': u'Pi is roughly 3.136000'},
  153. u'execution_count': 12,
  154. u'status': u'ok'},
  155. u'state': u'running'}
  156. Community
  157. =========
  158. * User group: http://groups.google.com/a/cloudera.org/group/hue-user
  159. * Jira: https://issues.cloudera.org/browse/HUE-2588
  160. * Reviews: https://review.cloudera.org/dashboard/?view=to-group&group=hue (repo 'hue-rw')
  161. REST API
  162. ========
  163. GET /sessions
  164. -------------
  165. Returns all the active interactive sessions.
  166. Response Body
  167. ^^^^^^^^^^^^^
  168. +----------+-----------------+------+
  169. | name | description | type |
  170. +==========+=================+======+
  171. | sessions | `session`_ list | list |
  172. +----------+-----------------+------+
  173. POST /sessions
  174. --------------
  175. Creates a new interative Scala or Python shell in the cluster.
  176. Request Body
  177. ^^^^^^^^^^^^
  178. +----------------+--------------------------------------------------+----------------------------+
  179. | name | description | type |
  180. +================+==================================================+============================+
  181. | kind | session kind (spark, pyspark, or sparkr) | `session kind`_ (required) |
  182. +----------------+--------------------------------------------------+----------------------------+
  183. | file | archive holding the file | path (required) |
  184. +----------------+--------------------------------------------------+----------------------------+
  185. | args | command line arguments | list of strings |
  186. +----------------+--------------------------------------------------+----------------------------+
  187. | className | application's java/spark main class | string |
  188. +----------------+--------------------------------------------------+----------------------------+
  189. | jars | files to be placed on the java classpath | list of paths |
  190. +----------------+--------------------------------------------------+----------------------------+
  191. | pyFiles | files to be placed on the PYTHONPATH | list of paths |
  192. +----------------+--------------------------------------------------+----------------------------+
  193. | files | files to be placed in executor working directory | list of paths |
  194. +----------------+--------------------------------------------------+----------------------------+
  195. | driverMemory | memory for driver | string |
  196. +----------------+--------------------------------------------------+----------------------------+
  197. | driverCores | number of cores used by driver | int |
  198. +----------------+--------------------------------------------------+----------------------------+
  199. | executorMemory | memory for executor | string |
  200. +----------------+--------------------------------------------------+----------------------------+
  201. | executorCores | number of cores used by executor | int |
  202. +----------------+--------------------------------------------------+----------------------------+
  203. | numExecutors | number of executor | int |
  204. +----------------+--------------------------------------------------+----------------------------+
  205. | archives | | list of paths |
  206. +----------------+--------------------------------------------------+----------------------------+
  207. Response Body
  208. ^^^^^^^^^^^^^
  209. The created `Session`_.
  210. GET /sessions/{sessionId}
  211. -------------------------
  212. Return the session information
  213. Response
  214. ^^^^^^^^
  215. The `Session`_.
  216. DELETE /sessions/{sessionId}
  217. -------------------------
  218. Kill the `Session`_ job.
  219. GET /sessions/{sessionId}/statements
  220. ------------------------------------
  221. Return all the statements in a session.
  222. Response Body
  223. ^^^^^^^^^^^^^
  224. +------------+-------------------+------+
  225. | name | description | type |
  226. +============+===================+======+
  227. | statements | `statement`_ list | list |
  228. +------------+-------------------+------+
  229. POST /sessions/{sessionId}/statements
  230. -------------------------------------
  231. Execute a statement in a session.
  232. Request Body
  233. ^^^^^^^^^^^^
  234. +------+---------------------+--------+
  235. | name | description | type |
  236. +======+=====================+========+
  237. | code | The code to execute | string |
  238. +------+---------------------+--------+
  239. Response Body
  240. ^^^^^^^^^^^^^
  241. The `statement`_ object.
  242. GET /batches
  243. ------------
  244. Return all the active batch jobs.
  245. Response Body
  246. ^^^^^^^^^^^^^
  247. +---------+---------------+------+
  248. | name | description | type |
  249. +=========+===============+======+
  250. | batches | `batch`_ list | list |
  251. +---------+---------------+------+
  252. POST /batches
  253. -------------
  254. Request Body
  255. ^^^^^^^^^^^^
  256. +----------------+--------------------------------------------------+-----------------+
  257. | name | description | type |
  258. +================+==================================================+=================+
  259. | file | archive holding the file | path (required) |
  260. +----------------+--------------------------------------------------+-----------------+
  261. | args | command line arguments | list of strings |
  262. +----------------+--------------------------------------------------+-----------------+
  263. | className | application's java/spark main class | string |
  264. +----------------+--------------------------------------------------+-----------------+
  265. | jars | files to be placed on the java classpath | list of paths |
  266. +----------------+--------------------------------------------------+-----------------+
  267. | pyFiles | files to be placed on the PYTHONPATH | list of paths |
  268. +----------------+--------------------------------------------------+-----------------+
  269. | files | files to be placed in executor working directory | list of paths |
  270. +----------------+--------------------------------------------------+-----------------+
  271. | driverMemory | memory for driver | string |
  272. +----------------+--------------------------------------------------+-----------------+
  273. | driverCores | number of cores used by driver | int |
  274. +----------------+--------------------------------------------------+-----------------+
  275. | executorMemory | memory for executor | string |
  276. +----------------+--------------------------------------------------+-----------------+
  277. | executorCores | number of cores used by executor | int |
  278. +----------------+--------------------------------------------------+-----------------+
  279. | numExecutors | number of executor | int |
  280. +----------------+--------------------------------------------------+-----------------+
  281. | archives | | list of paths |
  282. +----------------+--------------------------------------------------+-----------------+
  283. Response Body
  284. ^^^^^^^^^^^^^
  285. The created `Batch`_ object.
  286. GET /batches/{batchId}
  287. ----------------------
  288. Request Parameters
  289. ^^^^^^^^^^^^^^^^^^
  290. +------+-----------------------------+------+
  291. | name | description | type |
  292. +======+=============================+======+
  293. | from | offset | int |
  294. +------+-----------------------------+------+
  295. | size | amount of batches to return | int |
  296. +------+-----------------------------+------+
  297. Response Body
  298. ^^^^^^^^^^^^^
  299. +-------+-----------------------------+-----------------+
  300. | name | description | type |
  301. +=======+=============================+=================+
  302. | id | `batch`_ list | list |
  303. +-------+-----------------------------+-----------------+
  304. | state | The state of the batch | `batch`_ state |
  305. +-------+-----------------------------+-----------------+
  306. | lines | The output of the batch job | list of strings |
  307. +-------+-----------------------------+-----------------+
  308. DELETE /batches/{batchId}
  309. -------------------------
  310. Kill the `Batch`_ job.
  311. REST Objects
  312. ============
  313. Session
  314. -------
  315. Sessions represent an interactive shell.
  316. +----------------+--------------------------------------------------------------------------------+------------------+
  317. | name | description | type |
  318. +================+================================================================================+==================+
  319. | id | The session id | string |
  320. +----------------+--------------------------------------------------------------------------------+------------------+
  321. | state | The state of the session | `session state`_ |
  322. +----------------+--------------------------------------------------------------------------------+------------------+
  323. | kind | The session kind | `session kind`_ |
  324. +----------------+--------------------------------------------------------------------------------+------------------+
  325. | proxyUser | The user running this session | optional string |
  326. +----------------+--------------------------------------------------------------------------------+------------------+
  327. | jars | files to be placed on the java classpath | list of paths |
  328. +----------------+--------------------------------------------------------------------------------+------------------+
  329. | pyFiles | files to be placed on the PYTHONPATH | list of paths |
  330. +----------------+--------------------------------------------------------------------------------+------------------+
  331. | files | files to be placed in executor working directory | list of paths |
  332. +----------------+--------------------------------------------------------------------------------+------------------+
  333. | driverMemory | memory for driver | string |
  334. +----------------+--------------------------------------------------------------------------------+------------------+
  335. | driverCores | number of cores used by driver (YARN mode only) | int |
  336. +----------------+--------------------------------------------------------------------------------+------------------+
  337. | executorMemory | memory for executor | string |
  338. +----------------+--------------------------------------------------------------------------------+------------------+
  339. | executorCores | number of cores used by executor | int |
  340. +----------------+--------------------------------------------------------------------------------+------------------+
  341. | numExecutors | number of executors (YARN mode only) | int |
  342. +----------------+--------------------------------------------------------------------------------+------------------+
  343. | archives | Archives to be uncompressed in the executor working directory (YARN mode only) | list of paths |
  344. +----------------+--------------------------------------------------------------------------------+------------------+
  345. Session State
  346. ^^^^^^^^^^^^^
  347. +-------------+----------------------------------+
  348. | name | description |
  349. +=============+==================================+
  350. | not_started | session has not been started |
  351. +-------------+----------------------------------+
  352. | starting | session is starting |
  353. +-------------+----------------------------------+
  354. | idle | session is waiting for input |
  355. +-------------+----------------------------------+
  356. | busy | session is executing a statement |
  357. +-------------+----------------------------------+
  358. | error | session errored out |
  359. +-------------+----------------------------------+
  360. | dead | session has exited |
  361. +-------------+----------------------------------+
  362. Session Kind
  363. ^^^^^^^^^^^^
  364. +---------+----------------------------------+
  365. | name | description |
  366. +=========+==================================+
  367. | spark | interactive scala/spark session |
  368. +---------+----------------------------------+
  369. | pyspark | interactive python/spark session |
  370. +---------+----------------------------------+
  371. Statement
  372. ---------
  373. Statements represent the result of an execution statement.
  374. +--------+----------------------+---------------------+
  375. | name | description | type |
  376. +========+======================+=====================+
  377. | id | The statement id | integer |
  378. +--------+----------------------+---------------------+
  379. | state | The execution state | `statement state`_ |
  380. +--------+----------------------+---------------------+
  381. | output | The execution output | `statement output`_ |
  382. +--------+----------------------+---------------------+
  383. Statement State
  384. ^^^^^^^^^^^^^^^
  385. +-----------+----------------------------------+
  386. | name | description |
  387. +===========+==================================+
  388. | running | Statement is currently executing |
  389. +-----------+----------------------------------+
  390. | available | Statement has a ready response |
  391. +-----------+----------------------------------+
  392. | error | Statement failed |
  393. +-----------+----------------------------------+
  394. Statement Output
  395. ^^^^^^^^^^^^^^^^
  396. +-----------------+-------------------+----------------------------------+
  397. | name | description | type |
  398. +=================+===================+==================================+
  399. | status | execution status | string |
  400. +-----------------+-------------------+----------------------------------+
  401. | execution_count | a monotomically | integer |
  402. | | increasing number | |
  403. +-----------------+-------------------+----------------------------------+
  404. | data | statement output | an object mapping a mime type to |
  405. | | | the result. If the mime type is |
  406. | | | ``application/json``, the value |
  407. | | | will be a JSON value |
  408. +-----------------+-------------------+----------------------------------+
  409. Batch
  410. -----
  411. +----------------+--------------------------------------------------------------------------------+-----------------+
  412. | name | description | type |
  413. +================+================================================================================+=================+
  414. | file | archive holding the file | path (required) |
  415. +----------------+--------------------------------------------------------------------------------+-----------------+
  416. | args | command line arguments | list of strings |
  417. +----------------+--------------------------------------------------------------------------------+-----------------+
  418. | className | application's java/spark main class | string |
  419. +----------------+--------------------------------------------------------------------------------+-----------------+
  420. | jars | files to be placed on the java classpath | list of paths |
  421. +----------------+--------------------------------------------------------------------------------+-----------------+
  422. | pyFiles | files to be placed on the PYTHONPATH | list of paths |
  423. +----------------+--------------------------------------------------------------------------------+-----------------+
  424. | files | files to be placed in executor working directory | list of paths |
  425. +----------------+--------------------------------------------------------------------------------+-----------------+
  426. | driverMemory | memory for driver | string |
  427. +----------------+--------------------------------------------------------------------------------+-----------------+
  428. | driverCores | number of cores used by driver (YARN mode only) | int |
  429. +----------------+--------------------------------------------------------------------------------+-----------------+
  430. | executorMemory | memory for executor | string |
  431. +----------------+--------------------------------------------------------------------------------+-----------------+
  432. | executorCores | number of cores used by executor | int |
  433. +----------------+--------------------------------------------------------------------------------+-----------------+
  434. | numExecutors | number of executors (YARN mode only) | int |
  435. +----------------+--------------------------------------------------------------------------------+-----------------+
  436. | archives | Archives to be uncompressed in the executor working directory (YARN mode only) | list of paths |
  437. +----------------+--------------------------------------------------------------------------------+-----------------+
  438. License
  439. =======
  440. Apache License, Version 2.0
  441. http://www.apache.org/licenses/LICENSE-2.0