README.rst 24 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589
  1. Welcome to Livy, the REST Spark Server
  2. ======================================
  3. Livy is an open source REST interface (in **Beta**) for interacting with a
  4. remote Spark Shell running locally or from inside YARN.
  5. * Interactive Spark Scala, Python shells
  6. * Batch submissions in Scala, Java, Python
  7. * Multi users can submit to the same server with their own credentials in YARN mode (impersonation support)
  8. * Can be used for submitting jobs from anywhere with REST
  9. * Does not require any code change to your programs
  10. Livy is used for powering the `Spark Notebook`_ of `Hue 3.8`_, which you can see the
  11. `implementation here`_.
  12. .. _Spark Notebook: http://gethue.com/new-notebook-application-for-spark-sql/
  13. .. _Hue 3.8: http://gethue.com/hue-3-8-with-an-oozie-editor-revamp-better-performances-improved-spark-ui-is-out/
  14. .. _implementation here: https://github.com/cloudera/hue/blob/master/apps/spark/src/spark/job_server_api.py
  15. Prerequisites
  16. =============
  17. To build/run Livy, you will need:
  18. Debian/Ubuntu:
  19. * mvn (from ``maven`` package or maven3 tarball)
  20. * openjdk-7-jdk (or Oracle Java7 jdk)
  21. * spark 1.4 from (from `Apache Spark tarball`_)
  22. * Python 2.6+
  23. * R 3.x
  24. Redhat/CentOS:
  25. * mvn (from ``maven`` package or maven3 tarball)
  26. * java-1.7.0-openjdk (or Oracle Java7 jdk)
  27. * spark 1.4 (from `Apache Spark tarball`_)
  28. * Python 2.6+
  29. * R 3.x
  30. MacOS:
  31. * Xcode command line tools
  32. * Oracle's JDK 1.7+
  33. * Maven (Homebrew)
  34. * apache-spark 1.4 (Homebrew)
  35. * Python 2.6+
  36. * R 3.x
  37. .. _Apache Spark Tarball: https://spark.apache.org/downloads.html
  38. Building Livy
  39. =============
  40. Livy is currently built by the `Hue Build System`_, it can also be built on
  41. it's own (aka without any other Hue dependency) with `Apache Maven`_. To build,
  42. run:
  43. .. code:: shell
  44. % cd $HUE_HOME/apps/spark/java
  45. % mvn -DskipTests clean package
  46. .. _Hue Build System: https://github.com/cloudera/hue/#getting-started
  47. .. _Apache Maven: http://maven.apache.org
  48. Running Tests
  49. =============
  50. In order to run the Livy Tests, first follow the instructions in `Building
  51. Livy`_. Then run:
  52. .. code:: shell
  53. % cd $HUE_HOME/apps/spark/java
  54. % export SPARK_HOME=/usr/lib/spark
  55. % mvn test
  56. Running Livy
  57. ============
  58. In order to run Livy with local sessions, start the server with:
  59. .. code:: shell
  60. % export SPARK_HOME=/usr/lib/spark
  61. % ./bin/livy-server
  62. Or with YARN sessions by running:
  63. .. code:: shell
  64. % env \
  65. LIVY_SERVER_JAVA_OPTS="-Dlivy.server.session.factory=yarn" \
  66. CLASSPATH=`hadoop classpath` \
  67. ./bin/livy-server
  68. Spark Example
  69. =============
  70. Now to see it in action by interacting with it in Python with the `Requests`_
  71. library. By default livy runs on port 8998 (which can be changed with the
  72. ``livy_server_port config`` option). We’ll start off with a Spark session that
  73. takes Scala code:
  74. .. code:: python
  75. >>> import json, pprint, requests, textwrap
  76. >>> host = 'http://localhost:8998'
  77. >>> data = {'kind': 'spark'}
  78. >>> headers = {'Content-Type': 'application/json'}
  79. >>> r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
  80. >>> r.json()
  81. {u'state': u'starting', u'id': 0, u’kind’: u’spark’}
  82. Once the session has completed starting up, it transitions to the idle state:
  83. .. code:: python
  84. >>> session_url = host + r.headers['location']
  85. >>> r = requests.get(session_url, headers=headers)
  86. >>> r.json()
  87. {u'state': u'idle', u'id': 0, u’kind’: u’spark’}
  88. Now we can execute Scala by passing in a simple JSON command:
  89. .. code:: python
  90. >>> data = {'code': '1 + 1'}
  91. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  92. >>> r.json()
  93. {u'output': None, u'state': u'running', u'id': 0}
  94. If a statement takes longer than a few milliseconds to execute, Livy returns
  95. early and provides a URL that can be polled until it is complete:
  96. .. code:: python
  97. >>> statement_url = host + r.headers['location']
  98. >>> r = requests.get(statement_url, headers=headers)
  99. >>> pprint.pprint(r.json())
  100. [{u'id': 0,
  101. u'output': {u'data': {u'text/plain': u'res0: Int = 2'},
  102. u'execution_count': 0,
  103. u'status': u'ok'},
  104. u'state': u'available'}]
  105. That was a pretty simple example. More interesting is using Spark to estimate
  106. PI. This is from the `Spark Examples`_:
  107. .. code:: python
  108. >>> data = {
  109. ... 'code': textwrap.dedent("""\
  110. ... val NUM_SAMPLES = 100000;
  111. ... val count = sc.parallelize(1 to NUM_SAMPLES).map { i =>
  112. ... val x = Math.random();
  113. ... val y = Math.random();
  114. ... if (x*x + y*y < 1) 1 else 0
  115. ... }.reduce(_ + _);
  116. ... println(\"Pi is roughly \" + 4.0 * count / NUM_SAMPLES)
  117. ... """)
  118. ... }
  119. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  120. >>> pprint.pprint(r.json())
  121. {u'id': 1,
  122. u'output': {u'data': {u'text/plain': u'Pi is roughly 3.14004\nNUM_SAMPLES: Int = 100000\ncount: Int = 78501'},
  123. u'execution_count': 1,
  124. u'status': u'ok'},
  125. u'state': u'available'}
  126. Finally, lets close our session:
  127. .. code:: python
  128. >>> session_url = 'http://localhost:8998/sessions/0'
  129. >>> requests.delete(session_url, headers=headers)
  130. <Response [204]>
  131. .. _Requests: http://docs.python-requests.org/en/latest/
  132. .. _Spark Examples: https://spark.apache.org/examples.html
  133. PySpark Example
  134. ===============
  135. pyspark has the exact same API, just with a different initial command:
  136. .. code:: python
  137. >>> data = {'kind': 'pyspark'}
  138. >>> r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
  139. >>> r.json()
  140. {u'id': 1, u'state': u'idle'}
  141. The PI example from before then can be run as:
  142. .. code:: python
  143. >>> data = {
  144. ... 'code': textwrap.dedent("""\
  145. ... import random
  146. ... NUM_SAMPLES = 100000
  147. ... def sample(p):
  148. ... x, y = random.random(), random.random()
  149. ... return 1 if x*x + y*y < 1 else 0
  150. ...
  151. ... count = sc.parallelize(xrange(0, NUM_SAMPLES)).map(sample) \
  152. ... .reduce(lambda a, b: a + b)
  153. ... print "Pi is roughly %f" % (4.0 * count / NUM_SAMPLES)
  154. ... """)
  155. ... }
  156. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  157. >>> pprint.pprint(r.json())
  158. {u'id': 12,
  159. u'output': {u'data': {u'text/plain': u'Pi is roughly 3.136000'},
  160. u'execution_count': 12,
  161. u'status': u'ok'},
  162. u'state': u'running'}
  163. Community
  164. =========
  165. * User group: http://groups.google.com/a/cloudera.org/group/hue-user
  166. * Jira: https://issues.cloudera.org/browse/HUE-2588
  167. * Reviews: https://review.cloudera.org/dashboard/?view=to-group&group=hue (repo 'hue-rw')
  168. REST API
  169. ========
  170. GET /sessions
  171. -------------
  172. Returns all the active interactive sessions.
  173. Response Body
  174. ^^^^^^^^^^^^^
  175. +----------+-----------------+------+
  176. | name | description | type |
  177. +==========+=================+======+
  178. | sessions | `session`_ list | list |
  179. +----------+-----------------+------+
  180. POST /sessions
  181. --------------
  182. Creates a new interative Scala or Python shell in the cluster.
  183. Request Body
  184. ^^^^^^^^^^^^
  185. +----------------+--------------------------------------------------+----------------------------+
  186. | name | description | type |
  187. +================+==================================================+============================+
  188. | kind | session kind (spark, pyspark, or sparkr) | `session kind`_ (required) |
  189. +----------------+--------------------------------------------------+----------------------------+
  190. | file | archive holding the file | path (required) |
  191. +----------------+--------------------------------------------------+----------------------------+
  192. | args | command line arguments | list of strings |
  193. +----------------+--------------------------------------------------+----------------------------+
  194. | className | application's java/spark main class | string |
  195. +----------------+--------------------------------------------------+----------------------------+
  196. | jars | files to be placed on the java classpath | list of paths |
  197. +----------------+--------------------------------------------------+----------------------------+
  198. | pyFiles | files to be placed on the PYTHONPATH | list of paths |
  199. +----------------+--------------------------------------------------+----------------------------+
  200. | files | files to be placed in executor working directory | list of paths |
  201. +----------------+--------------------------------------------------+----------------------------+
  202. | driverMemory | memory for driver | string |
  203. +----------------+--------------------------------------------------+----------------------------+
  204. | driverCores | number of cores used by driver | int |
  205. +----------------+--------------------------------------------------+----------------------------+
  206. | executorMemory | memory for executor | string |
  207. +----------------+--------------------------------------------------+----------------------------+
  208. | executorCores | number of cores used by executor | int |
  209. +----------------+--------------------------------------------------+----------------------------+
  210. | numExecutors | number of executor | int |
  211. +----------------+--------------------------------------------------+----------------------------+
  212. | archives | | list of paths |
  213. +----------------+--------------------------------------------------+----------------------------+
  214. Response Body
  215. ^^^^^^^^^^^^^
  216. The created `Session`_.
  217. GET /sessions/{sessionId}
  218. -------------------------
  219. Return the session information
  220. Response
  221. ^^^^^^^^
  222. The `Session`_.
  223. DELETE /sessions/{sessionId}
  224. -------------------------
  225. Kill the `Session`_ job.
  226. GET /sessions/{sessionId}/statements
  227. ------------------------------------
  228. Return all the statements in a session.
  229. Response Body
  230. ^^^^^^^^^^^^^
  231. +------------+-------------------+------+
  232. | name | description | type |
  233. +============+===================+======+
  234. | statements | `statement`_ list | list |
  235. +------------+-------------------+------+
  236. POST /sessions/{sessionId}/statements
  237. -------------------------------------
  238. Execute a statement in a session.
  239. Request Body
  240. ^^^^^^^^^^^^
  241. +------+---------------------+--------+
  242. | name | description | type |
  243. +======+=====================+========+
  244. | code | The code to execute | string |
  245. +------+---------------------+--------+
  246. Response Body
  247. ^^^^^^^^^^^^^
  248. The `statement`_ object.
  249. GET /batches
  250. ------------
  251. Return all the active batch jobs.
  252. Response Body
  253. ^^^^^^^^^^^^^
  254. +---------+---------------+------+
  255. | name | description | type |
  256. +=========+===============+======+
  257. | batches | `batch`_ list | list |
  258. +---------+---------------+------+
  259. POST /batches
  260. -------------
  261. Request Body
  262. ^^^^^^^^^^^^
  263. +----------------+--------------------------------------------------+-----------------+
  264. | name | description | type |
  265. +================+==================================================+=================+
  266. | file | archive holding the file | path (required) |
  267. +----------------+--------------------------------------------------+-----------------+
  268. | args | command line arguments | list of strings |
  269. +----------------+--------------------------------------------------+-----------------+
  270. | className | application's java/spark main class | string |
  271. +----------------+--------------------------------------------------+-----------------+
  272. | jars | files to be placed on the java classpath | list of paths |
  273. +----------------+--------------------------------------------------+-----------------+
  274. | pyFiles | files to be placed on the PYTHONPATH | list of paths |
  275. +----------------+--------------------------------------------------+-----------------+
  276. | files | files to be placed in executor working directory | list of paths |
  277. +----------------+--------------------------------------------------+-----------------+
  278. | driverMemory | memory for driver | string |
  279. +----------------+--------------------------------------------------+-----------------+
  280. | driverCores | number of cores used by driver | int |
  281. +----------------+--------------------------------------------------+-----------------+
  282. | executorMemory | memory for executor | string |
  283. +----------------+--------------------------------------------------+-----------------+
  284. | executorCores | number of cores used by executor | int |
  285. +----------------+--------------------------------------------------+-----------------+
  286. | numExecutors | number of executor | int |
  287. +----------------+--------------------------------------------------+-----------------+
  288. | archives | | list of paths |
  289. +----------------+--------------------------------------------------+-----------------+
  290. Response Body
  291. ^^^^^^^^^^^^^
  292. The created `Batch`_ object.
  293. GET /batches/{batchId}
  294. ----------------------
  295. Request Parameters
  296. ^^^^^^^^^^^^^^^^^^
  297. +------+-----------------------------+------+
  298. | name | description | type |
  299. +======+=============================+======+
  300. | from | offset | int |
  301. +------+-----------------------------+------+
  302. | size | amount of batches to return | int |
  303. +------+-----------------------------+------+
  304. Response Body
  305. ^^^^^^^^^^^^^
  306. +-------+-----------------------------+-----------------+
  307. | name | description | type |
  308. +=======+=============================+=================+
  309. | id | `batch`_ list | list |
  310. +-------+-----------------------------+-----------------+
  311. | state | The state of the batch | `batch`_ state |
  312. +-------+-----------------------------+-----------------+
  313. | lines | The output of the batch job | list of strings |
  314. +-------+-----------------------------+-----------------+
  315. DELETE /batches/{batchId}
  316. -------------------------
  317. Kill the `Batch`_ job.
  318. REST Objects
  319. ============
  320. Session
  321. -------
  322. Sessions represent an interactive shell.
  323. +----------------+--------------------------------------------------------------------------------+------------------+
  324. | name | description | type |
  325. +================+================================================================================+==================+
  326. | id | The session id | string |
  327. +----------------+--------------------------------------------------------------------------------+------------------+
  328. | state | The state of the session | `session state`_ |
  329. +----------------+--------------------------------------------------------------------------------+------------------+
  330. | kind | The session kind | `session kind`_ |
  331. +----------------+--------------------------------------------------------------------------------+------------------+
  332. | proxyUser | The user running this session | optional string |
  333. +----------------+--------------------------------------------------------------------------------+------------------+
  334. | jars | files to be placed on the java classpath | list of paths |
  335. +----------------+--------------------------------------------------------------------------------+------------------+
  336. | pyFiles | files to be placed on the PYTHONPATH | list of paths |
  337. +----------------+--------------------------------------------------------------------------------+------------------+
  338. | files | files to be placed in executor working directory | list of paths |
  339. +----------------+--------------------------------------------------------------------------------+------------------+
  340. | driverMemory | memory for driver | string |
  341. +----------------+--------------------------------------------------------------------------------+------------------+
  342. | driverCores | number of cores used by driver (YARN mode only) | int |
  343. +----------------+--------------------------------------------------------------------------------+------------------+
  344. | executorMemory | memory for executor | string |
  345. +----------------+--------------------------------------------------------------------------------+------------------+
  346. | executorCores | number of cores used by executor | int |
  347. +----------------+--------------------------------------------------------------------------------+------------------+
  348. | numExecutors | number of executors (YARN mode only) | int |
  349. +----------------+--------------------------------------------------------------------------------+------------------+
  350. | archives | Archives to be uncompressed in the executor working directory (YARN mode only) | list of paths |
  351. +----------------+--------------------------------------------------------------------------------+------------------+
  352. Session State
  353. ^^^^^^^^^^^^^
  354. +-------------+----------------------------------+
  355. | name | description |
  356. +=============+==================================+
  357. | not_started | session has not been started |
  358. +-------------+----------------------------------+
  359. | starting | session is starting |
  360. +-------------+----------------------------------+
  361. | idle | session is waiting for input |
  362. +-------------+----------------------------------+
  363. | busy | session is executing a statement |
  364. +-------------+----------------------------------+
  365. | error | session errored out |
  366. +-------------+----------------------------------+
  367. | dead | session has exited |
  368. +-------------+----------------------------------+
  369. Session Kind
  370. ^^^^^^^^^^^^
  371. +---------+----------------------------------+
  372. | name | description |
  373. +=========+==================================+
  374. | spark | interactive scala/spark session |
  375. +---------+----------------------------------+
  376. | pyspark | interactive python/spark session |
  377. +---------+----------------------------------+
  378. Statement
  379. ---------
  380. Statements represent the result of an execution statement.
  381. +--------+----------------------+---------------------+
  382. | name | description | type |
  383. +========+======================+=====================+
  384. | id | The statement id | integer |
  385. +--------+----------------------+---------------------+
  386. | state | The execution state | `statement state`_ |
  387. +--------+----------------------+---------------------+
  388. | output | The execution output | `statement output`_ |
  389. +--------+----------------------+---------------------+
  390. Statement State
  391. ^^^^^^^^^^^^^^^
  392. +-----------+----------------------------------+
  393. | name | description |
  394. +===========+==================================+
  395. | running | Statement is currently executing |
  396. +-----------+----------------------------------+
  397. | available | Statement has a ready response |
  398. +-----------+----------------------------------+
  399. | error | Statement failed |
  400. +-----------+----------------------------------+
  401. Statement Output
  402. ^^^^^^^^^^^^^^^^
  403. +-----------------+-------------------+----------------------------------+
  404. | name | description | type |
  405. +=================+===================+==================================+
  406. | status | execution status | string |
  407. +-----------------+-------------------+----------------------------------+
  408. | execution_count | a monotomically | integer |
  409. | | increasing number | |
  410. +-----------------+-------------------+----------------------------------+
  411. | data | statement output | an object mapping a mime type to |
  412. | | | the result. If the mime type is |
  413. | | | ``application/json``, the value |
  414. | | | will be a JSON value |
  415. +-----------------+-------------------+----------------------------------+
  416. Batch
  417. -----
  418. +----------------+--------------------------------------------------------------------------------+-----------------+
  419. | name | description | type |
  420. +================+================================================================================+=================+
  421. | file | archive holding the file | path (required) |
  422. +----------------+--------------------------------------------------------------------------------+-----------------+
  423. | args | command line arguments | list of strings |
  424. +----------------+--------------------------------------------------------------------------------+-----------------+
  425. | className | application's java/spark main class | string |
  426. +----------------+--------------------------------------------------------------------------------+-----------------+
  427. | jars | files to be placed on the java classpath | list of paths |
  428. +----------------+--------------------------------------------------------------------------------+-----------------+
  429. | pyFiles | files to be placed on the PYTHONPATH | list of paths |
  430. +----------------+--------------------------------------------------------------------------------+-----------------+
  431. | files | files to be placed in executor working directory | list of paths |
  432. +----------------+--------------------------------------------------------------------------------+-----------------+
  433. | driverMemory | memory for driver | string |
  434. +----------------+--------------------------------------------------------------------------------+-----------------+
  435. | driverCores | number of cores used by driver (YARN mode only) | int |
  436. +----------------+--------------------------------------------------------------------------------+-----------------+
  437. | executorMemory | memory for executor | string |
  438. +----------------+--------------------------------------------------------------------------------+-----------------+
  439. | executorCores | number of cores used by executor | int |
  440. +----------------+--------------------------------------------------------------------------------+-----------------+
  441. | numExecutors | number of executors (YARN mode only) | int |
  442. +----------------+--------------------------------------------------------------------------------+-----------------+
  443. | archives | Archives to be uncompressed in the executor working directory (YARN mode only) | list of paths |
  444. +----------------+--------------------------------------------------------------------------------+-----------------+
  445. License
  446. =======
  447. Apache License, Version 2.0
  448. http://www.apache.org/licenses/LICENSE-2.0