README.rst 25 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696
  1. Welcome to Livy, the REST Spark Server
  2. ======================================
  3. Livy is an open source REST interface for interacting with Spark from anywhere. It supports executing snippets of code or programs in a Spark context that runs locally or in YARN.
  4. * Interactive Scala, Python and R shells
  5. * Batch submissions in Scala, Java, Python
  6. * Multi users can share the same server (impersonation support)
  7. * Can be used for submitting jobs from anywhere with REST
  8. * Does not require any code change to your programs
  9. The code is currently incubating in Hue but hopefully will eventually graduate in its top project. `Pull requests` are welcomed!
  10. .. _Pull requests: https://github.com/cloudera/hue/pulls
  11. Quick Start
  12. =============
  13. Livy is used for powering the Spark snippets of the `Hadoop Notebook`_ of `Hue 3.8`_, which you can see the
  14. `implementation here`_.
  15. See some `curl examples`_ and the API documentation is available below.
  16. .. _curl examples: http://gethue.com/how-to-use-the-livy-spark-rest-job-server-for-interactive-spark/
  17. .. _Hadoop Notebook: http://gethue.com/new-notebook-application-for-spark-sql/
  18. .. _Hue 3.8: http://gethue.com/hue-3-8-with-an-oozie-editor-revamp-better-performances-improved-spark-ui-is-out/
  19. .. _implementation here: https://github.com/cloudera/hue/blob/master/apps/spark/src/spark/job_server_api.py
  20. Prerequisites
  21. =============
  22. To build/run Livy, you will need:
  23. Debian/Ubuntu:
  24. * mvn (from ``maven`` package or maven3 tarball)
  25. * openjdk-7-jdk (or Oracle Java7 jdk)
  26. * spark 1.5 from (from `Apache Spark tarball`_)
  27. * Python 2.6+
  28. * R 3.x
  29. Redhat/CentOS:
  30. * mvn (from ``maven`` package or maven3 tarball)
  31. * java-1.7.0-openjdk (or Oracle Java7 jdk)
  32. * spark 1.5 (from `Apache Spark tarball`_)
  33. * Python 2.6+
  34. * R 3.x
  35. MacOS:
  36. * Xcode command line tools
  37. * Oracle's JDK 1.7+
  38. * Maven (Homebrew)
  39. * apache-spark 1.5 (Homebrew)
  40. * Python 2.6+
  41. * R 3.x
  42. .. _Apache Spark Tarball: https://spark.apache.org/downloads.html
  43. Building Livy
  44. =============
  45. Livy is currently built by the `Hue Build System`_, it can also be built on
  46. it's own (aka without any other Hue dependency) with `Apache Maven`_. To build,
  47. checks out the code, go to the Livy directory and run:
  48. .. code:: shell
  49. git clone git@github.com:cloudera/hue.git
  50. cd hue
  51. .. code:: shell
  52. % cd apps/spark/java
  53. % mvn -DskipTests clean package
  54. .. _Hue Build System: https://github.com/cloudera/hue/#getting-started
  55. .. _Apache Maven: http://maven.apache.org
  56. Running Tests
  57. =============
  58. In order to run the Livy Tests, first follow the instructions in `Building
  59. Livy`_. Then run:
  60. .. code:: shell
  61. % export SPARK_HOME=/usr/lib/spark
  62. % mvn test
  63. Running Livy
  64. ============
  65. In order to run Livy with local sessions, start the server with:
  66. .. code:: shell
  67. % export SPARK_HOME=/usr/lib/spark
  68. % ./bin/livy-server
  69. Or with YARN sessions by running:
  70. .. code:: shell
  71. % export SPARK_HOME=/usr/lib/spark
  72. % env \
  73. LIVY_SERVER_JAVA_OPTS="-Dlivy.server.session.factory=yarn" \
  74. CLASSPATH=`hadoop classpath` \
  75. ./bin/livy-server
  76. Livy Configuration
  77. ==================
  78. The properties of the server can be modified by copying <https://github.com/cloudera/hue/blob/master/apps/spark/java/conf/livy-defaults.conf.tmpl>
  79. and renaming it ``livy-defaults.conf``.
  80. In particular the ``YARN mode`` (default is ``local`` process for development) can be set with:
  81. .. code:: shell
  82. livy.server.session.factory = yarn
  83. Spark Example
  84. =============
  85. Now to see it in action by interacting with it in Python with the `Requests`_
  86. library. By default livy runs on port 8998 (which can be changed with the
  87. ``livy_server_port config`` option). We’ll start off with a Spark session that
  88. takes Scala code:
  89. .. code:: shell
  90. % sudo pip install requests
  91. .. code:: python
  92. >>> import json, pprint, requests, textwrap
  93. >>> host = 'http://localhost:8998'
  94. >>> data = {'kind': 'spark'}
  95. >>> headers = {'Content-Type': 'application/json'}
  96. >>> r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
  97. >>> r.json()
  98. {u'state': u'starting', u'id': 0, u’kind’: u’spark’}
  99. Once the session has completed starting up, it transitions to the idle state:
  100. .. code:: python
  101. >>> session_url = host + r.headers['location']
  102. >>> r = requests.get(session_url, headers=headers)
  103. >>> r.json()
  104. {u'state': u'idle', u'id': 0, u’kind’: u’spark’}
  105. Now we can execute Scala by passing in a simple JSON command:
  106. .. code:: python
  107. >>> statements_url = session_url + '/statements'
  108. >>> data = {'code': '1 + 1'}
  109. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  110. >>> r.json()
  111. {u'output': None, u'state': u'running', u'id': 0}
  112. If a statement takes longer than a few milliseconds to execute, Livy returns
  113. early and provides a URL that can be polled until it is complete:
  114. .. code:: python
  115. >>> statement_url = host + r.headers['location']
  116. >>> r = requests.get(statement_url, headers=headers)
  117. >>> pprint.pprint(r.json())
  118. [{u'id': 0,
  119. u'output': {u'data': {u'text/plain': u'res0: Int = 2'},
  120. u'execution_count': 0,
  121. u'status': u'ok'},
  122. u'state': u'available'}]
  123. That was a pretty simple example. More interesting is using Spark to estimate
  124. PI. This is from the `Spark Examples`_:
  125. .. code:: python
  126. >>> data = {
  127. ... 'code': textwrap.dedent("""\
  128. ... val NUM_SAMPLES = 100000;
  129. ... val count = sc.parallelize(1 to NUM_SAMPLES).map { i =>
  130. ... val x = Math.random();
  131. ... val y = Math.random();
  132. ... if (x*x + y*y < 1) 1 else 0
  133. ... }.reduce(_ + _);
  134. ... println(\"Pi is roughly \" + 4.0 * count / NUM_SAMPLES)
  135. ... """)
  136. ... }
  137. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  138. >>> pprint.pprint(r.json())
  139. {u'id': 1,
  140. u'output': {u'data': {u'text/plain': u'Pi is roughly 3.14004\nNUM_SAMPLES: Int = 100000\ncount: Int = 78501'},
  141. u'execution_count': 1,
  142. u'status': u'ok'},
  143. u'state': u'available'}
  144. Finally, lets close our session:
  145. .. code:: python
  146. >>> session_url = 'http://localhost:8998/sessions/0'
  147. >>> requests.delete(session_url, headers=headers)
  148. <Response [204]>
  149. .. _Requests: http://docs.python-requests.org/en/latest/
  150. .. _Spark Examples: https://spark.apache.org/examples.html
  151. PySpark Example
  152. ===============
  153. pyspark has the exact same API, just with a different initial command:
  154. .. code:: python
  155. >>> data = {'kind': 'pyspark'}
  156. >>> r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
  157. >>> r.json()
  158. {u'id': 1, u'state': u'idle'}
  159. The PI example from before then can be run as:
  160. .. code:: python
  161. >>> data = {
  162. ... 'code': textwrap.dedent("""\
  163. ... import random
  164. ... NUM_SAMPLES = 100000
  165. ... def sample(p):
  166. ... x, y = random.random(), random.random()
  167. ... return 1 if x*x + y*y < 1 else 0
  168. ...
  169. ... count = sc.parallelize(xrange(0, NUM_SAMPLES)).map(sample) \
  170. ... .reduce(lambda a, b: a + b)
  171. ... print "Pi is roughly %f" % (4.0 * count / NUM_SAMPLES)
  172. ... """)
  173. ... }
  174. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  175. >>> pprint.pprint(r.json())
  176. {u'id': 12,
  177. u'output': {u'data': {u'text/plain': u'Pi is roughly 3.136000'},
  178. u'execution_count': 12,
  179. u'status': u'ok'},
  180. u'state': u'running'}
  181. SparkR Example
  182. ==============
  183. SparkR also has the same API:
  184. .. code:: python
  185. >>> data = {'kind': 'sparkR'}
  186. >>> r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
  187. >>> r.json()
  188. {u'id': 1, u'state': u'idle'}
  189. The PI example from before then can be run as:
  190. .. code:: python
  191. >>> data = {
  192. ... 'code': textwrap.dedent("""\
  193. ... n <- 100000
  194. ... piFunc <- function(elem) {
  195. ... rands <- runif(n = 2, min = -1, max = 1)
  196. ... val <- ifelse((rands[1]^2 + rands[2]^2) < 1, 1.0, 0.0)
  197. ... val
  198. ... }
  199. ... piFuncVec <- function(elems) {
  200. ... message(length(elems))
  201. ... rands1 <- runif(n = length(elems), min = -1, max = 1)
  202. ... rands2 <- runif(n = length(elems), min = -1, max = 1)
  203. ... val <- ifelse((rands1^2 + rands2^2) < 1, 1.0, 0.0)
  204. ... sum(val)
  205. ... }
  206. ... rdd <- parallelize(sc, 1:n, slices)
  207. ... count <- reduce(lapplyPartition(rdd, piFuncVec), sum)
  208. ... cat("Pi is roughly", 4.0 * count / n, "\n")
  209. ... """)
  210. ... }
  211. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  212. >>> pprint.pprint(r.json())
  213. {u'id': 12,
  214. u'output': {u'data': {u'text/plain': u'Pi is roughly 3.136000'},
  215. u'execution_count': 12,
  216. u'status': u'ok'},
  217. u'state': u'running'}
  218. Community
  219. =========
  220. * User group: http://groups.google.com/a/cloudera.org/group/hue-user
  221. * Umbrella Jira: https://issues.cloudera.org/browse/HUE-2588
  222. * Pull requests: https://github.com/cloudera/hue/pulls
  223. REST API
  224. ========
  225. GET /sessions
  226. -------------
  227. Returns all the active interactive sessions.
  228. Response Body
  229. ^^^^^^^^^^^^^
  230. +----------+-----------------+------+
  231. | name | description | type |
  232. +==========+=================+======+
  233. | sessions | `session`_ list | list |
  234. +----------+-----------------+------+
  235. POST /sessions
  236. --------------
  237. Creates a new interative Scala, Python or R shell in the cluster.
  238. Request Body
  239. ^^^^^^^^^^^^
  240. +----------------+--------------------------------------------------------------------------------+------------------+
  241. | name | description | type |
  242. +================+================================================================================+==================+
  243. | kind | The session kind (required) | `session kind`_ |
  244. +----------------+--------------------------------------------------------------------------------+------------------+
  245. | proxyUser | The user to impersonate that will run this session (e.g. bob) | string |
  246. +----------------+--------------------------------------------------------------------------------+------------------+
  247. | jars | files to be placed on the java classpath | list of paths |
  248. +----------------+--------------------------------------------------------------------------------+------------------+
  249. | pyFiles | files to be placed on the PYTHONPATH | list of paths |
  250. +----------------+--------------------------------------------------------------------------------+------------------+
  251. | files | files to be placed in executor working directory | list of paths |
  252. +----------------+--------------------------------------------------------------------------------+------------------+
  253. | driverMemory | memory for driver (e.g. 1000M, 2G) | string |
  254. +----------------+--------------------------------------------------------------------------------+------------------+
  255. | driverCores | number of cores used by driver (YARN mode only) | int |
  256. +----------------+--------------------------------------------------------------------------------+------------------+
  257. | executorMemory | memory for executor (e.g. 1000M, 2G) | string |
  258. +----------------+--------------------------------------------------------------------------------+------------------+
  259. | executorCores | number of cores used by executor | int |
  260. +----------------+--------------------------------------------------------------------------------+------------------+
  261. | numExecutors | number of executors (YARN mode only) | int |
  262. +----------------+--------------------------------------------------------------------------------+------------------+
  263. | archives | Archives to be uncompressed in the executor working directory (YARN mode only) | list of paths |
  264. +----------------+--------------------------------------------------------------------------------+------------------+
  265. Response Body
  266. ^^^^^^^^^^^^^
  267. The created `Session`_.
  268. GET /sessions/{sessionId}
  269. -------------------------
  270. Return the session information
  271. Response
  272. ^^^^^^^^
  273. The `Session`_.
  274. DELETE /sessions/{sessionId}
  275. -------------------------
  276. Kill the `Session`_ job.
  277. GET /sessions/{sessionId}/logs
  278. ---------------------------
  279. Get the log lines from this session.
  280. Request Parameters
  281. ^^^^^^^^^^^^^^^^^^
  282. +------+-----------------------------+------+
  283. | name | description | type |
  284. +======+=============================+======+
  285. | from | offset | int |
  286. +------+-----------------------------+------+
  287. | size | amount of batches to return | int |
  288. +------+-----------------------------+------+
  289. Response Body
  290. ^^^^^^^^^^^^^
  291. +------+-----------------------+-----------------+
  292. | name | description | type |
  293. +======+=======================+=================+
  294. | id | The session id | int |
  295. +------+-----------------------+-----------------+
  296. | from | offset | int |
  297. +------+-----------------------+-----------------+
  298. | size | total amount of lines | int |
  299. +------+-----------------------+-----------------+
  300. | log | The log lines | list of strings |
  301. +------+-----------------------+-----------------+
  302. GET /sessions/{sessionId}/statements
  303. ------------------------------------
  304. Return all the statements in a session.
  305. Response Body
  306. ^^^^^^^^^^^^^
  307. +------------+-------------------+------+
  308. | name | description | type |
  309. +============+===================+======+
  310. | statements | `statement`_ list | list |
  311. +------------+-------------------+------+
  312. POST /sessions/{sessionId}/statements
  313. -------------------------------------
  314. Execute a statement in a session.
  315. Request Body
  316. ^^^^^^^^^^^^
  317. +------+---------------------+--------+
  318. | name | description | type |
  319. +======+=====================+========+
  320. | code | The code to execute | string |
  321. +------+---------------------+--------+
  322. Response Body
  323. ^^^^^^^^^^^^^
  324. The `statement`_ object.
  325. GET /batches
  326. ------------
  327. Return all the active batch jobs.
  328. Response Body
  329. ^^^^^^^^^^^^^
  330. +---------+---------------+------+
  331. | name | description | type |
  332. +=========+===============+======+
  333. | batches | `batch`_ list | list |
  334. +---------+---------------+------+
  335. POST /batches
  336. -------------
  337. Request Body
  338. ^^^^^^^^^^^^
  339. +----------------+--------------------------------------------------+-----------------+
  340. | name | description | type |
  341. +================+==================================================+=================+
  342. | proxyUser | The user to impersonate that will execute the job| string |
  343. +----------------+--------------------------------------------------+-----------------+
  344. | file | archive holding the file | path (required) |
  345. +----------------+--------------------------------------------------+-----------------+
  346. | args | command line arguments | list of strings |
  347. +----------------+--------------------------------------------------+-----------------+
  348. | className | application's java/spark main class | string |
  349. +----------------+--------------------------------------------------+-----------------+
  350. | jars | files to be placed on the java classpath | list of paths |
  351. +----------------+--------------------------------------------------+-----------------+
  352. | pyFiles | files to be placed on the PYTHONPATH | list of paths |
  353. +----------------+--------------------------------------------------+-----------------+
  354. | files | files to be placed in executor working directory | list of paths |
  355. +----------------+--------------------------------------------------+-----------------+
  356. | driverMemory | memory for driver (e.g. 1000M, 2G) | string |
  357. +----------------+--------------------------------------------------+-----------------+
  358. | driverCores | number of cores used by driver | int |
  359. +----------------+--------------------------------------------------+-----------------+
  360. | executorMemory | memory for executor (e.g. 1000M, 2G) | string |
  361. +----------------+--------------------------------------------------+-----------------+
  362. | executorCores | number of cores used by executor | int |
  363. +----------------+--------------------------------------------------+-----------------+
  364. | numExecutors | number of executor | int |
  365. +----------------+--------------------------------------------------+-----------------+
  366. | archives | Archives to be uncompressed (YARN mode only) | list of paths |
  367. +----------------+--------------------------------------------------+-----------------+
  368. Response Body
  369. ^^^^^^^^^^^^^
  370. The created `Batch`_ object.
  371. GET /batches/{batchId}
  372. ----------------------
  373. Request Parameters
  374. ^^^^^^^^^^^^^^^^^^
  375. +------+-----------------------------+------+
  376. | name | description | type |
  377. +======+=============================+======+
  378. | from | offset | int |
  379. +------+-----------------------------+------+
  380. | size | amount of batches to return | int |
  381. +------+-----------------------------+------+
  382. Response Body
  383. ^^^^^^^^^^^^^
  384. +-------+-----------------------------+-----------------+
  385. | name | description | type |
  386. +=======+=============================+=================+
  387. | id | The batch id | int |
  388. +-------+-----------------------------+-----------------+
  389. | state | The state of the batch | `batch`_ state |
  390. +-------+-----------------------------+-----------------+
  391. | log | The output of the batch job | list of strings |
  392. +-------+-----------------------------+-----------------+
  393. DELETE /batches/{batchId}
  394. -------------------------
  395. Kill the `Batch`_ job.
  396. GET /batches/{batchId}/logs
  397. ---------------------------
  398. Get the log lines from this batch.
  399. Request Parameters
  400. ^^^^^^^^^^^^^^^^^^
  401. +------+-----------------------------+------+
  402. | name | description | type |
  403. +======+=============================+======+
  404. | from | offset | int |
  405. +------+-----------------------------+------+
  406. | size | amount of batches to return | int |
  407. +------+-----------------------------+------+
  408. Response Body
  409. ^^^^^^^^^^^^^
  410. +------+-----------------------+-----------------+
  411. | name | description | type |
  412. +======+=======================+=================+
  413. | id | The batch id | int |
  414. +------+-----------------------+-----------------+
  415. | from | offset | int |
  416. +------+-----------------------+-----------------+
  417. | size | total amount of lines | int |
  418. +------+-----------------------+-----------------+
  419. | log | The log lines | list of strings |
  420. +------+-----------------------+-----------------+
  421. REST Objects
  422. ============
  423. Session
  424. -------
  425. Sessions represent an interactive shell.
  426. +----------------+--------------------------------------------------+----------------------------+
  427. | name | description | type |
  428. +================+==================================================+============================+
  429. | id | The session id | int |
  430. +----------------+--------------------------------------------------+----------------------------+
  431. | kind | session kind (spark, pyspark, or sparkr) | `session kind`_ (required) |
  432. +----------------+--------------------------------------------------+----------------------------+
  433. | log | The log lines | list of strings |
  434. +----------------+--------------------------------------------------+----------------------------+
  435. | state | The session state | string |
  436. +----------------+--------------------------------------------------+----------------------------+
  437. Session State
  438. ^^^^^^^^^^^^^
  439. +-------------+----------------------------------+
  440. | name | description |
  441. +=============+==================================+
  442. | not_started | session has not been started |
  443. +-------------+----------------------------------+
  444. | starting | session is starting |
  445. +-------------+----------------------------------+
  446. | idle | session is waiting for input |
  447. +-------------+----------------------------------+
  448. | busy | session is executing a statement |
  449. +-------------+----------------------------------+
  450. | error | session errored out |
  451. +-------------+----------------------------------+
  452. | dead | session has exited |
  453. +-------------+----------------------------------+
  454. Session Kind
  455. ^^^^^^^^^^^^
  456. +---------+----------------------------------+
  457. | name | description |
  458. +=========+==================================+
  459. | spark | interactive scala/spark session |
  460. +---------+----------------------------------+
  461. | pyspark | interactive python/spark session |
  462. +---------+----------------------------------+
  463. | sparkr | interactive R/spark session |
  464. +---------+----------------------------------+
  465. Statement
  466. ---------
  467. Statements represent the result of an execution statement.
  468. +--------+----------------------+---------------------+
  469. | name | description | type |
  470. +========+======================+=====================+
  471. | id | The statement id | integer |
  472. +--------+----------------------+---------------------+
  473. | state | The execution state | `statement state`_ |
  474. +--------+----------------------+---------------------+
  475. | output | The execution output | `statement output`_ |
  476. +--------+----------------------+---------------------+
  477. Statement State
  478. ^^^^^^^^^^^^^^^
  479. +-----------+----------------------------------+
  480. | name | description |
  481. +===========+==================================+
  482. | running | Statement is currently executing |
  483. +-----------+----------------------------------+
  484. | available | Statement has a ready response |
  485. +-----------+----------------------------------+
  486. | error | Statement failed |
  487. +-----------+----------------------------------+
  488. Statement Output
  489. ^^^^^^^^^^^^^^^^
  490. +-----------------+-------------------+----------------------------------+
  491. | name | description | type |
  492. +=================+===================+==================================+
  493. | status | execution status | string |
  494. +-----------------+-------------------+----------------------------------+
  495. | execution_count | a monotomically | integer |
  496. | | increasing number | |
  497. +-----------------+-------------------+----------------------------------+
  498. | data | statement output | an object mapping a mime type to |
  499. | | | the result. If the mime type is |
  500. | | | ``application/json``, the value |
  501. | | | will be a JSON value |
  502. +-----------------+-------------------+----------------------------------+
  503. Batch
  504. -----
  505. +----------------+--------------------------------------------------+----------------------------+
  506. | name | description | type |
  507. +================+==================================================+============================+
  508. | id | The session id | int |
  509. +----------------+--------------------------------------------------+----------------------------+
  510. | kind | session kind (spark, pyspark, or sparkr) | `session kind`_ (required) |
  511. +----------------+--------------------------------------------------+----------------------------+
  512. | log | The log lines | list of strings |
  513. +----------------+--------------------------------------------------+----------------------------+
  514. | state | The session state | string |
  515. +----------------+--------------------------------------------------+----------------------------+
  516. License
  517. =======
  518. Apache License, Version 2.0
  519. http://www.apache.org/licenses/LICENSE-2.0