README.rst 28 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751
  1. Welcome to Livy, the REST Spark Server
  2. ======================================
  3. Livy is an open source REST interface for interacting with Spark from anywhere. It supports executing snippets of code or programs in a Spark context that runs locally or in YARN.
  4. * Interactive Scala, Python and R shells
  5. * Batch submissions in Scala, Java, Python
  6. * Multi users can share the same server (impersonation support)
  7. * Can be used for submitting jobs from anywhere with REST
  8. * Does not require any code change to your programs
  9. The code is currently incubating in Hue but hopefully will eventually graduate in its top
  10. project. `Pull requests`_ are welcomed!
  11. .. _Pull requests: https://github.com/cloudera/hue/pulls
  12. Quick Start
  13. ===========
  14. Livy is used for powering the Spark snippets of the `Hadoop Notebook`_ of `Hue 3.8`_, which you can see the
  15. `implementation here`_.
  16. See the API documentation below and some curl examples:
  17. * `Interactive shells`_
  18. * `Batch jobs`_
  19. * `Shared RDDs`_
  20. .. _Interactive shells: http://gethue.com/how-to-use-the-livy-spark-rest-job-server-for-interactive-spark/
  21. .. _Batch jobs: http://gethue.com/how-to-use-the-livy-spark-rest-job-server-api-for-sharing-spark-rdds-and-contexts/
  22. .. _Shared RDDs: http://gethue.com/how-to-use-the-livy-spark-rest-job-server-api-for-submitting-batch-jar-python-and-streaming-spark-jobs/
  23. .. _Hadoop Notebook: http://gethue.com/new-notebook-application-for-spark-sql/
  24. .. _Hue 3.8: http://gethue.com/hue-3-8-with-an-oozie-editor-revamp-better-performances-improved-spark-ui-is-out/
  25. .. _implementation here: https://github.com/cloudera/hue/blob/master/apps/spark/src/spark/job_server_api.py
  26. Prerequisites
  27. =============
  28. To build/run Livy, you will need:
  29. Debian/Ubuntu:
  30. * mvn (from ``maven`` package or maven3 tarball)
  31. * openjdk-7-jdk (or Oracle Java7 jdk)
  32. * spark 1.5 from (from `Apache Spark tarball`_)
  33. * Python 2.6+
  34. * R 3.x
  35. Redhat/CentOS:
  36. * mvn (from ``maven`` package or maven3 tarball)
  37. * java-1.7.0-openjdk (or Oracle Java7 jdk)
  38. * spark 1.5 (from `Apache Spark tarball`_)
  39. * Python 2.6+
  40. * R 3.x
  41. MacOS:
  42. * Xcode command line tools
  43. * Oracle's JDK 1.7+
  44. * Maven (Homebrew)
  45. * apache-spark 1.5 (Homebrew)
  46. * Python 2.6+
  47. * R 3.x
  48. .. _Apache Spark Tarball: https://spark.apache.org/downloads.html
  49. Building Livy
  50. =============
  51. Livy is currently built by the `Hue Build System`_, it can also be built on
  52. it's own (aka without any other Hue dependency) with `Apache Maven`_. To build,
  53. checks out the code, go to the Livy directory and run:
  54. .. code:: shell
  55. git clone git@github.com:cloudera/hue.git
  56. cd hue
  57. .. code:: shell
  58. % cd apps/spark/java
  59. % mvn -DskipTests clean package
  60. .. _Hue Build System: https://github.com/cloudera/hue/#getting-started
  61. .. _Apache Maven: http://maven.apache.org
  62. Running Tests
  63. =============
  64. In order to run the Livy Tests, first follow the instructions in `Building
  65. Livy`_. Then run:
  66. .. code:: shell
  67. % export SPARK_HOME=/usr/lib/spark
  68. % export HADOOP_CONF_DIR=/etc/hadoop/conf
  69. % mvn test
  70. Running Livy
  71. ============
  72. In order to run Livy with local sessions, first export these variables:
  73. .. code:: shell
  74. % export SPARK_HOME=/usr/lib/spark
  75. % export HADOOP_CONF_DIR=/etc/hadoop/conf
  76. Then start the server with:
  77. .. code:: shell
  78. % ./bin/livy-server
  79. Or with YARN sessions by running:
  80. .. code:: shell
  81. % env \
  82. LIVY_SERVER_JAVA_OPTS="-Dlivy.server.session.factory=yarn" \
  83. CLASSPATH=`hadoop classpath` \
  84. $LIVY_HOME/bin/livy-server
  85. Livy Configuration
  86. ==================
  87. The properties of the server can be modified by copying
  88. `livy-defaults.conf.template`_ and renaming it ``conf/livy-defaults.conf``. The
  89. Livy configuration directory can be placed in an alternative directory by defining
  90. ``LIVY_CONF_DIR``.
  91. In particular the ``YARN mode`` (default is ``local`` process for development) can be set with:
  92. .. code:: shell
  93. livy.server.session.factory = yarn
  94. .. _livy-defaults.conf.template: https://github.com/cloudera/hue/blob/master/apps/spark/java/conf/livy-defaults.conf.template
  95. Spark Configuration
  96. ===================
  97. Livy's Spark sessions are configured through two mechanisms. First, is by way of the local
  98. `Spark configuration`_. Create, or modify the Spark configuration files as directed, and point
  99. Livy at this directory with:
  100. .. code:: shell
  101. % env \
  102. SPARK_CONF_DIR=... \
  103. $LIVY_HOME/bin/livy-server
  104. The second mechanism is by white listing `Spark configuration`_ options that can be set by the user
  105. creating a Spark session. This list can be created by copying
  106. `spark-user-configurable-options.template`_ to ``spark-user-configurable-options`` and listing
  107. the options the user may specify in the ``conf`` session field.
  108. *warning*: Be careful before enabling options. Some options may allow a malicious user to
  109. read files that are accessible by the Livy Server process user. Among other things, this might
  110. allow a user to access the Livy TLS private key, Kerberos tickets, or more.
  111. .. _Spark configuration: https://spark.apache.org/docs/latest/configuration.html
  112. .. _spark-user-configurable-options.template: https://github.com/cloudera/hue/blob/master/apps/spark/java/conf/spark-user-configurable-options.template
  113. Spark Example
  114. =============
  115. Now to see it in action by interacting with it in Python with the `Requests`_
  116. library. By default livy runs on port 8998 (which can be changed with the
  117. ``livy_server_port config`` option). We’ll start off with a Spark session that
  118. takes Scala code:
  119. .. code:: shell
  120. % sudo pip install requests
  121. .. code:: python
  122. >>> import json, pprint, requests, textwrap
  123. >>> host = 'http://localhost:8998'
  124. >>> data = {'kind': 'spark'}
  125. >>> headers = {'Content-Type': 'application/json'}
  126. >>> r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
  127. >>> r.json()
  128. {u'state': u'starting', u'id': 0, u’kind’: u’spark’}
  129. Once the session has completed starting up, it transitions to the idle state:
  130. .. code:: python
  131. >>> session_url = host + r.headers['location']
  132. >>> r = requests.get(session_url, headers=headers)
  133. >>> r.json()
  134. {u'state': u'idle', u'id': 0, u’kind’: u’spark’}
  135. Now we can execute Scala by passing in a simple JSON command:
  136. .. code:: python
  137. >>> statements_url = session_url + '/statements'
  138. >>> data = {'code': '1 + 1'}
  139. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  140. >>> r.json()
  141. {u'output': None, u'state': u'running', u'id': 0}
  142. If a statement takes longer than a few milliseconds to execute, Livy returns
  143. early and provides a URL that can be polled until it is complete:
  144. .. code:: python
  145. >>> statement_url = host + r.headers['location']
  146. >>> r = requests.get(statement_url, headers=headers)
  147. >>> pprint.pprint(r.json())
  148. [{u'id': 0,
  149. u'output': {u'data': {u'text/plain': u'res0: Int = 2'},
  150. u'execution_count': 0,
  151. u'status': u'ok'},
  152. u'state': u'available'}]
  153. That was a pretty simple example. More interesting is using Spark to estimate
  154. PI. This is from the `Spark Examples`_:
  155. .. code:: python
  156. >>> data = {
  157. ... 'code': textwrap.dedent("""\
  158. ... val NUM_SAMPLES = 100000;
  159. ... val count = sc.parallelize(1 to NUM_SAMPLES).map { i =>
  160. ... val x = Math.random();
  161. ... val y = Math.random();
  162. ... if (x*x + y*y < 1) 1 else 0
  163. ... }.reduce(_ + _);
  164. ... println(\"Pi is roughly \" + 4.0 * count / NUM_SAMPLES)
  165. ... """)
  166. ... }
  167. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  168. >>> pprint.pprint(r.json())
  169. {u'id': 1,
  170. u'output': {u'data': {u'text/plain': u'Pi is roughly 3.14004\nNUM_SAMPLES: Int = 100000\ncount: Int = 78501'},
  171. u'execution_count': 1,
  172. u'status': u'ok'},
  173. u'state': u'available'}
  174. Finally, lets close our session:
  175. .. code:: python
  176. >>> session_url = 'http://localhost:8998/sessions/0'
  177. >>> requests.delete(session_url, headers=headers)
  178. <Response [204]>
  179. .. _Requests: http://docs.python-requests.org/en/latest/
  180. .. _Spark Examples: https://spark.apache.org/examples.html
  181. PySpark Example
  182. ===============
  183. pyspark has the exact same API, just with a different initial command:
  184. .. code:: python
  185. >>> data = {'kind': 'pyspark'}
  186. >>> r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
  187. >>> r.json()
  188. {u'id': 1, u'state': u'idle'}
  189. The PI example from before then can be run as:
  190. .. code:: python
  191. >>> data = {
  192. ... 'code': textwrap.dedent("""\
  193. ... import random
  194. ... NUM_SAMPLES = 100000
  195. ... def sample(p):
  196. ... x, y = random.random(), random.random()
  197. ... return 1 if x*x + y*y < 1 else 0
  198. ...
  199. ... count = sc.parallelize(xrange(0, NUM_SAMPLES)).map(sample) \
  200. ... .reduce(lambda a, b: a + b)
  201. ... print "Pi is roughly %f" % (4.0 * count / NUM_SAMPLES)
  202. ... """)
  203. ... }
  204. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  205. >>> pprint.pprint(r.json())
  206. {u'id': 12,
  207. u'output': {u'data': {u'text/plain': u'Pi is roughly 3.136000'},
  208. u'execution_count': 12,
  209. u'status': u'ok'},
  210. u'state': u'running'}
  211. SparkR Example
  212. ==============
  213. SparkR also has the same API:
  214. .. code:: python
  215. >>> data = {'kind': 'sparkR'}
  216. >>> r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
  217. >>> r.json()
  218. {u'id': 1, u'state': u'idle'}
  219. The PI example from before then can be run as:
  220. .. code:: python
  221. >>> data = {
  222. ... 'code': textwrap.dedent("""\
  223. ... n <- 100000
  224. ... piFunc <- function(elem) {
  225. ... rands <- runif(n = 2, min = -1, max = 1)
  226. ... val <- ifelse((rands[1]^2 + rands[2]^2) < 1, 1.0, 0.0)
  227. ... val
  228. ... }
  229. ... piFuncVec <- function(elems) {
  230. ... message(length(elems))
  231. ... rands1 <- runif(n = length(elems), min = -1, max = 1)
  232. ... rands2 <- runif(n = length(elems), min = -1, max = 1)
  233. ... val <- ifelse((rands1^2 + rands2^2) < 1, 1.0, 0.0)
  234. ... sum(val)
  235. ... }
  236. ... rdd <- parallelize(sc, 1:n, slices)
  237. ... count <- reduce(lapplyPartition(rdd, piFuncVec), sum)
  238. ... cat("Pi is roughly", 4.0 * count / n, "\n")
  239. ... """)
  240. ... }
  241. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  242. >>> pprint.pprint(r.json())
  243. {u'id': 12,
  244. u'output': {u'data': {u'text/plain': u'Pi is roughly 3.136000'},
  245. u'execution_count': 12,
  246. u'status': u'ok'},
  247. u'state': u'running'}
  248. Community
  249. =========
  250. * User group: http://groups.google.com/a/cloudera.org/group/hue-user
  251. * Umbrella Jira: https://issues.cloudera.org/browse/HUE-2588
  252. * Pull requests: https://github.com/cloudera/hue/pulls
  253. REST API
  254. ========
  255. GET /sessions
  256. -------------
  257. Returns all the active interactive sessions.
  258. Response Body
  259. ^^^^^^^^^^^^^
  260. +----------+-----------------+------+
  261. | name | description | type |
  262. +==========+=================+======+
  263. | sessions | `session`_ list | list |
  264. +----------+-----------------+------+
  265. POST /sessions
  266. --------------
  267. Creates a new interative Scala, Python or R shell in the cluster.
  268. Request Body
  269. ^^^^^^^^^^^^
  270. +----------------+--------------------------------------------------------------------------------+-----------------+
  271. | name | description | type |
  272. +================+================================================================================+=================+
  273. | kind | The session kind (required) | `session kind`_ |
  274. +----------------+--------------------------------------------------------------------------------+-----------------+
  275. | proxyUser | The user to impersonate that will run this session (e.g. bob) | string |
  276. +----------------+--------------------------------------------------------------------------------+-----------------+
  277. | jars | Files to be placed on the java classpath | list of paths |
  278. +----------------+--------------------------------------------------------------------------------+-----------------+
  279. | pyFiles | Files to be placed on the PYTHONPATH | list of paths |
  280. +----------------+--------------------------------------------------------------------------------+-----------------+
  281. | files | Files to be placed in executor working directory | list of paths |
  282. +----------------+--------------------------------------------------------------------------------+-----------------+
  283. | driverMemory | Memory for driver (e.g. 1000M, 2G) | string |
  284. +----------------+--------------------------------------------------------------------------------+-----------------+
  285. | driverCores | Number of cores used by driver (YARN mode only) | int |
  286. +----------------+--------------------------------------------------------------------------------+-----------------+
  287. | executorMemory | Memory for executor (e.g. 1000M, 2G) | string |
  288. +----------------+--------------------------------------------------------------------------------+-----------------+
  289. | executorCores | Number of cores used by executor | int |
  290. +----------------+--------------------------------------------------------------------------------+-----------------+
  291. | numExecutors | Number of executors (YARN mode only) | int |
  292. +----------------+--------------------------------------------------------------------------------+-----------------+
  293. | archives | Archives to be uncompressed in the executor working directory (YARN mode only) | list of paths |
  294. +----------------+--------------------------------------------------------------------------------+-----------------+
  295. | queue | The YARN queue to submit too (YARN mode only) | string |
  296. +----------------+--------------------------------------------------------------------------------+-----------------+
  297. | name | Name of the application | string |
  298. +----------------+--------------------------------------------------------------------------------+-----------------+
  299. | conf | Spark configuration property | Map of key=val |
  300. +----------------+--------------------------------------------------------------------------------+-----------------+
  301. Response Body
  302. ^^^^^^^^^^^^^
  303. The created `Session`_.
  304. GET /sessions/{sessionId}
  305. -------------------------
  306. Return the session information
  307. Response
  308. ^^^^^^^^
  309. The `Session`_.
  310. DELETE /sessions/{sessionId}
  311. ----------------------------
  312. Kill the `Session`_ job.
  313. GET /sessions/{sessionId}/logs
  314. ------------------------------
  315. Get the log lines from this session.
  316. Request Parameters
  317. ^^^^^^^^^^^^^^^^^^
  318. +------+-----------------------------+------+
  319. | name | description | type |
  320. +======+=============================+======+
  321. | from | offset | int |
  322. +------+-----------------------------+------+
  323. | size | amount of batches to return | int |
  324. +------+-----------------------------+------+
  325. Response Body
  326. ^^^^^^^^^^^^^
  327. +------+-----------------------+-----------------+
  328. | name | description | type |
  329. +======+=======================+=================+
  330. | id | The session id | int |
  331. +------+-----------------------+-----------------+
  332. | from | offset | int |
  333. +------+-----------------------+-----------------+
  334. | size | total amount of lines | int |
  335. +------+-----------------------+-----------------+
  336. | log | The log lines | list of strings |
  337. +------+-----------------------+-----------------+
  338. GET /sessions/{sessionId}/statements
  339. ------------------------------------
  340. Return all the statements in a session.
  341. Response Body
  342. ^^^^^^^^^^^^^
  343. +------------+-------------------+------+
  344. | name | description | type |
  345. +============+===================+======+
  346. | statements | `statement`_ list | list |
  347. +------------+-------------------+------+
  348. POST /sessions/{sessionId}/statements
  349. -------------------------------------
  350. Execute a statement in a session.
  351. Request Body
  352. ^^^^^^^^^^^^
  353. +------+---------------------+--------+
  354. | name | description | type |
  355. +======+=====================+========+
  356. | code | The code to execute | string |
  357. +------+---------------------+--------+
  358. Response Body
  359. ^^^^^^^^^^^^^
  360. The `statement`_ object.
  361. GET /batches
  362. ------------
  363. Return all the active batch jobs.
  364. Response Body
  365. ^^^^^^^^^^^^^
  366. +---------+---------------+------+
  367. | name | description | type |
  368. +=========+===============+======+
  369. | batches | `batch`_ list | list |
  370. +---------+---------------+------+
  371. POST /batches
  372. -------------
  373. Request Body
  374. ^^^^^^^^^^^^
  375. +----------------+---------------------------------------------------+-----------------+
  376. | name | description | type |
  377. +================+===================================================+=================+
  378. | proxyUser | The user to impersonate that will execute the job | string |
  379. +----------------+---------------------------------------------------+-----------------+
  380. | file | Archive holding the file | path (required) |
  381. +----------------+---------------------------------------------------+-----------------+
  382. | args | Command line arguments | list of strings |
  383. +----------------+---------------------------------------------------+-----------------+
  384. | className | Application's java/spark main class | string |
  385. +----------------+---------------------------------------------------+-----------------+
  386. | jars | Files to be placed on the java classpath | list of paths |
  387. +----------------+---------------------------------------------------+-----------------+
  388. | pyFiles | Files to be placed on the PYTHONPATH | list of paths |
  389. +----------------+---------------------------------------------------+-----------------+
  390. | files | Files to be placed in executor working directory | list of paths |
  391. +----------------+---------------------------------------------------+-----------------+
  392. | driverMemory | Memory for driver (e.g. 1000M, 2G) | string |
  393. +----------------+---------------------------------------------------+-----------------+
  394. | driverCores | Number of cores used by driver | int |
  395. +----------------+---------------------------------------------------+-----------------+
  396. | executorMemory | Memory for executor (e.g. 1000M, 2G) | string |
  397. +----------------+---------------------------------------------------+-----------------+
  398. | executorCores | Number of cores used by executor | int |
  399. +----------------+---------------------------------------------------+-----------------+
  400. | numExecutors | Number of executor | int |
  401. +----------------+---------------------------------------------------+-----------------+
  402. | archives | Archives to be uncompressed (YARN mode only) | list of paths |
  403. +----------------+---------------------------------------------------+-----------------+
  404. | queue | The YARN queue to submit too (YARN mode only) | string |
  405. +----------------+---------------------------------------------------+-----------------+
  406. | name | Name of the application | string |
  407. +----------------+---------------------------------------------------+-----------------+
  408. | conf | Spark configuration property | Map of key=val |
  409. +----------------+---------------------------------------------------+-----------------+
  410. Response Body
  411. ^^^^^^^^^^^^^
  412. The created `Batch`_ object.
  413. GET /batches/{batchId}
  414. ----------------------
  415. Request Parameters
  416. ^^^^^^^^^^^^^^^^^^
  417. +------+-----------------------------+------+
  418. | name | description | type |
  419. +======+=============================+======+
  420. | from | offset | int |
  421. +------+-----------------------------+------+
  422. | size | amount of batches to return | int |
  423. +------+-----------------------------+------+
  424. Response Body
  425. ^^^^^^^^^^^^^
  426. +-------+-----------------------------+-----------------+
  427. | name | description | type |
  428. +=======+=============================+=================+
  429. | id | The batch id | int |
  430. +-------+-----------------------------+-----------------+
  431. | state | The state of the batch | `batch`_ state |
  432. +-------+-----------------------------+-----------------+
  433. | log | The output of the batch job | list of strings |
  434. +-------+-----------------------------+-----------------+
  435. DELETE /batches/{batchId}
  436. -------------------------
  437. Kill the `Batch`_ job.
  438. GET /batches/{batchId}/logs
  439. ---------------------------
  440. Get the log lines from this batch.
  441. Request Parameters
  442. ^^^^^^^^^^^^^^^^^^
  443. +------+-----------------------------+------+
  444. | name | description | type |
  445. +======+=============================+======+
  446. | from | offset | int |
  447. +------+-----------------------------+------+
  448. | size | amount of batches to return | int |
  449. +------+-----------------------------+------+
  450. Response Body
  451. ^^^^^^^^^^^^^
  452. +------+-----------------------+-----------------+
  453. | name | description | type |
  454. +======+=======================+=================+
  455. | id | The batch id | int |
  456. +------+-----------------------+-----------------+
  457. | from | offset | int |
  458. +------+-----------------------+-----------------+
  459. | size | total amount of lines | int |
  460. +------+-----------------------+-----------------+
  461. | log | The log lines | list of strings |
  462. +------+-----------------------+-----------------+
  463. REST Objects
  464. ============
  465. Session
  466. -------
  467. Sessions represent an interactive shell.
  468. +----------------+--------------------------------------------------+----------------------------+
  469. | name | description | type |
  470. +================+==================================================+============================+
  471. | id | The session id | int |
  472. +----------------+--------------------------------------------------+----------------------------+
  473. | kind | session kind (spark, pyspark, or sparkr) | `session kind`_ (required) |
  474. +----------------+--------------------------------------------------+----------------------------+
  475. | log | The log lines | list of strings |
  476. +----------------+--------------------------------------------------+----------------------------+
  477. | state | The session state | string |
  478. +----------------+--------------------------------------------------+----------------------------+
  479. Session State
  480. ^^^^^^^^^^^^^
  481. +-------------+----------------------------------+
  482. | name | description |
  483. +=============+==================================+
  484. | not_started | session has not been started |
  485. +-------------+----------------------------------+
  486. | starting | session is starting |
  487. +-------------+----------------------------------+
  488. | idle | session is waiting for input |
  489. +-------------+----------------------------------+
  490. | busy | session is executing a statement |
  491. +-------------+----------------------------------+
  492. | error | session errored out |
  493. +-------------+----------------------------------+
  494. | dead | session has exited |
  495. +-------------+----------------------------------+
  496. Session Kind
  497. ^^^^^^^^^^^^
  498. +---------+----------------------------------+
  499. | name | description |
  500. +=========+==================================+
  501. | spark | interactive scala/spark session |
  502. +---------+----------------------------------+
  503. | pyspark | interactive python/spark session |
  504. +---------+----------------------------------+
  505. | sparkr | interactive R/spark session |
  506. +---------+----------------------------------+
  507. Statement
  508. ---------
  509. Statements represent the result of an execution statement.
  510. +--------+----------------------+---------------------+
  511. | name | description | type |
  512. +========+======================+=====================+
  513. | id | The statement id | integer |
  514. +--------+----------------------+---------------------+
  515. | state | The execution state | `statement state`_ |
  516. +--------+----------------------+---------------------+
  517. | output | The execution output | `statement output`_ |
  518. +--------+----------------------+---------------------+
  519. Statement State
  520. ^^^^^^^^^^^^^^^
  521. +-----------+----------------------------------+
  522. | name | description |
  523. +===========+==================================+
  524. | running | Statement is currently executing |
  525. +-----------+----------------------------------+
  526. | available | Statement has a ready response |
  527. +-----------+----------------------------------+
  528. | error | Statement failed |
  529. +-----------+----------------------------------+
  530. Statement Output
  531. ^^^^^^^^^^^^^^^^
  532. +-----------------+-------------------+----------------------------------+
  533. | name | description | type |
  534. +=================+===================+==================================+
  535. | status | execution status | string |
  536. +-----------------+-------------------+----------------------------------+
  537. | execution_count | a monotomically | integer |
  538. | | increasing number | |
  539. +-----------------+-------------------+----------------------------------+
  540. | data | statement output | an object mapping a mime type to |
  541. | | | the result. If the mime type is |
  542. | | | ``application/json``, the value |
  543. | | | will be a JSON value |
  544. +-----------------+-------------------+----------------------------------+
  545. Batch
  546. -----
  547. +----------------+--------------------------------------------------+----------------------------+
  548. | name | description | type |
  549. +================+==================================================+============================+
  550. | id | The session id | int |
  551. +----------------+--------------------------------------------------+----------------------------+
  552. | kind | session kind (spark, pyspark, or sparkr) | `session kind`_ (required) |
  553. +----------------+--------------------------------------------------+----------------------------+
  554. | log | The log lines | list of strings |
  555. +----------------+--------------------------------------------------+----------------------------+
  556. | state | The session state | string |
  557. +----------------+--------------------------------------------------+----------------------------+
  558. License
  559. =======
  560. Apache License, Version 2.0
  561. http://www.apache.org/licenses/LICENSE-2.0