README.rst 30 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725
  1. Welcome to Livy, the REST Spark Server
  2. ======================================
  3. Livy is an open source REST interface for interacting with Spark from anywhere. It supports executing snippets of code or programs in a Spark context that runs locally or in YARN.
  4. * Interactive Scala, Python and R shells
  5. * Batch submissions in Scala, Java, Python
  6. * Multi users can share the same server (impersonation support)
  7. * Can be used for submitting jobs from anywhere with REST
  8. * Does not require any code change to your programs
  9. The code is currently incubating in Hue but hopefully will eventually graduate in its top project. `Pull requests` are welcomed!
  10. Livy is used for powering the `Spark Notebook`_ of `Hue 3.8`_, which you can see the
  11. `implementation here`_.
  12. .. _Pull requests: https://github.com/cloudera/hue/pulls
  13. .. _Spark Notebook: http://gethue.com/new-notebook-application-for-spark-sql/
  14. .. _Hue 3.8: http://gethue.com/hue-3-8-with-an-oozie-editor-revamp-better-performances-improved-spark-ui-is-out/
  15. .. _implementation here: https://github.com/cloudera/hue/blob/master/apps/spark/src/spark/job_server_api.py
  16. Prerequisites
  17. =============
  18. To build/run Livy, you will need:
  19. Debian/Ubuntu:
  20. * mvn (from ``maven`` package or maven3 tarball)
  21. * openjdk-7-jdk (or Oracle Java7 jdk)
  22. * spark 1.5 from (from `Apache Spark tarball`_)
  23. * Python 2.6+
  24. * R 3.x
  25. Redhat/CentOS:
  26. * mvn (from ``maven`` package or maven3 tarball)
  27. * java-1.7.0-openjdk (or Oracle Java7 jdk)
  28. * spark 1.5 (from `Apache Spark tarball`_)
  29. * Python 2.6+
  30. * R 3.x
  31. MacOS:
  32. * Xcode command line tools
  33. * Oracle's JDK 1.7+
  34. * Maven (Homebrew)
  35. * apache-spark 1.5 (Homebrew)
  36. * Python 2.6+
  37. * R 3.x
  38. .. _Apache Spark Tarball: https://spark.apache.org/downloads.html
  39. Building Livy
  40. =============
  41. Livy is currently built by the `Hue Build System`_, it can also be built on
  42. it's own (aka without any other Hue dependency) with `Apache Maven`_. To build,
  43. run:
  44. .. code:: shell
  45. % cd $HUE_HOME/apps/spark/java
  46. % mvn -DskipTests clean package
  47. .. _Hue Build System: https://github.com/cloudera/hue/#getting-started
  48. .. _Apache Maven: http://maven.apache.org
  49. Running Tests
  50. =============
  51. In order to run the Livy Tests, first follow the instructions in `Building
  52. Livy`_. Then run:
  53. .. code:: shell
  54. % cd $HUE_HOME/apps/spark/java
  55. % export SPARK_HOME=/usr/lib/spark
  56. % mvn test
  57. Running Livy
  58. ============
  59. In order to run Livy with local sessions, start the server with:
  60. .. code:: shell
  61. % export SPARK_HOME=/usr/lib/spark
  62. % ./bin/livy-server
  63. Or with YARN sessions by running:
  64. .. code:: shell
  65. % export SPARK_HOME=/usr/lib/spark
  66. % env \
  67. LIVY_SERVER_JAVA_OPTS="-Dlivy.server.session.factory=yarn" \
  68. CLASSPATH=`hadoop classpath` \
  69. ./bin/livy-server
  70. Livy Configuration
  71. ==================
  72. The properties of the server can be modified by copying <https://github.com/cloudera/hue/blob/master/apps/spark/java/conf/livy-defaults.conf.tmpl>
  73. and renaming it ``livy-defaults.conf``.
  74. In particular the ``YARN mode`` (default is ``local`` process for development) can be set with:
  75. .. code:: shell
  76. livy.server.session.factory = yarn
  77. Spark Example
  78. =============
  79. Now to see it in action by interacting with it in Python with the `Requests`_
  80. library. By default livy runs on port 8998 (which can be changed with the
  81. ``livy_server_port config`` option). We’ll start off with a Spark session that
  82. takes Scala code:
  83. .. code:: python
  84. >>> import json, pprint, requests, textwrap
  85. >>> host = 'http://localhost:8998'
  86. >>> data = {'kind': 'spark'}
  87. >>> headers = {'Content-Type': 'application/json'}
  88. >>> r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
  89. >>> r.json()
  90. {u'state': u'starting', u'id': 0, u’kind’: u’spark’}
  91. Once the session has completed starting up, it transitions to the idle state:
  92. .. code:: python
  93. >>> session_url = host + r.headers['location']
  94. >>> r = requests.get(session_url, headers=headers)
  95. >>> r.json()
  96. {u'state': u'idle', u'id': 0, u’kind’: u’spark’}
  97. Now we can execute Scala by passing in a simple JSON command:
  98. .. code:: python
  99. >>> statements_url = session_url + '/statements'
  100. >>> data = {'code': '1 + 1'}
  101. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  102. >>> r.json()
  103. {u'output': None, u'state': u'running', u'id': 0}
  104. If a statement takes longer than a few milliseconds to execute, Livy returns
  105. early and provides a URL that can be polled until it is complete:
  106. .. code:: python
  107. >>> statement_url = host + r.headers['location']
  108. >>> r = requests.get(statement_url, headers=headers)
  109. >>> pprint.pprint(r.json())
  110. [{u'id': 0,
  111. u'output': {u'data': {u'text/plain': u'res0: Int = 2'},
  112. u'execution_count': 0,
  113. u'status': u'ok'},
  114. u'state': u'available'}]
  115. That was a pretty simple example. More interesting is using Spark to estimate
  116. PI. This is from the `Spark Examples`_:
  117. .. code:: python
  118. >>> data = {
  119. ... 'code': textwrap.dedent("""\
  120. ... val NUM_SAMPLES = 100000;
  121. ... val count = sc.parallelize(1 to NUM_SAMPLES).map { i =>
  122. ... val x = Math.random();
  123. ... val y = Math.random();
  124. ... if (x*x + y*y < 1) 1 else 0
  125. ... }.reduce(_ + _);
  126. ... println(\"Pi is roughly \" + 4.0 * count / NUM_SAMPLES)
  127. ... """)
  128. ... }
  129. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  130. >>> pprint.pprint(r.json())
  131. {u'id': 1,
  132. u'output': {u'data': {u'text/plain': u'Pi is roughly 3.14004\nNUM_SAMPLES: Int = 100000\ncount: Int = 78501'},
  133. u'execution_count': 1,
  134. u'status': u'ok'},
  135. u'state': u'available'}
  136. Finally, lets close our session:
  137. .. code:: python
  138. >>> session_url = 'http://localhost:8998/sessions/0'
  139. >>> requests.delete(session_url, headers=headers)
  140. <Response [204]>
  141. .. _Requests: http://docs.python-requests.org/en/latest/
  142. .. _Spark Examples: https://spark.apache.org/examples.html
  143. PySpark Example
  144. ===============
  145. pyspark has the exact same API, just with a different initial command:
  146. .. code:: python
  147. >>> data = {'kind': 'pyspark'}
  148. >>> r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
  149. >>> r.json()
  150. {u'id': 1, u'state': u'idle'}
  151. The PI example from before then can be run as:
  152. .. code:: python
  153. >>> data = {
  154. ... 'code': textwrap.dedent("""\
  155. ... import random
  156. ... NUM_SAMPLES = 100000
  157. ... def sample(p):
  158. ... x, y = random.random(), random.random()
  159. ... return 1 if x*x + y*y < 1 else 0
  160. ...
  161. ... count = sc.parallelize(xrange(0, NUM_SAMPLES)).map(sample) \
  162. ... .reduce(lambda a, b: a + b)
  163. ... print "Pi is roughly %f" % (4.0 * count / NUM_SAMPLES)
  164. ... """)
  165. ... }
  166. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  167. >>> pprint.pprint(r.json())
  168. {u'id': 12,
  169. u'output': {u'data': {u'text/plain': u'Pi is roughly 3.136000'},
  170. u'execution_count': 12,
  171. u'status': u'ok'},
  172. u'state': u'running'}
  173. SparkR Example
  174. ==============
  175. SparkR also has the same API:
  176. .. code:: python
  177. >>> data = {'kind': 'sparkR'}
  178. >>> r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
  179. >>> r.json()
  180. {u'id': 1, u'state': u'idle'}
  181. The PI example from before then can be run as:
  182. .. code:: python
  183. >>> data = {
  184. ... 'code': textwrap.dedent("""\
  185. ... n <- 100000
  186. ... piFunc <- function(elem) {
  187. ... rands <- runif(n = 2, min = -1, max = 1)
  188. ... val <- ifelse((rands[1]^2 + rands[2]^2) < 1, 1.0, 0.0)
  189. ... val
  190. ... }
  191. ... piFuncVec <- function(elems) {
  192. ... message(length(elems))
  193. ... rands1 <- runif(n = length(elems), min = -1, max = 1)
  194. ... rands2 <- runif(n = length(elems), min = -1, max = 1)
  195. ... val <- ifelse((rands1^2 + rands2^2) < 1, 1.0, 0.0)
  196. ... sum(val)
  197. ... }
  198. ... rdd <- parallelize(sc, 1:n, slices)
  199. ... count <- reduce(lapplyPartition(rdd, piFuncVec), sum)
  200. ... cat("Pi is roughly", 4.0 * count / n, "\n")
  201. ... """)
  202. ... }
  203. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  204. >>> pprint.pprint(r.json())
  205. {u'id': 12,
  206. u'output': {u'data': {u'text/plain': u'Pi is roughly 3.136000'},
  207. u'execution_count': 12,
  208. u'status': u'ok'},
  209. u'state': u'running'}
  210. Community
  211. =========
  212. * User group: http://groups.google.com/a/cloudera.org/group/hue-user
  213. * Jira: https://issues.cloudera.org/browse/HUE-2588
  214. * Reviews: https://review.cloudera.org/dashboard/?view=to-group&group=hue (repo 'hue-rw')
  215. REST API
  216. ========
  217. GET /sessions
  218. -------------
  219. Returns all the active interactive sessions.
  220. Response Body
  221. ^^^^^^^^^^^^^
  222. +----------+-----------------+------+
  223. | name | description | type |
  224. +==========+=================+======+
  225. | sessions | `session`_ list | list |
  226. +----------+-----------------+------+
  227. POST /sessions
  228. --------------
  229. Creates a new interative Scala or Python shell in the cluster.
  230. Request Body
  231. ^^^^^^^^^^^^
  232. +----------------+--------------------------------------------------+----------------------------+
  233. | name | description | type |
  234. +================+==================================================+============================+
  235. | id | The session id | int |
  236. +----------------+--------------------------------------------------+----------------------------+
  237. | kind | session kind (spark, pyspark, or sparkr) | `session kind`_ (required) |
  238. +----------------+--------------------------------------------------+----------------------------+
  239. | log | The log lines | list of strings |
  240. +----------------+--------------------------------------------------+----------------------------+
  241. | state | The session state | string |
  242. +----------------+--------------------------------------------------+----------------------------+
  243. | file | archive holding the file | path (required) |
  244. +----------------+--------------------------------------------------+----------------------------+
  245. | args | command line arguments | list of strings |
  246. +----------------+--------------------------------------------------+----------------------------+
  247. | className | application's java/spark main class | string |
  248. +----------------+--------------------------------------------------+----------------------------+
  249. | jars | files to be placed on the java classpath | list of paths |
  250. +----------------+--------------------------------------------------+----------------------------+
  251. | pyFiles | files to be placed on the PYTHONPATH | list of paths |
  252. +----------------+--------------------------------------------------+----------------------------+
  253. | files | files to be placed in executor working directory | list of paths |
  254. +----------------+--------------------------------------------------+----------------------------+
  255. | driverMemory | memory for driver | string |
  256. +----------------+--------------------------------------------------+----------------------------+
  257. | driverCores | number of cores used by driver | int |
  258. +----------------+--------------------------------------------------+----------------------------+
  259. | executorMemory | memory for executor | string |
  260. +----------------+--------------------------------------------------+----------------------------+
  261. | executorCores | number of cores used by executor | int |
  262. +----------------+--------------------------------------------------+----------------------------+
  263. | numExecutors | number of executor | int |
  264. +----------------+--------------------------------------------------+----------------------------+
  265. | archives | | list of paths |
  266. +----------------+--------------------------------------------------+----------------------------+
  267. Response Body
  268. ^^^^^^^^^^^^^
  269. The created `Session`_.
  270. GET /sessions/{sessionId}
  271. -------------------------
  272. Return the session information
  273. Response
  274. ^^^^^^^^
  275. The `Session`_.
  276. DELETE /sessions/{sessionId}
  277. -------------------------
  278. Kill the `Session`_ job.
  279. GET /sessions/{sessionId}/logs
  280. ---------------------------
  281. Get the log lines from this session.
  282. Request Parameters
  283. ^^^^^^^^^^^^^^^^^^
  284. +------+-----------------------------+------+
  285. | name | description | type |
  286. +======+=============================+======+
  287. | from | offset | int |
  288. +------+-----------------------------+------+
  289. | size | amount of batches to return | int |
  290. +------+-----------------------------+------+
  291. Response Body
  292. ^^^^^^^^^^^^^
  293. +------+-----------------------+-----------------+
  294. | name | description | type |
  295. +======+=======================+=================+
  296. | id | The session id | int |
  297. +------+-----------------------+-----------------+
  298. | from | offset | int |
  299. +------+-----------------------+-----------------+
  300. | size | total amount of lines | int |
  301. +------+-----------------------+-----------------+
  302. | log | The log lines | list of strings |
  303. +------+-----------------------+-----------------+
  304. GET /sessions/{sessionId}/statements
  305. ------------------------------------
  306. Return all the statements in a session.
  307. Response Body
  308. ^^^^^^^^^^^^^
  309. +------------+-------------------+------+
  310. | name | description | type |
  311. +============+===================+======+
  312. | statements | `statement`_ list | list |
  313. +------------+-------------------+------+
  314. POST /sessions/{sessionId}/statements
  315. -------------------------------------
  316. Execute a statement in a session.
  317. Request Body
  318. ^^^^^^^^^^^^
  319. +------+---------------------+--------+
  320. | name | description | type |
  321. +======+=====================+========+
  322. | code | The code to execute | string |
  323. +------+---------------------+--------+
  324. Response Body
  325. ^^^^^^^^^^^^^
  326. The `statement`_ object.
  327. GET /batches
  328. ------------
  329. Return all the active batch jobs.
  330. Response Body
  331. ^^^^^^^^^^^^^
  332. +---------+---------------+------+
  333. | name | description | type |
  334. +=========+===============+======+
  335. | batches | `batch`_ list | list |
  336. +---------+---------------+------+
  337. POST /batches
  338. -------------
  339. Request Body
  340. ^^^^^^^^^^^^
  341. +----------------+--------------------------------------------------+-----------------+
  342. | name | description | type |
  343. +================+==================================================+=================+
  344. | id | The session id | int |
  345. +----------------+--------------------------------------------------+-----------------+
  346. | log | The log lines | list of strings |
  347. +----------------+--------------------------------------------------+-----------------+
  348. | state | The session state | string |
  349. +----------------+--------------------------------------------------+-----------------+
  350. | file | archive holding the file | path (required) |
  351. +----------------+--------------------------------------------------+-----------------+
  352. | args | command line arguments | list of strings |
  353. +----------------+--------------------------------------------------+-----------------+
  354. | className | application's java/spark main class | string |
  355. +----------------+--------------------------------------------------+-----------------+
  356. | jars | files to be placed on the java classpath | list of paths |
  357. +----------------+--------------------------------------------------+-----------------+
  358. | pyFiles | files to be placed on the PYTHONPATH | list of paths |
  359. +----------------+--------------------------------------------------+-----------------+
  360. | files | files to be placed in executor working directory | list of paths |
  361. +----------------+--------------------------------------------------+-----------------+
  362. | driverMemory | memory for driver | string |
  363. +----------------+--------------------------------------------------+-----------------+
  364. | driverCores | number of cores used by driver | int |
  365. +----------------+--------------------------------------------------+-----------------+
  366. | executorMemory | memory for executor | string |
  367. +----------------+--------------------------------------------------+-----------------+
  368. | executorCores | number of cores used by executor | int |
  369. +----------------+--------------------------------------------------+-----------------+
  370. | numExecutors | number of executor | int |
  371. +----------------+--------------------------------------------------+-----------------+
  372. | archives | | list of paths |
  373. +----------------+--------------------------------------------------+-----------------+
  374. Response Body
  375. ^^^^^^^^^^^^^
  376. The created `Batch`_ object.
  377. GET /batches/{batchId}
  378. ----------------------
  379. Request Parameters
  380. ^^^^^^^^^^^^^^^^^^
  381. +------+-----------------------------+------+
  382. | name | description | type |
  383. +======+=============================+======+
  384. | from | offset | int |
  385. +------+-----------------------------+------+
  386. | size | amount of batches to return | int |
  387. +------+-----------------------------+------+
  388. Response Body
  389. ^^^^^^^^^^^^^
  390. +-------+-----------------------------+-----------------+
  391. | name | description | type |
  392. +=======+=============================+=================+
  393. | id | The batch id | int |
  394. +-------+-----------------------------+-----------------+
  395. | state | The state of the batch | `batch`_ state |
  396. +-------+-----------------------------+-----------------+
  397. | log | The output of the batch job | list of strings |
  398. +-------+-----------------------------+-----------------+
  399. DELETE /batches/{batchId}
  400. -------------------------
  401. Kill the `Batch`_ job.
  402. GET /batches/{batchId}/logs
  403. ---------------------------
  404. Get the log lines from this batch.
  405. Request Parameters
  406. ^^^^^^^^^^^^^^^^^^
  407. +------+-----------------------------+------+
  408. | name | description | type |
  409. +======+=============================+======+
  410. | from | offset | int |
  411. +------+-----------------------------+------+
  412. | size | amount of batches to return | int |
  413. +------+-----------------------------+------+
  414. Response Body
  415. ^^^^^^^^^^^^^
  416. +------+-----------------------+-----------------+
  417. | name | description | type |
  418. +======+=======================+=================+
  419. | id | The batch id | int |
  420. +------+-----------------------+-----------------+
  421. | from | offset | int |
  422. +------+-----------------------+-----------------+
  423. | size | total amount of lines | int |
  424. +------+-----------------------+-----------------+
  425. | log | The log lines | list of strings |
  426. +------+-----------------------+-----------------+
  427. REST Objects
  428. ============
  429. Session
  430. -------
  431. Sessions represent an interactive shell.
  432. +----------------+--------------------------------------------------------------------------------+------------------+
  433. | name | description | type |
  434. +================+================================================================================+==================+
  435. | id | The session id | string |
  436. +----------------+--------------------------------------------------------------------------------+------------------+
  437. | state | The state of the session | `session state`_ |
  438. +----------------+--------------------------------------------------------------------------------+------------------+
  439. | kind | The session kind | `session kind`_ |
  440. +----------------+--------------------------------------------------------------------------------+------------------+
  441. | proxyUser | The user running this session | optional string |
  442. +----------------+--------------------------------------------------------------------------------+------------------+
  443. | jars | files to be placed on the java classpath | list of paths |
  444. +----------------+--------------------------------------------------------------------------------+------------------+
  445. | pyFiles | files to be placed on the PYTHONPATH | list of paths |
  446. +----------------+--------------------------------------------------------------------------------+------------------+
  447. | files | files to be placed in executor working directory | list of paths |
  448. +----------------+--------------------------------------------------------------------------------+------------------+
  449. | driverMemory | memory for driver | string |
  450. +----------------+--------------------------------------------------------------------------------+------------------+
  451. | driverCores | number of cores used by driver (YARN mode only) | int |
  452. +----------------+--------------------------------------------------------------------------------+------------------+
  453. | executorMemory | memory for executor | string |
  454. +----------------+--------------------------------------------------------------------------------+------------------+
  455. | executorCores | number of cores used by executor | int |
  456. +----------------+--------------------------------------------------------------------------------+------------------+
  457. | numExecutors | number of executors (YARN mode only) | int |
  458. +----------------+--------------------------------------------------------------------------------+------------------+
  459. | archives | Archives to be uncompressed in the executor working directory (YARN mode only) | list of paths |
  460. +----------------+--------------------------------------------------------------------------------+------------------+
  461. Session State
  462. ^^^^^^^^^^^^^
  463. +-------------+----------------------------------+
  464. | name | description |
  465. +=============+==================================+
  466. | not_started | session has not been started |
  467. +-------------+----------------------------------+
  468. | starting | session is starting |
  469. +-------------+----------------------------------+
  470. | idle | session is waiting for input |
  471. +-------------+----------------------------------+
  472. | busy | session is executing a statement |
  473. +-------------+----------------------------------+
  474. | error | session errored out |
  475. +-------------+----------------------------------+
  476. | dead | session has exited |
  477. +-------------+----------------------------------+
  478. Session Kind
  479. ^^^^^^^^^^^^
  480. +---------+----------------------------------+
  481. | name | description |
  482. +=========+==================================+
  483. | spark | interactive scala/spark session |
  484. +---------+----------------------------------+
  485. | pyspark | interactive python/spark session |
  486. +---------+----------------------------------+
  487. Statement
  488. ---------
  489. Statements represent the result of an execution statement.
  490. +--------+----------------------+---------------------+
  491. | name | description | type |
  492. +========+======================+=====================+
  493. | id | The statement id | integer |
  494. +--------+----------------------+---------------------+
  495. | state | The execution state | `statement state`_ |
  496. +--------+----------------------+---------------------+
  497. | output | The execution output | `statement output`_ |
  498. +--------+----------------------+---------------------+
  499. Statement State
  500. ^^^^^^^^^^^^^^^
  501. +-----------+----------------------------------+
  502. | name | description |
  503. +===========+==================================+
  504. | running | Statement is currently executing |
  505. +-----------+----------------------------------+
  506. | available | Statement has a ready response |
  507. +-----------+----------------------------------+
  508. | error | Statement failed |
  509. +-----------+----------------------------------+
  510. Statement Output
  511. ^^^^^^^^^^^^^^^^
  512. +-----------------+-------------------+----------------------------------+
  513. | name | description | type |
  514. +=================+===================+==================================+
  515. | status | execution status | string |
  516. +-----------------+-------------------+----------------------------------+
  517. | execution_count | a monotomically | integer |
  518. | | increasing number | |
  519. +-----------------+-------------------+----------------------------------+
  520. | data | statement output | an object mapping a mime type to |
  521. | | | the result. If the mime type is |
  522. | | | ``application/json``, the value |
  523. | | | will be a JSON value |
  524. +-----------------+-------------------+----------------------------------+
  525. Batch
  526. -----
  527. +----------------+--------------------------------------------------------------------------------+-----------------+
  528. | name | description | type |
  529. +================+================================================================================+=================+
  530. | file | archive holding the file | path (required) |
  531. +----------------+--------------------------------------------------------------------------------+-----------------+
  532. | args | command line arguments | list of strings |
  533. +----------------+--------------------------------------------------------------------------------+-----------------+
  534. | className | application's java/spark main class | string |
  535. +----------------+--------------------------------------------------------------------------------+-----------------+
  536. | jars | files to be placed on the java classpath | list of paths |
  537. +----------------+--------------------------------------------------------------------------------+-----------------+
  538. | pyFiles | files to be placed on the PYTHONPATH | list of paths |
  539. +----------------+--------------------------------------------------------------------------------+-----------------+
  540. | files | files to be placed in executor working directory | list of paths |
  541. +----------------+--------------------------------------------------------------------------------+-----------------+
  542. | driverMemory | memory for driver | string |
  543. +----------------+--------------------------------------------------------------------------------+-----------------+
  544. | driverCores | number of cores used by driver (YARN mode only) | int |
  545. +----------------+--------------------------------------------------------------------------------+-----------------+
  546. | executorMemory | memory for executor | string |
  547. +----------------+--------------------------------------------------------------------------------+-----------------+
  548. | executorCores | number of cores used by executor | int |
  549. +----------------+--------------------------------------------------------------------------------+-----------------+
  550. | numExecutors | number of executors (YARN mode only) | int |
  551. +----------------+--------------------------------------------------------------------------------+-----------------+
  552. | archives | Archives to be uncompressed in the executor working directory (YARN mode only) | list of paths |
  553. +----------------+--------------------------------------------------------------------------------+-----------------+
  554. License
  555. =======
  556. Apache License, Version 2.0
  557. http://www.apache.org/licenses/LICENSE-2.0