README.rst 29 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710
  1. Welcome to Livy, the REST Spark Server
  2. ======================================
  3. Livy is an open source REST interface for interacting with Spark from anywhere. It supports executing snippets of code or programs in a Spark context that runs locally or in YARN.
  4. * Interactive Scala, Python and R shells
  5. * Batch submissions in Scala, Java, Python
  6. * Multi users can share the same server (impersonation support)
  7. * Can be used for submitting jobs from anywhere with REST
  8. * Does not require any code change to your programs
  9. The code is currently incubating in Hue but hopefully will eventually graduate in its top project. `Pull requests` are welcomed!
  10. Livy is used for powering the `Spark Notebook`_ of `Hue 3.8`_, which you can see the
  11. `implementation here`_.
  12. .. _Pull requests: https://github.com/cloudera/hue/pulls
  13. .. _Spark Notebook: http://gethue.com/new-notebook-application-for-spark-sql/
  14. .. _Hue 3.8: http://gethue.com/hue-3-8-with-an-oozie-editor-revamp-better-performances-improved-spark-ui-is-out/
  15. .. _implementation here: https://github.com/cloudera/hue/blob/master/apps/spark/src/spark/job_server_api.py
  16. Prerequisites
  17. =============
  18. To build/run Livy, you will need:
  19. Debian/Ubuntu:
  20. * mvn (from ``maven`` package or maven3 tarball)
  21. * openjdk-7-jdk (or Oracle Java7 jdk)
  22. * spark 1.4 from (from `Apache Spark tarball`_)
  23. * Python 2.6+
  24. * R 3.x
  25. Redhat/CentOS:
  26. * mvn (from ``maven`` package or maven3 tarball)
  27. * java-1.7.0-openjdk (or Oracle Java7 jdk)
  28. * spark 1.4 (from `Apache Spark tarball`_)
  29. * Python 2.6+
  30. * R 3.x
  31. MacOS:
  32. * Xcode command line tools
  33. * Oracle's JDK 1.7+
  34. * Maven (Homebrew)
  35. * apache-spark 1.4 (Homebrew)
  36. * Python 2.6+
  37. * R 3.x
  38. .. _Apache Spark Tarball: https://spark.apache.org/downloads.html
  39. Building Livy
  40. =============
  41. Livy is currently built by the `Hue Build System`_, it can also be built on
  42. it's own (aka without any other Hue dependency) with `Apache Maven`_. To build,
  43. run:
  44. .. code:: shell
  45. % cd $HUE_HOME/apps/spark/java
  46. % mvn -DskipTests clean package
  47. .. _Hue Build System: https://github.com/cloudera/hue/#getting-started
  48. .. _Apache Maven: http://maven.apache.org
  49. Running Tests
  50. =============
  51. In order to run the Livy Tests, first follow the instructions in `Building
  52. Livy`_. Then run:
  53. .. code:: shell
  54. % cd $HUE_HOME/apps/spark/java
  55. % export SPARK_HOME=/usr/lib/spark
  56. % mvn test
  57. Running Livy
  58. ============
  59. In order to run Livy with local sessions, start the server with:
  60. .. code:: shell
  61. % export SPARK_HOME=/usr/lib/spark
  62. % ./bin/livy-server
  63. Or with YARN sessions by running:
  64. .. code:: shell
  65. % env \
  66. LIVY_SERVER_JAVA_OPTS="-Dlivy.server.session.factory=yarn" \
  67. CLASSPATH=`hadoop classpath` \
  68. ./bin/livy-server
  69. Spark Example
  70. =============
  71. Now to see it in action by interacting with it in Python with the `Requests`_
  72. library. By default livy runs on port 8998 (which can be changed with the
  73. ``livy_server_port config`` option). We’ll start off with a Spark session that
  74. takes Scala code:
  75. .. code:: python
  76. >>> import json, pprint, requests, textwrap
  77. >>> host = 'http://localhost:8998'
  78. >>> data = {'kind': 'spark'}
  79. >>> headers = {'Content-Type': 'application/json'}
  80. >>> r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
  81. >>> r.json()
  82. {u'state': u'starting', u'id': 0, u’kind’: u’spark’}
  83. Once the session has completed starting up, it transitions to the idle state:
  84. .. code:: python
  85. >>> session_url = host + r.headers['location']
  86. >>> r = requests.get(session_url, headers=headers)
  87. >>> r.json()
  88. {u'state': u'idle', u'id': 0, u’kind’: u’spark’}
  89. Now we can execute Scala by passing in a simple JSON command:
  90. .. code:: python
  91. >>> data = {'code': '1 + 1'}
  92. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  93. >>> r.json()
  94. {u'output': None, u'state': u'running', u'id': 0}
  95. If a statement takes longer than a few milliseconds to execute, Livy returns
  96. early and provides a URL that can be polled until it is complete:
  97. .. code:: python
  98. >>> statement_url = host + r.headers['location']
  99. >>> r = requests.get(statement_url, headers=headers)
  100. >>> pprint.pprint(r.json())
  101. [{u'id': 0,
  102. u'output': {u'data': {u'text/plain': u'res0: Int = 2'},
  103. u'execution_count': 0,
  104. u'status': u'ok'},
  105. u'state': u'available'}]
  106. That was a pretty simple example. More interesting is using Spark to estimate
  107. PI. This is from the `Spark Examples`_:
  108. .. code:: python
  109. >>> data = {
  110. ... 'code': textwrap.dedent("""\
  111. ... val NUM_SAMPLES = 100000;
  112. ... val count = sc.parallelize(1 to NUM_SAMPLES).map { i =>
  113. ... val x = Math.random();
  114. ... val y = Math.random();
  115. ... if (x*x + y*y < 1) 1 else 0
  116. ... }.reduce(_ + _);
  117. ... println(\"Pi is roughly \" + 4.0 * count / NUM_SAMPLES)
  118. ... """)
  119. ... }
  120. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  121. >>> pprint.pprint(r.json())
  122. {u'id': 1,
  123. u'output': {u'data': {u'text/plain': u'Pi is roughly 3.14004\nNUM_SAMPLES: Int = 100000\ncount: Int = 78501'},
  124. u'execution_count': 1,
  125. u'status': u'ok'},
  126. u'state': u'available'}
  127. Finally, lets close our session:
  128. .. code:: python
  129. >>> session_url = 'http://localhost:8998/sessions/0'
  130. >>> requests.delete(session_url, headers=headers)
  131. <Response [204]>
  132. .. _Requests: http://docs.python-requests.org/en/latest/
  133. .. _Spark Examples: https://spark.apache.org/examples.html
  134. PySpark Example
  135. ===============
  136. pyspark has the exact same API, just with a different initial command:
  137. .. code:: python
  138. >>> data = {'kind': 'pyspark'}
  139. >>> r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
  140. >>> r.json()
  141. {u'id': 1, u'state': u'idle'}
  142. The PI example from before then can be run as:
  143. .. code:: python
  144. >>> data = {
  145. ... 'code': textwrap.dedent("""\
  146. ... import random
  147. ... NUM_SAMPLES = 100000
  148. ... def sample(p):
  149. ... x, y = random.random(), random.random()
  150. ... return 1 if x*x + y*y < 1 else 0
  151. ...
  152. ... count = sc.parallelize(xrange(0, NUM_SAMPLES)).map(sample) \
  153. ... .reduce(lambda a, b: a + b)
  154. ... print "Pi is roughly %f" % (4.0 * count / NUM_SAMPLES)
  155. ... """)
  156. ... }
  157. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  158. >>> pprint.pprint(r.json())
  159. {u'id': 12,
  160. u'output': {u'data': {u'text/plain': u'Pi is roughly 3.136000'},
  161. u'execution_count': 12,
  162. u'status': u'ok'},
  163. u'state': u'running'}
  164. SparkR Example
  165. ==============
  166. SparkR also has the same API:
  167. .. code:: python
  168. >>> data = {'kind': 'sparkR'}
  169. >>> r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
  170. >>> r.json()
  171. {u'id': 1, u'state': u'idle'}
  172. The PI example from before then can be run as:
  173. .. code:: python
  174. >>> data = {
  175. ... 'code': textwrap.dedent("""\
  176. ... n <- 100000
  177. ... piFunc <- function(elem) {
  178. ... rands <- runif(n = 2, min = -1, max = 1)
  179. ... val <- ifelse((rands[1]^2 + rands[2]^2) < 1, 1.0, 0.0)
  180. ... val
  181. ... }
  182. ... piFuncVec <- function(elems) {
  183. ... message(length(elems))
  184. ... rands1 <- runif(n = length(elems), min = -1, max = 1)
  185. ... rands2 <- runif(n = length(elems), min = -1, max = 1)
  186. ... val <- ifelse((rands1^2 + rands2^2) < 1, 1.0, 0.0)
  187. ... sum(val)
  188. ... }
  189. ... rdd <- parallelize(sc, 1:n, slices)
  190. ... count <- reduce(lapplyPartition(rdd, piFuncVec), sum)
  191. ... cat("Pi is roughly", 4.0 * count / n, "\n")
  192. ... """)
  193. ... }
  194. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  195. >>> pprint.pprint(r.json())
  196. {u'id': 12,
  197. u'output': {u'data': {u'text/plain': u'Pi is roughly 3.136000'},
  198. u'execution_count': 12,
  199. u'status': u'ok'},
  200. u'state': u'running'}
  201. Community
  202. =========
  203. * User group: http://groups.google.com/a/cloudera.org/group/hue-user
  204. * Jira: https://issues.cloudera.org/browse/HUE-2588
  205. * Reviews: https://review.cloudera.org/dashboard/?view=to-group&group=hue (repo 'hue-rw')
  206. REST API
  207. ========
  208. GET /sessions
  209. -------------
  210. Returns all the active interactive sessions.
  211. Response Body
  212. ^^^^^^^^^^^^^
  213. +----------+-----------------+------+
  214. | name | description | type |
  215. +==========+=================+======+
  216. | sessions | `session`_ list | list |
  217. +----------+-----------------+------+
  218. POST /sessions
  219. --------------
  220. Creates a new interative Scala or Python shell in the cluster.
  221. Request Body
  222. ^^^^^^^^^^^^
  223. +----------------+--------------------------------------------------+----------------------------+
  224. | name | description | type |
  225. +================+==================================================+============================+
  226. | id | The session id | int |
  227. +----------------+--------------------------------------------------+----------------------------+
  228. | kind | session kind (spark, pyspark, or sparkr) | `session kind`_ (required) |
  229. +----------------+--------------------------------------------------+----------------------------+
  230. | log | The log lines | list of strings |
  231. +----------------+--------------------------------------------------+----------------------------+
  232. | state | The session state | string |
  233. +----------------+--------------------------------------------------+----------------------------+
  234. | file | archive holding the file | path (required) |
  235. +----------------+--------------------------------------------------+----------------------------+
  236. | args | command line arguments | list of strings |
  237. +----------------+--------------------------------------------------+----------------------------+
  238. | className | application's java/spark main class | string |
  239. +----------------+--------------------------------------------------+----------------------------+
  240. | jars | files to be placed on the java classpath | list of paths |
  241. +----------------+--------------------------------------------------+----------------------------+
  242. | pyFiles | files to be placed on the PYTHONPATH | list of paths |
  243. +----------------+--------------------------------------------------+----------------------------+
  244. | files | files to be placed in executor working directory | list of paths |
  245. +----------------+--------------------------------------------------+----------------------------+
  246. | driverMemory | memory for driver | string |
  247. +----------------+--------------------------------------------------+----------------------------+
  248. | driverCores | number of cores used by driver | int |
  249. +----------------+--------------------------------------------------+----------------------------+
  250. | executorMemory | memory for executor | string |
  251. +----------------+--------------------------------------------------+----------------------------+
  252. | executorCores | number of cores used by executor | int |
  253. +----------------+--------------------------------------------------+----------------------------+
  254. | numExecutors | number of executor | int |
  255. +----------------+--------------------------------------------------+----------------------------+
  256. | archives | | list of paths |
  257. +----------------+--------------------------------------------------+----------------------------+
  258. Response Body
  259. ^^^^^^^^^^^^^
  260. The created `Session`_.
  261. GET /sessions/{sessionId}
  262. -------------------------
  263. Return the session information
  264. Response
  265. ^^^^^^^^
  266. The `Session`_.
  267. DELETE /sessions/{sessionId}
  268. -------------------------
  269. Kill the `Session`_ job.
  270. GET /sessions/{sessionId}/logs
  271. ---------------------------
  272. Get the log lines from this session.
  273. Request Parameters
  274. ^^^^^^^^^^^^^^^^^^
  275. +------+-----------------------------+------+
  276. | name | description | type |
  277. +======+=============================+======+
  278. | from | offset | int |
  279. +------+-----------------------------+------+
  280. | size | amount of batches to return | int |
  281. +------+-----------------------------+------+
  282. Response Body
  283. ^^^^^^^^^^^^^
  284. +------+-----------------------+-----------------+
  285. | name | description | type |
  286. +======+=======================+=================+
  287. | id | The session id | int |
  288. +------+-----------------------+-----------------+
  289. | from | offset | int |
  290. +------+-----------------------+-----------------+
  291. | size | total amount of lines | int |
  292. +------+-----------------------+-----------------+
  293. | log | The log lines | list of strings |
  294. +------+-----------------------+-----------------+
  295. GET /sessions/{sessionId}/statements
  296. ------------------------------------
  297. Return all the statements in a session.
  298. Response Body
  299. ^^^^^^^^^^^^^
  300. +------------+-------------------+------+
  301. | name | description | type |
  302. +============+===================+======+
  303. | statements | `statement`_ list | list |
  304. +------------+-------------------+------+
  305. POST /sessions/{sessionId}/statements
  306. -------------------------------------
  307. Execute a statement in a session.
  308. Request Body
  309. ^^^^^^^^^^^^
  310. +------+---------------------+--------+
  311. | name | description | type |
  312. +======+=====================+========+
  313. | code | The code to execute | string |
  314. +------+---------------------+--------+
  315. Response Body
  316. ^^^^^^^^^^^^^
  317. The `statement`_ object.
  318. GET /batches
  319. ------------
  320. Return all the active batch jobs.
  321. Response Body
  322. ^^^^^^^^^^^^^
  323. +---------+---------------+------+
  324. | name | description | type |
  325. +=========+===============+======+
  326. | batches | `batch`_ list | list |
  327. +---------+---------------+------+
  328. POST /batches
  329. -------------
  330. Request Body
  331. ^^^^^^^^^^^^
  332. +----------------+--------------------------------------------------+-----------------+
  333. | name | description | type |
  334. +================+==================================================+=================+
  335. | id | The session id | int |
  336. +----------------+--------------------------------------------------+-----------------+
  337. | log | The log lines | list of strings |
  338. +----------------+--------------------------------------------------+-----------------+
  339. | state | The session state | string |
  340. +----------------+--------------------------------------------------+-----------------+
  341. | file | archive holding the file | path (required) |
  342. +----------------+--------------------------------------------------+-----------------+
  343. | args | command line arguments | list of strings |
  344. +----------------+--------------------------------------------------+-----------------+
  345. | className | application's java/spark main class | string |
  346. +----------------+--------------------------------------------------+-----------------+
  347. | jars | files to be placed on the java classpath | list of paths |
  348. +----------------+--------------------------------------------------+-----------------+
  349. | pyFiles | files to be placed on the PYTHONPATH | list of paths |
  350. +----------------+--------------------------------------------------+-----------------+
  351. | files | files to be placed in executor working directory | list of paths |
  352. +----------------+--------------------------------------------------+-----------------+
  353. | driverMemory | memory for driver | string |
  354. +----------------+--------------------------------------------------+-----------------+
  355. | driverCores | number of cores used by driver | int |
  356. +----------------+--------------------------------------------------+-----------------+
  357. | executorMemory | memory for executor | string |
  358. +----------------+--------------------------------------------------+-----------------+
  359. | executorCores | number of cores used by executor | int |
  360. +----------------+--------------------------------------------------+-----------------+
  361. | numExecutors | number of executor | int |
  362. +----------------+--------------------------------------------------+-----------------+
  363. | archives | | list of paths |
  364. +----------------+--------------------------------------------------+-----------------+
  365. Response Body
  366. ^^^^^^^^^^^^^
  367. The created `Batch`_ object.
  368. GET /batches/{batchId}
  369. ----------------------
  370. Request Parameters
  371. ^^^^^^^^^^^^^^^^^^
  372. +------+-----------------------------+------+
  373. | name | description | type |
  374. +======+=============================+======+
  375. | from | offset | int |
  376. +------+-----------------------------+------+
  377. | size | amount of batches to return | int |
  378. +------+-----------------------------+------+
  379. Response Body
  380. ^^^^^^^^^^^^^
  381. +-------+-----------------------------+-----------------+
  382. | name | description | type |
  383. +=======+=============================+=================+
  384. | id | The batch id | int |
  385. +-------+-----------------------------+-----------------+
  386. | state | The state of the batch | `batch`_ state |
  387. +-------+-----------------------------+-----------------+
  388. | log | The output of the batch job | list of strings |
  389. +-------+-----------------------------+-----------------+
  390. DELETE /batches/{batchId}
  391. -------------------------
  392. Kill the `Batch`_ job.
  393. GET /batches/{batchId}/logs
  394. ---------------------------
  395. Get the log lines from this batch.
  396. Request Parameters
  397. ^^^^^^^^^^^^^^^^^^
  398. +------+-----------------------------+------+
  399. | name | description | type |
  400. +======+=============================+======+
  401. | from | offset | int |
  402. +------+-----------------------------+------+
  403. | size | amount of batches to return | int |
  404. +------+-----------------------------+------+
  405. Response Body
  406. ^^^^^^^^^^^^^
  407. +------+-----------------------+-----------------+
  408. | name | description | type |
  409. +======+=======================+=================+
  410. | id | The batch id | int |
  411. +------+-----------------------+-----------------+
  412. | from | offset | int |
  413. +------+-----------------------+-----------------+
  414. | size | total amount of lines | int |
  415. +------+-----------------------+-----------------+
  416. | log | The log lines | list of strings |
  417. +------+-----------------------+-----------------+
  418. REST Objects
  419. ============
  420. Session
  421. -------
  422. Sessions represent an interactive shell.
  423. +----------------+--------------------------------------------------------------------------------+------------------+
  424. | name | description | type |
  425. +================+================================================================================+==================+
  426. | id | The session id | string |
  427. +----------------+--------------------------------------------------------------------------------+------------------+
  428. | state | The state of the session | `session state`_ |
  429. +----------------+--------------------------------------------------------------------------------+------------------+
  430. | kind | The session kind | `session kind`_ |
  431. +----------------+--------------------------------------------------------------------------------+------------------+
  432. | proxyUser | The user running this session | optional string |
  433. +----------------+--------------------------------------------------------------------------------+------------------+
  434. | jars | files to be placed on the java classpath | list of paths |
  435. +----------------+--------------------------------------------------------------------------------+------------------+
  436. | pyFiles | files to be placed on the PYTHONPATH | list of paths |
  437. +----------------+--------------------------------------------------------------------------------+------------------+
  438. | files | files to be placed in executor working directory | list of paths |
  439. +----------------+--------------------------------------------------------------------------------+------------------+
  440. | driverMemory | memory for driver | string |
  441. +----------------+--------------------------------------------------------------------------------+------------------+
  442. | driverCores | number of cores used by driver (YARN mode only) | int |
  443. +----------------+--------------------------------------------------------------------------------+------------------+
  444. | executorMemory | memory for executor | string |
  445. +----------------+--------------------------------------------------------------------------------+------------------+
  446. | executorCores | number of cores used by executor | int |
  447. +----------------+--------------------------------------------------------------------------------+------------------+
  448. | numExecutors | number of executors (YARN mode only) | int |
  449. +----------------+--------------------------------------------------------------------------------+------------------+
  450. | archives | Archives to be uncompressed in the executor working directory (YARN mode only) | list of paths |
  451. +----------------+--------------------------------------------------------------------------------+------------------+
  452. Session State
  453. ^^^^^^^^^^^^^
  454. +-------------+----------------------------------+
  455. | name | description |
  456. +=============+==================================+
  457. | not_started | session has not been started |
  458. +-------------+----------------------------------+
  459. | starting | session is starting |
  460. +-------------+----------------------------------+
  461. | idle | session is waiting for input |
  462. +-------------+----------------------------------+
  463. | busy | session is executing a statement |
  464. +-------------+----------------------------------+
  465. | error | session errored out |
  466. +-------------+----------------------------------+
  467. | dead | session has exited |
  468. +-------------+----------------------------------+
  469. Session Kind
  470. ^^^^^^^^^^^^
  471. +---------+----------------------------------+
  472. | name | description |
  473. +=========+==================================+
  474. | spark | interactive scala/spark session |
  475. +---------+----------------------------------+
  476. | pyspark | interactive python/spark session |
  477. +---------+----------------------------------+
  478. Statement
  479. ---------
  480. Statements represent the result of an execution statement.
  481. +--------+----------------------+---------------------+
  482. | name | description | type |
  483. +========+======================+=====================+
  484. | id | The statement id | integer |
  485. +--------+----------------------+---------------------+
  486. | state | The execution state | `statement state`_ |
  487. +--------+----------------------+---------------------+
  488. | output | The execution output | `statement output`_ |
  489. +--------+----------------------+---------------------+
  490. Statement State
  491. ^^^^^^^^^^^^^^^
  492. +-----------+----------------------------------+
  493. | name | description |
  494. +===========+==================================+
  495. | running | Statement is currently executing |
  496. +-----------+----------------------------------+
  497. | available | Statement has a ready response |
  498. +-----------+----------------------------------+
  499. | error | Statement failed |
  500. +-----------+----------------------------------+
  501. Statement Output
  502. ^^^^^^^^^^^^^^^^
  503. +-----------------+-------------------+----------------------------------+
  504. | name | description | type |
  505. +=================+===================+==================================+
  506. | status | execution status | string |
  507. +-----------------+-------------------+----------------------------------+
  508. | execution_count | a monotomically | integer |
  509. | | increasing number | |
  510. +-----------------+-------------------+----------------------------------+
  511. | data | statement output | an object mapping a mime type to |
  512. | | | the result. If the mime type is |
  513. | | | ``application/json``, the value |
  514. | | | will be a JSON value |
  515. +-----------------+-------------------+----------------------------------+
  516. Batch
  517. -----
  518. +----------------+--------------------------------------------------------------------------------+-----------------+
  519. | name | description | type |
  520. +================+================================================================================+=================+
  521. | file | archive holding the file | path (required) |
  522. +----------------+--------------------------------------------------------------------------------+-----------------+
  523. | args | command line arguments | list of strings |
  524. +----------------+--------------------------------------------------------------------------------+-----------------+
  525. | className | application's java/spark main class | string |
  526. +----------------+--------------------------------------------------------------------------------+-----------------+
  527. | jars | files to be placed on the java classpath | list of paths |
  528. +----------------+--------------------------------------------------------------------------------+-----------------+
  529. | pyFiles | files to be placed on the PYTHONPATH | list of paths |
  530. +----------------+--------------------------------------------------------------------------------+-----------------+
  531. | files | files to be placed in executor working directory | list of paths |
  532. +----------------+--------------------------------------------------------------------------------+-----------------+
  533. | driverMemory | memory for driver | string |
  534. +----------------+--------------------------------------------------------------------------------+-----------------+
  535. | driverCores | number of cores used by driver (YARN mode only) | int |
  536. +----------------+--------------------------------------------------------------------------------+-----------------+
  537. | executorMemory | memory for executor | string |
  538. +----------------+--------------------------------------------------------------------------------+-----------------+
  539. | executorCores | number of cores used by executor | int |
  540. +----------------+--------------------------------------------------------------------------------+-----------------+
  541. | numExecutors | number of executors (YARN mode only) | int |
  542. +----------------+--------------------------------------------------------------------------------+-----------------+
  543. | archives | Archives to be uncompressed in the executor working directory (YARN mode only) | list of paths |
  544. +----------------+--------------------------------------------------------------------------------+-----------------+
  545. License
  546. =======
  547. Apache License, Version 2.0
  548. http://www.apache.org/licenses/LICENSE-2.0