README.rst 29 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709
  1. Welcome to Livy, the REST Spark Server
  2. ======================================
  3. Livy is an open source REST interface (in **Beta**) for interacting with a
  4. remote Spark Shell running locally or from inside YARN.
  5. * Interactive Spark Scala, Python shells
  6. * Batch submissions in Scala, Java, Python
  7. * Multi users can submit to the same server with their own credentials in YARN mode (impersonation support)
  8. * Can be used for submitting jobs from anywhere with REST
  9. * Does not require any code change to your programs
  10. Livy is used for powering the `Spark Notebook`_ of `Hue 3.8`_, which you can see the
  11. `implementation here`_.
  12. .. _Spark Notebook: http://gethue.com/new-notebook-application-for-spark-sql/
  13. .. _Hue 3.8: http://gethue.com/hue-3-8-with-an-oozie-editor-revamp-better-performances-improved-spark-ui-is-out/
  14. .. _implementation here: https://github.com/cloudera/hue/blob/master/apps/spark/src/spark/job_server_api.py
  15. Prerequisites
  16. =============
  17. To build/run Livy, you will need:
  18. Debian/Ubuntu:
  19. * mvn (from ``maven`` package or maven3 tarball)
  20. * openjdk-7-jdk (or Oracle Java7 jdk)
  21. * spark 1.4 from (from `Apache Spark tarball`_)
  22. * Python 2.6+
  23. * R 3.x
  24. Redhat/CentOS:
  25. * mvn (from ``maven`` package or maven3 tarball)
  26. * java-1.7.0-openjdk (or Oracle Java7 jdk)
  27. * spark 1.4 (from `Apache Spark tarball`_)
  28. * Python 2.6+
  29. * R 3.x
  30. MacOS:
  31. * Xcode command line tools
  32. * Oracle's JDK 1.7+
  33. * Maven (Homebrew)
  34. * apache-spark 1.4 (Homebrew)
  35. * Python 2.6+
  36. * R 3.x
  37. .. _Apache Spark Tarball: https://spark.apache.org/downloads.html
  38. Building Livy
  39. =============
  40. Livy is currently built by the `Hue Build System`_, it can also be built on
  41. it's own (aka without any other Hue dependency) with `Apache Maven`_. To build,
  42. run:
  43. .. code:: shell
  44. % cd $HUE_HOME/apps/spark/java
  45. % mvn -DskipTests clean package
  46. .. _Hue Build System: https://github.com/cloudera/hue/#getting-started
  47. .. _Apache Maven: http://maven.apache.org
  48. Running Tests
  49. =============
  50. In order to run the Livy Tests, first follow the instructions in `Building
  51. Livy`_. Then run:
  52. .. code:: shell
  53. % cd $HUE_HOME/apps/spark/java
  54. % export SPARK_HOME=/usr/lib/spark
  55. % mvn test
  56. Running Livy
  57. ============
  58. In order to run Livy with local sessions, start the server with:
  59. .. code:: shell
  60. % export SPARK_HOME=/usr/lib/spark
  61. % ./bin/livy-server
  62. Or with YARN sessions by running:
  63. .. code:: shell
  64. % env \
  65. LIVY_SERVER_JAVA_OPTS="-Dlivy.server.session.factory=yarn" \
  66. CLASSPATH=`hadoop classpath` \
  67. ./bin/livy-server
  68. Spark Example
  69. =============
  70. Now to see it in action by interacting with it in Python with the `Requests`_
  71. library. By default livy runs on port 8998 (which can be changed with the
  72. ``livy_server_port config`` option). We’ll start off with a Spark session that
  73. takes Scala code:
  74. .. code:: python
  75. >>> import json, pprint, requests, textwrap
  76. >>> host = 'http://localhost:8998'
  77. >>> data = {'kind': 'spark'}
  78. >>> headers = {'Content-Type': 'application/json'}
  79. >>> r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
  80. >>> r.json()
  81. {u'state': u'starting', u'id': 0, u’kind’: u’spark’}
  82. Once the session has completed starting up, it transitions to the idle state:
  83. .. code:: python
  84. >>> session_url = host + r.headers['location']
  85. >>> r = requests.get(session_url, headers=headers)
  86. >>> r.json()
  87. {u'state': u'idle', u'id': 0, u’kind’: u’spark’}
  88. Now we can execute Scala by passing in a simple JSON command:
  89. .. code:: python
  90. >>> data = {'code': '1 + 1'}
  91. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  92. >>> r.json()
  93. {u'output': None, u'state': u'running', u'id': 0}
  94. If a statement takes longer than a few milliseconds to execute, Livy returns
  95. early and provides a URL that can be polled until it is complete:
  96. .. code:: python
  97. >>> statement_url = host + r.headers['location']
  98. >>> r = requests.get(statement_url, headers=headers)
  99. >>> pprint.pprint(r.json())
  100. [{u'id': 0,
  101. u'output': {u'data': {u'text/plain': u'res0: Int = 2'},
  102. u'execution_count': 0,
  103. u'status': u'ok'},
  104. u'state': u'available'}]
  105. That was a pretty simple example. More interesting is using Spark to estimate
  106. PI. This is from the `Spark Examples`_:
  107. .. code:: python
  108. >>> data = {
  109. ... 'code': textwrap.dedent("""\
  110. ... val NUM_SAMPLES = 100000;
  111. ... val count = sc.parallelize(1 to NUM_SAMPLES).map { i =>
  112. ... val x = Math.random();
  113. ... val y = Math.random();
  114. ... if (x*x + y*y < 1) 1 else 0
  115. ... }.reduce(_ + _);
  116. ... println(\"Pi is roughly \" + 4.0 * count / NUM_SAMPLES)
  117. ... """)
  118. ... }
  119. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  120. >>> pprint.pprint(r.json())
  121. {u'id': 1,
  122. u'output': {u'data': {u'text/plain': u'Pi is roughly 3.14004\nNUM_SAMPLES: Int = 100000\ncount: Int = 78501'},
  123. u'execution_count': 1,
  124. u'status': u'ok'},
  125. u'state': u'available'}
  126. Finally, lets close our session:
  127. .. code:: python
  128. >>> session_url = 'http://localhost:8998/sessions/0'
  129. >>> requests.delete(session_url, headers=headers)
  130. <Response [204]>
  131. .. _Requests: http://docs.python-requests.org/en/latest/
  132. .. _Spark Examples: https://spark.apache.org/examples.html
  133. PySpark Example
  134. ===============
  135. pyspark has the exact same API, just with a different initial command:
  136. .. code:: python
  137. >>> data = {'kind': 'pyspark'}
  138. >>> r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
  139. >>> r.json()
  140. {u'id': 1, u'state': u'idle'}
  141. The PI example from before then can be run as:
  142. .. code:: python
  143. >>> data = {
  144. ... 'code': textwrap.dedent("""\
  145. ... import random
  146. ... NUM_SAMPLES = 100000
  147. ... def sample(p):
  148. ... x, y = random.random(), random.random()
  149. ... return 1 if x*x + y*y < 1 else 0
  150. ...
  151. ... count = sc.parallelize(xrange(0, NUM_SAMPLES)).map(sample) \
  152. ... .reduce(lambda a, b: a + b)
  153. ... print "Pi is roughly %f" % (4.0 * count / NUM_SAMPLES)
  154. ... """)
  155. ... }
  156. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  157. >>> pprint.pprint(r.json())
  158. {u'id': 12,
  159. u'output': {u'data': {u'text/plain': u'Pi is roughly 3.136000'},
  160. u'execution_count': 12,
  161. u'status': u'ok'},
  162. u'state': u'running'}
  163. SparkR Example
  164. ==============
  165. SparkR also has the same API:
  166. .. code:: python
  167. >>> data = {'kind': 'sparkR'}
  168. >>> r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
  169. >>> r.json()
  170. {u'id': 1, u'state': u'idle'}
  171. The PI example from before then can be run as:
  172. .. code:: python
  173. >>> data = {
  174. ... 'code': textwrap.dedent("""\
  175. ... n <- 100000
  176. ... piFunc <- function(elem) {
  177. ... rands <- runif(n = 2, min = -1, max = 1)
  178. ... val <- ifelse((rands[1]^2 + rands[2]^2) < 1, 1.0, 0.0)
  179. ... val
  180. ... }
  181. ... piFuncVec <- function(elems) {
  182. ... message(length(elems))
  183. ... rands1 <- runif(n = length(elems), min = -1, max = 1)
  184. ... rands2 <- runif(n = length(elems), min = -1, max = 1)
  185. ... val <- ifelse((rands1^2 + rands2^2) < 1, 1.0, 0.0)
  186. ... sum(val)
  187. ... }
  188. ... rdd <- parallelize(sc, 1:n, slices)
  189. ... count <- reduce(lapplyPartition(rdd, piFuncVec), sum)
  190. ... cat("Pi is roughly", 4.0 * count / n, "\n")
  191. ... """)
  192. ... }
  193. >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
  194. >>> pprint.pprint(r.json())
  195. {u'id': 12,
  196. u'output': {u'data': {u'text/plain': u'Pi is roughly 3.136000'},
  197. u'execution_count': 12,
  198. u'status': u'ok'},
  199. u'state': u'running'}
  200. Community
  201. =========
  202. * User group: http://groups.google.com/a/cloudera.org/group/hue-user
  203. * Jira: https://issues.cloudera.org/browse/HUE-2588
  204. * Reviews: https://review.cloudera.org/dashboard/?view=to-group&group=hue (repo 'hue-rw')
  205. REST API
  206. ========
  207. GET /sessions
  208. -------------
  209. Returns all the active interactive sessions.
  210. Response Body
  211. ^^^^^^^^^^^^^
  212. +----------+-----------------+------+
  213. | name | description | type |
  214. +==========+=================+======+
  215. | sessions | `session`_ list | list |
  216. +----------+-----------------+------+
  217. POST /sessions
  218. --------------
  219. Creates a new interative Scala or Python shell in the cluster.
  220. Request Body
  221. ^^^^^^^^^^^^
  222. +----------------+--------------------------------------------------+----------------------------+
  223. | name | description | type |
  224. +================+==================================================+============================+
  225. | id | The session id | int |
  226. +----------------+--------------------------------------------------+----------------------------+
  227. | kind | session kind (spark, pyspark, or sparkr) | `session kind`_ (required) |
  228. +----------------+--------------------------------------------------+----------------------------+
  229. | log | The log lines | list of strings |
  230. +----------------+--------------------------------------------------+----------------------------+
  231. | state | The session state | string |
  232. +----------------+--------------------------------------------------+----------------------------+
  233. | file | archive holding the file | path (required) |
  234. +----------------+--------------------------------------------------+----------------------------+
  235. | args | command line arguments | list of strings |
  236. +----------------+--------------------------------------------------+----------------------------+
  237. | className | application's java/spark main class | string |
  238. +----------------+--------------------------------------------------+----------------------------+
  239. | jars | files to be placed on the java classpath | list of paths |
  240. +----------------+--------------------------------------------------+----------------------------+
  241. | pyFiles | files to be placed on the PYTHONPATH | list of paths |
  242. +----------------+--------------------------------------------------+----------------------------+
  243. | files | files to be placed in executor working directory | list of paths |
  244. +----------------+--------------------------------------------------+----------------------------+
  245. | driverMemory | memory for driver | string |
  246. +----------------+--------------------------------------------------+----------------------------+
  247. | driverCores | number of cores used by driver | int |
  248. +----------------+--------------------------------------------------+----------------------------+
  249. | executorMemory | memory for executor | string |
  250. +----------------+--------------------------------------------------+----------------------------+
  251. | executorCores | number of cores used by executor | int |
  252. +----------------+--------------------------------------------------+----------------------------+
  253. | numExecutors | number of executor | int |
  254. +----------------+--------------------------------------------------+----------------------------+
  255. | archives | | list of paths |
  256. +----------------+--------------------------------------------------+----------------------------+
  257. Response Body
  258. ^^^^^^^^^^^^^
  259. The created `Session`_.
  260. GET /sessions/{sessionId}
  261. -------------------------
  262. Return the session information
  263. Response
  264. ^^^^^^^^
  265. The `Session`_.
  266. DELETE /sessions/{sessionId}
  267. -------------------------
  268. Kill the `Session`_ job.
  269. GET /sessions/{sessionId}/logs
  270. ---------------------------
  271. Get the log lines from this session.
  272. Request Parameters
  273. ^^^^^^^^^^^^^^^^^^
  274. +------+-----------------------------+------+
  275. | name | description | type |
  276. +======+=============================+======+
  277. | from | offset | int |
  278. +------+-----------------------------+------+
  279. | size | amount of batches to return | int |
  280. +------+-----------------------------+------+
  281. Response Body
  282. ^^^^^^^^^^^^^
  283. +------+-----------------------+-----------------+
  284. | name | description | type |
  285. +======+=======================+=================+
  286. | id | The session id | int |
  287. +------+-----------------------+-----------------+
  288. | from | offset | int |
  289. +------+-----------------------+-----------------+
  290. | size | total amount of lines | int |
  291. +------+-----------------------+-----------------+
  292. | log | The log lines | list of strings |
  293. +------+-----------------------+-----------------+
  294. GET /sessions/{sessionId}/statements
  295. ------------------------------------
  296. Return all the statements in a session.
  297. Response Body
  298. ^^^^^^^^^^^^^
  299. +------------+-------------------+------+
  300. | name | description | type |
  301. +============+===================+======+
  302. | statements | `statement`_ list | list |
  303. +------------+-------------------+------+
  304. POST /sessions/{sessionId}/statements
  305. -------------------------------------
  306. Execute a statement in a session.
  307. Request Body
  308. ^^^^^^^^^^^^
  309. +------+---------------------+--------+
  310. | name | description | type |
  311. +======+=====================+========+
  312. | code | The code to execute | string |
  313. +------+---------------------+--------+
  314. Response Body
  315. ^^^^^^^^^^^^^
  316. The `statement`_ object.
  317. GET /batches
  318. ------------
  319. Return all the active batch jobs.
  320. Response Body
  321. ^^^^^^^^^^^^^
  322. +---------+---------------+------+
  323. | name | description | type |
  324. +=========+===============+======+
  325. | batches | `batch`_ list | list |
  326. +---------+---------------+------+
  327. POST /batches
  328. -------------
  329. Request Body
  330. ^^^^^^^^^^^^
  331. +----------------+--------------------------------------------------+-----------------+
  332. | name | description | type |
  333. +================+==================================================+=================+
  334. | id | The session id | int |
  335. +----------------+--------------------------------------------------+-----------------+
  336. | log | The log lines | list of strings |
  337. +----------------+--------------------------------------------------+-----------------+
  338. | state | The session state | string |
  339. +----------------+--------------------------------------------------+-----------------+
  340. | file | archive holding the file | path (required) |
  341. +----------------+--------------------------------------------------+-----------------+
  342. | args | command line arguments | list of strings |
  343. +----------------+--------------------------------------------------+-----------------+
  344. | className | application's java/spark main class | string |
  345. +----------------+--------------------------------------------------+-----------------+
  346. | jars | files to be placed on the java classpath | list of paths |
  347. +----------------+--------------------------------------------------+-----------------+
  348. | pyFiles | files to be placed on the PYTHONPATH | list of paths |
  349. +----------------+--------------------------------------------------+-----------------+
  350. | files | files to be placed in executor working directory | list of paths |
  351. +----------------+--------------------------------------------------+-----------------+
  352. | driverMemory | memory for driver | string |
  353. +----------------+--------------------------------------------------+-----------------+
  354. | driverCores | number of cores used by driver | int |
  355. +----------------+--------------------------------------------------+-----------------+
  356. | executorMemory | memory for executor | string |
  357. +----------------+--------------------------------------------------+-----------------+
  358. | executorCores | number of cores used by executor | int |
  359. +----------------+--------------------------------------------------+-----------------+
  360. | numExecutors | number of executor | int |
  361. +----------------+--------------------------------------------------+-----------------+
  362. | archives | | list of paths |
  363. +----------------+--------------------------------------------------+-----------------+
  364. Response Body
  365. ^^^^^^^^^^^^^
  366. The created `Batch`_ object.
  367. GET /batches/{batchId}
  368. ----------------------
  369. Request Parameters
  370. ^^^^^^^^^^^^^^^^^^
  371. +------+-----------------------------+------+
  372. | name | description | type |
  373. +======+=============================+======+
  374. | from | offset | int |
  375. +------+-----------------------------+------+
  376. | size | amount of batches to return | int |
  377. +------+-----------------------------+------+
  378. Response Body
  379. ^^^^^^^^^^^^^
  380. +-------+-----------------------------+-----------------+
  381. | name | description | type |
  382. +=======+=============================+=================+
  383. | id | The batch id | int |
  384. +-------+-----------------------------+-----------------+
  385. | state | The state of the batch | `batch`_ state |
  386. +-------+-----------------------------+-----------------+
  387. | log | The output of the batch job | list of strings |
  388. +-------+-----------------------------+-----------------+
  389. DELETE /batches/{batchId}
  390. -------------------------
  391. Kill the `Batch`_ job.
  392. GET /batches/{batchId}/logs
  393. ---------------------------
  394. Get the log lines from this batch.
  395. Request Parameters
  396. ^^^^^^^^^^^^^^^^^^
  397. +------+-----------------------------+------+
  398. | name | description | type |
  399. +======+=============================+======+
  400. | from | offset | int |
  401. +------+-----------------------------+------+
  402. | size | amount of batches to return | int |
  403. +------+-----------------------------+------+
  404. Response Body
  405. ^^^^^^^^^^^^^
  406. +------+-----------------------+-----------------+
  407. | name | description | type |
  408. +======+=======================+=================+
  409. | id | The batch id | int |
  410. +------+-----------------------+-----------------+
  411. | from | offset | int |
  412. +------+-----------------------+-----------------+
  413. | size | total amount of lines | int |
  414. +------+-----------------------+-----------------+
  415. | log | The log lines | list of strings |
  416. +------+-----------------------+-----------------+
  417. REST Objects
  418. ============
  419. Session
  420. -------
  421. Sessions represent an interactive shell.
  422. +----------------+--------------------------------------------------------------------------------+------------------+
  423. | name | description | type |
  424. +================+================================================================================+==================+
  425. | id | The session id | string |
  426. +----------------+--------------------------------------------------------------------------------+------------------+
  427. | state | The state of the session | `session state`_ |
  428. +----------------+--------------------------------------------------------------------------------+------------------+
  429. | kind | The session kind | `session kind`_ |
  430. +----------------+--------------------------------------------------------------------------------+------------------+
  431. | proxyUser | The user running this session | optional string |
  432. +----------------+--------------------------------------------------------------------------------+------------------+
  433. | jars | files to be placed on the java classpath | list of paths |
  434. +----------------+--------------------------------------------------------------------------------+------------------+
  435. | pyFiles | files to be placed on the PYTHONPATH | list of paths |
  436. +----------------+--------------------------------------------------------------------------------+------------------+
  437. | files | files to be placed in executor working directory | list of paths |
  438. +----------------+--------------------------------------------------------------------------------+------------------+
  439. | driverMemory | memory for driver | string |
  440. +----------------+--------------------------------------------------------------------------------+------------------+
  441. | driverCores | number of cores used by driver (YARN mode only) | int |
  442. +----------------+--------------------------------------------------------------------------------+------------------+
  443. | executorMemory | memory for executor | string |
  444. +----------------+--------------------------------------------------------------------------------+------------------+
  445. | executorCores | number of cores used by executor | int |
  446. +----------------+--------------------------------------------------------------------------------+------------------+
  447. | numExecutors | number of executors (YARN mode only) | int |
  448. +----------------+--------------------------------------------------------------------------------+------------------+
  449. | archives | Archives to be uncompressed in the executor working directory (YARN mode only) | list of paths |
  450. +----------------+--------------------------------------------------------------------------------+------------------+
  451. Session State
  452. ^^^^^^^^^^^^^
  453. +-------------+----------------------------------+
  454. | name | description |
  455. +=============+==================================+
  456. | not_started | session has not been started |
  457. +-------------+----------------------------------+
  458. | starting | session is starting |
  459. +-------------+----------------------------------+
  460. | idle | session is waiting for input |
  461. +-------------+----------------------------------+
  462. | busy | session is executing a statement |
  463. +-------------+----------------------------------+
  464. | error | session errored out |
  465. +-------------+----------------------------------+
  466. | dead | session has exited |
  467. +-------------+----------------------------------+
  468. Session Kind
  469. ^^^^^^^^^^^^
  470. +---------+----------------------------------+
  471. | name | description |
  472. +=========+==================================+
  473. | spark | interactive scala/spark session |
  474. +---------+----------------------------------+
  475. | pyspark | interactive python/spark session |
  476. +---------+----------------------------------+
  477. Statement
  478. ---------
  479. Statements represent the result of an execution statement.
  480. +--------+----------------------+---------------------+
  481. | name | description | type |
  482. +========+======================+=====================+
  483. | id | The statement id | integer |
  484. +--------+----------------------+---------------------+
  485. | state | The execution state | `statement state`_ |
  486. +--------+----------------------+---------------------+
  487. | output | The execution output | `statement output`_ |
  488. +--------+----------------------+---------------------+
  489. Statement State
  490. ^^^^^^^^^^^^^^^
  491. +-----------+----------------------------------+
  492. | name | description |
  493. +===========+==================================+
  494. | running | Statement is currently executing |
  495. +-----------+----------------------------------+
  496. | available | Statement has a ready response |
  497. +-----------+----------------------------------+
  498. | error | Statement failed |
  499. +-----------+----------------------------------+
  500. Statement Output
  501. ^^^^^^^^^^^^^^^^
  502. +-----------------+-------------------+----------------------------------+
  503. | name | description | type |
  504. +=================+===================+==================================+
  505. | status | execution status | string |
  506. +-----------------+-------------------+----------------------------------+
  507. | execution_count | a monotomically | integer |
  508. | | increasing number | |
  509. +-----------------+-------------------+----------------------------------+
  510. | data | statement output | an object mapping a mime type to |
  511. | | | the result. If the mime type is |
  512. | | | ``application/json``, the value |
  513. | | | will be a JSON value |
  514. +-----------------+-------------------+----------------------------------+
  515. Batch
  516. -----
  517. +----------------+--------------------------------------------------------------------------------+-----------------+
  518. | name | description | type |
  519. +================+================================================================================+=================+
  520. | file | archive holding the file | path (required) |
  521. +----------------+--------------------------------------------------------------------------------+-----------------+
  522. | args | command line arguments | list of strings |
  523. +----------------+--------------------------------------------------------------------------------+-----------------+
  524. | className | application's java/spark main class | string |
  525. +----------------+--------------------------------------------------------------------------------+-----------------+
  526. | jars | files to be placed on the java classpath | list of paths |
  527. +----------------+--------------------------------------------------------------------------------+-----------------+
  528. | pyFiles | files to be placed on the PYTHONPATH | list of paths |
  529. +----------------+--------------------------------------------------------------------------------+-----------------+
  530. | files | files to be placed in executor working directory | list of paths |
  531. +----------------+--------------------------------------------------------------------------------+-----------------+
  532. | driverMemory | memory for driver | string |
  533. +----------------+--------------------------------------------------------------------------------+-----------------+
  534. | driverCores | number of cores used by driver (YARN mode only) | int |
  535. +----------------+--------------------------------------------------------------------------------+-----------------+
  536. | executorMemory | memory for executor | string |
  537. +----------------+--------------------------------------------------------------------------------+-----------------+
  538. | executorCores | number of cores used by executor | int |
  539. +----------------+--------------------------------------------------------------------------------+-----------------+
  540. | numExecutors | number of executors (YARN mode only) | int |
  541. +----------------+--------------------------------------------------------------------------------+-----------------+
  542. | archives | Archives to be uncompressed in the executor working directory (YARN mode only) | list of paths |
  543. +----------------+--------------------------------------------------------------------------------+-----------------+
  544. License
  545. =======
  546. Apache License, Version 2.0
  547. http://www.apache.org/licenses/LICENSE-2.0