processing-ref.txt 13 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340
  1. .. include:: header.txt
  2. ==============================
  3. processing package reference
  4. ==============================
  5. The `processing` package mostly replicates the API of the `threading`
  6. module.
  7. Classes and exceptions
  8. ----------------------
  9. **class** `Process(group=None, target=None, name=None, args=(), kwargs={})`
  10. An analogue of `threading.Thread`.
  11. See `Process objects`_.
  12. **exception** `BufferTooShort`
  13. Exception raised by the `recvBytesInto()` method of a
  14. `connection object <connection-objects.html>`_
  15. when the supplied buffer object is too small for the message
  16. read.
  17. If `e` is an instance of `BufferTooShort` then `e.args[0]`
  18. will give the message as a byte string.
  19. Pipes and Queues
  20. ----------------
  21. When using multiple processes one generally uses message passing for
  22. communication between processes and avoids having to use any
  23. synchronization primitives like locks.
  24. For passing messages one can use a pipe (for a connection between two
  25. processes) or a queue (which allows multiple producers and consumers).
  26. Note that one can also create a shared queue by using a manager object
  27. -- see `Managers`_.
  28. For an example of the usage of queues for interprocess communication
  29. see `ex_workers.py <../examples/ex_workers.py>`_.
  30. `Pipe(duplex=True)`
  31. Returns a pair `(conn1, conn2)` of connection objects
  32. representing the ends of a pipe.
  33. If `duplex` is true then the pipe is two way; otherwise
  34. `conn1` can only be used for receiving messages and `conn2`
  35. can only be used for sending messages.
  36. See `Connection objects <connection-objects.html>`_.
  37. `Queue(maxsize=0)`
  38. Returns a process shared queue object. The usual `Empty` and
  39. `Full` exceptions from the standard library's `Queue` module
  40. are raised to signal timeouts.
  41. See `Queue objects <queue-objects.html>`_.
  42. Synchronization primitives
  43. --------------------------
  44. Generally synchronization primitives are not as necessary in a
  45. multiprocess program as they are in a mulithreaded program. See the
  46. documentation for the standard library's `threading` module.
  47. Note that one can also create synchronization primitves by using a
  48. manager object -- see `Managers`_.
  49. `BoundedSemaphore(value=1)`
  50. Returns a bounded semaphore object: a clone of
  51. `threading.BoundedSemaphore`.
  52. (On Mac OSX this is indistiguishable from `Semaphore()`
  53. because `sem_getvalue()` is not implemented on that platform).
  54. `Condition(lock=None)`
  55. Returns a condition variable: a clone of `threading.Condition`.
  56. If `lock` is specified then it should be a `Lock` or `RLock`
  57. object from `processing`.
  58. `Event()`
  59. Returns an event object: a clone of `threading.Event`.
  60. `Lock()`
  61. Returns a non-recursive lock object: a clone of `threading.Lock`.
  62. `RLock()`
  63. Returns a recursive lock object: a clone of `threading.RLock`.
  64. `Semaphore(value=1)`
  65. Returns a bounded semaphore object: a clone of
  66. `threading.Semaphore`.
  67. .. admonition:: Acquiring with a timeout
  68. The `acquire()` method of `BoundedSemaphore`, `Lock`, `RLock` and
  69. `Semaphore` has a timeout parameter not supported by the
  70. equivalents in `threading`. The signature is `acquire(block=True,
  71. timeout=None)` with keyword parameters being acceptable. If
  72. `block` is true and `timeout` is not `None` then it specifies a
  73. timeout in seconds. If `block` is false then `timeout` is ignored.
  74. .. admonition:: Interrupting the main thread
  75. If the SIGINT signal generated by Ctrl-C arrives while the main
  76. thread is blocked by a call to `BoundedSemaphore.acquire()`,
  77. `Lock.acquire()`, `RLock.acquire()`, `Semaphore.acquire()`,
  78. `Condition.acquire()` or `Condition.wait()` then the call will be
  79. immediately interrupted and `KeyboardInterrupt` will be raised.
  80. This differs from the behaviour of `threading` where SIGINT will be
  81. ignored while the equivalent blocking calls are in progress.
  82. Shared Objects
  83. --------------
  84. It is possible to create shared objects using shared memory which can
  85. be inherited by child processes.
  86. `Value(typecode_or_type, *args, **, lock=True)`
  87. Returns a ctypes object allocated from shared memory. By
  88. default the return value is actually a synchronized wrapper
  89. for the object.
  90. `typecode_or_type` determines the type of the returned object:
  91. it is either a ctypes type or a one character typecode of the
  92. kind used by the `array` module. `*args` is passed on to the
  93. constructor for the type.
  94. If `lock` is true (the default) then a new lock object is
  95. created to synchronize access to the value. If `lock` is a
  96. `Lock` or `RLock` object then that will be used to synchronize
  97. access to the value. If `lock` is false then access to the
  98. returned object will not be automatically protected by a lock,
  99. so it will not necessarily be "process-safe".
  100. Note that `lock` is a keyword only argument.
  101. `Array(typecode_or_type, size_or_initializer, **, lock=True)`
  102. Returns a ctypes array allocated from shared memory. By
  103. default the return value is actually a synchronized wrapper
  104. for the array.
  105. `typecode_or_type` determines the type of the elements of the
  106. returned array: it is either a ctypes type or a one character
  107. typecode of the kind used by the `array` module. If
  108. `size_or_initializer` is an integer then it determines the
  109. length of the array, and the array will be initially zeroed.
  110. Otherwise `size_or_initializer` is a sequence which is used to
  111. initialize the array and whose length determines the length of
  112. the array.
  113. If `lock` is true (the default) then a new lock object is
  114. created to synchronize access to the value. If `lock` is a
  115. `Lock` or `RLock` object then that will be used to synchronize
  116. access to the value. If `lock` is false then access to the
  117. returned object will not be automatically protected by a lock,
  118. so it will not necessarily be "process-safe".
  119. Note that `lock` is a keyword only argument.
  120. Note that an array of `ctypes.c_char` has `value` and
  121. `rawvalue` attributes which allow one to use it to store and
  122. retrieve strings -- see the documentation for ctypes in the
  123. standard library.
  124. See also `sharedctypes <sharedctypes.html>`_.
  125. Managers
  126. --------
  127. Managers provide a way to create data which can be shared between
  128. different processes.
  129. `Manager()`
  130. Returns a started `SyncManager` object which can be
  131. used for sharing objects between processes. The returned
  132. manager object corresponds to a spawned child process and has
  133. methods which will create shared objects and return
  134. corresponding proxies.
  135. The methods for creating shared objects are
  136. `list()`, `dict()`, `Namespace()`, `Value()`,
  137. `Array()`, `Lock()`, `RLock()`, `Semaphore()`,
  138. `BoundedSemaphore()`, `Condition()`, `Event()`, `Queue()`.
  139. See `SyncManager <manager-objects.html#sync-manager>`_.
  140. It is possible to create managers which support other types -- see
  141. `Customized managers <manager-objects.html#customized-managers>`_.
  142. Process Pools
  143. -------------
  144. One can create a pool of processes which will carry out tasks
  145. submitted to it.
  146. `Pool(processes=None, initializer=None, initargs=())`
  147. Returns a process pool object which controls a pool of worker
  148. processes to which jobs can be submitted.
  149. It supports asynchronous results with timeouts and
  150. callbacks and has a parallel map implementation.
  151. `processes` is the number of worker processes to use. If
  152. `processes` is `None` then the number returned by `cpuCount()`
  153. is used. If `initializer` is not `None` then each worker
  154. process will call `initializer(*initargs)` when it starts.
  155. See `Pool objects <pool-objects.html>`_.
  156. Logging
  157. -------
  158. Some support for logging is available. Note, however, that the
  159. `logging` package does not use process shared locks so it is possible
  160. (depending on the handler type) for messages from different processes
  161. to get mixed up.
  162. `enableLogging(level, HandlerType=None, handlerArgs=(), format=None)`
  163. Enables logging and sets the debug level used by the package's
  164. logger to `level`. See documentation for the `logging` module
  165. in the standard library.
  166. If `HandlerType` is specified then a handler is created using
  167. `HandlerType(*handlerArgs)` and this will be used by the
  168. logger -- any previous handlers will be discarded. If
  169. `format` is specified then this will be used for the handler;
  170. otherwise `format` defaults to
  171. `'[%(levelname)s/%(processName)s] %(message)s'`. (The logger
  172. used by `processing` allows use of the non-standard
  173. `'%(processName)s'` format.)
  174. If `HandlerType` is not specified and the logger has no
  175. handlers then a default one is created which prints to
  176. `sys.stderr`.
  177. *Note*: on Windows a child process does not directly inherit
  178. its parent's logger; instead it will automatically call
  179. `enableLogging()` with the same arguments which were used when
  180. its parent process last called `enableLogging()` (if it ever
  181. did).
  182. `getLogger()`
  183. Returns the logger used by `processing`. If `enableLogging()`
  184. has not yet been called then `None` is returned.
  185. Below is an example session with logging turned on::
  186. >>> import processing, logging
  187. >>> processing.enableLogging(level=logging.INFO)
  188. >>> processing.getLogger().warning('doomed')
  189. [WARNING/MainProcess] doomed
  190. >>> m = processing.Manager()
  191. [INFO/SyncManager-1] child process calling self.run()
  192. [INFO/SyncManager-1] manager bound to '\\\\.\\pipe\\pyc-2776-0-lj0tfa'
  193. >>> del m
  194. [INFO/MainProcess] sending shutdown message to manager
  195. [INFO/SyncManager-1] manager received shutdown message
  196. [INFO/SyncManager-1] manager exiting with exitcode 0
  197. Miscellaneous
  198. -------------
  199. `activeChildren()`
  200. Return list of all live children of the current process.
  201. Calling this has the side affect of "joining" any processes
  202. which have already finished.
  203. `cpuCount()`
  204. Returns the number of CPUs in the system. May raise
  205. `NotImplementedError`.
  206. `currentProcess()`
  207. An analogue of `threading.currentThread()`.
  208. Returns the object corresponding to the current process.
  209. `freezeSupport()`
  210. Adds support for when a program which uses the `processing`
  211. package has been frozen to produce a Windows executable. (Has
  212. been tested with `py2exe`, `PyInstaller` and `cx_Freeze`.)
  213. One needs to call this function straight after the `if __name__
  214. == '__main__'` line of the main module. For example ::
  215. from processing import Process, freezeSupport
  216. def f():
  217. print 'hello world!'
  218. if __name__ == '__main__':
  219. freezeSupport()
  220. Process(target=f).start()
  221. If the `freezeSupport()` line is missed out then trying to run
  222. the frozen executable will raise `RuntimeError`.
  223. If the module is being run normally by the python interpreter
  224. then `freezeSupport()` has no effect.
  225. .. note::
  226. * The `processing.dummy` package replicates the API of `processing`
  227. but is no more than a wrapper around the `threading` module.
  228. * `processing` contains no analogues of `activeCount`,
  229. `enumerate`, `settrace`, `setprofile`, `Timer`, or
  230. `local` from the `threading` module.
  231. Subsections
  232. -----------
  233. + `Process objects <process-objects.html>`_
  234. + `Queue objects <queue-objects.html>`_
  235. + `Connection objects <connection-objects.html>`_
  236. + `Manager objects <manager-objects.html>`_
  237. + `Proxy objects <proxy-objects.html>`_
  238. + `Pool objects <pool-objects.html>`_
  239. + `Shared ctypes object <sharedctypes.html>`_
  240. + `Listeners and Clients <connection-ref.html>`_
  241. .. _Prev: intro.html
  242. .. _Up: index.html
  243. .. _Next: process-objects.html