| 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340 |
- .. include:: header.txt
- ==============================
- processing package reference
- ==============================
- The `processing` package mostly replicates the API of the `threading`
- module.
- Classes and exceptions
- ----------------------
- **class** `Process(group=None, target=None, name=None, args=(), kwargs={})`
- An analogue of `threading.Thread`.
- See `Process objects`_.
- **exception** `BufferTooShort`
- Exception raised by the `recvBytesInto()` method of a
- `connection object <connection-objects.html>`_
- when the supplied buffer object is too small for the message
- read.
- If `e` is an instance of `BufferTooShort` then `e.args[0]`
- will give the message as a byte string.
- Pipes and Queues
- ----------------
- When using multiple processes one generally uses message passing for
- communication between processes and avoids having to use any
- synchronization primitives like locks.
- For passing messages one can use a pipe (for a connection between two
- processes) or a queue (which allows multiple producers and consumers).
- Note that one can also create a shared queue by using a manager object
- -- see `Managers`_.
- For an example of the usage of queues for interprocess communication
- see `ex_workers.py <../examples/ex_workers.py>`_.
- `Pipe(duplex=True)`
- Returns a pair `(conn1, conn2)` of connection objects
- representing the ends of a pipe.
- If `duplex` is true then the pipe is two way; otherwise
- `conn1` can only be used for receiving messages and `conn2`
- can only be used for sending messages.
-
- See `Connection objects <connection-objects.html>`_.
- `Queue(maxsize=0)`
- Returns a process shared queue object. The usual `Empty` and
- `Full` exceptions from the standard library's `Queue` module
- are raised to signal timeouts.
- See `Queue objects <queue-objects.html>`_.
-
- Synchronization primitives
- --------------------------
- Generally synchronization primitives are not as necessary in a
- multiprocess program as they are in a mulithreaded program. See the
- documentation for the standard library's `threading` module.
- Note that one can also create synchronization primitves by using a
- manager object -- see `Managers`_.
- `BoundedSemaphore(value=1)`
- Returns a bounded semaphore object: a clone of
- `threading.BoundedSemaphore`.
- (On Mac OSX this is indistiguishable from `Semaphore()`
- because `sem_getvalue()` is not implemented on that platform).
- `Condition(lock=None)`
- Returns a condition variable: a clone of `threading.Condition`.
-
- If `lock` is specified then it should be a `Lock` or `RLock`
- object from `processing`.
-
- `Event()`
- Returns an event object: a clone of `threading.Event`.
- `Lock()`
- Returns a non-recursive lock object: a clone of `threading.Lock`.
-
- `RLock()`
- Returns a recursive lock object: a clone of `threading.RLock`.
-
- `Semaphore(value=1)`
- Returns a bounded semaphore object: a clone of
- `threading.Semaphore`.
- .. admonition:: Acquiring with a timeout
- The `acquire()` method of `BoundedSemaphore`, `Lock`, `RLock` and
- `Semaphore` has a timeout parameter not supported by the
- equivalents in `threading`. The signature is `acquire(block=True,
- timeout=None)` with keyword parameters being acceptable. If
- `block` is true and `timeout` is not `None` then it specifies a
- timeout in seconds. If `block` is false then `timeout` is ignored.
- .. admonition:: Interrupting the main thread
- If the SIGINT signal generated by Ctrl-C arrives while the main
- thread is blocked by a call to `BoundedSemaphore.acquire()`,
- `Lock.acquire()`, `RLock.acquire()`, `Semaphore.acquire()`,
- `Condition.acquire()` or `Condition.wait()` then the call will be
- immediately interrupted and `KeyboardInterrupt` will be raised.
- This differs from the behaviour of `threading` where SIGINT will be
- ignored while the equivalent blocking calls are in progress.
- Shared Objects
- --------------
- It is possible to create shared objects using shared memory which can
- be inherited by child processes.
- `Value(typecode_or_type, *args, **, lock=True)`
- Returns a ctypes object allocated from shared memory. By
- default the return value is actually a synchronized wrapper
- for the object.
- `typecode_or_type` determines the type of the returned object:
- it is either a ctypes type or a one character typecode of the
- kind used by the `array` module. `*args` is passed on to the
- constructor for the type.
- If `lock` is true (the default) then a new lock object is
- created to synchronize access to the value. If `lock` is a
- `Lock` or `RLock` object then that will be used to synchronize
- access to the value. If `lock` is false then access to the
- returned object will not be automatically protected by a lock,
- so it will not necessarily be "process-safe".
- Note that `lock` is a keyword only argument.
- `Array(typecode_or_type, size_or_initializer, **, lock=True)`
- Returns a ctypes array allocated from shared memory. By
- default the return value is actually a synchronized wrapper
- for the array.
- `typecode_or_type` determines the type of the elements of the
- returned array: it is either a ctypes type or a one character
- typecode of the kind used by the `array` module. If
- `size_or_initializer` is an integer then it determines the
- length of the array, and the array will be initially zeroed.
- Otherwise `size_or_initializer` is a sequence which is used to
- initialize the array and whose length determines the length of
- the array.
- If `lock` is true (the default) then a new lock object is
- created to synchronize access to the value. If `lock` is a
- `Lock` or `RLock` object then that will be used to synchronize
- access to the value. If `lock` is false then access to the
- returned object will not be automatically protected by a lock,
- so it will not necessarily be "process-safe".
- Note that `lock` is a keyword only argument.
- Note that an array of `ctypes.c_char` has `value` and
- `rawvalue` attributes which allow one to use it to store and
- retrieve strings -- see the documentation for ctypes in the
- standard library.
- See also `sharedctypes <sharedctypes.html>`_.
- Managers
- --------
- Managers provide a way to create data which can be shared between
- different processes.
- `Manager()`
- Returns a started `SyncManager` object which can be
- used for sharing objects between processes. The returned
- manager object corresponds to a spawned child process and has
- methods which will create shared objects and return
- corresponding proxies.
- The methods for creating shared objects are
- `list()`, `dict()`, `Namespace()`, `Value()`,
- `Array()`, `Lock()`, `RLock()`, `Semaphore()`,
- `BoundedSemaphore()`, `Condition()`, `Event()`, `Queue()`.
-
- See `SyncManager <manager-objects.html#sync-manager>`_.
- It is possible to create managers which support other types -- see
- `Customized managers <manager-objects.html#customized-managers>`_.
- Process Pools
- -------------
- One can create a pool of processes which will carry out tasks
- submitted to it.
- `Pool(processes=None, initializer=None, initargs=())`
- Returns a process pool object which controls a pool of worker
- processes to which jobs can be submitted.
- It supports asynchronous results with timeouts and
- callbacks and has a parallel map implementation.
- `processes` is the number of worker processes to use. If
- `processes` is `None` then the number returned by `cpuCount()`
- is used. If `initializer` is not `None` then each worker
- process will call `initializer(*initargs)` when it starts.
- See `Pool objects <pool-objects.html>`_.
-
- Logging
- -------
- Some support for logging is available. Note, however, that the
- `logging` package does not use process shared locks so it is possible
- (depending on the handler type) for messages from different processes
- to get mixed up.
- `enableLogging(level, HandlerType=None, handlerArgs=(), format=None)`
- Enables logging and sets the debug level used by the package's
- logger to `level`. See documentation for the `logging` module
- in the standard library.
- If `HandlerType` is specified then a handler is created using
- `HandlerType(*handlerArgs)` and this will be used by the
- logger -- any previous handlers will be discarded. If
- `format` is specified then this will be used for the handler;
- otherwise `format` defaults to
- `'[%(levelname)s/%(processName)s] %(message)s'`. (The logger
- used by `processing` allows use of the non-standard
- `'%(processName)s'` format.)
- If `HandlerType` is not specified and the logger has no
- handlers then a default one is created which prints to
- `sys.stderr`.
- *Note*: on Windows a child process does not directly inherit
- its parent's logger; instead it will automatically call
- `enableLogging()` with the same arguments which were used when
- its parent process last called `enableLogging()` (if it ever
- did).
- `getLogger()`
- Returns the logger used by `processing`. If `enableLogging()`
- has not yet been called then `None` is returned.
- Below is an example session with logging turned on::
- >>> import processing, logging
- >>> processing.enableLogging(level=logging.INFO)
- >>> processing.getLogger().warning('doomed')
- [WARNING/MainProcess] doomed
- >>> m = processing.Manager()
- [INFO/SyncManager-1] child process calling self.run()
- [INFO/SyncManager-1] manager bound to '\\\\.\\pipe\\pyc-2776-0-lj0tfa'
- >>> del m
- [INFO/MainProcess] sending shutdown message to manager
- [INFO/SyncManager-1] manager received shutdown message
- [INFO/SyncManager-1] manager exiting with exitcode 0
- Miscellaneous
- -------------
- `activeChildren()`
- Return list of all live children of the current process.
-
- Calling this has the side affect of "joining" any processes
- which have already finished.
- `cpuCount()`
- Returns the number of CPUs in the system. May raise
- `NotImplementedError`.
- `currentProcess()`
- An analogue of `threading.currentThread()`.
- Returns the object corresponding to the current process.
- `freezeSupport()`
- Adds support for when a program which uses the `processing`
- package has been frozen to produce a Windows executable. (Has
- been tested with `py2exe`, `PyInstaller` and `cx_Freeze`.)
- One needs to call this function straight after the `if __name__
- == '__main__'` line of the main module. For example ::
- from processing import Process, freezeSupport
- def f():
- print 'hello world!'
- if __name__ == '__main__':
- freezeSupport()
- Process(target=f).start()
- If the `freezeSupport()` line is missed out then trying to run
- the frozen executable will raise `RuntimeError`.
- If the module is being run normally by the python interpreter
- then `freezeSupport()` has no effect.
-
- .. note::
- * The `processing.dummy` package replicates the API of `processing`
- but is no more than a wrapper around the `threading` module.
-
- * `processing` contains no analogues of `activeCount`,
- `enumerate`, `settrace`, `setprofile`, `Timer`, or
- `local` from the `threading` module.
- Subsections
- -----------
- + `Process objects <process-objects.html>`_
- + `Queue objects <queue-objects.html>`_
- + `Connection objects <connection-objects.html>`_
- + `Manager objects <manager-objects.html>`_
- + `Proxy objects <proxy-objects.html>`_
- + `Pool objects <pool-objects.html>`_
- + `Shared ctypes object <sharedctypes.html>`_
- + `Listeners and Clients <connection-ref.html>`_
- .. _Prev: intro.html
- .. _Up: index.html
- .. _Next: process-objects.html
|