pool-objects.txt 5.9 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160
  1. .. include:: header.txt
  2. ===============
  3. Process Pools
  4. ===============
  5. The `processing.pool` module has one public class:
  6. **class** `Pool(processes=None, initializer=None, initargs=())`
  7. A class representing a pool of worker processes.
  8. Tasks can be offloaded to the pool and the results dealt with
  9. when they become available.
  10. Note that tasks can only be submitted (or retrieved) by the
  11. process which created the pool object.
  12. `processes` is the number of worker processes to use. If
  13. `processes` is `None` then the number returned by `cpuCount()`
  14. is used. If `initializer` is not `None` then each worker
  15. process will call `initializer(*initargs)` when it starts.
  16. Pool objects
  17. ============
  18. `Pool` has the following public methods:
  19. `__init__(processes=None)`
  20. The constructor creates and starts `processes` worker
  21. processes. If `processes` is `None` then `cpuCount()` is used
  22. to find a default or 1 if `cpuCount()` raises `NotImplemented`.
  23. `apply(func, args=(), kwds={})`
  24. Equivalent of the `apply()` builtin function. It blocks till
  25. the result is ready.
  26. `applyAsync(func, args=(), kwds={}, callback=None)`
  27. A variant of the `apply()` method which returns a
  28. result object --- see `Asynchronous result objects`_.
  29. If `callback` is specified then it should be a callable which
  30. accepts a single argument. When the result becomes ready
  31. `callback` is applied to it (unless the call failed).
  32. `callback` should complete immediately since otherwise the
  33. thread which handles the results will get blocked.
  34. `map(func, iterable, chunksize=None)`
  35. A parallel equivalent of the `map()` builtin function. It
  36. blocks till the result is ready.
  37. This method chops the iterable into a number of chunks which
  38. it submits to the process pool as separate tasks. The
  39. (approximate) size of these chunks can be specified by setting
  40. `chunksize` to a positive integer.
  41. `mapAsync(func, iterable, chunksize=None, callback=None)`
  42. A variant of the `map()` method which returns a result object
  43. --- see `Asynchronous result objects`_.
  44. If `callback` is specified then it should be a callable which
  45. accepts a single argument. When the result becomes ready
  46. `callback` is applied to it (unless the call failed).
  47. `callback` should complete immediately since otherwise the
  48. thread which handles the results will get blocked.
  49. `imap(func, iterable, chunksize=1)`
  50. An equivalent of `itertools.imap()`.
  51. The `chunksize` argument is the same as the one used by the
  52. `map()` method. For very long iterables using a large value
  53. for `chunksize` can make make the job complete **much** faster
  54. than using the default value of `1`.
  55. Also if `chunksize` is `1` then the `next()` method of the
  56. iterator returned by the `imap()` method has an optional
  57. `timeout` parameter: `next(timeout)` will raise
  58. `processing.TimeoutError` if the result cannot be returned
  59. within `timeout` seconds.
  60. `imapUnordered(func, iterable, chunksize=1)`
  61. The same as `imap()` except that the ordering of the results
  62. from the returned iterator should be considered arbitrary.
  63. (Only when there is only one worker process is the order
  64. guaranteed to be "correct".)
  65. `close()`
  66. Prevents any more tasks from being submitted to the pool.
  67. Once all the tasks have been completed the worker processes
  68. will exit.
  69. `terminate()`
  70. Stops the worker processes immediately without completing
  71. outstanding work. When the pool object is garbage collected
  72. `terminate()` will be called immediately.
  73. `join()`
  74. Wait for the worker processes to exit. One must call
  75. `close()` or `terminate()` before using `join()`.
  76. Asynchronous result objects
  77. ===========================
  78. The result objects returns by `applyAsync()` and `mapAsync()` have
  79. the following public methods:
  80. `get(timeout=None)`
  81. Returns the result when it arrives. If `timeout` is not
  82. `None` and the result does not arrive within `timeout` seconds
  83. then `processing.TimeoutError` is raised. If the remote call
  84. raised an exception then that exception will be reraised by `get()`.
  85. `wait(timeout=None)`
  86. Waits until the result is available or until `timeout` seconds
  87. pass.
  88. `ready()`
  89. Returns whether the call has completed.
  90. `successful()`
  91. Returns whether the call completed without raising an
  92. exception. Will raise `AssertionError` if the result is not
  93. ready.
  94. Examples
  95. ========
  96. The following example demonstrates the use of a pool::
  97. from processing import Pool
  98. def f(x):
  99. return x*x
  100. if __name__ == '__main__':
  101. pool = Pool(processes=4) # start 4 worker processes
  102. result = pool.applyAsync(f, (10,)) # evaluate "f(10)" asynchronously
  103. print result.get(timeout=1) # prints "100" unless your computer is *very* slow
  104. print pool.map(f, range(10)) # prints "[0, 1, 4,..., 81]"
  105. it = pool.imap(f, range(10))
  106. print it.next() # prints "0"
  107. print it.next() # prints "1"
  108. print it.next(timeout=1) # prints "4" unless your computer is *very* slow
  109. import time
  110. result = pool.applyAsync(time.sleep, (10,))
  111. print result.get(timeout=1) # raises `TimeoutError`
  112. See also `ex_pool.py <../examples/ex_pool.py>`_.
  113. .. _Prev: proxy-objects.html
  114. .. _Up: processing-ref.html
  115. .. _Next: sharedctypes.html