pool-objects.html 11 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159
  1. <?xml version="1.0" encoding="utf-8" ?>
  2. <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
  3. <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  4. <head>
  5. <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
  6. <meta name="generator" content="Docutils 0.4: http://docutils.sourceforge.net/" />
  7. <title>Process Pools</title>
  8. <link rel="stylesheet" href="html4css1.css" type="text/css" />
  9. </head>
  10. <body>
  11. <div class="header">
  12. <a class="reference" href="proxy-objects.html">Prev</a> &nbsp; &nbsp; &nbsp; &nbsp; <a class="reference" href="processing-ref.html">Up</a> &nbsp; &nbsp; &nbsp; &nbsp; <a class="reference" href="sharedctypes.html">Next</a>
  13. <hr class="header"/>
  14. </div>
  15. <div class="document" id="process-pools">
  16. <h1 class="title">Process Pools</h1>
  17. <p>The <tt class="docutils literal"><span class="pre">processing.pool</span></tt> module has one public class:</p>
  18. <blockquote>
  19. <dl class="docutils">
  20. <dt><strong>class</strong> <tt class="docutils literal"><span class="pre">Pool(processes=None,</span> <span class="pre">initializer=None,</span> <span class="pre">initargs=())</span></tt></dt>
  21. <dd><p class="first">A class representing a pool of worker processes.</p>
  22. <p>Tasks can be offloaded to the pool and the results dealt with
  23. when they become available.</p>
  24. <p>Note that tasks can only be submitted (or retrieved) by the
  25. process which created the pool object.</p>
  26. <p class="last"><tt class="docutils literal"><span class="pre">processes</span></tt> is the number of worker processes to use. If
  27. <tt class="docutils literal"><span class="pre">processes</span></tt> is <tt class="docutils literal"><span class="pre">None</span></tt> then the number returned by <tt class="docutils literal"><span class="pre">cpuCount()</span></tt>
  28. is used. If <tt class="docutils literal"><span class="pre">initializer</span></tt> is not <tt class="docutils literal"><span class="pre">None</span></tt> then each worker
  29. process will call <tt class="docutils literal"><span class="pre">initializer(*initargs)</span></tt> when it starts.</p>
  30. </dd>
  31. </dl>
  32. </blockquote>
  33. <div class="section">
  34. <h1><a id="pool-objects" name="pool-objects">Pool objects</a></h1>
  35. <p><tt class="docutils literal"><span class="pre">Pool</span></tt> has the following public methods:</p>
  36. <blockquote>
  37. <dl class="docutils">
  38. <dt><tt class="docutils literal"><span class="pre">__init__(processes=None)</span></tt></dt>
  39. <dd>The constructor creates and starts <tt class="docutils literal"><span class="pre">processes</span></tt> worker
  40. processes. If <tt class="docutils literal"><span class="pre">processes</span></tt> is <tt class="docutils literal"><span class="pre">None</span></tt> then <tt class="docutils literal"><span class="pre">cpuCount()</span></tt> is used
  41. to find a default or 1 if <tt class="docutils literal"><span class="pre">cpuCount()</span></tt> raises <tt class="docutils literal"><span class="pre">NotImplemented</span></tt>.</dd>
  42. <dt><tt class="docutils literal"><span class="pre">apply(func,</span> <span class="pre">args=(),</span> <span class="pre">kwds={})</span></tt></dt>
  43. <dd>Equivalent of the <tt class="docutils literal"><span class="pre">apply()</span></tt> builtin function. It blocks till
  44. the result is ready.</dd>
  45. <dt><tt class="docutils literal"><span class="pre">applyAsync(func,</span> <span class="pre">args=(),</span> <span class="pre">kwds={},</span> <span class="pre">callback=None)</span></tt></dt>
  46. <dd><p class="first">A variant of the <tt class="docutils literal"><span class="pre">apply()</span></tt> method which returns a
  47. result object --- see <a class="reference" href="#asynchronous-result-objects">Asynchronous result objects</a>.</p>
  48. <p class="last">If <tt class="docutils literal"><span class="pre">callback</span></tt> is specified then it should be a callable which
  49. accepts a single argument. When the result becomes ready
  50. <tt class="docutils literal"><span class="pre">callback</span></tt> is applied to it (unless the call failed).
  51. <tt class="docutils literal"><span class="pre">callback</span></tt> should complete immediately since otherwise the
  52. thread which handles the results will get blocked.</p>
  53. </dd>
  54. <dt><tt class="docutils literal"><span class="pre">map(func,</span> <span class="pre">iterable,</span> <span class="pre">chunksize=None)</span></tt></dt>
  55. <dd><p class="first">A parallel equivalent of the <tt class="docutils literal"><span class="pre">map()</span></tt> builtin function. It
  56. blocks till the result is ready.</p>
  57. <p class="last">This method chops the iterable into a number of chunks which
  58. it submits to the process pool as separate tasks. The
  59. (approximate) size of these chunks can be specified by setting
  60. <tt class="docutils literal"><span class="pre">chunksize</span></tt> to a positive integer.</p>
  61. </dd>
  62. <dt><tt class="docutils literal"><span class="pre">mapAsync(func,</span> <span class="pre">iterable,</span> <span class="pre">chunksize=None,</span> <span class="pre">callback=None)</span></tt></dt>
  63. <dd><p class="first">A variant of the <tt class="docutils literal"><span class="pre">map()</span></tt> method which returns a result object
  64. --- see <a class="reference" href="#asynchronous-result-objects">Asynchronous result objects</a>.</p>
  65. <p class="last">If <tt class="docutils literal"><span class="pre">callback</span></tt> is specified then it should be a callable which
  66. accepts a single argument. When the result becomes ready
  67. <tt class="docutils literal"><span class="pre">callback</span></tt> is applied to it (unless the call failed).
  68. <tt class="docutils literal"><span class="pre">callback</span></tt> should complete immediately since otherwise the
  69. thread which handles the results will get blocked.</p>
  70. </dd>
  71. <dt><tt class="docutils literal"><span class="pre">imap(func,</span> <span class="pre">iterable,</span> <span class="pre">chunksize=1)</span></tt></dt>
  72. <dd><p class="first">An equivalent of <tt class="docutils literal"><span class="pre">itertools.imap()</span></tt>.</p>
  73. <p>The <tt class="docutils literal"><span class="pre">chunksize</span></tt> argument is the same as the one used by the
  74. <tt class="docutils literal"><span class="pre">map()</span></tt> method. For very long iterables using a large value
  75. for <tt class="docutils literal"><span class="pre">chunksize</span></tt> can make make the job complete <strong>much</strong> faster
  76. than using the default value of <tt class="docutils literal"><span class="pre">1</span></tt>.</p>
  77. <p class="last">Also if <tt class="docutils literal"><span class="pre">chunksize</span></tt> is <tt class="docutils literal"><span class="pre">1</span></tt> then the <tt class="docutils literal"><span class="pre">next()</span></tt> method of the
  78. iterator returned by the <tt class="docutils literal"><span class="pre">imap()</span></tt> method has an optional
  79. <tt class="docutils literal"><span class="pre">timeout</span></tt> parameter: <tt class="docutils literal"><span class="pre">next(timeout)</span></tt> will raise
  80. <tt class="docutils literal"><span class="pre">processing.TimeoutError</span></tt> if the result cannot be returned
  81. within <tt class="docutils literal"><span class="pre">timeout</span></tt> seconds.</p>
  82. </dd>
  83. <dt><tt class="docutils literal"><span class="pre">imapUnordered(func,</span> <span class="pre">iterable,</span> <span class="pre">chunksize=1)</span></tt></dt>
  84. <dd>The same as <tt class="docutils literal"><span class="pre">imap()</span></tt> except that the ordering of the results
  85. from the returned iterator should be considered arbitrary.
  86. (Only when there is only one worker process is the order
  87. guaranteed to be &quot;correct&quot;.)</dd>
  88. <dt><tt class="docutils literal"><span class="pre">close()</span></tt></dt>
  89. <dd>Prevents any more tasks from being submitted to the pool.
  90. Once all the tasks have been completed the worker processes
  91. will exit.</dd>
  92. <dt><tt class="docutils literal"><span class="pre">terminate()</span></tt></dt>
  93. <dd>Stops the worker processes immediately without completing
  94. outstanding work. When the pool object is garbage collected
  95. <tt class="docutils literal"><span class="pre">terminate()</span></tt> will be called immediately.</dd>
  96. <dt><tt class="docutils literal"><span class="pre">join()</span></tt></dt>
  97. <dd>Wait for the worker processes to exit. One must call
  98. <tt class="docutils literal"><span class="pre">close()</span></tt> or <tt class="docutils literal"><span class="pre">terminate()</span></tt> before using <tt class="docutils literal"><span class="pre">join()</span></tt>.</dd>
  99. </dl>
  100. </blockquote>
  101. </div>
  102. <div class="section">
  103. <h1><a id="asynchronous-result-objects" name="asynchronous-result-objects">Asynchronous result objects</a></h1>
  104. <p>The result objects returns by <tt class="docutils literal"><span class="pre">applyAsync()</span></tt> and <tt class="docutils literal"><span class="pre">mapAsync()</span></tt> have
  105. the following public methods:</p>
  106. <blockquote>
  107. <dl class="docutils">
  108. <dt><tt class="docutils literal"><span class="pre">get(timeout=None)</span></tt></dt>
  109. <dd>Returns the result when it arrives. If <tt class="docutils literal"><span class="pre">timeout</span></tt> is not
  110. <tt class="docutils literal"><span class="pre">None</span></tt> and the result does not arrive within <tt class="docutils literal"><span class="pre">timeout</span></tt> seconds
  111. then <tt class="docutils literal"><span class="pre">processing.TimeoutError</span></tt> is raised. If the remote call
  112. raised an exception then that exception will be reraised by <tt class="docutils literal"><span class="pre">get()</span></tt>.</dd>
  113. <dt><tt class="docutils literal"><span class="pre">wait(timeout=None)</span></tt></dt>
  114. <dd>Waits until the result is available or until <tt class="docutils literal"><span class="pre">timeout</span></tt> seconds
  115. pass.</dd>
  116. <dt><tt class="docutils literal"><span class="pre">ready()</span></tt></dt>
  117. <dd>Returns whether the call has completed.</dd>
  118. <dt><tt class="docutils literal"><span class="pre">successful()</span></tt></dt>
  119. <dd>Returns whether the call completed without raising an
  120. exception. Will raise <tt class="docutils literal"><span class="pre">AssertionError</span></tt> if the result is not
  121. ready.</dd>
  122. </dl>
  123. </blockquote>
  124. </div>
  125. <div class="section">
  126. <h1><a id="examples" name="examples">Examples</a></h1>
  127. <p>The following example demonstrates the use of a pool:</p>
  128. <pre class="literal-block">
  129. from processing import Pool
  130. def f(x):
  131. return x*x
  132. if __name__ == '__main__':
  133. pool = Pool(processes=4) # start 4 worker processes
  134. result = pool.applyAsync(f, (10,)) # evaluate &quot;f(10)&quot; asynchronously
  135. print result.get(timeout=1) # prints &quot;100&quot; unless your computer is *very* slow
  136. print pool.map(f, range(10)) # prints &quot;[0, 1, 4,..., 81]&quot;
  137. it = pool.imap(f, range(10))
  138. print it.next() # prints &quot;0&quot;
  139. print it.next() # prints &quot;1&quot;
  140. print it.next(timeout=1) # prints &quot;4&quot; unless your computer is *very* slow
  141. import time
  142. result = pool.applyAsync(time.sleep, (10,))
  143. print result.get(timeout=1) # raises `TimeoutError`
  144. </pre>
  145. <p>See also <a class="reference" href="../examples/ex_pool.py">ex_pool.py</a>.</p>
  146. </div>
  147. </div>
  148. <div class="footer">
  149. <hr class="footer" />
  150. <a class="reference" href="proxy-objects.html">Prev</a> &nbsp; &nbsp; &nbsp; &nbsp; <a class="reference" href="processing-ref.html">Up</a> &nbsp; &nbsp; &nbsp; &nbsp; <a class="reference" href="sharedctypes.html">Next</a>
  151. </div>
  152. </body>
  153. </html>