queue-objects.html 9.4 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129
  1. <?xml version="1.0" encoding="utf-8" ?>
  2. <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
  3. <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  4. <head>
  5. <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
  6. <meta name="generator" content="Docutils 0.4: http://docutils.sourceforge.net/" />
  7. <title>Queue objects</title>
  8. <link rel="stylesheet" href="html4css1.css" type="text/css" />
  9. </head>
  10. <body>
  11. <div class="header">
  12. <a class="reference" href="process-objects.html">Prev</a> &nbsp; &nbsp; &nbsp; &nbsp; <a class="reference" href="processing-ref.html">Up</a> &nbsp; &nbsp; &nbsp; &nbsp; <a class="reference" href="connection-objects.html">Next</a>
  13. <hr class="header"/>
  14. </div>
  15. <div class="document" id="queue-objects">
  16. <h1 class="title">Queue objects</h1>
  17. <p>The queue type provided by <tt class="docutils literal"><span class="pre">processing</span></tt> is a multi-producer,
  18. multi-consumer FIFO queue modelled on the <tt class="docutils literal"><span class="pre">Queue.Queue</span></tt> class in the
  19. standard library.</p>
  20. <dl class="docutils">
  21. <dt><tt class="docutils literal"><span class="pre">Queue(maxsize=0)</span></tt></dt>
  22. <dd><p class="first">Returns a process shared queue implemented using a pipe and a few
  23. locks/semaphores. When a process first puts an item on the queue
  24. a feeder thread is started which transfers objects from a
  25. buffer into the pipe.</p>
  26. <p><tt class="docutils literal"><span class="pre">Queue.Queue</span></tt> implements all the methods of <tt class="docutils literal"><span class="pre">Queue.Queue</span></tt> except for
  27. <tt class="docutils literal"><span class="pre">qsize()</span></tt>, <tt class="docutils literal"><span class="pre">task_done()</span></tt> and <tt class="docutils literal"><span class="pre">join()</span></tt>.</p>
  28. <blockquote>
  29. <dl class="docutils">
  30. <dt><tt class="docutils literal"><span class="pre">empty()</span></tt></dt>
  31. <dd>Return <tt class="docutils literal"><span class="pre">True</span></tt> if the queue is empty, <tt class="docutils literal"><span class="pre">False</span></tt>
  32. otherwise. Because of multithreading/multiprocessing
  33. semantics, this is not reliable.</dd>
  34. <dt><tt class="docutils literal"><span class="pre">full()</span></tt></dt>
  35. <dd>Return <tt class="docutils literal"><span class="pre">True</span></tt> if the queue is full, <tt class="docutils literal"><span class="pre">False</span></tt> otherwise. Because
  36. of multithreading/multiprocessing semantics, this is not
  37. reliable.</dd>
  38. <dt><tt class="docutils literal"><span class="pre">put(item,</span> <span class="pre">block=True,</span> <span class="pre">timeout=None)</span></tt></dt>
  39. <dd>Put item into the queue. If optional args <tt class="docutils literal"><span class="pre">block</span></tt> is true
  40. and <tt class="docutils literal"><span class="pre">timeout</span></tt> is <tt class="docutils literal"><span class="pre">None</span></tt> (the default), block if necessary
  41. until a free slot is available. If <tt class="docutils literal"><span class="pre">timeout</span></tt> is a positive
  42. number, it blocks at most <tt class="docutils literal"><span class="pre">timeout</span></tt> seconds and raises the
  43. <tt class="docutils literal"><span class="pre">Full</span></tt> exception if no free slot was available within that
  44. time. Otherwise (<tt class="docutils literal"><span class="pre">block</span></tt> is false), put an item on the queue
  45. if a free slot is immediately available, else raise the
  46. <tt class="docutils literal"><span class="pre">Full</span></tt> exception (<tt class="docutils literal"><span class="pre">timeout</span></tt> is ignored in that case).</dd>
  47. <dt><tt class="docutils literal"><span class="pre">put_nowait(item)</span></tt>, <tt class="docutils literal"><span class="pre">putNoWait(item)</span></tt></dt>
  48. <dd>Equivalent to <tt class="docutils literal"><span class="pre">put(item,</span> <span class="pre">False)</span></tt>.</dd>
  49. <dt><tt class="docutils literal"><span class="pre">get(block=True,</span> <span class="pre">timeout=None)</span></tt></dt>
  50. <dd>Remove and return an item from the queue. If optional args
  51. <tt class="docutils literal"><span class="pre">block</span></tt> is true and <tt class="docutils literal"><span class="pre">timeout</span></tt> is <tt class="docutils literal"><span class="pre">None</span></tt> (the default),
  52. block if necessary until an item is available. If
  53. <tt class="docutils literal"><span class="pre">timeout</span></tt> is a positive number, it blocks at most
  54. <tt class="docutils literal"><span class="pre">timeout</span></tt> seconds and raises the <tt class="docutils literal"><span class="pre">Empty</span></tt> exception if no
  55. item was available within that time. Otherwise (block is
  56. false), return an item if one is immediately available,
  57. else raise the <tt class="docutils literal"><span class="pre">Empty</span></tt> exception (<tt class="docutils literal"><span class="pre">timeout</span></tt> is ignored in
  58. that case).</dd>
  59. <dt><tt class="docutils literal"><span class="pre">get_nowait()</span></tt>, <tt class="docutils literal"><span class="pre">getNoWait()</span></tt></dt>
  60. <dd>Equivalent to <tt class="docutils literal"><span class="pre">get(False)</span></tt>.</dd>
  61. </dl>
  62. </blockquote>
  63. <p><tt class="docutils literal"><span class="pre">processing.Queue</span></tt> has a few additional methods not found in
  64. <tt class="docutils literal"><span class="pre">Queue.Queue</span></tt> which are usually unnecessary:</p>
  65. <blockquote class="last">
  66. <dl class="docutils">
  67. <dt><tt class="docutils literal"><span class="pre">putMany(iterable)</span></tt></dt>
  68. <dd>If the queue has infinite size then this adds all
  69. items in the iterable to the queue's buffer. So
  70. <tt class="docutils literal"><span class="pre">q.putMany(X)</span></tt> is a faster alternative to <tt class="docutils literal"><span class="pre">for</span> <span class="pre">x</span> <span class="pre">in</span> <span class="pre">X:</span>
  71. <span class="pre">q.put(x)</span></tt>. Raises an error if the queue has finite
  72. size.</dd>
  73. <dt><tt class="docutils literal"><span class="pre">close()</span></tt></dt>
  74. <dd>Indicates that no more data will be put on this queue by
  75. the current process. The background thread will quit once
  76. it has flushed all buffered data to the pipe. This is
  77. called automatically when the queue is garbage collected.</dd>
  78. <dt><tt class="docutils literal"><span class="pre">joinThread()</span></tt></dt>
  79. <dd><p class="first">This joins the background thread and can only be used
  80. after <tt class="docutils literal"><span class="pre">close()</span></tt> has been called. This blocks until
  81. the background thread exits, ensuring that all data in
  82. the buffer has been flushed to the pipe.</p>
  83. <p class="last">By default if a process is not the creator of the
  84. queue then on exit it will attempt to join the queue's
  85. background thread. The process can call
  86. <tt class="docutils literal"><span class="pre">cancelJoin()</span></tt> to prevent this behaviour.</p>
  87. </dd>
  88. <dt><tt class="docutils literal"><span class="pre">cancelJoin()</span></tt></dt>
  89. <dd>Prevents the background thread from being joined
  90. automatically when the process exits. Unnecessary if
  91. the current process created the queue.</dd>
  92. </dl>
  93. </blockquote>
  94. </dd>
  95. </dl>
  96. <div class="admonition-empty-and-full admonition">
  97. <p class="first admonition-title"><tt class="docutils literal"><span class="pre">Empty</span></tt> and <tt class="docutils literal"><span class="pre">Full</span></tt></p>
  98. <p class="last"><tt class="docutils literal"><span class="pre">processing</span></tt> uses the usual <tt class="docutils literal"><span class="pre">Queue.Empty</span></tt> and <tt class="docutils literal"><span class="pre">Queue.Full</span></tt>
  99. exceptions to signal a timeout. They are not available in the
  100. <tt class="docutils literal"><span class="pre">processing</span></tt> namespace so you need to import them from <tt class="docutils literal"><span class="pre">Queue</span></tt>.</p>
  101. </div>
  102. <div class="warning">
  103. <p class="first admonition-title">Warning</p>
  104. <p class="last">If a process is killed using the <tt class="docutils literal"><span class="pre">terminate()</span></tt> method or
  105. <tt class="docutils literal"><span class="pre">os.kill()</span></tt> while it is trying to use a <tt class="docutils literal"><span class="pre">Queue</span></tt> then the data in
  106. the queue is likely to become corrupted. This may cause any other
  107. processes to get an exception when it tries to use the queue later
  108. on.</p>
  109. </div>
  110. <div class="warning">
  111. <p class="first admonition-title">Warning</p>
  112. <p>As mentioned above, if a child process has put items on a queue
  113. (and it has not used <tt class="docutils literal"><span class="pre">cancelJoin()</span></tt>) then that process will not
  114. terminate until all buffered items have been flushed to the pipe.</p>
  115. <p>This means that if you try joining that process you may get a
  116. deadlock unless you are sure that all items which have been put on
  117. the queue have been consumed. Similarly, if the child process is
  118. non-daemonic then the parent process may hang on exit when it tries
  119. to join all it non-daemonic children.</p>
  120. <p class="last">Note that a queue created using a manager does not have this issue.
  121. See <a class="reference" href="programming-guidelines.html">Programming Guidelines</a>.</p>
  122. </div>
  123. </div>
  124. <div class="footer">
  125. <hr class="footer" />
  126. <a class="reference" href="process-objects.html">Prev</a> &nbsp; &nbsp; &nbsp; &nbsp; <a class="reference" href="processing-ref.html">Up</a> &nbsp; &nbsp; &nbsp; &nbsp; <a class="reference" href="connection-objects.html">Next</a>
  127. </div>
  128. </body>
  129. </html>