programming-guidelines.txt 6.5 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193
  1. .. include:: header.txt
  2. ========================
  3. Programming guidelines
  4. ========================
  5. There are certain guidelines and idioms which should be adhered to
  6. when using the `processing` package.
  7. All platforms
  8. -------------
  9. *Avoid shared state*
  10. As far as possible one should try to avoid shifting large amounts
  11. of data between processes.
  12. It is probably best to stick to using queues or pipes for
  13. communication between processes rather than using the lower level
  14. synchronization primitives from the `threading` module.
  15. *Picklability*:
  16. Ensure that the arguments to the methods of proxies are
  17. picklable.
  18. *Thread safety of proxies*:
  19. Do not use a proxy object from more than one thread unless you
  20. protect it with a lock.
  21. (There is never a problem with different processes using the
  22. 'same' proxy.)
  23. *Joining zombie processes*
  24. On Unix when a process finishes but has not been joined it becomes
  25. a zombie. There should never be very many because each time a new
  26. process starts (or `activeChildren()` is called) all completed
  27. processes which have not yet been joined will be joined. Also
  28. calling a finished process's `isAlive()` will join the process.
  29. Even so it is probably good practice to explicitly join all the
  30. processes that you start.
  31. *Better to inherit than pickle/unpickle*
  32. On Windows many of types from the `processing` package need to be
  33. picklable so that child processes can use them. However, one
  34. should generally avoid sending shared objects to other processes
  35. using pipes or queues. Instead you should arrange the program so
  36. that a process which need access to a shared resource created
  37. elsewhere can inherit it from an ancestor process.
  38. *Avoid terminating processes*
  39. Using the `terminate()` method to stop a process is liable to
  40. cause any shared resources (such as locks, semaphores, pipes and
  41. queues) currently being used by the process to become broken or
  42. unavailable to other processes.
  43. Therefore it is probably best to only consider using `terminate()`
  44. on processes which never use any shared resources.
  45. *Joining processes that use queues*
  46. Bear in mind that a process that has put items in a queue will
  47. wait before terminating until all the buffered items are fed by
  48. the "feeder" thread to the underlying pipe. (The child process
  49. can call the `cancelJoin()` method of the queue to avoid this
  50. behaviour.)
  51. This means that whenever you use a queue you need to make sure
  52. that all items which have been put on the queue will eventually be
  53. removed before the process is joined. Otherwise you cannot be
  54. sure that processes which have put items on the queue will
  55. terminate. Remember also that non-daemonic processes will be
  56. automatically be joined.
  57. An example which will deadlock is the following::
  58. from processing import Process, Queue
  59. def f(q):
  60. q.put('X' * 1000000)
  61. if __name__ == '__main__':
  62. queue = Queue()
  63. p = Process(target=f, args=(queue,))
  64. p.start()
  65. p.join() # this deadlocks
  66. obj = queue.get()
  67. A fix here would be to swap the last two lines round (or simply
  68. remove the `p.join()` line).
  69. *Explicity pass resources to child processes*
  70. On Unix a child process can make use of a shared resource created
  71. in a parent process using a global resource. However, it is
  72. better to pass the object as an argument to the constructor for
  73. the child process.
  74. Apart from making the code (potentially) compatible with Windows
  75. this also ensures that as long as the child process is still alive
  76. the object will not be garbage collected in the parent process.
  77. This might be important if some resource is freed when the object
  78. is garbage collected in the parent process.
  79. So for instance ::
  80. from processing import Process, Lock
  81. def f():
  82. ... do something using "lock" ...
  83. if __name__ == '__main__':
  84. lock = Lock()
  85. for i in range(10):
  86. Process(target=f).start()
  87. should be rewritten as ::
  88. from processing import Process, Lock
  89. def f(l):
  90. ... do something using "l" ...
  91. if __name__ == '__main__':
  92. lock = Lock()
  93. for i in range(10):
  94. Process(target=f, args=(lock,)).start()
  95. Windows
  96. -------
  97. Since Windows lacks `os.fork()` it has a few extra restrictions:
  98. *More picklability*:
  99. Ensure that all arguments to `Process.__init__()` are picklable.
  100. This means, in particular, that bound or unbound methods cannot be
  101. used directly as the `target` argument on Windows --- just define
  102. a function and use that instead.
  103. Also, if you subclass `Process` then make sure that instances
  104. will be picklable when the `start()` method is called.
  105. *Global variables*:
  106. Bear in mind that if code run in a child process tries to access a
  107. global variable, then the value it sees (if any) may not be the
  108. same as the value in the parent process at the time that
  109. `start()` was called.
  110. However, global variables which are just module level constants
  111. cause no problems.
  112. *Safe importing of main module*:
  113. Make sure that the main module can be safely imported by a new
  114. Python interpreter without causing unintended side effects (such a
  115. starting a new process).
  116. For example, under Windows running the following module would
  117. fail with a `RuntimeError`::
  118. from processing import Process
  119. def foo():
  120. print 'hello'
  121. p = Process(target=foo)
  122. p.start()
  123. Instead one should protect the "entry point" of the program by
  124. using `if __name__ == '__main__':` as follows::
  125. from processing import Process
  126. def foo():
  127. print 'hello'
  128. if __name__ == '__main__':
  129. freezeSupport()
  130. p = Process(target=foo)
  131. p.start()
  132. (The `freezeSupport()` line can be ommitted if the program will
  133. be run normally instead of frozen.)
  134. This allows the newly spawned Python interpreter to safely import
  135. the module and then run the module's `foo()` function.
  136. Similar restrictions apply if a pool or manager is created in the
  137. main module.
  138. .. _Prev: connection-ref.html
  139. .. _Up: index.html
  140. .. _Next: tests.html