CHANGES.txt 20 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553
  1. .. default-role:: literal
  2. ==========================
  3. Changelog for processing
  4. ==========================
  5. Changes in 0.52
  6. ---------------
  7. * On versions 0.50 and 0.51 Mac OSX `Lock.release()` would fail with
  8. `OSError(errno.ENOSYS, "[Errno 78] Function not implemented")`.
  9. This appears to be because on Mac OSX `sem_getvalue()` has not been
  10. implemented.
  11. Now `sem_getvalue()` is no longer needed. Unfortunately, however,
  12. on Mac OSX `BoundedSemaphore()` will not raise `ValueError` if it
  13. exceeds its initial value.
  14. * Some changes to the code for the reduction/rebuilding of connection
  15. and socket objects so that things work the same on Windows and Unix.
  16. This should fix a couple of bugs.
  17. * The code has been changed to consistently use "camelCase" for
  18. methods and (non-factory) functions. In the few cases where this
  19. has meant a change to the documented API, the old name has been
  20. retained as an alias.
  21. Changes in 0.51
  22. ---------------
  23. * In 0.50 `processing.Value()` and `processing.sharedctypes.Value()`
  24. were related but had different signatures, which was rather
  25. confusing.
  26. Now `processing.sharedctypes.Value()` has been renamed
  27. `processing.sharedctypes.RawValue()` and
  28. `processing.sharedctypes.Value()` is the same as `processing.Value()`.
  29. * In version 0.50 `sendfd()` and `recvfd()` apparently did not work on
  30. 64bit Linux. This has been fixed by reverting to using the CMSG_*
  31. macros as was done in 0.40.
  32. However, this means that systems without all the necessary CMSG_*
  33. macros (such as Solaris 8) will have to disable compilation of
  34. `sendfd()` and `recvfd()` by setting `macros['HAVE_FD_TRANSFRER'] = 0`
  35. in `setup.py`.
  36. * Fixed an authentication error when using a "remote" manager created
  37. using `BaseManager.from_address()`.
  38. * Fixed a couple of bugs which only affected Python 2.4.
  39. Changes in 0.50
  40. ---------------
  41. * `ctypes` is now a prerequisite if you want to use shared memory --
  42. with Python 2.4 you will need to install it separately.
  43. * `LocalManager()` has been removed.
  44. * Added `processing.Value()` and `processing.Array()`
  45. which are similar to `LocalManager.SharedValue()` and
  46. `LocalManager.SharedArray()`.
  47. * In the `sharedctypes` module `new_value()` and `new_array()` have
  48. been renamed `Value()` and `Array()`.
  49. * `Process.stop()`, `Process.getStoppable()` and
  50. `Process.setStoppable()` have been removed. Use
  51. `Process.terminate()` instead.
  52. * `procesing.Lock` now matches `threading.Lock` behaviour more
  53. closely: now a thread can release a lock it does not own, and now
  54. when a thread tries acquiring a lock it already owns a deadlock
  55. results instead of an exception.
  56. * On Windows when the main thread is blocking on a method of `Lock`,
  57. `RLock`, `Semaphore`, `BoundedSemaphore`, `Condition` it will no
  58. longer ignore Ctrl-C. (The same was already true on Unix.)
  59. This differs from the behaviour of the equivalent objects in
  60. `threading` which will completely ignore Ctrl-C.
  61. * The `test` sub-package has been replaced by lots of unit tests in a
  62. `tests` sub-package. Some of the old test files have been moved
  63. over to a new `examples` sub-package.
  64. * On Windows it is now possible for a non-console python program
  65. (i.e. one using `pythonw.exe` instead of `python.exe`) to use
  66. `processing`.
  67. Previously an exception was raised when `subprocess.py` tried to
  68. duplicate stdin, stdout, stderr.
  69. * Proxy objects should now be thread safe -- they now use thread local
  70. storage.
  71. * Trying to transfer shared resources such as locks, queues etc
  72. between processes over a pipe or queue will now raise `RuntimeError`
  73. with a message saying that the object should only be shared between
  74. processes using inheritance.
  75. Previously, this worked unreliably on Windows but would fail with an
  76. unexplained `AssertionError` on Unix.
  77. * The names of some of the macros used for compiling the extension
  78. have changed. See `INSTALL.txt` and `setup.py`.
  79. * A few changes which (hopefully) make compilation possible on Solaris.
  80. * Lots of refactoring of the code.
  81. * Fixed reference leaks so that unit tests pass with "regrtest -R::"
  82. (at least on Linux).
  83. Changes in 0.40
  84. ---------------
  85. * Removed `SimpleQueue` and `PosixQueue` types. Just use `Queue` instead.
  86. * Previously if you forgot to use the ::
  87. if __name__ == '__main__':
  88. freezeSupport()
  89. ...
  90. idiom on Windows then processes could be created recursively
  91. bringing the computer to its knees. Now `RuntimeError` will be
  92. raised instead.
  93. * Some refactoring of the code.
  94. * A Unix specific bug meant that a child process might fail to start a
  95. feeder thread for a queue if its parent process had already started
  96. its own feeder thread. Fixed.
  97. Changes in 0.39
  98. ---------------
  99. * One can now create one-way pipes by doing
  100. `reader, writer = Pipe(duplex=False)`.
  101. * Rewrote code for managing shared memory maps.
  102. * Added a `sharedctypes` module for creating `ctypes` objects allocated
  103. from shared memory. On Python 2.4 this requires the installation of
  104. `ctypes`.
  105. `ctypes` objects are not protected by any locks so you will need to
  106. synchronize access to them (such as by using a lock). However they
  107. can be much faster to access than equivalent objects allocated using
  108. a `LocalManager`.
  109. * Rearranged documentation.
  110. * Previously the C extension caused a segfault on 64 bit machines with
  111. Python 2.5 because it used `int` instead of `Py_ssize_t` in certain
  112. places. This is now fixed. Thanks to Alexy Khrabrov for the report.
  113. * A fix for `Pool.terminate()`.
  114. * A fix for cleanup behaviour of `Queue`.
  115. Changes in 0.38
  116. ---------------
  117. * Have revamped the queue types. Now the queue types are
  118. `Queue`, `SimpleQueue` and (on systems which support it)
  119. `PosixQueue`.
  120. Now `Queue` should behave just like Python's normal `Queue.Queue`
  121. class except that `qsize()`, `task_done()` and `join()` are not
  122. implemented. In particular, if no maximum size was specified when
  123. the queue was created then `put()` will always succeed without
  124. blocking.
  125. A `SimpleQueue` instance is really just a pipe protected by a couple
  126. of locks. It has `get()`, `put()` and `empty()` methods but does
  127. not not support timeouts or non-blocking.
  128. `BufferedPipeQueue()` and `PipeQueue()` remain as deprecated
  129. aliases of `Queue()` but `BufferedPosixQueue()` has been removed.
  130. (Not sure if we really need to keep `PosixQueue()`...)
  131. * Previously the `Pool.shutdown()` method was a little dodgy -- it
  132. could block indefinitely if `map()` or `imap*()` were used and did
  133. not try to terminate workers while they were doing a task.
  134. Now there are three new methods `close()`, `terminate()` and
  135. `join()` -- `shutdown()` is retained as a deprecated alias of
  136. `terminate()`. Thanks to Gerald John M. Manipon for feature
  137. request/suggested patch to `shutdown()`.
  138. * `Pool.imap()` and `Pool.imap_unordered()` has gained a `chunksize`
  139. argument which allows the iterable to be submitted to the pool in
  140. chunks. Choosing `chunksize` appropriately makes `Pool.imap()`
  141. almost as fast as `Pool.map()` even for long iterables and cheap
  142. functions.
  143. * Previously on Windows when the cleanup code for a `LocalManager`
  144. attempts to unlink the name of the file which backs the shared
  145. memory map an exception is raised if a child process still exists
  146. which has a handle open for that mmap. This is likely to happen if
  147. a daemon process inherits a `LocalManager` instance.
  148. Now the parent process will remember the filename and attempt to
  149. unlink the file name again once all the child processes have been
  150. joined or terminated. Reported by Paul Rudin.
  151. * `types.MethodType` is registered with `copy_reg` so now instance
  152. methods and class methods should be picklable. (Unfortunately there is
  153. no obvious way of supporting the pickling of staticmethods since
  154. they are not marked with the class in which they were defined.)
  155. This means that on Windows it is now possible to use an instance
  156. method or class method as the target callable of a Process object.
  157. * On Windows `reduction.fromfd()` now returns true instances of
  158. `_socket.socket`, so there is no more need for the
  159. `_processing.falsesocket` type.
  160. Changes in 0.37
  161. ---------------
  162. * Updated metadata and documentation because the project is now hosted
  163. at `developer.berlios.de/projects/pyprocessing`.
  164. * The `Pool.join()` method has been removed. `Pool.shutdown()` will
  165. now join the worker processes automatically.
  166. * A pool object no longer participates in a reference cycle so
  167. `Pool.shutdown()` should get called as soon as its reference count
  168. falls to zero.
  169. * On Windows if `enableLogging()` was used at module scope then the
  170. logger used by a child process would often get two copies of the
  171. same handler. To fix this, now specifiying a handler type in
  172. `enableLogging()` will cause any previous handlers used by the
  173. logger to be discarded.
  174. Changes in 0.36
  175. ---------------
  176. * In recent versions on Unix the finalizers in a manager process were
  177. never given a chance to run before `os._exit()` was called, so old
  178. unlinked AF_UNIX sockets could accumulate in '/tmp'. Fixed.
  179. * The shutting down of managers has been cleaned up.
  180. * In previous versions on Windows trying to acquire a lock owned by a
  181. different thread of the current process would raise an exception.
  182. Fixed.
  183. * In previous versions on Windows trying to use an event object for
  184. synchronization between two threads of the same process was likely
  185. to raise an exception. (This was caused by the bug described
  186. above.) Fixed.
  187. * Previously the arguments to `processing.Semaphore()` and
  188. `processing.BoundedSemaphore()` did not have any defaults. The
  189. defaults should be 1 to match `threading`. Fixed.
  190. * It should now be possible for a Windows Service created by using
  191. `pywin32` to spawn processes using the `processing` package.
  192. Note that `pywin32` apparently has a bug meaning that `Py_Finalize()`
  193. is never called when the service exits so functions registered with
  194. `atexit` never get a chance to run. Therefore it is advisable to
  195. explicitly call `sys.exitfunc()` or `atexit._run_exitfuncs()` at the
  196. end of `ServiceFramework.DoSvcRun()`. Otherwise child processes are
  197. liable to survive the service when it is stopped. Thanks to Charlie
  198. Hull for the report.
  199. * Added `getLogger()` and `enableLogging()` to support logging.
  200. Changes in 0.35
  201. ---------------
  202. * By default processes are no longer be stoppable using the `stop()`
  203. method: one must call `setStoppable(True)` before `start()` in order
  204. to use the `stop()` method. (Note that `terminate()` will work
  205. regardless of whether the process is marked as being "stoppable".)
  206. The reason for this is that on Windows getting `stop()` to work
  207. involves starting a new console for the child process and installing
  208. a signal handler for the `SIGBREAK` signal. This unfortunately
  209. means that Ctrl-Break cannot not be used to kill all processes of
  210. the program.
  211. * Added `setStoppable()` and `getStoppable()` methods -- see above.
  212. * Added `BufferedQueue`/`BufferedPipeQueue`/`BufferedPosixQueue`.
  213. Putting an object on a buffered queue will always succeed without
  214. blocking (just like with `Queue.Queue` if no maximum size is
  215. specified). This makes them potentially safer than the normal queue
  216. types provided by `processing` which have finite capacity and may
  217. cause deadlocks if they fill.
  218. `test/test_worker.py` has been updated to use `BufferedQueue` for
  219. the task queue instead of explicitly spawning a thread to feed tasks
  220. to the queue without risking a deadlock.
  221. * Now when the NO_SEM_TIMED macro is set polling will be used to get
  222. around the lack of `sem_timedwait()`. This means that
  223. `Condition.wait()` and `Queue.get()` should now work with timeouts
  224. on Mac OS X.
  225. * Added a `callback` argument to `Pool.apply_async()`.
  226. * Added `test/test_httpserverpool.py` which runs a pool of http
  227. servers which share a single listening socket.
  228. * Previously on Windows the process object was passed to the child
  229. process on the commandline (after pickling and hex encoding it).
  230. This caused errors when the pickled string was too large. Now if
  231. the pickled string is large then it will be passed to the child
  232. over a pipe or socket.
  233. * Fixed bug in the iterator returned by `Pool.imap()`.
  234. * Fixed bug in `Condition.__repr__()`.
  235. * Fixed a handle/file descriptor leak when sockets or connections are
  236. unpickled.
  237. Changes in 0.34
  238. ---------------
  239. * Although version 0.33 the C extension would compile on Mac OSX
  240. trying to import it failed with "undefined symbol: _sem_timedwait".
  241. Unfortunately the `ImportError` exception was silently swallowed.
  242. This is now fixed by using the `NO_SEM_TIMED` macro. Unfortunately
  243. this means that some methods like `Condition.wait()` and
  244. `Queue.get()` will not work with timeouts on Mac OS X. If you
  245. really need to be able to use timeouts then you can always use the
  246. equivalent objects created with a manager. Thanks to Doug Hellmann
  247. for report and testing.
  248. * Added a `terminate()` method to process objects which is more
  249. forceful than `stop()`.
  250. * Fixed bug in the cleanup function registered with `atexit` which on
  251. Windows could cause a process which is shutting down to deadlock
  252. waiting for a manager to exit. Thanks to Dominique Wahli for report
  253. and testing.
  254. * Added `test/test_workers.py` which gives an example of how to create
  255. a collection of worker processes which execute tasks from one queue
  256. and return results on another.
  257. * Added `processing.Pool()` which returns a process pool object. This
  258. allows one to execute functions asynchronously. It also has a
  259. parallel implementation of the `map()` builtin. This is still
  260. *experimental* and undocumented --- see `test/test_pool.py` for
  261. example usage.
  262. Changes in 0.33
  263. ---------------
  264. * Added a `recvbytes_into()` method for receiving byte data into
  265. objects with the writable buffer interface. Also renamed the
  266. `_recv_string()` and `_send_string()` methods of connection objects
  267. to `recvbytes()` and `sendbytes()`.
  268. * Some optimizations for the transferring of large blocks of data
  269. using connection objects.
  270. * On Unix `os.sysconf()` is now used by default to determine whether
  271. to compile in support for posix semaphores or posix message queues.
  272. By using the `NO_SEM_TIMED` and `NO_MQ_TIMED` macros (see
  273. `INSTALL.txt`) it should now also be possible to compile in
  274. (partial) semaphore or queue support on Unix systems which lack the
  275. timeout functions `sem_timedwait()` or `mq_timedreceive()` and
  276. `mq_timesend()`.
  277. * `gettimeofday()` is now used instead of `clock_gettime()` making
  278. compilation of the C extension (hopefully) possible on Mac OSX. No
  279. modificaton of `setup.py` should be necessary. Thanks to Michele
  280. Bertoldi for report and proposed patch.
  281. * `cpuCount()` function added which returns the number of CPUs
  282. in the system.
  283. * Bugfixes to `PosixQueue` class.
  284. Changes in 0.32
  285. ---------------
  286. * Refactored and simplified `_nonforking` module -- info about
  287. `sys.modules` of parent process is no longer passed on to child
  288. process. Also `pkgutil` is no longer used.
  289. * Allocated space from an mmap used by `LocalManager` will now be
  290. recycled.
  291. * Better tests for `LocalManager`.
  292. * Fixed bug in `managers.py` concerning refcounting of shared objects.
  293. Bug affects the case where the callable used to create a shared
  294. object does not return a unique object each time it is called.
  295. Thanks to Alexey Akimov for the report.
  296. * Added a `freezeSupport()` function. Calling this at the appropriate
  297. point in the main module is necessary when freezing a multiprocess
  298. program to produce a Windows executable. (Has been tested with
  299. `py2exe`, `PyInstaller` and `cx_Freeze`.)
  300. Changes in 0.31
  301. ---------------
  302. * Fixed one line bug in `localmanager.py` which caused shared memory maps
  303. not to be resized properly.
  304. * Added tests for shared values/structs/arrays to `test/test_processing`.
  305. Changes in 0.30
  306. ----------------
  307. * Process objects now support the complete API of thread objects.
  308. In particular `isAlive()`, `isDaemon()`, `setDaemon()` have been
  309. added and `join()` now supports the `timeout` paramater.
  310. There are also new methods `stop()`, `getPid()` and `getExitCode()`.
  311. * Implemented synchronization primitives based on the Windows mutexes
  312. and semaphores and posix named semaphores.
  313. * Added support for sharing simple objects between processes by using
  314. a shared memory map and the `struct` or `array` modules.
  315. * An `activeChildren()` function has been added to `processing` which
  316. returns a list of the child processes which are still alive.
  317. * A `Pipe()` function has been added which returns a pair of
  318. connection objects representing the ends of a duplex connection over
  319. which picklable objects can be sent.
  320. * socket objects etc are now picklable and can be transferred between
  321. processes. (Requires compilation of the `_processing` extension.)
  322. * Subclasses of `managers.BaseManager` no longer automatically spawn a
  323. child process when an instance is created: the `start()` method must be
  324. called explicitly.
  325. * On Windows child processes are now spawned using `subprocess`.
  326. * On Windows the Python 2.5 version of `pkgutil` is now used for
  327. loading modules by the `_nonforking` module. On Python 2.4 this
  328. version of `pkgutil` (which uses the standard Python licence) is
  329. included in `processing.compat`.
  330. * The arguments to the functions in `processing.connection` have
  331. changed slightly.
  332. * Connection objects now have a `poll()` method which tests whether
  333. there is any data available for reading.
  334. * The `test/py2exedemo` folder shows how to get `py2exe` to create a
  335. Windows executable from a program using the `processing` package.
  336. * More tests.
  337. * Bugfixes.
  338. * Rearrangement of various stuff.
  339. Changes in 0.21
  340. ---------------
  341. * By default a proxy is now only able to access those methods of its
  342. referent which have been explicitly exposed.
  343. * The `connection` sub-package now supports digest authentication.
  344. * Process objects are now given randomly generated 'inheritable'
  345. authentication keys.
  346. * A manager process will now only accept connections from processes
  347. using the same authentication key.
  348. * Previously `get_module()` from `_nonforking.py` was seriously messed
  349. up (though it generally worked). It is a lot saner now.
  350. * Python 2.4 or higher is now required.
  351. Changes in 0.20
  352. ---------------
  353. * The `doc` folder contains HTML documentation.
  354. * `test` is now a subpackage. Running `processing.test.main()`
  355. will run test scripts using both processes and threads.
  356. * `nonforking.py` has been renamed `_nonforking.py`.
  357. `manager.py` has been renamed `manager.py`.
  358. `connection.py` has become a sub-package `connection`
  359. * `Listener` and `Client` have been removed from
  360. `processing`, but still exist in `processing.connection`.
  361. * The package is now *probably* compatible with versions of Python
  362. earlier than 2.4.
  363. * `set` is no longer a type supported by the default manager type.
  364. * Many more changes.
  365. Changes in 0.12
  366. ---------------
  367. * Fixed bug where the arguments to `processing.Manager()` were passed on
  368. to `processing.manager.DefaultManager()` in the wrong order.
  369. * `processing.dummy` is now a subpackage of `processing`
  370. instead of a module.
  371. * Rearranged package so that the `test` folder, `README.txt` and
  372. `CHANGES.txt` are copied when the package is installed.
  373. Changes in 0.11
  374. ---------------
  375. * Fixed bug on windows when the full path of `nonforking.py` contains a
  376. space.
  377. * On unix there is no longer a need to make the arguments to the
  378. constructor of `Process` be picklable or for and instance of a
  379. subclass of `Process` to be picklable when you call the start method.
  380. * On unix proxies which a child process inherits from its parent can
  381. be used by the child without any problem, so there is no longer a
  382. need to pass them as arguments to `Process`. (This will never be
  383. possible on windows.)