| 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553 |
- .. default-role:: literal
- ==========================
- Changelog for processing
- ==========================
- Changes in 0.52
- ---------------
- * On versions 0.50 and 0.51 Mac OSX `Lock.release()` would fail with
- `OSError(errno.ENOSYS, "[Errno 78] Function not implemented")`.
- This appears to be because on Mac OSX `sem_getvalue()` has not been
- implemented.
- Now `sem_getvalue()` is no longer needed. Unfortunately, however,
- on Mac OSX `BoundedSemaphore()` will not raise `ValueError` if it
- exceeds its initial value.
- * Some changes to the code for the reduction/rebuilding of connection
- and socket objects so that things work the same on Windows and Unix.
- This should fix a couple of bugs.
- * The code has been changed to consistently use "camelCase" for
- methods and (non-factory) functions. In the few cases where this
- has meant a change to the documented API, the old name has been
- retained as an alias.
- Changes in 0.51
- ---------------
- * In 0.50 `processing.Value()` and `processing.sharedctypes.Value()`
- were related but had different signatures, which was rather
- confusing.
- Now `processing.sharedctypes.Value()` has been renamed
- `processing.sharedctypes.RawValue()` and
- `processing.sharedctypes.Value()` is the same as `processing.Value()`.
- * In version 0.50 `sendfd()` and `recvfd()` apparently did not work on
- 64bit Linux. This has been fixed by reverting to using the CMSG_*
- macros as was done in 0.40.
- However, this means that systems without all the necessary CMSG_*
- macros (such as Solaris 8) will have to disable compilation of
- `sendfd()` and `recvfd()` by setting `macros['HAVE_FD_TRANSFRER'] = 0`
- in `setup.py`.
- * Fixed an authentication error when using a "remote" manager created
- using `BaseManager.from_address()`.
- * Fixed a couple of bugs which only affected Python 2.4.
- Changes in 0.50
- ---------------
- * `ctypes` is now a prerequisite if you want to use shared memory --
- with Python 2.4 you will need to install it separately.
- * `LocalManager()` has been removed.
- * Added `processing.Value()` and `processing.Array()`
- which are similar to `LocalManager.SharedValue()` and
- `LocalManager.SharedArray()`.
- * In the `sharedctypes` module `new_value()` and `new_array()` have
- been renamed `Value()` and `Array()`.
- * `Process.stop()`, `Process.getStoppable()` and
- `Process.setStoppable()` have been removed. Use
- `Process.terminate()` instead.
- * `procesing.Lock` now matches `threading.Lock` behaviour more
- closely: now a thread can release a lock it does not own, and now
- when a thread tries acquiring a lock it already owns a deadlock
- results instead of an exception.
- * On Windows when the main thread is blocking on a method of `Lock`,
- `RLock`, `Semaphore`, `BoundedSemaphore`, `Condition` it will no
- longer ignore Ctrl-C. (The same was already true on Unix.)
- This differs from the behaviour of the equivalent objects in
- `threading` which will completely ignore Ctrl-C.
- * The `test` sub-package has been replaced by lots of unit tests in a
- `tests` sub-package. Some of the old test files have been moved
- over to a new `examples` sub-package.
- * On Windows it is now possible for a non-console python program
- (i.e. one using `pythonw.exe` instead of `python.exe`) to use
- `processing`.
- Previously an exception was raised when `subprocess.py` tried to
- duplicate stdin, stdout, stderr.
- * Proxy objects should now be thread safe -- they now use thread local
- storage.
- * Trying to transfer shared resources such as locks, queues etc
- between processes over a pipe or queue will now raise `RuntimeError`
- with a message saying that the object should only be shared between
- processes using inheritance.
- Previously, this worked unreliably on Windows but would fail with an
- unexplained `AssertionError` on Unix.
- * The names of some of the macros used for compiling the extension
- have changed. See `INSTALL.txt` and `setup.py`.
- * A few changes which (hopefully) make compilation possible on Solaris.
- * Lots of refactoring of the code.
- * Fixed reference leaks so that unit tests pass with "regrtest -R::"
- (at least on Linux).
- Changes in 0.40
- ---------------
- * Removed `SimpleQueue` and `PosixQueue` types. Just use `Queue` instead.
- * Previously if you forgot to use the ::
- if __name__ == '__main__':
- freezeSupport()
- ...
- idiom on Windows then processes could be created recursively
- bringing the computer to its knees. Now `RuntimeError` will be
- raised instead.
- * Some refactoring of the code.
- * A Unix specific bug meant that a child process might fail to start a
- feeder thread for a queue if its parent process had already started
- its own feeder thread. Fixed.
- Changes in 0.39
- ---------------
- * One can now create one-way pipes by doing
- `reader, writer = Pipe(duplex=False)`.
- * Rewrote code for managing shared memory maps.
- * Added a `sharedctypes` module for creating `ctypes` objects allocated
- from shared memory. On Python 2.4 this requires the installation of
- `ctypes`.
- `ctypes` objects are not protected by any locks so you will need to
- synchronize access to them (such as by using a lock). However they
- can be much faster to access than equivalent objects allocated using
- a `LocalManager`.
- * Rearranged documentation.
- * Previously the C extension caused a segfault on 64 bit machines with
- Python 2.5 because it used `int` instead of `Py_ssize_t` in certain
- places. This is now fixed. Thanks to Alexy Khrabrov for the report.
- * A fix for `Pool.terminate()`.
- * A fix for cleanup behaviour of `Queue`.
- Changes in 0.38
- ---------------
- * Have revamped the queue types. Now the queue types are
- `Queue`, `SimpleQueue` and (on systems which support it)
- `PosixQueue`.
- Now `Queue` should behave just like Python's normal `Queue.Queue`
- class except that `qsize()`, `task_done()` and `join()` are not
- implemented. In particular, if no maximum size was specified when
- the queue was created then `put()` will always succeed without
- blocking.
- A `SimpleQueue` instance is really just a pipe protected by a couple
- of locks. It has `get()`, `put()` and `empty()` methods but does
- not not support timeouts or non-blocking.
- `BufferedPipeQueue()` and `PipeQueue()` remain as deprecated
- aliases of `Queue()` but `BufferedPosixQueue()` has been removed.
- (Not sure if we really need to keep `PosixQueue()`...)
- * Previously the `Pool.shutdown()` method was a little dodgy -- it
- could block indefinitely if `map()` or `imap*()` were used and did
- not try to terminate workers while they were doing a task.
- Now there are three new methods `close()`, `terminate()` and
- `join()` -- `shutdown()` is retained as a deprecated alias of
- `terminate()`. Thanks to Gerald John M. Manipon for feature
- request/suggested patch to `shutdown()`.
- * `Pool.imap()` and `Pool.imap_unordered()` has gained a `chunksize`
- argument which allows the iterable to be submitted to the pool in
- chunks. Choosing `chunksize` appropriately makes `Pool.imap()`
- almost as fast as `Pool.map()` even for long iterables and cheap
- functions.
- * Previously on Windows when the cleanup code for a `LocalManager`
- attempts to unlink the name of the file which backs the shared
- memory map an exception is raised if a child process still exists
- which has a handle open for that mmap. This is likely to happen if
- a daemon process inherits a `LocalManager` instance.
- Now the parent process will remember the filename and attempt to
- unlink the file name again once all the child processes have been
- joined or terminated. Reported by Paul Rudin.
- * `types.MethodType` is registered with `copy_reg` so now instance
- methods and class methods should be picklable. (Unfortunately there is
- no obvious way of supporting the pickling of staticmethods since
- they are not marked with the class in which they were defined.)
- This means that on Windows it is now possible to use an instance
- method or class method as the target callable of a Process object.
- * On Windows `reduction.fromfd()` now returns true instances of
- `_socket.socket`, so there is no more need for the
- `_processing.falsesocket` type.
- Changes in 0.37
- ---------------
- * Updated metadata and documentation because the project is now hosted
- at `developer.berlios.de/projects/pyprocessing`.
- * The `Pool.join()` method has been removed. `Pool.shutdown()` will
- now join the worker processes automatically.
- * A pool object no longer participates in a reference cycle so
- `Pool.shutdown()` should get called as soon as its reference count
- falls to zero.
-
- * On Windows if `enableLogging()` was used at module scope then the
- logger used by a child process would often get two copies of the
- same handler. To fix this, now specifiying a handler type in
- `enableLogging()` will cause any previous handlers used by the
- logger to be discarded.
- Changes in 0.36
- ---------------
- * In recent versions on Unix the finalizers in a manager process were
- never given a chance to run before `os._exit()` was called, so old
- unlinked AF_UNIX sockets could accumulate in '/tmp'. Fixed.
- * The shutting down of managers has been cleaned up.
- * In previous versions on Windows trying to acquire a lock owned by a
- different thread of the current process would raise an exception.
- Fixed.
- * In previous versions on Windows trying to use an event object for
- synchronization between two threads of the same process was likely
- to raise an exception. (This was caused by the bug described
- above.) Fixed.
- * Previously the arguments to `processing.Semaphore()` and
- `processing.BoundedSemaphore()` did not have any defaults. The
- defaults should be 1 to match `threading`. Fixed.
- * It should now be possible for a Windows Service created by using
- `pywin32` to spawn processes using the `processing` package.
- Note that `pywin32` apparently has a bug meaning that `Py_Finalize()`
- is never called when the service exits so functions registered with
- `atexit` never get a chance to run. Therefore it is advisable to
- explicitly call `sys.exitfunc()` or `atexit._run_exitfuncs()` at the
- end of `ServiceFramework.DoSvcRun()`. Otherwise child processes are
- liable to survive the service when it is stopped. Thanks to Charlie
- Hull for the report.
- * Added `getLogger()` and `enableLogging()` to support logging.
- Changes in 0.35
- ---------------
- * By default processes are no longer be stoppable using the `stop()`
- method: one must call `setStoppable(True)` before `start()` in order
- to use the `stop()` method. (Note that `terminate()` will work
- regardless of whether the process is marked as being "stoppable".)
- The reason for this is that on Windows getting `stop()` to work
- involves starting a new console for the child process and installing
- a signal handler for the `SIGBREAK` signal. This unfortunately
- means that Ctrl-Break cannot not be used to kill all processes of
- the program.
- * Added `setStoppable()` and `getStoppable()` methods -- see above.
- * Added `BufferedQueue`/`BufferedPipeQueue`/`BufferedPosixQueue`.
- Putting an object on a buffered queue will always succeed without
- blocking (just like with `Queue.Queue` if no maximum size is
- specified). This makes them potentially safer than the normal queue
- types provided by `processing` which have finite capacity and may
- cause deadlocks if they fill.
- `test/test_worker.py` has been updated to use `BufferedQueue` for
- the task queue instead of explicitly spawning a thread to feed tasks
- to the queue without risking a deadlock.
- * Now when the NO_SEM_TIMED macro is set polling will be used to get
- around the lack of `sem_timedwait()`. This means that
- `Condition.wait()` and `Queue.get()` should now work with timeouts
- on Mac OS X.
- * Added a `callback` argument to `Pool.apply_async()`.
- * Added `test/test_httpserverpool.py` which runs a pool of http
- servers which share a single listening socket.
- * Previously on Windows the process object was passed to the child
- process on the commandline (after pickling and hex encoding it).
- This caused errors when the pickled string was too large. Now if
- the pickled string is large then it will be passed to the child
- over a pipe or socket.
- * Fixed bug in the iterator returned by `Pool.imap()`.
- * Fixed bug in `Condition.__repr__()`.
- * Fixed a handle/file descriptor leak when sockets or connections are
- unpickled.
- Changes in 0.34
- ---------------
- * Although version 0.33 the C extension would compile on Mac OSX
- trying to import it failed with "undefined symbol: _sem_timedwait".
- Unfortunately the `ImportError` exception was silently swallowed.
-
- This is now fixed by using the `NO_SEM_TIMED` macro. Unfortunately
- this means that some methods like `Condition.wait()` and
- `Queue.get()` will not work with timeouts on Mac OS X. If you
- really need to be able to use timeouts then you can always use the
- equivalent objects created with a manager. Thanks to Doug Hellmann
- for report and testing.
-
- * Added a `terminate()` method to process objects which is more
- forceful than `stop()`.
- * Fixed bug in the cleanup function registered with `atexit` which on
- Windows could cause a process which is shutting down to deadlock
- waiting for a manager to exit. Thanks to Dominique Wahli for report
- and testing.
-
- * Added `test/test_workers.py` which gives an example of how to create
- a collection of worker processes which execute tasks from one queue
- and return results on another.
- * Added `processing.Pool()` which returns a process pool object. This
- allows one to execute functions asynchronously. It also has a
- parallel implementation of the `map()` builtin. This is still
- *experimental* and undocumented --- see `test/test_pool.py` for
- example usage.
- Changes in 0.33
- ---------------
- * Added a `recvbytes_into()` method for receiving byte data into
- objects with the writable buffer interface. Also renamed the
- `_recv_string()` and `_send_string()` methods of connection objects
- to `recvbytes()` and `sendbytes()`.
- * Some optimizations for the transferring of large blocks of data
- using connection objects.
- * On Unix `os.sysconf()` is now used by default to determine whether
- to compile in support for posix semaphores or posix message queues.
-
- By using the `NO_SEM_TIMED` and `NO_MQ_TIMED` macros (see
- `INSTALL.txt`) it should now also be possible to compile in
- (partial) semaphore or queue support on Unix systems which lack the
- timeout functions `sem_timedwait()` or `mq_timedreceive()` and
- `mq_timesend()`.
- * `gettimeofday()` is now used instead of `clock_gettime()` making
- compilation of the C extension (hopefully) possible on Mac OSX. No
- modificaton of `setup.py` should be necessary. Thanks to Michele
- Bertoldi for report and proposed patch.
- * `cpuCount()` function added which returns the number of CPUs
- in the system.
- * Bugfixes to `PosixQueue` class.
- Changes in 0.32
- ---------------
- * Refactored and simplified `_nonforking` module -- info about
- `sys.modules` of parent process is no longer passed on to child
- process. Also `pkgutil` is no longer used.
- * Allocated space from an mmap used by `LocalManager` will now be
- recycled.
- * Better tests for `LocalManager`.
- * Fixed bug in `managers.py` concerning refcounting of shared objects.
- Bug affects the case where the callable used to create a shared
- object does not return a unique object each time it is called.
- Thanks to Alexey Akimov for the report.
- * Added a `freezeSupport()` function. Calling this at the appropriate
- point in the main module is necessary when freezing a multiprocess
- program to produce a Windows executable. (Has been tested with
- `py2exe`, `PyInstaller` and `cx_Freeze`.)
- Changes in 0.31
- ---------------
- * Fixed one line bug in `localmanager.py` which caused shared memory maps
- not to be resized properly.
- * Added tests for shared values/structs/arrays to `test/test_processing`.
- Changes in 0.30
- ----------------
- * Process objects now support the complete API of thread objects.
- In particular `isAlive()`, `isDaemon()`, `setDaemon()` have been
- added and `join()` now supports the `timeout` paramater.
- There are also new methods `stop()`, `getPid()` and `getExitCode()`.
-
- * Implemented synchronization primitives based on the Windows mutexes
- and semaphores and posix named semaphores.
- * Added support for sharing simple objects between processes by using
- a shared memory map and the `struct` or `array` modules.
- * An `activeChildren()` function has been added to `processing` which
- returns a list of the child processes which are still alive.
- * A `Pipe()` function has been added which returns a pair of
- connection objects representing the ends of a duplex connection over
- which picklable objects can be sent.
- * socket objects etc are now picklable and can be transferred between
- processes. (Requires compilation of the `_processing` extension.)
- * Subclasses of `managers.BaseManager` no longer automatically spawn a
- child process when an instance is created: the `start()` method must be
- called explicitly.
- * On Windows child processes are now spawned using `subprocess`.
- * On Windows the Python 2.5 version of `pkgutil` is now used for
- loading modules by the `_nonforking` module. On Python 2.4 this
- version of `pkgutil` (which uses the standard Python licence) is
- included in `processing.compat`.
- * The arguments to the functions in `processing.connection` have
- changed slightly.
- * Connection objects now have a `poll()` method which tests whether
- there is any data available for reading.
- * The `test/py2exedemo` folder shows how to get `py2exe` to create a
- Windows executable from a program using the `processing` package.
- * More tests.
- * Bugfixes.
- * Rearrangement of various stuff.
- Changes in 0.21
- ---------------
- * By default a proxy is now only able to access those methods of its
- referent which have been explicitly exposed.
- * The `connection` sub-package now supports digest authentication.
- * Process objects are now given randomly generated 'inheritable'
- authentication keys.
- * A manager process will now only accept connections from processes
- using the same authentication key.
- * Previously `get_module()` from `_nonforking.py` was seriously messed
- up (though it generally worked). It is a lot saner now.
- * Python 2.4 or higher is now required.
- Changes in 0.20
- ---------------
- * The `doc` folder contains HTML documentation.
- * `test` is now a subpackage. Running `processing.test.main()`
- will run test scripts using both processes and threads.
- * `nonforking.py` has been renamed `_nonforking.py`.
- `manager.py` has been renamed `manager.py`.
- `connection.py` has become a sub-package `connection`
- * `Listener` and `Client` have been removed from
- `processing`, but still exist in `processing.connection`.
- * The package is now *probably* compatible with versions of Python
- earlier than 2.4.
- * `set` is no longer a type supported by the default manager type.
- * Many more changes.
- Changes in 0.12
- ---------------
- * Fixed bug where the arguments to `processing.Manager()` were passed on
- to `processing.manager.DefaultManager()` in the wrong order.
- * `processing.dummy` is now a subpackage of `processing`
- instead of a module.
- * Rearranged package so that the `test` folder, `README.txt` and
- `CHANGES.txt` are copied when the package is installed.
- Changes in 0.11
- ---------------
- * Fixed bug on windows when the full path of `nonforking.py` contains a
- space.
- * On unix there is no longer a need to make the arguments to the
- constructor of `Process` be picklable or for and instance of a
- subclass of `Process` to be picklable when you call the start method.
- * On unix proxies which a child process inherits from its parent can
- be used by the child without any problem, so there is no longer a
- need to pass them as arguments to `Process`. (This will never be
- possible on windows.)
|