CHANGES.txt 20 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751752
  1. 3.5.0.5 - 2018-12-03
  2. --------------------
  3. - Fix a crash when handling a timeout.
  4. - Improve billiard.einfo.Frame compatibility with Python to avoid crashes when using Django.
  5. - Cleanups.
  6. 3.5.0.4 - 2018-07-07
  7. --------------------
  8. - Fix rebuild_ctype losing BufferedWrapper reference on Python 3.
  9. - Deprecation warning with python 3.6 fixed.
  10. - Correct spawn start method.
  11. 3.5.0.3 - 2017-07-16
  12. --------------------
  13. - Adds Process._authkey alias to .authkey for 2.7 compat.
  14. - Remove superfluous else clause from max_memory_per_child_check.
  15. - Document and test all supported Python versions.
  16. - Extend 'Process' to be compatible with < Py3.5.
  17. - Use a properly initialized logger in pool.py error logging.
  18. - _trywaitkill can now kill a whole process group if the worker process declares itself as a group leader.
  19. - Fix cpython issue 14881 (See https://bugs.python.org/issue14881).
  20. - Fix for a crash on windows.
  21. - Fix messaging in case of worker exceeds max memory.
  22. 3.5.0.2 - 2016-10-03
  23. --------------------
  24. - max_memory_per_child was measured in kilobytes on Linux, but bytes on
  25. *BSD/MacOS, it's now always kilobytes.
  26. - Windows: Adds support for max_memory_per_child, but requires the
  27. ``psutil`` package to be installed.
  28. - Fixed bug in ForkingPickler.loadbuf, where it tried to pass
  29. a BytesIO instance directly to ``pickle.loads`` on Python 2.7.
  30. 3.5.0.1 - 2016-09-07
  31. --------------------
  32. - Connection: Properly handle EINTR (Issue #191).
  33. - Fixed bug with missing CreateProcess for Windows on Python 2.7.
  34. - Adds Process._counter for compatibility with Python <3.5.
  35. 3.5.0.0 - 2016-07-28
  36. --------------------
  37. - No longer supports Python 2.6
  38. You need Python 2.7 or later to use this version of billiard.
  39. - Merged changes from CPython 3.5
  40. 3.3.0.20 - 2015-04-17
  41. ---------------------
  42. - Pool: Timeouts will attempt to send SIGKILL, but this signal
  43. does not exist on Windows. Replaced with SIGTERM.
  44. 3.3.0.19 - 2014-10-13
  45. ---------------------
  46. - Pool: Exceptions in user timeout callbacks are now logged instead
  47. of crashing the pool.
  48. Contributed by Pierre Fersing.
  49. - Pool: Exit codes in errors were improperly being represented as signals.
  50. - Pool: ``.map``. and ``.imap`` now working again.
  51. - Now builds on FreeBSD 10.
  52. Contributed by Michael Fladischer.
  53. 3.3.0.18 - 2014-06-20
  54. ---------------------
  55. - Now compiles on GNU/kFreeBSD
  56. Contributed by Michael Fladischer.
  57. - Pool: `AF_PIPE` address fixed so that it works on recent Windows versions
  58. in combination with Python 2.7.7.
  59. Fix contributed by Joshua Tacoma.
  60. - Pool: Fix for `Supervisor object has no attribute _children` error.
  61. Fix contributed by Andres Riancho.
  62. - Pool: Fixed bug with human_status(None).
  63. - Pool: shrink did not work properly if asked to remove more than 1 process.
  64. 3.3.0.17 - 2014-04-16
  65. ---------------------
  66. - Fixes SemLock on Python 3.4 (Issue #107) when using
  67. ``forking_enable(False)``.
  68. - Pool: Include more useful exitcode information when processes exit.
  69. 3.3.0.16 - 2014-02-11
  70. ---------------------
  71. - Previous release was missing the billiard.py3 package from MANIFEST
  72. so the installation would not work on Python 3.
  73. 3.3.0.15 - 2014-02-10
  74. ---------------------
  75. - Pool: Fixed "cannot join process not started" error.
  76. - Now uses billiard.py2 and billiard.py3 specific packages that are installed
  77. depending on the python version used.
  78. This way the installation will not import version specific modules (and
  79. possibly crash).
  80. 3.3.0.14 - 2014-01-17
  81. ---------------------
  82. - Fixed problem with our backwards compatible ``bytes`` wrapper
  83. (Issue #103).
  84. - No longer expects frozen applications to have a valid ``__file__``
  85. attribute.
  86. Fix contributed by George Sibble.
  87. 3.3.0.13 - 2013-12-13
  88. ---------------------
  89. - Fixes compatability with Python < 2.7.6
  90. - No longer attempts to handle ``SIGBUS``
  91. Contributed by Vishal Vatsa.
  92. - Non-thread based pool now only handles signals:
  93. ``SIGHUP``, ``SIGQUIT``, ``SIGTERM``, ``SIGUSR1``,
  94. ``SIGUSR2``.
  95. - setup.py: Only show compilation warning for build related commands.
  96. 3.3.0.12 - 2013-12-09
  97. ---------------------
  98. - Fixed installation for Python 3.
  99. Contributed by Rickert Mulder.
  100. - Pool: Fixed bug with maxtasksperchild.
  101. Fix contributed by Ionel Cristian Maries.
  102. - Pool: Fixed bug in maintain_pool.
  103. 3.3.0.11 - 2013-12-03
  104. ---------------------
  105. - Fixed Unicode error when installing the distribution (Issue #89).
  106. - Daemonic processes are now allowed to have children.
  107. But note that it will not be possible to automatically
  108. terminate them when the process exits.
  109. See discussion at https://github.com/celery/celery/issues/1709
  110. - Pool: Would not always be able to detect that a process exited.
  111. 3.3.0.10 - 2013-12-02
  112. ---------------------
  113. - Windows: Fixed problem with missing ``WAITABANDONED_0``
  114. Fix contributed by Matthias Wagner
  115. - Windows: PipeConnection can now be inherited.
  116. Fix contributed by Matthias Wagner
  117. 3.3.0.9 - 2013-12-02
  118. --------------------
  119. - Temporary workaround for Celery maxtasksperchild issue.
  120. Fix contributed by Ionel Cristian Maries.
  121. 3.3.0.8 - 2013-11-21
  122. --------------------
  123. - Now also sets ``multiprocessing.current_process`` for compatibility
  124. with loggings ``processName`` field.
  125. 3.3.0.7 - 2013-11-15
  126. --------------------
  127. - Fixed compatibility with PyPy 2.1 + 2.2.
  128. - Fixed problem in pypy detection.
  129. Fix contributed by Tin Tvrtkovic.
  130. - Now uses ``ctypes.find_library`` instead of hardcoded path to find
  131. the macOS CoreServices framework.
  132. Fix contributed by Moritz Kassner.
  133. 3.3.0.6 - 2013-11-12
  134. --------------------
  135. - Now works without C extension again.
  136. - New ``_billiard.read(fd, buffer, [len, ])`` function
  137. implements os.read with buffer support (new buffer API)
  138. - New pure-python implementation of ``Connection.send_offset``.
  139. 3.3.0.5 - 2013-11-11
  140. --------------------
  141. - All platforms except for Windows/PyPy/Jython now requires the C extension.
  142. 3.3.0.4 - 2013-11-11
  143. --------------------
  144. - Fixed problem with Python3 and setblocking.
  145. 3.3.0.3 - 2013-11-09
  146. --------------------
  147. - Now works on Windows again.
  148. 3.3.0.2 - 2013-11-08
  149. --------------------
  150. - ApplyResult.terminate() may be set to signify that the job
  151. must not be executed. It can be used in combination with
  152. Pool.terminate_job.
  153. - Pipe/_SimpleQueue: Now supports rnonblock/wnonblock arguments
  154. to set the read or write end of the pipe to be nonblocking.
  155. - Pool: Log message included exception info but exception happened
  156. in another process so the resulting traceback was wrong.
  157. - Pool: Worker process can now prepare results before they are sent
  158. back to the main process (using ``Worker.prepare_result``).
  159. 3.3.0.1 - 2013-11-04
  160. --------------------
  161. - Pool: New ``correlation_id`` argument to ``apply_async`` can be
  162. used to set a related id for the ``ApplyResult`` object returned:
  163. >>> r = pool.apply_async(target, args, kwargs, correlation_id='foo')
  164. >>> r.correlation_id
  165. 'foo'
  166. - Pool: New callback `on_process_exit` is called when a pool
  167. process exits, with signature ``(pid, exitcode)``.
  168. Contributed by Daniel M. Taub.
  169. - Pool: Improved the too many restarts detection.
  170. 3.3.0.0 - 2013-10-14
  171. --------------------
  172. - Dual code base now runs on Python 2.6+ and Python 3.
  173. - No longer compatible with Python 2.5
  174. - Includes many changes from multiprocessing in 3.4.
  175. - Now uses ``time.monotonic`` when available, also including
  176. fallback implementations for Linux and macOS.
  177. - No longer cleans up after receiving SIGILL, SIGSEGV or SIGFPE
  178. Contributed by Kevin Blackham
  179. - ``Finalize`` and ``register_after_fork`` is now aliases to multiprocessing.
  180. It's better to import these from multiprocessing directly now
  181. so that there aren't multiple registries.
  182. - New `billiard.queues._SimpleQueue` that does not use semaphores.
  183. - Pool: Can now be extended to support using multiple IPC queues.
  184. - Pool: Can now use async I/O to write to pool IPC queues.
  185. - Pool: New ``Worker.on_loop_stop`` handler can be used to add actions
  186. at pool worker process shutdown.
  187. Note that, like all finalization handlers, there is no guarantee that
  188. this will be executed.
  189. Contributed by dmtaub.
  190. 2.7.3.30 - 2013-06-28
  191. ---------------------
  192. - Fixed ImportError in billiard._ext
  193. 2.7.3.29 - 2013-06-28
  194. ---------------------
  195. - Compilation: Fixed improper handling of HAVE_SEM_OPEN (Issue #55)
  196. Fix contributed by Krzysztof Jagiello.
  197. - Process now releases logging locks after fork.
  198. This previously happened in Pool, but it was done too late
  199. as processes logs when they bootstrap.
  200. - Pool.terminate_job now ignores `No such process` errors.
  201. - billiard.Pool entrypoint did not support new arguments
  202. to billiard.pool.Pool
  203. - Connection inbound buffer size increased from 1kb to 128kb.
  204. - C extension cleaned up by properly adding a namespace to symbols.
  205. - _exit_function now works even if thread wakes up after gc collect.
  206. 2.7.3.28 - 2013-04-16
  207. ---------------------
  208. - Pool: Fixed regression that disabled the deadlock
  209. fix in 2.7.3.24
  210. - Pool: RestartFreqExceeded could be raised prematurely.
  211. - Process: Include pid in startup and process INFO logs.
  212. 2.7.3.27 - 2013-04-12
  213. ---------------------
  214. - Manager now works again.
  215. - Python 3 fixes for billiard.connection.
  216. - Fixed invalid argument bug when running on Python 3.3
  217. Fix contributed by Nathan Wan.
  218. - Ignore OSError when setting up signal handlers.
  219. 2.7.3.26 - 2013-04-09
  220. ---------------------
  221. - Pool: Child processes must ignore SIGINT.
  222. 2.7.3.25 - 2013-04-09
  223. ---------------------
  224. - Pool: 2.7.3.24 broke support for subprocesses (Issue #48).
  225. Signals that should be ignored were instead handled
  226. by terminating.
  227. 2.7.3.24 - 2013-04-08
  228. ---------------------
  229. - Pool: Make sure finally blocks are called when process exits
  230. due to a signal.
  231. This fixes a deadlock problem when the process is killed
  232. while having acquired the shared semaphore. However, this solution
  233. does not protect against the processes being killed, a more elaborate
  234. solution is required for that. Hopefully this will be fixed soon in a
  235. later version.
  236. - Pool: Can now use GDB to debug pool child processes.
  237. - Fixes Python 3 compatibility problems.
  238. Contributed by Albertas Agejevas.
  239. 2.7.3.23 - 2013-03-22
  240. ---------------------
  241. - Windows: Now catches SystemExit from setuptools while trying to build
  242. the C extension (Issue #41).
  243. 2.7.3.22 - 2013-03-08
  244. ---------------------
  245. - Pool: apply_async now supports a ``callbacks_propagate`` keyword
  246. argument that can be a tuple of exceptions to propagate in callbacks.
  247. (callback, errback, accept_callback, timeout_callback).
  248. - Errors are no longer logged for OK and recycle exit codes.
  249. This would cause normal maxtasksperchild recycled process
  250. to log an error.
  251. - Fixed Python 2.5 compatibility problem (Issue #33).
  252. - FreeBSD: Compilation now disables semaphores if Python was built
  253. without it (Issue #40).
  254. Contributed by William Grzybowski
  255. 2.7.3.21 - 2013-02-11
  256. ---------------------
  257. - Fixed typo EX_REUSE -> EX_RECYCLE
  258. - Code now conforms to new pep8.py rules.
  259. 2.7.3.20 - 2013-02-08
  260. ---------------------
  261. - Pool: Disable restart limit if maxR is not set.
  262. - Pool: Now uses os.kill instead of signal.signal.
  263. Contributed by Lukasz Langa
  264. - Fixed name error in process.py
  265. - Pool: ApplyResult.get now properly raises exceptions.
  266. Fix contributed by xentac.
  267. 2.7.3.19 - 2012-11-30
  268. ---------------------
  269. - Fixes problem at shutdown when gc has collected symbols.
  270. - Pool now always uses _kill for Py2.5 compatibility on Windows (Issue #32).
  271. - Fixes Python 3 compatibility issues
  272. 2.7.3.18 - 2012-11-05
  273. ---------------------
  274. - [Pool] Fix for check_timeouts if not set.
  275. Fix contributed by Dmitry Sukhov
  276. - Fixed pickle problem with Traceback.
  277. Code.frame.__loader__ is now ignored as it may be set to
  278. an unpickleable object.
  279. - The Django old-layout warning was always showing.
  280. 2.7.3.17 - 2012-09-26
  281. ---------------------
  282. - Fixes typo
  283. 2.7.3.16 - 2012-09-26
  284. ---------------------
  285. - Windows: Fixes for SemLock._rebuild (Issue #24).
  286. - Pool: Job terminated with terminate_job now raises
  287. billiard.exceptions.Terminated.
  288. 2.7.3.15 - 2012-09-21
  289. ---------------------
  290. - Windows: Fixes unpickling of SemLock when using fallback.
  291. - Windows: Fixes installation when no C compiler.
  292. 2.7.3.14 - 2012-09-20
  293. ---------------------
  294. - Installation now works again for Python 3.
  295. 2.7.3.13 - 2012-09-14
  296. ---------------------
  297. - Merged with Python trunk (many authors, many fixes: see Python changelog in
  298. trunk).
  299. - Using execv now also works with older Django projects using setup_environ
  300. (Issue #10).
  301. - Billiard now installs with a warning that the C extension could not be built
  302. if a compiler is not installed or the build fails in some other way.
  303. It really is recommended to have the C extension installed when running
  304. with force execv, but this change also makes it easier to install.
  305. - Pool: Hard timeouts now sends KILL shortly after TERM so that C extensions
  306. cannot block the signal.
  307. Python signal handlers are called in the interpreter, so they cannot
  308. be called while a C extension is blocking the interpreter from running.
  309. - Now uses a timeout value for Thread.join that doesn't exceed the maximum
  310. on some platforms.
  311. - Fixed bug in the SemLock fallback used when C extensions not installed.
  312. Fix contributed by Mher Movsisyan.
  313. - Pool: Now sets a Process.index attribute for every process in the pool.
  314. This number will always be between 0 and concurrency-1, and
  315. can be used to e.g. create a logfile for each process in the pool
  316. without creating a new logfile whenever a process is replaced.
  317. 2.7.3.12 - 2012-08-05
  318. ---------------------
  319. - Fixed Python 2.5 compatibility issue.
  320. - New Pool.terminate_job(pid) to terminate a job without raising WorkerLostError
  321. 2.7.3.11 - 2012-08-01
  322. ---------------------
  323. - Adds support for FreeBSD 7+
  324. Fix contributed by koobs.
  325. - Pool: New argument ``allow_restart`` is now required to enable
  326. the pool process sentinel that is required to restart the pool.
  327. It's disabled by default, which reduces the number of file
  328. descriptors/semaphores required to run the pool.
  329. - Pool: Now emits a warning if a worker process exited with error-code.
  330. But not if the error code is 155, which is now returned if the worker
  331. process was recycled (maxtasksperchild).
  332. - Python 3 compatibility fixes.
  333. - Python 2.5 compatibility fixes.
  334. 2.7.3.10 - 2012-06-26
  335. ---------------------
  336. - The ``TimeLimitExceeded`` exception string representation
  337. only included the seconds as a number, it now gives a more human
  338. friendly description.
  339. - Fixed typo in ``LaxBoundedSemaphore.shrink``.
  340. - Pool: ``ResultHandler.handle_event`` no longer requires
  341. any arguments.
  342. - setup.py bdist now works
  343. 2.7.3.9 - 2012-06-03
  344. --------------------
  345. - Environment variable ``MP_MAIN_FILE`` envvar is now set to
  346. the path of the ``__main__`` module when execv is enabled.
  347. - Pool: Errors occurring in the TaskHandler are now reported.
  348. 2.7.3.8 - 2012-06-01
  349. --------------------
  350. - Can now be installed on Py 3.2
  351. - Issue #12091: simplify ApplyResult and MapResult with threading.Event
  352. Patch by Charles-Francois Natali
  353. - Pool: Support running without TimeoutHandler thread.
  354. - The with_*_thread arguments has also been replaced with
  355. a single `threads=True` argument.
  356. - Two new pool callbacks:
  357. - ``on_timeout_set(job, soft, hard)``
  358. Applied when a task is executed with a timeout.
  359. - ``on_timeout_cancel(job)``
  360. Applied when a timeout is cancelled (the job completed)
  361. 2.7.3.7 - 2012-05-21
  362. --------------------
  363. - Fixes Python 2.5 support.
  364. 2.7.3.6 - 2012-05-21
  365. --------------------
  366. - Pool: Can now be used in an event loop, without starting the supporting
  367. threads (TimeoutHandler still not supported)
  368. To facilitate this the pool has gained the following keyword arguments:
  369. - ``with_task_thread``
  370. - ``with_result_thread``
  371. - ``with_supervisor_thread``
  372. - ``on_process_up``
  373. Callback called with Process instance as argument
  374. whenever a new worker process is added.
  375. Used to add new process fds to the eventloop::
  376. def on_process_up(proc):
  377. hub.add_reader(proc.sentinel, pool.maintain_pool)
  378. - ``on_process_down``
  379. Callback called with Process instance as argument
  380. whenever a new worker process is found dead.
  381. Used to remove process fds from the eventloop::
  382. def on_process_down(proc):
  383. hub.remove(proc.sentinel)
  384. - ``semaphore``
  385. Sets the semaphore used to protect from adding new items to the
  386. pool when no processes available. The default is a threaded
  387. one, so this can be used to change to an async semaphore.
  388. And the following attributes::
  389. - ``readers``
  390. A map of ``fd`` -> ``callback``, to be registered in an eventloop.
  391. Currently this is only the result outqueue with a callback
  392. that processes all currently incoming results.
  393. And the following methods::
  394. - ``did_start_ok``
  395. To be called after starting the pool, and after setting up the
  396. eventloop with the pool fds, to ensure that the worker processes
  397. didn't immediately exit caused by an error (internal/memory).
  398. - ``maintain_pool``
  399. Public version of ``_maintain_pool`` that handles max restarts.
  400. - Pool: Process too frequent restart protection now only counts if the process
  401. had a non-successful exit-code.
  402. This to take into account the maxtasksperchild option, and allowing
  403. processes to exit cleanly on their own.
  404. - Pool: New options max_restart + max_restart_freq
  405. This means that the supervisor can't restart processes
  406. faster than max_restart' times per max_restart_freq seconds
  407. (like the Erlang supervisor maxR & maxT settings).
  408. The pool is closed and joined if the max restart
  409. frequency is exceeded, where previously it would keep restarting
  410. at an unlimited rate, possibly crashing the system.
  411. The current default value is to stop if it exceeds
  412. 100 * process_count restarts in 1 seconds. This may change later.
  413. It will only count processes with an unsuccessful exit code,
  414. this is to take into account the ``maxtasksperchild`` setting
  415. and code that voluntarily exits.
  416. - Pool: The ``WorkerLostError`` message now includes the exit-code of the
  417. process that disappeared.
  418. 2.7.3.5 - 2012-05-09
  419. --------------------
  420. - Now always cleans up after ``sys.exc_info()`` to avoid
  421. cyclic references.
  422. - ExceptionInfo without arguments now defaults to ``sys.exc_info``.
  423. - Forking can now be disabled using the
  424. ``MULTIPROCESSING_FORKING_DISABLE`` environment variable.
  425. Also this envvar is set so that the behavior is inherited
  426. after execv.
  427. - The semaphore cleanup process started when execv is used
  428. now sets a useful process name if the ``setproctitle``
  429. module is installed.
  430. - Sets the ``FORKED_BY_MULTIPROCESSING``
  431. environment variable if forking is disabled.
  432. 2.7.3.4 - 2012-04-27
  433. --------------------
  434. - Added `billiard.ensure_multiprocessing()`
  435. Raises NotImplementedError if the platform does not support
  436. multiprocessing (e.g. Jython).
  437. 2.7.3.3 - 2012-04-23
  438. --------------------
  439. - PyPy now falls back to using its internal _multiprocessing module,
  440. so everything works except for forking_enable(False) (which
  441. silently degrades).
  442. - Fixed Python 2.5 compat. issues.
  443. - Uses more with statements
  444. - Merged some of the changes from the Python 3 branch.
  445. 2.7.3.2 - 2012-04-20
  446. --------------------
  447. - Now installs on PyPy/Jython (but does not work).
  448. 2.7.3.1 - 2012-04-20
  449. --------------------
  450. - Python 2.5 support added.
  451. 2.7.3.0 - 2012-04-20
  452. --------------------
  453. - Updated from Python 2.7.3
  454. - Python 2.4 support removed, now only supports 2.5, 2.6 and 2.7.
  455. (may consider py3k support at some point).
  456. - Pool improvements from Celery.
  457. - no-execv patch added (https://bugs.python.org/issue8713)