paste-httpserver-threadpool.txt 6.3 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150
  1. The Paste HTTP Server Thread Pool
  2. =================================
  3. This document describes how the thread pool in ``paste.httpserver``
  4. works, and how it can adapt to problems.
  5. Note all of the configuration parameters listed here are prefixed with
  6. ``threadpool_`` when running through a Paste Deploy configuration.
  7. Error Cases
  8. -----------
  9. When a WSGI application is called, it's possible that it will block
  10. indefinitely. There's two basic ways you can manage threads:
  11. * Start a thread on every request, close it down when the thread stops
  12. * Start a pool of threads, and reuse those threads for subsequent
  13. requests
  14. In both cases things go wrong -- if you start a thread every request
  15. you will have an explosion of threads, and with it memory and a loss
  16. of performance. This can culminate in really high loads, swapping,
  17. and the whole site grinds to a halt.
  18. If you are using a pool of threads, all the threads can simply be used
  19. up. New requests go into a queue to be processed, but since that
  20. queue never moves forward everyone will just block. The site
  21. basically freezes, though memory usage doesn't generally get worse.
  22. Paste Thread Pool
  23. -----------------
  24. The thread pool in Paste has some options to walk the razor's edge
  25. between the two techniques, and to try to respond usefully in most
  26. cases.
  27. The pool tracks all workers threads. Threads can be in a few states:
  28. * Idle, waiting for a request ("idle")
  29. * Working on a request
  30. * For a reasonable amount of time ("busy")
  31. * For an unreasonably long amount of time ("hung")
  32. * Thread that should die
  33. * An exception has been injected that should kill the thread, but it
  34. hasn't happened yet ("dying")
  35. * An exception has been injected, but the thread has persisted for
  36. an unreasonable amount of time ("zombie")
  37. When a request comes in, if there are no idle worker threads waiting
  38. then the server looks at the workers; all workers are busy or hung.
  39. If too many are hung, another thread is opened up. The limit is if
  40. there are less than ``spawn_if_under`` busy threads. So if you have
  41. 10 workers, ``spawn_if_under`` is 5, and there are 6 hung threads and
  42. 4 busy threads, another thread will be opened (bringing the number of
  43. busy threads back to 5). Later those threads may be collected again
  44. if some of the threads become un-hung. A thread is hung if it has
  45. been working for longer than ``hung_thread_limit`` (default 30
  46. seconds).
  47. Every so often, the server will check all the threads for error
  48. conditions. This happens every ``hung_check_period`` requests
  49. (default 100). At this time if there are more than enough threads
  50. (because of ``spawn_if_under``) some threads may be collected. If any
  51. threads have been working for longer than ``kill_thread_limit``
  52. (default 1800 seconds, i.e., 30 minutes) then the thread will be
  53. killed.
  54. To kill a thread the ``ctypes`` module must be installed. This will
  55. raise an exception (``SystemExit``) in the thread, which should cause
  56. the thread to stop. It can take quite a while for this to actually
  57. take effect, sometimes on the order of several minutes. This uses a
  58. non-public API (hence the ``ctypes`` requirement), and so it might not
  59. work in all cases. I've tried it in pure Python code and with a hung
  60. socket, and in both cases it worked. As soon as the thread is killed
  61. (before it is actually dead) another worker is added to the pool.
  62. If the killed thread lives longer than ``dying_thread_limit`` (default
  63. 300 seconds, 5 minutes) then it is considered a zombie.
  64. Zombie threads are not handled specially unless you set
  65. ``max_zombies_before_die``. If you set this and there are more than
  66. this many zombie threads, then the entire process will be killed.
  67. This is useful if you are running the server under some process
  68. monitor, such as ``start-stop-daemon``, ``daemontools``, ``runit``, or
  69. with ``paster serve --monitor``. To make the process die, it may run
  70. ``os._exit``, which is considered an impolite way to exit a process
  71. (akin to ``kill -9``). It *will* try to run the functions registered
  72. with ``atexit`` (except for the thread cleanup functions, which are
  73. the ones which will block so long as there are living threads).
  74. Notification
  75. ------------
  76. If you set ``error_email`` (including setting it globally in a Paste
  77. Deploy ``[DEFAULT]`` section) then you will be notified of two error
  78. conditions: when hung threads are killed, and when the process is
  79. killed due to too many zombie threads.
  80. Missed Cases
  81. ------------
  82. If you have a worker pool size of 10, and 11 slow or hung requests
  83. come in, the first 10 will get handed off but the server won't know
  84. yet that they will hang. The last request will stay stuck in a queue
  85. until another request comes in. When a later request comes later
  86. (after ``hung_thread_limit`` seconds) the server will notice the
  87. problem and add more threads, and the 11th request will come through.
  88. If a trickle of bad requests keeps coming in, the number of hung
  89. threads will keep increasing. At 100 the ``hung_check_period`` may
  90. not clean them up fast enough.
  91. Killing threads is not something Python really supports. Corruption
  92. of the process, memory leaks, or who knows what might occur. For the
  93. most part the threads seem to be killed in a fairly simple manner --
  94. an exception is raised, and ``finally`` blocks do get executed. But
  95. this hasn't been tried much in production, so there's not much
  96. experience with it.
  97. watch_threads
  98. -------------
  99. If you want to see what's going on in your process, you can install
  100. the application ``egg:Paste#watch_threads`` (in the
  101. ``paste.debug.watchthreads`` module). This lets you see requests and
  102. how long they have been running. In Python 2.5 you can see tracebacks
  103. of the running requests; before that you can only see request data
  104. (URLs, User-Agent, etc). If you set ``allow_kill = true`` then you
  105. can also kill threads from the application. The thread pool is
  106. intended to run reliably without intervention, but this can help debug
  107. problems or give you some feeling of what causes problems in the site.
  108. This does open up privacy problems, as it gives you access to all the
  109. request data in the site, including cookies, IP addresses, etc. It
  110. shouldn't be left on in a public setting.
  111. socket_timeout
  112. --------------
  113. The HTTP server (not the thread pool) also accepts an argument
  114. ``socket_timeout``. It is turned off by default. You might find it
  115. helpful to turn it on.