| 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150 |
- The Paste HTTP Server Thread Pool
- =================================
- This document describes how the thread pool in ``paste.httpserver``
- works, and how it can adapt to problems.
- Note all of the configuration parameters listed here are prefixed with
- ``threadpool_`` when running through a Paste Deploy configuration.
- Error Cases
- -----------
- When a WSGI application is called, it's possible that it will block
- indefinitely. There's two basic ways you can manage threads:
- * Start a thread on every request, close it down when the thread stops
- * Start a pool of threads, and reuse those threads for subsequent
- requests
- In both cases things go wrong -- if you start a thread every request
- you will have an explosion of threads, and with it memory and a loss
- of performance. This can culminate in really high loads, swapping,
- and the whole site grinds to a halt.
- If you are using a pool of threads, all the threads can simply be used
- up. New requests go into a queue to be processed, but since that
- queue never moves forward everyone will just block. The site
- basically freezes, though memory usage doesn't generally get worse.
- Paste Thread Pool
- -----------------
- The thread pool in Paste has some options to walk the razor's edge
- between the two techniques, and to try to respond usefully in most
- cases.
- The pool tracks all workers threads. Threads can be in a few states:
- * Idle, waiting for a request ("idle")
- * Working on a request
- * For a reasonable amount of time ("busy")
- * For an unreasonably long amount of time ("hung")
- * Thread that should die
- * An exception has been injected that should kill the thread, but it
- hasn't happened yet ("dying")
- * An exception has been injected, but the thread has persisted for
- an unreasonable amount of time ("zombie")
- When a request comes in, if there are no idle worker threads waiting
- then the server looks at the workers; all workers are busy or hung.
- If too many are hung, another thread is opened up. The limit is if
- there are less than ``spawn_if_under`` busy threads. So if you have
- 10 workers, ``spawn_if_under`` is 5, and there are 6 hung threads and
- 4 busy threads, another thread will be opened (bringing the number of
- busy threads back to 5). Later those threads may be collected again
- if some of the threads become un-hung. A thread is hung if it has
- been working for longer than ``hung_thread_limit`` (default 30
- seconds).
- Every so often, the server will check all the threads for error
- conditions. This happens every ``hung_check_period`` requests
- (default 100). At this time if there are more than enough threads
- (because of ``spawn_if_under``) some threads may be collected. If any
- threads have been working for longer than ``kill_thread_limit``
- (default 1800 seconds, i.e., 30 minutes) then the thread will be
- killed.
- To kill a thread the ``ctypes`` module must be installed. This will
- raise an exception (``SystemExit``) in the thread, which should cause
- the thread to stop. It can take quite a while for this to actually
- take effect, sometimes on the order of several minutes. This uses a
- non-public API (hence the ``ctypes`` requirement), and so it might not
- work in all cases. I've tried it in pure Python code and with a hung
- socket, and in both cases it worked. As soon as the thread is killed
- (before it is actually dead) another worker is added to the pool.
- If the killed thread lives longer than ``dying_thread_limit`` (default
- 300 seconds, 5 minutes) then it is considered a zombie.
- Zombie threads are not handled specially unless you set
- ``max_zombies_before_die``. If you set this and there are more than
- this many zombie threads, then the entire process will be killed.
- This is useful if you are running the server under some process
- monitor, such as ``start-stop-daemon``, ``daemontools``, ``runit``, or
- with ``paster serve --monitor``. To make the process die, it may run
- ``os._exit``, which is considered an impolite way to exit a process
- (akin to ``kill -9``). It *will* try to run the functions registered
- with ``atexit`` (except for the thread cleanup functions, which are
- the ones which will block so long as there are living threads).
- Notification
- ------------
- If you set ``error_email`` (including setting it globally in a Paste
- Deploy ``[DEFAULT]`` section) then you will be notified of two error
- conditions: when hung threads are killed, and when the process is
- killed due to too many zombie threads.
- Missed Cases
- ------------
- If you have a worker pool size of 10, and 11 slow or hung requests
- come in, the first 10 will get handed off but the server won't know
- yet that they will hang. The last request will stay stuck in a queue
- until another request comes in. When a later request comes later
- (after ``hung_thread_limit`` seconds) the server will notice the
- problem and add more threads, and the 11th request will come through.
- If a trickle of bad requests keeps coming in, the number of hung
- threads will keep increasing. At 100 the ``hung_check_period`` may
- not clean them up fast enough.
- Killing threads is not something Python really supports. Corruption
- of the process, memory leaks, or who knows what might occur. For the
- most part the threads seem to be killed in a fairly simple manner --
- an exception is raised, and ``finally`` blocks do get executed. But
- this hasn't been tried much in production, so there's not much
- experience with it.
- watch_threads
- -------------
- If you want to see what's going on in your process, you can install
- the application ``egg:Paste#watch_threads`` (in the
- ``paste.debug.watchthreads`` module). This lets you see requests and
- how long they have been running. In Python 2.5 you can see tracebacks
- of the running requests; before that you can only see request data
- (URLs, User-Agent, etc). If you set ``allow_kill = true`` then you
- can also kill threads from the application. The thread pool is
- intended to run reliably without intervention, but this can help debug
- problems or give you some feeling of what causes problems in the site.
- This does open up privacy problems, as it gives you access to all the
- request data in the site, including cookies, IP addresses, etc. It
- shouldn't be left on in a public setting.
- socket_timeout
- --------------
- The HTTP server (not the thread pool) also accepts an argument
- ``socket_timeout``. It is turned off by default. You might find it
- helpful to turn it on.
|