README.twisted 7.6 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181
  1. --work in progress--
  2. Introduction
  3. ------------
  4. Twisted provides solid foundation for asynchronous programming in Python.
  5. Eventlet makes asynchronous programming look like synchronous, thus
  6. achieving higher signal-to-noise ratio than traditional twisted programs have.
  7. Eventlet on top of twisted provides:
  8. * stable twisted
  9. * usable and readable synchronous style
  10. * existing twisted code can be used without any changes
  11. * existing blocking code can be used after trivial changes applied
  12. NOTE: the maintainer of Eventlet's Twisted support no longer supports it; it still exists but may have had some breakage along the way. Please treat it as experimental, and if you'd like to maintain it, please do!
  13. Eventlet features:
  14. * utilities for spawning and controlling greenlet execution:
  15. api.spawn, api.kill, proc module
  16. * utilities for communicating between greenlets:
  17. event.Event, queue.Queue, semaphore.Semaphore
  18. * standard Python modules that won't block the reactor:
  19. eventlet.green package
  20. * utilities specific to twisted hub:
  21. eventlet.twistedutil package
  22. Getting started with eventlet on twisted
  23. ----------------------------------------
  24. This section will only mention stuff that may be useful but it
  25. won't explain in details how to use it. For that, refer to the
  26. docstrings of the modules and the examples.
  27. There are 2 ways of using twisted with eventlet, one that is
  28. familiar to twisted developers and another that is familiar
  29. to eventlet developers:
  30. 1. explicitly start the main loop in the main greenlet;
  31. 2. implicitly start the main loop in a dedicated greenlet.
  32. To enable (1), add this line at the top of your program:
  33. from eventlet.twistedutil import join_reactor
  34. then start the reactor as you would do in a regular twisted application.
  35. For (2) just make sure that you have reactor installed before using
  36. any of eventlet functions. Otherwise a non-twisted hub will be selected
  37. and twisted code won't work.
  38. Most of examples/twisted_* use twisted style with the exception of
  39. twisted_client.py and twisted_srvconnector.py. All of the non-twisted
  40. examples in examples directory use eventlet-style (they work with any
  41. of eventlet's hubs, not just twisted-based).
  42. Eventlet implements "blocking" operations by switching to the main loop
  43. greenlet, thus it's impossible to call a blocking function when you are
  44. already in the main loop. Therefore one must be cautious in a twisted
  45. callback, calling only a non-blocking subset of eventlet API here. The
  46. following functions won't unschedule the current greenlet and are safe
  47. to call from anywhere:
  48. 1. Greenlet creation functions: api.spawn, proc.spawn,
  49. twistedutil.deferToGreenThread and others based on api.spawn.
  50. 2. send(), send_exception(), poll(), ready() methods of event.Event
  51. and queue.Queue.
  52. 3. wait(timeout=0) is identical to poll(). Currently only Proc.wait
  53. supports timeout parameter.
  54. 4. Proc.link/link_value/link_exception
  55. Other classes that use these names should follow the convention.
  56. For an example on how to take advantage of eventlet in a twisted
  57. application using deferToGreenThread see examples/twisted_http_proxy.py
  58. Although eventlet provides eventlet.green.socket module that implements
  59. interface of the standard Python socket, there's also a way to use twisted's
  60. network code in a synchronous fashion via GreenTransport class.
  61. A GreenTransport interface is reminiscent of socket but it's not a drop-in
  62. replacement. It combines features of TCPTransport and Protocol in a single
  63. object:
  64. * all of transport methods (like getPeer()) are available directly on
  65. a GreenTransport instance; in addition, underlying transport object
  66. is available via 'transport' attribute;
  67. * write method is overriden: it may block if transport write buffer is full;
  68. * read() and recv() methods are provided to retrieve the data from protocol
  69. synchronously.
  70. To make a GreenTransport instance use twistedutil.protocol.GreenClientCreator
  71. (usage is similar to that of twisted.internet.protocol.ClientCreator)
  72. For an example on how to get a connected GreenTransport instance,
  73. see twisted_client.py, twisted_srvconnect.py or twisted_portforward.py.
  74. For an example on how to use GreenTransport for incoming connections,
  75. see twisted_server.py, twisted_portforward.py.
  76. also
  77. * twistedutil.block_on - wait for a deferred to fire
  78. block_on(reactor.callInThread(func, args))
  79. * twistedutil.protocol.basic.LineOnlyReceiverTransport - a green transport
  80. variant built on top of LineOnlyReceiver protocol. Demonstrates how
  81. to convert a protocol to a synchronous mode.
  82. Coroutines
  83. ----------
  84. To understand how eventlet works, one has to understand how to use greenlet:
  85. http://codespeak.net/py/dist/greenlet.html
  86. Essential points
  87. * There always exists MAIN greenlet
  88. * Every greenlet except MAIN has a parent. MAIN therefore could be detected as g.parent is None
  89. * When greenlet is finished it's return value is propagated to the parent (i.e. switch() call
  90. in the parent greenlet returns it)
  91. * When an exception leaves a greelen, it's propagated to the parent (i.e. switch() in the parent
  92. re-raises it) unless it's a subclass of GreenletExit, which is returned as a value.
  93. * parent can be reassigned (by simply setting 'parent' attribute). A cycle would be detected and
  94. rejected with ValueError
  95. Note, that there's no scheduler of any sort; if a coroutine wants to be
  96. scheduled again it must take care of it itself. As an application developer,
  97. however, you don't need to worry about it as that's what eventlet does behind
  98. the scenes. The cost of that is that you should not use greenlet's switch() and
  99. throw() methods, they will likely leave the current greenlet unscheduled
  100. forever. Eventlet also takes advantage of greenlet's `parent' attribute,
  101. so you should not meddle with it either.
  102. How does eventlet work
  103. ----------------------
  104. Twisted's reactor and eventlet's hub are very similar in what they do.
  105. Both continuously perform polling on the list of registered descriptors
  106. and each time a specific event is fired, the associated callback function
  107. is called. In addition, both maintain a list of scheduled calls.
  108. Polling is performed by the main loop - a function that both reactor and hub have.
  109. When twisted calls user's callback it's expected to return almost immediately,
  110. without any blocking I/O calls.
  111. Eventlet runs the main loop in a dedicated greenlet (MAIN_LOOP). It is the same
  112. greenlet as MAIN if you use join_reactor. Otherwise it's a separate greenlet
  113. started implicitly. The execution is organized in a such way that the switching
  114. always involves MAIN_LOOP. All of functions in eventlet that appear "blocking"
  115. use the following algorithm:
  116. 1. register a callback that switches back to the current greenlet when
  117. an event of interest happens
  118. 2. switch to the MAIN_LOOP
  119. For example, here's what eventlet's socket recv() does:
  120. = blocking operation RECV on socket d =
  121. user's greenlet (USER) main loop's greenlet (MAIN_LOOP)
  122. |
  123. (inside d.recv() call)
  124. |
  125. add_descriptor(d, RECV)
  126. |
  127. data=MAIN_LOOP.switch() ---------> poll for events
  128. ^---------------------\ |
  129. | ... ---------------------------> may execute other greenlets here
  130. | |
  131. | event RECV on descriptor d?
  132. | |
  133. | d.remove_descriptor(d, RECV)
  134. | |
  135. | data = d.recv() # calling blocking op that will return immediately
  136. | |
  137. \--------- USER.switch(data) # argument data here becomes return value in user's switch
  138. return data