README.rst 12 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360
  1. ========================================
  2. kombu - Messaging library for Python
  3. ========================================
  4. |build-status| |coverage| |license| |wheel| |pyversion| |pyimp|
  5. :Version: 4.3.0
  6. :Documentation: https://kombu.readthedocs.io/
  7. :Download: https://pypi.org/project/kombu/
  8. :Source: https://github.com/celery/kombu/
  9. :Keywords: messaging, amqp, rabbitmq, redis, mongodb, python, queue
  10. About
  11. =====
  12. `Kombu` is a messaging library for Python.
  13. The aim of `Kombu` is to make messaging in Python as easy as possible by
  14. providing an idiomatic high-level interface for the AMQ protocol, and also
  15. provide proven and tested solutions to common messaging problems.
  16. `AMQP`_ is the Advanced Message Queuing Protocol, an open standard protocol
  17. for message orientation, queuing, routing, reliability and security,
  18. for which the `RabbitMQ`_ messaging server is the most popular implementation.
  19. Features
  20. ========
  21. * Allows application authors to support several message server
  22. solutions by using pluggable transports.
  23. * AMQP transport using the `py-amqp`_, `librabbitmq`_, or `qpid-python`_ libraries.
  24. * High performance AMQP transport written in C - when using `librabbitmq`_
  25. This is automatically enabled if librabbitmq is installed:
  26. ::
  27. $ pip install librabbitmq
  28. * Virtual transports makes it really easy to add support for non-AMQP
  29. transports. There is already built-in support for `Redis`_,
  30. `Amazon SQS`_, `ZooKeeper`_, `SoftLayer MQ`_ and `Pyro`_.
  31. * In-memory transport for unit testing.
  32. * Supports automatic encoding, serialization and compression of message
  33. payloads.
  34. * Consistent exception handling across transports.
  35. * The ability to ensure that an operation is performed by gracefully
  36. handling connection and channel errors.
  37. * Several annoyances with `amqplib`_ has been fixed, like supporting
  38. timeouts and the ability to wait for events on more than one channel.
  39. * Projects already using `carrot`_ can easily be ported by using
  40. a compatibility layer.
  41. For an introduction to AMQP you should read the article `Rabbits and warrens`_,
  42. and the `Wikipedia article about AMQP`_.
  43. .. _`RabbitMQ`: https://www.rabbitmq.com/
  44. .. _`AMQP`: https://amqp.org
  45. .. _`py-amqp`: https://pypi.org/project/amqp/
  46. .. _`qpid-python`: https://pypi.org/project/qpid-python/
  47. .. _`Redis`: https://redis.io
  48. .. _`Amazon SQS`: https://aws.amazon.com/sqs/
  49. .. _`Zookeeper`: https://zookeeper.apache.org/
  50. .. _`Rabbits and warrens`: http://web.archive.org/web/20160323134044/http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/
  51. .. _`amqplib`: https://barryp.org/software/py-amqplib/
  52. .. _`Wikipedia article about AMQP`: https://en.wikipedia.org/wiki/AMQP
  53. .. _`carrot`: https://pypi.org/project/carrot/
  54. .. _`librabbitmq`: https://pypi.org/project/librabbitmq/
  55. .. _`Pyro`: https://pyro4.readthedocs.io/
  56. .. _`SoftLayer MQ`: https://sldn.softlayer.com/reference/messagequeueapi
  57. .. _transport-comparison:
  58. Transport Comparison
  59. ====================
  60. +---------------+----------+------------+------------+---------------+--------------+-----------------------+
  61. | **Client** | **Type** | **Direct** | **Topic** | **Fanout** | **Priority** | **TTL** |
  62. +---------------+----------+------------+------------+---------------+--------------+-----------------------+
  63. | *amqp* | Native | Yes | Yes | Yes | Yes [#f3]_ | Yes [#f4]_ |
  64. +---------------+----------+------------+------------+---------------+--------------+-----------------------+
  65. | *qpid* | Native | Yes | Yes | Yes | No | No |
  66. +---------------+----------+------------+------------+---------------+--------------+-----------------------+
  67. | *redis* | Virtual | Yes | Yes | Yes (PUB/SUB) | Yes | No |
  68. +---------------+----------+------------+------------+---------------+--------------+-----------------------+
  69. | *mongodb* | Virtual | Yes | Yes | Yes | Yes | Yes |
  70. +---------------+----------+------------+------------+---------------+--------------+-----------------------+
  71. | *SQS* | Virtual | Yes | Yes [#f1]_ | Yes [#f2]_ | No | No |
  72. +---------------+----------+------------+------------+---------------+--------------+-----------------------+
  73. | *zookeeper* | Virtual | Yes | Yes [#f1]_ | No | Yes | No |
  74. +---------------+----------+------------+------------+---------------+--------------+-----------------------+
  75. | *in-memory* | Virtual | Yes | Yes [#f1]_ | No | No | No |
  76. +---------------+----------+------------+------------+---------------+--------------+-----------------------+
  77. | *SLMQ* | Virtual | Yes | Yes [#f1]_ | No | No | No |
  78. +---------------+----------+------------+------------+---------------+--------------+-----------------------+
  79. | *Pyro* | Virtual | Yes | Yes [#f1]_ | No | No | No |
  80. +---------------+----------+------------+------------+---------------+--------------+-----------------------+
  81. .. [#f1] Declarations only kept in memory, so exchanges/queues
  82. must be declared by all clients that needs them.
  83. .. [#f2] Fanout supported via storing routing tables in SimpleDB.
  84. Disabled by default, but can be enabled by using the
  85. ``supports_fanout`` transport option.
  86. .. [#f3] AMQP Message priority support depends on broker implementation.
  87. .. [#f4] AMQP Message/Queue TTL support depends on broker implementation.
  88. Documentation
  89. -------------
  90. Kombu is using Sphinx, and the latest documentation can be found here:
  91. https://kombu.readthedocs.io/
  92. Quick overview
  93. --------------
  94. .. code:: python
  95. from kombu import Connection, Exchange, Queue
  96. media_exchange = Exchange('media', 'direct', durable=True)
  97. video_queue = Queue('video', exchange=media_exchange, routing_key='video')
  98. def process_media(body, message):
  99. print body
  100. message.ack()
  101. # connections
  102. with Connection('amqp://guest:guest@localhost//') as conn:
  103. # produce
  104. producer = conn.Producer(serializer='json')
  105. producer.publish({'name': '/tmp/lolcat1.avi', 'size': 1301013},
  106. exchange=media_exchange, routing_key='video',
  107. declare=[video_queue])
  108. # the declare above, makes sure the video queue is declared
  109. # so that the messages can be delivered.
  110. # It's a best practice in Kombu to have both publishers and
  111. # consumers declare the queue. You can also declare the
  112. # queue manually using:
  113. # video_queue(conn).declare()
  114. # consume
  115. with conn.Consumer(video_queue, callbacks=[process_media]) as consumer:
  116. # Process messages and handle events on all channels
  117. while True:
  118. conn.drain_events()
  119. # Consume from several queues on the same channel:
  120. video_queue = Queue('video', exchange=media_exchange, key='video')
  121. image_queue = Queue('image', exchange=media_exchange, key='image')
  122. with connection.Consumer([video_queue, image_queue],
  123. callbacks=[process_media]) as consumer:
  124. while True:
  125. connection.drain_events()
  126. Or handle channels manually:
  127. .. code:: python
  128. with connection.channel() as channel:
  129. producer = Producer(channel, ...)
  130. consumer = Producer(channel)
  131. All objects can be used outside of with statements too,
  132. just remember to close the objects after use:
  133. .. code:: python
  134. from kombu import Connection, Consumer, Producer
  135. connection = Connection()
  136. # ...
  137. connection.release()
  138. consumer = Consumer(channel_or_connection, ...)
  139. consumer.register_callback(my_callback)
  140. consumer.consume()
  141. # ....
  142. consumer.cancel()
  143. `Exchange` and `Queue` are simply declarations that can be pickled
  144. and used in configuration files etc.
  145. They also support operations, but to do so they need to be bound
  146. to a channel.
  147. Binding exchanges and queues to a connection will make it use
  148. that connections default channel.
  149. ::
  150. >>> exchange = Exchange('tasks', 'direct')
  151. >>> connection = Connection()
  152. >>> bound_exchange = exchange(connection)
  153. >>> bound_exchange.delete()
  154. # the original exchange is not affected, and stays unbound.
  155. >>> exchange.delete()
  156. raise NotBoundError: Can't call delete on Exchange not bound to
  157. a channel.
  158. Terminology
  159. ===========
  160. There are some concepts you should be familiar with before starting:
  161. * Producers
  162. Producers sends messages to an exchange.
  163. * Exchanges
  164. Messages are sent to exchanges. Exchanges are named and can be
  165. configured to use one of several routing algorithms. The exchange
  166. routes the messages to consumers by matching the routing key in the
  167. message with the routing key the consumer provides when binding to
  168. the exchange.
  169. * Consumers
  170. Consumers declares a queue, binds it to a exchange and receives
  171. messages from it.
  172. * Queues
  173. Queues receive messages sent to exchanges. The queues are declared
  174. by consumers.
  175. * Routing keys
  176. Every message has a routing key. The interpretation of the routing
  177. key depends on the exchange type. There are four default exchange
  178. types defined by the AMQP standard, and vendors can define custom
  179. types (so see your vendors manual for details).
  180. These are the default exchange types defined by AMQP/0.8:
  181. * Direct exchange
  182. Matches if the routing key property of the message and
  183. the `routing_key` attribute of the consumer are identical.
  184. * Fan-out exchange
  185. Always matches, even if the binding does not have a routing
  186. key.
  187. * Topic exchange
  188. Matches the routing key property of the message by a primitive
  189. pattern matching scheme. The message routing key then consists
  190. of words separated by dots (`"."`, like domain names), and
  191. two special characters are available; star (`"*"`) and hash
  192. (`"#"`). The star matches any word, and the hash matches
  193. zero or more words. For example `"*.stock.#"` matches the
  194. routing keys `"usd.stock"` and `"eur.stock.db"` but not
  195. `"stock.nasdaq"`.
  196. Installation
  197. ============
  198. You can install `Kombu` either via the Python Package Index (PyPI)
  199. or from source.
  200. To install using `pip`,:
  201. ::
  202. $ pip install kombu
  203. To install using `easy_install`,:
  204. ::
  205. $ easy_install kombu
  206. If you have downloaded a source tarball you can install it
  207. by doing the following,:
  208. ::
  209. $ python setup.py build
  210. # python setup.py install # as root
  211. Getting Help
  212. ============
  213. Mailing list
  214. ------------
  215. Join the `celery-users`_ mailing list.
  216. .. _`celery-users`: https://groups.google.com/group/celery-users/
  217. Bug tracker
  218. ===========
  219. If you have any suggestions, bug reports or annoyances please report them
  220. to our issue tracker at https://github.com/celery/kombu/issues/
  221. Contributing
  222. ============
  223. Development of `Kombu` happens at Github: https://github.com/celery/kombu
  224. You are highly encouraged to participate in the development. If you don't
  225. like Github (for some reason) you're welcome to send regular patches.
  226. License
  227. =======
  228. This software is licensed under the `New BSD License`. See the `LICENSE`
  229. file in the top distribution directory for the full license text.
  230. .. |build-status| image:: https://secure.travis-ci.org/celery/kombu.png?branch=master
  231. :alt: Build status
  232. :target: https://travis-ci.org/celery/kombu
  233. .. |coverage| image:: https://codecov.io/github/celery/kombu/coverage.svg?branch=master
  234. :target: https://codecov.io/github/celery/kombu?branch=master
  235. .. |license| image:: https://img.shields.io/pypi/l/kombu.svg
  236. :alt: BSD License
  237. :target: https://opensource.org/licenses/BSD-3-Clause
  238. .. |wheel| image:: https://img.shields.io/pypi/wheel/kombu.svg
  239. :alt: Kombu can be installed via wheel
  240. :target: https://pypi.org/project/kombu/
  241. .. |pyversion| image:: https://img.shields.io/pypi/pyversions/kombu.svg
  242. :alt: Supported Python versions.
  243. :target: https://pypi.org/project/kombu/
  244. .. |pyimp| image:: https://img.shields.io/pypi/implementation/kombu.svg
  245. :alt: Support Python implementations.
  246. :target: https://pypi.org/project/kombu/
  247. --