advanced-usage.rst 11 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291
  1. Advanced Usage
  2. ==============
  3. .. currentmodule:: urllib3
  4. Customizing pool behavior
  5. -------------------------
  6. The :class:`~poolmanager.PoolManager` class automatically handles creating
  7. :class:`~connectionpool.ConnectionPool` instances for each host as needed. By
  8. default, it will keep a maximum of 10 :class:`~connectionpool.ConnectionPool`
  9. instances. If you're making requests to many different hosts it might improve
  10. performance to increase this number::
  11. >>> import urllib3
  12. >>> http = urllib3.PoolManager(num_pools=50)
  13. However, keep in mind that this does increase memory and socket consumption.
  14. Similarly, the :class:`~connectionpool.ConnectionPool` class keeps a pool
  15. of individual :class:`~connection.HTTPConnection` instances. These connections
  16. are used during an individual request and returned to the pool when the request
  17. is complete. By default only one connection will be saved for re-use. If you
  18. are making many requests to the same host simultaneously it might improve
  19. performance to increase this number::
  20. >>> import urllib3
  21. >>> http = urllib3.PoolManager(maxsize=10)
  22. # Alternatively
  23. >>> http = urllib3.HTTPConnectionPool('google.com', maxsize=10)
  24. The behavior of the pooling for :class:`~connectionpool.ConnectionPool` is
  25. different from :class:`~poolmanager.PoolManager`. By default, if a new
  26. request is made and there is no free connection in the pool then a new
  27. connection will be created. However, this connection will not be saved if more
  28. than ``maxsize`` connections exist. This means that ``maxsize`` does not
  29. determine the maximum number of connections that can be open to a particular
  30. host, just the maximum number of connections to keep in the pool. However, if you specify ``block=True`` then there can be at most ``maxsize`` connections
  31. open to a particular host::
  32. >>> http = urllib3.PoolManager(maxsize=10, block=True)
  33. # Alternatively
  34. >>> http = urllib3.HTTPConnectionPool('google.com', maxsize=10, block=True)
  35. Any new requests will block until a connection is available from the pool.
  36. This is a great way to prevent flooding a host with too many connections in
  37. multi-threaded applications.
  38. .. _stream:
  39. Streaming and IO
  40. ----------------
  41. When dealing with large responses it's often better to stream the response
  42. content::
  43. >>> import urllib3
  44. >>> http = urllib3.PoolManager()
  45. >>> r = http.request(
  46. ... 'GET',
  47. ... 'http://httpbin.org/bytes/1024',
  48. ... preload_content=False)
  49. >>> for chunk in r.stream(32):
  50. ... print(chunk)
  51. b'...'
  52. b'...'
  53. ...
  54. >>> r.release_conn()
  55. Setting ``preload_content`` to ``False`` means that urllib3 will stream the
  56. response content. :meth:`~response.HTTPResponse.stream` lets you iterate over
  57. chunks of the response content.
  58. .. note:: When using ``preload_content=False``, you should call
  59. :meth:`~response.HTTPResponse.release_conn` to release the http connection
  60. back to the connection pool so that it can be re-used.
  61. However, you can also treat the :class:`~response.HTTPResponse` instance as
  62. a file-like object. This allows you to do buffering::
  63. >>> r = http.request(
  64. ... 'GET',
  65. ... 'http://httpbin.org/bytes/1024',
  66. ... preload_content=False)
  67. >>> r.read(4)
  68. b'\x88\x1f\x8b\xe5'
  69. Calls to :meth:`~response.HTTPResponse.read()` will block until more response
  70. data is available.
  71. >>> import io
  72. >>> reader = io.BufferedReader(r, 8)
  73. >>> reader.read(4)
  74. >>> r.release_conn()
  75. You can use this file-like object to do things like decode the content using
  76. :mod:`codecs`::
  77. >>> import codecs
  78. >>> reader = codecs.getreader('utf-8')
  79. >>> r = http.request(
  80. ... 'GET',
  81. ... 'http://httpbin.org/ip',
  82. ... preload_content=False)
  83. >>> json.load(reader(r))
  84. {'origin': '127.0.0.1'}
  85. >>> r.release_conn()
  86. .. _proxies:
  87. Proxies
  88. -------
  89. You can use :class:`~poolmanager.ProxyManager` to tunnel requests through an
  90. HTTP proxy::
  91. >>> import urllib3
  92. >>> proxy = urllib3.ProxyManager('http://localhost:3128/')
  93. >>> proxy.request('GET', 'http://google.com/')
  94. The usage of :class:`~poolmanager.ProxyManager` is the same as
  95. :class:`~poolmanager.PoolManager`.
  96. You can use :class:`~contrib.socks.SOCKSProxyManager` to connect to SOCKS4 or
  97. SOCKS5 proxies. In order to use SOCKS proxies you will need to install
  98. `PySocks <https://pypi.org/project/PySocks/>`_ or install urllib3 with the
  99. ``socks`` extra::
  100. pip install urllib3[socks]
  101. Once PySocks is installed, you can use
  102. :class:`~contrib.socks.SOCKSProxyManager`::
  103. >>> from urllib3.contrib.socks import SOCKSProxyManager
  104. >>> proxy = SOCKSProxyManager('socks5://localhost:8889/')
  105. >>> proxy.request('GET', 'http://google.com/')
  106. .. _ssl_custom:
  107. Custom SSL certificates
  108. -----------------------
  109. Instead of using `certifi <https://certifi.io/>`_ you can provide your
  110. own certificate authority bundle. This is useful for cases where you've
  111. generated your own certificates or when you're using a private certificate
  112. authority. Just provide the full path to the certificate bundle when creating a
  113. :class:`~poolmanager.PoolManager`::
  114. >>> import urllib3
  115. >>> http = urllib3.PoolManager(
  116. ... cert_reqs='CERT_REQUIRED',
  117. ... ca_certs='/path/to/your/certificate_bundle')
  118. When you specify your own certificate bundle only requests that can be
  119. verified with that bundle will succeed. It's recommended to use a separate
  120. :class:`~poolmanager.PoolManager` to make requests to URLs that do not need
  121. the custom certificate.
  122. .. _ssl_client:
  123. Client certificates
  124. -------------------
  125. You can also specify a client certificate. This is useful when both the server
  126. and the client need to verify each other's identity. Typically these
  127. certificates are issued from the same authority. To use a client certificate,
  128. provide the full path when creating a :class:`~poolmanager.PoolManager`::
  129. >>> http = urllib3.PoolManager(
  130. ... cert_file='/path/to/your/client_cert.pem',
  131. ... cert_reqs='CERT_REQUIRED',
  132. ... ca_certs='/path/to/your/certificate_bundle')
  133. If you have an encrypted client certificate private key you can use
  134. the ``key_password`` parameter to specify a password to decrypt the key. ::
  135. >>> http = urllib3.PoolManager(
  136. ... cert_file='/path/to/your/client_cert.pem',
  137. ... cert_reqs='CERT_REQUIRED',
  138. ... key_file='/path/to/your/client.key',
  139. ... key_password='keyfile_password')
  140. If your key isn't encrypted the ``key_password`` parameter isn't required.
  141. .. _ssl_mac:
  142. Certificate validation and Mac OS X
  143. -----------------------------------
  144. Apple-provided Python and OpenSSL libraries contain a patches that make them
  145. automatically check the system keychain's certificates. This can be
  146. surprising if you specify custom certificates and see requests unexpectedly
  147. succeed. For example, if you are specifying your own certificate for validation
  148. and the server presents a different certificate you would expect the connection
  149. to fail. However, if that server presents a certificate that is in the system
  150. keychain then the connection will succeed.
  151. `This article <https://hynek.me/articles/apple-openssl-verification-surprises/>`_
  152. has more in-depth analysis and explanation.
  153. .. _ssl_warnings:
  154. SSL Warnings
  155. ------------
  156. urllib3 will issue several different warnings based on the level of certificate
  157. verification support. These warnings indicate particular situations and can
  158. be resolved in different ways.
  159. * :class:`~exceptions.InsecureRequestWarning`
  160. This happens when a request is made to an HTTPS URL without certificate
  161. verification enabled. Follow the :ref:`certificate verification <ssl>`
  162. guide to resolve this warning.
  163. * :class:`~exceptions.InsecurePlatformWarning`
  164. This happens on Python 2 platforms that have an outdated :mod:`ssl` module.
  165. These older :mod:`ssl` modules can cause some insecure requests to succeed
  166. where they should fail and secure requests to fail where they should
  167. succeed. Follow the :ref:`pyOpenSSL <ssl_py2>` guide to resolve this
  168. warning.
  169. .. _sni_warning:
  170. * :class:`~exceptions.SNIMissingWarning`
  171. This happens on Python 2 versions older than 2.7.9. These older versions
  172. lack `SNI <https://en.wikipedia.org/wiki/Server_Name_Indication>`_ support.
  173. This can cause servers to present a certificate that the client thinks is
  174. invalid. Follow the :ref:`pyOpenSSL <ssl_py2>` guide to resolve this
  175. warning.
  176. .. _disable_ssl_warnings:
  177. Making unverified HTTPS requests is **strongly** discouraged, however, if you
  178. understand the risks and wish to disable these warnings, you can use :func:`~urllib3.disable_warnings`::
  179. >>> import urllib3
  180. >>> urllib3.disable_warnings()
  181. Alternatively you can capture the warnings with the standard :mod:`logging` module::
  182. >>> logging.captureWarnings(True)
  183. Finally, you can suppress the warnings at the interpreter level by setting the
  184. ``PYTHONWARNINGS`` environment variable or by using the
  185. `-W flag <https://docs.python.org/3/using/cmdline.html#cmdoption-w>`_.
  186. Google App Engine
  187. -----------------
  188. urllib3 supports `Google App Engine <https://cloud.google.com/appengine>`_ with
  189. some caveats.
  190. If you're using the `Flexible environment
  191. <https://cloud.google.com/appengine/docs/flexible/>`_, you do not have to do
  192. any configuration- urllib3 will just work. However, if you're using the
  193. `Standard environment <https://cloud.google.com/appengine/docs/python/>`_ then
  194. you either have to use :mod:`urllib3.contrib.appengine`'s
  195. :class:`~urllib3.contrib.appengine.AppEngineManager` or use the `Sockets API
  196. <https://cloud.google.com/appengine/docs/python/sockets/>`_
  197. To use :class:`~urllib3.contrib.appengine.AppEngineManager`::
  198. >>> from urllib3.contrib.appengine import AppEngineManager
  199. >>> http = AppEngineManager()
  200. >>> http.request('GET', 'https://google.com/')
  201. To use the Sockets API, add the following to your app.yaml and use
  202. :class:`~urllib3.poolmanager.PoolManager` as usual::
  203. env_variables:
  204. GAE_USE_SOCKETS_HTTPLIB : 'true'
  205. For more details on the limitations and gotchas, see
  206. :mod:`urllib3.contrib.appengine`.
  207. Brotli Encoding
  208. ---------------
  209. Brotli is a compression algorithm created by Google with better compression
  210. than gzip and deflate and is supported by urllib3 if the
  211. `brotlipy <https://github.com/python-hyper/brotlipy>`_ package is installed.
  212. You may also request the package be installed via the ``urllib3[brotli]`` extra::
  213. python -m pip install urllib3[brotli]
  214. Here's an example using brotli encoding via the ``Accept-Encoding`` header::
  215. >>> from urllib3 import PoolManager
  216. >>> http = PoolManager()
  217. >>> http.request('GET', 'https://www.google.com/', headers={'Accept-Encoding': 'br'})