advanced-usage.rst 19 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503
  1. Advanced Usage
  2. ==============
  3. .. currentmodule:: urllib3
  4. Customizing Pool Behavior
  5. -------------------------
  6. The :class:`~poolmanager.PoolManager` class automatically handles creating
  7. :class:`~connectionpool.ConnectionPool` instances for each host as needed. By
  8. default, it will keep a maximum of 10 :class:`~connectionpool.ConnectionPool`
  9. instances. If you're making requests to many different hosts it might improve
  10. performance to increase this number::
  11. >>> import urllib3
  12. >>> http = urllib3.PoolManager(num_pools=50)
  13. However, keep in mind that this does increase memory and socket consumption.
  14. Similarly, the :class:`~connectionpool.ConnectionPool` class keeps a pool
  15. of individual :class:`~connection.HTTPConnection` instances. These connections
  16. are used during an individual request and returned to the pool when the request
  17. is complete. By default only one connection will be saved for re-use. If you
  18. are making many requests to the same host simultaneously it might improve
  19. performance to increase this number::
  20. >>> import urllib3
  21. >>> http = urllib3.PoolManager(maxsize=10)
  22. # Alternatively
  23. >>> http = urllib3.HTTPConnectionPool('google.com', maxsize=10)
  24. The behavior of the pooling for :class:`~connectionpool.ConnectionPool` is
  25. different from :class:`~poolmanager.PoolManager`. By default, if a new
  26. request is made and there is no free connection in the pool then a new
  27. connection will be created. However, this connection will not be saved if more
  28. than ``maxsize`` connections exist. This means that ``maxsize`` does not
  29. determine the maximum number of connections that can be open to a particular
  30. host, just the maximum number of connections to keep in the pool. However, if you specify ``block=True`` then there can be at most ``maxsize`` connections
  31. open to a particular host::
  32. >>> http = urllib3.PoolManager(maxsize=10, block=True)
  33. # Alternatively
  34. >>> http = urllib3.HTTPConnectionPool('google.com', maxsize=10, block=True)
  35. Any new requests will block until a connection is available from the pool.
  36. This is a great way to prevent flooding a host with too many connections in
  37. multi-threaded applications.
  38. .. _stream:
  39. .. _streaming_and_io:
  40. Streaming and I/O
  41. -----------------
  42. When using ``preload_content=True`` (the default setting) the
  43. response body will be read immediately into memory and the HTTP connection
  44. will be released back into the pool without manual intervention.
  45. However, when dealing with large responses it's often better to stream the response
  46. content using ``preload_content=False``. Setting ``preload_content`` to ``False`` means
  47. that urllib3 will only read from the socket when data is requested.
  48. .. note:: When using ``preload_content=False``, you need to manually release
  49. the HTTP connection back to the connection pool so that it can be re-used.
  50. To ensure the HTTP connection is in a valid state before being re-used
  51. all data should be read off the wire.
  52. You can call the :meth:`~response.HTTPResponse.drain_conn` to throw away
  53. unread data still on the wire. This call isn't necessary if data has already
  54. been completely read from the response.
  55. After all data is read you can call :meth:`~response.HTTPResponse.release_conn`
  56. to release the connection into the pool.
  57. You can call the :meth:`~response.HTTPResponse.close` to close the connection,
  58. but this call doesn’t return the connection to the pool, throws away the unread
  59. data on the wire, and leaves the connection in an undefined protocol state.
  60. This is desirable if you prefer not reading data from the socket to re-using the
  61. HTTP connection.
  62. :meth:`~response.HTTPResponse.stream` lets you iterate over chunks of the response content.
  63. >>> import urllib3
  64. >>> http = urllib3.PoolManager()
  65. >>> r = http.request(
  66. ... 'GET',
  67. ... 'http://httpbin.org/bytes/1024',
  68. ... preload_content=False)
  69. >>> for chunk in r.stream(32):
  70. ... print(chunk)
  71. b'...'
  72. b'...'
  73. ...
  74. >>> r.release_conn()
  75. However, you can also treat the :class:`~response.HTTPResponse` instance as
  76. a file-like object. This allows you to do buffering::
  77. >>> r = http.request(
  78. ... 'GET',
  79. ... 'http://httpbin.org/bytes/1024',
  80. ... preload_content=False)
  81. >>> r.read(4)
  82. b'\x88\x1f\x8b\xe5'
  83. Calls to :meth:`~response.HTTPResponse.read()` will block until more response
  84. data is available.
  85. >>> import io
  86. >>> reader = io.BufferedReader(r, 8)
  87. >>> reader.read(4)
  88. >>> r.release_conn()
  89. You can use this file-like object to do things like decode the content using
  90. :mod:`codecs`::
  91. >>> import codecs
  92. >>> reader = codecs.getreader('utf-8')
  93. >>> r = http.request(
  94. ... 'GET',
  95. ... 'http://httpbin.org/ip',
  96. ... preload_content=False)
  97. >>> json.load(reader(r))
  98. {'origin': '127.0.0.1'}
  99. >>> r.release_conn()
  100. .. _proxies:
  101. Proxies
  102. -------
  103. You can use :class:`~poolmanager.ProxyManager` to tunnel requests through an
  104. HTTP proxy::
  105. >>> import urllib3
  106. >>> proxy = urllib3.ProxyManager('http://localhost:3128/')
  107. >>> proxy.request('GET', 'http://google.com/')
  108. The usage of :class:`~poolmanager.ProxyManager` is the same as
  109. :class:`~poolmanager.PoolManager`.
  110. You can connect to a proxy using HTTP, HTTPS or SOCKS. urllib3's behavior will
  111. be different depending on the type of proxy you selected and the destination
  112. you're contacting.
  113. HTTP and HTTPS Proxies
  114. ~~~~~~~~~~~~~~~~~~~~~~
  115. Both HTTP/HTTPS proxies support HTTP and HTTPS destinations. The only
  116. difference between them is if you need to establish a TLS connection to the
  117. proxy first. You can specify which proxy you need to contact by specifying the
  118. proper proxy scheme. (i.e ``http://`` or ``https://``)
  119. urllib3's behavior will be different depending on your proxy and destination:
  120. * HTTP proxy + HTTP destination
  121. Your request will be forwarded with the `absolute URI
  122. <https://tools.ietf.org/html/rfc7230#section-5.3.2>`_.
  123. * HTTP proxy + HTTPS destination
  124. A TCP tunnel will be established with a `HTTP
  125. CONNECT <https://tools.ietf.org/html/rfc7231#section-4.3.6>`_. Afterward a
  126. TLS connection will be established with the destination and your request
  127. will be sent.
  128. * HTTPS proxy + HTTP destination
  129. A TLS connection will be established to the proxy and later your request
  130. will be forwarded with the `absolute URI
  131. <https://tools.ietf.org/html/rfc7230#section-5.3.2>`_.
  132. * HTTPS proxy + HTTPS destination
  133. A TLS-in-TLS tunnel will be established. An initial TLS connection will be
  134. established to the proxy, then an `HTTP CONNECT
  135. <https://tools.ietf.org/html/rfc7231#section-4.3.6>`_ will be sent to
  136. establish a TCP connection to the destination and finally a second TLS
  137. connection will be established to the destination. You can customize the
  138. :class:`ssl.SSLContext` used for the proxy TLS connection through the
  139. ``proxy_ssl_context`` argument of the :class:`~poolmanager.ProxyManager`
  140. class.
  141. For HTTPS proxies we also support forwarding your requests to HTTPS destinations with
  142. an `absolute URI <https://tools.ietf.org/html/rfc7230#section-5.3.2>`_ if the
  143. ``use_forwarding_for_https`` argument is set to ``True``. We strongly recommend you
  144. **only use this option with trusted or corporate proxies** as the proxy will have
  145. full visibility of your requests.
  146. .. _https_proxy_error_http_proxy:
  147. Your proxy appears to only use HTTP and not HTTPS
  148. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  149. If you're receiving the :class:`~urllib3.exceptions.ProxyError` and it mentions
  150. your proxy only speaks HTTP and not HTTPS here's what to do to solve your issue:
  151. If you're using ``urllib3`` directly, make sure the URL you're passing into :class:`urllib3.ProxyManager`
  152. starts with ``http://`` instead of ``https://``:
  153. .. code-block:: python
  154. # Do this:
  155. http = urllib3.ProxyManager("http://...")
  156. # Not this:
  157. http = urllib3.ProxyManager("https://...")
  158. If instead you're using ``urllib3`` through another library like Requests
  159. there are multiple ways your proxy could be mis-configured. You need to figure out
  160. where the configuration isn't correct and make the fix there. Some common places
  161. to look are environment variables like ``HTTP_PROXY``, ``HTTPS_PROXY``, and ``ALL_PROXY``.
  162. Ensure that the values for all of these environment variables starts with ``http://``
  163. and not ``https://``:
  164. .. code-block:: bash
  165. # Check your existing environment variables in bash
  166. $ env | grep "_PROXY"
  167. HTTP_PROXY=http://127.0.0.1:8888
  168. HTTPS_PROXY=https://127.0.0.1:8888 # <--- This setting is the problem!
  169. # Make the fix in your current session and test your script
  170. $ export HTTPS_PROXY="http://127.0.0.1:8888"
  171. $ python test-proxy.py # This should now pass.
  172. # Persist your change in your shell 'profile' (~/.bashrc, ~/.profile, ~/.bash_profile, etc)
  173. # You may need to logout and log back in to ensure this works across all programs.
  174. $ vim ~/.bashrc
  175. If you're on Windows or macOS your proxy may be getting set at a system level.
  176. To check this first ensure that the above environment variables aren't set
  177. then run the following:
  178. .. code-block:: bash
  179. $ python -c 'import urllib.request; print(urllib.request.getproxies())'
  180. If the output of the above command isn't empty and looks like this:
  181. .. code-block:: python
  182. {
  183. "http": "http://127.0.0.1:8888",
  184. "https": "https://127.0.0.1:8888" # <--- This setting is the problem!
  185. }
  186. Search how to configure proxies on your operating system and change the ``https://...`` URL into ``http://``.
  187. After you make the change the return value of ``urllib.request.getproxies()`` should be:
  188. .. code-block:: python
  189. { # Everything is good here! :)
  190. "http": "http://127.0.0.1:8888",
  191. "https": "http://127.0.0.1:8888"
  192. }
  193. If you still can't figure out how to configure your proxy after all these steps
  194. please `join our community Discord <https://discord.gg/urllib3>`_ and we'll try to help you with your issue.
  195. SOCKS Proxies
  196. ~~~~~~~~~~~~~
  197. For SOCKS, you can use :class:`~contrib.socks.SOCKSProxyManager` to connect to
  198. SOCKS4 or SOCKS5 proxies. In order to use SOCKS proxies you will need to
  199. install `PySocks <https://pypi.org/project/PySocks/>`_ or install urllib3 with
  200. the ``socks`` extra::
  201. python -m pip install urllib3[socks]
  202. Once PySocks is installed, you can use
  203. :class:`~contrib.socks.SOCKSProxyManager`::
  204. >>> from urllib3.contrib.socks import SOCKSProxyManager
  205. >>> proxy = SOCKSProxyManager('socks5h://localhost:8889/')
  206. >>> proxy.request('GET', 'http://google.com/')
  207. .. note::
  208. It is recommended to use ``socks5h://`` or ``socks4a://`` schemes in
  209. your ``proxy_url`` to ensure that DNS resolution is done from the remote
  210. server instead of client-side when connecting to a domain name.
  211. .. _ssl_custom:
  212. .. _custom_ssl_certificates:
  213. Custom TLS Certificates
  214. -----------------------
  215. Instead of using `certifi <https://certifi.io/>`_ you can provide your
  216. own certificate authority bundle. This is useful for cases where you've
  217. generated your own certificates or when you're using a private certificate
  218. authority. Just provide the full path to the certificate bundle when creating a
  219. :class:`~poolmanager.PoolManager`::
  220. >>> import urllib3
  221. >>> http = urllib3.PoolManager(
  222. ... cert_reqs='CERT_REQUIRED',
  223. ... ca_certs='/path/to/your/certificate_bundle')
  224. When you specify your own certificate bundle only requests that can be
  225. verified with that bundle will succeed. It's recommended to use a separate
  226. :class:`~poolmanager.PoolManager` to make requests to URLs that do not need
  227. the custom certificate.
  228. .. _sni_custom:
  229. Custom SNI Hostname
  230. -------------------
  231. If you want to create a connection to a host over HTTPS which uses SNI, there
  232. are two places where the hostname is expected. It must be included in the Host
  233. header sent, so that the server will know which host is being requested. The
  234. hostname should also match the certificate served by the server, which is
  235. checked by urllib3.
  236. Normally, urllib3 takes care of setting and checking these values for you when
  237. you connect to a host by name. However, it's sometimes useful to set a
  238. connection's expected Host header and certificate hostname (subject),
  239. especially when you are connecting without using name resolution. For example,
  240. you could connect to a server by IP using HTTPS like so::
  241. >>> import urllib3
  242. >>> pool = urllib3.HTTPSConnectionPool(
  243. ... "10.0.0.10",
  244. ... assert_hostname="example.org",
  245. ... server_hostname="example.org"
  246. ... )
  247. >>> pool.urlopen(
  248. ... "GET",
  249. ... "/",
  250. ... headers={"Host": "example.org"},
  251. ... assert_same_host=False
  252. ... )
  253. Note that when you use a connection in this way, you must specify
  254. ``assert_same_host=False``.
  255. This is useful when DNS resolution for ``example.org`` does not match the
  256. address that you would like to use. The IP may be for a private interface, or
  257. you may want to use a specific host under round-robin DNS.
  258. .. _ssl_client:
  259. Client Certificates
  260. -------------------
  261. You can also specify a client certificate. This is useful when both the server
  262. and the client need to verify each other's identity. Typically these
  263. certificates are issued from the same authority. To use a client certificate,
  264. provide the full path when creating a :class:`~poolmanager.PoolManager`::
  265. >>> http = urllib3.PoolManager(
  266. ... cert_file='/path/to/your/client_cert.pem',
  267. ... cert_reqs='CERT_REQUIRED',
  268. ... ca_certs='/path/to/your/certificate_bundle')
  269. If you have an encrypted client certificate private key you can use
  270. the ``key_password`` parameter to specify a password to decrypt the key. ::
  271. >>> http = urllib3.PoolManager(
  272. ... cert_file='/path/to/your/client_cert.pem',
  273. ... cert_reqs='CERT_REQUIRED',
  274. ... key_file='/path/to/your/client.key',
  275. ... key_password='keyfile_password')
  276. If your key isn't encrypted the ``key_password`` parameter isn't required.
  277. .. _ssl_mac:
  278. .. _certificate_validation_and_mac_os_x:
  279. Certificate Validation and macOS
  280. --------------------------------
  281. Apple-provided Python and OpenSSL libraries contain a patches that make them
  282. automatically check the system keychain's certificates. This can be
  283. surprising if you specify custom certificates and see requests unexpectedly
  284. succeed. For example, if you are specifying your own certificate for validation
  285. and the server presents a different certificate you would expect the connection
  286. to fail. However, if that server presents a certificate that is in the system
  287. keychain then the connection will succeed.
  288. `This article <https://hynek.me/articles/apple-openssl-verification-surprises/>`_
  289. has more in-depth analysis and explanation.
  290. .. _ssl_warnings:
  291. TLS Warnings
  292. ------------
  293. urllib3 will issue several different warnings based on the level of certificate
  294. verification support. These warnings indicate particular situations and can
  295. be resolved in different ways.
  296. * :class:`~exceptions.InsecureRequestWarning`
  297. This happens when a request is made to an HTTPS URL without certificate
  298. verification enabled. Follow the :ref:`certificate verification <ssl>`
  299. guide to resolve this warning.
  300. * :class:`~exceptions.InsecurePlatformWarning`
  301. This happens on Python 2 platforms that have an outdated :mod:`ssl` module.
  302. These older :mod:`ssl` modules can cause some insecure requests to succeed
  303. where they should fail and secure requests to fail where they should
  304. succeed. Follow the :ref:`pyOpenSSL <ssl_py2>` guide to resolve this
  305. warning.
  306. .. _sni_warning:
  307. * :class:`~exceptions.SNIMissingWarning`
  308. This happens on Python 2 versions older than 2.7.9. These older versions
  309. lack `SNI <https://en.wikipedia.org/wiki/Server_Name_Indication>`_ support.
  310. This can cause servers to present a certificate that the client thinks is
  311. invalid. Follow the :ref:`pyOpenSSL <ssl_py2>` guide to resolve this
  312. warning.
  313. .. _disable_ssl_warnings:
  314. Making unverified HTTPS requests is **strongly** discouraged, however, if you
  315. understand the risks and wish to disable these warnings, you can use :func:`~urllib3.disable_warnings`:
  316. .. code-block:: pycon
  317. >>> import urllib3
  318. >>> urllib3.disable_warnings()
  319. Alternatively you can capture the warnings with the standard :mod:`logging` module:
  320. .. code-block:: pycon
  321. >>> logging.captureWarnings(True)
  322. Finally, you can suppress the warnings at the interpreter level by setting the
  323. ``PYTHONWARNINGS`` environment variable or by using the
  324. `-W flag <https://docs.python.org/3/using/cmdline.html#cmdoption-w>`_.
  325. Google App Engine
  326. -----------------
  327. urllib3 supports `Google App Engine <https://cloud.google.com/appengine>`_ with
  328. some caveats.
  329. If you're using the `Flexible environment
  330. <https://cloud.google.com/appengine/docs/flexible/>`_, you do not have to do
  331. any configuration- urllib3 will just work. However, if you're using the
  332. `Standard environment <https://cloud.google.com/appengine/docs/python/>`_ then
  333. you either have to use :mod:`urllib3.contrib.appengine`'s
  334. :class:`~urllib3.contrib.appengine.AppEngineManager` or use the `Sockets API
  335. <https://cloud.google.com/appengine/docs/python/sockets/>`_
  336. To use :class:`~urllib3.contrib.appengine.AppEngineManager`:
  337. .. code-block:: pycon
  338. >>> from urllib3.contrib.appengine import AppEngineManager
  339. >>> http = AppEngineManager()
  340. >>> http.request('GET', 'https://google.com/')
  341. To use the Sockets API, add the following to your app.yaml and use
  342. :class:`~urllib3.poolmanager.PoolManager` as usual:
  343. .. code-block:: yaml
  344. env_variables:
  345. GAE_USE_SOCKETS_HTTPLIB : 'true'
  346. For more details on the limitations and gotchas, see
  347. :mod:`urllib3.contrib.appengine`.
  348. Brotli Encoding
  349. ---------------
  350. Brotli is a compression algorithm created by Google with better compression
  351. than gzip and deflate and is supported by urllib3 if the
  352. `Brotli <https://pypi.org/Brotli>`_ package or
  353. `brotlicffi <https://github.com/python-hyper/brotlicffi>`_ package is installed.
  354. You may also request the package be installed via the ``urllib3[brotli]`` extra:
  355. .. code-block:: bash
  356. $ python -m pip install urllib3[brotli]
  357. Here's an example using brotli encoding via the ``Accept-Encoding`` header:
  358. .. code-block:: pycon
  359. >>> from urllib3 import PoolManager
  360. >>> http = PoolManager()
  361. >>> http.request('GET', 'https://www.google.com/', headers={'Accept-Encoding': 'br'})
  362. Decrypting Captured TLS Sessions with Wireshark
  363. -----------------------------------------------
  364. Python 3.8 and higher support logging of TLS pre-master secrets.
  365. With these secrets tools like `Wireshark <https://wireshark.org>`_ can decrypt captured
  366. network traffic.
  367. To enable this simply define environment variable `SSLKEYLOGFILE`:
  368. .. code-block:: bash
  369. export SSLKEYLOGFILE=/path/to/keylogfile.txt
  370. Then configure the key logfile in `Wireshark <https://wireshark.org>`_, see
  371. `Wireshark TLS Decryption <https://wiki.wireshark.org/TLS#TLS_Decryption>`_ for instructions.