user-guide.rst 14 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435
  1. User Guide
  2. ==========
  3. .. currentmodule:: urllib3
  4. Making requests
  5. ---------------
  6. First things first, import the urllib3 module::
  7. >>> import urllib3
  8. You'll need a :class:`~poolmanager.PoolManager` instance to make requests.
  9. This object handles all of the details of connection pooling and thread safety
  10. so that you don't have to::
  11. >>> http = urllib3.PoolManager()
  12. To make a request use :meth:`~poolmanager.PoolManager.request`::
  13. >>> r = http.request('GET', 'http://httpbin.org/robots.txt')
  14. >>> r.data
  15. b'User-agent: *\nDisallow: /deny\n'
  16. ``request()`` returns a :class:`~response.HTTPResponse` object, the
  17. :ref:`response_content` section explains how to handle various responses.
  18. You can use :meth:`~poolmanager.PoolManager.request` to make requests using any
  19. HTTP verb::
  20. >>> r = http.request(
  21. ... 'POST',
  22. ... 'http://httpbin.org/post',
  23. ... fields={'hello': 'world'})
  24. The :ref:`request_data` section covers sending other kinds of requests data,
  25. including JSON, files, and binary data.
  26. .. _response_content:
  27. Response content
  28. ----------------
  29. The :class:`~response.HTTPResponse` object provides
  30. :attr:`~response.HTTPResponse.status`, :attr:`~response.HTTPResponse.data`, and
  31. :attr:`~response.HTTPResponse.header` attributes::
  32. >>> r = http.request('GET', 'http://httpbin.org/ip')
  33. >>> r.status
  34. 200
  35. >>> r.data
  36. b'{\n "origin": "104.232.115.37"\n}\n'
  37. >>> r.headers
  38. HTTPHeaderDict({'Content-Length': '33', ...})
  39. JSON content
  40. ~~~~~~~~~~~~
  41. JSON content can be loaded by decoding and deserializing the
  42. :attr:`~response.HTTPResponse.data` attribute of the request::
  43. >>> import json
  44. >>> r = http.request('GET', 'http://httpbin.org/ip')
  45. >>> json.loads(r.data.decode('utf-8'))
  46. {'origin': '127.0.0.1'}
  47. Binary content
  48. ~~~~~~~~~~~~~~
  49. The :attr:`~response.HTTPResponse.data` attribute of the response is always set
  50. to a byte string representing the response content::
  51. >>> r = http.request('GET', 'http://httpbin.org/bytes/8')
  52. >>> r.data
  53. b'\xaa\xa5H?\x95\xe9\x9b\x11'
  54. .. note:: For larger responses, it's sometimes better to :ref:`stream <stream>`
  55. the response.
  56. Using io Wrappers with Response content
  57. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  58. Sometimes you want to use :class:`io.TextIOWrapper` or similar objects like a CSV reader
  59. directly with :class:`~response.HTTPResponse` data. Making these two interfaces play nice
  60. together requires using the :attr:`~response.HTTPResponse.auto_close` attribute by setting it
  61. to ``False``. By default HTTP responses are closed after reading all bytes, this disables that behavior::
  62. >>> import io
  63. >>> r = http.request('GET', 'https://example.com', preload_content=False)
  64. >>> r.auto_close = False
  65. >>> for line in io.TextIOWrapper(r):
  66. >>> print(line)
  67. .. _request_data:
  68. Request data
  69. ------------
  70. Headers
  71. ~~~~~~~
  72. You can specify headers as a dictionary in the ``headers`` argument in :meth:`~poolmanager.PoolManager.request`::
  73. >>> r = http.request(
  74. ... 'GET',
  75. ... 'http://httpbin.org/headers',
  76. ... headers={
  77. ... 'X-Something': 'value'
  78. ... })
  79. >>> json.loads(r.data.decode('utf-8'))['headers']
  80. {'X-Something': 'value', ...}
  81. Query parameters
  82. ~~~~~~~~~~~~~~~~
  83. For ``GET``, ``HEAD``, and ``DELETE`` requests, you can simply pass the
  84. arguments as a dictionary in the ``fields`` argument to
  85. :meth:`~poolmanager.PoolManager.request`::
  86. >>> r = http.request(
  87. ... 'GET',
  88. ... 'http://httpbin.org/get',
  89. ... fields={'arg': 'value'})
  90. >>> json.loads(r.data.decode('utf-8'))['args']
  91. {'arg': 'value'}
  92. For ``POST`` and ``PUT`` requests, you need to manually encode query parameters
  93. in the URL::
  94. >>> from urllib.parse import urlencode
  95. >>> encoded_args = urlencode({'arg': 'value'})
  96. >>> url = 'http://httpbin.org/post?' + encoded_args
  97. >>> r = http.request('POST', url)
  98. >>> json.loads(r.data.decode('utf-8'))['args']
  99. {'arg': 'value'}
  100. .. _form_data:
  101. Form data
  102. ~~~~~~~~~
  103. For ``PUT`` and ``POST`` requests, urllib3 will automatically form-encode the
  104. dictionary in the ``fields`` argument provided to
  105. :meth:`~poolmanager.PoolManager.request`::
  106. >>> r = http.request(
  107. ... 'POST',
  108. ... 'http://httpbin.org/post',
  109. ... fields={'field': 'value'})
  110. >>> json.loads(r.data.decode('utf-8'))['form']
  111. {'field': 'value'}
  112. JSON
  113. ~~~~
  114. You can send a JSON request by specifying the encoded data as the ``body``
  115. argument and setting the ``Content-Type`` header when calling
  116. :meth:`~poolmanager.PoolManager.request`::
  117. >>> import json
  118. >>> data = {'attribute': 'value'}
  119. >>> encoded_data = json.dumps(data).encode('utf-8')
  120. >>> r = http.request(
  121. ... 'POST',
  122. ... 'http://httpbin.org/post',
  123. ... body=encoded_data,
  124. ... headers={'Content-Type': 'application/json'})
  125. >>> json.loads(r.data.decode('utf-8'))['json']
  126. {'attribute': 'value'}
  127. Files & binary data
  128. ~~~~~~~~~~~~~~~~~~~
  129. For uploading files using ``multipart/form-data`` encoding you can use the same
  130. approach as :ref:`form_data` and specify the file field as a tuple of
  131. ``(file_name, file_data)``::
  132. >>> with open('example.txt') as fp:
  133. ... file_data = fp.read()
  134. >>> r = http.request(
  135. ... 'POST',
  136. ... 'http://httpbin.org/post',
  137. ... fields={
  138. ... 'filefield': ('example.txt', file_data),
  139. ... })
  140. >>> json.loads(r.data.decode('utf-8'))['files']
  141. {'filefield': '...'}
  142. While specifying the filename is not strictly required, it's recommended in
  143. order to match browser behavior. You can also pass a third item in the tuple
  144. to specify the file's MIME type explicitly::
  145. >>> r = http.request(
  146. ... 'POST',
  147. ... 'http://httpbin.org/post',
  148. ... fields={
  149. ... 'filefield': ('example.txt', file_data, 'text/plain'),
  150. ... })
  151. For sending raw binary data simply specify the ``body`` argument. It's also
  152. recommended to set the ``Content-Type`` header::
  153. >>> with open('example.jpg', 'rb') as fp:
  154. ... binary_data = fp.read()
  155. >>> r = http.request(
  156. ... 'POST',
  157. ... 'http://httpbin.org/post',
  158. ... body=binary_data,
  159. ... headers={'Content-Type': 'image/jpeg'})
  160. >>> json.loads(r.data.decode('utf-8'))['data']
  161. b'...'
  162. .. _ssl:
  163. Certificate verification
  164. ------------------------
  165. .. note:: *New in version 1.25*
  166. HTTPS connections are now verified by default (``cert_reqs = 'CERT_REQUIRED'``).
  167. While you can disable certification verification, it is highly recommend to leave it on.
  168. Unless otherwise specified urllib3 will try to load the default system certificate stores.
  169. The most reliable cross-platform method is to use the `certifi <https://certifi.io/>`_
  170. package which provides Mozilla's root certificate bundle::
  171. pip install certifi
  172. You can also install certifi along with urllib3 by using the ``secure``
  173. extra::
  174. pip install urllib3[secure]
  175. .. warning:: If you're using Python 2 you may need additional packages. See the :ref:`section below <ssl_py2>` for more details.
  176. Once you have certificates, you can create a :class:`~poolmanager.PoolManager`
  177. that verifies certificates when making requests::
  178. >>> import certifi
  179. >>> import urllib3
  180. >>> http = urllib3.PoolManager(
  181. ... cert_reqs='CERT_REQUIRED',
  182. ... ca_certs=certifi.where())
  183. The :class:`~poolmanager.PoolManager` will automatically handle certificate
  184. verification and will raise :class:`~exceptions.SSLError` if verification fails::
  185. >>> http.request('GET', 'https://google.com')
  186. (No exception)
  187. >>> http.request('GET', 'https://expired.badssl.com')
  188. urllib3.exceptions.SSLError ...
  189. .. note:: You can use OS-provided certificates if desired. Just specify the full
  190. path to the certificate bundle as the ``ca_certs`` argument instead of
  191. ``certifi.where()``. For example, most Linux systems store the certificates
  192. at ``/etc/ssl/certs/ca-certificates.crt``. Other operating systems can
  193. be `difficult <https://stackoverflow.com/questions/10095676/openssl-reasonable-default-for-trusted-ca-certificates>`_.
  194. .. _ssl_py2:
  195. Certificate verification in Python 2
  196. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  197. Older versions of Python 2 are built with an :mod:`ssl` module that lacks
  198. :ref:`SNI support <sni_warning>` and can lag behind security updates. For these reasons it's recommended to use
  199. `pyOpenSSL <https://pyopenssl.readthedocs.io/en/latest/>`_.
  200. If you install urllib3 with the ``secure`` extra, all required packages for
  201. certificate verification on Python 2 will be installed::
  202. pip install urllib3[secure]
  203. If you want to install the packages manually, you will need ``pyOpenSSL``,
  204. ``cryptography``, ``idna``, and ``certifi``.
  205. .. note:: If you are not using macOS or Windows, note that `cryptography
  206. <https://cryptography.io/en/latest/>`_ requires additional system packages
  207. to compile. See `building cryptography on Linux
  208. <https://cryptography.io/en/latest/installation/#building-cryptography-on-linux>`_
  209. for the list of packages required.
  210. Once installed, you can tell urllib3 to use pyOpenSSL by using :mod:`urllib3.contrib.pyopenssl`::
  211. >>> import urllib3.contrib.pyopenssl
  212. >>> urllib3.contrib.pyopenssl.inject_into_urllib3()
  213. Finally, you can create a :class:`~poolmanager.PoolManager` that verifies
  214. certificates when performing requests::
  215. >>> import certifi
  216. >>> import urllib3
  217. >>> http = urllib3.PoolManager(
  218. ... cert_reqs='CERT_REQUIRED',
  219. ... ca_certs=certifi.where())
  220. If you do not wish to use pyOpenSSL, you can simply omit the call to
  221. :func:`urllib3.contrib.pyopenssl.inject_into_urllib3`. urllib3 will fall back
  222. to the standard-library :mod:`ssl` module. You may experience
  223. :ref:`several warnings <ssl_warnings>` when doing this.
  224. .. warning:: If you do not use pyOpenSSL, Python must be compiled with ssl
  225. support for certificate verification to work. It is uncommon, but it is
  226. possible to compile Python without SSL support. See this
  227. `Stackoverflow thread <https://stackoverflow.com/questions/5128845/importerror-no-module-named-ssl>`_
  228. for more details.
  229. If you are on Google App Engine, you must explicitly enable SSL
  230. support in your ``app.yaml``::
  231. libraries:
  232. - name: ssl
  233. version: latest
  234. Using timeouts
  235. --------------
  236. Timeouts allow you to control how long (in seconds) requests are allowed to run
  237. before being aborted. In simple cases, you can specify a timeout as a ``float``
  238. to :meth:`~poolmanager.PoolManager.request`::
  239. >>> http.request(
  240. ... 'GET', 'http://httpbin.org/delay/3', timeout=4.0)
  241. <urllib3.response.HTTPResponse>
  242. >>> http.request(
  243. ... 'GET', 'http://httpbin.org/delay/3', timeout=2.5)
  244. MaxRetryError caused by ReadTimeoutError
  245. For more granular control you can use a :class:`~util.timeout.Timeout`
  246. instance which lets you specify separate connect and read timeouts::
  247. >>> http.request(
  248. ... 'GET',
  249. ... 'http://httpbin.org/delay/3',
  250. ... timeout=urllib3.Timeout(connect=1.0))
  251. <urllib3.response.HTTPResponse>
  252. >>> http.request(
  253. ... 'GET',
  254. ... 'http://httpbin.org/delay/3',
  255. ... timeout=urllib3.Timeout(connect=1.0, read=2.0))
  256. MaxRetryError caused by ReadTimeoutError
  257. If you want all requests to be subject to the same timeout, you can specify
  258. the timeout at the :class:`~urllib3.poolmanager.PoolManager` level::
  259. >>> http = urllib3.PoolManager(timeout=3.0)
  260. >>> http = urllib3.PoolManager(
  261. ... timeout=urllib3.Timeout(connect=1.0, read=2.0))
  262. You still override this pool-level timeout by specifying ``timeout`` to
  263. :meth:`~poolmanager.PoolManager.request`.
  264. Retrying requests
  265. -----------------
  266. urllib3 can automatically retry idempotent requests. This same mechanism also
  267. handles redirects. You can control the retries using the ``retries`` parameter
  268. to :meth:`~poolmanager.PoolManager.request`. By default, urllib3 will retry
  269. requests 3 times and follow up to 3 redirects.
  270. To change the number of retries just specify an integer::
  271. >>> http.requests('GET', 'http://httpbin.org/ip', retries=10)
  272. To disable all retry and redirect logic specify ``retries=False``::
  273. >>> http.request(
  274. ... 'GET', 'http://nxdomain.example.com', retries=False)
  275. NewConnectionError
  276. >>> r = http.request(
  277. ... 'GET', 'http://httpbin.org/redirect/1', retries=False)
  278. >>> r.status
  279. 302
  280. To disable redirects but keep the retrying logic, specify ``redirect=False``::
  281. >>> r = http.request(
  282. ... 'GET', 'http://httpbin.org/redirect/1', redirect=False)
  283. >>> r.status
  284. 302
  285. For more granular control you can use a :class:`~util.retry.Retry` instance.
  286. This class allows you far greater control of how requests are retried.
  287. For example, to do a total of 3 retries, but limit to only 2 redirects::
  288. >>> http.request(
  289. ... 'GET',
  290. ... 'http://httpbin.org/redirect/3',
  291. ... retries=urllib3.Retry(3, redirect=2))
  292. MaxRetryError
  293. You can also disable exceptions for too many redirects and just return the
  294. ``302`` response::
  295. >>> r = http.request(
  296. ... 'GET',
  297. ... 'http://httpbin.org/redirect/3',
  298. ... retries=urllib3.Retry(
  299. ... redirect=2, raise_on_redirect=False))
  300. >>> r.status
  301. 302
  302. If you want all requests to be subject to the same retry policy, you can
  303. specify the retry at the :class:`~urllib3.poolmanager.PoolManager` level::
  304. >>> http = urllib3.PoolManager(retries=False)
  305. >>> http = urllib3.PoolManager(
  306. ... retries=urllib3.Retry(5, redirect=2))
  307. You still override this pool-level retry policy by specifying ``retries`` to
  308. :meth:`~poolmanager.PoolManager.request`.
  309. Errors & Exceptions
  310. -------------------
  311. urllib3 wraps lower-level exceptions, for example::
  312. >>> try:
  313. ... http.request('GET', 'nx.example.com', retries=False)
  314. >>> except urllib3.exceptions.NewConnectionError:
  315. ... print('Connection failed.')
  316. See :mod:`~urllib3.exceptions` for the full list of all exceptions.
  317. Logging
  318. -------
  319. If you are using the standard library :mod:`logging` module urllib3 will
  320. emit several logs. In some cases this can be undesirable. You can use the
  321. standard logger interface to change the log level for urllib3's logger::
  322. >>> logging.getLogger("urllib3").setLevel(logging.WARNING)