user-guide.rst 13 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417
  1. User Guide
  2. ==========
  3. .. currentmodule:: urllib3
  4. Making requests
  5. ---------------
  6. First things first, import the urllib3 module::
  7. >>> import urllib3
  8. You'll need a :class:`~poolmanager.PoolManager` instance to make requests.
  9. This object handles all of the details of connection pooling and thread safety
  10. so that you don't have to::
  11. >>> http = urllib3.PoolManager()
  12. To make a request use :meth:`~poolmanager.PoolManager.request`::
  13. >>> r = http.request('GET', 'http://httpbin.org/robots.txt')
  14. >>> r.data
  15. b'User-agent: *\nDisallow: /deny\n'
  16. ``request()`` returns a :class:`~response.HTTPResponse` object, the
  17. :ref:`response_content` section explains how to handle various responses.
  18. You can use :meth:`~poolmanager.PoolManager.request` to make requests using any
  19. HTTP verb::
  20. >>> r = http.request(
  21. ... 'POST',
  22. ... 'http://httpbin.org/post',
  23. ... fields={'hello': 'world'})
  24. The :ref:`request_data` section covers sending other kinds of requests data,
  25. including JSON, files, and binary data.
  26. .. _response_content:
  27. Response content
  28. ----------------
  29. The :class:`~response.HTTPResponse` object provides
  30. :attr:`~response.HTTPResponse.status`, :attr:`~response.HTTPResponse.data`, and
  31. :attr:`~response.HTTPResponse.header` attributes::
  32. >>> r = http.request('GET', 'http://httpbin.org/ip')
  33. >>> r.status
  34. 200
  35. >>> r.data
  36. b'{\n "origin": "104.232.115.37"\n}\n'
  37. >>> r.headers
  38. HTTPHeaderDict({'Content-Length': '33', ...})
  39. JSON content
  40. ~~~~~~~~~~~~
  41. JSON content can be loaded by decoding and deserializing the
  42. :attr:`~response.HTTPResponse.data` attribute of the request::
  43. >>> import json
  44. >>> r = http.request('GET', 'http://httpbin.org/ip')
  45. >>> json.loads(r.data.decode('utf-8'))
  46. {'origin': '127.0.0.1'}
  47. Binary content
  48. ~~~~~~~~~~~~~~
  49. The :attr:`~response.HTTPResponse.data` attribute of the response is always set
  50. to a byte string representing the response content::
  51. >>> r = http.request('GET', 'http://httpbin.org/bytes/8')
  52. >>> r.data
  53. b'\xaa\xa5H?\x95\xe9\x9b\x11'
  54. .. note:: For larger responses, it's sometimes better to :ref:`stream <stream>`
  55. the response.
  56. .. _request_data:
  57. Request data
  58. ------------
  59. Headers
  60. ~~~~~~~
  61. You can specify headers as a dictionary in the ``headers`` argument in :meth:`~poolmanager.PoolManager.request`::
  62. >>> r = http.request(
  63. ... 'GET',
  64. ... 'http://httpbin.org/headers',
  65. ... headers={
  66. ... 'X-Something': 'value'
  67. ... })
  68. >>> json.loads(r.data.decode('utf-8'))['headers']
  69. {'X-Something': 'value', ...}
  70. Query parameters
  71. ~~~~~~~~~~~~~~~~
  72. For ``GET``, ``HEAD``, and ``DELETE`` requests, you can simply pass the
  73. arguments as a dictionary in the ``fields`` argument to
  74. :meth:`~poolmanager.PoolManager.request`::
  75. >>> r = http.request(
  76. ... 'GET',
  77. ... 'http://httpbin.org/get',
  78. ... fields={'arg': 'value'})
  79. >>> json.loads(r.data.decode('utf-8'))['args']
  80. {'arg': 'value'}
  81. For ``POST`` and ``PUT`` requests, you need to manually encode query parameters
  82. in the URL::
  83. >>> from urllib.parse import urlencode
  84. >>> encoded_args = urlencode({'arg': 'value'})
  85. >>> url = 'http://httpbin.org/post?' + encoded_args
  86. >>> r = http.request('POST', url)
  87. >>> json.loads(r.data.decode('utf-8'))['args']
  88. {'arg': 'value'}
  89. .. _form_data:
  90. Form data
  91. ~~~~~~~~~
  92. For ``PUT`` and ``POST`` requests, urllib3 will automatically form-encode the
  93. dictionary in the ``fields`` argument provided to
  94. :meth:`~poolmanager.PoolManager.request`::
  95. >>> r = http.request(
  96. ... 'POST',
  97. ... 'http://httpbin.org/post',
  98. ... fields={'field': 'value'})
  99. >>> json.loads(r.data.decode('utf-8'))['form']
  100. {'field': 'value'}
  101. JSON
  102. ~~~~
  103. You can sent JSON a request by specifying the encoded data as the ``body``
  104. argument and setting the ``Content-Type`` header when calling
  105. :meth:`~poolmanager.PoolManager.request`::
  106. >>> import json
  107. >>> data = {'attribute': 'value'}
  108. >>> encoded_data = json.dumps(data).encode('utf-8')
  109. >>> r = http.request(
  110. ... 'POST',
  111. ... 'http://httpbin.org/post',
  112. ... body=encoded_data,
  113. ... headers={'Content-Type': 'application/json'})
  114. >>> json.loads(r.data.decode('utf-8'))['json']
  115. {'attribute': 'value'}
  116. Files & binary data
  117. ~~~~~~~~~~~~~~~~~~~
  118. For uploading files using ``multipart/form-data`` encoding you can use the same
  119. approach as :ref:`form_data` and specify the file field as a tuple of
  120. ``(file_name, file_data)``::
  121. >>> with open('example.txt') as fp:
  122. ... file_data = fp.read()
  123. >>> r = http.request(
  124. ... 'POST',
  125. ... 'http://httpbin.org/post',
  126. ... fields={
  127. ... 'filefield': ('example.txt', file_data),
  128. ... })
  129. >>> json.loads(r.data.decode('utf-8'))['files']
  130. {'filefield': '...'}
  131. While specifying the filename is not strictly required, it's recommended in
  132. order to match browser behavior. You can also pass a third item in the tuple
  133. to specify the file's MIME type explicitly::
  134. >>> r = http.request(
  135. ... 'POST',
  136. ... 'http://httpbin.org/post',
  137. ... fields={
  138. ... 'filefield': ('example.txt', file_data, 'text/plain'),
  139. ... })
  140. For sending raw binary data simply specify the ``body`` argument. It's also
  141. recommended to set the ``Content-Type`` header::
  142. >>> with open('example.jpg', 'rb') as fp:
  143. ... binary_data = fp.read()
  144. >>> r = http.request(
  145. ... 'POST',
  146. ... 'http://httpbin.org/post',
  147. ... body=binary_data,
  148. ... headers={'Content-Type': 'image/jpeg'})
  149. >>> json.loads(r.data.decode('utf-8'))['data']
  150. b'...'
  151. .. _ssl:
  152. Certificate verification
  153. ------------------------
  154. It is highly recommended to always use SSL certificate verification.
  155. **By default, urllib3 does not verify HTTPS requests**.
  156. In order to enable verification you will need a set of root certificates. The easiest
  157. and most reliable method is to use the `certifi <https://certifi.io/en/latest>`_ package which provides Mozilla's root certificate bundle::
  158. pip install certifi
  159. You can also install certifi along with urllib3 by using the ``secure``
  160. extra::
  161. pip install urllib3[secure]
  162. .. warning:: If you're using Python 2 you may need additional packages. See the :ref:`section below <ssl_py2>` for more details.
  163. Once you have certificates, you can create a :class:`~poolmanager.PoolManager`
  164. that verifies certificates when making requests::
  165. >>> import certifi
  166. >>> import urllib3
  167. >>> http = urllib3.PoolManager(
  168. ... cert_reqs='CERT_REQUIRED',
  169. ... ca_certs=certifi.where())
  170. The :class:`~poolmanager.PoolManager` will automatically handle certificate
  171. verification and will raise :class:`~exceptions.SSLError` if verification fails::
  172. >>> http.request('GET', 'https://google.com')
  173. (No exception)
  174. >>> http.request('GET', 'https://expired.badssl.com')
  175. urllib3.exceptions.SSLError ...
  176. .. note:: You can use OS-provided certificates if desired. Just specify the full
  177. path to the certificate bundle as the ``ca_certs`` argument instead of
  178. ``certifi.where()``. For example, most Linux systems store the certificates
  179. at ``/etc/ssl/certs/ca-certificates.crt``. Other operating systems can
  180. be `difficult <https://stackoverflow.com/questions/10095676/openssl-reasonable-default-for-trusted-ca-certificates>`_.
  181. .. _ssl_py2:
  182. Certificate verification in Python 2
  183. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  184. Older versions of Python 2 are built with an :mod:`ssl` module that lacks
  185. :ref:`SNI support <sni_warning>` and can lag behind security updates. For these reasons it's recommended to use
  186. `pyOpenSSL <https://pyopenssl.readthedocs.io/en/latest/>`_.
  187. If you install urllib3 with the ``secure`` extra, all required packages for
  188. certificate verification on Python 2 will be installed::
  189. pip install urllib3[secure]
  190. If you want to install the packages manually, you will need ``pyOpenSSL``,
  191. ``cryptography``, ``idna``, and ``certifi``.
  192. .. note:: If you are not using macOS or Windows, note that `cryptography
  193. <https://cryptography.io/en/latest/>`_ requires additional system packages
  194. to compile. See `building cryptography on Linux
  195. <https://cryptography.io/en/latest/installation/#building-cryptography-on-linux>`_
  196. for the list of packages required.
  197. Once installed, you can tell urllib3 to use pyOpenSSL by using :mod:`urllib3.contrib.pyopenssl`::
  198. >>> import urllib3.contrib.pyopenssl
  199. >>> urllib3.contrib.pyopenssl.inject_into_urllib3()
  200. Finally, you can create a :class:`~poolmanager.PoolManager` that verifies
  201. certificates when performing requests::
  202. >>> import certifi
  203. >>> import urllib3
  204. >>> http = urllib3.PoolManager(
  205. ... cert_reqs='CERT_REQUIRED',
  206. ... ca_certs=certifi.where())
  207. If you do not wish to use pyOpenSSL, you can simply omit the call to
  208. :func:`urllib3.contrib.pyopenssl.inject_into_urllib3`. urllib3 will fall back
  209. to the standard-library :mod:`ssl` module. You may experience
  210. :ref:`several warnings <ssl_warnings>` when doing this.
  211. .. warning:: If you do not use pyOpenSSL, Python must be compiled with ssl
  212. support for certificate verification to work. It is uncommon, but it is
  213. possible to compile Python without SSL support. See this
  214. `Stackoverflow thread <https://stackoverflow.com/questions/5128845/importerror-no-module-named-ssl>`_
  215. for more details.
  216. If you are on Google App Engine, you must explicitly enable SSL
  217. support in your ``app.yaml``::
  218. libraries:
  219. - name: ssl
  220. version: latest
  221. Using timeouts
  222. --------------
  223. Timeouts allow you to control how long requests are allowed to run before
  224. being aborted. In simple cases, you can specify a timeout as a ``float``
  225. to :meth:`~poolmanager.PoolManager.request`::
  226. >>> http.request(
  227. ... 'GET', 'http://httpbin.org/delay/3', timeout=4.0)
  228. <urllib3.response.HTTPResponse>
  229. >>> http.request(
  230. ... 'GET', 'http://httpbin.org/delay/3', timeout=2.5)
  231. MaxRetryError caused by ReadTimeoutError
  232. For more granular control you can use a :class:`~util.timeout.Timeout`
  233. instance which lets you specify separate connect and read timeouts::
  234. >>> http.request(
  235. ... 'GET',
  236. ... 'http://httpbin.org/delay/3',
  237. ... timeout=urllib3.Timeout(connect=1.0))
  238. <urllib3.response.HTTPResponse>
  239. >>> http.request(
  240. ... 'GET',
  241. ... 'http://httpbin.org/delay/3',
  242. ... timeout=urllib3.Timeout(connect=1.0, read=2.0))
  243. MaxRetryError caused by ReadTimeoutError
  244. If you want all requests to be subject to the same timeout, you can specify
  245. the timeout at the :class:`~urllib3.poolmanager.PoolManager` level::
  246. >>> http = urllib3.PoolManager(timeout=3.0)
  247. >>> http = urllib3.PoolManager(
  248. ... timeout=urllib3.Timeout(connect=1.0, read=2.0))
  249. You still override this pool-level timeout by specifying ``timeout`` to
  250. :meth:`~poolmanager.PoolManager.request`.
  251. Retrying requests
  252. -----------------
  253. urllib3 can automatically retry idempotent requests. This same mechanism also
  254. handles redirects. You can control the retries using the ``retries`` parameter
  255. to :meth:`~poolmanager.PoolManager.request`. By default, urllib3 will retry
  256. requests 3 times and follow up to 3 redirects.
  257. To change the number of retries just specify an integer::
  258. >>> http.requests('GET', 'http://httpbin.org/ip', retries=10)
  259. To disable all retry and redirect logic specify ``retries=False``::
  260. >>> http.request(
  261. ... 'GET', 'http://nxdomain.example.com', retries=False)
  262. NewConnectionError
  263. >>> r = http.request(
  264. ... 'GET', 'http://httpbin.org/redirect/1', retries=False)
  265. >>> r.status
  266. 302
  267. To disable redirects but keep the retrying logic, specify ``redirect=False``::
  268. >>> r = http.request(
  269. ... 'GET', 'http://httpbin.org/redirect/1', redirect=False)
  270. >>> r.status
  271. 302
  272. For more granular control you can use a :class:`~util.retry.Retry` instance.
  273. This class allows you far greater control of how requests are retried.
  274. For example, to do a total of 3 retries, but limit to only 2 redirects::
  275. >>> http.request(
  276. ... 'GET',
  277. ... 'http://httpbin.org/redirect/3',
  278. ... retries=urllib3.Retries(3, redirect=2))
  279. MaxRetryError
  280. You can also disable exceptions for too many redirects and just return the
  281. ``302`` response::
  282. >>> r = http.request(
  283. ... 'GET',
  284. ... 'http://httpbin.org/redirect/3',
  285. ... retries=urllib3.Retries(
  286. ... redirect=2, raise_on_redirect=False))
  287. >>> r.status
  288. 302
  289. If you want all requests to be subject to the same retry policy, you can
  290. specify the retry at the :class:`~urllib3.poolmanager.PoolManager` level::
  291. >>> http = urllib3.PoolManager(retries=False)
  292. >>> http = urllib3.PoolManager(
  293. ... retries=urllib3.Retry(5, redirect=2))
  294. You still override this pool-level retry policy by specifying ``retries`` to
  295. :meth:`~poolmanager.PoolManager.request`.
  296. Errors & Exceptions
  297. -------------------
  298. urllib3 wraps lower-level exceptions, for example::
  299. >>> try:
  300. ... http.request('GET', 'nx.example.com', retries=False)
  301. >>> except urllib3.exceptions.NewConnectionError:
  302. ... print('Connection failed.')
  303. See :mod:`~urllib3.exceptions` for the full list of all exceptions.
  304. Logging
  305. -------
  306. If you are using the standard library :mod:`logging` module urllib3 will
  307. emit several logs. In some cases this can be undesirable. You can use the
  308. standard logger interface to change the log level for urllib3's logger::
  309. >>> logging.getLogger("urllib3").setLevel(logging.WARNING)