user-guide.rst 15 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536
  1. User Guide
  2. ==========
  3. .. currentmodule:: urllib3
  4. Installing
  5. ----------
  6. urllib3 can be installed with `pip <https://pip.pypa.io>`_
  7. .. code-block:: bash
  8. $ python -m pip install urllib3
  9. Making Requests
  10. ---------------
  11. First things first, import the urllib3 module:
  12. .. code-block:: pycon
  13. >>> import urllib3
  14. You'll need a :class:`~poolmanager.PoolManager` instance to make requests.
  15. This object handles all of the details of connection pooling and thread safety
  16. so that you don't have to:
  17. .. code-block:: pycon
  18. >>> http = urllib3.PoolManager()
  19. To make a request use :meth:`~poolmanager.PoolManager.request`:
  20. .. code-block:: pycon
  21. >>> r = http.request('GET', 'http://httpbin.org/robots.txt')
  22. >>> r.data
  23. b'User-agent: *\nDisallow: /deny\n'
  24. ``request()`` returns a :class:`~response.HTTPResponse` object, the
  25. :ref:`response_content` section explains how to handle various responses.
  26. You can use :meth:`~poolmanager.PoolManager.request` to make requests using any
  27. HTTP verb:
  28. .. code-block:: pycon
  29. >>> r = http.request(
  30. ... 'POST',
  31. ... 'http://httpbin.org/post',
  32. ... fields={'hello': 'world'}
  33. ... )
  34. The :ref:`request_data` section covers sending other kinds of requests data,
  35. including JSON, files, and binary data.
  36. .. _response_content:
  37. Response Content
  38. ----------------
  39. The :class:`~response.HTTPResponse` object provides
  40. :attr:`~response.HTTPResponse.status`, :attr:`~response.HTTPResponse.data`, and
  41. :attr:`~response.HTTPResponse.headers` attributes:
  42. .. code-block:: pycon
  43. >>> r = http.request('GET', 'http://httpbin.org/ip')
  44. >>> r.status
  45. 200
  46. >>> r.data
  47. b'{\n "origin": "104.232.115.37"\n}\n'
  48. >>> r.headers
  49. HTTPHeaderDict({'Content-Length': '33', ...})
  50. JSON Content
  51. ~~~~~~~~~~~~
  52. JSON content can be loaded by decoding and deserializing the
  53. :attr:`~response.HTTPResponse.data` attribute of the request:
  54. .. code-block:: pycon
  55. >>> import json
  56. >>> r = http.request('GET', 'http://httpbin.org/ip')
  57. >>> json.loads(r.data.decode('utf-8'))
  58. {'origin': '127.0.0.1'}
  59. Binary Content
  60. ~~~~~~~~~~~~~~
  61. The :attr:`~response.HTTPResponse.data` attribute of the response is always set
  62. to a byte string representing the response content:
  63. .. code-block:: pycon
  64. >>> r = http.request('GET', 'http://httpbin.org/bytes/8')
  65. >>> r.data
  66. b'\xaa\xa5H?\x95\xe9\x9b\x11'
  67. .. note:: For larger responses, it's sometimes better to :ref:`stream <stream>`
  68. the response.
  69. Using io Wrappers with Response Content
  70. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  71. Sometimes you want to use :class:`io.TextIOWrapper` or similar objects like a CSV reader
  72. directly with :class:`~response.HTTPResponse` data. Making these two interfaces play nice
  73. together requires using the :attr:`~response.HTTPResponse.auto_close` attribute by setting it
  74. to ``False``. By default HTTP responses are closed after reading all bytes, this disables that behavior:
  75. .. code-block:: pycon
  76. >>> import io
  77. >>> r = http.request('GET', 'https://example.com', preload_content=False)
  78. >>> r.auto_close = False
  79. >>> for line in io.TextIOWrapper(r):
  80. >>> print(line)
  81. .. _request_data:
  82. Request Data
  83. ------------
  84. Headers
  85. ~~~~~~~
  86. You can specify headers as a dictionary in the ``headers`` argument in :meth:`~poolmanager.PoolManager.request`:
  87. .. code-block:: pycon
  88. >>> r = http.request(
  89. ... 'GET',
  90. ... 'http://httpbin.org/headers',
  91. ... headers={
  92. ... 'X-Something': 'value'
  93. ... }
  94. ... )
  95. >>> json.loads(r.data.decode('utf-8'))['headers']
  96. {'X-Something': 'value', ...}
  97. Query Parameters
  98. ~~~~~~~~~~~~~~~~
  99. For ``GET``, ``HEAD``, and ``DELETE`` requests, you can simply pass the
  100. arguments as a dictionary in the ``fields`` argument to
  101. :meth:`~poolmanager.PoolManager.request`:
  102. .. code-block:: pycon
  103. >>> r = http.request(
  104. ... 'GET',
  105. ... 'http://httpbin.org/get',
  106. ... fields={'arg': 'value'}
  107. ... )
  108. >>> json.loads(r.data.decode('utf-8'))['args']
  109. {'arg': 'value'}
  110. For ``POST`` and ``PUT`` requests, you need to manually encode query parameters
  111. in the URL:
  112. .. code-block:: pycon
  113. >>> from urllib.parse import urlencode
  114. >>> encoded_args = urlencode({'arg': 'value'})
  115. >>> url = 'http://httpbin.org/post?' + encoded_args
  116. >>> r = http.request('POST', url)
  117. >>> json.loads(r.data.decode('utf-8'))['args']
  118. {'arg': 'value'}
  119. .. _form_data:
  120. Form Data
  121. ~~~~~~~~~
  122. For ``PUT`` and ``POST`` requests, urllib3 will automatically form-encode the
  123. dictionary in the ``fields`` argument provided to
  124. :meth:`~poolmanager.PoolManager.request`:
  125. .. code-block:: pycon
  126. >>> r = http.request(
  127. ... 'POST',
  128. ... 'http://httpbin.org/post',
  129. ... fields={'field': 'value'}
  130. ... )
  131. >>> json.loads(r.data.decode('utf-8'))['form']
  132. {'field': 'value'}
  133. JSON
  134. ~~~~
  135. You can send a JSON request by specifying the encoded data as the ``body``
  136. argument and setting the ``Content-Type`` header when calling
  137. :meth:`~poolmanager.PoolManager.request`:
  138. .. code-block:: pycon
  139. >>> import json
  140. >>> data = {'attribute': 'value'}
  141. >>> encoded_data = json.dumps(data).encode('utf-8')
  142. >>> r = http.request(
  143. ... 'POST',
  144. ... 'http://httpbin.org/post',
  145. ... body=encoded_data,
  146. ... headers={'Content-Type': 'application/json'}
  147. ... )
  148. >>> json.loads(r.data.decode('utf-8'))['json']
  149. {'attribute': 'value'}
  150. Files & Binary Data
  151. ~~~~~~~~~~~~~~~~~~~
  152. For uploading files using ``multipart/form-data`` encoding you can use the same
  153. approach as :ref:`form_data` and specify the file field as a tuple of
  154. ``(file_name, file_data)``:
  155. .. code-block:: pycon
  156. >>> with open('example.txt') as fp:
  157. ... file_data = fp.read()
  158. >>> r = http.request(
  159. ... 'POST',
  160. ... 'http://httpbin.org/post',
  161. ... fields={
  162. ... 'filefield': ('example.txt', file_data),
  163. ... }
  164. ... )
  165. >>> json.loads(r.data.decode('utf-8'))['files']
  166. {'filefield': '...'}
  167. While specifying the filename is not strictly required, it's recommended in
  168. order to match browser behavior. You can also pass a third item in the tuple
  169. to specify the file's MIME type explicitly:
  170. .. code-block:: pycon
  171. >>> r = http.request(
  172. ... 'POST',
  173. ... 'http://httpbin.org/post',
  174. ... fields={
  175. ... 'filefield': ('example.txt', file_data, 'text/plain'),
  176. ... }
  177. ... )
  178. For sending raw binary data simply specify the ``body`` argument. It's also
  179. recommended to set the ``Content-Type`` header:
  180. .. code-block:: pycon
  181. >>> with open('example.jpg', 'rb') as fp:
  182. ... binary_data = fp.read()
  183. >>> r = http.request(
  184. ... 'POST',
  185. ... 'http://httpbin.org/post',
  186. ... body=binary_data,
  187. ... headers={'Content-Type': 'image/jpeg'}
  188. ... )
  189. >>> json.loads(r.data.decode('utf-8'))['data']
  190. b'...'
  191. .. _ssl:
  192. Certificate Verification
  193. ------------------------
  194. .. note:: *New in version 1.25:*
  195. HTTPS connections are now verified by default (``cert_reqs = 'CERT_REQUIRED'``).
  196. While you can disable certification verification by setting ``cert_reqs = 'CERT_NONE'``, it is highly recommend to leave it on.
  197. Unless otherwise specified urllib3 will try to load the default system certificate stores.
  198. The most reliable cross-platform method is to use the `certifi <https://certifi.io/>`_
  199. package which provides Mozilla's root certificate bundle:
  200. .. code-block:: bash
  201. $ python -m pip install certifi
  202. You can also install certifi along with urllib3 by using the ``secure``
  203. extra:
  204. .. code-block:: bash
  205. $ python -m pip install urllib3[secure]
  206. .. warning:: If you're using Python 2 you may need additional packages. See the :ref:`section below <ssl_py2>` for more details.
  207. Once you have certificates, you can create a :class:`~poolmanager.PoolManager`
  208. that verifies certificates when making requests:
  209. .. code-block:: pycon
  210. >>> import certifi
  211. >>> import urllib3
  212. >>> http = urllib3.PoolManager(
  213. ... cert_reqs='CERT_REQUIRED',
  214. ... ca_certs=certifi.where()
  215. ... )
  216. The :class:`~poolmanager.PoolManager` will automatically handle certificate
  217. verification and will raise :class:`~exceptions.SSLError` if verification fails:
  218. .. code-block:: pycon
  219. >>> http.request('GET', 'https://google.com')
  220. (No exception)
  221. >>> http.request('GET', 'https://expired.badssl.com')
  222. urllib3.exceptions.SSLError ...
  223. .. note:: You can use OS-provided certificates if desired. Just specify the full
  224. path to the certificate bundle as the ``ca_certs`` argument instead of
  225. ``certifi.where()``. For example, most Linux systems store the certificates
  226. at ``/etc/ssl/certs/ca-certificates.crt``. Other operating systems can
  227. be `difficult <https://stackoverflow.com/questions/10095676/openssl-reasonable-default-for-trusted-ca-certificates>`_.
  228. .. _ssl_py2:
  229. Certificate Verification in Python 2
  230. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  231. Older versions of Python 2 are built with an :mod:`ssl` module that lacks
  232. :ref:`SNI support <sni_warning>` and can lag behind security updates. For these reasons it's recommended to use
  233. `pyOpenSSL <https://pyopenssl.readthedocs.io/en/latest/>`_.
  234. If you install urllib3 with the ``secure`` extra, all required packages for
  235. certificate verification on Python 2 will be installed:
  236. .. code-block:: bash
  237. $ python -m pip install urllib3[secure]
  238. If you want to install the packages manually, you will need ``pyOpenSSL``,
  239. ``cryptography``, ``idna``, and ``certifi``.
  240. .. note:: If you are not using macOS or Windows, note that `cryptography
  241. <https://cryptography.io/en/latest/>`_ requires additional system packages
  242. to compile. See `building cryptography on Linux
  243. <https://cryptography.io/en/latest/installation/#building-cryptography-on-linux>`_
  244. for the list of packages required.
  245. Once installed, you can tell urllib3 to use pyOpenSSL by using :mod:`urllib3.contrib.pyopenssl`:
  246. .. code-block:: pycon
  247. >>> import urllib3.contrib.pyopenssl
  248. >>> urllib3.contrib.pyopenssl.inject_into_urllib3()
  249. Finally, you can create a :class:`~poolmanager.PoolManager` that verifies
  250. certificates when performing requests:
  251. .. code-block:: pycon
  252. >>> import certifi
  253. >>> import urllib3
  254. >>> http = urllib3.PoolManager(
  255. ... cert_reqs='CERT_REQUIRED',
  256. ... ca_certs=certifi.where()
  257. ... )
  258. If you do not wish to use pyOpenSSL, you can simply omit the call to
  259. :func:`urllib3.contrib.pyopenssl.inject_into_urllib3`. urllib3 will fall back
  260. to the standard-library :mod:`ssl` module. You may experience
  261. :ref:`several warnings <ssl_warnings>` when doing this.
  262. .. warning:: If you do not use pyOpenSSL, Python must be compiled with ssl
  263. support for certificate verification to work. It is uncommon, but it is
  264. possible to compile Python without SSL support. See this
  265. `StackOverflow thread <https://stackoverflow.com/questions/5128845/importerror-no-module-named-ssl>`_
  266. for more details.
  267. If you are on Google App Engine, you must explicitly enable SSL
  268. support in your ``app.yaml``:
  269. .. code-block:: yaml
  270. libraries:
  271. - name: ssl
  272. version: latest
  273. Using Timeouts
  274. --------------
  275. Timeouts allow you to control how long (in seconds) requests are allowed to run
  276. before being aborted. In simple cases, you can specify a timeout as a ``float``
  277. to :meth:`~poolmanager.PoolManager.request`:
  278. .. code-block:: pycon
  279. >>> http.request(
  280. ... 'GET', 'http://httpbin.org/delay/3', timeout=4.0
  281. ... )
  282. <urllib3.response.HTTPResponse>
  283. >>> http.request(
  284. ... 'GET', 'http://httpbin.org/delay/3', timeout=2.5
  285. ... )
  286. MaxRetryError caused by ReadTimeoutError
  287. For more granular control you can use a :class:`~util.timeout.Timeout`
  288. instance which lets you specify separate connect and read timeouts:
  289. .. code-block:: pycon
  290. >>> http.request(
  291. ... 'GET',
  292. ... 'http://httpbin.org/delay/3',
  293. ... timeout=urllib3.Timeout(connect=1.0)
  294. ... )
  295. <urllib3.response.HTTPResponse>
  296. >>> http.request(
  297. ... 'GET',
  298. ... 'http://httpbin.org/delay/3',
  299. ... timeout=urllib3.Timeout(connect=1.0, read=2.0)
  300. ... )
  301. MaxRetryError caused by ReadTimeoutError
  302. If you want all requests to be subject to the same timeout, you can specify
  303. the timeout at the :class:`~urllib3.poolmanager.PoolManager` level:
  304. .. code-block:: pycon
  305. >>> http = urllib3.PoolManager(timeout=3.0)
  306. >>> http = urllib3.PoolManager(
  307. ... timeout=urllib3.Timeout(connect=1.0, read=2.0)
  308. ... )
  309. You still override this pool-level timeout by specifying ``timeout`` to
  310. :meth:`~poolmanager.PoolManager.request`.
  311. Retrying Requests
  312. -----------------
  313. urllib3 can automatically retry idempotent requests. This same mechanism also
  314. handles redirects. You can control the retries using the ``retries`` parameter
  315. to :meth:`~poolmanager.PoolManager.request`. By default, urllib3 will retry
  316. requests 3 times and follow up to 3 redirects.
  317. To change the number of retries just specify an integer:
  318. .. code-block:: pycon
  319. >>> http.requests('GET', 'http://httpbin.org/ip', retries=10)
  320. To disable all retry and redirect logic specify ``retries=False``:
  321. .. code-block:: pycon
  322. >>> http.request(
  323. ... 'GET', 'http://nxdomain.example.com', retries=False
  324. ... )
  325. NewConnectionError
  326. >>> r = http.request(
  327. ... 'GET', 'http://httpbin.org/redirect/1', retries=False
  328. ... )
  329. >>> r.status
  330. 302
  331. To disable redirects but keep the retrying logic, specify ``redirect=False``:
  332. .. code-block:: pycon
  333. >>> r = http.request(
  334. ... 'GET', 'http://httpbin.org/redirect/1', redirect=False
  335. ... )
  336. >>> r.status
  337. 302
  338. For more granular control you can use a :class:`~util.retry.Retry` instance.
  339. This class allows you far greater control of how requests are retried.
  340. For example, to do a total of 3 retries, but limit to only 2 redirects:
  341. .. code-block:: pycon
  342. >>> http.request(
  343. ... 'GET',
  344. ... 'http://httpbin.org/redirect/3',
  345. ... retries=urllib3.Retry(3, redirect=2)
  346. ... )
  347. MaxRetryError
  348. You can also disable exceptions for too many redirects and just return the
  349. ``302`` response:
  350. .. code-block:: pycon
  351. >>> r = http.request(
  352. ... 'GET',
  353. ... 'http://httpbin.org/redirect/3',
  354. ... retries=urllib3.Retry(
  355. ... redirect=2, raise_on_redirect=False)
  356. ... )
  357. >>> r.status
  358. 302
  359. If you want all requests to be subject to the same retry policy, you can
  360. specify the retry at the :class:`~urllib3.poolmanager.PoolManager` level:
  361. .. code-block:: pycon
  362. >>> http = urllib3.PoolManager(retries=False)
  363. >>> http = urllib3.PoolManager(
  364. ... retries=urllib3.Retry(5, redirect=2)
  365. ... )
  366. You still override this pool-level retry policy by specifying ``retries`` to
  367. :meth:`~poolmanager.PoolManager.request`.
  368. Errors & Exceptions
  369. -------------------
  370. urllib3 wraps lower-level exceptions, for example:
  371. .. code-block:: pycon
  372. >>> try:
  373. ... http.request('GET', 'nx.example.com', retries=False)
  374. ... except urllib3.exceptions.NewConnectionError:
  375. ... print('Connection failed.')
  376. See :mod:`~urllib3.exceptions` for the full list of all exceptions.
  377. Logging
  378. -------
  379. If you are using the standard library :mod:`logging` module urllib3 will
  380. emit several logs. In some cases this can be undesirable. You can use the
  381. standard logger interface to change the log level for urllib3's logger:
  382. .. code-block:: pycon
  383. >>> logging.getLogger("urllib3").setLevel(logging.WARNING)