elementsoup.txt 7.5 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222
  1. ====================
  2. BeautifulSoup Parser
  3. ====================
  4. BeautifulSoup_ is a Python package for working with real-world and broken HTML,
  5. just like `lxml.html <lxmlhtml.html>`_. As of version 4.x, it can use
  6. `different HTML parsers
  7. <http://www.crummy.com/software/BeautifulSoup/bs4/doc/#installing-a-parser>`_,
  8. each of which has its advantages and disadvantages (see the link).
  9. lxml can make use of BeautifulSoup as a parser backend, just like BeautifulSoup
  10. can employ lxml as a parser. When using BeautifulSoup from lxml, however, the
  11. default is to use Python's integrated HTML parser in the
  12. `html.parser <https://docs.python.org/3/library/html.parser.html>`_ module.
  13. In order to make use of the HTML5 parser of
  14. `html5lib <https://pypi.python.org/pypi/html5lib>`_ instead, it is better
  15. to go directly through the `html5parser module <html5parser.html>`_ in
  16. ``lxml.html``.
  17. A very nice feature of BeautifulSoup is its excellent `support for encoding
  18. detection`_ which can provide better results for real-world HTML pages that
  19. do not (correctly) declare their encoding.
  20. .. _BeautifulSoup: http://www.crummy.com/software/BeautifulSoup/
  21. .. _`support for encoding detection`: http://www.crummy.com/software/BeautifulSoup/bs4/doc/#unicode-dammit
  22. .. _ElementSoup: http://effbot.org/zone/element-soup.htm
  23. lxml interfaces with BeautifulSoup through the ``lxml.html.soupparser``
  24. module. It provides three main functions: ``fromstring()`` and ``parse()``
  25. to parse a string or file using BeautifulSoup into an ``lxml.html``
  26. document, and ``convert_tree()`` to convert an existing BeautifulSoup
  27. tree into a list of top-level Elements.
  28. .. contents::
  29. ..
  30. 1 Parsing with the soupparser
  31. 2 Entity handling
  32. 3 Using soupparser as a fallback
  33. 4 Using only the encoding detection
  34. Parsing with the soupparser
  35. ===========================
  36. The functions ``fromstring()`` and ``parse()`` behave as known from
  37. lxml. The first returns a root Element, the latter returns an
  38. ElementTree.
  39. There is also a legacy module called ``lxml.html.ElementSoup``, which
  40. mimics the interface provided by Fredrik Lundh's ElementSoup_
  41. module. Note that the ``soupparser`` module was added in lxml 2.0.3.
  42. Previous versions of lxml 2.0.x only have the ``ElementSoup`` module.
  43. Here is a document full of tag soup, similar to, but not quite like, HTML:
  44. .. sourcecode:: pycon
  45. >>> tag_soup = '''
  46. ... <meta/><head><title>Hello</head><body onload=crash()>Hi all<p>'''
  47. All you need to do is pass it to the ``fromstring()`` function:
  48. .. sourcecode:: pycon
  49. >>> from lxml.html.soupparser import fromstring
  50. >>> root = fromstring(tag_soup)
  51. To see what we have here, you can serialise it:
  52. .. sourcecode:: pycon
  53. >>> from lxml.etree import tostring
  54. >>> print(tostring(root, pretty_print=True).strip())
  55. <html>
  56. <meta/>
  57. <head>
  58. <title>Hello</title>
  59. </head>
  60. <body onload="crash()">Hi all<p/></body>
  61. </html>
  62. Not quite what you'd expect from an HTML page, but, well, it was broken
  63. already, right? The parser did its best, and so now it's a tree.
  64. To control how Element objects are created during the conversion
  65. of the tree, you can pass a ``makeelement`` factory function to
  66. ``parse()`` and ``fromstring()``. By default, this is based on the
  67. HTML parser defined in ``lxml.html``.
  68. For a quick comparison, libxml2 2.9.1 parses the same tag soup as
  69. follows. The only difference is that libxml2 tries harder to adhere
  70. to the structure of an HTML document and moves misplaced tags where
  71. they (likely) belong. Note, however, that the result can vary between
  72. parser versions.
  73. .. sourcecode:: html
  74. <html>
  75. <head>
  76. <meta/>
  77. <title>Hello</title>
  78. </head>
  79. <body onload="crash()">Hi all<p/></body>
  80. </html>
  81. Entity handling
  82. ===============
  83. By default, the BeautifulSoup parser also replaces the entities it
  84. finds by their character equivalent.
  85. .. sourcecode:: pycon
  86. >>> tag_soup = '<body>&copy;&euro;&#45;&#245;&#445;<p>'
  87. >>> body = fromstring(tag_soup).find('.//body')
  88. >>> body.text
  89. u'\xa9\u20ac-\xf5\u01bd'
  90. If you want them back on the way out, you can just serialise with the
  91. default encoding, which is 'US-ASCII'.
  92. .. sourcecode:: pycon
  93. >>> tostring(body)
  94. '<body>&#169;&#8364;-&#245;&#445;<p/></body>'
  95. >>> tostring(body, method="html")
  96. '<body>&#169;&#8364;-&#245;&#445;<p></p></body>'
  97. Any other encoding will output the respective byte sequences.
  98. .. sourcecode:: pycon
  99. >>> tostring(body, encoding="utf-8")
  100. '<body>\xc2\xa9\xe2\x82\xac-\xc3\xb5\xc6\xbd<p/></body>'
  101. >>> tostring(body, method="html", encoding="utf-8")
  102. '<body>\xc2\xa9\xe2\x82\xac-\xc3\xb5\xc6\xbd<p></p></body>'
  103. >>> tostring(body, encoding='unicode')
  104. u'<body>\xa9\u20ac-\xf5\u01bd<p/></body>'
  105. >>> tostring(body, method="html", encoding='unicode')
  106. u'<body>\xa9\u20ac-\xf5\u01bd<p></p></body>'
  107. Using soupparser as a fallback
  108. ==============================
  109. The downside of using this parser is that it is `much slower`_ than
  110. the C implemented HTML parser of libxml2 that lxml uses. So if
  111. performance matters, you might want to consider using ``soupparser``
  112. only as a fallback for certain cases.
  113. .. _`much slower`: http://blog.ianbicking.org/2008/03/30/python-html-parser-performance/
  114. One common problem of lxml's parser is that it might not get the
  115. encoding right in cases where the document contains a ``<meta>`` tag
  116. at the wrong place. In this case, you can exploit the fact that lxml
  117. serialises much faster than most other HTML libraries for Python.
  118. Just serialise the document to unicode and if that gives you an
  119. exception, re-parse it with BeautifulSoup to see if that works
  120. better.
  121. .. sourcecode:: pycon
  122. >>> tag_soup = '''\
  123. ... <meta http-equiv="Content-Type"
  124. ... content="text/html;charset=utf-8" />
  125. ... <html>
  126. ... <head>
  127. ... <title>Hello W\xc3\xb6rld!</title>
  128. ... </head>
  129. ... <body>Hi all</body>
  130. ... </html>'''
  131. >>> import lxml.html
  132. >>> import lxml.html.soupparser
  133. >>> root = lxml.html.fromstring(tag_soup)
  134. >>> try:
  135. ... ignore = tostring(root, encoding='unicode')
  136. ... except UnicodeDecodeError:
  137. ... root = lxml.html.soupparser.fromstring(tag_soup)
  138. Using only the encoding detection
  139. =================================
  140. Even if you prefer lxml's fast HTML parser, you can still benefit
  141. from BeautifulSoup's `support for encoding detection`_ in the
  142. ``UnicodeDammit`` class. Once it succeeds in decoding the data,
  143. you can simply pass the resulting Unicode string into lxml's parser.
  144. .. sourcecode:: pycon
  145. >>> try:
  146. ... from bs4 import UnicodeDammit # BeautifulSoup 4
  147. ...
  148. ... def decode_html(html_string):
  149. ... converted = UnicodeDammit(html_string)
  150. ... if not converted.unicode_markup:
  151. ... raise UnicodeDecodeError(
  152. ... "Failed to detect encoding, tried [%s]",
  153. ... ', '.join(converted.tried_encodings))
  154. ... # print converted.original_encoding
  155. ... return converted.unicode_markup
  156. ...
  157. ... except ImportError:
  158. ... from BeautifulSoup import UnicodeDammit # BeautifulSoup 3
  159. ...
  160. ... def decode_html(html_string):
  161. ... converted = UnicodeDammit(html_string, isHTML=True)
  162. ... if not converted.unicode:
  163. ... raise UnicodeDecodeError(
  164. ... "Failed to detect encoding, tried [%s]",
  165. ... ', '.join(converted.triedEncodings))
  166. ... # print converted.originalEncoding
  167. ... return converted.unicode
  168. >>> root = lxml.html.fromstring(decode_html(tag_soup))