elementsoup.txt 6.5 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201
  1. ====================
  2. BeautifulSoup Parser
  3. ====================
  4. BeautifulSoup_ is a Python package that parses broken HTML, just like
  5. lxml supports it based on the parser of libxml2. BeautifulSoup uses a
  6. different parsing approach. It is not a real HTML parser but uses
  7. regular expressions to dive through tag soup. It is therefore more
  8. forgiving in some cases and less good in others. It is not uncommon
  9. that lxml/libxml2 parses and fixes broken HTML better, but
  10. BeautifulSoup has superiour `support for encoding detection`_. It
  11. very much depends on the input which parser works better.
  12. .. _BeautifulSoup: http://www.crummy.com/software/BeautifulSoup/
  13. .. _`support for encoding detection`: http://www.crummy.com/software/BeautifulSoup/documentation.html#Beautiful%20Soup%20Gives%20You%20Unicode%2C%20Dammit
  14. .. _ElementSoup: http://effbot.org/zone/element-soup.htm
  15. To prevent users from having to choose their parser library in
  16. advance, lxml can interface to the parsing capabilities of
  17. BeautifulSoup through the ``lxml.html.soupparser`` module. It
  18. provides three main functions: ``fromstring()`` and ``parse()`` to
  19. parse a string or file using BeautifulSoup into an ``lxml.html``
  20. document, and ``convert_tree()`` to convert an existing BeautifulSoup
  21. tree into a list of top-level Elements.
  22. .. contents::
  23. ..
  24. 1 Parsing with the soupparser
  25. 2 Entity handling
  26. 3 Using soupparser as a fallback
  27. 4 Using only the encoding detection
  28. Parsing with the soupparser
  29. ===========================
  30. The functions ``fromstring()`` and ``parse()`` behave as known from
  31. ElementTree. The first returns a root Element, the latter returns an
  32. ElementTree.
  33. There is also a legacy module called ``lxml.html.ElementSoup``, which
  34. mimics the interface provided by ElementTree's own ElementSoup_
  35. module. Note that the ``soupparser`` module was added in lxml 2.0.3.
  36. Previous versions of lxml 2.0.x only have the ``ElementSoup`` module.
  37. Here is a document full of tag soup, similar to, but not quite like, HTML:
  38. .. sourcecode:: pycon
  39. >>> tag_soup = '<meta><head><title>Hello</head><body onload=crash()>Hi all<p>'
  40. all you need to do is pass it to the ``fromstring()`` function:
  41. .. sourcecode:: pycon
  42. >>> from lxml.html.soupparser import fromstring
  43. >>> root = fromstring(tag_soup)
  44. To see what we have here, you can serialise it:
  45. .. sourcecode:: pycon
  46. >>> from lxml.etree import tostring
  47. >>> print(tostring(root, pretty_print=True).strip())
  48. <html>
  49. <meta/>
  50. <head>
  51. <title>Hello</title>
  52. </head>
  53. <body onload="crash()">Hi all<p/></body>
  54. </html>
  55. Not quite what you'd expect from an HTML page, but, well, it was broken
  56. already, right? BeautifulSoup did its best, and so now it's a tree.
  57. To control which Element implementation is used, you can pass a
  58. ``makeelement`` factory function to ``parse()`` and ``fromstring()``.
  59. By default, this is based on the HTML parser defined in ``lxml.html``.
  60. For a quick comparison, libxml2 2.6.32 parses the same tag soup as
  61. follows. The main difference is that libxml2 tries harder to adhere
  62. to the structure of an HTML document and moves misplaced tags where
  63. they (likely) belong. Note, however, that the result can vary between
  64. parser versions.
  65. .. sourcecode:: html
  66. <html>
  67. <head>
  68. <meta/>
  69. <title>Hello</title>
  70. </head>
  71. <body>
  72. <p>Hi all</p>
  73. <p/>
  74. </body>
  75. </html>
  76. Entity handling
  77. ===============
  78. By default, the BeautifulSoup parser also replaces the entities it
  79. finds by their character equivalent.
  80. .. sourcecode:: pycon
  81. >>> tag_soup = '<body>&copy;&euro;&#45;&#245;&#445;<p>'
  82. >>> body = fromstring(tag_soup).find('.//body')
  83. >>> body.text
  84. u'\xa9\u20ac-\xf5\u01bd'
  85. If you want them back on the way out, you can just serialise with the
  86. default encoding, which is 'US-ASCII'.
  87. .. sourcecode:: pycon
  88. >>> tostring(body)
  89. '<body>&#169;&#8364;-&#245;&#445;<p/></body>'
  90. >>> tostring(body, method="html")
  91. '<body>&#169;&#8364;-&#245;&#445;<p></p></body>'
  92. Any other encoding will output the respective byte sequences.
  93. .. sourcecode:: pycon
  94. >>> tostring(body, encoding="utf-8")
  95. '<body>\xc2\xa9\xe2\x82\xac-\xc3\xb5\xc6\xbd<p/></body>'
  96. >>> tostring(body, method="html", encoding="utf-8")
  97. '<body>\xc2\xa9\xe2\x82\xac-\xc3\xb5\xc6\xbd<p></p></body>'
  98. >>> tostring(body, encoding='unicode')
  99. u'<body>\xa9\u20ac-\xf5\u01bd<p/></body>'
  100. >>> tostring(body, method="html", encoding='unicode')
  101. u'<body>\xa9\u20ac-\xf5\u01bd<p></p></body>'
  102. Using soupparser as a fallback
  103. ==============================
  104. The downside of using this parser is that it is `much slower`_ than
  105. the HTML parser of lxml. So if performance matters, you might want to
  106. consider using ``soupparser`` only as a fallback for certain cases.
  107. .. _`much slower`: http://blog.ianbicking.org/2008/03/30/python-html-parser-performance/
  108. One common problem of lxml's parser is that it might not get the
  109. encoding right in cases where the document contains a ``<meta>`` tag
  110. at the wrong place. In this case, you can exploit the fact that lxml
  111. serialises much faster than most other HTML libraries for Python.
  112. Just serialise the document to unicode and if that gives you an
  113. exception, re-parse it with BeautifulSoup to see if that works
  114. better.
  115. .. sourcecode:: pycon
  116. >>> tag_soup = '''\
  117. ... <meta http-equiv="Content-Type"
  118. ... content="text/html;charset=utf-8" />
  119. ... <html>
  120. ... <head>
  121. ... <title>Hello W\xc3\xb6rld!</title>
  122. ... </head>
  123. ... <body>Hi all</body>
  124. ... </html>'''
  125. >>> import lxml.html
  126. >>> import lxml.html.soupparser
  127. >>> root = lxml.html.fromstring(tag_soup)
  128. >>> try:
  129. ... ignore = tostring(root, encoding='unicode')
  130. ... except UnicodeDecodeError:
  131. ... root = lxml.html.soupparser.fromstring(tag_soup)
  132. Using only the encoding detection
  133. =================================
  134. If you prefer a 'real' (and fast) HTML parser instead of the regular
  135. expression based one in BeautifulSoup, you can still benefit from
  136. BeautifulSoup's `support for encoding detection`_ in the
  137. ``UnicodeDammit`` class.
  138. .. sourcecode:: pycon
  139. >>> from BeautifulSoup import UnicodeDammit
  140. >>> def decode_html(html_string):
  141. ... converted = UnicodeDammit(html_string, isHTML=True)
  142. ... if not converted.unicode:
  143. ... raise UnicodeDecodeError(
  144. ... "Failed to detect encoding, tried [%s]",
  145. ... ', '.join(converted.triedEncodings))
  146. ... # print converted.originalEncoding
  147. ... return converted.unicode
  148. >>> root = lxml.html.fromstring(decode_html(tag_soup))