| 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151 |
- ====================
- BeautifulSoup Parser
- ====================
- BeautifulSoup_ is a Python package that parses broken HTML, just like
- lxml supports it based on the parser of libxml2. BeautifulSoup uses a
- different parsing approach. It is not a real HTML parser but uses
- regular expressions to dive through tag soup. It is therefore more
- forgiving in some cases and less good in others. It is not uncommon
- that lxml/libxml2 parses and fixes broken HTML better, but
- BeautifulSoup has superiour `support for encoding detection`_. It
- very much depends on the input which parser works better.
- .. _BeautifulSoup: http://www.crummy.com/software/BeautifulSoup/
- .. _`support for encoding detection`: http://www.crummy.com/software/BeautifulSoup/documentation.html#Beautiful%20Soup%20Gives%20You%20Unicode%2C%20Dammit
- .. _ElementSoup: http://effbot.org/zone/element-soup.htm
- To prevent users from having to choose their parser library in
- advance, lxml can interface to the parsing capabilities of
- BeautifulSoup through the ``lxml.html.soupparser`` module. It
- provides three main functions: ``fromstring()`` and ``parse()`` to
- parse a string or file using BeautifulSoup into an ``lxml.html``
- document, and ``convert_tree()`` to convert an existing BeautifulSoup
- tree into a list of top-level Elements.
- Parsing with the soupparser
- ===========================
- The functions ``fromstring()`` and ``parse()`` behave as known from
- ElementTree. The first returns a root Element, the latter returns an
- ElementTree.
- There is also a legacy module called ``lxml.html.ElementSoup``, which
- mimics the interface provided by ElementTree's own ElementSoup_
- module. Note that the ``soupparser`` module was added in lxml 2.0.3.
- Previous versions of lxml 2.0.x only have the ``ElementSoup`` module.
- Here is a document full of tag soup, similar to, but not quite like, HTML:
- .. sourcecode:: pycon
- >>> tag_soup = '<meta><head><title>Hello</head<body onload=crash()>Hi all<p>'
- all you need to do is pass it to the ``fromstring()`` function:
- .. sourcecode:: pycon
- >>> from lxml.html.soupparser import fromstring
- >>> root = fromstring(tag_soup)
- To see what we have here, you can serialise it:
- .. sourcecode:: pycon
- >>> from lxml.etree import tostring
- >>> print tostring(root, pretty_print=True),
- <html>
- <meta/>
- <head>
- <title>Hello</title>
- </head>
- <body onload="crash()">Hi all<p/></body>
- </html>
- Not quite what you'd expect from an HTML page, but, well, it was broken
- already, right? BeautifulSoup did its best, and so now it's a tree.
- To control which Element implementation is used, you can pass a
- ``makeelement`` factory function to ``parse()`` and ``fromstring()``.
- By default, this is based on the HTML parser defined in ``lxml.html``.
- Entity handling
- ===============
- By default, the BeautifulSoup parser also replaces the entities it
- finds by their character equivalent.
- .. sourcecode:: pycon
- >>> tag_soup = '<body>©€-õƽ<p>'
- >>> body = fromstring(tag_soup).find('.//body')
- >>> body.text
- u'\xa9\u20ac-\xf5\u01bd'
- If you want them back on the way out, you can just serialise with the
- default encoding, which is 'US-ASCII'.
- .. sourcecode:: pycon
- >>> tostring(body)
- '<body>©€-õƽ<p/></body>'
- >>> tostring(body, method="html")
- '<body>©€-õƽ<p></p></body>'
- Any other encoding will output the respective byte sequences.
- .. sourcecode:: pycon
- >>> tostring(body, encoding="utf-8")
- '<body>\xc2\xa9\xe2\x82\xac-\xc3\xb5\xc6\xbd<p/></body>'
- >>> tostring(body, method="html", encoding="utf-8")
- '<body>\xc2\xa9\xe2\x82\xac-\xc3\xb5\xc6\xbd<p></p></body>'
- >>> tostring(body, encoding=unicode)
- u'<body>\xa9\u20ac-\xf5\u01bd<p/></body>'
- >>> tostring(body, method="html", encoding=unicode)
- u'<body>\xa9\u20ac-\xf5\u01bd<p></p></body>'
- Using soupparser as a fallback
- ==============================
- The downside of using this parser is that it is `much slower`_ than
- the HTML parser of lxml. So if performance matters, you might want to
- consider using ``soupparser`` only as a fallback for certain cases.
- .. _`much slower`: http://blog.ianbicking.org/2008/03/30/python-html-parser-performance/
- One common problem of lxml's parser is that it might not get the
- encoding right in cases where the document contains a ``<meta>`` tag
- at the wrong place. In this case, you can exploit the fact that lxml
- serialises much faster than most other HTML libraries for Python.
- Just serialise the document to unicode and if that gives you an
- exception, re-parse it with BeautifulSoup to see if that works
- better.
- .. sourcecode:: pycon
- >>> tag_soup = '''\
- ... <meta http-equiv="Content-Type"
- ... content="text/html;charset=utf-8" />
- ... <html>
- ... <head>
- ... <title>Hello W\xc3\xb6rld!</title>
- ... </head>
- ... <body>Hi all</body>
- ... </html>'''
- >>> import lxml.html
- >>> import lxml.html.soupparser
- >>> root = lxml.html.fromstring(tag_soup)
- >>> try:
- ... ignore = tostring(root, encoding=unicode)
- ... except UnicodeDecodeError:
- ... root = lxml.html.soupparser.fromstring(tag_soup)
|