| 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639 |
- ===========================
- APIs specific to lxml.etree
- ===========================
- lxml.etree tries to follow established APIs wherever possible. Sometimes,
- however, the need to expose a feature in an easy way led to the invention of a
- new API. This page describes the major differences and a few additions to the
- main ElementTree API.
- For a complete reference of the API, see the `generated API
- documentation`_.
- Separate pages describe the support for `parsing XML`_, executing `XPath and
- XSLT`_, `validating XML`_ and interfacing with other XML tools through the
- `SAX-API`_.
- lxml is extremely extensible through `XPath functions in Python`_, custom
- `Python element classes`_, custom `URL resolvers`_ and even `at the C-level`_.
- .. _`parsing XML`: parsing.html
- .. _`XPath and XSLT`: xpathxslt.html
- .. _`validating XML`: validation.html
- .. _`SAX-API`: sax.html
- .. _`XPath functions in Python`: extensions.html
- .. _`Python element classes`: element_classes.html
- .. _`at the C-level`: capi.html
- .. _`URL resolvers`: resolvers.html
- .. _`generated API documentation`: api/index.html
- .. contents::
- ..
- 1 lxml.etree
- 2 Other Element APIs
- 3 Trees and Documents
- 4 Iteration
- 5 Error handling on exceptions
- 6 Error logging
- 7 Serialisation
- 8 Incremental XML generation
- 8 CDATA
- 9 XInclude and ElementInclude
- 10 write_c14n on ElementTree
- ..
- >>> try:
- ... from io import BytesIO
- ... def StringIO(s=None):
- ... if isinstance(s, str): s = s.encode("UTF-8")
- ... return BytesIO(s)
- ... except ImportError:
- ... from StringIO import StringIO # Py2 <= 2.5
- ... BytesIO = StringIO
- >>> try: from collections import deque
- ... except ImportError:
- ... class deque(list):
- ... def popleft(self): return self.pop(0)
- >>> try: unicode = unicode
- ... except NameError: unicode = str
- lxml.etree
- ----------
- lxml.etree tries to follow the `ElementTree API`_ wherever it can. There are
- however some incompatibilities (see `compatibility`_). The extensions are
- documented here.
- .. _`ElementTree API`: http://effbot.org/zone/element-index.htm
- .. _`compatibility`: compatibility.html
- If you need to know which version of lxml is installed, you can access the
- ``lxml.etree.LXML_VERSION`` attribute to retrieve a version tuple. Note,
- however, that it did not exist before version 1.0, so you will get an
- AttributeError in older versions. The versions of libxml2 and libxslt are
- available through the attributes ``LIBXML_VERSION`` and ``LIBXSLT_VERSION``.
- The following examples usually assume this to be executed first:
- .. sourcecode:: pycon
- >>> from lxml import etree
- ..
- >>> import sys
- >>> from lxml import etree as _etree
- >>> if sys.version_info[0] >= 3:
- ... class etree_mock(object):
- ... def __getattr__(self, name): return getattr(_etree, name)
- ... def tostring(self, *args, **kwargs):
- ... s = _etree.tostring(*args, **kwargs)
- ... if isinstance(s, bytes) and bytes([10]) in s: s = s.decode("utf-8") # CR
- ... if s[-1] == '\n': s = s[:-1]
- ... return s
- ... else:
- ... class etree_mock(object):
- ... def __getattr__(self, name): return getattr(_etree, name)
- ... def tostring(self, *args, **kwargs):
- ... s = _etree.tostring(*args, **kwargs)
- ... if s[-1] == '\n': s = s[:-1]
- ... return s
- >>> etree = etree_mock()
- Other Element APIs
- ------------------
- While lxml.etree itself uses the ElementTree API, it is possible to replace
- the Element implementation by `custom element subclasses`_. This has been
- used to implement well-known XML APIs on top of lxml. For example, lxml ships
- with a data-binding implementation called `objectify`_, which is similar to
- the `Amara bindery`_ tool.
- lxml.etree comes with a number of `different lookup schemes`_ to customize the
- mapping between libxml2 nodes and the Element classes used by lxml.etree.
- .. _`custom element subclasses`: element_classes.html
- .. _`objectify`: objectify.html
- .. _`different lookup schemes`: element_classes.html#setting-up-a-class-lookup-scheme
- .. _`Amara bindery`: http://uche.ogbuji.net/tech/4suite/amara/
- Trees and Documents
- -------------------
- Compared to the original ElementTree API, lxml.etree has an extended tree
- model. It knows about parents and siblings of elements:
- .. sourcecode:: pycon
- >>> root = etree.Element("root")
- >>> a = etree.SubElement(root, "a")
- >>> b = etree.SubElement(root, "b")
- >>> c = etree.SubElement(root, "c")
- >>> d = etree.SubElement(root, "d")
- >>> e = etree.SubElement(d, "e")
- >>> b.getparent() == root
- True
- >>> print(b.getnext().tag)
- c
- >>> print(c.getprevious().tag)
- b
- Elements always live within a document context in lxml. This implies that
- there is also a notion of an absolute document root. You can retrieve an
- ElementTree for the root node of a document from any of its elements.
- .. sourcecode:: pycon
- >>> tree = d.getroottree()
- >>> print(tree.getroot().tag)
- root
- Note that this is different from wrapping an Element in an ElementTree. You
- can use ElementTrees to create XML trees with an explicit root node:
- .. sourcecode:: pycon
- >>> tree = etree.ElementTree(d)
- >>> print(tree.getroot().tag)
- d
- >>> etree.tostring(tree)
- b'<d><e/></d>'
- ElementTree objects are serialised as complete documents, including
- preceding or trailing processing instructions and comments.
- All operations that you run on such an ElementTree (like XPath, XSLT, etc.)
- will understand the explicitly chosen root as root node of a document. They
- will not see any elements outside the ElementTree. However, ElementTrees do
- not modify their Elements:
- .. sourcecode:: pycon
- >>> element = tree.getroot()
- >>> print(element.tag)
- d
- >>> print(element.getparent().tag)
- root
- >>> print(element.getroottree().getroot().tag)
- root
- The rule is that all operations that are applied to Elements use either the
- Element itself as reference point, or the absolute root of the document that
- contains this Element (e.g. for absolute XPath expressions). All operations
- on an ElementTree use its explicit root node as reference.
- Iteration
- ---------
- The ElementTree API makes Elements iterable to supports iteration over their
- children. Using the tree defined above, we get:
- .. sourcecode:: pycon
- >>> [ child.tag for child in root ]
- ['a', 'b', 'c', 'd']
- To iterate in the opposite direction, use the builtin ``reversed()`` function
- that exists in Python 2.4 and later.
- Tree traversal should use the ``element.iter()`` method:
- .. sourcecode:: pycon
- >>> [ el.tag for el in root.iter() ]
- ['root', 'a', 'b', 'c', 'd', 'e']
- lxml.etree also supports this, but additionally features an extended API for
- iteration over the children, following/preceding siblings, ancestors and
- descendants of an element, as defined by the respective XPath axis:
- .. sourcecode:: pycon
- >>> [ child.tag for child in root.iterchildren() ]
- ['a', 'b', 'c', 'd']
- >>> [ child.tag for child in root.iterchildren(reversed=True) ]
- ['d', 'c', 'b', 'a']
- >>> [ sibling.tag for sibling in b.itersiblings() ]
- ['c', 'd']
- >>> [ sibling.tag for sibling in c.itersiblings(preceding=True) ]
- ['b', 'a']
- >>> [ ancestor.tag for ancestor in e.iterancestors() ]
- ['d', 'root']
- >>> [ el.tag for el in root.iterdescendants() ]
- ['a', 'b', 'c', 'd', 'e']
- Note how ``element.iterdescendants()`` does not include the element
- itself, as opposed to ``element.iter()``. The latter effectively
- implements the 'descendant-or-self' axis in XPath.
- All of these iterators support one (or more, since lxml 2.4) additional
- arguments that filter the generated elements by tag name:
- .. sourcecode:: pycon
- >>> [ child.tag for child in root.iterchildren('a') ]
- ['a']
- >>> [ child.tag for child in d.iterchildren('a') ]
- []
- >>> [ el.tag for el in root.iterdescendants('d') ]
- ['d']
- >>> [ el.tag for el in root.iter('d') ]
- ['d']
- >>> [ el.tag for el in root.iter('d', 'a') ]
- ['a', 'd']
- Note that the order of the elements is determined by the iteration order,
- which is the document order in most cases (except for preceding siblings
- and ancestors, where it is the reversed document order). The order of
- the tag selection arguments is irrelevant, as you can see in the last
- example.
- The most common way to traverse an XML tree is depth-first, which
- traverses the tree in document order. This is implemented by the
- ``.iter()`` method. While there is no dedicated method for
- breadth-first traversal, it is almost as simple if you use the
- ``collections.deque`` type that is available in Python 2.4 and later.
- .. sourcecode:: pycon
- >>> root = etree.XML('<root><a><b/><c/></a><d><e/></d></root>')
- >>> print(etree.tostring(root, pretty_print=True, encoding='unicode'))
- <root>
- <a>
- <b/>
- <c/>
- </a>
- <d>
- <e/>
- </d>
- </root>
- >>> queue = deque([root])
- >>> while queue:
- ... el = queue.popleft() # pop next element
- ... queue.extend(el) # append its children
- ... print(el.tag)
- root
- a
- d
- b
- c
- e
- See also the section on the utility functions ``iterparse()`` and
- ``iterwalk()`` in the `parser documentation`_.
- .. _`parser documentation`: parsing.html#iterparse-and-iterwalk
- Error handling on exceptions
- ----------------------------
- Libxml2 provides error messages for failures, be it during parsing, XPath
- evaluation or schema validation. The preferred way of accessing them is
- through the local ``error_log`` property of the respective evaluator or
- transformer object. See their documentation for details.
- However, lxml also keeps a global error log of all errors that occurred at the
- application level. Whenever an exception is raised, you can retrieve the
- errors that occured and "might have" lead to the problem from the error log
- copy attached to the exception:
- .. sourcecode:: pycon
- >>> etree.clear_error_log()
- >>> broken_xml = '''
- ... <root>
- ... <a>
- ... </root>
- ... '''
- >>> try:
- ... etree.parse(StringIO(broken_xml))
- ... except etree.XMLSyntaxError, e:
- ... pass # just put the exception into e
- ..
- >>> etree.clear_error_log()
- >>> try:
- ... etree.parse(StringIO(broken_xml))
- ... except etree.XMLSyntaxError:
- ... import sys; e = sys.exc_info()[1]
- Once you have caught this exception, you can access its ``error_log`` property
- to retrieve the log entries or filter them by a specific type, error domain or
- error level:
- .. sourcecode:: pycon
- >>> log = e.error_log.filter_from_level(etree.ErrorLevels.FATAL)
- >>> print(log)
- <string>:4:8:FATAL:PARSER:ERR_TAG_NAME_MISMATCH: Opening and ending tag mismatch: a line 3 and root
- <string>:5:1:FATAL:PARSER:ERR_TAG_NOT_FINISHED: Premature end of data in tag root line 2
- This might look a little cryptic at first, but it is the information that
- libxml2 gives you. At least the message at the end should give you a hint
- what went wrong and you can see that the fatal errors (FATAL) happened during
- parsing (PARSER) lines 4, column 8 and line 5, column 1 of a string (<string>,
- or the filename if available). Here, PARSER is the so-called error domain,
- see ``lxml.etree.ErrorDomains`` for that. You can get it from a log entry
- like this:
- .. sourcecode:: pycon
- >>> entry = log[0]
- >>> print(entry.domain_name)
- PARSER
- >>> print(entry.type_name)
- ERR_TAG_NAME_MISMATCH
- >>> print(entry.filename)
- <string>
- There is also a convenience attribute ``last_error`` that returns the last
- error or fatal error that occurred:
- .. sourcecode:: pycon
- >>> entry = e.error_log.last_error
- >>> print(entry.domain_name)
- PARSER
- >>> print(entry.type_name)
- ERR_TAG_NOT_FINISHED
- >>> print(entry.filename)
- <string>
- Error logging
- -------------
- lxml.etree supports logging libxml2 messages to the Python stdlib logging
- module. This is done through the ``etree.PyErrorLog`` class. It disables the
- error reporting from exceptions and forwards log messages to a Python logger.
- To use it, see the descriptions of the function ``etree.useGlobalPythonLog``
- and the class ``etree.PyErrorLog`` for help. Note that this does not affect
- the local error logs of XSLT, XMLSchema, etc.
- Serialisation
- -------------
- lxml.etree has direct support for pretty printing XML output. Functions like
- ``ElementTree.write()`` and ``tostring()`` support it through a keyword
- argument:
- .. sourcecode:: pycon
- >>> root = etree.XML("<root><test/></root>")
- >>> etree.tostring(root)
- b'<root><test/></root>'
- >>> print(etree.tostring(root, pretty_print=True))
- <root>
- <test/>
- </root>
- Note the newline that is appended at the end when pretty printing the
- output. It was added in lxml 2.0.
- By default, lxml (just as ElementTree) outputs the XML declaration only if it
- is required by the standard:
- .. sourcecode:: pycon
- >>> unicode_root = etree.Element( u"t\u3120st" )
- >>> unicode_root.text = u"t\u0A0Ast"
- >>> etree.tostring(unicode_root, encoding="utf-8")
- b'<t\xe3\x84\xa0st>t\xe0\xa8\x8ast</t\xe3\x84\xa0st>'
- >>> print(etree.tostring(unicode_root, encoding="iso-8859-1"))
- <?xml version='1.0' encoding='iso-8859-1'?>
- <tㄠst>tਊst</tㄠst>
- Also see the general remarks on `Unicode support`_.
- .. _`Unicode support`: parsing.html#python-unicode-strings
- You can enable or disable the declaration explicitly by passing another
- keyword argument for the serialisation:
- .. sourcecode:: pycon
- >>> print(etree.tostring(root, xml_declaration=True))
- <?xml version='1.0' encoding='ASCII'?>
- <root><test/></root>
- >>> unicode_root.clear()
- >>> etree.tostring(unicode_root, encoding="UTF-16LE",
- ... xml_declaration=False)
- b'<\x00t\x00 1s\x00t\x00/\x00>\x00'
- Note that a standard compliant XML parser will not consider the last line
- well-formed XML if the encoding is not explicitly provided somehow, e.g. in an
- underlying transport protocol:
- .. sourcecode:: pycon
- >>> notxml = etree.tostring(unicode_root, encoding="UTF-16LE",
- ... xml_declaration=False)
- >>> root = etree.XML(notxml) #doctest: +ELLIPSIS
- Traceback (most recent call last):
- ...
- lxml.etree.XMLSyntaxError: ...
- Since version 2.3, the serialisation can override the internal subset
- of the document with a user provided DOCTYPE:
- .. sourcecode:: pycon
- >>> xml = '<!DOCTYPE root>\n<root/>'
- >>> tree = etree.parse(StringIO(xml))
- >>> print(etree.tostring(tree))
- <!DOCTYPE root>
- <root/>
- >>> print(etree.tostring(tree,
- ... doctype='<!DOCTYPE root SYSTEM "/tmp/test.dtd">'))
- <!DOCTYPE root SYSTEM "/tmp/test.dtd">
- <root/>
- The content will be encoded, but otherwise copied verbatimly into the
- output stream. It is therefore left to the user to take care for a
- correct doctype format, including the name of the root node.
- Incremental XML generation
- --------------------------
- Since version 3.1, lxml has an API for incrementally generating XML
- using the ``with`` statement. It's main purpose is to freely and
- safely mix surrounding elements with pre-built in-memory trees, e.g.
- to write out large documents that consist mostly of repetitive
- subtrees (like database dumps). But it can be useful in many cases
- where memory consumption matters or where XML is naturally generated
- sequentially.
- The API can serialise to real files (also given as file paths) as
- well as file-like objects, e.g. ``io.BytesIO()``. Here is a simple
- example::
- >>> f = BytesIO()
- >>> with etree.xmlfile(f) as xf:
- ... with xf.element('abc'):
- ... xf.write('text')
- >>> print(f.getvalue().decode('utf-8'))
- <abc>text</abc>
- To insert pre-constructed Elements and subtrees, just pass them
- into ``write()``::
- >>> f = BytesIO()
- >>> with etree.xmlfile(f) as xf:
- ... with xf.element('abc'):
- ... with xf.element('in'):
- ... for value in '123':
- ... # construct a really complex XML tree
- ... el = etree.Element('xyz', attr=value)
- ...
- ... xf.write(el)
- ...
- ... # no longer needed, discard it right away!
- ... el = None
- >>> print(f.getvalue().decode('utf-8'))
- <abc><in><xyz attr="1"/><xyz attr="2"/><xyz attr="3"/></in></abc>
- It is a common pattern to have one or more nested ``element()``
- blocks, and then build in-memory XML subtrees in a loop (using the
- ElementTree API, the builder API, XSLT, or whatever) to write them
- out into the XML file one after the other. That way, they can be
- removed from memory right after their construction, which can largely
- reduce the memory footprint of an application, while keeping the
- overall XML generation easy, safe and correct.
- Together with Python coroutines, this can be used to generate XML
- in an asynchronous, non-blocking fashion, e.g. for a stream protocol
- like the instant messaging protocol
- `XMPP <https://en.wikipedia.org/wiki/Extensible_Messaging_and_Presence_Protocol>`_::
- def writer(out_stream):
- with xmlfile(out_stream) as xf:
- with xf.element('{http://etherx.jabber.org/streams}stream'):
- try:
- while True:
- el = (yield)
- xf.write(el)
- except GeneratorExit:
- pass
- w = writer(stream)
- next(w) # start writing (run up to 'yield')
- Then, whenever XML elements are available for writing, call
- ::
- w.send(element)
- And when done::
- w.close()
- CDATA
- -----
- By default, lxml's parser will strip CDATA sections from the tree and
- replace them by their plain text content. As real applications for
- CDATA are rare, this is the best way to deal with this issue.
- However, in some cases, keeping CDATA sections or creating them in a
- document is required to adhere to existing XML language definitions.
- For these special cases, you can instruct the parser to leave CDATA
- sections in the document:
- .. sourcecode:: pycon
- >>> parser = etree.XMLParser(strip_cdata=False)
- >>> root = etree.XML('<root><![CDATA[test]]></root>', parser)
- >>> root.text
- 'test'
- >>> etree.tostring(root)
- b'<root><![CDATA[test]]></root>'
- Note how the ``.text`` property does not give any indication that the
- text content is wrapped by a CDATA section. If you want to make sure
- your data is wrapped by a CDATA block, you can use the ``CDATA()``
- text wrapper:
- .. sourcecode:: pycon
- >>> root.text = 'test'
- >>> root.text
- 'test'
- >>> etree.tostring(root)
- b'<root>test</root>'
- >>> root.text = etree.CDATA(root.text)
- >>> root.text
- 'test'
- >>> etree.tostring(root)
- b'<root><![CDATA[test]]></root>'
- XInclude and ElementInclude
- ---------------------------
- You can let lxml process xinclude statements in a document by calling the
- xinclude() method on a tree:
- .. sourcecode:: pycon
- >>> data = StringIO('''\
- ... <doc xmlns:xi="http://www.w3.org/2001/XInclude">
- ... <foo/>
- ... <xi:include href="doc/test.xml" />
- ... </doc>''')
- >>> tree = etree.parse(data)
- >>> tree.xinclude()
- >>> print(etree.tostring(tree.getroot()))
- <doc xmlns:xi="http://www.w3.org/2001/XInclude">
- <foo/>
- <a xml:base="doc/test.xml"/>
- </doc>
- Note that the ElementTree compatible ElementInclude_ module is also supported
- as ``lxml.ElementInclude``. It has the additional advantage of supporting
- custom `URL resolvers`_ at the Python level. The normal XInclude mechanism
- cannot deploy these. If you need ElementTree compatibility or custom
- resolvers, you have to stick to the external Python module.
- .. _ElementInclude: http://effbot.org/zone/element-xinclude.htm
- write_c14n on ElementTree
- -------------------------
- The lxml.etree.ElementTree class has a method write_c14n, which takes a file
- object as argument. This file object will receive an UTF-8 representation of
- the canonicalized form of the XML, following the W3C C14N recommendation. For
- example:
- .. sourcecode:: pycon
- >>> f = StringIO('<a><b/></a>')
- >>> tree = etree.parse(f)
- >>> f2 = StringIO()
- >>> tree.write_c14n(f2)
- >>> print(f2.getvalue().decode("utf-8"))
- <a><b></b></a>
|