api.txt 20 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639
  1. ===========================
  2. APIs specific to lxml.etree
  3. ===========================
  4. lxml.etree tries to follow established APIs wherever possible. Sometimes,
  5. however, the need to expose a feature in an easy way led to the invention of a
  6. new API. This page describes the major differences and a few additions to the
  7. main ElementTree API.
  8. For a complete reference of the API, see the `generated API
  9. documentation`_.
  10. Separate pages describe the support for `parsing XML`_, executing `XPath and
  11. XSLT`_, `validating XML`_ and interfacing with other XML tools through the
  12. `SAX-API`_.
  13. lxml is extremely extensible through `XPath functions in Python`_, custom
  14. `Python element classes`_, custom `URL resolvers`_ and even `at the C-level`_.
  15. .. _`parsing XML`: parsing.html
  16. .. _`XPath and XSLT`: xpathxslt.html
  17. .. _`validating XML`: validation.html
  18. .. _`SAX-API`: sax.html
  19. .. _`XPath functions in Python`: extensions.html
  20. .. _`Python element classes`: element_classes.html
  21. .. _`at the C-level`: capi.html
  22. .. _`URL resolvers`: resolvers.html
  23. .. _`generated API documentation`: api/index.html
  24. .. contents::
  25. ..
  26. 1 lxml.etree
  27. 2 Other Element APIs
  28. 3 Trees and Documents
  29. 4 Iteration
  30. 5 Error handling on exceptions
  31. 6 Error logging
  32. 7 Serialisation
  33. 8 Incremental XML generation
  34. 8 CDATA
  35. 9 XInclude and ElementInclude
  36. 10 write_c14n on ElementTree
  37. ..
  38. >>> try:
  39. ... from io import BytesIO
  40. ... def StringIO(s=None):
  41. ... if isinstance(s, str): s = s.encode("UTF-8")
  42. ... return BytesIO(s)
  43. ... except ImportError:
  44. ... from StringIO import StringIO # Py2 <= 2.5
  45. ... BytesIO = StringIO
  46. >>> try: from collections import deque
  47. ... except ImportError:
  48. ... class deque(list):
  49. ... def popleft(self): return self.pop(0)
  50. >>> try: unicode = unicode
  51. ... except NameError: unicode = str
  52. lxml.etree
  53. ----------
  54. lxml.etree tries to follow the `ElementTree API`_ wherever it can. There are
  55. however some incompatibilities (see `compatibility`_). The extensions are
  56. documented here.
  57. .. _`ElementTree API`: http://effbot.org/zone/element-index.htm
  58. .. _`compatibility`: compatibility.html
  59. If you need to know which version of lxml is installed, you can access the
  60. ``lxml.etree.LXML_VERSION`` attribute to retrieve a version tuple. Note,
  61. however, that it did not exist before version 1.0, so you will get an
  62. AttributeError in older versions. The versions of libxml2 and libxslt are
  63. available through the attributes ``LIBXML_VERSION`` and ``LIBXSLT_VERSION``.
  64. The following examples usually assume this to be executed first:
  65. .. sourcecode:: pycon
  66. >>> from lxml import etree
  67. ..
  68. >>> import sys
  69. >>> from lxml import etree as _etree
  70. >>> if sys.version_info[0] >= 3:
  71. ... class etree_mock(object):
  72. ... def __getattr__(self, name): return getattr(_etree, name)
  73. ... def tostring(self, *args, **kwargs):
  74. ... s = _etree.tostring(*args, **kwargs)
  75. ... if isinstance(s, bytes) and bytes([10]) in s: s = s.decode("utf-8") # CR
  76. ... if s[-1] == '\n': s = s[:-1]
  77. ... return s
  78. ... else:
  79. ... class etree_mock(object):
  80. ... def __getattr__(self, name): return getattr(_etree, name)
  81. ... def tostring(self, *args, **kwargs):
  82. ... s = _etree.tostring(*args, **kwargs)
  83. ... if s[-1] == '\n': s = s[:-1]
  84. ... return s
  85. >>> etree = etree_mock()
  86. Other Element APIs
  87. ------------------
  88. While lxml.etree itself uses the ElementTree API, it is possible to replace
  89. the Element implementation by `custom element subclasses`_. This has been
  90. used to implement well-known XML APIs on top of lxml. For example, lxml ships
  91. with a data-binding implementation called `objectify`_, which is similar to
  92. the `Amara bindery`_ tool.
  93. lxml.etree comes with a number of `different lookup schemes`_ to customize the
  94. mapping between libxml2 nodes and the Element classes used by lxml.etree.
  95. .. _`custom element subclasses`: element_classes.html
  96. .. _`objectify`: objectify.html
  97. .. _`different lookup schemes`: element_classes.html#setting-up-a-class-lookup-scheme
  98. .. _`Amara bindery`: http://uche.ogbuji.net/tech/4suite/amara/
  99. Trees and Documents
  100. -------------------
  101. Compared to the original ElementTree API, lxml.etree has an extended tree
  102. model. It knows about parents and siblings of elements:
  103. .. sourcecode:: pycon
  104. >>> root = etree.Element("root")
  105. >>> a = etree.SubElement(root, "a")
  106. >>> b = etree.SubElement(root, "b")
  107. >>> c = etree.SubElement(root, "c")
  108. >>> d = etree.SubElement(root, "d")
  109. >>> e = etree.SubElement(d, "e")
  110. >>> b.getparent() == root
  111. True
  112. >>> print(b.getnext().tag)
  113. c
  114. >>> print(c.getprevious().tag)
  115. b
  116. Elements always live within a document context in lxml. This implies that
  117. there is also a notion of an absolute document root. You can retrieve an
  118. ElementTree for the root node of a document from any of its elements.
  119. .. sourcecode:: pycon
  120. >>> tree = d.getroottree()
  121. >>> print(tree.getroot().tag)
  122. root
  123. Note that this is different from wrapping an Element in an ElementTree. You
  124. can use ElementTrees to create XML trees with an explicit root node:
  125. .. sourcecode:: pycon
  126. >>> tree = etree.ElementTree(d)
  127. >>> print(tree.getroot().tag)
  128. d
  129. >>> etree.tostring(tree)
  130. b'<d><e/></d>'
  131. ElementTree objects are serialised as complete documents, including
  132. preceding or trailing processing instructions and comments.
  133. All operations that you run on such an ElementTree (like XPath, XSLT, etc.)
  134. will understand the explicitly chosen root as root node of a document. They
  135. will not see any elements outside the ElementTree. However, ElementTrees do
  136. not modify their Elements:
  137. .. sourcecode:: pycon
  138. >>> element = tree.getroot()
  139. >>> print(element.tag)
  140. d
  141. >>> print(element.getparent().tag)
  142. root
  143. >>> print(element.getroottree().getroot().tag)
  144. root
  145. The rule is that all operations that are applied to Elements use either the
  146. Element itself as reference point, or the absolute root of the document that
  147. contains this Element (e.g. for absolute XPath expressions). All operations
  148. on an ElementTree use its explicit root node as reference.
  149. Iteration
  150. ---------
  151. The ElementTree API makes Elements iterable to supports iteration over their
  152. children. Using the tree defined above, we get:
  153. .. sourcecode:: pycon
  154. >>> [ child.tag for child in root ]
  155. ['a', 'b', 'c', 'd']
  156. To iterate in the opposite direction, use the builtin ``reversed()`` function
  157. that exists in Python 2.4 and later.
  158. Tree traversal should use the ``element.iter()`` method:
  159. .. sourcecode:: pycon
  160. >>> [ el.tag for el in root.iter() ]
  161. ['root', 'a', 'b', 'c', 'd', 'e']
  162. lxml.etree also supports this, but additionally features an extended API for
  163. iteration over the children, following/preceding siblings, ancestors and
  164. descendants of an element, as defined by the respective XPath axis:
  165. .. sourcecode:: pycon
  166. >>> [ child.tag for child in root.iterchildren() ]
  167. ['a', 'b', 'c', 'd']
  168. >>> [ child.tag for child in root.iterchildren(reversed=True) ]
  169. ['d', 'c', 'b', 'a']
  170. >>> [ sibling.tag for sibling in b.itersiblings() ]
  171. ['c', 'd']
  172. >>> [ sibling.tag for sibling in c.itersiblings(preceding=True) ]
  173. ['b', 'a']
  174. >>> [ ancestor.tag for ancestor in e.iterancestors() ]
  175. ['d', 'root']
  176. >>> [ el.tag for el in root.iterdescendants() ]
  177. ['a', 'b', 'c', 'd', 'e']
  178. Note how ``element.iterdescendants()`` does not include the element
  179. itself, as opposed to ``element.iter()``. The latter effectively
  180. implements the 'descendant-or-self' axis in XPath.
  181. All of these iterators support one (or more, since lxml 2.4) additional
  182. arguments that filter the generated elements by tag name:
  183. .. sourcecode:: pycon
  184. >>> [ child.tag for child in root.iterchildren('a') ]
  185. ['a']
  186. >>> [ child.tag for child in d.iterchildren('a') ]
  187. []
  188. >>> [ el.tag for el in root.iterdescendants('d') ]
  189. ['d']
  190. >>> [ el.tag for el in root.iter('d') ]
  191. ['d']
  192. >>> [ el.tag for el in root.iter('d', 'a') ]
  193. ['a', 'd']
  194. Note that the order of the elements is determined by the iteration order,
  195. which is the document order in most cases (except for preceding siblings
  196. and ancestors, where it is the reversed document order). The order of
  197. the tag selection arguments is irrelevant, as you can see in the last
  198. example.
  199. The most common way to traverse an XML tree is depth-first, which
  200. traverses the tree in document order. This is implemented by the
  201. ``.iter()`` method. While there is no dedicated method for
  202. breadth-first traversal, it is almost as simple if you use the
  203. ``collections.deque`` type that is available in Python 2.4 and later.
  204. .. sourcecode:: pycon
  205. >>> root = etree.XML('<root><a><b/><c/></a><d><e/></d></root>')
  206. >>> print(etree.tostring(root, pretty_print=True, encoding='unicode'))
  207. <root>
  208. <a>
  209. <b/>
  210. <c/>
  211. </a>
  212. <d>
  213. <e/>
  214. </d>
  215. </root>
  216. >>> queue = deque([root])
  217. >>> while queue:
  218. ... el = queue.popleft() # pop next element
  219. ... queue.extend(el) # append its children
  220. ... print(el.tag)
  221. root
  222. a
  223. d
  224. b
  225. c
  226. e
  227. See also the section on the utility functions ``iterparse()`` and
  228. ``iterwalk()`` in the `parser documentation`_.
  229. .. _`parser documentation`: parsing.html#iterparse-and-iterwalk
  230. Error handling on exceptions
  231. ----------------------------
  232. Libxml2 provides error messages for failures, be it during parsing, XPath
  233. evaluation or schema validation. The preferred way of accessing them is
  234. through the local ``error_log`` property of the respective evaluator or
  235. transformer object. See their documentation for details.
  236. However, lxml also keeps a global error log of all errors that occurred at the
  237. application level. Whenever an exception is raised, you can retrieve the
  238. errors that occured and "might have" lead to the problem from the error log
  239. copy attached to the exception:
  240. .. sourcecode:: pycon
  241. >>> etree.clear_error_log()
  242. >>> broken_xml = '''
  243. ... <root>
  244. ... <a>
  245. ... </root>
  246. ... '''
  247. >>> try:
  248. ... etree.parse(StringIO(broken_xml))
  249. ... except etree.XMLSyntaxError, e:
  250. ... pass # just put the exception into e
  251. ..
  252. >>> etree.clear_error_log()
  253. >>> try:
  254. ... etree.parse(StringIO(broken_xml))
  255. ... except etree.XMLSyntaxError:
  256. ... import sys; e = sys.exc_info()[1]
  257. Once you have caught this exception, you can access its ``error_log`` property
  258. to retrieve the log entries or filter them by a specific type, error domain or
  259. error level:
  260. .. sourcecode:: pycon
  261. >>> log = e.error_log.filter_from_level(etree.ErrorLevels.FATAL)
  262. >>> print(log)
  263. <string>:4:8:FATAL:PARSER:ERR_TAG_NAME_MISMATCH: Opening and ending tag mismatch: a line 3 and root
  264. <string>:5:1:FATAL:PARSER:ERR_TAG_NOT_FINISHED: Premature end of data in tag root line 2
  265. This might look a little cryptic at first, but it is the information that
  266. libxml2 gives you. At least the message at the end should give you a hint
  267. what went wrong and you can see that the fatal errors (FATAL) happened during
  268. parsing (PARSER) lines 4, column 8 and line 5, column 1 of a string (<string>,
  269. or the filename if available). Here, PARSER is the so-called error domain,
  270. see ``lxml.etree.ErrorDomains`` for that. You can get it from a log entry
  271. like this:
  272. .. sourcecode:: pycon
  273. >>> entry = log[0]
  274. >>> print(entry.domain_name)
  275. PARSER
  276. >>> print(entry.type_name)
  277. ERR_TAG_NAME_MISMATCH
  278. >>> print(entry.filename)
  279. <string>
  280. There is also a convenience attribute ``last_error`` that returns the last
  281. error or fatal error that occurred:
  282. .. sourcecode:: pycon
  283. >>> entry = e.error_log.last_error
  284. >>> print(entry.domain_name)
  285. PARSER
  286. >>> print(entry.type_name)
  287. ERR_TAG_NOT_FINISHED
  288. >>> print(entry.filename)
  289. <string>
  290. Error logging
  291. -------------
  292. lxml.etree supports logging libxml2 messages to the Python stdlib logging
  293. module. This is done through the ``etree.PyErrorLog`` class. It disables the
  294. error reporting from exceptions and forwards log messages to a Python logger.
  295. To use it, see the descriptions of the function ``etree.useGlobalPythonLog``
  296. and the class ``etree.PyErrorLog`` for help. Note that this does not affect
  297. the local error logs of XSLT, XMLSchema, etc.
  298. Serialisation
  299. -------------
  300. lxml.etree has direct support for pretty printing XML output. Functions like
  301. ``ElementTree.write()`` and ``tostring()`` support it through a keyword
  302. argument:
  303. .. sourcecode:: pycon
  304. >>> root = etree.XML("<root><test/></root>")
  305. >>> etree.tostring(root)
  306. b'<root><test/></root>'
  307. >>> print(etree.tostring(root, pretty_print=True))
  308. <root>
  309. <test/>
  310. </root>
  311. Note the newline that is appended at the end when pretty printing the
  312. output. It was added in lxml 2.0.
  313. By default, lxml (just as ElementTree) outputs the XML declaration only if it
  314. is required by the standard:
  315. .. sourcecode:: pycon
  316. >>> unicode_root = etree.Element( u"t\u3120st" )
  317. >>> unicode_root.text = u"t\u0A0Ast"
  318. >>> etree.tostring(unicode_root, encoding="utf-8")
  319. b'<t\xe3\x84\xa0st>t\xe0\xa8\x8ast</t\xe3\x84\xa0st>'
  320. >>> print(etree.tostring(unicode_root, encoding="iso-8859-1"))
  321. <?xml version='1.0' encoding='iso-8859-1'?>
  322. <t&#12576;st>t&#2570;st</t&#12576;st>
  323. Also see the general remarks on `Unicode support`_.
  324. .. _`Unicode support`: parsing.html#python-unicode-strings
  325. You can enable or disable the declaration explicitly by passing another
  326. keyword argument for the serialisation:
  327. .. sourcecode:: pycon
  328. >>> print(etree.tostring(root, xml_declaration=True))
  329. <?xml version='1.0' encoding='ASCII'?>
  330. <root><test/></root>
  331. >>> unicode_root.clear()
  332. >>> etree.tostring(unicode_root, encoding="UTF-16LE",
  333. ... xml_declaration=False)
  334. b'<\x00t\x00 1s\x00t\x00/\x00>\x00'
  335. Note that a standard compliant XML parser will not consider the last line
  336. well-formed XML if the encoding is not explicitly provided somehow, e.g. in an
  337. underlying transport protocol:
  338. .. sourcecode:: pycon
  339. >>> notxml = etree.tostring(unicode_root, encoding="UTF-16LE",
  340. ... xml_declaration=False)
  341. >>> root = etree.XML(notxml) #doctest: +ELLIPSIS
  342. Traceback (most recent call last):
  343. ...
  344. lxml.etree.XMLSyntaxError: ...
  345. Since version 2.3, the serialisation can override the internal subset
  346. of the document with a user provided DOCTYPE:
  347. .. sourcecode:: pycon
  348. >>> xml = '<!DOCTYPE root>\n<root/>'
  349. >>> tree = etree.parse(StringIO(xml))
  350. >>> print(etree.tostring(tree))
  351. <!DOCTYPE root>
  352. <root/>
  353. >>> print(etree.tostring(tree,
  354. ... doctype='<!DOCTYPE root SYSTEM "/tmp/test.dtd">'))
  355. <!DOCTYPE root SYSTEM "/tmp/test.dtd">
  356. <root/>
  357. The content will be encoded, but otherwise copied verbatimly into the
  358. output stream. It is therefore left to the user to take care for a
  359. correct doctype format, including the name of the root node.
  360. Incremental XML generation
  361. --------------------------
  362. Since version 3.1, lxml has an API for incrementally generating XML
  363. using the ``with`` statement. It's main purpose is to freely and
  364. safely mix surrounding elements with pre-built in-memory trees, e.g.
  365. to write out large documents that consist mostly of repetitive
  366. subtrees (like database dumps). But it can be useful in many cases
  367. where memory consumption matters or where XML is naturally generated
  368. sequentially.
  369. The API can serialise to real files (also given as file paths) as
  370. well as file-like objects, e.g. ``io.BytesIO()``. Here is a simple
  371. example::
  372. >>> f = BytesIO()
  373. >>> with etree.xmlfile(f) as xf:
  374. ... with xf.element('abc'):
  375. ... xf.write('text')
  376. >>> print(f.getvalue().decode('utf-8'))
  377. <abc>text</abc>
  378. To insert pre-constructed Elements and subtrees, just pass them
  379. into ``write()``::
  380. >>> f = BytesIO()
  381. >>> with etree.xmlfile(f) as xf:
  382. ... with xf.element('abc'):
  383. ... with xf.element('in'):
  384. ... for value in '123':
  385. ... # construct a really complex XML tree
  386. ... el = etree.Element('xyz', attr=value)
  387. ...
  388. ... xf.write(el)
  389. ...
  390. ... # no longer needed, discard it right away!
  391. ... el = None
  392. >>> print(f.getvalue().decode('utf-8'))
  393. <abc><in><xyz attr="1"/><xyz attr="2"/><xyz attr="3"/></in></abc>
  394. It is a common pattern to have one or more nested ``element()``
  395. blocks, and then build in-memory XML subtrees in a loop (using the
  396. ElementTree API, the builder API, XSLT, or whatever) to write them
  397. out into the XML file one after the other. That way, they can be
  398. removed from memory right after their construction, which can largely
  399. reduce the memory footprint of an application, while keeping the
  400. overall XML generation easy, safe and correct.
  401. Together with Python coroutines, this can be used to generate XML
  402. in an asynchronous, non-blocking fashion, e.g. for a stream protocol
  403. like the instant messaging protocol
  404. `XMPP <https://en.wikipedia.org/wiki/Extensible_Messaging_and_Presence_Protocol>`_::
  405. def writer(out_stream):
  406. with xmlfile(out_stream) as xf:
  407. with xf.element('{http://etherx.jabber.org/streams}stream'):
  408. try:
  409. while True:
  410. el = (yield)
  411. xf.write(el)
  412. except GeneratorExit:
  413. pass
  414. w = writer(stream)
  415. next(w) # start writing (run up to 'yield')
  416. Then, whenever XML elements are available for writing, call
  417. ::
  418. w.send(element)
  419. And when done::
  420. w.close()
  421. CDATA
  422. -----
  423. By default, lxml's parser will strip CDATA sections from the tree and
  424. replace them by their plain text content. As real applications for
  425. CDATA are rare, this is the best way to deal with this issue.
  426. However, in some cases, keeping CDATA sections or creating them in a
  427. document is required to adhere to existing XML language definitions.
  428. For these special cases, you can instruct the parser to leave CDATA
  429. sections in the document:
  430. .. sourcecode:: pycon
  431. >>> parser = etree.XMLParser(strip_cdata=False)
  432. >>> root = etree.XML('<root><![CDATA[test]]></root>', parser)
  433. >>> root.text
  434. 'test'
  435. >>> etree.tostring(root)
  436. b'<root><![CDATA[test]]></root>'
  437. Note how the ``.text`` property does not give any indication that the
  438. text content is wrapped by a CDATA section. If you want to make sure
  439. your data is wrapped by a CDATA block, you can use the ``CDATA()``
  440. text wrapper:
  441. .. sourcecode:: pycon
  442. >>> root.text = 'test'
  443. >>> root.text
  444. 'test'
  445. >>> etree.tostring(root)
  446. b'<root>test</root>'
  447. >>> root.text = etree.CDATA(root.text)
  448. >>> root.text
  449. 'test'
  450. >>> etree.tostring(root)
  451. b'<root><![CDATA[test]]></root>'
  452. XInclude and ElementInclude
  453. ---------------------------
  454. You can let lxml process xinclude statements in a document by calling the
  455. xinclude() method on a tree:
  456. .. sourcecode:: pycon
  457. >>> data = StringIO('''\
  458. ... <doc xmlns:xi="http://www.w3.org/2001/XInclude">
  459. ... <foo/>
  460. ... <xi:include href="doc/test.xml" />
  461. ... </doc>''')
  462. >>> tree = etree.parse(data)
  463. >>> tree.xinclude()
  464. >>> print(etree.tostring(tree.getroot()))
  465. <doc xmlns:xi="http://www.w3.org/2001/XInclude">
  466. <foo/>
  467. <a xml:base="doc/test.xml"/>
  468. </doc>
  469. Note that the ElementTree compatible ElementInclude_ module is also supported
  470. as ``lxml.ElementInclude``. It has the additional advantage of supporting
  471. custom `URL resolvers`_ at the Python level. The normal XInclude mechanism
  472. cannot deploy these. If you need ElementTree compatibility or custom
  473. resolvers, you have to stick to the external Python module.
  474. .. _ElementInclude: http://effbot.org/zone/element-xinclude.htm
  475. write_c14n on ElementTree
  476. -------------------------
  477. The lxml.etree.ElementTree class has a method write_c14n, which takes a file
  478. object as argument. This file object will receive an UTF-8 representation of
  479. the canonicalized form of the XML, following the W3C C14N recommendation. For
  480. example:
  481. .. sourcecode:: pycon
  482. >>> f = StringIO('<a><b/></a>')
  483. >>> tree = etree.parse(f)
  484. >>> f2 = StringIO()
  485. >>> tree.write_c14n(f2)
  486. >>> print(f2.getvalue().decode("utf-8"))
  487. <a><b></b></a>