api.txt 16 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528
  1. ===========================
  2. APIs specific to lxml.etree
  3. ===========================
  4. lxml.etree tries to follow established APIs wherever possible. Sometimes,
  5. however, the need to expose a feature in an easy way led to the invention of a
  6. new API. This page describes the major differences and a few additions to the
  7. main ElementTree API.
  8. For a complete reference of the API, see the `generated API
  9. documentation`_.
  10. Separate pages describe the support for `parsing XML`_, executing `XPath and
  11. XSLT`_, `validating XML`_ and interfacing with other XML tools through the
  12. `SAX-API`_.
  13. lxml is extremely extensible through `XPath functions in Python`_, custom
  14. `Python element classes`_, custom `URL resolvers`_ and even `at the C-level`_.
  15. .. _`parsing XML`: parsing.html
  16. .. _`XPath and XSLT`: xpathxslt.html
  17. .. _`validating XML`: validation.html
  18. .. _`SAX-API`: sax.html
  19. .. _`XPath functions in Python`: extensions.html
  20. .. _`Python element classes`: element_classes.html
  21. .. _`at the C-level`: capi.html
  22. .. _`URL resolvers`: resolvers.html
  23. .. _`generated API documentation`: api/index.html
  24. .. contents::
  25. ..
  26. 1 lxml.etree
  27. 2 Other Element APIs
  28. 3 Trees and Documents
  29. 4 Iteration
  30. 5 Error handling on exceptions
  31. 6 Error logging
  32. 7 Serialisation
  33. 8 CDATA
  34. 9 XInclude and ElementInclude
  35. 10 write_c14n on ElementTree
  36. ..
  37. >>> try: from StringIO import StringIO
  38. ... except ImportError:
  39. ... from io import BytesIO
  40. ... def StringIO(s=None):
  41. ... if isinstance(s, str): s = s.encode("UTF-8")
  42. ... return BytesIO(s)
  43. >>> try: from collections import deque
  44. ... except ImportError:
  45. ... class deque(list):
  46. ... def popleft(self): return self.pop(0)
  47. >>> try: unicode = __builtins__["unicode"]
  48. ... except (NameError, KeyError): unicode = str
  49. lxml.etree
  50. ----------
  51. lxml.etree tries to follow the `ElementTree API`_ wherever it can. There are
  52. however some incompatibilities (see `compatibility`_). The extensions are
  53. documented here.
  54. .. _`ElementTree API`: http://effbot.org/zone/element-index.htm
  55. .. _`compatibility`: compatibility.html
  56. If you need to know which version of lxml is installed, you can access the
  57. ``lxml.etree.LXML_VERSION`` attribute to retrieve a version tuple. Note,
  58. however, that it did not exist before version 1.0, so you will get an
  59. AttributeError in older versions. The versions of libxml2 and libxslt are
  60. available through the attributes ``LIBXML_VERSION`` and ``LIBXSLT_VERSION``.
  61. The following examples usually assume this to be executed first:
  62. .. sourcecode:: pycon
  63. >>> from lxml import etree
  64. ..
  65. >>> import sys
  66. >>> from lxml import etree as _etree
  67. >>> if sys.version_info[0] >= 3:
  68. ... class etree_mock(object):
  69. ... def __getattr__(self, name): return getattr(_etree, name)
  70. ... def tostring(self, *args, **kwargs):
  71. ... s = _etree.tostring(*args, **kwargs)
  72. ... if isinstance(s, bytes) and bytes([10]) in s: s = s.decode("utf-8") # CR
  73. ... if s[-1] == '\n': s = s[:-1]
  74. ... return s
  75. ... else:
  76. ... class etree_mock(object):
  77. ... def __getattr__(self, name): return getattr(_etree, name)
  78. ... def tostring(self, *args, **kwargs):
  79. ... s = _etree.tostring(*args, **kwargs)
  80. ... if s[-1] == '\n': s = s[:-1]
  81. ... return s
  82. >>> etree = etree_mock()
  83. Other Element APIs
  84. ------------------
  85. While lxml.etree itself uses the ElementTree API, it is possible to replace
  86. the Element implementation by `custom element subclasses`_. This has been
  87. used to implement well-known XML APIs on top of lxml. For example, lxml ships
  88. with a data-binding implementation called `objectify`_, which is similar to
  89. the `Amara bindery`_ tool.
  90. lxml.etree comes with a number of `different lookup schemes`_ to customize the
  91. mapping between libxml2 nodes and the Element classes used by lxml.etree.
  92. .. _`custom element subclasses`: element_classes.html
  93. .. _`objectify`: objectify.html
  94. .. _`different lookup schemes`: element_classes.html#setting-up-a-class-lookup-scheme
  95. .. _`Amara bindery`: http://uche.ogbuji.net/tech/4suite/amara/
  96. Trees and Documents
  97. -------------------
  98. Compared to the original ElementTree API, lxml.etree has an extended tree
  99. model. It knows about parents and siblings of elements:
  100. .. sourcecode:: pycon
  101. >>> root = etree.Element("root")
  102. >>> a = etree.SubElement(root, "a")
  103. >>> b = etree.SubElement(root, "b")
  104. >>> c = etree.SubElement(root, "c")
  105. >>> d = etree.SubElement(root, "d")
  106. >>> e = etree.SubElement(d, "e")
  107. >>> b.getparent() == root
  108. True
  109. >>> print(b.getnext().tag)
  110. c
  111. >>> print(c.getprevious().tag)
  112. b
  113. Elements always live within a document context in lxml. This implies that
  114. there is also a notion of an absolute document root. You can retrieve an
  115. ElementTree for the root node of a document from any of its elements.
  116. .. sourcecode:: pycon
  117. >>> tree = d.getroottree()
  118. >>> print(tree.getroot().tag)
  119. root
  120. Note that this is different from wrapping an Element in an ElementTree. You
  121. can use ElementTrees to create XML trees with an explicit root node:
  122. .. sourcecode:: pycon
  123. >>> tree = etree.ElementTree(d)
  124. >>> print(tree.getroot().tag)
  125. d
  126. >>> etree.tostring(tree)
  127. b'<d><e/></d>'
  128. ElementTree objects are serialised as complete documents, including
  129. preceding or trailing processing instructions and comments.
  130. All operations that you run on such an ElementTree (like XPath, XSLT, etc.)
  131. will understand the explicitly chosen root as root node of a document. They
  132. will not see any elements outside the ElementTree. However, ElementTrees do
  133. not modify their Elements:
  134. .. sourcecode:: pycon
  135. >>> element = tree.getroot()
  136. >>> print(element.tag)
  137. d
  138. >>> print(element.getparent().tag)
  139. root
  140. >>> print(element.getroottree().getroot().tag)
  141. root
  142. The rule is that all operations that are applied to Elements use either the
  143. Element itself as reference point, or the absolute root of the document that
  144. contains this Element (e.g. for absolute XPath expressions). All operations
  145. on an ElementTree use its explicit root node as reference.
  146. Iteration
  147. ---------
  148. The ElementTree API makes Elements iterable to supports iteration over their
  149. children. Using the tree defined above, we get:
  150. .. sourcecode:: pycon
  151. >>> [ child.tag for child in root ]
  152. ['a', 'b', 'c', 'd']
  153. To iterate in the opposite direction, use the ``reversed()`` function
  154. that exists in Python 2.4 and later.
  155. Tree traversal should use the ``element.iter()`` method:
  156. .. sourcecode:: pycon
  157. >>> [ el.tag for el in root.iter() ]
  158. ['root', 'a', 'b', 'c', 'd', 'e']
  159. lxml.etree also supports this, but additionally features an extended API for
  160. iteration over the children, following/preceding siblings, ancestors and
  161. descendants of an element, as defined by the respective XPath axis:
  162. .. sourcecode:: pycon
  163. >>> [ child.tag for child in root.iterchildren() ]
  164. ['a', 'b', 'c', 'd']
  165. >>> [ child.tag for child in root.iterchildren(reversed=True) ]
  166. ['d', 'c', 'b', 'a']
  167. >>> [ sibling.tag for sibling in b.itersiblings() ]
  168. ['c', 'd']
  169. >>> [ sibling.tag for sibling in c.itersiblings(preceding=True) ]
  170. ['b', 'a']
  171. >>> [ ancestor.tag for ancestor in e.iterancestors() ]
  172. ['d', 'root']
  173. >>> [ el.tag for el in root.iterdescendants() ]
  174. ['a', 'b', 'c', 'd', 'e']
  175. Note how ``element.iterdescendants()`` does not include the element
  176. itself, as opposed to ``element.iter()``. The latter effectively
  177. implements the 'descendant-or-self' axis in XPath.
  178. All of these iterators support an additional ``tag`` keyword argument that
  179. filters the generated elements by tag name:
  180. .. sourcecode:: pycon
  181. >>> [ child.tag for child in root.iterchildren(tag='a') ]
  182. ['a']
  183. >>> [ child.tag for child in d.iterchildren(tag='a') ]
  184. []
  185. >>> [ el.tag for el in root.iterdescendants(tag='d') ]
  186. ['d']
  187. >>> [ el.tag for el in root.iter(tag='d') ]
  188. ['d']
  189. The most common way to traverse an XML tree is depth-first, which
  190. traverses the tree in document order. This is implemented by the
  191. ``.iter()`` method. While there is no dedicated method for
  192. breadth-first traversal, it is almost as simple if you use the
  193. ``collections.deque`` type from Python 2.4.
  194. .. sourcecode:: pycon
  195. >>> root = etree.XML('<root><a><b/><c/></a><d><e/></d></root>')
  196. >>> print(etree.tostring(root, pretty_print=True, encoding=unicode))
  197. <root>
  198. <a>
  199. <b/>
  200. <c/>
  201. </a>
  202. <d>
  203. <e/>
  204. </d>
  205. </root>
  206. >>> queue = deque([root])
  207. >>> while queue:
  208. ... el = queue.popleft() # pop next element
  209. ... queue.extend(el) # append its children
  210. ... print(el.tag)
  211. root
  212. a
  213. d
  214. b
  215. c
  216. e
  217. See also the section on the utility functions ``iterparse()`` and
  218. ``iterwalk()`` in the `parser documentation`_.
  219. .. _`parser documentation`: parsing.html#iterparse-and-iterwalk
  220. Error handling on exceptions
  221. ----------------------------
  222. Libxml2 provides error messages for failures, be it during parsing, XPath
  223. evaluation or schema validation. The preferred way of accessing them is
  224. through the local ``error_log`` property of the respective evaluator or
  225. transformer object. See their documentation for details.
  226. However, lxml also keeps a global error log of all errors that occurred at the
  227. application level. Whenever an exception is raised, you can retrieve the
  228. errors that occured and "might have" lead to the problem from the error log
  229. copy attached to the exception:
  230. .. sourcecode:: pycon
  231. >>> etree.clear_error_log()
  232. >>> broken_xml = '''
  233. ... <root>
  234. ... <a>
  235. ... </root>
  236. ... '''
  237. >>> try:
  238. ... etree.parse(StringIO(broken_xml))
  239. ... except etree.XMLSyntaxError, e:
  240. ... pass # just put the exception into e
  241. ..
  242. >>> etree.clear_error_log()
  243. >>> try:
  244. ... etree.parse(StringIO(broken_xml))
  245. ... except etree.XMLSyntaxError:
  246. ... import sys; e = sys.exc_info()[1]
  247. Once you have caught this exception, you can access its ``error_log`` property
  248. to retrieve the log entries or filter them by a specific type, error domain or
  249. error level:
  250. .. sourcecode:: pycon
  251. >>> log = e.error_log.filter_from_level(etree.ErrorLevels.FATAL)
  252. >>> print(log)
  253. <string>:4:8:FATAL:PARSER:ERR_TAG_NAME_MISMATCH: Opening and ending tag mismatch: a line 3 and root
  254. <string>:5:1:FATAL:PARSER:ERR_TAG_NOT_FINISHED: Premature end of data in tag root line 2
  255. This might look a little cryptic at first, but it is the information that
  256. libxml2 gives you. At least the message at the end should give you a hint
  257. what went wrong and you can see that the fatal errors (FATAL) happened during
  258. parsing (PARSER) lines 4, column 8 and line 5, column 1 of a string (<string>,
  259. or the filename if available). Here, PARSER is the so-called error domain,
  260. see ``lxml.etree.ErrorDomains`` for that. You can get it from a log entry
  261. like this:
  262. .. sourcecode:: pycon
  263. >>> entry = log[0]
  264. >>> print(entry.domain_name)
  265. PARSER
  266. >>> print(entry.type_name)
  267. ERR_TAG_NAME_MISMATCH
  268. >>> print(entry.filename)
  269. <string>
  270. There is also a convenience attribute ``last_error`` that returns the last
  271. error or fatal error that occurred:
  272. .. sourcecode:: pycon
  273. >>> entry = e.error_log.last_error
  274. >>> print(entry.domain_name)
  275. PARSER
  276. >>> print(entry.type_name)
  277. ERR_TAG_NOT_FINISHED
  278. >>> print(entry.filename)
  279. <string>
  280. Error logging
  281. -------------
  282. lxml.etree supports logging libxml2 messages to the Python stdlib logging
  283. module. This is done through the ``etree.PyErrorLog`` class. It disables the
  284. error reporting from exceptions and forwards log messages to a Python logger.
  285. To use it, see the descriptions of the function ``etree.useGlobalPythonLog``
  286. and the class ``etree.PyErrorLog`` for help. Note that this does not affect
  287. the local error logs of XSLT, XMLSchema, etc.
  288. Serialisation
  289. -------------
  290. lxml.etree has direct support for pretty printing XML output. Functions like
  291. ``ElementTree.write()`` and ``tostring()`` support it through a keyword
  292. argument:
  293. .. sourcecode:: pycon
  294. >>> root = etree.XML("<root><test/></root>")
  295. >>> etree.tostring(root)
  296. b'<root><test/></root>'
  297. >>> print(etree.tostring(root, pretty_print=True))
  298. <root>
  299. <test/>
  300. </root>
  301. Note the newline that is appended at the end when pretty printing the
  302. output. It was added in lxml 2.0.
  303. By default, lxml (just as ElementTree) outputs the XML declaration only if it
  304. is required by the standard:
  305. .. sourcecode:: pycon
  306. >>> unicode_root = etree.Element( u"t\u3120st" )
  307. >>> unicode_root.text = u"t\u0A0Ast"
  308. >>> etree.tostring(unicode_root, encoding="utf-8")
  309. b'<t\xe3\x84\xa0st>t\xe0\xa8\x8ast</t\xe3\x84\xa0st>'
  310. >>> print(etree.tostring(unicode_root, encoding="iso-8859-1"))
  311. <?xml version='1.0' encoding='iso-8859-1'?>
  312. <t&#12576;st>t&#2570;st</t&#12576;st>
  313. Also see the general remarks on `Unicode support`_.
  314. .. _`Unicode support`: parsing.html#python-unicode-strings
  315. You can enable or disable the declaration explicitly by passing another
  316. keyword argument for the serialisation:
  317. .. sourcecode:: pycon
  318. >>> print(etree.tostring(root, xml_declaration=True))
  319. <?xml version='1.0' encoding='ASCII'?>
  320. <root><test/></root>
  321. >>> unicode_root.clear()
  322. >>> etree.tostring(unicode_root, encoding="UTF-16LE",
  323. ... xml_declaration=False)
  324. b'<\x00t\x00 1s\x00t\x00/\x00>\x00'
  325. Note that a standard compliant XML parser will not consider the last line
  326. well-formed XML if the encoding is not explicitly provided somehow, e.g. in an
  327. underlying transport protocol:
  328. .. sourcecode:: pycon
  329. >>> notxml = etree.tostring(unicode_root, encoding="UTF-16LE",
  330. ... xml_declaration=False)
  331. >>> root = etree.XML(notxml) #doctest: +ELLIPSIS
  332. Traceback (most recent call last):
  333. ...
  334. lxml.etree.XMLSyntaxError: ...
  335. CDATA
  336. -----
  337. By default, lxml's parser will strip CDATA sections from the tree and
  338. replace them by their plain text content. As real applications for
  339. CDATA are rare, this is the best way to deal with this issue.
  340. However, in some cases, keeping CDATA sections or creating them in a
  341. document is required to adhere to existing XML language definitions.
  342. For these special cases, you can instruct the parser to leave CDATA
  343. sections in the document:
  344. .. sourcecode:: pycon
  345. >>> parser = etree.XMLParser(strip_cdata=False)
  346. >>> root = etree.XML('<root><![CDATA[test]]></root>', parser)
  347. >>> root.text
  348. 'test'
  349. >>> etree.tostring(root)
  350. b'<root><![CDATA[test]]></root>'
  351. Note how the ``.text`` property does not give any indication that the
  352. text content is wrapped by a CDATA section. If you want to make sure
  353. your data is wrapped by a CDATA block, you can use the ``CDATA()``
  354. text wrapper:
  355. .. sourcecode:: pycon
  356. >>> root.text = 'test'
  357. >>> root.text
  358. 'test'
  359. >>> etree.tostring(root)
  360. b'<root>test</root>'
  361. >>> root.text = etree.CDATA(root.text)
  362. >>> root.text
  363. 'test'
  364. >>> etree.tostring(root)
  365. b'<root><![CDATA[test]]></root>'
  366. XInclude and ElementInclude
  367. ---------------------------
  368. You can let lxml process xinclude statements in a document by calling the
  369. xinclude() method on a tree:
  370. .. sourcecode:: pycon
  371. >>> data = StringIO('''\
  372. ... <doc xmlns:xi="http://www.w3.org/2001/XInclude">
  373. ... <foo/>
  374. ... <xi:include href="doc/test.xml" />
  375. ... </doc>''')
  376. >>> tree = etree.parse(data)
  377. >>> tree.xinclude()
  378. >>> print(etree.tostring(tree.getroot()))
  379. <doc xmlns:xi="http://www.w3.org/2001/XInclude">
  380. <foo/>
  381. <a xml:base="doc/test.xml"/>
  382. </doc>
  383. Note that the ElementTree compatible ElementInclude_ module is also supported
  384. as ``lxml.ElementInclude``. It has the additional advantage of supporting
  385. custom `URL resolvers`_ at the Python level. The normal XInclude mechanism
  386. cannot deploy these. If you need ElementTree compatibility or custom
  387. resolvers, you have to stick to the external Python module.
  388. .. _ElementInclude: http://effbot.org/zone/element-xinclude.htm
  389. write_c14n on ElementTree
  390. -------------------------
  391. The lxml.etree.ElementTree class has a method write_c14n, which takes a file
  392. object as argument. This file object will receive an UTF-8 representation of
  393. the canonicalized form of the XML, following the W3C C14N recommendation. For
  394. example:
  395. .. sourcecode:: pycon
  396. >>> f = StringIO('<a><b/></a>')
  397. >>> tree = etree.parse(f)
  398. >>> f2 = StringIO()
  399. >>> tree.write_c14n(f2)
  400. >>> print(f2.getvalue().decode("utf-8"))
  401. <a><b></b></a>