xpathxslt.txt 19 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647
  1. ========================
  2. XPath and XSLT with lxml
  3. ========================
  4. lxml supports both XPath and XSLT through libxml2 and libxslt in a standards
  5. compliant way.
  6. .. contents::
  7. ..
  8. 1 XPath
  9. 1.1 The ``xpath()`` method
  10. 1.2 XPath return values
  11. 1.3 Generating XPath expressions
  12. 1.4 The ``XPath`` class
  13. 1.5 The ``XPathEvaluator`` classes
  14. 1.6 ``ETXPath``
  15. 1.7 Error handling
  16. 2 XSLT
  17. 2.1 XSLT result objects
  18. 2.2 Stylesheet parameters
  19. 2.3 The ``xslt()`` tree method
  20. 2.4 Dealing with stylesheet complexity
  21. 2.5 Profiling
  22. The usual setup procedure:
  23. .. sourcecode:: pycon
  24. >>> from lxml import etree
  25. ..
  26. >>> try: from StringIO import StringIO
  27. ... except ImportError:
  28. ... from io import BytesIO
  29. ... def StringIO(s):
  30. ... if isinstance(s, str): s = s.encode("UTF-8")
  31. ... return BytesIO(s)
  32. >>> try: unicode = __builtins__["unicode"]
  33. ... except (NameError, KeyError): unicode = str
  34. XPath
  35. =====
  36. lxml.etree supports the simple path syntax of the `find, findall and
  37. findtext`_ methods on ElementTree and Element, as known from the original
  38. ElementTree library (ElementPath_). As an lxml specific extension, these
  39. classes also provide an ``xpath()`` method that supports expressions in the
  40. complete XPath syntax, as well as `custom extension functions`_.
  41. .. _ElementPath: http://effbot.org/zone/element-xpath.htm
  42. .. _`find, findall and findtext`: http://effbot.org/zone/element.htm#searching-for-subelements
  43. .. _`custom extension functions`: extensions.html#xpath-extension-functions
  44. .. _`XSLT extension elements`: extensions.html#xslt-extension-elements
  45. There are also specialized XPath evaluator classes that are more efficient for
  46. frequent evaluation: ``XPath`` and ``XPathEvaluator``. See the `performance
  47. comparison`_ to learn when to use which. Their semantics when used on
  48. Elements and ElementTrees are the same as for the ``xpath()`` method described
  49. here.
  50. .. _`performance comparison`: performance.html#xpath
  51. The ``xpath()`` method
  52. ----------------------
  53. For ElementTree, the xpath method performs a global XPath query against the
  54. document (if absolute) or against the root node (if relative):
  55. .. sourcecode:: pycon
  56. >>> f = StringIO('<foo><bar></bar></foo>')
  57. >>> tree = etree.parse(f)
  58. >>> r = tree.xpath('/foo/bar')
  59. >>> len(r)
  60. 1
  61. >>> r[0].tag
  62. 'bar'
  63. >>> r = tree.xpath('bar')
  64. >>> r[0].tag
  65. 'bar'
  66. When ``xpath()`` is used on an Element, the XPath expression is evaluated
  67. against the element (if relative) or against the root tree (if absolute):
  68. .. sourcecode:: pycon
  69. >>> root = tree.getroot()
  70. >>> r = root.xpath('bar')
  71. >>> r[0].tag
  72. 'bar'
  73. >>> bar = root[0]
  74. >>> r = bar.xpath('/foo/bar')
  75. >>> r[0].tag
  76. 'bar'
  77. >>> tree = bar.getroottree()
  78. >>> r = tree.xpath('/foo/bar')
  79. >>> r[0].tag
  80. 'bar'
  81. The ``xpath()`` method has support for XPath variables:
  82. .. sourcecode:: pycon
  83. >>> expr = "//*[local-name() = $name]"
  84. >>> print(root.xpath(expr, name = "foo")[0].tag)
  85. foo
  86. >>> print(root.xpath(expr, name = "bar")[0].tag)
  87. bar
  88. >>> print(root.xpath("$text", text = "Hello World!"))
  89. Hello World!
  90. Optionally, you can provide a ``namespaces`` keyword argument, which should be
  91. a dictionary mapping the namespace prefixes used in the XPath expression to
  92. namespace URIs:
  93. .. sourcecode:: pycon
  94. >>> f = StringIO('''\
  95. ... <a:foo xmlns:a="http://codespeak.net/ns/test1"
  96. ... xmlns:b="http://codespeak.net/ns/test2">
  97. ... <b:bar>Text</b:bar>
  98. ... </a:foo>
  99. ... ''')
  100. >>> doc = etree.parse(f)
  101. >>> r = doc.xpath('/t:foo/b:bar',
  102. ... namespaces={'t': 'http://codespeak.net/ns/test1',
  103. ... 'b': 'http://codespeak.net/ns/test2'})
  104. >>> len(r)
  105. 1
  106. >>> r[0].tag
  107. '{http://codespeak.net/ns/test2}bar'
  108. >>> r[0].text
  109. 'Text'
  110. There is also an optional ``extensions`` argument which is used to define
  111. `custom extension functions`_ in Python that are local to this evaluation.
  112. XPath return values
  113. -------------------
  114. The return value types of XPath evaluations vary, depending on the
  115. XPath expression used:
  116. * True or False, when the XPath expression has a boolean result
  117. * a float, when the XPath expression has a numeric result (integer or float)
  118. * a 'smart' string (as described below), when the XPath expression has
  119. a string result.
  120. * a list of items, when the XPath expression has a list as result.
  121. The items may include Elements (also comments and processing
  122. instructions), strings and tuples. Text nodes and attributes in the
  123. result are returned as 'smart' string values. Namespace
  124. declarations are returned as tuples of strings: ``(prefix, URI)``.
  125. XPath string results are 'smart' in that they provide a
  126. ``getparent()`` method that knows their origin:
  127. * for attribute values, ``result.getparent()`` returns the Element
  128. that carries them. An example is ``//foo/@attribute``, where the
  129. parent would be a ``foo`` Element.
  130. * for the ``text()`` function (as in ``//text()``), it returns the
  131. Element that contains the text or tail that was returned.
  132. You can distinguish between different text origins with the boolean
  133. properties ``is_text``, ``is_tail`` and ``is_attribute``.
  134. Note that ``getparent()`` may not always return an Element. For
  135. example, the XPath functions ``string()`` and ``concat()`` will
  136. construct strings that do not have an origin. For them,
  137. ``getparent()`` will return None.
  138. There are certain cases where the smart string behaviour is
  139. undesirable. For example, it means that the tree will be kept alive
  140. by the string, which may have a considerable memory impact in the case
  141. that the string value is the only thing in the tree that is actually
  142. of interest. For these cases, you can deactivate the parental
  143. relationship using the keyword argument ``smart_strings``.
  144. >>> root = etree.XML("<root><a>TEXT</a></root>")
  145. >>> find_text = etree.XPath("//text()")
  146. >>> text = find_text(root)[0]
  147. >>> print(text)
  148. TEXT
  149. >>> print(text.getparent().text)
  150. TEXT
  151. >>> find_text = etree.XPath("//text()", smart_strings=False)
  152. >>> text = find_text(root)[0]
  153. >>> print(text)
  154. TEXT
  155. >>> hasattr(text, 'getparent')
  156. False
  157. Generating XPath expressions
  158. ----------------------------
  159. ElementTree objects have a method ``getpath(element)``, which returns a
  160. structural, absolute XPath expression to find that element:
  161. .. sourcecode:: pycon
  162. >>> a = etree.Element("a")
  163. >>> b = etree.SubElement(a, "b")
  164. >>> c = etree.SubElement(a, "c")
  165. >>> d1 = etree.SubElement(c, "d")
  166. >>> d2 = etree.SubElement(c, "d")
  167. >>> tree = etree.ElementTree(c)
  168. >>> print(tree.getpath(d2))
  169. /c/d[2]
  170. >>> tree.xpath(tree.getpath(d2)) == [d2]
  171. True
  172. The ``XPath`` class
  173. -------------------
  174. The ``XPath`` class compiles an XPath expression into a callable function:
  175. .. sourcecode:: pycon
  176. >>> root = etree.XML("<root><a><b/></a><b/></root>")
  177. >>> find = etree.XPath("//b")
  178. >>> print(find(root)[0].tag)
  179. b
  180. The compilation takes as much time as in the ``xpath()`` method, but it is
  181. done only once per class instantiation. This makes it especially efficient
  182. for repeated evaluation of the same XPath expression.
  183. Just like the ``xpath()`` method, the ``XPath`` class supports XPath
  184. variables:
  185. .. sourcecode:: pycon
  186. >>> count_elements = etree.XPath("count(//*[local-name() = $name])")
  187. >>> print(count_elements(root, name = "a"))
  188. 1.0
  189. >>> print(count_elements(root, name = "b"))
  190. 2.0
  191. This supports very efficient evaluation of modified versions of an XPath
  192. expression, as compilation is still only required once.
  193. Prefix-to-namespace mappings can be passed as second parameter:
  194. .. sourcecode:: pycon
  195. >>> root = etree.XML("<root xmlns='NS'><a><b/></a><b/></root>")
  196. >>> find = etree.XPath("//n:b", namespaces={'n':'NS'})
  197. >>> print(find(root)[0].tag)
  198. {NS}b
  199. By default, ``XPath`` supports regular expressions in the EXSLT_ namespace:
  200. .. sourcecode:: pycon
  201. >>> regexpNS = "http://exslt.org/regular-expressions"
  202. >>> find = etree.XPath("//*[re:test(., '^abc$', 'i')]",
  203. ... namespaces={'re':regexpNS})
  204. >>> root = etree.XML("<root><a>aB</a><b>aBc</b></root>")
  205. >>> print(find(root)[0].text)
  206. aBc
  207. .. _EXSLT: http://www.exslt.org/
  208. You can disable this with the boolean keyword argument ``regexp`` which
  209. defaults to True.
  210. The ``XPathEvaluator`` classes
  211. ------------------------------
  212. lxml.etree provides two other efficient XPath evaluators that work on
  213. ElementTrees or Elements respectively: ``XPathDocumentEvaluator`` and
  214. ``XPathElementEvaluator``. They are automatically selected if you use the
  215. XPathEvaluator helper for instantiation:
  216. .. sourcecode:: pycon
  217. >>> root = etree.XML("<root><a><b/></a><b/></root>")
  218. >>> xpatheval = etree.XPathEvaluator(root)
  219. >>> print(isinstance(xpatheval, etree.XPathElementEvaluator))
  220. True
  221. >>> print(xpatheval("//b")[0].tag)
  222. b
  223. This class provides efficient support for evaluating different XPath
  224. expressions on the same Element or ElementTree.
  225. ``ETXPath``
  226. -----------
  227. ElementTree supports a language named ElementPath_ in its ``find*()`` methods.
  228. One of the main differences between XPath and ElementPath is that the XPath
  229. language requires an indirection through prefixes for namespace support,
  230. whereas ElementTree uses the Clark notation (``{ns}name``) to avoid prefixes
  231. completely. The other major difference regards the capabilities of both path
  232. languages. Where XPath supports various sophisticated ways of restricting the
  233. result set through functions and boolean expressions, ElementPath only
  234. supports pure path traversal without nesting or further conditions. So, while
  235. the ElementPath syntax is self-contained and therefore easier to write and
  236. handle, XPath is much more powerful and expressive.
  237. lxml.etree bridges this gap through the class ``ETXPath``, which accepts XPath
  238. expressions with namespaces in Clark notation. It is identical to the
  239. ``XPath`` class, except for the namespace notation. Normally, you would
  240. write:
  241. .. sourcecode:: pycon
  242. >>> root = etree.XML("<root xmlns='ns'><a><b/></a><b/></root>")
  243. >>> find = etree.XPath("//p:b", namespaces={'p' : 'ns'})
  244. >>> print(find(root)[0].tag)
  245. {ns}b
  246. ``ETXPath`` allows you to change this to:
  247. .. sourcecode:: pycon
  248. >>> find = etree.ETXPath("//{ns}b")
  249. >>> print(find(root)[0].tag)
  250. {ns}b
  251. Error handling
  252. --------------
  253. lxml.etree raises exceptions when errors occur while parsing or evaluating an
  254. XPath expression:
  255. .. sourcecode:: pycon
  256. >>> find = etree.XPath("\\")
  257. Traceback (most recent call last):
  258. ...
  259. lxml.etree.XPathSyntaxError: Invalid expression
  260. lxml will also try to give you a hint what went wrong, so if you pass a more
  261. complex expression, you may get a somewhat more specific error:
  262. .. sourcecode:: pycon
  263. >>> find = etree.XPath("//*[1.1.1]")
  264. Traceback (most recent call last):
  265. ...
  266. lxml.etree.XPathSyntaxError: Invalid predicate
  267. During evaluation, lxml will emit an XPathEvalError on errors:
  268. .. sourcecode:: pycon
  269. >>> find = etree.XPath("//ns:a")
  270. >>> find(root)
  271. Traceback (most recent call last):
  272. ...
  273. lxml.etree.XPathEvalError: Undefined namespace prefix
  274. This works for the ``XPath`` class, however, the other evaluators (including
  275. the ``xpath()`` method) are one-shot operations that do parsing and evaluation
  276. in one step. They therefore raise evaluation exceptions in all cases:
  277. .. sourcecode:: pycon
  278. >>> root = etree.Element("test")
  279. >>> find = root.xpath("//*[1.1.1]")
  280. Traceback (most recent call last):
  281. ...
  282. lxml.etree.XPathEvalError: Invalid predicate
  283. >>> find = root.xpath("//ns:a")
  284. Traceback (most recent call last):
  285. ...
  286. lxml.etree.XPathEvalError: Undefined namespace prefix
  287. >>> find = root.xpath("\\")
  288. Traceback (most recent call last):
  289. ...
  290. lxml.etree.XPathEvalError: Invalid expression
  291. Note that lxml versions before 1.3 always raised an ``XPathSyntaxError`` for
  292. all errors, including evaluation errors. The best way to support older
  293. versions is to except on the superclass ``XPathError``.
  294. XSLT
  295. ====
  296. lxml.etree introduces a new class, lxml.etree.XSLT. The class can be
  297. given an ElementTree object to construct an XSLT transformer:
  298. .. sourcecode:: pycon
  299. >>> f = StringIO('''\
  300. ... <xsl:stylesheet version="1.0"
  301. ... xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  302. ... <xsl:template match="/">
  303. ... <foo><xsl:value-of select="/a/b/text()" /></foo>
  304. ... </xsl:template>
  305. ... </xsl:stylesheet>''')
  306. >>> xslt_doc = etree.parse(f)
  307. >>> transform = etree.XSLT(xslt_doc)
  308. You can then run the transformation on an ElementTree document by simply
  309. calling it, and this results in another ElementTree object:
  310. .. sourcecode:: pycon
  311. >>> f = StringIO('<a><b>Text</b></a>')
  312. >>> doc = etree.parse(f)
  313. >>> result_tree = transform(doc)
  314. By default, XSLT supports all extension functions from libxslt and
  315. libexslt as well as Python regular expressions through the `EXSLT
  316. regexp functions`_. Also see the documentation on `custom extension
  317. functions`_, `XSLT extension elements`_ and `document resolvers`_.
  318. There is a separate section on `controlling access`_ to external
  319. documents and resources.
  320. .. _`EXSLT regexp functions`: http://www.exslt.org/regexp/
  321. .. _`document resolvers`: resolvers.html
  322. .. _`controlling access`: resolvers.html#i-o-access-control-in-xslt
  323. XSLT result objects
  324. -------------------
  325. The result of an XSL transformation can be accessed like a normal ElementTree
  326. document:
  327. .. sourcecode:: pycon
  328. >>> f = StringIO('<a><b>Text</b></a>')
  329. >>> doc = etree.parse(f)
  330. >>> result = transform(doc)
  331. >>> result.getroot().text
  332. 'Text'
  333. but, as opposed to normal ElementTree objects, can also be turned into an (XML
  334. or text) string by applying the str() function:
  335. .. sourcecode:: pycon
  336. >>> str(result)
  337. '<?xml version="1.0"?>\n<foo>Text</foo>\n'
  338. The result is always a plain string, encoded as requested by the
  339. ``xsl:output`` element in the stylesheet. If you want a Python unicode string
  340. instead, you should set this encoding to ``UTF-8`` (unless the `ASCII` default
  341. is sufficient). This allows you to call the builtin ``unicode()`` function on
  342. the result:
  343. .. sourcecode:: pycon
  344. >>> unicode(result)
  345. u'<?xml version="1.0"?>\n<foo>Text</foo>\n'
  346. You can use other encodings at the cost of multiple recoding. Encodings that
  347. are not supported by Python will result in an error:
  348. .. sourcecode:: pycon
  349. >>> xslt_tree = etree.XML('''\
  350. ... <xsl:stylesheet version="1.0"
  351. ... xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  352. ... <xsl:output encoding="UCS4"/>
  353. ... <xsl:template match="/">
  354. ... <foo><xsl:value-of select="/a/b/text()" /></foo>
  355. ... </xsl:template>
  356. ... </xsl:stylesheet>''')
  357. >>> transform = etree.XSLT(xslt_tree)
  358. >>> result = transform(doc)
  359. >>> unicode(result)
  360. Traceback (most recent call last):
  361. ...
  362. LookupError: unknown encoding: UCS4
  363. Stylesheet parameters
  364. ---------------------
  365. It is possible to pass parameters, in the form of XPath expressions, to the
  366. XSLT template:
  367. .. sourcecode:: pycon
  368. >>> xslt_tree = etree.XML('''\
  369. ... <xsl:stylesheet version="1.0"
  370. ... xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  371. ... <xsl:param name="a" />
  372. ... <xsl:template match="/">
  373. ... <foo><xsl:value-of select="$a" /></foo>
  374. ... </xsl:template>
  375. ... </xsl:stylesheet>''')
  376. >>> transform = etree.XSLT(xslt_tree)
  377. >>> f = StringIO('<a><b>Text</b></a>')
  378. >>> doc = etree.parse(f)
  379. The parameters are passed as keyword parameters to the transform call.
  380. First, let's try passing in a simple integer expression:
  381. .. sourcecode:: pycon
  382. >>> result = transform(doc, a="5")
  383. >>> str(result)
  384. '<?xml version="1.0"?>\n<foo>5</foo>\n'
  385. You can use any valid XPath expression as parameter value:
  386. .. sourcecode:: pycon
  387. >>> result = transform(doc, a="/a/b/text()")
  388. >>> str(result)
  389. '<?xml version="1.0"?>\n<foo>Text</foo>\n'
  390. Passing a string expression looks like this:
  391. .. sourcecode:: pycon
  392. >>> result = transform(doc, a="'A'")
  393. >>> str(result)
  394. '<?xml version="1.0"?>\n<foo>A</foo>\n'
  395. To pass a string that (potentially) contains quotes, you can use the
  396. ``.strparam()`` class method. Note that it does not escape the
  397. string. Instead, it returns an opaque object that keeps the string
  398. value.
  399. .. sourcecode:: pycon
  400. >>> plain_string_value = etree.XSLT.strparam(
  401. ... """ It's "Monty Python" """)
  402. >>> result = transform(doc, a=plain_string_value)
  403. >>> str(result)
  404. '<?xml version="1.0"?>\n<foo> It\'s "Monty Python" </foo>\n'
  405. The ``xslt()`` tree method
  406. --------------------------
  407. There's also a convenience method on ElementTree objects for doing XSL
  408. transformations. This is less efficient if you want to apply the same XSL
  409. transformation to multiple documents, but is shorter to write for one-shot
  410. operations, as you do not have to instantiate a stylesheet yourself:
  411. .. sourcecode:: pycon
  412. >>> result = doc.xslt(xslt_tree, a="'A'")
  413. >>> str(result)
  414. '<?xml version="1.0"?>\n<foo>A</foo>\n'
  415. This is a shortcut for the following code:
  416. .. sourcecode:: pycon
  417. >>> transform = etree.XSLT(xslt_tree)
  418. >>> result = transform(doc, a="'A'")
  419. >>> str(result)
  420. '<?xml version="1.0"?>\n<foo>A</foo>\n'
  421. Dealing with stylesheet complexity
  422. ----------------------------------
  423. Some applications require a larger set of rather diverse stylesheets.
  424. lxml.etree allows you to deal with this in a number of ways. Here are
  425. some ideas to try.
  426. The most simple way to reduce the diversity is by using XSLT
  427. parameters that you pass at call time to configure the stylesheets.
  428. The ``partial()`` function in the ``functools`` module of Python 2.5
  429. may come in handy here. It allows you to bind a set of keyword
  430. arguments (i.e. stylesheet parameters) to a reference of a callable
  431. stylesheet. The same works for instances of the ``XPath()``
  432. evaluator, obviously.
  433. You may also consider creating stylesheets programmatically. Just
  434. create an XSL tree, e.g. from a parsed template, and then add or
  435. replace parts as you see fit. Passing an XSL tree into the ``XSLT()``
  436. constructor multiple times will create independent stylesheets, so
  437. later modifications of the tree will not be reflected in the already
  438. created stylesheets. This makes stylesheet generation very straight
  439. forward.
  440. A third thing to remember is the support for `custom extension
  441. functions`_ and `XSLT extension elements`_. Some things are much
  442. easier to express in XSLT than in Python, while for others it is the
  443. complete opposite. Finding the right mixture of Python code and XSL
  444. code can help a great deal in keeping applications well designed and
  445. maintainable.
  446. Profiling
  447. ---------
  448. If you want to know how your stylesheet performed, pass the ``profile_run``
  449. keyword to the transform:
  450. .. sourcecode:: pycon
  451. >>> result = transform(doc, a="/a/b/text()", profile_run=True)
  452. >>> profile = result.xslt_profile
  453. The value of the ``xslt_profile`` property is an ElementTree with profiling
  454. data about each template, similar to the following:
  455. .. sourcecode:: xml
  456. <profile>
  457. <template rank="1" match="/" name="" mode="" calls="1" time="1" average="1"/>
  458. </profile>
  459. Note that this is a read-only document. You must not move any of its elements
  460. to other documents. Please deep-copy the document if you need to modify it.
  461. If you want to free it from memory, just do:
  462. .. sourcecode:: pycon
  463. >>> del result.xslt_profile