extensions.txt 20 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621
  1. ====================================
  2. Python extensions for XPath and XSLT
  3. ====================================
  4. This document describes how to use Python extension functions in XPath
  5. and XSLT like this:
  6. .. sourcecode:: xml
  7. <xsl:value-of select="f:myPythonFunction(.//sometag)" />
  8. and extension elements in XSLT as in the following example:
  9. .. sourcecode:: xml
  10. <xsl:template match="*">
  11. <my:python-extension>
  12. <some-content />
  13. </my:python-extension>
  14. </xsl:template>
  15. .. contents::
  16. ..
  17. 1 XPath Extension functions
  18. 1.1 The FunctionNamespace
  19. 1.2 Global prefix assignment
  20. 1.3 The XPath context
  21. 1.4 Evaluators and XSLT
  22. 1.5 Evaluator-local extensions
  23. 1.6 What to return from a function
  24. 2 XSLT extension elements
  25. 2.1 Declaring extension elements
  26. 2.2 Applying XSL templates
  27. 2.3 Working with read-only elements
  28. ..
  29. >>> try: from StringIO import StringIO
  30. ... except ImportError:
  31. ... from io import BytesIO
  32. ... def StringIO(s):
  33. ... if isinstance(s, str): s = s.encode("UTF-8")
  34. ... return BytesIO(s)
  35. XPath Extension functions
  36. =========================
  37. Here is how an extension function looks like. As the first argument,
  38. it always receives a context object (see below). The other arguments
  39. are provided by the respective call in the XPath expression, one in
  40. the following examples. Any number of arguments is allowed:
  41. .. sourcecode:: pycon
  42. >>> def hello(context, a):
  43. ... return "Hello %s" % a
  44. >>> def ola(context, a):
  45. ... return "Ola %s" % a
  46. >>> def loadsofargs(context, *args):
  47. ... return "Got %d arguments." % len(args)
  48. The FunctionNamespace
  49. ---------------------
  50. In order to use a function in XPath or XSLT, it needs to have a
  51. (namespaced) name by which it can be called during evaluation. This
  52. is done using the FunctionNamespace class. For simplicity, we choose
  53. the empty namespace (None):
  54. .. sourcecode:: pycon
  55. >>> from lxml import etree
  56. >>> ns = etree.FunctionNamespace(None)
  57. >>> ns['hello'] = hello
  58. >>> ns['countargs'] = loadsofargs
  59. This registers the function `hello` with the name `hello` in the default
  60. namespace (None), and the function `loadsofargs` with the name `countargs`.
  61. Since lxml 4.1, it is preferred to use the ``FunctionNamespace`` as a decorator.
  62. Either pass an explicit function name (``@ns("countargs")``), or just use the
  63. bare decorator to register the function under its own name:
  64. .. sourcecode:: pycon
  65. >>> @ns
  66. ... def hello(context, a):
  67. ... return "Hello %s" % a
  68. Now we're going to create a document that we can run XPath expressions
  69. against:
  70. .. sourcecode:: pycon
  71. >>> root = etree.XML('<a><b>Haegar</b></a>')
  72. >>> doc = etree.ElementTree(root)
  73. Done. Now we can have XPath expressions call our new function:
  74. .. sourcecode:: pycon
  75. >>> print(root.xpath("hello('Dr. Falken')"))
  76. Hello Dr. Falken
  77. >>> print(root.xpath('hello(local-name(*))'))
  78. Hello b
  79. >>> print(root.xpath('hello(string(b))'))
  80. Hello Haegar
  81. >>> print(root.xpath('countargs(., b, ./*)'))
  82. Got 3 arguments.
  83. Note how we call both a Python function (``hello()``) and an XPath built-in
  84. function (``string()``) in exactly the same way. Normally, however, you would
  85. want to separate the two in different namespaces. The FunctionNamespace class
  86. allows you to do this:
  87. .. sourcecode:: pycon
  88. >>> ns = etree.FunctionNamespace('http://mydomain.org/myfunctions')
  89. >>> ns['hello'] = hello
  90. >>> prefixmap = {'f' : 'http://mydomain.org/myfunctions'}
  91. >>> print(root.xpath('f:hello(local-name(*))', namespaces=prefixmap))
  92. Hello b
  93. Global prefix assignment
  94. ------------------------
  95. In the last example, you had to specify a prefix for the function namespace.
  96. If you always use the same prefix for a function namespace, you can also
  97. register it with the namespace:
  98. .. sourcecode:: pycon
  99. >>> ns = etree.FunctionNamespace('http://mydomain.org/myother/functions')
  100. >>> ns.prefix = 'es'
  101. >>> ns['hello'] = ola
  102. >>> print(root.xpath('es:hello(local-name(*))'))
  103. Ola b
  104. This is a global assignment, so take care not to assign the same prefix to
  105. more than one namespace. The resulting behaviour in that case is completely
  106. undefined. It is always a good idea to consistently use the same meaningful
  107. prefix for each namespace throughout your application.
  108. The prefix assignment only works with functions and FunctionNamespace objects,
  109. not with the general Namespace object that registers element classes. The
  110. reasoning is that elements in lxml do not care about prefixes anyway, so it
  111. would rather complicate things than be of any help.
  112. The XPath context
  113. -----------------
  114. Functions get a context object as first parameter. In lxml 1.x, this value
  115. was None, but since lxml 2.0 it provides two properties: ``eval_context`` and
  116. ``context_node``. The context node is the Element where the current function
  117. is called:
  118. .. sourcecode:: pycon
  119. >>> def print_tag(context, nodes):
  120. ... print("%s: %s" % (context.context_node.tag, [ n.tag for n in nodes ]))
  121. >>> ns = etree.FunctionNamespace('http://mydomain.org/printtag')
  122. >>> ns.prefix = "pt"
  123. >>> ns["print_tag"] = print_tag
  124. >>> ignore = root.xpath("//*[pt:print_tag(.//*)]")
  125. a: ['b']
  126. b: []
  127. The ``eval_context`` is a dictionary that is local to the evaluation. It
  128. allows functions to keep state:
  129. .. sourcecode:: pycon
  130. >>> def print_context(context):
  131. ... context.eval_context[context.context_node.tag] = "done"
  132. ... print(sorted(context.eval_context.items()))
  133. >>> ns["print_context"] = print_context
  134. >>> ignore = root.xpath("//*[pt:print_context()]")
  135. [('a', 'done')]
  136. [('a', 'done'), ('b', 'done')]
  137. Evaluators and XSLT
  138. -------------------
  139. Extension functions work for all ways of evaluating XPath expressions and for
  140. XSL transformations:
  141. .. sourcecode:: pycon
  142. >>> e = etree.XPathEvaluator(doc)
  143. >>> print(e('es:hello(local-name(/a))'))
  144. Ola a
  145. >>> namespaces = {'f' : 'http://mydomain.org/myfunctions'}
  146. >>> e = etree.XPathEvaluator(doc, namespaces=namespaces)
  147. >>> print(e('f:hello(local-name(/a))'))
  148. Hello a
  149. >>> xslt = etree.XSLT(etree.XML('''
  150. ... <stylesheet version="1.0"
  151. ... xmlns="http://www.w3.org/1999/XSL/Transform"
  152. ... xmlns:es="http://mydomain.org/myother/functions">
  153. ... <output method="text" encoding="ASCII"/>
  154. ... <template match="/">
  155. ... <value-of select="es:hello(string(//b))"/>
  156. ... </template>
  157. ... </stylesheet>
  158. ... '''))
  159. >>> print(xslt(doc))
  160. Ola Haegar
  161. It is also possible to register namespaces with a single evaluator after its
  162. creation. While the following example involves no functions, the idea should
  163. still be clear:
  164. .. sourcecode:: pycon
  165. >>> f = StringIO('<a xmlns="http://mydomain.org/myfunctions" />')
  166. >>> ns_doc = etree.parse(f)
  167. >>> e = etree.XPathEvaluator(ns_doc)
  168. >>> e('/a')
  169. []
  170. This returns nothing, as we did not ask for the right namespace. When we
  171. register the namespace with the evaluator, however, we can access it via a
  172. prefix:
  173. .. sourcecode:: pycon
  174. >>> e.register_namespace('foo', 'http://mydomain.org/myfunctions')
  175. >>> e('/foo:a')[0].tag
  176. '{http://mydomain.org/myfunctions}a'
  177. Note that this prefix mapping is only known to this evaluator, as opposed to
  178. the global mapping of the FunctionNamespace objects:
  179. .. sourcecode:: pycon
  180. >>> e2 = etree.XPathEvaluator(ns_doc)
  181. >>> e2('/foo:a')
  182. Traceback (most recent call last):
  183. ...
  184. lxml.etree.XPathEvalError: Undefined namespace prefix
  185. Evaluator-local extensions
  186. --------------------------
  187. Apart from the global registration of extension functions, there is also a way
  188. of making extensions known to a single Evaluator or XSLT. All evaluators and
  189. the XSLT object accept a keyword argument ``extensions`` in their constructor.
  190. The value is a dictionary mapping (namespace, name) tuples to functions:
  191. .. sourcecode:: pycon
  192. >>> extensions = {('local-ns', 'local-hello') : hello}
  193. >>> namespaces = {'l' : 'local-ns'}
  194. >>> e = etree.XPathEvaluator(doc, namespaces=namespaces, extensions=extensions)
  195. >>> print(e('l:local-hello(string(b))'))
  196. Hello Haegar
  197. For larger numbers of extension functions, you can define classes or modules
  198. and use the ``Extension`` helper:
  199. .. sourcecode:: pycon
  200. >>> class MyExt:
  201. ... def function1(self, _, arg):
  202. ... return '1'+arg
  203. ... def function2(self, _, arg):
  204. ... return '2'+arg
  205. ... def function3(self, _, arg):
  206. ... return '3'+arg
  207. >>> ext_module = MyExt()
  208. >>> functions = ('function1', 'function2')
  209. >>> extensions = etree.Extension( ext_module, functions, ns='local-ns' )
  210. >>> e = etree.XPathEvaluator(doc, namespaces=namespaces, extensions=extensions)
  211. >>> print(e('l:function1(string(b))'))
  212. 1Haegar
  213. The optional second argument to ``Extension`` can either be a
  214. sequence of names to select from the module, a dictionary that
  215. explicitly maps function names to their XPath alter-ego or ``None``
  216. (explicitly passed) to take all available functions under their
  217. original name (if their name does not start with '_').
  218. The additional ``ns`` keyword argument takes a namespace URI or
  219. ``None`` (also if left out) for the default namespace. The following
  220. examples will therefore all do the same thing:
  221. .. sourcecode:: pycon
  222. >>> functions = ('function1', 'function2', 'function3')
  223. >>> extensions = etree.Extension( ext_module, functions )
  224. >>> e = etree.XPathEvaluator(doc, extensions=extensions)
  225. >>> print(e('function1(function2(function3(string(b))))'))
  226. 123Haegar
  227. >>> extensions = etree.Extension( ext_module, functions, ns=None )
  228. >>> e = etree.XPathEvaluator(doc, extensions=extensions)
  229. >>> print(e('function1(function2(function3(string(b))))'))
  230. 123Haegar
  231. >>> extensions = etree.Extension(ext_module)
  232. >>> e = etree.XPathEvaluator(doc, extensions=extensions)
  233. >>> print(e('function1(function2(function3(string(b))))'))
  234. 123Haegar
  235. >>> functions = {
  236. ... 'function1' : 'function1',
  237. ... 'function2' : 'function2',
  238. ... 'function3' : 'function3'
  239. ... }
  240. >>> extensions = etree.Extension(ext_module, functions)
  241. >>> e = etree.XPathEvaluator(doc, extensions=extensions)
  242. >>> print(e('function1(function2(function3(string(b))))'))
  243. 123Haegar
  244. For convenience, you can also pass a sequence of extensions:
  245. .. sourcecode:: pycon
  246. >>> extensions1 = etree.Extension(ext_module)
  247. >>> extensions2 = etree.Extension(ext_module, ns='local-ns')
  248. >>> e = etree.XPathEvaluator(doc, extensions=[extensions1, extensions2],
  249. ... namespaces=namespaces)
  250. >>> print(e('function1(l:function2(function3(string(b))))'))
  251. 123Haegar
  252. What to return from a function
  253. ------------------------------
  254. .. _`XPath return values`: xpathxslt.html#xpath-return-values
  255. Extension functions can return any data type for which there is an XPath
  256. equivalent (see the documentation on `XPath return values`). This includes
  257. numbers, boolean values, elements and lists of elements. Note that integers
  258. will also be returned as floats:
  259. .. sourcecode:: pycon
  260. >>> def returnsFloat(_):
  261. ... return 1.7
  262. >>> def returnsInteger(_):
  263. ... return 1
  264. >>> def returnsBool(_):
  265. ... return True
  266. >>> def returnFirstNode(_, nodes):
  267. ... return nodes[0]
  268. >>> ns = etree.FunctionNamespace(None)
  269. >>> ns['float'] = returnsFloat
  270. >>> ns['int'] = returnsInteger
  271. >>> ns['bool'] = returnsBool
  272. >>> ns['first'] = returnFirstNode
  273. >>> e = etree.XPathEvaluator(doc)
  274. >>> e("float()")
  275. 1.7
  276. >>> e("int()")
  277. 1.0
  278. >>> int( e("int()") )
  279. 1
  280. >>> e("bool()")
  281. True
  282. >>> e("count(first(//b))")
  283. 1.0
  284. As the last example shows, you can pass the results of functions back into
  285. the XPath expression. Elements and sequences of elements are treated as
  286. XPath node-sets:
  287. .. sourcecode:: pycon
  288. >>> def returnsNodeSet(_):
  289. ... results1 = etree.Element('results1')
  290. ... etree.SubElement(results1, 'result').text = "Alpha"
  291. ... etree.SubElement(results1, 'result').text = "Beta"
  292. ...
  293. ... results2 = etree.Element('results2')
  294. ... etree.SubElement(results2, 'result').text = "Gamma"
  295. ... etree.SubElement(results2, 'result').text = "Delta"
  296. ...
  297. ... results3 = etree.SubElement(results2, 'subresult')
  298. ... return [results1, results2, results3]
  299. >>> ns['new-node-set'] = returnsNodeSet
  300. >>> e = etree.XPathEvaluator(doc)
  301. >>> r = e("new-node-set()/result")
  302. >>> print([ t.text for t in r ])
  303. ['Alpha', 'Beta', 'Gamma', 'Delta']
  304. >>> r = e("new-node-set()")
  305. >>> print([ t.tag for t in r ])
  306. ['results1', 'results2', 'subresult']
  307. >>> print([ len(t) for t in r ])
  308. [2, 3, 0]
  309. >>> r[0][0].text
  310. 'Alpha'
  311. >>> etree.tostring(r[0])
  312. b'<results1><result>Alpha</result><result>Beta</result></results1>'
  313. >>> etree.tostring(r[1])
  314. b'<results2><result>Gamma</result><result>Delta</result><subresult/></results2>'
  315. >>> etree.tostring(r[2])
  316. b'<subresult/>'
  317. The current implementation deep-copies newly created elements in node-sets.
  318. Only the elements and their children are passed on, no outlying parents or
  319. tail texts will be available in the result. This also means that in the above
  320. example, the `subresult` elements in `results2` and `results3` are no longer
  321. identical within the node-set, they belong to independent trees:
  322. .. sourcecode:: pycon
  323. >>> print("%s - %s" % (r[1][-1].tag, r[2].tag))
  324. subresult - subresult
  325. >>> print(r[1][-1] == r[2])
  326. False
  327. >>> print(r[1][-1].getparent().tag)
  328. results2
  329. >>> print(r[2].getparent())
  330. None
  331. This is an implementation detail that you should be aware of, but you should
  332. avoid relying on it in your code. Note that elements taken from the source
  333. document (the most common case) do not suffer from this restriction. They
  334. will always be passed unchanged.
  335. XSLT extension elements
  336. =======================
  337. Just like the XPath extension functions described above, lxml supports
  338. custom extension *elements* in XSLT. This means, you can write XSLT
  339. code like this:
  340. .. sourcecode:: xml
  341. <xsl:template match="*">
  342. <my:python-extension>
  343. <some-content />
  344. </my:python-extension>
  345. </xsl:template>
  346. And then you can implement the element in Python like this:
  347. .. sourcecode:: pycon
  348. >>> class MyExtElement(etree.XSLTExtension):
  349. ... def execute(self, context, self_node, input_node, output_parent):
  350. ... print("Hello from XSLT!")
  351. ... output_parent.text = "I did it!"
  352. ... # just copy own content input to output
  353. ... output_parent.extend( list(self_node) )
  354. The arguments passed to the ``.execute()`` method are
  355. context
  356. The opaque evaluation context. You need this when calling back
  357. into the XSLT processor.
  358. self_node
  359. A read-only Element object that represents the extension element
  360. in the stylesheet.
  361. input_node
  362. The current context Element in the input document (also read-only).
  363. output_parent
  364. The current insertion point in the output document. You can
  365. append elements or set the text value (not the tail). Apart from
  366. that, the Element is read-only.
  367. Declaring extension elements
  368. ----------------------------
  369. In XSLT, extension elements can be used like any other XSLT element,
  370. except that they must be declared as extensions using the standard
  371. XSLT ``extension-element-prefixes`` option:
  372. .. sourcecode:: pycon
  373. >>> xslt_ext_tree = etree.XML('''
  374. ... <xsl:stylesheet version="1.0"
  375. ... xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  376. ... xmlns:my="testns"
  377. ... extension-element-prefixes="my">
  378. ... <xsl:template match="/">
  379. ... <foo><my:ext><child>XYZ</child></my:ext></foo>
  380. ... </xsl:template>
  381. ... <xsl:template match="child">
  382. ... <CHILD>--xyz--</CHILD>
  383. ... </xsl:template>
  384. ... </xsl:stylesheet>''')
  385. To register the extension, add its namespace and name to the extension
  386. mapping of the XSLT object:
  387. .. sourcecode:: pycon
  388. >>> my_extension = MyExtElement()
  389. >>> extensions = { ('testns', 'ext') : my_extension }
  390. >>> transform = etree.XSLT(xslt_ext_tree, extensions = extensions)
  391. Note how we pass an instance here, not the class of the extension.
  392. Now we can run the transformation and see how our extension is
  393. called:
  394. .. sourcecode:: pycon
  395. >>> root = etree.XML('<dummy/>')
  396. >>> result = transform(root)
  397. Hello from XSLT!
  398. >>> str(result)
  399. '<?xml version="1.0"?>\n<foo>I did it!<child>XYZ</child></foo>\n'
  400. Applying XSL templates
  401. ----------------------
  402. XSLT extensions are a very powerful feature that allows you to
  403. interact directly with the XSLT processor. You have full read-only
  404. access to the input document and the stylesheet, and you can even call
  405. back into the XSLT processor to process templates. Here is an example
  406. that passes an Element into the ``.apply_templates()`` method of the
  407. ``XSLTExtension`` instance:
  408. .. sourcecode:: pycon
  409. >>> class MyExtElement(etree.XSLTExtension):
  410. ... def execute(self, context, self_node, input_node, output_parent):
  411. ... child = self_node[0]
  412. ... results = self.apply_templates(context, child)
  413. ... output_parent.append(results[0])
  414. >>> my_extension = MyExtElement()
  415. >>> extensions = { ('testns', 'ext') : my_extension }
  416. >>> transform = etree.XSLT(xslt_ext_tree, extensions = extensions)
  417. >>> root = etree.XML('<dummy/>')
  418. >>> result = transform(root)
  419. >>> str(result)
  420. '<?xml version="1.0"?>\n<foo><CHILD>--xyz--</CHILD></foo>\n'
  421. Here, we applied the templates to a child of the extension element
  422. itself, i.e. to an element inside the stylesheet instead of an element
  423. of the input document.
  424. The return value of ``.apply_templates()`` is always a list. It may
  425. contain a mix of elements and strings, collected from the XSLT processing
  426. result. If you want to append these values to the output parent, be aware
  427. that you cannot use the ``.append()`` method to add strings. In many
  428. cases, you would only be interested in elements anyway, so you can discard
  429. strings (e.g. formatting whitespace) and append the rest.
  430. If you want to include string results in the output, you can either build
  431. an appropriate tree yourself and append that, or you can manually add the
  432. string values to the current output tree, e.g. by concatenating them with
  433. the ``.tail`` of the last element that was appended.
  434. Note that you can also let lxml build the result tree for you by passing
  435. the ``output_parent`` into the ``.apply_templates()`` method. In this
  436. case, the result will be None and all content found by applying templates
  437. will be appended to the output parent.
  438. If you do not care about string results at all, e.g. because you already
  439. know that they will only contain whitespace, you can pass the option
  440. ``elements_only=True`` to the ``.apply_templates()`` method, or pass
  441. ``remove_blank_text=True`` to remove only those strings that consist
  442. entirely of whitespace.
  443. Working with read-only elements
  444. -------------------------------
  445. There is one important thing to keep in mind: all Elements that the
  446. ``execute()`` method gets to deal with are read-only Elements, so you
  447. cannot modify them. They also will not easily work in the API. For
  448. example, you cannot pass them to the ``tostring()`` function or wrap
  449. them in an ``ElementTree``.
  450. What you can do, however, is to deepcopy them to make them normal
  451. Elements, and then modify them using the normal etree API. So this
  452. will work:
  453. .. sourcecode:: pycon
  454. >>> from copy import deepcopy
  455. >>> class MyExtElement(etree.XSLTExtension):
  456. ... def execute(self, context, self_node, input_node, output_parent):
  457. ... child = deepcopy(self_node[0])
  458. ... child.text = "NEW TEXT"
  459. ... output_parent.append(child)
  460. >>> my_extension = MyExtElement()
  461. >>> extensions = { ('testns', 'ext') : my_extension }
  462. >>> transform = etree.XSLT(xslt_ext_tree, extensions = extensions)
  463. >>> root = etree.XML('<dummy/>')
  464. >>> result = transform(root)
  465. >>> str(result)
  466. '<?xml version="1.0"?>\n<foo><child>NEW TEXT</child></foo>\n'