resolvers.txt 9.0 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257
  1. Document loading and URL resolving
  2. ==================================
  3. .. contents::
  4. ..
  5. 1 URI Resolvers
  6. 2 Document loading in context
  7. 3 I/O access control in XSLT
  8. Lxml has support for custom document loaders in both the parsers and XSL
  9. transformations. These so-called resolvers are subclasses of the
  10. etree.Resolver class.
  11. ..
  12. >>> try: from StringIO import StringIO
  13. ... except ImportError:
  14. ... from io import BytesIO
  15. ... def StringIO(s):
  16. ... if isinstance(s, str): s = s.encode("UTF-8")
  17. ... return BytesIO(s)
  18. URI Resolvers
  19. -------------
  20. Here is an example of a custom resolver:
  21. .. sourcecode:: pycon
  22. >>> from lxml import etree
  23. >>> class DTDResolver(etree.Resolver):
  24. ... def resolve(self, url, id, context):
  25. ... print("Resolving URL '%s'" % url)
  26. ... return self.resolve_string(
  27. ... '<!ENTITY myentity "[resolved text: %s]">' % url, context)
  28. This defines a resolver that always returns a dynamically generated DTD
  29. fragment defining an entity. The ``url`` argument passes the system URL of
  30. the requested document, the ``id`` argument is the public ID. Note that any
  31. of these may be None. The context object is not normally used by client code.
  32. Resolving is based on three methods of the Resolver object that build internal
  33. representations of the result document. The following methods exist:
  34. * ``resolve_string`` takes a parsable string as result document
  35. * ``resolve_filename`` takes a filename
  36. * ``resolve_file`` takes an open file-like object that has at least a read() method
  37. * ``resolve_empty`` resolves into an empty document
  38. The ``resolve()`` method may choose to return None, in which case the next
  39. registered resolver (or the default resolver) is consulted. Resolving always
  40. terminates if ``resolve()`` returns the result of any of the above
  41. ``resolve_*()`` methods.
  42. Resolvers are registered local to a parser:
  43. .. sourcecode:: pycon
  44. >>> parser = etree.XMLParser(load_dtd=True)
  45. >>> parser.resolvers.add( DTDResolver() )
  46. Note that we instantiate a parser that loads the DTD. This is not done by the
  47. default parser, which does no validation. When we use this parser to parse a
  48. document that requires resolving a URL, it will call our custom resolver:
  49. .. sourcecode:: pycon
  50. >>> xml = '<!DOCTYPE doc SYSTEM "MissingDTD.dtd"><doc>&myentity;</doc>'
  51. >>> tree = etree.parse(StringIO(xml), parser)
  52. Resolving URL 'MissingDTD.dtd'
  53. >>> root = tree.getroot()
  54. >>> print(root.text)
  55. [resolved text: MissingDTD.dtd]
  56. The entity in the document was correctly resolved by the generated DTD
  57. fragment.
  58. Document loading in context
  59. ---------------------------
  60. XML documents memorise their initial parser (and its resolvers) during their
  61. life-time. This means that a lookup process related to a document will use
  62. the resolvers of the document's parser. We can demonstrate this with a
  63. resolver that only responds to a specific prefix:
  64. .. sourcecode:: pycon
  65. >>> class PrefixResolver(etree.Resolver):
  66. ... def __init__(self, prefix):
  67. ... self.prefix = prefix
  68. ... self.result_xml = '''\
  69. ... <xsl:stylesheet
  70. ... xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  71. ... <test xmlns="testNS">%s-TEST</test>
  72. ... </xsl:stylesheet>
  73. ... ''' % prefix
  74. ... def resolve(self, url, pubid, context):
  75. ... if url.startswith(self.prefix):
  76. ... print("Resolved url %s as prefix %s" % (url, self.prefix))
  77. ... return self.resolve_string(self.result_xml, context)
  78. We demonstrate this in XSLT and use the following stylesheet as an example:
  79. .. sourcecode:: pycon
  80. >>> xml_text = """\
  81. ... <xsl:stylesheet version="1.0"
  82. ... xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  83. ... <xsl:include href="honk:test"/>
  84. ... <xsl:template match="/">
  85. ... <test>
  86. ... <xsl:value-of select="document('hoi:test')/*/*/text()"/>
  87. ... </test>
  88. ... </xsl:template>
  89. ... </xsl:stylesheet>
  90. ... """
  91. Note that it needs to resolve two URIs: ``honk:test`` when compiling the XSLT
  92. document (i.e. when resolving ``xsl:import`` and ``xsl:include`` elements) and
  93. ``hoi:test`` at transformation time, when calls to the ``document`` function
  94. are resolved. If we now register different resolvers with two different
  95. parsers, we can parse our document twice in different resolver contexts:
  96. .. sourcecode:: pycon
  97. >>> hoi_parser = etree.XMLParser()
  98. >>> normal_doc = etree.parse(StringIO(xml_text), hoi_parser)
  99. >>> hoi_parser.resolvers.add( PrefixResolver("hoi") )
  100. >>> hoi_doc = etree.parse(StringIO(xml_text), hoi_parser)
  101. >>> honk_parser = etree.XMLParser()
  102. >>> honk_parser.resolvers.add( PrefixResolver("honk") )
  103. >>> honk_doc = etree.parse(StringIO(xml_text), honk_parser)
  104. These contexts are important for the further behaviour of the documents. They
  105. memorise their original parser so that the correct set of resolvers is used in
  106. subsequent lookups. To compile the stylesheet, XSLT must resolve the
  107. ``honk:test`` URI in the ``xsl:include`` element. The ``hoi`` resolver cannot
  108. do that:
  109. .. sourcecode:: pycon
  110. >>> transform = etree.XSLT(normal_doc)
  111. Traceback (most recent call last):
  112. ...
  113. lxml.etree.XSLTParseError: Cannot resolve URI honk:test
  114. >>> transform = etree.XSLT(hoi_doc)
  115. Traceback (most recent call last):
  116. ...
  117. lxml.etree.XSLTParseError: Cannot resolve URI honk:test
  118. However, if we use the ``honk`` resolver associated with the respective
  119. document, everything works fine:
  120. .. sourcecode:: pycon
  121. >>> transform = etree.XSLT(honk_doc)
  122. Resolved url honk:test as prefix honk
  123. Running the transform accesses the same parser context again, but since it now
  124. needs to resolve the ``hoi`` URI in the call to the document function, its
  125. ``honk`` resolver will fail to do so:
  126. .. sourcecode:: pycon
  127. >>> result = transform(normal_doc)
  128. Traceback (most recent call last):
  129. ...
  130. lxml.etree.XSLTApplyError: Cannot resolve URI hoi:test
  131. >>> result = transform(hoi_doc)
  132. Traceback (most recent call last):
  133. ...
  134. lxml.etree.XSLTApplyError: Cannot resolve URI hoi:test
  135. >>> result = transform(honk_doc)
  136. Traceback (most recent call last):
  137. ...
  138. lxml.etree.XSLTApplyError: Cannot resolve URI hoi:test
  139. This can only be solved by adding a ``hoi`` resolver to the original parser:
  140. .. sourcecode:: pycon
  141. >>> honk_parser.resolvers.add( PrefixResolver("hoi") )
  142. >>> result = transform(honk_doc)
  143. Resolved url hoi:test as prefix hoi
  144. >>> print(str(result)[:-1])
  145. <?xml version="1.0"?>
  146. <test>hoi-TEST</test>
  147. We can see that the ``hoi`` resolver was called to generate a document that
  148. was then inserted into the result document by the XSLT transformation. Note
  149. that this is completely independent of the XML file you transform, as the URI
  150. is resolved from within the stylesheet context:
  151. .. sourcecode:: pycon
  152. >>> result = transform(normal_doc)
  153. Resolved url hoi:test as prefix hoi
  154. >>> print(str(result)[:-1])
  155. <?xml version="1.0"?>
  156. <test>hoi-TEST</test>
  157. It may be seen as a matter of taste what resolvers the generated document
  158. inherits. For XSLT, the output document inherits the resolvers of the input
  159. document and not those of the stylesheet. Therefore, the last result does not
  160. inherit any resolvers at all.
  161. I/O access control in XSLT
  162. --------------------------
  163. By default, XSLT supports all extension functions from libxslt and libexslt as
  164. well as Python regular expressions through EXSLT. Some extensions enable
  165. style sheets to read and write files on the local file system.
  166. XSLT has a mechanism to control the access to certain I/O operations during
  167. the transformation process. This is most interesting where XSL scripts come
  168. from potentially insecure sources and must be prevented from modifying the
  169. local file system. Note, however, that there is no way to keep them from
  170. eating up your precious CPU time, so this should not stop you from thinking
  171. about what XSLT you execute.
  172. Access control is configured using the ``XSLTAccessControl`` class. It can be
  173. called with a number of keyword arguments that allow or deny specific
  174. operations:
  175. .. sourcecode:: pycon
  176. >>> transform = etree.XSLT(honk_doc)
  177. Resolved url honk:test as prefix honk
  178. >>> result = transform(normal_doc)
  179. Resolved url hoi:test as prefix hoi
  180. >>> ac = etree.XSLTAccessControl(read_network=False)
  181. >>> transform = etree.XSLT(honk_doc, access_control=ac)
  182. Resolved url honk:test as prefix honk
  183. >>> result = transform(normal_doc)
  184. Traceback (most recent call last):
  185. ...
  186. lxml.etree.XSLTApplyError: xsltLoadDocument: read rights for hoi:test denied
  187. There are a few things to keep in mind:
  188. * XSL parsing (``xsl:import``, etc.) is not affected by this mechanism
  189. * ``read_file=False`` does not imply ``write_file=False``, all controls are
  190. independent.
  191. * ``read_file`` only applies to files in the file system. Any other scheme
  192. for URLs is controlled by the ``*_network`` keywords.
  193. * If you need more fine-grained control than switching access on and off, you
  194. should consider writing a custom document loader that returns empty
  195. documents or raises exceptions if access is denied.