resolvers.txt 10 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283
  1. Document loading and URL resolving
  2. ==================================
  3. .. contents::
  4. ..
  5. 1 XML Catalogs
  6. 2 URI Resolvers
  7. 3 Document loading in context
  8. 4 I/O access control in XSLT
  9. The normal way to load external entities (such as DTDs) is by using
  10. XML catalogs. Lxml also has support for user provided document
  11. loaders in both the parsers and XSL transformations. These so-called
  12. resolvers are subclasses of the etree.Resolver class.
  13. ..
  14. >>> try: from StringIO import StringIO
  15. ... except ImportError:
  16. ... from io import BytesIO
  17. ... def StringIO(s):
  18. ... if isinstance(s, str): s = s.encode("UTF-8")
  19. ... return BytesIO(s)
  20. XML Catalogs
  21. ------------
  22. When loading an external entity for a document, e.g. a DTD, the parser
  23. is normally configured to prevent network access (see the
  24. ``no_network`` parser option). Instead, it will try to load the
  25. entity from their local file system path or, in the most common case
  26. that the entity uses a network URL as reference, from a local XML
  27. catalog.
  28. `XML catalogs`_ are the preferred and agreed-on mechanism to load
  29. external entities from XML processors. Most tools will use them, so
  30. it is worth configuring them properly on a system. Many Linux
  31. installations use them by default, but on other systems they may need
  32. to get enabled manually. The `libxml2 site`_ has some documentation
  33. on `how to set up XML catalogs`_
  34. .. _`XML catalogs`: http://www.oasis-open.org/committees/entity/spec.html
  35. .. _`libxml2 site`: http://xmlsoft.org/
  36. .. _`how to set up XML catalogs`: http://xmlsoft.org/catalog.html
  37. URI Resolvers
  38. -------------
  39. Here is an example of a custom resolver:
  40. .. sourcecode:: pycon
  41. >>> from lxml import etree
  42. >>> class DTDResolver(etree.Resolver):
  43. ... def resolve(self, url, id, context):
  44. ... print("Resolving URL '%s'" % url)
  45. ... return self.resolve_string(
  46. ... '<!ENTITY myentity "[resolved text: %s]">' % url, context)
  47. This defines a resolver that always returns a dynamically generated DTD
  48. fragment defining an entity. The ``url`` argument passes the system URL of
  49. the requested document, the ``id`` argument is the public ID. Note that any
  50. of these may be None. The context object is not normally used by client code.
  51. Resolving is based on the methods of the Resolver object that build
  52. internal representations of the result document. The following
  53. methods exist:
  54. * ``resolve_string`` takes a parsable string as result document
  55. * ``resolve_filename`` takes a filename
  56. * ``resolve_file`` takes an open file-like object that has at least a read() method
  57. * ``resolve_empty`` resolves into an empty document
  58. The ``resolve()`` method may choose to return None, in which case the next
  59. registered resolver (or the default resolver) is consulted. Resolving always
  60. terminates if ``resolve()`` returns the result of any of the above
  61. ``resolve_*()`` methods.
  62. Resolvers are registered local to a parser:
  63. .. sourcecode:: pycon
  64. >>> parser = etree.XMLParser(load_dtd=True)
  65. >>> parser.resolvers.add( DTDResolver() )
  66. Note that we instantiate a parser that loads the DTD. This is not done by the
  67. default parser, which does no validation. When we use this parser to parse a
  68. document that requires resolving a URL, it will call our custom resolver:
  69. .. sourcecode:: pycon
  70. >>> xml = '<!DOCTYPE doc SYSTEM "MissingDTD.dtd"><doc>&myentity;</doc>'
  71. >>> tree = etree.parse(StringIO(xml), parser)
  72. Resolving URL 'MissingDTD.dtd'
  73. >>> root = tree.getroot()
  74. >>> print(root.text)
  75. [resolved text: MissingDTD.dtd]
  76. The entity in the document was correctly resolved by the generated DTD
  77. fragment.
  78. Document loading in context
  79. ---------------------------
  80. XML documents memorise their initial parser (and its resolvers) during their
  81. life-time. This means that a lookup process related to a document will use
  82. the resolvers of the document's parser. We can demonstrate this with a
  83. resolver that only responds to a specific prefix:
  84. .. sourcecode:: pycon
  85. >>> class PrefixResolver(etree.Resolver):
  86. ... def __init__(self, prefix):
  87. ... self.prefix = prefix
  88. ... self.result_xml = '''\
  89. ... <xsl:stylesheet
  90. ... xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  91. ... <test xmlns="testNS">%s-TEST</test>
  92. ... </xsl:stylesheet>
  93. ... ''' % prefix
  94. ... def resolve(self, url, pubid, context):
  95. ... if url.startswith(self.prefix):
  96. ... print("Resolved url %s as prefix %s" % (url, self.prefix))
  97. ... return self.resolve_string(self.result_xml, context)
  98. We demonstrate this in XSLT and use the following stylesheet as an example:
  99. .. sourcecode:: pycon
  100. >>> xml_text = """\
  101. ... <xsl:stylesheet version="1.0"
  102. ... xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  103. ... <xsl:include href="honk:test"/>
  104. ... <xsl:template match="/">
  105. ... <test>
  106. ... <xsl:value-of select="document('hoi:test')/*/*/text()"/>
  107. ... </test>
  108. ... </xsl:template>
  109. ... </xsl:stylesheet>
  110. ... """
  111. Note that it needs to resolve two URIs: ``honk:test`` when compiling the XSLT
  112. document (i.e. when resolving ``xsl:import`` and ``xsl:include`` elements) and
  113. ``hoi:test`` at transformation time, when calls to the ``document`` function
  114. are resolved. If we now register different resolvers with two different
  115. parsers, we can parse our document twice in different resolver contexts:
  116. .. sourcecode:: pycon
  117. >>> hoi_parser = etree.XMLParser()
  118. >>> normal_doc = etree.parse(StringIO(xml_text), hoi_parser)
  119. >>> hoi_parser.resolvers.add( PrefixResolver("hoi") )
  120. >>> hoi_doc = etree.parse(StringIO(xml_text), hoi_parser)
  121. >>> honk_parser = etree.XMLParser()
  122. >>> honk_parser.resolvers.add( PrefixResolver("honk") )
  123. >>> honk_doc = etree.parse(StringIO(xml_text), honk_parser)
  124. These contexts are important for the further behaviour of the documents. They
  125. memorise their original parser so that the correct set of resolvers is used in
  126. subsequent lookups. To compile the stylesheet, XSLT must resolve the
  127. ``honk:test`` URI in the ``xsl:include`` element. The ``hoi`` resolver cannot
  128. do that:
  129. .. sourcecode:: pycon
  130. >>> transform = etree.XSLT(normal_doc)
  131. Traceback (most recent call last):
  132. ...
  133. lxml.etree.XSLTParseError: Cannot resolve URI honk:test
  134. >>> transform = etree.XSLT(hoi_doc)
  135. Traceback (most recent call last):
  136. ...
  137. lxml.etree.XSLTParseError: Cannot resolve URI honk:test
  138. However, if we use the ``honk`` resolver associated with the respective
  139. document, everything works fine:
  140. .. sourcecode:: pycon
  141. >>> transform = etree.XSLT(honk_doc)
  142. Resolved url honk:test as prefix honk
  143. Running the transform accesses the same parser context again, but since it now
  144. needs to resolve the ``hoi`` URI in the call to the document function, its
  145. ``honk`` resolver will fail to do so:
  146. .. sourcecode:: pycon
  147. >>> result = transform(normal_doc)
  148. Traceback (most recent call last):
  149. ...
  150. lxml.etree.XSLTApplyError: Cannot resolve URI hoi:test
  151. >>> result = transform(hoi_doc)
  152. Traceback (most recent call last):
  153. ...
  154. lxml.etree.XSLTApplyError: Cannot resolve URI hoi:test
  155. >>> result = transform(honk_doc)
  156. Traceback (most recent call last):
  157. ...
  158. lxml.etree.XSLTApplyError: Cannot resolve URI hoi:test
  159. This can only be solved by adding a ``hoi`` resolver to the original parser:
  160. .. sourcecode:: pycon
  161. >>> honk_parser.resolvers.add( PrefixResolver("hoi") )
  162. >>> result = transform(honk_doc)
  163. Resolved url hoi:test as prefix hoi
  164. >>> print(str(result)[:-1])
  165. <?xml version="1.0"?>
  166. <test>hoi-TEST</test>
  167. We can see that the ``hoi`` resolver was called to generate a document that
  168. was then inserted into the result document by the XSLT transformation. Note
  169. that this is completely independent of the XML file you transform, as the URI
  170. is resolved from within the stylesheet context:
  171. .. sourcecode:: pycon
  172. >>> result = transform(normal_doc)
  173. Resolved url hoi:test as prefix hoi
  174. >>> print(str(result)[:-1])
  175. <?xml version="1.0"?>
  176. <test>hoi-TEST</test>
  177. It may be seen as a matter of taste what resolvers the generated document
  178. inherits. For XSLT, the output document inherits the resolvers of the input
  179. document and not those of the stylesheet. Therefore, the last result does not
  180. inherit any resolvers at all.
  181. I/O access control in XSLT
  182. --------------------------
  183. By default, XSLT supports all extension functions from libxslt and libexslt as
  184. well as Python regular expressions through EXSLT. Some extensions enable
  185. style sheets to read and write files on the local file system.
  186. XSLT has a mechanism to control the access to certain I/O operations during
  187. the transformation process. This is most interesting where XSL scripts come
  188. from potentially insecure sources and must be prevented from modifying the
  189. local file system. Note, however, that there is no way to keep them from
  190. eating up your precious CPU time, so this should not stop you from thinking
  191. about what XSLT you execute.
  192. Access control is configured using the ``XSLTAccessControl`` class. It can be
  193. called with a number of keyword arguments that allow or deny specific
  194. operations:
  195. .. sourcecode:: pycon
  196. >>> transform = etree.XSLT(honk_doc)
  197. Resolved url honk:test as prefix honk
  198. >>> result = transform(normal_doc)
  199. Resolved url hoi:test as prefix hoi
  200. >>> ac = etree.XSLTAccessControl(read_network=False)
  201. >>> transform = etree.XSLT(honk_doc, access_control=ac)
  202. Resolved url honk:test as prefix honk
  203. >>> result = transform(normal_doc)
  204. Traceback (most recent call last):
  205. ...
  206. lxml.etree.XSLTApplyError: xsltLoadDocument: read rights for hoi:test denied
  207. There are a few things to keep in mind:
  208. * XSL parsing (``xsl:import``, etc.) is not affected by this mechanism
  209. * ``read_file=False`` does not imply ``write_file=False``, all controls are
  210. independent.
  211. * ``read_file`` only applies to files in the file system. Any other scheme
  212. for URLs is controlled by the ``*_network`` keywords.
  213. * If you need more fine-grained control than switching access on and off, you
  214. should consider writing a custom document loader that returns empty
  215. documents or raises exceptions if access is denied.