| 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283 |
- Document loading and URL resolving
- ==================================
- .. contents::
- ..
- 1 XML Catalogs
- 2 URI Resolvers
- 3 Document loading in context
- 4 I/O access control in XSLT
- The normal way to load external entities (such as DTDs) is by using
- XML catalogs. Lxml also has support for user provided document
- loaders in both the parsers and XSL transformations. These so-called
- resolvers are subclasses of the etree.Resolver class.
- ..
- >>> try: from StringIO import StringIO
- ... except ImportError:
- ... from io import BytesIO
- ... def StringIO(s):
- ... if isinstance(s, str): s = s.encode("UTF-8")
- ... return BytesIO(s)
- XML Catalogs
- ------------
- When loading an external entity for a document, e.g. a DTD, the parser
- is normally configured to prevent network access (see the
- ``no_network`` parser option). Instead, it will try to load the
- entity from their local file system path or, in the most common case
- that the entity uses a network URL as reference, from a local XML
- catalog.
- `XML catalogs`_ are the preferred and agreed-on mechanism to load
- external entities from XML processors. Most tools will use them, so
- it is worth configuring them properly on a system. Many Linux
- installations use them by default, but on other systems they may need
- to get enabled manually. The `libxml2 site`_ has some documentation
- on `how to set up XML catalogs`_
- .. _`XML catalogs`: http://www.oasis-open.org/committees/entity/spec.html
- .. _`libxml2 site`: http://xmlsoft.org/
- .. _`how to set up XML catalogs`: http://xmlsoft.org/catalog.html
- URI Resolvers
- -------------
- Here is an example of a custom resolver:
- .. sourcecode:: pycon
- >>> from lxml import etree
- >>> class DTDResolver(etree.Resolver):
- ... def resolve(self, url, id, context):
- ... print("Resolving URL '%s'" % url)
- ... return self.resolve_string(
- ... '<!ENTITY myentity "[resolved text: %s]">' % url, context)
- This defines a resolver that always returns a dynamically generated DTD
- fragment defining an entity. The ``url`` argument passes the system URL of
- the requested document, the ``id`` argument is the public ID. Note that any
- of these may be None. The context object is not normally used by client code.
- Resolving is based on the methods of the Resolver object that build
- internal representations of the result document. The following
- methods exist:
- * ``resolve_string`` takes a parsable string as result document
- * ``resolve_filename`` takes a filename
- * ``resolve_file`` takes an open file-like object that has at least a read() method
- * ``resolve_empty`` resolves into an empty document
- The ``resolve()`` method may choose to return None, in which case the next
- registered resolver (or the default resolver) is consulted. Resolving always
- terminates if ``resolve()`` returns the result of any of the above
- ``resolve_*()`` methods.
- Resolvers are registered local to a parser:
- .. sourcecode:: pycon
- >>> parser = etree.XMLParser(load_dtd=True)
- >>> parser.resolvers.add( DTDResolver() )
- Note that we instantiate a parser that loads the DTD. This is not done by the
- default parser, which does no validation. When we use this parser to parse a
- document that requires resolving a URL, it will call our custom resolver:
- .. sourcecode:: pycon
- >>> xml = '<!DOCTYPE doc SYSTEM "MissingDTD.dtd"><doc>&myentity;</doc>'
- >>> tree = etree.parse(StringIO(xml), parser)
- Resolving URL 'MissingDTD.dtd'
- >>> root = tree.getroot()
- >>> print(root.text)
- [resolved text: MissingDTD.dtd]
- The entity in the document was correctly resolved by the generated DTD
- fragment.
- Document loading in context
- ---------------------------
- XML documents memorise their initial parser (and its resolvers) during their
- life-time. This means that a lookup process related to a document will use
- the resolvers of the document's parser. We can demonstrate this with a
- resolver that only responds to a specific prefix:
- .. sourcecode:: pycon
- >>> class PrefixResolver(etree.Resolver):
- ... def __init__(self, prefix):
- ... self.prefix = prefix
- ... self.result_xml = '''\
- ... <xsl:stylesheet
- ... xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
- ... <test xmlns="testNS">%s-TEST</test>
- ... </xsl:stylesheet>
- ... ''' % prefix
- ... def resolve(self, url, pubid, context):
- ... if url.startswith(self.prefix):
- ... print("Resolved url %s as prefix %s" % (url, self.prefix))
- ... return self.resolve_string(self.result_xml, context)
- We demonstrate this in XSLT and use the following stylesheet as an example:
- .. sourcecode:: pycon
- >>> xml_text = """\
- ... <xsl:stylesheet version="1.0"
- ... xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
- ... <xsl:include href="honk:test"/>
- ... <xsl:template match="/">
- ... <test>
- ... <xsl:value-of select="document('hoi:test')/*/*/text()"/>
- ... </test>
- ... </xsl:template>
- ... </xsl:stylesheet>
- ... """
- Note that it needs to resolve two URIs: ``honk:test`` when compiling the XSLT
- document (i.e. when resolving ``xsl:import`` and ``xsl:include`` elements) and
- ``hoi:test`` at transformation time, when calls to the ``document`` function
- are resolved. If we now register different resolvers with two different
- parsers, we can parse our document twice in different resolver contexts:
- .. sourcecode:: pycon
- >>> hoi_parser = etree.XMLParser()
- >>> normal_doc = etree.parse(StringIO(xml_text), hoi_parser)
- >>> hoi_parser.resolvers.add( PrefixResolver("hoi") )
- >>> hoi_doc = etree.parse(StringIO(xml_text), hoi_parser)
- >>> honk_parser = etree.XMLParser()
- >>> honk_parser.resolvers.add( PrefixResolver("honk") )
- >>> honk_doc = etree.parse(StringIO(xml_text), honk_parser)
- These contexts are important for the further behaviour of the documents. They
- memorise their original parser so that the correct set of resolvers is used in
- subsequent lookups. To compile the stylesheet, XSLT must resolve the
- ``honk:test`` URI in the ``xsl:include`` element. The ``hoi`` resolver cannot
- do that:
- .. sourcecode:: pycon
- >>> transform = etree.XSLT(normal_doc)
- Traceback (most recent call last):
- ...
- lxml.etree.XSLTParseError: Cannot resolve URI honk:test
- >>> transform = etree.XSLT(hoi_doc)
- Traceback (most recent call last):
- ...
- lxml.etree.XSLTParseError: Cannot resolve URI honk:test
- However, if we use the ``honk`` resolver associated with the respective
- document, everything works fine:
- .. sourcecode:: pycon
- >>> transform = etree.XSLT(honk_doc)
- Resolved url honk:test as prefix honk
- Running the transform accesses the same parser context again, but since it now
- needs to resolve the ``hoi`` URI in the call to the document function, its
- ``honk`` resolver will fail to do so:
- .. sourcecode:: pycon
- >>> result = transform(normal_doc)
- Traceback (most recent call last):
- ...
- lxml.etree.XSLTApplyError: Cannot resolve URI hoi:test
- >>> result = transform(hoi_doc)
- Traceback (most recent call last):
- ...
- lxml.etree.XSLTApplyError: Cannot resolve URI hoi:test
- >>> result = transform(honk_doc)
- Traceback (most recent call last):
- ...
- lxml.etree.XSLTApplyError: Cannot resolve URI hoi:test
- This can only be solved by adding a ``hoi`` resolver to the original parser:
- .. sourcecode:: pycon
- >>> honk_parser.resolvers.add( PrefixResolver("hoi") )
- >>> result = transform(honk_doc)
- Resolved url hoi:test as prefix hoi
- >>> print(str(result)[:-1])
- <?xml version="1.0"?>
- <test>hoi-TEST</test>
- We can see that the ``hoi`` resolver was called to generate a document that
- was then inserted into the result document by the XSLT transformation. Note
- that this is completely independent of the XML file you transform, as the URI
- is resolved from within the stylesheet context:
- .. sourcecode:: pycon
- >>> result = transform(normal_doc)
- Resolved url hoi:test as prefix hoi
- >>> print(str(result)[:-1])
- <?xml version="1.0"?>
- <test>hoi-TEST</test>
- It may be seen as a matter of taste what resolvers the generated document
- inherits. For XSLT, the output document inherits the resolvers of the input
- document and not those of the stylesheet. Therefore, the last result does not
- inherit any resolvers at all.
- I/O access control in XSLT
- --------------------------
- By default, XSLT supports all extension functions from libxslt and libexslt as
- well as Python regular expressions through EXSLT. Some extensions enable
- style sheets to read and write files on the local file system.
- XSLT has a mechanism to control the access to certain I/O operations during
- the transformation process. This is most interesting where XSL scripts come
- from potentially insecure sources and must be prevented from modifying the
- local file system. Note, however, that there is no way to keep them from
- eating up your precious CPU time, so this should not stop you from thinking
- about what XSLT you execute.
- Access control is configured using the ``XSLTAccessControl`` class. It can be
- called with a number of keyword arguments that allow or deny specific
- operations:
- .. sourcecode:: pycon
- >>> transform = etree.XSLT(honk_doc)
- Resolved url honk:test as prefix honk
- >>> result = transform(normal_doc)
- Resolved url hoi:test as prefix hoi
- >>> ac = etree.XSLTAccessControl(read_network=False)
- >>> transform = etree.XSLT(honk_doc, access_control=ac)
- Resolved url honk:test as prefix honk
- >>> result = transform(normal_doc)
- Traceback (most recent call last):
- ...
- lxml.etree.XSLTApplyError: xsltLoadDocument: read rights for hoi:test denied
- There are a few things to keep in mind:
- * XSL parsing (``xsl:import``, etc.) is not affected by this mechanism
- * ``read_file=False`` does not imply ``write_file=False``, all controls are
- independent.
- * ``read_file`` only applies to files in the file system. Any other scheme
- for URLs is controlled by the ``*_network`` keywords.
- * If you need more fine-grained control than switching access on and off, you
- should consider writing a custom document loader that returns empty
- documents or raises exceptions if access is denied.
|