lxml2.txt 11 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269
  1. =======================
  2. What's new in lxml 2.0?
  3. =======================
  4. .. contents::
  5. ..
  6. 1 Changes in etree and objectify
  7. 1.1 Incompatible changes
  8. 1.2 Enhancements
  9. 1.3 Deprecation
  10. 2 New modules
  11. 2.1 lxml.usedoctest
  12. 2.2 lxml.html
  13. 2.3 lxml.cssselect
  14. During the development of the lxml 1.x series, a couple of quirks were
  15. discovered in the design that made the API less obvious and its future
  16. extensions harder than necessary. lxml 2.0 is a soft evolution of lxml 1.x
  17. towards a simpler, more consistent and more powerful API - with some major
  18. extensions. Wherever possible, lxml 1.3 comes close to the semantics of lxml
  19. 2.0, so that migrating should be easier for code that currently runs with 1.3.
  20. One of the important internal changes was the switch from the Pyrex_
  21. compiler to Cython_, which provides better optimisation and improved
  22. support for newer Python language features. This allows the code of
  23. lxml to become more Python-like again, while the performance improves
  24. as Cython continues its own development. The code simplification,
  25. which will continue throughout the 2.x series, will hopefully make it
  26. even easier for users to contribute.
  27. .. _Cython: http://www.cython.org/
  28. .. _Pyrex: http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/
  29. Changes in etree and objectify
  30. ==============================
  31. A graduation towards a more consistent API cannot go without a certain amount
  32. of incompatible changes. The following is a list of those differences that
  33. applications need to take into account when migrating from lxml 1.x to lxml
  34. 2.0.
  35. Incompatible changes
  36. --------------------
  37. * lxml 0.9 introduced a feature called `namespace implementation`_. The
  38. global ``Namespace`` factory was added to register custom element classes
  39. and have lxml.etree look them up automatically. However, the later
  40. development of further class lookup mechanisms made it appear less and less
  41. adequate to register this mapping at a global level, so lxml 1.1 first
  42. removed the namespace based lookup from the default setup and lxml 2.0
  43. finally removes the global namespace registry completely. As all other
  44. lookup mechanisms, the namespace lookup is now local to a parser, including
  45. the registry itself. Applications that use a module-level parser can easily
  46. map its ``get_namespace()`` method to a global ``Namespace`` function to
  47. mimic the old behaviour.
  48. .. _`namespace implementation`: element_classes.html#implementing-namespaces
  49. * Some API functions now require passing options as keyword arguments,
  50. as opposed to positional arguments. This restriction was introduced
  51. to make the API usage independent of future extensions such as the
  52. addition of new positional arguments. Users should not rely on the
  53. position of an optional argument in function signatures and instead
  54. pass it explicitly named. This also improves code readability - it
  55. is common good practice to pass options in a consistent way
  56. independent of their position, so many people may not even notice
  57. the change in their code. Another important reason is compatibility
  58. with cElementTree, which also enforces keyword-only arguments in a
  59. couple of places.
  60. * XML tag names are validated when creating an Element. This does not
  61. apply to HTML tags, where only HTML special characters are
  62. forbidden. The distinction is made by the ``SubElement()`` factory,
  63. which tests if the tree it works on is an HTML tree, and by the
  64. ``.makeelement()`` methods of parsers, which behave differently for
  65. the ``XMLParser()`` and the ``HTMLParser()``.
  66. * XPath now raises exceptions specific to the part of the execution that
  67. failed: ``XPathSyntaxError`` for parser errors and ``XPathEvalError`` for
  68. errors that occurred during the evaluation. Note that the distinction only
  69. works for the ``XPath()`` class. The other two evaluators only have a
  70. single evaluation call that includes the parsing step, and will therefore
  71. only raise an ``XPathEvalError``. Applications can catch both exceptions
  72. through the common base class ``XPathError`` (which also exists in earlier
  73. lxml versions).
  74. * Network access in parsers is now disabled by default, i.e. the
  75. ``no_network`` option defaults to True. Due to a somewhat 'interesting'
  76. implementation in libxml2, this does not affect the first document (i.e. the
  77. URL that is parsed), but only subsequent documents, such as a DTD when
  78. parsing with validation. This means that you will have to check the URL you
  79. pass, instead of relying on lxml to prevent *any* access to external
  80. resources. As this can be helpful in some use cases, lxml does not work
  81. around it.
  82. * The type annotations in lxml.objectify (the ``pytype`` attribute) now use
  83. ``NoneType`` for the None value as this is the correct Python type name.
  84. Previously, lxml 1.x used a lower case ``none``.
  85. * Another change in objectify regards the way it deals with ambiguous types.
  86. Previously, setting a value like the string ``"3"`` through normal attribute
  87. access would let it come back as an integer when reading the object
  88. attribute. lxml 2.0 prevents this by always setting the ``pytype``
  89. attribute to the type the user passed in, so ``"3"`` will come back as a
  90. string, while the number ``3`` will come back as a number. To remove the
  91. type annotation on serialisation, you can use the ``deannotate()`` function.
  92. * The C-API function ``findOrBuildNodeNs()`` was replaced by the more generic
  93. ``findOrBuildNodeNsPrefix()`` that accepts an additional default prefix.
  94. Enhancements
  95. ------------
  96. Most of the enhancements of lxml 2.0 were made under the hood. Most people
  97. won't even notice them, but they make the maintenance of lxml easier and thus
  98. facilitate further enhancements and an improved integration between lxml's
  99. features.
  100. * lxml.objectify now has its own implementation of the `E factory`_. It uses
  101. the built-in type lookup mechanism of lxml.objectify, thus removing the need
  102. for an additional type registry mechanism (as previously available through
  103. the ``typemap`` parameter).
  104. * XML entities are supported through the ``Entity()`` factory, an Entity
  105. element class and a parser option ``resolve_entities`` that allows to keep
  106. entities in the element tree when set to False. Also, the parser will now
  107. report undefined entities as errors if it needs to resolve them (which is
  108. still the default, as in lxml 1.x).
  109. * A major part of the XPath code was rewritten and can now benefit from a
  110. bigger overlap with the XSLT code. The main benefits are improved thread
  111. safety in the XPath evaluators and Python RegExp support in standard XPath.
  112. * The string results of an XPath evaluation have become 'smart' string
  113. subclasses. Formerly, there was no easy way to find out where a
  114. string originated from. In lxml 2.0, you can call its
  115. ``getparent()`` method to `find the Element that carries it`_. This
  116. works for attributes (``//@attribute``) and for ``text()`` nodes,
  117. i.e. Element text and tails. Strings that were constructed in the
  118. path expression, e.g. by the ``string()`` function or extension
  119. functions, will return None as their parent.
  120. * Setting a ``QName`` object as value of the ``.text`` property or as
  121. an attribute value will resolve its prefix in the respective context
  122. * Following ElementTree 1.3, the ``iterfind()`` method supports
  123. efficient iteration based on XPath-like expressions.
  124. The parsers also received some major enhancements:
  125. * ``iterparse()`` can parse HTML when passing the boolean ``html``
  126. keyword.
  127. * Parse time XML Schema validation by passing an
  128. XMLSchema object to the ``schema`` keyword argument of a parser.
  129. * Support for a ``target`` object that implements ElementTree's
  130. `TreeBuilder interface`_.
  131. * The ``encoding`` keyword allows overriding the document encoding.
  132. .. _`E factory`: objectify.html#tree-generation-with-the-e-factory
  133. .. _`find the Element that carries it`: tutorial.html#using-xpath-to-find-text
  134. .. _`TreeBuilder interface`: http://effbot.org/elementtree/elementtree-treebuilder.htm
  135. Deprecation
  136. -----------
  137. The following functions and methods are now deprecated. They are
  138. still available in lxml 2.0 and will be removed in lxml 2.1:
  139. * The ``tounicode()`` function was replaced by the call
  140. ``tostring(encoding='unicode')``.
  141. * CamelCaseNamed module functions and methods were renamed to their
  142. underscore equivalents to follow `PEP 8`_ in naming.
  143. - ``etree.clearErrorLog()``, use ``etree.clear_error_log()``
  144. - ``etree.useGlobalPythonLog()``, use
  145. ``etree.use_global_python_log()``
  146. - ``etree.ElementClassLookup.setFallback()``, use
  147. ``etree.ElementClassLookup.set_fallback()``
  148. - ``etree.getDefaultParser()``, use ``etree.get_default_parser()``
  149. - ``etree.setDefaultParser()``, use ``etree.set_default_parser()``
  150. - ``etree.setElementClassLookup()``, use
  151. ``etree.set_element_class_lookup()``
  152. - ``XMLParser.setElementClassLookup()``, use ``.set_element_class_lookup()``
  153. - ``HTMLParser.setElementClassLookup()``, use ``.set_element_class_lookup()``
  154. Note that ``parser.setElementClassLookup()`` has not been removed
  155. yet, although ``parser.set_element_class_lookup()`` should be used
  156. instead.
  157. - ``xpath_evaluator.registerNamespace()``, use
  158. ``xpath_evaluator.register_namespace()``
  159. - ``xpath_evaluator.registerNamespaces()``, use
  160. ``xpath_evaluator.register_namespaces()``
  161. - ``objectify.setPytypeAttributeTag``, use
  162. ``objectify.set_pytype_attribute_tag``
  163. - ``objectify.setDefaultParser()``, use
  164. ``objectify.set_default_parser()``
  165. * The ``.getiterator()`` method on Elements and ElementTrees was
  166. renamed to ``.iter()`` to follow ElementTree 1.3.
  167. .. _`PEP 8`: http://www.python.org/dev/peps/pep-0008/
  168. New modules
  169. ===========
  170. The most visible changes in lxml 2.0 regard the new modules that were added.
  171. lxml.usedoctest
  172. ---------------
  173. A very useful module for doctests based on XML or HTML is
  174. ``lxml.doctestcompare``. It provides a relaxed comparison mechanism
  175. for XML and HTML in doctests. Using it for XML comparisons is as
  176. simple as:
  177. .. sourcecode:: pycon
  178. >>> import lxml.usedoctest
  179. and for HTML comparisons:
  180. .. sourcecode:: pycon
  181. >>> import lxml.html.usedoctest
  182. lxml.html
  183. ---------
  184. The largest new package that was added to lxml 2.0 is `lxml.html`_. It
  185. contains various tools and modules for HTML handling. The major features
  186. include support for cleaning up HTML (removing unwanted content), a readable
  187. HTML diff and various tools for working with links.
  188. .. _`lxml.html`: lxmlhtml.html
  189. lxml.cssselect
  190. --------------
  191. The Cascading Stylesheet Language (CSS_) has a very short and generic path
  192. language for pointing at elements in XML/HTML trees (`CSS selectors`_). The module
  193. lxml.cssselect_ provides an implementation based on XPath.
  194. .. _lxml.cssselect: cssselect.html
  195. .. _CSS: http://www.w3.org/Style/CSS/
  196. .. _`CSS selectors`: http://www.w3.org/TR/CSS21/selector.html