lxml-source-howto.txt 11 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313
  1. ==============================
  2. How to read the source of lxml
  3. ==============================
  4. :Author:
  5. Stefan Behnel
  6. .. meta::
  7. :description: How to read and work on the source code of lxml
  8. :keywords: lxml, XML, Cython, source code, develop, comprehend, understand
  9. This document describes how to read the source code of lxml_ and how
  10. to start working on it. You might also be interested in the companion
  11. document that describes `how to build lxml from sources`_.
  12. .. _lxml: http://lxml.de/
  13. .. _`how to build lxml from sources`: build.html
  14. .. _`ReStructured Text`: http://docutils.sourceforge.net/rst.html
  15. .. _epydoc: http://epydoc.sourceforge.net/
  16. .. _docutils: http://docutils.sourceforge.net/
  17. .. _`C-level API`: capi.html
  18. .. contents::
  19. ..
  20. 1 What is Cython?
  21. 2 Where to start?
  22. 2.1 Concepts
  23. 2.2 The documentation
  24. 3 lxml.etree
  25. 4 lxml.objectify
  26. 5 lxml.html
  27. What is Cython?
  28. ===============
  29. .. _Cython: http://cython.org/
  30. .. _Pyrex: http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/
  31. Cython_ is the language that lxml is written in. It is a very
  32. Python-like language that was specifically designed for writing Python
  33. extension modules.
  34. The reason why Cython (or actually its predecessor Pyrex_ at the time)
  35. was chosen as an implementation language for lxml, is that it makes it
  36. very easy to interface with both the Python world and external C code.
  37. Cython generates all the necessary glue code for the Python API,
  38. including Python types, calling conventions and reference counting.
  39. On the other side of the table, calling into C code is not more than
  40. declaring the signature of the function and maybe some variables as
  41. being C types, pointers or structs, and then calling it. The rest of
  42. the code is just plain Python code.
  43. The Cython language is so close to Python that the Cython compiler can
  44. actually compile many, many Python programs to C without major
  45. modifications. But the real speed gains of a C compilation come from
  46. type annotations that were added to the language and that allow Cython
  47. to generate very efficient C code.
  48. Even if you are not familiar with Cython, you should keep in mind that
  49. a slow implementation of a feature is better than none. So, if you
  50. want to contribute and have an idea what code you want to write, feel
  51. free to start with a pure Python implementation. Chances are, if you
  52. get the change officially accepted and integrated, others will take
  53. the time to optimise it so that it runs fast in Cython.
  54. Where to start?
  55. ===============
  56. First of all, read `how to build lxml from sources`_ to learn how to
  57. retrieve the source code from the Subversion repository and how to
  58. build it. The source code lives in the subdirectory ``src`` of the
  59. checkout.
  60. The main extension modules in lxml are ``lxml.etree`` and
  61. ``lxml.objectify``. All main modules have the file extension
  62. ``.pyx``, which shows the descendence from Pyrex. As usual in Python,
  63. the main files start with a short description and a couple of imports.
  64. Cython distinguishes between the run-time ``import`` statement (as
  65. known from Python) and the compile-time ``cimport`` statement, which
  66. imports C declarations, either from external libraries or from other
  67. Cython modules.
  68. Concepts
  69. --------
  70. lxml's tree API is based on proxy objects. That means, every Element
  71. object (or rather ``_Element`` object) is a proxy for a libxml2 node
  72. structure. The class declaration is (mainly)::
  73. cdef class _Element:
  74. cdef _Document _doc
  75. cdef xmlNode* _c_node
  76. It is a naming convention that C variables and C level class members
  77. that are passed into libxml2 start with a prefixed ``c_`` (commonly
  78. libxml2 struct pointers), and that C level class members are prefixed
  79. with an underscore. So you will often see names like ``c_doc`` for an
  80. ``xmlDoc*`` variable (or ``c_node`` for an ``xmlNode*``), or the above
  81. ``_c_node`` for a class member that points to an ``xmlNode`` struct
  82. (or ``_c_doc`` for an ``xmlDoc*``).
  83. It is important to know that every proxy in lxml has a factory
  84. function that properly sets up C level members. Proxy objects must
  85. *never* be instantiated outside of that factory. For example, to
  86. instantiate an _Element object or its subclasses, you must always call
  87. its factory function::
  88. cdef xmlNode* c_node
  89. cdef _Document doc
  90. cdef _Element element
  91. ...
  92. element = _elementFactory(doc, c_node)
  93. A good place to see how this factory is used are the Element methods
  94. ``getparent()``, ``getnext()`` and ``getprevious()``.
  95. The documentation
  96. -----------------
  97. An important part of lxml is the documentation that lives in the
  98. ``doc`` directory. It describes a large part of the API and comprises
  99. a lot of example code in the form of doctests.
  100. The documentation is written in the `ReStructured Text`_ format, a
  101. very powerful text markup language that looks almost like plain text.
  102. It is part of the docutils_ package.
  103. The project web site of lxml_ is completely generated from these text
  104. documents. Even the side menu is just collected from the table of
  105. contents that the ReST processor writes into each HTML page.
  106. Obviously, we use lxml for this.
  107. The easiest way to generate the HTML pages is by calling::
  108. make html
  109. This will call the script ``doc/mkhtml.py`` to run the ReST processor
  110. on the files. After generating an HTML page the script parses it back
  111. in to build the side menu, and injects the complete menu into each
  112. page at the very end.
  113. Running the ``make`` command will also generate the API documentation
  114. if you have epydoc_ installed. The epydoc package will import and
  115. introspect the extension modules and also introspect and parse the
  116. Python modules of lxml. The aggregated information will then be
  117. written out into an HTML documentation site.
  118. lxml.etree
  119. ==========
  120. The main module, ``lxml.etree``, is in the file `lxml.etree.pyx
  121. <https://github.com/lxml/lxml/blob/master/src/lxml/lxml.etree.pyx>`_. It
  122. implements the main functions and types of the ElementTree API, as
  123. well as all the factory functions for proxies. It is the best place
  124. to start if you want to find out how a specific feature is
  125. implemented.
  126. At the very end of the file, it contains a series of ``include``
  127. statements that merge the rest of the implementation into the
  128. generated C code. Yes, you read right: no importing, no source file
  129. namespacing, just plain good old include and a huge C code result of
  130. more than 100,000 lines that we throw right into the C compiler.
  131. The main include files are:
  132. apihelpers.pxi
  133. Private C helper functions. Except for the factory functions,
  134. most of the little functions that are used all over the place are
  135. defined here. This includes things like reading out the text
  136. content of a libxml2 tree node, checking input from the API level,
  137. creating a new Element node or handling attribute values. If you
  138. want to work on the lxml code, you should keep these functions in
  139. the back of your head, as they will definitely make your life
  140. easier.
  141. classlookup.pxi
  142. Element class lookup mechanisms. The main API and engines for
  143. those who want to define custom Element classes and inject them
  144. into lxml.
  145. docloader.pxi
  146. Support for custom document loaders. Base class and registry for
  147. custom document resolvers.
  148. extensions.pxi
  149. Infrastructure for extension functions in XPath/XSLT, including
  150. XPath value conversion and function registration.
  151. iterparse.pxi
  152. Incremental XML parsing. An iterator class that builds iterparse
  153. events while parsing.
  154. nsclasses.pxi
  155. Namespace implementation and registry. The registry and engine
  156. for Element classes that use the ElementNamespaceClassLookup
  157. scheme.
  158. parser.pxi
  159. Parsers for XML and HTML. This is the main parser engine. It's
  160. the reason why you can parse a document from various sources in
  161. two lines of Python code. It's definitely not the right place to
  162. start reading lxml's soure code.
  163. parsertarget.pxi
  164. An ElementTree compatible parser target implementation based on
  165. the SAX2 interface of libxml2.
  166. proxy.pxi
  167. Very low-level functions for memory allocation/deallocation
  168. and Element proxy handling. Ignoring this for the beginning
  169. will safe your head from exploding.
  170. public-api.pxi
  171. The set of C functions that are exported to other extension
  172. modules at the C level. For example, ``lxml.objectify`` makes use
  173. of these. See the `C-level API` documentation.
  174. readonlytree.pxi
  175. A separate read-only implementation of the Element API. This is
  176. used in places where non-intrusive access to a tree is required,
  177. such as the ``PythonElementClassLookup`` or XSLT extension
  178. elements.
  179. saxparser.pxi
  180. SAX-like parser interfaces as known from ElementTree's TreeBuilder.
  181. serializer.pxi
  182. XML output functions. Basically everything that creates byte
  183. sequences from XML trees.
  184. xinclude.pxi
  185. XInclude support.
  186. xmlerror.pxi
  187. Error log handling. All error messages that libxml2 generates
  188. internally walk through the code in this file to end up in lxml's
  189. Python level error logs.
  190. At the end of the file, you will find a long list of named error
  191. codes. It is generated from the libxml2 HTML documentation (using
  192. lxml, of course). See the script ``update-error-constants.py``
  193. for this.
  194. xmlid.pxi
  195. XMLID and IDDict, a dictionary-like way to find Elements by their
  196. XML-ID attribute.
  197. xpath.pxi
  198. XPath evaluators.
  199. xslt.pxi
  200. XSL transformations, including the ``XSLT`` class, document lookup
  201. handling and access control.
  202. The different schema languages (DTD, RelaxNG, XML Schema and
  203. Schematron) are implemented in the following include files:
  204. * dtd.pxi
  205. * relaxng.pxi
  206. * schematron.pxi
  207. * xmlschema.pxi
  208. Python modules
  209. ==============
  210. The ``lxml`` package also contains a number of pure Python modules:
  211. builder.py
  212. The E-factory and the ElementBuilder class. These provide a
  213. simple interface to XML tree generation.
  214. cssselect.py
  215. A CSS selector implementation based on XPath. The main class is
  216. called ``CSSSelector``.
  217. doctestcompare.py
  218. A relaxed comparison scheme for XML/HTML markup in doctest.
  219. ElementInclude.py
  220. XInclude-like document inclusion, compatible with ElementTree.
  221. _elementpath.py
  222. XPath-like path language, compatible with ElementTree.
  223. sax.py
  224. SAX2 compatible interfaces to copy lxml trees from/to SAX compatible
  225. tools.
  226. usedoctest.py
  227. Wrapper module for ``doctestcompare.py`` that simplifies its usage
  228. from inside a doctest.
  229. lxml.objectify
  230. ==============
  231. A Cython implemented extension module that uses the public C-API of
  232. lxml.etree. It provides a Python object-like interface to XML trees.
  233. The implementation resides in the file `lxml.objectify.pyx
  234. <https://github.com/lxml/lxml/blob/master/src/lxml/lxml.objectify.pyx>`_.
  235. lxml.html
  236. =========
  237. A specialised toolkit for HTML handling, based on lxml.etree. This is
  238. implemented in pure Python.