| 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313 |
- ==============================
- How to read the source of lxml
- ==============================
- :Author:
- Stefan Behnel
- .. meta::
- :description: How to read and work on the source code of lxml
- :keywords: lxml, XML, Cython, source code, develop, comprehend, understand
- This document describes how to read the source code of lxml_ and how
- to start working on it. You might also be interested in the companion
- document that describes `how to build lxml from sources`_.
- .. _lxml: http://lxml.de/
- .. _`how to build lxml from sources`: build.html
- .. _`ReStructured Text`: http://docutils.sourceforge.net/rst.html
- .. _epydoc: http://epydoc.sourceforge.net/
- .. _docutils: http://docutils.sourceforge.net/
- .. _`C-level API`: capi.html
- .. contents::
- ..
- 1 What is Cython?
- 2 Where to start?
- 2.1 Concepts
- 2.2 The documentation
- 3 lxml.etree
- 4 lxml.objectify
- 5 lxml.html
- What is Cython?
- ===============
- .. _Cython: http://cython.org/
- .. _Pyrex: http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/
- Cython_ is the language that lxml is written in. It is a very
- Python-like language that was specifically designed for writing Python
- extension modules.
- The reason why Cython (or actually its predecessor Pyrex_ at the time)
- was chosen as an implementation language for lxml, is that it makes it
- very easy to interface with both the Python world and external C code.
- Cython generates all the necessary glue code for the Python API,
- including Python types, calling conventions and reference counting.
- On the other side of the table, calling into C code is not more than
- declaring the signature of the function and maybe some variables as
- being C types, pointers or structs, and then calling it. The rest of
- the code is just plain Python code.
- The Cython language is so close to Python that the Cython compiler can
- actually compile many, many Python programs to C without major
- modifications. But the real speed gains of a C compilation come from
- type annotations that were added to the language and that allow Cython
- to generate very efficient C code.
- Even if you are not familiar with Cython, you should keep in mind that
- a slow implementation of a feature is better than none. So, if you
- want to contribute and have an idea what code you want to write, feel
- free to start with a pure Python implementation. Chances are, if you
- get the change officially accepted and integrated, others will take
- the time to optimise it so that it runs fast in Cython.
- Where to start?
- ===============
- First of all, read `how to build lxml from sources`_ to learn how to
- retrieve the source code from the Subversion repository and how to
- build it. The source code lives in the subdirectory ``src`` of the
- checkout.
- The main extension modules in lxml are ``lxml.etree`` and
- ``lxml.objectify``. All main modules have the file extension
- ``.pyx``, which shows the descendence from Pyrex. As usual in Python,
- the main files start with a short description and a couple of imports.
- Cython distinguishes between the run-time ``import`` statement (as
- known from Python) and the compile-time ``cimport`` statement, which
- imports C declarations, either from external libraries or from other
- Cython modules.
- Concepts
- --------
- lxml's tree API is based on proxy objects. That means, every Element
- object (or rather ``_Element`` object) is a proxy for a libxml2 node
- structure. The class declaration is (mainly)::
- cdef class _Element:
- cdef _Document _doc
- cdef xmlNode* _c_node
- It is a naming convention that C variables and C level class members
- that are passed into libxml2 start with a prefixed ``c_`` (commonly
- libxml2 struct pointers), and that C level class members are prefixed
- with an underscore. So you will often see names like ``c_doc`` for an
- ``xmlDoc*`` variable (or ``c_node`` for an ``xmlNode*``), or the above
- ``_c_node`` for a class member that points to an ``xmlNode`` struct
- (or ``_c_doc`` for an ``xmlDoc*``).
- It is important to know that every proxy in lxml has a factory
- function that properly sets up C level members. Proxy objects must
- *never* be instantiated outside of that factory. For example, to
- instantiate an _Element object or its subclasses, you must always call
- its factory function::
- cdef xmlNode* c_node
- cdef _Document doc
- cdef _Element element
- ...
- element = _elementFactory(doc, c_node)
- A good place to see how this factory is used are the Element methods
- ``getparent()``, ``getnext()`` and ``getprevious()``.
- The documentation
- -----------------
- An important part of lxml is the documentation that lives in the
- ``doc`` directory. It describes a large part of the API and comprises
- a lot of example code in the form of doctests.
- The documentation is written in the `ReStructured Text`_ format, a
- very powerful text markup language that looks almost like plain text.
- It is part of the docutils_ package.
- The project web site of lxml_ is completely generated from these text
- documents. Even the side menu is just collected from the table of
- contents that the ReST processor writes into each HTML page.
- Obviously, we use lxml for this.
- The easiest way to generate the HTML pages is by calling::
- make html
- This will call the script ``doc/mkhtml.py`` to run the ReST processor
- on the files. After generating an HTML page the script parses it back
- in to build the side menu, and injects the complete menu into each
- page at the very end.
- Running the ``make`` command will also generate the API documentation
- if you have epydoc_ installed. The epydoc package will import and
- introspect the extension modules and also introspect and parse the
- Python modules of lxml. The aggregated information will then be
- written out into an HTML documentation site.
- lxml.etree
- ==========
- The main module, ``lxml.etree``, is in the file `lxml.etree.pyx
- <https://github.com/lxml/lxml/blob/master/src/lxml/lxml.etree.pyx>`_. It
- implements the main functions and types of the ElementTree API, as
- well as all the factory functions for proxies. It is the best place
- to start if you want to find out how a specific feature is
- implemented.
- At the very end of the file, it contains a series of ``include``
- statements that merge the rest of the implementation into the
- generated C code. Yes, you read right: no importing, no source file
- namespacing, just plain good old include and a huge C code result of
- more than 100,000 lines that we throw right into the C compiler.
- The main include files are:
- apihelpers.pxi
- Private C helper functions. Except for the factory functions,
- most of the little functions that are used all over the place are
- defined here. This includes things like reading out the text
- content of a libxml2 tree node, checking input from the API level,
- creating a new Element node or handling attribute values. If you
- want to work on the lxml code, you should keep these functions in
- the back of your head, as they will definitely make your life
- easier.
- classlookup.pxi
- Element class lookup mechanisms. The main API and engines for
- those who want to define custom Element classes and inject them
- into lxml.
- docloader.pxi
- Support for custom document loaders. Base class and registry for
- custom document resolvers.
- extensions.pxi
- Infrastructure for extension functions in XPath/XSLT, including
- XPath value conversion and function registration.
- iterparse.pxi
- Incremental XML parsing. An iterator class that builds iterparse
- events while parsing.
- nsclasses.pxi
- Namespace implementation and registry. The registry and engine
- for Element classes that use the ElementNamespaceClassLookup
- scheme.
- parser.pxi
- Parsers for XML and HTML. This is the main parser engine. It's
- the reason why you can parse a document from various sources in
- two lines of Python code. It's definitely not the right place to
- start reading lxml's soure code.
- parsertarget.pxi
- An ElementTree compatible parser target implementation based on
- the SAX2 interface of libxml2.
- proxy.pxi
- Very low-level functions for memory allocation/deallocation
- and Element proxy handling. Ignoring this for the beginning
- will safe your head from exploding.
- public-api.pxi
- The set of C functions that are exported to other extension
- modules at the C level. For example, ``lxml.objectify`` makes use
- of these. See the `C-level API` documentation.
- readonlytree.pxi
- A separate read-only implementation of the Element API. This is
- used in places where non-intrusive access to a tree is required,
- such as the ``PythonElementClassLookup`` or XSLT extension
- elements.
- saxparser.pxi
- SAX-like parser interfaces as known from ElementTree's TreeBuilder.
- serializer.pxi
- XML output functions. Basically everything that creates byte
- sequences from XML trees.
- xinclude.pxi
- XInclude support.
- xmlerror.pxi
- Error log handling. All error messages that libxml2 generates
- internally walk through the code in this file to end up in lxml's
- Python level error logs.
- At the end of the file, you will find a long list of named error
- codes. It is generated from the libxml2 HTML documentation (using
- lxml, of course). See the script ``update-error-constants.py``
- for this.
- xmlid.pxi
- XMLID and IDDict, a dictionary-like way to find Elements by their
- XML-ID attribute.
- xpath.pxi
- XPath evaluators.
- xslt.pxi
- XSL transformations, including the ``XSLT`` class, document lookup
- handling and access control.
- The different schema languages (DTD, RelaxNG, XML Schema and
- Schematron) are implemented in the following include files:
- * dtd.pxi
- * relaxng.pxi
- * schematron.pxi
- * xmlschema.pxi
- Python modules
- ==============
- The ``lxml`` package also contains a number of pure Python modules:
- builder.py
- The E-factory and the ElementBuilder class. These provide a
- simple interface to XML tree generation.
- cssselect.py
- A CSS selector implementation based on XPath. The main class is
- called ``CSSSelector``.
- doctestcompare.py
- A relaxed comparison scheme for XML/HTML markup in doctest.
- ElementInclude.py
- XInclude-like document inclusion, compatible with ElementTree.
- _elementpath.py
- XPath-like path language, compatible with ElementTree.
- sax.py
- SAX2 compatible interfaces to copy lxml trees from/to SAX compatible
- tools.
- usedoctest.py
- Wrapper module for ``doctestcompare.py`` that simplifies its usage
- from inside a doctest.
- lxml.objectify
- ==============
- A Cython implemented extension module that uses the public C-API of
- lxml.etree. It provides a Python object-like interface to XML trees.
- The implementation resides in the file `lxml.objectify.pyx
- <https://github.com/lxml/lxml/blob/master/src/lxml/lxml.objectify.pyx>`_.
- lxml.html
- =========
- A specialised toolkit for HTML handling, based on lxml.etree. This is
- implemented in pure Python.
|