build.txt 11 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341
  1. How to build lxml from source
  2. =============================
  3. To build lxml from source, you need libxml2 and libxslt properly
  4. installed, *including the header files*. These are likely shipped in
  5. separate ``-dev`` or ``-devel`` packages like ``libxml2-dev``, which
  6. you must install before trying to build lxml.
  7. .. contents::
  8. ..
  9. 1 Cython
  10. 2 Github, git and hg
  11. 3 Building the sources
  12. 4 Running the tests and reporting errors
  13. 5 Building an egg
  14. 6 Building lxml on MacOS-X
  15. 7 Static linking on Windows
  16. 8 Building Debian packages from SVN sources
  17. Cython
  18. ------
  19. .. _pip: http://pypi.python.org/pypi/pip
  20. .. _Cython: http://cython.org
  21. .. _wheel: https://wheel.readthedocs.io/en/latest/
  22. The lxml.etree and lxml.objectify modules are written in Cython_.
  23. Since we distribute the Cython-generated .c files with lxml releases,
  24. however, you do not need Cython to build lxml from the normal release
  25. sources. We even encourage you to *not install Cython* for a normal
  26. release build, as the generated C code can vary quite heavily between
  27. Cython versions, which may or may not generate correct code for lxml.
  28. The pre-generated release sources were tested and therefore are known
  29. to work.
  30. So, if you want a reliable build of lxml, we suggest to a) use a
  31. source release of lxml and b) disable or uninstall Cython for the
  32. build.
  33. *Only* if you are interested in building lxml from a checkout of the
  34. developer sources (e.g. to test a bug fix that has not been release
  35. yet) or if you want to be an lxml developer, then you do need a
  36. working Cython installation. You can use pip_ to install it::
  37. pip install -r requirements.txt
  38. https://github.com/lxml/lxml/blob/master/requirements.txt
  39. lxml currently requires at least Cython 0.29. Later release versions
  40. are generally preferred.
  41. Github, git and hg
  42. -------------------
  43. The lxml package is developed in a repository on Github_ using
  44. Mercurial_ and the `hg-git`_ plugin. You can retrieve the current
  45. developer version using::
  46. hg clone git+ssh://git@github.com/lxml/lxml.git lxml
  47. Or, using git::
  48. git clone ssh://git@github.com/lxml/lxml.git lxml
  49. This will create a directory ``lxml`` and download the source into it,
  50. including the complete development history. Don't be afraid, the
  51. repository download is fairly quick. You can also browse the
  52. `lxml repository`_ through the web or download a ZIP archive with the
  53. `latest master branch <https://github.com/lxml/lxml/archive/master.zip>`_.
  54. .. _Github: https://github.com/lxml/
  55. .. _Mercurial: http://mercurial.selenic.com/
  56. .. _`hg-git`: http://hg-git.github.com/
  57. .. _`lxml repository`: https://github.com/lxml/lxml
  58. .. _`source tar-ball`: https://github.com/lxml/lxml/tarball/master
  59. Building the sources
  60. ---------------------
  61. Clone the source repository as described above (or download
  62. the `source tar-ball`_ and unpack it) and then type::
  63. python setup.py build
  64. or::
  65. python setup.py bdist_egg # requires 'setuptools' or 'distribute'
  66. To (re-)build the C sources with Cython, you must additionally pass the
  67. option ``--with-cython``::
  68. python setup.py build --with-cython
  69. If you want to test lxml from the source directory, it is better to build it
  70. in-place like this::
  71. python setup.py build_ext -i --with-cython
  72. or, in Unix-like environments::
  73. make inplace
  74. To speed up the build in test environments (e.g. on a continuous
  75. integration server), set the ``CFLAGS`` environment variable to
  76. disable C compiler optimisations (e.g. "-O0" for gcc, that's
  77. minus-oh-zero), for example::
  78. CFLAGS="-O0" make inplace
  79. If you get errors about missing header files (e.g. ``Python.h`` or
  80. ``libxml/xmlversion.h``) then you need to make sure the development
  81. packages of Python, libxml2 and libxslt are properly installed. On
  82. Linux distributions, they are usually called something like
  83. ``libxml2-dev`` or ``libxslt-devel``. If these packages were
  84. installed in non-standard places, try passing the following option to
  85. setup.py to make sure the right config is found::
  86. python setup.py build --with-xslt-config=/path/to/xslt-config
  87. There are also env vars to allow overriding the config tool::
  88. env XML2_CONFIG=/path/to/xml2-config python build
  89. You may also use ``pkg-config`` as the tools::
  90. env XSLT_CONFIG="pkg-config libxslt" python setup.py build
  91. If this doesn't help, you may have to add the location of the header
  92. files to the include path like::
  93. python setup.py build_ext -i -I /usr/include/libxml2
  94. where the file is in ``/usr/include/libxml2/libxml/xmlversion.h``
  95. To use lxml.etree in-place, you can place lxml's ``src`` directory
  96. on your Python module search path (PYTHONPATH) and then import
  97. ``lxml.etree`` to play with it::
  98. # cd lxml
  99. # PYTHONPATH=src python
  100. Python 2.7.2
  101. Type "help", "copyright", "credits" or "license" for more information.
  102. >>> from lxml import etree
  103. >>>
  104. To make sure everything gets recompiled cleanly after changes, you can
  105. run ``make clean`` or delete the file ``src/lxml/etree.c``.
  106. Running the tests and reporting errors
  107. --------------------------------------
  108. The source distribution (tgz) and the source repository contain a test
  109. suite for lxml. You can run it from the top-level directory::
  110. python test.py
  111. Note that the test script only tests the in-place build (see distutils
  112. building above), as it searches the ``src`` directory. You can use the
  113. following one-step command to trigger an in-place build and test it::
  114. make test
  115. This also runs the ElementTree and cElementTree compatibility tests. To call
  116. them separately, make sure you have lxml on your PYTHONPATH first, then run::
  117. python selftest.py
  118. and::
  119. python selftest2.py
  120. If the tests give failures, errors, or worse, segmentation faults, we'd really
  121. like to know. Please contact us on the `mailing list`_, and please specify
  122. the version of lxml, libxml2, libxslt and Python you were using, as well as
  123. your operating system type (Linux, Windows, MacOS-X, ...).
  124. .. _`mailing list`: https://lxml.de/mailinglist/
  125. Building an egg or wheel
  126. ------------------------
  127. This is the procedure to make an lxml egg or wheel_ for your platform.
  128. It assumes that you have ``setuptools`` or ``distribute`` installed, as well
  129. as the ``wheel`` package.
  130. First, download the lxml-x.y.tar.gz release. This contains the pregenerated
  131. C files so that you can be sure you build exactly from the release sources.
  132. Unpack them and ``cd`` into the resulting directory. Then, to build a wheel,
  133. simply run the command
  134. ::
  135. python setup.py bdist_wheel
  136. or, to build a statically linked wheel with all of libxml2, libxslt and
  137. friends compiled in, run
  138. python setup.py bdist_wheel --static-deps
  139. The resulting .whl file will be written into the ``dist`` directory.
  140. To build an egg file, run
  141. ::
  142. python setup.py build_egg
  143. If you are on a Unix-like platform, you can first build the extension modules
  144. using
  145. ::
  146. python setup.py build
  147. and then ``cd`` into the directory ``build/lib.your.platform`` to call
  148. ``strip`` on any ``.so`` file you find there. This reduces the size of
  149. the binary distribution considerably. Then, from the package root directory,
  150. call
  151. ::
  152. python setup.py bdist_egg
  153. This will quickly package the pre-built packages into an egg file and
  154. drop it into the ``dist`` directory.
  155. Building lxml on MacOS-X
  156. ------------------------
  157. Apple regularly ships new system releases with horribly outdated
  158. system libraries. This is specifically the case for libxml2 and
  159. libxslt, where the system provided versions used to be too old
  160. to even build lxml for a long time.
  161. While the Unix environment in MacOS-X makes it relatively easy to
  162. install Unix/Linux style package management tools and new software, it
  163. actually seems to be hard to get libraries set up for exclusive usage
  164. that MacOS-X ships in an older version. Alternative distributions
  165. (like macports) install their libraries in addition to the system
  166. libraries, but the compiler and the runtime loader on MacOS still sees
  167. the system libraries before the new libraries. This can lead to
  168. undebuggable crashes where the newer library seems to be loaded but
  169. the older system library is used.
  170. Apple discourages static building against libraries, which would help
  171. working around this problem. Apple does not ship static library
  172. binaries with its system and several package management systems follow
  173. this decision. Therefore, building static binaries requires building
  174. the dependencies first. The ``setup.py`` script does this
  175. automatically when you call it like this::
  176. python setup.py build --static-deps
  177. This will download and build the latest versions of libxml2 and
  178. libxslt from the official FTP download site. If you want to use
  179. specific versions, or want to prevent any online access, you can
  180. download both ``tar.gz`` release files yourself, place them into a
  181. subdirectory ``libs`` in the lxml distribution, and call ``setup.py``
  182. with the desired target versions like this::
  183. python setup.py build --static-deps \
  184. --libxml2-version=2.9.12 \
  185. --libxslt-version=1.1.34 \
  186. sudo python setup.py install
  187. Instead of ``build``, you can use any target, like ``bdist_egg``
  188. if you want to use setuptools to build an installable egg, or
  189. ``bdist_wheel`` for a wheel package.
  190. Note that this also works with pip_. Since you can't pass
  191. command line options in this case, you have to use an environment
  192. variable instead::
  193. STATIC_DEPS=true pip install lxml
  194. To install the package into the system Python package directory,
  195. run the installation with "sudo"::
  196. STATIC_DEPS=true sudo pip install lxml
  197. The ``STATICBUILD`` environment variable is handled equivalently to
  198. the ``STATIC_DEPS`` variable, but is used by some other extension
  199. packages, too.
  200. If you decide to do a non-static build, you may also have to install
  201. the command line tools in addition to the XCode build environment.
  202. They are available as a restricted download from here:
  203. https://developer.apple.com/downloads/index.action?=command%20line%20tools#
  204. Without them, the compiler may not find the necessary header files of
  205. the XML libraries, according to the second comment in this ticket:
  206. https://bugs.launchpad.net/lxml/+bug/1244094
  207. Static linking on Windows
  208. -------------------------
  209. Most operating systems have proper package management that makes installing
  210. current versions of libxml2 and libxslt easy. The most famous exception is
  211. Microsoft Windows, which entirely lacks these capabilities. To work around
  212. the limits of this platform, lxml's installation can download pre-built
  213. packages of the dependencies and build statically against them. Assuming
  214. you have a proper C compiler setup to build Python extensions, this should
  215. work::
  216. python setup.py bdist_wininst --static-deps
  217. It should create a windows installer in the ``pkg`` directory.
  218. Building Debian packages from SVN sources
  219. -----------------------------------------
  220. `Andreas Pakulat`_ proposed the following approach.
  221. .. _`Andreas Pakulat`: http://thread.gmane.org/gmane.comp.python.lxml.devel/1239/focus=1249
  222. * ``apt-get source lxml``
  223. * remove the unpacked directory
  224. * tar.gz the lxml SVN version and replace the orig.tar.gz that lies in the
  225. directory
  226. * check md5sum of created tar.gz file and place new sum and size in dsc file
  227. * do ``dpkg-source -x lxml-[VERSION].dsc`` and cd into the newly created directory
  228. * run ``dch -i`` and add a comment like "use trunk version", this will
  229. increase the debian version number so apt/dpkg won't get confused
  230. * run ``dpkg-buildpackage -rfakeroot -us -uc`` to build the package
  231. In case ``dpkg-buildpackage`` tells you that some dependencies are missing, you
  232. can either install them manually or run ``apt-get build-dep lxml``.
  233. That will give you .deb packages in the parent directory which can be
  234. installed using ``dpkg -i``.