FAQ.txt 41 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751752753754755756757758759760761762763764765766767768769770771772773774775776777778779780781782783784785786787788789790791792793794795796797798799800801802803804805806807808809810811812813814815816817818819820821822823824825826827828829830831832833834835836837838839840841842843844845846847848849850851852853854855856857858859860861862863864865866867868869870871872873874875876877878879880881882883884885886887888889890891892893894895896897898899900901902903904905906907908909910911912913914915916917918919920921922923924925926927928929930931932933934935936937938939940941942943944945946
  1. =====================================
  2. lxml FAQ - Frequently Asked Questions
  3. =====================================
  4. .. meta::
  5. :description: Frequently Asked Questions about lxml (FAQ)
  6. :keywords: lxml, lxml.etree, FAQ, frequently asked questions
  7. Frequently asked questions on lxml. See also the notes on compatibility_ to
  8. ElementTree_.
  9. .. _compatibility: compatibility.html
  10. .. _ElementTree: http://effbot.org/zone/element-index.htm
  11. .. _`build instructions`: build.html
  12. .. _`MacOS-X` : build.html#building-lxml-on-macos-x
  13. .. contents::
  14. ..
  15. 1 General Questions
  16. 1.1 Is there a tutorial?
  17. 1.2 Where can I find more documentation about lxml?
  18. 1.3 What standards does lxml implement?
  19. 1.4 Who uses lxml?
  20. 1.5 What is the difference between lxml.etree and lxml.objectify?
  21. 1.6 How can I make my application run faster?
  22. 1.7 What about that trailing text on serialised Elements?
  23. 1.8 How can I find out if an Element is a comment or PI?
  24. 1.9 How can I map an XML tree into a dict of dicts?
  25. 2 Installation
  26. 2.1 Which version of libxml2 and libxslt should I use or require?
  27. 2.2 Where are the binary builds?
  28. 2.3 Why do I get errors about missing UCS4 symbols when installing lxml?
  29. 3 Contributing
  30. 3.1 Why is lxml not written in Python?
  31. 3.2 How can I contribute?
  32. 4 Bugs
  33. 4.1 My application crashes!
  34. 4.2 My application crashes on MacOS-X!
  35. 4.3 I think I have found a bug in lxml. What should I do?
  36. 4.4 How do I know a bug is really in lxml and not in libxml2?
  37. 5 Threading
  38. 5.1 Can I use threads to concurrently access the lxml API?
  39. 5.2 Does my program run faster if I use threads?
  40. 5.3 Would my single-threaded program run faster if I turned off threading?
  41. 5.4 Why can't I reuse XSLT stylesheets in other threads?
  42. 5.5 My program crashes when run with mod_python/Pyro/Zope/Plone/...
  43. 6 Parsing and Serialisation
  44. 6.1 Why doesn't the ``pretty_print`` option reformat my XML output?
  45. 6.2 Why can't lxml parse my XML from unicode strings?
  46. 6.3 What is the difference between str(xslt(doc)) and xslt(doc).write() ?
  47. 6.4 Why can't I just delete parents or clear the root node in iterparse()?
  48. 6.5 How do I output null characters in XML text?
  49. 7 XPath and Document Traversal
  50. 7.1 What are the ``findall()`` and ``xpath()`` methods on Element(Tree)?
  51. 7.2 Why doesn't ``findall()`` support full XPath expressions?
  52. 7.3 How can I find out which namespace prefixes are used in a document?
  53. 7.4 How can I specify a default namespace for XPath expressions?
  54. ..
  55. >>> import sys
  56. >>> from lxml import etree as _etree
  57. >>> if sys.version_info[0] >= 3:
  58. ... class etree_mock(object):
  59. ... def __getattr__(self, name): return getattr(_etree, name)
  60. ... def tostring(self, *args, **kwargs):
  61. ... s = _etree.tostring(*args, **kwargs)
  62. ... if isinstance(s, bytes): s = s.decode("utf-8") # CR
  63. ... if s[-1] == '\n': s = s[:-1]
  64. ... return s
  65. ... else:
  66. ... class etree_mock(object):
  67. ... def __getattr__(self, name): return getattr(_etree, name)
  68. ... def tostring(self, *args, **kwargs):
  69. ... s = _etree.tostring(*args, **kwargs)
  70. ... if s[-1] == '\n': s = s[:-1]
  71. ... return s
  72. >>> etree = etree_mock()
  73. General Questions
  74. =================
  75. Is there a tutorial?
  76. --------------------
  77. Read the `lxml.etree Tutorial`_. While this is still work in progress
  78. (just as any good documentation), it provides an overview of the most
  79. important concepts in ``lxml.etree``. If you want to help out,
  80. improving the tutorial is a very good place to start.
  81. There is also a `tutorial for ElementTree`_ which works for
  82. ``lxml.etree``. The documentation of the `extended etree API`_ also
  83. contains many examples for ``lxml.etree``. Fredrik Lundh's `element
  84. library`_ contains a lot of nice recipes that show how to solve common
  85. tasks in ElementTree and lxml.etree. To learn using
  86. ``lxml.objectify``, read the `objectify documentation`_.
  87. John Shipman has written another tutorial called `Python XML
  88. processing with lxml`_ that contains lots of examples. Liza Daly
  89. wrote a nice article about high-performance aspects when `parsing
  90. large files with lxml`_.
  91. .. _`lxml.etree Tutorial`: tutorial.html
  92. .. _`tutorial for ElementTree`: http://effbot.org/zone/element.htm
  93. .. _`extended etree API`: api.html
  94. .. _`objectify documentation`: objectify.html
  95. .. _`Python XML processing with lxml`: http://www.nmt.edu/tcc/help/pubs/pylxml/
  96. .. _`element library`: http://effbot.org/zone/element-lib.htm
  97. .. _`parsing large files with lxml`: http://www.ibm.com/developerworks/xml/library/x-hiperfparse/
  98. Where can I find more documentation about lxml?
  99. -----------------------------------------------
  100. There is a lot of documentation on the web and also in the Python
  101. standard library documentation, as lxml implements the well-known
  102. `ElementTree API`_ and tries to follow its documentation as closely as
  103. possible. The recipes in Fredrik Lundh's `element library`_ are
  104. generally worth taking a look at. There are a couple of issues where
  105. lxml cannot keep up compatibility. They are described in the
  106. compatibility_ documentation.
  107. The lxml specific extensions to the API are described by individual
  108. files in the ``doc`` directory of the source distribution and on `the
  109. web page`_.
  110. The `generated API documentation`_ is a comprehensive API reference
  111. for the lxml package.
  112. .. _`ElementTree API`: http://effbot.org/zone/element-index.htm
  113. .. _`the web page`: http://codespeak.net/lxml/#documentation
  114. .. _`generated API documentation`: api/index.html
  115. What standards does lxml implement?
  116. -----------------------------------
  117. The compliance to XML Standards depends on the support in libxml2 and libxslt.
  118. Here is a quote from `http://xmlsoft.org/ <http://xmlsoft.org/>`_:
  119. In most cases libxml2 tries to implement the specifications in a relatively
  120. strictly compliant way. As of release 2.4.16, libxml2 passed all 1800+ tests
  121. from the OASIS XML Tests Suite.
  122. lxml currently supports libxml2 2.6.20 or later, which has even better
  123. support for various XML standards. The important ones are:
  124. * XML 1.0
  125. * HTML 4
  126. * XML namespaces
  127. * XML Schema 1.0
  128. * XPath 1.0
  129. * XInclude 1.0
  130. * XSLT 1.0
  131. * EXSLT
  132. * XML catalogs
  133. * canonical XML
  134. * RelaxNG
  135. * xml:id
  136. * xml:base
  137. Support for XML Schema is currently not 100% complete in libxml2, but
  138. is definitely very close to compliance. Schematron is supported,
  139. although not necessarily complete. libxml2 also supports loading
  140. documents through HTTP and FTP.
  141. Who uses lxml?
  142. --------------
  143. As an XML library, lxml is often used under the hood of in-house
  144. server applications, such as web servers or applications that
  145. facilitate some kind of document management. Many people who deploy
  146. Zope_ or Plone_ use it together with lxml. Therefore, it is hard to
  147. get an idea of who uses it, and the following list of 'users and
  148. projects we know of' is definitely not a complete list of lxml's
  149. users.
  150. Also note that the compatibility to the ElementTree library does not
  151. require projects to set a hard dependency on lxml - as long as they do
  152. not take advantage of lxml's enhanced feature set.
  153. * cssutils_, a CSS parser and toolkit, can be used with ``lxml.cssselect``
  154. * Deliverance_, a content theming tool
  155. * `Enfold Proxy 4`_, a web server accelerator with on-the-fly XSLT processing
  156. * Inteproxy_, a secure HTTP proxy
  157. * lwebstring_, an XML template engine
  158. * OpenXMLlib_, a library for handling OpenXML document meta data
  159. * Pycoon_, a WSGI web development framework based on XML pipelines
  160. * PyQuery_, a query framework for XML/HTML, similar to jQuery for JavaScript
  161. * Rambler_, a meta search engine that aggregates different data sources
  162. * rdfadict_, an RDFa parser with a simple dictionary-like interface.
  163. Zope3 and some of its extensions have good support for lxml:
  164. * gocept.lxml_, Zope3 interface bindings for lxml
  165. * z3c.rml_, an implementation of ReportLab's RML format
  166. * zif.sedna_, an XQuery based interface to the Sedna OpenSource XML database
  167. And don't miss the quotes by our generally happy_ users_, and other
  168. `sites that link to lxml`_. As `Liza Daly`_ puts it: "Many software
  169. products come with the pick-two caveat, meaning that you must choose
  170. only two: speed, flexibility, or readability. When used carefully,
  171. lxml can provide all three."
  172. .. _Zope: http://www.zope.org/
  173. .. _Plone: http://www.plone.org/
  174. .. _cssutils: http://code.google.com/p/cssutils/source/browse/trunk/examples/style.py?r=917
  175. .. _Deliverance: http://www.openplans.org/projects/deliverance/project-home
  176. .. _`Enfold Proxy 4`: http://www.enfoldsystems.com/Products/Proxy/4
  177. .. _gocept.lxml: http://pypi.python.org/pypi/gocept.lxml
  178. .. _Inteproxy: http://lists.wald.intevation.org/pipermail/inteproxy-devel/2007-February/000000.html
  179. .. _lwebstring: http://pypi.python.org/pypi/lwebstring
  180. .. _OpenXMLlib: http://permalink.gmane.org/gmane.comp.python.lxml.devel/3250
  181. .. _Pycoon: http://pypi.python.org/pypi/pycoon
  182. .. _PyQuery: http://pypi.python.org/pypi/pyquery
  183. .. _Rambler: http://beta.rambler.ru/srch?query=python+lxml&searchtype=web
  184. .. _rdfadict: http://pypi.python.org/pypi/rdfadict
  185. .. _z3c.rml: http://pypi.python.org/pypi/z3c.rml
  186. .. _zif.sedna: http://pypi.python.org/pypi/zif.sedna
  187. .. _happy: http://thread.gmane.org/gmane.comp.python.lxml.devel/3244/focus=3244
  188. .. _users: http://article.gmane.org/gmane.comp.python.lxml.devel/3246
  189. .. _`sites that link to lxml`: http://www.google.com/search?as_lq=http:%2F%2Fcodespeak.net%2Flxml
  190. .. _`Liza Daly`: http://www.ibm.com/developerworks/xml/library/x-hiperfparse/
  191. What is the difference between lxml.etree and lxml.objectify?
  192. -------------------------------------------------------------
  193. The two modules provide different ways of handling XML. However, objectify
  194. builds on top of lxml.etree and therefore inherits most of its capabilities
  195. and a large portion of its API.
  196. * lxml.etree is a generic API for XML and HTML handling. It aims for
  197. ElementTree compatibility_ and supports the entire XML infoset. It is well
  198. suited for both mixed content and data centric XML. Its generality makes it
  199. the best choice for most applications.
  200. * lxml.objectify is a specialized API for XML data handling in a Python object
  201. syntax. It provides a very natural way to deal with data fields stored in a
  202. structurally well defined XML format. Data is automatically converted to
  203. Python data types and can be manipulated with normal Python operators. Look
  204. at the examples in the `objectify documentation`_ to see what it feels like
  205. to use it.
  206. Objectify is not well suited for mixed contents or HTML documents. As it is
  207. built on top of lxml.etree, however, it inherits the normal support for
  208. XPath, XSLT or validation.
  209. How can I make my application run faster?
  210. -----------------------------------------
  211. lxml.etree is a very fast library for processing XML. There are, however, `a
  212. few caveats`_ involved in the mapping of the powerful libxml2 library to the
  213. simple and convenient ElementTree API. Not all operations are as fast as the
  214. simplicity of the API might suggest, while some use cases can heavily benefit
  215. from finding the right way of doing them. The `benchmark page`_ has a
  216. comparison to other ElementTree implementations and a number of tips for
  217. performance tweaking. As with any Python application, the rule of thumb is:
  218. the more of your processing runs in C, the faster your application gets. See
  219. also the section on threading_.
  220. .. _`a few caveats`: performance.html#the-elementtree-api
  221. .. _`benchmark page`: performance.html
  222. .. _threading: #threading
  223. What about that trailing text on serialised Elements?
  224. -----------------------------------------------------
  225. The ElementTree tree model defines an Element as a container with a tag name,
  226. contained text, child Elements and a tail text. This means that whenever you
  227. serialise an Element, you will get all parts of that Element:
  228. .. sourcecode:: pycon
  229. >>> root = etree.XML("<root><tag>text<child/></tag>tail</root>")
  230. >>> print(etree.tostring(root[0]))
  231. <tag>text<child/></tag>tail
  232. Here is an example that shows why not serialising the tail would be
  233. even more surprising from an object point of view:
  234. .. sourcecode:: pycon
  235. >>> root = etree.Element("test")
  236. >>> root.text = "TEXT"
  237. >>> print(etree.tostring(root))
  238. <test>TEXT</test>
  239. >>> root.tail = "TAIL"
  240. >>> print(etree.tostring(root))
  241. <test>TEXT</test>TAIL
  242. >>> root.tail = None
  243. >>> print(etree.tostring(root))
  244. <test>TEXT</test>
  245. Just imagine a Python list where you append an item and it doesn't
  246. show up when you look at the list.
  247. The ``.tail`` property is a huge simplification for the tree model as
  248. it avoids text nodes to appear in the list of children and makes
  249. access to them quick and simple. So this is a benefit in most
  250. applications and simplifies many, many XML tree algorithms.
  251. However, in document-like XML (and especially HTML), the above result can be
  252. unexpected to new users and can sometimes require a bit more overhead. A good
  253. way to deal with this is to use helper functions that copy the Element without
  254. its tail. The ``lxml.html`` package also deals with this in a couple of
  255. places, as most HTML algorithms benefit from a tail-free behaviour.
  256. How can I find out if an Element is a comment or PI?
  257. ----------------------------------------------------
  258. .. sourcecode:: pycon
  259. >>> root = etree.XML("<?my PI?><root><!-- empty --></root>")
  260. >>> root.tag
  261. 'root'
  262. >>> root.getprevious().tag is etree.PI
  263. True
  264. >>> root[0].tag is etree.Comment
  265. True
  266. How can I map an XML tree into a dict of dicts?
  267. -----------------------------------------------
  268. I'm glad you asked.
  269. .. sourcecode:: python
  270. def recursive_dict(element):
  271. return element.tag, \
  272. dict(map(recursive_dict, element)) or element.text
  273. Installation
  274. ============
  275. Which version of libxml2 and libxslt should I use or require?
  276. -------------------------------------------------------------
  277. It really depends on your application, but the rule of thumb is: more recent
  278. versions contain less bugs and provide more features.
  279. * Do not use libxml2 2.6.27 if you want to use XPath (including XSLT). You
  280. will get crashes when XPath errors occur during the evaluation (e.g. for
  281. unknown functions). This happens inside the evaluation call to libxml2, so
  282. there is nothing that lxml can do about it.
  283. * Try to use versions of both libraries that were released together. At least
  284. the libxml2 version should not be older than the libxslt version.
  285. * If you use XML Schema or Schematron which are still under development, the
  286. most recent version of libxml2 is usually a good bet.
  287. * The same applies to XPath, where a substantial number of bugs and memory
  288. leaks were fixed over time. If you encounter crashes or memory leaks in
  289. XPath applications, try a more recent version of libxml2.
  290. * For parsing and fixing broken HTML, lxml requires at least libxml2 2.6.21.
  291. * For the normal tree handling, however, any libxml2 version starting with
  292. 2.6.20 should do.
  293. Read the `release notes of libxml2`_ and the `release notes of libxslt`_ to
  294. see when (or if) a specific bug has been fixed.
  295. .. _`release notes of libxml2`: http://xmlsoft.org/news.html
  296. .. _`release notes of libxslt`: http://xmlsoft.org/XSLT/news.html
  297. Where are the binary builds?
  298. ----------------------------
  299. Sidnei da Silva regularly contributes Windows binaries for new
  300. releases. This is because two of the major problems of Microsoft
  301. Windows make it non-trivial for users to build lxml on this platform:
  302. the lack of a pre-installed standard compiler and the missing package
  303. management.
  304. If there is not currently a binary distribution of the most recent
  305. lxml release for this platform available from the Python Package Index
  306. (PyPI), please look through the older versions to see if they provide
  307. a binary build. This is done by appending the version number to the
  308. PyPI URL, e.g.::
  309. http://pypi.python.org/pypi/lxml/2.1.5
  310. Apart from that, we generally do not provide binary builds of lxml, as
  311. most of the other operating systems out there can build lxml without
  312. problems (with the exception of `MacOS-X`_), and the sheer mass of
  313. variations between platforms makes it futile to provide builds for
  314. everyone.
  315. Why do I get errors about missing UCS4 symbols when installing lxml?
  316. --------------------------------------------------------------------
  317. Most likely, you use a Python installation that was configured for internal
  318. use of UCS2 unicode, meaning 16-bit unicode. The lxml egg distributions are
  319. generally compiled on platforms that use UCS4, a 32-bit unicode encoding, as
  320. this is used on the majority of platforms. Sadly, both are not compatible, so
  321. the eggs can only support the one they were compiled with.
  322. This means that you have to compile lxml from sources for your system. Note
  323. that you do not need Cython for this, the lxml source distribution is directly
  324. compilable on both platform types. See the `build instructions`_ on how to do
  325. this.
  326. Contributing
  327. ============
  328. Why is lxml not written in Python?
  329. ----------------------------------
  330. It *almost* is.
  331. lxml is not written in plain Python, because it interfaces with two C
  332. libraries: libxml2 and libxslt. Accessing them at the C-level is
  333. required for performance reasons.
  334. However, to avoid writing plain C-code and caring too much about the
  335. details of built-in types and reference counting, lxml is written in
  336. Cython_, a Python-like language that is translated into C-code.
  337. Chances are that if you know Python, you can write `code that Cython
  338. accepts`_. Again, the C-ish style used in the lxml code is just for
  339. performance optimisations. If you want to contribute, don't bother
  340. with the details, a Python implementation of your contribution is
  341. better than none. And keep in mind that lxml's flexible API often
  342. favours an implementation of features in pure Python, without
  343. bothering with C-code at all. For example, the ``lxml.html`` package
  344. is entirely written in Python.
  345. Please contact the `mailing list`_ if you need any help.
  346. .. _Cython: http://www.cython.org/
  347. .. _`code that Cython accepts`: http://docs.cython.org/docs/tutorial.html
  348. How can I contribute?
  349. ---------------------
  350. If you find something that you would like lxml to do (or do better),
  351. then please tell us about it on the `mailing list`_. Patches are
  352. always appreciated, especially when accompanied by unit tests and
  353. documentation (doctests would be great). See the ``tests``
  354. subdirectories in the lxml source tree (below the ``src`` directory)
  355. and the ReST_ `text files`_ in the ``doc`` directory.
  356. We also have a `list of missing features`_ that we would like to
  357. implement but didn't due to lack if time. If *you* find the time,
  358. patches are very welcome.
  359. .. _ReST: http://docutils.sourceforge.net/rst.html
  360. .. _`text files`: http://codespeak.net/svn/lxml/trunk/doc/
  361. .. _`list of missing features`: http://codespeak.net/svn/lxml/trunk/IDEAS.txt
  362. Besides enhancing the code, there are a lot of places where you can help the
  363. project and its user base. You can
  364. * spread the word and write about lxml. Many users (especially new Python
  365. users) have not yet heared about lxml, although our user base is constantly
  366. growing. If you write your own blog and feel like saying something about
  367. lxml, go ahead and do so. If we think your contribution or criticism is
  368. valuable to other users, we may even put a link or a quote on the project
  369. page.
  370. * provide code examples for the general usage of lxml or specific problems
  371. solved with lxml. Readable code is a very good way of showing how a library
  372. can be used and what great things you can do with it. Again, if we hear
  373. about it, we can set a link on the project page.
  374. * work on the documentation. The web page is generated from a set of ReST_
  375. `text files`_. It is meant both as a representative project page for lxml
  376. and as a site for documenting lxml's API and usage. If you have questions
  377. or an idea how to make it more readable and accessible while you are reading
  378. it, please send a comment to the `mailing list`_.
  379. * enhance the web site. We put some work into making the web site
  380. usable, understandable and also easy to find, but there's always
  381. things that can be done better. You may notice that we are not
  382. top-ranked when searching the web for "Python and XML", so maybe you
  383. have an idea how to improve that.
  384. * help with the tutorial. A tutorial is the most important stating point for
  385. new users, so it is important for us to provide an easy to understand guide
  386. into lxml. As allo documentation, the tutorial is work in progress, so we
  387. appreciate every helping hand.
  388. * improve the docstrings. lxml uses docstrings to support Python's integrated
  389. online ``help()`` function. However, sometimes these are not sufficient to
  390. grasp the details of the function in question. If you find such a place,
  391. you can try to write up a better description and send it to the `mailing
  392. list`_.
  393. Bugs
  394. ====
  395. My application crashes!
  396. -----------------------
  397. One of the goals of lxml is "no segfaults", so if there is no clear
  398. warning in the documentation that you were doing something potentially
  399. harmful, you have found a bug and we would like to hear about it.
  400. Please report this bug to the `mailing list`_. See the section on bug
  401. reporting to learn how to do that.
  402. If your application (or e.g. your web container) uses threads, please
  403. see the FAQ section on threading_ to check if you touch on one of the
  404. potential pitfalls.
  405. In any case, try to reproduce the problem with the latest versions of
  406. libxml2 and libxslt. From time to time, bugs and race conditions are found
  407. in these libraries, so a more recent version might already contain a fix for
  408. your problem.
  409. Remember: even if you see lxml appear in a crash stack trace, it is
  410. not necessarily lxml that *caused* the crash.
  411. My application crashes on MacOS-X!
  412. ----------------------------------
  413. This was a common problem up to lxml 2.1.x. Since lxml 2.2, the only
  414. officially supported way to use it on this platform is through a
  415. static build against freshly downloaded versions of libxml2 and
  416. libxslt. See the build instructions for `MacOS-X`_.
  417. I think I have found a bug in lxml. What should I do?
  418. -----------------------------------------------------
  419. First, you should look at the `current developer changelog`_ to see if this
  420. is a known problem that has already been fixed in the SVN trunk since the
  421. release you are using.
  422. .. _`current developer changelog`: http://codespeak.net/svn/lxml/trunk/CHANGES.txt
  423. Also, the 'crash' section above has a few good advices what to try to see if
  424. the problem is really in lxml - and not in your setup. Believe it or not,
  425. that happens more often than you might think, especially when old libraries
  426. or even multiple library versions are installed.
  427. You should always try to reproduce the problem with the latest
  428. versions of libxml2 and libxslt - and make sure they are used.
  429. ``lxml.etree`` can tell you what it runs with:
  430. .. sourcecode:: python
  431. from lxml import etree
  432. print "lxml.etree: ", etree.LXML_VERSION
  433. print "libxml used: ", etree.LIBXML_VERSION
  434. print "libxml compiled: ", etree.LIBXML_COMPILED_VERSION
  435. print "libxslt used: ", etree.LIBXSLT_VERSION
  436. print "libxslt compiled: ", etree.LIBXSLT_COMPILED_VERSION
  437. If you can figure that the problem is not in lxml but in the
  438. underlying libxml2 or libxslt, you can ask right on the respective
  439. mailing lists, which may considerably reduce the time to find a fix or
  440. work-around. See the next question for some hints on how to do that.
  441. Otherwise, we would really like to hear about it. Please report it to
  442. the `mailing list`_ so that we can fix it. It is very helpful in this
  443. case if you can come up with a short code snippet that demonstrates
  444. your problem. If others can reproduce and see the problem, it is much
  445. easier for them to fix it - and maybe even easier for you to describe
  446. it and get people convinced that it really is a problem to fix.
  447. It is important that you always report the version of lxml, libxml2
  448. and libxslt that you get from the code snippet above. If we do not
  449. know the library versions you are using, we will ask back, so it will
  450. take longer for you to get a helpful answer.
  451. Since as a user of lxml you are likely a programmer, you might find
  452. `this article on bug reports`_ an interesting read.
  453. .. _`mailing list`: http://codespeak.net/mailman/listinfo/lxml-dev
  454. .. _`this article on bug reports`: http://www.chiark.greenend.org.uk/~sgtatham/bugs.html
  455. How do I know a bug is really in lxml and not in libxml2?
  456. ---------------------------------------------------------
  457. A large part of lxml's functionality is implemented by libxml2 and
  458. libxslt, so problems that you encounter may be in one or the other.
  459. Knowing the right place to ask will reduce the time it takes to fix
  460. the problem, or to find a work-around.
  461. Both libxml2 and libxslt come with their own command line frontends,
  462. namely ``xmllint`` and ``xsltproc``. If you encounter problems with
  463. XSLT processing for specific stylesheets or with validation for
  464. specific schemas, try to run the XSLT with ``xsltproc`` or the
  465. validation with ``xmllint`` respectively to find out if it fails there
  466. as well. If it does, please report directly to the mailing lists of
  467. the respective project, namely:
  468. * `libxml2 mailing list <http://mail.gnome.org/mailman/listinfo/xml>`_
  469. * `libxslt mailing list <http://mail.gnome.org/mailman/listinfo/xslt>`_
  470. On the other hand, everything that seems to be related to Python code,
  471. including custom resolvers, custom XPath functions, etc. is likely
  472. outside of the scope of libxml2/libxslt. If you encounter problems
  473. here or you are not sure where there the problem may come from, please
  474. ask on the lxml mailing list first.
  475. In any case, a good explanation of the problem including some simple
  476. test code and some input data will help us (or the libxml2 developers)
  477. see and understand the problem, which largely increases your chance of
  478. getting help. See the question above for a few hints on what is
  479. helpful here.
  480. Threading
  481. =========
  482. Can I use threads to concurrently access the lxml API?
  483. ------------------------------------------------------
  484. Short answer: yes, if you use lxml 2.2 and later.
  485. Since version 1.1, lxml frees the GIL (Python's global interpreter
  486. lock) internally when parsing from disk and memory, as long as you use
  487. either the default parser (which is replicated for each thread) or
  488. create a parser for each thread yourself. lxml also allows
  489. concurrency during validation (RelaxNG and XMLSchema) and XSL
  490. transformation. You can share RelaxNG, XMLSchema and XSLT objects
  491. between threads.
  492. While you can also share parsers between threads, this will serialize
  493. the access to each of them, so it is better to ``.copy()`` parsers or
  494. to just use the default parser if you do not need any special
  495. configuration. The same applies to the XPath evaluators, which use an
  496. internal lock to protect their prepared evaluation contexts. It is
  497. therefore best to use separate evaluator instances in threads.
  498. Warning: Before lxml 2.2, and especially before 2.1, there were
  499. various issues when moving subtrees between different threads, or when
  500. applying XSLT objects from one thread to trees parsed or modified in
  501. another. If you need code to run with older versions, you should
  502. generally avoid modifying trees in other threads than the one it was
  503. generated in. Although this should work in many cases, there are
  504. certain scenarios where the termination of a thread that parsed a tree
  505. can crash the application if subtrees of this tree were moved to other
  506. documents. You should be on the safe side when passing trees between
  507. threads if you either
  508. - do not modify these trees and do not move their elements to other
  509. trees, or
  510. - do not terminate threads while the trees they parsed are still in
  511. use (e.g. by using a fixed size thread-pool or long-running threads
  512. in processing chains)
  513. Since lxml 2.2, even multi-thread pipelines are supported. However,
  514. note that it is more efficient to do all tree work inside one thread,
  515. than to let multiple threads work on a tree one after the other. This
  516. is because trees inherit state from the thread that created them,
  517. which must be maintained when the tree is modified inside another
  518. thread.
  519. Does my program run faster if I use threads?
  520. --------------------------------------------
  521. Depends. The best way to answer this is timing and profiling.
  522. The global interpreter lock (GIL) in Python serializes access to the
  523. interpreter, so if the majority of your processing is done in Python
  524. code (walking trees, modifying elements, etc.), your gain will be
  525. close to zero. The more of your XML processing moves into lxml,
  526. however, the higher your gain. If your application is bound by XML
  527. parsing and serialisation, or by very selective XPath expressions and
  528. complex XSLTs, your speedup on multi-processor machines can be
  529. substantial.
  530. See the question above to learn which operations free the GIL to support
  531. multi-threading.
  532. Would my single-threaded program run faster if I turned off threading?
  533. ----------------------------------------------------------------------
  534. Possibly, yes. You can see for yourself by compiling lxml entirely
  535. without threading support. Pass the ``--without-threading`` option to
  536. setup.py when building lxml from source. You can also build libxml2
  537. without pthread support (``--without-pthreads`` option), which may add
  538. another bit of performance. Note that this will leave internal data
  539. structures entirely without thread protection, so make sure you really
  540. do not use lxml outside of the main application thread in this case.
  541. Why can't I reuse XSLT stylesheets in other threads?
  542. ----------------------------------------------------
  543. Since later lxml 2.0 versions, you can do this. There is some
  544. overhead involved as the result document needs an additional cleanup
  545. traversal when the input document and/or the stylesheet were created
  546. in other threads. However, on a multi-processor machine, the gain of
  547. freeing the GIL easily covers this drawback.
  548. If you need even the last bit of performance, consider keeping (a copy
  549. of) the stylesheet in thread-local storage, and try creating the input
  550. document(s) in the same thread. And do not forget to benchmark your
  551. code to see if the increased code complexity is really worth it.
  552. My program crashes when run with mod_python/Pyro/Zope/Plone/...
  553. ---------------------------------------------------------------
  554. These environments can use threads in a way that may not make it obvious when
  555. threads are created and what happens in which thread. This makes it hard to
  556. ensure lxml's threading support is used in a reliable way. Sadly, if problems
  557. arise, they are as diverse as the applications, so it is difficult to provide
  558. any generally applicable solution. Also, these environments are so complex
  559. that problems become hard to debug and even harder to reproduce in a
  560. predictable way. If you encounter crashes in one of these systems, but your
  561. code runs perfectly when started by hand, the following gives you a few hints
  562. for possible approaches to solve your specific problem:
  563. * make sure you use recent versions of libxml2, libxslt and lxml. The
  564. libxml2 developers keep fixing bugs in each release, and lxml also
  565. tries to become more robust against possible pitfalls. So newer
  566. versions might already fix your problem in a reliable way. Version
  567. 2.2 of lxml contains many improvements.
  568. * make sure the library versions you installed are really used. Do
  569. not rely on what your operating system tells you! Print the version
  570. constants in ``lxml.etree`` from within your runtime environment to
  571. make sure it is the case. This is especially a problem under
  572. MacOS-X when newer library versions were installed in addition to
  573. the outdated system libraries. Please read the bugs section
  574. regarding MacOS-X in this FAQ.
  575. * if you use ``mod_python``, try setting this option:
  576. PythonInterpreter main_interpreter
  577. There was a discussion on the mailing list about this problem:
  578. http://comments.gmane.org/gmane.comp.python.lxml.devel/2942
  579. * compile lxml without threading support by running ``setup.py`` with the
  580. ``--without-threading`` option. While this might be slower in certain
  581. scenarios on multi-processor systems, it *might* also keep your application
  582. from crashing, which should be worth more to you than peek performance.
  583. Remember that lxml is fast anyway, so concurrency may not even be worth it.
  584. * look out for fancy XSLT stuff like foreign document access or
  585. passing in subtrees trough XSLT variables. This might or might not
  586. work, depending on your specific usage. Again, later versions of
  587. lxml and libxslt provide safer support here.
  588. * try copying trees at suspicious places in your code and working with
  589. those instead of a tree shared between threads. Note that the
  590. copying must happen inside the target thread to be effective, not in
  591. the thread that created the tree. Serialising in one thread and
  592. parsing in another is also a simple (and fast) way of separating
  593. thread contexts.
  594. * try keeping thread-local copies of XSLT stylesheets, i.e. one per thread,
  595. instead of sharing one. Also see the question above.
  596. * you can try to serialise suspicious parts of your code with explicit thread
  597. locks, thus disabling the concurrency of the runtime system.
  598. * report back on the mailing list to see if there are other ways to work
  599. around your specific problems. Do not forget to report the version numbers
  600. of lxml, libxml2 and libxslt you are using (see the question on reporting
  601. a bug).
  602. Note that most of these options will degrade performance and/or your
  603. code quality. If you are unsure what to do, please ask on the mailing
  604. list.
  605. Parsing and Serialisation
  606. =========================
  607. ..
  608. making doctest happy:
  609. >>> try: from StringIO import StringIO
  610. ... except ImportError: from io import StringIO # Py3
  611. >>> filename = StringIO("<root/>")
  612. Why doesn't the ``pretty_print`` option reformat my XML output?
  613. ---------------------------------------------------------------
  614. Pretty printing (or formatting) an XML document means adding white space to
  615. the content. These modifications are harmless if they only impact elements in
  616. the document that do not carry (text) data. They corrupt your data if they
  617. impact elements that contain data. If lxml cannot distinguish between
  618. whitespace and data, it will not alter your data. Whitespace is therefore
  619. only added between nodes that do not contain data. This is always the case
  620. for trees constructed element-by-element, so no problems should be expected
  621. here. For parsed trees, a good way to assure that no conflicting whitespace
  622. is left in the tree is the ``remove_blank_text`` option:
  623. .. sourcecode:: pycon
  624. >>> parser = etree.XMLParser(remove_blank_text=True)
  625. >>> tree = etree.parse(filename, parser)
  626. This will allow the parser to drop blank text nodes when constructing the
  627. tree. If you now call a serialization function to pretty print this tree,
  628. lxml can add fresh whitespace to the XML tree to indent it.
  629. Fredrik Lundh also has a Python-level function for indenting XML by
  630. appending whitespace to tags. It can be found on his `element
  631. library`_ recipe page.
  632. Why can't lxml parse my XML from unicode strings?
  633. -------------------------------------------------
  634. lxml can read Python unicode strings and even tries to support them if libxml2
  635. does not. However, if the unicode string declares an XML encoding internally
  636. (``<?xml encoding="..."?>``), parsing is bound to fail, as this encoding is
  637. most likely not the real encoding used in Python unicode. The same is true
  638. for HTML unicode strings that contain charset meta tags, although the problems
  639. may be more subtle here. The libxml2 HTML parser may not be able to parse the
  640. meta tags in broken HTML and may end up ignoring them, so even if parsing
  641. succeeds, later handling may still fail with character encoding errors.
  642. Note that Python uses different encodings for unicode on different platforms,
  643. so even specifying the real internal unicode encoding is not portable between
  644. Python interpreters. Don't do it.
  645. Python unicode strings with XML data or HTML data that carry encoding
  646. information are broken. lxml will not parse them. You must provide parsable
  647. data in a valid encoding.
  648. What is the difference between str(xslt(doc)) and xslt(doc).write() ?
  649. ---------------------------------------------------------------------
  650. The str() implementation of the XSLTResultTree class (a subclass of the
  651. ElementTree class) knows about the output method chosen in the stylesheet
  652. (xsl:output), write() doesn't. If you call write(), the result will be a
  653. normal XML tree serialization in the requested encoding. Calling this method
  654. may also fail for XSLT results that are not XML trees (e.g. string results).
  655. If you call str(), it will return the serialized result as specified by the
  656. XSL transform. This correctly serializes string results to encoded Python
  657. strings and honours ``xsl:output`` options like ``indent``. This almost
  658. certainly does what you want, so you should only use ``write()`` if you are
  659. sure that the XSLT result is an XML tree and you want to override the encoding
  660. and indentation options requested by the stylesheet.
  661. Why can't I just delete parents or clear the root node in iterparse()?
  662. ----------------------------------------------------------------------
  663. The ``iterparse()`` implementation is based on the libxml2 parser. It
  664. requires the tree to be intact to finish parsing. If you delete or modify
  665. parents of the current node, chances are you modify the structure in a way
  666. that breaks the parser. Normally, this will result in a segfault. Please
  667. refer to the `iterparse section`_ of the lxml API documentation to find out
  668. what you can do and what you can't do.
  669. .. _`iterparse section`: parsing.html#iterparse-and-iterwalk
  670. How do I output null characters in XML text?
  671. --------------------------------------------
  672. Don't. What you would produce is not well-formed XML. XML parsers
  673. will refuse to parse a document that contains null characters. The
  674. right way to embed binary data in XML is using a text encoding such as
  675. uuencode or base64.
  676. XPath and Document Traversal
  677. ============================
  678. What are the ``findall()`` and ``xpath()`` methods on Element(Tree)?
  679. --------------------------------------------------------------------
  680. ``findall()`` is part of the original `ElementTree API`_. It supports a
  681. `simple subset of the XPath language`_, without predicates, conditions and
  682. other advanced features. It is very handy for finding specific tags in a
  683. tree. Another important difference is namespace handling, which uses the
  684. ``{namespace}tagname`` notation. This is not supported by XPath. The
  685. findall, find and findtext methods are compatible with other ElementTree
  686. implementations and allow writing portable code that runs on ElementTree,
  687. cElementTree and lxml.etree.
  688. ``xpath()``, on the other hand, supports the complete power of the XPath
  689. language, including predicates, XPath functions and Python extension
  690. functions. The syntax is defined by the `XPath specification`_. If you need
  691. the expressiveness and selectivity of XPath, the ``xpath()`` method, the
  692. ``XPath`` class and the ``XPathEvaluator`` are the best choice_.
  693. .. _`simple subset of the XPath language`: http://effbot.org/zone/element-xpath.htm
  694. .. _`XPath specification`: http://www.w3.org/TR/xpath
  695. .. _choice: performance.html#xpath
  696. Why doesn't ``findall()`` support full XPath expressions?
  697. ---------------------------------------------------------
  698. It was decided that it is more important to keep compatibility with
  699. ElementTree_ to simplify code migration between the libraries. The main
  700. difference compared to XPath is the ``{namespace}tagname`` notation used in
  701. ``findall()``, which is not valid XPath.
  702. ElementTree and lxml.etree use the same implementation, which assures 100%
  703. compatibility. Note that ``findall()`` is `so fast`_ in lxml that a native
  704. implementation would not bring any performance benefits.
  705. .. _`so fast`: performance.html#tree-traversal
  706. How can I find out which namespace prefixes are used in a document?
  707. -------------------------------------------------------------------
  708. You can traverse the document (``root.iter()``) and collect the prefix
  709. attributes from all Elements into a set. However, it is unlikely that you
  710. really want to do that. You do not need these prefixes, honestly. You only
  711. need the namespace URIs. All namespace comparisons use these, so feel free to
  712. make up your own prefixes when you use XPath expressions or extension
  713. functions.
  714. The only place where you might consider specifying prefixes is the
  715. serialization of Elements that were created through the API. Here, you can
  716. specify a prefix mapping through the ``nsmap`` argument when creating the root
  717. Element. Its children will then inherit this prefix for serialization.
  718. How can I specify a default namespace for XPath expressions?
  719. ------------------------------------------------------------
  720. You can't. In XPath, there is no such thing as a default namespace. Just use
  721. an arbitrary prefix and let the namespace dictionary of the XPath evaluators
  722. map it to your namespace. See also the question above.