FAQ.txt 56 KB

12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989910010110210310410510610710810911011111211311411511611711811912012112212312412512612712812913013113213313413513613713813914014114214314414514614714814915015115215315415515615715815916016116216316416516616716816917017117217317417517617717817918018118218318418518618718818919019119219319419519619719819920020120220320420520620720820921021121221321421521621721821922022122222322422522622722822923023123223323423523623723823924024124224324424524624724824925025125225325425525625725825926026126226326426526626726826927027127227327427527627727827928028128228328428528628728828929029129229329429529629729829930030130230330430530630730830931031131231331431531631731831932032132232332432532632732832933033133233333433533633733833934034134234334434534634734834935035135235335435535635735835936036136236336436536636736836937037137237337437537637737837938038138238338438538638738838939039139239339439539639739839940040140240340440540640740840941041141241341441541641741841942042142242342442542642742842943043143243343443543643743843944044144244344444544644744844945045145245345445545645745845946046146246346446546646746846947047147247347447547647747847948048148248348448548648748848949049149249349449549649749849950050150250350450550650750850951051151251351451551651751851952052152252352452552652752852953053153253353453553653753853954054154254354454554654754854955055155255355455555655755855956056156256356456556656756856957057157257357457557657757857958058158258358458558658758858959059159259359459559659759859960060160260360460560660760860961061161261361461561661761861962062162262362462562662762862963063163263363463563663763863964064164264364464564664764864965065165265365465565665765865966066166266366466566666766866967067167267367467567667767867968068168268368468568668768868969069169269369469569669769869970070170270370470570670770870971071171271371471571671771871972072172272372472572672772872973073173273373473573673773873974074174274374474574674774874975075175275375475575675775875976076176276376476576676776876977077177277377477577677777877978078178278378478578678778878979079179279379479579679779879980080180280380480580680780880981081181281381481581681781881982082182282382482582682782882983083183283383483583683783883984084184284384484584684784884985085185285385485585685785885986086186286386486586686786886987087187287387487587687787887988088188288388488588688788888989089189289389489589689789889990090190290390490590690790890991091191291391491591691791891992092192292392492592692792892993093193293393493593693793893994094194294394494594694794894995095195295395495595695795895996096196296396496596696796896997097197297397497597697797897998098198298398498598698798898999099199299399499599699799899910001001100210031004100510061007100810091010101110121013101410151016101710181019102010211022102310241025102610271028102910301031103210331034103510361037103810391040104110421043104410451046104710481049105010511052105310541055105610571058105910601061106210631064106510661067106810691070107110721073107410751076107710781079108010811082108310841085108610871088108910901091109210931094109510961097109810991100110111021103110411051106110711081109111011111112111311141115111611171118111911201121112211231124112511261127112811291130113111321133113411351136113711381139114011411142114311441145114611471148114911501151115211531154115511561157115811591160116111621163116411651166116711681169117011711172117311741175117611771178117911801181118211831184118511861187118811891190119111921193119411951196119711981199120012011202120312041205120612071208120912101211121212131214121512161217121812191220122112221223122412251226122712281229123012311232123312341235123612371238123912401241124212431244124512461247124812491250125112521253125412551256125712581259126012611262126312641265126612671268126912701271127212731274127512761277
  1. =====================================
  2. lxml FAQ - Frequently Asked Questions
  3. =====================================
  4. .. meta::
  5. :description: Frequently Asked Questions about lxml (FAQ)
  6. :keywords: lxml, lxml.etree, FAQ, frequently asked questions
  7. Frequently asked questions on lxml. See also the notes on compatibility_ to
  8. ElementTree_.
  9. .. _compatibility: compatibility.html
  10. .. _ElementTree: http://effbot.org/zone/element-index.htm
  11. .. _`build instructions`: build.html
  12. .. _`MacOS-X` : build.html#building-lxml-on-macos-x
  13. .. contents::
  14. ..
  15. 1 General Questions
  16. 1.1 Is there a tutorial?
  17. 1.2 Where can I find more documentation about lxml?
  18. 1.3 What standards does lxml implement?
  19. 1.4 Who uses lxml?
  20. 1.5 What is the difference between lxml.etree and lxml.objectify?
  21. 1.6 How can I make my application run faster?
  22. 1.7 What about that trailing text on serialised Elements?
  23. 1.8 How can I find out if an Element is a comment or PI?
  24. 1.9 How can I map an XML tree into a dict of dicts?
  25. 1.10 Why does lxml sometimes return 'str' values for text in Python 2?
  26. 1.11 Why do I get XInclude or DTD lookup failures on some systems but not on others?
  27. 1.12 How do namespaces work in lxml?
  28. 2 Installation
  29. 2.1 Which version of libxml2 and libxslt should I use or require?
  30. 2.2 Where are the binary builds?
  31. 2.3 Why do I get errors about missing UCS4 symbols when installing lxml?
  32. 2.4 My C compiler crashes on installation
  33. 3 Contributing
  34. 3.1 Why is lxml not written in Python?
  35. 3.2 How can I contribute?
  36. 4 Bugs
  37. 4.1 My application crashes!
  38. 4.2 My application crashes on MacOS-X!
  39. 4.3 I think I have found a bug in lxml. What should I do?
  40. 4.4 How do I know a bug is really in lxml and not in libxml2?
  41. 5 Threading
  42. 5.1 Can I use threads to concurrently access the lxml API?
  43. 5.2 Does my program run faster if I use threads?
  44. 5.3 Would my single-threaded program run faster if I turned off threading?
  45. 5.4 Why can't I reuse XSLT stylesheets in other threads?
  46. 5.5 My program crashes when run with mod_python/Pyro/Zope/Plone/...
  47. 6 Parsing and Serialisation
  48. 6.1 Why doesn't the ``pretty_print`` option reformat my XML output?
  49. 6.2 Why can't lxml parse my XML from unicode strings?
  50. 6.3 Can lxml parse from file objects opened in unicode mode?
  51. 6.4 What is the difference between str(xslt(doc)) and xslt(doc).write() ?
  52. 6.5 Why can't I just delete parents or clear the root node in iterparse()?
  53. 6.6 How do I output null characters in XML text?
  54. 6.7 Is lxml vulnerable to XML bombs?
  55. 6.8 How do I configure lxml safely as a web-service endpoint?
  56. 6.9 How can I sort the attributes?
  57. 7 XPath and Document Traversal
  58. 7.1 What are the ``findall()`` and ``xpath()`` methods on Element(Tree)?
  59. 7.2 Why doesn't ``findall()`` support full XPath expressions?
  60. 7.3 How can I find out which namespace prefixes are used in a document?
  61. 7.4 How can I specify a default namespace for XPath expressions?
  62. 7.5 How can I modify the tree during iteration?
  63. The code examples below use the `'lxml.etree`` module:
  64. .. sourcecode:: pycon
  65. >>> from lxml import etree
  66. ..
  67. >>> import sys
  68. >>> _etree = etree
  69. >>> if sys.version_info[0] >= 3:
  70. ... class etree_mock(object):
  71. ... def __getattr__(self, name): return getattr(_etree, name)
  72. ... def tostring(self, *args, **kwargs):
  73. ... s = _etree.tostring(*args, **kwargs)
  74. ... if isinstance(s, bytes): s = s.decode("utf-8") # CR
  75. ... if s[-1] == '\n': s = s[:-1]
  76. ... return s
  77. ... else:
  78. ... class etree_mock(object):
  79. ... def __getattr__(self, name): return getattr(_etree, name)
  80. ... def tostring(self, *args, **kwargs):
  81. ... s = _etree.tostring(*args, **kwargs)
  82. ... if s[-1] == '\n': s = s[:-1]
  83. ... return s
  84. >>> etree = etree_mock()
  85. General Questions
  86. =================
  87. Is there a tutorial?
  88. --------------------
  89. Read the `lxml.etree Tutorial`_. While this is still work in progress
  90. (just as any good documentation), it provides an overview of the most
  91. important concepts in ``lxml.etree``. If you want to help out,
  92. improving the tutorial is a very good place to start.
  93. There is also a `tutorial for ElementTree`_ which works for
  94. ``lxml.etree``. The documentation of the `extended etree API`_ also
  95. contains many examples for ``lxml.etree``. Fredrik Lundh's `element
  96. library`_ contains a lot of nice recipes that show how to solve common
  97. tasks in ElementTree and lxml.etree. To learn using
  98. ``lxml.objectify``, read the `objectify documentation`_.
  99. John Shipman has written another tutorial called `Python XML
  100. processing with lxml`_ that contains lots of examples. Liza Daly
  101. wrote a nice article about high-performance aspects when `parsing
  102. large files with lxml`_.
  103. .. _`lxml.etree Tutorial`: tutorial.html
  104. .. _`tutorial for ElementTree`: https://web.archive.org/web/20200720191942/https://effbot.org/zone/element.htm
  105. .. _`extended etree API`: api.html
  106. .. _`objectify documentation`: objectify.html
  107. .. _`Python XML processing with lxml`: https://web.archive.org/web/20190522191656/http://infohost.nmt.edu/tcc/help/pubs/pylxml/web/index.html
  108. .. _`element library`: https://web.archive.org/web/20200703234431/http://www.effbot.org/zone/element-lib.htm
  109. .. _`parsing large files with lxml`: http://www.ibm.com/developerworks/xml/library/x-hiperfparse/
  110. Where can I find more documentation about lxml?
  111. -----------------------------------------------
  112. There is a lot of documentation on the web and also in the Python
  113. standard library documentation, as lxml implements the well-known
  114. `ElementTree API`_ and tries to follow its documentation as closely as
  115. possible. The recipes in Fredrik Lundh's `element library`_ are
  116. generally worth taking a look at. There are a couple of issues where
  117. lxml cannot keep up compatibility. They are described in the
  118. compatibility_ documentation.
  119. The lxml specific extensions to the API are described by individual
  120. files in the ``doc`` directory of the source distribution and on `the
  121. web page`_.
  122. The `generated API documentation`_ is a comprehensive API reference
  123. for the lxml package.
  124. .. _`ElementTree API`: https://web.archive.org/web/20200703191710/http://www.effbot.org/zone/element-index.htm
  125. .. _`the web page`: https://lxml.de/#documentation
  126. .. _`generated API documentation`: api/index.html
  127. What standards does lxml implement?
  128. -----------------------------------
  129. The compliance to XML Standards depends on the support in libxml2 and libxslt.
  130. Here is a quote from `http://xmlsoft.org/ <http://xmlsoft.org/>`_:
  131. In most cases libxml2 tries to implement the specifications in a relatively
  132. strictly compliant way. As of release 2.4.16, libxml2 passed all 1800+ tests
  133. from the OASIS XML Tests Suite.
  134. lxml currently supports libxml2 2.6.20 or later, which has even better
  135. support for various XML standards. The important ones are:
  136. * XML 1.0
  137. * HTML 4
  138. * XML namespaces
  139. * XML Schema 1.0
  140. * XPath 1.0
  141. * XInclude 1.0
  142. * XSLT 1.0
  143. * EXSLT
  144. * XML catalogs
  145. * canonical XML
  146. * RelaxNG
  147. * xml:id
  148. * xml:base
  149. Support for XML Schema is currently not 100% complete in libxml2, but
  150. is definitely very close to compliance. Schematron is supported in
  151. two ways, the best being the original ISO Schematron reference
  152. implementation via XSLT. libxml2 also supports loading documents
  153. through HTTP and FTP.
  154. For `RelaxNG Compact Syntax <http://relaxng.org/compact-tutorial-20030326.html>`_
  155. support, there is a tool called `rnc2rng <http://www.gnosis.cx/download/relax/>`_,
  156. written by David Mertz, which you might be able to use from Python. Failing
  157. that, `trang <http://code.google.com/p/jing-trang/>`_ is the 'official'
  158. command line tool (written in Java) to do the conversion.
  159. Who uses lxml?
  160. --------------
  161. As an XML library, lxml is often used under the hood of in-house
  162. server applications, such as web servers or applications that
  163. facilitate some kind of content management. Many people who deploy
  164. Zope_, Plone_ or Django_ use it together with lxml in the background,
  165. without speaking publicly about it. Therefore, it is hard to get an
  166. idea of who uses it, and the following list of 'users and projects we
  167. know of' is very far from a complete list of lxml's users.
  168. Also note that the compatibility to the ElementTree library does not
  169. require projects to set a hard dependency on lxml - as long as they do
  170. not take advantage of lxml's enhanced feature set.
  171. * `cssutils <http://code.google.com/p/cssutils/source/browse/trunk/examples/style.py?r=917>`_,
  172. a CSS parser and toolkit, can be used with ``lxml.cssselect``
  173. * `Deliverance <http://www.openplans.org/projects/deliverance/project-home>`_,
  174. a content theming tool
  175. * `Enfold Proxy 4 <http://www.enfoldsystems.com/Products/Proxy/4>`_,
  176. a web server accelerator with on-the-fly XSLT processing
  177. * `Inteproxy <http://lists.wald.intevation.org/pipermail/inteproxy-devel/2007-February/000000.html>`_,
  178. a secure HTTP proxy
  179. * `lwebstring <http://pypi.python.org/pypi/lwebstring>`_,
  180. an XML template engine
  181. * `openpyxl <https://openpyxl.readthedocs.io/>`_,
  182. a library to read/write MS Excel 2007 files
  183. * `OpenXMLlib <http://permalink.gmane.org/gmane.comp.python.lxml.devel/3250>`_,
  184. a library for handling OpenXML document meta data
  185. * `PsychoPy <http://www.psychopy.org/>`_,
  186. psychology software in Python
  187. * `Pycoon <http://pypi.python.org/pypi/pycoon>`_,
  188. a WSGI web development framework based on XML pipelines
  189. * `pycsw <http://pycsw.org>`_,
  190. an `OGC CSW <http://opengeospatial.org/standards/cat>`_ server implementation written in Python
  191. * `PyQuery <http://pypi.python.org/pypi/pyquery>`_,
  192. a query framework for XML/HTML, similar to jQuery for JavaScript
  193. * `python-docx <http://github.com/mikemaccana/python-docx>`_,
  194. a package for handling Microsoft's Word OpenXML format
  195. * `Rambler <https://www.rambler.ru/>`_,
  196. news aggregator on Runet
  197. * `rdfadict <http://pypi.python.org/pypi/rdfadict>`_,
  198. an RDFa parser with a simple dictionary-like interface.
  199. * `xupdate-processor <http://pypi.python.org/pypi/xupdate-processor>`_,
  200. an XUpdate implementation for lxml.etree
  201. * `Diazo <http://docs.diazo.org/>`_,
  202. an XSLT-under-the-hood web site theming engine
  203. Zope3 and some of its extensions have good support for lxml:
  204. * `gocept.lxml <http://pypi.python.org/pypi/gocept.lxml>`_,
  205. Zope3 interface bindings for lxml
  206. * `z3c.rml <http://pypi.python.org/pypi/z3c.rml>`_,
  207. an implementation of ReportLab's RML format
  208. * `zif.sedna <http://pypi.python.org/pypi/zif.sedna>`_,
  209. an XQuery based interface to the Sedna OpenSource XML database
  210. And don't miss the quotes by our generally happy_ users_, and other
  211. `sites that link to lxml`_. As `Liza Daly`_ puts it: "Many software
  212. products come with the pick-two caveat, meaning that you must choose
  213. only two: speed, flexibility, or readability. When used carefully,
  214. lxml can provide all three."
  215. .. _Zope: http://www.zope.org/
  216. .. _Plone: http://www.plone.org/
  217. .. _Django: https://www.djangoproject.com/
  218. .. _happy: http://thread.gmane.org/gmane.comp.python.lxml.devel/3244/focus=3244
  219. .. _users: http://article.gmane.org/gmane.comp.python.lxml.devel/3246
  220. .. _`sites that link to lxml`: http://www.google.com/search?as_lq=http:%2F%2Flxml.de%2F
  221. .. _`Liza Daly`: http://www.ibm.com/developerworks/xml/library/x-hiperfparse/
  222. What is the difference between lxml.etree and lxml.objectify?
  223. -------------------------------------------------------------
  224. The two modules provide different ways of handling XML. However, objectify
  225. builds on top of lxml.etree and therefore inherits most of its capabilities
  226. and a large portion of its API.
  227. * lxml.etree is a generic API for XML and HTML handling. It aims for
  228. ElementTree compatibility_ and supports the entire XML infoset. It is well
  229. suited for both mixed content and data centric XML. Its generality makes it
  230. the best choice for most applications.
  231. * lxml.objectify is a specialized API for XML data handling in a Python object
  232. syntax. It provides a very natural way to deal with data fields stored in a
  233. structurally well defined XML format. Data is automatically converted to
  234. Python data types and can be manipulated with normal Python operators. Look
  235. at the examples in the `objectify documentation`_ to see what it feels like
  236. to use it.
  237. Objectify is not well suited for mixed contents or HTML documents. As it is
  238. built on top of lxml.etree, however, it inherits the normal support for
  239. XPath, XSLT or validation.
  240. How can I make my application run faster?
  241. -----------------------------------------
  242. lxml.etree is a very fast library for processing XML. There are, however, `a
  243. few caveats`_ involved in the mapping of the powerful libxml2 library to the
  244. simple and convenient ElementTree API. Not all operations are as fast as the
  245. simplicity of the API might suggest, while some use cases can heavily benefit
  246. from finding the right way of doing them. The `benchmark page`_ has a
  247. comparison to other ElementTree implementations and a number of tips for
  248. performance tweaking. As with any Python application, the rule of thumb is:
  249. the more of your processing runs in C, the faster your application gets. See
  250. also the section on threading_.
  251. .. _`a few caveats`: performance.html#the-elementtree-api
  252. .. _`benchmark page`: performance.html
  253. .. _threading: #threading
  254. What about that trailing text on serialised Elements?
  255. -----------------------------------------------------
  256. The ElementTree tree model defines an Element as a container with a tag name,
  257. contained text, child Elements and a tail text. This means that whenever you
  258. serialise an Element, you will get all parts of that Element:
  259. .. sourcecode:: pycon
  260. >>> root = etree.XML("<root><tag>text<child/></tag>tail</root>")
  261. >>> print(etree.tostring(root[0]))
  262. <tag>text<child/></tag>tail
  263. Here is an example that shows why not serialising the tail would be
  264. even more surprising from an object point of view:
  265. .. sourcecode:: pycon
  266. >>> root = etree.Element("test")
  267. >>> root.text = "TEXT"
  268. >>> print(etree.tostring(root))
  269. <test>TEXT</test>
  270. >>> root.tail = "TAIL"
  271. >>> print(etree.tostring(root))
  272. <test>TEXT</test>TAIL
  273. >>> root.tail = None
  274. >>> print(etree.tostring(root))
  275. <test>TEXT</test>
  276. Just imagine a Python list where you append an item and it doesn't
  277. show up when you look at the list.
  278. The ``.tail`` property is a huge simplification for the tree model as
  279. it avoids text nodes to appear in the list of children and makes
  280. access to them quick and simple. So this is a benefit in most
  281. applications and simplifies many, many XML tree algorithms.
  282. However, in document-like XML (and especially HTML), the above result can be
  283. unexpected to new users and can sometimes require a bit more overhead. A good
  284. way to deal with this is to use helper functions that copy the Element without
  285. its tail. The ``lxml.html`` package also deals with this in a couple of
  286. places, as most HTML algorithms benefit from a tail-free behaviour.
  287. How can I find out if an Element is a comment or PI?
  288. ----------------------------------------------------
  289. .. sourcecode:: pycon
  290. >>> root = etree.XML("<?my PI?><root><!-- empty --></root>")
  291. >>> root.tag
  292. 'root'
  293. >>> root.getprevious().tag is etree.PI
  294. True
  295. >>> root[0].tag is etree.Comment
  296. True
  297. How can I map an XML tree into a dict of dicts?
  298. -----------------------------------------------
  299. I'm glad you asked.
  300. .. sourcecode:: python
  301. def recursive_dict(element):
  302. return element.tag, \
  303. dict(map(recursive_dict, element)) or element.text
  304. Note that this beautiful quick-and-dirty converter expects children
  305. to have unique tag names and will silently overwrite any data that
  306. was contained in preceding siblings with the same name. For any
  307. real-world application of xml-to-dict conversion, you would better
  308. write your own, longer version of this.
  309. Why does lxml sometimes return 'str' values for text in Python 2?
  310. -----------------------------------------------------------------
  311. In Python 2, lxml's API returns byte strings for plain ASCII text
  312. values, be it for tag names or text in Element content. This is the
  313. same behaviour as known from ElementTree. The reasoning is that ASCII
  314. encoded byte strings are compatible with Unicode strings in Python 2,
  315. but consume less memory (usually by a factor of 2 or 4) and are faster
  316. to create because they do not require decoding. Plain ASCII string
  317. values are very common in XML, so this optimisation is generally worth
  318. it.
  319. In Python 3, lxml always returns Unicode strings for text and names,
  320. as does ElementTree. Since Python 3.3, Unicode strings containing
  321. only characters that can be encoded in ASCII or Latin-1 are generally
  322. as efficient as byte strings. In older versions of Python 3, the
  323. above mentioned drawbacks apply.
  324. Why do I get XInclude or DTD lookup failures on some systems but not on others?
  325. -------------------------------------------------------------------------------
  326. To avoid network access, external resources are first looked up in
  327. `XML catalogues <https://www.oasis-open.org/committees/entity/spec.html>`_.
  328. Many systems have them installed by default, but some don't.
  329. On Linux systems, the default place to look is the index file
  330. ``/etc/xml/catalog``, which most importantly provides a mapping from
  331. doctype IDs to locally installed DTD files.
  332. See the `libxml2 catalogue documentation <http://xmlsoft.org/catalog.html>`_
  333. for further information.
  334. How do namespaces work in lxml?
  335. -------------------------------
  336. The same as in ElementTree. See the `tutorial <tutorial.html#namespaces>`_.
  337. Installation
  338. ============
  339. Which version of libxml2 and libxslt should I use or require?
  340. -------------------------------------------------------------
  341. It really depends on your application, but the rule of thumb is: more recent
  342. versions contain less bugs and provide more features.
  343. * Do not use the stock libxml2 versions 2.9.11 or 2.9.12. They are incompatible
  344. with lxml and lead to excess output on serialisation. For static builds
  345. against 2.9.12, lxml automatically downloads a post-release version that
  346. contains a work-around.
  347. * Try to use versions of both libraries that were released together. At least
  348. the libxml2 version should not be older than the libxslt version.
  349. * If you use XML Schema or Schematron which are still under development, the
  350. most recent version of libxml2 is usually a good bet.
  351. * The same applies to XPath, where a substantial number of bugs and memory
  352. leaks were fixed over time. If you encounter crashes or memory leaks in
  353. XPath applications, try a more recent version of libxml2.
  354. * For the normal tree handling, however, any libxml2 version starting with
  355. 2.7.x should do.
  356. Read the `release notes of libxml2`_ and the `release notes of libxslt`_ to
  357. see when (or if) a specific bug has been fixed.
  358. .. _`release notes of libxml2`: http://xmlsoft.org/news.html
  359. .. _`release notes of libxslt`: http://xmlsoft.org/XSLT/news.html
  360. Where are the binary builds?
  361. ----------------------------
  362. Thanks to the help by Joar Wandborg, we try to make "manylinux_" binary
  363. builds for Linux available shortly after each source release, as they
  364. are very frequently used by continuous integration and/or build servers.
  365. Thanks to the help by Maximilian Hils and the Appveyor build service,
  366. we also try to serve the frequent requests for binary builds available
  367. for Microsoft Windows in a timely fashion, since users of that platform
  368. usually fail to build lxml themselves. Two of the major design issues
  369. of this operating system make this non-trivial for its users: the lack
  370. of a pre-installed standard compiler and the missing package management.
  371. Besides that, Christoph Gohlke generously provides `unofficial lxml binary
  372. builds for Windows <http://www.lfd.uci.edu/~gohlke/pythonlibs/#lxml>`_
  373. that are usually very up to date. Consider using them if you prefer a
  374. binary build over a signed official source release.
  375. .. _manylinux: https://www.python.org/dev/peps/pep-0513
  376. Why do I get errors about missing UCS4 symbols when installing lxml?
  377. --------------------------------------------------------------------
  378. You are using a Python installation that was configured for a different
  379. internal Unicode representation than the lxml package you are trying to
  380. install. CPython versions before 3.3 allowed to switch between two types
  381. at build time: the 32 bit encoding UCS4 and the 16 bit encoding UCS2.
  382. Sadly, both are not compatible, so eggs and other binary distributions
  383. can only support the one they were compiled with.
  384. This means that you have to compile lxml from sources for your system. Note
  385. that you do not need Cython for this, the lxml source distribution is directly
  386. compilable on both platform types. See the `build instructions`_ on how to do
  387. this.
  388. My C compiler crashes on installation
  389. -------------------------------------
  390. lxml consists of a relatively large amount of (Cython) generated C code
  391. in a single source module. Compiling this module requires a lot of free
  392. memory, usually more than half a GB, which can pose problems especially on
  393. shared/cloud build systems.
  394. If your C compiler crashes while building lxml from sources, consider
  395. using one of the binary wheels that we provide. The "manylinux_" binaries
  396. should generally work well on most build systems and install substantially
  397. faster than a source build.
  398. Contributing
  399. ============
  400. Why is lxml not written in Python?
  401. ----------------------------------
  402. It *almost* is.
  403. lxml is not written in plain Python, because it interfaces with two C
  404. libraries: libxml2 and libxslt. Accessing them at the C-level is
  405. required for performance reasons.
  406. However, to avoid writing plain C-code and caring too much about the
  407. details of built-in types and reference counting, lxml is written in
  408. Cython_, a superset of the Python language that translates to C-code.
  409. Chances are that if you know Python, you can write `code that Cython
  410. accepts`_. Again, the C-ish style used in the lxml code is just for
  411. performance optimisations. If you want to contribute, don't bother
  412. with the details, a Python implementation of your contribution is
  413. better than none. And keep in mind that lxml's flexible API often
  414. favours an implementation of features in pure Python, without
  415. bothering with C-code at all. For example, the ``lxml.html`` package
  416. is written entirely in Python.
  417. Please contact the `mailing list`_ if you need any help.
  418. .. _Cython: http://cython.org/
  419. .. _`code that Cython accepts`: http://docs.cython.org/docs/tutorial.html
  420. How can I contribute?
  421. ---------------------
  422. If you find something that you would like lxml to do (or do better),
  423. then please tell us about it on the `mailing list`_. Pull requests
  424. on github are always appreciated, especially when accompanied by unit
  425. tests and documentation (doctests would be great). See the ``tests``
  426. subdirectories in the lxml source tree (below the ``src`` directory)
  427. and the ReST_ `text files`_ in the ``doc`` directory.
  428. We also have a `list of missing features`_ that we would like to
  429. implement but didn't due to lack if time. If *you* find the time,
  430. patches are very welcome.
  431. .. _ReST: http://docutils.sourceforge.net/rst.html
  432. .. _`text files`: https://github.com/lxml/lxml/tree/master/doc
  433. .. _`list of missing features`: https://github.com/lxml/lxml/blob/master/IDEAS.txt
  434. Besides enhancing the code, there are a lot of places where you can help the
  435. project and its user base. You can
  436. * spread the word and write about lxml. Many users (especially new Python
  437. users) have not yet heard about lxml, although our user base is constantly
  438. growing. If you write your own blog and feel like saying something about
  439. lxml, go ahead and do so. If we think your contribution or criticism is
  440. valuable to other users, we may even put a link or a quote on the project
  441. page.
  442. * provide code examples for the general usage of lxml or specific problems
  443. solved with lxml. Readable code is a very good way of showing how a library
  444. can be used and what great things you can do with it. Again, if we hear
  445. about it, we can set a link on the project page.
  446. * work on the documentation. The web page is generated from a set of ReST_
  447. `text files`_. It is meant both as a representative project page for lxml
  448. and as a site for documenting lxml's API and usage. If you have questions
  449. or an idea how to make it more readable and accessible while you are reading
  450. it, please send a comment to the `mailing list`_.
  451. * enhance the web site. We put some work into making the web site
  452. usable, understandable and also easy to find, but there's always
  453. things that can be done better. You may notice that we are not
  454. top-ranked when searching the web for "Python and XML", so maybe you
  455. have an idea how to improve that.
  456. * help with the tutorial. A tutorial is the most important starting point for
  457. new users, so it is important for us to provide an easy to understand guide
  458. into lxml. As all documentation, the tutorial is work in progress, so we
  459. appreciate every helping hand.
  460. * improve the docstrings. lxml uses docstrings to support Python's integrated
  461. online ``help()`` function. However, sometimes these are not sufficient to
  462. grasp the details of the function in question. If you find such a place,
  463. you can try to write up a better description and send it to the `mailing
  464. list`_.
  465. Bugs
  466. ====
  467. My application crashes!
  468. -----------------------
  469. One of the goals of lxml is "no segfaults", so if there is no clear
  470. warning in the documentation that you were doing something potentially
  471. harmful, you have found a bug and we would like to hear about it.
  472. Please report this bug to the `mailing list`_. See the section on bug
  473. reporting to learn how to do that.
  474. If your application (or e.g. your web container) uses threads, please
  475. see the FAQ section on threading_ to check if you touch on one of the
  476. potential pitfalls.
  477. In any case, try to reproduce the problem with the latest versions of
  478. libxml2 and libxslt. From time to time, bugs and race conditions are found
  479. in these libraries, so a more recent version might already contain a fix for
  480. your problem.
  481. Remember: even if you see lxml appear in a crash stack trace, it is
  482. not necessarily lxml that *caused* the crash.
  483. My application crashes on MacOS-X!
  484. ----------------------------------
  485. This was a common problem up to lxml 2.1.x. Since lxml 2.2, the only
  486. officially supported way to use it on this platform is through a
  487. static build against freshly downloaded versions of libxml2 and
  488. libxslt. See the build instructions for `MacOS-X`_.
  489. I think I have found a bug in lxml. What should I do?
  490. -----------------------------------------------------
  491. First, you should look at the `current developer changelog`_ to see if this
  492. is a known problem that has already been fixed in the master branch since the
  493. release you are using.
  494. .. _`current developer changelog`: https://github.com/lxml/lxml/blob/master/CHANGES.txt
  495. Also, the 'crash' section above has a few good advices what to try to see if
  496. the problem is really in lxml - and not in your setup. Believe it or not,
  497. that happens more often than you might think, especially when old libraries
  498. or even multiple library versions are installed.
  499. You should always try to reproduce the problem with the latest
  500. versions of libxml2 and libxslt - and make sure they are used.
  501. ``lxml.etree`` can tell you what it runs with:
  502. .. sourcecode:: python
  503. import sys
  504. from lxml import etree
  505. print("%-20s: %s" % ('Python', sys.version_info))
  506. print("%-20s: %s" % ('lxml.etree', etree.LXML_VERSION))
  507. print("%-20s: %s" % ('libxml used', etree.LIBXML_VERSION))
  508. print("%-20s: %s" % ('libxml compiled', etree.LIBXML_COMPILED_VERSION))
  509. print("%-20s: %s" % ('libxslt used', etree.LIBXSLT_VERSION))
  510. print("%-20s: %s" % ('libxslt compiled', etree.LIBXSLT_COMPILED_VERSION))
  511. If you can figure that the problem is not in lxml but in the
  512. underlying libxml2 or libxslt, you can ask right on the respective
  513. mailing lists, which may considerably reduce the time to find a fix or
  514. work-around. See the next question for some hints on how to do that.
  515. Otherwise, we would really like to hear about it. Please report it to
  516. the `bug tracker`_ or to the `mailing list`_ so that we can fix it.
  517. It is very helpful in this case if you can come up with a short code
  518. snippet that demonstrates your problem. If others can reproduce and
  519. see the problem, it is much easier for them to fix it - and maybe even
  520. easier for you to describe it and get people convinced that it really
  521. is a problem to fix.
  522. It is important that you always report the version of lxml, libxml2
  523. and libxslt that you get from the code snippet above. If we do not
  524. know the library versions you are using, we will ask back, so it will
  525. take longer for you to get a helpful answer.
  526. Since as a user of lxml you are likely a programmer, you might find
  527. `this article on bug reports`_ an interesting read.
  528. .. _`bug tracker`: https://bugs.launchpad.net/lxml/
  529. .. _`mailing list`: https://lxml.de/mailinglist/
  530. .. _`this article on bug reports`: http://www.chiark.greenend.org.uk/~sgtatham/bugs.html
  531. How do I know a bug is really in lxml and not in libxml2?
  532. ---------------------------------------------------------
  533. A large part of lxml's functionality is implemented by libxml2 and
  534. libxslt, so problems that you encounter may be in one or the other.
  535. Knowing the right place to ask will reduce the time it takes to fix
  536. the problem, or to find a work-around.
  537. Both libxml2 and libxslt come with their own command line frontends,
  538. namely ``xmllint`` and ``xsltproc``. If you encounter problems with
  539. XSLT processing for specific stylesheets or with validation for
  540. specific schemas, try to run the XSLT with ``xsltproc`` or the
  541. validation with ``xmllint`` respectively to find out if it fails there
  542. as well. If it does, please report directly to the mailing lists of
  543. the respective project, namely:
  544. * `libxml2 mailing list <http://mail.gnome.org/mailman/listinfo/xml>`_
  545. * `libxslt mailing list <http://mail.gnome.org/mailman/listinfo/xslt>`_
  546. On the other hand, everything that seems to be related to Python code,
  547. including custom resolvers, custom XPath functions, etc. is likely
  548. outside of the scope of libxml2/libxslt. If you encounter problems
  549. here or you are not sure where there the problem may come from, please
  550. ask on the lxml mailing list first.
  551. In any case, a good explanation of the problem including some simple
  552. test code and some input data will help us (or the libxml2 developers)
  553. see and understand the problem, which largely increases your chance of
  554. getting help. See the question above for a few hints on what is
  555. helpful here.
  556. Threading
  557. =========
  558. Can I use threads to concurrently access the lxml API?
  559. ------------------------------------------------------
  560. Short answer: yes, if you use lxml 2.2 and later.
  561. Since version 1.1, lxml frees the GIL (Python's global interpreter
  562. lock) internally when parsing from disk and memory, as long as you use
  563. either the default parser (which is replicated for each thread) or
  564. create a parser for each thread yourself. lxml also allows
  565. concurrency during validation (RelaxNG and XMLSchema) and XSL
  566. transformation. You can share RelaxNG, XMLSchema and XSLT objects
  567. between threads.
  568. While you can also share parsers between threads, this will serialize
  569. the access to each of them, so it is better to ``.copy()`` parsers or
  570. to just use the default parser if you do not need any special
  571. configuration. The same applies to the XPath evaluators, which use an
  572. internal lock to protect their prepared evaluation contexts. It is
  573. therefore best to use separate evaluator instances in threads.
  574. Warning: Before lxml 2.2, and especially before 2.1, there were
  575. various issues when moving subtrees between different threads, or when
  576. applying XSLT objects from one thread to trees parsed or modified in
  577. another. If you need code to run with older versions, you should
  578. generally avoid modifying trees in other threads than the one it was
  579. generated in. Although this should work in many cases, there are
  580. certain scenarios where the termination of a thread that parsed a tree
  581. can crash the application if subtrees of this tree were moved to other
  582. documents. You should be on the safe side when passing trees between
  583. threads if you either
  584. - do not modify these trees and do not move their elements to other
  585. trees, or
  586. - do not terminate threads while the trees they parsed are still in
  587. use (e.g. by using a fixed size thread-pool or long-running threads
  588. in processing chains)
  589. Since lxml 2.2, even multi-thread pipelines are supported. However,
  590. note that it is more efficient to do all tree work inside one thread,
  591. than to let multiple threads work on a tree one after the other. This
  592. is because trees inherit state from the thread that created them,
  593. which must be maintained when the tree is modified inside another
  594. thread.
  595. Does my program run faster if I use threads?
  596. --------------------------------------------
  597. Depends. The best way to answer this is timing and profiling.
  598. The global interpreter lock (GIL) in Python serializes access to the
  599. interpreter, so if the majority of your processing is done in Python
  600. code (walking trees, modifying elements, etc.), your gain will be
  601. close to zero. The more of your XML processing moves into lxml,
  602. however, the higher your gain. If your application is bound by XML
  603. parsing and serialisation, or by very selective XPath expressions and
  604. complex XSLTs, your speedup on multi-processor machines can be
  605. substantial.
  606. See the question above to learn which operations free the GIL to support
  607. multi-threading.
  608. Would my single-threaded program run faster if I turned off threading?
  609. ----------------------------------------------------------------------
  610. Possibly, yes. You can see for yourself by compiling lxml entirely
  611. without threading support. Pass the ``--without-threading`` option to
  612. setup.py when building lxml from source. You can also build libxml2
  613. without pthread support (``--without-pthreads`` option), which may add
  614. another bit of performance. Note that this will leave internal data
  615. structures entirely without thread protection, so make sure you really
  616. do not use lxml outside of the main application thread in this case.
  617. Why can't I reuse XSLT stylesheets in other threads?
  618. ----------------------------------------------------
  619. Since later lxml 2.0 versions, you can do this. There is some
  620. overhead involved as the result document needs an additional cleanup
  621. traversal when the input document and/or the stylesheet were created
  622. in other threads. However, on a multi-processor machine, the gain of
  623. freeing the GIL easily covers this drawback.
  624. If you need even the last bit of performance, consider keeping (a copy
  625. of) the stylesheet in thread-local storage, and try creating the input
  626. document(s) in the same thread. And do not forget to benchmark your
  627. code to see if the increased code complexity is really worth it.
  628. My program crashes when run with mod_python/Pyro/Zope/Plone/...
  629. ---------------------------------------------------------------
  630. These environments can use threads in a way that may not make it obvious when
  631. threads are created and what happens in which thread. This makes it hard to
  632. ensure lxml's threading support is used in a reliable way. Sadly, if problems
  633. arise, they are as diverse as the applications, so it is difficult to provide
  634. any generally applicable solution. Also, these environments are so complex
  635. that problems become hard to debug and even harder to reproduce in a
  636. predictable way. If you encounter crashes in one of these systems, but your
  637. code runs perfectly when started by hand, the following gives you a few hints
  638. for possible approaches to solve your specific problem:
  639. * make sure you use recent versions of libxml2, libxslt and lxml. The
  640. libxml2 developers keep fixing bugs in each release, and lxml also
  641. tries to become more robust against possible pitfalls. So newer
  642. versions might already fix your problem in a reliable way. Version
  643. 2.2 of lxml contains many improvements.
  644. * make sure the library versions you installed are really used. Do
  645. not rely on what your operating system tells you! Print the version
  646. constants in ``lxml.etree`` from within your runtime environment to
  647. make sure it is the case. This is especially a problem under
  648. MacOS-X when newer library versions were installed in addition to
  649. the outdated system libraries. Please read the bugs section
  650. regarding MacOS-X in this FAQ.
  651. * if you use ``mod_python``, try setting this option:
  652. PythonInterpreter main_interpreter
  653. There was a discussion on the mailing list about this problem:
  654. http://comments.gmane.org/gmane.comp.python.lxml.devel/2942
  655. * in a threaded environment, try to initially import ``lxml.etree``
  656. from the main application thread instead of doing first-time imports
  657. separately in each spawned worker thread. If you cannot control the
  658. thread spawning of your web/application server, an import of
  659. ``lxml.etree`` in sitecustomize.py or usercustomize.py may still do
  660. the trick.
  661. * compile lxml without threading support by running ``setup.py`` with the
  662. ``--without-threading`` option. While this might be slower in certain
  663. scenarios on multi-processor systems, it *might* also keep your application
  664. from crashing, which should be worth more to you than peek performance.
  665. Remember that lxml is fast anyway, so concurrency may not even be worth it.
  666. * look out for fancy XSLT stuff like foreign document access or
  667. passing in subtrees through XSLT variables. This might or might not
  668. work, depending on your specific usage. Again, later versions of
  669. lxml and libxslt provide safer support here.
  670. * try copying trees at suspicious places in your code and working with
  671. those instead of a tree shared between threads. Note that the
  672. copying must happen inside the target thread to be effective, not in
  673. the thread that created the tree. Serialising in one thread and
  674. parsing in another is also a simple (and fast) way of separating
  675. thread contexts.
  676. * try keeping thread-local copies of XSLT stylesheets, i.e. one per thread,
  677. instead of sharing one. Also see the question above.
  678. * you can try to serialise suspicious parts of your code with explicit thread
  679. locks, thus disabling the concurrency of the runtime system.
  680. * report back on the mailing list to see if there are other ways to work
  681. around your specific problems. Do not forget to report the version numbers
  682. of lxml, libxml2 and libxslt you are using (see the question on reporting
  683. a bug).
  684. Note that most of these options will degrade performance and/or your
  685. code quality. If you are unsure what to do, please ask on the mailing
  686. list.
  687. Parsing and Serialisation
  688. =========================
  689. ..
  690. making doctest happy:
  691. >>> try: from StringIO import StringIO
  692. ... except ImportError: from io import StringIO # Py3
  693. >>> filename = StringIO("<root/>")
  694. Why doesn't the ``pretty_print`` option reformat my XML output?
  695. ---------------------------------------------------------------
  696. Pretty printing (or formatting) an XML document means adding white space to
  697. the content. These modifications are harmless if they only impact elements in
  698. the document that do not carry (text) data. They corrupt your data if they
  699. impact elements that contain data. If lxml cannot distinguish between
  700. whitespace and data, it will not alter your data. Whitespace is therefore
  701. only added between nodes that do not contain data. This is always the case
  702. for trees constructed element-by-element, so no problems should be expected
  703. here. For parsed trees, a good way to assure that no conflicting whitespace
  704. is left in the tree is the ``remove_blank_text`` option:
  705. .. sourcecode:: pycon
  706. >>> parser = etree.XMLParser(remove_blank_text=True)
  707. >>> tree = etree.parse(filename, parser)
  708. This will allow the parser to drop blank text nodes when constructing the
  709. tree. If you now call a serialization function to pretty print this tree,
  710. lxml can add fresh whitespace to the XML tree to indent it.
  711. Note that the ``remove_blank_text`` option also uses a heuristic if it
  712. has no definite knowledge about the document's ignorable whitespace.
  713. It will keep blank text nodes that appear after non-blank text nodes
  714. at the same level. This is to prevent document-style XML from losing
  715. content.
  716. The HTMLParser has this structural knowledge built-in, which means that
  717. most whitespace that appears between tags in HTML documents will *not*
  718. be removed by this option, except in places where it is truly ignorable,
  719. e.g. in the page header, between table structure tags, etc. Therefore,
  720. it is also safe to use this option with the HTMLParser, as it will keep
  721. content like the following intact (i.e. it will not remove the space
  722. that separates the two words):
  723. .. sourcecode:: html
  724. <p><b>some</b> <em>text</em></p>
  725. If you want to be sure all blank text is removed from an XML document
  726. (or just more blank text than the parser does by itself), you have to
  727. use either a DTD to tell the parser which whitespace it can safely
  728. ignore, or remove the ignorable whitespace manually after parsing,
  729. e.g. by setting all tail text to None:
  730. .. sourcecode:: python
  731. for element in root.iter():
  732. element.tail = None
  733. Fredrik Lundh also has a Python-level function for indenting XML by
  734. appending whitespace to tags. It can be found on his `element library
  735. recipes page <http://effbot.org/zone/element-lib.htm#prettyprint>`_.
  736. Why can't lxml parse my XML from unicode strings?
  737. -------------------------------------------------
  738. First of all, XML is explicitly defined as a stream of bytes. It's not
  739. Unicode text. Take a look at the `XML specification`_, it's all about byte
  740. sequences and how to map them to text and structure. That leads to rule
  741. number one: do not decode your XML data yourself. That's a part of the
  742. work of an XML parser, and it does it very well. Just pass it your data as
  743. a plain byte stream, it will always do the right thing, by specification.
  744. This also includes not opening XML files in text mode. Make sure you always
  745. use binary mode, or, even better, pass the file path into lxml's ``parse()``
  746. function to let it do the file opening, reading and closing itself. This
  747. is the most simple and most efficient way to do it.
  748. That being said, lxml can read Python unicode strings and even tries to
  749. support them if libxml2 does not. This is because there is one valid use
  750. case for parsing XML from text strings: literal XML fragments in source
  751. code.
  752. However, if the unicode string declares an XML encoding internally
  753. (``<?xml encoding="..."?>``), parsing is bound to fail, as this encoding is
  754. almost certainly not the real encoding used in Python unicode. The same is
  755. true for HTML unicode strings that contain charset meta tags, although the
  756. problems may be more subtle here. The libxml2 HTML parser may not be able
  757. to parse the meta tags in broken HTML and may end up ignoring them, so even
  758. if parsing succeeds, later handling may still fail with character encoding
  759. errors. Therefore, parsing HTML from unicode strings is a much saner thing
  760. to do than parsing XML from unicode strings.
  761. Note that Python uses different encodings for unicode on different platforms,
  762. so even specifying the real internal unicode encoding is not portable between
  763. Python interpreters. Don't do it.
  764. Python unicode strings with XML data that carry encoding information are
  765. broken. lxml will not parse them. You must provide parsable data in a
  766. valid encoding.
  767. .. _`XML specification`: http://www.w3.org/TR/REC-xml/
  768. Can lxml parse from file objects opened in unicode/text mode?
  769. -------------------------------------------------------------
  770. Technically, yes. However, you likely do not want to do that, because
  771. it is extremely inefficient. The text encoding that libxml2 uses
  772. internally is UTF-8, so parsing from a Unicode file means that Python
  773. first reads a chunk of data from the file, then decodes it into a new
  774. buffer, and then copies it into a new unicode string object, just to
  775. let libxml2 make yet another copy while encoding it down into UTF-8
  776. in order to parse it. It's clear that this involves a lot more
  777. recoding and copying than when parsing straight from the bytes that
  778. the file contains.
  779. If you really know the encoding better than the parser (e.g. when
  780. parsing HTML that lacks a content declaration), then instead of passing
  781. an encoding parameter into the file object when opening it, create a
  782. new instance of an XMLParser or HTMLParser and pass the encoding into
  783. its constructor. Afterwards, use that parser for parsing, e.g. by
  784. passing it into the ``etree.parse(file, parser)`` function. Remember
  785. to open the file in binary mode (mode="rb"), or, if possible, prefer
  786. passing the file path directly into ``parse()`` instead of an opened
  787. Python file object.
  788. What is the difference between str(xslt(doc)) and xslt(doc).write() ?
  789. ---------------------------------------------------------------------
  790. The str() implementation of the XSLTResultTree class (a subclass of the
  791. ElementTree class) knows about the output method chosen in the stylesheet
  792. (xsl:output), write() doesn't. If you call write(), the result will be a
  793. normal XML tree serialization in the requested encoding. Calling this method
  794. may also fail for XSLT results that are not XML trees (e.g. string results).
  795. If you call str(), it will return the serialized result as specified by the
  796. XSL transform. This correctly serializes string results to encoded Python
  797. strings and honours ``xsl:output`` options like ``indent``. This almost
  798. certainly does what you want, so you should only use ``write()`` if you are
  799. sure that the XSLT result is an XML tree and you want to override the encoding
  800. and indentation options requested by the stylesheet.
  801. Why can't I just delete parents or clear the root node in iterparse()?
  802. ----------------------------------------------------------------------
  803. The ``iterparse()`` implementation is based on the libxml2 parser. It
  804. requires the tree to be intact to finish parsing. If you delete or modify
  805. parents of the current node, chances are you modify the structure in a way
  806. that breaks the parser. Normally, this will result in a segfault. Please
  807. refer to the `iterparse section`_ of the lxml API documentation to find out
  808. what you can do and what you can't do.
  809. .. _`iterparse section`: parsing.html#iterparse-and-iterwalk
  810. How do I output null characters in XML text?
  811. --------------------------------------------
  812. Don't. What you would produce is not well-formed XML. XML parsers
  813. will refuse to parse a document that contains null characters. The
  814. right way to embed binary data in XML is using a text encoding such as
  815. uuencode or base64.
  816. Is lxml vulnerable to XML bombs?
  817. --------------------------------
  818. This has nothing to do with lxml itself, only with the parser of
  819. libxml2. Since libxml2 version 2.7, the parser imposes hard security
  820. limits on input documents to prevent DoS attacks with forged input
  821. data. Since lxml 2.2.1, you can disable these limits with the
  822. ``huge_tree`` parser option if you need to parse *really* large,
  823. trusted documents. All lxml versions will leave these restrictions
  824. enabled by default.
  825. Note that libxml2 versions of the 2.6 series do not restrict their
  826. parser and are therefore vulnerable to DoS attacks.
  827. Note also that these "hard limits" may still be high enough to
  828. allow for excessive resource usage in a given use case. They are
  829. compile time modifiable, so building your own library versions will
  830. allow you to change the limits to your own needs. Also see the next
  831. question.
  832. How do I use lxml safely as a web-service endpoint?
  833. ---------------------------------------------------
  834. XML based web-service endpoints are generally subject to several
  835. types of attacks if they allow some kind of untrusted input.
  836. From the point of view of the underlying XML tool, the most
  837. obvious attacks try to send a relatively small amount of data
  838. that induces a comparatively large resource consumption on the
  839. receiver side.
  840. First of all, make sure network access is not enabled for the XML
  841. parser that you use for parsing untrusted content and that it is
  842. not configured to load external DTDs. Otherwise, attackers can
  843. try to trick the parser into an attempt to load external resources
  844. that are overly slow or impossible to retrieve, thus wasting time
  845. and other valuable resources on your server such as socket
  846. connections. Note that you can register your own document loader
  847. in lxml, which allows for fine-grained control over any read access
  848. to resources.
  849. Some of the most famous excessive content expansion attacks
  850. use XML entity references. Luckily, entity expansion is mostly
  851. useless for the data commonly sent through web services and
  852. can simply be disabled, which rules out several types of
  853. denial of service attacks at once. This also involves an attack
  854. that reads local files from the server, as XML entities can be
  855. defined to expand into their content. Consequently, version
  856. 1.2 of the SOAP standard explicitly disallows entity references
  857. in the XML stream.
  858. To disable entity expansion, use an XML parser that is configured
  859. with the option ``resolve_entities=False``. Then, after (or
  860. while) parsing the document, use ``root.iter(etree.Entity)`` to
  861. recursively search for entity references. If it contains any,
  862. reject the entire input document with a suitable error response.
  863. In lxml 3.x, you can also use the new DTD introspection API to
  864. apply your own restrictions on input documents.
  865. Another attack to consider is compression bombs. If you allow
  866. compressed input into your web service, attackers can try to send
  867. well forged highly repetitive and thus very well compressing input
  868. that unpacks into a very large XML document in your server's main
  869. memory, potentially a thousand times larger than the compressed
  870. input data.
  871. As a counter measure, either disable compressed input for your
  872. web server, at least for untrusted sources, or use incremental
  873. parsing with ``iterparse()`` instead of parsing the whole input
  874. document into memory in one shot. That allows you to enforce
  875. suitable limits on the input by applying semantic checks that
  876. detect and prevent an illegitimate use of your service. If
  877. possible, you can also use this to reduce the amount of data
  878. that you need to keep in memory while parsing the document,
  879. thus further reducing the possibility of an attacker to trick
  880. your system into excessive resource usage.
  881. Finally, please be aware that XPath suffers from the same
  882. vulnerability as SQL when it comes to content injection. The
  883. obvious fix is to not build any XPath expressions via string
  884. formatting or concatenation when the parameters may come from
  885. untrusted sources, and instead use XPath variables, which
  886. safely expose their values to the evaluation engine.
  887. The defusedxml_ package comes with an example setup and a wrapper
  888. API for lxml that applies certain counter measures internally.
  889. .. _defusedxml: https://bitbucket.org/tiran/defusedxml
  890. How can I sort the attributes?
  891. ------------------------------
  892. lxml preserves the order in which attributes were originally created.
  893. There is one case in which this is difficult: when attributes are passed
  894. in a dict or as keyword arguments to the `Element()` factory. Before Python
  895. 3.6, dicts had no predictable order.
  896. Since Python 3.6, however, dicts also preserve the creation order of their keys,
  897. and lxml makes use of that since release 4.4.
  898. In earlier versions, lxml tries to assure at least reproducible output by
  899. sorting the attributes from the dict before creating them. All sequential
  900. ways to set attributes keep their order and do not apply sorting. Also,
  901. OrderedDict instances are recognised and not sorted.
  902. In cases where you cannot control the order in which attributes are created,
  903. you can still change it before serialisation. To sort them by name, for example,
  904. you can apply the following function:
  905. .. sourcecode:: python
  906. def sort_attributes(root):
  907. for el in root.iter():
  908. attrib = el.attrib
  909. if len(attrib) > 1:
  910. attributes = sorted(attrib.items())
  911. attrib.clear()
  912. attrib.update(attributes)
  913. XPath and Document Traversal
  914. ============================
  915. What are the ``findall()`` and ``xpath()`` methods on Element(Tree)?
  916. --------------------------------------------------------------------
  917. ``findall()`` is part of the original `ElementTree API`_. It supports a
  918. `simple subset of the XPath language`_, without predicates, conditions and
  919. other advanced features. It is very handy for finding specific tags in a
  920. tree. Another important difference is namespace handling, which uses the
  921. ``{namespace}tagname`` notation. This is not supported by XPath. The
  922. findall, find and findtext methods are compatible with other ElementTree
  923. implementations and allow writing portable code that runs on ElementTree,
  924. cElementTree and lxml.etree.
  925. ``xpath()``, on the other hand, supports the complete power of the XPath
  926. language, including predicates, XPath functions and Python extension
  927. functions. The syntax is defined by the `XPath specification`_. If you need
  928. the expressiveness and selectivity of XPath, the ``xpath()`` method, the
  929. ``XPath`` class and the ``XPathEvaluator`` are the best choice_.
  930. .. _`simple subset of the XPath language`: http://effbot.org/zone/element-xpath.htm
  931. .. _`XPath specification`: http://www.w3.org/TR/xpath
  932. .. _choice: performance.html#xpath
  933. Why doesn't ``findall()`` support full XPath expressions?
  934. ---------------------------------------------------------
  935. It was decided that it is more important to keep compatibility with
  936. ElementTree_ to simplify code migration between the libraries. The main
  937. difference compared to XPath is the ``{namespace}tagname`` notation used in
  938. ``findall()``, which is not valid XPath.
  939. ElementTree and lxml.etree use the same implementation, which assures 100%
  940. compatibility. Note that ``findall()`` is `so fast`_ in lxml that a native
  941. implementation would not bring any performance benefits.
  942. .. _`so fast`: performance.html#tree-traversal
  943. How can I find out which namespace prefixes are used in a document?
  944. -------------------------------------------------------------------
  945. You can traverse the document (``root.iter()``) and collect the prefix
  946. attributes from all Elements into a set. However, it is unlikely that you
  947. really want to do that. You do not need these prefixes, honestly. You only
  948. need the namespace URIs. All namespace comparisons use these, so feel free to
  949. make up your own prefixes when you use XPath expressions or extension
  950. functions.
  951. The only place where you might consider specifying prefixes is the
  952. serialization of Elements that were created through the API. Here, you can
  953. specify a prefix mapping through the ``nsmap`` argument when creating the root
  954. Element. Its children will then inherit this prefix for serialization.
  955. How can I specify a default namespace for XPath expressions?
  956. ------------------------------------------------------------
  957. You can't. In XPath 1.0, there is no such thing as a default namespace. Just
  958. use an arbitrary prefix and let the namespace dictionary of the XPath evaluators
  959. map it to your namespace. See also the question above.
  960. How can I modify the tree during iteration?
  961. -------------------------------------------
  962. lxml's iterators need to hold on to an element in the tree in order to remember
  963. their current position. Therefore, tree modifications between two calls into the
  964. iterator can lead to surprising results if such an element is deleted or moved
  965. around, for example.
  966. If your code risks modifying elements that the iterator might still need, and
  967. you know that the number of elements returned by the iterator is small, then just
  968. read them all into a list (or use ``.findall()``), and iterate over that list.
  969. If the number of elements can be larger and you really want to process the tree
  970. incrementally, you can often use a read-ahead generator to make the iterator
  971. advance beyond the critical point before touching the tree structure.
  972. For example:
  973. .. sourcecode:: python
  974. from itertools import islice
  975. from collections import deque
  976. def readahead(iterator, count=1):
  977. iterator = iter(iterator) # allow iterables as well
  978. elements = deque(islice(iterator, 0, count))
  979. for element in iterator:
  980. elements.append(element)
  981. yield elements.popleft()
  982. yield from elements
  983. for element in readahead(root.iterfind("path/to/children")):
  984. element.getparent().remove(element)