parsing.txt 32 KB

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980818283848586878889909192939495969798991001011021031041051061071081091101111121131141151161171181191201211221231241251261271281291301311321331341351361371381391401411421431441451461471481491501511521531541551561571581591601611621631641651661671681691701711721731741751761771781791801811821831841851861871881891901911921931941951961971981992002012022032042052062072082092102112122132142152162172182192202212222232242252262272282292302312322332342352362372382392402412422432442452462472482492502512522532542552562572582592602612622632642652662672682692702712722732742752762772782792802812822832842852862872882892902912922932942952962972982993003013023033043053063073083093103113123133143153163173183193203213223233243253263273283293303313323333343353363373383393403413423433443453463473483493503513523533543553563573583593603613623633643653663673683693703713723733743753763773783793803813823833843853863873883893903913923933943953963973983994004014024034044054064074084094104114124134144154164174184194204214224234244254264274284294304314324334344354364374384394404414424434444454464474484494504514524534544554564574584594604614624634644654664674684694704714724734744754764774784794804814824834844854864874884894904914924934944954964974984995005015025035045055065075085095105115125135145155165175185195205215225235245255265275285295305315325335345355365375385395405415425435445455465475485495505515525535545555565575585595605615625635645655665675685695705715725735745755765775785795805815825835845855865875885895905915925935945955965975985996006016026036046056066076086096106116126136146156166176186196206216226236246256266276286296306316326336346356366376386396406416426436446456466476486496506516526536546556566576586596606616626636646656666676686696706716726736746756766776786796806816826836846856866876886896906916926936946956966976986997007017027037047057067077087097107117127137147157167177187197207217227237247257267277287297307317327337347357367377387397407417427437447457467477487497507517527537547557567577587597607617627637647657667677687697707717727737747757767777787797807817827837847857867877887897907917927937947957967977987998008018028038048058068078088098108118128138148158168178188198208218228238248258268278288298308318328338348358368378388398408418428438448458468478488498508518528538548558568578588598608618628638648658668678688698708718728738748758768778788798808818828838848858868878888898908918928938948958968978988999009019029039049059069079089099109119129139149159169179189199209219229239249259269279289299309319329339349359369379389399409419429439449459469479489499509519529539549559569579589599609619629639649659669679689699709719729739749759769779789799809819829839849859869879889899909919929939949959969979989991000100110021003100410051006100710081009101010111012101310141015101610171018101910201021102210231024
  1. ==============================
  2. Parsing XML and HTML with lxml
  3. ==============================
  4. lxml provides a very simple and powerful API for parsing XML and HTML. It
  5. supports one-step parsing as well as step-by-step parsing using an
  6. event-driven API (currently only for XML).
  7. .. contents::
  8. ..
  9. 1 Parsers
  10. 1.1 Parser options
  11. 1.2 Error log
  12. 1.3 Parsing HTML
  13. 1.4 Doctype information
  14. 2 The target parser interface
  15. 3 The feed parser interface
  16. 4 Incremental event parsing
  17. 4.1 Event types
  18. 4.1 Modifying the tree
  19. 4.3 Selective tag events
  20. 4.4 Comments and PIs
  21. 4.5 Events with custom targets
  22. 5 iterparse and iterwalk
  23. 5.1 iterwalk
  24. 6 Python unicode strings
  25. 6.1 Serialising to Unicode strings
  26. The usual setup procedure:
  27. .. sourcecode:: pycon
  28. >>> from lxml import etree
  29. The following examples also use StringIO or BytesIO to show how to parse
  30. from files and file-like objects. Both are available in the ``io`` module:
  31. .. sourcecode:: python
  32. from io import StringIO, BytesIO
  33. ..
  34. >>> from lxml import usedoctest
  35. >>> try: from StringIO import StringIO
  36. ... except ImportError:
  37. ... from io import BytesIO
  38. ... def StringIO(s):
  39. ... if isinstance(s, str): s = s.encode("UTF-8")
  40. ... return BytesIO(s)
  41. >>> try: unicode = unicode
  42. ... except NameError: unicode = str
  43. >>> import sys
  44. >>> from lxml import etree as _etree
  45. >>> if sys.version_info[0] >= 3:
  46. ... class etree_mock(object):
  47. ... def __getattr__(self, name): return getattr(_etree, name)
  48. ... def tostring(self, *args, **kwargs):
  49. ... s = _etree.tostring(*args, **kwargs)
  50. ... if isinstance(s, bytes) and bytes([10]) in s: s = s.decode("utf-8") # CR
  51. ... if s[-1] == '\n': s = s[:-1]
  52. ... return s
  53. ... else:
  54. ... class etree_mock(object):
  55. ... def __getattr__(self, name): return getattr(_etree, name)
  56. ... def tostring(self, *args, **kwargs):
  57. ... s = _etree.tostring(*args, **kwargs)
  58. ... if s[-1] == '\n': s = s[:-1]
  59. ... return s
  60. >>> etree = etree_mock()
  61. Parsers
  62. =======
  63. Parsers are represented by parser objects. There is support for parsing both
  64. XML and (broken) HTML. Note that XHTML is best parsed as XML, parsing it with
  65. the HTML parser can lead to unexpected results. Here is a simple example for
  66. parsing XML from an in-memory string:
  67. .. sourcecode:: pycon
  68. >>> xml = '<a xmlns="test"><b xmlns="test"/></a>'
  69. >>> root = etree.fromstring(xml)
  70. >>> etree.tostring(root)
  71. b'<a xmlns="test"><b xmlns="test"/></a>'
  72. To read from a file or file-like object, you can use the ``parse()`` function,
  73. which returns an ``ElementTree`` object:
  74. .. sourcecode:: pycon
  75. >>> tree = etree.parse(StringIO(xml))
  76. >>> etree.tostring(tree.getroot())
  77. b'<a xmlns="test"><b xmlns="test"/></a>'
  78. Note how the ``parse()`` function reads from a file-like object here. If
  79. parsing is done from a real file, it is more common (and also somewhat more
  80. efficient) to pass a filename:
  81. .. sourcecode:: pycon
  82. >>> tree = etree.parse("doc/test.xml")
  83. lxml can parse from a local file, an HTTP URL or an FTP URL. It also
  84. auto-detects and reads gzip-compressed XML files (.gz).
  85. If you want to parse from memory and still provide a base URL for the document
  86. (e.g. to support relative paths in an XInclude), you can pass the ``base_url``
  87. keyword argument:
  88. .. sourcecode:: pycon
  89. >>> root = etree.fromstring(xml, base_url="http://where.it/is/from.xml")
  90. Parser options
  91. --------------
  92. The parsers accept a number of setup options as keyword arguments. The above
  93. example is easily extended to clean up namespaces during parsing:
  94. .. sourcecode:: pycon
  95. >>> parser = etree.XMLParser(ns_clean=True)
  96. >>> tree = etree.parse(StringIO(xml), parser)
  97. >>> etree.tostring(tree.getroot())
  98. b'<a xmlns="test"><b/></a>'
  99. The keyword arguments in the constructor are mainly based on the libxml2
  100. parser configuration. A DTD will also be loaded if validation or attribute
  101. default values are requested.
  102. Available boolean keyword arguments:
  103. * attribute_defaults - read the DTD (if referenced by the document) and add
  104. the default attributes from it
  105. * dtd_validation - validate while parsing (if a DTD was referenced)
  106. * load_dtd - load and parse the DTD while parsing (no validation is performed)
  107. * no_network - prevent network access when looking up external
  108. documents (on by default)
  109. * ns_clean - try to clean up redundant namespace declarations
  110. * recover - try hard to parse through broken XML
  111. * remove_blank_text - discard blank text nodes between tags, also known as
  112. ignorable whitespace. This is best used together with a DTD or schema
  113. (which tells data and noise apart), otherwise a heuristic will be applied.
  114. * remove_comments - discard comments
  115. * remove_pis - discard processing instructions
  116. * strip_cdata - replace CDATA sections by normal text content (on by
  117. default)
  118. * resolve_entities - replace entities by their text value (on by
  119. default)
  120. * huge_tree - disable security restrictions and support very deep trees
  121. and very long text content (only affects libxml2 2.7+)
  122. * compact - use compact storage for short text content (on by default)
  123. * collect_ids - collect XML IDs in a hash table while parsing (on by default).
  124. Disabling this can substantially speed up parsing of documents with many
  125. different IDs if the hash lookup is not used afterwards.
  126. Other keyword arguments:
  127. * encoding - override the document encoding
  128. * target - a parser target object that will receive the parse events
  129. (see `The target parser interface`_)
  130. * schema - an XMLSchema to validate against (see `validation <validation.html#xmlschema>`_)
  131. Error log
  132. ---------
  133. Parsers have an ``error_log`` property that lists the errors and
  134. warnings of the last parser run:
  135. .. sourcecode:: pycon
  136. >>> parser = etree.XMLParser()
  137. >>> print(len(parser.error_log))
  138. 0
  139. >>> tree = etree.XML("<root>\n</b>", parser)
  140. Traceback (most recent call last):
  141. ...
  142. lxml.etree.XMLSyntaxError: Opening and ending tag mismatch: root line 1 and b, line 2, column 5
  143. >>> print(len(parser.error_log))
  144. 1
  145. >>> error = parser.error_log[0]
  146. >>> print(error.message)
  147. Opening and ending tag mismatch: root line 1 and b
  148. >>> print(error.line)
  149. 2
  150. >>> print(error.column)
  151. 5
  152. Each entry in the log has the following properties:
  153. * ``message``: the message text
  154. * ``domain``: the domain ID (see the lxml.etree.ErrorDomains class)
  155. * ``type``: the message type ID (see the lxml.etree.ErrorTypes class)
  156. * ``level``: the log level ID (see the lxml.etree.ErrorLevels class)
  157. * ``line``: the line at which the message originated (if applicable)
  158. * ``column``: the character column at which the message originated (if applicable)
  159. * ``filename``: the name of the file in which the message originated (if applicable)
  160. For convenience, there are also three properties that provide readable
  161. names for the ID values:
  162. * ``domain_name``
  163. * ``type_name``
  164. * ``level_name``
  165. To filter for a specific kind of message, use the different
  166. ``filter_*()`` methods on the error log (see the
  167. lxml.etree._ListErrorLog class).
  168. Parsing HTML
  169. ------------
  170. HTML parsing is similarly simple. The parsers have a ``recover``
  171. keyword argument that the HTMLParser sets by default. It lets libxml2
  172. try its best to return a valid HTML tree with all content it can
  173. manage to parse. It will not raise an exception on parser errors.
  174. You should use libxml2 version 2.6.21 or newer to take advantage of
  175. this feature.
  176. .. sourcecode:: pycon
  177. >>> broken_html = "<html><head><title>test<body><h1>page title</h3>"
  178. >>> parser = etree.HTMLParser()
  179. >>> tree = etree.parse(StringIO(broken_html), parser)
  180. >>> result = etree.tostring(tree.getroot(),
  181. ... pretty_print=True, method="html")
  182. >>> print(result)
  183. <html>
  184. <head>
  185. <title>test</title>
  186. </head>
  187. <body>
  188. <h1>page title</h1>
  189. </body>
  190. </html>
  191. Lxml has an HTML function, similar to the XML shortcut known from
  192. ElementTree:
  193. .. sourcecode:: pycon
  194. >>> html = etree.HTML(broken_html)
  195. >>> result = etree.tostring(html, pretty_print=True, method="html")
  196. >>> print(result)
  197. <html>
  198. <head>
  199. <title>test</title>
  200. </head>
  201. <body>
  202. <h1>page title</h1>
  203. </body>
  204. </html>
  205. The support for parsing broken HTML depends entirely on libxml2's recovery
  206. algorithm. It is *not* the fault of lxml if you find documents that are so
  207. heavily broken that the parser cannot handle them. There is also no guarantee
  208. that the resulting tree will contain all data from the original document. The
  209. parser may have to drop seriously broken parts when struggling to keep
  210. parsing. Especially misplaced meta tags can suffer from this, which may lead
  211. to encoding problems.
  212. Note that the result is a valid HTML tree, but it may not be a
  213. well-formed XML tree. For example, XML forbids double hyphens in
  214. comments, which the HTML parser will happily accept in recovery mode.
  215. Therefore, if your goal is to serialise an HTML document as an
  216. XML/XHTML document after parsing, you may have to apply some manual
  217. preprocessing first.
  218. Also note that the HTML parser is meant to parse HTML documents. For
  219. XHTML documents, use the XML parser, which is namespace aware.
  220. Doctype information
  221. -------------------
  222. The use of the libxml2 parsers makes some additional information available at
  223. the API level. Currently, ElementTree objects can access the DOCTYPE
  224. information provided by a parsed document, as well as the XML version and the
  225. original encoding:
  226. .. sourcecode:: pycon
  227. >>> pub_id = "-//W3C//DTD XHTML 1.0 Transitional//EN"
  228. >>> sys_url = "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"
  229. >>> doctype_string = '<!DOCTYPE html PUBLIC "%s" "%s">' % (pub_id, sys_url)
  230. >>> xml_header = '<?xml version="1.0" encoding="ascii"?>'
  231. >>> xhtml = xml_header + doctype_string + '<html><body></body></html>'
  232. >>> tree = etree.parse(StringIO(xhtml))
  233. >>> docinfo = tree.docinfo
  234. >>> print(docinfo.public_id)
  235. -//W3C//DTD XHTML 1.0 Transitional//EN
  236. >>> print(docinfo.system_url)
  237. http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd
  238. >>> docinfo.doctype == doctype_string
  239. True
  240. >>> print(docinfo.xml_version)
  241. 1.0
  242. >>> print(docinfo.encoding)
  243. ascii
  244. The target parser interface
  245. ===========================
  246. .. _`As in ElementTree`: http://effbot.org/elementtree/elementtree-xmlparser.htm
  247. `As in ElementTree`_, and similar to a SAX event handler, you can pass
  248. a target object to the parser:
  249. .. sourcecode:: pycon
  250. >>> class EchoTarget(object):
  251. ... def start(self, tag, attrib):
  252. ... print("start %s %r" % (tag, dict(attrib)))
  253. ... def end(self, tag):
  254. ... print("end %s" % tag)
  255. ... def data(self, data):
  256. ... print("data %r" % data)
  257. ... def comment(self, text):
  258. ... print("comment %s" % text)
  259. ... def close(self):
  260. ... print("close")
  261. ... return "closed!"
  262. >>> parser = etree.XMLParser(target = EchoTarget())
  263. >>> result = etree.XML("<element>some<!--comment-->text</element>",
  264. ... parser)
  265. start element {}
  266. data u'some'
  267. comment comment
  268. data u'text'
  269. end element
  270. close
  271. >>> print(result)
  272. closed!
  273. It is important for the ``.close()`` method to reset the parser target
  274. to a usable state, so that you can reuse the parser as often as you
  275. like:
  276. .. sourcecode:: pycon
  277. >>> result = etree.XML("<element>some<!--comment-->text</element>",
  278. ... parser)
  279. start element {}
  280. data u'some'
  281. comment comment
  282. data u'text'
  283. end element
  284. close
  285. >>> print(result)
  286. closed!
  287. Starting with lxml 2.3, the ``.close()`` method will also be called in
  288. the error case. This diverges from the behaviour of ElementTree, but
  289. allows target objects to clean up their state in all situations, so
  290. that the parser can reuse them afterwards.
  291. .. sourcecode:: pycon
  292. >>> class CollectorTarget(object):
  293. ... def __init__(self):
  294. ... self.events = []
  295. ... def start(self, tag, attrib):
  296. ... self.events.append("start %s %r" % (tag, dict(attrib)))
  297. ... def end(self, tag):
  298. ... self.events.append("end %s" % tag)
  299. ... def data(self, data):
  300. ... self.events.append("data %r" % data)
  301. ... def comment(self, text):
  302. ... self.events.append("comment %s" % text)
  303. ... def close(self):
  304. ... self.events.append("close")
  305. ... return "closed!"
  306. >>> parser = etree.XMLParser(target = CollectorTarget())
  307. >>> result = etree.XML("<element>some</error>",
  308. ... parser) # doctest: +ELLIPSIS
  309. Traceback (most recent call last):
  310. ...
  311. lxml.etree.XMLSyntaxError: Opening and ending tag mismatch...
  312. >>> for event in parser.target.events:
  313. ... print(event)
  314. start element {}
  315. data u'some'
  316. close
  317. Note that the parser does *not* build a tree when using a parser
  318. target. The result of the parser run is whatever the target object
  319. returns from its ``.close()`` method. If you want to return an XML
  320. tree here, you have to create it programmatically in the target
  321. object. An example for a parser target that builds a tree is the
  322. ``TreeBuilder``:
  323. .. sourcecode:: pycon
  324. >>> parser = etree.XMLParser(target = etree.TreeBuilder())
  325. >>> result = etree.XML("<element>some<!--comment-->text</element>",
  326. ... parser)
  327. >>> print(result.tag)
  328. element
  329. >>> print(result[0].text)
  330. comment
  331. The feed parser interface
  332. =========================
  333. Since lxml 2.0, the parsers have a feed parser interface that is
  334. compatible to the `ElementTree parsers`_. You can use it to feed data
  335. into the parser in a controlled step-by-step way.
  336. In lxml.etree, you can use both interfaces to a parser at the same
  337. time: the ``parse()`` or ``XML()`` functions, and the feed parser
  338. interface. Both are independent and will not conflict (except if used
  339. in conjunction with a parser target object as described above).
  340. .. _`ElementTree parsers`: http://effbot.org/elementtree/elementtree-xmlparser.htm
  341. To start parsing with a feed parser, just call its ``feed()`` method
  342. to feed it some data.
  343. .. sourcecode:: pycon
  344. >>> parser = etree.XMLParser()
  345. >>> for data in ('<?xml versio', 'n="1.0"?', '><roo', 't><a', '/></root>'):
  346. ... parser.feed(data)
  347. When you are done parsing, you **must** call the ``close()`` method to
  348. retrieve the root Element of the parse result document, and to unlock the
  349. parser:
  350. .. sourcecode:: pycon
  351. >>> root = parser.close()
  352. >>> print(root.tag)
  353. root
  354. >>> print(root[0].tag)
  355. a
  356. If you do not call ``close()``, the parser will stay locked and
  357. subsequent feeds will keep appending data, usually resulting in a non
  358. well-formed document and an unexpected parser error. So make sure you
  359. always close the parser after use, also in the exception case.
  360. Another way of achieving the same step-by-step parsing is by writing your own
  361. file-like object that returns a chunk of data on each ``read()`` call. Where
  362. the feed parser interface allows you to actively pass data chunks into the
  363. parser, a file-like object passively responds to ``read()`` requests of the
  364. parser itself. Depending on the data source, either way may be more natural.
  365. Note that the feed parser has its own error log called
  366. ``feed_error_log``. Errors in the feed parser do not show up in the
  367. normal ``error_log`` and vice versa.
  368. You can also combine the feed parser interface with the target parser:
  369. .. sourcecode:: pycon
  370. >>> parser = etree.XMLParser(target = EchoTarget())
  371. >>> parser.feed("<eleme")
  372. >>> parser.feed("nt>some text</elem")
  373. start element {}
  374. data u'some text'
  375. >>> parser.feed("ent>")
  376. end element
  377. >>> result = parser.close()
  378. close
  379. >>> print(result)
  380. closed!
  381. Again, this prevents the automatic creation of an XML tree and leaves
  382. all the event handling to the target object. The ``close()`` method
  383. of the parser forwards the return value of the target's ``close()``
  384. method.
  385. Incremental event parsing
  386. =========================
  387. In Python 3.4, the ``xml.etree.ElementTree`` package gained an extension
  388. to the feed parser interface that is implemented by the ``XMLPullParser``
  389. class. It additionally allows processing parse events after each
  390. incremental parsing step, by calling the ``.read_events()`` method and
  391. iterating over the result. This is most useful for non-blocking execution
  392. environments where data chunks arrive one after the other and should be
  393. processed as far as possible in each step.
  394. The same feature is available in lxml 3.3. The basic usage is as follows:
  395. .. sourcecode:: pycon
  396. >>> parser = etree.XMLPullParser(events=('start', 'end'))
  397. >>> def print_events(parser):
  398. ... for action, element in parser.read_events():
  399. ... print('%s: %s' % (action, element.tag))
  400. >>> parser.feed('<root>some text')
  401. >>> print_events(parser)
  402. start: root
  403. >>> print_events(parser) # well, no more events, as before ...
  404. >>> parser.feed('<child><a />')
  405. >>> print_events(parser)
  406. start: child
  407. start: a
  408. end: a
  409. >>> parser.feed('</child></roo')
  410. >>> print_events(parser)
  411. end: child
  412. >>> parser.feed('t>')
  413. >>> print_events(parser)
  414. end: root
  415. Just like the normal feed parser, the ``XMLPullParser`` builds a tree in
  416. memory (and you should always call the ``.close()`` method when done with
  417. parsing):
  418. .. sourcecode:: pycon
  419. >>> root = parser.close()
  420. >>> etree.tostring(root)
  421. b'<root>some text<child><a/></child></root>'
  422. However, since the parser provides incremental access to that tree,
  423. you can explicitly delete content that you no longer need once you
  424. have processed it. Read the section on `Modifying the tree`_ below
  425. to see what you can do here and what kind of modifications you should
  426. avoid.
  427. In lxml, it is enough to call the ``.read_events()`` method once as
  428. the iterator it returns can be reused when new events are available.
  429. Also, as known from other iterators in lxml, you can pass a ``tag``
  430. argument that selects which parse events are returned by the
  431. ``.read_events()`` iterator.
  432. Event types
  433. -----------
  434. The parse events are tuples ``(event-type, object)``. The event types
  435. supported by ElementTree and lxml.etree are the strings 'start', 'end',
  436. 'start-ns' and 'end-ns'. The 'start' and 'end' events represent opening
  437. and closing elements. They are accompanied by the respective Element
  438. instance. By default, only 'end' events are generated, whereas the
  439. example above requested the generation of both 'start' and 'end' events.
  440. The 'start-ns' and 'end-ns' events notify about namespace declarations.
  441. They do not come with Elements. Instead, the value of the 'start-ns'
  442. event is a tuple ``(prefix, namespaceURI)`` that designates the beginning
  443. of a prefix-namespace mapping. The corresponding ``end-ns`` event does
  444. not have a value (None). It is common practice to use a list as namespace
  445. stack and pop the last entry on the 'end-ns' event.
  446. .. sourcecode:: pycon
  447. >>> def print_events(events):
  448. ... for action, obj in events:
  449. ... if action in ('start', 'end'):
  450. ... print("%s: %s" % (action, obj.tag))
  451. ... elif action == 'start-ns':
  452. ... print("%s: %s" % (action, obj))
  453. ... else:
  454. ... print(action)
  455. >>> event_types = ("start", "end", "start-ns", "end-ns")
  456. >>> parser = etree.XMLPullParser(event_types)
  457. >>> events = parser.read_events()
  458. >>> parser.feed('<root><element>')
  459. >>> print_events(events)
  460. start: root
  461. start: element
  462. >>> parser.feed('text</element><element>text</element>')
  463. >>> print_events(events)
  464. end: element
  465. start: element
  466. end: element
  467. >>> parser.feed('<empty-element xmlns="http://testns/" />')
  468. >>> print_events(events)
  469. start-ns: ('', 'http://testns/')
  470. start: {http://testns/}empty-element
  471. end: {http://testns/}empty-element
  472. end-ns
  473. >>> parser.feed('</root>')
  474. >>> print_events(events)
  475. end: root
  476. Modifying the tree
  477. ------------------
  478. You can modify the element and its descendants when handling the
  479. 'end' event. To save memory, for example, you can remove subtrees
  480. that are no longer needed:
  481. .. sourcecode:: pycon
  482. >>> parser = etree.XMLPullParser()
  483. >>> events = parser.read_events()
  484. >>> parser.feed('<root><element key="value">text</element>')
  485. >>> parser.feed('<element><child /></element>')
  486. >>> for action, elem in events:
  487. ... print('%s: %d' % (elem.tag, len(elem))) # processing
  488. ... elem.clear() # delete children
  489. element: 0
  490. child: 0
  491. element: 1
  492. >>> parser.feed('<empty-element xmlns="http://testns/" /></root>')
  493. >>> for action, elem in events:
  494. ... print('%s: %d' % (elem.tag, len(elem))) # processing
  495. ... elem.clear() # delete children
  496. {http://testns/}empty-element: 0
  497. root: 3
  498. >>> root = parser.close()
  499. >>> etree.tostring(root)
  500. b'<root/>'
  501. **WARNING**: During the 'start' event, any content of the element,
  502. such as the descendants, following siblings or text, is not yet
  503. available and should not be accessed. Only attributes are guaranteed
  504. to be set. During the 'end' event, the element and its descendants
  505. can be freely modified, but its following siblings should not be
  506. accessed. During either of the two events, you **must not** modify or
  507. move the ancestors (parents) of the current element. You should also
  508. avoid moving or discarding the element itself. The golden rule is: do
  509. not touch anything that will have to be touched again by the parser
  510. later on.
  511. If you have elements with a long list of children in your XML file and want
  512. to save more memory during parsing, you can clean up the preceding siblings
  513. of the current element:
  514. .. sourcecode:: pycon
  515. >>> for event, element in parser.read_events():
  516. ... # ... do something with the element
  517. ... element.clear() # clean up children
  518. ... while element.getprevious() is not None:
  519. ... del element.getparent()[0] # clean up preceding siblings
  520. The ``while`` loop deletes multiple siblings in a row. This is only necessary
  521. if you skipped over some of them using the ``tag`` keyword argument.
  522. Otherwise, a simple ``if`` should do. The more selective your tag is,
  523. however, the more thought you will have to put into finding the right way to
  524. clean up the elements that were skipped. Therefore, it is sometimes easier to
  525. traverse all elements and do the tag selection by hand in the event handler
  526. code.
  527. Selective tag events
  528. --------------------
  529. As an extension over ElementTree, lxml.etree accepts a ``tag`` keyword
  530. argument just like ``element.iter(tag)``. This restricts events to a
  531. specific tag or namespace:
  532. .. sourcecode:: pycon
  533. >>> parser = etree.XMLPullParser(tag="element")
  534. >>> parser.feed('<root><element key="value">text</element>')
  535. >>> parser.feed('<element><child /></element>')
  536. >>> parser.feed('<empty-element xmlns="http://testns/" /></root>')
  537. >>> for action, elem in parser.read_events():
  538. ... print("%s: %s" % (action, elem.tag))
  539. end: element
  540. end: element
  541. >>> event_types = ("start", "end")
  542. >>> parser = etree.XMLPullParser(event_types, tag="{http://testns/}*")
  543. >>> parser.feed('<root><element key="value">text</element>')
  544. >>> parser.feed('<element><child /></element>')
  545. >>> parser.feed('<empty-element xmlns="http://testns/" /></root>')
  546. >>> for action, elem in parser.read_events():
  547. ... print("%s: %s" % (action, elem.tag))
  548. start: {http://testns/}empty-element
  549. end: {http://testns/}empty-element
  550. Comments and PIs
  551. ----------------
  552. As an extension over ElementTree, the ``XMLPullParser`` in lxml.etree
  553. also supports the event types 'comment' and 'pi' for the respective
  554. XML structures.
  555. .. sourcecode:: pycon
  556. >>> event_types = ("start", "end", "comment", "pi")
  557. >>> parser = etree.XMLPullParser(event_types)
  558. >>> parser.feed('<?some pi ?><!-- a comment --><root>')
  559. >>> parser.feed('<element key="value">text</element>')
  560. >>> parser.feed('<!-- another comment -->')
  561. >>> parser.feed('<element>text</element>tail')
  562. >>> parser.feed('<empty-element xmlns="http://testns/" />')
  563. >>> parser.feed('</root>')
  564. >>> for action, elem in parser.read_events():
  565. ... if action in ('start', 'end'):
  566. ... print("%s: %s" % (action, elem.tag))
  567. ... elif action == 'pi':
  568. ... print("%s: -%s=%s-" % (action, elem.target, elem.text))
  569. ... else: # 'comment'
  570. ... print("%s: -%s-" % (action, elem.text))
  571. pi: -some=pi -
  572. comment: - a comment -
  573. start: root
  574. start: element
  575. end: element
  576. comment: - another comment -
  577. start: element
  578. end: element
  579. start: {http://testns/}empty-element
  580. end: {http://testns/}empty-element
  581. end: root
  582. >>> root = parser.close()
  583. >>> print(root.tag)
  584. root
  585. Events with custom targets
  586. --------------------------
  587. You can combine the pull parser with a parser target. In that case,
  588. it is the target's responsibility to generate event values. Whatever
  589. it returns from its ``.start()`` and ``.end()`` methods will be returned
  590. by the pull parser as the second item of the parse events tuple.
  591. .. sourcecode:: pycon
  592. >>> class Target(object):
  593. ... def start(self, tag, attrib):
  594. ... print('-> start(%s)' % tag)
  595. ... return '>>START: %s<<' % tag
  596. ... def end(self, tag):
  597. ... print('-> end(%s)' % tag)
  598. ... return '>>END: %s<<' % tag
  599. ... def close(self):
  600. ... print('-> close()')
  601. ... return "CLOSED!"
  602. >>> event_types = ('start', 'end')
  603. >>> parser = etree.XMLPullParser(event_types, target=Target())
  604. >>> parser.feed('<root><child1 /><child2 /></root>')
  605. -> start(root)
  606. -> start(child1)
  607. -> end(child1)
  608. -> start(child2)
  609. -> end(child2)
  610. -> end(root)
  611. >>> for action, value in parser.read_events():
  612. ... print('%s: %s' % (action, value))
  613. start: >>START: root<<
  614. start: >>START: child1<<
  615. end: >>END: child1<<
  616. start: >>START: child2<<
  617. end: >>END: child2<<
  618. end: >>END: root<<
  619. >>> print(parser.close())
  620. -> close()
  621. CLOSED!
  622. As you can see, the event values do not even have to be Element objects.
  623. The target is generally free to decide how it wants to create an XML tree
  624. or whatever else it wants to make of the parser callbacks. In many cases,
  625. however, you will want to make your custom target inherit from the
  626. ``TreeBuilder`` class in order to have it build a tree that you can process
  627. normally. The ``start()`` and ``.end()`` methods of ``TreeBuilder`` return
  628. the Element object that was created, so you can override them and modify
  629. the input or output according to your needs. Here is an example that
  630. filters attributes before they are being added to the tree:
  631. .. sourcecode:: pycon
  632. >>> class AttributeFilter(etree.TreeBuilder):
  633. ... def start(self, tag, attrib):
  634. ... attrib = dict(attrib)
  635. ... if 'evil' in attrib:
  636. ... del attrib['evil']
  637. ... return super(AttributeFilter, self).start(tag, attrib)
  638. >>> parser = etree.XMLPullParser(target=AttributeFilter())
  639. >>> parser.feed('<root><child1 test="123" /><child2 evil="YES" /></root>')
  640. >>> for action, element in parser.read_events():
  641. ... print('%s: %s(%r)' % (action, element.tag, element.attrib))
  642. end: child1({'test': '123'})
  643. end: child2({})
  644. end: root({})
  645. >>> root = parser.close()
  646. iterparse and iterwalk
  647. ======================
  648. As known from ElementTree, the ``iterparse()`` utility function
  649. returns an iterator that generates parser events for an XML file (or
  650. file-like object), while building the tree. You can think of it as
  651. a blocking wrapper around the ``XMLPullParser`` that automatically and
  652. incrementally reads data from the input file for you and provides a
  653. single iterator for them:
  654. .. sourcecode:: pycon
  655. >>> xml = '''
  656. ... <root>
  657. ... <element key='value'>text</element>
  658. ... <element>text</element>tail
  659. ... <empty-element xmlns="http://testns/" />
  660. ... </root>
  661. ... '''
  662. >>> context = etree.iterparse(StringIO(xml))
  663. >>> for action, elem in context:
  664. ... print("%s: %s" % (action, elem.tag))
  665. end: element
  666. end: element
  667. end: {http://testns/}empty-element
  668. end: root
  669. After parsing, the resulting tree is available through the ``root`` property
  670. of the iterator:
  671. .. sourcecode:: pycon
  672. >>> context.root.tag
  673. 'root'
  674. The other event types can be activated with the ``events`` keyword argument:
  675. .. sourcecode:: pycon
  676. >>> events = ("start", "end")
  677. >>> context = etree.iterparse(StringIO(xml), events=events)
  678. >>> for action, elem in context:
  679. ... print("%s: %s" % (action, elem.tag))
  680. start: root
  681. start: element
  682. end: element
  683. start: element
  684. end: element
  685. start: {http://testns/}empty-element
  686. end: {http://testns/}empty-element
  687. end: root
  688. ``iterparse()`` also supports the ``tag`` argument for selective event
  689. iteration and several other parameters that control the parser setup.
  690. You can also use it to parse HTML input by passing ``html=True``.
  691. iterwalk
  692. --------
  693. A second extension over ElementTree is the ``iterwalk()`` function.
  694. It behaves exactly like ``iterparse()``, but works on Elements and
  695. ElementTrees. Here is an example for a tree parsed by ``iterparse()``:
  696. .. sourcecode:: pycon
  697. >>> f = StringIO(xml)
  698. >>> context = etree.iterparse(
  699. ... f, events=("start", "end"), tag="element")
  700. >>> for action, elem in context:
  701. ... print("%s: %s" % (action, elem.tag))
  702. start: element
  703. end: element
  704. start: element
  705. end: element
  706. >>> root = context.root
  707. And now we can take the resulting in-memory tree and iterate over it
  708. using ``iterwalk()`` to get the exact same events without parsing the
  709. input again:
  710. >>> context = etree.iterwalk(
  711. ... root, events=("start", "end"), tag="element")
  712. >>> for action, elem in context:
  713. ... print("%s: %s" % (action, elem.tag))
  714. start: element
  715. end: element
  716. start: element
  717. end: element
  718. Python unicode strings
  719. ======================
  720. lxml.etree has broader support for Python unicode strings than the ElementTree
  721. library. First of all, where ElementTree would raise an exception, the
  722. parsers in lxml.etree can handle unicode strings straight away. This is most
  723. helpful for XML snippets embedded in source code using the ``XML()``
  724. function:
  725. .. sourcecode:: pycon
  726. >>> root = etree.XML( u'<test> \uf8d1 + \uf8d2 </test>' )
  727. This requires, however, that unicode strings do not specify a conflicting
  728. encoding themselves and thus lie about their real encoding:
  729. .. sourcecode:: pycon
  730. >>> etree.XML( u'<?xml version="1.0" encoding="ASCII"?>\n' +
  731. ... u'<test> \uf8d1 + \uf8d2 </test>' )
  732. Traceback (most recent call last):
  733. ...
  734. ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.
  735. Similarly, you will get errors when you try the same with HTML data in a
  736. unicode string that specifies a charset in a meta tag of the header. You
  737. should generally avoid converting XML/HTML data to unicode before passing it
  738. into the parsers. It is both slower and error prone.
  739. Serialising to Unicode strings
  740. ------------------------------
  741. To serialize the result, you would normally use the ``tostring()``
  742. module function, which serializes to plain ASCII by default or a
  743. number of other byte encodings if asked for:
  744. .. sourcecode:: pycon
  745. >>> etree.tostring(root)
  746. b'<test> &#63697; + &#63698; </test>'
  747. >>> etree.tostring(root, encoding='UTF-8', xml_declaration=False)
  748. b'<test> \xef\xa3\x91 + \xef\xa3\x92 </test>'
  749. As an extension, lxml.etree recognises the name 'unicode' as an argument
  750. to the encoding parameter to build a Python unicode representation of a tree:
  751. .. sourcecode:: pycon
  752. >>> etree.tostring(root, encoding='unicode')
  753. u'<test> \uf8d1 + \uf8d2 </test>'
  754. >>> el = etree.Element("test")
  755. >>> etree.tostring(el, encoding='unicode')
  756. u'<test/>'
  757. >>> subel = etree.SubElement(el, "subtest")
  758. >>> etree.tostring(el, encoding='unicode')
  759. u'<test><subtest/></test>'
  760. >>> tree = etree.ElementTree(el)
  761. >>> etree.tostring(tree, encoding='unicode')
  762. u'<test><subtest/></test>'
  763. The result of ``tostring(encoding='unicode')`` can be treated like any
  764. other Python unicode string and then passed back into the parsers.
  765. However, if you want to save the result to a file or pass it over the
  766. network, you should use ``write()`` or ``tostring()`` with a byte
  767. encoding (typically UTF-8) to serialize the XML. The main reason is
  768. that unicode strings returned by ``tostring(encoding='unicode')`` are
  769. not byte streams and they never have an XML declaration to specify
  770. their encoding. These strings are most likely not parsable by other
  771. XML libraries.
  772. For normal byte encodings, the ``tostring()`` function automatically
  773. adds a declaration as needed that reflects the encoding of the
  774. returned string. This makes it possible for other parsers to
  775. correctly parse the XML byte stream. Note that using ``tostring()``
  776. with UTF-8 is also considerably faster in most cases.