parsing.txt 33 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751752753754755756757758759760761762763764765766767768769770771772773774775776777778779780781782783784785786787788789790791792793794795796797798799800801802803804805806807808809810811812813814815816817818819820821822823824825826827828829830831832833834835836837838839840841842843844845846847848849850851852853854855856857858859860861862863864865866867868869870871872873874875876877878879880881882883884885886887888889890891892893894895896897898899900901902903904905906907908909910911912913914915916917918919920921922923924925926927928929930931932933934935936937938939940941942943944945946947948949950951952953954955956957958959960961962963964965966967968969970971972973974975976977978979980981982983984985986987988989990991992993994995996997998999100010011002100310041005100610071008100910101011101210131014101510161017101810191020102110221023102410251026102710281029103010311032103310341035103610371038103910401041104210431044104510461047104810491050105110521053105410551056105710581059106010611062
  1. ==============================
  2. Parsing XML and HTML with lxml
  3. ==============================
  4. lxml provides a very simple and powerful API for parsing XML and HTML. It
  5. supports one-step parsing as well as step-by-step parsing using an
  6. event-driven API (currently only for XML).
  7. .. contents::
  8. ..
  9. 1 Parsers
  10. 1.1 Parser options
  11. 1.2 Error log
  12. 1.3 Parsing HTML
  13. 1.4 Doctype information
  14. 2 The target parser interface
  15. 3 The feed parser interface
  16. 4 Incremental event parsing
  17. 4.1 Event types
  18. 4.1 Modifying the tree
  19. 4.3 Selective tag events
  20. 4.4 Comments and PIs
  21. 4.5 Events with custom targets
  22. 5 iterparse and iterwalk
  23. 5.1 iterwalk
  24. 6 Python unicode strings
  25. 6.1 Serialising to Unicode strings
  26. The usual setup procedure:
  27. .. sourcecode:: pycon
  28. >>> from lxml import etree
  29. The following examples also use StringIO or BytesIO to show how to parse
  30. from files and file-like objects. Both are available in the ``io`` module:
  31. .. sourcecode:: python
  32. from io import StringIO, BytesIO
  33. ..
  34. >>> from lxml import usedoctest
  35. >>> try: from StringIO import StringIO
  36. ... except ImportError:
  37. ... from io import BytesIO
  38. ... def StringIO(s):
  39. ... if isinstance(s, str): s = s.encode("UTF-8")
  40. ... return BytesIO(s)
  41. >>> try: unicode = unicode
  42. ... except NameError: unicode = str
  43. >>> import sys
  44. >>> from lxml import etree as _etree
  45. >>> if sys.version_info[0] >= 3:
  46. ... class etree_mock(object):
  47. ... def __getattr__(self, name): return getattr(_etree, name)
  48. ... def tostring(self, *args, **kwargs):
  49. ... s = _etree.tostring(*args, **kwargs)
  50. ... if isinstance(s, bytes) and bytes([10]) in s: s = s.decode("utf-8") # CR
  51. ... if s[-1] == '\n': s = s[:-1]
  52. ... return s
  53. ... else:
  54. ... class etree_mock(object):
  55. ... def __getattr__(self, name): return getattr(_etree, name)
  56. ... def tostring(self, *args, **kwargs):
  57. ... s = _etree.tostring(*args, **kwargs)
  58. ... if s[-1] == '\n': s = s[:-1]
  59. ... return s
  60. >>> etree = etree_mock()
  61. Parsers
  62. =======
  63. Parsers are represented by parser objects. There is support for parsing both
  64. XML and (broken) HTML. Note that XHTML is best parsed as XML, parsing it with
  65. the HTML parser can lead to unexpected results. Here is a simple example for
  66. parsing XML from an in-memory string:
  67. .. sourcecode:: pycon
  68. >>> xml = '<a xmlns="test"><b xmlns="test"/></a>'
  69. >>> root = etree.fromstring(xml)
  70. >>> etree.tostring(root)
  71. b'<a xmlns="test"><b xmlns="test"/></a>'
  72. To read from a file or file-like object, you can use the ``parse()`` function,
  73. which returns an ``ElementTree`` object:
  74. .. sourcecode:: pycon
  75. >>> tree = etree.parse(StringIO(xml))
  76. >>> etree.tostring(tree.getroot())
  77. b'<a xmlns="test"><b xmlns="test"/></a>'
  78. Note how the ``parse()`` function reads from a file-like object here. If
  79. parsing is done from a real file, it is more common (and also somewhat more
  80. efficient) to pass a filename:
  81. .. sourcecode:: pycon
  82. >>> tree = etree.parse("doc/test.xml")
  83. lxml can parse from a local file, an HTTP URL or an FTP URL. It also
  84. auto-detects and reads gzip-compressed XML files (.gz).
  85. If you want to parse from memory and still provide a base URL for the document
  86. (e.g. to support relative paths in an XInclude), you can pass the ``base_url``
  87. keyword argument:
  88. .. sourcecode:: pycon
  89. >>> root = etree.fromstring(xml, base_url="http://where.it/is/from.xml")
  90. Parser options
  91. --------------
  92. The parsers accept a number of setup options as keyword arguments. The above
  93. example is easily extended to clean up namespaces during parsing:
  94. .. sourcecode:: pycon
  95. >>> parser = etree.XMLParser(ns_clean=True)
  96. >>> tree = etree.parse(StringIO(xml), parser)
  97. >>> etree.tostring(tree.getroot())
  98. b'<a xmlns="test"><b/></a>'
  99. The keyword arguments in the constructor are mainly based on the libxml2
  100. parser configuration. A DTD will also be loaded if validation or attribute
  101. default values are requested.
  102. Available boolean keyword arguments:
  103. * attribute_defaults - read the DTD (if referenced by the document) and add
  104. the default attributes from it
  105. * dtd_validation - validate while parsing (if a DTD was referenced)
  106. * load_dtd - load and parse the DTD while parsing (no validation is performed)
  107. * no_network - prevent network access when looking up external
  108. documents (on by default)
  109. * ns_clean - try to clean up redundant namespace declarations
  110. * recover - try hard to parse through broken XML
  111. * remove_blank_text - discard blank text nodes between tags, also known as
  112. ignorable whitespace. This is best used together with a DTD or schema
  113. (which tells data and noise apart), otherwise a heuristic will be applied.
  114. * remove_comments - discard comments
  115. * remove_pis - discard processing instructions
  116. * strip_cdata - replace CDATA sections by normal text content (on by
  117. default)
  118. * resolve_entities - replace entities by their text value (on by
  119. default)
  120. * huge_tree - disable security restrictions and support very deep trees
  121. and very long text content (only affects libxml2 2.7+)
  122. * compact - use compact storage for short text content (on by default)
  123. * collect_ids - collect XML IDs in a hash table while parsing (on by default).
  124. Disabling this can substantially speed up parsing of documents with many
  125. different IDs if the hash lookup is not used afterwards.
  126. Other keyword arguments:
  127. * encoding - override the document encoding
  128. * target - a parser target object that will receive the parse events
  129. (see `The target parser interface`_)
  130. * schema - an XMLSchema to validate against (see `validation <validation.html#xmlschema>`_)
  131. Error log
  132. ---------
  133. Parsers have an ``error_log`` property that lists the errors and
  134. warnings of the last parser run:
  135. .. sourcecode:: pycon
  136. >>> parser = etree.XMLParser()
  137. >>> print(len(parser.error_log))
  138. 0
  139. >>> tree = etree.XML("<root>\n</b>", parser) # doctest: +ELLIPSIS
  140. Traceback (most recent call last):
  141. ...
  142. lxml.etree.XMLSyntaxError: Opening and ending tag mismatch: root line 1 and b, line 2, column 5...
  143. >>> print(len(parser.error_log))
  144. 1
  145. >>> error = parser.error_log[0]
  146. >>> print(error.message)
  147. Opening and ending tag mismatch: root line 1 and b
  148. >>> print(error.line)
  149. 2
  150. >>> print(error.column)
  151. 5
  152. Each entry in the log has the following properties:
  153. * ``message``: the message text
  154. * ``domain``: the domain ID (see the lxml.etree.ErrorDomains class)
  155. * ``type``: the message type ID (see the lxml.etree.ErrorTypes class)
  156. * ``level``: the log level ID (see the lxml.etree.ErrorLevels class)
  157. * ``line``: the line at which the message originated (if applicable)
  158. * ``column``: the character column at which the message originated (if applicable)
  159. * ``filename``: the name of the file in which the message originated (if applicable)
  160. For convenience, there are also three properties that provide readable
  161. names for the ID values:
  162. * ``domain_name``
  163. * ``type_name``
  164. * ``level_name``
  165. To filter for a specific kind of message, use the different
  166. ``filter_*()`` methods on the error log (see the
  167. lxml.etree._ListErrorLog class).
  168. Parsing HTML
  169. ------------
  170. HTML parsing is similarly simple. The parsers have a ``recover``
  171. keyword argument that the HTMLParser sets by default. It lets libxml2
  172. try its best to return a valid HTML tree with all content it can
  173. manage to parse. It will not raise an exception on parser errors.
  174. You should use libxml2 version 2.6.21 or newer to take advantage of
  175. this feature.
  176. .. sourcecode:: pycon
  177. >>> broken_html = "<html><head><title>test<body><h1>page title</h3>"
  178. >>> parser = etree.HTMLParser()
  179. >>> tree = etree.parse(StringIO(broken_html), parser)
  180. >>> result = etree.tostring(tree.getroot(),
  181. ... pretty_print=True, method="html")
  182. >>> print(result)
  183. <html>
  184. <head>
  185. <title>test</title>
  186. </head>
  187. <body>
  188. <h1>page title</h1>
  189. </body>
  190. </html>
  191. Lxml has an HTML function, similar to the XML shortcut known from
  192. ElementTree:
  193. .. sourcecode:: pycon
  194. >>> html = etree.HTML(broken_html)
  195. >>> result = etree.tostring(html, pretty_print=True, method="html")
  196. >>> print(result)
  197. <html>
  198. <head>
  199. <title>test</title>
  200. </head>
  201. <body>
  202. <h1>page title</h1>
  203. </body>
  204. </html>
  205. The support for parsing broken HTML depends entirely on libxml2's recovery
  206. algorithm. It is *not* the fault of lxml if you find documents that are so
  207. heavily broken that the parser cannot handle them. There is also no guarantee
  208. that the resulting tree will contain all data from the original document. The
  209. parser may have to drop seriously broken parts when struggling to keep
  210. parsing. Especially misplaced meta tags can suffer from this, which may lead
  211. to encoding problems.
  212. Note that the result is a valid HTML tree, but it may not be a
  213. well-formed XML tree. For example, XML forbids double hyphens in
  214. comments, which the HTML parser will happily accept in recovery mode.
  215. Therefore, if your goal is to serialise an HTML document as an
  216. XML/XHTML document after parsing, you may have to apply some manual
  217. preprocessing first.
  218. Also note that the HTML parser is meant to parse HTML documents. For
  219. XHTML documents, use the XML parser, which is namespace aware.
  220. Doctype information
  221. -------------------
  222. The use of the libxml2 parsers makes some additional information available at
  223. the API level. Currently, ElementTree objects can access the DOCTYPE
  224. information provided by a parsed document, as well as the XML version and the
  225. original encoding. Since lxml 3.5, the doctype references are mutable.
  226. .. sourcecode:: pycon
  227. >>> pub_id = "-//W3C//DTD XHTML 1.0 Transitional//EN"
  228. >>> sys_url = "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"
  229. >>> doctype_string = '<!DOCTYPE html PUBLIC "%s" "%s">' % (pub_id, sys_url)
  230. >>> xml_header = '<?xml version="1.0" encoding="ascii"?>'
  231. >>> xhtml = xml_header + doctype_string + '<html><body></body></html>'
  232. >>> tree = etree.parse(StringIO(xhtml))
  233. >>> docinfo = tree.docinfo
  234. >>> print(docinfo.public_id)
  235. -//W3C//DTD XHTML 1.0 Transitional//EN
  236. >>> print(docinfo.system_url)
  237. http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd
  238. >>> docinfo.doctype == doctype_string
  239. True
  240. >>> print(docinfo.xml_version)
  241. 1.0
  242. >>> print(docinfo.encoding)
  243. ascii
  244. >>> docinfo.system_url = None
  245. >>> docinfo.public_id = None
  246. >>> print(etree.tostring(tree))
  247. <!DOCTYPE html>
  248. <html><body/></html>
  249. The target parser interface
  250. ===========================
  251. .. _`As in ElementTree`: http://effbot.org/elementtree/elementtree-xmlparser.htm
  252. `As in ElementTree`_, and similar to a SAX event handler, you can pass
  253. a target object to the parser:
  254. .. sourcecode:: pycon
  255. >>> class EchoTarget(object):
  256. ... def start(self, tag, attrib):
  257. ... print("start %s %r" % (tag, dict(attrib)))
  258. ... def end(self, tag):
  259. ... print("end %s" % tag)
  260. ... def data(self, data):
  261. ... print("data %r" % data)
  262. ... def comment(self, text):
  263. ... print("comment %s" % text)
  264. ... def close(self):
  265. ... print("close")
  266. ... return "closed!"
  267. >>> parser = etree.XMLParser(target = EchoTarget())
  268. >>> result = etree.XML("<element>some<!--comment-->text</element>",
  269. ... parser)
  270. start element {}
  271. data u'some'
  272. comment comment
  273. data u'text'
  274. end element
  275. close
  276. >>> print(result)
  277. closed!
  278. It is important for the ``.close()`` method to reset the parser target
  279. to a usable state, so that you can reuse the parser as often as you
  280. like:
  281. .. sourcecode:: pycon
  282. >>> result = etree.XML("<element>some<!--comment-->text</element>",
  283. ... parser)
  284. start element {}
  285. data u'some'
  286. comment comment
  287. data u'text'
  288. end element
  289. close
  290. >>> print(result)
  291. closed!
  292. Starting with lxml 2.3, the ``.close()`` method will also be called in
  293. the error case. This diverges from the behaviour of ElementTree, but
  294. allows target objects to clean up their state in all situations, so
  295. that the parser can reuse them afterwards.
  296. .. sourcecode:: pycon
  297. >>> class CollectorTarget(object):
  298. ... def __init__(self):
  299. ... self.events = []
  300. ... def start(self, tag, attrib):
  301. ... self.events.append("start %s %r" % (tag, dict(attrib)))
  302. ... def end(self, tag):
  303. ... self.events.append("end %s" % tag)
  304. ... def data(self, data):
  305. ... self.events.append("data %r" % data)
  306. ... def comment(self, text):
  307. ... self.events.append("comment %s" % text)
  308. ... def close(self):
  309. ... self.events.append("close")
  310. ... return "closed!"
  311. >>> parser = etree.XMLParser(target = CollectorTarget())
  312. >>> result = etree.XML("<element>some</error>",
  313. ... parser) # doctest: +ELLIPSIS
  314. Traceback (most recent call last):
  315. ...
  316. lxml.etree.XMLSyntaxError: Opening and ending tag mismatch...
  317. >>> for event in parser.target.events:
  318. ... print(event)
  319. start element {}
  320. data u'some'
  321. close
  322. Note that the parser does *not* build a tree when using a parser
  323. target. The result of the parser run is whatever the target object
  324. returns from its ``.close()`` method. If you want to return an XML
  325. tree here, you have to create it programmatically in the target
  326. object. An example for a parser target that builds a tree is the
  327. ``TreeBuilder``:
  328. .. sourcecode:: pycon
  329. >>> parser = etree.XMLParser(target = etree.TreeBuilder())
  330. >>> result = etree.XML("<element>some<!--comment-->text</element>",
  331. ... parser)
  332. >>> print(result.tag)
  333. element
  334. >>> print(result[0].text)
  335. comment
  336. The feed parser interface
  337. =========================
  338. Since lxml 2.0, the parsers have a feed parser interface that is
  339. compatible to the `ElementTree parsers`_. You can use it to feed data
  340. into the parser in a controlled step-by-step way.
  341. In lxml.etree, you can use both interfaces to a parser at the same
  342. time: the ``parse()`` or ``XML()`` functions, and the feed parser
  343. interface. Both are independent and will not conflict (except if used
  344. in conjunction with a parser target object as described above).
  345. .. _`ElementTree parsers`: http://effbot.org/elementtree/elementtree-xmlparser.htm
  346. To start parsing with a feed parser, just call its ``feed()`` method
  347. to feed it some data.
  348. .. sourcecode:: pycon
  349. >>> parser = etree.XMLParser()
  350. >>> for data in ('<?xml versio', 'n="1.0"?', '><roo', 't><a', '/></root>'):
  351. ... parser.feed(data)
  352. When you are done parsing, you **must** call the ``close()`` method to
  353. retrieve the root Element of the parse result document, and to unlock the
  354. parser:
  355. .. sourcecode:: pycon
  356. >>> root = parser.close()
  357. >>> print(root.tag)
  358. root
  359. >>> print(root[0].tag)
  360. a
  361. If you do not call ``close()``, the parser will stay locked and
  362. subsequent feeds will keep appending data, usually resulting in a non
  363. well-formed document and an unexpected parser error. So make sure you
  364. always close the parser after use, also in the exception case.
  365. Another way of achieving the same step-by-step parsing is by writing your own
  366. file-like object that returns a chunk of data on each ``read()`` call. Where
  367. the feed parser interface allows you to actively pass data chunks into the
  368. parser, a file-like object passively responds to ``read()`` requests of the
  369. parser itself. Depending on the data source, either way may be more natural.
  370. Note that the feed parser has its own error log called
  371. ``feed_error_log``. Errors in the feed parser do not show up in the
  372. normal ``error_log`` and vice versa.
  373. You can also combine the feed parser interface with the target parser:
  374. .. sourcecode:: pycon
  375. >>> parser = etree.XMLParser(target = EchoTarget())
  376. >>> parser.feed("<eleme")
  377. >>> parser.feed("nt>some text</elem")
  378. start element {}
  379. data u'some text'
  380. >>> parser.feed("ent>")
  381. end element
  382. >>> result = parser.close()
  383. close
  384. >>> print(result)
  385. closed!
  386. Again, this prevents the automatic creation of an XML tree and leaves
  387. all the event handling to the target object. The ``close()`` method
  388. of the parser forwards the return value of the target's ``close()``
  389. method.
  390. Incremental event parsing
  391. =========================
  392. In Python 3.4, the ``xml.etree.ElementTree`` package gained an extension
  393. to the feed parser interface that is implemented by the ``XMLPullParser``
  394. class. It additionally allows processing parse events after each
  395. incremental parsing step, by calling the ``.read_events()`` method and
  396. iterating over the result. This is most useful for non-blocking execution
  397. environments where data chunks arrive one after the other and should be
  398. processed as far as possible in each step.
  399. The same feature is available in lxml 3.3. The basic usage is as follows:
  400. .. sourcecode:: pycon
  401. >>> parser = etree.XMLPullParser(events=('start', 'end'))
  402. >>> def print_events(parser):
  403. ... for action, element in parser.read_events():
  404. ... print('%s: %s' % (action, element.tag))
  405. >>> parser.feed('<root>some text')
  406. >>> print_events(parser)
  407. start: root
  408. >>> print_events(parser) # well, no more events, as before ...
  409. >>> parser.feed('<child><a />')
  410. >>> print_events(parser)
  411. start: child
  412. start: a
  413. end: a
  414. >>> parser.feed('</child></roo')
  415. >>> print_events(parser)
  416. end: child
  417. >>> parser.feed('t>')
  418. >>> print_events(parser)
  419. end: root
  420. Just like the normal feed parser, the ``XMLPullParser`` builds a tree in
  421. memory (and you should always call the ``.close()`` method when done with
  422. parsing):
  423. .. sourcecode:: pycon
  424. >>> root = parser.close()
  425. >>> etree.tostring(root)
  426. b'<root>some text<child><a/></child></root>'
  427. However, since the parser provides incremental access to that tree,
  428. you can explicitly delete content that you no longer need once you
  429. have processed it. Read the section on `Modifying the tree`_ below
  430. to see what you can do here and what kind of modifications you should
  431. avoid.
  432. In lxml, it is enough to call the ``.read_events()`` method once as
  433. the iterator it returns can be reused when new events are available.
  434. Also, as known from other iterators in lxml, you can pass a ``tag``
  435. argument that selects which parse events are returned by the
  436. ``.read_events()`` iterator.
  437. Event types
  438. -----------
  439. The parse events are tuples ``(event-type, object)``. The event types
  440. supported by ElementTree and lxml.etree are the strings 'start', 'end',
  441. 'start-ns' and 'end-ns'. The 'start' and 'end' events represent opening
  442. and closing elements. They are accompanied by the respective Element
  443. instance. By default, only 'end' events are generated, whereas the
  444. example above requested the generation of both 'start' and 'end' events.
  445. The 'start-ns' and 'end-ns' events notify about namespace declarations.
  446. They do not come with Elements. Instead, the value of the 'start-ns'
  447. event is a tuple ``(prefix, namespaceURI)`` that designates the beginning
  448. of a prefix-namespace mapping. The corresponding ``end-ns`` event does
  449. not have a value (None). It is common practice to use a list as namespace
  450. stack and pop the last entry on the 'end-ns' event.
  451. .. sourcecode:: pycon
  452. >>> def print_events(events):
  453. ... for action, obj in events:
  454. ... if action in ('start', 'end'):
  455. ... print("%s: %s" % (action, obj.tag))
  456. ... elif action == 'start-ns':
  457. ... print("%s: %s" % (action, obj))
  458. ... else:
  459. ... print(action)
  460. >>> event_types = ("start", "end", "start-ns", "end-ns")
  461. >>> parser = etree.XMLPullParser(event_types)
  462. >>> events = parser.read_events()
  463. >>> parser.feed('<root><element>')
  464. >>> print_events(events)
  465. start: root
  466. start: element
  467. >>> parser.feed('text</element><element>text</element>')
  468. >>> print_events(events)
  469. end: element
  470. start: element
  471. end: element
  472. >>> parser.feed('<empty-element xmlns="http://testns/" />')
  473. >>> print_events(events)
  474. start-ns: ('', 'http://testns/')
  475. start: {http://testns/}empty-element
  476. end: {http://testns/}empty-element
  477. end-ns
  478. >>> parser.feed('</root>')
  479. >>> print_events(events)
  480. end: root
  481. Modifying the tree
  482. ------------------
  483. You can modify the element and its descendants when handling the
  484. 'end' event. To save memory, for example, you can remove subtrees
  485. that are no longer needed:
  486. .. sourcecode:: pycon
  487. >>> parser = etree.XMLPullParser()
  488. >>> events = parser.read_events()
  489. >>> parser.feed('<root><element key="value">text</element>')
  490. >>> parser.feed('<element><child /></element>')
  491. >>> for action, elem in events:
  492. ... print('%s: %d' % (elem.tag, len(elem))) # processing
  493. ... elem.clear(keep_tail=True) # delete children
  494. element: 0
  495. child: 0
  496. element: 1
  497. >>> parser.feed('<empty-element xmlns="http://testns/" /></root>')
  498. >>> for action, elem in events:
  499. ... print('%s: %d' % (elem.tag, len(elem))) # processing
  500. ... elem.clear(keep_tail=True) # delete children
  501. {http://testns/}empty-element: 0
  502. root: 3
  503. >>> root = parser.close()
  504. >>> etree.tostring(root)
  505. b'<root/>'
  506. **WARNING**: During the 'start' event, any content of the element,
  507. such as the descendants, following siblings or text, is not yet
  508. available and should not be accessed. Only attributes are guaranteed
  509. to be set. During the 'end' event, the element and its descendants
  510. can be freely modified, but its following siblings should not be
  511. accessed. During either of the two events, you **must not** modify or
  512. move the ancestors (parents) of the current element. You should also
  513. avoid moving or discarding the element itself. The golden rule is: do
  514. not touch anything that will have to be touched again by the parser
  515. later on.
  516. If you have elements with a long list of children in your XML file and want
  517. to save more memory during parsing, you can clean up the preceding siblings
  518. of the current element:
  519. .. sourcecode:: pycon
  520. >>> for event, element in parser.read_events():
  521. ... # ... do something with the element
  522. ... element.clear(keep_tail=True) # clean up children
  523. ... while element.getprevious() is not None:
  524. ... del element.getparent()[0] # clean up preceding siblings
  525. The ``while`` loop deletes multiple siblings in a row. This is only necessary
  526. if you skipped over some of them using the ``tag`` keyword argument.
  527. Otherwise, a simple ``if`` should do. The more selective your tag is,
  528. however, the more thought you will have to put into finding the right way to
  529. clean up the elements that were skipped. Therefore, it is sometimes easier to
  530. traverse all elements and do the tag selection by hand in the event handler
  531. code.
  532. Selective tag events
  533. --------------------
  534. As an extension over ElementTree, lxml.etree accepts a ``tag`` keyword
  535. argument just like ``element.iter(tag)``. This restricts events to a
  536. specific tag or namespace:
  537. .. sourcecode:: pycon
  538. >>> parser = etree.XMLPullParser(tag="element")
  539. >>> parser.feed('<root><element key="value">text</element>')
  540. >>> parser.feed('<element><child /></element>')
  541. >>> parser.feed('<empty-element xmlns="http://testns/" /></root>')
  542. >>> for action, elem in parser.read_events():
  543. ... print("%s: %s" % (action, elem.tag))
  544. end: element
  545. end: element
  546. >>> event_types = ("start", "end")
  547. >>> parser = etree.XMLPullParser(event_types, tag="{http://testns/}*")
  548. >>> parser.feed('<root><element key="value">text</element>')
  549. >>> parser.feed('<element><child /></element>')
  550. >>> parser.feed('<empty-element xmlns="http://testns/" /></root>')
  551. >>> for action, elem in parser.read_events():
  552. ... print("%s: %s" % (action, elem.tag))
  553. start: {http://testns/}empty-element
  554. end: {http://testns/}empty-element
  555. Comments and PIs
  556. ----------------
  557. As an extension over ElementTree, the ``XMLPullParser`` in lxml.etree
  558. also supports the event types 'comment' and 'pi' for the respective
  559. XML structures.
  560. .. sourcecode:: pycon
  561. >>> event_types = ("start", "end", "comment", "pi")
  562. >>> parser = etree.XMLPullParser(event_types)
  563. >>> parser.feed('<?some pi ?><!-- a comment --><root>')
  564. >>> parser.feed('<element key="value">text</element>')
  565. >>> parser.feed('<!-- another comment -->')
  566. >>> parser.feed('<element>text</element>tail')
  567. >>> parser.feed('<empty-element xmlns="http://testns/" />')
  568. >>> parser.feed('</root>')
  569. >>> for action, elem in parser.read_events():
  570. ... if action in ('start', 'end'):
  571. ... print("%s: %s" % (action, elem.tag))
  572. ... elif action == 'pi':
  573. ... print("%s: -%s=%s-" % (action, elem.target, elem.text))
  574. ... else: # 'comment'
  575. ... print("%s: -%s-" % (action, elem.text))
  576. pi: -some=pi -
  577. comment: - a comment -
  578. start: root
  579. start: element
  580. end: element
  581. comment: - another comment -
  582. start: element
  583. end: element
  584. start: {http://testns/}empty-element
  585. end: {http://testns/}empty-element
  586. end: root
  587. >>> root = parser.close()
  588. >>> print(root.tag)
  589. root
  590. Events with custom targets
  591. --------------------------
  592. You can combine the pull parser with a parser target. In that case,
  593. it is the target's responsibility to generate event values. Whatever
  594. it returns from its ``.start()`` and ``.end()`` methods will be returned
  595. by the pull parser as the second item of the parse events tuple.
  596. .. sourcecode:: pycon
  597. >>> class Target(object):
  598. ... def start(self, tag, attrib):
  599. ... print('-> start(%s)' % tag)
  600. ... return '>>START: %s<<' % tag
  601. ... def end(self, tag):
  602. ... print('-> end(%s)' % tag)
  603. ... return '>>END: %s<<' % tag
  604. ... def close(self):
  605. ... print('-> close()')
  606. ... return "CLOSED!"
  607. >>> event_types = ('start', 'end')
  608. >>> parser = etree.XMLPullParser(event_types, target=Target())
  609. >>> parser.feed('<root><child1 /><child2 /></root>')
  610. -> start(root)
  611. -> start(child1)
  612. -> end(child1)
  613. -> start(child2)
  614. -> end(child2)
  615. -> end(root)
  616. >>> for action, value in parser.read_events():
  617. ... print('%s: %s' % (action, value))
  618. start: >>START: root<<
  619. start: >>START: child1<<
  620. end: >>END: child1<<
  621. start: >>START: child2<<
  622. end: >>END: child2<<
  623. end: >>END: root<<
  624. >>> print(parser.close())
  625. -> close()
  626. CLOSED!
  627. As you can see, the event values do not even have to be Element objects.
  628. The target is generally free to decide how it wants to create an XML tree
  629. or whatever else it wants to make of the parser callbacks. In many cases,
  630. however, you will want to make your custom target inherit from the
  631. ``TreeBuilder`` class in order to have it build a tree that you can process
  632. normally. The ``start()`` and ``.end()`` methods of ``TreeBuilder`` return
  633. the Element object that was created, so you can override them and modify
  634. the input or output according to your needs. Here is an example that
  635. filters attributes before they are being added to the tree:
  636. .. sourcecode:: pycon
  637. >>> class AttributeFilter(etree.TreeBuilder):
  638. ... def start(self, tag, attrib):
  639. ... attrib = dict(attrib)
  640. ... if 'evil' in attrib:
  641. ... del attrib['evil']
  642. ... return super(AttributeFilter, self).start(tag, attrib)
  643. >>> parser = etree.XMLPullParser(target=AttributeFilter())
  644. >>> parser.feed('<root><child1 test="123" /><child2 evil="YES" /></root>')
  645. >>> for action, element in parser.read_events():
  646. ... print('%s: %s(%r)' % (action, element.tag, element.attrib))
  647. end: child1({'test': '123'})
  648. end: child2({})
  649. end: root({})
  650. >>> root = parser.close()
  651. iterparse and iterwalk
  652. ======================
  653. As known from ElementTree, the ``iterparse()`` utility function
  654. returns an iterator that generates parser events for an XML file (or
  655. file-like object), while building the tree. You can think of it as
  656. a blocking wrapper around the ``XMLPullParser`` that automatically and
  657. incrementally reads data from the input file for you and provides a
  658. single iterator for them:
  659. .. sourcecode:: pycon
  660. >>> xml = '''
  661. ... <root>
  662. ... <element key='value'>text</element>
  663. ... <element>text</element>tail
  664. ... <empty-element xmlns="http://testns/" />
  665. ... </root>
  666. ... '''
  667. >>> context = etree.iterparse(StringIO(xml))
  668. >>> for action, elem in context:
  669. ... print("%s: %s" % (action, elem.tag))
  670. end: element
  671. end: element
  672. end: {http://testns/}empty-element
  673. end: root
  674. After parsing, the resulting tree is available through the ``root`` property
  675. of the iterator:
  676. .. sourcecode:: pycon
  677. >>> context.root.tag
  678. 'root'
  679. The other event types can be activated with the ``events`` keyword argument:
  680. .. sourcecode:: pycon
  681. >>> events = ("start", "end")
  682. >>> context = etree.iterparse(StringIO(xml), events=events)
  683. >>> for action, elem in context:
  684. ... print("%s: %s" % (action, elem.tag))
  685. start: root
  686. start: element
  687. end: element
  688. start: element
  689. end: element
  690. start: {http://testns/}empty-element
  691. end: {http://testns/}empty-element
  692. end: root
  693. ``iterparse()`` also supports the ``tag`` argument for selective event
  694. iteration and several other parameters that control the parser setup.
  695. The ``tag`` argument can be a single tag or a sequence of tags.
  696. You can also use it to parse HTML input by passing ``html=True``.
  697. iterwalk
  698. --------
  699. For convenience, lxml also provides an ``iterwalk()`` function.
  700. It behaves exactly like ``iterparse()``, but works on Elements and
  701. ElementTrees. Here is an example for a tree parsed by ``iterparse()``:
  702. .. sourcecode:: pycon
  703. >>> f = StringIO(xml)
  704. >>> context = etree.iterparse(
  705. ... f, events=("start", "end"), tag="element")
  706. >>> for action, elem in context:
  707. ... print("%s: %s" % (action, elem.tag))
  708. start: element
  709. end: element
  710. start: element
  711. end: element
  712. >>> root = context.root
  713. And now we can take the resulting in-memory tree and iterate over it
  714. using ``iterwalk()`` to get the exact same events without parsing the
  715. input again:
  716. .. sourcecode:: pycon
  717. >>> context = etree.iterwalk(
  718. ... root, events=("start", "end"), tag="element")
  719. >>> for action, elem in context:
  720. ... print("%s: %s" % (action, elem.tag))
  721. start: element
  722. end: element
  723. start: element
  724. end: element
  725. In order to avoid wasting time on uninteresting parts of the tree, the ``iterwalk``
  726. iterator can be instructed to skip over an entire subtree with its
  727. ``.skip_subtree()`` method.
  728. .. sourcecode:: pycon
  729. >>> root = etree.XML('''
  730. ... <root>
  731. ... <a> <b /> </a>
  732. ... <c />
  733. ... </root>
  734. ... ''')
  735. >>> context = etree.iterwalk(root, events=("start", "end"))
  736. >>> for action, elem in context:
  737. ... print("%s: %s" % (action, elem.tag))
  738. ... if action == 'start' and elem.tag == 'a':
  739. ... context.skip_subtree() # ignore <b>
  740. start: root
  741. start: a
  742. end: a
  743. start: c
  744. end: c
  745. end: root
  746. Note that ``.skip_subtree()`` only has an effect when handling ``start`` or
  747. ``start-ns`` events.
  748. Python unicode strings
  749. ======================
  750. lxml.etree has broader support for Python unicode strings than the ElementTree
  751. library. First of all, where ElementTree would raise an exception, the
  752. parsers in lxml.etree can handle unicode strings straight away. This is most
  753. helpful for XML snippets embedded in source code using the ``XML()``
  754. function:
  755. .. sourcecode:: pycon
  756. >>> root = etree.XML( u'<test> \uf8d1 + \uf8d2 </test>' )
  757. This requires, however, that unicode strings do not specify a conflicting
  758. encoding themselves and thus lie about their real encoding:
  759. .. sourcecode:: pycon
  760. >>> etree.XML( u'<?xml version="1.0" encoding="ASCII"?>\n' +
  761. ... u'<test> \uf8d1 + \uf8d2 </test>' )
  762. Traceback (most recent call last):
  763. ...
  764. ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.
  765. Similarly, you will get errors when you try the same with HTML data in a
  766. unicode string that specifies a charset in a meta tag of the header. You
  767. should generally avoid converting XML/HTML data to unicode before passing it
  768. into the parsers. It is both slower and error prone.
  769. Serialising to Unicode strings
  770. ------------------------------
  771. To serialize the result, you would normally use the ``tostring()``
  772. module function, which serializes to plain ASCII by default or a
  773. number of other byte encodings if asked for:
  774. .. sourcecode:: pycon
  775. >>> etree.tostring(root)
  776. b'<test> &#63697; + &#63698; </test>'
  777. >>> etree.tostring(root, encoding='UTF-8', xml_declaration=False)
  778. b'<test> \xef\xa3\x91 + \xef\xa3\x92 </test>'
  779. As an extension, lxml.etree recognises the name 'unicode' as an argument
  780. to the encoding parameter to build a Python unicode representation of a tree:
  781. .. sourcecode:: pycon
  782. >>> etree.tostring(root, encoding='unicode')
  783. u'<test> \uf8d1 + \uf8d2 </test>'
  784. >>> el = etree.Element("test")
  785. >>> etree.tostring(el, encoding='unicode')
  786. u'<test/>'
  787. >>> subel = etree.SubElement(el, "subtest")
  788. >>> etree.tostring(el, encoding='unicode')
  789. u'<test><subtest/></test>'
  790. >>> tree = etree.ElementTree(el)
  791. >>> etree.tostring(tree, encoding='unicode')
  792. u'<test><subtest/></test>'
  793. The result of ``tostring(encoding='unicode')`` can be treated like any
  794. other Python unicode string and then passed back into the parsers.
  795. However, if you want to save the result to a file or pass it over the
  796. network, you should use ``write()`` or ``tostring()`` with a byte
  797. encoding (typically UTF-8) to serialize the XML. The main reason is
  798. that unicode strings returned by ``tostring(encoding='unicode')`` are
  799. not byte streams and they never have an XML declaration to specify
  800. their encoding. These strings are most likely not parsable by other
  801. XML libraries.
  802. For normal byte encodings, the ``tostring()`` function automatically
  803. adds a declaration as needed that reflects the encoding of the
  804. returned string. This makes it possible for other parsers to
  805. correctly parse the XML byte stream. Note that using ``tostring()``
  806. with UTF-8 is also considerably faster in most cases.