parsing.txt 32 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751752753754755756757758759760761762763764765766767768769770771772773774775776777778779780781782783784785786787788789790791792793794795796797798799800801802803804805806807808809810811812813814815816817818819820821822823824825826827828829830831832833834835836837838839840841842843844845846847848849850851852853854855856857858859860861862863864865866867868869870871872873874875876877878879880881882883884885886887888889890891892893894895896897898899900901902903904905906907908909910911912913914915916917918919920921922923924925926927928929930931932933934935936937938939940941942943944945946947948949950951952953954955956957958959960961962963964965966967968969970971972973974975976977978979980981982983984985986987988989990991992993994995996997998999100010011002100310041005100610071008100910101011101210131014101510161017101810191020
  1. ==============================
  2. Parsing XML and HTML with lxml
  3. ==============================
  4. lxml provides a very simple and powerful API for parsing XML and HTML. It
  5. supports one-step parsing as well as step-by-step parsing using an
  6. event-driven API (currently only for XML).
  7. .. contents::
  8. ..
  9. 1 Parsers
  10. 1.1 Parser options
  11. 1.2 Error log
  12. 1.3 Parsing HTML
  13. 1.4 Doctype information
  14. 2 The target parser interface
  15. 3 The feed parser interface
  16. 4 Incremental event parsing
  17. 4.1 Event types
  18. 4.1 Modifying the tree
  19. 4.3 Selective tag events
  20. 4.4 Comments and PIs
  21. 4.5 Events with custom targets
  22. 5 iterparse and iterwalk
  23. 5.1 iterwalk
  24. 6 Python unicode strings
  25. 6.1 Serialising to Unicode strings
  26. The usual setup procedure:
  27. .. sourcecode:: pycon
  28. >>> from lxml import etree
  29. The following examples also use StringIO or BytesIO to show how to parse
  30. from files and file-like objects. Both are available in the ``io`` module:
  31. .. sourcecode:: python
  32. from io import StringIO, BytesIO
  33. ..
  34. >>> from lxml import usedoctest
  35. >>> try: from StringIO import StringIO
  36. ... except ImportError:
  37. ... from io import BytesIO
  38. ... def StringIO(s):
  39. ... if isinstance(s, str): s = s.encode("UTF-8")
  40. ... return BytesIO(s)
  41. >>> try: unicode = unicode
  42. ... except NameError: unicode = str
  43. >>> import sys
  44. >>> from lxml import etree as _etree
  45. >>> if sys.version_info[0] >= 3:
  46. ... class etree_mock(object):
  47. ... def __getattr__(self, name): return getattr(_etree, name)
  48. ... def tostring(self, *args, **kwargs):
  49. ... s = _etree.tostring(*args, **kwargs)
  50. ... if isinstance(s, bytes) and bytes([10]) in s: s = s.decode("utf-8") # CR
  51. ... if s[-1] == '\n': s = s[:-1]
  52. ... return s
  53. ... else:
  54. ... class etree_mock(object):
  55. ... def __getattr__(self, name): return getattr(_etree, name)
  56. ... def tostring(self, *args, **kwargs):
  57. ... s = _etree.tostring(*args, **kwargs)
  58. ... if s[-1] == '\n': s = s[:-1]
  59. ... return s
  60. >>> etree = etree_mock()
  61. Parsers
  62. =======
  63. Parsers are represented by parser objects. There is support for parsing both
  64. XML and (broken) HTML. Note that XHTML is best parsed as XML, parsing it with
  65. the HTML parser can lead to unexpected results. Here is a simple example for
  66. parsing XML from an in-memory string:
  67. .. sourcecode:: pycon
  68. >>> xml = '<a xmlns="test"><b xmlns="test"/></a>'
  69. >>> root = etree.fromstring(xml)
  70. >>> etree.tostring(root)
  71. b'<a xmlns="test"><b xmlns="test"/></a>'
  72. To read from a file or file-like object, you can use the ``parse()`` function,
  73. which returns an ``ElementTree`` object:
  74. .. sourcecode:: pycon
  75. >>> tree = etree.parse(StringIO(xml))
  76. >>> etree.tostring(tree.getroot())
  77. b'<a xmlns="test"><b xmlns="test"/></a>'
  78. Note how the ``parse()`` function reads from a file-like object here. If
  79. parsing is done from a real file, it is more common (and also somewhat more
  80. efficient) to pass a filename:
  81. .. sourcecode:: pycon
  82. >>> tree = etree.parse("doc/test.xml")
  83. lxml can parse from a local file, an HTTP URL or an FTP URL. It also
  84. auto-detects and reads gzip-compressed XML files (.gz).
  85. If you want to parse from memory and still provide a base URL for the document
  86. (e.g. to support relative paths in an XInclude), you can pass the ``base_url``
  87. keyword argument:
  88. .. sourcecode:: pycon
  89. >>> root = etree.fromstring(xml, base_url="http://where.it/is/from.xml")
  90. Parser options
  91. --------------
  92. The parsers accept a number of setup options as keyword arguments. The above
  93. example is easily extended to clean up namespaces during parsing:
  94. .. sourcecode:: pycon
  95. >>> parser = etree.XMLParser(ns_clean=True)
  96. >>> tree = etree.parse(StringIO(xml), parser)
  97. >>> etree.tostring(tree.getroot())
  98. b'<a xmlns="test"><b/></a>'
  99. The keyword arguments in the constructor are mainly based on the libxml2
  100. parser configuration. A DTD will also be loaded if validation or attribute
  101. default values are requested.
  102. Available boolean keyword arguments:
  103. * attribute_defaults - read the DTD (if referenced by the document) and add
  104. the default attributes from it
  105. * dtd_validation - validate while parsing (if a DTD was referenced)
  106. * load_dtd - load and parse the DTD while parsing (no validation is performed)
  107. * no_network - prevent network access when looking up external
  108. documents (on by default)
  109. * ns_clean - try to clean up redundant namespace declarations
  110. * recover - try hard to parse through broken XML
  111. * remove_blank_text - discard blank text nodes between tags, also known as
  112. ignorable whitespace. This is best used together with a DTD or schema
  113. (which tells data and noise apart), otherwise a heuristic will be applied.
  114. * remove_comments - discard comments
  115. * remove_pis - discard processing instructions
  116. * strip_cdata - replace CDATA sections by normal text content (on by
  117. default)
  118. * resolve_entities - replace entities by their text value (on by
  119. default)
  120. * huge_tree - disable security restrictions and support very deep trees
  121. and very long text content (only affects libxml2 2.7+)
  122. * compact - use compact storage for short text content (on by default)
  123. Other keyword arguments:
  124. * encoding - override the document encoding
  125. * target - a parser target object that will receive the parse events
  126. (see `The target parser interface`_)
  127. * schema - an XMLSchema to validate against (see `validation <validation.html#xmlschema>`_)
  128. Error log
  129. ---------
  130. Parsers have an ``error_log`` property that lists the errors and
  131. warnings of the last parser run:
  132. .. sourcecode:: pycon
  133. >>> parser = etree.XMLParser()
  134. >>> print(len(parser.error_log))
  135. 0
  136. >>> tree = etree.XML("<root>\n</b>", parser)
  137. Traceback (most recent call last):
  138. ...
  139. lxml.etree.XMLSyntaxError: Opening and ending tag mismatch: root line 1 and b, line 2, column 5
  140. >>> print(len(parser.error_log))
  141. 1
  142. >>> error = parser.error_log[0]
  143. >>> print(error.message)
  144. Opening and ending tag mismatch: root line 1 and b
  145. >>> print(error.line)
  146. 2
  147. >>> print(error.column)
  148. 5
  149. Each entry in the log has the following properties:
  150. * ``message``: the message text
  151. * ``domain``: the domain ID (see the lxml.etree.ErrorDomains class)
  152. * ``type``: the message type ID (see the lxml.etree.ErrorTypes class)
  153. * ``level``: the log level ID (see the lxml.etree.ErrorLevels class)
  154. * ``line``: the line at which the message originated (if applicable)
  155. * ``column``: the character column at which the message originated (if applicable)
  156. * ``filename``: the name of the file in which the message originated (if applicable)
  157. For convenience, there are also three properties that provide readable
  158. names for the ID values:
  159. * ``domain_name``
  160. * ``type_name``
  161. * ``level_name``
  162. To filter for a specific kind of message, use the different
  163. ``filter_*()`` methods on the error log (see the
  164. lxml.etree._ListErrorLog class).
  165. Parsing HTML
  166. ------------
  167. HTML parsing is similarly simple. The parsers have a ``recover``
  168. keyword argument that the HTMLParser sets by default. It lets libxml2
  169. try its best to return a valid HTML tree with all content it can
  170. manage to parse. It will not raise an exception on parser errors.
  171. You should use libxml2 version 2.6.21 or newer to take advantage of
  172. this feature.
  173. .. sourcecode:: pycon
  174. >>> broken_html = "<html><head><title>test<body><h1>page title</h3>"
  175. >>> parser = etree.HTMLParser()
  176. >>> tree = etree.parse(StringIO(broken_html), parser)
  177. >>> result = etree.tostring(tree.getroot(),
  178. ... pretty_print=True, method="html")
  179. >>> print(result)
  180. <html>
  181. <head>
  182. <title>test</title>
  183. </head>
  184. <body>
  185. <h1>page title</h1>
  186. </body>
  187. </html>
  188. Lxml has an HTML function, similar to the XML shortcut known from
  189. ElementTree:
  190. .. sourcecode:: pycon
  191. >>> html = etree.HTML(broken_html)
  192. >>> result = etree.tostring(html, pretty_print=True, method="html")
  193. >>> print(result)
  194. <html>
  195. <head>
  196. <title>test</title>
  197. </head>
  198. <body>
  199. <h1>page title</h1>
  200. </body>
  201. </html>
  202. The support for parsing broken HTML depends entirely on libxml2's recovery
  203. algorithm. It is *not* the fault of lxml if you find documents that are so
  204. heavily broken that the parser cannot handle them. There is also no guarantee
  205. that the resulting tree will contain all data from the original document. The
  206. parser may have to drop seriously broken parts when struggling to keep
  207. parsing. Especially misplaced meta tags can suffer from this, which may lead
  208. to encoding problems.
  209. Note that the result is a valid HTML tree, but it may not be a
  210. well-formed XML tree. For example, XML forbids double hyphens in
  211. comments, which the HTML parser will happily accept in recovery mode.
  212. Therefore, if your goal is to serialise an HTML document as an
  213. XML/XHTML document after parsing, you may have to apply some manual
  214. preprocessing first.
  215. Also note that the HTML parser is meant to parse HTML documents. For
  216. XHTML documents, use the XML parser, which is namespace aware.
  217. Doctype information
  218. -------------------
  219. The use of the libxml2 parsers makes some additional information available at
  220. the API level. Currently, ElementTree objects can access the DOCTYPE
  221. information provided by a parsed document, as well as the XML version and the
  222. original encoding:
  223. .. sourcecode:: pycon
  224. >>> pub_id = "-//W3C//DTD XHTML 1.0 Transitional//EN"
  225. >>> sys_url = "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"
  226. >>> doctype_string = '<!DOCTYPE html PUBLIC "%s" "%s">' % (pub_id, sys_url)
  227. >>> xml_header = '<?xml version="1.0" encoding="ascii"?>'
  228. >>> xhtml = xml_header + doctype_string + '<html><body></body></html>'
  229. >>> tree = etree.parse(StringIO(xhtml))
  230. >>> docinfo = tree.docinfo
  231. >>> print(docinfo.public_id)
  232. -//W3C//DTD XHTML 1.0 Transitional//EN
  233. >>> print(docinfo.system_url)
  234. http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd
  235. >>> docinfo.doctype == doctype_string
  236. True
  237. >>> print(docinfo.xml_version)
  238. 1.0
  239. >>> print(docinfo.encoding)
  240. ascii
  241. The target parser interface
  242. ===========================
  243. .. _`As in ElementTree`: http://effbot.org/elementtree/elementtree-xmlparser.htm
  244. `As in ElementTree`_, and similar to a SAX event handler, you can pass
  245. a target object to the parser:
  246. .. sourcecode:: pycon
  247. >>> class EchoTarget(object):
  248. ... def start(self, tag, attrib):
  249. ... print("start %s %r" % (tag, dict(attrib)))
  250. ... def end(self, tag):
  251. ... print("end %s" % tag)
  252. ... def data(self, data):
  253. ... print("data %r" % data)
  254. ... def comment(self, text):
  255. ... print("comment %s" % text)
  256. ... def close(self):
  257. ... print("close")
  258. ... return "closed!"
  259. >>> parser = etree.XMLParser(target = EchoTarget())
  260. >>> result = etree.XML("<element>some<!--comment-->text</element>",
  261. ... parser)
  262. start element {}
  263. data u'some'
  264. comment comment
  265. data u'text'
  266. end element
  267. close
  268. >>> print(result)
  269. closed!
  270. It is important for the ``.close()`` method to reset the parser target
  271. to a usable state, so that you can reuse the parser as often as you
  272. like:
  273. .. sourcecode:: pycon
  274. >>> result = etree.XML("<element>some<!--comment-->text</element>",
  275. ... parser)
  276. start element {}
  277. data u'some'
  278. comment comment
  279. data u'text'
  280. end element
  281. close
  282. >>> print(result)
  283. closed!
  284. Starting with lxml 2.3, the ``.close()`` method will also be called in
  285. the error case. This diverges from the behaviour of ElementTree, but
  286. allows target objects to clean up their state in all situations, so
  287. that the parser can reuse them afterwards.
  288. .. sourcecode:: pycon
  289. >>> class CollectorTarget(object):
  290. ... def __init__(self):
  291. ... self.events = []
  292. ... def start(self, tag, attrib):
  293. ... self.events.append("start %s %r" % (tag, dict(attrib)))
  294. ... def end(self, tag):
  295. ... self.events.append("end %s" % tag)
  296. ... def data(self, data):
  297. ... self.events.append("data %r" % data)
  298. ... def comment(self, text):
  299. ... self.events.append("comment %s" % text)
  300. ... def close(self):
  301. ... self.events.append("close")
  302. ... return "closed!"
  303. >>> parser = etree.XMLParser(target = CollectorTarget())
  304. >>> result = etree.XML("<element>some</error>",
  305. ... parser) # doctest: +ELLIPSIS
  306. Traceback (most recent call last):
  307. ...
  308. lxml.etree.XMLSyntaxError: Opening and ending tag mismatch...
  309. >>> for event in parser.target.events:
  310. ... print(event)
  311. start element {}
  312. data u'some'
  313. close
  314. Note that the parser does *not* build a tree when using a parser
  315. target. The result of the parser run is whatever the target object
  316. returns from its ``.close()`` method. If you want to return an XML
  317. tree here, you have to create it programmatically in the target
  318. object. An example for a parser target that builds a tree is the
  319. ``TreeBuilder``:
  320. .. sourcecode:: pycon
  321. >>> parser = etree.XMLParser(target = etree.TreeBuilder())
  322. >>> result = etree.XML("<element>some<!--comment-->text</element>",
  323. ... parser)
  324. >>> print(result.tag)
  325. element
  326. >>> print(result[0].text)
  327. comment
  328. The feed parser interface
  329. =========================
  330. Since lxml 2.0, the parsers have a feed parser interface that is
  331. compatible to the `ElementTree parsers`_. You can use it to feed data
  332. into the parser in a controlled step-by-step way.
  333. In lxml.etree, you can use both interfaces to a parser at the same
  334. time: the ``parse()`` or ``XML()`` functions, and the feed parser
  335. interface. Both are independent and will not conflict (except if used
  336. in conjunction with a parser target object as described above).
  337. .. _`ElementTree parsers`: http://effbot.org/elementtree/elementtree-xmlparser.htm
  338. To start parsing with a feed parser, just call its ``feed()`` method
  339. to feed it some data.
  340. .. sourcecode:: pycon
  341. >>> parser = etree.XMLParser()
  342. >>> for data in ('<?xml versio', 'n="1.0"?', '><roo', 't><a', '/></root>'):
  343. ... parser.feed(data)
  344. When you are done parsing, you **must** call the ``close()`` method to
  345. retrieve the root Element of the parse result document, and to unlock the
  346. parser:
  347. .. sourcecode:: pycon
  348. >>> root = parser.close()
  349. >>> print(root.tag)
  350. root
  351. >>> print(root[0].tag)
  352. a
  353. If you do not call ``close()``, the parser will stay locked and
  354. subsequent feeds will keep appending data, usually resulting in a non
  355. well-formed document and an unexpected parser error. So make sure you
  356. always close the parser after use, also in the exception case.
  357. Another way of achieving the same step-by-step parsing is by writing your own
  358. file-like object that returns a chunk of data on each ``read()`` call. Where
  359. the feed parser interface allows you to actively pass data chunks into the
  360. parser, a file-like object passively responds to ``read()`` requests of the
  361. parser itself. Depending on the data source, either way may be more natural.
  362. Note that the feed parser has its own error log called
  363. ``feed_error_log``. Errors in the feed parser do not show up in the
  364. normal ``error_log`` and vice versa.
  365. You can also combine the feed parser interface with the target parser:
  366. .. sourcecode:: pycon
  367. >>> parser = etree.XMLParser(target = EchoTarget())
  368. >>> parser.feed("<eleme")
  369. >>> parser.feed("nt>some text</elem")
  370. start element {}
  371. data u'some text'
  372. >>> parser.feed("ent>")
  373. end element
  374. >>> result = parser.close()
  375. close
  376. >>> print(result)
  377. closed!
  378. Again, this prevents the automatic creation of an XML tree and leaves
  379. all the event handling to the target object. The ``close()`` method
  380. of the parser forwards the return value of the target's ``close()``
  381. method.
  382. Incremental event parsing
  383. =========================
  384. In Python 3.4, the ``xml.etree.ElementTree`` package gained an extension
  385. to the feed parser interface that is implemented by the ``XMLPullParser``
  386. class. It additionally allows processing parse events after each
  387. incremental parsing step, by calling the ``.read_events()`` method and
  388. iterating over the result. This is most useful for non-blocking execution
  389. environments where data chunks arrive one after the other and should be
  390. processed as far as possible in each step.
  391. The same feature is available in lxml 3.3. The basic usage is as follows:
  392. .. sourcecode:: pycon
  393. >>> parser = etree.XMLPullParser(events=('start', 'end'))
  394. >>> def print_events(parser):
  395. ... for action, element in parser.read_events():
  396. ... print('%s: %s' % (action, element.tag))
  397. >>> parser.feed('<root>some text')
  398. >>> print_events(parser)
  399. start: root
  400. >>> print_events(parser) # well, no more events, as before ...
  401. >>> parser.feed('<child><a />')
  402. >>> print_events(parser)
  403. start: child
  404. start: a
  405. end: a
  406. >>> parser.feed('</child></roo')
  407. >>> print_events(parser)
  408. end: child
  409. >>> parser.feed('t>')
  410. >>> print_events(parser)
  411. end: root
  412. Just like the normal feed parser, the ``XMLPullParser`` builds a tree in
  413. memory (and you should always call the ``.close()`` method when done with
  414. parsing):
  415. .. sourcecode:: pycon
  416. >>> root = parser.close()
  417. >>> etree.tostring(root)
  418. b'<root>some text<child><a/></child></root>'
  419. However, since the parser provides incremental access to that tree,
  420. you can explicitly delete content that you no longer need once you
  421. have processed it. Read the section on `Modifying the tree`_ below
  422. to see what you can do here and what kind of modifications you should
  423. avoid.
  424. In lxml, it is enough to call the ``.read_events()`` method once as
  425. the iterator it returns can be reused when new events are available.
  426. Also, as known from other iterators in lxml, you can pass a ``tag``
  427. argument that selects which parse events are returned by the
  428. ``.read_events()`` iterator.
  429. Event types
  430. -----------
  431. The parse events are tuples ``(event-type, object)``. The event types
  432. supported by ElementTree and lxml.etree are the strings 'start', 'end',
  433. 'start-ns' and 'end-ns'. The 'start' and 'end' events represent opening
  434. and closing elements. They are accompanied by the respective Element
  435. instance. By default, only 'end' events are generated, whereas the
  436. example above requested the generation of both 'start' and 'end' events.
  437. The 'start-ns' and 'end-ns' events notify about namespace declarations.
  438. They do not come with Elements. Instead, the value of the 'start-ns'
  439. event is a tuple ``(prefix, namespaceURI)`` that designates the beginning
  440. of a prefix-namespace mapping. The corresponding ``end-ns`` event does
  441. not have a value (None). It is common practice to use a list as namespace
  442. stack and pop the last entry on the 'end-ns' event.
  443. .. sourcecode:: pycon
  444. >>> def print_events(events):
  445. ... for action, obj in events:
  446. ... if action in ('start', 'end'):
  447. ... print("%s: %s" % (action, obj.tag))
  448. ... elif action == 'start-ns':
  449. ... print("%s: %s" % (action, obj))
  450. ... else:
  451. ... print(action)
  452. >>> event_types = ("start", "end", "start-ns", "end-ns")
  453. >>> parser = etree.XMLPullParser(event_types)
  454. >>> events = parser.read_events()
  455. >>> parser.feed('<root><element>')
  456. >>> print_events(events)
  457. start: root
  458. start: element
  459. >>> parser.feed('text</element><element>text</element>')
  460. >>> print_events(events)
  461. end: element
  462. start: element
  463. end: element
  464. >>> parser.feed('<empty-element xmlns="http://testns/" />')
  465. >>> print_events(events)
  466. start-ns: ('', 'http://testns/')
  467. start: {http://testns/}empty-element
  468. end: {http://testns/}empty-element
  469. end-ns
  470. >>> parser.feed('</root>')
  471. >>> print_events(events)
  472. end: root
  473. Modifying the tree
  474. ------------------
  475. You can modify the element and its descendants when handling the
  476. 'end' event. To save memory, for example, you can remove subtrees
  477. that are no longer needed:
  478. .. sourcecode:: pycon
  479. >>> parser = etree.XMLPullParser()
  480. >>> events = parser.read_events()
  481. >>> parser.feed('<root><element key="value">text</element>')
  482. >>> parser.feed('<element><child /></element>')
  483. >>> for action, elem in events:
  484. ... print('%s: %d' % (elem.tag, len(elem))) # processing
  485. ... elem.clear() # delete children
  486. element: 0
  487. child: 0
  488. element: 1
  489. >>> parser.feed('<empty-element xmlns="http://testns/" /></root>')
  490. >>> for action, elem in events:
  491. ... print('%s: %d' % (elem.tag, len(elem))) # processing
  492. ... elem.clear() # delete children
  493. {http://testns/}empty-element: 0
  494. root: 3
  495. >>> root = parser.close()
  496. >>> etree.tostring(root)
  497. b'<root/>'
  498. **WARNING**: During the 'start' event, any content of the element,
  499. such as the descendants, following siblings or text, is not yet
  500. available and should not be accessed. Only attributes are guaranteed
  501. to be set. During the 'end' event, the element and its descendants
  502. can be freely modified, but its following siblings should not be
  503. accessed. During either of the two events, you **must not** modify or
  504. move the ancestors (parents) of the current element. You should also
  505. avoid moving or discarding the element itself. The golden rule is: do
  506. not touch anything that will have to be touched again by the parser
  507. later on.
  508. If you have elements with a long list of children in your XML file and want
  509. to save more memory during parsing, you can clean up the preceding siblings
  510. of the current element:
  511. .. sourcecode:: pycon
  512. >>> for event, element in parser.read_events():
  513. ... # ... do something with the element
  514. ... element.clear() # clean up children
  515. ... while element.getprevious() is not None:
  516. ... del element.getparent()[0] # clean up preceding siblings
  517. The ``while`` loop deletes multiple siblings in a row. This is only necessary
  518. if you skipped over some of them using the ``tag`` keyword argument.
  519. Otherwise, a simple ``if`` should do. The more selective your tag is,
  520. however, the more thought you will have to put into finding the right way to
  521. clean up the elements that were skipped. Therefore, it is sometimes easier to
  522. traverse all elements and do the tag selection by hand in the event handler
  523. code.
  524. Selective tag events
  525. --------------------
  526. As an extension over ElementTree, lxml.etree accepts a ``tag`` keyword
  527. argument just like ``element.iter(tag)``. This restricts events to a
  528. specific tag or namespace:
  529. .. sourcecode:: pycon
  530. >>> parser = etree.XMLPullParser(tag="element")
  531. >>> parser.feed('<root><element key="value">text</element>')
  532. >>> parser.feed('<element><child /></element>')
  533. >>> parser.feed('<empty-element xmlns="http://testns/" /></root>')
  534. >>> for action, elem in parser.read_events():
  535. ... print("%s: %s" % (action, elem.tag))
  536. end: element
  537. end: element
  538. >>> event_types = ("start", "end")
  539. >>> parser = etree.XMLPullParser(event_types, tag="{http://testns/}*")
  540. >>> parser.feed('<root><element key="value">text</element>')
  541. >>> parser.feed('<element><child /></element>')
  542. >>> parser.feed('<empty-element xmlns="http://testns/" /></root>')
  543. >>> for action, elem in parser.read_events():
  544. ... print("%s: %s" % (action, elem.tag))
  545. start: {http://testns/}empty-element
  546. end: {http://testns/}empty-element
  547. Comments and PIs
  548. ----------------
  549. As an extension over ElementTree, the ``XMLPullParser`` in lxml.etree
  550. also supports the event types 'comment' and 'pi' for the respective
  551. XML structures.
  552. .. sourcecode:: pycon
  553. >>> event_types = ("start", "end", "comment", "pi")
  554. >>> parser = etree.XMLPullParser(event_types)
  555. >>> parser.feed('<?some pi ?><!-- a comment --><root>')
  556. >>> parser.feed('<element key="value">text</element>')
  557. >>> parser.feed('<!-- another comment -->')
  558. >>> parser.feed('<element>text</element>tail')
  559. >>> parser.feed('<empty-element xmlns="http://testns/" />')
  560. >>> parser.feed('</root>')
  561. >>> for action, elem in parser.read_events():
  562. ... if action in ('start', 'end'):
  563. ... print("%s: %s" % (action, elem.tag))
  564. ... elif action == 'pi':
  565. ... print("%s: -%s=%s-" % (action, elem.target, elem.text))
  566. ... else: # 'comment'
  567. ... print("%s: -%s-" % (action, elem.text))
  568. pi: -some=pi -
  569. comment: - a comment -
  570. start: root
  571. start: element
  572. end: element
  573. comment: - another comment -
  574. start: element
  575. end: element
  576. start: {http://testns/}empty-element
  577. end: {http://testns/}empty-element
  578. end: root
  579. >>> root = parser.close()
  580. >>> print(root.tag)
  581. root
  582. Events with custom targets
  583. --------------------------
  584. You can combine the pull parser with a parser target. In that case,
  585. it is the target's responsibility to generate event values. Whatever
  586. it returns from its ``.start()`` and ``.end()`` methods will be returned
  587. by the pull parser as the second item of the parse events tuple.
  588. .. sourcecode:: pycon
  589. >>> class Target(object):
  590. ... def start(self, tag, attrib):
  591. ... print('-> start(%s)' % tag)
  592. ... return '>>START: %s<<' % tag
  593. ... def end(self, tag):
  594. ... print('-> end(%s)' % tag)
  595. ... return '>>END: %s<<' % tag
  596. ... def close(self):
  597. ... print('-> close()')
  598. ... return "CLOSED!"
  599. >>> event_types = ('start', 'end')
  600. >>> parser = etree.XMLPullParser(event_types, target=Target())
  601. >>> parser.feed('<root><child1 /><child2 /></root>')
  602. -> start(root)
  603. -> start(child1)
  604. -> end(child1)
  605. -> start(child2)
  606. -> end(child2)
  607. -> end(root)
  608. >>> for action, value in parser.read_events():
  609. ... print('%s: %s' % (action, value))
  610. start: >>START: root<<
  611. start: >>START: child1<<
  612. end: >>END: child1<<
  613. start: >>START: child2<<
  614. end: >>END: child2<<
  615. end: >>END: root<<
  616. >>> print(parser.close())
  617. -> close()
  618. CLOSED!
  619. As you can see, the event values do not even have to be Element objects.
  620. The target is generally free to decide how it wants to create an XML tree
  621. or whatever else it wants to make of the parser callbacks. In many cases,
  622. however, you will want to make your custom target inherit from the
  623. ``TreeBuilder`` class in order to have it build a tree that you can process
  624. normally. The ``start()`` and ``.end()`` methods of ``TreeBuilder`` return
  625. the Element object that was created, so you can override them and modify
  626. the input or output according to your needs. Here is an example that
  627. filters attributes before they are being added to the tree:
  628. .. sourcecode:: pycon
  629. >>> class AttributeFilter(etree.TreeBuilder):
  630. ... def start(self, tag, attrib):
  631. ... attrib = dict(attrib)
  632. ... if 'evil' in attrib:
  633. ... del attrib['evil']
  634. ... return super(AttributeFilter, self).start(tag, attrib)
  635. >>> parser = etree.XMLPullParser(target=AttributeFilter())
  636. >>> parser.feed('<root><child1 test="123" /><child2 evil="YES" /></root>')
  637. >>> for action, element in parser.read_events():
  638. ... print('%s: %s(%r)' % (action, element.tag, element.attrib))
  639. end: child1({'test': '123'})
  640. end: child2({})
  641. end: root({})
  642. >>> root = parser.close()
  643. iterparse and iterwalk
  644. ======================
  645. As known from ElementTree, the ``iterparse()`` utility function
  646. returns an iterator that generates parser events for an XML file (or
  647. file-like object), while building the tree. You can think of it as
  648. a blocking wrapper around the ``XMLPullParser`` that automatically and
  649. incrementally reads data from the input file for you and provides a
  650. single iterator for them:
  651. .. sourcecode:: pycon
  652. >>> xml = '''
  653. ... <root>
  654. ... <element key='value'>text</element>
  655. ... <element>text</element>tail
  656. ... <empty-element xmlns="http://testns/" />
  657. ... </root>
  658. ... '''
  659. >>> context = etree.iterparse(StringIO(xml))
  660. >>> for action, elem in context:
  661. ... print("%s: %s" % (action, elem.tag))
  662. end: element
  663. end: element
  664. end: {http://testns/}empty-element
  665. end: root
  666. After parsing, the resulting tree is available through the ``root`` property
  667. of the iterator:
  668. .. sourcecode:: pycon
  669. >>> context.root.tag
  670. 'root'
  671. The other event types can be activated with the ``events`` keyword argument:
  672. .. sourcecode:: pycon
  673. >>> events = ("start", "end")
  674. >>> context = etree.iterparse(StringIO(xml), events=events)
  675. >>> for action, elem in context:
  676. ... print("%s: %s" % (action, elem.tag))
  677. start: root
  678. start: element
  679. end: element
  680. start: element
  681. end: element
  682. start: {http://testns/}empty-element
  683. end: {http://testns/}empty-element
  684. end: root
  685. ``iterparse()`` also supports the ``tag`` argument for selective event
  686. iteration and several other parameters that control the parser setup.
  687. You can also use it to parse HTML input by passing ``html=True``.
  688. iterwalk
  689. --------
  690. A second extension over ElementTree is the ``iterwalk()`` function.
  691. It behaves exactly like ``iterparse()``, but works on Elements and
  692. ElementTrees. Here is an example for a tree parsed by ``iterparse()``:
  693. .. sourcecode:: pycon
  694. >>> f = StringIO(xml)
  695. >>> context = etree.iterparse(
  696. ... f, events=("start", "end"), tag="element")
  697. >>> for action, elem in context:
  698. ... print("%s: %s" % (action, elem.tag))
  699. start: element
  700. end: element
  701. start: element
  702. end: element
  703. >>> root = context.root
  704. And now we can take the resulting in-memory tree and iterate over it
  705. using ``iterwalk()`` to get the exact same events without parsing the
  706. input again:
  707. >>> context = etree.iterwalk(
  708. ... root, events=("start", "end"), tag="element")
  709. >>> for action, elem in context:
  710. ... print("%s: %s" % (action, elem.tag))
  711. start: element
  712. end: element
  713. start: element
  714. end: element
  715. Python unicode strings
  716. ======================
  717. lxml.etree has broader support for Python unicode strings than the ElementTree
  718. library. First of all, where ElementTree would raise an exception, the
  719. parsers in lxml.etree can handle unicode strings straight away. This is most
  720. helpful for XML snippets embedded in source code using the ``XML()``
  721. function:
  722. .. sourcecode:: pycon
  723. >>> root = etree.XML( u'<test> \uf8d1 + \uf8d2 </test>' )
  724. This requires, however, that unicode strings do not specify a conflicting
  725. encoding themselves and thus lie about their real encoding:
  726. .. sourcecode:: pycon
  727. >>> etree.XML( u'<?xml version="1.0" encoding="ASCII"?>\n' +
  728. ... u'<test> \uf8d1 + \uf8d2 </test>' )
  729. Traceback (most recent call last):
  730. ...
  731. ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.
  732. Similarly, you will get errors when you try the same with HTML data in a
  733. unicode string that specifies a charset in a meta tag of the header. You
  734. should generally avoid converting XML/HTML data to unicode before passing it
  735. into the parsers. It is both slower and error prone.
  736. Serialising to Unicode strings
  737. ------------------------------
  738. To serialize the result, you would normally use the ``tostring()``
  739. module function, which serializes to plain ASCII by default or a
  740. number of other byte encodings if asked for:
  741. .. sourcecode:: pycon
  742. >>> etree.tostring(root)
  743. b'<test> &#63697; + &#63698; </test>'
  744. >>> etree.tostring(root, encoding='UTF-8', xml_declaration=False)
  745. b'<test> \xef\xa3\x91 + \xef\xa3\x92 </test>'
  746. As an extension, lxml.etree recognises the name 'unicode' as an argument
  747. to the encoding parameter to build a Python unicode representation of a tree:
  748. .. sourcecode:: pycon
  749. >>> etree.tostring(root, encoding='unicode')
  750. u'<test> \uf8d1 + \uf8d2 </test>'
  751. >>> el = etree.Element("test")
  752. >>> etree.tostring(el, encoding='unicode')
  753. u'<test/>'
  754. >>> subel = etree.SubElement(el, "subtest")
  755. >>> etree.tostring(el, encoding='unicode')
  756. u'<test><subtest/></test>'
  757. >>> tree = etree.ElementTree(el)
  758. >>> etree.tostring(tree, encoding='unicode')
  759. u'<test><subtest/></test>'
  760. The result of ``tostring(encoding='unicode')`` can be treated like any
  761. other Python unicode string and then passed back into the parsers.
  762. However, if you want to save the result to a file or pass it over the
  763. network, you should use ``write()`` or ``tostring()`` with a byte
  764. encoding (typically UTF-8) to serialize the XML. The main reason is
  765. that unicode strings returned by ``tostring(encoding='unicode')`` are
  766. not byte streams and they never have an XML declaration to specify
  767. their encoding. These strings are most likely not parsable by other
  768. XML libraries.
  769. For normal byte encodings, the ``tostring()`` function automatically
  770. adds a declaration as needed that reflects the encoding of the
  771. returned string. This makes it possible for other parsers to
  772. correctly parse the XML byte stream. Note that using ``tostring()``
  773. with UTF-8 is also considerably faster in most cases.