parsing.txt 23 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751752753754755756757
  1. ==============================
  2. Parsing XML and HTML with lxml
  3. ==============================
  4. lxml provides a very simple and powerful API for parsing XML and HTML. It
  5. supports one-step parsing as well as step-by-step parsing using an
  6. event-driven API (currently only for XML).
  7. .. contents::
  8. ..
  9. 1 Parsers
  10. 1.1 Parser options
  11. 1.2 Error log
  12. 1.3 Parsing HTML
  13. 1.4 Doctype information
  14. 2 The target parser interface
  15. 3 The feed parser interface
  16. 4 iterparse and iterwalk
  17. 4.1 Selective tag events
  18. 4.2 Comments and PIs
  19. 4.3 Modifying the tree
  20. 4.4 iterwalk
  21. 5 Python unicode strings
  22. 5.1 Serialising to Unicode strings
  23. The usual setup procedure:
  24. .. sourcecode:: pycon
  25. >>> from lxml import etree
  26. ..
  27. >>> from lxml import usedoctest
  28. >>> try: from StringIO import StringIO
  29. ... except ImportError:
  30. ... from io import BytesIO
  31. ... def StringIO(s):
  32. ... if isinstance(s, str): s = s.encode("UTF-8")
  33. ... return BytesIO(s)
  34. >>> try: unicode = __builtins__["unicode"]
  35. ... except (NameError, KeyError): unicode = str
  36. >>> import sys
  37. >>> from lxml import etree as _etree
  38. >>> if sys.version_info[0] >= 3:
  39. ... class etree_mock(object):
  40. ... def __getattr__(self, name): return getattr(_etree, name)
  41. ... def tostring(self, *args, **kwargs):
  42. ... s = _etree.tostring(*args, **kwargs)
  43. ... if isinstance(s, bytes) and bytes([10]) in s: s = s.decode("utf-8") # CR
  44. ... if s[-1] == '\n': s = s[:-1]
  45. ... return s
  46. ... else:
  47. ... class etree_mock(object):
  48. ... def __getattr__(self, name): return getattr(_etree, name)
  49. ... def tostring(self, *args, **kwargs):
  50. ... s = _etree.tostring(*args, **kwargs)
  51. ... if s[-1] == '\n': s = s[:-1]
  52. ... return s
  53. >>> etree = etree_mock()
  54. Parsers
  55. =======
  56. Parsers are represented by parser objects. There is support for parsing both
  57. XML and (broken) HTML. Note that XHTML is best parsed as XML, parsing it with
  58. the HTML parser can lead to unexpected results. Here is a simple example for
  59. parsing XML from an in-memory string:
  60. .. sourcecode:: pycon
  61. >>> xml = '<a xmlns="test"><b xmlns="test"/></a>'
  62. >>> root = etree.fromstring(xml)
  63. >>> etree.tostring(root)
  64. b'<a xmlns="test"><b xmlns="test"/></a>'
  65. To read from a file or file-like object, you can use the ``parse()`` function,
  66. which returns an ``ElementTree`` object:
  67. .. sourcecode:: pycon
  68. >>> tree = etree.parse(StringIO(xml))
  69. >>> etree.tostring(tree.getroot())
  70. b'<a xmlns="test"><b xmlns="test"/></a>'
  71. Note how the ``parse()`` function reads from a file-like object here. If
  72. parsing is done from a real file, it is more common (and also somewhat more
  73. efficient) to pass a filename:
  74. .. sourcecode:: pycon
  75. >>> tree = etree.parse("doc/test.xml")
  76. lxml can parse from a local file, an HTTP URL or an FTP URL. It also
  77. auto-detects and reads gzip-compressed XML files (.gz).
  78. If you want to parse from memory and still provide a base URL for the document
  79. (e.g. to support relative paths in an XInclude), you can pass the ``base_url``
  80. keyword argument:
  81. .. sourcecode:: pycon
  82. >>> root = etree.fromstring(xml, base_url="http://where.it/is/from.xml")
  83. Parser options
  84. --------------
  85. The parsers accept a number of setup options as keyword arguments. The above
  86. example is easily extended to clean up namespaces during parsing:
  87. .. sourcecode:: pycon
  88. >>> parser = etree.XMLParser(ns_clean=True)
  89. >>> tree = etree.parse(StringIO(xml), parser)
  90. >>> etree.tostring(tree.getroot())
  91. b'<a xmlns="test"><b/></a>'
  92. The keyword arguments in the constructor are mainly based on the libxml2
  93. parser configuration. A DTD will also be loaded if validation or attribute
  94. default values are requested.
  95. Available boolean keyword arguments:
  96. * attribute_defaults - read the DTD (if referenced by the document) and add
  97. the default attributes from it
  98. * dtd_validation - validate while parsing (if a DTD was referenced)
  99. * load_dtd - load and parse the DTD while parsing (no validation is performed)
  100. * no_network - prevent network access when looking up external
  101. documents (on by default)
  102. * ns_clean - try to clean up redundant namespace declarations
  103. * recover - try hard to parse through broken XML
  104. * remove_blank_text - discard blank text nodes between tags
  105. * remove_comments - discard comments
  106. * remove_pis - discard processing instructions
  107. * strip_cdata - replace CDATA sections by normal text content (on by
  108. default)
  109. * resolve_entities - replace entities by their text value (on by
  110. default)
  111. * huge_tree - disable security restrictions and support very deep trees
  112. and very long text content (only affects libxml2 2.7+)
  113. * compact - use compact storage for short text content (on by default)
  114. Error log
  115. ---------
  116. Parsers have an ``error_log`` property that lists the errors of the
  117. last parser run:
  118. .. sourcecode:: pycon
  119. >>> parser = etree.XMLParser()
  120. >>> print(len(parser.error_log))
  121. 0
  122. >>> tree = etree.XML("<root></b>", parser)
  123. Traceback (most recent call last):
  124. ...
  125. lxml.etree.XMLSyntaxError: Opening and ending tag mismatch: root line 1 and b, line 1, column 11
  126. >>> print(len(parser.error_log))
  127. 1
  128. >>> error = parser.error_log[0]
  129. >>> print(error.message)
  130. Opening and ending tag mismatch: root line 1 and b
  131. >>> print(error.line)
  132. 1
  133. >>> print(error.column)
  134. 11
  135. Parsing HTML
  136. ------------
  137. HTML parsing is similarly simple. The parsers have a ``recover``
  138. keyword argument that the HTMLParser sets by default. It lets libxml2
  139. try its best to return a valid HTML tree with all content it can
  140. manage to parse. It will not raise an exception on parser errors.
  141. You should use libxml2 version 2.6.21 or newer to take advantage of
  142. this feature.
  143. .. sourcecode:: pycon
  144. >>> broken_html = "<html><head><title>test<body><h1>page title</h3>"
  145. >>> parser = etree.HTMLParser()
  146. >>> tree = etree.parse(StringIO(broken_html), parser)
  147. >>> result = etree.tostring(tree.getroot(),
  148. ... pretty_print=True, method="html")
  149. >>> print(result)
  150. <html>
  151. <head>
  152. <title>test</title>
  153. </head>
  154. <body>
  155. <h1>page title</h1>
  156. </body>
  157. </html>
  158. Lxml has an HTML function, similar to the XML shortcut known from
  159. ElementTree:
  160. .. sourcecode:: pycon
  161. >>> html = etree.HTML(broken_html)
  162. >>> result = etree.tostring(html, pretty_print=True, method="html")
  163. >>> print(result)
  164. <html>
  165. <head>
  166. <title>test</title>
  167. </head>
  168. <body>
  169. <h1>page title</h1>
  170. </body>
  171. </html>
  172. The support for parsing broken HTML depends entirely on libxml2's recovery
  173. algorithm. It is *not* the fault of lxml if you find documents that are so
  174. heavily broken that the parser cannot handle them. There is also no guarantee
  175. that the resulting tree will contain all data from the original document. The
  176. parser may have to drop seriously broken parts when struggling to keep
  177. parsing. Especially misplaced meta tags can suffer from this, which may lead
  178. to encoding problems.
  179. Note that the result is a valid HTML tree, but it may not be a
  180. well-formed XML tree. For example, XML forbids double hyphens in
  181. comments, which the HTML parser will happily accept in recovery mode.
  182. Therefore, if your goal is to serialise an HTML document as an
  183. XML/XHTML document after parsing, you may have to apply some manual
  184. preprocessing first.
  185. Doctype information
  186. -------------------
  187. The use of the libxml2 parsers makes some additional information available at
  188. the API level. Currently, ElementTree objects can access the DOCTYPE
  189. information provided by a parsed document, as well as the XML version and the
  190. original encoding:
  191. .. sourcecode:: pycon
  192. >>> pub_id = "-//W3C//DTD XHTML 1.0 Transitional//EN"
  193. >>> sys_url = "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"
  194. >>> doctype_string = '<!DOCTYPE html PUBLIC "%s" "%s">' % (pub_id, sys_url)
  195. >>> xml_header = '<?xml version="1.0" encoding="ascii"?>'
  196. >>> xhtml = xml_header + doctype_string + '<html><body></body></html>'
  197. >>> tree = etree.parse(StringIO(xhtml))
  198. >>> docinfo = tree.docinfo
  199. >>> print(docinfo.public_id)
  200. -//W3C//DTD XHTML 1.0 Transitional//EN
  201. >>> print(docinfo.system_url)
  202. http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd
  203. >>> docinfo.doctype == doctype_string
  204. True
  205. >>> print(docinfo.xml_version)
  206. 1.0
  207. >>> print(docinfo.encoding)
  208. ascii
  209. The target parser interface
  210. ===========================
  211. .. _`As in ElementTree`: http://effbot.org/elementtree/elementtree-xmlparser.htm
  212. `As in ElementTree`_, and similar to a SAX event handler, you can pass
  213. a target object to the parser:
  214. .. sourcecode:: pycon
  215. >>> class EchoTarget:
  216. ... def start(self, tag, attrib):
  217. ... print("start %s %s" % (tag, attrib))
  218. ... def end(self, tag):
  219. ... print("end %s" % tag)
  220. ... def data(self, data):
  221. ... print("data %r" % data)
  222. ... def comment(self, text):
  223. ... print("comment %s" % text)
  224. ... def close(self):
  225. ... print("close")
  226. ... return "closed!"
  227. >>> parser = etree.XMLParser(target = EchoTarget())
  228. >>> result = etree.XML("<element>some<!--comment-->text</element>",
  229. ... parser)
  230. start element {}
  231. data u'some'
  232. comment comment
  233. data u'text'
  234. end element
  235. close
  236. >>> print(result)
  237. closed!
  238. It is important for the ``.close()`` method to reset the parser target
  239. to a usable state, so that you can reuse the parser as often as you
  240. like:
  241. .. sourcecode:: pycon
  242. >>> result = etree.XML("<element>some<!--comment-->text</element>",
  243. ... parser)
  244. start element {}
  245. data u'some'
  246. comment comment
  247. data u'text'
  248. end element
  249. close
  250. >>> print(result)
  251. closed!
  252. Note that the parser does *not* build a tree when using a parser
  253. target. The result of the parser run is whatever the target object
  254. returns from its ``.close()`` method. If you want to return an XML
  255. tree here, you have to create it programmatically in the target
  256. object. An example for a parser target that builds a tree is the
  257. ``TreeBuilder``.
  258. >>> parser = etree.XMLParser(target = etree.TreeBuilder())
  259. >>> result = etree.XML("<element>some<!--comment-->text</element>",
  260. ... parser)
  261. >>> print(result.tag)
  262. element
  263. >>> print(result[0].text)
  264. comment
  265. The feed parser interface
  266. =========================
  267. Since lxml 2.0, the parsers have a feed parser interface that is
  268. compatible to the `ElementTree parsers`_. You can use it to feed data
  269. into the parser in a controlled step-by-step way.
  270. In lxml.etree, you can use both interfaces to a parser at the same
  271. time: the ``parse()`` or ``XML()`` functions, and the feed parser
  272. interface. Both are independent and will not conflict (except if used
  273. in conjunction with a parser target object as described above).
  274. .. _`ElementTree parsers`: http://effbot.org/elementtree/elementtree-xmlparser.htm
  275. To start parsing with a feed parser, just call its ``feed()`` method
  276. to feed it some data.
  277. .. sourcecode:: pycon
  278. >>> parser = etree.XMLParser()
  279. >>> for data in ('<?xml versio', 'n="1.0"?', '><roo', 't><a', '/></root>'):
  280. ... parser.feed(data)
  281. When you are done parsing, you **must** call the ``close()`` method to
  282. retrieve the root Element of the parse result document, and to unlock the
  283. parser:
  284. .. sourcecode:: pycon
  285. >>> root = parser.close()
  286. >>> print(root.tag)
  287. root
  288. >>> print(root[0].tag)
  289. a
  290. If you do not call ``close()``, the parser will stay locked and
  291. subsequent feeds will keep appending data, usually resulting in a non
  292. well-formed document and an unexpected parser error. So make sure you
  293. always close the parser after use, also in the exception case.
  294. Another way of achieving the same step-by-step parsing is by writing your own
  295. file-like object that returns a chunk of data on each ``read()`` call. Where
  296. the feed parser interface allows you to actively pass data chunks into the
  297. parser, a file-like object passively responds to ``read()`` requests of the
  298. parser itself. Depending on the data source, either way may be more natural.
  299. Note that the feed parser has its own error log called
  300. ``feed_error_log``. Errors in the feed parser do not show up in the
  301. normal ``error_log`` and vice versa.
  302. You can also combine the feed parser interface with the target parser:
  303. .. sourcecode:: pycon
  304. >>> parser = etree.XMLParser(target = EchoTarget())
  305. >>> parser.feed("<eleme")
  306. >>> parser.feed("nt>some text</elem")
  307. start element {}
  308. data u'some text'
  309. >>> parser.feed("ent>")
  310. end element
  311. >>> result = parser.close()
  312. close
  313. >>> print(result)
  314. closed!
  315. Again, this prevents the automatic creation of an XML tree and leaves
  316. all the event handling to the target object. The ``close()`` method
  317. of the parser forwards the return value of the target's ``close()``
  318. method.
  319. iterparse and iterwalk
  320. ======================
  321. As known from ElementTree, the ``iterparse()`` utility function
  322. returns an iterator that generates parser events for an XML file (or
  323. file-like object), while building the tree. The values are tuples
  324. ``(event-type, object)``. The event types supported by ElementTree
  325. and lxml.etree are the strings 'start', 'end', 'start-ns' and
  326. 'end-ns'.
  327. The 'start' and 'end' events represent opening and closing elements.
  328. They are accompanied by the respective Element instance. By default,
  329. only 'end' events are generated:
  330. .. sourcecode:: pycon
  331. >>> xml = '''\
  332. ... <root>
  333. ... <element key='value'>text</element>
  334. ... <element>text</element>tail
  335. ... <empty-element xmlns="http://testns/" />
  336. ... </root>
  337. ... '''
  338. >>> context = etree.iterparse(StringIO(xml))
  339. >>> for action, elem in context:
  340. ... print("%s: %s" % (action, elem.tag))
  341. end: element
  342. end: element
  343. end: {http://testns/}empty-element
  344. end: root
  345. The resulting tree is available through the ``root`` property of the iterator:
  346. .. sourcecode:: pycon
  347. >>> context.root.tag
  348. 'root'
  349. The other event types can be activated with the ``events`` keyword argument:
  350. .. sourcecode:: pycon
  351. >>> events = ("start", "end")
  352. >>> context = etree.iterparse(StringIO(xml), events=events)
  353. >>> for action, elem in context:
  354. ... print("%s: %s" % (action, elem.tag))
  355. start: root
  356. start: element
  357. end: element
  358. start: element
  359. end: element
  360. start: {http://testns/}empty-element
  361. end: {http://testns/}empty-element
  362. end: root
  363. The 'start-ns' and 'end-ns' events notify about namespace
  364. declarations. They do not come with Elements. Instead, the value of
  365. the 'start-ns' event is a tuple ``(prefix, namespaceURI)`` that
  366. designates the beginning of a prefix-namespace mapping. The
  367. corresponding ``end-ns`` event does not have a value (None). It is
  368. common practice to use a list as namespace stack and pop the last
  369. entry on the 'end-ns' event.
  370. .. sourcecode:: pycon
  371. >>> print(xml[:-1])
  372. <root>
  373. <element key='value'>text</element>
  374. <element>text</element>tail
  375. <empty-element xmlns="http://testns/" />
  376. </root>
  377. >>> events = ("start", "end", "start-ns", "end-ns")
  378. >>> context = etree.iterparse(StringIO(xml), events=events)
  379. >>> for action, elem in context:
  380. ... if action in ('start', 'end'):
  381. ... print("%s: %s" % (action, elem.tag))
  382. ... elif action == 'start-ns':
  383. ... print("%s: %s" % (action, elem))
  384. ... else:
  385. ... print(action)
  386. start: root
  387. start: element
  388. end: element
  389. start: element
  390. end: element
  391. start-ns: ('', 'http://testns/')
  392. start: {http://testns/}empty-element
  393. end: {http://testns/}empty-element
  394. end-ns
  395. end: root
  396. Selective tag events
  397. --------------------
  398. As an extension over ElementTree, lxml.etree accepts a ``tag`` keyword
  399. argument just like ``element.iter(tag)``. This restricts events to a
  400. specific tag or namespace:
  401. .. sourcecode:: pycon
  402. >>> context = etree.iterparse(StringIO(xml), tag="element")
  403. >>> for action, elem in context:
  404. ... print("%s: %s" % (action, elem.tag))
  405. end: element
  406. end: element
  407. >>> events = ("start", "end")
  408. >>> context = etree.iterparse(
  409. ... StringIO(xml), events=events, tag="{http://testns/}*")
  410. >>> for action, elem in context:
  411. ... print("%s: %s" % (action, elem.tag))
  412. start: {http://testns/}empty-element
  413. end: {http://testns/}empty-element
  414. Comments and PIs
  415. ----------------
  416. As an extension over ElementTree, the ``iterparse()`` function in
  417. lxml.etree also supports the event types 'comment' and 'pi' for the
  418. respective XML structures.
  419. .. sourcecode:: pycon
  420. >>> commented_xml = '''\
  421. ... <?some pi ?>
  422. ... <!-- a comment -->
  423. ... <root>
  424. ... <element key='value'>text</element>
  425. ... <!-- another comment -->
  426. ... <element>text</element>tail
  427. ... <empty-element xmlns="http://testns/" />
  428. ... </root>
  429. ... '''
  430. >>> events = ("start", "end", "comment", "pi")
  431. >>> context = etree.iterparse(StringIO(commented_xml), events=events)
  432. >>> for action, elem in context:
  433. ... if action in ('start', 'end'):
  434. ... print("%s: %s" % (action, elem.tag))
  435. ... elif action == 'pi':
  436. ... print("%s: -%s=%s-" % (action, elem.target, elem.text))
  437. ... else: # 'comment'
  438. ... print("%s: -%s-" % (action, elem.text))
  439. pi: -some=pi -
  440. comment: - a comment -
  441. start: root
  442. start: element
  443. end: element
  444. comment: - another comment -
  445. start: element
  446. end: element
  447. start: {http://testns/}empty-element
  448. end: {http://testns/}empty-element
  449. end: root
  450. >>> print(context.root.tag)
  451. root
  452. Modifying the tree
  453. ------------------
  454. You can modify the element and its descendants when handling the 'end' event.
  455. To save memory, for example, you can remove subtrees that are no longer
  456. needed:
  457. .. sourcecode:: pycon
  458. >>> context = etree.iterparse(StringIO(xml))
  459. >>> for action, elem in context:
  460. ... print(len(elem))
  461. ... elem.clear()
  462. 0
  463. 0
  464. 0
  465. 3
  466. >>> context.root.getchildren()
  467. []
  468. **WARNING**: During the 'start' event, the descendants and following siblings
  469. are not yet available and should not be accessed. During the 'end' event, the
  470. element and its descendants can be freely modified, but its following siblings
  471. should not be accessed. During either of the two events, you **must not**
  472. modify or move the ancestors (parents) of the current element. You should
  473. also avoid moving or discarding the element itself. The golden rule is: do
  474. not touch anything that will have to be touched again by the parser later on.
  475. If you have elements with a long list of children in your XML file and want to
  476. save more memory during parsing, you can clean up the preceding siblings of
  477. the current element:
  478. .. sourcecode:: pycon
  479. >>> for event, element in etree.iterparse(StringIO(xml)):
  480. ... # ... do something with the element
  481. ... element.clear() # clean up children
  482. ... while element.getprevious() is not None:
  483. ... del element.getparent()[0] # clean up preceding siblings
  484. The ``while`` loop deletes multiple siblings in a row. This is only necessary
  485. if you skipped over some of them using the ``tag`` keyword argument.
  486. Otherwise, a simple ``if`` should do. The more selective your tag is,
  487. however, the more thought you will have to put into finding the right way to
  488. clean up the elements that were skipped. Therefore, it is sometimes easier to
  489. traverse all elements and do the tag selection by hand in the event handler
  490. code.
  491. iterwalk
  492. --------
  493. A second extension over ElementTree is the ``iterwalk()`` function. It
  494. behaves exactly like ``iterparse()``, but works on Elements and ElementTrees:
  495. .. sourcecode:: pycon
  496. >>> root = etree.XML(xml)
  497. >>> context = etree.iterwalk(
  498. ... root, events=("start", "end"), tag="element")
  499. >>> for action, elem in context:
  500. ... print("%s: %s" % (action, elem.tag))
  501. start: element
  502. end: element
  503. start: element
  504. end: element
  505. >>> f = StringIO(xml)
  506. >>> context = etree.iterparse(
  507. ... f, events=("start", "end"), tag="element")
  508. >>> for action, elem in context:
  509. ... print("%s: %s" % (action, elem.tag))
  510. start: element
  511. end: element
  512. start: element
  513. end: element
  514. Python unicode strings
  515. ======================
  516. lxml.etree has broader support for Python unicode strings than the ElementTree
  517. library. First of all, where ElementTree would raise an exception, the
  518. parsers in lxml.etree can handle unicode strings straight away. This is most
  519. helpful for XML snippets embedded in source code using the ``XML()``
  520. function:
  521. .. sourcecode:: pycon
  522. >>> uxml = u'<test> \uf8d1 + \uf8d2 </test>'
  523. >>> uxml
  524. u'<test> \uf8d1 + \uf8d2 </test>'
  525. >>> root = etree.XML(uxml)
  526. This requires, however, that unicode strings do not specify a conflicting
  527. encoding themselves and thus lie about their real encoding:
  528. .. sourcecode:: pycon
  529. >>> etree.XML( u'<?xml version="1.0" encoding="ASCII"?>\n' + uxml )
  530. Traceback (most recent call last):
  531. ...
  532. ValueError: Unicode strings with encoding declaration are not supported.
  533. Similarly, you will get errors when you try the same with HTML data in a
  534. unicode string that specifies a charset in a meta tag of the header. You
  535. should generally avoid converting XML/HTML data to unicode before passing it
  536. into the parsers. It is both slower and error prone.
  537. Serialising to Unicode strings
  538. ------------------------------
  539. To serialize the result, you would normally use the ``tostring()``
  540. module function, which serializes to plain ASCII by default or a
  541. number of other byte encodings if asked for:
  542. .. sourcecode:: pycon
  543. >>> etree.tostring(root)
  544. b'<test> &#63697; + &#63698; </test>'
  545. >>> etree.tostring(root, encoding='UTF-8', xml_declaration=False)
  546. b'<test> \xef\xa3\x91 + \xef\xa3\x92 </test>'
  547. As an extension, lxml.etree recognises the unicode type as an argument to the
  548. encoding parameter to build a Python unicode representation of a tree:
  549. .. sourcecode:: pycon
  550. >>> etree.tostring(root, encoding=unicode)
  551. u'<test> \uf8d1 + \uf8d2 </test>'
  552. >>> el = etree.Element("test")
  553. >>> etree.tostring(el, encoding=unicode)
  554. u'<test/>'
  555. >>> subel = etree.SubElement(el, "subtest")
  556. >>> etree.tostring(el, encoding=unicode)
  557. u'<test><subtest/></test>'
  558. >>> tree = etree.ElementTree(el)
  559. >>> etree.tostring(tree, encoding=unicode)
  560. u'<test><subtest/></test>'
  561. The result of ``tostring(encoding=unicode)`` can be treated like any
  562. other Python unicode string and then passed back into the parsers.
  563. However, if you want to save the result to a file or pass it over the
  564. network, you should use ``write()`` or ``tostring()`` with a byte
  565. encoding (typically UTF-8) to serialize the XML. The main reason is
  566. that unicode strings returned by ``tostring(encoding=unicode)`` are
  567. not byte streams and they never have an XML declaration to specify
  568. their encoding. These strings are most likely not parsable by other
  569. XML libraries.
  570. For normal byte encodings, the ``tostring()`` function automatically
  571. adds a declaration as needed that reflects the encoding of the
  572. returned string. This makes it possible for other parsers to
  573. correctly parse the XML byte stream. Note that using ``tostring()``
  574. with UTF-8 is also considerably faster in most cases.