unicode.html 31 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474
  1. <html xmlns="http://www.w3.org/1999/xhtml">
  2. <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
  3. <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
  4. <head>
  5. <title>
  6. The Unicode Chapter
  7. &mdash;
  8. Mako 0.7.2 Documentation
  9. </title>
  10. <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
  11. <link rel="stylesheet" href="_static/docs.css" type="text/css" />
  12. <script type="text/javascript">
  13. var DOCUMENTATION_OPTIONS = {
  14. URL_ROOT: '#',
  15. VERSION: '0.7.2',
  16. COLLAPSE_MODINDEX: false,
  17. FILE_SUFFIX: '.html'
  18. };
  19. </script>
  20. <script type="text/javascript" src="_static/jquery.js"></script>
  21. <script type="text/javascript" src="_static/underscore.js"></script>
  22. <script type="text/javascript" src="_static/doctools.js"></script>
  23. <link rel="index" title="Index" href="genindex.html" />
  24. <link rel="search" title="Search" href="search.html" />
  25. <link rel="top" title="Mako 0.7.2 Documentation" href="index.html" />
  26. <link rel="next" title="Caching" href="caching.html" />
  27. <link rel="prev" title="Filtering and Buffering" href="filtering.html" />
  28. <link rel="stylesheet" href="_static/site.css"></link>
  29. </head>
  30. <body>
  31. <div id="wrap">
  32. <div class="rightbar">
  33. <div class="slogan">
  34. Hyperfast and lightweight templating for the Python platform.
  35. </div>
  36. </div>
  37. <a href="http://www.makotemplates.org/"><img src="_static/makoLogo.png" /></a>
  38. <hr/>
  39. <div id="docs-container">
  40. <div id="docs-header">
  41. <h1>Mako 0.7.2 Documentation</h1>
  42. <div id="docs-search">
  43. Search:
  44. <form class="search" action="search.html" method="get">
  45. <input type="text" name="q" size="18" /> <input type="submit" value="Search" />
  46. <input type="hidden" name="check_keywords" value="yes" />
  47. <input type="hidden" name="area" value="default" />
  48. </form>
  49. </div>
  50. <div id="docs-version-header">
  51. Release: <span class="version-num">0.7.2</span>
  52. </div>
  53. </div>
  54. <div id="docs-top-navigation">
  55. <div id="docs-top-page-control" class="docs-navigation-links">
  56. <ul>
  57. <li>Prev:
  58. <a href="filtering.html" title="previous chapter">Filtering and Buffering</a>
  59. </li>
  60. <li>Next:
  61. <a href="caching.html" title="next chapter">Caching</a>
  62. </li>
  63. <li>
  64. <a href="index.html">Table of Contents</a> |
  65. <a href="genindex.html">Index</a>
  66. | <a href="_sources/unicode.txt">view source
  67. </li>
  68. </ul>
  69. </div>
  70. <div id="docs-navigation-banner">
  71. <a href="index.html">Mako 0.7.2 Documentation</a>
  72. »
  73. The Unicode Chapter
  74. <h2>
  75. The Unicode Chapter
  76. </h2>
  77. </div>
  78. </div>
  79. <div id="docs-body-container">
  80. <div id="docs-sidebar">
  81. <h3><a href="index.html">Table of Contents</a></h3>
  82. <ul>
  83. <li><a class="reference internal" href="#">The Unicode Chapter</a><ul>
  84. <li><a class="reference internal" href="#specifying-the-encoding-of-a-template-file">Specifying the Encoding of a Template File</a></li>
  85. <li><a class="reference internal" href="#handling-expressions">Handling Expressions</a></li>
  86. <li><a class="reference internal" href="#defining-output-encoding">Defining Output Encoding</a><ul>
  87. <li><a class="reference internal" href="#buffer-selection">Buffer Selection</a></li>
  88. </ul>
  89. </li>
  90. <li><a class="reference internal" href="#saying-to-heck-with-it-disabling-the-usage-of-unicode-entirely">Saying to Heck with It: Disabling the Usage of Unicode Entirely</a><ul>
  91. <li><a class="reference internal" href="#rules-for-using-disable-unicode-true">Rules for using <tt class="docutils literal"><span class="pre">disable_unicode=True</span></tt></a></li>
  92. </ul>
  93. </li>
  94. </ul>
  95. </li>
  96. </ul>
  97. <h4>Previous Topic</h4>
  98. <p>
  99. <a href="filtering.html" title="previous chapter">Filtering and Buffering</a>
  100. </p>
  101. <h4>Next Topic</h4>
  102. <p>
  103. <a href="caching.html" title="next chapter">Caching</a>
  104. </p>
  105. <h4>Quick Search</h4>
  106. <p>
  107. <form class="search" action="search.html" method="get">
  108. <input type="text" name="q" size="18" /> <input type="submit" value="Search" />
  109. <input type="hidden" name="check_keywords" value="yes" />
  110. <input type="hidden" name="area" value="default" />
  111. </form>
  112. </p>
  113. </div>
  114. <div id="docs-body" class="withsidebar" >
  115. <div class="section" id="the-unicode-chapter">
  116. <span id="unicode-toplevel"></span><h1>The Unicode Chapter<a class="headerlink" href="#the-unicode-chapter" title="Permalink to this headline">¶</a></h1>
  117. <p>The Python language supports two ways of representing what we
  118. know as &#8220;strings&#8221;, i.e. series of characters. In Python 2, the
  119. two types are <tt class="docutils literal"><span class="pre">string</span></tt> and <tt class="docutils literal"><span class="pre">unicode</span></tt>, and in Python 3 they are
  120. <tt class="docutils literal"><span class="pre">bytes</span></tt> and <tt class="docutils literal"><span class="pre">string</span></tt>. A key aspect of the Python 2 <tt class="docutils literal"><span class="pre">string</span></tt> and
  121. Python 3 <tt class="docutils literal"><span class="pre">bytes</span></tt> types are that they contain no information
  122. regarding what <strong>encoding</strong> the data is stored in. For this
  123. reason they were commonly referred to as <strong>byte strings</strong> on
  124. Python 2, and Python 3 makes this name more explicit. The
  125. origins of this come from Python&#8217;s background of being developed
  126. before the Unicode standard was even available, back when
  127. strings were C-style strings and were just that, a series of
  128. bytes. Strings that had only values below 128 just happened to
  129. be <strong>ASCII</strong> strings and were printable on the console, whereas
  130. strings with values above 128 would produce all kinds of
  131. graphical characters and bells.</p>
  132. <p>Contrast the &#8220;byte-string&#8221; type with the &#8220;unicode/string&#8221; type.
  133. Objects of this latter type are created whenever you say something like
  134. <tt class="docutils literal"><span class="pre">u&quot;hello</span> <span class="pre">world&quot;</span></tt> (or in Python 3, just <tt class="docutils literal"><span class="pre">&quot;hello</span> <span class="pre">world&quot;</span></tt>). In this
  135. case, Python represents each character in the string internally
  136. using multiple bytes per character (something similar to
  137. UTF-16). What&#8217;s important is that when using the
  138. <tt class="docutils literal"><span class="pre">unicode</span></tt>/<tt class="docutils literal"><span class="pre">string</span></tt> type to store strings, Python knows the
  139. data&#8217;s encoding; it&#8217;s in its own internal format. Whereas when
  140. using the <tt class="docutils literal"><span class="pre">string</span></tt>/<tt class="docutils literal"><span class="pre">bytes</span></tt> type, it does not.</p>
  141. <p>When Python 2 attempts to treat a byte-string as a string, which
  142. means it&#8217;s attempting to compare/parse its characters, to coerce
  143. it into another encoding, or to decode it to a unicode object,
  144. it has to guess what the encoding is. In this case, it will
  145. pretty much always guess the encoding as <tt class="docutils literal"><span class="pre">ascii</span></tt>... and if the
  146. byte-string contains bytes above value 128, you&#8217;ll get an error.
  147. Python 3 eliminates much of this confusion by just raising an
  148. error unconditionally if a byte-string is used in a
  149. character-aware context.</p>
  150. <p>There is one operation that Python <em>can</em> do with a non-ASCII
  151. byte-string, and it&#8217;s a great source of confusion: it can dump the
  152. byte-string straight out to a stream or a file, with nary a care
  153. what the encoding is. To Python, this is pretty much like
  154. dumping any other kind of binary data (like an image) to a
  155. stream somewhere. In Python 2, it is common to see programs that
  156. embed all kinds of international characters and encodings into
  157. plain byte-strings (i.e. using <tt class="docutils literal"><span class="pre">&quot;hello</span> <span class="pre">world&quot;</span></tt> style literals)
  158. can fly right through their run, sending reams of strings out to
  159. wherever they are going, and the programmer, seeing the same
  160. output as was expressed in the input, is now under the illusion
  161. that his or her program is Unicode-compliant. In fact, their
  162. program has no unicode awareness whatsoever, and similarly has
  163. no ability to interact with libraries that <em>are</em> unicode aware.
  164. Python 3 makes this much less likely by defaulting to unicode as
  165. the storage format for strings.</p>
  166. <p>The &#8220;pass through encoded data&#8221; scheme is what template
  167. languages like Cheetah and earlier versions of Myghty do by
  168. default. Mako as of version 0.2 also supports this mode of
  169. operation when using Python 2, using the <tt class="docutils literal"><span class="pre">disable_unicode=True</span></tt>
  170. flag. However, when using Mako in its default mode of
  171. unicode-aware, it requires explicitness when dealing with
  172. non-ASCII encodings. Additionally, if you ever need to handle
  173. unicode strings and other kinds of encoding conversions more
  174. intelligently, the usage of raw byte-strings quickly becomes a
  175. nightmare, since you are sending the Python interpreter
  176. collections of bytes for which it can make no intelligent
  177. decisions with regards to encoding. In Python 3 Mako only allows
  178. usage of native, unicode strings.</p>
  179. <p>In normal Mako operation, all parsed template constructs and
  180. output streams are handled internally as Python <tt class="docutils literal"><span class="pre">unicode</span></tt>
  181. objects. It&#8217;s only at the point of <a class="reference internal" href="usage.html#mako.template.Template.render" title="mako.template.Template.render"><tt class="xref py py-meth docutils literal"><span class="pre">render()</span></tt></a> that this unicode
  182. stream may be rendered into whatever the desired output encoding
  183. is. The implication here is that the template developer must
  184. :ensure that <a class="reference internal" href="#set-template-file-encoding"><em>the encoding of all non-ASCII templates is explicit</em></a> (still required in Python 3),
  185. that <a class="reference internal" href="#handling-non-ascii-expressions"><em>all non-ASCII-encoded expressions are in one way or another
  186. converted to unicode</em></a>
  187. (not much of a burden in Python 3), and that <a class="reference internal" href="#defining-output-encoding"><em>the output stream of the
  188. template is handled as a unicode stream being encoded to some
  189. encoding</em></a> (still required in Python 3).</p>
  190. <div class="section" id="specifying-the-encoding-of-a-template-file">
  191. <span id="set-template-file-encoding"></span><h2>Specifying the Encoding of a Template File<a class="headerlink" href="#specifying-the-encoding-of-a-template-file" title="Permalink to this headline">¶</a></h2>
  192. <p>This is the most basic encoding-related setting, and it is
  193. equivalent to Python&#8217;s &#8220;magic encoding comment&#8221;, as described in
  194. <a class="reference external" href="http://www.python.org/dev/peps/pep-0263/">pep-0263</a>. Any
  195. template that contains non-ASCII characters requires that this
  196. comment be present so that Mako can decode to unicode (and also
  197. make usage of Python&#8217;s AST parsing services). Mako&#8217;s lexer will
  198. use this encoding in order to convert the template source into a
  199. <tt class="docutils literal"><span class="pre">unicode</span></tt> object before continuing its parsing:</p>
  200. <div class="highlight-mako"><div class="highlight"><pre><span class="cp">## -*- coding: utf-8 -*-</span><span class="x"></span>
  201. <span class="x">Alors vous imaginez ma surprise, au lever du jour, quand</span>
  202. <span class="x">une drôle de petite voix m’a réveillé. Elle disait:</span>
  203. <span class="x"> « S’il vous plaît… dessine-moi un mouton! »</span>
  204. </pre></div>
  205. </div>
  206. <p>For the picky, the regular expression used is derived from that
  207. of the above mentioned pep:</p>
  208. <div class="highlight-python"><div class="highlight"><pre><span class="c">#.*coding[:=]\s*([-\w.]+).*\n</span>
  209. </pre></div>
  210. </div>
  211. <p>The lexer will convert to unicode in all cases, so that if any
  212. characters exist in the template that are outside of the
  213. specified encoding (or the default of <tt class="docutils literal"><span class="pre">ascii</span></tt>), the error will
  214. be immediate.</p>
  215. <p>As an alternative, the template encoding can be specified
  216. programmatically to either <a class="reference internal" href="usage.html#mako.template.Template" title="mako.template.Template"><tt class="xref py py-class docutils literal"><span class="pre">Template</span></tt></a> or <a class="reference internal" href="usage.html#mako.lookup.TemplateLookup" title="mako.lookup.TemplateLookup"><tt class="xref py py-class docutils literal"><span class="pre">TemplateLookup</span></tt></a> via
  217. the <tt class="docutils literal"><span class="pre">input_encoding</span></tt> parameter:</p>
  218. <div class="highlight-python"><div class="highlight"><pre><span class="n">t</span> <span class="o">=</span> <span class="n">TemplateLookup</span><span class="p">(</span><span class="n">directories</span><span class="o">=</span><span class="p">[</span><span class="s">&#39;./&#39;</span><span class="p">],</span> <span class="n">input_encoding</span><span class="o">=</span><span class="s">&#39;utf-8&#39;</span><span class="p">)</span>
  219. </pre></div>
  220. </div>
  221. <p>The above will assume all located templates specify <tt class="docutils literal"><span class="pre">utf-8</span></tt>
  222. encoding, unless the template itself contains its own magic
  223. encoding comment, which takes precedence.</p>
  224. </div>
  225. <div class="section" id="handling-expressions">
  226. <span id="handling-non-ascii-expressions"></span><h2>Handling Expressions<a class="headerlink" href="#handling-expressions" title="Permalink to this headline">¶</a></h2>
  227. <p>The next area that encoding comes into play is in expression
  228. constructs. By default, Mako&#8217;s treatment of an expression like
  229. this:</p>
  230. <div class="highlight-mako"><div class="highlight"><pre><span class="cp">${</span><span class="s">&quot;hello world&quot;</span><span class="cp">}</span><span class="x"></span>
  231. </pre></div>
  232. </div>
  233. <p>looks something like this:</p>
  234. <div class="highlight-python"><div class="highlight"><pre><span class="n">context</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="nb">unicode</span><span class="p">(</span><span class="s">&quot;hello world&quot;</span><span class="p">))</span>
  235. </pre></div>
  236. </div>
  237. <p>In Python 3, it&#8217;s just:</p>
  238. <div class="highlight-python"><div class="highlight"><pre><span class="n">context</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="s">&quot;hello world&quot;</span><span class="p">))</span>
  239. </pre></div>
  240. </div>
  241. <p>That is, <strong>the output of all expressions is run through the
  242. ``unicode`` built-in</strong>. This is the default setting, and can be
  243. modified to expect various encodings. The <tt class="docutils literal"><span class="pre">unicode</span></tt> step serves
  244. both the purpose of rendering non-string expressions into
  245. strings (such as integers or objects which contain <tt class="docutils literal"><span class="pre">__str()__</span></tt>
  246. methods), and to ensure that the final output stream is
  247. constructed as a unicode object. The main implication of this is
  248. that <strong>any raw byte-strings that contain an encoding other than
  249. ASCII must first be decoded to a Python unicode object</strong>. It
  250. means you can&#8217;t say this in Python 2:</p>
  251. <div class="highlight-mako"><div class="highlight"><pre><span class="cp">${</span><span class="s">&quot;voix m’a réveillé.&quot;</span><span class="cp">}</span> <span class="cp">## error in Python 2!</span><span class="x"></span>
  252. </pre></div>
  253. </div>
  254. <p>You must instead say this:</p>
  255. <div class="highlight-mako"><div class="highlight"><pre><span class="cp">${</span><span class="s">u&quot;voix m’a réveillé.&quot;</span><span class="cp">}</span> <span class="cp">## OK !</span><span class="x"></span>
  256. </pre></div>
  257. </div>
  258. <p>Similarly, if you are reading data from a file that is streaming
  259. bytes, or returning data from some object that is returning a
  260. Python byte-string containing a non-ASCII encoding, you have to
  261. explicitly decode to unicode first, such as:</p>
  262. <div class="highlight-mako"><div class="highlight"><pre><span class="cp">${</span><span class="n">call_my_object</span><span class="p">()</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s">&#39;utf-8&#39;</span><span class="p">)</span><span class="cp">}</span><span class="x"></span>
  263. </pre></div>
  264. </div>
  265. <p>Note that filehandles acquired by <tt class="docutils literal"><span class="pre">open()</span></tt> in Python 3 default
  266. to returning &#8220;text&#8221;, that is the decoding is done for you. See
  267. Python 3&#8217;s documentation for the <tt class="docutils literal"><span class="pre">open()</span></tt> built-in for details on
  268. this.</p>
  269. <p>If you want a certain encoding applied to <em>all</em> expressions,
  270. override the <tt class="docutils literal"><span class="pre">unicode</span></tt> builtin with the <tt class="docutils literal"><span class="pre">decode</span></tt> built-in at the
  271. <a class="reference internal" href="usage.html#mako.template.Template" title="mako.template.Template"><tt class="xref py py-class docutils literal"><span class="pre">Template</span></tt></a> or <a class="reference internal" href="usage.html#mako.lookup.TemplateLookup" title="mako.lookup.TemplateLookup"><tt class="xref py py-class docutils literal"><span class="pre">TemplateLookup</span></tt></a> level:</p>
  272. <div class="highlight-python"><div class="highlight"><pre><span class="n">t</span> <span class="o">=</span> <span class="n">Template</span><span class="p">(</span><span class="n">templatetext</span><span class="p">,</span> <span class="n">default_filters</span><span class="o">=</span><span class="p">[</span><span class="s">&#39;decode.utf8&#39;</span><span class="p">])</span>
  273. </pre></div>
  274. </div>
  275. <p>Note that the built-in <tt class="docutils literal"><span class="pre">decode</span></tt> object is slower than the
  276. <tt class="docutils literal"><span class="pre">unicode</span></tt> function, since unlike <tt class="docutils literal"><span class="pre">unicode</span></tt> it&#8217;s not a Python
  277. built-in, and it also checks the type of the incoming data to
  278. determine if string conversion is needed first.</p>
  279. <p>The <tt class="docutils literal"><span class="pre">default_filters</span></tt> argument can be used to entirely customize
  280. the filtering process of expressions. This argument is described
  281. in <a class="reference internal" href="filtering.html#filtering-default-filters"><em>The default_filters Argument</em></a>.</p>
  282. </div>
  283. <div class="section" id="defining-output-encoding">
  284. <span id="id1"></span><h2>Defining Output Encoding<a class="headerlink" href="#defining-output-encoding" title="Permalink to this headline">¶</a></h2>
  285. <p>Now that we have a template which produces a pure unicode output
  286. stream, all the hard work is done. We can take the output and do
  287. anything with it.</p>
  288. <p>As stated in the <a class="reference internal" href="usage.html"><em>&#8220;Usage&#8221; chapter</em></a>, both <a class="reference internal" href="usage.html#mako.template.Template" title="mako.template.Template"><tt class="xref py py-class docutils literal"><span class="pre">Template</span></tt></a> and
  289. <a class="reference internal" href="usage.html#mako.lookup.TemplateLookup" title="mako.lookup.TemplateLookup"><tt class="xref py py-class docutils literal"><span class="pre">TemplateLookup</span></tt></a> accept <tt class="docutils literal"><span class="pre">output_encoding</span></tt> and <tt class="docutils literal"><span class="pre">encoding_errors</span></tt>
  290. parameters which can be used to encode the output in any Python
  291. supported codec:</p>
  292. <div class="highlight-python"><div class="highlight"><pre><span class="kn">from</span> <span class="nn">mako.template</span> <span class="kn">import</span> <span class="n">Template</span>
  293. <span class="kn">from</span> <span class="nn">mako.lookup</span> <span class="kn">import</span> <span class="n">TemplateLookup</span>
  294. <span class="n">mylookup</span> <span class="o">=</span> <span class="n">TemplateLookup</span><span class="p">(</span><span class="n">directories</span><span class="o">=</span><span class="p">[</span><span class="s">&#39;/docs&#39;</span><span class="p">],</span> <span class="n">output_encoding</span><span class="o">=</span><span class="s">&#39;utf-8&#39;</span><span class="p">,</span> <span class="n">encoding_errors</span><span class="o">=</span><span class="s">&#39;replace&#39;</span><span class="p">)</span>
  295. <span class="n">mytemplate</span> <span class="o">=</span> <span class="n">mylookup</span><span class="o">.</span><span class="n">get_template</span><span class="p">(</span><span class="s">&quot;foo.txt&quot;</span><span class="p">)</span>
  296. <span class="k">print</span> <span class="n">mytemplate</span><span class="o">.</span><span class="n">render</span><span class="p">()</span>
  297. </pre></div>
  298. </div>
  299. <p><a class="reference internal" href="usage.html#mako.template.Template.render" title="mako.template.Template.render"><tt class="xref py py-meth docutils literal"><span class="pre">render()</span></tt></a> will return a <tt class="docutils literal"><span class="pre">bytes</span></tt> object in Python 3 if an output
  300. encoding is specified. By default it performs no encoding and
  301. returns a native string.</p>
  302. <p><a class="reference internal" href="usage.html#mako.template.Template.render_unicode" title="mako.template.Template.render_unicode"><tt class="xref py py-meth docutils literal"><span class="pre">render_unicode()</span></tt></a> will return the template output as a Python
  303. <tt class="docutils literal"><span class="pre">unicode</span></tt> object (or <tt class="docutils literal"><span class="pre">string</span></tt> in Python 3):</p>
  304. <div class="highlight-python"><div class="highlight"><pre><span class="k">print</span> <span class="n">mytemplate</span><span class="o">.</span><span class="n">render_unicode</span><span class="p">()</span>
  305. </pre></div>
  306. </div>
  307. <p>The above method disgards the output encoding keyword argument;
  308. you can encode yourself by saying:</p>
  309. <div class="highlight-python"><div class="highlight"><pre><span class="k">print</span> <span class="n">mytemplate</span><span class="o">.</span><span class="n">render_unicode</span><span class="p">()</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s">&#39;utf-8&#39;</span><span class="p">,</span> <span class="s">&#39;replace&#39;</span><span class="p">)</span>
  310. </pre></div>
  311. </div>
  312. <div class="section" id="buffer-selection">
  313. <h3>Buffer Selection<a class="headerlink" href="#buffer-selection" title="Permalink to this headline">¶</a></h3>
  314. <p>Mako does play some games with the style of buffering used
  315. internally, to maximize performance. Since the buffer is by far
  316. the most heavily used object in a render operation, it&#8217;s
  317. important!</p>
  318. <p>When calling <a class="reference internal" href="usage.html#mako.template.Template.render" title="mako.template.Template.render"><tt class="xref py py-meth docutils literal"><span class="pre">render()</span></tt></a> on a template that does not specify any
  319. output encoding (i.e. it&#8217;s <tt class="docutils literal"><span class="pre">ascii</span></tt>), Python&#8217;s <tt class="docutils literal"><span class="pre">cStringIO</span></tt> module,
  320. which cannot handle encoding of non-ASCII <tt class="docutils literal"><span class="pre">unicode</span></tt> objects
  321. (even though it can send raw byte-strings through), is used for
  322. buffering. Otherwise, a custom Mako class called
  323. <tt class="docutils literal"><span class="pre">FastEncodingBuffer</span></tt> is used, which essentially is a super
  324. dumbed-down version of <tt class="docutils literal"><span class="pre">StringIO</span></tt> that gathers all strings into
  325. a list and uses <tt class="docutils literal"><span class="pre">u''.join(elements)</span></tt> to produce the final output
  326. &#8211; it&#8217;s markedly faster than <tt class="docutils literal"><span class="pre">StringIO</span></tt>.</p>
  327. </div>
  328. </div>
  329. <div class="section" id="saying-to-heck-with-it-disabling-the-usage-of-unicode-entirely">
  330. <span id="unicode-disabled"></span><h2>Saying to Heck with It: Disabling the Usage of Unicode Entirely<a class="headerlink" href="#saying-to-heck-with-it-disabling-the-usage-of-unicode-entirely" title="Permalink to this headline">¶</a></h2>
  331. <p>Some segments of Mako&#8217;s userbase choose to make no usage of
  332. Unicode whatsoever, and instead would prefer the &#8220;pass through&#8221;
  333. approach; all string expressions in their templates return
  334. encoded byte-strings, and they would like these strings to pass
  335. right through. The only advantage to this approach is that
  336. templates need not use <tt class="docutils literal"><span class="pre">u&quot;&quot;</span></tt> for literal strings; there&#8217;s an
  337. arguable speed improvement as well since raw byte-strings
  338. generally perform slightly faster than unicode objects in
  339. Python. For these users, assuming they&#8217;re sticking with Python
  340. 2, they can hit the <tt class="docutils literal"><span class="pre">disable_unicode=True</span></tt> flag as so:</p>
  341. <div class="highlight-python"><div class="highlight"><pre><span class="c"># -*- encoding:utf-8 -*-</span>
  342. <span class="kn">from</span> <span class="nn">mako.template</span> <span class="kn">import</span> <span class="n">Template</span>
  343. <span class="n">t</span> <span class="o">=</span> <span class="n">Template</span><span class="p">(</span><span class="s">&quot;drôle de petite voix m’a réveillé.&quot;</span><span class="p">,</span> <span class="n">disable_unicode</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">input_encoding</span><span class="o">=</span><span class="s">&#39;utf-8&#39;</span><span class="p">)</span>
  344. <span class="k">print</span> <span class="n">t</span><span class="o">.</span><span class="n">code</span>
  345. </pre></div>
  346. </div>
  347. <p>The <tt class="docutils literal"><span class="pre">disable_unicode</span></tt> mode is strictly a Python 2 thing. It is
  348. not supported at all in Python 3.</p>
  349. <p>The generated module source code will contain elements like
  350. these:</p>
  351. <div class="highlight-python"><div class="highlight"><pre><span class="c"># -*- encoding:utf-8 -*-</span>
  352. <span class="c"># ...more generated code ...</span>
  353. <span class="k">def</span> <span class="nf">render_body</span><span class="p">(</span><span class="n">context</span><span class="p">,</span><span class="o">**</span><span class="n">pageargs</span><span class="p">):</span>
  354. <span class="n">context</span><span class="o">.</span><span class="n">caller_stack</span><span class="o">.</span><span class="n">push_frame</span><span class="p">()</span>
  355. <span class="k">try</span><span class="p">:</span>
  356. <span class="n">__M_locals</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">(</span><span class="n">pageargs</span><span class="o">=</span><span class="n">pageargs</span><span class="p">)</span>
  357. <span class="c"># SOURCE LINE 1</span>
  358. <span class="n">context</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s">&#39;dr</span><span class="se">\xc3\xb4</span><span class="s">le de petite voix m</span><span class="se">\xe2\x80\x99</span><span class="s">a r</span><span class="se">\xc3\xa9</span><span class="s">veill</span><span class="se">\xc3\xa9</span><span class="s">.&#39;</span><span class="p">)</span>
  359. <span class="k">return</span> <span class="s">&#39;&#39;</span>
  360. <span class="k">finally</span><span class="p">:</span>
  361. <span class="n">context</span><span class="o">.</span><span class="n">caller_stack</span><span class="o">.</span><span class="n">pop_frame</span><span class="p">()</span>
  362. </pre></div>
  363. </div>
  364. <p>Where above that the string literal used within <a class="reference internal" href="runtime.html#mako.runtime.Context.write" title="mako.runtime.Context.write"><tt class="xref py py-meth docutils literal"><span class="pre">Context.write()</span></tt></a>
  365. is a regular byte-string.</p>
  366. <p>When <tt class="docutils literal"><span class="pre">disable_unicode=True</span></tt> is turned on, the <tt class="docutils literal"><span class="pre">default_filters</span></tt>
  367. argument which normally defaults to <tt class="docutils literal"><span class="pre">[&quot;unicode&quot;]</span></tt> now defaults
  368. to <tt class="docutils literal"><span class="pre">[&quot;str&quot;]</span></tt> instead. Setting <tt class="docutils literal"><span class="pre">default_filters</span></tt> to the empty list
  369. <tt class="docutils literal"><span class="pre">[]</span></tt> can remove the overhead of the <tt class="docutils literal"><span class="pre">str</span></tt> call. Also, in this
  370. mode you <strong>cannot</strong> safely call <a class="reference internal" href="usage.html#mako.template.Template.render_unicode" title="mako.template.Template.render_unicode"><tt class="xref py py-meth docutils literal"><span class="pre">render_unicode()</span></tt></a> &#8211; you&#8217;ll get
  371. unicode/decode errors.</p>
  372. <p>The <tt class="docutils literal"><span class="pre">h</span></tt> filter (HTML escape) uses a less performant pure Python
  373. escape function in non-unicode mode. This because
  374. MarkupSafe only supports Python unicode objects for non-ASCII
  375. strings.</p>
  376. <p class="versionchanged">
  377. <span class="versionmodified">Changed in version 0.3.4: </span>In prior versions, it used <tt class="docutils literal"><span class="pre">cgi.escape()</span></tt>, which has been replaced
  378. with a function that also escapes single quotes.</p>
  379. <div class="section" id="rules-for-using-disable-unicode-true">
  380. <h3>Rules for using <tt class="docutils literal"><span class="pre">disable_unicode=True</span></tt><a class="headerlink" href="#rules-for-using-disable-unicode-true" title="Permalink to this headline">¶</a></h3>
  381. <ul class="simple">
  382. <li>Don&#8217;t use this mode unless you really, really want to and you
  383. absolutely understand what you&#8217;re doing.</li>
  384. <li>Don&#8217;t use this option just because you don&#8217;t want to learn to
  385. use Unicode properly; we aren&#8217;t supporting user issues in this
  386. mode of operation. We will however offer generous help for the
  387. vast majority of users who stick to the Unicode program.</li>
  388. <li>Python 3 is unicode by default, and the flag is not available
  389. when running on Python 3.</li>
  390. </ul>
  391. </div>
  392. </div>
  393. </div>
  394. </div>
  395. </div>
  396. <div id="docs-bottom-navigation" class="docs-navigation-links">
  397. Previous:
  398. <a href="filtering.html" title="previous chapter">Filtering and Buffering</a>
  399. Next:
  400. <a href="caching.html" title="next chapter">Caching</a>
  401. <div id="docs-copyright">
  402. &copy; Copyright the Mako authors and contributors.
  403. Documentation generated using <a href="http://sphinx.pocoo.org/">Sphinx</a> 1.1.3
  404. with Mako templates.
  405. </div>
  406. </div>
  407. </div>
  408. <div class="clearfix">
  409. <hr/>
  410. <div class="copyright">Website content copyright &copy; by Michael Bayer.
  411. All rights reserved. Mako and its documentation are licensed
  412. under the MIT license. mike(&)zzzcomputing.com</div>
  413. </div>
  414. </div>
  415. </body>
  416. </html>