messages.rst 11 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308
  1. .. -*- mode: rst; encoding: utf-8 -*-
  2. .. _messages:
  3. =============================
  4. Working with Message Catalogs
  5. =============================
  6. Introduction
  7. ============
  8. The ``gettext`` translation system enables you to mark any strings used in your
  9. application as subject to localization, by wrapping them in functions such as
  10. ``gettext(str)`` and ``ngettext(singular, plural, num)``. For brevity, the
  11. ``gettext`` function is often aliased to ``_(str)``, so you can write:
  12. .. code-block:: python
  13. print _("Hello")
  14. instead of just:
  15. .. code-block:: python
  16. print "Hello"
  17. to make the string "Hello" localizable.
  18. Message catalogs are collections of translations for such localizable messages
  19. used in an application. They are commonly stored in PO (Portable Object) and MO
  20. (Machine Object) files, the formats of which are defined by the GNU `gettext`_
  21. tools and the GNU `translation project`_.
  22. .. _`gettext`: http://www.gnu.org/software/gettext/
  23. .. _`translation project`: http://sourceforge.net/projects/translation
  24. The general procedure for building message catalogs looks something like this:
  25. * use a tool (such as ``xgettext``) to extract localizable strings from the
  26. code base and write them to a POT (PO Template) file.
  27. * make a copy of the POT file for a specific locale (for example, "en_US")
  28. and start translating the messages
  29. * use a tool such as ``msgfmt`` to compile the locale PO file into a binary
  30. MO file
  31. * later, when code changes make it necessary to update the translations, you
  32. regenerate the POT file and merge the changes into the various
  33. locale-specific PO files, for example using ``msgmerge``
  34. Python provides the :mod:`gettext` module as part of the standard library,
  35. which enables applications to work with appropriately generated MO files.
  36. As ``gettext`` provides a solid and well supported foundation for translating
  37. application messages, Babel does not reinvent the wheel, but rather reuses this
  38. infrastructure, and makes it easier to build message catalogs for Python
  39. applications.
  40. Message Extraction
  41. ==================
  42. Babel provides functionality similar to that of the ``xgettext`` program,
  43. except that only extraction from Python source files is built-in, while support
  44. for other file formats can be added using a simple extension mechanism.
  45. Unlike ``xgettext``, which is usually invoked once for every file, the routines
  46. for message extraction in Babel operate on directories. While the per-file
  47. approach of ``xgettext`` works nicely with projects using a ``Makefile``,
  48. Python projects rarely use ``make``, and thus a different mechanism is needed
  49. for extracting messages from the heterogeneous collection of source files that
  50. many Python projects are composed of.
  51. When message extraction is based on directories instead of individual files,
  52. there needs to be a way to configure which files should be treated in which
  53. manner. For example, while many projects may contain ``.html`` files, some of
  54. those files may be static HTML files that don't contain localizable message,
  55. while others may be `Jinja2`_ templates, and still others may contain `Genshi`_
  56. markup templates. Some projects may even mix HTML files for different templates
  57. languages (for whatever reason). Therefore the way in which messages are
  58. extracted from source files can not only depend on the file extension, but
  59. needs to be controllable in a precise manner.
  60. .. _`Jinja2`: http://jinja.pocoo.org/
  61. .. _`Genshi`: http://genshi.edgewall.org/
  62. Babel accepts a configuration file to specify this mapping of files to
  63. extraction methods, which is described below.
  64. .. _`frontends`:
  65. ----------
  66. Front-Ends
  67. ----------
  68. Babel provides two different front-ends to access its functionality for working
  69. with message catalogs:
  70. * A :ref:`cmdline`, and
  71. * :ref:`setup-integration`
  72. Which one you choose depends on the nature of your project. For most modern
  73. Python projects, the distutils/setuptools integration is probably more
  74. convenient.
  75. .. _`mapping`:
  76. -------------------------------------------
  77. Extraction Method Mapping and Configuration
  78. -------------------------------------------
  79. The mapping of extraction methods to files in Babel is done via a configuration
  80. file. This file maps extended glob patterns to the names of the extraction
  81. methods, and can also set various options for each pattern (which options are
  82. available depends on the specific extraction method).
  83. For example, the following configuration adds extraction of messages from both
  84. Genshi markup templates and text templates:
  85. .. code-block:: ini
  86. # Extraction from Python source files
  87. [python: **.py]
  88. # Extraction from Genshi HTML and text templates
  89. [genshi: **/templates/**.html]
  90. ignore_tags = script,style
  91. include_attrs = alt title summary
  92. [genshi: **/templates/**.txt]
  93. template_class = genshi.template:TextTemplate
  94. encoding = ISO-8819-15
  95. # Extraction from JavaScript files
  96. [javascript: **.js]
  97. extract_messages = $._, jQuery._
  98. The configuration file syntax is based on the format commonly found in ``.INI``
  99. files on Windows systems, and as supported by the ``ConfigParser`` module in
  100. the Python standard library. Section names (the strings enclosed in square
  101. brackets) specify both the name of the extraction method, and the extended glob
  102. pattern to specify the files that this extraction method should be used for,
  103. separated by a colon. The options in the sections are passed to the extraction
  104. method. Which options are available is specific to the extraction method used.
  105. The extended glob patterns used in this configuration are similar to the glob
  106. patterns provided by most shells. A single asterisk (``*``) is a wildcard for
  107. any number of characters (except for the pathname component separator "/"),
  108. while a question mark (``?``) only matches a single character. In addition,
  109. two subsequent asterisk characters (``**``) can be used to make the wildcard
  110. match any directory level, so the pattern ``**.txt`` matches any file with the
  111. extension ``.txt`` in any directory.
  112. Lines that start with a ``#`` or ``;`` character are ignored and can be used
  113. for comments. Empty lines are ignored, too.
  114. .. note:: if you're performing message extraction using the command Babel
  115. provides for integration into ``setup.py`` scripts, you can also
  116. provide this configuration in a different way, namely as a keyword
  117. argument to the ``setup()`` function. See
  118. :ref:`setup-integration` for more information.
  119. Default Extraction Methods
  120. --------------------------
  121. Babel comes with a few builtin extractors: ``python`` (which extracts
  122. messages from Python source files), ``javascript``, and ``ignore`` (which
  123. extracts nothing).
  124. The ``python`` extractor is by default mapped to the glob pattern ``**.py``,
  125. meaning it'll be applied to all files with the ``.py`` extension in any
  126. directory. If you specify your own mapping configuration, this default mapping
  127. is discarded, so you need to explicitly add it to your mapping (as shown in the
  128. example above.)
  129. .. _`referencing extraction methods`:
  130. Referencing Extraction Methods
  131. ------------------------------
  132. To be able to use short extraction method names such as “genshi”, you need to
  133. have `pkg_resources`_ installed, and the package implementing that extraction
  134. method needs to have been installed with its meta data (the `egg-info`_).
  135. If this is not possible for some reason, you need to map the short names to
  136. fully qualified function names in an extract section in the mapping
  137. configuration. For example:
  138. .. code-block:: ini
  139. # Some custom extraction method
  140. [extractors]
  141. custom = mypackage.module:extract_custom
  142. [custom: **.ctm]
  143. some_option = foo
  144. Note that the builtin extraction methods ``python`` and ``ignore`` are available
  145. by default, even if `pkg_resources`_ is not installed. You should never need to
  146. explicitly define them in the ``[extractors]`` section.
  147. .. _`egg-info`: http://peak.telecommunity.com/DevCenter/PythonEggs
  148. .. _`pkg_resources`: http://peak.telecommunity.com/DevCenter/PkgResources
  149. --------------------------
  150. Writing Extraction Methods
  151. --------------------------
  152. Adding new methods for extracting localizable methods is easy. First, you'll
  153. need to implement a function that complies with the following interface:
  154. .. code-block:: python
  155. def extract_xxx(fileobj, keywords, comment_tags, options):
  156. """Extract messages from XXX files.
  157. :param fileobj: the file-like object the messages should be extracted
  158. from
  159. :param keywords: a list of keywords (i.e. function names) that should
  160. be recognized as translation functions
  161. :param comment_tags: a list of translator tags to search for and
  162. include in the results
  163. :param options: a dictionary of additional options (optional)
  164. :return: an iterator over ``(lineno, funcname, message, comments)``
  165. tuples
  166. :rtype: ``iterator``
  167. """
  168. .. note:: Any strings in the tuples produced by this function must be either
  169. ``unicode`` objects, or ``str`` objects using plain ASCII characters.
  170. That means that if sources contain strings using other encodings, it
  171. is the job of the extractor implementation to do the decoding to
  172. ``unicode`` objects.
  173. Next, you should register that function as an entry point. This requires your
  174. ``setup.py`` script to use `setuptools`_, and your package to be installed with
  175. the necessary metadata. If that's taken care of, add something like the
  176. following to your ``setup.py`` script:
  177. .. code-block:: python
  178. def setup(...
  179. entry_points = """
  180. [babel.extractors]
  181. xxx = your.package:extract_xxx
  182. """,
  183. That is, add your extraction method to the entry point group
  184. ``babel.extractors``, where the name of the entry point is the name that people
  185. will use to reference the extraction method, and the value being the module and
  186. the name of the function (separated by a colon) implementing the actual
  187. extraction.
  188. .. note:: As shown in `Referencing Extraction Methods`_, declaring an entry
  189. point is not strictly required, as users can still reference the
  190. extraction function directly. But whenever possible, the entry point
  191. should be declared to make configuration more convenient.
  192. .. _`setuptools`: http://peak.telecommunity.com/DevCenter/setuptools
  193. -------------------
  194. Translator Comments
  195. -------------------
  196. First of all what are comments tags. Comments tags are excerpts of text to
  197. search for in comments, only comments, right before the python :mod:`gettext`
  198. calls, as shown on the following example:
  199. .. code-block:: python
  200. # NOTE: This is a comment about `Foo Bar`
  201. _('Foo Bar')
  202. The comments tag for the above example would be ``NOTE:``, and the translator
  203. comment for that tag would be ``This is a comment about `Foo Bar```.
  204. The resulting output in the catalog template would be something like::
  205. #. This is a comment about `Foo Bar`
  206. #: main.py:2
  207. msgid "Foo Bar"
  208. msgstr ""
  209. Now, you might ask, why would I need that?
  210. Consider this simple case; you have a menu item called “manual”. You know what
  211. it means, but when the translator sees this they will wonder did you mean:
  212. 1. a document or help manual, or
  213. 2. a manual process?
  214. This is the simplest case where a translation comment such as
  215. “The installation manual” helps to clarify the situation and makes a translator
  216. more productive.
  217. .. note:: Whether translator comments can be extracted depends on the extraction
  218. method in use. The Python extractor provided by Babel does implement
  219. this feature, but others may not.