HowToUsePyparsing.rst 40 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751752753754755756757758759760761762763764765766767768769770771772773774775776777778779780781782783784785786787788789790791792793794795796797798799800801802803804805806807808809810811812813814815816817818819820821822823824825826827828829830831832833834835836837838839840841842843844845846847848849850851852853854855856857858859860861862863864865866867868869870871872873874875876877878879880881882883884885886887888889890891892893894895896897898899900901902903904905906907908909910911912913914915916917918919920921922923924925926927928929930931932933934935936937938939940941942943944945946947948949950951952953954955956957958959960961962963964965966967968969970971972973
  1. ==========================
  2. Using the pyparsing module
  3. ==========================
  4. :author: Paul McGuire
  5. :address: ptmcg@users.sourceforge.net
  6. :revision: 2.0.1a
  7. :date: July, 2013 (minor update August, 2018)
  8. :copyright: Copyright |copy| 2003-2013 Paul McGuire.
  9. .. |copy| unicode:: 0xA9
  10. :abstract: This document provides how-to instructions for the
  11. pyparsing library, an easy-to-use Python module for constructing
  12. and executing basic text parsers. The pyparsing module is useful
  13. for evaluating user-definable
  14. expressions, processing custom application language commands, or
  15. extracting data from formatted reports.
  16. .. sectnum:: :depth: 4
  17. .. contents:: :depth: 4
  18. Note: While this content is still valid, there are more detailed
  19. descriptions and examples at the online doc server at
  20. https://pythonhosted.org/pyparsing/pyparsing-module.html
  21. Steps to follow
  22. ===============
  23. To parse an incoming data string, the client code must follow these steps:
  24. 1. First define the tokens and patterns to be matched, and assign
  25. this to a program variable. Optional results names or parsing
  26. actions can also be defined at this time.
  27. 2. Call ``parseString()`` or ``scanString()`` on this variable, passing in
  28. the string to
  29. be parsed. During the matching process, whitespace between
  30. tokens is skipped by default (although this can be changed).
  31. When token matches occur, any defined parse action methods are
  32. called.
  33. 3. Process the parsed results, returned as a list of strings.
  34. Matching results may also be accessed as named attributes of
  35. the returned results, if names are defined in the definition of
  36. the token pattern, using ``setResultsName()``.
  37. Hello, World!
  38. -------------
  39. The following complete Python program will parse the greeting "Hello, World!",
  40. or any other greeting of the form "<salutation>, <addressee>!"::
  41. from pyparsing import Word, alphas
  42. greet = Word(alphas) + "," + Word(alphas) + "!"
  43. greeting = greet.parseString("Hello, World!")
  44. print greeting
  45. The parsed tokens are returned in the following form::
  46. ['Hello', ',', 'World', '!']
  47. Usage notes
  48. -----------
  49. - The pyparsing module can be used to interpret simple command
  50. strings or algebraic expressions, or can be used to extract data
  51. from text reports with complicated format and structure ("screen
  52. or report scraping"). However, it is possible that your defined
  53. matching patterns may accept invalid inputs. Use pyparsing to
  54. extract data from strings assumed to be well-formatted.
  55. - To keep up the readability of your code, use operators_ such as ``+``, ``|``,
  56. ``^``, and ``~`` to combine expressions. You can also combine
  57. string literals with ParseExpressions - they will be
  58. automatically converted to Literal objects. For example::
  59. integer = Word(nums) # simple unsigned integer
  60. variable = Word(alphas, max=1) # single letter variable, such as x, z, m, etc.
  61. arithOp = Word("+-*/", max=1) # arithmetic operators
  62. equation = variable + "=" + integer + arithOp + integer # will match "x=2+2", etc.
  63. In the definition of ``equation``, the string ``"="`` will get added as
  64. a ``Literal("=")``, but in a more readable way.
  65. - The pyparsing module's default behavior is to ignore whitespace. This is the
  66. case for 99% of all parsers ever written. This allows you to write simple, clean,
  67. grammars, such as the above ``equation``, without having to clutter it up with
  68. extraneous ``ws`` markers. The ``equation`` grammar will successfully parse all of the
  69. following statements::
  70. x=2+2
  71. x = 2+2
  72. a = 10 * 4
  73. r= 1234/ 100000
  74. Of course, it is quite simple to extend this example to support more elaborate expressions, with
  75. nesting with parentheses, floating point numbers, scientific notation, and named constants
  76. (such as ``e`` or ``pi``). See ``fourFn.py``, included in the examples directory.
  77. - To modify pyparsing's default whitespace skipping, you can use one or
  78. more of the following methods:
  79. - use the static method ``ParserElement.setDefaultWhitespaceChars``
  80. to override the normal set of whitespace chars (' \t\n'). For instance
  81. when defining a grammar in which newlines are significant, you should
  82. call ``ParserElement.setDefaultWhitespaceChars(' \t')`` to remove
  83. newline from the set of skippable whitespace characters. Calling
  84. this method will affect all pyparsing expressions defined afterward.
  85. - call ``leaveWhitespace()`` on individual expressions, to suppress the
  86. skipping of whitespace before trying to match the expression
  87. - use ``Combine`` to require that successive expressions must be
  88. adjacent in the input string. For instance, this expression::
  89. real = Word(nums) + '.' + Word(nums)
  90. will match "3.14159", but will also match "3 . 12". It will also
  91. return the matched results as ['3', '.', '14159']. By changing this
  92. expression to::
  93. real = Combine(Word(nums) + '.' + Word(nums))
  94. it will not match numbers with embedded spaces, and it will return a
  95. single concatenated string '3.14159' as the parsed token.
  96. - Repetition of expressions can be indicated using ``*`` or ``[]`` notation. An
  97. expression may be multiplied by an integer value (to indicate an exact
  98. repetition count), or indexed with a tuple, representing min and max repetitions
  99. (with ``...`` representing no min or no max, depending whether it is the first or
  100. second tuple element). See the following examples, where n is used to
  101. indicate an integer value:
  102. - ``expr*3`` is equivalent to ``expr + expr + expr``
  103. - ``expr[2, 3]`` is equivalent to ``expr + expr + Optional(expr)``
  104. - ``expr[n, ...]`` or ``expr[n,]`` is equivalent
  105. to ``expr*n + ZeroOrMore(expr)`` (read as "at least n instances of expr")
  106. - ``expr[... ,n]`` is equivalent to ``expr*(0, n)``
  107. (read as "0 to n instances of expr")
  108. - ``expr[...]`` and ``expr[0, ...]`` are equivalent to ``ZeroOrMore(expr)``
  109. - ``expr[1, ...]`` is equivalent to ``OneOrMore(expr)``
  110. Note that ``expr[..., n]`` does not raise an exception if
  111. more than n exprs exist in the input stream; that is,
  112. ``expr[..., n]`` does not enforce a maximum number of expr
  113. occurrences. If this behavior is desired, then write
  114. ``expr[..., n] + ~expr``.
  115. - ``MatchFirst`` expressions are matched left-to-right, and the first
  116. match found will skip all later expressions within, so be sure
  117. to define less-specific patterns after more-specific patterns.
  118. If you are not sure which expressions are most specific, use Or
  119. expressions (defined using the ``^`` operator) - they will always
  120. match the longest expression, although they are more
  121. compute-intensive.
  122. - ``Or`` expressions will evaluate all of the specified subexpressions
  123. to determine which is the "best" match, that is, which matches
  124. the longest string in the input data. In case of a tie, the
  125. left-most expression in the ``Or`` list will win.
  126. - If parsing the contents of an entire file, pass it to the
  127. ``parseFile`` method using::
  128. expr.parseFile(sourceFile)
  129. - ``ParseExceptions`` will report the location where an expected token
  130. or expression failed to match. For example, if we tried to use our
  131. "Hello, World!" parser to parse "Hello World!" (leaving out the separating
  132. comma), we would get an exception, with the message::
  133. pyparsing.ParseException: Expected "," (6), (1,7)
  134. In the case of complex
  135. expressions, the reported location may not be exactly where you
  136. would expect. See more information under ParseException_ .
  137. - Use the ``Group`` class to enclose logical groups of tokens within a
  138. sublist. This will help organize your results into more
  139. hierarchical form (the default behavior is to return matching
  140. tokens as a flat list of matching input strings).
  141. - Punctuation may be significant for matching, but is rarely of
  142. much interest in the parsed results. Use the ``suppress()`` method
  143. to keep these tokens from cluttering up your returned lists of
  144. tokens. For example, ``delimitedList()`` matches a succession of
  145. one or more expressions, separated by delimiters (commas by
  146. default), but only returns a list of the actual expressions -
  147. the delimiters are used for parsing, but are suppressed from the
  148. returned output.
  149. - Parse actions can be used to convert values from strings to
  150. other data types (ints, floats, booleans, etc.).
  151. - Results names are recommended for retrieving tokens from complex
  152. expressions. It is much easier to access a token using its field
  153. name than using a positional index, especially if the expression
  154. contains optional elements. You can also shortcut
  155. the ``setResultsName`` call::
  156. stats = ("AVE:" + realNum.setResultsName("average")
  157. + "MIN:" + realNum.setResultsName("min")
  158. + "MAX:" + realNum.setResultsName("max"))
  159. can now be written as this::
  160. stats = ("AVE:" + realNum("average")
  161. + "MIN:" + realNum("min")
  162. + "MAX:" + realNum("max"))
  163. - Be careful when defining parse actions that modify global variables or
  164. data structures (as in ``fourFn.py``), especially for low level tokens
  165. or expressions that may occur within an ``And`` expression; an early element
  166. of an ``And`` may match, but the overall expression may fail.
  167. Classes
  168. =======
  169. Classes in the pyparsing module
  170. -------------------------------
  171. ``ParserElement`` - abstract base class for all pyparsing classes;
  172. methods for code to use are:
  173. - ``parseString(sourceString, parseAll=False)`` - only called once, on the overall
  174. matching pattern; returns a ParseResults_ object that makes the
  175. matched tokens available as a list, and optionally as a dictionary,
  176. or as an object with named attributes; if parseAll is set to True, then
  177. parseString will raise a ParseException if the grammar does not process
  178. the complete input string.
  179. - ``parseFile(sourceFile)`` - a convenience function, that accepts an
  180. input file object or filename. The file contents are passed as a
  181. string to ``parseString()``. ``parseFile`` also supports the ``parseAll`` argument.
  182. - ``scanString(sourceString)`` - generator function, used to find and
  183. extract matching text in the given source string; for each matched text,
  184. returns a tuple of:
  185. - matched tokens (packaged as a ParseResults_ object)
  186. - start location of the matched text in the given source string
  187. - end location in the given source string
  188. ``scanString`` allows you to scan through the input source string for
  189. random matches, instead of exhaustively defining the grammar for the entire
  190. source text (as would be required with ``parseString``).
  191. - ``transformString(sourceString)`` - convenience wrapper function for
  192. ``scanString``, to process the input source string, and replace matching
  193. text with the tokens returned from parse actions defined in the grammar
  194. (see setParseAction_).
  195. - ``searchString(sourceString)`` - another convenience wrapper function for
  196. ``scanString``, returns a list of the matching tokens returned from each
  197. call to ``scanString``.
  198. - ``setName(name)`` - associate a short descriptive name for this
  199. element, useful in displaying exceptions and trace information
  200. - ``setResultsName(string, listAllMatches=False)`` - name to be given
  201. to tokens matching
  202. the element; if multiple tokens within
  203. a repetition group (such as ``ZeroOrMore`` or ``delimitedList``) the
  204. default is to return only the last matching token - if listAllMatches
  205. is set to True, then a list of all the matching tokens is returned.
  206. (New in 1.5.6 - a results name with a trailing '*' character will be
  207. interpreted as setting listAllMatches to True.)
  208. Note:
  209. ``setResultsName`` returns a *copy* of the element so that a single
  210. basic element can be referenced multiple times and given
  211. different names within a complex grammar.
  212. .. _setParseAction:
  213. - ``setParseAction(*fn)`` - specify one or more functions to call after successful
  214. matching of the element; each function is defined as ``fn(s, loc, toks)``, where:
  215. - ``s`` is the original parse string
  216. - ``loc`` is the location in the string where matching started
  217. - ``toks`` is the list of the matched tokens, packaged as a ParseResults_ object
  218. Multiple functions can be attached to a ParserElement by specifying multiple
  219. arguments to setParseAction, or by calling setParseAction multiple times.
  220. Each parse action function can return a modified ``toks`` list, to perform conversion, or
  221. string modifications. For brevity, ``fn`` may also be a
  222. lambda - here is an example of using a parse action to convert matched
  223. integer tokens from strings to integers::
  224. intNumber = Word(nums).setParseAction(lambda s,l,t: [int(t[0])])
  225. If ``fn`` does not modify the ``toks`` list, it does not need to return
  226. anything at all.
  227. - ``setBreak(breakFlag=True)`` - if breakFlag is True, calls pdb.set_break()
  228. as this expression is about to be parsed
  229. - ``copy()`` - returns a copy of a ParserElement; can be used to use the same
  230. parse expression in different places in a grammar, with different parse actions
  231. attached to each
  232. - ``leaveWhitespace()`` - change default behavior of skipping
  233. whitespace before starting matching (mostly used internally to the
  234. pyparsing module, rarely used by client code)
  235. - ``setWhitespaceChars(chars)`` - define the set of chars to be ignored
  236. as whitespace before trying to match a specific ParserElement, in place of the
  237. default set of whitespace (space, tab, newline, and return)
  238. - ``setDefaultWhitespaceChars(chars)`` - class-level method to override
  239. the default set of whitespace chars for all subsequently created ParserElements
  240. (including copies); useful when defining grammars that treat one or more of the
  241. default whitespace characters as significant (such as a line-sensitive grammar, to
  242. omit newline from the list of ignorable whitespace)
  243. - ``suppress()`` - convenience function to suppress the output of the
  244. given element, instead of wrapping it with a Suppress object.
  245. - ``ignore(expr)`` - function to specify parse expression to be
  246. ignored while matching defined patterns; can be called
  247. repeatedly to specify multiple expressions; useful to specify
  248. patterns of comment syntax, for example
  249. - ``setDebug(dbgFlag=True)`` - function to enable/disable tracing output
  250. when trying to match this element
  251. - ``validate()`` - function to verify that the defined grammar does not
  252. contain infinitely recursive constructs
  253. .. _parseWithTabs:
  254. - ``parseWithTabs()`` - function to override default behavior of converting
  255. tabs to spaces before parsing the input string; rarely used, except when
  256. specifying whitespace-significant grammars using the White_ class.
  257. - ``enablePackrat()`` - a class-level static method to enable a memoizing
  258. performance enhancement, known as "packrat parsing". packrat parsing is
  259. disabled by default, since it may conflict with some user programs that use
  260. parse actions. To activate the packrat feature, your
  261. program must call the class method ParserElement.enablePackrat(). For best
  262. results, call enablePackrat() immediately after importing pyparsing.
  263. Basic ParserElement subclasses
  264. ------------------------------
  265. - ``Literal`` - construct with a string to be matched exactly
  266. - ``CaselessLiteral`` - construct with a string to be matched, but
  267. without case checking; results are always returned as the
  268. defining literal, NOT as they are found in the input string
  269. - ``Keyword`` - similar to Literal, but must be immediately followed by
  270. whitespace, punctuation, or other non-keyword characters; prevents
  271. accidental matching of a non-keyword that happens to begin with a
  272. defined keyword
  273. - ``CaselessKeyword`` - similar to Keyword, but with caseless matching
  274. behavior
  275. .. _Word:
  276. - ``Word`` - one or more contiguous characters; construct with a
  277. string containing the set of allowed initial characters, and an
  278. optional second string of allowed body characters; for instance,
  279. a common Word construct is to match a code identifier - in C, a
  280. valid identifier must start with an alphabetic character or an
  281. underscore ('_'), followed by a body that can also include numeric
  282. digits. That is, ``a``, ``i``, ``MAX_LENGTH``, ``_a1``, ``b_109_``, and
  283. ``plan9FromOuterSpace``
  284. are all valid identifiers; ``9b7z``, ``$a``, ``.section``, and ``0debug``
  285. are not. To
  286. define an identifier using a Word, use either of the following::
  287. - Word(alphas+"_", alphanums+"_")
  288. - Word(srange("[a-zA-Z_]"), srange("[a-zA-Z0-9_]"))
  289. If only one
  290. string given, it specifies that the same character set defined
  291. for the initial character is used for the word body; for instance, to
  292. define an identifier that can only be composed of capital letters and
  293. underscores, use::
  294. - Word("ABCDEFGHIJKLMNOPQRSTUVWXYZ_")
  295. - Word(srange("[A-Z_]"))
  296. A Word may
  297. also be constructed with any of the following optional parameters:
  298. - ``min`` - indicating a minimum length of matching characters
  299. - ``max`` - indicating a maximum length of matching characters
  300. - ``exact`` - indicating an exact length of matching characters
  301. If ``exact`` is specified, it will override any values for ``min`` or ``max``.
  302. New in 1.5.6 - Sometimes you want to define a word using all
  303. characters in a range except for one or two of them; you can do this
  304. with the new ``excludeChars`` argument. This is helpful if you want to define
  305. a word with all printables except for a single delimiter character, such
  306. as '.'. Previously, you would have to create a custom string to pass to Word.
  307. With this change, you can just create ``Word(printables, excludeChars='.')``.
  308. - ``CharsNotIn`` - similar to Word_, but matches characters not
  309. in the given constructor string (accepts only one string for both
  310. initial and body characters); also supports ``min``, ``max``, and ``exact``
  311. optional parameters.
  312. - ``Regex`` - a powerful construct, that accepts a regular expression
  313. to be matched at the current parse position; accepts an optional
  314. ``flags`` parameter, corresponding to the flags parameter in the re.compile
  315. method; if the expression includes named sub-fields, they will be
  316. represented in the returned ParseResults_
  317. - ``QuotedString`` - supports the definition of custom quoted string
  318. formats, in addition to pyparsing's built-in ``dblQuotedString`` and
  319. ``sglQuotedString``. ``QuotedString`` allows you to specify the following
  320. parameters:
  321. - ``quoteChar`` - string of one or more characters defining the quote delimiting string
  322. - ``escChar`` - character to escape quotes, typically backslash (default=None)
  323. - ``escQuote`` - special quote sequence to escape an embedded quote string (such as SQL's "" to escape an embedded ") (default=None)
  324. - ``multiline`` - boolean indicating whether quotes can span multiple lines (default=False)
  325. - ``unquoteResults`` - boolean indicating whether the matched text should be unquoted (default=True)
  326. - ``endQuoteChar`` - string of one or more characters defining the end of the quote delimited string (default=None => same as quoteChar)
  327. - ``SkipTo`` - skips ahead in the input string, accepting any
  328. characters up to the specified pattern; may be constructed with
  329. the following optional parameters:
  330. - ``include`` - if set to true, also consumes the match expression
  331. (default is false)
  332. - ``ignore`` - allows the user to specify patterns to not be matched,
  333. to prevent false matches
  334. - ``failOn`` - if a literal string or expression is given for this argument, it defines an expression that
  335. should cause the ``SkipTo`` expression to fail, and not skip over that expression
  336. .. _White:
  337. - ``White`` - also similar to Word_, but matches whitespace
  338. characters. Not usually needed, as whitespace is implicitly
  339. ignored by pyparsing. However, some grammars are whitespace-sensitive,
  340. such as those that use leading tabs or spaces to indicating grouping
  341. or hierarchy. (If matching on tab characters, be sure to call
  342. parseWithTabs_ on the top-level parse element.)
  343. - ``Empty`` - a null expression, requiring no characters - will always
  344. match; useful for debugging and for specialized grammars
  345. - ``NoMatch`` - opposite of Empty, will never match; useful for debugging
  346. and for specialized grammars
  347. Expression subclasses
  348. ---------------------
  349. - ``And`` - construct with a list of ParserElements, all of which must
  350. match for And to match; can also be created using the '+'
  351. operator; multiple expressions can be Anded together using the '*'
  352. operator as in::
  353. ipAddress = Word(nums) + ('.' + Word(nums)) * 3
  354. A tuple can be used as the multiplier, indicating a min/max::
  355. usPhoneNumber = Word(nums) + ('-' + Word(nums)) * (1,2)
  356. A special form of ``And`` is created if the '-' operator is used
  357. instead of the '+' operator. In the ipAddress example above, if
  358. no trailing '.' and Word(nums) are found after matching the initial
  359. Word(nums), then pyparsing will back up in the grammar and try other
  360. alternatives to ipAddress. However, if ipAddress is defined as::
  361. strictIpAddress = Word(nums) - ('.'+Word(nums))*3
  362. then no backing up is done. If the first Word(nums) of strictIpAddress
  363. is matched, then any mismatch after that will raise a ParseSyntaxException,
  364. which will halt the parsing process immediately. By careful use of the
  365. '-' operator, grammars can provide meaningful error messages close to
  366. the location where the incoming text does not match the specified
  367. grammar.
  368. - ``Or`` - construct with a list of ParserElements, any of which must
  369. match for Or to match; if more than one expression matches, the
  370. expression that makes the longest match will be used; can also
  371. be created using the '^' operator
  372. - ``MatchFirst`` - construct with a list of ParserElements, any of
  373. which must match for MatchFirst to match; matching is done
  374. left-to-right, taking the first expression that matches; can
  375. also be created using the '|' operator
  376. - ``Each`` - similar to And, in that all of the provided expressions
  377. must match; however, Each permits matching to be done in any order;
  378. can also be created using the '&' operator
  379. - ``Optional`` - construct with a ParserElement, but this element is
  380. not required to match; can be constructed with an optional ``default`` argument,
  381. containing a default string or object to be supplied if the given optional
  382. parse element is not found in the input string; parse action will only
  383. be called if a match is found, or if a default is specified
  384. - ``ZeroOrMore`` - similar to Optional, but can be repeated
  385. - ``OneOrMore`` - similar to ZeroOrMore, but at least one match must
  386. be present
  387. - ``FollowedBy`` - a lookahead expression, requires matching of the given
  388. expressions, but does not advance the parsing position within the input string
  389. - ``NotAny`` - a negative lookahead expression, prevents matching of named
  390. expressions, does not advance the parsing position within the input string;
  391. can also be created using the unary '~' operator
  392. .. _operators:
  393. Expression operators
  394. --------------------
  395. - ``~`` - creates NotAny using the expression after the operator
  396. - ``+`` - creates And using the expressions before and after the operator
  397. - ``|`` - creates MatchFirst (first left-to-right match) using the expressions before and after the operator
  398. - ``^`` - creates Or (longest match) using the expressions before and after the operator
  399. - ``&`` - creates Each using the expressions before and after the operator
  400. - ``*`` - creates And by multiplying the expression by the integer operand; if
  401. expression is multiplied by a 2-tuple, creates an And of (min,max)
  402. expressions (similar to "{min,max}" form in regular expressions); if
  403. min is None, intepret as (0,max); if max is None, interpret as
  404. expr*min + ZeroOrMore(expr)
  405. - ``-`` - like ``+`` but with no backup and retry of alternatives
  406. - ``*`` - repetition of expression
  407. - ``==`` - matching expression to string; returns True if the string matches the given expression
  408. - ``<<=`` - inserts the expression following the operator as the body of the
  409. Forward expression before the operator
  410. Positional subclasses
  411. ---------------------
  412. - ``StringStart`` - matches beginning of the text
  413. - ``StringEnd`` - matches the end of the text
  414. - ``LineStart`` - matches beginning of a line (lines delimited by ``\n`` characters)
  415. - ``LineEnd`` - matches the end of a line
  416. - ``WordStart`` - matches a leading word boundary
  417. - ``WordEnd`` - matches a trailing word boundary
  418. Converter subclasses
  419. --------------------
  420. - ``Combine`` - joins all matched tokens into a single string, using
  421. specified joinString (default ``joinString=""``); expects
  422. all matching tokens to be adjacent, with no intervening
  423. whitespace (can be overridden by specifying ``adjacent=False`` in constructor)
  424. - ``Suppress`` - clears matched tokens; useful to keep returned
  425. results from being cluttered with required but uninteresting
  426. tokens (such as list delimiters)
  427. Special subclasses
  428. ------------------
  429. - ``Group`` - causes the matched tokens to be enclosed in a list;
  430. useful in repeated elements like ``ZeroOrMore`` and ``OneOrMore`` to
  431. break up matched tokens into groups for each repeated pattern
  432. - ``Dict`` - like ``Group``, but also constructs a dictionary, using the
  433. [0]'th elements of all enclosed token lists as the keys, and
  434. each token list as the value
  435. - ``SkipTo`` - catch-all matching expression that accepts all characters
  436. up until the given pattern is found to match; useful for specifying
  437. incomplete grammars
  438. - ``Forward`` - placeholder token used to define recursive token
  439. patterns; when defining the actual expression later in the
  440. program, insert it into the ``Forward`` object using the ``<<``
  441. operator (see ``fourFn.py`` for an example).
  442. Other classes
  443. -------------
  444. .. _ParseResults:
  445. - ``ParseResults`` - class used to contain and manage the lists of tokens
  446. created from parsing the input using the user-defined parse
  447. expression. ParseResults can be accessed in a number of ways:
  448. - as a list
  449. - total list of elements can be found using len()
  450. - individual elements can be found using [0], [1], [-1], etc.
  451. - elements can be deleted using ``del``
  452. - the -1th element can be extracted and removed in a single operation
  453. using ``pop()``, or any element can be extracted and removed
  454. using ``pop(n)``
  455. - as a dictionary
  456. - if ``setResultsName()`` is used to name elements within the
  457. overall parse expression, then these fields can be referenced
  458. as dictionary elements or as attributes
  459. - the Dict class generates dictionary entries using the data of the
  460. input text - in addition to ParseResults listed as ``[ [ a1, b1, c1, ...], [ a2, b2, c2, ...] ]``
  461. it also acts as a dictionary with entries defined as ``{ a1 : [ b1, c1, ... ] }, { a2 : [ b2, c2, ... ] }``;
  462. this is especially useful when processing tabular data where the first column contains a key
  463. value for that line of data
  464. - list elements that are deleted using ``del`` will still be accessible by their
  465. dictionary keys
  466. - supports ``get()``, ``items()`` and ``keys()`` methods, similar to a dictionary
  467. - a keyed item can be extracted and removed using ``pop(key)``. Here
  468. key must be non-numeric (such as a string), in order to use dict
  469. extraction instead of list extraction.
  470. - new named elements can be added (in a parse action, for instance), using the same
  471. syntax as adding an item to a dict (``parseResults["X"] = "new item"``); named elements can be removed using ``del parseResults["X"]``
  472. - as a nested list
  473. - results returned from the Group class are encapsulated within their
  474. own list structure, so that the tokens can be handled as a hierarchical
  475. tree
  476. ParseResults can also be converted to an ordinary list of strings
  477. by calling ``asList()``. Note that this will strip the results of any
  478. field names that have been defined for any embedded parse elements.
  479. (The ``pprint`` module is especially good at printing out the nested contents
  480. given by ``asList()``.)
  481. Finally, ParseResults can be viewed by calling ``dump()``. ``dump()` will first show
  482. the ``asList()`` output, followed by an indented structure listing parsed tokens that
  483. have been assigned results names.
  484. Exception classes and Troubleshooting
  485. -------------------------------------
  486. .. _ParseException:
  487. - ``ParseException`` - exception returned when a grammar parse fails;
  488. ParseExceptions have attributes loc, msg, line, lineno, and column; to view the
  489. text line and location where the reported ParseException occurs, use::
  490. except ParseException, err:
  491. print err.line
  492. print " " * (err.column - 1) + "^"
  493. print err
  494. - ``RecursiveGrammarException`` - exception returned by ``validate()`` if
  495. the grammar contains a recursive infinite loop, such as::
  496. badGrammar = Forward()
  497. goodToken = Literal("A")
  498. badGrammar <<= Optional(goodToken) + badGrammar
  499. - ``ParseFatalException`` - exception that parse actions can raise to stop parsing
  500. immediately. Should be used when a semantic error is found in the input text, such
  501. as a mismatched XML tag.
  502. - ``ParseSyntaxException`` - subclass of ``ParseFatalException`` raised when a
  503. syntax error is found, based on the use of the '-' operator when defining
  504. a sequence of expressions in an ``And`` expression.
  505. You can also get some insights into the parsing logic using diagnostic parse actions,
  506. and setDebug(), or test the matching of expression fragments by testing them using
  507. scanString().
  508. Miscellaneous attributes and methods
  509. ====================================
  510. Helper methods
  511. --------------
  512. - ``delimitedList(expr, delim=',')`` - convenience function for
  513. matching one or more occurrences of expr, separated by delim.
  514. By default, the delimiters are suppressed, so the returned results contain
  515. only the separate list elements. Can optionally specify ``combine=True``,
  516. indicating that the expressions and delimiters should be returned as one
  517. combined value (useful for scoped variables, such as ``"a.b.c"``, or
  518. ``"a::b::c"``, or paths such as ``"a/b/c"``).
  519. - ``countedArray(expr)`` - convenience function for a pattern where an list of
  520. instances of the given expression are preceded by an integer giving the count of
  521. elements in the list. Returns an expression that parses the leading integer,
  522. reads exactly that many expressions, and returns the array of expressions in the
  523. parse results - the leading integer is suppressed from the results (although it
  524. is easily reconstructed by using len on the returned array).
  525. - ``oneOf(string, caseless=False)`` - convenience function for quickly declaring an
  526. alternative set of ``Literal`` tokens, by splitting the given string on
  527. whitespace boundaries. The tokens are sorted so that longer
  528. matches are attempted first; this ensures that a short token does
  529. not mask a longer one that starts with the same characters. If ``caseless=True``,
  530. will create an alternative set of CaselessLiteral tokens.
  531. - ``dictOf(key, value)`` - convenience function for quickly declaring a
  532. dictionary pattern of ``Dict(ZeroOrMore(Group(key + value)))``.
  533. - ``makeHTMLTags(tagName)`` and ``makeXMLTags(tagName)`` - convenience
  534. functions to create definitions of opening and closing tag expressions. Returns
  535. a pair of expressions, for the corresponding <tag> and </tag> strings. Includes
  536. support for attributes in the opening tag, such as <tag attr1="abc"> - attributes
  537. are returned as keyed tokens in the returned ParseResults. ``makeHTMLTags`` is less
  538. restrictive than ``makeXMLTags``, especially with respect to case sensitivity.
  539. - ``infixNotation(baseOperand, operatorList)`` - (formerly named ``operatorPrecedence``)
  540. convenience function to define a grammar for parsing infix notation
  541. expressions with a hierarchical precedence of operators. To use the ``infixNotation``
  542. helper:
  543. 1. Define the base "atom" operand term of the grammar.
  544. For this simple grammar, the smallest operand is either
  545. and integer or a variable. This will be the first argument
  546. to the ``infixNotation`` method.
  547. 2. Define a list of tuples for each level of operator
  548. precendence. Each tuple is of the form
  549. ``(opExpr, numTerms, rightLeftAssoc, parseAction)``, where:
  550. - ``opExpr`` - the pyparsing expression for the operator;
  551. may also be a string, which will be converted to a Literal; if
  552. None, indicates an empty operator, such as the implied
  553. multiplication operation between 'm' and 'x' in "y = mx + b".
  554. - ``numTerms`` - the number of terms for this operator (must
  555. be 1, 2, or 3)
  556. - ``rightLeftAssoc`` is the indicator whether the operator is
  557. right or left associative, using the pyparsing-defined
  558. constants ``opAssoc.RIGHT`` and ``opAssoc.LEFT``.
  559. - ``parseAction`` is the parse action to be associated with
  560. expressions matching this operator expression (the
  561. ``parseAction`` tuple member may be omitted)
  562. 3. Call ``infixNotation`` passing the operand expression and
  563. the operator precedence list, and save the returned value
  564. as the generated pyparsing expression. You can then use
  565. this expression to parse input strings, or incorporate it
  566. into a larger, more complex grammar.
  567. - ``matchPreviousLiteral`` and ``matchPreviousExpr`` - function to define and
  568. expression that matches the same content
  569. as was parsed in a previous parse expression. For instance::
  570. first = Word(nums)
  571. matchExpr = first + ":" + matchPreviousLiteral(first)
  572. will match "1:1", but not "1:2". Since this matches at the literal
  573. level, this will also match the leading "1:1" in "1:10".
  574. In contrast::
  575. first = Word(nums)
  576. matchExpr = first + ":" + matchPreviousExpr(first)
  577. will *not* match the leading "1:1" in "1:10"; the expressions are
  578. evaluated first, and then compared, so "1" is compared with "10".
  579. - ``nestedExpr(opener, closer, content=None, ignoreExpr=quotedString)`` - method for defining nested
  580. lists enclosed in opening and closing delimiters.
  581. - ``opener`` - opening character for a nested list (default="("); can also be a pyparsing expression
  582. - ``closer`` - closing character for a nested list (default=")"); can also be a pyparsing expression
  583. - ``content`` - expression for items within the nested lists (default=None)
  584. - ``ignoreExpr`` - expression for ignoring opening and closing delimiters (default=quotedString)
  585. If an expression is not provided for the content argument, the nested
  586. expression will capture all whitespace-delimited content between delimiters
  587. as a list of separate values.
  588. Use the ignoreExpr argument to define expressions that may contain
  589. opening or closing characters that should not be treated as opening
  590. or closing characters for nesting, such as quotedString or a comment
  591. expression. Specify multiple expressions using an Or or MatchFirst.
  592. The default is quotedString, but if no expressions are to be ignored,
  593. then pass None for this argument.
  594. - ``indentedBlock(statementExpr, indentationStackVar, indent=True)`` -
  595. function to define an indented block of statements, similar to
  596. indentation-based blocking in Python source code:
  597. - ``statementExpr`` - the expression defining a statement that
  598. will be found in the indented block; a valid ``indentedBlock``
  599. must contain at least 1 matching ``statementExpr``
  600. - ``indentationStackVar`` - a Python list variable; this variable
  601. should be common to all ``indentedBlock`` expressions defined
  602. within the same grammar, and should be reinitialized to [1]
  603. each time the grammar is to be used
  604. - ``indent`` - a boolean flag indicating whether the expressions
  605. within the block must be indented from the current parse
  606. location; if using ``indentedBlock`` to define the left-most
  607. statements (all starting in column 1), set ``indent`` to False
  608. .. _originalTextFor:
  609. - ``originalTextFor(expr)`` - helper function to preserve the originally parsed text, regardless of any
  610. token processing or conversion done by the contained expression. For instance, the following expression::
  611. fullName = Word(alphas) + Word(alphas)
  612. will return the parse of "John Smith" as ['John', 'Smith']. In some applications, the actual name as it
  613. was given in the input string is what is desired. To do this, use ``originalTextFor``::
  614. fullName = originalTextFor(Word(alphas) + Word(alphas))
  615. - ``ungroup(expr)`` - function to "ungroup" returned tokens; useful
  616. to undo the default behavior of And to always group the returned tokens, even
  617. if there is only one in the list. (New in 1.5.6)
  618. - ``lineno(loc, string)`` - function to give the line number of the
  619. location within the string; the first line is line 1, newlines
  620. start new rows
  621. - ``col(loc, string)`` - function to give the column number of the
  622. location within the string; the first column is column 1,
  623. newlines reset the column number to 1
  624. - ``line(loc, string)`` - function to retrieve the line of text
  625. representing ``lineno(loc, string)``; useful when printing out diagnostic
  626. messages for exceptions
  627. - ``srange(rangeSpec)`` - function to define a string of characters,
  628. given a string of the form used by regexp string ranges, such as ``"[0-9]"`` for
  629. all numeric digits, ``"[A-Z_]"`` for uppercase characters plus underscore, and
  630. so on (note that rangeSpec does not include support for generic regular
  631. expressions, just string range specs)
  632. - ``getTokensEndLoc()`` - function to call from within a parse action to get
  633. the ending location for the matched tokens
  634. - ``traceParseAction(fn)`` - decorator function to debug parse actions. Lists
  635. each call, called arguments, and return value or exception
  636. Helper parse actions
  637. --------------------
  638. - ``removeQuotes`` - removes the first and last characters of a quoted string;
  639. useful to remove the delimiting quotes from quoted strings
  640. - ``replaceWith(replString)`` - returns a parse action that simply returns the
  641. replString; useful when using transformString, or converting HTML entities, as in::
  642. nbsp = Literal("&nbsp;").setParseAction( replaceWith("<BLANK>") )
  643. - ``keepOriginalText``- (deprecated, use originalTextFor_ instead) restores any internal whitespace or suppressed
  644. text within the tokens for a matched parse
  645. expression. This is especially useful when defining expressions
  646. for scanString or transformString applications.
  647. - ``withAttribute( *args, **kwargs )`` - helper to create a validating parse action to be used with start tags created
  648. with ``makeXMLTags`` or ``makeHTMLTags``. Use ``withAttribute`` to qualify a starting tag
  649. with a required attribute value, to avoid false matches on common tags such as
  650. ``<TD>`` or ``<DIV>``.
  651. ``withAttribute`` can be called with:
  652. - keyword arguments, as in ``(class="Customer", align="right")``, or
  653. - a list of name-value tuples, as in ``(("ns1:class", "Customer"), ("ns2:align", "right"))``
  654. An attribute can be specified to have the special value
  655. ``withAttribute.ANY_VALUE``, which will match any value - use this to
  656. ensure that an attribute is present but any attribute value is
  657. acceptable.
  658. - ``downcaseTokens`` - converts all matched tokens to lowercase
  659. - ``upcaseTokens`` - converts all matched tokens to uppercase
  660. - ``matchOnlyAtCol(columnNumber)`` - a parse action that verifies that
  661. an expression was matched at a particular column, raising a
  662. ParseException if matching at a different column number; useful when parsing
  663. tabular data
  664. Common string and token constants
  665. ---------------------------------
  666. - ``alphas`` - same as ``string.letters``
  667. - ``nums`` - same as ``string.digits``
  668. - ``alphanums`` - a string containing ``alphas + nums``
  669. - ``alphas8bit`` - a string containing alphabetic 8-bit characters::
  670. ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõöøùúûüýþ
  671. - ``printables`` - same as ``string.printable``, minus the space (``' '``) character
  672. - ``empty`` - a global ``Empty()``; will always match
  673. - ``sglQuotedString`` - a string of characters enclosed in 's; may
  674. include whitespace, but not newlines
  675. - ``dblQuotedString`` - a string of characters enclosed in "s; may
  676. include whitespace, but not newlines
  677. - ``quotedString`` - ``sglQuotedString | dblQuotedString``
  678. - ``cStyleComment`` - a comment block delimited by ``'/*'`` and ``'*/'`` sequences; can span
  679. multiple lines, but does not support nesting of comments
  680. - ``htmlComment`` - a comment block delimited by ``'<!--'`` and ``'-->'`` sequences; can span
  681. multiple lines, but does not support nesting of comments
  682. - ``commaSeparatedList`` - similar to ``delimitedList``, except that the
  683. list expressions can be any text value, or a quoted string; quoted strings can
  684. safely include commas without incorrectly breaking the string into two tokens
  685. - ``restOfLine`` - all remaining printable characters up to but not including the next
  686. newline