codecs.html 15 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503
  1. <html>
  2. <title>
  3. PyASN1 codecs
  4. </title>
  5. <head>
  6. </head>
  7. <body>
  8. <center>
  9. <table width=60%>
  10. <tr>
  11. <td>
  12. <h3>
  13. 2. PyASN1 Codecs
  14. </h3>
  15. <p>
  16. In ASN.1 context,
  17. <a href=http://en.wikipedia.org/wiki/Codec>codec</a>
  18. is a program that transforms between concrete data structures and a stream
  19. of octets, suitable for transmission over the wire. This serialized form of
  20. data is sometimes called <i>substrate</i> or <i>essence</i>.
  21. </p>
  22. <p>
  23. In pyasn1 implementation, substrate takes shape of Python 3 bytes or
  24. Python 2 string objects.
  25. </p>
  26. <p>
  27. One of the properties of a codec is its ability to cope with incomplete
  28. data and/or substrate what implies codec to be stateful. In other words,
  29. when decoder runs out of substrate and data item being recovered is still
  30. incomplete, stateful codec would suspend and complete data item recovery
  31. whenever the rest of substrate becomes available. Similarly, stateful encoder
  32. would encode data items in multiple steps waiting for source data to
  33. arrive. Codec restartability is especially important when application deals
  34. with large volumes of data and/or runs on low RAM. For an interesting
  35. discussion on codecs options and design choices, refer to
  36. <a href=http://directory.apache.org/subprojects/asn1/>Apache ASN.1 project</a>
  37. .
  38. </p>
  39. <p>
  40. As of this writing, codecs implemented in pyasn1 are all stateless, mostly
  41. to keep the code simple.
  42. </p>
  43. <p>
  44. The pyasn1 package currently supports
  45. <a href=http://en.wikipedia.org/wiki/Basic_encoding_rules>BER</a> codec and
  46. its variations --
  47. <a href=http://en.wikipedia.org/wiki/Canonical_encoding_rules>CER</a> and
  48. <a href=http://en.wikipedia.org/wiki/Distinguished_encoding_rules>DER</a>.
  49. More ASN.1 codecs are planned for implementation in the future.
  50. </p>
  51. <a name="2.1"></a>
  52. <h4>
  53. 2.1 Encoders
  54. </h4>
  55. <p>
  56. Encoder is used for transforming pyasn1 value objects into substrate. Only
  57. pyasn1 value objects could be serialized, attempts to process pyasn1 type
  58. objects will cause encoder failure.
  59. </p>
  60. <p>
  61. The following code will create a pyasn1 Integer object and serialize it with
  62. BER encoder:
  63. </p>
  64. <table bgcolor="lightgray" border=0 width=100%><TR><TD>
  65. <pre>
  66. >>> from pyasn1.type import univ
  67. >>> from pyasn1.codec.ber import encoder
  68. >>> encoder.encode(univ.Integer(123456))
  69. b'\x02\x03\x01\xe2@'
  70. >>>
  71. </pre>
  72. </td></tr></table>
  73. <p>
  74. BER standard also defines a so-called <i>indefinite length</i> encoding form
  75. which makes large data items processing more memory efficient. It is mostly
  76. useful when encoder does not have the whole value all at once and the
  77. length of the value can not be determined at the beginning of encoding.
  78. </p>
  79. <p>
  80. <i>Constructed encoding</i> is another feature of BER closely related to the
  81. indefinite length form. In essence, a large scalar value (such as ASN.1
  82. character BitString type) could be chopped into smaller chunks by encoder
  83. and transmitted incrementally to limit memory consumption. Unlike indefinite
  84. length case, the length of the whole value must be known in advance when
  85. using constructed, definite length encoding form.
  86. </p>
  87. <p>
  88. Since pyasn1 codecs are not restartable, pyasn1 encoder may only encode data
  89. item all at once. However, even in this case, generating indefinite length
  90. encoding may help a low-memory receiver, running a restartable decoder,
  91. to process a large data item.
  92. </p>
  93. <table bgcolor="lightgray" border=0 width=100%><TR><TD>
  94. <pre>
  95. >>> from pyasn1.type import univ
  96. >>> from pyasn1.codec.ber import encoder
  97. >>> encoder.encode(
  98. ... univ.OctetString('The quick brown fox jumps over the lazy dog'),
  99. ... defMode=False,
  100. ... maxChunkSize=8
  101. ... )
  102. b'$\x80\x04\x08The quic\x04\x08k brown \x04\x08fox jump\x04\x08s over \
  103. t\x04\x08he lazy \x04\x03dog\x00\x00'
  104. >>>
  105. >>> encoder.encode(
  106. ... univ.OctetString('The quick brown fox jumps over the lazy dog'),
  107. ... maxChunkSize=8
  108. ... )
  109. b'$7\x04\x08The quic\x04\x08k brown \x04\x08fox jump\x04\x08s over \
  110. t\x04\x08he lazy \x04\x03dog'
  111. </pre>
  112. </td></tr></table>
  113. <p>
  114. The <b>defMode</b> encoder parameter disables definite length encoding mode,
  115. while the optional <b>maxChunkSize</b> parameter specifies desired
  116. substrate chunk size that influences memory requirements at the decoder's end.
  117. </p>
  118. <p>
  119. To use CER or DER encoders one needs to explicitly import and call them - the
  120. APIs are all compatible.
  121. </p>
  122. <table bgcolor="lightgray" border=0 width=100%><TR><TD>
  123. <pre>
  124. >>> from pyasn1.type import univ
  125. >>> from pyasn1.codec.ber import encoder as ber_encoder
  126. >>> from pyasn1.codec.cer import encoder as cer_encoder
  127. >>> from pyasn1.codec.der import encoder as der_encoder
  128. >>> ber_encoder.encode(univ.Boolean(True))
  129. b'\x01\x01\x01'
  130. >>> cer_encoder.encode(univ.Boolean(True))
  131. b'\x01\x01\xff'
  132. >>> der_encoder.encode(univ.Boolean(True))
  133. b'\x01\x01\xff'
  134. >>>
  135. </pre>
  136. </td></tr></table>
  137. <a name="2.2"></a>
  138. <h4>
  139. 2.2 Decoders
  140. </h4>
  141. <p>
  142. In the process of decoding, pyasn1 value objects are created and linked to
  143. each other, based on the information containted in the substrate. Thus,
  144. the original pyasn1 value object(s) are recovered.
  145. </p>
  146. <table bgcolor="lightgray" border=0 width=100%><TR><TD>
  147. <pre>
  148. >>> from pyasn1.type import univ
  149. >>> from pyasn1.codec.ber import encoder, decoder
  150. >>> substrate = encoder.encode(univ.Boolean(True))
  151. >>> decoder.decode(substrate)
  152. (Boolean('True(1)'), b'')
  153. >>>
  154. </pre>
  155. </td></tr></table>
  156. <p>
  157. Commenting on the code snippet above, pyasn1 decoder accepts substrate
  158. as an argument and returns a tuple of pyasn1 value object (possibly
  159. a top-level one in case of constructed object) and unprocessed part
  160. of input substrate.
  161. </p>
  162. <p>
  163. All pyasn1 decoders can handle both definite and indefinite length
  164. encoding modes automatically, explicit switching into one mode
  165. to another is not required.
  166. </p>
  167. <table bgcolor="lightgray" border=0 width=100%><TR><TD>
  168. <pre>
  169. >>> from pyasn1.type import univ
  170. >>> from pyasn1.codec.ber import encoder, decoder
  171. >>> substrate = encoder.encode(
  172. ... univ.OctetString('The quick brown fox jumps over the lazy dog'),
  173. ... defMode=False,
  174. ... maxChunkSize=8
  175. ... )
  176. >>> decoder.decode(substrate)
  177. (OctetString(b'The quick brown fox jumps over the lazy dog'), b'')
  178. >>>
  179. </pre>
  180. </td></tr></table>
  181. <p>
  182. Speaking of BER/CER/DER encoding, in many situations substrate may not contain
  183. all necessary information needed for complete and accurate ASN.1 values
  184. recovery. The most obvious cases include implicitly tagged ASN.1 types
  185. and constrained types.
  186. </p>
  187. <p>
  188. As discussed earlier in this handbook, when an ASN.1 type is implicitly
  189. tagged, previous outermost tag is lost and never appears in substrate.
  190. If it is the base tag that gets lost, decoder is unable to pick type-specific
  191. value decoder at its table of built-in types, and therefore recover
  192. the value part, based only on the information contained in substrate. The
  193. approach taken by pyasn1 decoder is to use a prototype pyasn1 type object (or
  194. a set of them) to <i>guide</i> the decoding process by matching [possibly
  195. incomplete] tags recovered from substrate with those found in prototype pyasn1
  196. type objects (also called pyasn1 specification object further in this paper).
  197. </p>
  198. <table bgcolor="lightgray" border=0 width=100%><TR><TD>
  199. <pre>
  200. >>> from pyasn1.codec.ber import decoder
  201. >>> decoder.decode(b'\x02\x01\x0c', asn1Spec=univ.Integer())
  202. Integer(12), b''
  203. >>>
  204. </pre>
  205. </td></tr></table>
  206. <p>
  207. Decoder would neither modify pyasn1 specification object nor use
  208. its current values (if it's a pyasn1 value object), but rather use it as
  209. a hint for choosing proper decoder and as a pattern for creating new objects:
  210. </p>
  211. <table bgcolor="lightgray" border=0 width=100%><TR><TD>
  212. <pre>
  213. >>> from pyasn1.type import univ, tag
  214. >>> from pyasn1.codec.ber import encoder, decoder
  215. >>> i = univ.Integer(12345).subtype(
  216. ... implicitTag=tag.Tag(tag.tagClassContext, tag.tagFormatSimple, 40)
  217. ... )
  218. >>> substrate = encoder.encode(i)
  219. >>> substrate
  220. b'\x9f(\x0209'
  221. >>> decoder.decode(substrate)
  222. Traceback (most recent call last):
  223. ...
  224. pyasn1.error.PyAsn1Error:
  225. TagSet(Tag(tagClass=128, tagFormat=0, tagId=40)) not in asn1Spec
  226. >>> decoder.decode(substrate, asn1Spec=i)
  227. (Integer(12345), b'')
  228. >>>
  229. </pre>
  230. </td></tr></table>
  231. <p>
  232. Notice in the example above, that an attempt to run decoder without passing
  233. pyasn1 specification object fails because recovered tag does not belong
  234. to any of the built-in types.
  235. </p>
  236. <p>
  237. Another important feature of guided decoder operation is the use of
  238. values constraints possibly present in pyasn1 specification object.
  239. To explain this, we will decode a random integer object into generic Integer
  240. and the constrained one.
  241. </p>
  242. <table bgcolor="lightgray" border=0 width=100%><TR><TD>
  243. <pre>
  244. >>> from pyasn1.type import univ, constraint
  245. >>> from pyasn1.codec.ber import encoder, decoder
  246. >>> class DialDigit(univ.Integer):
  247. ... subtypeSpec = constraint.ValueRangeConstraint(0,9)
  248. >>> substrate = encoder.encode(univ.Integer(13))
  249. >>> decoder.decode(substrate)
  250. (Integer(13), b'')
  251. >>> decoder.decode(substrate, asn1Spec=DialDigit())
  252. Traceback (most recent call last):
  253. ...
  254. pyasn1.type.error.ValueConstraintError:
  255. ValueRangeConstraint(0, 9) failed at: 13
  256. >>>
  257. </pre>
  258. </td></tr></table>
  259. <p>
  260. Similarily to encoders, to use CER or DER decoders application has to
  261. explicitly import and call them - all APIs are compatible.
  262. </p>
  263. <table bgcolor="lightgray" border=0 width=100%><TR><TD>
  264. <pre>
  265. >>> from pyasn1.type import univ
  266. >>> from pyasn1.codec.ber import encoder as ber_encoder
  267. >>> substrate = ber_encoder.encode(univ.OctetString('http://pyasn1.sf.net'))
  268. >>>
  269. >>> from pyasn1.codec.ber import decoder as ber_decoder
  270. >>> from pyasn1.codec.cer import decoder as cer_decoder
  271. >>> from pyasn1.codec.der import decoder as der_decoder
  272. >>>
  273. >>> ber_decoder.decode(substrate)
  274. (OctetString(b'http://pyasn1.sf.net'), b'')
  275. >>> cer_decoder.decode(substrate)
  276. (OctetString(b'http://pyasn1.sf.net'), b'')
  277. >>> der_decoder.decode(substrate)
  278. (OctetString(b'http://pyasn1.sf.net'), b'')
  279. >>>
  280. </pre>
  281. </td></tr></table>
  282. <a name="2.2.1"></a>
  283. <h4>
  284. 2.2.1 Decoding untagged types
  285. </h4>
  286. <p>
  287. It has already been mentioned, that ASN.1 has two "special case" types:
  288. CHOICE and ANY. They are different from other types in part of
  289. tagging - unless these two are additionally tagged, neither of them will
  290. have their own tag. Therefore these types become invisible in substrate
  291. and can not be recovered without passing pyasn1 specification object to
  292. decoder.
  293. </p>
  294. <p>
  295. To explain the issue, we will first prepare a Choice object to deal with:
  296. </p>
  297. <table bgcolor="lightgray" border=0 width=100%><TR><TD>
  298. <pre>
  299. >>> from pyasn1.type import univ, namedtype
  300. >>> class CodeOrMessage(univ.Choice):
  301. ... componentType = namedtype.NamedTypes(
  302. ... namedtype.NamedType('code', univ.Integer()),
  303. ... namedtype.NamedType('message', univ.OctetString())
  304. ... )
  305. >>>
  306. >>> codeOrMessage = CodeOrMessage()
  307. >>> codeOrMessage.setComponentByName('message', 'my string value')
  308. >>> print(codeOrMessage.prettyPrint())
  309. CodeOrMessage:
  310. message=b'my string value'
  311. >>>
  312. </pre>
  313. </td></tr></table>
  314. <p>
  315. Let's now encode this Choice object and then decode its substrate
  316. with and without pyasn1 specification object:
  317. </p>
  318. <table bgcolor="lightgray" border=0 width=100%><TR><TD>
  319. <pre>
  320. >>> from pyasn1.codec.ber import encoder, decoder
  321. >>> substrate = encoder.encode(codeOrMessage)
  322. >>> substrate
  323. b'\x04\x0fmy string value'
  324. >>> encoder.encode(univ.OctetString('my string value'))
  325. b'\x04\x0fmy string value'
  326. >>>
  327. >>> decoder.decode(substrate)
  328. (OctetString(b'my string value'), b'')
  329. >>> codeOrMessage, substrate = decoder.decode(substrate, asn1Spec=CodeOrMessage())
  330. >>> print(codeOrMessage.prettyPrint())
  331. CodeOrMessage:
  332. message=b'my string value'
  333. >>>
  334. </pre>
  335. </td></tr></table>
  336. <p>
  337. First thing to notice in the listing above is that the substrate produced
  338. for our Choice value object is equivalent to the substrate for an OctetString
  339. object initialized to the same value. In other words, any information about
  340. the Choice component is absent in encoding.
  341. </p>
  342. <p>
  343. Sure enough, that kind of substrate will decode into an OctetString object,
  344. unless original Choice type object is passed to decoder to guide the decoding
  345. process.
  346. </p>
  347. <p>
  348. Similarily untagged ANY type behaves differently on decoding phase - when
  349. decoder bumps into an Any object in pyasn1 specification, it stops decoding
  350. and puts all the substrate into a new Any value object in form of an octet
  351. string. Concerned application could then re-run decoder with an additional,
  352. more exact pyasn1 specification object to recover the contents of Any
  353. object.
  354. </p>
  355. <p>
  356. As it was mentioned elsewhere in this paper, Any type allows for incomplete
  357. or changing ASN.1 specification to be handled gracefully by decoder and
  358. applications.
  359. </p>
  360. <p>
  361. To illustrate the working of Any type, we'll have to make the stage
  362. by encoding a pyasn1 object and then putting its substrate into an any
  363. object.
  364. </p>
  365. <table bgcolor="lightgray" border=0 width=100%><TR><TD>
  366. <pre>
  367. >>> from pyasn1.type import univ
  368. >>> from pyasn1.codec.ber import encoder, decoder
  369. >>> innerSubstrate = encoder.encode(univ.Integer(1234))
  370. >>> innerSubstrate
  371. b'\x02\x02\x04\xd2'
  372. >>> any = univ.Any(innerSubstrate)
  373. >>> any
  374. Any(b'\x02\x02\x04\xd2')
  375. >>> substrate = encoder.encode(any)
  376. >>> substrate
  377. b'\x02\x02\x04\xd2'
  378. >>>
  379. </pre>
  380. </td></tr></table>
  381. <p>
  382. As with Choice type encoding, there is no traces of Any type in substrate.
  383. Obviously, the substrate we are dealing with, will decode into the inner
  384. [Integer] component, unless pyasn1 specification is given to guide the
  385. decoder. Continuing previous code:
  386. </p>
  387. <table bgcolor="lightgray" border=0 width=100%><TR><TD>
  388. <pre>
  389. >>> from pyasn1.type import univ
  390. >>> from pyasn1.codec.ber import encoder, decoder
  391. >>> decoder.decode(substrate)
  392. (Integer(1234), b'')
  393. >>> any, substrate = decoder.decode(substrate, asn1Spec=univ.Any())
  394. >>> any
  395. Any(b'\x02\x02\x04\xd2')
  396. >>> decoder.decode(str(any))
  397. (Integer(1234), b'')
  398. >>>
  399. </pre>
  400. </td></tr></table>
  401. <p>
  402. Both CHOICE and ANY types are widely used in practice. Reader is welcome to
  403. take a look at
  404. <a href=http://www.cs.auckland.ac.nz/~pgut001/pubs/x509guide.txt>
  405. ASN.1 specifications of X.509 applications</a> for more information.
  406. </p>
  407. <a name="2.2.2"></a>
  408. <h4>
  409. 2.2.2 Ignoring unknown types
  410. </h4>
  411. <p>
  412. When dealing with a loosely specified ASN.1 structure, the receiving
  413. end may not be aware of some types present in the substrate. It may be
  414. convenient then to turn decoder into a recovery mode. Whilst there, decoder
  415. will not bail out when hit an unknown tag but rather treat it as an Any
  416. type.
  417. </p>
  418. <table bgcolor="lightgray" border=0 width=100%><TR><TD>
  419. <pre>
  420. >>> from pyasn1.type import univ, tag
  421. >>> from pyasn1.codec.ber import encoder, decoder
  422. >>> taggedInt = univ.Integer(12345).subtype(
  423. ... implicitTag=tag.Tag(tag.tagClassContext, tag.tagFormatSimple, 40)
  424. ... )
  425. >>> substrate = encoder.encode(taggedInt)
  426. >>> decoder.decode(substrate)
  427. Traceback (most recent call last):
  428. ...
  429. pyasn1.error.PyAsn1Error: TagSet(Tag(tagClass=128, tagFormat=0, tagId=40)) not in asn1Spec
  430. >>>
  431. >>> decoder.decode.defaultErrorState = decoder.stDumpRawValue
  432. >>> decoder.decode(substrate)
  433. (Any(b'\x9f(\x0209'), '')
  434. >>>
  435. </pre>
  436. </td></tr></table>
  437. <p>
  438. It's also possible to configure a custom decoder, to handle unknown tags
  439. found in substrate. This can be done by means of <b>defaultRawDecoder</b>
  440. attribute holding a reference to type decoder object. Refer to the source
  441. for API details.
  442. </p>
  443. <hr>
  444. </td>
  445. </tr>
  446. </table>
  447. </center>
  448. </body>
  449. </html>