| 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503 |
- <html>
- <title>
- PyASN1 codecs
- </title>
- <head>
- </head>
- <body>
- <center>
- <table width=60%>
- <tr>
- <td>
- <h3>
- 2. PyASN1 Codecs
- </h3>
- <p>
- In ASN.1 context,
- <a href=http://en.wikipedia.org/wiki/Codec>codec</a>
- is a program that transforms between concrete data structures and a stream
- of octets, suitable for transmission over the wire. This serialized form of
- data is sometimes called <i>substrate</i> or <i>essence</i>.
- </p>
- <p>
- In pyasn1 implementation, substrate takes shape of Python 3 bytes or
- Python 2 string objects.
- </p>
- <p>
- One of the properties of a codec is its ability to cope with incomplete
- data and/or substrate what implies codec to be stateful. In other words,
- when decoder runs out of substrate and data item being recovered is still
- incomplete, stateful codec would suspend and complete data item recovery
- whenever the rest of substrate becomes available. Similarly, stateful encoder
- would encode data items in multiple steps waiting for source data to
- arrive. Codec restartability is especially important when application deals
- with large volumes of data and/or runs on low RAM. For an interesting
- discussion on codecs options and design choices, refer to
- <a href=http://directory.apache.org/subprojects/asn1/>Apache ASN.1 project</a>
- .
- </p>
- <p>
- As of this writing, codecs implemented in pyasn1 are all stateless, mostly
- to keep the code simple.
- </p>
- <p>
- The pyasn1 package currently supports
- <a href=http://en.wikipedia.org/wiki/Basic_encoding_rules>BER</a> codec and
- its variations --
- <a href=http://en.wikipedia.org/wiki/Canonical_encoding_rules>CER</a> and
- <a href=http://en.wikipedia.org/wiki/Distinguished_encoding_rules>DER</a>.
- More ASN.1 codecs are planned for implementation in the future.
- </p>
- <a name="2.1"></a>
- <h4>
- 2.1 Encoders
- </h4>
- <p>
- Encoder is used for transforming pyasn1 value objects into substrate. Only
- pyasn1 value objects could be serialized, attempts to process pyasn1 type
- objects will cause encoder failure.
- </p>
- <p>
- The following code will create a pyasn1 Integer object and serialize it with
- BER encoder:
- </p>
- <table bgcolor="lightgray" border=0 width=100%><TR><TD>
- <pre>
- >>> from pyasn1.type import univ
- >>> from pyasn1.codec.ber import encoder
- >>> encoder.encode(univ.Integer(123456))
- b'\x02\x03\x01\xe2@'
- >>>
- </pre>
- </td></tr></table>
- <p>
- BER standard also defines a so-called <i>indefinite length</i> encoding form
- which makes large data items processing more memory efficient. It is mostly
- useful when encoder does not have the whole value all at once and the
- length of the value can not be determined at the beginning of encoding.
- </p>
- <p>
- <i>Constructed encoding</i> is another feature of BER closely related to the
- indefinite length form. In essence, a large scalar value (such as ASN.1
- character BitString type) could be chopped into smaller chunks by encoder
- and transmitted incrementally to limit memory consumption. Unlike indefinite
- length case, the length of the whole value must be known in advance when
- using constructed, definite length encoding form.
- </p>
- <p>
- Since pyasn1 codecs are not restartable, pyasn1 encoder may only encode data
- item all at once. However, even in this case, generating indefinite length
- encoding may help a low-memory receiver, running a restartable decoder,
- to process a large data item.
- </p>
- <table bgcolor="lightgray" border=0 width=100%><TR><TD>
- <pre>
- >>> from pyasn1.type import univ
- >>> from pyasn1.codec.ber import encoder
- >>> encoder.encode(
- ... univ.OctetString('The quick brown fox jumps over the lazy dog'),
- ... defMode=False,
- ... maxChunkSize=8
- ... )
- b'$\x80\x04\x08The quic\x04\x08k brown \x04\x08fox jump\x04\x08s over \
- t\x04\x08he lazy \x04\x03dog\x00\x00'
- >>>
- >>> encoder.encode(
- ... univ.OctetString('The quick brown fox jumps over the lazy dog'),
- ... maxChunkSize=8
- ... )
- b'$7\x04\x08The quic\x04\x08k brown \x04\x08fox jump\x04\x08s over \
- t\x04\x08he lazy \x04\x03dog'
- </pre>
- </td></tr></table>
- <p>
- The <b>defMode</b> encoder parameter disables definite length encoding mode,
- while the optional <b>maxChunkSize</b> parameter specifies desired
- substrate chunk size that influences memory requirements at the decoder's end.
- </p>
- <p>
- To use CER or DER encoders one needs to explicitly import and call them - the
- APIs are all compatible.
- </p>
- <table bgcolor="lightgray" border=0 width=100%><TR><TD>
- <pre>
- >>> from pyasn1.type import univ
- >>> from pyasn1.codec.ber import encoder as ber_encoder
- >>> from pyasn1.codec.cer import encoder as cer_encoder
- >>> from pyasn1.codec.der import encoder as der_encoder
- >>> ber_encoder.encode(univ.Boolean(True))
- b'\x01\x01\x01'
- >>> cer_encoder.encode(univ.Boolean(True))
- b'\x01\x01\xff'
- >>> der_encoder.encode(univ.Boolean(True))
- b'\x01\x01\xff'
- >>>
- </pre>
- </td></tr></table>
- <a name="2.2"></a>
- <h4>
- 2.2 Decoders
- </h4>
- <p>
- In the process of decoding, pyasn1 value objects are created and linked to
- each other, based on the information containted in the substrate. Thus,
- the original pyasn1 value object(s) are recovered.
- </p>
- <table bgcolor="lightgray" border=0 width=100%><TR><TD>
- <pre>
- >>> from pyasn1.type import univ
- >>> from pyasn1.codec.ber import encoder, decoder
- >>> substrate = encoder.encode(univ.Boolean(True))
- >>> decoder.decode(substrate)
- (Boolean('True(1)'), b'')
- >>>
- </pre>
- </td></tr></table>
- <p>
- Commenting on the code snippet above, pyasn1 decoder accepts substrate
- as an argument and returns a tuple of pyasn1 value object (possibly
- a top-level one in case of constructed object) and unprocessed part
- of input substrate.
- </p>
- <p>
- All pyasn1 decoders can handle both definite and indefinite length
- encoding modes automatically, explicit switching into one mode
- to another is not required.
- </p>
- <table bgcolor="lightgray" border=0 width=100%><TR><TD>
- <pre>
- >>> from pyasn1.type import univ
- >>> from pyasn1.codec.ber import encoder, decoder
- >>> substrate = encoder.encode(
- ... univ.OctetString('The quick brown fox jumps over the lazy dog'),
- ... defMode=False,
- ... maxChunkSize=8
- ... )
- >>> decoder.decode(substrate)
- (OctetString(b'The quick brown fox jumps over the lazy dog'), b'')
- >>>
- </pre>
- </td></tr></table>
- <p>
- Speaking of BER/CER/DER encoding, in many situations substrate may not contain
- all necessary information needed for complete and accurate ASN.1 values
- recovery. The most obvious cases include implicitly tagged ASN.1 types
- and constrained types.
- </p>
- <p>
- As discussed earlier in this handbook, when an ASN.1 type is implicitly
- tagged, previous outermost tag is lost and never appears in substrate.
- If it is the base tag that gets lost, decoder is unable to pick type-specific
- value decoder at its table of built-in types, and therefore recover
- the value part, based only on the information contained in substrate. The
- approach taken by pyasn1 decoder is to use a prototype pyasn1 type object (or
- a set of them) to <i>guide</i> the decoding process by matching [possibly
- incomplete] tags recovered from substrate with those found in prototype pyasn1
- type objects (also called pyasn1 specification object further in this paper).
- </p>
- <table bgcolor="lightgray" border=0 width=100%><TR><TD>
- <pre>
- >>> from pyasn1.codec.ber import decoder
- >>> decoder.decode(b'\x02\x01\x0c', asn1Spec=univ.Integer())
- Integer(12), b''
- >>>
- </pre>
- </td></tr></table>
- <p>
- Decoder would neither modify pyasn1 specification object nor use
- its current values (if it's a pyasn1 value object), but rather use it as
- a hint for choosing proper decoder and as a pattern for creating new objects:
- </p>
- <table bgcolor="lightgray" border=0 width=100%><TR><TD>
- <pre>
- >>> from pyasn1.type import univ, tag
- >>> from pyasn1.codec.ber import encoder, decoder
- >>> i = univ.Integer(12345).subtype(
- ... implicitTag=tag.Tag(tag.tagClassContext, tag.tagFormatSimple, 40)
- ... )
- >>> substrate = encoder.encode(i)
- >>> substrate
- b'\x9f(\x0209'
- >>> decoder.decode(substrate)
- Traceback (most recent call last):
- ...
- pyasn1.error.PyAsn1Error:
- TagSet(Tag(tagClass=128, tagFormat=0, tagId=40)) not in asn1Spec
- >>> decoder.decode(substrate, asn1Spec=i)
- (Integer(12345), b'')
- >>>
- </pre>
- </td></tr></table>
- <p>
- Notice in the example above, that an attempt to run decoder without passing
- pyasn1 specification object fails because recovered tag does not belong
- to any of the built-in types.
- </p>
- <p>
- Another important feature of guided decoder operation is the use of
- values constraints possibly present in pyasn1 specification object.
- To explain this, we will decode a random integer object into generic Integer
- and the constrained one.
- </p>
- <table bgcolor="lightgray" border=0 width=100%><TR><TD>
- <pre>
- >>> from pyasn1.type import univ, constraint
- >>> from pyasn1.codec.ber import encoder, decoder
- >>> class DialDigit(univ.Integer):
- ... subtypeSpec = constraint.ValueRangeConstraint(0,9)
- >>> substrate = encoder.encode(univ.Integer(13))
- >>> decoder.decode(substrate)
- (Integer(13), b'')
- >>> decoder.decode(substrate, asn1Spec=DialDigit())
- Traceback (most recent call last):
- ...
- pyasn1.type.error.ValueConstraintError:
- ValueRangeConstraint(0, 9) failed at: 13
- >>>
- </pre>
- </td></tr></table>
- <p>
- Similarily to encoders, to use CER or DER decoders application has to
- explicitly import and call them - all APIs are compatible.
- </p>
- <table bgcolor="lightgray" border=0 width=100%><TR><TD>
- <pre>
- >>> from pyasn1.type import univ
- >>> from pyasn1.codec.ber import encoder as ber_encoder
- >>> substrate = ber_encoder.encode(univ.OctetString('http://pyasn1.sf.net'))
- >>>
- >>> from pyasn1.codec.ber import decoder as ber_decoder
- >>> from pyasn1.codec.cer import decoder as cer_decoder
- >>> from pyasn1.codec.der import decoder as der_decoder
- >>>
- >>> ber_decoder.decode(substrate)
- (OctetString(b'http://pyasn1.sf.net'), b'')
- >>> cer_decoder.decode(substrate)
- (OctetString(b'http://pyasn1.sf.net'), b'')
- >>> der_decoder.decode(substrate)
- (OctetString(b'http://pyasn1.sf.net'), b'')
- >>>
- </pre>
- </td></tr></table>
- <a name="2.2.1"></a>
- <h4>
- 2.2.1 Decoding untagged types
- </h4>
- <p>
- It has already been mentioned, that ASN.1 has two "special case" types:
- CHOICE and ANY. They are different from other types in part of
- tagging - unless these two are additionally tagged, neither of them will
- have their own tag. Therefore these types become invisible in substrate
- and can not be recovered without passing pyasn1 specification object to
- decoder.
- </p>
- <p>
- To explain the issue, we will first prepare a Choice object to deal with:
- </p>
- <table bgcolor="lightgray" border=0 width=100%><TR><TD>
- <pre>
- >>> from pyasn1.type import univ, namedtype
- >>> class CodeOrMessage(univ.Choice):
- ... componentType = namedtype.NamedTypes(
- ... namedtype.NamedType('code', univ.Integer()),
- ... namedtype.NamedType('message', univ.OctetString())
- ... )
- >>>
- >>> codeOrMessage = CodeOrMessage()
- >>> codeOrMessage.setComponentByName('message', 'my string value')
- >>> print(codeOrMessage.prettyPrint())
- CodeOrMessage:
- message=b'my string value'
- >>>
- </pre>
- </td></tr></table>
- <p>
- Let's now encode this Choice object and then decode its substrate
- with and without pyasn1 specification object:
- </p>
- <table bgcolor="lightgray" border=0 width=100%><TR><TD>
- <pre>
- >>> from pyasn1.codec.ber import encoder, decoder
- >>> substrate = encoder.encode(codeOrMessage)
- >>> substrate
- b'\x04\x0fmy string value'
- >>> encoder.encode(univ.OctetString('my string value'))
- b'\x04\x0fmy string value'
- >>>
- >>> decoder.decode(substrate)
- (OctetString(b'my string value'), b'')
- >>> codeOrMessage, substrate = decoder.decode(substrate, asn1Spec=CodeOrMessage())
- >>> print(codeOrMessage.prettyPrint())
- CodeOrMessage:
- message=b'my string value'
- >>>
- </pre>
- </td></tr></table>
- <p>
- First thing to notice in the listing above is that the substrate produced
- for our Choice value object is equivalent to the substrate for an OctetString
- object initialized to the same value. In other words, any information about
- the Choice component is absent in encoding.
- </p>
- <p>
- Sure enough, that kind of substrate will decode into an OctetString object,
- unless original Choice type object is passed to decoder to guide the decoding
- process.
- </p>
- <p>
- Similarily untagged ANY type behaves differently on decoding phase - when
- decoder bumps into an Any object in pyasn1 specification, it stops decoding
- and puts all the substrate into a new Any value object in form of an octet
- string. Concerned application could then re-run decoder with an additional,
- more exact pyasn1 specification object to recover the contents of Any
- object.
- </p>
- <p>
- As it was mentioned elsewhere in this paper, Any type allows for incomplete
- or changing ASN.1 specification to be handled gracefully by decoder and
- applications.
- </p>
- <p>
- To illustrate the working of Any type, we'll have to make the stage
- by encoding a pyasn1 object and then putting its substrate into an any
- object.
- </p>
- <table bgcolor="lightgray" border=0 width=100%><TR><TD>
- <pre>
- >>> from pyasn1.type import univ
- >>> from pyasn1.codec.ber import encoder, decoder
- >>> innerSubstrate = encoder.encode(univ.Integer(1234))
- >>> innerSubstrate
- b'\x02\x02\x04\xd2'
- >>> any = univ.Any(innerSubstrate)
- >>> any
- Any(b'\x02\x02\x04\xd2')
- >>> substrate = encoder.encode(any)
- >>> substrate
- b'\x02\x02\x04\xd2'
- >>>
- </pre>
- </td></tr></table>
- <p>
- As with Choice type encoding, there is no traces of Any type in substrate.
- Obviously, the substrate we are dealing with, will decode into the inner
- [Integer] component, unless pyasn1 specification is given to guide the
- decoder. Continuing previous code:
- </p>
- <table bgcolor="lightgray" border=0 width=100%><TR><TD>
- <pre>
- >>> from pyasn1.type import univ
- >>> from pyasn1.codec.ber import encoder, decoder
- >>> decoder.decode(substrate)
- (Integer(1234), b'')
- >>> any, substrate = decoder.decode(substrate, asn1Spec=univ.Any())
- >>> any
- Any(b'\x02\x02\x04\xd2')
- >>> decoder.decode(str(any))
- (Integer(1234), b'')
- >>>
- </pre>
- </td></tr></table>
- <p>
- Both CHOICE and ANY types are widely used in practice. Reader is welcome to
- take a look at
- <a href=http://www.cs.auckland.ac.nz/~pgut001/pubs/x509guide.txt>
- ASN.1 specifications of X.509 applications</a> for more information.
- </p>
- <a name="2.2.2"></a>
- <h4>
- 2.2.2 Ignoring unknown types
- </h4>
- <p>
- When dealing with a loosely specified ASN.1 structure, the receiving
- end may not be aware of some types present in the substrate. It may be
- convenient then to turn decoder into a recovery mode. Whilst there, decoder
- will not bail out when hit an unknown tag but rather treat it as an Any
- type.
- </p>
- <table bgcolor="lightgray" border=0 width=100%><TR><TD>
- <pre>
- >>> from pyasn1.type import univ, tag
- >>> from pyasn1.codec.ber import encoder, decoder
- >>> taggedInt = univ.Integer(12345).subtype(
- ... implicitTag=tag.Tag(tag.tagClassContext, tag.tagFormatSimple, 40)
- ... )
- >>> substrate = encoder.encode(taggedInt)
- >>> decoder.decode(substrate)
- Traceback (most recent call last):
- ...
- pyasn1.error.PyAsn1Error: TagSet(Tag(tagClass=128, tagFormat=0, tagId=40)) not in asn1Spec
- >>>
- >>> decoder.decode.defaultErrorState = decoder.stDumpRawValue
- >>> decoder.decode(substrate)
- (Any(b'\x9f(\x0209'), '')
- >>>
- </pre>
- </td></tr></table>
- <p>
- It's also possible to configure a custom decoder, to handle unknown tags
- found in substrate. This can be done by means of <b>defaultRawDecoder</b>
- attribute holding a reference to type decoder object. Refer to the source
- for API details.
- </p>
- <hr>
- </td>
- </tr>
- </table>
- </center>
- </body>
- </html>
|