NOTES.rst 3.7 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140
  1. Class Hierarchy for chardet
  2. ===========================
  3. Universal Detector
  4. ------------------
  5. Has a list of probers.
  6. CharSetProber
  7. -------------
  8. Mostly abstract parent class.
  9. CharSetGroupProber
  10. ------------------
  11. Runs a bunch of related probers at the same time and decides which is best.
  12. SBCSGroupProber
  13. ---------------
  14. SBCS = Single-ByteCharSet. Runs a bunch of SingleByteCharSetProbers. Always
  15. contains the same SingleByteCharSetProbers.
  16. SingleByteCharSetProber
  17. -----------------------
  18. A CharSetProber that is used for detecting single-byte encodings by using
  19. a "precedence matrix" (i.e., a character bigram model).
  20. MBCSGroupProber
  21. ---------------
  22. Runs a bunch of MultiByteCharSetProbers. It also uses a UTF8Prober, which is
  23. essentially a MultiByteCharSetProber that only has a state machine. Always
  24. contains the same MultiByteCharSetProbers.
  25. MultiByteCharSetProber
  26. ----------------------
  27. A CharSetProber that uses both a character unigram model (or "character
  28. distribution analysis") and an independent state machine for trying to
  29. detect and encoding.
  30. CodingStateMachine
  31. ------------------
  32. Used for "coding scheme" detection, where we just look for either invalid
  33. byte sequences or sequences that only occur for that particular encoding.
  34. CharDistributionAnalysis
  35. ------------------------
  36. Used for character unigram distribution encoding detection. Takes a mapping
  37. from characters to a "frequency order" (i.e., what frequency rank that byte has
  38. in the given encoding) and a "typical distribution ratio", which is the number
  39. of occurrences of the 512 most frequently used characters divided by the number
  40. of occurrences of the rest of the characters for a typical document.
  41. The "characters" in this case are 2-byte sequences and they are first converted
  42. to an "order" (name comes from ord() function, I believe). This "order" is used
  43. to index into the frequency order table to determine the frequency rank of that
  44. byte sequence. The reason this extra step is necessary is that the frequency
  45. rank table is language-specific (and not encoding-specific).
  46. What's where
  47. ============
  48. Bigram files
  49. ------------
  50. - ``hebrewprober.py``
  51. - ``jpcntxprober.py``
  52. - ``langbulgarianmodel.py``
  53. - ``langcyrillicmodel.py``
  54. - ``langgreekmodel.py``
  55. - ``langhebrewmodel.py``
  56. - ``langhungarianmodel.py``
  57. - ``langthaimodel.py``
  58. - ``latin1prober.py``
  59. - ``sbcharsetprober.py``
  60. - ``sbcsgroupprober.py``
  61. Coding Scheme files
  62. -------------------
  63. - ``escprober.py``
  64. - ``escsm.py``
  65. - ``utf8prober.py``
  66. - ``codingstatemachine.py``
  67. - ``mbcssmprober.py``
  68. Unigram files
  69. -------------
  70. - ``big5freqprober.py``
  71. - ``chardistribution.py``
  72. - ``euckrfreqprober.py``
  73. - ``euctwfreqprober.py``
  74. - ``gb2312freqprober.py``
  75. - ``jisfreqprober.py``
  76. Multibyte probers
  77. -----------------
  78. - ``big5prober.py``
  79. - ``cp949prober.py``
  80. - ``eucjpprober.py``
  81. - ``euckrprober.py``
  82. - ``euctwprober.py``
  83. - ``gb2312prober.py``
  84. - ``mbcharsetprober.py``
  85. - ``mbcsgroupprober.py``
  86. - ``sjisprober.py``
  87. Misc files
  88. ----------
  89. - ``__init__.py`` (currently has ``detect`` function in it)
  90. - ``compat.py``
  91. - ``enums.py``
  92. - ``universaldetector.py``
  93. - ``version.py``
  94. Useful links
  95. ============
  96. This is just a collection of information that I've found useful or thought
  97. might be useful in the future:
  98. - `BOM by Encoding`_
  99. - `A Composite Approach to Language/Encoding Detection`_
  100. - `What Every Programmer Absolutely...`_
  101. - The actual `source`_
  102. .. _BOM by Encoding:
  103. https://en.wikipedia.org/wiki/Byte_order_mark#Representations_of_byte_order_marks_by_encoding
  104. .. _A Composite Approach to Language/Encoding Detection:
  105. http://www-archive.mozilla.org/projects/intl/UniversalCharsetDetection.html
  106. .. _What Every Programmer Absolutely...: http://kunststube.net/encoding/
  107. .. _source: https://mxr.mozilla.org/mozilla/source/intl/chardet/