usage.rst 2.7 KB

12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788
  1. Usage
  2. =====
  3. Basic usage
  4. -----------
  5. The easiest way to use the Universal Encoding Detector library is with
  6. the ``detect`` function.
  7. Example: Using the ``detect`` function
  8. --------------------------------------
  9. The ``detect`` function takes one argument, a non-Unicode string. It
  10. returns a dictionary containing the auto-detected character encoding and
  11. a confidence level from ``0`` to ``1``.
  12. .. code:: python
  13. >>> import urllib
  14. >>> rawdata = urllib.urlopen('http://yahoo.co.jp/').read()
  15. >>> import chardet
  16. >>> chardet.detect(rawdata)
  17. {'encoding': 'EUC-JP', 'confidence': 0.99}
  18. Advanced usage
  19. --------------
  20. If you’re dealing with a large amount of text, you can call the
  21. Universal Encoding Detector library incrementally, and it will stop as
  22. soon as it is confident enough to report its results.
  23. Create a ``UniversalDetector`` object, then call its ``feed`` method
  24. repeatedly with each block of text. If the detector reaches a minimum
  25. threshold of confidence, it will set ``detector.done`` to ``True``.
  26. Once you’ve exhausted the source text, call ``detector.close()``, which
  27. will do some final calculations in case the detector didn’t hit its
  28. minimum confidence threshold earlier. Then ``detector.result`` will be a
  29. dictionary containing the auto-detected character encoding and
  30. confidence level (the same as the ``chardet.detect`` function
  31. `returns <usage.html#example-using-the-detect-function>`__).
  32. Example: Detecting encoding incrementally
  33. -----------------------------------------
  34. .. code:: python
  35. import urllib
  36. from chardet.universaldetector import UniversalDetector
  37. usock = urllib.urlopen('http://yahoo.co.jp/')
  38. detector = UniversalDetector()
  39. for line in usock.readlines():
  40. detector.feed(line)
  41. if detector.done: break
  42. detector.close()
  43. usock.close()
  44. print detector.result
  45. .. code:: python
  46. {'encoding': 'EUC-JP', 'confidence': 0.99}
  47. If you want to detect the encoding of multiple texts (such as separate
  48. files), you can re-use a single ``UniversalDetector`` object. Just call
  49. ``detector.reset()`` at the start of each file, call ``detector.feed``
  50. as many times as you like, and then call ``detector.close()`` and check
  51. the ``detector.result`` dictionary for the file’s results.
  52. Example: Detecting encodings of multiple files
  53. ----------------------------------------------
  54. .. code:: python
  55. import glob
  56. from chardet.universaldetector import UniversalDetector
  57. detector = UniversalDetector()
  58. for filename in glob.glob('*.xml'):
  59. print filename.ljust(60),
  60. detector.reset()
  61. for line in file(filename, 'rb'):
  62. detector.feed(line)
  63. if detector.done: break
  64. detector.close()
  65. print detector.result