PKG-INFO 4.6 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116
  1. Metadata-Version: 2.1
  2. Name: parquet
  3. Version: 1.3.1
  4. Summary: Python support for Parquet file format
  5. Home-page: https://github.com/jcrobak/parquet-python
  6. Author: Joe Crobak
  7. Author-email: joecrow@gmail.com
  8. License: Apache License 2.0
  9. Description: parquet-python
  10. ==============
  11. .. image:: https://travis-ci.org/jcrobak/parquet-python.svg?branch=master
  12. :target: https://travis-ci.org/jcrobak/parquet-python
  13. parquet-python is a pure-python implementation (currently with only
  14. read-support) of the `parquet
  15. format <https://github.com/Parquet/parquet-format>`_. It comes with a
  16. script for reading parquet files and outputting the data to stdout as
  17. JSON or TSV (without the overhead of JVM startup). Performance has not
  18. yet been optimized, but it's useful for debugging and quick viewing of
  19. data in files.
  20. Not all parts of the parquet-format have been implemented yet or tested
  21. e.g. nested data—see Todos below for a full list. With that said,
  22. parquet-python is capable of reading all the data files from the
  23. `parquet-compatability <https://github.com/Parquet/parquet-compatibility>`_
  24. project.
  25. requirements
  26. ============
  27. parquet-python has been tested on python 2.7, 3.6, and 3.7. It depends
  28. on ``pythrift2`` and optionally on ``python-snappy`` (for snappy compressed
  29. files, please also install ``parquet-python[snappy]``).
  30. getting started
  31. ===============
  32. parquet-python is available via PyPi and can be installed using
  33. `pip install parquet`. The package includes the `parquet`
  34. command for reading python files, e.g. `parquet test.parquet`.
  35. See `parquet --help` for full usage.
  36. Example
  37. -------
  38. parquet-python currently has two programatic interfaces with similar
  39. functionality to Python's csv reader. First, it supports a DictReader
  40. which returns a dictionary per row. Second, it has a reader which
  41. returns a list of values for each row. Both function require a file-like
  42. object and support an optional ``columns`` field to only read the
  43. specified columns.
  44. .. code:: python
  45. import parquet
  46. import json
  47. ## assuming parquet file with two rows and three columns:
  48. ## foo bar baz
  49. ## 1 2 3
  50. ## 4 5 6
  51. with open("test.parquet") as fo:
  52. # prints:
  53. # {"foo": 1, "bar": 2}
  54. # {"foo": 4, "bar": 5}
  55. for row in parquet.DictReader(fo, columns=['foo', 'bar']):
  56. print(json.dumps(row))
  57. with open("test.parquet") as fo:
  58. # prints:
  59. # 1,2
  60. # 4,5
  61. for row in parquet.reader(fo, columns=['foo', 'bar]):
  62. print(",".join([str(r) for r in row]))
  63. Todos
  64. =====
  65. - Support the deprecated bitpacking
  66. - Fix handling of repetition-levels and definition-levels
  67. - Tests for nested schemas, null data
  68. - Support reading of data from HDFS via snakebite and/or webhdfs.
  69. - Implement writing
  70. - performance evaluation and optimization (i.e. how does it compare to
  71. the c++, java implementations)
  72. Contributing
  73. ============
  74. Is done via Pull Requests. Please include tests with your changes and
  75. follow `pep8 <http://www.python.org/dev/peps/pep-0008/>`_.
  76. To run the tests you must install and execute ``tox`` (``pip install tox``) to
  77. run for all supported versions. If you want to run just for your current
  78. version, execute: ``pip install -r requirements-development.txt`` and then
  79. ``nosetests``.
  80. Platform: UNKNOWN
  81. Classifier: Development Status :: 3 - Alpha
  82. Classifier: Intended Audience :: Developers
  83. Classifier: Intended Audience :: System Administrators
  84. Classifier: License :: OSI Approved :: Apache Software License
  85. Classifier: Programming Language :: Python
  86. Classifier: Programming Language :: Python :: 2
  87. Classifier: Programming Language :: Python :: 3
  88. Classifier: Programming Language :: Python :: 2.7
  89. Classifier: Programming Language :: Python :: 3.6
  90. Classifier: Programming Language :: Python :: 3.7
  91. Classifier: Programming Language :: Python :: Implementation :: CPython
  92. Classifier: Programming Language :: Python :: Implementation :: PyPy
  93. Description-Content-Type: text/x-rst
  94. Provides-Extra: snappy