README.rst 3.0 KB

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980818283848586878889909192
  1. parquet-python
  2. ==============
  3. .. image:: https://travis-ci.org/jcrobak/parquet-python.svg?branch=master
  4. :target: https://travis-ci.org/jcrobak/parquet-python
  5. parquet-python is a pure-python implementation (currently with only
  6. read-support) of the `parquet
  7. format <https://github.com/Parquet/parquet-format>`_. It comes with a
  8. script for reading parquet files and outputting the data to stdout as
  9. JSON or TSV (without the overhead of JVM startup). Performance has not
  10. yet been optimized, but it's useful for debugging and quick viewing of
  11. data in files.
  12. Not all parts of the parquet-format have been implemented yet or tested
  13. e.g. nested data—see Todos below for a full list. With that said,
  14. parquet-python is capable of reading all the data files from the
  15. `parquet-compatability <https://github.com/Parquet/parquet-compatibility>`_
  16. project.
  17. requirements
  18. ============
  19. parquet-python has been tested on python 2.7, 3.6, and 3.7. It depends
  20. on ``pythrift2`` and optionally on ``python-snappy`` (for snappy compressed
  21. files, please also install ``parquet-python[snappy]``).
  22. getting started
  23. ===============
  24. parquet-python is available via PyPi and can be installed using
  25. `pip install parquet`. The package includes the `parquet`
  26. command for reading python files, e.g. `parquet test.parquet`.
  27. See `parquet --help` for full usage.
  28. Example
  29. -------
  30. parquet-python currently has two programatic interfaces with similar
  31. functionality to Python's csv reader. First, it supports a DictReader
  32. which returns a dictionary per row. Second, it has a reader which
  33. returns a list of values for each row. Both function require a file-like
  34. object and support an optional ``columns`` field to only read the
  35. specified columns.
  36. .. code:: python
  37. import parquet
  38. import json
  39. ## assuming parquet file with two rows and three columns:
  40. ## foo bar baz
  41. ## 1 2 3
  42. ## 4 5 6
  43. with open("test.parquet") as fo:
  44. # prints:
  45. # {"foo": 1, "bar": 2}
  46. # {"foo": 4, "bar": 5}
  47. for row in parquet.DictReader(fo, columns=['foo', 'bar']):
  48. print(json.dumps(row))
  49. with open("test.parquet") as fo:
  50. # prints:
  51. # 1,2
  52. # 4,5
  53. for row in parquet.reader(fo, columns=['foo', 'bar]):
  54. print(",".join([str(r) for r in row]))
  55. Todos
  56. =====
  57. - Support the deprecated bitpacking
  58. - Fix handling of repetition-levels and definition-levels
  59. - Tests for nested schemas, null data
  60. - Support reading of data from HDFS via snakebite and/or webhdfs.
  61. - Implement writing
  62. - performance evaluation and optimization (i.e. how does it compare to
  63. the c++, java implementations)
  64. Contributing
  65. ============
  66. Is done via Pull Requests. Please include tests with your changes and
  67. follow `pep8 <http://www.python.org/dev/peps/pep-0008/>`_.
  68. To run the tests you must install and execute ``tox`` (``pip install tox``) to
  69. run for all supported versions. If you want to run just for your current
  70. version, execute: ``pip install -r requirements-development.txt`` and then
  71. ``nosetests``.