README.rst 2.5 KB

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980818283
  1. parquet-python
  2. ==============
  3. parquet-python is a pure-python implementation (currently with only
  4. read-support) of the `parquet
  5. format <https://github.com/Parquet/parquet-format>`_. It comes with a
  6. script for reading parquet files and outputting the data to stdout as
  7. JSON or TSV (without the overhead of JVM startup). Performance has not
  8. yet been optimized, but it's useful for debugging and quick viewing of
  9. data in files.
  10. Not all parts of the parquet-format have been implemented yet or tested
  11. e.g. nested data—see Todos below for a full list. With that said,
  12. parquet-python is capable of reading all the data files from the
  13. `parquet-compatability <https://github.com/Parquet/parquet-compatibility>`_
  14. project.
  15. requirements
  16. ============
  17. parquet-python has been tested on python 2.7, 3.4, and 3.5. It depends
  18. on ``thrift`` (0.9) and ``python-snappy`` (for snappy compressed files).
  19. getting started
  20. ===============
  21. parquet-python is available via PyPi and can be installed using
  22. `pip install parquet`. The package includes the `parquet`
  23. command for reading python files, e.g. `parquet test.parquet`.
  24. See `parquet --help` for full usage.
  25. Example
  26. -------
  27. parquet-python currently has two programatic interfaces with similar
  28. functionality to Python's csv reader. First, it supports a DictReader
  29. which returns a dictionary per row. Second, it has a reader which
  30. returns a list of values for each row. Both function require a file-like
  31. object and support an optional ``columns`` field to only read the
  32. specified columns.
  33. .. code:: python
  34. import parquet
  35. import json
  36. ## assuming parquet file with two rows and three columns:
  37. ## foo bar baz
  38. ## 1 2 3
  39. ## 4 5 6
  40. with open("test.parquet") as fo:
  41. # prints:
  42. # {"foo": 1, "bar": 2}
  43. # {"foo": 4, "bar": 5}
  44. for row in parquet.DictReader(fo, columns=['foo', 'bar']):
  45. print(json.dumps(row))
  46. with open("test.parquet") as fo:
  47. # prints:
  48. # 1,2
  49. # 4,5
  50. for row in parquet.reader(fo, columns=['foo', 'bar]):
  51. print(",".join([str(r) for r in row]))
  52. Todos
  53. =====
  54. - Support the deprecated bitpacking
  55. - Fix handling of repetition-levels and definition-levels
  56. - Tests for nested schemas, null data
  57. - Support reading of data from HDFS via snakebite and/or webhdfs.
  58. - Implement writing
  59. - performance evaluation and optimization (i.e. how does it compare to
  60. the c++, java implementations)
  61. Contributing
  62. ============
  63. Is done via Pull Requests. Please include tests with your changes and
  64. follow `pep8 <http://www.python.org/dev/peps/pep-0008/>`_.