Jenny Kim 64e31b7187 HUE-4931 [fb] Fix parquet decoding error on debug log messages 9 years ago
..
parquet 64e31b7187 HUE-4931 [fb] Fix parquet decoding error on debug log messages 9 years ago
test 0f2fde4775 HUE-5009 [core] Backport parquet-python Fix issues invoking struct methods. (#42) 9 years ago
MANIFEST.in b4aac26136 HUE-4726 [core] Upgrade parquet-python to 1.1 and include dependencies and backports 9 years ago
PKG-INFO b4aac26136 HUE-4726 [core] Upgrade parquet-python to 1.1 and include dependencies and backports 9 years ago
README.rst b4aac26136 HUE-4726 [core] Upgrade parquet-python to 1.1 and include dependencies and backports 9 years ago
setup.cfg b4aac26136 HUE-4726 [core] Upgrade parquet-python to 1.1 and include dependencies and backports 9 years ago
setup.py e895374b6b HUE-5009 [core] Backport parquet-python Fixes errors reported by flake8 and pylint 9 years ago
tox.ini e895374b6b HUE-5009 [core] Backport parquet-python Fixes errors reported by flake8 and pylint 9 years ago

README.rst

parquet-python
==============

parquet-python is a pure-python implementation (currently with only
read-support) of the `parquet
format `_. It comes with a
script for reading parquet files and outputting the data to stdout as
JSON or TSV (without the overhead of JVM startup). Performance has not
yet been optimized, but it's useful for debugging and quick viewing of
data in files.

Not all parts of the parquet-format have been implemented yet or tested
e.g. nested data—see Todos below for a full list. With that said,
parquet-python is capable of reading all the data files from the
`parquet-compatability `_
project.

requirements
============

parquet-python has been tested on python 2.7, 3.4, and 3.5. It depends
on ``thrift`` (0.9) and ``python-snappy`` (for snappy compressed files).

getting started
===============

parquet-python is available via PyPi and can be installed using
`pip install parquet`. The package includes the `parquet`
command for reading python files, e.g. `parquet test.parquet`.
See `parquet --help` for full usage.

Example
-------

parquet-python currently has two programatic interfaces with similar
functionality to Python's csv reader. First, it supports a DictReader
which returns a dictionary per row. Second, it has a reader which
returns a list of values for each row. Both function require a file-like
object and support an optional ``columns`` field to only read the
specified columns.

.. code:: python


import parquet
import json

## assuming parquet file with two rows and three columns:
## foo bar baz
## 1 2 3
## 4 5 6

with open("test.parquet") as fo:
# prints:
# {"foo": 1, "bar": 2}
# {"foo": 4, "bar": 5}
for row in parquet.DictReader(fo, columns=['foo', 'bar']):
print(json.dumps(row))


with open("test.parquet") as fo:
# prints:
# 1,2
# 4,5
for row in parquet.reader(fo, columns=['foo', 'bar]):
print(",".join([str(r) for r in row]))

Todos
=====

- Support the deprecated bitpacking
- Fix handling of repetition-levels and definition-levels
- Tests for nested schemas, null data
- Support reading of data from HDFS via snakebite and/or webhdfs.
- Implement writing
- performance evaluation and optimization (i.e. how does it compare to
the c++, java implementations)

Contributing
============

Is done via Pull Requests. Please include tests with your changes and
follow `pep8 `_.