瀏覽代碼

HUE-4344 [fb] Buffer reading parquet file and limit in size

If not, we get 1 WebHDFS call by byte.
Reading full 128 block is very slow, even 10MB to convert is too slow.
Would need to tweak the lib to actually apply a limit.
Romain Rigaux 9 年之前
父節點
當前提交
6e508da
共有 1 個文件被更改,包括 4 次插入1 次删除
  1. 4 1
      apps/filebrowser/src/filebrowser/views.py

+ 4 - 1
apps/filebrowser/src/filebrowser/views.py

@@ -762,8 +762,11 @@ def _read_avro(fhandle, path, offset, length, stats):
 
 def _read_parquet(fhandle, path, offset, length, stats):
     try:
+        size = 1 * 1024 * 1024 # Limit readable file, ParquetOptions.limit does not help
+        data = StringIO(fhandle.read(size)) # There is a footer
+
         dumped_data = StringIO()
-        parquet._dump(fhandle, ParquetOptions(), out=dumped_data)
+        parquet._dump(data, ParquetOptions(), out=dumped_data)
         dumped_data.seek(offset)
         return dumped_data.read()
     except: