Daniel here, cofounder at Aperiodic. We live in microstucture data all day (trades, quotes, order books, etc) and tparquet got ours to about half the ZSTD size while its still just parquet.
Time series are everywhere though thats why we open sourced it!
Im really curious what else it could speed up (sensors, metrics, telemetry?). Contributions very welcome!
Another co-founder here, Mark - we're publicly backing PCodec and committing to getting it registered as an official compression codec. We've been using internally for years now, and want to contribute to its adaption. It has an excellent reference implementation from the original author, Martin Loncaric.
It's actually a very small fork of the standard parquet libraries, plus DuckDB, Polars, etc. plugins.
After Pcodec becoming an official parquet codec, the switch is only rewriting a couple of bytes in the parquet files. Until then, it's free disk space reduction, which may matter a lot if you have hundreds of terabytes of market data like we do.
Daniel here, cofounder at Aperiodic. We live in microstucture data all day (trades, quotes, order books, etc) and tparquet got ours to about half the ZSTD size while its still just parquet.
Time series are everywhere though thats why we open sourced it! Im really curious what else it could speed up (sensors, metrics, telemetry?). Contributions very welcome!
Another co-founder here, Mark - we're publicly backing PCodec and committing to getting it registered as an official compression codec. We've been using internally for years now, and want to contribute to its adaption. It has an excellent reference implementation from the original author, Martin Loncaric.
It's actually a very small fork of the standard parquet libraries, plus DuckDB, Polars, etc. plugins. After Pcodec becoming an official parquet codec, the switch is only rewriting a couple of bytes in the parquet files. Until then, it's free disk space reduction, which may matter a lot if you have hundreds of terabytes of market data like we do.