Pcodec support

[mwlon]

I'm excited that Nimble has such flexible encodings/compressions! It shouldn't be too hard to add Pcodec, which generally gets much better compression ratio on numerical data than the traditional dictionary/rle/.../LZ approach. Compression and decompression speeds could benefit too. This seems important, especially for an ML-focused columnar format.

[pedroerp]

Hi @mwlon, I'm just reading about pcodec and it does seem like something that would be interesting to try out. Is this something you would like to do? We can help in ensure that Nimble has the right extensibility APIs for you to add it, and would be interested in experimental results.

Cc: @helfman @Yuhta

[mwlon]

I'm looking at the repo more now, but I don't see a spec doc. Does Nimble have a concept equivalent to Parquet's fine-grained "data pages"? If not, does it plan to have finer-grained pages in the future? This might affect a Pcodec implementation.

[Yuhta]

@mwlon Nimble has the same concept as Parquet page, which is called "chunk" inside one stream.