CSV Reader

[zuyu]

CSV file support in Velox? Using Apache Arrow?

Apache Gluten also uses Apache Arrow to read CSV files as velox backend: apache/incubator-gluten#5414

https://github.com/UWQuickstep/splitting uses Velox and Apache Arrow to parse CSV files in C++.

[majetideepak]

@pedroerp, @Yuhta Do you have any thoughts on this?

[Yuhta]

We have a TEXTFILE reader internally that has support for arbitrary delimiter, we will create an open source version some time next year based on the internal version (with some optimization).

[pedroerp]

+1. I think it's ok for now to reuse Arrow's for now, but ideally we should have a native implementation. We could either start with our own internal implementation (like @Yuhta pointed out), or by forking Arrow's or DuckDB's).

[yingsu00]

@Yuhta @pedroerp Can we know the timeline please? We need it to load TPCDS flat files. Is it possible to prioritize open sourcing it? Thanks!

[pedroerp]

The one we have internally has a few internal dependencies, so it's a bit more work than just moving the code around.

Another option would be to look into other open source (and potentially more efficient) readers. I know DuckDB has one that is supposed to be quite good.

[Yuhta]

The problem with other open source readers is that they do not provide special column support like $row_number or $row_id. We will have to change the source code to get the current row number somehow anyway, so they have to be vendored and become our own "special" version anyway.

[yingsu00]

So it seems it's easier to copy Arrow's implementation first. We can support $row_number or $row_id based on it. @zuyu has tried it and made it work.

[zhouyuan]

CC @jinchengchenghh