When using external JSON materalization: bumping into default maximum_object_size limit.

[firewall413]

dbt-duckdb/dbt/include/duckdb/macros/materializations/external.sql

Line 52 in ab16970

create or replace view {{ intermediate_relation }} as (

When materializing a table to a JSON file bigger than 30MB, we bump into the following:

Invalid Input Error: "maximum_object_size" of 16777216 bytes exceeded while reading file "s3://xxxxxx.json" (>33554428 bytes). Try increasing "maximum_object_size".

This is likely due to the select * from '{{ read_location }}' trying to build a view with the default read_json_auto() and default options params.

Would it be possible to pass the read_json/read_parquet/read_csv functions and their options params?

[jwills]

Yes, I think; there would need to be a PR that modified this function to let you override more of the defaults using the rendered_options dictionary (like we do for external materializations that use partitioning): https://github.com/duckdb/dbt-duckdb/blob/master/dbt/adapters/duckdb/impl.py#L166