Commit limit reached: 15,000 files for 'copy into table'.

[flaneur2020]

Summary

error:

code: 1001, message: Commit limit reached: 15,000 files for 'copy into table'. To handle more files, adjust 'CopyOption' with 'max_files='(e.g., 'max_files=10000') and perform several operations until all files are processed.

is there any background information on the design about this error? will it be more intuitive to just grab 10,000 files directly instead of raising errors?

[wubx]

This error is by design.

https://docs.databend.com/sql/sql-commands/dml/dml-copy-into-table#copyoptions

The maximum number of files allowed is 15,000, and no default value is set for max_files.

If the number of files handled by the copy process exceeds 15,000, an error will be outputted, prompting the user to define a value for max_files.

Please note that this process may need to be handled multiple times.

If a batch is too large, it can cause significant databend-meta instability.