Parser fails if uploading more than 331186 columns

[h2oai]

Parser fails if uploading more than 331186 columns

[exalate-issue-sync]

Sebastien Poirier commented: comments from internal discussion:

{noformat}michalkurka 4 days ago

There is a magic number, the default chunk size for Uploaded data is 4MB, your file is about 22MB, and it is split into 6 chunks. It looks like the header is longer than one chunk and the data is incorrectly parsed.

Something like this should only happen for datasets like yours, super-short and ultra-wide.

It is a bug that should be documented.{noformat}

[h2o-ops]

JIRA Issue Details

Jira Issue: PUBDEV-8898
Assignee: New H2O Bugs
Reporter: Sebastien Poirier
State: Open
Fix Version: N/A
Attachments: N/A
Development PRs: N/A

[wendycwong]

The error is caused by the header data takes more then one chunk of memory to store. In this case, if we can auto detect this condition and re-assign the chunk size, this should avoid the problem.

[wendycwong]

Could be related to this GH: #6610