Not all records from SAS file being loaded into DataFrame

[saurfang]

Not all records from SAS file being loaded into DataFrame

[thesuperzapper]

Can you provide a sas file (without sensitive data) which has this issue?

If possible, also provide a CSV version so we can see which rows get dropped .

[westonsankey]

Unfortunately I can't provide the SAS file, as it contains sensitive information. Was hoping that you have possibly heard of this issue and could point me in the right direction.

I tried setting the minSplitSize so that it forces a single partition, but that resulted in the same incorrect number of records.

[westonsankey]

I did some additional troubleshooting and this appears to be a Parso issue.

[thesuperzapper]

@westonsankey even though you can't provide that specific file, can you create a new file which also has that issue, and then submit it on the parso issue tracker?

[marchub]

I have a similar issue in that we read a large SAS file with 2.1.0 and it works but misses a few rows. It appears that the rows we miss at the ones that were appended or modified later manually via the SAS UI by users. Is this an issue with Parso missing these modified/appended rows? Or is this related to spark-sas7bdat?

@westonsankey did you find the source of your issue? It might be related. Thanks.

[thesuperzapper]

@marchub can you raise an new GitHub Issue, with an example file displaying the behaviour you reported?

Preferably, could you also upload a CSV version of file, so we can see what it should contain?