Fix fst write routines

[mpadge]

Fix fst write routines

[rafapereirabr]

Hi @mpadge . Quick question. Why did you choose to use fst instead of parquet?

[mpadge]

Hiya Rafa! Chose just because it's more lightweight, and arrow for this workflow is overkill. Everything here is internally indexed by city name; arrow would enable city to be used an index, but then you'd lose oversight of what individual files were. Does that make sense?

[rafapereirabr]

Yes, it makes perfect sense! Thanks for clarifying! I'm following your work on m4ra with great interest!

[mpadge]

Have to re-open because fst still produces unreliable values. The package is clearly going to have to be dumped here, because the unreliability breaks all analyses here. It seems that the row-wise ordering in different columns is randomly rearranged, so that rows are not recovered, but include random data mixed from other rows.

[mpadge]

Can't use arrow, because even with current dev version, it errors because

Error in rawToChar(out): long vectors not supported yet: raw.c:68

That's on a data frame with 6,5 million rows, 21 columns. I guess arrow still primarily envisions numerical data ... so it's back to plain old saveRDS. That will slow these routines down somewhat, but at least everything will once again be reliable.

[mpadge]

Confirmed that full accuracy has been regained, and load times even for the data.frame above with 6.5M rows are still well under 1 minute, which is okay