Dasel query crashes with go panic

[I-Al-Istannen]

Dasel query crashes with go panic

[I-Al-Istannen]

Streaming parsers are hard enough on there own - imagine trying to streaming readers + writers in 5 file types.

:^)

---

You can find the full file for e.g. Baden-Württemberg here. They offer data in the only two capable data formats this world has come up with, semicolon-separated comma separated values and XLSX. You can follow the "csv Datei" link (or click here directly).

The query is in my last comment, it just takes the json array, filters the entries to the ones in a specific municipality and then trims down the json object to the relevant elements.

[TomWright]

Hey @I-Al-Istannen,

Just for your information I got some time to dig at this. This is possible without hitting the panic you experienced:

./dasel -r csv -f <(iconv -f ISO-8859-1 -t UTF-8 ~/Downloads/file.csv) --csv-comma ';' -w json 'all().filterOr(equal(Gemeinde,Stuttgart),equal(Gemeinde,Mahlstetten)).mapOf(Gemeinde,Gemeinde,Haltestelle,Haltestelle,Haltestelle_lang,Haltestelle_lang,globaleID,globaleID)'
6.79s user 0.10s system 164% cpu 4.185 total

This shows just the fields:

• Gemeinde
• Haltestelle
• Haltestelle_lang
• globaleID

For any stations in:

• Stuttgart
• Mahlstetten

I have also merged the panic fix to master.

[TomWright]

Hey @I-Al-Istannen,

Thanks for raising this. I'll do a bit of digging.

Out of interest, what were you actually trying to get from the CSV?

[TomWright]

It seems this panic could occur when appending to a list of objects. I have fixed it here: #393 however it requires the deletion of some code that I need to verify is OK to be removed.

[I-Al-Istannen]

Thanks for raising this. I'll do a bit of digging.

Thanks :)

Out of interest, what were you actually trying to get from the CSV?

I was parsing stop data for the German train network, which of course only offers data in a large CSV file :P The full command for finding stops I was interested in looked something like this:

dasel -r csv -f <(iconv -f ISO-8859-1 -t UTF-8 haltestellen.csv) --csv-comma ';' -w json | jq 'map(select(.Gemeinde | contains("Name"))) | map({Gemeinde, Haltestelle, Haltestelle_lang, globaleID})'

I was trying to do the select with dasel first, but it was sufficiently different from jq that I didn't manage to cook something up quickly and then ran into the crash. dasel+jq worked just fine. Though the above command takes around 7 seconds, with dasel taking up 5 of them, but I don't really care about the performance :D

[TomWright]

Yeah dasel unfortunately loads the entire file into memory because of the change in file type. Streaming parsers are hard enough on there own - imagine trying to streaming readers + writers in 5 file types.

Performance aside, I'd be interested in a subset of your file that I can create some tests against to:
A: be sure no panic will occur
B: Potentially help you with your query without the need for jq (I'm always looking for ways to increase the general usefulness of dasel)

[I-Al-Istannen]

Ah, I see. You need to also duplicate all the fields in the map as it is key,value,key,value,…. I was trying more of a jq-like approach for the filter apparently 'equal([].Landkreis, "Sigmaringen")', but that already maps the array values so you do not need to further deal with the outer array here.

[TomWright]

That's right - I want a neater version of that map function for exactly this purpose. It works for now though