'plain' output with a defined field-separator

[Jamie-Landeg-Jones]

'plain' output with a defined field-separator

[philshafer]

I've been thinking about this and realized that with a little work, I can make a "csv" encoder that does what you need. Sort of. If I'm seeing what you need correctly.

So imagine:

• a customer "csv" encoder
• the ability to pass "options" to encoders
• the csv encoder supports three options:
  • path=some/path/in/the/hierarchy (emit only content under this specific hierarchy)
  • leafs=some.leafs.that.you.want (emit only leafs "some", "leafs", "that", "you", and "want")
  • no-header (so avoid the normal CSV header)

With no explicit "path", it emits all "lists". With no explict "leafs", it emits the set of fields under the first instance of the first list. Without the no-header, it emits a list of leafs that it's emitting.

Using the example of tests/core/test-01.c:

% test_01.test --libxo warn,encoder=csv:path=top/data/item
sku,name,sold,in-stock,on-order
GRO-000-415,gum,1412,54,10
HRD-000-212,rope,85,4,2
HRD-000-517,ladder,0,2,1
HRD-000-632,bolt,4123,144,42
% test_01.test --libxo warn,encoder=csv:path=top/data/item+leafs=name.sold
name,sold
gum,1412
rope,85
ladder,0
bolt,4123
water,17

So add the "no-header" to strip the header and "leafs=xxx" and you can trim the content to what you need.

Does that sounds reasonable and usable?

Check out commit a8e2070 (and the one before that). To see where I'm at. It's mostly coded, but needs some finishing touches and documentation.

Thanks,
Phil

[philshafer]

On Dec 30, 2017, at 6:26 PM, Jamie Landeg Jones ***@***.***> wrote: 'df' does indeed have libxo functionality, and whilst this would do the trick, parsing xml or json is overkill. I was hoping it would be possible to have something like: --libxo=delimited,delimiter="",record-separator=""

The problem is that we emit data that is hierarchical, so field-based access would be fragile. Better than text/regex, but still fragile. We have a set of shell functions that eat XML (sort of) and provide access to data. I’ll add this functionality to a future libxo release. Thanks, Phil

[Jamie-Landeg-Jones]

Apologies for the delay in replying.

That's brilliant - better than what I was hoping for (the ability to filter out leafs)

Hopefully this will get libxo added to more utilities - this is much better for shell scripts than parsing human orientated output!

I notice you've committed this to FreeBSD-Current. Are there plans to MFC to 12-stable?

Thanks again, Jamie

[philshafer]

Nov 11, 2019, at 12:48 PM, Jamie Landeg Jones <[email protected]> wrote:

That's brilliant - better than what I was hoping for (the ability to filter out leafs)

Thanks!

Hopefully this will get libxo added to more utilities - this is much better for shell scripts than parsing human orientated output!

Yes, I need to do some more conversions.

I notice you've committed this to FreeBSD-Current. Are there plans to MFC to 12-stable?

It’s not _entirely_ in. The Makefile for encoders/csv/ broke the build in ways that are beyond me. I’ve a fix from sjg@ but am waiting on confirmation from bapt@ before I can commit it. But I’ll likely do it anyway if I don’t hear something soon (>1wk already). It seems to work fine, but I’ve no interest in breaking things again. After that, I’d be happy for more testing before MFCing it. Thanks, Phil

[philshafer]

Oops. I need to recalibrate my “patience meter”. Turns out my email was only sent on Thursday night. Thanks, Phil

…

On Nov 11, 2019, at 2:37 PM, Phil Shafer ***@***.***> wrote: It’s not _entirely_ in. The Makefile for encoders/csv/ broke the build in ways that are beyond me. I’ve a fix from sjg@ but am waiting on confirmation from bapt@ before I can commit it. But I’ll likely do it anyway if I don’t hear something soon (>1wk already). It seems to work fine, but I’ve no interest in breaking things again. After that, I’d be happy for more testing before MFCing it.

[Jamie-Landeg-Jones]

Ha! You must have been terrible as a kid leading up to Christmas, and presents!

As for freebsd, I'm on 12-stable, I just installed the latest from git (seeing as you'd released it by the time I got around to it). It installed in /usr/local/... and seems ok to me (df --libxo=encoders=csv works fine)

I used your install instructions - I did have to use "gmake" rather than "make" to get it to install though.

I couldn't get "xo" command line to work with the csv encoder, but I haven't really used that command before, so put that down to my own stupidity!

I am a bit puzzled by why it's --html/--xml/--json/--text/--encoder=csv and not --encoder=html/xml/json/text/csv but it's not important!

By the way, talking of Christmas, I don't want to be the guy who tells you that santa doesn't exist.... but apparently, the rail-gauge size is a myth! https://www.snopes.com/fact-check/railroad-gauge-chariots/

[philshafer]

On Nov 12, 2019, at 4:10 AM, Jamie Landeg Jones ***@***.***> wrote: I used your install instructions - I did have to use "gmake" rather than "make" to get it to install though.

You may also need “env MAKE=gmake ../configure” to get it to work.

I couldn't get "xo" command line to work with the csv encoder, but I haven't really used that command before, so put that down to my own stupidity!

Strange; does “ldd `which xo`” say anything helpful? Or “xo —version”, which will report both the “xo” version and the libxo version.

I am a bit puzzled by why it's --html/--xml/--json/--text/--encoder=csv and not --encoder=html/xml/json/text/csv but it's not important!

xml/html/text/json are built in libxo. The “encoder” api allows dynamically loadable encoders to be add in an independent manner, so I can’t (and don’t want to) capture them as native options. “encoder=csv” tells libxo to load /usr/lib/libxo/encoders/csv.enc and use that library. It’s definitely awkward, but the truth is the rest of the options (to csv) bother me more, but it’s all unavoidable imho.

By the way, talking of Christmas, I don't want to be the guy who tells you that santa doesn't exist.... but apparently, the rail-gauge size is a myth! https://www.snopes.com/fact-check/railroad-gauge-chariots/ <https://www.snopes.com/fact-check/railroad-gauge-chariots/>

Thanks. You just cost me 1/2 an hour learning about railroad gauges and English rail tycoons, and other bits of useless information, like the English 7’0.25” gauge (which must have been an impressive engine) and the the fact that Toronto has two different gauges for streetcars. In the end, the story is mostly right, since Stephenson picked his width to match his house-drawn carriages, and his was the one picked as the standard. The "built by English expatriates” bit is rubbish; better to say the first engines were built by Englishmen. A completely fascinating waste of time ;^) Thanks, Phil

[philshafer]

On Nov 12, 2019, at 9:40 PM, Phil Shafer ***@***.***> wrote: I couldn't get "xo" command line to work with the csv encoder, but I haven't really used that command before, so put that down to my own stupidity!

I realized last night that you might be hitting the issue described in: https://libxo.readthedocs.io/en/latest/encoders.html#csv-comma-separated-values <https://libxo.readthedocs.io/en/latest/encoders.html#csv-comma-separated-values> Be aware that since the CSV encoder looks for data instances, when used with The “xo” Utility <https://libxo.readthedocs.io/en/latest/xo.html#xo>, the --instance option will be needed: % xo --libxo encoder=csv --instance foo 'The {:product} is {:status}\n' stereo "in route" product,status stereo,in route HTH, Phil

[philshafer]

The functionality has been added to the current libxo release.

Thanks,
Phil