Should extension metadata tag be more specific?

[DrChainsaw]

Should extension metadata tag be more specific?

[ericphanson]

Just below they suggest name spacing the value passed there: https://arrow.apache.org/docs/format/Columnar.html#extension-types. That bit reads to me like it is not a placeholder, but rather the customization is in the value (not the key).

[DrChainsaw]

But the metadata is a dict, so the namespacing they suggest would be pointless if only applied to values since they will be overwritten.

[ericphanson]

Maybe we can check against another implementation by seeing the metadata produced by an extension type there, e.g. following https://arrow.apache.org/docs/python/generated/pyarrow.ExtensionType.html#pyarrow.ExtensionType.extension_name.

It doesn't seem clear if a column can have more than one extension type though. It could be there's only 1 key on purpose so that different implementations can share that key to define an extension to the arrow spec overall (e.g. we if we all agree what a foo is, we define an extension name for that, serialize that metadata, and then read it in as a foo when possible). Which maybe then means your suggestion is that arrow-julia shouldn't be using "extension types" specifically for metadata that is only used by that implementation, and should use other keys for that.

Would appreciate any feedback from someone who understands the spec better

[DrChainsaw]

After searching through the arrow repo after arrow:extension is seems like you might be right. Here it is defined as a const in the cpp code for example and I could not find any trace of it being manipulated or changed anywhere (which would be a quite strange thing to do as well).

The c code seems to accept the metadata as a vector of pairs through, so it would in theory allow for multiple identical keys, but python, java and Julia use dicts so there is no way to have it through any of them.

It is a bit unclear to me what the ExtensionType stuff does in arrow and what one gets for buying in to it. However, most things points to it being a mechanism for your foo example.

I close the issue now since my initial understanding of the extension metadata was incorrect and I don't think there is any action needed here. Just reopen or open another issue if there is a point about late conversion stuff (e.g. String <-> Symbol) not fitting into the definition of extensions.