Some of my posts don't decode

[bluesky-social]

Some of my posts don't decode

[bnewbold]

I will need to look, but off the top of my head:

• this known issue with some posts not being output correctly (probably not a parse issue though)
• a month ago, the repo MST tree would be "complete" based on spidering CID links out of DAG-CBOR records. aka, every target of a CID would be in the repo. during a lexicon refactor, we now make "blobs" (eg, images) contain CID links. the blobs are not "in" the repo, they are stored separately. there is no current way to get a CAR file with repo and all the blobs as blocks. so if this code is still spidering it will fail. the ipfs CLI tool chokes in this particular way on newer records

[bradfitz]

(I haven't looked into what the panic is; just filing before I forget.)

[bradfitz]

Removing the recover, here's one of the panic stacks:

app.bsky.actor.profile/self
panic: unreachable

goroutine 1 [running]:
github.com/polydawn/refmt/json.(*Encoder).flushValue(0x140002104d0?, 0x101ebddf9?)
        github.com/polydawn/[email protected]/json/jsonEncoder.go:218 +0x204
github.com/polydawn/refmt/json.(*Encoder).Step(0x140002104d0, 0x14000562150)
        github.com/polydawn/[email protected]/json/jsonEncoder.go:121 +0x89c
github.com/polydawn/refmt/shared.TokenPump.Run({{0x1019c9000?, 0x14000032960?}, {0x1019c9080?, 0x140002104d0?}})
        github.com/polydawn/[email protected]/shared/pump.go:35 +0x80
main.cborToJson({0x14000afb824, 0x1da, 0x1da})
        github.com/bluesky-social/indigo/cmd/gosky/main.go:763 +0x33c
main.glob..func18.1({0x140000b2e00, 0x1b}, {{0x1400015d620?, 0x0?}})
        github.com/bluesky-social/indigo/cmd/gosky/main.go:640 +0x1b0
github.com/bluesky-social/indigo/repo.(*Repo).ForEach.func1({0x1, {0x140000b2e00, 0x1b}, {{0x1400015d620, 0x24}}, 0x0})
        github.com/bluesky-social/indigo/repo/repo.go:314 +0x4c
github.com/bluesky-social/indigo/mst.(*MerkleSearchTree).WalkLeavesFrom(0x14000133178?, {0x1019d2018, 0x140005886c0}, {0x0, 0x0}, 0x140001335d0)
        github.com/bluesky-social/indigo/mst/mst.go:936 +0x210
github.com/bluesky-social/indigo/mst.(*MerkleSearchTree).WalkLeavesFrom(0x140001332e8?, {0x1019d2018, 0x140005886c0}, {0x0, 0x0}, 0x140001335d0)
        github.com/bluesky-social/indigo/mst/mst.go:928 +0x110
github.com/bluesky-social/indigo/mst.(*MerkleSearchTree).WalkLeavesFrom(0x14000032230?, {0x1019d2018, 0x140005886c0}, {0x0, 0x0}, 0x140001335d0)
        github.com/bluesky-social/indigo/mst/mst.go:928 +0x110
github.com/bluesky-social/indigo/mst.(*MerkleSearchTree).WalkLeavesFrom(0x1400004c040?, {0x1019d2018, 0x140005886c0}, {0x0, 0x0}, 0x140001335d0)
        github.com/bluesky-social/indigo/mst/mst.go:928 +0x110
github.com/bluesky-social/indigo/repo.(*Repo).ForEach(0x140001c8000, {0x1019d1fa8, 0x140001a2008}, {0x0, 0x0}, 0x14000133700)
        github.com/bluesky-social/indigo/repo/repo.go:313 +0x148
main.glob..func18(0x14000527340)
        github.com/bluesky-social/indigo/cmd/gosky/main.go:623 +0x248
github.com/urfave/cli/v2.(*Command).Run(0x101f2d680, 0x14000527340, {0x14000527380, 0x4, 0x4})
        github.com/urfave/cli/[email protected]/command.go:274 +0x75c
github.com/urfave/cli/v2.(*Command).Run(0x140002286e0, 0x14000527140, {0x1400018e050, 0x5, 0x5})
        github.com/urfave/cli/[email protected]/command.go:267 +0x97c
github.com/urfave/cli/v2.(*App).RunContext(0x14000564000, {0x1019d1fa8?, 0x140001a2000}, {0x1400018e050, 0x5, 0x5})
        github.com/urfave/cli/[email protected]/app.go:332 +0x604
github.com/urfave/cli/v2.(*App).Run(...)
        github.com/urfave/cli/[email protected]/app.go:309
github.com/urfave/cli/v2.(*App).RunAndExitOnError(0x14000564000)
        github.com/urfave/cli/[email protected]/app.go:373 +0x48
main.run({0x1400006e768?, 0x100ec50e8?, 0x101ecd850?})
        github.com/bluesky-social/indigo/cmd/gosky/main.go:91 +0x5fc
main.main()
        github.com/bluesky-social/indigo/cmd/gosky/main.go:42 +0x30
exit status 2

[whyrusleeping]

oh yeah, this is a bug in the refmt library, it doesnt properly convert certain things. Need to just swap that out i think

[jmhodges]

I think this might be a generic problem with any CBOR to JSON conversion. For instance, CBOR allows for non-string keys in its maps and JSON libraries typically don't (because the JSON spec says not to). The panic in remft might be caused by that.

As evidence, I did a quick conversion from github.com/polydawn/refmt in cborToJson to github.com/fxamacker/cbor/v2 and encoding/json with a generic map[string]interface{} in the middle. That errored out because fxamacker/cbor returns the avatar and banner maps as map[interface{}][interface{}]. Those maps have only string keys, but a generic cbor parser can't assume that. Code in a branch on my fork

I believe there's a few options available:

1. Give up on the JSON encoding dream and print the raw Go types returned
2. Have an explicit set of types that can be pretty-printed and fallback to raw Go types or the original data when an unknown type is found (That is, parse the CBOR twice. Once to get the top-level $type value and the second to parse as the matching Go type for that value. It's unfortunate that the $type key is at the same level as the object's fields, but whaddaya gonna do)
3. Walk the generic maps to copy all of them and flip the key types to strings for maps and then error when the conversion is impossible. This assumes the cbor would never contain non-string keys for maps
4. Convince a cbor library like fxamacker/cbor to provide pretty-print tooling for generic data

[bnewbold]

FWIW, I'm in the middle of writing up better public specs for our data model (use of JSON and CBOR) and the Lexicon schema definition library. Those should end up in review over at https://github.com/bluesky-social/atproto-website/pulls in the next couple days, and might help nail down the space of data objects that need to be handled (or not).