bluesky-social/indigo

Go source code for Bluesky's atproto services.

indigo: atproto libraries and services in golang

Some Bluesky software is developed in Typescript, and lives in the bluesky-social/atproto repository. Some is developed in Go, and lives here.

What is in here?

Go Services:

• bigsky (README): "Big Graph Service" (BGS) reference implementation, running at bsky.network
• palomar (README): fulltext search service for https://bsky.app

Go Packages:

⚠️ All the packages in this repository are under active development. Features and software interfaces have not stabilized and may break or be removed.

Package	Docs
api/atproto: generated types for com.atproto.* Lexicons
api/bsky: generated types for app.bsky.* Lexicons
atproto/crypto: crytographic signing and key serialization
atproto/identity: DID and handle resolution
atproto/syntax: string types and parsers for identifiers
mst: Merkle Search Tree implementation
repo: account data storage
xrpc: HTTP API client

The TypeScript reference implementation, including PDS and bsky AppView services, is at bluesky-social/atproto. Source code for the Bluesky Social client app (for web and mobile) can be found at bluesky-social/social-app.

Development Quickstart

First, you will need the Go toolchain installed. We develop using the latest stable version of the language.

The Makefile provides wrapper commands for basic development:

make build
make test
make fmt
make lint

Individual commands can be run like:

go run ./cmd/bigsky

The HACKING file has a list of commands and packages in this repository and some other development tips.

What is atproto?

not to be confused with the AT command set or Adenosine triphosphate

The Authenticated Transfer Protocol ("ATP" or "atproto") is a decentralized social media protocol, developed by Bluesky PBC. Learn more at:

• Overview and Guides 👈 Best starting point
• Github Discussions 👈 Great place to ask questions
• Protocol Specifications
• Blogpost on self-authenticating data structures

The Bluesky Social application encompasses a set of schemas and APIs built in the overall AT Protocol framework. The namespace for these "Lexicons" is app.bsky.*.

Contributions

While we do accept contributions, we prioritize high quality issues and pull requests. Adhering to the below guidelines will ensure a more timely review.

Rules:

• We may not respond to your issue or PR.
• We may close an issue or PR without much feedback.
• We may lock discussions or contributions if our attention is getting DDOSed.
• We do not provide support for build issues.

Guidelines:

• Check for existing issues before filing a new one, please.
• Open an issue and give some time for discussion before submitting a PR.
• Issues are for bugs & feature requests related to the golang implementation of atproto and related services.
  • For high-level discussions, please use the Discussion Forum.
  • For client issues, please use the relevant social-app repo.
• Stay away from PRs that:
  • Refactor large parts of the codebase
  • Add entirely new features without prior discussion
  • Change the tooling or libraries used without prior discussion
  • Introduce new unnecessary dependencies

Remember, we serve a wide community of users. Our day-to-day involves us constantly asking "which top priority is our top priority." If you submit well-written PRs that solve problems concisely, that's an awesome contribution. Otherwise, as much as we'd love to accept your ideas and contributions, we really don't have the bandwidth.

Are you a developer interested in building on atproto?

Bluesky is an open social network built on the AT Protocol, a flexible technology that will never lock developers out of the ecosystems that they help build. With atproto, third-party can be as seamless as first-party through custom feeds, federated services, clients, and more.

If you're a developer interested in building on atproto, we'd love to email you a Bluesky invite code. Simply share your GitHub (or similar) profile with us via this form.

License

This project is dual-licensed under MIT and Apache 2.0 terms:

• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)
• Apache License, Version 2.0, (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)

Downstream projects and end users may chose either license individually, or both together, at their discretion. The motivation for this dual-licensing is the additional software patent assurance provided by Apache 2.0.

indigo's Issues

pds: `limit` not enforced on several endpoints, including `getTimeline`

Queried with limit=100, got back 108 records, while testing with fakermaker.

Not a huge priority, but note that this would result in un-boounded return sets, and slow SQL queries, as repos get larger.

pds: Passwords stored as raw text

By default, raw passwords are stored in data/laputa/pds.sqlite.
I think we should use password-hashing or something.

cborgen: support for 'float' types

We aspire to support the full IPLD data model, even if floats are considered a poor practice in content-addressed systems.

Upstream (cbor-gen) issue: whyrusleeping/cbor-gen#80

com.atproto.repo.describeRepo returns something unparseable by indigo

I've got the following response on this endpoint:

{
  "handle": "go-bluesky-tester.bsky.social",
  "did": "did:plc:wflozfzpewefv46qof26vbzm",
  "didDoc": {
    "@context": [
      "https://www.w3.org/ns/did/v1",
      "https://w3id.org/security/suites/secp256k1-2019/v1"
    ],
    "id": "did:plc:wflozfzpewefv46qof26vbzm",
    "alsoKnownAs": [
      "at://go-bluesky-tester.bsky.social"
    ],
    "verificationMethod": [
      {
        "id": "#atproto",
        "type": "EcdsaSecp256k1VerificationKey2019",
        "controller": "did:plc:wflozfzpewefv46qof26vbzm",
        "publicKeyMultibase": "zQYEBzXeuTM9UR3rfvNag6L3RNAs5pQZyYPsomTsgQhsxLdEgCrPTLgFna8yqCnxPpNT7DBk6Ym3dgPKNu86vt9GR"
      }
    ],
    "service": [
      {
        "id": "#atproto_pds",
        "type": "AtprotoPersonalDataServer",
        "serviceEndpoint": "https://bsky.social"
      }
    ]
  },
  "collections": [
    "app.bsky.actor.profile",
    "app.bsky.feed.post",
    "app.bsky.graph.follow"
  ],
  "handleIsCorrect": true
}

The indigo parser chokes on it however because the diddoc does not have a type field and the parser wants one. Unsure who's at fault here, just that I don't know how to fix it.

PDS: missing features

Not all endpoints and features are implemented in the current PDS. Some notable missing things (turned up by fakermaker test; you can all search for panic under pds/handlers.go).

Some notable/priority features:

• blob uploads
• mutes
• repo list records
• repo sync commands (new)

XRPC client: retry with backoffs on 5xx

Particularly for beemo, getting some XRPC ERROR 502: 502 Bad Gateway.

Segfault during VerifyUserSignature

I run into this segfault consistently when pointing bigsky at the PDS dev environment setup. I am running bigsky on 9a357fe with the following command:

./bigsky --aggregation=false --ssl-events=false --plc-host="http://localhost:2582"

Output:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x100220d86]

goroutine 63 [running]:
math/big.(*Int).Sign(...)
        /usr/local/go/src/math/big/int.go:41
crypto/ecdsa.(*nistCurve[...]).pointFromAffine(0x100f86be0, 0x0, 0x0)
        /usr/local/go/src/crypto/ecdsa/ecdsa.go:575 +0x46
crypto/ecdsa.verifyNISTEC[...](0xc00061ee40, 0xc000333568, {0xc000333508, 0x20, 0x20}, {0xc000616600, 0x0?, 0x0?})
        /usr/local/go/src/crypto/ecdsa/ecdsa.go:494 +0xc5
crypto/ecdsa.VerifyASN1(0xc000333568, {0xc000333508, 0x20, 0x20}, {0xc000616600, 0x48, 0x60})
        /usr/local/go/src/crypto/ecdsa/ecdsa.go:478 +0x285
crypto/ecdsa.Verify(0x11000371f53b00?, {0xc000333508, 0x20, 0x20}, 0xc0007366f0?, 0xc00033a7c0)
        /usr/local/go/src/crypto/ecdsa/ecdsa_legacy.go:126 +0x10b
github.com/whyrusleeping/go-did.(*PubKey).Verify(0xc00033a780, {0xc0007366f0, 0x24, 0x30}, {0xc000618380, 0x40, 0x40})
        /Users/devinivy/go/pkg/mod/github.com/whyrusleeping/[email protected]/key.go:158 +0x2f9
github.com/bluesky-social/indigo/indexer.(*KeyManager).VerifyUserSignature(0xc0006125f0?, {0x100f89d60, 0xc00061b020}, {0xc00062a3c0, 0x20}, {0xc000618380, 0x40, 0x40}, {0xc0007366f0, 0x24, ...})
        /project/indigo/indexer/keymgr.go:37 +0x185
github.com/bluesky-social/indigo/repomgr.(*RepoManager).HandleExternalUserEvent(0xc00071ffc0, {0x100f89cf0, 0xc000120008}, 0x1, 0x1, {0xc00071c7a0, 0x20}, 0x0, {0xc0005ee000, 0x2b3, ...})
        /project/indigo/repomgr/repomgr.go:548 +0x4ec
github.com/bluesky-social/indigo/bgs.(*BGS).handleFedEvent(0xc0002bf4f0, {0x100f89cf0, 0xc000120008}, 0xc000308090, 0xc0005ee001?)
        /project/indigo/bgs/bgs.go:269 +0x434
github.com/bluesky-social/indigo/bgs.(*Slurper).handleConnection.func1(0xc0005d8300)
        /project/indigo/bgs/fedmgr.go:139 +0x1a7
github.com/bluesky-social/indigo/events.HandleRepoStream({0x100f89cb8?, 0xc0005c2a50?}, 0xc000455e40, 0xc000333df0)
        /project/indigo/events/consumer.go:63 +0x213
github.com/bluesky-social/indigo/bgs.(*Slurper).handleConnection(0xc000300990, 0xc000308090, 0x0?, 0xc000323ea8)
        /project/indigo/bgs/fedmgr.go:135 +0x117
github.com/bluesky-social/indigo/bgs.(*Slurper).subscribeWithRedialer(0xc000300990, 0xc000308090)
        /project/indigo/bgs/fedmgr.go:105 +0x370
created by github.com/bluesky-social/indigo/bgs.(*Slurper).SubscribeToPds
        /project/indigo/bgs/fedmgr.go:70 +0x378

Some of my posts don't decode

gosky list --all works on most of my repo, but some values don't parse:

% gosky list --all --values $(gosky handle resolve bradfitz.com) | grep -A 1 panic
panic:  unreachable
bad blob: a5652474797065766170702e62736b792e6163746f722e70726f66696c6566617661746172a463726566d82a58250001551220cb04502240be648abdc77c7e9a5ef1750a2b0943ab6bb5f6b4fc7a16dc59092a6473697a651a0001afd965247479706564626c6f62686d696d65547970656a696d6167652f6a7065676662616e6e6572a463726566d82a5825000155122050509cd8bd1f3e4834eb5d369c712d6bd84475a795d5f309fbd26dabc07476386473697a651a0002b1fc65247479706564626c6f62686d696d65547970656a696d6167652f6a7065676b6465736372697074696f6e78b54920646f20636f6d7075746572732e204d61727269656420746f20407261646b61742e6669747a7061742e636f6d2e205468726565206b6964732e20586f6f676c65722e200a476f202823676f6c616e6729207465616d20323031307e323032302e204d616465204c6976654a6f75726e616c2c204f70656e49442c206d656d6361636865642e2043757272656e746c79206174205461696c7363616c65206d616b696e672057697265477561726420656173792e6b646973706c61794e616d657042726164204669747a7061747269636b
--
panic:  unreachable
bad blob: a5647465787478254c6f6f6b73206c696b6520746869732077617320796f757220666c696768743f2046756e21652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a58250001551220773ea1e3c22c394429a71709b216ded26d0c939890351ddd37bbdb45dee5ad306473697a651a000f140065247479706564626c6f62686d696d65547970656a696d6167652f6a706567657265706c79a264726f6f74a263636964783b62616679726569673277653373676d3663653278356e666a75717a6171327169653571666c7963366b7068777a376368377535736632756366666163757269784661743a2f2f6469643a706c633a6966327475673562756335613363727a32643269323469332f6170702e62736b792e666565642e706f73742f336a74713371376235657a326d66706172656e74a263636964783b62616679726569673277653373676d3663653278356e666a75717a6171327169653571666c7963366b7068777a376368377535736632756366666163757269784661743a2f2f6469643a706c633a6966327475673562756335613363727a32643269323469332f6170702e62736b792e666565642e706f73742f336a74713371376235657a326d696372656174656441747818323032332d30342d32345431373a31383a32332e3739385a
--
panic:  unreachable
bad blob: a4647465787464f09f9299652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a5825000155122055fcd00bd248cdc46dfed852b93aab637706e8daf716c43504da1dc02ad71ed96473697a651a000d4fa565247479706564626c6f62686d696d65547970656a696d6167652f6a706567696372656174656441747818323032332d30342d32345432323a30363a30312e3738355a
--
panic:  unreachable
bad blob: a56474657874787e4f682c206e6963652c206d7920667269656e642040627261646669747a2e69732e74657374696e672e686f772e62656175746966756c2e6f662e612e757365726e616d652e796f752e63616e2e686176652e302e302e662e652e622e662e322e302e362e322e6970362e61727061206a6f696e656420426c7565736b7921652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a58250001551220526572a012f96be8743f6388062b9127a0fc332b8f461ce29c85cd42a7e32d636473697a651a0004e8e165247479706564626c6f62686d696d65547970656a696d6167652f6a7065676666616365747381a3652474797065776170702e62736b792e72696368746578742e666163657465696e646578a26762797465456e64186e696279746553746172741468666561747572657381a26364696478206469643a706c633a736637786a68736a3461666368326769366e743633636c65652474797065781f6170702e62736b792e72696368746578742e6661636574236d656e74696f6e696372656174656441747818323032332d30342d32355430333a35303a31312e3731315a
--
panic:  unreachable
bad blob: a5647465787478b45475726e73206f7574206d792077696665206f66206e6561726c7920736576656e2079656172732c20407261646b61742e6669747a7061742e636f6d2c206469646e2774206b6e6f772074686174206d7920706572736f6e616c207765627369746520697320627261646669747a2e636f6d2e0a0a5468616e6b7320426c7565736b7920666f72206c657474696e67207573206c6561726e206d6f72652061626f75742065616368206f746865722120f09f9882652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a58250001551220e71409e2ab359e01241ccc7705b84ecac6ff638e11e071ed8cbd7211e61cf6516473697a651a0001362f65247479706564626c6f62686d696d65547970656a696d6167652f6a7065676666616365747382a3652474797065776170702e62736b792e72696368746578742e666163657465696e646578a26762797465456e64183c69627974655374617274182968666561747572657381a26364696478206469643a706c633a6f6b6b33656b7575736b7062736476777972616e72726c71652474797065781f6170702e62736b792e72696368746578742e6661636574236d656e74696f6ea265696e646578a26762797465456e64187269627974655374617274186668666561747572657381a2637572697468747470733a2f2f627261646669747a2e636f6d652474797065781c6170702e62736b792e72696368746578742e6661636574236c696e6b696372656174656441747818323032332d30342d32355431373a35393a34372e3739335a
--
panic:  unreachable
bad blob: a4647465787468f09f9299f09f8cb8652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a58250001551220ee70559a326bc006796f3371d9dece0f31c318bf45569ad87d1445cbb9b79d486473697a651a000ee56a65247479706564626c6f62686d696d65547970656a696d6167652f6a706567696372656174656441747818323032332d30342d32355432303a35353a30382e3538395a
--
panic:  unreachable
bad blob: a564746578747873546f64617920406361747a6b6f726e2e62736b792e736f6369616c20706f73746564207468697320696e206f6e65206f66206f757220766172696f757320536c61636b2073686974706f7374696e67206368616e6e656c7320617420407461696c7363616c652e636f6d202e2e2e20f09f9882652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a582500015512205b3dbac27e31007345f2b54ca68dc5b6430a245a5a814c60494c59992318cf296473697a651a0006220365247479706564626c6f62686d696d65547970656a696d6167652f6a7065676666616365747382a3652474797065776170702e62736b792e72696368746578742e666163657465696e646578a26762797465456e64181b696279746553746172740668666561747572657381a26364696478206469643a706c633a7a6e6f71776568777369616563656f757534747471366277652474797065781f6170702e62736b792e72696368746578742e6661636574236d656e74696f6ea3652474797065776170702e62736b792e72696368746578742e666163657465696e646578a26762797465456e64186a69627974655374617274185c68666561747572657381a26364696478206469643a706c633a746b633378697961346b37757978336a7365693362703635652474797065781f6170702e62736b792e72696368746578742e6661636574236d656e74696f6e696372656174656441747818323032332d30342d32365431383a34393a30392e3833375a
--
panic:  unreachable
bad blob: a46474657874781a4f682c20746865206a6f7973206f6620696e66616e74732e2e2e652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a582500015512202bf480f10f5fd233413a06177473f6c4d2d6ea40a8bfbf9122b675067a24945d6473697a651a000d21f565247479706564626c6f62686d696d65547970656a696d6167652f6a706567696372656174656441747818323032332d30342d32375431373a30393a31322e3038345a
--
panic:  unreachable
bad blob: a4647465787479012e4920686164206120636f6e74726163746f72206f757420646f696e672061206d61696e74656e616e6365207669736974206f6e206f757220686561742070756d70732e2048652068616420746f20676f207468726f756768206d79206f666669636520746f2067657420746f2074686520726f6f6620756e697420616e6420667265616b6564206f7574207768656e20686520736177206d79206465736b2c20667265657a696e6720616e642073746172696e67206174206d652c2061736b696e673a0a0a2257687920646f20796f7520686176652061206b6e696665206f6e20796f7572206465736b3f213f220a0a492068616420746f206578706c61696e20776861742061206c6574746572206f70656e6572206973202620686f7720697420776f726b65642e20f09f9882652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a582500015512201d115053c4bf8de9fc6bb243a5b23e82ac20d4f01dfce4ae31a71a0570752b3b6473697a651a000e383a65247479706564626c6f62686d696d65547970656a696d6167652f6a706567696372656174656441747818323032332d30342d32375431383a33373a34332e3735315a
--
panic:  unreachable
bad blob: a4647465787478d1496620796f75206e65656420746f207761726e206d6520696e20616476616e6365207768656e20796f752068616e64206d652074686520515220636f6465206d656e7520746861742069742070726f6261626c7920776f6e2774207363616e2c206d6179626520697427732074696d6520666f72206e657720515220636f646573207468617420617265206d6f726520626f72696e673f0a0a427574206f6e207468652062726967687420736964652c207468697320776561746865722120416e64206265657221204f7574736964652e652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a582500015512202ba9760cd191f411c3a8f31700c4ced2cd7a0f61a4e3e0d0c74488b05b3732aa6473697a651a000eecbb65247479706564626c6f62686d696d65547970656a696d6167652f6a706567696372656174656441747818323032332d30342d32375432323a32313a32372e3036335a
--
panic:  unreachable
bad blob: a5647465787479012c49207361772070656f706c6520686572652064697363757373696e6720656e6a6f79696e67206c696b696e6720227468697273742220706f737473206f6e207768696c65206c696b65732061726520707269766174652e0a0a4265206361726566756c2e2054686579277265206e6f742e20546865207765622f61707073206a75737420646f6e27742073686f7720796f752c206275742069742773207075626c69632e0a0a652e672e2048657265206172652040616e696c646173682e636f6d2773206c696b65732e204c6173742055524c2069730a0a68747470733a2f2f73746167696e672e62736b792e6170702f70726f66696c652f6469643a706c633a6b646a6d336d7776627662766171746236723369743571712f706f73742f336a75666d7276677069753270652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a582500015512200a15cfd7eb305eef376e743752c05754c6bbcabf28c801aac0d347dc49cd067e6473697a651a000ba21f65247479706564626c6f62686d696d65547970656a696d6167652f6a7065676666616365747382a3652474797065776170702e62736b792e72696368746578742e666163657465696e646578a26762797465456e6418c16962797465537461727418b468666561747572657381a26364696478206469643a706c633a73673265326b71647364703271327a6c3434747861646270652474797065781f6170702e62736b792e72696368746578742e6661636574236d656e74696f6ea265696e646578a26762797465456e6419012c6962797465537461727418d868666561747572657381a263757269785468747470733a2f2f73746167696e672e62736b792e6170702f70726f66696c652f6469643a706c633a6b646a6d336d7776627662766171746236723369743571712f706f73742f336a75666d7276677069753270652474797065781c6170702e62736b792e72696368746578742e6661636574236c696e6b696372656174656441747818323032332d30342d32385430333a32383a32362e3832355a
--
panic:  unreachable
bad blob: a5647465787460652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a582500015512202751a5b99841ef99b3b195e750cb2f6e8b8f47b12f5b99781a09326991f30b076473697a651a000142b465247479706564626c6f62686d696d65547970656a696d6167652f6a706567657265706c79a264726f6f74a263636964783b6261667972656968716b62707236796171376a6c617a666637636763336863326468706a6b616b79677765723362657a676f7869786d763567356163757269784661743a2f2f6469643a706c633a377232667933623475376d6d6e68676264786e666c6f76762f6170702e62736b792e666565642e706f73742f336a75666f636a6763716e327066706172656e74a263636964783b626166797265696470656a3261736b737566716f79647668787736796a66376d33756c71616336787a6174666177666a696e377970356c6977336163757269784661743a2f2f6469643a706c633a3634727976757271777a72366c6a6e3576376c776e696e682f6170702e62736b792e666565642e706f73742f336a75666f6b32656468353270696372656174656441747818323032332d30342d32385430333a33353a32332e3136385a

/cc @whyrusleeping @bnewbold

legacy blob parsing broken

Still investigating, but can trigger errors like:

decoding xrpc response: parsing legacy blob: json: Unmarshal(nil *util.LegacyBlob)

Debugging this right now

cmd/gosky: plc flag not defined for didGet

Can I haz:

		&cli.StringFlag{
			Name:    "plc",
			Value:   "https://plc.staging.bsky.dev",
			EnvVars: []string{"BSKY_PLC_URL"},
		},

in https://github.com/bluesky-social/indigo/blob/main/cmd/gosky/main.go#L155, otherwise https://github.com/bluesky-social/indigo/blob/main/cmd/gosky/util/util.go#L24 is a noop and the command goes boom

$ go run ./cmd/gosky did get @karalabe.bsky.social
Get "/@karalabe.bsky.social": unsupported protocol scheme ""
exit status 1

Not easy to build on cross-platform

indigo uses go-did. Currently, go-did use cgo module ipsn/go-secp256k1. So this is not easy to build arm64 binary release on GitHub Actions. I created pull-request to switch another pure Go module for go-did. Could you please take a look?

whyrusleeping/go-did#2

Missing GraphGetBlocks function

Looking to use some of the functions via indigo but saw that GraphGetBlocks seems to be missing. I would expect that function to use app.bsky.graph.getBlocks (https://atproto.com/lexicons/app-bsky-graph#appbskygraphgetblocks) similar as it is done in the GraphGetMutes -> https://github.com/bluesky-social/indigo/blob/main/api/bsky/graphgetMutes.go

I saw the GraphGetMutes is generated based on the comment in the first line but I'm unsure if the missing function is due to cmd/lexgen not being run recently, but it also seems last changes to https://github.com/bluesky-social/atproto/blob/main/lexicons/app/bsky/graph/getBlocks.json were 3 weeks back, when it was last run. (+/- a day maybe).

refactors to event code and BGS to be flexible to labelmaker use case

• #490
• find ways to reduce code duplication between repoappend and label events (eg, single EventManager?)

small refactors and docs needed

Keeping a single issue to track these small things:

• repomgr.InitNewActor does some com.atproto things and some app.bsky things. would be good to separate those steps. for example, labeling service is not an app.bsky actor, doesn't need a profile record created
• in lexgen, replace package schemagen with actually-meaningful package names. for example, comatproto/appbsky, or just atproto/bsky
• don't use util or api as package names, too generic. I think this is a common style guide recommendation
• consider renaming key to indicate it is a did:key thing. possibly even spin out in to separate repo/package
• consider moving a bunch of lex/util in to util (or whatever renamed to)
• api/atproto.go and api/bsky.go record types should be deprecated and replaced with api/atproto/*.go and api/bsky/*.go types, respectively
• move explicitly public API packages to pkg/, and explicitly internal packages to internal/ (maybe opinionated/controversial?). feels to me that this helps with "scaling" as a monorepo (containing multiple commands and libraries) (update: why is against internal/)
• add CLI/PDS/PLC integration test coverage, possibly using cmdtest or icmd
• deprecate/remove ATProto (in api/atproto.go) and BskyApp (in api/bsky.go). Use xrpc.Client and lexgen types instead

I want to participate in this project

I want to be involved in this project, where do I start? Aren't there any good first issues?

run some tests against postgresql in CI

Using ephemeral/parallel postgresql databases (eg, ramdisk). Goal is to get additional confidence that we are testing the same configuration that we deploy in production and staging. Eg, should be easy to test against different versions of postgresql, and surface any possible differences between sqlite and postgresql DB drivers.

Should be fairly simple using containerization in CI.

A related goal is to make it easy to develop and run tests locally.

flaky test failures

I've seen some occasional test failures recently, both locally and in CI. Re-running usually fixes. I didn't capture a link or output until just now. The output is very verbose, below is just the snip at the end.

2023/02/11 21:16:56 /home/bnewbold/code/indigo/indexer/indexer.go:569 record not found
[0.047ms] [rows:0] SELECT * FROM `pds` WHERE id = 0 AND `pds`.`deleted_at` IS NULL ORDER BY `pds`.`id` LIMIT 1
2023-02-11T21:16:56.079-0800	ERROR	indexer	indexer/crawler.go:157	failed to perform repo crawl of "did:plc:4475991f29aeeeee": expected to find pds record in db for crawling one of their users: record not found

2023/02/11 21:16:56 /home/bnewbold/code/indigo/indexer/indexer.go:528 record not found
[0.313ms] [rows:0] SELECT * FROM `feed_posts` WHERE (rkey = "3jojbfhdi34m" AND author = (SELECT `id` FROM `actor_infos` WHERE did = "did:plc:5b48e5c9059a135b" AND `actor_infos`.`deleted_at` IS NULL)) AND `feed_posts`.`deleted_at` IS NULL ORDER BY `feed_posts`.`id` LIMIT 1
--- FAIL: TestBGSMultiGap (0.42s)
    integ_test.go:190: record not found
FAIL
FAIL	github.com/bluesky-social/indigo/testing	5.778s

lexgen: handle types without "main" in defs

Running lexgen on current atproto lexicons fails on the com.atproto.admin section:

failed to process schema "/home/bnewbold/bsky/atproto/lexicons/com/atproto/admin/moderationAction.json": schema "com.atproto.admin.moderationAction" doesn't have a main def

Skipping that section, it also fails in com.atproto.report:

failed to process schema "/home/bnewbold/bsky/atproto/lexicons/com/atproto/report/subject.json": schema "com.atproto.report.subject" doesn't have a main def

These were previously segfaults, I added a check to print an error and bail instead of crashing.

The solution might be to generate types for all defs (not just main) in some contexts, and just skip generation in other situations? I didn't really try to read the lexgen code and solve this myself yet, didn't want to submit a patch that would break things subtly without understanding.

repo hardening test ideas

This issue is a list of things i'm intending to review and test in the indigo MST and repo/repomgr implementations. I'll also be cross-referencing and expanding from the atproto (Typescript) and adenosine (Rust) implementations for cross-language interop.

MST

Existing tests:

• merge across projects
• convert fanout=32 edge case tests to fanout=16

When receiving/validating external MST structures:

• invalid: duplicate exact keys in MST
• invalid: wrong order in node child list
• invalid: additional fields in MST node CBOR
• invalid: wrong CID type/encoding in MST refs
• invalid: non-existant CID to an MST or record in MST refs ("broken IPLD ref")
• invalid: broken CID in MST refs
• invalid: point to MST node w/o IPLD flag/type set in DAG-CBOR
• invalid: key (path) with whitespace; null bytes; non-ASCII; control chars; etc
• restrict: huge lists of entities per node (similar to collision attack)
• invalid: keys at wrong "depth" in tree (based on hash prefix)
• limit: MST key length (path) should have some limit
• invalid: CBOR, but not canonical DAG-CBOR

Edge cases:

• path as empty string (zero length)
• key is UTF-8 with partial match (bytes share prefix; string char different)
• key points to a non-IPLD blob

Repo (on top of MST)

• path as empty string (zero length) (distinct from MST)
• no-op commits: same MST root CID
• edge case: MST with greater or fewer than 2 path segments
• path with leading, trailing, or doubled slashes (//)
• key is UTF-8 with partial match (bytes share prefix; string char different) (same as MST test)

CIDs

Mostly handled by IPLD libraries, but we have some additional constraints, and also have some CID string fields in records that could (should?) be validated.

Invalid in string form:

• truncated (too short)
• extended (too long)
• wrong characters
• trailing null byte
• non-existent encoding or hash

Places to check:

• lexicon record 'cid' fields (string)
• XRPC query params (string)
• XRPC post body fields (string)

Stricter constraints on generally valid CIDs:

• encoding
• hash algorithm (eg, MD5)

Handles

Presumably there is a validator function, which might just be a regex. It should check:

• leading/trailing dot
• leading numbers
• dashes, underlines, slashes, etc
• whitespace
• unicode (not ASCII)
• internationalized domains (puny code)

If we do want to support puny code and internationalized domains, will need converter to/from display form, with tests.

DIDs

Generally, should verify:

• valid DID format (based on W3C spec?)
• overall length restriction

And more specific constraints in atproto, for now:

• restricted set of supported methods (eg, did:plc and did:web to start)

ATURI

Similar to handles and DIDs, should have tests for weird/mangled ATURIs.

• invalid: null-byte, whitespace
• "authority" must be a valid DID or valid handle (see above)

TID/rkey

• all the same restrictions as MST key strings

Post Records

Lexicon should catch most things, but some additional checks, specifically for post records.

• invalid or empty embed URLs
• invalid DIDs in entity embeds (mentions)
• timestamps: partial/truncated (eg, just year or date)

Strings Generally

Basically, just stick a bunch of interesting strings and interesting places and ensure they are handled correctly.

• "naughty strings" list: https://github.com/minimaxir/big-list-of-naughty-strings
• i18n: CJK, arabic (RTL), diacritics, emoji
• with/without UTF-8 byte order mark (BOM)
• empty string
• invalid/broken UTF-8 encoding (eg, bad JSON)

XRPC (API) hardening

Probably handled at the load-balancer level:

• huge individual header fields (MByte+)
• many headers/cookies (MByte+ sum)
• client SSL attempt
• huge query params
• mangled JWT tokens
• huge request body (over limits)
• GET request with a body
• /xrpc/../blah (relative path chars)
• /xrpc//blah (double slash)
• /xrpc/blah/ (trailing slash)

At application layer (XRPC server generically):

• random additional query params (like UTM tracker)
• JSON corner case: double-backslashes
• JSON corner case: un-escaped UTF-8 in string
• JSON corner case: invalid UTF-8 in string

PDS: proper XRPC/HTTP error handling

Currently, if you do something like getRecord on a record that doesn't exist, or an unknown XRPC endpoint, you get an HTTP 500 error, without a properly formatted XRPC error reponse (aka, JSON with an error message).

cborgen: support nullable booleans

Trying to have a not-required boolean field on a record, lexgen seemed to work fine, but then I got:

# github.com/bluesky-social/indigo/api/label
api/label/cbor_gen.go:111:30: cannot use t.Neg (variable of type *bool) as bool value in argument to cbg.WriteBool
api/label/cbor_gen.go:288:13: cannot use false (untyped bool constant) as *bool value in assignment
api/label/cbor_gen.go:290:13: cannot use true (untyped bool constant) as *bool value in assignment

This might be an issue for upstream https://github.com/whyrusleeping/cbor-gen

brainstorm ideas for Cool Developer Tools

I love Cool Tools! There are some nifty existing tools for wrangling IPLD objects, CIDs, etc. It would be nice if we eventually had similar things for atproto repos, XRPC APIs, Lexicons, etc.

This issue is a wiki-like brainstorm of ideas for nifty developer tooling.

CLI/UNIX Tools

MST tree visualizer: prints out a graph of nodes, showing depth, number of entries, leading-zeros, CIDs, etc. Maybe a "compact" form and a "verbose" form? Could be horizontal (matches "classic" way of thinking about trees) or vertical with indentation (easier to dump large trees). Could also export graphviz or something. Bonus points for doing "diffs" between two trees, with some kind of color/visualization. Would be helpful for docs, interop tests, etc.

MST and/or repo to JSON object(s): I think the IPFS/IPLD tools (kubo?) can already convert a CAR or arbitrary blockstore to JSON files on disk. A possible improvement, for debugging and testing, would be to dump a single giant JSON object, with CID "links" being nested sub-objects. There might already be a known format or best practice for this. Then could use jq to pretty-print; would map to things like python dicts for integration testing; human-inspect-able format for test vectors. Could include actual CID values. Another mode would allow editing the JSON (invalidating CID links), and recomputing CID links. Maybe a "JSON patch" variant to show diffs between trees.

repo dump to/from folder: for well-behaved repos, a way to take a CAR file and unpack it into DAG-JSON files in filenames/folders which match the MST key names. And a way to parse such a thing, along with a DID and signing key, into a repo CAR file. Should reproduce. Useful for inspecting and sharing test vectors, also for folks to work with their exported CAR files. Maybe a variant would create a .tar.gz in memory from the CAR, as a parallel export format. Optionally a "standard" way to store any extra metadata and keys in the base directory. Optionally a "standard" way to store blobs (images, etc) in the same directory and .tar.gz file.

mount repo as FUSE filesystem: DAG-JSON objects at MST key paths. could be local (working on a blockstore or CAR file) or remote (PDS via XRPC). listing, reading, writing should work. Validation errors would be some IO error I guess. Could even mount the full atproto world with DID prefix directory (!). Blob support?

lexicon converters: like pandoc, but for schemas? lexicon-to-... openapi-v3; JSON schema; protobuf interface. Opinionated/controversial?

Daemons / Proxies / Services

Some of these could be simple features in PDS implementations.

GraphQL proxy to XRPC: could be codegen or not

RSS feeds to/from bsky: the trivial thing is making author feeds available as RSS (and/or Atom, JSON feed). a general-purpose bot would consume an RSS feed and push to bsky (why's hnbot maybe already does this?)

Micropub interface to PDS: micropub is a relatively simple indieweb HTTP protocol for posting microblog-like content. There are several mobile apps. Much simpler than activitypub, IIRC. Might unlock a small ecosystem of bots, apps, integrations, etc? But might also be extremely niche.

Web Tools

Simple web interfaces to all the CLI things above, works by passing in a DID or ATURI to inspect.

Lexicon verifier: paste in a lexicon, get told if it is valid and if not why. then, in a second box, paste in arbitrary JSON and get feedback on whether it matches the lexicon or not, and if not why, in a human-readable form. Should be able to do this in-browser with typescript implementation, or compile whatever else to WASM. The Rust ecosystem has some nice validators that generate human-meaningful error messages; i'm sure similar things exist in other ecosystems.

DID+ATURI debugger: plug in a DID or ATURI, the web service live attempts to resolve the DID and connect to PDS; for ATURIs also tries to fetch and verify content. Similar to web tools like BGP looking glass; whois lookup; SSL score; "is my email setup (MX/SPF/DKIM/etc) working right". Might need some rate limits?

Existing Stuff

I'm pretty happy with the CLI "API" (argument structure) for adenosine-cli, which is basically general purpose com.atproto + app.bsky CLI: https://gitlab.com/bnewbold/adenosine/-/blob/main/extra/adenosine.1.md

subscription clients should backoff on errors

Right now, at least labelmaker will try to re-connect to the upstream server very aggressively (hundreds/thousands of times per second) if it receives a valid error. It should probably do an exponential backoff thing.

I vaguely remember the BGS doing some backoff when dialing PDS instances, but I could be confused.

cc: @whyrusleeping

gosky CLI: command to consume event streams (websocket) and pretty-print events as JSON

Would be helpful for debugging. https://github.com/vi/websocat exists and is partially helpful, but dumps binary CBOR and doesn't handle "paired" objects (header and payload) in any special way.

As some example use-cases, could pipe output though | pv -l > /dev/null to get an event rate counter; pipe through | rg whatever | jq for filtering and extracting specific event fields; etc.

I could also implement in adenosine-cli, which i've been using for some development.

HACKING.md needs to be updated

I think the atproto Makefile has been updated to use target "run-dev-pds" for running pds locally but the integrated development section in HACKING.md still needs to be updated accordingly.
Am I missing something? can raise a quick pr for the same

brainstorm: package refactoring

I find the current package names and layout confusing, even after working with them for a while. I'm also not always sure where to put stuff. Here is a draft proposal for how to re-organize things, particularly coming out of the recent lex refactor.

Would be good to push these through before too many folks start building on this repo.

Could do this in stages, or rip the bandaid all at once. Timing-wise, would like to get labelmaker landed before doing any of the more disruptive refactors.

ATP Services

• bgs/: "big graph server"; command name is bigsky
• pds/: "personal data server"; command name is laputa
• labeler/: "labeling service"; command name is labelmaker

Service Components

These generally mange state, either on-disk or in a SQL database. Sometimes these align with "ecosystem roles" (a "service" might fulfill multiple "roles").

• carstore/: on-disk repo store (in CAR files), plus SQL database indexing
• blobstore/: on-disk file storage, plus SQL database indexing
• eventmgr/: handles event subscription from producer side, including persisting the stream (SQL database or other backens) and re-play buffer
• repomgr/: integrates a storage engine, key management, and event generation
• crawlmgr/: rename of indexer/, though we might be able to split that functionality into eventmgr/ (for single consumption, like gosky, labelmaker, and search service) and bgs/ (for crawling multiple endpoints). might be simplest to keep a single implementation which works with multiple upstream endpoints
• aggrmgr/: for parts of indexer related to persisting notifications, backlinks, etc
• didmgr/ (NEW): persisted cache of DID identifies and keys, including both local accounts and remote identities. supersedes previous keymgr code
• labelmgr/ (NEW): not needed to start, but might end up existing if PDS or other services need to persist and access labels from a SQL database

atproto libraries

These are specific to atproto and might be reused by third parties.

• atproto/identifiers/ (NEW): string wrapper types for DID, NSID, at-uri, handles, cid-str (as a string, not parsed) and other Lexicon-defined types. only string validation, not external code/helpers (eg, no DID resolution stuff). many other packages would import this
• atproto/did/ (NEW): replaces whyrusleeping/go-did. parse and represent only the DIDs supported by atproto (did:plc and did:web). no crypto! parse DID docs. possibly clients for doing did:web and DNS lookups. not a general-purpose DID library.
• atproto/crypto/ (NEW): replaces whyrusleeping/go-did. parses/generates all the supported atproto key types in all the various formats (did:key, multibase-in-did-doc, hex for signing keys, etc). only works with the curves and formats used in atproto. clear naming for, eg, "HashAndSign" (which does SHA-256 then signs; instead of just "Sign" or "Verify" names)
• atproto/xrpc/: generic XRPC client, and possibly some server-side helpers. may also include some subscription (websocket) client and helper code
• atproto/repo/ and atproto/repo/mst/: low-level types and algorithms for working with repo DAG structure. agnostic to storage details.

lexicon packages

This is kind of bike-sheddy, and i'm not really certain this is the best way to go. But the current api/atproto/ setup, resulting in package name atproto by default, feels pretty confusing to me.

It really feels like the lexutil stuff (LexBlob etc) should live closer to the actual generated lexicons.

• lexicon/comatproto/: the current api/atproto/. types for com.atproto.* lexicons
• lexicon/appbsky/: the current api/bsky/. types for app.bsky.* lexicons
• lexicon/lexutil/: the current lex/util.
• lexicon/gen: implementation code for lexgen

Other Packages

• fakedata/: new package with most of fakermaker code. should be in a non-main package for easier re-use in integration tests
• plc/: did:plc API client; types for encoding and verifying PLC operations; fake/mock/testing server implementation. could go under atproto/did/plc/?
• dbmodels/: only for database models that are actual shared across services (or service components)
• cmd/: actual binaries, CLI handling, etc
• testing/: any inter-package or high-level tests. open to renaming this (tests/?)
• util/: grab bag stuff that still doesn't fit elsewhere
• cborgen/ or /gen/cbor/: clearer name for current gen/. alternatively, use the go:generate functionality in golang to do this per-package instead of top-level

Other Changes

• move cmd/gosky/util/ to util/cmdutil/
• move version/ to util/version/
• move util/dbcid.go and util/uid.go to dbmodels/
• move util/time.go to lexutil. or maybe copy? and possibly rename to, eg, LexDatetime
• move util/fakekey.go to didmgr (or wherever keymgr ends up)
• delete testscripts/, or at least move under testing/

handling typed identifier strings

The lexicon refactor comes with better-defined string-based identifier types. These include handle, nsid, at-uri, did, and others.

These don't change the encoded formats, and the golang lex code doesn't directly need to do anything with these types directly, but it is important to not generate invalid records or other lex objects (eg, query params or request bodies), and it is an opportunity to add type info to our codebase.

Throwing out some ideas of how to work with these types:

• CheckNSID(raw string) error and similar functions in lexutil or similar package
• have lexgen call these check functions in places like XRPC handlers and JSON marshaling. maybe also generate func (r *Record) Check() error style functions for all generated structs, which would call these methods on fields recursively
• declare string type aliases for each of the identifier types. put these in lexutil, and have them implement Parse(raw string) (Identifier, error) and (ident *Identifier) ToString() string helpers (though type coercion could also work) and JSON/CBOR marshaling helpers.

This last option is basically what I was doing in adenosine (Rust). It makes validation clear (happens only once, when the string is converted), and to-string is cheap. It might make parsing records and handlers a bit slower though, especially in cases where validation doesn't really matter? But I think in all those cases the record just shouldn't be un-marshaled at all.

Integration tests need randomly assigned ports

Our integration test setup (testing/) currently hardcodes a port for each service it spins up. This is cumbersome and error prone, we should instead listen on port 0 and detect the port the OS chooses for us, then set that field in the struct from there.

indigo: followsAddCmd return XRPC ERROR 400: 400 Bad Request

repomgr: transaction failure under load

Running fakermaker, in parallel, resulted in this SQL transaction failure:

2023/02/13 15:39:48 /home/bnewbold/bsky/indigo/repomgr/repomgr.go:166 context canceled; sql: transaction has already been committed or rolled back
[228.911ms] [rows:0] INSERT INTO `repo_heads` (`created_at`,`updated_at`,`deleted_at`,`usr`,`root`) VALUES ("2023-02-13 15:39:48.542","2023-02-13 15:39:48.542",NULL,282,"bafyreihnvmu2p27co2trp5pokjolay5x3vmacl7ttfrswe7byvfb47lwem") ON CONFLICT (`usr`) DO UPDATE SET `root`=`excluded`.`root` RETURNING `id`

Lex refactor needs

Cross-reference to main upstream (atproto) branches: bluesky-social/atproto#658

Things that will require new lexgen feature support:

• type="cid-link", encodes as an object like { "$link": "<cid>" }. For binary CIDs, aka "Link" in IPLD data model)
• type="bytes", encodes as an object like { "$bytes": "<base64-encoded>" } for embedded bytes (which specific base64 config?)
• no image/video/audio, just type="blob" now. blob gains a size field (integer, size in bytes)
• type="number" changed to type="float" (though not yet used, and might change how that works)
• string types, which will work like type="string" format="at-uri"

The string types are not strictly needed to get lexgen to run, but we probably might as well wire up placeholder validation for functions in codegen code. The formats are at least: handle, nsid, at-identifier (handle or DID), uri (generic), datetime, at-uri, did, cid. I can port over the validation functions and their tests. I guess we haven't actually implemented the other validators yet, like maxLength. Maybe we punt on validation until a second pass (separate PR).

There are some semantics/benavioral changes, like idempotency of batch mutations and deleteRecord not failing if record already does not exist. I don't think the existing golang PDS is complete enough to care about those.

Once lexgen runs successfully, presumably there will be a bunch of broken stuff that we'll just need to get to compile.

cc: @whyrusleeping

search timestamp handling

This is an old issue, but assuming it will still be present in pulumi.

failed to handle op: indexing post: post (41345, app.bsky.feed.post/3jvdilglu7j2n) had invalid timestamp ("2023-05-10T00:05:55.720835Z"): parsing time "2023-05-10T00:05:55.720835Z" as "2006-01-02T15:04:05.000Z": cannot parse "835Z" as "Z"

When creating timestamps we have been doing milisecond precision (three digits after "second"), but it seems reasonable to expect pulumi to be able to handle longer timestamp precision (at least discarding the extra digits). This is a known ambiguity in the current Lexicon datetime specification.

gosky: bsky.RepoGetRecord return 400 Bad Request

getting record: XRPC ERROR 400: 400 Bad Request

PDS<>BGS Sync Megaissue

Making this issue to keep track of any known event stream/sync issues.

• PDS occasionally sends an event for a repo with an empty car file and a null prev field, despite the repo having pre-existing commits
• BGS occasionally gets into a backfill mode and claims that it has missing blocks.
  • In most (but not all) cases, these are encountered when processing the 'next' commit beyond the one currently set as the head in the repo
  • In other cases, the missing blocks are in the middle of the backfilled commit range.
  • It has been observed in some cases (unknown if all cases) that the missing blocks occur in multiple places in the commit history for the repo in question

get `GOLOG_LOG_LEVEL` working with `dotenv`

This was an issue with the atproto PDS as well, and was resolved by loading the dotenv stuff before initializing the logger. Maybe we can re-initialize the logger if a dotenv file is found?

pipeline event stream playback

The database event persisters playback loop needs a bit of optimization.
Relevant code here:
https://github.com/bluesky-social/indigo/blob/main/events/dbpersist.go#L128

The easy first pass should be to just pipeline the database loading and the event hydration. Also some amount of parallelism and/or caching in the event hydration is likely warranted too.

PDS: update profile fails if already exists

Running fakermaker gen-profile, got an error:

mst.Add failed: value already set at key: app.bsky.actor.profile/self

Fix should include a regression test.

lexgen: optional query params for concrete types (bool, integer)

I could be wrong about this, but I think we have a problem with lexgen-erated client functions.

For example: https://github.com/bluesky-social/indigo/blob/main/api/atproto/admingetModerationReports.go#L20

func AdminGetModerationReports(ctx context.Context, c *xrpc.Client, cursor string, limit int64, resolved bool, subject string) (*AdminGetModerationReports_Output, error) {

limit and resolved query parameters here are optional. For resolved specifically, "true", "false", and "no param supplied" are all distinct behaviors.

I'm not sure if just *bool as the type would resolve this or not.

some form of interop testing between indigo (golang) and atproto (typescript)

Does not have to happen on every CI run, but want to start making it possible to run, eg, fakermaker and bigsky against current (main branch) PDS/PLC/etc from atproto repo.

Probably using containers. Maybe in the form of special indigo golang tests behind a flag.

How do we use ./cmd/gosky to do things like log in and post?

There doesn't seem to be any documentation and the help is minimal.

There's a command createSession, that looks like how you log in. So … how? I tried things like

go run ./cmd/gosky createSession -u <username> -p <password>

and

go run ./cmd/gosky createSession --username <username>

but they don't work. When I do

go run ./cmd/gosky help createSession

I just get this:

NAME:
   gosky createSession

USAGE:
   gosky createSession [command options] [arguments...]

update all github.com/lestrrat-go/jwx/* to v2

eg, use of jwk and jwa. We currently have a mix of the two packages, should use just one.

capture git revision at build time, and inject as needed

Eg, CLI --version; HTTP client User-Agent; etc.

The git invocation I have used before is:

git describe --tags --long --always

Need to take care in docker builds to grab this while .git is available.

bigsky as-a-crawler polish

A couple things we should look at for bigsky, in the context of it reaching out and crawling an increasingly large number of hosts on the web.

• going to make HTTP(S) connections to random web servers. it can self rate-limit, but should ensure it has robust timeouts to not get honey-trapped
• we should ensure that the "ding/dong" requests can't be used to make us DDoS other servers (outgoing requests)
• needs to have fresh SSL cert database when deployed: need to include in alpine docker image? or maybe just switch to ubuntu base for deployed images?
• when bigsky connects to servers, it should set a User-Agent identifying itself, and linking back to some contact info (webpage about crawling, or email address). this is standard best practice for bots/crawlers
• should think about robots.txt a bit. I'm not very concerned because we mostly crawl on request, and are connecting at an atproto-specific URL (/xrpc/...)

atproto.RepoDescribeRepo fails with a decode error for did:plc:wflozfzpewefv46qof26vbzm

decoding xrpc response: unrecognized type: ""

notifications: mentions not handled

With fakermaker, got an error:

HANDLER ERROR: (/xrpc/app.bsky.notification.list) attempted to hydrate unknown notif kind: 2

The "mentioned" notification kind is commented out and un-implemented in notifs/notifs.go.

BGS: transport compression for non-trivial CAR files

Eg, if doing a sync method on a full repo download, should get HTTP transport compression. CAR files can get big, and compress quite well. IMO so well that we should do this at the BGS (app) layer, not at load-balancer level.

Might already even be doing this, for free! But need to check. In particular, mimetype "compressible" database might not be configured for the CAR mimetype.

XRPC client: handle 5xx retries with backoff

(I didn't check if the default HTTP client config for gosky already does this)

beemo is failing immediately on a single 5xx error, for example with session auth. It would be nice to have a bit of shock absorber behavior where the XRPC client generically does a small number of retries on 5xx errors. Bots and things like beemo probably want to set that even longer (like up to a couple minutes of retries during planned maintenance?).

Lex refactor follow-ups

There are a number of things in the lex refactor that we are skipping for now, but should come back around to soon.

• string "subtype" validation, for things like handles. there already exist validation functions, regexes, and test cases in the typescript repo, we should reuse those. but need to add some lexgen hook or API for this. eg, Validate() methods on all structs? string alias types with validating constructors?
• size constraints on blobs (max bytes)
• size constraints on strings, both in bytes (UTF-8) and graphemes
• review when to populate $type field (see below)
• cleanup deprecated terms: "vote" (now "like"), repo "append" (now "commit"), ActorRef (now "ActorViewBasic" or similar)

From conversation with Devin, the times the $type field needs to be set are:

• on every record type object, at the top level, with no # in the type
• on every "new" style blob ($type = "blob")
• on every union (enum) object sub-type, with a # indicating the variant

remaining did:plc changes

Already handled a couple changes in #50 and #49.

@dholms mentioned:

Last related thing is the new canonical way of finding the handle is: "first value in alsoKnownAs that starts with at://"

We should update indigo to match that.

lexgen+cborgen: support arrays-of-arrays

This has only come up in the context of lexicon tests, not in any of our own lexicons. But it feels to me like a correctness issue.

May be partially/entirely an issue in upstream https://github.com/whyrusleeping/cbor-gen