bluesky-social / feed-generator Goto Github PK
View Code? Open in Web Editor NEWATProto Feed Generator Starter Kit
License: MIT License
ATProto Feed Generator Starter Kit
License: MIT License
Items that contain labels (so far i have only seen Adult content be kept in) are still shown in the feed even if i use the remove if item has label block, i tried changing its position, not using regex, but NSFW Content is still displayed.
I'm trying to determine if a certain repost has been un-reposted. For example, person A reposts person B; person A eventually decides to undo that action. When this happens in the app, I see the atproto.repo.deleteRecord API is called with a reference to user A's DID so I figured this counted as a "delete" action. Inside the record object I assumed the the opsByType.reposts.deletes (here) would have all the reposted items that have since been deleted (un-reposted) but I'm getting nothing when I console.log there.
(Title only)
Not sure if there has been much talk about this amongst the bsky team, but I'm wondering if people are open to using ES Module resolution over CommonJs?
From my (limited) understanding, ES Modules are the official standard, they are now supported by Node.js, and newer libraries are dropping support for CommonJs.
It's not difficult to make the switch in my own fork, but I do find myself updating the imports in all of the auto-generated files and it feels wrong.
This would also be less of an issue of the lexicon was a published package rather than a copy and paste.
@dholms don't you mind adding https://github.com/MarshalX/bluesky-feed-generator in the README? I think many people and the Python ML community will be glad to know about it
Twitter has decorators on tweets that explain why you're seeing it -- e.g. "someone you're following liked this" or a specific topic. It would be very nice to be able to specify something like this for atproto feeds. As an example, I'm working on something that will identify a specific anime that a post is talking about and it would be nice to be able to include the name of the show (and maybe even an anilist link?) above the post in the feed.
I've got this running on my server and can verify that it's reachable at the domain i'm providing in the .env file. Posts are being indexed properly and its basically operating almost exactly like the whats-alf feed does.
I'm able to successfully run yarn publishFeed
, and if i log the return from agent.api.com.atproto.repo.putRecord
at the end of the script, I can see it was successful? Unfortunately, I think I'm doing something wrong, though; I can't see my feed in the web or ios clients.
Any pointers would be appreciated.
I setup a custom firehose feed a couple weeks ago but it hasn't updated in a week. When I clone this repo and run yarn
and yarn start
no posts get written to console as expected.
Was there a change to the protocol that broke this repo?
I did a dumb copy/paste job in the lexicon folder. clean that up to only necessary lexicons
Hello! Been trying to wrap my head around this for a few days now. I'm struggling with building feed generators which rely upon user profiles or an authenticated user's personalized feed. Would it be possible to get some examples of say, a "mutuals" feed and a feed based on something to do with the poster's profile? Maybe a feed with only posts from users whose handles start with the letter 'A'?
The spec shows a limit parameter being available, which should default to 50. As best as I can tell this repo doesn't support it, and always returns all the posts. This seems especially odd since it does have a cursor parameter, but it's never needed if the API always returns all the posts since the beginning of time.
Hey Guys
I just published a feed hosted at https://bsky.flipboard.com/ but i do not see it under discover feeds on bluesky app.
I got the uri from bsky and the following endpoint seems to work :
https://bsky.flipboard.com/xrpc/app.bsky.feed.getFeedSkeleton?feed=at://did:plc:cndfx4udwgvpjaakvxvh7wm5/app.bsky.feed.generator/Flipboard-tech
with response
{ cursor: "1686067188000::zdpuAykYqGecnAruErcrLusqVqHyNNMQx47QxLXvruvMHUkoC", feed: [ { post: "at://did:plc:m5soh62ifdpq443v3rnjnfrm/app.bsky.feed.post/3jxj5j2jsub22" }, { post: "at://did:plc:6f3xxnzgosa77ljoajfryihf/app.bsky.feed.post/3jxj5c2swuu2h" }, { post: "at://did:plc:7dh44snmqoa4gyzv3652gm3j/app.bsky.feed.post/3jxj57jhvyp23" }, { post: "at://did:plc:kgpihmrsir64cq5gbo4fam7e/app.bsky.feed.post/3jxj55lehek2i" }, { post: "at://did:plc:4qbgho2ec2fbhyizgzipl7gg/app.bsky.feed.post/3jxj54dsz7q2s" }, { post: "at://did:plc:ir2b7mnp3f7qouemawhhew72/app.bsky.feed.post/3jxj4zv3k6a2b" }, { post: "at://did:plc:6a7yxehur7eohyxp2wv7efru/app.bsky.feed.post/3jxj4qsiope2h" }, { post: "at://did:plc:e37k2mocg7e6tnund62tmpmz/app.bsky.feed.post/3jxj4hxbl6e2h" }, { post: "at://did:plc:xxdwn6ygf56j4ge5xsjw5l2t/app.bsky.feed.post/3jxj476m4vu2f" }, { post: "at://did:plc:sfc3dijwyvbh7r3te5v5ty3b/app.bsky.feed.post/3jxj3wntpdr2v" }, { post: "at://did:plc:fukey2sryiz53sqvu2wiuba6/app.bsky.feed.post/3jxj3u3u2cq2s" }, { post: "at://did:plc:lohurgscqafxogwp3f7tsd3l/app.bsky.feed.post/3jxj3mbawyh23" }, { post: "at://did:plc:ctphcgyhnllfluywokibtrgb/app.bsky.feed.post/3jxj3ikphf32a" }, { post: "at://did:plc:bj6a7q7sbqu2vil6bviysjgg/app.bsky.feed.post/3jxj3bpuvee2h" }, { post: "at://did:plc:kowyosk2m56ibvwwi5vf32hz/app.bsky.feed.post/3jxj2zg6k5u2l" }, { post: "at://did:plc:gatxphlbrw3hych5avk5ipci/app.bsky.feed.post/3jxj2vgmswh23" }, { post: "at://did:plc:rvlyeda73kxm7l2weegk73pa/app.bsky.feed.post/3jxj2v5ojru2f" }, { post: "at://did:plc:s6j27rxb3ic2rxw73ixgqv2p/app.bsky.feed.post/3jxj2owmc4y2b" }, { post: "at://did:plc:th2ktgiz43y6dtdgz5zflcxg/app.bsky.feed.post/3jxj2kb7xgs2i" }, { post: "at://did:plc:dhvvqy2dcefrk35kudxclkyd/app.bsky.feed.post/3jxj2hvgh7m2f" } ] }
Any help would be appreciated.
The skeleton metadata example looks like this:
[
{post: 'at://did:example:1234/app.bsky.feed.post/1'},
{post: 'at://did:example:1234/app.bsky.feed.post/2'},
{post: 'at://did:example:1234/app.bsky.feed.post/3'}
]
But your actual feed in production needs to look like this (at minimum), which I found out the hard way:
{
"feed": [
{"post": "at://did:example:1234/app.bsky.feed.post/1"},
{"post": "at://did:example:1234/app.bsky.feed.post/2"},
{"post": "at://did:example:1234/app.bsky.feed.post/3"}
]
}
I’m curious if there is any way with the current support to create a custom feed that is just a list of accounts. (Consuming the feed would populate it with posts from those accounts.) This would allow for easy implementation of communities/lists functionality without having to run a server since such a feed could be hosted statically. It seems to me such ”algorithms” are going to be popular but take a lot of overhead and maintenance.
I published a few feeds using this modification to scripts/publishFeedGen.ts
await agent.api.com.atproto.repo.createRecord()
I tried to run the script a second time with the same params to add an avatar to a feed and got the following error:
alan@dojo feed-generator % yarn publishFeed
yarn run v1.22.17
$ ts-node scripts/publishFeedGen.ts
/Users/alan/Desktop/feed-generator/node_modules/@atproto/xrpc/src/client.ts:125
throw new XRPCError(resCode, res.body.error, res.body.message)
^
XRPCError: Internal Server Error
at ServiceClient.call (/Users/alan/Desktop/feed-generator/node_modules/@atproto/xrpc/src/client.ts:125:15)
at processTicksAndRejections (node:internal/process/task_queues:95:5) {
status: 500,
error: 'InternalServerError',
success: false
}
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
I'm not super surprised since the name is createRecord()
. I tried updateRecord()
record just to see if it worked, but it didn't.
Let me know if I can provide more info
I am not entirely sure how this works, but I'd be interested in getting it working. If I wanted to publish a very simple feed generator that filters for a certain phrase / hashtag, what would I need to do exactly?
I just followed the quick start and ran publishFeed a couple of times, but it's not showing up on bluesky.
I am also wondering whether I don't need to change the uri, which is not mentioned in the quickstart, as well as how to find out my did
?:
// /src/alogs/whats-alf.ts
export const uri = 'at://did:example:alice/app.bsky.feed.generator/whats-alf'
Sorry, I am a complete newbie here, but I'd really like to publish a very simple feed generator.
The code should be more resilient to server errors and try to reconnect automatically and handle errors gracefully. It exited for me on a 502 error.
node_modules/node_modules/ws/lib/websocket.js:888
abortHandshake(
^
Error: Unexpected server response: 502
at ClientRequest.<anonymous> (/home/thomas/projects/twistori-feed-generator/node_modules/node_modules/ws/lib/websocket.js:888:7)
at ClientRequest.emit (node:events:511:28)
at ClientRequest.emit (node:domain:489:12)
at HTTPParser.parserOnIncomingClient (node:_http_client:693:27)
at HTTPParser.parserOnHeadersComplete (node:_http_common:119:17)
at TLSSocket.socketOnData (node:_http_client:535:22)
at TLSSocket.emit (node:events:511:28)
at TLSSocket.emit (node:domain:489:12)
at addChunk (node:internal/streams/readable:332:12)
at readableAddChunk (node:internal/streams/readable:305:9)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command
Trying to deploy a new feed I'm getting a XRPC Not Supported when calling await agent.api.app.bsky.feed.describeFeedGenerator()
Not sure what needs to be fixed.
It happens when trying to publish a new one and republishing an old one. Is there something wrong in that FeedGenerator outcome?
Thanks and regards
This is the describeFeedGenerator outcome if I understand it right: https://bsky-feeds.adrian.uy/xrpc/app.bsky.feed.describeFeedGenerator
feed-generator/src/util/subscription.ts
Line 84 in e849ac7
still commented in the code that updates are not handled.
After testing Deck.blue, I feel like Bluesky app filters content by language as a default? For instance in my Next.js feed, I see only posts in English, but Deck.blue shows many posts in Japanese.
If that's the case, is a feed expected to filter posts by language, or should it be the responsibility of the app view instead?
Related to eric-burel#1
Thus if you switch out sqlite3, it will fail to create the table. I set up one called atproto_user
instead. I think it should default to another name. If it's beyond the scope of the started it should at least be noted in the README because I'm sure plenty of people will run into this.
Hi,
I've been playing around the generator and I must admit I haven't been that enthusiastic about technology in a while.
The architecture is simple to host on a traditional server, however it would be interesting to evaluate a serverless alternative:
I plan to investigate relying on Vercel architecture specifically, which I know as a Next.js dev and that seems to provides all the required primitives (KV database, serverless hosting, API for a multi-tenant system).
However the part that I don't fully grasp yet is the websocket that receives the hose. Is it doable to have a serverless consumer? Or maybe I need a "serverful" architecture just for this part? It's not doing intensive work so that could do as long as we move the database + the feed algorithm to serverless.
Some services seem to offer stateful serverless systems that could work with websockets: https://sunilpai.dev/posts/the-future-of-serverless/
There is a problem in the FirehoseSubscriptionBase
implementation of updateCursor()
. It makes an update to the sub_state
table to put the current cursor state there, but since no one has inserted any rows there previously, the update doesn't do anything.
The method should either first make a select and then either insert or update, or some kind of "insert / on conflict" (but there would need to be a unique index), or insert a null row there at launch. I don't know how you'd prefer to fix this, so I'm leaving this an an issue w/o a PR.
I want a sample script for unpublishing a feed, which is launched such as yarn unpublishFeed
.
As of yesterday my feed simply stopped showing posts. I have a list of DIDs for those who will appear on the feed, and I am checking in the subsctiption.ts indexing logic if any of the incoming posts match those DIDs. It was working perfectly fine until yesterday, and now when I run it in prod or even locally I can't get a single post to show from someone in that list, including myself.
I haven't changed any code. Did something change with the firehose? With the post metadata? I'm really at a loss here. If this keeps up I don't really know what to do about the feed.
It's been two days that feeds in Hebrew are not receiving new publications.
Even a in the log that shows all the publications that arrive does not have any lines in Hebrew.
The code is the original code.
The only change is the keyword I'm looking for.
Hi,
Me again, I am trying to figure out how to prevent spamming or explicit content in a feed. I've read the arxiv paper, I've understood that labelers are in charge of detecting spam (or whatever kind of labelling you want) and will provide their result either to the PDS or the AppView. If I rephrase, the label is used upstream to filter posts in the PDS, so the feed generator may check the label, or downstream in the user App View, so user can select what they want to see.
However, feeds seems to be parallel to this process. Therefore, how do I guarantee that a feed contains no spam or explicit content? It has access to labels from the PDS? For instance think a feed around web development, they are usual preys for spamming and you'd expect safe for work content only there.
I have created a feed using the documentation, and got all the way to publishing it onto Bluesky, but in Bluesky, it says Unsupported Algorithm.
Im successfully getting a feed over at
https://vrfurry-bsky.oasislab.co.uk/xrpc/app.bsky.feed.getFeedSkeleton?feed=at://did:plc:mkklr4smns2baynajhq3in6g/app.bsky.feed.generator/sfw-feed
on both HTTPS and HTTP using a Nginx reverse proxy.
I have barely made any changes to the algorithm. I'm using the base whats-alf.ts (renamed to sfw-feed.ts, and shortname changed to 'sfw-feed'), and I modified the subscription.ts to filter for a different word (a hashtag).
What could be causing this Unsupported Algorithm error?
Thanks!
it would be nice if the generated lexicon files were a separate package. copying them from this repo or from the atproto repo is pretty hacky, and after that you still need to run tsc on them.
.. to make a feed, we have to have our own server? What are the server requirements like, particularly with respect to disk space? I'd use my home server, but I don't think I can guarantee sufficient uptime for this sort of task.
There's one feed that REALLY seems to be missing, and since nobody has coded it, I was thinking about giving it a shot. Specifically, "NeverMiss" - a "Following"-type feed that prioritizes those you follow who post less often over those who post more often, so you don't miss your good friends' post in a sea of spammy posts from really active users (but you'll still get the latters' bangers... or all of their posts, if you scroll down far enough). Possibly further prioritizing users whose posts you like/share/reply to more as a percentage basis of their total, over those that you like/share/reply to less.
This repo currently splits the cursor parameter by ..
.
However, it returns a cursor split by ::
, and it sounds like other cursors in the protocol are split by ::
as well.
I'm assuming the code should be changed to split the incoming cursor parameter by ::
, but if not the format of the returned cursor should be probable be changed to match the parsed one.
Hello !
I think I properly made a custom feed, i published it, but when i try to add it to my feeds Bluesky gives me this error and I'm not sure what's causing this :/
Here is the link to the feed (might delete it at some point as it doesnt work, so i can't promise the link stays valid):
https://bsky.app/profile/did:plc:xitaf2i6zwhmmjuxlum5xofy/feed/diy
My service seems to work properly:
https://bsky-diy.durss.com/xrpc/app.bsky.feed.getFeedSkeleton?feed=at://did:plc:xitaf2i6zwhmmjuxlum5xofy/app.bsky.feed.generator/diy
I saw this issue mentionning that having an IP V6 server may cause this, I disabled it but it changed nothing:
bluesky-social/atproto#1338
I really have no idea what could be wrong.
If someone had a clue it would be greatly appreciated 🙏
I'm not sure what the distinction between feeds and algorithms is.
But the lexicon for getFeedSkeleton only defines an UnknownFeed
error, not UnsupportedAlgorithm
. I don't see a reason to use an undefined error type when UnknownFeed could be more relevant.
Trying out the repo locally, I can see all the contents from the "firehose" as they happen live, this is useful for getting new posts as they come, but is there a way to get all posts from a specific did from when the account was created?
Hi! I'm (and not only me) confused about how we can host Feed without deal with DID:PLC.
Few notes:
did:web
will be useddid:plc
if we are using did:web...Well... On one side. Service DID is optional that gives as the ability to forget about "DID:PLC". On the other side, the "feed" param verification in the official example uses URI including "DID:PLC". So we can't forget about it!
The current implementation looks like this:
feed-generator/src/algos/whats-alf.ts
Line 5 in e849ac7
It will be nice to not deal with DID:PLC at all if we are using DID:WEB. And it's pretty easy to achieve! Let's split the "feed" param and use only rkey to match the algo.
And here is my final questions. Is it fine to do so? Does the feed-generator repository show the best practice?
UPD. Maybe Bluesky server could send requests with URI that contains DID:WEB instead?
Hi,
I feel like naming the example algorithm whats-alf
can be confusing as well as the following part of the readme:
1. Implement indexing logic in `src/subscription.ts`.
This will subscribe to the repo subscription stream on startup, parse events and index them according to your provided logic.
2. Implement feed generation logic in `src/algos`
For inspiration, we've provided a very simple feed algorithm (`whats-alf`) that returns all posts related to the titular character of the TV show ALF.
When you look at it, it's actually an algorithm that show the latest posts. The logic that filters ALF related posts is actually living in the subscription, when messages are indexed.
This makes sense because for simple feeds, you'll probably want to index as little as possible, so most of the filtering will happen there.
The change I would propose would be:
whats-alf
to latest
What do you think?
I tried updating my fork to use the latest version of the @atproto packages, but there are lexicon changes that seem to be required (and fixed). I'm fine fixing them on my own, but I'm not 100% sure how to use the @atproto/lex-cli package to update the src/lexicon directory properly.
Could someone explain it? Happy to update the README as a PR afterwards.
Hello,
I'm currently implementing my own feed, and it works pretty well but I have these errors at random intervals:
repo subscription could not handle message RangeError: Could not decode varint
at Object.read3 [as decode] (/home/bsky/app/node_modules/@atproto/repo/dist/index.js:13095:17)
at readVarint (/home/bsky/app/node_modules/@atproto/repo/dist/index.js:24770:36)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async readHeader (/home/bsky/app/node_modules/@atproto/repo/dist/index.js:24775:19)
at async decodeReaderComplete (/home/bsky/app/node_modules/@atproto/repo/dist/index.js:25015:40)
If a reproduction repo is needed, there you go: https://github.com/maximesimoncelli/dungeon-synth-bsky-feed
For information, I encounter these errors on node 20, and I haven't tested it on older versions.
Thank you very much for your help :)
Hi, currently from the readme I cannot tell what happens when I implement two algorithms. Will the feed be associated with a button to select the relevant algorithm?
Give an example: in the README or in code of a minimum viable feed generator - static routes & no firehose subscription
Anyone getting a fetch error when attempting to publishFeed with no modification to source code?
/Users/jakeliebert/Documents/bluesky-PM-feed-main/node_modules/@atproto/xrpc/src/client.ts:156
throw new XRPCError(ResponseType.Unknown, String(e))
^
XRPCError: TypeError: fetch failed
at Function.defaultFetchHandler [as fetch] (/Users/jakeliebert/Documents/bluesky-PM-feed-main/node_modules/@atproto/xrpc/src/client.ts:156:11)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at async _AtpAgent._fetch (/Users/jakeliebert/Documents/bluesky-PM-feed-main/node_modules/@atproto/api/src/agent.ts:201:15)
at async ServiceClient.call (/Users/jakeliebert/Documents/bluesky-PM-feed-main/node_modules/@atproto/xrpc/src/client.ts:104:17)
at async _AtpAgent.login (/Users/jakeliebert/Documents/bluesky-PM-feed-main/node_modules/@atproto/api/src/agent.ts:120:19) {
status: 1,
error: 'TypeError: fetch failed',
success: false
}
error Command failed with exit code 1.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.