pantonshire/goldcrest

Twitter API proxy server using gRPC

Goldcrest Twitter API Proxy

Goldcrest is a proxy server using gRPC for interacting with the Twitter API v1.1. Its main focus is on providing centralised rate-limit tracking so that several processes can concurrently use the Twitter API without having to worry about rate-limits.

Currently, there are clients in Go and Rust.

Please note that this project is still in alpha stage; use at your own risk!

Supported endpoints

Goldcrest currently supports the following Twitter API endpoints:

Twitter API endpoint	gRPC method
statuses/update	PublishTweet
statuses/retweet	RetweetTweet
search/tweets	SearchTweets
statuses/unretweet	UnretweetTweet
statuses/show	GetTweet
statuses/lookup	GetTweets
statuses/destroy	DeleteTweet
statuses/home_timeline	GetHomeTimeline
statuses/mentions_timeline	GetMentionTimeline
statuses/user_timeline	GetUserTimeline
favorites/create	LikeTweet
favorites/destroy	UnlikeTweet
account/update_profile	UpdateProfile

Setup

Docker

Pre-built images are available on Docker Hub.

docker run -d -p 127.0.0.1:8080:8080 --read-only pantonshire/goldcrest

Currently, images are available for amd64 and arm64. If you're using a different architecture, you'll probably need to build the image yourself:

docker build -t goldcrest https://github.com/Pantonshire/goldcrest.git#main
docker run -d -p 127.0.0.1:8080:8080 --read-only goldcrest

Building from source

1. To compile Goldcrest from source, you will first need the following:
   • Go ≥ 1.15
   • Protocol Buffers Compiler v3
   • protoc-gen-go
     (go install google.golang.org/protobuf/cmd/protoc-gen-go)
2. Run make proto from the repository root.
3. Run make from the repository root.
4. cp default.goldcrest.yaml goldcrest.yaml to get a correctly-named config file.

goldcrest's Issues

Tokio 1.0

Waiting on Tonic update to upgrade from Tokio 0.2 to 1.0 🦀🦀🦀

Do not attempt request if authentication is not provided

If any of the fields in the Authentication message is empty, the proxy should not attempt to make a request to Twitter as it will always fail.

Implement local client

A local client allows the proxy to be within the same process as the application, so there is no need for inter-process communication by gRPC between the application and the proxy in this case. The exposed interface should be identical to the remote client.

Option for TTL for stored user data

A TTL would mean that the server will not have to retain old data about users who have not interacted with the server recently. A user interacting with the server should reset the TTL for that user. The TTL should be optional and configurable.

Tweet streaming

Add support for Tweet streaming endpoints to the proxy.

Direct messages

Provide support for direct message endpoints

Custom error for failed connections to Twitter

twitterClient.request should return a custom error when twitterClient.client.Do fails, rather than returning whatever error the Do method returns. This error can then be serialised and included in the response message.

Retry Twitter API request on 429 "Too Many Requests" with exponential backoff

A 429 can occur if:

• Goldcrest doesn't yet have information about rate limits for the endpoint being used
• Another process is using the Twitter API simultaneously, causing Goldcrest's rate limit information to become out-of-date
• There is a bug with Goldcrest's rate limit tracking

Rather than returning the 429 to the client, it would be preferable for Goldcrest to retry the request. Exponential backoff should be used to avoid getting flagged for misuse by Twitter.

Client search query builder

The Twitter search API has syntax for filtering Tweets by certain parameters. For example, including (from:account) will only return Tweets sent by @account (a user interface for this exists at twitter.com/search-advanced). The clients could implement query builders which automatically generate valid queries according to this syntax.

For example, in Rust this might look something like:

QueryBuilder::new()
    .words("food data rent")
    .exact_phrase("help me budget this")
    .from_account("dril");