Giter Site home page Giter Site logo

superseriousbusiness / gotosocial Goto Github PK

View Code? Open in Web Editor NEW
3.5K 38.0 290.0 152.05 MB

Fast, fun, small ActivityPub server.

Home Page: https://docs.gotosocial.org

License: GNU Affero General Public License v3.0

Go 93.05% Shell 0.29% Dockerfile 0.03% CSS 1.85% JavaScript 0.14% Python 0.03% TypeScript 4.60%
mastodon fediverse fediverse-server activitypub social-media social-network agplv3 golang federation activitystreams

gotosocial's Introduction

GoToSocial

GoToSocial is an ActivityPub social network server, written in Golang.

With GoToSocial, you can keep in touch with your friends, post, read, and share images and articles. All without being tracked or advertised to!

GoToSocial is still ALPHA SOFTWARE. It is already deployable and useable, and it federates cleanly with many other Fediverse servers (not yet all). However, many things are not yet implemented, and there are plenty of bugs! We foresee entering beta around the beginning of 2024.

Documentation is at docs.gotosocial.org. You can skip straight to the API documentation here. To build from source, check the CONTRIBUTING.md file.

Here's a screenshot of the instance landing page!

Screenshot of the landing page for the GoToSocial instance goblin.technology. It shows basic information about the instance; number of users and posts etc.

Table of Contents

What is GoToSocial?

GoToSocial provides a lightweight, customizable, and safety-focused entryway into the Fediverse, and is comparable to (but distinct from) existing projects such as Mastodon, Pleroma, Friendica, and PixelFed.

If you've ever used something like Twitter or Tumblr (or even Myspace!) GoToSocial will probably feel familiar to you: You can follow people and have followers, you make posts which people can favourite and reply to and share, and you scroll through posts from people you follow using a timeline. You can write long posts or short posts, or just post images, it's up to you. You can also, of course, block people or otherwise limit interactions that you don't want by posting just to your friends.

Screenshot of the web view of a profile in GoToSocial, showing header and avatar, bio, and numbers of followers/following.

GoToSocial does NOT use recommendation algorithms or collect data about you to suggest content or 'improve your experience'. The timeline is chronological: whatever you see at the top of your timeline is there because it's just been posted, not because it's been selected as interesting (or controversial) based on your personal profile.

GoToSocial is not designed for 'must-follow' influencers with tens of thousands of followers, and it's not designed to be addictive. Your timeline and your experience are shaped by who you follow and how you interact with people, not by metrics of engagement!

GoToSocial doesn't claim to be better than any other application, but it offers something that might be better for you in particular.

Federation

Because GoToSocial uses ActivityPub, you can hang out not just with people on your home server, but with people all over the Fediverse, seamlessly.

the activitypub logo

Federation means that your home server is part of a network of servers all over the world that all communicate using the same protocol. Your data is no longer centralized on one company's servers, but resides on your own server and is shared — as you see fit — across a resilient web of servers run by other people.

This federated approach also means that you aren't beholden to arbitrary rules from some gigantic corporation potentially thousands of miles away. Your server has its own rules and culture; your fellow server residents are your neighbors; you will likely get to know your server admins and moderators, or be an admin yourself.

GoToSocial advocates for many small, weird, specialist servers where people can feel at home, rather than a few big and generic ones where one person's voice can get lost in the crowd.

History and Status

This project sprang up in February/March 2021 out of a dissatisfaction with the safety + privacy features of other Federated microblogging/social media applications, and a desire to implement something a little different.

It began as a solo project, and then picked up steam as more developers became interested and jumped on.

For a detailed view on what's implemented and what's not, and progress made towards beta release, please see the roadmap document. The FAQ contains a higher-level overview.

Features

Mastodon API compatibility

The Mastodon API has become the de facto standard for client communication with federated servers, so GoToSocial has implemented and extended the API with custom functionality.

Though most apps that implement the Mastodon API should work, GoToSocial works reliably with beautiful apps like:

If you've used Mastodon with any of these apps before, you'll find using GoToSocial a breeze.

Granular post settings

It's important that when you post something, you can choose who sees it.

GoToSocial offers public/unlisted/friends-only/mutuals-only/and direct posts (slide in DMs! -- with consent).

It also allows you to customize how people interact with your posts:

  • Local-only posts.
  • Rebloggable/boostable toggle.
  • 'Likeable' toggle.
  • 'Replyable' toggle.

Customizability for admins

Plenty of config options for admins to play around with, including:

  • Easily adjustable post length.
  • Media upload size settings.

Easy to run

No external dependencies apart from a database (or just use SQLite!). Simply download the binary + assets (or Docker container), and run.

GoToSocial plays nice with lower-powered machines like Raspberry Pi, old laptops and tiny $5/month VPSes.

Safety + security features

  • Built-in, automatic support for secure HTTPS with Let's Encrypt.
  • Strict privacy enforcement for posts and strict blocking logic.
  • Import and export allow lists and deny lists. Subscribe to community-created block lists (think Ad blocker, but for federation!).
  • HTTP signature authentication: GoToSocial requires HTTP Signatures when sending and receiving messages, to ensure that your messages can't be tampered with and your identity can't be forged.

Various federation modes

GoToSocial doesn't apply a one-size-fits-all approach to federation. Who your server federates with should be up to you.

  • 'blocklist' mode (default): discover new servers; block servers you don't like.
  • 'allowlist' mode (experimental); opt-in to federation with trusted servers.
  • 'zero' federation mode; keep your server private (not yet implemented).

See the docs for more info.

OIDC integration

GoToSocial supports OpenID Connect (OIDC) identity providers, meaning you can integrate it with existing user management services like Auth0, Gitlab, etc., or run your own and hook GtS up to that (we recommend Dex).

Backend-first design

Unlike other federated server projects, GoToSocial doesn't include an integrated client front-end (i.e., a web app).

Instead, like Matrix.org's Synapse project, it provides a relatively generic backend server implementation, some beautiful static pages for profiles and posts, and a well-documented API.

On top of this API, web developers are encouraged to build any front-end implementation or mobile application that they wish, whether Tumblr-like, Facebook-like, Twitter-like, or something else entirely.

Wishlist

These cool things will be implemented if time allows (because we really want them):

  • Groups and group posting!
  • Reputation-based 'slow' federation.
  • Community decision-making for federation and moderation actions.
  • User-selectable custom templates for rendering public posts:
    • Twitter-style
    • Blogpost
    • Gallery
    • Etc.

Getting Started

All docs for installation and configuration are hosted at docs.gotosocial.org.

Third-Party Packaging

Thank you so much to the cool people who have put time and energy into packaging GoToSocial!

Distribution packaging

These packages are not maintained by GoToSocial, so please direct questions and issues to the repository maintainers (and donate to them!).

Packaging status

Self-hosting

You can deploy your own instance of GoToSocial with the help of:

Known Issues

Since GoToSocial is still in alpha, there are plenty of bugs. We use GitHub issues to track these. The FAQ also describes some of the features that haven't been implemented yet.

Client App Issues

GoToSocial works great with Tusky and Semaphore, but some other client applications still need work or have issues connecting to GoToSocial. We're tracking them right here. It's our goal to make any app that's compatible with the Mastodon API work seamlessly with GoToSocial.

Federation Issues

Since every ActivityPub server implementation has a slightly different interpretation of the protocol, some servers don't quite federate properly with GoToSocial yet. We're tracking these issues in this project. Eventually, we want to make sure that any implementation that can federate nicely with Mastodon should also be able to federate with GoToSocial.

Contributing

You would like to contribute to GtS? Great! ❤️❤️❤️ Check out the issues page to see if there's anything you intend to jump in on, and read the CONTRIBUTING.md file for guidelines and setting up your dev environment.

Building

Instructions for building GoToSocial from source are in the CONTRIBUTING.md file.

Releases

Stable

We package our stable releases for both binary builds and Docker containers, so that you don't have to build from source yourself.

Check our releases page and our getting started documentation.

The Docker image superseriousbusiness/gotosocial:latest will always correspond to the latest stable release. Since this tag is overwritten frequently, you may want to use Docker CLI flag --pull always to ensure that you always have the most up-to-date image every time you run using this tag. Alternatively, run docker pull superseriousbusiness/gotosocial:latest manually just before use.

Snapshots

We also make snapshot builds every time something is merged into the main branch, so you can run from whatever code is on main if you wish.

Please be warned that you do so at your own risk! We try to keep main working properly, but we make absolutely no guarantees. Take a stable release instead if you're unsure.

Docker

To run from main using Docker, use the snapshot Docker tag. The Docker image superseriousbusiness/gotosocial:snapshot will always correspond to the latest commit on main. Since this tag is overwritten frequently, you may want to use Docker CLI flag --pull always to ensure that you always have the most up-to-date image every time you run using this tag. Alternatively, run docker pull superseriousbusiness/gotosocial:snapshot manually just before use.

Binary release .tar.gz

To run from main using a binary release, download the appropriate .tar.gz file for your architecture from our self-hosted Minio S3 repository.

Snapshot binary releases in the S3 bucket are keyed by Github commit hash. To get the latest one, sort by Last Modified, or check out the list of commits here, copy the SHA of the latest one, and paste it in the Minio console filter. Snapshot binary releases are expired after 28 days, to keep our hosting costs down.

Contact

For questions and comments, you can join our Matrix space at #gotosocial-space:superseriousbusiness.org. This is the quickest way to reach the devs. You can also mail [email protected].

For bugs and feature requests, please check to see if there's already an issue, and if not, open one or use one of the above channels to make a request (if you don't have a Github account).

Credits

Libraries

The following open source libraries, frameworks, and tools are used by GoToSocial, with gratitude 💕

Image Attribution and Licensing

Sloth logo by Anna Abramek.

Creative Commons License
The GoToSocial sloth mascot is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

The Creative Commons Attribution-ShareAlike 4.0 International License license applies specifically to the following files and subdirectories of this repository:

Under the terms of the license, you are free to:

  • Share — copy and redistribute the abovementioned material in any medium or format.
  • Adapt — remix, transform, and build upon the abovementioned material for any purpose, even commercially.

Team

In alphabetical order (... and order of smell):

Special Thanks

A huge thank you to CJ from go-fed: without your work, GoToSocial would not have been possible.

Thanks to everyone who has used GtS, opened an issue, suggested something, given funding, and otherwise encouraged or supported the project!

Sponsorship + Funding

Please note: GoToSocial has NO CORPORATE SPONSORS and does not desire corporate sponsorship. In addition, we do not take donations from any of the following: adult websites, affiliate and review websites, casinos and gambling, insurance and financial products (credit), pharmacy products, SEO services and social media buying, VPN and proxy services, and essay writing services. Donations from such sources will be automatically rejected.

Crowdfunding

open collective Standard Sloth badge open collective Stable Sloth badge open collective Special Sloth badge open collective Sugar Sloth badge

If you would like to donate to GoToSocial to keep the lights on during development, you can do so via our OpenCollective page!

LiberaPay patrons receives via LiberaPay

If you prefer, we also have an account on LiberaPay! You can find that right here.

Crowdfunded donations to our OpenCollective and Liberapay accounts go towards paying the core team, paying server costs, and paying for GtS art, design, and other bits and bobs.

💕 🦥 💕 Thank you!

NLnet

NGIZero logo

Combined with the above crowdfunding sources, 2023 Alpha development of GoToSocial is also funded by a 50,000 EUR grant from the NGI0 Entrust Fund, via NLnet. See here for more details. The successful grant application is archived here.

License

the gnu AGPL logo

GoToSocial is free software, licensed under the GNU AGPL v3 LICENSE. We encourage forking and changing the code, hacking around with it, and experimenting.

See here for the differences between AGPL versus GPL licensing, and here for FAQ's about GPL licenses, including the AGPL.

If you modify the GoToSocial source code, and run that modified code in a way that's accessible over a network, you must make your modifications to the source code available following the guidelines of the license:

[I]f you modify the Program, your modified version must prominently offer all users interacting with it remotely through a computer network (if your version supports such interaction) an opportunity to receive the Corresponding Source of your version by providing access to the Corresponding Source from a network server at no charge, through some standard or customary means of facilitating copying of software.

Copyright (C) GoToSocial Authors

(I'm adding this here to take the crown of having the 1000th commit ~ kim)

gotosocial's People

Contributors

blackle avatar daenney avatar decentral1se avatar dependabot[bot] avatar echedellelr avatar f0x52 avatar forestjohnson avatar fruitsbat avatar gaykitty avatar handlerug avatar i5heu avatar igalic avatar illfygli avatar kaja47 avatar liclac avatar littlefox94 avatar lzap avatar martijndeb avatar mirabilos avatar moan0s avatar noracodes avatar nyaaawhatsupdoc avatar oniricorpe avatar sentynel avatar technomancy avatar terinjokes avatar thesuess avatar tsmethurst avatar tsuribori avatar vyrcossont avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

gotosocial's Issues

Don't log sensitive info

We need to make sure we're not logging sensitive information with logrus or the in-build gin-gonic request logger.

An example of sensitive information that's logged: access tokens in streaming request query parameters.

We should be able to customize the gin logger: https://github.com/gin-gonic/gin/blob/master/logger.go

For the logrus logging, we just need to be sensible and make sure we're not logging passwords / tokens etc

[feature] Proper parsing of statuses into HTML representation upon creation

Right now when a new status is created by a local account through a POST to /api/v1/statuses (entrypoint https://github.com/superseriousbusiness/gotosocial/blob/main/internal/api/client/status/statuscreate.go), it is not formatted in HTML but stored as plaintext.

This should be changed in line with other fediverse implementations like Mastodon, which add proper <p> tags and <href>s and whatnot to statuses.

This functionality should be added here: https://github.com/superseriousbusiness/gotosocial/blob/main/internal/util/statustools.go#L109

However, it could be the case that the database is required to properly format the status, including links to other profiles created by mentions, tags, emojis, etc.

If so, then this function should be moved into the processor rather than the util package: https://github.com/superseriousbusiness/gotosocial/blob/main/internal/message/statusprocess.go

[feature] Account `move` activity for instance migration

Mastodon allows an account to migrate from one instance to another, as documented here:

https://docs.joinmastodon.org/user/moving/#migration

Under the hood, this uses a combination of the activitypub move activity, and the alsoKnownAs field on profiles, as documented here:

https://docs.joinmastodon.org/spec/activitypub/#profile

It would be very cool if GoToSocial was able to support this as well, to allow accounts to move freely across instances with different AP implementations and features.

In order to do this, we need to first add alsoKnownAs to the library that GoToSocial uses for federation: https://github.com/go-fed/activity -- using astool: https://github.com/go-fed/activity/tree/master/astool. Then, either we make a PR against that repo (preferred), or we fork the repo and use our own fork (much less preferred, and silly).

Once that's done, we need to implement some functionality in GtS for setting alsoKnownAs -- probably through a PATCH to the /api/v1/accounts/update_credentials path (https://github.com/superseriousbusiness/gotosocial/blob/main/internal/api/client/account/accountupdate.go). This PATCH would update this field in the account model: https://github.com/superseriousbusiness/gotosocial/blob/main/internal/gtsmodel/account.go#L112

Finally, we'd need to implement the move activity in the federating database -- https://github.com/superseriousbusiness/gotosocial/tree/main/internal/federation/federatingdb -- and process side effects in the processor somewhere -- https://github.com/superseriousbusiness/gotosocial/blob/main/internal/processing/fromfederator.go

[feature] Follower vs friend distinction

There should be a way of distinguishing between a 'follower' -- someone who gets your public and unlisted posts delivered to them -- and a 'friend' -- someone who sees your 'friends only' posts.

[feature] import AP json objects into account

mastodon exports the raw activitypub json for posts for each account, i think it would be nice to not worry about losing all my posts when/if a user decided to change to gotosocial

(automatic) asset bundling

The html templates currently use some npm stuff to bundle the stylesheet variables etc, and the /admin interface will probably use more (#78).
We can provide pre-bundled files in the repo so instances happy with the defaults don't have to install node/npm and do bundling, but it would probably be good to make sure the bundles in repo stay up to date.
Github Action stuff maybe? idk
The Dockerfile could also do bundling itself I think

Hide\blur images when sensitive flag set to true

This is a known missing feature so far as I understand (as field was missing from go-fed) so just raising the ticket as a placeholder.

When I post a sensitive\content warning status via the API with an image attached, the status is correctly collapsed behind the subject but currently the image is not hidden, even if sensitive=true.

[security] Make sure 404s from invisible statuses and 404s from bad routes are identical

A baddie shouldn't be able to make inferences about whether or not a status/route exists by comparing error responses.

Example:

  1. a route doesn't exist, someone visits it, gets a nice 404 page.
  2. a route does exist but it gives a 404 because it's not meant to be visible to the caller, someone visits, gets a very simple json error message.

By comparing the responses, someone can eventually figure out which routes exist but they can't see, and which routes don't exist, leading to information leakage.

Returned errors should be consistent, and the returned content type should also (probably) be based on the request's Accept headers as well.

unable to follow accounts with appDomain set up

I've set my GtS instance up using the config.yaml and a different appDomain and host domain and when I attempt to follow anyone (via pinafore or tusky) the instance is logging either 400 Bad Request or 401 Unauthorized

It seems like the request are accompanied with a webfinger request which is using the host domain rather than appDomain.

appDomain: example.org
host: gts.example.org

format of resource=acct:[email protected] rather than [email protected]

If I manually check webfinger for the host formatted account I get an error "no 'resource' in request query" whereas if I manually check with the appDomain formatted account ([email protected]), I get the correct account record returned.

nginx redirects work for appDomain to host for .well-known/webfinger and .well-known/nodeinfo

When I try to log in to either pinafore or tusky using the appDomain, I get an error, only logging in via the host url works.

pinafore error: Error: NetworkError when attempting to fetch resource.. Is this a valid Mastodon instance? Is a browser extension blocking the request? Are you in private browsing mode? If you believe this is a problem with your instance, please send this link to the administrator of your instance.

tusky error: Failed authenticating with that instance

[feature] Rate limiting of API requests

There's no rate-limiting for inbound requests currently, and this should be implemented to avoid the server being spammed/scraped by bad eggs.

There are a couple of middlewares available for Gin to do rate limiting:

These should be evaluated to see if they fit the needs of GoToSocial.

Ideally, whatever rate limiting implementation is used should have the following characteristics:

  1. Use the same rate-limit response headers as Mastodon for compatibility with Mastodon applications.
  2. Use the client IP as a rate-limiting key.
  3. In-memory implementation (good-enough for now).
  4. Sliding window limits (might be a bit complicated though -- fixed limits per minute would already be a fine start).

The rate limit implementation should be added to the security package here: https://github.com/superseriousbusiness/gotosocial/tree/main/internal/api/security

[feature] Landing page at /

There should be a nice landing page at the root of the instance that shows:

  • Some explanation about the instance (description) with instance header and profile images.
  • Basic server configuration/software version.
  • Number of users/number of statuses.
  • Links to apps for using the instance (Tusky, Pinafore.social, etc)
  • Instructions for registering.
  • Link to GoToSocial source code.
  • Etc.

This should be created as a template in the /web/template directory of the repository, which allows instance owners to overwrite it as they wish.

cross-platform build issue

My Go gets compiled with (I guess) GLIBC_2.32 on Arch Linux, so that's an issue on the Ubuntu LTS testingtesting123.xyz server:
./gotosocial: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.32' not found (required by ./gotosocial-test)`

html special characters encoded in status text

I have a php script that is triggered when I post to my website via micropub. The script, amongst other things, does a htmlspecialchars_decode on the status text prior to syndication.

Locally, when I run the code and print the output special characters such as ' are correctly rendered so I think it works OK.

When the status is posted via GtS API the special character is encoded so ' is displayed as $#39;

Example from my website - rendered as posted:
Screenshot_2021-08-11_13-06-27

Example from pinafore of the same snippet, after syndication to GtS:
Screenshot_2021-08-11_13-07-07

GtS doesn't properly handle remote status searches for URL rather than URI

Searching for a note by its URL doesn't always work, because GtS gets confused by redirects.

For example, searching https://social.schumacher.im/notice/A7Vctol0jQdJEdsYTo doesn't resolve.

However, searching https://social.schumacher.im/objects/acc304f2-e509-4eb1-b87b-66a1b6b506f3 does resolve, because this is the ActivityPub IRI of the status, rather than the web URL.

Users should be able to search using either the URL or the URI of a status and have the same results.

The issue is likely either here: https://github.com/superseriousbusiness/gotosocial/blob/main/internal/processing/search.go#L117-L142 or here:

func (d *deref) GetRemoteStatus(username string, remoteStatusID *url.URL, refresh bool) (*gtsmodel.Status, ap.Statusable, bool, error) {
new := true
// check if we already have the status in our db
maybeStatus, err := d.db.GetStatusByURI(remoteStatusID.String())
if err == nil {
// we've seen this status before so it's not new
new = false
// if we're not being asked to refresh, we can just return the maybeStatus as-is and avoid doing any external calls
if !refresh {
return maybeStatus, nil, new, nil
}
}
statusable, err := d.dereferenceStatusable(username, remoteStatusID)
if err != nil {
return nil, statusable, new, fmt.Errorf("GetRemoteStatus: error dereferencing statusable: %s", err)
}
accountURI, err := ap.ExtractAttributedTo(statusable)
if err != nil {
return nil, statusable, new, fmt.Errorf("GetRemoteStatus: error extracting attributedTo: %s", err)
}
// do this so we know we have the remote account of the status in the db
_, _, err = d.GetRemoteAccount(username, accountURI, false)
if err != nil {
return nil, statusable, new, fmt.Errorf("GetRemoteStatus: couldn't derive status author: %s", err)
}
gtsStatus, err := d.typeConverter.ASStatusToStatus(statusable)
if err != nil {
return nil, statusable, new, fmt.Errorf("GetRemoteStatus: error converting statusable to status: %s", err)
}
if new {
ulid, err := id.NewULIDFromTime(gtsStatus.CreatedAt)
if err != nil {
return nil, statusable, new, fmt.Errorf("GetRemoteStatus: error generating new id for status: %s", err)
}
gtsStatus.ID = ulid
if err := d.populateStatusFields(gtsStatus, username); err != nil {
return nil, statusable, new, fmt.Errorf("GetRemoteStatus: error populating status fields: %s", err)
}
if err := d.db.PutStatus(gtsStatus); err != nil {
return nil, statusable, new, fmt.Errorf("GetRemoteStatus: error putting new status: %s", err)
}
} else {
gtsStatus.ID = maybeStatus.ID
if err := d.populateStatusFields(gtsStatus, username); err != nil {
return nil, statusable, new, fmt.Errorf("GetRemoteStatus: error populating status fields: %s", err)
}
if err := d.db.UpdateByID(gtsStatus.ID, gtsStatus); err != nil {
return nil, statusable, new, fmt.Errorf("GetRemoteStatus: error updating status: %s", err)
}
}
return gtsStatus, statusable, new, nil
}

Note that this issue probably isn't caused by the http client not following redirects, because the default client used in GtS for dereferencing should follow up to 10 redirects before stopping.

[bug] Tusky: searching for @user@instance goes into a UI discovery loop

I don't have a nice way to take a screencast from my phone right now but when I search for @user@instance in the tusky interface, when the account is found, the interface starts to endlessly list them in a loop. You can still choose the first user found and the follow works out, so this is not blocking me in any way. Hope that is clear 👍

Consider _FILE convention for secrets in deployment

GTS_DB_PASSWORD could be read from a file then by passing GTS_DB_PASSWORD_FILE.

This is something https://hub.docker.com/_/postgres supports for example:

image

This makes it possible to avoid having to work around the fact that Docker swarm does not make it possible to pass secrets in via the environment with a custom entrypoint hack like in https://git.autonomic.zone/coop-cloud/gotosocial/src/commit/62ecb0fd879c8ac0145d87414e24cbd9435fd305/entrypoint.sh#L23.

SQLite usage

For small (and single-user) instances, it would alleviate the maintenance burden to allow running an instance using SQLite as RDBMS.

Would you consider it something possible for this project?

Feature request: Markdown support

A consideration for a bit further down the road perhaps? I know that opinions a bit split in the fediverse with regards to the degree that users should be able to manipulate formatting but for my somewhat unique use case it would be great to have markdown input handled by the API.

That use case being that I post to my site using markdown and then it is syndicates to GtS, even now with pleroma they only handle markdown at the front-end, not via the API so posts including markdown formatted links are somewhat ugly when they hit pleroma.

[feature] Subscribe-able allowlists / denylists

Instances should be able to subscribe to a public list (some kind of JSON format hosted on, for example, github or anywhere else) to populate their allow/deny lists. This allows for the possibility of community-moderated blocklists or indeed community-moderated allowlists.

Eg., you have a github repo (or whatever) where people can make PRs to add entries to the allowlist or the denylist. Discussion happens. The entry is added or not. Instances subscribed to that allowlist/denylist update (perhaps every 24hrs automatically or with a manual action or even a git webhook??). If you don't like the list, fork it and make your own. Or just don't use it at all.

Note: this should, of course, happen at an instance level not a user level, to avoid targeting individual users.

Migrate go-pg => bun

Currently we use go-pg as our ORM for managing Postgres connections, but this has now been put in maintenance mode: https://github.com/go-pg/pg#maintenance-mode

Bun is a rewrite of go-pg from the same devs that should also be able to connect to SQLite using the same query language and syntax. https://bun.uptrace.dev/guide/pg-migration.html

It also claims to be a little bit faster.

We should look at migrating from go-pg to bun in order to stay up to date and get these new features/improvements.

this will affect #122

[infrastructure] Move away from Github to a better solution (Gitea?)

Since Github is full-on stealing licensed code these days, among other problems, GoToSocial shouldn't stay here in the long run.

One good looking alternative is https://gitea.io/en-us/ which we could self-host somewhere.

We'd have to decide where to host it exactly -- on some cloud infrastructure? On a raspberry pi somewhere? (i have a spare in my drawer).

We'd also have to decide under which domain name to host it. git.superseriousbusiness.org is the most likely candidate, but we also could do git.gotosocial.org if it's just going to have GtS on it.

Co-op Cloud deployment questions: reverse proxy & CLI flags / env vars

Thanks for all this wonderful work!

I'm looking into packaging gotosocial for https://coopcloud.tech for early testing and as we use https://doc.traefik.io/traefik/ as our default automagic "take care of TLS for me" reverse proxy, I am wondering how that interacts with the built-in (!) automagic TLS handling in gotosocial. Can gotosocial be run in the usual HTTP mode? Do I just set --letsencrypt-enabled=false?

Also, I'm wondering what is the default minimal configuration I need to run the binary? When I run it just plain server start, I see FATA[0000] error parsing config: host was not set and I am wondering are there docs to help with knowing what is mandatory and what isn't?

Finally, is it possible to pass command line flags as environment variables? That woule be handy.

I can help write docs if you point me in the right direction 🚀

GtS <> Pixelfed incompatibility

Hi,

as already pointed out at the matrix channel, there seems to be a bug/incompatibility between gts & pixelfed.

here are some logs from both sides:

follow request gotosocial > pixelfed

pixelfed:

app_1     | 172.29.0.3 - - [08/Aug/2021:18:59:29 +0000] "GET /users/egon0 HTTP/1.1" 200 2771 "-" "gotosocial social.netzspielplatz.de (go-fed/activity v1.0.0)"
app_1     | 172.29.0.3 - - [08/Aug/2021:18:59:29 +0000] "POST /users/egon0/inbox HTTP/1.1" 200 207 "-" "gotosocial social.netzspielplatz.de (go-fed/activity v1.0.0)"
app_1     | 172.29.0.3 - - [08/Aug/2021:18:59:29 +0000] "GET /i/actor HTTP/1.1" 200 1212 "-" "gotosocial social.netzspielplatz.de (go-fed/activity v1.0.0)"
worker_1  | [2021-08-08 18:59:29][80d666e0-fc2e-4463-8c38-858cc08f9ec4] Processing: App\Jobs\InboxPipeline\InboxValidator
worker_1  | [2021-08-08 18:59:29][80d666e0-fc2e-4463-8c38-858cc08f9ec4] Processed:  App\Jobs\InboxPipeline\InboxValidator

gotosocial:

app_1    | [GIN] 2021/08/08 - 18:59:28 | 204 |      23.397µs |      172.29.0.3 | OPTIONS  "/api/v1/accounts/019J5XHB1WYP8Y9M42HB34XVQJ/follow"
app_1    | time="2021-08-08T18:59:29Z" level=debug msg="received NEWID request for asType {\"@context\":\"https://www.w3.org/ns/activitystreams\",\"actor\":\"https://social.netzspielplatz.de/users/egon0\",\"id\":\"https://social.netzspielplatz.de/users/egon0/follow/016WGEGGVVHKV78XJ926Y3KDX8\",\"object\":\"https://pixel.netzspielplatz.de/users/egon0\",\"to\":\"https://pixel.netzspielplatz.de/users/egon0\",\"type\":\"Follow\"}" asType=Follow func=NewID
app_1    | time="2021-08-08T18:59:29Z" level=debug msg="received CREATE asType {\"@context\":\"https://www.w3.org/ns/activitystreams\",\"actor\":\"https://social.netzspielplatz.de/users/egon0\",\"id\":\"https://social.netzspielplatz.de/users/egon0/follow/016WGEGGVVHKV78XJ926Y3KDX8\",\"object\":\"https://pixel.netzspielplatz.de/users/egon0\",\"to\":\"https://pixel.netzspielplatz.de/users/egon0\",\"type\":\"Follow\"}" asType=Follow func=Create
app_1    | time="2021-08-08T18:59:29Z" level=debug msg="entering GETOUTBOX function" func=GetOutbox
app_1    | time="2021-08-08T18:59:29Z" level=debug msg="entering SETOUTBOX function" func=SetOutbox
app_1    | time="2021-08-08T18:59:29Z" level=debug msg="performing GET to https://pixel.netzspielplatz.de/users/egon0" func=Dereference
app_1    | [GIN] 2021/08/08 - 18:59:29 | 200 |  473.550687ms |      172.29.0.3 | POST     "/api/v1/accounts/019J5XHB1WYP8Y9M42HB34XVQJ/follow"
app_1    | time="2021-08-08T18:59:29Z" level=debug msg="entering ACTORFOROUTBOX function with outboxIRI https://social.netzspielplatz.de/users/egon0/outbox" func=ActorForOutbox inboxIRI="https://social.netzspielplatz.de/users/egon0/outbox"
app_1    | time="2021-08-08T18:59:29Z" level=debug msg="entering GET function" func=Get id="https://social.netzspielplatz.de/users/egon0"
app_1    | time="2021-08-08T18:59:29Z" level=debug msg="is user path! returning account" func=Get id="https://social.netzspielplatz.de/users/egon0"
app_1    | time="2021-08-08T18:59:29Z" level=debug msg="performing GET to https://pixel.netzspielplatz.de/i/actor#main-key" func=Dereference
app_1    | time="2021-08-08T18:59:29Z" level=info msg="not authorized" func=UsersGETHandler url=/users/egon0
app_1    | [GIN] 2021/08/08 - 18:59:29 | 401 |   84.709351ms |      172.29.0.3 | GET      "/users/egon0"

search on pixelfed for a gotosocial user:

gotosocial:

app_1    | time="2021-08-08T19:01:47Z" level=debug msg="aborting request because resource query [email protected] could not be split by 'acct:'" func=WebfingerGETRequest user-agent="GuzzleHttp/6.5.5 curl/7.64.0 PHP/7.4.22"
app_1    | [GIN] 2021/08/08 - 19:01:47 | 400 |      88.051µs |      172.29.0.3 | GET      "/.well-known/webfinger?resource=egon0%40social.netzspielplatz.de"

pixelfed:

app_1     | 172.29.0.3 - - [08/Aug/2021:19:03:36 +0000] "GET /api/search?q=%40egon0%40social.netzspielplatz.de&src=metro&v=2&scope=webfinger HTTP/1.1" 200 1146 "https://pixel.netzspielplatz.de/i/results?q=%40egon0%40social.netzspielplatz.de" "Mozilla/5.0 (X11; Linux x86_64; rv:90.0) Gecko/20100101 Firefox/90.0"
app_1     | 172.29.0.3 - - [08/Aug/2021:19:03:36 +0000] "GET /api/pixelfed/v1/accounts/verify_credentials HTTP/1.1" 200 1653 "https://pixel.netzspielplatz.de/i/results?q=%40egon0%40social.netzspielplatz.de" "Mozilla/5.0 (X11; Linux x86_64; rv:90.0) Gecko/20100101 Firefox/90.0"
app_1     | 172.29.0.3 - - [08/Aug/2021:19:03:37 +0000] "GET /storage/avatars/008/792/922/758/381/977/6/2nkbJNnkLA3vL4R1t5rO_avatar.jpeg?v=2 HTTP/1.1" 304 126 "https://pixel.netzspielplatz.de/i/results?q=%40egon0%40social.netzspielplatz.de" "Mozilla/5.0 (X11; Linux x86_64; rv:90.0) Gecko/20100101 Firefox/90.0"

Dereference prev + next posts in public threads

Right now threads often appear incomplete because GtS doesn't fetch any of the previous or next posts in a thread, so users see only fragments of conversations.

To offer a more complete view, we should dereference the whole chain of posts that a given post replies to or is replied to by. But we should only do this for posts that are Public (or possibly Unlisted), and not followers-only posts.

This dereferencing could be done here for example:

func (f *federator) DereferenceStatusFields(status *gtsmodel.Status, requestingUsername string) error {

To figure out where replies etc are, we can use the replies field on status activity representations:

  "replies": {
    "id": "https://example.org/users/some_user/statuses/106691726504418392/replies",
    "type": "Collection",
    "first": {
      "type": "CollectionPage",
      "next": "https://example.org/users/some_user/106691726504418392/replies?only_other_accounts=true&page=true",
      "partOf": "https://example.org/users/some_user/106691726504418392/replies",
      "items": []
    }

and the inReplyTo field:

 "inReplyTo": "https://example.org/users/a_different_user/statuses/106691640442578059"

Support `manuallyApprovesFollowers` field in Activitypub profiles

This is the field that Mastodon etc use to determine whether an account is 'locked' -- in other words, whether it manually approves follow requests or not.

The default behavior in GoToSocial is to assume accounts are locked, which leads to confusing behavior where bot + public accounts appear locked, and the user will get a 'sent follow request' status back, rather than a 'following' status.

To support this in GtS, we need to fork https://github.com/go-fed/activity and add the field, then submit a PR for it.

[feature] Database Migration Path for Mastodon => GoToSocial

It should be possible for a Mastodon database to be migrated to a GoToSocial one, since the underlying models are not really dissimilar (the GtS database model is based loosely on the Mastodon one after all).

A plan for this (imo, but I'm very open to debate) might look something like this:

Mastodon => GoToSocial Migration Plan

The most direct way of migrating a whole instance from Mastodon => GoToSocial would be to convert the Mastodon database into a format that GoToSocial can understand + use. So this plan focuses on that. The plan assumes that some kind of command like gotosocial database migrate ... would be added to the GtS command line tool.

Requirements

What do we need to keep, what can we afford to lose.

Crucial

  1. Keep host name and account domain of the instance the same.
  2. Migrate local account data and user data from masto => gts.
  3. Migrate following and followers from masto => gts.
  4. Migrate account public + private keys from masto => gts so remote servers aren't confused as hell when the keys change.
  5. Migrate domain blocks, suspensions, and personal blocks from masto => gts.

Nice-to-have but not sure if crucial

  1. Migrate local acount posts from masto => gts.
  2. Migrate local account media (images, videos, etc) from masto => gts.
  3. Migrate instance settings from masto => gts.
  4. Migrate account preferences from masto => gts.

Probably not necessary

  1. Migrate remote account entries, posts, media. This is probably not necessary because the server can just populate them again moving onwards, assuming they're still available on the remote servers.

Possible fucker-uppers

Obviously in a big migration like this there are gonna be problems we can foresee and problems we can't. Here are some possible issues.

URLs + Paths

ActivityPub IDs are URLs. GoToSocial uses a slightly different template from Mastodon for generating URLs for everything from follow requests, to posts, to account URLs (inbox, following, followers, likes, etc). We have to go through these one by one and make a comparison of what would change and what would stay the same, and figure out how remote servers would cope with that. Otherwise, there's a real risk of totally breaking federation for an account, if the remote servers are still trying to access an account's resources on paths that GoToSocial can't recognize or serve from.

Implementation Ideas

Some ideas for how to actually implement this monster.

Parallel Databases

Let's say Mastodon is running with a Postgres server, with a database called mastodon.

One way of doing the migration would be to have the gotosocial database migrate ... command create a new database on the server called gotosocial.

Then, the tool would parse through entries in mastodon, convert them, and stick them in gotosocial. This would leave the mastodon database intact, in case the database admin wanted to archive it or whatever.

After the migration, Mastodon could be stopped, and GoToSocial could be started.

Advantages

Preserve info in the Mastodon database in case something goes wrong during migration, or the instance admin just changes their mind and wants to go back to Mastodon.

Drawbacks

This is gonna be really intensive on the machine running the migration -- we'd have to throw a lot of CPU and memory at the problem, not to mention disk space.

To be continued....

[feature] Admin web page for instance management

We should have a web page served at https://example.org/admin (and sub-paths) which allows an instance admin to manage settings for the instance. Web pages should follow the templating pattern established for the instance base path by @f0x52, for showing current settings, and make simple POST or PATCH form requests for updating information.

An admin should be able to:

  • view and update instance information
  • create, view, delete domain blocks
  • import and export domain block lists

Instance information updating eg https://example.org/admin/instance

Current instance info can be retrieved by calling /api/v1/instance (see eg: https://testingtesting123.xyz/api/v1/instance), which calls the Processor function here: https://github.com/superseriousbusiness/gotosocial/blob/main/internal/processing/instance.go#L31

Instance info can be updated by calling a PATCH to /api/v1/instance, which calls the Processor function here: https://github.com/superseriousbusiness/gotosocial/blob/main/internal/processing/instance.go#L45

The form to submit for an update is modeled here: https://github.com/superseriousbusiness/gotosocial/blob/main/internal/api/model/instance.go#L77

Domain block creation, viewing, and deletion eg https://example.org/admin/federation

Admin should be able to view a list of current domain blocks, probably with pagination since there could be a lot of them (pixie.town and rage.love for example both have hundreds of domain blocks).

Domain blocks can be retrieved by calling /api/v1/admin/domain_blocks, which calls the Processor function here: https://github.com/superseriousbusiness/gotosocial/blob/main/internal/processing/admin.go#L39

A new domain block can be created by a POST to /api/v1/admin/domain_blocks, which calls the Processor function here: https://github.com/superseriousbusiness/gotosocial/blob/main/internal/processing/admin.go#L31 with the following model: https://github.com/superseriousbusiness/gotosocial/blob/main/internal/api/model/domainblock.go#L36

An individual domain block can be viewed by calling /api/v1/admin/domain_blocks/DOMAIN_BLOCK_ID, and deleted by a DELETE to that path.

Domain blocklist import/export https://example.org/admin/federation

To mass import domain blocks, a form can be submitted as a POST to the /api/v1/admin/domain_blocks path given above, where the form field domains is a json file containing an array of domains to block, and the query parameter import is set to true. See here: #77

An admin should be able to upload this file through the admin page as well.

To retrieve a list of blocked domains for sharing elsewhere, without all the domain block IDs and other extraneous information, an admin can make a GET to /api/v1/admin/domain_blocks with the query parameter export set to true. -- This should also be possible through the admin page, a la Grafana's 'export this' functionality where it returns some copy-pasteable JSON.

Text duplication within posts containing hashtags and links

As per the huge wall of logs and screenshots in the matrix room (which I'll add below) - I am experiencing an issue when posting a status containing a hashtag - only noticed now because I almost always forget to add them.

All of the text before and after the hashtag is duplicate in place, as in the text before the hashtag is duplicated before the hashtag and text following it is duplicated following the hashtag.

However, this appears only to happen when there is another link in the post. If there is no link the hashtag itself is linked once posted and there is no duplication.

Screenshots of posts in different scenarios - haven't included the logs posted in the matrix chat as I'm not sure that they are relevant to the duplication issue, possibly just API elements which aren't yet exposed.

gts1
gts2
gts3
gts4
gts5
gts6

[feature] Format links correctly in statuses etc

We should allow users to post links and format them nicely so they're clickable HREFs.

To determine if part of a post is a link, we can use something like this (if we don't feel like rolling our own regular expressions):

https://github.com/mvdan/xurls

We need to parse and replace links with HREF's before extracting hashtags from a status, to make sure we don't include link fragments as hashtags.

[feature] Avatar & profile header descriptions

Image descriptions are useful, accessibility-wise. They are used in posts, but there's at the moment no way of describing profile avatars or head images. Adding these would be a nice change.

[feature/admin] Automatically add CW on federating instance posts

From a post by a Mastodon user who would rather remain anonymous:

'damn it'd be really cool if i could somehow make my instance about cooking automatically add a "food" cw when posts federate out, but not show the cw on the instance itself, this sounds possible but i have no idea how to do it'

also:

'as far as glitch-soc implements it, an existing CW will overwrite the automatic one. the automatic one only applies to posts that don't have a CW'

GoToSocial should be able to do this, it's a really cool idea.

Confusing CLI flag parsing

apparently the correct format is something like ./gotosocial --host test server start
but when doing the imo more logical ./gotosocial server start --host test you just get a very vague flag parsing error :(

[bug] pleroma issues

Having a few issues with pleroma <-> GtS interactivity in relation to searches and follows, as discussed with @tsmethurst in the matrix room.

When I search for @[email protected] from my pleroma instance, the account is returned but it is the @[email protected] format (the same is seen when searching for my GtS account with the same formatting). There are no errors in this scenario, just the incorrect nickname is returned (visually). I can follow this returned account from pleroma.

However, when I attempt to follow my pleroma account from my GtS account it fails.

At the GtS end I see the following error:

[GIN] 2021/07/21 - 16:59:05 | 204 |     467.249µs |    192.168.1.83 | OPTIONS  "/api/v1/accounts/01FB4N70SVQPHBE9F766DG8ND1/follow"
time="2021-07-21T16:59:05+01:00" level=error msg="target account wasn't set on context" asType=Follow func=Create
[GIN] 2021/07/21 - 16:59:05 | 200 |   51.900197ms |    192.168.1.83 | POST     "/api/v1/accounts/01FB4N70SVQPHBE9F766DG8ND1/follow"
[GIN] 2021/07/21 - 16:59:06 | 200 |    7.007773ms |    192.168.1.83 | GET      "/users/jon/main-key"
[GIN] 2021/07/21 - 16:59:06 | 200 |    7.330689ms |    192.168.1.83 | GET      "/users/jon/main-key"
time="2021-07-21T16:59:06+01:00" level=error msg="batch deliver had at least one failure: POST request to https://social.nipponalba.scot/users/jk/inbox failed (400): 400 Bad Request"

At the pleroma end I see the following errors:
Could not validate against known public keys: {:error, :error}
Signature validation error for: https://gts.kelbie.scot/users/jon, make sure you are forwarding the HTTP Host header!

Pleroma version is: 2.3.0

[feature] Instance terms of service \ code of conduct pages

Lots of responsible fediverse users and admin sensibly don't follow users or allow instances to use their relay services without a comprehensive terms of service or code of conduct. It would be good to have this and also great if this was easily editable via the admin interface.

Support migrations for new versions

When new commits roll out and they require database migrations, it would be nice to have some way to automatically run those migrations on the deployment side when deploying the new binary version. I use the Docker deployment myself, so having it there would be handy too. For now, having the commands to run in psql is fine ofc.

(Also discussed in the matrix room and wanted to help keep track of this one too 👍)

Publishing a container image

Forgot to mention that in #62 but an "upstream" (e.g. published from here) container image would be lovely. For now, I can build my own image for now but yeah, would be cool to have a superseriousbusiness/gotosocial:latest thing from hub.docker.com or whatever. No rush on this!

[feature] Profile page for a user

A user should (if they wish) be able to display their profile page at their account's URL. This can be pretty basic and static, but should at least show:

  • Their profile pic and header.
  • Followers/following (if desired).
  • Number of posts (if desired).
  • Public posts (if desired).

The template for the page should be added at /web/templates so it can be overwritten by server admins if desired.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.