incplusplus/beacon-backend

beacon-backend's Issues

Add a permanent storage for the CIS

The CIS needs to permanently store some data like profile pictures. We should use the same DO Space that was used in #64.

Add basic message handling API to the CIS

Add some basic message sending and retrieving functionality to the CIS server for DMs. This does NOT include Websocket updates. Although, it might be worth looking into whether the client should send new messages using HTTP or via a Websocket message.

Other people's profile pictures don't update

When someone updates their profile picture, it doesn't change for everyone else until they refresh the page. This is because the CIS doesn't notify clients about, well, anything. The CIS doesn't even have any websocket capabilities. While this is a bug behaviorally, it's just an unimplemented feature in reality. This is sorta relevant to IncPlusPlus/beacon-frontend#49.

Implement file storage properly

Currently, file uploads sit on the Heroku dyno. That's a problem because that storage space is ephemeral. Pick an online storage solution and interact with it on the backend. This storage needs to be S3 compatible. We can't just "use" Dropbox or Mega or something.

Current solutions under consideration:

• DigitalOcean Spaces
  • $5/mo for 250 GB of storage and 1TB outbound transfer/mo
  • There's $100 platform credit for new users with the GH Student Pack
• Linode Buckets
  • Linode Object Storage costs a flat rate of $5/month, and includes 250 GB of storage and 1 TB of outbound data transfer. Beyond that, it's $0.02 per GB per month.
• DreamHost Cloud Object Storage
  • $0.95/mo for 40 GB
  • $4.50/mo for 200 GB
• Vultr
  • $5/mo for 250 GB with 1 TB of transfer
• Storj
  • Up to 150 GB/mo free
  • The Java client library only provides bindings for something we have to build ourselves. That's a pain in the ass.

As a follow-up issue, we could keep track of objects so that if the message containing a file is deleted, we could potentially free up storage on our end by deleting the object which no longer has any references to it.

Make towers generate with a #general channel

Currently, new towers are empty on creation. They should start with a #general channel

Split the OpenAPI spec files

This is so that they can use shared schemas. For instance, the CIS is going to be capable of handling DMs while the City is capable of hosting Beacons. Both of these will use the same schema for what constitutes a Message. To avoid duplicating these definitions across two OpenAPI specs, it would be advisable to split them and use $refs to reference them.

So that the OpenAPI spec can still be viewable on the deployed services, they will have to be bundled into one spec file at build time before they have the version applied to them.

See this guide and its complete example for more info.

Fix error status codes

I think I've left a bunch of RuntimeExceptions littered in the codebase. These result in erroneous 500 errors when something like a 404 might be appropriate instead. For instance, fetching the messages of a non-existent channel will result in a 500 instead of a 404.

The production City instance currently uses the QA CIS instance

This was done temporarily just so that we had two City instances running on separate machines with separate databases for demonstration purposes. This should be reverted after we're done with the demo.

Mark required schema properties as required

If there are schema properties that must be specified, it seems like they can be included in the required array which is one of the object-level attributes of the Schema Object (example). Unlike #16, this actually will affect validation, adding the @NotNull annotation to all fields that are marked as required. This means that if a client fails to specify a required field, they'll receive an error.

Fix deprecated `antMatchers` usage

In Spring Boot 2.6+, our usage of antMatchers is deprecated. See https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-2.6-Release-Notes#pathpattern-based-path-matching-strategy-for-spring-mvc.

Add permission checks to endpoints that require it

For example, to delete a message, the user attempting to do so must be the owner of that message. Pretty much any of the endpoints in the City that fall under /tower will need to be gone through and we'll have to verify that the user is a member of that Tower.

Get MongoDB connection functioning in production

We need to get a MongoDB connection working in production.

• I've set up two MongoDB Atlas databases.
• beacon-east.lw78i.mongodb.net (beacon-backend-qa) is to be used by the beacon-cis-main-staging and beacon-city-main-staging Heroku apps
• prodcluster0.bdjwu.mongodb.net (beacon-backend-prod) is to be used by the CIS and City Heroku apps which are promoted to production

Have City check its version with the CIS

People self-hosting their Cities might end up with versions that are too old to be compatible with the CIS and/or the frontend client. We could have a very simple CIS endpoint that the City can contact to check what the latest version is. If we wanted to get really fancy with it, we could have the City determine whether there's been a major version change. However, that'll probably be a follow-up feature to this one.

We might also want a way for the client to ask the City if the City believes it is out of date so this information could be displayed to users. Alternatively, we could have the City push the message to the client by itself through a websocket message.

Create profile & instructions for running a self-hosted City

With the introduction of #15, services now communicate between each other and have database connectivity. The next logical step is to create a profile that is configured to use a local database that the user set up. Then, instructions on how to run the service have to be setup. The profile should be added to .gitignore and the file should be named selfhosted-TEMPLATE.yml.

Add customization fields to Tower

Each Tower needs to store

• Icon displayed in tower list
• Primary Color (Beacon yellow by default)
• Tower Banner image?

Look into graceful shutdown at least for the City

How to send the shutdown command
Make the web side gracefully shut down
Wait for other tasks to complete

Implement a way to find Tower memberships & better City-to-CIS communications

The endpoint to get tower memberships needs to be implemented. That's a simple way of putting it. To be more specific, I need to find a better way for Cities to be registered and for communications to not break down if somebody shuts down their City temporarily or permanently.

• The CIS knows what Cities are registered with it. It can return that list of Cities which includes hostnames that the client could iterate over to find if the user is a member of any Towers within it. However, that's a huge chore and a one-time request would be a lot easier.
• If a one-time request is to be implemented, here are my different ideas of how that could be implemented
  1. Instead of making the client iterate over all the hostnames of the different registered cities, the backend could do it and send back the result to the client.
     • This could cause errors that will be difficult to make clear to the client
  2. Change the City registration system so that the any Cities that are no longer online for 5 minutes are removed from the registry.
     • I'd have to refactor the way that Cities manage their config. However, that's probably not a big deal. The City will just register itself every time on startup and won't retain any config.
     • The CIS could use Spring's task scheduler to periodically (maybe every five minutes) send some kind of request to the City
       • This request could potentially be authenticated by the City with some kind of shared key that the CIS establishes when the City first registers itself on startup.
       • If the City doesn't respond to the CIS request (effectively a reverse heartbeat)
  3. I could stop reinventing the wheel and actually perform service discovery
     • For example, https://blog.scottlogic.com/2019/10/31/building-microservices-with-spring-boot.html

Regardless of which of these I choose, I'll still need to think of a better way to do this. If we make the City registrations ephemeral, we want some way to permanently persist some kind of static City identifier. That way, if the City has to re-register itself, it won't lose its ID which would cause invites and stuff to fail (due to changing the City ID).

Add query params to the GET /messages endpoint for time range and limit

When fetching the messages in a channel, we don't necessarily need every message since that channel's inception. Instead, we most likely want the most recent N messages or potentially a range of messages. One might even want to obtain messages between two timestamps. This issue will be closed once query parameters that would allow such functionality are implemented in this endpoint.

Fix some of the `POST`, `PUT`, and `DELETE` responses

Some of the POST and PUT endpoints respond with 200 instead of 201. I think the endpoint to delete a tower also responds with a 204 which isn't documented in the spec. This is a really small issue and won't cause issues but it just bugs me.

Make some of the OpenAPI schema properties read-only or write-only

@Sciman101 was looking over the OpenAPI spec and asked me if the client was supposed to set things like the ID of a given entity. I completely forgot that the OpenAPI 3.0 specifications allow me to mark properties as read only. This issue may be closed when I've gone through the various objects I've made in the API and add readOnly: true where applicable.

Also, using writeOnly: true on properties like password is something I should've done instead of making a separate CreateAccountRequest.yml for requests to create a new account.

OH YEAH! Also the array of attachment URLs in the Message schema is read only. That gets populated by the server and never the client.

I keep remembering more things. We'd also wanna mark any timestamps as read-only as well.

Fix old OpenAPI specs being cached by Chrome

While this won't cause issues with communication between the services nor between the frontend and the backend, this is a huge PITA. Say the CIS spec is changed and I visit the CIS staging Heroku app, I don't see the new specification until I clear my browser cache. This might be solvable with one of the suggestions here. If not, there's a chance it's springdoc's fault.

Add STOMP topics for when users leave and join a Tower

Right now, a client will need to refresh to get the updated list of Towers they're a member of. This would solve that issue.

There's still one caveat. If the user joins a Tower that's on a City they're not a part of yet, there won't yet be an established websocket connection to that City so there's no way to let the client know about this event. That's a problem for later.

I've realized that with the way the flow between the client, CIS, and City works, we could make the client temporarily listen to the City if it's not already being listened to. That way, we'd receive the event from that City telling us that the user is now a member of a tower we weren't previously aware of.

Profile pictures as well as tower icons + banners not updating

These images, when replaced, won't update. This is because the CDN doesn't get notified about files being modified and keeps the old version. To fix this, a different file path will need to be generated each time one of these images is updated.

Implement means to determine if a message has been edited

Ideally, just a boolean field that is sent alongside other data relevant to a message.