kryptokrona/hugin-api

RESTful API and WebSocket server for the Hugin Messenger service.

Hugin API

Hugin is a messaging service where messages are stored on the Kryptokrona Blockchain. Messages are secured with industry leading encryption so that only you and your chat partner have the possibility to read them. The purpose is not to gather any personal information about the users, unlike most apps and social media.

Hugin also packs some powerful decentralized P2P encrypted features like video, voice, screen sharing and filesharing.

To be able to get data more easily from Hugin this RESTful API and websocket server called Hugin API was created. It provides useful endpoints for within the Kryptokrona project as well as other external 3rd party interests that wants to host their own Hugin API.

Table of Contents

• Development Resources
• Sponsors
• Usages
• Contributors
• License

Development Resources

• Web: https://kryptokrona.org
• Docs: https://docs.kryptokrona.org
• GitHub: https://github.com/kryptokrona/hugin-api
• Slack: https://join.slack.com/t/kryptokrona/shared_invite/zt-1qmt8cz9j-s~2RKYtFfshqeLC1dhU5aA
• Discord: https://discord.gg/VTgsTGS9b7
• Twitter: https://twitter.com/mjovanc

Sponsors

The following sponsors are either helping the project with free usage, great discounts of services or in other ways helping via advertisement.

Usages

Our API is useful! Projects using Hugin API:

• Official Kryptokrona Website
• Kryptokrona Block Explorer
• Hugin Mobile
• Hugin Bridge
• Hugin Node.JS Client
• Hugin Desktop
• Hugin API Twitter
• Hugin Full Node

If you are using Hugin API, please fork and make a pull request into our repo with your added link.

Contributors

The following contributors have either helped to start this project, have contributed code, are actively maintaining it (including documentation), or in other ways being awesome contributors to this project. We'd like to take a moment to recognize them.

License

The license is BSD-3 License.

hugin-api's Issues

Add threads of posts

We need to store information about threads of a specific post when someone replies. Probably we need a junction table to store this data.

Add sample database dump

We need a database dump of sample data for testing the endpoints when starting the application so we don't need to wait for incoming messages.

Definition of done (DOD)

• A PostgreSQL database dump inside https://github.com/kryptokrona/hugin-cache/database/samples
• Updated README.md under the section "Test environment" https://github.com/kryptokrona/hugin-cache#test-environment with instruction on how to import the PostgreSQL dump file.

Add deployment stage in Pipeline using Ansible

Sync data between Hugin Cache apps

Sync data between Hugin Cache databases currently running to avoid mismatch of stored data.

Create integration tests

Add monitoring software

It would be nice to have a monitoring software to setup with Ansible such as Prometheus so we can see statistics of the VPS graphically on the web such as CPU/RAM/disk usage etc.

Definition of Done (DOD):

• Add new role in https://github.com/kryptokrona/hugin-cache/tree/main/ansible/roles called "prometheus" or "grafana"
• Add the role to provision_vps.yml (https://github.com/kryptokrona/hugin-cache/blob/main/ansible/provision_vps.yml)
• Update NGINX conf with a new location block "/monitoring" and point to the prometheus server running (https://github.com/kryptokrona/hugin-cache/blob/main/ansible/roles/nginx/templates/nginx-le.j2)
• Update README.md with instructions of how to use it and where to find it etc.
• Make Prometheus display metrics such as CPU, RAM, Disk usage
• Configure alerts to specified email so it will be sent if it exceeds some disk usage or other metrics.

Bonus additions:

• Add metric such as posts added to API over time
• Add metric of how many hashtags added to API
• Add metric of how many encrypted posts added to API

Store reply to a post

Check if a reply exists and save it to database for a post.

Files that needs to be updated (perhaps more):

• database/models/post.js
• database/migrations/20220426141549-create-post.js
• configs/huginSyncer.js

All unit tests needs to be updated in with new field:

• tests/controllers/post.js

Implement rollback pipeline

If we run the main pipeline and deploy and the smoke test fails we need to handle rollbacks. It should get the previous docker container with the previous commit SHA and deploy it once again. Then we only need to run a deploy job and smoke test again since the docker image is already built since before.

Change date to datetime of post database model

Add websocket endpoint for encrypted posts

Categorize content based on language

All messages should be analysed and categorised into a language category so a user can then select what kind of messages to subscribe to. To be more selective.

Definition of done (DOD):

• Create new function in huginSyncer.js called detectLanguage(text)
• Add new column on post table called "language"
• Create migration file using command

npx sequelize-cli migration:create --name create_language_column

• Update post model with new field
• Use i18n.detectLanguage() see link in comment for detecting the language used for a string
• Update huginSyncer to save the new field in the messageObj
• Update unit test in tests/utils/messageCriteria.js with new field language.
• Update unit test in tests/controllers/post.js with new field language.
• Update seeder in database/seeders/20220426144921-Post.js with new field so the database will populate correctly.

Stress testing of API endpoints

Either using Apache JMeter https://jmeter.apache.org/ or https://www.npmjs.com/package/loadtest for conducting API stress testing.

Configure a clean method to remove old data from previous week/month/year when the maximum storage is getting close (perhaps 80-90%) to the storage limit

Fix NGINX conf SSL directive

2022/05/10 21:14:35 [warn] 59348#59348: the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/sites-enabled/le:13

Show if the API is up and running in the HTML document

We need a way to display in the HTML document if it can not access the API or if it's up and running.

Definition of done (DOD):

• When the API is down, show a message as "Could not connect to Hugin Cache API".
• When the API is up, show a subtile message like "Hugin Cache API is operational".

.dockerignore doesn't ignore files properly when building images

The app crashes with "parent: error: value "1655331326420" is out of range for type integer"

The app crashes with "parent: error: value "1655331326420" is out of range for type integer". See the images below:

Configurable system

We need to implement a way so that a user running the system should be able to:

• Configure a clean method to work directly if the user wants even if the maximum storage is below (80-90%) to remove old data from previous week/month/year

Configure a way to exclude/include incoming data based on criteria

Based on which user, board, content in message, date and so forth

Implement logging

We need to save the log output to a file. This should be mapped to a volume locating outside the Docker container.

Remove frontpage

We don't need the frontend. All the API endpoints is much better suited to be viewed/tested on our Postman account. And since people are going to use this and host it, it just feels unnecessary.

Make API use variable to handle version in source code

We need this to avoid manual work.

Add search/filtering/sorting

Update database diagram of new schema

Get trending posts by tipped, replies, and tipped + replies

Get trending post by how many replies a post has.

Setup iptables/ufw firewall rules for Ansible deployment

Setup docker build/publish stage in pipeline

Onion service

Look over expected responses in unit tests

Look over if there is other responses not covered in implemented unit tests that we need to test.

Implement a websocket server for real time communication

We need a way to get real time data from Hugin Cache to present new incoming data directly to clients using the websocket server on Hugin Cache.

Refactor huginSyncer.js

Too big function, need to split it some parts of it to separate functions.

Configure a maximum storage of data in the database

No Cache control for Kryptokrona.org doesn't work

Check server.js

Add code coverage

Save your own encrypted posts with keypair

Private Hugin Syncer to save encrypted messages that you write using Hugin Messenger.

This could be used by third party to allow for private communication with Hugin Cache server.

known_keys (public keys) which the user can decrypt.

This is to be used with both boards posts and/or encrypted posts.

New table: posts_encrypted

Add rate limit

Add rate limiting to avoid abuse of the API.

API endpoint documentation

We need to generate an API endpoint documentation. Perhaps through GitHub Pages?

Example to use: https://github.com/slatedocs/slate

Usage of Ansible facts

Look over for future usage of Ansible facts instead.

Cache Control

Fix Cache Control settings.

Implement GraphQL

Future improvement for the API.

Creating Docker container with Ansible using recreate option with yes causes failure

Error:

Null values of properties of a post

Some posts that get's created in the DB gets null values for some reason. We need to investigate why. This is what it looks like:

{
            "id": 7,
            "message": null,
            "key": null,
            "signature": null,
            "board": null,
            "time": null,
            "nickname": null,
            "tx_hash": "946a5d2552e3082ab8ec5d1166e08f320cc695a2122855a9585ab8869a2a88bc",
            "createdAt": "2022-05-11",
            "updatedAt": "2022-05-11"
        },

Usage of appropriate status codes

We need to implement appropriate status codes when user request something with for example faulty parameters or similar, currently we get a 500s and that's not a good way.

Remove old junk data in cache.hugin.chat with null values

Write SQL query to remove posts that have null values.

Handle cleanup of old docker images

We need to handle the cleanup of old docker images since it takes up a lot of disk space. We also need to look into volumes.

Add staging server

Add a staging server that we can test deploy to and conduct load testing and smoke test before starting production deployment.

Cache data to improve performance

Add caching to return data from the local memory cache instead of querying the database to get the data every time we want to retrieve some data that users request.

Add pagination

Create mock data for unit tests

Add smoke test