The frontend, editor panel, and API of the TheIndex.
The easiest way is to get started is by using Docker Compose. You need to copy
the docker-compose.yml and example.env file from GitHub. Rename example.env
to .env and adjust the environment variables as you need to. with the following command:
docker-compose up -dYou'll need to change <host-port> to your port of choice. The web-server is not secured via SSL/TLS, it is in your
responsibility to put a reverse proxy in front of this container. When you run the image for the first time, don't
forget to set your own discord-id in the
env SETUP_WHITELIST_DISCORD_ID to be able to login and edit. Once your container has set up itself once, you can
remove the env variable from your setup.
We use mongodb as our database server. You can deploy your own mongo setup as HA service or just a simple single docker container via e.g.:
Note: The database will start empty, you have to fill the data yourself.
docker run -d \
--name mongo \
-v ./db:/data/db \
mongoFor development or testing purposes it is highly recommended to use mongo-express for accessing, viewing and editing the current state of the database. If you make it publicly accessible, don't forget to secure it with login credentials.
To simply spin up a mongo-express docker container, run:
docker run -d \
--name mongo-express \
-p 8081:8081 \
mongo-expressYou can also take a look at our provided docker-compose file on how to set it up.
To increase performance we use redis to cache results from the mongoDB. The cache is being autopopulated on cache-misses.
You can create a new instance with docker by running:
docker run -d \
--name redis \
redisThe redis db is already included in the example docker-compose file
Warning: be aware, that we do not offer any kind of official support and every update may be with breaking changes. Be sure to make backups before you update
To get the newest version of the container image, you will need to run:
docker pull ghcr.io/snaacky/theindex:latestAfterwards, you will need to stop and remove your current running instance and start it again.
Here is a collection of the possible environment variables with their default values you should set in your .env file:
| Parameter | Function | Default |
|---|---|---|
NEXT_PUBLIC_SITE_NAME |
The name of your site | "The Index" |
NEXT_PUBLIC_DOMAIN |
Your domain or IP, remove trailing slash | "https://theindex.moe" |
NEXTAUTH_URL |
Your domain or IP, remove trailing slash | $NEXT_PUBLIC_DOMAIN |
DATABASE_URL |
Take a look at mongodb docs | "mongodb://mongo:27017/index" |
CACHE_URL |
Connection string for the redis cache database | "redis://redis:6379" |
CHROME_URL |
WebSocket URL to a running chrome instance | "ws://chrome:3300" |
AUDIT_WEBHOOK |
WebHook-URL for audit-log, leave empty to disable support | "" |
DISCORD_CLIENT_ID |
Discord OAuth2 client ID | "your_discord_oauth_client_id" |
DISCORD_CLIENT_SECRET |
Discord OAuth2 client secret | "your_discord_oauth_client_secret" |
DISCORD_BOT_TOKEN |
Required to access BOT resources | "your_discord_bot_token" |
SETUP_WHITELIST_DISCORD_ID |
If you need help getting your id, check out this guide | "your_discord_id" |
And the following env variables are only needed when you are in dev mode and debugging the db
| Parameter | Function | Default |
|---|---|---|
ME_CONFIG_BASICAUTH_USERNAME |
mongo-express username | "admin" |
ME_CONFIG_BASICAUTH_PASSWORD |
mongo-express password | "SUPER_SECRET" |
If you want to verify how the docker-compose file fills the envs in, use docker-compose config
-
Getting started isn't that straight forward. You will need to have installed the latest version of docker with docker-compose on your machine.
-
Start by cloning the repo via a graphical git client (highly recommended) or use the cli via
git clone https://github.com/snaacky/theindex- Copy the
example.envfile to.env.
- Replace
NEXT_PUBLIC_DOMAINandNEXTAUTH_URLwithhttp://localhost:3000 - Generate a random strong string of at least 32 characters and use it for
NEXTAUTH_SECRET. You can use generators e.g. 1Password or create it yourself. - Change
DATABASE_URL,CACHE_URLandCHROME_URLto use localhost instead ofmongo,redisandchromefor example:mongodb://mongo:27017-> becomesmongodb://localhost:27017 - Go to
https://discord.com/developers-> Create a new Application -> Go intoAuth2inside your Application panel -> Copy theCLIENT IDandCLIENT SECRETinto the.envfile. - In
RedirectsinAuth2copy and paste the following URL needed to verify your Discord loginhttp://localhost:3000/api/auth/callback/discord. - Fill
SETUP_WHITELIST_DISCORD_IDwith your Discord ID to have an admin account when you login. - Generate again another random strong string of at least 32 characters and use it for
MEILI_MASTER_KEY. You can use generators e.g. 1Password or create it yourself. - Lastly if you want to change the password for mongo-express, you can do so with the last two variables, usually though none of us do while developing locally.
- Add the following ports to the images in the
docker-composefile:
| service | port-mapping |
|---|---|
mongo |
27017:27017 |
redis |
6379:6379 |
meili |
7700:7700 |
chrome |
3300:3000 |
As an example, the setup for mongo should look similar to this:
mongo:
image: mongo
container_name: index-db
restart: unless-stopped
ports:
- '27017:27017'
volumes:
- ./db:/data/db- Now run the command to start all the needed backend processes
docker-compose up -d mongo redis meili chrome mongo-expressAlternatively you can also just comment or remove the index service and run the command
docker-compose up -dTo start coding on the frontend, you will need to make sure, you have the latest version of node.js correctly installed. To install all the required dependencies run once:
npm installNote: We decided to stick with npm instead of yarn to manage dependencies.
You should now have a folder called node_modules, which contains all the dependencies we need. We use
Next.js as framework for our React web service. To test the web service you
will have to run a db server in the background and start the frontend via:
npm run devAfter compiling you can open http://localhost:3000 in your browser of choice and see the running web application.
As we use Next.js, the frontend supports hot reloading, so you can just leave the page open, while you modify the code and see the changes on the fly in your browser.
To create a ready-made docker image, just run:
docker build . -t indexYou now have a local image with tag index that contains a distributable version of the code which can now be run.
We use prettier to ensure a consistent code style across all participants. You can simply auto-format everything with e.g. running the command
npx prettier --write .Where possible we use ISR to pre generate all publicly accessible pages for caching by CDNs or proxies while validating and fetching new data with SWR requesting our own api.
We serve every api request over the endpoint /api, the corresponding code can be viewed at pages/api.
/api/authis reserved for NextAuth.js./api/edit/...requires a logged-in user and usually (at least) editor rights and is for modifying or creating new content. The_idkeyword_newis reserved for creating new content./api/delete/...requires a logged-in user and usually (at least) editor rights and is for deleting content./api/[content]sare public endpoints for fetching a list of all items of a certain content./api/[content]/...are public endpoints for fetching information about a specific content.
Every page request first has to go through _app.ts, where a basic layout is being applied and if a
page has an auth property, it also validates whether the user can access the given page. Valid auth attributes are:
authitself isnullortypeof auth === "undefined", no restrictions. This seems to be a public page.requireLogin, not really needed, but set it for logic reasons. User must be logged-in.requireAdmin, only a user with admin rights can access this page.requireEditor, only editors can view this page.
Pull-requests are always welcome, but may not be always merged as it has to be in align with our idea of the index. If
you want a certain feature or have an idea, you can always open a feature request
in Issues
or report it on our Discord in #index to be discussed. If it is not bad, in align with
our ideas, and we find some time, we will certainly implement your requested feature (sometime...).
- next.js as Webserver
- React as JS framework
- Styles from bootstrap
- Icons from Font Awesome
- We bake everything into a docker image
- Running on node js
- User-authentication via NextAuth.js
- mongodb as database backend
- redis as cache
- puppeteer to create screenshots of websites
- browserless/chrome to run our chrome instance
- react-in-viewport for checking if components are visible or not
- react-draggable for dragging components by the end user
- react-toastify for handling notifications
- iso-639-3 for the languages
And most importantly:
The help of our awesome community :3
- We use JS instead of TS (currently in migration)
- Unify
Editor-views - Unify db insert and updates to the format of
func(_id, dataObject)and update only as needed, GraphQL would be nice... - Move to
add,removeapi instead of having to send whole arrays to update lists -> GraphQL?
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent