dailydotdev/daily-api

Daily API V2

Provide the daily.dev feed and engagement layer

The project started as a monolith service for everything the extension needed (thus Daily API). Safely and slowly it was torn apart to other services to make every service have only one responsibility. The so called Daily API kept to itself the content domain. The feed you know very well is delivered from here, along with other very useful endpoints.

Stack

• Node v18 (a .nvmrc is presented for nvm users).
• NPM for managing dependencies.
• Fastify as the web framework
• Apollo for GraphQL
• Typeorm as a database layer

Project structure

• __tests__ - There you can find all the tests and fixtures. Tests are written using jest.
• bin - Folder with utilities and executables.
• helm - The home of the service helm chart for easily deploying it to kubernetes.
• seeds - JSON files with seed data for local development.
• src - This is obviously the place where you can find the source files.
  • common - Utility functions that are used across the project.
  • compatibility - Fastify routes to keep backwards compatibility with API v1.
  • cron - Tasks that will be deployed as cron jobs.
  • directive - GraphQL schema directives.
  • entity - Typeorm entities that are used to communicate with the database and sync its schema.
  • migration - Typeorm migrations folder to update the database schema.
  • schema - Apollo GraphQL resolvers, including also types.
  • workers - Pub/Sub message handlers that are deployed as part of the background processor.

Local environment

Daily API requires a running instance of PostgreSQL, you can easily set it up using the provided docker-compose file. Check out this guide of how to install Docker Compose. Once installed, you can run docker-compose up -d and viola!

Make sure to apply the latest migrations by running: npm run db:migrate:latest

.env is used to set the required environment variables. It is loaded automatically by the project.

If you want some seed data you can run: npm run db:seed:import

Finally run npm run dev to run the service and listen to port 5000.

Caveat

Currently, there is no staging environment for Algolia so there is no search functionality for local development.

GraphORM

We have an internal solution to tackle problems we have encountered along the way. The library is in its early stages so we are continually writing the documentation to provide better developer experience. Have a look at the link below: https://github.com/dailydotdev/daily-api/wiki/GraphORM

Want to Help?

So you want to contribute to Daily API and make an impact, we are glad to hear it. 😍

Before you proceed we have a few guidelines for contribution that will make everything much easier. We would appreciate if you dedicate the time and read them carefully: https://github.com/dailydotdev/.github/blob/master/CONTRIBUTING.md

daily-api's Issues

Remove read articles from the main feed

As stated in dailydotdev/daily#142, we would like to filter out the read articles from the main feed of the extension.
The first step is to add a filter that will do exactly this in the generateFeed function (https://github.com/dailydotdev/daily-api/blob/master/src/models/post.js#L261). This function is the core that generates all the feeds we can browse. The good news is that a field which indicates whether the post has been read already exists and is called read (obviously :P), now we just need to filter by it.
In the second step, we have to add a parameter to the graphql query which states whether the user would like to filter out the read articles or not (https://github.com/dailydotdev/daily-api/blob/master/src/graphql/schema/post.js#L93).
Lastly, we need to add the user preference so we can sync it across the devices (https://github.com/dailydotdev/daily-api/blob/master/src/models/settings.js). Please make sure to create a knex migration file.
It is highly important to add tests to check everything, this is a core functionality of the API and we have to make sure it works and will keep working on every change.
I would love to provide more information if needed :)

Create Swagger file

Document the different routes with a swagger file to improve accessibility of the services

🐛 BUG: Redirector doesn't have access to posts of private sources

Redirector asks the API to get a post's link by its ID. Due to access control, the API returns 404 as it thinks the redirector is an anonymous user. This should be fixed to allow navigating to the post of a private source.

Create database snapshot for local development

To setup a local environment a pre-populated database is required. It is mandatory for contributing both for this project and daily-apps project.
A mechanism which exports the current database state to csv/json files on a weekly basis and publicly publish them is needed.

Only the following tables are required:

• posts
• publications
• tags
• tags_count

Add GraphQL query for getting feeds

Currently there are many endpoints for getting different feeds:

• Main feed which queries posts based on publications and tags and rank them by popularity (actually inside, it has two branches one for anonymous users and one for registered users)
• A single tag feed which obviously returns all the posts of a single tag ordered by time
• The same goes for a single publication feed

The current design is very limited for the applications as every change in one of the feeds must coordinated with the API. We would like to present a query which will provide the frontend a single yet flexible endpoint it need to create ad-hoc feeds and will lay the ground for a future search mechanism as well.

The query should support the following:

• Filters (everything is optional):
  • Before and after - used to filter posts based on the creation time
  • Exclude publications - posts which belong to these publications will be filtered out
  • Include publication - posts will belong only to these list of publications
  • Tags - posts must contain at least one tag in the list
• Ordering:
  • Popularity - rank posts by combining number of views and time since creation
  • Time - rank posts only by time

Query must also support pagination (probably can be achieved with before and after filters).

This issue is complex and can be split to smaller tasks as in the list below.
Would love to get the community thoughts on this one as well.

• Write a GraphQL schema
• Create a generic function to query the data with SQL
• Refactor existing endpoints to use the mentioned above function
• Choose a GraphQL server package (with Koa support)
• Add a GraphQL query

Why are there no issues in any backend and frontend repo?

Why are there no issues in any backend and frontend repo?

export personalized rss feeds from api

Hi all. I want to read daily.dev suggestions in my rss feed reader from mobile. Is it possible to export endpoint with rss support?

/rss?cookie=...
maybe
/rss?id=<idFromExtension>&mail=<mailFromExtension>

Send Pub/Sub message on every event of publication request

Send a Pub/Sub when a publication request changes due to creation, approval, decline or any other event. This will enable a cloud function to send an email notification with a relevant update

[BUG] "This email is already used" while updating Account details

"This email is already used" while updating Account details

I tried adding my hashnode handle to daily.dev profile, but whenever I clicked on save changes, it displayed "This email is already used" under the email input section.

The request (copied as fetch):

fetch("https://app.daily.dev/api/v1/users/me", {
  "headers": {
    "accept": "*/*",
    "accept-language": "en-IN,en-GB;q=0.9,en-US;q=0.8,en;q=0.7",
    "content-type": "application/json",
    "sec-ch-ua": "\"Chromium\";v=\"94\", \"Google Chrome\";v=\"94\", \";Not A Brand\";v=\"99\"",
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": "\"Linux\"",
    "sec-fetch-dest": "empty",
    "sec-fetch-mode": "cors",
    "sec-fetch-site": "same-origin"
  },
  "referrer": "https://app.daily.dev/nimishjn",
  "referrerPolicy": "strict-origin-when-cross-origin",
  "body": "{\"name\":\"NIMISH JAIN\",\"email\":\"[email protected]\",\"username\":\"nimishjn\",\"bio\":null,\"company\":null,\"title\":null,\"twitter\":\"nimishjain_\",\"github\":\"nimishjn\",\"hashnode\":\"nimishjn\",\"portfolio\":\"https://www.nimish-jain.com\",\"acceptedMarketing\":true}",
  "method": "PUT",
  "mode": "cors",
  "credentials": "include"
});

The response:

{
    "code": 1,
    "message": "email already exists",
    "field": "email",
    "reason": "email already exists"
}

Reuse the same source in case of same private sources

If user A and user B add the same private source to their feed, it should not add twice the same RSS and not create two different sources.

Migrate to CircleCI

Migrate CI/CD platform to CircleCI from Cloud Build

API documentation?

Hi, I just want to know whether API documentation is available?

Thanks.

Add seed data to docker

To complete dailydotdev/daily-apps#110, we must copy seed data to docker so it will be available to docker-compose based setup.

Add full text search filter to generate feed function

We would like to create a new filter for the generateFeed function (https://github.com/dailynowco/daily-api/blob/master/src/models/post.js) for searching posts with free text.
The name of the filter should be text and the value should be string.
Using the value of the text filter a full text search should be executed as part of the query.
One thing to note is when using this filter we must make sure rankBy is set to creation to keep query performance safe.

Follow up issue of dailydotdev/daily#76

Get bookmarks endpoint have degraded performance

Due to the recent changes in feed query system (#23), there seems to be a regression in the get bookmarks endpoint performance. Further analysis is required.

Add retry mechanism and logging to Superfeedr operations

When subscribing or unsubscribing from a feed, sometimes the request to superfeedr might fail.
A retry mechanism should be added and also proper logging with all information in case it still fails.

Docs: Improve developer experience when understanding the code base

I think it would be great if there was a section in which we would guide the developer on how to easily understand the code base.

I'm particularly referring to understanding what's going on through using the debugger while running the tests.
I personally saved a lot of time by following the above approach.

These are the steps that I followed in order to really understand the magic behind this API(I'm assuming you're using VS Code):

1. Open the debugger tab and add a new configuration
2. In your launch.json, add the following lines:

"configurations": [
        {
             "type": "node",
             "request": "launch",
             "name": "Mocha Tests",
             "program": "${workspaceRoot}/node_modules/mocha/bin/mocha",
             "args": [
               "--inspect-brk",
               "${workspaceFolder}/test/integration/routes/posts.js"
             ],
             "port": 9229,
             "runtimeVersion": "10.12.0",
             "envFile": "${workspaceFolder}/.env.test",
             "internalConsoleOptions": "neverOpen"
       }
    ]

Note: the .env.test file is where you put the environment variables so you don't have to issue them manually

3. Add these options in your .babelrc file:

"retainLines": true,
 "sourceMaps": "inline"

4. Open a test file, place a some breakpoints and start running the tests in debugging mode by pressing F5 or clicking on the Start Debugging button.