tokitouq/manga-api

A Python based web scraping api built with fastapi that provides easy access to manga contents

manga-api's Issues

feat: new endpoint `/genre/`

New endpoint to get list of Manga with genre passed as parameter. eg: /genre/action, returns list.
Utilize BaseSearchScraper for this since HTML structure is same.

also add other queries like page offset limit and sort

feat: new `Completed` endpoint

A new endpoint: completed which returns list of Mangas which is completed airing.
scraping link: https://mangareader.to/completed

features:

• Offset and Limit queries
• Page query

refactor: move `__get_parser` to `helpers`

https://github.com/tokitou-san/MangaAPI/blob/e09ad4d36c7001a95c9ece318cb5d3c23bc6fa97/app/api/scrapers/base_manga.py#L13-L15

[`Feature`] : Revert to single provider

Our current workflow is like, now we've 2 providers for getting manga contents.
Due to this, we could only provide limited informations for each endpoints ( need to provide synced info ).
So I think its better if we can provide more information.
For this goal, we need to rely on single provider.

AniList seems promising and has server side rendering.
means we can use selectolax to get info ( faster ).

feat: add `pre-commit`

Feat: add response_model for each endpoints

Create a pydantic model and follow this:
https://fastapi.tiangolo.com/tutorial/response-model/

[`Bug`]: exclude Index route from OpenAPI

https://github.com/tokitouq/manga-api/blob/aa6570d8f9ea777103c8fbc45548016f1dd7d34f/server/api.py#L25-L27

Exclude this route from OpenAPI
use this: https://fastapi.tiangolo.com/advanced/path-operation-advanced-configuration/#exclude-from-openapi

Merge codes with coreproject

The scraper should be in shinobi

the rest of the logcis should be ported to django-rest-framework

feat: add `Search` endpoint

Add a new endpoint which search mangas and return a list.
Endpoint eg: /v1/search/?keyword=one+piece
Scraping url: https://mangareader.to/search?keyword=one+piece

refactor: add better `404` response

https://github.com/tokitou-san/MangaAPI/blob/4a0660358d2e40053921458c2588e0f0af82fdfd/app/api/scrapers/base_manga.py#L30

Currently it returns empty fields if manga is not found or any error occurs
Instead I'd like to return a better 404 response with message

[`Backend`]: refactor codebase for supporting more providers

Unfortunately mangareader seems down ( showing its for sale ).
Currently manga-api only relies on mangareader for data, but since it went down we gotta change provider.

Well I think its better if we can change whole codebase for supporting different providers, we need to scale ( currently this architecture isnt scalable ).
We will prolly change folder structure as well like
url can be like: /api/<provider>/<query>/

List of manga providers we can use: https://github.com/anshumanv/awesome-anime-sources#manga
Note: we might need to use selenium instead, well but its better if we can do this with selectolax itself.

feat: add endpoint `/type/`

Add new endpoint /type/ which returns list of Mangas of specific types.

feat: migrate to `Django` and `drf`

Currently this project uses FastAPI, but now I'm thinking to rewrite this in django and django-rest-framework.
since I use this kinda scraping logics for mangacore ( a project for streaming mangas ), it would be better to try this in django and drf.

I'd like hear your opinions.

[`Ideas`] : change API architecture

Hi there, hope you like this API.

What do you think about making this a single provider? I mean, scrape datas from just 1 site?
currently you can get same data from 2 different providers.

I think if we move to single provider, we can provider more info such as "latest, trending, sorting, etc..." like extra queries.
duplicate(#175) nvm.

Please share your thoughts bout this.
have a nice day.... ❤️

docs: add description to endpoints

Helpful links:

https://fastapi.tiangolo.com/tutorial/metadata/#create-metadata-for-tags
https://fastapi.tiangolo.com/tutorial/metadata/#use-your-tags
https://fastapi.tiangolo.com/tutorial/path-operation-configuration/#description-from-docstring
https://fastapi.tiangolo.com/tutorial/path-operation-configuration/#summary-and-description

refactor: use `self.parser` instead `utilities.get-*`

https://github.com/tokitou-san/MangaAPI/blob/e09ad4d36c7001a95c9ece318cb5d3c23bc6fa97/app/api/utils.py#L1-L15

Use self.parser.* instead functions from utilities.
eg: instead this code
https://github.com/tokitou-san/MangaAPI/blob/cccbcb6437861ebaf8aea7ef3d52bd00881cc3be/app/api/scrapers/popular.py#L16

Do something like ( might different for other cases )

slug = self.parser.css("div#manga-trending div.swiper-slide a.link-mask").attributes["href"] 

Lmk if you want more info or anything :)

fix: Typo in endpoint details

https://github.com/tokitou-san/MangaAPI/blob/4a0660358d2e40053921458c2588e0f0af82fdfd/app/api/endpoints.py#L139-L143

Here, used summary and description of genre instead type ( mistake )
fix: Change summary and description for type endpoint

feat: add `markdown`

Add markdown for index page, currently its rendering in markdown format, we need in html
use markdown and pass to template
then render with |safe filter