ilyaglow/dor

🌦️ Domain Ranker

dor's Issues

Add PageRank top 10 million domains

About: https://www.domcop.com/top-10-million-domains

Link to zipped CSV: https://www.domcop.com/files/top/top10milliondomains.csv.zip

Extend dor.ExtendedRank with SourceData field

It would be useful to store raw data from the provider. Could be a marshaled bytes.

Insert data gracefully

Insert data only if there is no domain ranks today for a particular source. It is useful for testing and future trends functions.

Refactor all constructors to variadic functional options

Native cloud support

Consider using Amazon DynamoDB / Google Cloud Storage as a backend computed by Amazon Lambda / Google Functions triggered once a day with CloudWatch Events / AppEngine with cron

Add tranco list

About: https://tranco-list.eu/

https://tranco-list.eu/top-1m.csv.zip

Add quantcast list

https://www.quantcast.com/top-sites/

https://ak.quantcast.com/quantcast-top-sites.zip

Clickhouse storage

Make a separated table with the data for each day / week. Data retention can be implemented via simple drop table by cron.

Broaden project use cases

Currently, a map-only strict implementation for the data storage limits the number of possible usages.

Storage should be an interface.

Consider moving to fasthttp

I do not use all power of Iris, so probably it's too heavy dependency for Dor

Remove yandex radar data provider

Yandex decomissioned the top10k in 2020:
https://www.rbc.ru/technology_and_media/22/06/2020/5ef0c17c9a7947b822cb0302

Remove Statvoo list

Hello,
The Statvoo ZIP file is an outdated copy of the Alexa list from November 23, 2016 - this can be checked through the timestamp of the unzipped CSV file. It has never been updated since, so its ranks are not representative of current popularity and stay the same every day.
I think it can therefore safely be ignored/removed from the service.

Handle configuration for a storage

Consider using viper for managing storage options like data retention, TTL etc

clickhouse_storage: Create table with TTL

https://clickhouse.yandex/docs/en/operations/table_engines/mergetree/#table_engine-mergetree-ttl

Add radar.yandex.ru top 10k

Top 10k popular online projects in Russia

https://radar.yandex.ru/export?options=%7B%22title%22%3A%22%D0%AF%D0%BD%D0%B4%D0%B5%D0%BA%D1%81.%D0%A0%D0%B0%D0%B4%D0%B0%D1%80%2B-%2B%D0%A2%D0%BE%D0%BF%2B%D0%98%D0%BD%D1%82%D0%B5%D1%80%D0%BD%D0%B5%D1%82-%D0%BF%D1%80%D0%BE%D0%B5%D0%BA%D1%82%D0%BE%D0%B2%22%2C%22exportFormat%22%3A%22csv%22%2C%22limit%22%3A0%2C%22offset%22%3A1000000%7D&mode=top-sites

Docker support