Comments (4)
This issue also includes making the sql
object thread-safe. So some sort of locking should be implemented to ensure that the sql
object is not mutated by multiple users at once.
This will likely involve using asyncio.lock
and/or asyncio.event
from statbot.
I have created the crawler
branch. I will limit my implementation until the design is more solidified, but there is some code that can be written regardless.
My proposal for the crawler is that it runs as an asyncio
task that plays round-robin for all the channels in all the guilds it must process. It will ingest a large block of messages before going to the next.
Within each channel, it will look for the earliest possible set of messages it can fetch in bulk, and then process those. It will do this by storing a range of snowflakes for each channel that records what parts of the history have been inserted already. It then updates the history record.
from statbot.
The first thing to implement is the Range
class and similar, to represent abstract integer ranges for tracking which snowflakes have already been covered. See the range
branch for info on this.
from statbot.
The range class has been completed and merged. Now the crawler proper (found on the crawler
branch) needs to be completed.
from statbot.
Related Issues (20)
- Read deployment data from environment HOT 5
- Crawling sets first_message_id incorrectly
- Store information about pinned messages
- Track guild membership
- Audit log entries not appearing
- Add animated emoji support
- Fix "is_member" column
- Update SCHEMA.md to add information about int_user_id
- Audit log live updates
- Split sql.py and move model conversion functions HOT 1
- Add logging for status changes/activities
- Change config file to TOML
- Add optional tracking for user avatar changes
- Allow for optional data collection
- Add message_history table
- Crawler doesn't crawl unless Read Message History perms are explicitly granted
- Add "bans" table for tracking users who are banned
- error: unrecognized arguments:
- SQL Lookup Tables will not initialize HOT 1
- Relax foreign key constraints on Audit logs and reactions
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from statbot.