ail-project/ail-feeder-discord

Discord feeder for AIL

ail-feeder-discord's Issues

Join new servers

Find a way to join servers without being banned. This can be joining 10-20 servers, then scanning, then joining again etc. Find a way which works all the time.

Make the program completely automated

Currently the joining process is manual, maybe this can be automated to be:

Do until server-invite-codes.txt is empty:

1. Join new servers
2. Remove the joined servers from the server-invite-codes.txt file
3. Scan the servers
4. Restart from point 1

Extend found invite links

Invite links are not only discord.gg but also discord.com.
This can be added in the code.

Find servers to join

There needs to be a way to find the servers that need to be crawled.

These servers can either be public communities or servers, which have a permanent invite link openly accessible somewhere. If we find other invite links somewhere, we can also use those.

This could maybe be automated and the file etc/server-invite-codes.txt could be filled automatically.

Tags:
discord.gg
discord.com
top.gg

Finish up README

The README should contain everything you need to know to use the project.

Wait time should be configurable in the config file

Wait time should be configurable in the config file

Replace printing with logging

To keep track of what the code is doing, the project currently uses print statements. These can be replaced with logs.

logging library for Python

Resolve TODOs

There are a few TODOs in the code, that need to be resolved.

"Recursive" crawling

If we find a new invite link in the messages, we can crawl the new server as well.
To avoid possible infinite loops, we need to implement caching, see issue #5.

Test functionality on big servers

Try the program on big servers with lots of messages, to find remaining bugs.

• Get messages
• #20
• Leave servers

Continuously monitor the feed of a channel

Sometimes we want to keep monitoring a chat and continue to listen for new messages after the initial scan.

This can be implemented using the listener for new message and then as soon as a new one arrives, feed it if necessary.

This method should be terminated at some point. Either through a keyboard interrupt or after a certain amount of time.

Connect to AIL

Once everything is parsed and ready, we need to send it to AIL.

Cleanup code

The code is a bit messy on some places, clean up and make it look good and more readable!

FInd a workaround to the 100 servers per user

A user can only be part of 100 servers at a time. To crawl all of them, the user should leave a few servers before joining new ones.

Add more filter options

Messages can be filtered on many options, which ones should be used in the build?

• Text filter
• Follow threads
• Maximum size to be downloaded
• How long to keep listening for new messages to come in before terminating

Token generator

For automated account creation, the following repository could be used: https://github.com/silvanohirtie/Discord-Token-Generator

Adapt README

Due to some changes, the README needs to be adapted.

• discum installer from github, not pip
• How to join servers
• How the program flow works
• etc

Treat ALL urls separately

Next to the filter option for text and so on, every URL should be extracted and analysed.

Sometimes a URL cannot be scanned with article.nlp(). In this case metadata should be extracted instead.

Asyncio Bug in `chats` options

Hello,

each time after command python3 bin/feeder.py chats I am getting following error:

Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7f880ff073a0>
transport: <_SelectorSocketTransport closing fd=7>
Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 924, in write
    n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/sslproto.py", line 690, in _process_write_backlog
    self._transport.write(chunk)
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 930, in write
    self._fatal_error(exc, 'Fatal write error on socket transport')
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 725, in _fatal_error
    self._force_close(exc)
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 737, in _force_close
    self._loop.call_soon(self._call_connection_lost, exc)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
    self._check_closed()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

I think it can be connected with following bug..

Create new account on launch

Create a new Discord account every time the application is run.

This would require the token or email+password to be passed as a parameter.

Implement caching

To not send message to AIL multiple times, we can implement caching and make sure, every message is only send once.

Finalise README

Adapt the README to the latest changes that were made to the program.

Reduce the wait time between requests

To avoid being rate limited, wait times were implemented. However, the current wait times are 10 seconds between each request, which is quite slow.

Try to find the best value to not be rate limited while keeping the speed up.

Try to prevent being rate limited

When fetching a lot of messages, it is quite common to be rate limited. Implement small wait times between requests to prevent this.