ajayk-01/ktu-notifier

An NLP based Telegram Bot that pushes KTU Announcements Notifications

Hi there 👋, I'm Ajay Krishna K V

• 🌱 I’m currently learning DevOps and Go.
• 💻 I work on flutter, react, python and nodejs projects.
• 🔗 I also tinker around Blockchain.

Projects

• Cherava - An open source, zero-code web scraping automation tool. Prize winner at Foss Hack 3.0
• KTU Notifier - An NLP based Telegram Bot that pushes KTU Announcements Notifications.
• SUSya - AI powered plant disease detection and assistance application. Prize winner at Impulse IEEE Hackathon.
• moniTOR - A dark web crawler and search engine which collects and stores links from various corners of the dark web for easier access and monitoring

Web3 Projects

• unHODL - NFT based GHO Token loan/stake platform, built at EthGlobal LFGHO.
• Project Mudrika - Web3 based disaster management platform proposal for the NDMA, SDMA and DDMA of India.
• SuperToken Deployer - A user friendly website which can be used to create custom super tokens and deploy them without having to write contracts.
• CampusVote - Blockchain based Voting Platform for Transparent Campus Elections.

Socials

• Linkedin
• Twitter

Ethereum - 0x6d60F08347deDEcbC6661C252c6Ad73b65808a49

ktu-notifier's Issues

Bot fails to Update

Describe the bug
While trying to run the bot, the scrapper function returns an empty list and fails to send notifications.
While trying to run scrape() function in scrapper.py, It returns the following error:
list index out of range

To Reproduce
Steps to reproduce the behavior:

1. Run the function scrape() from scrapper.py

Expected behavior
scrapper.py returns non-empty array of announcements

Screenshots

Additional context
Error is supposedly produced while trying to parse an announcement which is in a different format as shown below

Python Version Support

The packages in requirements.py are mostly deprecated and not available in latest python versions.
The packages for numpy1.19.1 and tensorflow2.0.0 needs Python Version 3.7 or lower
Yet the package pycrypto is deprecated and unsupported in python versions 3.3 and above
Therefore, package pycrypto must be changed to pycryptodome.
(Use pycryptodome instead! It is a project still maintained (june 2017) and includes the functions of pycrypto for python 3.xx (It worked for me on python 3.7) Refer this link for more details : https://www.pycryptodome.org/en/latest/index.html )

Or else please specify the exact version number on which the project is built/currently is working.

High memory useage while running scheduled check for new notifcations

Describe the bug:

When the APScheduler runs the function scheduledjob (line 70, bot.py), the RAM useage spikes up over time, reaching an unreasonable high value (upto 1GB useage in less than a day's use! ), thereby making the bot unhosteable on any of the free plans of common hosting providers.

To Reproduce:

Steps to reproduce the behavior:

1. Create your own bot token and set up the bot as mentioned in the README instructions.
2. Run python bot.py
3. Open up your local memory useage monitor application (task manager) and monitor the memory useage of the bot.
4. Wait for a few hours or just increase the frequency of the scheduler from 10 mins in line 181, bot.py to see the memory spike sooner.
5. Try removing a latest notification from your notifications cache (database), to trigger the bot to send a notification to the users, the memory spike is more profound in the case when new notification triggers.

Expected behavior:

The memory useage spikes up with each run of schedule job slowly, and the spike is much more profound when a new notification comes in the KTU website and the bot starts to send users the notification.

Screenshots:

Gradual increase of memory useage:

Eventually crossing free tier quota on heroku:

Additional context:

Currently the bot is being restarted by a systemd timer every hour in a custom VPS, but this hacky solution can't be used when hosting on hosting providers like heroku.

Scrapper no longer needed

Hey there. KTU recently completely shifted their website to React and started making use of APIs for almost everything in their page. So this project might require a rewrite at some parts. Good thing is that we don't need to go through the pain of scraping anymore.

I have hacked together some code and tried to make a replica of this in typescript (see here) although I still haven't tested it out completely yet.

Provide an email service similar to a newsletter.

Add email service that would allow forwarding of notifications to subscribed emails alongside a client application that'd allow users to subscribe to the notifications.

🍻 Do assign the issue to me if it's something you guys are planning at.