emotebot/emotecollector

Collects emotes from other servers for use by people who don't have Nitro

emotecollector's Issues

persist emote replies using a new table

The dict containing which reply messages correspond to which user messages should persist across restarts. We can use Redis to do this.

https://redis.io/topics/lru-cache

preserve newlines in messages

allows users to do shit like this

The code is 🗑

• duplicate code
• very long functions
• multiple database queries are done for the same emote sometimes
• it's just overall garbage

probably the on_raw_message* events is the only passable part of the code, but it still needs comments

to fix

• give the architecture a good hard think
• use custom converters
• maybe keep using custom contexts
• rewrite almost everything

requires #5

ec/react: Add message search

[9:19 PM] Vex: I'd check through client.state._messages first personally
[9:19 PM] Vex: except in very rare cases it should be as up to date as fresh-getting

[9:28 PM] Cute: Check cache -> get_message -> format content -> send

May want to also try channel.history but that's p expensive. It shouldn't be necessary unless someone wants to react to an old message.

ec/add: support no-embed links

Links surrounded in <> do not show an embed, which can be useful if the image embed is pretty big.
This should be easy to implement: if < and > are found in a message, and it's not an emote (which always have <>), just remove them.

on_guild_remove: DELETE FROM guilds when a backend guild is removed

we have a CASCADE set up on emotes so the deletion will propagate to the emotes table

Add a quote command which preserves text other than emojis

May require some sort of parser or something.
Usage:
ec/quote foo :notlikeblob: bar baz
→ foo :notlikeblob: bar baz

move some funcs from api cog to ec-website

Create a new cog in ec-website which extends the current API cog, and move the functions which are only needed to run the API, such as validate_token, to the ec-website subclass cog.

can't change locale back to en_US

Isolate the database code into its own class

underpins #3 #4 and #14
Reference:
https://gitlab.com/Pandentia/element-zero/blob/ca7d7f97e068e89334e66692922d9a8744e3e9be/element_zero/util/database.py

Fix jishaku

May require forking the cog and adding 3.5 support. Otherwise I'll have to figure out 3.6 on the pi.

only send typing if the emote takes too long

Model it after jishaku's ReplResponseReactor.
Prevents the typing notification persisting after the emote is created.

Use a custom class for backend emotes

It should:

• have a convert method to use as a custom converter from :name: or name to an instance
• have everything that get returns, but as attributes
• have a method that formats itself

command to steal from message

specify a Message (use emote_collector.utils.converter) and it yanks all the emotes out

perhaps if there's more than one, ask which ones wanna be added

Collect statistics

All that's needed is a count column on the emojis table, and then various aggregate stats can be collected either in ec/info (no args), ec/stats (would have to change the name of the "send bot list stats" command first) or ec/about (probably not).

The aggregate statistics would be:

• mean/median emotes per owner
• mean/median uses per emote
• top used emotes (ec/popular?)

use timing safe string comparison for api token validation

add mod role support

For #34

memory leak, possibly in wand resize code

Basic testing seems to indicate that emote_collector.utils.resize_until_small (see below) may be causing a memory leak.

https://github.com/EmoteCollector/bot/blob/18091ca4b3f3c11d6c640d306a0a9bd35b9c8589/emote_collector/utils/image.py#L17-L52

Command:
ec/add ttl <https://cdn.discordapp.com/attachments/381963689470984203/509133128418328576/terry_the_legend.gif>

Memory usage before: 39.2 MB
Memory usage after: 699.5 MB

objgraph.show_growth() seems to indicate that multiple BytesIO objects stick around after the image resize is complete.

Two things to notice here:

1. The context manager for wand.Image is not being used. According to the Wand docs, the context manager allows created image objects to be GC'd. Implying that not using the ctxman or explicitly calling close() is basically a guaranteed memory leak.
2. A new BytesIO object is created in each loop iteration. That's fine but I'd like to find a way to delete each image that was unsuccessfully resized when it is determined that the image data is no longer needed.

The while loop essentially boils down to this each iteration:

if some_blob is too_big:
    some_blob = frobnicate(some_blob)

Notice that the old some_blob is still in memory while frobnicate is running. After using the context manager, it may be worth assigning to separate variables and using del manually.

Another tool I was made aware of is the ImageMagick environment variables. These can be used to limit ImageMagick's memory usage before it writes to disk instead, and can be done by resolution or image size. Very cool. If I use these env vars, make sure to include them in a fix for #44.

ec/add: support webp images

Emote aliases

ec/alias alias-name target, or
ec/ln target alias-name, for those Unix users :)

Two ways to do this, schema wise:
Approach 1: Add a new table for aliases. Columns: name, id (foreign key on emotes.id).
Upon lookup of a name, either
A) first look in emotes, then in aliases, or
B) duplicate information in both tables, as RoboDanny does with tags, in order to make lookup faster

Approach 2:
Store primary emotes and aliases in the same table. Remove the unique constraint on emote IDs, and maybe use a SERIAL type as a primary key instead of ID. Add a boolean flag to indicate if the emote is an alias or the primary.

In both cases, upon deletion of an emote, delete all corresponding aliases. This would be easiest with Approach 1 (ON DELETE CASCADE).

Note: sending an emote with a different name than its name in Discord is actually not a problem:

.

Add description support

It'd be cool if people could describe the meaning or purpose of their emotes. It would work like this:

ec/description speedtest Used like "this sucks" or "well, fuck".
Originally from http://i0.kym-cdn.com/photos/images/original/001/285/389/9ab.jpg. 

Obviously the description would have to be one of those fancy "consume rest" parameters.

Then people could view the description on ec/info speedtest, when that's implemented (#2).

“ec/help --no-embed” displays an embed

Add a web interface/API of some sort

It doesn't need to report individual usages, just summaries by day. Requires #3.

The bot will fetch any arbitrary amount of data

Steps to reproduce

1. dd if=/dev/zero of=1-GiB.bin bs=(math 1024^3) count=1 iflag=fullblock
2. Start up ngrok, and copy the hostname it gives you
3. Put this in your Caddyfile

localhost:80 $ngrok_hostname:80
mime .bin image/gif

4. Now just run ec/add StressTest $ngrok_hostname/1-GiB.bin

database transactions sorely needed

extensions/db is riddled with race conditions

add `add-from-e0` command

Copies an emote from e0 into EC. I have an archived list of emotes from E0 saved as JSON and HTML (only the HTML page has author IDs).

per server mode

Opt in mode per server:

• Only show emotes from the database that were created by people on this server.
• Require a nullable guild_created column, for emotes that were created in this mode.
• Allow mods to edit emotes created in that server.

Maybe not too useful of a feature.

ec/rm -f should require a reason

Role white/black-listing

Allow server admins to configure a role which prevents or allows people to use the emote commands.

Add a command to upload emotes to the server

command name: ec/adopt?
would just create a guild emoji on the server it's in, and use the name and image from the database

ec/add: check name for validity before sending

Reduces API calls, and allows for prettier error messages.

ec/react: fail early if that message already has that reaction

end-human meat popsicle doesn't need the bot if that emote already exists on that message.
scold them if they try

Unicode emote names

On attempting to save an emote with invalid chars:

1. Replace the emote name, e.g. 你好Ben, with "unicode_emote" in Discord, and "你好Ben" in the name column (the "display name").
2. When the user says :你好Ben:, send back <:unicode_emote:1234>.
3. Have a boolean unicode column for emotes.
4. No char limit on unicode emojis.

Display the display name in ec/info et al.

Match the ec/nsfw interface to the ec/preserve interface and require a reason for moderators

Add a way to anonymously add emotes

Do it ptpb.pw-style:

• Give the user a deletion key, and then the user can run ec/delete-with-key <name> <key> to delete their emote
• Store author = NULL in the database

bundle the systemd unit file i use as an example

may help people set up their own instance

Add an "info" command

Shows the name, owner, usage, and creation date of a specific emote.

Add toggle/blacklist support

E0 has a "state" command which toggles the bot responding to :name: on that server.
It accepts a boolean argument, but it'd be easier if the bot just toggled the state.

This should be stored in the blacklists table, with a new column called "guild".
The logic would work like this:

1. For each user, if they don't have a row in the table, go ahead and reply.
2. If they do have at least one row in the table, and one of those rows has no guild entry, the user is globally plonked. Don't reply.
3. If they have at least one row in the table, and they all have a guild entry, check the current guild against the table, and if they match, the user has plonked themselves on this guild. Don't reply.

This should probably be a commands.check.

add a command to split an emote into a wall

The reply message would contain a link to each generated emote and a reaction button to add all of them to the database

Weird mention in the paginator's info tab

Steps to reproduce:

• ec/list <any_user>
• Click on the ℹ️ reaction and see what happens

add ec/rm -f

Restricted to mods only. Required for force removal.

get rid of argparse

It's causing too much pain.
Would fix #27

Can't test the bot without affecting prod

Unfortunately, the nature of the bot requires several backend servers that are owned by a specific person, and no other guilds that are owned by the bot owner. This means that in order to test the bot without affecting prod, I need a separate set of backend guilds, which requires another account.
I tried to create a new account but it got "verify locked" again, requiring yet another phone number to be put in. Hopefully [email protected] will help me here.

This isn't really a code issue, although https://github.com/bmintz/emoji-connoisseur/blob/4846a970d8f4d382f3a18b7a0e854d84fcb5156f/utils.py#L14 will have to be changed to pull from data/config.json afterwards.

rewrite backend_creator to stay within the law

the ToS police are out to get me, man! and they think i'm still running backend_creator.py lol.

To qualm their fears, let's rewrite this with a new technique:

• 1 bot and N user accounts are created. The users will each own 100 backend guilds.
• the +1 bot is the guild creator bot. backend_creator.py logs in to that bot account, and does the following in a loop:
  • create up to 10 guilds (that's the limit)
  • invite the backend user account to the guild(s)
  • when the user joins, transfer ownership, leave, and goto step 1

remove dependency on client cache

instead of enumerating backend guilds entirely from cache:

1. fetch from db. if rows exist, guilds = rows
2. else, try to fetch from cache (use old approach)

requires update to everything that uses the backend guilds.

make the bot case insensitive

A lot of phone keyboards capitalize the first word of every sentence.
Fortunately, we don't have to add custom prefixes. Newest discord.py rewrite saves the day!

ec/add: support "," delimiter for adding multiple emotes at the same time

this would obviate ec/steal-these

Support marking emotes as NSFW

DB column should store who marked it as NSFW, so that a user can toggle their own emotes, unless the NSFW status was set by an admin.

Per-server emote blacklisting

Would prevent people within that server from using that emote unless they had Manage Messages (Manage Emojis?) permissions. Would also send a report to a private logs channel.

add emote reports

should be logged either to Logs/#emotes or #report-emotes