Picture ratings

[The-furf-of-July]

Picture ratings

[matjojo]

Hmm, this seems feasible. That is, the ranking system can be used to create a "prediction" of the outcome of a match between two "teams".

We use the tags as teams and then predict the outcome as a "better/worse" score. That could be done in some "tournament-style" to rank your images.

To rank all images, I believe we'll need to do something like log2 n comparisons. This is not prohibitively many, but it is something to watch out for.

I assume (have not done any investigation here) that the "win percentage" calculation in some way calculates the "power" of the teams. If we can steal that "power" number we can do the sorting a lot faster since we can do the comparisons on the number, instead of having to do the outcome prediction every time.

Anyways, this will not synthesize comparisons for the sake of rating tags, since the comparisons here are derived from the tags already it'll probably do funky stuff to your ratings.

[The-furf-of-July]

We could probably make inferences about what images can do good based on the tags. Sometimes there's art that is better for reasons that can't really be tagged, though.

There's also the potential issue of databases that aren't properly tagged. One of the sites I source from doesn't export tags at all, and I haven't had the time or willpower to clean it up. Not sure if this program could accommodate or help with that or if I'd need a different software to help me with tagging.

I think it requiring a lot of comparisons is fine because the ranking system doubles as a photo viewer. As long as progress is saved and can adjust to the database growing, it should be fine in the long run.

A very rough google search mentioned pairwise comparison and using win rates, but I have no clue how many comparisons that would take even with strategic algorithms involved.

[matjojo]

Not sure if this program could accommodate or help with that or if I'd need a different software to help me with tagging.

I have been working on a program that helps you train neural networks for just one tag, but it's very early days so it's all local for now. One of the main problems I found there is this tagging issue.

My thoughts are a program that has you make decisions on a number of tags, say 5-15, at once, for an image using hotkeys and some clear UI. I was thinking of having the list of tags on the side of the screen, and then using maybe the numpad or the home row on your keyboard, to select or de-select the tags. This would make tagging so much faster than having to type the tags by hand every time.

Anyway, those are just two ideas that I'm working on. I'm quite busy, however, so all of this is going in short spurts when I have time left.

As for your original idea, I think I only now realise what you mean, you want to use the comparisons that the user makes to rank images. This is an okay idea, and can be done, but will take a lot of comparisons. Think for example about how a football world cup needs all those matches just to find one winner, and really knows very little about the other teams.

[matjojo]

Pushed a new commit such that we're now at least tracking all comparisons in a file.

I'm not sure yet how we can use this to sort files in hydrus, so keeping this issue open for now.

[The-furf-of-July]

I have been working on a program that helps you train neural networks for just one tag, but it's very early days so it's all local for now. One of the main problems I found there is this tagging issue.

Should we make a separate thread for the tagging ideas?

My initial thought was that it could cycle randomly through the images with the least number of tags, and the user could enter a few of the most relevant ones. If you're training a neural network, that would help too.

Another thing that I think would help is being able to add tag relationships while viewing an image.

[matjojo]

Should we make a separate thread for the tagging ideas?

Yeah do that, or join the hydrus discord, I'm there as well. Other users will also have opinions on this I bet.

[The-furf-of-July]

Another thing that may help if the end goal is ranking images accurately, would be the ability to make hard vs soft decisions.

Soft decisionmaking is what I assume it already does, images can go against each other multiple times, and everything sort of averages out over time. No single decision will have a permanent irreversible effect on the rankings.

For an example of hard decisions, if a user presses Shift+A on a pair, it would connect the two such that the image on the right will never have a higher rating than the image on the left. If the image on the bottom would have surpassed the other, increase the score of both. If the image on the top of this link would have gone down in ranking, lower both images.

This could be useful for alternates or other cases where it can be determined with confidence that one image is always lesser than another.

[matjojo]

I pushed an update that allows you to sort your images based on your tag rankings.

A few responses to your questions:

Soft decisionmaking is what I assume it already does, images can go against each other multiple times, and everything sort of averages out over time. No single decision will have a permanent irreversible effect on the rankings.

There is no such thing as a "file ranking". Imagine you're playing an online team game with rankings, like COD, or Overwatch, stuff like that. Now imagine that each tag is a player, and that the files are teams made up of the tags that are on the file. Now each time that you make a decision the ranking system acts like those "players" played a game, and that the outcome was what you decided. Then the TrueSkill algorithm (by Microsoft) adjusts the "skill" of the "players" such that they come closer to their "real skill".

TagRank takes this "skill" number, and acts like it represents how much you like the tag. This works up to some measure, but is also deeply flawed.

For an example of hard decisions, if a user presses Shift+A on a pair, it would connect the two such that the image on the right will never have a higher rating than the image on the left. If the image on the bottom would have surpassed the other, increase the score of both. If the image on the top of this link would have gone down in ranking, lower both images.

This again is about the misconception about file rankings, there is no such things, only tag rankings.

As for making a "file ranking" by hand, I'm not really interested in that feature myself, so I'm not sure about spending time creating it right now. I have some ideas on how to make it, but I have to think about it for a while before I have a good pllan. At least now that we are storing the file comparisons your time use in making comparisons is not in vain.

[The-furf-of-July]

There is no such thing as a "file ranking".

I understand that there are not file rankings currently. Part of the original point of this issue thread was discussing the possibility and how it would be implemented.

It being low priority is definitely understandable. I would still like to throw ideas around, however, if that is okay.

TagRank takes this "skill" number, and acts like it represents how much you like the tag. This works up to some measure, but is also deeply flawed.

Like you said, the skill-based ranking has benefits and drawbacks, and may not be suited for file ranking. I did brief searching, and pairwise ranking was something I saw thrown around a lot as an option for subjective measurement. One of the main drawbacks being that it requires a large number of comparisons.

Since the UI for this fits both methods so well, it may be a possibility to have the same software run both ranking systems, providing skill based ranking in the short term and pairwise in the long term. Getting value for both from the same comparisons sounds appealing.

[The-furf-of-July]

I pushed an update that allows you to sort your images based on your tag rankings.

I tried running the new feature and it caused my hydrus client to lag out and crash.

[matjojo]

I tried running the new feature and it caused my hydrus client to lag out and crash.

That is a bit of a bummer haha, how many images do you have? And can you show the log of the crash?

The feature is expected to lag out hydrus since it is possible that it may ask for information on almost every image. But crashing is not the goal.

[The-furf-of-July]

I have 2.5k images. I'm not sure if it was a crash exactly, as it stopped responding and I forced it to close, so I'm not sure if there's a crash log or where it would be.
Maybe if the information being requested is spaced out more, or if there were a progress bar on the terminal, that could help with user patience.

[matjojo]

@The-furf-of-July

I added a function that chunks these large requests. There is some tuning involved in the size of the chunks. I've set the chunk size to 1000 for now. It would be ideal if you could test this before I release it on master.

You can install this different version by using the following page/branch: https://github.com/matjojo/tagrank/tree/try-chunking-requests

I'd really appreciate a response on the size of the chunks. Is the client locking up still? If so, how long? My client only has 124 images, so this is hard to test for me.

Also relevant is the kind of computer you are running this on. Do you have hydrus on an SSD? Do you have an especially old computer/laptop?

[The-furf-of-July]

Acer Predator Helios 300. Hydrus is currently on an SSD I think, but I do have a hard disk for backups.

There is still noticable lag, but it was usable for a while. I returned to it after a bit and it had frozen again. I'm not sure how long I should wait to see if it'll unfreeze.

[The-furf-of-July]

It has unfrozen but I'm not sure what it's supposed to have done? The main.py window closed itself.
Update, I see the new tags, not sure what I'm supposed to do with them or what search to do to get it sorted.

Are there API options for using a rating system? I think that might make a bit more sense.

[matjojo]

Ah, I had not considered the option that people would run the application by double-clicking it, instead of running from the commandline.

I'm not sure how long I should wait to see if it'll unfreeze.

TagRank now has a text that says if it is still busy, you can wait for that.

Update, I see the new tags, not sure what I'm supposed to do with them or what search to do to get it sorted.

The program gives some console output to inform you of how to use them. It's really quite easy.

I've updated this branch to ensure that the widow keeps itself open.

Can you please tell me how long the chunks take? Is it in the minutes range? Since that is way longer than for me, I have less than 1 second of runtime for my 124 images, so fetching all the data should not take that long even with 1000 images per chunk.

Note that sorting will indeed take long-ish, but when sorting hydrus should not be locked up, since I am not making requests in that time.

[The-furf-of-July]

Ah, I had not considered the option that people would run the application by double-clicking it, instead of running from the commandline.

My original solution was to make a Windows shortcut so I could add the runtime argument.

Can you please tell me how long the chunks take?

Note that sorting will indeed take long-ish, but when sorting hydrus should not be locked up, since I am not making requests in that time.

The chunks don't take very long at all. Seconds. The hydrus client lag appears to be occurring after "Existing sort tags deleted." So it's either the sort or the deletion, not the requests.
New theories, it may be a CPU bottleneck and my entire computer might be complaining. Though it is very weird that even if I don't interact with Hydrus, it still goes Not Responding during the sort.
Maybe deleting the tags all at once is the issue, and batching those could help?

Hope this helps!

[matjojo]

The hydrus client lag appears to be occurring after "Existing sort tags deleted."

At this point, the tags have been removed. Due to the way hydrus API works, it's easier to delete them one by one, so they are already chunked, just in small chunks haha.

It is possible that Hydrus needs to work for a while to process those deletions, but that is not something I can do anything about.

I assume that it is indeed your PC that is the bottleneck. As I had said before, the comparisons are quite heavy.

The way "Not Responding" works is quite technical, but in short it can happen any time that your PC is hard at work, even on tasks that do not seem to have a lot to do with the program that is "Not Responding".

It sounds like everything is working. Did the help text explain what to do with the tag ranking well enough?

[The-furf-of-July]

Yep! Thank you for your help!