Importing directory hierarchies as tag groups

[amoffat]

Importing directory hierarchies as tag groups

[SeanPedersen]

Where can I find the mentioned SQLite database? I want to take a look at the schema. From my current understanding it should not be too challenging (complexity wise). So I might actually give it a shot if you welcome the abstraction change.

[amoffat]

Sure! So here is the schema. You can access the db here, or on your normal file system directly through (on linux) ~/.config/supertag/collections/<collection>/<collection>.db

Let me know if you run into blockers

[amoffat]

The tests are pretty comprehensive, so if you can make them pass with your proposed changes, then I think you're on the right track.

[amoffat]

Would the tag groups stop at 1 level deep? For example, suppose there exists Wallpapers/art/munch/1890. Would both munch and 1890 be tags inside of the tag group art+ ?. Related, how could supertag be told that art+ is the tag group you want to create, and not Wallpapers+ ?

[SeanPedersen]

Yes IMO it should only be 1 level deep. So Wallpapers/art/munch/1890 would result in only munch+ being part of art+ and 1890+ being part of munch+. So recursive 1 level deep group membership.

That would be solved by not using the current working directory as a tag group. So if you cd Wallpapers and then invoke tag ln import . only art/munch/1890 would be added as tag groups as described above.

[amoffat]

So Wallpapers/art/munch/1890 would result in only munch+ being part of art+ and 1890+ being part of munch+. So recursive 1 level deep group membership.

This isn't possible with supertag currently. A tag group can only contain tags, and a tag cannot also be a tag group. So you could make munch be a part of art+, but 1890 couldn't be a part of munch (only tag groups have the + suffix).

It's an interesting idea, but I think this issue is more in line with something I would support, specifically:

treat all path components as tags

Tag groups are sort of a necessary kludge that I haven't designed to be built on top of, so I'm not as interested in expanding them as I am expanding the more things you can do with tags themselves.

[SeanPedersen]

Ok that sounds like an abstraction issue. Why not unify tag groups and tags into a single entity? So a tag may contain files or other tags (to represent the necessary hierarchy). What do you think?

[amoffat]

It's a big undertaking. Not saying it is impossible, but it would probably necessitate using a graph database (instead of sqlite) because we've introduced complicated relationships that could be cyclical and challenging to traverse. Making something like this performant would also be a challenge. For example, to see all of the other tags that should be listed under a tag's directory, it is no longer enough to just find the files that share the tags' intersection. You would also need to traverse all children tags, their children, etc, breaking cycles along the way. It is a big overhaul.

You are welcome to try for it, but learning Rust and redesigning the architecture would be a big challenge :)

[SeanPedersen]

Thanks for the pointers!

So after taking a look, I would suggest following schema update:
keep files, tags, file_tag add tag_tag remove tag_group_tag, tag_groups

tag_tag would represent tags that contain tags with the columns parent_tag_id, children_tag_id.

When querying members for a single tag or an intersection of tags, we query first file_tag to retrieve associated files and than proceed to retrieve contained tags by querying tag_tag. By only looking ahead 1 level, we prevent any bad cycles causing havoc as far as I can tell.

This should provide my earlier described behavior. What do you think?

[SeanPedersen]

Alright, I kicked off the development by trying to remove all tag groups related code. See my fork: https://github.com/SeanPedersen/supertag/tree/tag_generalization

I believe to have succeeded with cutting out all tag groups related code but I think I might cut a little bit too much, since 6 tests fail right now. It would be cool if you could take a look and see where I messed up, as I am not too familiar with Rust and the code base yet.

After all tests pass again, I will continue by adding the tag_tag table to SQLite DB and than add the code that will allow to associate tags as children of existing tags (parents).

[amoffat]

Glad to hear you are getting started on it. Unfortunately I don't have the bandwidth to help work on the re-architecture right now. I am working on sh v2.0.0 for the next month or so. I can answer specific questions though when you get really stuck.

[SeanPedersen]

Alright, thanks for the heads up. Hope you will quickly make progress on sh!
Alright thanks for the offer, will get back to you when I get stuck.

[SeanPedersen]

Ok I changed my plans. I will start working on my own implementation of a tag based file manager using Python as I am more familiar with Python and it offering a generally better development velocity to try out new ideas. Maybe you will get around to implementing this feature yourself, I still think supertag would benefit greatly from it.