feature request cached library

[ltguillaume]

feature request cached library

[whysthatso]

ohboiohboiohboi will do tonight when the kids are in bed :D

[ltguillaume]

No problems there, if anyone would save a playlist called library, it would be saved as library.mfp.json, but the actual library is saved as library.json.

[ltguillaume]

Hi! Wow, that's a great compliment, love your enthusiasm! 😃

I understand your frustration, though the amount of time you need to wait is largely dependent on the system you're running. For instance, I'm running one instance with 25373 songs, and one with 18155 songs. It takes about 6 seconds and 3 seconds respectively to loads the instances the first time, then for quite a while the system caches the folder structure, so it's down to 1-2 seconds on a reload.

The client can also be of influence, but if we would cache the library locally, whatever time it takes to rebuild the tree will not be mitigated by caching.

We can look into it what causes the delay for you. What kind of setup do you have? If you load music.php in a browser, does that take up most of the time? Does it take an equal amount of time when reloading music.php?

[whysthatso]

alright, so some really basic preliminary benchmarking follows:
system: raspberry pi 4B, 2gb
storage: synology nas, mounted via nfs
Song count: 23695
first load: php-fpm on 30% cpu usage for 30sec
load right after: php-fpm on 45% cpu usage for 10sec

so i can see the caching working indeed, but then i ran this again after a reboot, and stripping the system down to the bare minimum, process state was mostly D, with short R bursts, only using about 10% of cpu, so i presumed there might be some issues with i/o.

loading time was a whooping 130sec, even on reload, so i am not sure what's going on there... but i cannot see any i/o, really (using iotop). strace does show that the process seems to check every file; same it does on reload.

it's all a bit weird but i will certainly accept that this is a system-related, rather than software-related optimization that this issue needs.

if you or anybody else has any experience with that kind of setup, chime in. otherwise, issue can be closed, i think.

[ltguillaume]

Wow, 130s is long indeed, don't know what you didn't load the second time and if it has anything to do with caching nfs locations. When you started this topic, I think you meant the 30s delay instead of the 130s delay, right?

There may be a way to improve caching the nfs share, perhaps by mounting it differently. I'll have a look.

Edit: This may help https://askubuntu.com/questions/4572/how-can-i-cache-nfs-shares-on-a-local-disk/4578#4578.

By the way, https://www.musicforprogramming.net is also pretty cool, yeah. Almost forgot about it.

[whysthatso]

thanks for the link, will check if i can apply it

yes, first it was 30sec/10sec split, then i rebooted the machine (first time after 28 days, it was also a fresh install back then), removed snapd, multipathd services from ubuntu and then it then was indeed 130 sec.

i'll update this issue when i find out more, it might be helpful for people later.

[ltguillaume]

i'll update this issue when i find out more, it might be helpful for people later.

Great! Keep me posted 😃

[ltguillaume]

@whysthatso Any news on this? 🙂

[whysthatso]

sadly not so far. i just haven't gotten around to improve on this

[ltguillaume]

I'll have a look at a refresh button + the option to save the library variable on the server.

[ltguillaume]

Have a look-see over here: https://github.com/ltGuillaume/MusicFolderPlayer/tree/reloadlibrary @whysthatso

[ltguillaume]

Just saw I made a stupid mistake with the URL to reload the library (via the button). Fixed now!

[whysthatso]

cool, just tried and it does work nicely for me 👍

might it possibly create a conflict if anybody tries to save a playlist named 'library'? i think for caching makes sense, as one should reduce the amount of writable locations to a minimum. so maybe a more obscure filename?

[whysthatso]

i guess #49 closes this

[whysthatso]

you might want to consider versioning / releasing some of your commits, so it's easier to reference stuff in the future, or rather group features and development. i.e. i prefer watching all these gh repos along releases, since i watch so many that i loose track easily when i'd watch all activities. so watching releases only is a good way of keeping up with an interesting project and see if i have to update something on my own deploy. just a thought.

[ltguillaume]

I completely understand and feel the same way about other projects, but that's mainly useful for those project that are very actively developed. I'm doing this on my own, and there are still some things I want to add/change, but it's not something I think needs versioning/"release management". The last commit is the best version and most of the time there are only a few commits a month.

A similar approach is taken by a developer of a much bigger project, Tiny Tiny RSS, and what I have done to keep up with its development is to just subscribe to its commits via RSS (in Tiny Tiny RSS itself). Then again, that's sort of what you do when watching a project on GitHub, too, but I don't like GitHub's interface much.

[whysthatso]

yep, makes sense.

also thanks for the tip of following gh repos via rss. i'm using ttrss as well, that would be ideal. do you know if there's an easy way to 'export' all the current watched repos as rss feeds? i guess it's probably possible via the gh api...