Improve speed rom scan

[finalburnneo]

Improve speed rom scan

[dinkc64]

good news friends!
by watching closely what goes on during a rom scan, I was able to optimize some things...
scanning roms on my system went from 11 minutes to 2.5 minutes after the optimizations! successs!

also, I didn't have to change the way things work internally - as it is rather complex. :)

[barbudreadmon]

The only way to get the same hash every time you zip a file is to use torrentzip, i don't think it would be a good idea :

• we would have to maintain a database of crc for every torrentzipped roms, seems like a pain in the ass
• what about 7z ?
• what about custom neogeo.zip with "custom" unibios ?

If we wanted to speed up scanning, i would go with the following instead :

• only care about the basename of the zip/7z file
• only check content (wrong crc and/or missing files) when launching the game, show a warning but allow the game to be launched anyway if file list match but not crcs, show an error if file list doesn't match.

[tmaul]

Whilst I am sure there is some scope for speeding it up, there are a lot of different ways of creating a zip file with different compression levels, etc so I am not sure that hashing the zip would be the best idea.

[Speedvicio]

I don't know what could be the best way, but anyway old mame release as mame32k or fba shuffle can perform the rom scan in more less time.
On old pc like eeepc series a entire romset scan during 30 or too much time.

[dinkc64]

I've been thinking about this for years, actually. This is what I came up with and on todo list:
Make an option to just check if the zip is there, then when the game loads, it can tell you if something is missing. This feature will, of course be configurable, (normal scan, fast scan..)

best regards,

• dink

[dinkc64]

Addendum to previous comment:
Fast Scan mode has a drawback which really can't be worked around:
lets say you have pacman.zip for arcade, it will also show that you have pacman set for MSX (for example), and coleco.
.. but you don't have to use fastscan mode if you don't want to.. That's why it will be an option :)

[tmaul]

One work around for that could be to have system specific directories? Will be a pain for anyone moving from FBA though, but if we are going to change things, now would be a good time to do it :)

[dinkc64]

tmaul, thought about that - but I don't like that idea at all. :/

[dinkc64]

In the test above, I have 5479 romsets total.

A friend tested with a new laptop and nearly complete fb-romset, times are:
4:00 old version
3:06 new version

Saving a minute doesn't seem like much, but lets hope the new code will have more effect on older pc's.

best regards,

• dink

[tmaul]

I have about 9000 roms, some duplicated, and an initial scan takes about 2 mins. Clicking "scan roms" a second time takes about 20 seconds, so maybe windows is caching some filesystem stuff? It's loads quicker though, was taking about 5-8mins before.

[dinkc64]

2 mins is better than 5-8, success! :D
The subsequent scans taking just 20 seconds is def. benefit of caching.
thanks for testing!