manage ROMs and DATs collections
ROMba is a command-line tool (and WEB shell) for the management, and collection of ROMs and DAT files within the Linux and Mac OS X environments. While its core functionality is similar to tools like ROMVault and CLRMamePRO, ROMba takes a unique approach by storing ROMs in a de-duplicated way, and allowing you to "build" any set you need on demand.
docker run --rm --publish 4204:4204 --volume /xxx_PATH_TO_ROMBA_DIR_xxx:/var/romba uwedeportivo/romba:v1xx
ROMba is under the Go license.
When processing DATs with the line
romba correctly recreated the files and folder structure BUT added ".zip" to the folder name. This should NOT happen.
At this moment in time romba seems to stall at the start of the archive command - when looking at the terminal output one can see that it is working on the initial scan - the progress command though only shows:
progress
running archive: (0 of 0 files) and (0B of 0B)
7z files can cause romba to hard crash - example log and file on dropbox
Both standards have same behaviour
Override the .ini verbosity for say an archive command with -verbosity=5
Enable romba to deal with .7z files like .zip and .gz
Instead romba creates in /tmp entries for both the db "romba_dedup028058326" and the individual in process torrentzip files it is building. On larger sets this is causing the root system to run out of space.
Please fix in such a way that that a different tmp in the ini can be selected for the "temp db" and "temp rom build files" so that for instance a faster SSD path can be chosen by the user for the "temp rom build files"
For large folder structures this takes up a lot of time and might improve with multiple workers
Example of a brand new archive command, taking place right after a scan that had 9 failed files
started archiving
finished archive roms
total number of files: 1
number of files processed: 1
number of files with errors: 9 <------------
total number of bytes: 723MB
number of bytes processed: 723MB
elapsed time: 43s
throughput: 17MB/s
Input for archive of .gz files is now treated as "unpack" exclusively - please add option to allow for external SHA1 and storage of .gz as option - "-include-gz=true" (this should be standard set to false)
Archive and merge show 6720 and 6721 files amd respectively - whereas input folder is 20,083 items, totalling 90.1 GB - output IS correctly creating 20,083 items, totalling 90.1 GB
total number of files: 6720
number of files processed: 6720
number of files with errors: 0
total number of bytes: 84GiB
number of bytes processed: 84GiB
elapsed time: 0s
throughput: 99GiB/s
zip and gz are SOMETIMES used in old systems as an non compression standard - romba will hard crash on those files - either add option to ignore these - or (better) trap these and prevent romba topple over
(1) fixdat
(2) diffdat
Build should not create a folder by default - only build a folder when there are indeed files being built into it,
Intent would be to have NO fixdat have a repeat crc+size/md5/sha1 entry, independent if 1 or more fixdats are being created.
This command got removed from the feature list and is thus no longer required
Currently progress command shows:
progress
running archive: (10239 of 523674 files) and (17GiB of 100GB)
Please add the current list of worker files e.g.
/mnt/roms02/depot-input/0005/62/27/24/8d/6227248ddc01698687edfd1fc9eff8cdd2744a57.gz
/mnt/roms02/depot-input/0005/62/27/27/01/622727010de01d312b76dd53209619e86a38493c.gz
/mnt/roms02/depot-input/0005/62/27/2d/98/62272d9834297fdbaa8bb12fd85302419f313268.gz
/mnt/roms02/depot-input/0005/62/27/3b/a7/62273ba7c380db17852e613d95f2da4af7c90d6e.gz
/mnt/roms02/depot-input/0005/62/27/45/22/622745227fabc3bb6a2f20675efca0430ee186e0.gz
/mnt/roms02/depot-input/0005/62/5f/d5/6b/625fd56bb104a85b7a06fe07787cba3bd7cc66dd.gz
/mnt/roms02/depot-input/0005/62/5f/eb/de/625febde2bf1ee0d9ab03efdd83e6670a07c5250.gz
/mnt/roms02/depot-input/0005/62/60/03/ed/626003edbfa34a4d7453677937e7a6f320fd06bc.gz
/mnt/roms02/depot-input/0005/62/60/05/c7/626005c78c5a0763acc8a15c81014d7097b4d1be.gz
For certain jobs (e.g. build it may be useful to increase the number of workers from the standard assigned by the .ini)
Allow for a build -numberworks=X command line option.
When a path or folder from the ini is not reachable romba hard crashes:
I0709 23:13:57.117710 07445 depot.go:74] establishing size of /mnt/roms01/depot
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x20 pc=0x4dc79f]
goroutine 16 [running]:
runtime.panic(0x84fd00, 0xd25cb3)
/usr/local/go/src/pkg/runtime/panic.c:279 +0xf5
and pages more of mumbojumbo
romba is listing much detail in ther terminal when verbosity is set to 1. This applies to the refresh-dats command only.
romba will be stuck on:
running archive: (0 of 0 files) and (0B of 0B)
The header of the gzip file that gets created as part of the archive command should include both the originating original gz/zip/7z name (if available) - as well as the filename of the individual ROM name. This helps with the research of files that do not (currently) match an existing DAT
In theory a single disk delivers about 100 IOPS so any value >1MB might be sufficient already. 1MB-16MB is a safe value however.
cgzip/writer.go and reader both use:
DEFAULT_COMPRESSED_BUFFER_SIZE = 32 * 1024
This would allow for multiple files to be processed in parallel and optimizing CPU use to 100%
This would allow for further investigation on user - in the ini destination path - recreate folder path to ease location of original content.
This log would allow a user to pick up where left at the last stage.
"when parsing DATs for rebuild, base the rebuild on the existence (or non-existence) of the following flag:
Normal operation (torrentzip the target) when
(1) NO flag
(2)
(3) clrmamepro (forcezipping yes)
HOWEVER when one of these flags is set:
(1)
(2) clrmamepro (forcezipping no)
A special build sequence is needed. Using the following example:
In this case, the BUILD command instead of creating a torrentzip file named "Old school game fun.zip" with the roms and rom names in it, it would create a FOLDER named "Old school game fun" and put into those simply the renamed files "lgc-doad.nfo", "lgc-doad.r00", "lgc-doad.rar", "lgc-doad.sfv" as separate files (NO torrentzip, nothing, just unpacked from the archived sha1.gz and renamed properly."
When building sets the "" sign in entries in BOTH GAME NAME and ROM NAME is a folder separator (so to be used as "/")
<rom name="Assets\global\twine.flo" size="4496" crc="5e7b1bed" sha1="2904b6311a7e331a0de91b793c65105c16d8795e"/>
Should lead to the following folder setup:
Assets
|--->global
|-->twine.flo
Instead - the build command now creates a folder called "Assets\global"
Assets\global
|-->twine.flo
romba calculates disk-space to be used as part of the assigned by assuming that all the roms called in the "archive" command actually make it into the depot, ignoring dupes that may actually never make it.
Additionally it will as a result of this also assume the wrong total disk space occupied by the depot
GoLang sets these by default somewhere, too confusing for end-users, please override with an ini entry option.
When processing sets with "archive" sometimes workers seem to get stuck - no activity seems to take place, "cancel" won't do anything and only "shutdown" stops romba.
A set that can reproduce the issue is available, reference "/!0000Feed to ROMba/Raine"
Seems to stall romba
With the completion of the new "Copy to BAD" feature in romba, the cancel archive after 100 failed zip/gz files has become obsolete
romba Does not build 0 byte files - and will report these files as missing in fixdats, where romba should be just creating these filenames
Expected behaviour
If a DAT contains an entry for a rom it is expected that romba builds this zero byte file. (UNLESS the already romba included exception of is in the same line.
Actual behaviour
romba will not create the file, and list it as missing during build of fixdat command
Steps to reproduce
Dropbox has a few sample DAT files - just create a fresh romba setup - use these DATs without any files in the depot. - Romba should be able to build a few partial files games now due to the 0-byte file rom entries in the DAT file
Example line:
<game name="Sleepwalker & Pinball Fantasies (Europe)">
<description>Sleepwalker & Pinball Fantasies (Europe)</description>
<rom name="Sleepwalker & Pinball Fantasies (Europe) (Track 01).iso" size="53168447" crc="c899917b"/>
<rom name="Sleepwalker & Pinball Fantasies (Europe) (Track 02).wav" size="0" crc="-"/>
<rom name="Sleepwalker & Pinball Fantasies (Europe) (Track 03).wav" size="0" crc="-"/>
<rom name="Sleepwalker & Pinball Fantasies (Europe) (Track 04).wav" size="0" crc="-"/>
<rom name="Sleepwalker & Pinball Fantasies (Europe) (Track 05).wav" size="0" crc="-"/>
<rom name="Sleepwalker & Pinball Fantasies (Europe) (Track 06).wav" size="0" crc="-"/>
<rom name="Sleepwalker & Pinball Fantasies (Europe) (Track 07).wav" size="0" crc="-"/>
<rom name="Sleepwalker & Pinball Fantasies (Europe) (Track 08).wav" size="0" crc="-"/>
<rom name="Sleepwalker & Pinball Fantasies (Europe) (Track 09).wav" size="0" crc="-"/>
<rom name="Sleepwalker & Pinball Fantasies (Europe) (Track 10).wav" size="0" crc="-"/>
<rom name="Sleepwalker & Pinball Fantasies (Europe) (Track 11).wav" size="0" crc="-"/>
<rom name="Sleepwalker & Pinball Fantasies (Europe).cue" size="1468" crc="7ae01178"/>
</game>
The command should pause at the earliest possible moment in time.
Use the sha1name.gz and header entries (crc+md5) to integrate a previous depot into another collection.
Usage e.g.:
archive -usegzinfo /mnt/test/olddepot
this would prevent recalculating crc, md5 and gz
This would give user the chance to quicker process really large archive jobs
When processing sets with "archive" sometimes workers seem to get stuck - no activity seems to take place.
A set that can reproduce the issue is available, reference "/!0000Feed to ROMba/Raine"
This eases the -resume= command - alternatively, enhance the resume command to be "latest" e.g -resume=latest will take the latest available resume file as a basis
As an attempt to see which method might improve performance
Certain zip files get marked as unreadable for no reason - as they can be unpacked both OSX and linux terminal environment without any issues
Sample zip has been provided to aid in bug fix
It appears now to be related to any zip file >4GB
http://www.zlib.net/zlib_faq.html#faq32 talks about setting different compiler parameters
Allow for an argument like "rombaserver test.ini" to be used.
This would allow for easy test setups with alternative folder structures
The following 2 sections in readme.md need swapping due to dependencies:
Install zlib, download latest and unpack in Downloads folder, then do in the unpacked directory:
./configure
make
sudo make install
Install Git and G++:
sudo apt-get install git
sudo apt-get install g++
When using resume, at any point it seems that X an Y, A and B, always represent the initial values. It is therefor unclear IF resume actually even works as designed
Format of dat DATE field is YYYY-MM-DD 24HH:MM:SS
Example entry:
The dir2dat tool would store this as well - but ONLY as OPTION (default ignore store)
Allow for purging all-but-latest logs
Example:
1943: The Battle of Midway (US, Rev C) 1987 Capcomin above example the entry row that has status="nodump" should be ignored the next row however SHOULD be processed as normal.
In case of a build command, a fixdat or other commands - the status="nodump" entry row should be completely ignored
Right now romba requires DATs to be part of the DB in order to be used. Allow for non DB DATs to be parsed for commands like build, diffdat and fixdat.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.