theypsilon/update_all_mister

All-in-one script for updating your MiSTer

Update All

All-in-one script for keeping up-to-date with the MiSTer ecosystem.

This script runs the MiSTer Downloader under the hood. It expands it by selecting additional downloader databases.

Update All has a Settings Screen that allows you to configure which databases and tools you want to use. In said screen the menu includes:

1. Main Distribution. Downloads most essential files, including cores and firmware. You are able to select between two versions:
   • The official MiSTer Distribution in the MiSTer-devel organization. [Default option]
   • The DB9 Fork MiSTer Distribution with Extended Native Controller Compatibility for Genesis and NeoGeo/Supergun controllers. Requires SNAC8 compatible adapter.
2. JTCORES for MiSTer. Downloads many cores made by Jotego Team. [Default option]
   • You may additionally enable patreon-only cores.
3. Coin-Op Collection. Downloads cores developed by the Coin-Op Collection organization. [Default option]
4. Other Cores. A selection of curated databases that provide unofficial cores, including:
   • Arcade Offset. Downloads patched arcade games curated by Toya. [Disabled by default]
   • LLAPI Forks Folder. Downloads LLAPI cores which are compatible with BlisSTer and LLAMA. [Disabled by default]
   • And more...
5. Other Tools & Scripts. Additional tools & scripts.
   • Arcade Organizer. Creates a folder structure under _Arcade/_Organized for easy navigation to all the MRA files. [Default option]
   • Names TXT. Downloads a complete names.txt file curated by Threepwood that enables better core names in the menus. [Disabled by default]
   • And more...
6. Extra Content. A selection of curated databases that provide extra content.
   • BIOS Database. Downloads bios from https://archive.org/ for your installed systems. [Disabled by default]
   • Arcade ROMs Database. Downloads arcade roms from https://archive.org/ that are compatible with MRAs. [Disabled by default]
   • And more...
7. Analogue Pocket. Tools to connect your MiSTer with your Analogue Pocket.
8. Patron Menu. A menu with exclusive contents for members of my Patreon community.

Setup

Download this ZIP file and extract update_all.sh to your /Scripts folder on your primary SD card.

Then turn on your MiSTer, go to the Scripts menu and run this script from there.

It will take around 15 minutes the first time you run it, but later runs should take much less time.

Accessing the Settings Screen

The Settings screen will show up if you press UP in your controller or keyboard during the countdown showing up right after starting update_all.sh in your MiSTer. Please, don't forget to select SAVE to keep all the changes you have done before leaving.

PC Launcher (for Windows, Mac, and Linux)

Check MiSTer Downloader's PC Launcher to download all MiSTer files on your PC. Useful if you can't connect your MiSTer to the internet.

To install the same files that you get with Update All, use the same downloader.ini file that you have in your MiSTer at the root of the SD card. Keep in mind that Update All writes into that file every time you change something in the Settings Screen.

Funding

Consider funding Alexey Melnikov "Sorgelig" for his invaluable work on the MiSTer project.

Check also other core developers that you might want to support:

• Sergey Dvodnenko "srg320"
• José Tejada "jotego"
• Robert Peip "FPGAzumSpass"
• Josh Bassett "nullobject"
• MiSTer-X
• furrtek
• Ace
• Blackwine
• atrac17
• Darren

And finally, if you would like to support my work, you may also subscribe to my patreon here: https://www.patreon.com/bePatron?u=37499475

Supporters+ shout-out!

Blum Chillins, Daniel Tarsky, Koala Koa, Tony Escobar, turbochop3300 and Wayne Booker

Thank you so much for supporting this project on Patreon! You'll show up here if you become a Supporter+!

Special thanks to Locutus73 for all his great work on the older Updater script. That Updater script is no longer used by Update All but it was an amazing source of inspiration.

License

Copyright © 2020-2024, José Manuel Barroso Galindo. Released under the GPL v3 License.

Warning

MiSTer Scripts are run with root access in a pretty powerful device that has internet access. Be careful and do proper research before running any script on your device.

update_all_mister's Issues

In standard configuration Popeye.mra from jt overwrites Popeye.mra from Mister-devel

update with corrupted or damaged files.

thanks for this tool. I don't know if there is already a verification routine, but my first use brought defective files, is there any that can be done?

Incorrect syntax on line 99

Line 99 of the update_all.sh has a syntax error that occurs for some folks when running this for the first time due to the condition check on the "if" which does not have the double brackets.

Current:
if ! ${SCRIPT_PATH} ; then

Should be:
if [[ ! ${SCRIPT_PATH} ]] ; then

line 3 BASE_PATH: unbound variable error

I believe it has something to do with the 3rd line of the update_all.ini file that's generated

ADDITIONAL_REPOSITORIES+=( "https://github.com/MiSTer-devel/Fonts_MiSTer|pf|$BASE_PATH/font" )

It looks like this got generated because I didn't have a fonts directory beforehand, and hadn't downloaded the fonts. Unsure if it's a pre-requisite before starting the process, as I didn't see it there.

After I did so, I still got the unbound variable error. I deleted the ini and it generated again properly. This time, the file didn't have that line.

Suggestion would be to have a better error message for that, and also to make a pre-requisite in your readme to need the fonts to be installed

download folder

i have sd with system files (rbf, mra's and so on) and usb ssd drive containing only games folder, the script always download the mame rom on sd card creating another games folder

update_all_linux folder creation issue (not yet reproduced)

From MiSTer discord:

Kasreyn: I'm wondering if this is a bug. When I run `bash  update_all_linux.sh' it creates a new directory in games for each core. Except for AtariST, GAMEBOY, GBA, AO486 on those cores it creates a directory with UART setting text is appended to the name e.g. "Creating ../games/AtariSTUART19200:9600:4800:2400:1200,MIDI directory"

Feature: Dialog for asking to clean duplicated mame folders if one is more full than the other

Question about the PC Updater Scripts

Is it possible to have everything Download to a specfic location or BASE_PATH? It looks like it defaults to ../ but I was wondering if it's possible to use the INI or some other way to Specifiy an Alternate Directory. I tried a few things with no luck so far.

Can't display the SETTINGS screen because fb terminal is off.

Hello, I try to run Update_all_Mister directly from ssh like this :

$ ssh [email protected] "cd /media/fat/Scripts/ && ./update_all.sh"
Password: 
Downloading
https://github.com/theypsilon/Update_All_MiSTer

Executing 'Update All' script
The All-in-One Updater for MiSTer
Version 1.3

Reading INI file './update_all.ini':
OK.

Sequence:
- Main Updater: MiSTer-devel
- Jotego Updater
- Unofficial Updater
- BIOS Getter
- MAME Getter
- HBMAME Getter
- Arcade Organizer

SORRY!
Can't display the SETTINGS screen because fb terminal is off.
Maybe you have fb_terminal=0 on MiSTer.ini?
NOTE: It could still work if you also add the following lines in MiSTer.ini:
    [Menu]
    vga_scaler=1

Start time: Sun Jan 31 16:25:26 UTC 2021

So : the script runs normally after skipping the settings menu : SORRY! Can't display the SETTINGS screen because fb terminal is off.
I guess I have to modify a variable before running ssh ? Or run ssh with a special parameter ?
Or maybe a setting from my terminal emulator ? (I use Gnome Terminal 3.38 with bash)

Request: have script create TGFX16-CD folder

Currently the script ran on a fresh image does not make a TGFX16-CD folder, and thus the BIOS-getter gives an error. Can we get an addition to the script that makes this folder if not present?

I will also suggest to the Mr. Fusion installer that they add this folder, since it's pretty standardly used.

Feature: Autoreboot

High priority

Update_all script overwrites my setting in update.ini

I have a custon config in update.ini. When i choose to use the Mister update in update_all, it overwrites my custom config with an empty file. If the i add my custom config again and start the updater, it still makes a full resinc, which i don't want, i wann only the cores that are defined in my custom config.

Add support for MiSTer Super Attract Mode (SAM) script

from here:
https://github.com/mrchrisster/MiSTer_SAM

Thank you for your work. Cheers.

MAME-GETTER ignores path

When configuring a custom path for the MAME roms (ex /media/usb0/mame) it ignores the custom path no matter if i place it in the update_all.ini or the udpate_mame-getter.ini and places them in the default path of /media/fat/games/mame.

When I run the mame-getter script on its own - it downloads to the correct location

Feature: LLAPI Updater

Just static after update_all ran today

I ran the update_all and upon reboot it was just static. appeared as though the MiSTer main framework, or menu core, or something on the linux side must have been messed up. I got another SD card, ran the newest release of the sd card installer from sorg, and overwrote all the files on the one that wasn't working from the new ones. It then booted. Not sure what happened, but could it be that the linux files extracting to do their update screwed something up?

Invalid arguments:

Noticed 4 entries the log file for invalid arguments, and this is due to using characters that are not allowed. Here they are in sequential order:

mkdir: cannot create directory ‘/media/fat/games/AO486UART115200:4000000(Turbo 115200),MIDI’: Invalid argument

mkdir: cannot create directory ‘/media/fat/games/GAMEBOYSS3E000000:100000’: Invalid argument

mkdir: cannot create directory ‘/media/fat/games/GBASS3E000000:100000’: Invalid argument

error: cannot create /media/fat/cheats/TGFX16/Ranma ? (CD)(Jpn) [].zip
Invalid argument

Bug: Somebody show me that dialog didn't work on his MiSTer setup

Investigate. Reporter: J. C. in Spanish Telegram. Look for "O por lo menos hace cosas diferentes" there.

Feature: names.txt

Keep more than one global log

I think it could be an INI option to opt-in.

This is a suggestion from @wiggy808 #22

Feature: ZIP download link

Instead of raw link to script.

mame updater seems to not update properly

I've seen that on jotego's updates, new roms are not updated properly and it always requires to get into the console and delete them manually.

It might be caused by wrong dates?

Maybe it's enough to provide a 'force mame full update' option somewhere.

Feature: Log file with script output

High priority

Feature: Additional third party scripts

They should be from a trusted source.

error line 6

just tried to run the script, i get error on line 6 !

dave

Avoid AUTO_UPDATE_LAUNCHER=true deleting launcher when scripts are in CIFS

By now, the workaround is adding AUTO_UPDATE_LAUNCHER="false" to update_all.ini

PC Updater - Linux Issue

I may be doing something wrong on my end but no matter what INI Settings I use it seems to ignore them and use the defaults. For example if I add UNOFFICIAL_UPDATER="true" to the INI it doesn't work and seems to use the default "false" value. If I use the same INI settings and use the Windows Batch script within Windows it will work as expected.

Suggestion: SCUMMVM support

I'd like to suggest the Update_All script support installing and updating SCUMMVM for Mister.

Feature: Remove broken symlinks from Arcade Organizer orgdir.

Feature: Customize BASE_PATH

High priority

Request: Add RetroSpy installer

Wondering if you could add the option to add the installer script for RetroSpy as an option to update? It is an input display reader that currently supports USB reading - hopefully SNAC soon. I created the script, all it does is download the file to the SD card - no logic in it at the moment, but thought it would be a cool optional add on.

https://github.com/retrospy/RetroSpy/tree/master/MiSTer

Sega System 1 alternatives are missing

Hi,

update_all includes the "alternatives" for the existing arcade cores. But the alternatives for SEGA System 1 are missing. They are available here: https://github.com/eubrunosilva/MiSTerMRA/blob/master/MRA/MRA_system1_0218_CRC_20200204.zip

Feature: Try to identify and remove broken symlinks from the _Organized folder

Not sure if it's better to do it here or in https://github.com/MAME-GETTER/_arcade-organizer

Apparently they break when copying the _Organized folder under Windows.

Flush update_all.log to folder before rebooting

Wallpapers

Are you interested in adding a script that downloads wallpapers?

This one is an example:
https://github.com/RetroDriven/MiSTerWallpapers

Mind RetroDriven has stated he will discontinue his scripts as of now.

Feature: INI setup menu on when button pressed at start

/media/fat/Scripts/update_all.sh: line 103: ho: command not found

I've just run the update_all.sh script (v1.3) and when it finished it displayed the following error:

/media/fat/Scripts/update_all.sh: line 103: ho: command not found

I've tried to upload the log to pastebin but it's too large (527 KB). However, the error doesn't appear in the log so I think it's something trying to run at the very end of the updater?

Feature: Clean Start

It deletes everything that should be installed after, and then runs he script.

Move ZIPs to GitHub Releases

Issue with downloading the new arcade MRA's

EDIT
PLEASE IGNORE IT! I have just noticed that this issue has been already reported. Thanks!

Hi there,

Perhaps you are already aware of this issue. Anyway, I thought it was a good idea to mention it.

There was an issue with downloading the new arcade MRA's. However, this has already been fixed by Locutus73 with the updater v4.0.12. Please see link below:

Thanks a lot!

Kind regards,

Alberto

Legal warning when scripts that download from archive.org are used for first time

There are countries where is illegal downloading roms from archive.org or from any other server hosting roms. In practice, I don't think is a real problem, since tracking downloads from archive.org is never gonna be enforced probably. But it might be a good idea showing a notice to first-time users displaying that this is a thing, and giving the option to disable those downloads if they prefer to be on the safer side.

Feature Request: Please have the update all download the GETTER Scripts & _arcade-organizer

Please have the update_all.sh script download the following scripts to the "Scripts" Directory:

https://github.com/MAME-GETTER/MiSTer_MAME_SCRIPTS/raw/master/update_mame-getter.sh

https://github.com/MAME-GETTER/MiSTer_MAME_SCRIPTS/raw/master/update_hbmame-getter.sh

https://github.com/MAME-GETTER/_arcade-organizer/raw/master/update_arcade-organizer.sh

Thanks!

Feature: PC Updater

Jotego updater not downloading Popeye mra or core

I've just run a full update from scratch, and the Jotego portion of the updater doesn't seem to be grabbing the Popeye core or mra.

I have CORE_CATEGORY_PATHS["arcade-cores"]="/media/fat/_Arcade/_Jotego" in my update_jtcores.ini if that makes any difference.

Curl: (35) Unknown SSL protocol error in connection to archive.org:443

Hi,
I have got a new Mister FPGA set up. Every time a try to use the mame script I receive this error message:

Curl: (35) Unknown SSL protocol error in connection to archive.org:443

It works perfectly fine with the other Mister FPGA setup. The only difference is the way I connect to internet. In fact with my second setup I use the hotspot function on my phone as the WIFI connection is very poor in the room where the Mister is.

I have also tried to run the security script but to no avail.

Many thanks for looking at my issue!

Feature: Testing suite and automatic build to verify changes

It's a feature since it provides trust to the project.

I have a lot of files for that already, just need to clean them and create an automated infrastructure around them.

MRA files not downloading

I created a fresh sd card and ran update_all and everything downloads except the MRA files. The alternative mra files download fine though. I checked the logs and it shows the following for every arcade game:

Checking Arcade-[ArcadeNameHere]_MISTer
Nothing to update
Checking MRA files

corrupted rbf not replaced

if during update something goes wrong and a rbf is not downloaded fully, it remain truncated on sd and a new run of the script do not fix the situation. i have to manually remove the partial downloaded file

Various errors with updater-pc

From BIOS-GETTER (not sure what these are about):

mkdir: created directory '../BIOS/TurboGrafx16'
Downloading https://archive.org/download/mi-ster-console-bios-pack/MiSTer_Console_BIOS_PACK.zip/TurboGrafx16.zip
Archive:  ../BIOS/TurboGrafx16.zip
  inflating: ../BIOS/TurboGrafx16/[BIOS] CD-ROM System (Japan) (v1.0)_autoboot.pce   (warning) cannot set modif./access times
          Invalid argument
  inflating: ../BIOS/TurboGrafx16/[BIOS] CD-ROM System (Japan) (v2.0)_autoboot.pce   (warning) cannot set modif./access times
          Invalid argument
  inflating: ../BIOS/TurboGrafx16/[BIOS] CD-ROM System (Japan) (v2.1)_autoboot.pce   (warning) cannot set modif./access times
          Invalid argument
  inflating: ../BIOS/TurboGrafx16/[BIOS] Super CD-ROM System (Japan) (v3.0)_autoboot.pce   (warning) cannot set modif./access times
          Invalid argument
  inflating: ../BIOS/TurboGrafx16/[BIOS] TurboGrafx CD Super System Card (USA) (v3.0)_autoboot.pce   (warning) cannot set modif./access times
          Invalid argument
  inflating: ../BIOS/TurboGrafx16/[BIOS] TurboGrafx CD System Card (USA) (v2.0)_autoboot.pce   (warning) cannot set modif./access times
          Invalid argument

From Jotego Updater (just error messages for 404, but the MRA's and roms are fine):

Checking 1942
curl: (22) The requested URL returned error: 404 Not Found
Nothing to update
Checking MRA files

Checking 1943
curl: (22) The requested URL returned error: 404 Not Found
Nothing to update
Checking MRA files

Checking biocom
curl: (22) The requested URL returned error: 404 Not Found
Nothing to update
Checking MRA files

Checking btiger
curl: (22) The requested URL returned error: 404 Not Found
Nothing to update
Checking MRA files

Checking bubl
curl: (22) The requested URL returned error: 404 Not Found
Nothing to update
Checking MRA files

Checking commando
curl: (22) The requested URL returned error: 404 Not Found
Nothing to update
Checking MRA files

Checking cps1
curl: (22) The requested URL returned error: 404 Not Found
Nothing to update
Checking MRA files

Checking dd
curl: (22) The requested URL returned error: 404 Not Found
Nothing to update
Checking MRA files

Checking dd2
curl: (22) The requested URL returned error: 404 Not Found
Nothing to update
Checking MRA files

Checking f1dream
curl: (22) The requested URL returned error: 404 Not Found
Nothing to update
Checking MRA files

Checking gng
curl: (22) The requested URL returned error: 404 Not Found
Nothing to update
Checking MRA files

Checking gunsmoke
curl: (22) The requested URL returned error: 404 Not Found
Nothing to update
Checking MRA files

Checking hige
curl: (22) The requested URL returned error: 404 Not Found
Nothing to update
Checking MRA files

Checking sarms
curl: (22) The requested URL returned error: 404 Not Found
Nothing to update
Checking MRA files

Checking tora
curl: (22) The requested URL returned error: 404 Not Found
Nothing to update
Checking MRA files

Checking trojan
curl: (22) The requested URL returned error: 404 Not Found
Nothing to update
Checking MRA files

Checking sectionz
curl: (22) The requested URL returned error: 404 Not Found
Nothing to update
Checking MRA files

Checking vulgus
curl: (22) The requested URL returned error: 404 Not Found
Nothing to update
Checking MRA files

Checking MRA-Alternatives
curl: (22) The requested URL returned error: 404 Not Found
Nothing to update

On last step (doubled forward-slash likely the cause of the problem here):

Installing update_all.sh in MiSTer /Scripts directory.
           update_all.ini too.
           /tmp/ua_settings_update.ini too.
cp: cannot create regular file '../Scripts//tmp/ua_settings_update.ini': No such file or directory

Script /tmp/update_all.sh failed!

Add support for more unofficial cores (mandelbrot)

There are a few fun and interesting FPGA cores for MiSTer that are available - but not included in the unofficial core updater.

https://github.com/d18c7db/Gauntlet_FPGA
https://github.com/jacquesdriessen/MiSTer-mandelbrot