The CRS-INPE performs daily simulations of the total electronic content of the ionosphere through an ionosphere prediction model (SUPIM-DAVS).
These simulations are done using observational data executed on the research institute clusters.
However, there wasn't an intelligent backup system to save SUPIM-DAVS simulation data and virtualized machines at INPE.
In this way, this software was developed to realize a backup on INPE sensitive data.
P.S.: All sensitive data, such as paths, external and local IPs, folder names and files were hidden for security reasons.
Given that there was no system to keep sensitive files on INPE and prevent failures, I developed Shell Script algorithms to realize SUPIM backup (daily) and VM's backup (weekly).
The scripts were all developed for a LINUX system and configured to run on CRON in the backup machine.
To backup the ionospheric simulation data, we developed the julianday.sh script to calculate the current Julian day and Julian previous two days to facilitate the process of backup data, given that the ionospheric simulation system is based on julian days.
The backup.sh file is responsible for running the julianday.sh script and thus check whether is a leap year. In addition, it check for management of backup folders, thus not overwrite folders or save files of interest in a wrong place.
So, the backup machine is ssh accessed and it is verified the existence of the last data to be copied.
Then the backup is made to keep sensitive simulation files safe (if they are available to be copied).
The cronbackupmachines.sh file is responsible for turn off INPE VMs one time a week, create specific images of each machine and reattach them after this process.
This is a very complex work, because INPE sensitive virtualized machines need to be switched off.
Therefore, it is a procedure that should not admit fault.
Given that the machine image files are large, it is necessary to make a memory management on the server, so there is always space for backup.
The backupVMs.sh file is responsible for accessing the server, and then copy the .img files to the backup machine and do all the local memory management.
Moreover, perform a verification if the backup server was successful.
For this task some points need to be checked, such as size and existence of files in the backup folder, etc..
In addition, has been configured a postfix system in the backup machine.
Thus, always that a new backup procedure is done it is emailed a backup report to people of interest.
Below there is an image of a received backup email referent at julian day #268.
Hardly the project itself can be reproduced because it is a backup system quite complex to a specific case.
However, some search ideas and memory management in backup situations can be reused.
If you want to reproduce some part of system or if you want to copy a piece of code, please, just keep credits to the author.
Enjoy!
Hollweg
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent