Alfresco BART is a tool written in shell script on top of Duplicity to do Alfresco backups and restore from a local file system, FTP, SCP or Amazon S3.
License: Apache License 2.0
Hi, i created a cron script that launches alfresco-bart.sh collection weekly, to check backup status. I noticed that in cron log it warns me of "command not found" at line 745 of alfresco-bart.sh.
At the end of that line there is a backslash, indeed...as far as I understood it is not necessary, or am I wrong?
Here is the line:
$DUPLICITYBIN collection-status -v0 ${NOENCFLAG} --log-file=${ALFBRT_LOG_FILE} $DEST/$INDEXTYPE/backup; \
Is there any updated version of alfresco bart for containerized version of alfresco?
Hi Tony
Just reporting a problem we found when testing BART using S3. Seems I needed to get a version of duplicity 0.7+ as the one bundled with at least Centos 7 is having problems. We have a ticket open on honeycomb build (Order of the bee), marsbard/puppet-alfresco#86
cheers
Daren
When using SCP for backup, there is not way to setup a custom port, if required.
Besides that, nowadays is recommended to use SFTP instead of SCP when a SSH connection is used.
Details can be found at http://duplicity.nongnu.org/duplicity.1.html#sect23
Why use sftp instead of scp? The change to sftp was made in order to allow the remote system to chroot the backup, thus providing better security and because it does not suffer from shell quoting issues like scp. Scp also does not support any kind of file listing, so sftp or ssh access will always be needed in addition for this backend mode to work properly. Sftp does not have these limitations but needs an sftp service running on the backend server, which is sometimes not an option.
I am working on this also, but because of "needs an sftp service running on the backend server, which is sometimes not an option" part, I think simply replacing SCP with SFTP could be a problem for some users, so the ideal would be to include a new backup option.
I will think about this.
I have been facing problems with "No space left on device" when bart is executed.
This is happening because when duplicity processes the volumes, it uses disc space, and if the partition is small, that can cause the disc to be completely filled.
I have this fix almost done here, as I am fixing it for a customer.
Once I finish it, I will create a PR.
Hi Toni,
first of all thanks for your precius work, I tested your script yesterday and it worked fine!
Now I have two questions for you:
2)My Alfresco is installed on an Ubuntu 12.04 LTS, so I installed duplicity with apt, and version 0.6.18 was installed; I did some test backups and everything worked fine, but do you suggest to upgrade to duplicity 0.6.22 anyhow? I am asking you this because the latest version is not still available on Ubuntu ufficial repositories, however I found source tarballs and .deb on the Internet.
Have a nice day!
Was getting this error:
NoAuthHandlerFound: No handler was ready to authenticate. 1 handlers were checked. ['HmacAuthV1Handler'] Check your credentials
Debugged the Boto Python code (storage_uri.py) and found out the access key id and secret key are None (and not Strings as they should be).
Fixed it by adding two export statements at line 80 of alfresco-bart.sh (just below the s3 case):
export AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}"
export AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}"
Hello,
I am getting the below errors again and again while backuping to s3 bucket.
ERROR 30 AssertionError
. Traceback (most recent call last):
. File "/usr/bin/duplicity", line 1532, in
. with_tempdir(main)
. File "/usr/bin/duplicity", line 1526, in with_tempdir
. fn()
. File "/usr/bin/duplicity", line 1380, in main
. do_backup(action)
. File "/usr/bin/duplicity", line 1405, in do_backup
. globals.archive_dir).set_values()
. File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 710, in set_values
. self.get_backup_chains(partials + backend_filename_list)
. File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 835, in get_backup_chains
. add_to_sets(f)
. File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 829, in add_to_sets
. if new_set.add_filename(filename):
. File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 101, in add_filename
. self.set_manifest(filename)
. File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 149, in set_manifest
. self.set_files_changed()
. File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 128, in set_files_changed
. mf = self.get_manifest()
. File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 250, in get_manifest
. return self.get_local_manifest()
. File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 225, in get_local_manifest
. return manifest.Manifest().from_string(manifest_buffer)
. File "/usr/lib/python2.7/dist-packages/duplicity/manifest.py", line 207, in from_string
. assert filecount == len(self.files_changed)
. AssertionError
.
Its was working fine till before 5-6 days. but now I am getting the above errors and its not working.
Please guide me what to do to resolve this issue?.
Hey @toniblyx
Shouldn't we mark all the latest changes as the version 0.3?
When running ./alfresco-bart.sh (0.2 tag) I get the following in the log files
2014-09-02-10:08:11 - [BART-Loglevel4] ContentStore backup done!
2014-09-02-10:08:11 - [BART-Loglevel4] Backing up the Alfresco files to local
2014-09-02-10:08:11 - [BART-Loglevel4] Starting backup - Alfresco files
2014-09-02-10:08:11 - [BART-Loglevel4] Running command - /usr/bin/duplicity -v4 --volsize=25 --log-file=/opt/bart/logs/alfresco-bart-2014-09-02.log --full-if-older-than 30D --asynchronous-upload --no-encryption --exclude /var/lib/tomcat7/alf_data/contentstore --exclude /var/lib/tomcat7/alf_data/alfresco-db-backup file:///mnt/backup/files
ERROR 2
. Command line error: Expected 2 args, got 1
. Enter 'duplicity --help' for help screen.
2014-09-02-10:08:11 - [BART-Loglevel4] Files backup done!
2014-09-02-10:08:11 - [BART-Loglevel4] Running maintenance commands
Not sure if the backup ended up successfully or not, but I see some files created in /mnt/backup.
Will continue with the investigation.
In my case I have Alfresco installed into /opt/alfresco, but my contentstore and index files are in different disks, mounted as /alfresco/alf_data and /alfresco/solr
When the script tries to build the backup line in the function "filesBackup", the script fails when it tries to ignore everything from line 229 to 258.
There must be a duplicity parameter to say something like "if the folder I'm telling you to ignore is not into the folder I'm backing up, ignore the non existence and go ahead"
I quick solved my problem by commenting all those lines.
I hope my explanation was enough, and sorry for the level of details, as I'm writing this in hurry while trying to recover a server.
When trying to export in alfresco 6.0, it throws a 400 browser error
Hi,
sometimes we noticed a one day shift during the execution of full backup (ie: configuration settings for full backup is "7D", but backup is executed after 8D).
The event is triggered because the time of the last full backup is before current start time.
Example log:
`Data dell'ultimo backup completo: Sat Mar 16 19:21:18 2019
NOTICE 1
. Data dell'ultimo backup completo: Sat Mar 16 19:21:18 2019
--------------[ Statistiche del backup ]--------------
StartTime 1552846872.62 (Sun Mar 17 19:21:12 2019)
EndTime 1552847954.54 (Sun Mar 17 19:39:14 2019)
ElapsedTime 1081.92 (18 minutes 1.92 seconds)
SourceFiles 1755775
SourceFileSize 1137161314644 (1.03 TB)
NewFiles 0
NewFileSize 0 (0 bytes)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 0
RawDeltaSize 0 (0 bytes)
TotalDestinationSizeChange 20 (20 bytes)
Errors 0
------------------------------------------------------`
As you can see the Last full backup date is Sat Mar 16 19:21:18 2019, but the start time of the current backup is Sun Mar 17 19:21:12 2019. For only 6 seconds scripts not trigger full backup, but instead incremental backup.
This is a common behavior in production environment and sometimes is a problem because full backup is very large and it should be scheduled the same day of week (ie: for bandwith problem).
Do you have any solutions?
Thanks in advance
Domenico
Hello!
Is it posible to use it with Community edition 5.0.a? I get the error:
alfresco-bart.sh: 112: alfresco-bart.sh: function: not found
alfresco-bart.sh: 118: [: solr: unexpected operator
alfresco-bart.sh: 123: alfresco-bart.sh: Syntax error: "}" unexpected
Thank you!
Hello,
I am trying to use this tool for backuping data to gcloud.
I have added some lines in alfresco-bart.sh :
case $BACKUPTYPE in
"s3" )
export AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}"
export AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}"
DEST=${S3FILESYSLOCATION}
PARAMS="${GLOBAL_DUPLICITY_PARMS} ${S3OPTIONS} ${NOENCFLAG}"
;;
"gcloud" )
export GS_ACCESS_KEY_ID="${GS_ACCESS_KEY_ID}"
export GS_SECRET_ACCESS_KEY="${GS_SECRET_ACCESS_KEY}"
DEST=${GCLOUD_FILE_SYS_LOCATION}
PARAMS="${GLOBAL_DUPLICITY_PARMS} ${NOENCFLAG}"
;;
"ftp" )
if [ ${FTPS_ENABLE} == 'false' ]; then
DEST=ftp://${FTP_USER}:${FTP_PASSWORD}@${FTP_SERVER}:${FTP_PORT}/${FTP_FOLDER}
PARAMS="${GLOBAL_DUPLICITY_PARMS} ${NOENCFLAG}"
else
DEST=ftps://${FTP_USER}:${FTP_PASSWORD}@${FTP_SERVER}:${FTP_PORT}/${FTP_FOLDER}
PARAMS="${GLOBAL_DUPLICITY_PARMS} ${NOENCFLAG}"
fi
;;
"scp" )
DEST=scp://${SCP_USER}@${SCP_SERVER}/${SCP_FOLDER}
PARAMS="${GLOBAL_DUPLICITY_PARMS} ${NOENCFLAG}"
;;
I think, I need more configuration or something.
I need help with backup to gcloud bucket.
Thanks.
Hello,
First, congrats for this amazing job.
My question concerns the line $DUPLICITYBIN $PARAMS $INDEXES_DIR --exclude $INDEXES_DIR/archive --exclude $INDEXES_DIR/workspace $DEST/$INDEXTYPE/config of your script.
My $INDEXES is ALF_DIRROOT/solr4/index and contains only 2 directories "workspace" and "archive" and both are excluded in your command.
Could you explain the goal of this command and tell me if the backup of INDEXES_BACKUP_DIR could be enough to restore Solr ?
Thank you.
Christophe
Hi All
I am trying to backup a test Alfresco 5.2 community installation to AWS S3. This is running on Ubuntu 16.04 LTS.
For some reason it doesn't backup the contentstore and I get the below error message:
Fatal Error: The file specification
true
cannot match any files in the base directory
/opt/installations/systest/alf_data
Useful file specifications begin with the base directory or some
pattern (such as '**') which matches the base directory.
I am running this as root - and the root user has full access to all folders in alf_data.
Also attaching the alfresco-bart.properties file here for reference (had to add .txt at end of file name, as github refuses to upload a .properties file)
alfresco-bart.properties.txt
Any ideas would be appreciated.
Thanks
Vipul
Hai Friend,
I am facing this issue while taking backup of alfresco
ERROR 31 GPG Error,
GPG Error: GPG Failed,
gpg: problem with the agent: Not implemented
and while generating gpg key it throwing
gpg: problem with the agent: Not implemented
how to solve this issue help me please
I am using
gnupg 2.0.30
redhat linux
duplicity
Hello,
Are you planning to migrate the module to Alfresco 5.2?
If so, when is it planned?
Thanks in advance,
Regards,
NESSON Melvin
Hi - firstly - loving Alfresco BART.
Now we have started using S3 for contentstore (rather than local disk) in Alfresco installations. I understand that BART only reads the local disk filesystem and backs that up to S3 (among other destinations).
My question is - can S3 be the Source and Destination (different buckets, of course)?
We are on Alfresco 5.2 Community Edition, on Ubuntu 16.04.
Thanks
As discussed at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=724594, using the --extra-clean parameter can make the "not-latest" full backup to be corrupted, and then, not recoverable.
from the man page:
--extra-clean
When cleaning up, be more aggressive about saving space. For
example, this may delete signature files for old backup chains.
@toniblyx What do you think?
Hello Tony,
We have stored our alfresco contentstore on a NAS server and linked to that data with symbolic links in the alf_data folder.
root@Alfresco-test:/opt/alfresco/scripts# cd /opt/alfresco/alf_data/
root@Alfresco-test:/opt/alfresco/alf_data# ll
drwxr-xr-x 2 alfresco-db-backup/
lrwxrwxrwx 1 contentstore -> /mnt/Transport/alfresco_data_storage/contentstore/
lrwxrwxrwx 1 contentstore.deleted -> /mnt/Transport/alfresco_data_storage/contentstore.deleted//
lrwxrwxrwx 1 keystore -> /mnt/Transport/alfresco_data_storage/keystore//
drwx------ 12 postgresql/
lrwxrwxrwx 1 solr -> /mnt/Transport/alfresco_data_storage/solr//
drwxr-xr-x 4 solrBackup/
The ALF_CONTENTSTORE property is set to the default value ${ALF_DIRROOT}/contentstore and then the backup is run (backup all).
Unfortunately nothing gets written in the backup.
Then I changed the ALF_CONTENTSTORE property to point to the real location of the data (/mnt/Transport/alfresco_data_storage/contentstore) and start again.
Then alfresco-bart throws the following exception:
root@Alfresco-test:/opt/alfresco/scripts# ./alfresco-bart.sh backup cs
Fatal Error: Backup source directory has changed.
Current directory: /mnt/Transport/alfresco_data_storage
Previous directory: /opt/alfresco-4.1.1.3/alf_data
Aborting because you may have accidentally tried to backup two different data sets to the same remote location, or using the same archive directory. If this is not a mistake, use the --allow-source-mismatch switch to avoid seeing this message
Ok I try the allow-source-mismatch option, but that does not exists so I used the force parameter.
root@Alfresco-test:/opt/alfresco/scripts# ./alfresco-bart.sh backup all force
Then the following fatal error is shown (after the backup process i think)
Fatal Error: The file specification
/mnt/Transport/alfresco_data_storage/contentstore
cannot match any files in the base directory
/opt/alfresco-4.1.1.3
Useful file specifications begin with the base directory or some
pattern (such as '**') which matches the base directory.
root@Alfresco-test:/opt/alfresco/scripts#
Is there a way to avoid this problem?
Or should we configure the alf_data structure differently?
Kind regards,
Bart Berden
APS Group BV
Hello Toni,
Thanks a lot for your work !
Afresco bart works very well with small and medium repositories, no problem so far.
I've tested alfresco bart with a 1300GB repository and it takes more than 12 hours to complete.
Do you know how I could decrease the time needed for the backup ? 4-5 hours would be acceptable.
(disabling encryption ? or compression ? any other seting ?)
My setup:
Intel i7 4770
8GB RAM
2 * 4TB HDD
Ubuntu 12.04 LTS
Alfresco-4.2.d Community edition
My repository:
mix of few thousands office documents (.doc, .xls, .pdf ...) and about 60'000 pics and videos.
Thanks in advance for your reply.
Jean-Pierre
Hey @toniblyx
I have a suggestion about the destination paths for database and index.
For contentstore, you use $DEST/cs
For config files, you use $DEST/files
Why not to use something like $DEST/database or $DEST/db for database instead of $DEST/$DBTYPE and $DEST/index for the index, instead of $DEST/$INDEXTYPE
If you agree on that, I can do the changes and provide a pull request.
Hi.
The backup works fine in local mode. The collection switch returns the correct information.
However when I try to use s3, files seem to be correctly loaded in s3 but then the collection switch returns BackendException: No connection to backend.
Here is my configuration file:
# Copyright (c) 2013 Toni de la Fuente.
#
# This program is free software: you can redistribute it and/or modify it under
# the terms of the Apache License as published by the Apache Foundation.
#
# This program is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
# FOR A PARTICULAR PURPOSE.
#
# Most recent information about this tool is available in:
# http://blyx.com/alfresco-bart
#
# Latest code available at:
# http://blyx.com/alfresco-bart
#
#########################################################################################
# alfresco-bart: ALFRESCO BACKUP AND RECOVERY TOOL
# Version 0.2
#########################################################################################
# ACTION REQUIRED:
# this file should contains passwords, please, set this file as read only
# for root (chmod 400 alfresco-bart.properties)
# Copy this file in you ALFRESCO_INSTALLATION_PATH/scripts.
#########################################################################################
## Logging
# Log configuration
LOG_DATE_FILES=`date +%F`
LOG_DATE_LOG=`date +%F-%X`
# Directory to left all Alfresco BART tasks logs with reports.
ALFBRT_LOG_DIR=/opt/alfresco/logs/bart
ALFBRT_LOG_FILE=${ALFBRT_LOG_DIR}/alfresco-bart-${LOG_DATE_FILES}.log
## Duplicity Setup ##
# If yes, please make sure you specify either PASSPHRASE, see INSTALL file.
ENCRYPTION_ENABLED=false
PASSPHRASE=xxx
DUPLICITYBIN=`which duplicity`
GZIP=`which gzip`
## Backup policies
# Number of days of every full backup (if not backup found it does a full)
FULLDAYS=30D
# Backup policies to apply all backups collections (retention and cleanup)
BACKUP_POLICIES_ENABLED=true
# Number of moths to remove all backups older than or backup retention period
CLEAN_TIME=3M
# After MAXFULL counter, all incrementals will be deleted and all full will be kept until CLEAN_TIME applies,
MAXFULL=6
# If you want to keep full backups of last 12 months but only with incremental in last 6 months
# you must set CLEAN_TIME=12M and MAXFULL=6
# Volume size in MB, default is 25MB per backup volume, consider reduce or increase it if you
# are doing tape backup (if a backup takes 60MB you will get 3 volumes, 25+25+10)
VOLUME_SIZE=25
# Alfresco root installation path. It has to be a real directory not a symlink
ALF_INSTALLATION_DIR=/opt/alfresco
# Alfresco alf_data path
ALF_DIRROOT=${ALF_INSTALLATION_DIR}/alf_data
# Duplicity log vervosity 0 Error, 2 Warning, 4 Notice, 8 Info, 9 Debug (noisiest)
#0 recommended for production
DUPLICITY_LOG_VERBOSITY=4
BART_LOG_TAG=[BART-Loglevel$DUPLICITY_LOG_VERBOSITY]
GLOBAL_DUPLICITY_PARMS="-v${DUPLICITY_LOG_VERBOSITY} --volsize=${VOLUME_SIZE} --log-file=${ALFBRT_LOG_F
## Indexes Configuration
BACKUP_INDEX_ENABLED=false
# Alfresco index type, use solr or lucene
INDEXTYPE=lucene
# If Solr is selected, all Solr installation and config files will be included in your backup
INDEXES_DIR=${ALF_DIRROOT}/lucene
# If Lucene change to ${ALF_DIRROOT}/lucene
# SOLR or Lucene backup location
INDEXES_BACKUP_DIR=${ALF_DIRROOT}/backup-lucene-indexes
# If lucene add the apropiate location like ${ALF_DIRROOT}/backup-lucene-indexes
# If solr, please go to Alfresco Share Admin Console - Search Solr and set "Backups to
# keep to 1" in both Main and Archive
# Other that do a backup of the "backed up" indexes, to easy restore of your Solr
# indexes is recommendable to backup all configuration files but the live files,
# To enable Solr installation and configuration backup set next property
## DB Configuration ##
BACKUP_DB_ENABLED=true
# use mysql, postgresql or oracle
DBTYPE=mysql
LOCAL_BACKUP_DB_DIR=${ALF_DIRROOT}/alfresco-db-backup
# if you are using the bundle posgresql db, with next value it will include pg live files
# for easy restore. For remote, different or external db leave it as is.
LOCAL_DB_DIR=${ALF_DIRROOT}/postgresql
DATE_FILE=`date +%Y-%m-%d_%H%M%S`
# Global DB parameters
DBNAME=alfresco
DBUSER=alfresco
DBPASS=xxxxxxxxxxxxxx
DBHOST=localhost
# MySQL - Alfresco DB Configuration
MYSQL_BINDIR=/usr/bin
MYSQLDUMP_BIN=mysqldump
# PostgreSQL - Alfresco DB Configuration
PGSQL_BINDIR=${ALF_INSTALLATION_DIR}/postgresql/bin
PGSQLDUMP_BIN=pg_dump
# Add PGUSER and PGPASSWORD if required for your installation and you are not doing
# running BART as root.
PGUSER=
PGPASSWORD=${DBPASS}
# If you found problems to dump your Postgres DB try creating a file .pgpass with
# content "::::alfresco" without quotes, while alfresco is the password of your alfresco db
PGPASSFILE=${ALF_DIRROOT}/.pgpass
# Oracle - Alfresco DB Configuration
ORACLE_BINDIR=
ORASQLDUMP_BIN=exp
## ContentStore Configuration
BACKUP_CONTENTSTORE_ENABLED=true
ALF_CONTENTSTORE=${ALF_DIRROOT}/contentstore
# To exclude contentstore.deleted leave next value empty
ALF_CONTENSTORE_DELETED=
ALF_CACHED_CONTENTSTORE=
ALF_CONTENTSTORE2=
ALF_CONTENTSTORE3=
ALF_CONTENTSTORE4=
ALF_CONTENTSTORE5=
## Alfresco configuration and application files backup
# This backup includes deployments and configuration files excludes
# the ContentStore, Indexes and DB as above
BACKUP_FILES_ENABLED=true
## Backup type ##
# use s3, ftp, scp, local
BACKUPTYPE=s3
## LOCAL backup destination folder ##
# absolute path starting with /
LOCAL_BACKUP_FOLDER=/home/bastien/backup
## Amazon S3 information ##
AWS_ACCESS_KEY_ID=xxxxxxxxxxxxxxxxxxxxxxxxxx
AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
# Upper case bucket name is not allow
S3FILESYSLOCATION="s3+http://my_bucket"
S3OPTIONS="--s3-use-new-style --s3-use-rrs"
S3_EUROPEAN_BUCKET=false # not used yet
## FTP Information ##
FTP_SERVER=
FTP_USER=
FTP_PASSWORD=
FTP_FOLDER=
FTP_PORT=21
FTPS_ENABLE=false
## SCP Information ##
# SSH shared keys required
SCP_SERVER=
SCP_USER=
SCP_FOLDER=
## Backup temp folder definition ##
TEMPFOLDERNAME="alfresco-backup-`date +%F-%s`"
## Recovery configuration
ENABLE_SINGLE_FILE_RECOVERY=false
# Temporary DB for recovery and get the phisical path of a single content.
REC_DBTYPE=mysql
# Recovery database must be already created but EMPTY! Alfresco-BART will recover your
# backup database and find the files for you, after this task the recovery database
# tables will be deleted.
# MySQL - Recovery DB Configuration ##
REC_MYDBNAME=alfresco_rec
REC_MYUSER=alfresco
REC_MYPASS=xxxxxxxxxxxxxxxxxxxx
REC_MYHOST=localhost
REC_MYSQL_BIN=/usr/bin/mysql
REC_MYSQLDUMP_BIN=/usr/bin/mysqldump
# PgSQL - Recovery DB Configuration ##
REC_PGDBNAME=alfresco_rec
REC_PGUSER=alfresco
REC_PGPASS=alfresco
REC_PGHOST=localhost
REC_PGSQL_BIN=/opt/alfresco/postgresql/bin/psql
REC_PGSQLDUMP_BIN=/opt/alfresco/postgresql/bin/pg_dump
# Oracle - Recovery DB Configuration ##
REC_ORADBNAME=alfresco_rec
REC_ORAUSER=alfresco
REC_ORAPASS=alfresco
REC_ORAHOST=localhost
REC_ORASQL_BIN=
REC_ORASQLDUMP_BIN=exp
alfresco-bart.sh hardcodes the location of alfresco-bart.properties, making it difficult to use and error-prone.
I think this could be greatly improved by:
Is BART available for windows? Would be a great tool for development purposes too to test code and have a quick possibility to rollback...
Thanks,
Jens
I don't know why, and I am going to investigate it, but the backup statistics is always show twice in the logs, just like this:
--------------[ Backup Statistics ]--------------
StartTime 1496606481.20 (Sun Jun 4 22:01:21 2017)
EndTime 1496606484.76 (Sun Jun 4 22:01:24 2017)
ElapsedTime 3.56 (3.56 seconds)
SourceFiles 12851
SourceFileSize 1223592859 (1.14 GB)
NewFiles 6
NewFileSize 104430 (102 KB)
DeletedFiles 0
ChangedFiles 3
ChangedFileSize 71607 (69.9 KB)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 9
RawDeltaSize 131018 (128 KB)
TotalDestinationSizeChange 8623 (8.42 KB)
Errors 0
-------------------------------------------------
NOTICE 1
. --------------[ Backup Statistics ]--------------
. StartTime 1496606481.20 (Sun Jun 4 22:01:21 2017)
. EndTime 1496606484.76 (Sun Jun 4 22:01:24 2017)
. ElapsedTime 3.56 (3.56 seconds)
. SourceFiles 12851
. SourceFileSize 1223592859 (1.14 GB)
. NewFiles 6
. NewFileSize 104430 (102 KB)
. DeletedFiles 0
. ChangedFiles 3
. ChangedFileSize 71607 (69.9 KB)
. ChangedDeltaSize 0 (0 bytes)
. DeltaEntries 9
. RawDeltaSize 131018 (128 KB)
. TotalDestinationSizeChange 8623 (8.42 KB)
. Errors 0
. -------------------------------------------------
When you perform a restore using BART, after the DB restore, the user gets an informative message about how to restore the database using the dump file.
The command is printed like
$PGSQL_BINDIR/psql --host=$DBHOST -U $DBUSER -d $DBNAME -f $DBNAME.dump.gz
That command doesn't work for the dump file generated by using the --format=p
The way I do the restore is like this
gunzip -c alfresco.dump | sudo -u postgres psql --host=localhost -U alfresco -d alfresco
The message I am talking about is defined at https://github.com/toniblyx/alfresco-backup-and-recovery-tool/blob/master/src/alfresco-bart.sh#L328
The script run ok but sometime once per 1 ou 2 month, it's not finish so when the script want start the next day i get a message of error "alfresco-bart.sh or duplicity is already running."
What i can check for find the problem. My skill in linux was limited
Thank for your help
Hello, I set up BART with encryption disabled, but today I tried to restore a single file and it asked me for a GPG password. If I did non write anything it gave me an error saying that the password can not be empty; so I tried to insert a random password and the restore process started. Before completing the restore, the script asked for the password again and the restore completed successfully.
Is it the normal behaviour?
I have installed PostgreSQL and Alfresco in different server. I am not able to take back up using BART. Getting below error
Fatal Error: The file specification /home/thirumal/alfresco-db-backup cannot match any files in the base directory /opt/alfresco-community Useful file specifications begin with the base directory or some pattern (such as '**') which matches the base directory.
Any suggestion ?
Hi there
I followed your instructions on how to use the bart.sh changed the necessary as recommended, but when i run the ./alfresco-bart.sh backup [all] i get the following errors:
which: no duplicity in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin)
./alfresco-bart.sh: line 121: -v4: command not found
./alfresco-bart.sh: line 124: -v4: command not found
./alfresco-bart.sh: line 165: -v4: command not found
./alfresco-bart.sh: line 222: -v4: command not found
./alfresco-bart.sh: line 269: -v4: command not found
./alfresco-bart.sh: line 769: remove-older-than: command not found
./alfresco-bart.sh: line 771: remove-all-inc-of-but-n-full: command not found
./alfresco-bart.sh: line 773: remove-older-than: command not found
./alfresco-bart.sh: line 774: remove-all-inc-of-but-n-full: command not found
./alfresco-bart.sh: line 788: remove-older-than: command not found
./alfresco-bart.sh: line 790: remove-all-inc-of-but-n-full: command not found
I then check my log files 👍
2017-03-28-04:54:22 AM - [BART-Loglevel4] Starting backup
2017-03-28-04:54:22 AM - [BART-Loglevel4] Set script varibles done
2017-03-28-04:54:22 AM - [BART-Loglevel4] Backing up the Alfresco indexes to local
2017-03-28-04:54:22 AM - [BART-Loglevel4] Starting backup - Alfresco solr indexes
2017-03-28-04:54:22 AM - [BART-Loglevel4] Running command - -v4 --volsize=25 --log-file=/opt/alfresco/scripts/alfresco-bart-2017-03-28.log --full-if-older-than 30D --asynchronous-upload /opt/alfresco/alf_data/solr4Backup file:///var/backups/solr
2017-03-28-04:54:22 AM - [BART-Loglevel4] Running command - -v4 --volsize=25 --log-file=/opt/alfresco/scripts/alfresco-bart-2017-03-28.log --full-if-older-than 30D --asynchronous-upload /opt/alfresco/alf_data/solr --exclude /opt/alfresco/alf_data/solr/archive --exclude /opt/alfresco/alf_data/solr/workspace file:///var/backups/solr/config
2017-03-28-04:54:22 AM - [BART-Loglevel4] Indexes backup finished
2017-03-28-04:54:22 AM - [BART-Loglevel4] Backing up the Alfresco db to local
2017-03-28-04:54:22 AM - [BART-Loglevel4] Starting backup - Alfresco postgresql db
2017-03-28-04:54:22 AM - [BART-Loglevel4] Backing up the Alfresco DB to local
2017-03-28-04:54:22 AM - [BART-Loglevel4] Starting backup - Alfresco postgresql DB
2017-03-28-04:54:22 AM - [BART-Loglevel4] Running command - /opt/alfresco/postgresql/bin/pg_dump --host=localhost --username=alfresco --format=p --compress=9 --file=/opt/alfresco/alf_data/alfresco-db-backup/alfresco.dump alfresco -w
2017-03-28-04:54:22 AM - [BART-Loglevel4] Running command - -v4 --volsize=25 --log-file=/opt/alfresco/scripts/alfresco-bart-2017-03-28.log --full-if-older-than 30D --asynchronous-upload /opt/alfresco/alf_data/alfresco-db-backup file:///var/backups/postgresql
2017-03-28-04:54:22 AM - [BART-Loglevel4] cleaning DB backup
2017-03-28-04:54:22 AM - [BART-Loglevel4] DB backup finished
2017-03-28-04:54:22 AM - [BART-Loglevel4] Backing up the Alfresco ContentStore to local
2017-03-28-04:54:22 AM - [BART-Loglevel4] Starting backup - Alfresco ContentStore
2017-03-28-04:54:22 AM - [BART-Loglevel4] Running command - -v4 --volsize=25 --log-file=/opt/alfresco/scripts/alfresco-bart-2017-03-28.log --full-if-older-than 30D --asynchronous-upload --include /opt/alfresco/alf_data/contentstore --exclude /opt/alfresco/alf_data /opt/alfresco/alf_data file:///var/backups/cs
2017-03-28-04:54:22 AM - [BART-Loglevel4] ContentStore backup done!
2017-03-28-04:54:22 AM - [BART-Loglevel4] Backing up the Alfresco files to local
2017-03-28-04:54:22 AM - [BART-Loglevel4] Starting backup - Alfresco files
2017-03-28-04:54:22 AM - [BART-Loglevel4] Running command - -v4 --volsize=25 --log-file=/opt/alfresco/scripts/alfresco-bart-2017-03-28.log --full-if-older-than 30D --asynchronous-upload /opt/alfresco --exclude /opt/alfresco/alf_data/solr4Backup --exclude /opt/alfresco/alf_data/contentstore --exclude /opt/alfresco/alf_data/contentstore.deleted --exclude /opt/alfresco/alf_data/alfresco-db-backup file:///var/backups/files
2017-03-28-04:54:22 AM - [BART-Loglevel4] Files backup done!
2017-03-28-04:54:22 AM - [BART-Loglevel4] Running maintenance commands
2017-03-28-04:54:22 AM - [BART-Loglevel4] Running command - remove-older-than 12M -v4 --log-file=/opt/alfresco/scripts/alfresco-bart-2017-03-28.log --force --extra-clean file:///var/backups/solr/backup
2017-03-28-04:54:22 AM - [BART-Loglevel4] Running command - remove-all-but-n-full 6 -v4 --log-file=/opt/alfresco/scripts/alfresco-bart-2017-03-28.log --force file:///var/backups/solr/backup
2017-03-28-04:54:22 AM - [BART-Loglevel4] Running command - remove-older-than 12M -v4 --log-file=/opt/alfresco/scripts/alfresco-bart-2017-03-28.log --force --extra-clean file:///var/backups/postgresql
2017-03-28-04:54:22 AM - [BART-Loglevel4] Running command - remove-all-but-n-full 6 -v4 --log-file=/opt/alfresco/scripts/alfresco-bart-2017-03-28.log --force file:///var/backups/postgresql
2017-03-28-04:54:22 AM - [BART-Loglevel4] Running command - remove-older-than 12M -v4 --log-file=/opt/alfresco/scripts/alfresco-bart-2017-03-28.log --force --extra-clean file:///var/backups/cs
./alfresco-bart.sh: line 803: remove-older-than: command not found
2017-03-28-04:54:22 AM - [BART-Loglevel4] Running command - remove-all-but-n-full 6 -v4 --log-file=/opt/alfresco/scripts/alfresco-bart-2017-03-28.log --force file:///var/backups/cs
./alfresco-bart.sh: line 805: remove-all-inc-of-but-n-full: command not found
2017-03-28-04:54:22 AM - [BART-Loglevel4] Running command - remove-older-than 12M -v4 --log-file=/opt/alfresco/scripts/alfresco-bart-2017-03-28.log --force --extra-clean file:///var/backups/files
./alfresco-bart.sh: line 809: remove-older-than: command not found
2017-03-28-04:54:22 AM - [BART-Loglevel4] Running command - remove-all-but-n-full 6 -v4 --log-file=/opt/alfresco/scripts/alfresco-bart-2017-03-28.log --force file:///var/backups/files
./alfresco-bart.sh: line 811: remove-all-inc-of-but-n-full: command not found
2017-03-28-04:54:22 AM - [BART-Loglevel4] Maintenance commands DONE!
Where it says my backup is complete, but i cant find the backup anywhere, not in my local_dir or alf_data folder at all.
Am i doing something wrong?
Hello,
I need help in cleaning/deleting old backups. Its not deleting old backups, just printing some messages like :
Local and Remote metadata are synchronized, no sync needed.
NOTICE 1
. Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Tue Sep 18 01:35:44 2018
NOTICE 1
. Last full backup date: Tue Sep 18 01:35:44 2018
There are backup set(s) at time(s):
Thu Jun 7 11:05:51 2018
Fri Jun 8 01:30:43 2018
Sat Jun 9 01:30:24 2018
Sun Jun 10 01:30:25 2018
Mon Jun 11 01:30:25 2018
Tue Jun 12 01:30:25 2018
Wed Jun 13 01:30:25 2018
Thu Jun 14 01:30:25 2018
Fri Jun 15 01:30:25 2018
Sat Jun 16 01:30:24 2018
Sun Jun 17 01:30:25 2018
Mon Jun 18 01:30:27 2018
Mon Jun 25 01:30:29 2018
Tue Jun 26 01:30:26 2018
Wed Jun 27 01:30:25 2018
Thu Jun 28 01:30:24 2018
Fri Jun 29 01:30:25 2018
Sat Jun 30 01:30:29 2018
Sun Jul 1 01:30:26 2018
Mon Jul 2 01:30:25 2018
Tue Jul 3 01:30:25 2018
Wed Jul 4 01:30:26 2018
Thu Jul 5 01:30:26 2018
Fri Jul 6 01:30:26 2018
Sat Jul 7 01:30:26 2018
Sun Jul 8 01:30:27 2018
Mon Jul 9 01:30:27 2018
Tue Jul 10 01:30:26 2018
Wed Jul 11 01:30:27 2018
Thu Jul 12 01:30:26 2018
Fri Jul 13 01:30:26 2018
Sat Jul 14 01:30:26 2018
Sun Jul 15 01:30:27 2018
Mon Jul 16 01:30:27 2018
Which can't be deleted because newer sets depend on them.
WARNING 1
. There are backup set(s) at time(s):
. Thu Jun 7 11:05:51 2018
. Fri Jun 8 01:30:43 2018
. Sat Jun 9 01:30:24 2018
. Sun Jun 10 01:30:25 2018
. Mon Jun 11 01:30:25 2018
. Tue Jun 12 01:30:25 2018
. Wed Jun 13 01:30:25 2018
. Thu Jun 14 01:30:25 2018
. Fri Jun 15 01:30:25 2018
. Sat Jun 16 01:30:24 2018
. Sun Jun 17 01:30:25 2018
. Mon Jun 18 01:30:27 2018
. Mon Jun 25 01:30:29 2018
. Tue Jun 26 01:30:26 2018
. Wed Jun 27 01:30:25 2018
. Thu Jun 28 01:30:24 2018
. Fri Jun 29 01:30:25 2018
. Sat Jun 30 01:30:29 2018
. Sun Jul 1 01:30:26 2018
. Mon Jul 2 01:30:25 2018
. Tue Jul 3 01:30:25 2018
. Wed Jul 4 01:30:26 2018
. Thu Jul 5 01:30:26 2018
. Fri Jul 6 01:30:26 2018
. Sat Jul 7 01:30:26 2018
. Sun Jul 8 01:30:27 2018
. Mon Jul 9 01:30:27 2018
. Tue Jul 10 01:30:26 2018
. Wed Jul 11 01:30:27 2018
. Thu Jul 12 01:30:26 2018
. Fri Jul 13 01:30:26 2018
. Sat Jul 14 01:30:26 2018
. Sun Jul 15 01:30:27 2018
. Mon Jul 16 01:30:27 2018
. Which can't be deleted because newer sets depend on them.
No old backup sets found, nothing deleted.
NOTICE 1
. No old backup sets found, nothing deleted.
I have saved value 2M for property CLEAN_TIME but its not actually deleting the old backups. Please guide me how it will work or How to make it work ?
Thanks.
duplicity creates files, uses a lot of space in the .cache/duplicity directory. Is there a preferred may to clean this up (as part of the BART process)?
Now for detecting is used:
ps axu | grep -v "grep" | grep --count "duplicity"
The a option shows processes for all users so if there is another duplicity task running on the same machine under different user then our script will not start.
Something like this (added some other minor cleanup) should work:
ps xu | grep --count 'dupli[c]ity'
or
pgrep --uid "${LOGNAME}" --count duplicity
Or delete detection from the script (lines 47-52) and use in crontab file something like:
flock -xn /path/to/.flock.alfresco-bart.sh -c /path/to/alfresco-bart.sh
Hi Toni,
Could you explain the "high level" process to include MSSQL support (or others) in Alfresco BART?
We are interested to collaborate with you to include this enhacement.
Kind regards!
I have noticed that the log file has all the lines with the same moment, like 2017-06-14-22:00:01
My backup is scheduled using cron, once a day, always on 10 PM, and all the lines keep repeating the moment it started.
This backup can take more than one hour, based on the statistics output for the files backup, the latest step:
--------------[ Backup Statistics ]--------------
StartTime 1497491193.45 (Wed Jun 14 22:46:33 2017)
EndTime 1497491202.35 (Wed Jun 14 22:46:42 2017)
@toniblyx The test you asked me to do.
we configured a new backup server with alfresco which will be used in case of disaster with our main alfresco server, Is it possible to use a common mount path to backup the data, so that we can backup from main server in daily basis and restore it to other server whenever the main server is down? any limitations are there in this case?
Will the entire production data documents get created in new server by using this method?
I'm a newbie please bear with my questions.
Regards
Amar.
In commit 23f1ced#commitcomment-36381643 last 3 lines of postgres dump has been removed.
As stated by the duplicity's manual, the force option can be dangerous:
--force
Proceed even if data loss might result. Duplicity will let the user know when this option is required.
Because of that, I believe it should be removed from the script.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.