Terminology

|product|-machine - The machine where the Guardium Big Data Intelligence system is running, and should be backed up (a.k.a. The source system).

Backup-machine - The system where the sonar remote backup is installed.

Prerequisites

Ensure that SELinux is set to Permissive:

setenforce permissive

Installation

Installation using a tarball

1.Download the tarball to the Backup-machine.

2.Un-tar the tarball to a local folder i.e.:

tar -xvf rhel7.x_sonarbackup_installer_*.tar.gz

3.Copy the setup script to the local folder:

cp sonarbackup_rhel7_local_repo/*.sh ./

4.Run the installer:

sudo ./sonarbackup_rhel7_install.sh | & tee sonarbackup-install.out

5.Check the sonarbackup-install.out for errors:

grep -inE "error|fail|not|non|missing|warning" sonarbackup-install.out

Installation using package repository

1.Enable Extra Packages for Enterprise Linux (epel):

sudo rpm -U https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

2.Add the jsonar RHEL7 repository:

sudo cat <<'EOF' | sudo tee /etc/yum.repos.d/jsonar.repo
[Jsonar]
name=Jsonar Repository
baseurl=http://<your user name>:<your password>@rpm.jsonar.com/
enabled=1
priority=1
gpgcheck=0
EOF

3.Install sonarremotebackup:

sudo yum -y install sonarremotebackup

Setting up the backup system

1.Copy cert.pem file (located in: /etc/sonar/ssl/client/admin/) from Guardium Big Data Intelligence-machine to the Backup-machine (to: /etc/sonar/backup folder).

2.Chown the cert.pem on the Backup-machine:

sudo chown sonarbackup:sonar /etc/sonar/backup/cert.pem

3.Add the rsa key from the Backup-machine to the sonarw authorized_keys:

3.1. Check if the .ssh directory exists in the SONAR_HOME directory on the |product|-machine,

(default location is: /var/lib/sonarw):

    sudo ls -la /var/lib/sonarw/

3.2.If it doesn’t exist, add the ".ssh" folder:

    sudo mkdir /var/lib/sonarw/.ssh

3.3. If the "authorized_keys" file doesn’t exist, add it:

    sudo touch /var/lib/sonarw/.ssh/authorized_keys

3.4. Chown the newly created folder and file:

    sudo chown -R sonarw:sonar /var/lib/sonarw/.ssh

3.5. Copy the content of "/etc/sonar/backup/backup-key.pub" (from the Backup-machine),
into the "authorized_keys" file in the sonarw home ".ssh" folder

3.6. Test ssh from the Backup-machine to the |product|-machine:

    sudo -u sonarbackup bash

    ssh -i /etc/sonar/backup/backup-key sonarw@<|product|-ip-address>


* Don't be alarmed if it says “Failed to add the host to the list of known hosts”.

3.7. Exit twice(from the ssh and the sonarbackup user shell):

    exit
    exit

4.Edit the config file on the Backup-machine:

sudo vi /etc/sonar/backup.conf

and set the next variables:

sonar_client_uri  = mongodb://CN%%3Dadmin@<|product|-machine-ip-address>:27117/admin?authSource=%%24external&authMechanism=PLAIN&certfile=%%2Fetc%%2Fsonar%%2Fbackup%%2Fcert.pem
remote_server_uri = sonarw@<|product|-machine-ip-address>
admin_email = <system admin email>
backup_dir = /path/to/backup/folder
reports_dir =  /path/to/reports/folder/on/|product|-machine
  • By default it will backup all the databases and collections on the Guardium Big Data Intelligence-machine, in case of backing up specific databases or collections, refer to the “backup.conf” file for instructions.

5.On the Guardium Big Data Intelligence-machine, edit “sonard.conf” file:

sudo vi /etc/sonar/sonard.conf

and set the next variables:

bind_ip = 0.0.0.0
accept_connections_from = 127.0.0.1/32,<Backup-machine-ip-address>/32

6.Restart sonard service (on the Guardium Big Data Intelligence-machine):

sudo systemctl restart sonard

7.On the Backup-machine, start the remote backup service timer (will run backup daily by default):

sudo systemctl start sonarremotebackup.timer

To change backup timing:

A. Create a copy of the timer in “/etc/systemd/system”:

    cp /usr/lib/systemd/system/sonarremotebackup.timer /etc/systemd/system

B. Edit the timer and set to the desired schedule:

    vi /etc/systemd/system/sonarremotebackup.timer

C. Change the [Timer] section to the desired schedule examples:

Every 12 hours:

    OnBootSec=15min
    OnActiveSec=12h

At a specific hour:

    OnCalendar=*-*-* 12:00:00

Every 12 Hr at specific times (1 AM & 1 PM):

    OnCalendar=*-*-* 01/13:00:00

* **For additional options refer to rhel7 systemd timers documentation**

D. Run:

    sudo systemctl daemon-reload
    sudo systemctl restart sonarremotebackup.timer

8.To force one run of the backup, do:

service sonarremotebackup start
  • The backup log file can be found at: /var/log/sonar/sonar-backup.log

Setting up a local backup

It is possible to set up a backup on the same machine that is running Guardium Big Data Intelligence. Follow the above steps treating the single machine as both the Guardium Big Data Intelligence machine and the backup machine, and use the localhost ip (e.g. 127.0.0.1) as the Guardium Big Data Intelligence-ip-address. While having the backup on the same machine is not recommended in the general case (because on a total machine failure, both the backup and the main database will be lost), this functionality can provide easier integration with existing backup strategies. In particular the backup directory can (and should) be configured to be a mount point to a location that the user can replicate, store on tape for long term etc…

Upgrade from sonarBackup package to the newer sonarremotebackup

1.Upgrade Guardium Big Data Intelligence

2.Setup the new sonarremotebackup (use different backup folder then the old backup).

3.Run sonarremotebackup and check that it finished without any errors.

4.Delete the old backup directory.

Full Machine Restore from Backup

This section explains how to restore a copy of the Guardium Big Data Intelligence-machine on the Backup-machine from a full backup.

1.Install Guardium Big Data Intelligence on the Backup-machine (use [doc](https://github.com/jsonar/sonarg/blob/master/docs/sonarg_install.rst)), it is recommended to use the same Guardium Big Data Intelligence version as the one that was used on the Guardium Big Data Intelligence-machine. When running the Guardium Big Data Intelligence-setup, use the same locations as on the original Guardium Big Data Intelligence-machine.

2.Stop all the sonar services. Sonard service should be stopped last:

# sudo systemctl stop sonarfinder sonargd sonares sonarkibana sonardispatcher ...
# sudo systemctl stop sonard

3.Move the data from the desired backup version to <sonarw home>/data/ and chown it to sonarw:sonar:

# sudo rm -rf /var/lib/sonarw/data/*
# sudo mv /var/lib/sonar/backup/database_backups/<Version>/* /var/lib/sonarw/data/
# sudo chown -R sonarw:sonar /var/lib/sonarw/data/

4.Run sonard to make sure there are no data related errors:

# sudo systemctl start sonard
  • Don’t be alarmed if you see “Rebuilding unique id index” errors, sonarw needs to update the metadata for the new data.
  1. Restore each of the files & folders under the “configuration_backup” folder to their appropriate location, and set the permissions, based on the information in the “config_file_metadata” file located in the backup directory (default location: /var/lib/sonar/backup/configuration_backup). The metadata file contains the relevant metadata (file locations, owners, permissions, etc…).

For example, to set the sonarfinder service “config.properties” file:

# grep "config.properties" config_file_metadatasonarFinder
-rw-r--r-- 1 sonarfinder sonar 8.3K Jan  4 21:18 /opt/sonarfinder/sonarFinder/config.properties

# sudo cp /var/lib/sonar/backup/configuration_backup/config_files/20190228080001/config.properties
/opt/sonarfinder/sonarFinder/config.properties
cp: overwrite ‘/opt/sonarfinder/sonarFinder/config.properties’? y

# sudo chown sonarfinder:sonar config.properties

6.Move all the reports from the backup directories (note: backup is only keeping the last two report versions, in the “reports” and “reports.bak folders”), to the reports folder. Then chown them to sonarfinder:sonar

# sudo cp /var/lib/backup/configuration_backup/reports/* <reports folder on |product|-machine>
# sudo chown -R sonarfinder:sonar <reports folder on |product|-machine>

7.Restart sonard and the other sonar services:

# sudo systemctl restart sonard
# sudo systemctl start sonarfinder sonargd sonares sonarkibana ...

Note: The backup doesn’t back-up the users .ssh folders and keys(like the sonarw .ssh folder being set in the HADR or if there is .ssh folder for sonargd user), so in case of using them on the |product| original machine, recreate them on the new machine as well.