Borg backup - Revision history

Gcormier: /* Initializing a borg repository */

2018-09-17T19:26:13Z

Initializing a borg repository

← Older revision		Revision as of 15:26, 17 September 2018
Line 8:		Line 8:
	Borg uses a repository (a special directory in your filesystem) to store the backup data. You will have to initialize a repository before writing backup data to it.		Borg uses a repository (a special directory in your filesystem) to store the backup data. You will have to initialize a repository before writing backup data to it.
	You can create the repository in any filesystem. In this example, I am storing the repository in my project filesystem.		You can create the repository in any filesystem. In this example, I am storing the repository in my project filesystem.
			'''Please note that you will need to be logged on the xfer node to issue these commands.'''
	Run the following command to initialize the repository.		Run the following command to initialize the repository.
	<pre>		<pre>

Gcormier: /* Accessing the data in the backup archives */

2018-09-17T19:21:38Z

Accessing the data in the backup archives

← Older revision		Revision as of 15:21, 17 September 2018
Line 52:		Line 52:
	</pre>		</pre>
	=== Accessing the data in the backup archives ===		=== Accessing the data in the backup archives ===
	One of the nice ~~feature~~ of borg backup is the ability to mount a backup archive as a FUSE filesystem and access the files in the backup as if they are in a real directory tree.		One of the nice features of borg backup is the ability to mount a backup archive as a FUSE filesystem and access the files in the backup as if they are in a real directory tree.
	Create a mount point in your home directory and mount the borg backup repo using the following commands.		Create a mount point in your home directory and mount the borg backup repo using the following commands.
	<pre>		<pre>

Gcormier: /* Initializing a borg repository */

2018-09-17T19:19:05Z

Initializing a borg repository

← Older revision		Revision as of 15:19, 17 September 2018
Line 7:		Line 7:
	=== Initializing a borg repository ===		=== Initializing a borg repository ===
	Borg uses a repository (a special directory in your filesystem) to store the backup data. You will have to initialize a repository before writing backup data to it.		Borg uses a repository (a special directory in your filesystem) to store the backup data. You will have to initialize a repository before writing backup data to it.
	You can create the repository in any filesystem. In this example I am storing the repository in my project filesystem.		You can create the repository in any filesystem. In this example, I am storing the repository in my project filesystem.
	Run the following command to initialize the repository.		Run the following command to initialize the repository.
	<pre>		<pre>
	$ borg init --encryption none /project/gclab/raj76/my_project		$ borg init --encryption none /project/gclab/raj76/my_project
	</pre>		</pre>
	If successful the command will not return/print anything.		If successful, the command will not return/print anything.

	=== Creating a backup ===		=== Creating a backup ===
	The next step is to create a backup to the repository.		The next step is to create a backup to the repository.

Gcormier: /* Example showing archiving lustre project directory */

2018-09-17T19:17:53Z

Example showing archiving lustre project directory

← Older revision		Revision as of 15:17, 17 September 2018
Line 4:		Line 4:
	The borg backup software is installed on all xfer nodes. You can use this software to archive your files. If you have a large amount of data this could possibly reduce storage by deduplicating and compressing your data. This is especially true for large genomic datasets.		The borg backup software is installed on all xfer nodes. You can use this software to archive your files. If you have a large amount of data this could possibly reduce storage by deduplicating and compressing your data. This is especially true for large genomic datasets.

	== Example showing archiving ~~lustre~~ project directory ==		== Example showing archiving Lustre data into your lab's project directory ==
	=== Initializing a borg repository ===		=== Initializing a borg repository ===
	Borg uses a repository (a special directory in your filesystem) to store the backup data. You will have to initialize a repository before writing backup data to it.		Borg uses a repository (a special directory in your filesystem) to store the backup data. You will have to initialize a repository before writing backup data to it.

Raj76 at 19:04, 17 September 2018

2018-09-17T19:04:26Z

← Older revision		Revision as of 15:04, 17 September 2018
Line 68:		Line 68:
	$ borg umount /home/raj76/backup_mount		$ borg umount /home/raj76/backup_mount
	</pre>		</pre>

			=== More information ===
			Please refer to the Borg backup manual for more information and detailed explanation of the available options.
			https://borgbackup.readthedocs.io/en/stable/index.html

Raj76 at 19:01, 17 September 2018

2018-09-17T19:01:58Z

← Older revision		Revision as of 15:01, 17 September 2018
Line 63:		Line 63:
	You can browse the archive directories and access the files in the backup and if need be copy the files to an area outside of the backup.		You can browse the archive directories and access the files in the backup and if need be copy the files to an area outside of the backup.
	The borg backup mount is a read-only mount and files and directories cannot be changed or deleted from the archive.		The borg backup mount is a read-only mount and files and directories cannot be changed or deleted from the archive.
			You can unmount the backup using the following command.
			<pre>
			$ cd
			$ borg umount /home/raj76/backup_mount
			</pre>

Raj76 at 19:00, 17 September 2018

2018-09-17T19:00:38Z

@@ Line 13: / Line 13: @@
 </pre>
 If successful the command will not return/print anything.
+=== Creating a backup ===
 The next step is to create a backup to the repository.
 Run the following command to create a backup with deduplication and fast compression.
@@ Line 21: / Line 21: @@
 In the above command ::lustre1-{now} is the name of the archive that will be created by this backup. The {now} shorthand tells borg backup to use the current time stamp as part of the archive name. This will be useful later to identify archives.
 The above command should be run in a screen or tmux session as this will take a while for large datasets.

Raj76: Created page with "= Introduction = The [https://borgbackup.readthedocs.io/en/stable/index.html Borg backup software] is a deduplicating backup program. It supports compression using various cod..."

2018-09-17T18:41:58Z

Created page with "= Introduction = The [https://borgbackup.readthedocs.io/en/stable/index.html Borg backup software] is a deduplicating backup program. It supports compression using various cod..."

New page

= Introduction =
The [https://borgbackup.readthedocs.io/en/stable/index.html Borg backup software] is a deduplicating backup program. It supports compression using various codecs as well.

The borg backup software is installed on all xfer nodes. You can use this software to archive your files. If you have a large amount of data this could possibly reduce storage by deduplicating and compressing your data. This is especially true for large genomic datasets.

== Example showing archiving lustre project directory ==
=== Initializing a borg repository ===
Borg uses a repository (a special directory in your filesystem) to store the backup data. You will have to initialize a repository before writing backup data to it.
You can create the repository in any filesystem. In this example I am storing the repository in my project filesystem.
Run the following command to initialize the repository.
<pre>
$ borg init --encryption none /project/gclab/raj76/my_project
</pre>
If successful the command will not return/print anything.

The next step is to create a backup to the repository.
Run the following command to create a backup with deduplication and fast compression.
<pre>
$ borg create -s --compression auto,lz4 /project/gclab/raj76/my_project::lustre1-{now} /lustre1/raj76/my_project
</pre>
In the above command ::lustre1-{now} is the name of the archive that will be created by this backup. The {now} shorthand tells borg backup to use the current time stamp as part of the archive name. This will be useful later to identify archives.
The above command should be run in a screen or tmux session as this will take a while for large datasets.