File Management

File Compression with pigz

In order to save space, we recommend that users compress their files before transferring them into their group's /project file system.

The gzip command can be used to compress files, but it uses a single thread on a single core.

The pigz command is a parallel implementation of gzip that can run with multiple threads, making use of multiple cores. The unpigz command is equivalent to gunzip and it can be used to uncompress gzip'ed files. The .gz files created by pigz is compatible with gzip/gunzip.

The pigz command is particularly helpful to compress a large number of files (or a folder) or to compress large files.

The compute nodes on Sapelo2 and on the teaching cluster have pigz installed centrally, so you don't need to load any modules in order to use this command. The help page for this command shows the available options, and it can be viewed with the command

pigz --help

Some simple examples

Compress a file

pigz filename

Compress a file with best compression rate

pigz -9 filename
pigz --best filename

Uncompress a file

unpigz filename.gz

We suggest that you run pigz on an interactive session that request multiple cores and run pigz with the '-p num_thread' option to specify the numnber of threads (num_threads) to use.

For example, start an interactive session with 10 cores and 4GB of RAM with

interact -c 10 --mem=4g 

and then run pigz with 10 threads with

pigz -9 -p 10 my_big_file

To recursive compress all files in a directory (e.g. called dirname) use the -r option. For example, using 10 threads

pigz -9 -r -p 10 dirname

To recursive uncompress all files in a directory:

unpigz -r dirname