Transferring Files

The File Transfer nodes (xfer.gacrc.uga.edu)

Users can transfer files between their local machines and GACRC storage using various programs, such as (a) Filezilla, (b) WinSCP, and (c) a secure copy (scp). To transfer files, you must have a file transfer program installed on your local machine and a connection to the UGA campus network.

All Sapelo2 users, as well as all GACRC PIs, can access the xfer nodes using the hostname xfer.gacrc.uga.edu and using their UGA MyID (not the 810 or 811 number) and the MyID password to authenticate. Two-factor authentication using Archpass Duo is also necessary. For more details of Archpass Duo, please refer to archpass_duo.

The GACRC file transfer nodes (xfer) are configured to facilitate file transfer. These nodes, which have a hostname of xfer.gacrc.uga.edu, are connected via a higher-bandwidth network connection. Hence, maximum transfer speed between a user's local machine and the GACRC systems can be achieved by transferring files to/from the host xfer.gacrc.uga.edu (instead of the login nodes of the clusters).

It is not necessary to connect to the UGA VPN when connecting to a file transfer node from off-campus. If you are transferring data from/to a server outside of campus (such as your local computer), you will get much faster transfer speeds if you are not connected to the VPN.

Note that a user's home directory on the xfer nodes is the same as the user's Sapelo2 home directory, which is not the same as the user's home directory on the teaching cluster.

The xfer node can access the following file systems using the full path:

1. Your home directory on an xfer node is the same as your Sapelo2 home directory, and the path is /home/username

2. The Sapelo2 scratch directory: /scratch/username

3. The Sapelo2 work directory: /work/groupname

4. The project file system for the lab: /project/groupname

Transferring Files from your local storage device to Sapelo2 or vice-versa

To transfer files from your local storage device to Sapelo2 or from Sapelo2 to your local storage device, please connect to xfer.gacrc.uga.edu (use your UGA MyID, the MyID password to authenticate). You will also be prompted for two-factor authentication with Archpass Duo.

Using FileZilla

In order to use FileZilla to transfer files between a local machine and Sapelo2, users will need to install the FileZilla software on the local machine. After installation, start FileZilla and follow these steps:

1. Go to File -> Site Manager

2. Add a New Site with the following configuration.

In the General tab select

Protocol: SFTP - SSH File Transfer Protocol

Host: sftp://xfer.gacrc.uga.edu

Port: 22

Logon Type: Interactive

User:  [your UGA MyID]

In the Transfer Settings tab, check the box for "Limit number of simultaneous connections" and set "Maximum number of connections" to 1.

If this step is omitted, then you will need to enter your password and Duo authentication for each file that you upload or download. By limiting the number of simultaneous connections to 1, you only need to enter your password and Duo authentication once per session.

With the above settings saved, you should be able to transfer files to txfer using filezilla. When you open the connection, you will be prompted for your MyID password (unless you use key based ssh). If the password authenticates successfully, then another pop-up window will prompt for Duo. Note that the Duo window looks very similar to the password window, but the text in the box will describe the options you can use for Duo (for example, a push, enter a passcode, etc).

Once the connection is established you can upload files from your local machine to Sapelo2 or download files from Sapelo2 to your local machine.

Here is the PDF to check step-by-step screen shots:

Media:Xfer_Filezilla_12032018.pdf

Using WinSCP

In order to use WinSCP to transfer files between a local Windows machine and Sapelo2, users will need to install the WinSCP software on the local machine. After installation, when you click the WinSCP shortcut button, Login window pops up, in that you select new site and enter the following information to start the new session:

Session:

File Protocol: SFTP

Host name: xfer.gacrc.uga.edu

Port number: 22

Username: [your UGA MyID]

Password: [your MyID password]

If the password authenticates successfully, then another pop-up window will prompt for Duo two-factor authentication. It says, Using Keyboard-interactive authentication and lists the Duo options you can choose from: Duo push, Phone Call, or SMS passcode, etc. Once the connection is established you can upload or download files from Local Machine to Cluster or vice-versa. Note: Under Advanced, keep the "Default" option for Transfer settings Rule. With this default option, you will be able to transfer multiple files or folders during the session.

Using scp

To transfer files using scp you must have scp on your local machine and a connection to the UGA campus network. An scp software is included in recent releases of Unix based operating systems (including Linux and Mac OS X).

Sample commands to transfer files from your local Unix/Linux/Mac OS X machine to Sapelo2:

Open a terminal in your local machine, change directory (cd) to where the files are located in your local machine, and at the command prompt type:

scp filename  yourUGAMyID@xfer.gacrc.uga.edu:subdirectory

where filename is the name of the file to be transferred, yourUGAMyID is your UGA MyID, and subdirectory is the subdirectory in your Sapelo2 home directory to which files are being transferred. You will then be asked to enter your UGA MyID password. You will also be prompted for two-factor authentication with Archpass Duo.

To transfer files from Sapelo2 to your local machine, use the full path to your file on Sapelo2 followed by the location you would like to transfer the files locally. The following example transfers a file called "filename" to the local current working directory.

scp yourUGAMyID@xfer.gacrc.uga.edu:full/path/to/filename ./

(Note: yourUGAMyID@ can be omitted if your username on Sapelo2 is the same as on your local machine). Multiple files (e.g. file1, file2, and file3) can be transferred with a single command:

scp file1 file2 file3  yourUGAMyID@xfer.gacrc.uga.edu:subdirectory

Wildcards can be used for multiple file transfer, for example, to upload all files with .dat extension to your sapelo2 home directory, to a subdirectory called subdir:

scp *.dat yourUGAMyID@xfer.gacrc.uga.edu:subdir

An example that uses a wildcard to download all files with .dat extension in your Sapelo2 home dir, in a subdirectory called subdir to your local machine:

scp yourUGAMyID@xfer.gacrc.uga.edu:subdir/\*.dat   ./

(Note the backslash “ \ ” preceeding * in the last example.)

A directory and all its contents can be transferred using the scp option -r, for recursive file transferring. For example, to transfer a directory on your local machine called programs and all files in it to your Sapelo2 home directory, use:

scp -r programs yourUGAMyID@xfer.gacrc.uga.edu:

You can also use the scp option -p to preserve the file features, such as last modification time, file permissions, etc.

To transfer files directly to your Sapelo2 /scratch area (such as /scratch/johndoe/):

scp filename  yourUGAMyID@xfer.gacrc.uga.edu:/scratch/johndoe/

To transfer files directly to your Sapelo2 work directory:

scp filename  yourUGAMyID@xfer.gacrc.uga.edu:/work/groupname/subdirname

To transfer files directly to your lab's project file system (such as /project/groupname):

scp filename  yourUGAMyID@xfer.gacrc.uga.edu:/project/groupname

Download directly from internet to Sapelo2

Here is an example on how to download FastaX software from internet directly to Sapelo2

wget https://github.com/agordon/fastx_toolkit/releases/download/0.0.14/fastx_toolkit-0.0.14.tar.bz2

Some websites use indirect connections, it could be downloaded as

curl -OL https://github.com/agordon/fastx_toolkit/releases/download/0.0.14/fastx_toolkit-0.0.14.tar.bz2

Transferring Files between two file systems on Sapelo2

Use ssh to login to xfer.gacrc.uga.edu using your UGA MyID username and password (and two-factor authentication via Archpass Duo) to transfer files between different file systems on Sapelo2, including the project area. To transfer data between two file systems that are available on the xfer node, you can use the cp or the rsync commands. If you have many files to copy, then a good option is to use the fpsync command, as that will allow the use of multiple cores.

For example, if you want to use 4 cores to transfer all files from /scratch/myid/mydata to /project/mylab/myid/mydata, you can use the following command on an xfer node shell prompt:

fpsync -n 4 -t $HOME/fpsync /scratch/myid/mydata /project/mylab/myid/mydata

When using the fpysnc command, please use the -t $HOME/fpsync option to set the fpsync's temporary directory to be in your home directory.

The File Transfer node for the teaching cluster (txfer.gacrc.uga.edu)

Users can transfer files between their local machines and the teaching cluster using (a) Filezilla, (b) a secure copy (scp), or (c) WinSCP. To transfer files using scp (or SSH file transfer) you must have scp (or SSH) on your local machine and a connection to the UGA campus network. An scp software is included in recent releases of Unix based operating systems (including Linux and Mac OS X).

Access to txfer.gacrc.uga.edu requires an active MyID and password, and two-factor authentication with Archpass Duo. For more details of Archpass Duo, please refer to archpass_duo .

Transferring Files from your local storage device to the teaching cluster

To transfer files from your local storage device to the teaching cluster, please connect to txfer.gacrc.uga.edu (use your UGA MyID and the MyID password to authenticate). You will also be prompted for two-factor authentication with Archpass Duo.

Using FileZilla

In order to use FileZilla to transfer files between a local machine and the teaching cluster, users will need to install the FileZilla software on the local machine. After installation, start FileZilla and follow these steps:

1. Go to File -> Site Manager

2. Add a New Site with the following configuration.

In the General tab select

Protocol: SFTP - SSH File Transfer Protocol

Host: sftp://txfer.gacrc.uga.edu

Port: 22

Logon Type: Interactive

User:  [your UGA MyID]

In the Transfer Settings tab, check the box for "Limit number of simultaneous connections" and set "Maximum number of connections" to 1.

If this step is omitted, then you will need to enter your password and Duo authentication for each file that you upload or download. By limiting the number of simultaneous connections to 1, you only need to enter your password and Duo authentication once per session.

With the above settings saved, you should be able to transfer files to txfer using filezilla. When you open the connection, you will be prompted for your MyID password (unless you use key based ssh). If the password authenticates successfully, then another pop-up window will prompt for Duo. Note that the Duo window looks very similar to the password window, but the text in the box will describe the options you can use for Duo (for example, a push, enter a passcode, etc).

Using WinSCP

In order to use WinSCP to transfer files between a local Windows machine and the teaching cluster, users will need to install the WinSCP software on the local machine. After installation, when you click the WinSCP shortcut button, Login window pops up, in that you select new site and enter the following information to start the new session:

Session:

File Protocol: SFTP

Host name: txfer.gacrc.uga.edu

Port number: 22

Username: [your UGA MyID]

Password: [your MyID password]

If the password authenticates successfully, then another pop-up window will prompt for Duo two-factor authentication. It says, Using Keyboard-interactive authentication and lists the Duo options you can choose from: Duo push, Phone Call, or SMS passcode, etc. Once the connection is established you can upload or download files from Local Machine to Cluster or vice-versa. Note: Under Advanced, kept the "Default" option for Transfer settings Rule. With this default option, you will be able to transfer multiple files or folders during the session.

Using scp

Sample commands to transfer files from your local Unix/Linux/Mac OS X machine to the teaching cluster:

Open a terminal in your local machine, change directory (cd) to where the files are located in your local machine, and at the command prompt type:

scp filename  yourUGAMyID@txfer.gacrc.uga.edu:subdirectory

where filename is the name of the file to be transferred, yourUGAMyID is your UGA MyID, and subdirectory is the subdirectory in your teaching cluster home directory to which files are being transferred. You will then be asked to enter your UGA MyID password.

(Note: yourUGAMyID@ can be omitted if your username on the teaching cluster is the same as on your local machine). Multiple files (e.g. file1, file2, and file3) can be transferred with a single command:

scp file1 file2 file3  yourUGAMyID@txfer.gacrc.uga.edu:subdirectory

Wildcards can be used for multiple file transfer (e.g. all files with .dat extension):

scp *.dat yourUGAMyID@txfer.gacrc.uga.edu:subdirectory

Transferring Files between your Sapelo2 directories and the teaching cluster

To transfer files between your Sapelo2 directories (e.g. your Sapelo2 home directory or /scratch directory) and your teaching cluster home directory, please first use ssh to login into xfer.gacrc.uga.edu. Then use the scp command to copy files to/from your home directory at txfer.gacrc.uga.edu.

For example, to transfer a directory called subdir2 in your /scratch directory to your teaching cluster home directory. On xfer.gacrc.uga.edu issue the command:

scp -r -p /scratch/yourUGAMyID/subdir2 txfer.gacrc.uga.edu:

UGA DUO authentication is required for SSH/SCP access to
GACRC systems.

UGA DUO is a two-factor authentication service which
requires a password (one factor) and a code, phone,
or device (second factor) to successfully authenticate.

If you are not enrolled in the UGA DUO service please 
visit the UGA DUO service self-service portal to enroll
and configure or manage your DUO enabled devices.

https://eits.uga.edu/access_and_security/infosec/tools/duo/portal/

For additional help with UGA DUO authentication or to 
report an issue please visit:

https://eits.uga.edu/access_and_security/infosec/tools/archpass/

Password: 


Duo two-factor login for yourUGAMyID

Enter a passcode or select one of the following options:

 1. Duo Push to XXX-XXX-4304
 2. Phone call to XXX-XXX-4304
 3. SMS passcodes to XXX-XXX-4304

Passcode or option (1-3): 

When you enter your password, you will be prompted for the two-factor authentication step. Once you enter a passcode or select an option, the file transfer will start.

Transferring Files between your project area and the teaching cluster

To transfer files between your /project area and your teaching cluster home directory, please first use ssh to login into xfer.gacrc.uga.edu. Then use the scp command to e.g. copy files from /project to your home directory at txfer.gacrc.uga.edu (or to copy files from your home directory at txfer.gacrc.uga.edu to your /project area).

For example, to copy a file called species.fa from your teaching cluster home directory to a subdirectory called binf1234 in your /project area, use the following command on xfer.gacrc.uga.edu:

scp txfer.gacrc.uga.edu:species.fa /project/abclab/binf1234

where abclab needs to be changed to the name of your group.

Globus

Globus is a high-performance data-transfer platform that allows you to perform and/or automate data transfers. These transfers can be to and from other institutions, your local computer, or your GACRC /project area. Collaborators can also share data with you by sharing a Collection or an "endpoint" with you. The shared Collection/Endpoint can be on another institution, on a desktop or laptop, or on their GACRC storage. Data transfers happen unattended and are faster than SCP/SFTP, data verification is on by default, and automatic restarts or continuation of transfers happen after a disruption.

To use Globus you will need a Globus Identity. At a minimum you will need to setup your identity using the University of Georgia organizational login in order to access UGA systems. More information about how to get started using Globus can be found on the Globus page. You can also get more information about Globus from the Official Globus Documentation

Using iCommands to access the Cyverse Data Store

Cyverse is a secure data storage and management service. iCommands is a collection of commands for Linux and Mac OS operating systems that are used to interact with the CyVerse Data Store. iCommands can be used by CyVerse account users to download files that have been shared by other users and to upload files to the Data Store, as well as add metadata, change permissions, and more. A CyVerse account is not required to download a public data file via iCommands.

version 4.2.8 of iCommands is installed on Sapelo2 IRODS transfer node (xfer-irods.gacrc.uga.edu). Please note that this command is not available on the other transfer nodes.

You will need to log on a Sapelo2 transfer node xfer-irods.gacrc.uga.edu and then initialize the connection to iRODS by using the iinit command to use iCommands

More information about using iCommands can be found on the ICommands-Sapelo2 page. You can also read Official iCommands Documentation