Transferring Files
The File Transfer nodes (xfer.gacrc.uga.edu)
Users can transfer files between their local machines and GACRC storage using various programs, such as (a) Filezilla, (b) WinSCP, and (c) a secure copy (scp). To transfer files, you must have a file transfer program installed on your local machine and a connection to the UGA campus network.
All Sapelo2 users, as well as all GACRC PIs, can access the xfer nodes using the hostname xfer.gacrc.uga.edu and using their UGA MyID (not the 810 or 811 number) and the MyID password to authenticate. Two-factor authentication using Archpass Duo is also necessary. For more details of Archpass Duo, please refer to archpass_duo.
The GACRC file transfer nodes (xfer) are configured to facilitate file transfer. These nodes, which have a hostname of xfer.gacrc.uga.edu, are connected via a higher-bandwidth network connection. Hence, maximum transfer speed between a user's local machine and the GACRC systems can be achieved by transferring files to/from the host xfer.gacrc.uga.edu (instead of the login nodes of the clusters).
It is not necessary to connect to the UGA VPN when connecting to a file transfer node from off-campus. If you are transferring data from/to a server outside of campus (such as downloading data from NCBI into the cluster), you will get much faster transfer speeds if you are not connected to the VPN.
Note that a user's home directory on the xfer nodes is the same as the user's Sapelo2 home directory, which is not the same as the user's home directory on the teaching cluster.
The xfer node can access the following file systems using the full path:
1. Your home directory on an xfer node is the same as your Sapelo2 home directory, and the path is /home/username
2. The Sapelo2 scratch directory: /scratch/username
3. The Sapelo2 work directory: /work/groupname
4. The project file system for the lab: /project/groupname
Transferring Files from your local storage device to Sapelo2 or vice-versa
To transfer files from your local storage device to Sapelo2 or from Sapelo2 to your local storage device, please connect to xfer.gacrc.uga.edu (use your UGA MyID, the MyID password to authenticate). You will also be prompted for two-factor authentication with Archpass Duo.
Using FileZilla
In order to use FileZilla to transfer files between a local machine and Sapelo2, users will need to install the FileZilla software on the local machine. After installation, start FileZilla and follow these steps:
1. Go to File -> Site Manager
2. Add a New Site with the following configuration.
In the General tab select
Protocol: SFTP - SSH File Transfer Protocol Host: sftp://xfer.gacrc.uga.edu Port: 22 Logon Type: Interactive User: [your UGA MyID]
In the Transfer Settings tab, check the box for "Limit number of simultaneous connections" and set "Maximum number of connections" to 1.
If this step is omitted, then you will need to enter your password and Duo authentication for each file that you upload or download. By limiting the number of simultaneous connections to 1, you only need to enter your password and Duo authentication once per session.
With the above settings saved, you should be able to transfer files to txfer using filezilla. When you open the connection, you will be prompted for your MyID password (unless you use key based ssh). If the password authenticates successfully, then another pop-up window will prompt for Duo. Note that the Duo window looks very similar to the password window, but the text in the box will describe the options you can use for Duo (for example, a push, enter a passcode, etc).
Once the connection is established you can upload files from your local machine to Sapelo2 or download files from Sapelo2 to your local machine.
Here is the PDF to check step-by-step screen shots:
| media:Xfer Filezilla 12032018.pdf |
Using WinSCP
In order to use WinSCP to transfer files between a local Windows machine and Sapelo2, users will need to install the WinSCP software on the local machine. After installation, when you click the WinSCP shortcut button, Login window pops up, in that you select new site and enter the following information to start the new session:
Session: File Protocol: SFTP Host name: xfer.gacrc.uga.edu Port number: 22 Username: [your UGA MyID] Password: [your MyID password]
If the password authenticates successfully, then another pop-up window will prompt for Duo two-factor authentication. It says, Using Keyboard-interactive authentication and lists the Duo options you can choose from: Duo push, Phone Call, or SMS passcode, etc. Once the connection is established you can upload or download files from Local Machine to Cluster or vice-versa. Note: Under Advanced, kept the "Default" option for Transfer settings Rule. With this default option, you will be able to transfer multiple files or folders during the session.
Using scp
To transfer files using scp you must have scp on your local machine and a connection to the UGA campus network. An scp software is included in recent releases of Unix based operating systems (including Linux and Mac OS X).
Sample commands to transfer files from your local Unix/Linux/Mac OS X machine to Sapelo2:
Open a terminal in your local machine, change directory (cd) to where the files are located in your local machine, and at the command prompt type:
scp filename yourUGAMyID@xfer.gacrc.uga.edu:subdirectory
where filename is the name of the file to be transferred, yourUGAMyID is your UGA MyID, and subdirectory is the subdirectory in your Sapelo2 home directory to which files are being transferred. You will then be asked to enter your UGA MyID password. You will also be prompted for two-factor authentication with Archpass Duo.
(Note: yourUGAMyID@ can be omitted if your username on Sapelo2 is the same as on your local machine). Multiple files (e.g. file1, file2, and file3) can be transferred with a single command:
scp file1 file2 file3 yourUGAMyID@xfer.gacrc.uga.edu:subdirectory
Wildcards can be used for multiple file transfer, for example, to upload all files with .dat extension to your sapelo2 home directory, to a subdirectory called subdir:
scp *.dat yourUGAMyID@xfer.gacrc.uga.edu:subdir
An example that uses a wildcard to download all files with .dat extension in your Sapelo2 home dir, in a subdirectory called subdir to your local machine:
scp yourUGAMyID@xfer.gacrc.uga.edu:subdir/\*.dat ./
(Note the backslash “ \ ” preceeding * in the last example.)
A directory and all its contents can be transferred using the scp option -r, for recursive file transferring. For example, to transfer a directory on your local machine called programs and all files in it to your Sapelo2 home directory, use:
scp -r programs yourUGAMyID@xfer.gacrc.uga.edu:
You can also use the scp option -p to preserve the file features, such as last modification time, file permissions, etc.
To transfer files directly to your Sapelo2 /scratch area (such as /scratch/johndoe/):
scp filename yourUGAMyID@xfer.gacrc.uga.edu:/scratch/johndoe/
To transfer files directly to your Sapelo2 work directory:
scp filename yourUGAMyID@xfer.gacrc.uga.edu:/work/groupname/subdirname
To transfer files directly to your lab's project file system (such as /project/groupname):
scp filename yourUGAMyID@xfer.gacrc.uga.edu:/project/groupname
Download directly from internet to Sapelo2
Here is an example on how to download FastaX software from internet directly to Sapelo2
wget https://github.com/agordon/fastx_toolkit/releases/download/0.0.14/fastx_toolkit-0.0.14.tar.bz2
Some websites use indirect connections, it could be downloaded as
curl -OL https://github.com/agordon/fastx_toolkit/releases/download/0.0.14/fastx_toolkit-0.0.14.tar.bz2
Transferring Files from the decommissioned zcluster to Sapelo2
Use ssh to login to xfer.gacrc.uga.edu using your UGA MyID username and MyID password. You will also be prompted for two-factor authentication with Archpass Duo. This file transfer node has the following file systems mounted:
- zcluster /escratch4 file system. To access it, use e.g.
cd /escratch4/username
- zcluster home file systems. To access your zcluster home directory, use the following
cd /panfs/pstor.storage/home/groupname/username
For example, if your group is named abclab and your username is jsmith use
cd /panfs/pstor.storage/home/abclab/jsmith
- Sapelo2 home file systems. To access your Sapelo2 home directory, use the following
cd /home/username
- Sapelo2 scratch file system on the Lustre file system. To access your /scratch space use the following
cd /scratch/username
- Sapelo2 work file system on the Lustre file system. To access your group's /work space use the following
cd /work/abclab
- /project file system. To access your group's /project space use the following (please note that this file system is mounted when it is first accessed, so you need to access it with the full path):
cd /project/abclab
Because the xfer node mounts all these file systems, users can ssh into an xfer node and use cp or rsync to copy the files between any two of these file systems. Users can also copy files from their zcluster home dirs or /escratch4 to their project file system (/project/groupname).
Transferring Files between two file systems on Sapelo2
Use ssh to login to xfer.gacrc.uga.edu using your UGA MyID username and password (and two-factor authentication via Archpass Duo) to transfer files between different file systems on Sapelo2, including the project area.
The File Transfer node for the teaching cluster (txfer.gacrc.uga.edu)
Users can transfer files between their local machines and the teaching cluster using (a) Filezilla, (b) a secure copy (scp), or (c) an SSH secure File Transfer software (SSH file transfer). To transfer files using scp (or SSH file transfer) you must have scp (or SSH) on your local machine and a connection to the UGA campus network. An scp software is included in recent releases of Unix based operating systems (including Linux and Mac OS X). An SSH Secure File Transfer Utility is available for Windows on the UGA Sitesoft page (you will need to enter your UGA myID and password to download the software).
Access to txfer.gacrc.uga.edu requires an active MyID and password, and two-factor authentication with Archpass Duo. For more details of Archpass Duo, please refer to archpass_duo .
Transferring Files from your local storage device to the teaching cluster
To transfer files from your local storage device to the teaching cluster, please connect to txfer.gacrc.uga.edu (use your UGA MyID and the MyID password to authenticate). You will also be prompted for two-factor authentication with Archpass Duo.
Using FileZilla
In order to use FileZilla to transfer files between a local machine and the teaching cluster, users will need to install the FileZilla software on the local machine. After installation, start FileZilla and follow these steps:
1. Go to File -> Site Manager
2. Add a New Site with the following configuration.
In the General tab select
Protocol: SFTP - SSH File Transfer Protocol Host: sftp://txfer.gacrc.uga.edu Port: 22 Logon Type: Interactive User: [your UGA MyID]
In the Transfer Settings tab, check the box for "Limit number of simultaneous connections" and set "Maximum number of connections" to 1.
If this step is omitted, then you will need to enter your password and Duo authentication for each file that you upload or download. By limiting the number of simultaneous connections to 1, you only need to enter your password and Duo authentication once per session.
With the above settings saved, you should be able to transfer files to txfer using filezilla. When you open the connection, you will be prompted for your MyID password (unless you use key based ssh). If the password authenticates successfully, then another pop-up window will prompt for Duo. Note that the Duo window looks very similar to the password window, but the text in the box will describe the options you can use for Duo (for example, a push, enter a passcode, etc).
Using WinSCP
In order to use WinSCP to transfer files between a local Windows machine and the teaching cluster, users will need to install the WinSCP software on the local machine. After installation, when you click the WinSCP shortcut button, Login window pops up, in that you select new site and enter the following information to start the new session:
Session: File Protocol: SFTP Host name: txfer.gacrc.uga.edu Port number: 22 Username: [your UGA MyID] Password: [your MyID password]
If the password authenticates successfully, then another pop-up window will prompt for Duo two-factor authentication. It says, Using Keyboard-interactive authentication and lists the Duo options you can choose from: Duo push, Phone Call, or SMS passcode, etc. Once the connection is established you can upload or download files from Local Machine to Cluster or vice-versa. Note: Under Advanced, kept the "Default" option for Transfer settings Rule. With this default option, you will be able to transfer multiple files or folders during the session.
Using SSH Secure File Transfer
In order to use the SSH Secure File Transfer software that is installed as part of SSH (Secure Shell Utilities for Windows) available on the UGA Sitesoft page, users need to start the Secure File Transfer application and open a connection to host txfer.gacrc.uga.edu. You will have to enter your UGA MyID as the username, but other fields (port number, etc) can be left empty (or leave the default value, if any). You will be prompted for your MyID password and after that you will be prompted to enter DUO two factor authentication code. Please choose one of the options like DUO push, Phone call or SMS passcodes. Once the connection is established, you can upload or download files.
Here is the PDF to check step-by-step screen shots:
| media: Txfer SSH File Transfer Nov302018.pdf |
Using scp
Sample commands to transfer files from your local Unix/Linux/Mac OS X machine to the teaching cluster:
Open a terminal in your local machine, change directory (cd) to where the files are located in your local machine, and at the command prompt type:
scp filename yourUGAMyID@txfer.gacrc.uga.edu:subdirectory
where filename is the name of the file to be transferred, yourUGAMyID is your UGA MyID, and subdirectory is the subdirectory in your teaching cluster home directory to which files are being transferred. You will then be asked to enter your UGA MyID password.
(Note: username@ can be omitted if your username on the teaching cluster is the same as on your local machine). Multiple files (e.g. file1, file2, and file3) can be transferred with a single command:
scp file1 file2 file3 yourUGAMyID@txfer.gacrc.uga.edu:subdirectory
Wildcards can be used for multiple file transfer (e.g. all files with .dat extension):
scp *.dat yourUGAMyID@txfer.gacrc.uga.edu:subdirectory
Transferring Files between your Sapelo2 directories and the teaching cluster
To transfer files between your Sapelo2 directories (e.g. your Sapelo2 home directory or /lustre1 directory) and your teaching cluster home directory, please first use ssh to login into xfer.gacrc.uga.edu. Then use the scp command to copy files to/from your home directory at txfer.gacrc.uga.edu.
For example, to transfer a directory called subdir2 in your /lustre1 directory to your teaching cluster home directory. On xfer.gacrc.uga.edu issue the command:
scp -r -p /lustre1/yourUGAMyID/subdir2 yourUGAMyID@txfer.gacrc.uga.edu: UGA DUO authentication is required for SSH/SCP access to GACRC systems. UGA DUO is a two-factor authentication service which requires a password (one factor) and a code, phone, or device (second factor) to successfully authenticate. If you are not enrolled in the UGA DUO service please visit the UGA DUO service self-service portal to enroll and configure or manage your DUO enabled devices. https://eits.uga.edu/access_and_security/infosec/tools/duo/portal/ For additional help with UGA DUO authentication or to report an issue please visit: https://eits.uga.edu/access_and_security/infosec/tools/archpass/ Password: Duo two-factor login for yourUGAMyID Enter a passcode or select one of the following options: 1. Duo Push to XXX-XXX-4304 2. Phone call to XXX-XXX-4304 3. SMS passcodes to XXX-XXX-4304 Passcode or option (1-3):
When you enter your password, you will be prompted for the two-factor authentication step. Once you enter a passcode or select an option, the file transfer will start.
Transferring Files between your project area and the teaching cluster
To transfer files between your /project area and your teaching cluster home directory, please first use ssh to login into xfer.gacrc.uga.edu. Then use the scp command to e.g. copy files from /project to your home directory at txfer.gacrc.uga.edu (or to copy files from your home directory at txfer.gacrc.uga.edu to your /project area).
For example, to copy a file called species.fa from your teaching cluster home directory to a subdirectory called binf1234 in your /project area, use the following command on xfer.gacrc.uga.edu:
scp txfer.gacrc.uga.edu:species.fa /project/abclab/binf1234
where abclab needs to be changed to the name of your group.