Transferring Files

From Research Computing Center Wiki
Revision as of 15:39, 3 December 2018 by Pakala (talk | contribs) (→‎Using FileZilla)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search


The File Transfer nodes (xfer.gacrc.uga.edu)

Users can transfer files between their local machines and GACRC storage using various programs, such as (a) Filezilla, (b) WinSCP, (c) a secure copy (scp), and (d) an SSH secure File Transfer software (SSH file transfer). To transfer files, you must have a file transfer program installed on your local machine and a connection to the UGA campus network.

All Sapelo2 users, as well as all GACRC PIs, can access the xfer nodes using the hostname xfer.gacrc.uga.edu and using their UGA MyID (not the 810 or 811 number) and the MyID password to authenticate. Two-factor authentication using Archpass Duo is also necessary. For more details of Archpass Duo, please refer to archpass_duo.

For connecting from off-campus, please first connect to the UGA VPN and then connect to xfer.gacrc.uga.edu. Information on how to use the VPN is available at UGA_VPN.

To transfer files using scp (or SSH file transfer) you must have scp (or SSH) on your local machine and a connection to the UGA campus network. An scp software is included in recent releases of Unix based operating systems (including Linux and Mac OS X). An SSH Secure File Transfer Utility is available for Windows on the UGA Sitesoft page (you will need to enter your UGA myID and password to download the software).

The GACRC file transfer nodes (xfer) are configured to facilitate file transfer. These nodes, which have a hostname of xfer.gacrc.uga.edu, are connected via a higher-bandwidth network connection. Hence, maximum transfer speed between a user's local machine and the GACRC systems can be achieved by transferring files to/from the host xfer.gacrc.uga.edu (instead of the login nodes of the clusters).


Note that a user's home directory on the xfer nodes is the same as the user's Sapelo2 home directory, which is not the same as the user's home directory on the teaching cluster.

The xfer node can access the following file systems using the full path:

1. Your home directory on an xfer node is the same as your Sapelo2 home directory, and the path is /home/username

2. The zcluster home directory: /panfs/pstor.storage/home/groupname/username (for former zcluster users only)

3. The zcluster escratch4 directory: /escratch4/username (for former zcluster users only)

4. The Sapelo2 scratch directory: /scratch/username

5. The Sapelo2 work directory: /work/groupname

6. The project file system for the lab: /project/groupname


Note: Starting Nov. 16th, the old /lustre1 file system was replaced by a /scratch file system. All files that were present in /lustre1 on Nov. 16th have been copied to /scratch. When submitting new jobs, please use /scratch and do not use /lustre1 anymore.


Transferring Files from your local storage device to Sapelo2 or vice-versa

To transfer files from your local storage device to Sapelo2 or from Sapelo2 to your local storage device, please connect to xfer.gacrc.uga.edu (use your UGA MyID, the MyID password to authenticate). You will also be prompted for two-factor authentication with Archpass Duo.


Using FileZilla

In order to use FileZilla to transfer files between a local machine and Sapelo2, users will need to install the FileZilla software on the local machine. After installation, start FileZilla and follow these steps:

1. Go to File -> Site Manager

2. Add a New Site with the following configuration.

In the General tab select 


Protocol: SFTP - SSH File Transfer Protocol

Host: sftp://xfer.gacrc.uga.edu

Port: 22

Logon Type: Interactive

User:  [your UGA MyID]

In the Transfer Settings tab, check the box for "Limit number of simultaneous connections" and set "Maximum number of connections" to 1.

If this step is omitted, then you will need to enter your password and Duo authentication for each file that you upload or download. By limiting the number of simultaneous connections to 1, you only need to enter your password and Duo authentication once per session.

With the above settings saved, you should be able to transfer files to txfer using filezilla. When you open the connection, you will be prompted for your MyID password (unless you use key based ssh). If the password authenticates successfully, then another pop-up window will prompt for Duo. Note that the Duo window looks very similar to the password window, but the text in the box will describe the options you can use for Duo (for example, a push, enter a passcode, etc).

Once the connection is established you can upload files from your local machine to Sapelo2 or download files from Sapelo2 to your local machine.

Here is the PDF to check step-by-step screen shots:

media:Xfer Filezilla 12032018.pdf

Using WinSCP

In order to use WinSCP to transfer files between a local Windows machine and Sapelo2, users will need to install the WinSCP software on the local machine. After installation, when you click the WinSCP shortcut button, Login window pops up, in that you select new site and enter the following information to start the new session: 


Session:

File Protocol: SFTP

Host name: xfer.gacrc.uga.edu

Port number: 22

Username: [your UGA MyID]

Password: [your MyID password]

If the password authenticates successfully, then another pop-up window will prompt for Duo two-factor authentication. It says, Using Keyboard-interactive authentication and lists the Duo options you can choose from: Duo push, Phone Call, or SMS passcode, etc. Once the connection is established you can upload or download files from Local Machine to Cluster or vice-versa. Note: Under Advanced, kept the "Default" option for Transfer settings Rule. With this default option, you will be able to transfer multiple files or folders during the session.


Using SSH Secure File Transfer

In order to use the SSH Secure File Transfer software that is installed as part of SSH (Secure Shell Utilities for Windows) available on the UGA Sitesoft page, users need to start the Secure File Transfer application and open a connection to host xfer.gacrc.uga.edu. You will have to enter your UGA MyID as the username, but other fields (port number, etc) can be left empty (or leave the default value, if any). You will be prompted for your MyID password and after that you will be prompted to enter DUO two factor authentication code. Please choose one of the options like DUO push, Phone call or SMS passcodes. Once the connection is established, you can upload or download files.

Here is the PDF to check step-by-step screen shots:

media:Xfer SSH File Transfer Nov292018.pdf

Using scp

Sample commands to transfer files from your local Unix/Linux/Mac OS X machine to Sapelo2:

Open a terminal in your local machine, change directory (cd) to where the files are located in your local machine, and at the command prompt type: 

scp filename  yourUGAMyID@xfer.gacrc.uga.edu:subdirectory

where filename is the name of the file to be transferred, yourUGAMyID is your UGA MyID, and subdirectory is the subdirectory in your Sapelo2 home directory to which files are being transferred. You will then be asked to enter your UGA MyID password. You will also be prompted for two-factor authentication with Archpass Duo.


(Note: yourUGAMyID@ can be omitted if your username on Sapelo2 is the same as on your local machine). Multiple files (e.g. file1, file2, and file3) can be transferred with a single command: 

scp file1 file2 file3  yourUGAMyID@xfer.gacrc.uga.edu:subdirectory

Wildcards can be used for multiple file transfer (e.g. all files with .dat extension): 

scp *.dat yourUGAMyID@xfer.gacrc.uga.edu:subdirectory


A directory and all its contents can be transferred using the scp option -r, for recursive file transferring. For example, to transfer a directory on your local machine called programs and all files in it to your Sapelo2 home directory, use: 

scp -r programs yourUGAMyID@xfer.gacrc.uga.edu:

You can also use the scp option -p to preserve the file features, such as last modification time, file permissions, etc.


To transfer files directly to your Sapelo2 /scratch area (such as /scratch/johndoe/): 

scp filename  yourUGAMyID@xfer.gacrc.uga.edu:/scratch/johndoe/subdirectory

To transfer files directly to your Sapelo2 work directory: 

scp filename  yourUGAMyID@xfer.gacrc.uga.edu:/work/groupname/subdirname

To transfer files directly to your lab's project file system (such as /project/groupname): 

scp filename  yourUGAMyID@xfer.gacrc.uga.edu:/project/groupname


To transfer files from the decommissioned zcluster home dir to your local machine

From your local machine, issue the command: 

scp yourUGAMyID@xfer.gacrc.uga.edu:/panfs/pstor.storage/home/groupname/username/subdirectory/filename ./

The transferred file will be at your working directory. (Note: yourUGAMyID@ can be omitted if your username on Sapelo2 is the same as on your local machine)

Wildcards can be used for multiple file transfer (e.g. all files with .dat extension in your Sapelo2 home dir, in a subdirectory called subdir): 

scp yourUGAMyID@xfer.gacrc.uga.edu:subdir/\*.dat   ./

(Note the backslash “ \ ” preceeding * in the last example.)


Download directly from internet to Sapelo2

Here is an example on how to download FastaX software from internet directly to Sapelo2 

wget https://github.com/agordon/fastx_toolkit/releases/download/0.0.14/fastx_toolkit-0.0.14.tar.bz2

Some websites use indirect connections, it could be downloaded as 

curl -OL https://github.com/agordon/fastx_toolkit/releases/download/0.0.14/fastx_toolkit-0.0.14.tar.bz2


Transferring Files from the decommissioned zcluster to Sapelo2

Use ssh to login to xfer.gacrc.uga.edu using your UGA MyID username and MyID password. You will also be prompted for two-factor authentication with Archpass Duo. This file transfer node has the following file systems mounted:

  • zcluster /escratch4 file system. To access it, use e.g.
cd /escratch4/username
  • zcluster home file systems. To access your zcluster home directory, use the following
cd /panfs/pstor.storage/home/groupname/username

For example, if your group is named abclab and your username is jsmith use 

cd /panfs/pstor.storage/home/abclab/jsmith
  • Sapelo2 home file systems. To access your Sapelo2 home directory, use the following
cd /home/username
  • Sapelo2 scratch file system on the Lustre file system. To access your /scratch space use the following
cd /scratch/username
  • Sapelo2 work file system on the Lustre file system. To access your group's /work space use the following
cd /work/abclab
  • /project file system. To access your group's /project space use the following (please note that this file system is mounted when it is first accessed, so you need to access it with the full path):
cd /project/abclab

Because the xfer node mounts all these file systems, users can ssh into an xfer node and use cp or rsync to copy the files between any two of these file systems. Users can also copy files from their zcluster home dirs or /escratch4 to their project file system (/project/groupname).


Transferring Files between two file systems on Sapelo2

Use ssh to login to xfer.gacrc.uga.edu using your UGA MyID username and password (and two-factor authentication via Archpass Duo) to transfer files between different file systems on Sapelo2, including the project area.


The File Transfer node for the teaching cluster (txfer.gacrc.uga.edu)

Users can transfer files between their local machines and the teaching cluster using (a) Filezilla, (b) a secure copy (scp), or (c) an SSH secure File Transfer software (SSH file transfer). To transfer files using scp (or SSH file transfer) you must have scp (or SSH) on your local machine and a connection to the UGA campus network. An scp software is included in recent releases of Unix based operating systems (including Linux and Mac OS X). An SSH Secure File Transfer Utility is available for Windows on the UGA Sitesoft page (you will need to enter your UGA myID and password to download the software).

Access to txfer.gacrc.uga.edu requires an active MyID and password, and two-factor authentication with Archpass Duo. For more details of Archpass Duo, please refer to archpass_duo .

Transferring Files from your local storage device to the teaching cluster

To transfer files from your local storage device to the teaching cluster, please connect to txfer.gacrc.uga.edu (use your UGA MyID and the MyID password to authenticate). You will also be prompted for two-factor authentication with Archpass Duo.


Using FileZilla

In order to use FileZilla to transfer files between a local machine and the teaching cluster, users will need to install the FileZilla software on the local machine. After installation, start FileZilla and follow these steps:

1. Go to File -> Site Manager

2. Add a New Site with the following configuration.

In the General tab select 


Protocol: SFTP - SSH File Transfer Protocol

Host: sftp://txfer.gacrc.uga.edu

Port: 22

Logon Type: Interactive

User:  [your UGA MyID]

In the Transfer Settings tab, check the box for "Limit number of simultaneous connections" and set "Maximum number of connections" to 1.

If this step is omitted, then you will need to enter your password and Duo authentication for each file that you upload or download. By limiting the number of simultaneous connections to 1, you only need to enter your password and Duo authentication once per session.

With the above settings saved, you should be able to transfer files to txfer using filezilla. When you open the connection, you will be prompted for your MyID password (unless you use key based ssh). If the password authenticates successfully, then another pop-up window will prompt for Duo. Note that the Duo window looks very similar to the password window, but the text in the box will describe the options you can use for Duo (for example, a push, enter a passcode, etc).


Using WinSCP

In order to use WinSCP to transfer files between a local Windows machine and the teaching cluster, users will need to install the WinSCP software on the local machine. After installation, when you click the WinSCP shortcut button, Login window pops up, in that you select new site and enter the following information to start the new session: 


Session:

File Protocol: SFTP

Host name: txfer.gacrc.uga.edu

Port number: 22

Username: [your UGA MyID]

Password: [your MyID password]

If the password authenticates successfully, then another pop-up window will prompt for Duo two-factor authentication. It says, Using Keyboard-interactive authentication and lists the Duo options you can choose from: Duo push, Phone Call, or SMS passcode, etc. Once the connection is established you can upload or download files from Local Machine to Cluster or vice-versa. Note: Under Advanced, kept the "Default" option for Transfer settings Rule. With this default option, you will be able to transfer multiple files or folders during the session.

Using SSH Secure File Transfer

In order to use the SSH Secure File Transfer software that is installed as part of SSH (Secure Shell Utilities for Windows) available on the UGA Sitesoft page, users need to start the Secure File Transfer application and open a connection to host txfer.gacrc.uga.edu. You will have to enter your UGA MyID as the username, but other fields (port number, etc) can be left empty (or leave the default value, if any). You will be prompted for your MyID password and after that you will be prompted to enter DUO two factor authentication code. Please choose one of the options like DUO push, Phone call or SMS passcodes. Once the connection is established, you can upload or download files.

Here is the PDF to check step-by-step screen shots:

media: Txfer SSH File Transfer Nov302018.pdf

Using scp

Sample commands to transfer files from your local Unix/Linux/Mac OS X machine to the teaching cluster:

Open a terminal in your local machine, change directory (cd) to where the files are located in your local machine, and at the command prompt type: 

scp filename  yourUGAMyID@txfer.gacrc.uga.edu:subdirectory

where filename is the name of the file to be transferred, yourUGAMyID is your UGA MyID, and subdirectory is the subdirectory in your teaching cluster home directory to which files are being transferred. You will then be asked to enter your UGA MyID password.

(Note: username@ can be omitted if your username on the teaching cluster is the same as on your local machine). Multiple files (e.g. file1, file2, and file3) can be transferred with a single command: 

scp file1 file2 file3  yourUGAMyID@txfer.gacrc.uga.edu:subdirectory

Wildcards can be used for multiple file transfer (e.g. all files with .dat extension): 

scp *.dat yourUGAMyID@txfer.gacrc.uga.edu:subdirectory

Transferring Files between your Sapelo2 directories and the teaching cluster

To transfer files between your Sapelo2 directories (e.g. your Sapelo2 home directory or /lustre1 directory) and your teaching cluster home directory, please first use ssh to login into xfer.gacrc.uga.edu. Then use the scp command to copy files to/from your home directory at txfer.gacrc.uga.edu.

For example, to transfer a directory called subdir2 in your /lustre1 directory to your teaching cluster home directory. On xfer.gacrc.uga.edu issue the command: 

scp -r -p /lustre1/yourUGAMyID/subdir2 yourUGAMyID@txfer.gacrc.uga.edu:

UGA DUO authentication is required for SSH/SCP access to
GACRC systems.

UGA DUO is a two-factor authentication service which
requires a password (one factor) and a code, phone,
or device (second factor) to successfully authenticate.

If you are not enrolled in the UGA DUO service please 
visit the UGA DUO service self-service portal to enroll
and configure or manage your DUO enabled devices.

https://eits.uga.edu/access_and_security/infosec/tools/duo/portal/

For additional help with UGA DUO authentication or to 
report an issue please visit:

https://eits.uga.edu/access_and_security/infosec/tools/archpass/

Password: 


Duo two-factor login for yourUGAMyID

Enter a passcode or select one of the following options:

 1. Duo Push to XXX-XXX-4304
 2. Phone call to XXX-XXX-4304
 3. SMS passcodes to XXX-XXX-4304

Passcode or option (1-3):

When you enter your password, you will be prompted for the two-factor authentication step. Once you enter a passcode or select an option, the file transfer will start.

Transferring Files between your project area and the teaching cluster

To transfer files between your /project area and your teaching cluster home directory, please first use ssh to login into xfer.gacrc.uga.edu. Then use the scp command to e.g. copy files from /project to your home directory at txfer.gacrc.uga.edu (or to copy files from your home directory at txfer.gacrc.uga.edu to your /project area).

For example, to copy a file called species.fa from your teaching cluster home directory to a subdirectory called binf1234 in your /project area, use the following command on xfer.gacrc.uga.edu: 

scp txfer.gacrc.uga.edu:species.fa /project/abclab/binf1234

where abclab needs to be changed to the name of your group.

Retrieved from "https://wiki.gacrc.uga.edu/index.php?title=Transferring_Files&oldid=14984"