Dropbox

Project Status

One of the next steps is a discussion of possible data transfer scenarios.

Questions and Concerns

• What is the maximum file size supported for transfer?
• Is there an issue when large numbers of files must be kept up with?
• Do we want a Dropbox style service or an FTP style service? Or do we need both...
• How well does the solution handle file permissions, especially group permissions?
• Do we want to survey the users to see what features are most important to them?
• What about a pilot with a subset of users?
• How much service level maintenance or intervention does the service require?

Software

ZCluster Copy Nodes

Pros and Cons

• Pros

1. Already installed and configured

• Cons

SparkleShare

http://sparkleshare.org/

Pros and Cons

• Pros

1. Open Source
2. Availibility for Mac, Windows, and Linux
3. Version control (Git based)
4. Notifications of changes to folder contents

• Cons

1. Sharing folders not built into desktop client
2. Git isn’t built for handling large, binary files
3. Version control not built into desktop client

FileZilla

https://filezilla-project.org/

Features

• Easy to use
• Supports FTP, FTP over SSL/TLS (FTPS) and SSH File Transfer Protocol (SFTP)
• Cross-platform. Runs on Windows, Linux, *BSD, Mac OS X and more
• IPv6 support
• Available in many languages
• Supports resume and transfer of large files >4GB
• Tabbed user interface
• Powerful Site Manager and transfer queue
• Bookmarks
• Drag & drop support
• Configurable transfer speed limits
• Filename filters
• Directory comparison
• Network configuration wizard
• Remote file editing
• Keep-alive
• HTTP/1.1, SOCKS5 and FTP-Proxy support
• Logging to file
• Synchronized directory browsing
• Remote file search

Pros and Cons

• Pros

1. Free
2. Supports FTP, FTPS and SFTP
3. Supports resume and transfer of files > 4GB.
4. Client supports Linux, Mac OSX and Windows

• Cons

1. Server requires Windows XP, Vista or 7.

RhinoSoft Serv-U

http://www.serv-u.com/

Features

• Popular Features
  • Phone and Tablet Transfers
  • Web Browser Transfers
  • Web Administration
  • Branding
  • FTP, FTPS and SFTP
  • International Support
  • Event-Driven Automation
• Popular Deployment Options
  • Linux
  • High Availability
  • Reverse Proxy
  • 3rd Party Integration
• Easy Deployment
  • Up and running in minutes
  • Web-based administration
  • iPad administration
• High Performance
  • Native 64-bit and 32-bit applications written in C++
  • N+1 scalable on all tiers
• Total Compatibility
  • Major web browsers
  • Major smart phones and tablets
  • FTP, FTPS, SFTP (SSH), HTTP and HTTPS
  • Guaranteed fidelity to RFCs
  • Supports both IPv4 and IPv6
  • Supports large files (>2GB) on all interfaces
• Industry’s Best Security
  • FIPS 140-2 validated cryptography
  • United States Army Certificate of Networthiness (CoN)
  • No data in DMZ (w/ Serv-U Gateway)
• Total Control
  • Ratios, quotas and bandwidth throttling
  • Mix and match Active Directory, ODBC database, and local authentication
  • Realtime session monitoring and statistics
  • Virtual directories and advanced permissions
  • Multiple domains and options for delegated administration
  • iPad Administration

Pros and Cons

• Pros

1. Web based admin
2. Supports FTP, SFTP and FTPS
3. Supports web and mobile transfers
4. Guaranteed fidelity to RFCs
5. Supports both IPv4 and IPv6
6. Supports large files (>2GB) on all interfaces

• Cons

1. Liscensed ($500 for 250, $3000 unlimited for 1 year)

F*EX Frams' Fast File Exchange

http://fex.rus.uni-stuttgart.de/

Features

• file transfer of virtually unlimited file size
• recipient and sender only need an e-mail program and a web browser - of any kind, they do not have to install any software
• RESEND and REGET for resuming after link failures at last sent byte
• auto-notification of recipient
• auto-deletion after download
• auto-deletion after expiration date (default: 5 days)
• full-users can create one time upload URLs for foreign users
• full-users can create sub-users, who can send only to this full-user
• full-users can create groups, an analogy to mailing lists, but for files
• admin can allow (internal or external) user self-registration
• admin can allow upload to public recipients without authentification
• admin can allow upload for LAN users without registration (anonymous upload)
• Sending to multiple recipients needs storage on the server only once
• F*EX is a HTTP web-service and needs no firewall-tunnels
• maintenance-free: no admin interaction necessary besides creating new F*EX accounts
• support for streams, too
• for real UNIX users, there are shell programs to avoid annoying web browser usage
• protocol and source-code free available

Pros and Cons

• Pros

1. Web based so no client software required

• Cons

1. Requires F*EX accounts

OwnCloud

http://owncloud.org/

Features

• Data Access
• Desktop Sync
• Share Data
• Calendars
• Mount External Storage (Dropbox, ect)
• Docunment Viewer
• Application store
• Tasks
• Theming
• Galleries
• Contacts
• File Versioning
• Undelete
• Powerful Search

Pros and Cons

• Pros

1. Present in EPEL repository

• Cons

1. Possible problem in file size based on webserver settings and memory size?

SeaFile

http://seafile.com/en/home/

Features

• Seafile uses file libraries which can be synchronized separately;
• Users can create and join groups;
• File revisions;
• File preview, comments, event notifications;
• Library encryption;
• Share files;
• Clients available for Windows, Mac OS X, Linux, Android and iOS.

Pros and Cons

• Pros

1. Item 1

• Cons

1. Possible problem in file size based on webserver settings and memory size?
2. Possible issue with large number of files

Globus Online

https://www.globusonline.org/

Features

https://www.globusonline.org/features/

https://www.globusonline.org/plus/ NOTE: This is not in beta yet.

A web seminar on it: https://anl.adobeconnect.com/_a752132587/p5fcn8zvrhv/?launcher=false&fcsContent=true&pbMode=normal

https://twiki.grid.iu.edu/bin/view/ReleaseDocumentation/GlobusOnlineTutorial

• File Movement
  • Web UI
  • CLI Commands
  • Globus Connect (to connect personal PC as an endpoint)
• Performance Optimization
• Error Handling and Retry
• Monitoring
• Endpoint Management
• Conditional Transfers and Integrity Checking

Pros and Cons

• Pros

1. Secure file transfer
2. Globus Connect Multiuser packages the GridFTP server as well as Proxy and Certificate services.
3. Does not require an external cloud storage... making it big data friendly. Use a shared endpoint.
4. Works across multiple security domains.
5. Third party transfer.
6. Support for xsede
7. CLI for scripting.
8. Service enhancements automatic via web login.

• Cons

1. Requires a Globus Online account
2. Endpoints require Globus GridFTP server
3. Single user computers such as laptops require Globus Connect
4. Data sharing/P2P via Globus Online Plus Plan at $7/month or $70/year. This has not reached beta according to the website

Installation

https://support.globusonline.org/entries/23857088-Installing-Globus-Connect-Multiuser

This is a cange for the adding of the repository:

yum install http://www.globus.org/ftppub/gt5/5.2/stable/installers/repo/Globus-5.2.stable-config.centos-6-1.noarch.rpm

Scenarios

Who does the copying: UGA Researcher, UGA Research Lab, Research Associate

Where does the data go: User Workstation, GACRC Filesystems, Research Associate Filesystems, Supercomputing Center

Misc Scenarios: UGA Lab to another UGA Lab

Links

Random links I am using to sort through the project.

http://git-scm.com/

http://blogs.bath.ac.uk/research360/2012/05/mrd-hack-days-file-backup-sync-and-versioning-or-the-academic-dropbox/

http://orbital.blogs.lincoln.ac.uk/project-plan/

https://wiki.filezilla-project.org/Main_Page

https://wiki.filezilla-project.org/FAQ

http://orbital.blogs.lincoln.ac.uk/2012/05/04/shared-versioned-network-drives/

http://www.webupd8.org/2013/02/seafile-robust-file-synchronization-and.html

http://www.nersc.gov/users/data-and-file-systems/transferring-data/globus-online/