Mac Ftp Download Command Line

Question & Answer

 

Oct 19, 2013. You can use FTP to transfer files with an FTP server by using the command-line interface (CLI); to use the CLI, open a Terminal (or shell) session. To use a Terminal session, click the Launchpad icon in the Dock and click the Utilities group; then click the Terminal icon.

Mac Os Ftp Command

Question

Answer

Mac Ftp Download Command Line Installer

IN THIS ARTICLE:

Description

The National Center for Biotechnology Information (NCBI) offers a wealth of databases analysis tools and reports for use in research by the medical and scientific community.

These resources are freely available to download from the NCBI website. Because of the large sizes of most of the datasets (on the level of gigabytes or terabytes) the recommended method of transfer is with the Aspera Connect browser plugin.

You can use Aspera Connect directly through the NCBI website on your browser by clicking and downloading the datasets of your choice.

Alternatively you can also choose to download data from NCBI through the command line with ascp Asperas transfer tool which comes bundled with your Connect installation.

Usage

The general syntax for downloading data from NCBI is the following:

The components of the command can be broken down as follows:

• /path/to/ascp You will need to specify the full path to theascp program a reference for which can be found in the next section.
• -k 1 If the transfer stops because of connection loss or other issues the k option tells the transfer to resume from where it left off rather than restarting the entire transfer over. This is important because of the large size of most NCBI data. The 1 specifies that a sparse checksum will be performed before resuming a transfer which is the best choice for NCBI data because a full checksum on large files may be slow. For more information on the resume transfer option see this Knowledge Base article.
• -T This option tells the server not to encrypt the transfer as NCBIs download server doesnt offer encryption.
• -i /path/to/private/key This is an option which specifies the path to the private key used to authenticate this transfer. Ensure that you specify the FULL path to the key (in other words ~/path/to/key or similar shortcuts will not work).
• anonftp is the transfer user configured on NCBIs Aspera server.
• ftp.ncbi.nlm.nih.gov is the hostname of NCBIs Aspera server.
• /path/to/data is the path to the data you are downloading. You can find a reference of these paths here.
• /local/location is the path to the folder on your own machine that you want the NCBI files to be downloaded to.

Private key and ascp locations

The private key you will use is asperaweb_id_dsa.openssh which comes with your Connect installation.

Below are locations where you can generally find the private key and the ascp executable. Where applicable replace usernamewith the name of the user you're logged in as.

Mac

Private key

• Local installation of connect - /Users/username/Applications/Aspera Connect.app/Contents/Resources/asperaweb_id_dsa.openssh
• System wide installation of Connect - /Applications/Aspera Connect.app/Contents/Resources/asperaweb_id_dsa.openssh

ascp

• Local installation of connect - /Users/username/Applications/Aspera Connect.app/Contents/Resources/ascp
• System wide installation of Connect - /Applications/Aspera Connect.app/Contents/Resources/ascp

Linux

Private key

• /home/username/.aspera/connect/etc/asperaweb_id_dsa.openssh
• /opt/aspera/etc/asperaweb_id_dsa.openssh

ascp

• /opt/aspera/bin/ascp

Windows

Private key

• 'C:Program Files (x86)AsperaAspera Connectetcasperaweb_id_dsa.openssh'
• C:UsersusernameAppDataLocalProgramsAsperaAspera Connectetcasperaweb_id_dsa.openssh

ascp

• C:Program FilesAsperaAspera Connectbinascp.exe
• C:UsersusernameAppDataLocalProgramsAsperaAspera Connectbinascp.exe

Examples

The following examples demonstrate usage of ascp to download real data from NCBI. Commands for Mac Linux and Windows will be shown with the assumption that we are downloading from a user account on the system named janedoe and downloaded data will go to the folder NCBI_data in janedoes home directory. The path locations of the datasets are shown on NCBI's public download directory.
1. Say you need to download all the data NCBI offers on epigenomics. There is a 223.79 GB sized folder on the topic containing 5 subfolders worth of data. In order to download the entire folder via ascp you would use the following command:

On a Mac:

On a Windows:

On Linux:

2. Perhaps you are conducting a study on tree-dwelling lizards and want to examine the genome data NCBI offers for the Anolis carolinensis species. To download the genome data for this species you would use the following command:

On a Mac:

On a Windows:

On Linux:

3. As part of a research paper youre writing you need to look at NCBIs RefSeq project data concerning protein and RNA sequencing data in humans. You know there is 1.69 GB worth of available data on NCBI and you proceed to download it with the following command:

On a Mac:

On a Windows:

On Linux:

4. Another Windows example an actual command line and the file download status:

[{'Business Unit':{'code':'BU053','label':'Cloud & Data Platform'},'Product':{'code':'SSMURG','label':'IBM Aspera Connect'},'Component':','Platform':[{'code':'PF025','label':'Platform Independent'}],'Version':'All Versions','Edition':','Line of Business':{'code':'LOB15','label':'Integration'}}]

Document Information

Modified date:
18 February 2020

There is no shortage of GUI FTP programs, but kicking it old school on the command line allows you to easily automate uploads and downloads. The best part is, there is nothing to install. Everything you need waits patiently behind the warm glow of a Terminal session.

The Mac Command Line FTP Program

The default command line FTP program in OS X 10.5 Leopard resides at /usr/bin/ftp

Mac Terminal Ftp

By all outward appearances and behavior, the Mac FTP program is the standard BSD version. The man page is the standard BSD main page and contains a wealth of useful information.

A typical command line FTP session is interactive and goes something like this:

• login to an FTP server
• issue commands (ls (list), get (download), put (upload))
• quit

If you have a repetitive FTP task, the fun quickly fades into a mind numbing exercise. This is where FTP automation shines.

The Magical .netrc File

What makes FTP automation possible is a magical, little known file called .netrc. The .netrc file is a plain text file that is hidden (the file name starts with a period) and lives in the root of your home directory. The .netrc file allows FTP to perform automatic logins to FTP servers based on the name.

The .netrc is not created by default. You have to create it manually. To create an empty .netrc file, open a Terminal and use the following commands:

It is critical that you issue the chmod command to set the permissions so that only the owner of the file can view it. If the permissions are not set correctly, the FTP client will assume it has been compromised and will refuse to use it.

Inside the .netrc, you define a block of settings for each FTP server you use, including the machine name, the login ID, and the password. Here is a typical block for a mythical FTP server:

Ftp Commands Examples

There are additional settings that can be included. Check the FTP man page for more. You can test your settings by typing “ftp myftpserver.com” at a Terminal prompt, and it should automatically login. Note that you can store multiple FTP server logins in the .netrc file.

Sending FTP Commands from a BASH Shell Script

Mac Os Ftp

Once logins are automated, the final piece of the puzzle is to script a set of FTP commands. The following example uses an advanced BASH shell scripting technique called a “here” document to group the FTP commands to be sent to the server.

The FTP command is issued with the -d flag (debug mode) to make it more verbose. That makes any kind of error more obvious. The connection is made to myftpserver.com using the ID and password from the .netrc file. Once the connection is made, the rest of the commands are issued one at a time until the end of the “here” document at the second “ftpEOF”. Note that any valid FTP commands can be sent. In the example, the prompt command tells FTP not to prompt for multiple file operations, then the put uploads all files with an .html extension. If you want to go the extra mile, you can extend the shell script and do things like reconnect to the FTP server to verify the file sizes of your uploads.

While there are several ways you can automate FTP, the nice thing about this method is that it is portable to Linux or any other Unix system.

Short link: http://goo.gl/1gxlfd

searchword: ftp

Download From Ftp Command Line Mac

Make a donation to Keith Winston’s Tip Jar.