Skip to content
Contact Support

Moving files to and from the cluster

Prerequisite

Have an active account and project.

Find more information on the NeSI Filesystem.

Using the Jupyter interface

The Jupyter interface useful for running code on NeSI requiring only a web browser; the instructions are same whether your are connecting from a Windows, Mac or a Linux computer.

To upload a file, click on the

up arrow

button, near the top left and generally under the Run button.

To download a file, navigate the file browser on the left and right-click on the file to see the menu below,

right click menu

The Download button is at the bottom.

Standard Terminal

Prerequisite

Have SSH setup as described in Standard Terminal Setup

In a local terminal the following commands can be used to:

Move a file from your local machine to Mahuika.

scp <path/filename> mahuika:<path/filename>

Move a file from Mahuika to your local machine.

scp mahuika:<path/filename> <path/filename>

Note

  • This will only work if you have set up aliases as described in Terminal Setup.
  • As the terms 'maui' and 'mahuika' are defined locally, the above commands only works when using a local terminal (i.e. not on Mahuika).
  • If you are using Windows subsystem, the root paths are different as shown by Windows. e.g. C: is located at /mnt/c/

scp stands for Secure CoPy and operates in a similar way to regular cp with the source file as the left term and destination on the right.

These commands make use of multiplexing, this means that if you already have a connection to the cluster you will not be prompted for your password.

File Managers

Prerequisite

Have SSH setup as described in Standard Terminal Setup

Most file managers can be used to connect to a remote directory simply by typing in the address bar provided your have an active connection to the cluster.

For Nautilus (Ubuntu default) just prepend the path you want to connect to with sftp://mahuika. (ctrl + L opens address bar)

This does not work for File Explorer (Windows default)

This does not work for Finder (Mac default)

files

If your default file manager does not support mounting over SFTP, see Can I use SSHFS to mount the cluster filesystem on my local machine?.

MobaXterm

Prerequisite

MobaXterm Setup Windows

Clicking the "Scp" tab (located on the left-hand side of the
window) opens up a graphical user interface that can be used for basic file operations. You can drag and drop files in the file explorer or use the up and down arrows on the toolbar to upload and download files.

moba terminal

You may also transfer files as described under 'Standard Terminal' (provided Windows_Subsystem_for_Linux is enabled).

WinSCP

Prerequisite

WinSCP-PuTTY Setup Windows

As WinSCP uses multiple tunnels for file transfer you will be required to authenticate again on your first file operation of the session. The second prompt for your 2FA can be skipped, just the same as with login authentication.

Globus

Globus is available for those with large amounts of data, security concerns, or connection consistency issues. You can find more details in Data_Transfer_using_Globus_V5.

Rclone

Rclone is available for those that need to transfer data from cloud storage services like Google drive or OneDrive.

The basic command syntax of Rclone:

rclone subcommand options source:path dest:path

The most frequently used Rclone subcommands:

  • rclone copy - Copy files from the source to the destination, skipping what has already been copied.
  • rclone sync - Make the source and destination identical, modifying only the destination.
  • rclone move - Move files from the source to the destination.
  • rclone delete - Remove the contents of a path.

A more extensive list can be found in the Rclone documentation.

Rsync

Rsync is an utility that provides fast incremental file transfer and efficient file synchronization between a computer and a storage disk. The basic command syntax of:

rsync -options source target

If the data source or target location is a remote site, it is defined with syntax:

userame@server:/path/in/server

The most frequently used Rsync options:

  • -r - Recurse into directories.
  • -a - Use archive mode: copy files and directories recursively and preserve access permissions and time stamps.
  • -z - Compress.
  • -e ssh - Specify the remote shell to use.
  • -n - Show what files would be transferred.

A more extensive list can be found in the Rsync documentation.