Data transfer between Atlas and UL HPC Clusters#

This How-to Card describes the different methods to transfer data between Atlas and the UL HPC Clusters. The four recommended methods to transfer data are:

• Mounting Atlas on HPC directly
• Using laptop
• Via dedicated Virtual Machine (VM)
• Using Large File Transfer (LFT) tool

Please refer to the dedicated knowledge bases to see how to connect to UL HPC Clusters and to mount Atlas.

Mounting Atlas on HPC directly#

To transfer files directly between Atlas and the UL HPC Clusters, you must mount Atlas on the UL HPC Clusters via SMB first. You can then use scp or rsync commands to perform the transfer. While both commands ensure a secure transfer of data between the UL HPC Clusters and Atlas, rsync may be much faster for large amounts of files (transferred very quickly in batches), and for selective incremental updates of large datasets (it is capable to automatically transferring only the changed files, thus saving time).

• scp: transfers all files and directories.
• rsync: transfers only the files which differ between the source and the destination.

Please visit the UL HPC documentation for instructions on how to use rsync and scp.

Due to the high-speed connection between Atlas and the UL HPC Clusters, it is highly recommended to use this method for transferring large files.

Note: You may want to run large transfers in the background using screen or tmux. It will prevent any interruptions of the transfer - e.g. when your ssh connection gets interrupted. Refer to the documentation about attaching and detaching sessions in the Getting Started guide for tmux.

Using laptop#

When using the UL laptop to transfer data between UL HPC Clusters and Atlas, you must mount Atlas via smb on laptop. Use scp or rsync for the transfer, as explained in Mounting Atlas on HPC directly.

Via dedicated Virtual Machine (VM)#

Data can be transferred via a dedicated VM, which can be requested via ServiceNow. Instead of transferring data between Atlas and UL HPC Clusters through the laptop as described above, the transfer will go through the dedicated VM. Once connected to the VM and mounted to Atlas, the rsync command can be used in the same way as described in the UL HPC documentation. This method is recommended for recurring transfers of large datasets that benefit from high-speed network connection between the VM and the HPC.

Note: You may want to run large transfers in the background using screen or tmux. It will prevent any interruptions of the transfer. Refer to the documentation about attaching and detaching sessions in the Getting Started guide for tmux.

Using LCSB File Transfer (LFT) tool#

An alternative solution is to use LFT for transferring data between Atlas and UL HPC Clusters. This method can reliably transfer large data volumes (typically several terabytes). If the data is already on LFT (e.g. received from external collaborators), you can make a copy of the data and directly download it to the UL HPC Clusters for computational analysis, and the same for Atlas, which is recommended for internal archival.

Please refer to the dedicated How-to Card on LFT for detailed information.

Note: If the data is not already on the LFT server, we strongly recommend to use one of the above methods instead.