...
After evaluating several tools, the HPC admins settled on 2 tools, ecs-sync and rclone, as 'best of breed' for moving data between ROSS and Grace's cluster storage system, where the /home, /scratch, and /storage directory live. The ecs-sync program is the more efficient and the speedier of the two for bulk data moves. It consumes fewer compute resources and less memory than rclone. When properly tweaked for number of threads (i.e. when the sweet spot is found) it moves data significantly faster than rclone. The rclone program has more features than ecs-sync, including ways to browse data in ROSS, to mount a ROSS bucket as if it were a POSIX file system, and to synchronize content using a familiar rsync-like command syntax. While ecs-sync is great for fast, bulk moves, rclone works very well for nuanced access to ROSS and small transfers. The rclone program also works quite nicely for moving data between a workstation or laptop and ROSS.
The following sections describe using ecs-sync and rclone for moving data between ROSS and Grace's cluster storage.
ecs-sync
The ecs-sync program is specifically designed for the parallel bulk moving of data from one storage technology to another. It comes from the Dell/EMC support labs, and is what EMC support engineers use for migrating data. It certainly is possible to install ecs-sync outside of Grace, for example on a lab workstation to rapidly move data to or from ROSS. However in this article we only discuss the use case of moving data between ROSS and Grace using the ecs-sync installed by the HPC Admins.
...