In this part you will find documentation about rclone. Rclone is the rsync for cloud storage. Information on how to install rclone and other things may be found at: https://rclone.org.

Installing rclone from package managers is not recommended as these versions are often quite old. Please follow the instructions on https://rclone.org

Generate a WebDAV Password

Before you can use WebDAV, go through the following steps how to generated you WebDAV password first.


Contents

What can Rclone do?

Apart from being an rsync-type tool for cloud storage, it has the following features:

  • MD5/SHA1 hashes checked at all times for file integrity
  • Timestamps preserved on files
  • Partial syncs supported on a whole file basis
  • Copy mode to just copy new/changed files
  • Sync (one way) mode to make a directory identical
  • Check mode to check for file hash equality
  • Can sync to and from network, eg two different cloud accounts
  • Optional encryption ( Crypt )
  • Optional FUSE mount ( rclone mount )

Rclone configuration

In order to use Rclone, an Rclone configuration must be created first. Start the rclone config utility to generatie a Rclone configuration;

    rclone config

Then you get the following options:

 

  1. Select n for new remote and choose the name, RD in this example
  2. Select the storage system. Select WebDAV
  3. Fill in the URL, that is displayed when Getting your WebDAV credentials. In this example we use: https://researchdrive.surfsara.nl/remote.php/webdav
  4. Select the type of WebDAV storage system. Select Owncloud
  5. Type in your user name, displayed when creating the WebDAV credentials
  6. Select: y) Yes type in my own password
  7. Type in your password and type it in again for confirmation. This is the WebDAV password generated in step 3 on your personal settings page.
  8. Confirm your settings


Environment URL can be different

Here you enter the web address of the environment that you are using.
- For example; https://researchdrive.surfsara.nl or look up here: Institutional instances

The rclone.conf file resides in .config/rclone/ by default and will contains the following contents: 

[RD]
type = webdav
url = https://researchdrive.surfsara.nl/remote.php/webdav/
vendor = owncloud
user = <user name>
pass = <encrypted password>

If you want to put the rclone.conf file in a non-standard place, then that is possible too, but then you need to run your rclone commands in the following manner:

    rclone --config /path/to/rclone.conf <command> ......
Timeout

It is important to set the '--timeout' option high enough. As a rule of thumb, set it to 10 minutes for every GB of the biggest file in a collection. So if the biggest file you want to upload in a collection is 10GB, set --timeout 100m

This may look ridiculously large, but it provides a safe margin to avoid problems with timeout issues.

Avoid locking usage

Use also the flag `--use-cookies` to return always on the same Research Drive back end, to prevent file lock between Research Drive back-ends.

Command examples

Listing destination directory

To get an overview of your current files on Research Drive enter the following:

    rclone ls RD:


Spaces in path

When your path contains a space, place it then between quotation marks. For example;

./rclone ls "RD:My project with spaces (Projectfolder)"

Copy source directory to destination directory

Copy the source to the destination. Doesn’t transfer unchanged files, testing by size and modification time or MD5SUM. Doesn’t delete files from the destination.
If my/destination/folder doesn’t exist, it is created and the contents of /my/folder goes there:

    rclone copy /my/folder RD:my/destination/folder

Working with large objects

When you want to upload large files to Research Drive, we recommend using a timeout of 10 minutes per gigabyte of the largest source file.
As an example, the largest file in the source directory is 5GB. Calculating the argument for --timeout gives: 10 minutes x 5GB = 50 minutes:

    rclone copy --use-cookies --timeout 50m ~/my_5gb_file.bin RD:my/destination/folder

Sync source directory with destination directory

Sync the source to the destination, changing the destination only. Doesn’t transfer unchanged files, testing by size and modification time or MD5SUM. Destination is updated to match source, including deleting files if necessary:

    rclone sync /my/folder RD:my/destination/folder

Important: Since this can cause data loss, test first with the --dry-run flag to see exactly what would be copied and deleted.

Note that files in the destination won’t be deleted if there were any errors at any point.

If my/destination/folder doesn’t exist, it is created and the contents of /my/folder goes there.

Check if files in the source and destination match

Checks the files in the source and destination match. It compares sizes and hashes (MD5 or SHA1) and logs a report of files which don’t match. It doesn’t alter the source or destination.

    rclone check /my/folder RD:my/destination/folder

Use Rclone to mount file systems in user space

Using Rclone to mount a file system in user space is done as follows:

    rclone mount --use-cookies --timeout 15m RD:[path/to/dir] /path/to/local/mount

The flag `--use-cookies` is needed, to get you always on the same Research Drive back-end, to prevent file lock between back-ends. The timeout flag is useful for uploading large files, we recommend using a timeout of 10 minutes per gigabyte of the largest source file.

Enable verbos mode

If you are not receving any response from the mount command, enable the verbos mode using "-v" or "-vv". Rclone will then tell you about each file that is transferred and a small number of significant events.

rclone -v ...
rclone -vv ...

You can unmount this file system by:

    fusermount -u /path/to/local/mount

An example is shown below:




  • No labels