To migrate data between two instances (for example, between researchdrive.surfsara.nl and a branded instance), we advice to use rclone, which you can download here: https://rclone.org/downloads/
Create federated share
To create a server-side copy, you need to create a federated share between the source and the destination.
Log in to the destination environment, and in the menu on the top-right corner go to 'settings'. Scroll down to 'Federated Cloud' and copy 'Your Federated Cloud ID'
Log in to the source environment, and share the folder you wish to migrate with your Federated Cloud ID, and select 'Federated':
You are now sharing the folder on the source environment with your account on the destination environment. On the destination environment, you will have a notification asking you to accept (or decline) the federated share. Click accept.
Now you should be able to see the federated share (don't worry about the pending state):
Create rclone config
Log in to the destination environment, go to 'settings' in the top right menu, and then to 'security'. Scroll down and create a new app password. Don't click 'done' yet! You need the information that it shows there:
You need from this:
- Username
- Password
- URL
Configure rclone
After installing rclone, begin by opening a command line terminal
Start the rclone cofigurator:
rclone config
First, give a clear name to your remote:
n) New remote s) Set configuration password q) Quit config n/s/q> n name> destination-example
Now rclone will give you a (long) list of possible storages. Select 'Webdav'
Type of storage to configure. Choose a number from below, or type in your own value 1 / Alias for a existing remote \ "alias" 2 / Amazon Drive \ "amazon cloud drive" (...) 19 / SSH/SFTP Connection \ "sftp" 20 / Webdav \ "webdav" (...) Storage> 20
Now copy/paste the full URL from the webdav settings (that you got earlier)
URL of http host to connect to Choose a number from below, or type in your own value 1 / Connect to example.com \ "https://example.com" url> https://destonation-example.data.surfsara.nl/remote.php/nonshib-webdav/
Select owncloud
Name of the Webdav site/service/software you are using Choose a number from below, or type in your own value 1 / Nextcloud \ "nextcloud" 2 / Owncloud \ "owncloud" 3 / Other site/service or software \ "other" vendor> 2
Copy/paste the username from the webdav settings (that you got earlier)
User name user> example_user
Copy/paste the password from the webdav settings (that you got earlier)
Password. y) Yes type in my own password g) Generate random password n) No leave this optional password blank y/g/n> y Enter the password: password: Confirm the password: ******** password: ********
Depending on your rclone version, it may ask for advanced config or bearer token here. You do not need those:
Edit advanced config? (y/n) y) Yes n) No (default) y/n> n
Check the configuration and confirm if it is correct
Remote config -------------------- [source-rdrive] type = webdav url = https://destination-example.data.surfsara.nl/remote.php/nonshib-webdav/ vendor = owncloud user = example_user pass = *** ENCRYPTED *** -------------------- y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y
Quit the configurator
Current remotes: Name Type ==== ==== destination-example webdav e) Edit existing remote n) New remote d) Delete remote r) Rename remote c) Copy remote s) Set configuration password q) Quit config e/n/d/r/c/s/q> q
Testing your configuration
If all went well, you have the destination defined:
rclone listremotes destination-example:
You should be able to list your files:
rclone ls "destination-example:/source_project/" rclone ls "destination-example:/destination_project (Projectfolder)/"
Migrating the data
See rclone performance considerations for some important performance tips!
It is important to set the '--timeout' option high enough. As a rule of thumb, set it to 10 minutes for every GB of the biggest file in a collection. So if the biggest file you want to upload in a collection is 10GB, set --timeout 100m
This may look ridiculously large, but it provides a safe margin to avoid problems with timeout issues.
And you should be able to transfer your files:
rclone -vvv copy --timeout 360m --use-cookies --transfers 24 "destination-example:/source_project/" "destination-example:/destination_project (Projectfolder)/"
Things not to worry about
Inactivity in the upload
The bigger the number of files, the longer it takes before the process starts running. You may see output like the below, with not much seemingly happening. This is fine, and should not be interrupted. It's creating a listing of all the files on the source, and that takes some time.
example_user@machine1 (2020-11-11 11:47) ~ $ time rclone -vvv copy --timeout 360m --use-cookies --transfers 24 "destination-example:/source_project/" "destination-example:/destination_project (Projectfolder)/" 2020/11/11 11:48:43 DEBUG : rclone: Version "v1.47.0" starting with parameters ["rclone" "-vvv" "copy" "--timeout" "360m" "--use-cookies" "--transfers" "24" "destination-example:/source_project/" "destination-example:/destination_project (Projectfolder)/"] 2020/11/11 11:48:43 DEBUG : Using config file from "/home/example_user/.config/rclone/rclone.conf" 2020/11/11 11:49:43 INFO : Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA - Errors: 0 Checks: 0 / 0, - Transferred: 0 / 0, - Elapsed time: 1m0s 2020/11/11 11:50:43 INFO : Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA - Errors: 0 Checks: 0 / 0, - Transferred: 0 / 0, - Elapsed time: 2m0s 2020/11/11 11:51:43 INFO : Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA - Errors: 0 Checks: 0 / 0, - Transferred: 0 / 0, - Elapsed time: 3m0s 2020/11/11 11:52:43 INFO : Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA - Errors: 0 Checks: 0 / 0, - Transferred: 0 / 0, - Elapsed time: 4m0s 2020/11/11 11:53:43 INFO : Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA - Errors: 0 Checks: 0 / 0, - Transferred: 0 / 0, - Elapsed time: 5m0s 2020/11/11 11:54:43 INFO : Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA - Errors: 0 Checks: 0 / 0, - Transferred: 0 / 0, - Elapsed time: 6m0s 2020/11/11 11:55:43 INFO : Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA - Errors: 0 Checks: 0 / 0, - Transferred: 0 / 0, - Elapsed time: 7m0s
Failed to copy: Copy call failed: The destination node already exists, and the overwrite header is set to false: Sabre\DAV\Exception\PreconditionFailed: 412 Precondition Failed
This means the file already exists on the destination. Because the server side copy also resets timestamps on the destination, it will think that the two files are different. But, the server side copy cannot overwrite this file, and so it flags it as an error.