To migrate data between two instances (for example, between researchdrive.surfsara.nl and a branded instance), we advice to use rclone, which you can download here: https://rclone.org/downloads/
Table of Contents |
---|
Create federated share
To create a server-side copy, you need to create a federated share between the source and the destination.
Log in to the destination environment, and in the menu on the top-right corner go to 'settings'. Scroll down to 'Federated Cloud' and copy 'Your Federated Cloud ID'
Log in to the source environment, and share the folder you wish to migrate with your Federated Cloud ID, and select 'Federated':
You are now sharing the folder on the source environment with your account on the destination environment. On the destination environment, you will have a notification asking you to accept (or decline) the federated share. Click accept.
Now you should be able to see the federated share (don't worry about the pending state):
Create rclone config
Log in to the destination environment, go to 'settings' in the top right menu, and then to 'security'. Scroll down and create a new app password. Don't click 'done' yet! You need the information that it shows there:
You need from this:
- Username
- Password
- URL
Tip | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
After installing rclone, begin by opening a command line terminal Start the rclone cofigurator:
First, give a clear name to your remote:
Now rclone will give you a (long) list of possible storages. Select 'Webdav'
Now copy/paste the full URL from the webdav settings (that you got earlier)
Select owncloud
Copy/paste the username from the webdav settings (that you got earlier)
Copy/paste the password from the webdav settings (that you got earlier)
Depending on your rclone version, it may ask for advanced config or bearer token here. You do not need those:
Check the configuration and confirm if it is correct
Quit the configurator
|
Testing your configuration
If all went well, you have the destination defined:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
rclone listremotes
destination-example: |
You should be able to list your files:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
rclone ls "destination-example:/source_project/"
rclone ls "destination-example:/destination_project (Projectfolder)/" |
Migrating the data
See rclone performance considerations for some important performance tips!
Panel | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||
It is important to set the '-- This may look ridiculously large, but it provides a safe margin to avoid problems with timeout issues. |
And you should be able to transfer your files:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
rclone -vvv copy --timeout 360m --use-cookies --transfers 24 "destination-example:/source_project/" "destination-example:/destination_project (Projectfolder)/" |
Things not to worry about
Inactivity in the upload
The bigger the number of files, the longer it takes before the process starts running. You may see output like the below, with not much seemingly happening. This is fine, and should not be interrupted. It's creating a listing of all the files on the source, and that takes some time.
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
example_user@machine1 (2020-11-11 11:47) ~ $ time rclone -vvv copy --timeout 360m --use-cookies --transfers 24 "destination-example:/source_project/" "destination-example:/destination_project (Projectfolder)/"
2020/11/11 11:48:43 DEBUG : rclone: Version "v1.47.0" starting with parameters ["rclone" "-vvv" "copy" "--timeout" "360m" "--use-cookies" "--transfers" "24" "destination-example:/source_project/" "destination-example:/destination_project (Projectfolder)/"]
2020/11/11 11:48:43 DEBUG : Using config file from "/home/example_user/.config/rclone/rclone.conf"
2020/11/11 11:49:43 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 0, -
Elapsed time: 1m0s
2020/11/11 11:50:43 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 0, -
Elapsed time: 2m0s
2020/11/11 11:51:43 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 0, -
Elapsed time: 3m0s
2020/11/11 11:52:43 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 0, -
Elapsed time: 4m0s
2020/11/11 11:53:43 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 0, -
Elapsed time: 5m0s
2020/11/11 11:54:43 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 0, -
Elapsed time: 6m0s
2020/11/11 11:55:43 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 0, -
Elapsed time: 7m0s |
Failed to copy: Copy call failed: The destination node already exists, and the overwrite header is set to false: Sabre\DAV\Exception\PreconditionFailed: 412 Precondition Failed
This means the file already exists on the destination. Because the server side copy also resets timestamps on the destination, it will think that the two files are different. But, the server side copy cannot overwrite this file, and so it flags it as an error.