Versions Compared

Old Version 1

changes.mady.by.user Kosja Dietz

Saved on

compared with

New Version Current

changes.mady.by.user Kosja Dietz

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

To migrate data between two instances (for example, between researchdrive.surfsara.nl and a branded instance), we advice to use rclone, which you can download here: https://rclone.org/downloads/


Table of Contents

Define data source and destination

First, log in to the source environment, go to 'settings' in the top right menu, and then to 'security'. Scroll down and create a new app password. Don't click 'done' yet! You need the information that it shows there:

Image Added

You need from this:

  1. Username
  2. Password
  3. URL


Tip
titleConfigure rclone

After installing rclone, begin by opening a command line terminal

Start the rclone cofigurator:

Code Block
languagebash
themeMidnight
titlerclone config
rclone config


First, give a clear name to your remote:

Code Block
languagebash
themeMidnight
titlerclone config
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> source-rdrive


Now rclone will give you a (long) list of possible storages. Select 'Webdav'

Code Block
languagebash
themeMidnight
titlerclone config
Type of storage to configure.
Choose a number from below, or type in your own value
 1 / Alias for a existing remote
   \ "alias"
 2 / Amazon Drive
   \ "amazon cloud drive"
(...)
19 / SSH/SFTP Connection
   \ "sftp"
20 / Webdav
   \ "webdav"
(...)
Storage> 20


Now copy/paste the full URL from the webdav settings (that you got earlier)

Code Block
languagebash
themeMidnight
titlerclone config
URL of http host to connect to
Choose a number from below, or type in your own value
 1 / Connect to example.com
   \ "https://example.com"
url> https://researchdrive.surfsara.nl/remote.php/nonshib-webdav/


Select owncloud

Code Block
languagebash
themeMidnight
titlerclone config
Name of the Webdav site/service/software you are using
Choose a number from below, or type in your own value
 1 / Nextcloud
   \ "nextcloud"
 2 / Owncloud
   \ "owncloud"
 3 / Other site/service or software
   \ "other"
vendor> 2


Copy/paste the username from the webdav settings (that you got earlier)

Code Block
languagebash
themeMidnight
titlerclone config
User name
user> example_user


Copy/paste the password from the webdav settings (that you got earlier)

Code Block
languagebash
themeMidnight
titlerclone config
Password.
y) Yes type in my own password
g) Generate random password
n) No leave this optional password blank
y/g/n> y
Enter the password:
password:
Confirm the password: ********
password: ********


Depending on your rclone version, it may ask for advanced config or bearer token here. You do not need those:

Code Block
languagebash
themeMidnight
titlerclone config
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n


Check the configuration and confirm if it is correct

Code Block
languagebash
themeMidnight
titlerclone config
Remote config
--------------------
[source-rdrive]
type = webdav
url = https://researchdrive.surfsara.nl/remote.php/nonshib-webdav/
vendor = owncloud
user = example_user
pass = *** ENCRYPTED ***
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y


Quit the configurator

Code Block
languagebash
themeMidnight
titlerclone config
Current remotes:

Name                 Type
====                 ====
source-rdrive        webdav

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q
Info
titlerepeat steps

You have to repeat the above steps for the destination environment


Testing your configuration

If all went well, you have a source and a destination defined:

Code Block
languagebash
themeMidnight
titlerclone config
rclone listremotes
source-rdrive:
target-example:


You should be able to list your files:

Code Block
languagebash
themeMidnight
titlerclone config
rclone ls "source-rdrive:/source_project (Projectfolder)/"
rclone ls "target-example:/destination_project (Projectfolder)/"


Migrating the data

See rclone performance considerations for some important performance tips!

Panel
borderColorred
titleColorwhite
titleBGColorred
borderStylesolid
titleTimeout

It is important to set the '--timeout' option high enough. As a rule of thumb, set it to 10 minutes for every GB of the biggest file in a collection. So if the biggest file you want to upload in a collection is 10GB, set --timeout 100m

This may look ridiculously large, but it provides a safe margin to avoid problems with timeout issues.

And you should be able to transfer your files:

Code Block
languagebash
themeMidnight
titlerclone config
rclone -vvv copy --timeout 360m --use-cookies --transfers 16 "source-rdrive:/source_project (Projectfolder)/" "target-example:/destination_project (Projectfolder)/"