“Instead of maximizing usability at the expense of utility, we should focus on making powerful tools and fostering a culture of mentorship".
The icommands are a powerful high performant set of tools that can be installed in any unix environment.
It can be used both for administrative tasks and (big) data handling.
It is especially powerful, because it makes full use of the ability of iRODS to transfer files in parallel.
It can securely send a large file completely saturating the available network.
Note that the icommands can be solely operated from within the terminal.
Following information is based on assuming basic proficiency of commandline tools.
Note that most of this information can also be found on docs.irods.org/4.2.8/icommands.
Installation
For Ubuntu 18.04:
wget -qO - https://packages.irods.org/irods-signing-key.asc | sudo apt-key add - echo "deb [arch=amd64] https://packages.irods.org/apt/ xenial main" | sudo tee /etc/apt/sources.list.d/renci-irods.list sudo apt-get update sudo apt -y install irods-icommands
For CentOS:
sudo yum -y install wget epel-release sudo rpm --import https://packages.irods.org/irods-signing-key.asc wget -qO - https://packages.irods.org/renci-irods.yum.repo | sudo tee /etc/yum.repos.d/renci-irods.yum.repo sudo yum -y install irods-icommands
For MacOS:
There are no up-to-date packages for MacOS writing this document.
The icommands can be built from source (https://github.com/irods/irods_client_icommands), however it is not trivial.
Please let us know at servicedesk@surfsara.nl if you would like a more supported version of icommands for MacOS.
For Windows:
The icommands work in Linux.
They should be installable via the Ubuntu method using the linux subsystem of Windows 10.
Connecting to Yoda/iRODS
To connect to Yoda/iRODS you have to create the `~/.irods/irods_environment.json` file and edit it as follows:
{
"irods_authentication_scheme": "pam",
"irods_client_server_negotiation": "request_server_negotiation",
"irods_encryption_algorithm": "AES-256-CBC",
"irods_encryption_key_size": 32,
"irods_encryption_num_hash_rounds": 16,
"irods_encryption_salt_size": 8,
"irods_host": "<YODA_HOST_ADDRESS>",
"irods_home": "/tempZone/home",
"irods_port": 1247,
"irods_user_name": "<YODA_USERNAME>",
"irods_zone_name": "tempZone"
}
Note to change the specific details:
- Check Yoda hostnames and URLs for choosing the correct YODA_HOST_ADDRESS
- Your username is either the identity from your institute or the e-mail address you were invited through.
Subsequently you can run the command:
iinit
Enter your password, and a token will be stored locally which will be used when running subsequent icommands.
You can find help for the icommands with:
ihelp
which will show you all the possible commands.
Basic data/collection handling
Note that in iRODS files are called data objects, and folders are called collections.
The icommands have basic file handling functionality that have their (almost) equivalent in the Linux bash commands but with an `i` in front of the name, *e.g.* `ls` and `ils`, `cp` and `icp`, `mv` and `imv`, `pwd` and `ipwd`, `mkdir` and `imkdir`, `cd` and `icd`. `rm` and `irm`.
Try out `icommand -h` if you want to know more about the available options.
However, please do realize iRODS is not a filesystem like you have on your personal computer.
Data handling is done differently.
Create collections and navigate
Use `ils` to see what collections are available to you and navigate to that collection with `icd`:
ils icd <collection name>
Note that you can't use bash tab completion with paths in iRODS.
You can create a collection with `imkdir`.
If you get an error that you are not allowed to create a collection, be sure to have navigated to a collection where you do have the permission to write.
You can use:
ils -A
to check the permissions (`-A` stands for Access Control Lists) of data objects and collections.
Sharing data objects and/or collections with other users
With `ils -A` you can view what the current Access Control Lists are, *i.e.* a list who has what type of access to the data objects and collections in the current working directory of iRODS.
The ACLs can be changed by using `ichmod`.
With this command you modify the access to data objects and collections for specific user or groups.
When you store a file, you are the owner and have full control - you can read, write or delete it and, by default, no one else can.
With chmod you can give access to other users or groups, either just read access, or read and write, or full ownership.
You can only give (or remove) access to others if you own the file yourself.
But if you give 'own' to someone else, they can also give (and remove) access to others.
For example if you like to give user or group `alice` recursive read permissions on folder `fruit` which is located in your iRODS current working directory (you could also use the full iRODS path), you should do the following:
ichmod -r read alice fruit
For more information, see `ichmod -h`.
Tickets are another way to provide access to iRODS data-objects (files) or collections (directories or folders).
The 'iticket' command allows you to create, modify, list, and delete tickets.
When you create a ticket its 15 character string is given to you which you can share with others.
For example, if you would like to give read permission for a collection `fruit`:
iticket create read fruit
This command will print out the ticket string to screen.
Note that you can view all active tickets with `iticket ls`.
You can use `iticket mod ticket-string-or-id` to modify the restrictions on the ticket.
Restrictions can be set on who is able to use the ticket, how many times the ticket can be used, which host or IP is able to use the ticket or the expiration date of the ticket.
Uploading a file with `iput`
In order to get a file in Yoda:
iput <source_file> <dest_do>
when no destination data object is given, the local filename will be used.
iRODS will issue a warning if the file already exists (which you can force to do with the `-f` option).
In order to get a folder in iRODS:
iput -r <source_folder> <dest_coll>
you need to add the `-r` option for recursively putting data in iRODS.
Downloading a file with `iget`
iget <source_do> <dest_file>
Again, if the local destination is not specified, the name of the data object will be used.
To download a collection you have to specify the `-r` option:
iget -r <source_coll> <dest_folder>
Metadata and querying for data
In Yoda/iRODS a data object is not only the bitstream and the filename, but user defined metadata is part of the data object. This makes it very powerful for the findability of data. Data objects can be found by querying for metadata.
Querying based on metadata
Metadata attached to data objects and collections becomes very useful when you want to query for data. Queries can mainly be performed by the `iquest` command, but simple queries can be done by the `imeta` command:
#find data objects where distance is larger than 12 imeta qu -d distance '<=' 12 #find collections where key a is value b and key c smaller than value 10 imeta qu -C a = b and c '<' 10 #find data objects where r is not in between 5 and 7 imeta qu -d r '<' 5 or '>' 7 #find data-objects with attribute 'a' with a value that starts with 'b'. imeta qu -d a like b% #find data-objects with attribute 'a' defined (with any value). imeta qu -d a like %
You can enter the `imeta` prompt by just typing `imeta` and Enter. You can get more helpful tips by then doing `help qu`.
You can look up the available keys you can query for via:
iquest attrs
General `iquest` queries look like:
iquest "select COLL_NAME, DATA_NAME, META_DATA_ATTR_VALUE where \ META_DATA_ATTR_NAME like 'author'"
In above command we wanted to look for everything which has the metadata attribute name (metadata key) that resembles 'author' and retrieve the collection name, data name and metadata attribute value (metadata value).
iRODS will respond something like:
COLL_NAME = /yoda/home/research-dms DATA_NAME = testfile META_DATA_ATTR_VALUE = val1 ------------------------------------------------------------
You can change the format of the iRODS response by using a C like format string after `iquest` like so:
iquest "User %-6.6s has %-5.5s access to file %s" "SELECT USER_NAME, DATA_ACCESS_NAME, DATA_NAME WHERE COLL_NAME = '/yoda/home/research-dms'"
where after `iquest` there is the format string and the second string is again the SQL like query. This can be convenient for making reports but also for retrieving absolute filepaths:
iquest "%s/%s" "select COLL_NAME, DATA_NAME where \ META_DATA_ATTR_NAME like 'author' and META_DATA_ATTR_VALUE = 'Lewis Carroll'"