Introduction
PyCharm is a popular IDE for the Python programming language, and is developed by Jetbrains.
Remote development means running the back-end of a development environment on a remote device other than the user's personal device.
Remote development using PyCharm is officially supported by Jetbrains, although still in Beta at the time of writing this document.
Prerequisites
It is assumed that:
- The user has already created a public key
- The user is already somewhat familiar with the PyCharm IDE
- The user has access to either PyCharm Professional or PyCharm Edu
- The user is comfortable manually editing configuration files
Table of Contents
Install prerequisites:
Jetbrains Gateway
First, download and install JetBrains gateway from the JetBrains website: https://www.jetbrains.com/remote-development/gateway/
Alternatively, you can install JetBrains gateway through their toolbox app: https://www.jetbrains.com/toolbox-app/
ssh-askpass
IF you have 2FA enabled on Snellius, AND you are on linux, you need to install the ssh-askpass tool.
You should be able to install ssh-askpass through the command line:
sudo apt install ssh-askpass-gnome
Configuring your ssh connection to the batch nodes
In order to be able to run the PyCharm backend on a batch node, PyCharm needs to know how to connect to it. Since connecting directly to batch nodes is not possible, we need to configure the login node as a proxy in your SSH config.
Essentially, this means that your computer will SSH to the login node, which will then forward the connection to the batch node.
On Unix-based systems, your SSH config is typically located at ~/.ssh/config.
Similarly, on Windows, your SSH config will probably be located at %userprofile%\.ssh\config.
Add the following snippet there:
Snellius
Host snellius.surf.nl IdentityFile ~/.ssh/id_surf # replace with the path to your own private key Port 22 # works for thin & fat CPU nodes, and gpu & gpu_vis nodes Host ?cn* ProxyJump snellius.surf.nl IdentityFile ~/.ssh/id_surf # replace with the path to your own private key Port 22
Changing node type
You can add other nodetypes as well; you can find a list of node flavours here
ControlMaster
It might also be useful to add a ControlMaster to your ssh config.
A ControlMaster allows your SSH connection to be multiplexed, meaning multiple SSH connections to the same host can share a connection.
Conversely, if the original connection is killed or drops, all others using this ControlMaster also will drop!
Host * ControlMaster auto ControlPath ~/.ssh/master-%r@%h:%p ServerAliveInterval 15
Allocate an interactive node
This process is more broadly described here, but it boils down to allocating an interactive node using the salloc command:
salloc command, e.g.:salloc -n 32 -t 4:00:00
SLURM will inform you whether the node allocation has been succesfull:
[robertsc@int6 ~]$ salloc -p thin -t 01:00:00 salloc: Single-node jobs run on a shared node by default. Add --exclusive if you want to use a node exclusively. salloc: A full node consists of 128 CPU cores, 229376 MiB of memory and 0 GPUs and can be shared by up to 4 jobs. salloc: By default shared jobs get 1792 MiB of memory per CPU core, unless explicitly overridden with --mem-per-cpu, --mem-per-gpu or --mem. salloc: You will be charged for 0.25 node, based on the number of CPUs, GPUs and the amount memory that you've requested. salloc: Pending job allocation 2961370 salloc: job 2961370 queued and waiting for resources salloc: job 2961370 has been allocated resources salloc: Granted job allocation 2961370 salloc: Waiting for resource configuration salloc: Nodes tcn463 are ready for job
Pay attention to the name of the node (tcn463 in this example) that gets allocated to you, you'll need it later.
You will be fully charged for the allocated resources, even if you are not actively running any Python scripts through PyCharm. Consider how many cores you really need for testing your code, since with interactive work you are leaving resources idle most of the time.
Connect your IDE to the allocated node
After launching JetBrains gateway, you will be greeted by the following screen:
Click "Connect via SSH"
Click the 
icon next to the <New Connection> bar, and then the 
button on the next page.
In the SSH configurations screen:
- Enter your username
- Fill in the node name as Host
- set "Authentication type" to "OpenSSH config and authentication agent"
- click "OK".
Select the added configuration from the drop-down menu, and then click "Check connection and Continue", after which a connection to the node will be made
The first time you connect to a specific node, you will be asked if you trust the RSA key. Press "Yes".
If you have 2FA enabled, you will be prompted to fill in your One-Time Password (OTP)
In the drop-down menu, select your IDE.
Next, fill in the directory path of your project as "Project directory".
Click "Download and Start IDE".
Your IDE will download and launch itself on the interactive node.
If everything went right, we find ourselves in the familiar JetBrains IDE environment:
It is worth pointing out that looking in the bottom right of our Development Environment, the IDE displays the bits/second it is sending and receiving from our node.
As a double check, we can see that the IDE launched itself on the node if we run the htop command on the interactive node:
Adding a Python interpreter to PyCharm
In general, adding a Python interpreter to your remote PyCharm project works no different than adding one to a local project.
To add a Python interpreter to your PyCharm project, first go to the general PyCharm settings, and then under "Project" go to "Python interpreter".
There, you can select the flavour of environment you're used to.
We will list some ways to add a Python interpreter to your project which may not immediately be intuitive.
System Python (i.e. from a module)
We need to find the path to the Python binary, which loaded when we execute module load Python/<xxx>.We can find out its exact location with the which command like so:
$ module load 2022 $ module load Python/3.10.4-GCCcore-11.3.0-bare $ which python /sw/arch/RHEL8/EB_production/2022/software/Python/3.10.4-GCCcore-11.3.0-bare/bin/python
Then, paste the above path at System Interpreter → triple dots '...' menu → Select Python Interpreter
Existing Conda environment on Snellius
Equally, we can find out the path to our Python binary with the with the which command, after activating our conda environment:
$ module load 2021 $ module load Miniconda3/4.9.2 $ conda activate 2D-VQ-AE-2 $ which python /home/robertsc/.conda/envs/2D-VQ-AE-2/bin/python
The correct conda executable path should select itself automatically.
Development
You should now be setup to work with your project!
Any time you execute any code, your code will be running on your allocated node.
Good luck!
Troubleshooting
"Cannot connect to remote host" error message
This error message is the most generic error you can encounter, and may mean a few different things.
The first step is always to check if you can connect to the node in any case, for example through a terminal.
After opening a terminal, try connecting to gcn1 :
ssh <username>@gcn1
If everything is setup correctly, you should be able to connect.
if not, you should debug connecting to gcn1 through the terminal first.
Gateway tries to use the wrong private key
It might happen that gateway tries to use the default ~/.ssh/id_rsa private key, while it should be something else.
You might observe this incorrect behavior after setting a password on your private key, since in the password promt for your private key an incorrect path is shown.
Workaround: in the 'SSH configurations' screen (where you insert the node name), select 'Key Pair' as Authentication type.
Then, insert the correct path to the private key in "Private key file".