Please refer to the "User Manual - Student" for non-teacher specific instructions (e.g. changing your password, accepting the usage agreement, starting a notebook, etc)
Basic Topics
How to request logins for a course
You can provide a list of all students (first name, last name, e-mail address) in semicolon ";"-separated file format (.csv) through the Servicedesk (https://servicedesk.surfsara.nl/jira/plugins/servlet/desk/portal/1).
Specifying course hours
The times at which the large reservation (intended for course hours) is available is agreed upon beforehand between SURF, the local ICT and/or the teacher. Please note that we need to put reservations in at least 5 days in advance in order to make sure the required amount of hardware is available. Allowing some time for us to process your request, this means that you'll typically need to notify us at least a week in advance on changes when this large reservation should be available. The maximum total duration of this large reservation is agreed upon in the contract with your local ICT, so it is important to only request the larger reservation when needed.
Logging in
Each course uses it's own hub. The hubs are running at https://jupyter.lisa.surfsara.nl/coursename. The 'coursename' will be agreed upon before the start of the course and will have either been communicated to you either by SURF or your local ICT contact.
Further instructions on loggin in can be found in the Student Manual.
The Jupyter Notebook session
After launching a Jupyter Notebook server, you should see something similar to the following screen
These are the files and folders present in your home directory. There are a couple of special folders starting with JHL_*, the use of which we will explain in this guide. In general, these give you means to share data, notebooks, and installations (e.g. additional Python packages) with students.
Basic navigation, create/move/remove folders, create/move/remove files, uploading/downloading files
Please see the User Manual for Students
Working in a notebook
Please see the User Manual for Students
The admin panel
An admin panel can be accessed. If you are not running a Jupyter Notebook server, click the 'Admin' button at the top to access it. If you are already running a Notebook Server yourself, you will need to click the 'Control Panel' in the top right first, after which the 'Admin' button will appear at the top.
The admin panel shows all the logins that have recently logged in, and shows which logins are currently running a Jupyter Notebook server. For example, in the image below you see that the student with login 'lcurs002' is running a Jupyter Notebook server. If needed, you as a teacher can shutdown this Jupyter Notebook server by clicking the 'stop server' button behind that particular login. Note that we use the login database of the host system (Lisa) to authenticate users, not the internal login database of JupyterHub. Thus, it is not possible to add users with the Add Users button. Also, please do not use the 'edit user' button. This will make the internal login database of Jupyterhub go out of sync with the login database of the host system. The delete user button is also not functional . Though clicking it will shutdown ánd logout the corresponding user, this change is not persistent and the user can simply re-login.
Sharing notebooks with your students
The JHL_notebooks folder and its sub-folders are intended for you to distribute your notebooks to the students. Anything placed in this folder is automatically synchronized (i.e. copied) to the students whenever the student starts a new Jupyter Notebook server. To prevent overwriting any of the student's changes afterwards, only files that do not exist yet in the student's home directory are copied. The only way for the student to receive your original copy again is for him/her to throw away their copy, shutdown the Jupyter Notebook server, and start a new Jupyter Notebook server again.
To prevent notebooks from being synchronized before you are finished with them, you can prepare your notebooks in the JHL_prepare folder. Files created there will get the correct permissions so that they can later be shared with the students, but are not synchronized yet. Once you are done with the notebook, you can move it from the JHL_prepare to the JHL_notebooks.
You may want to gradually add new notebooks to the JHL_notebooks folder during the course-progress, so that students only see the notebooks relevant for the current lessons. Also, you can organize your notebooks in subfolders with in the JHL_notebooks folder if you want: the complete directory structure is synchronized to the students.
Example: preparing and a sharing notebook with your students
In this example, we prepare a notebook "Course_01.ipynb". To prevent this file from being copied to students while we are still developing it we prepare it in the JHL_prepare folder, where we create a new notebook with the 'New' button, and renamed it as 'Course_01.ipynb'
Once we are finished preparing, we can move it to the JHL_notebooks folder by selecting the checkbox befor the 'Course_01.ipynb' file, click move, and enter '/JHL_notebooks' as destination:
The notebooks from the JHL_notebooks folder are synchronized to the student as soon as a student logs in to the Jupyter Hub.
Sharing data with your students
You want to use example data (e.g. a csv dataset, or a dataset of images) in your course and work with that data in the Jupyter Notebooks. There are two ways in which you can share such data with your students:
- You can place your data files in JHL_data
- You can place your data files next to your notebooks in the JHL_notebooks
We strongly encourage the first approach, as described in more detail below.
Sharing data through JHL_data
The JHL_data folder is intended for sharing read-only data with the students. The data is not copied to the student, but can be accessed from notebooks by referring to it with the full file path. This is a very efficient way of sharing files: the file only needs to be stored once (saving storage space), and read performance is much better if a large group of students read a single file, compared to when they would each read their own copy.
Example
Suppose we have a text file "example.txt" we'd like to use in one of our notebooks, we first construct the path where the file can be found with the following code snippet:
import os |
Note that the environment variable TEACHER_DIR refers to the top level directory of your course material. Thus, 'fullpath' now contains the full file path to your "example.txt" file. The nice thing about using the TEACHER_DIR environment variable over hardcoding is that this code is portable: next time you setup a new course, and would run the same notebook, it will refer to the top level directory of your course setup.
After constructing the filepath, you typically want to open and e.g. read the file
file = open(fullpath, 'r') |
(this part of the code of course depends on what type of file you're using in your notebook, for normal text files, we use the open() function). Since the 'fullpath' refers to a file in the JHL_data directory (which is readable to your students), all students will be able to execute this Notebook and work with the data.
Sharing data through JHL_notebooks
WARNING: sharing data through JHL_notebooks is acceptable only for small files (at most a few MB), but should be avoided for large files! If all students read their own copy of a large file when executing a notebook, it would cause the filesystem to slow down to a crawl.
The designated way of sharing files is through JHL_data. However, there may be cases where it is necessary that students have their own copy of the data files, for example because you want them to edit these. In that case, you can store files in JHL_notebooks: each student gets their own copy of the data file, just like each student gets their own copy of the notebooks. The downside is that it increases the amount of storage used and if a large group of students starts to simultaneously read files, this can substnatially impact the performance of the file system. That is why we ask that you use JHL_data for datasharing whenever possible.
Starting a Linux terminal
Apart from notebooks, you can start a Linux terminal from a Jupyter Notebook server. A Linux terminal is the most flexible way of interacting with a Linux system, and will for example enable you to install additional packages, or create/move/rename files from the terminal.
To start a terminal, from your running Jupyter Notebook server, press 'New' and click 'Terminal' from the dropdown menu. This will open a terminal like the one below:
Here, you can use the regular Linux shell commands to work on the system. One thing you can use this for is e.g. to install additional R or Python packages (see instructions below).
Installing additional Python packages
R and Python packages that are installed in the appropriate subfolder within JHL_installations are automatically picked up by the student logins as well. This is because the JHL_installations folder is also readable (read-only) for all students associated with your teacher login. Thus, if you perform a single installation of an additional R or Python package, all students can run that same R or Python package using your installation.
To install a Python package in the correct folder, run
pip install <package_name> --prefix=~/JHL_installations/Python |
IMPORTANT: In some cases, the file permissions are not set correctly by the pip installation process. To make sure the installed files have the correct file permissions, so that the are readable by your students, run
find ~/JHL_installations/Python -not -perm -g=rX -exec chmod g+rwX {} \; |
after having performed all your Python package installations. If you later decide to add another Python package, it is recommended to run the above 'find' command again to ensure the correct file permissions.
Installing additional R packages
To install an R package in the correct prefix, click 'New' and start an R notebook. In the notebook, use the install.packages command with a custom 'lib' argument to install in the ~/JHL_installations/R/ folder. For example, to install the 'metafor' package, one would run
install.packages("metafor", lib="~/JHL_installations/R/") |
IMPORTANT: In some cases, the file permissions are not set correctly by the R installation process. To make sure the installed files have the correct file permissions, so that the are readable by your students, open a Terminal (see instructions above) and run
find ~/JHL_installations/R -not -perm -g=rX -exec chmod g+rwX {} \; |
after having performed all your R package installations. If you later decideto add another R package, it is recommended to run the above 'find' command again to ensure the correct file permissions.
Advanced Topics
With the setup covered in the Basic Topics, most teachers will have enough flexibility to run a course according to their needs. The following section of advanced topics are intended for teachers who have a good knowledge of Linux systems, and who require the ability to further customize their Notebook environment. We support this in a limited way: we make sure that it is possible, e.g. by providing the JHL_hooks (see below), but can not help you set up your customized environment.
Logging in through SSH
If you are comfortable working with a terminal on Linux, you may want to connect to Lisa with a terminal through SSH, rather than through the Jupyter Hub / Jupyter Notebook interface. Instructions for connecting can be found in the general Lisa user manual. Typically, you'll just want to connect using
ssh <username>@lisa.surfsara.nl |
where your username is the same as your username for Jupyter Hub. More information can be found at https://userinfo.surfsara.nl/systems/lisa/user-guide/connecting-and-transferring-data#connecting
Downloading/uploading large numbers of files
The quickest way to upload or download a large number of files is by connecting to Lisa through an FTP client. This is explained in the Lisa documentation: https://userinfo.surfsara.nl/systems/lisa/user-guide/connecting-and-transferring-data#sftp
Using customized hooks
The JHL_hooks folder contains two hooks: one called env and one called script.
The environment hook
The env hook can be used to change the environment and is sourced (sourcing is the way Linux systems change the current environment) whenever you or a student starts a Jupyter Notebook server. The default env hook will load a number of so-called modules. Modules are the way that on Lisa, we can offer multiple versions of the same software. By default, we load the modules needed to run the Jupyter Notebook servers, and support a Python 3 and R kernel. At this time, these are the following modules:
module load 2019 |
Unless you really know what you are doing, we don't advise you to change these default modules or any of the environment variables that are set in the default env script. You can however add to it. For example, if you happen to be familiar with the module environment on Lisa, you can leverage the env hook to load any additional module you like. Do make sure that they are compatible with the modules that provide the Jupyterhub and R kernel (i.e. for the modules above, that means the foss-2019b, but please check your current env script to see what the current modules are that are loaded by default).
Additionally, you can set environment variables in this script. For example, the default script sets
PYTHONPATH=/home/<teacherlogin>/JHL_installations/Python/lib/python${PYTHON_MAJ_MIN}/site-packages:$PYTHONPATH |
This enables the default functionality that new Python / R packages installed in the corresponding prefixes are automatically found by Python / R. You can expand on these if you feel the need to install Python or R packages in other, non-default locations, but be aware that only the JHL_* directories are by default readable to your students.
Of course, you can also use the env hook to set any other environment variables here that you might need.
IMPORTANT: note that changes to the env hook only take effect after you've restarted the Jupyter Notebook Server.
The script hook
The script hook is executed whenever you or a student starts a Jupyter Nobeook server. The default script hook takes care of synchronizing the JHL_notebooks folder. Again, we advise to leave the default lines untouched, but your are free to add additional commands that you would like to get executed at startup. There are many possible uses, such as synchronizing additional folders, enabling notebook extensions, etc. A sample script might look like:
#!/bin/bash |
IMPORTANT: note that changes to the script hook only take effect after you've restarted the Jupyter Notebook Server.
Installing additional software
You may want to install additional software. As an example, let us install the 'ncdu' program (https://dev.yorhel.nl/ncdu), a utility to analyse disk usage on Linux systems. Here, we are again assuming your are in a terminal session - either logged in through SSH or by starting a terminal from the Jupyter Notebook server.
In this example, we start in the home directory, create an ncdu subfolder in the JHL_installations directory, download the sources, and then install the ncdu program in the ~/JHL_installations/ncdu prefix:
cd |
Now that we have installed the ncdu software, we need to make sure it is found by setting the correct PATH environment variable. We do this through the env hook, so that if our students start a notebook, they can run the ncdu executable. Thus, we add the following line to the env script:
PATH=/home/lcurs000/JHL_installations/ncdu/bin:$PATH |
After these steps, if you or the students start a new Jupyter Notebook server, the ncdu executable will be available. For example, if in the Jupyter Notebook environment we go to 'New' and start a 'Terminal', we can execute the ncdu command:
Note that you can also execute commands on the command line from e.g. a Python notebook by starting a command with an exclamation mark (!)
Using Python virtual environments
You can use Python virtual environments through Jupyter Notebooks, but a couple of steps are needed.
- Create a virtual environment in the JHL_installations directory
- Install the package 'ipykernel' in the virtual environment
- Install python packages in the virtual environment (as you normally would)
- Set Unix permissions so that files/folders within the virtual environment are group readable
- Create a hook that installs the ipython kernel for each student, so that your Jupyter Notebook server will find it
After these steps, you will see the Python virtual environment as an additional item if you go the the 'New' menu in a Jupyter Notebook server.
There are two possibilities for creating a virtual environment: either you build upon the python packages already available on the system
virtualenv --system-site-packages ~/JHL_installations/venvs/my_env |
or you start with a completely clean environment
virtualenv ~/JHL_installations/venvs/my_env |
The first approach will allow you to e.g. still use the system numpy installation, which is well optimized for our CPUs. The second approach however allows you more flexibility and control over the environment, since the only packages that will be available are the ones you install.
Example
In this example, we will install a Python virtual environment with the 'values' Python package. Again, for this example, we assume you are in a terminal session either logged in through SSH or by starting a terminal from the Jupyter Notebook server. In this example, we installed our environment in ~/JHL_installations/venvs/my_env, but any subdirectory of ~/JHL_installations will do. If you want to create multiple virtual environments, you'll need to put each of them in their own subdirectory.
With --system-site-packages
When installing with --system-site-packages, we need to pass the force flag to the ipykernel install, in order to force the ipython kernel to be (re)installed in the virtual environment.
mkdir ~/JHL_installations/venvs |
Without --system-site-packages
When not using any of the system packages, the force flag can be ommitted for the ipykernel installation:
mkdir ~/JHL_installations/venvs |
Create hook that installs the kernel for each student
Then, we add to the script hook:
module purge |
IMPORTANT: we purge all modules here, because otherwise the ipykernel install will pick up on the global installation of ipykernel and fail to install the local kernel spec. Since purging temporarily changes the environment, we advise to put this section of code all the way at the bottom of the script hook, so that other parts of the script hook are not affected by the purge.
Note that as '--name' you can provide whatever you feel is most descriptive. Now, this virtual environment shows up as a seperate kernel in the 'New' menu in a Jupyter Notebook (you may have to reload your Jupyter Notebook web page):
If we start the 'my_env' kernel, we will be able to import the 'values' Python package.