|Snellius under construction at the Amsterdam Science Park|
After a very successful 8 years of service, Cartesius will be replaced this year by a new Dutch National Supercomputer called Snellius.
Snellius will be built by Lenovo and will contain predominantly AMD technology, plus NVIDIA GPUs. A high-level news item on the new system can be read here.
This page will be the main source of information where we will keep you as user updated on the progress of the installation of Snellius and the corresponding transition from Cartesius to Snellius. The purpose of this page is to help you adapt your research schedule to take into account the upcoming transition.
The details of the system configuration and the transition schedule are currently still being decided.
This page will be continuously updated while Snellius is being prepared to go into production. Please check here regularly for new or updated information on the migration.
What is the timeline for this transition?
The hardware of Snellius will be delivered and installed in tranches. The first hardware has already been delivered to the data center at Amsterdam Science Park and is currently being installed next to Cartesius.
The projected start of production of Snellius will be September 1st. We expect this date will also be the end date of service for Cartesius.
Of course, these dates are only valid when everything goes according to plan and no major delays are encountered in the delivery, installation, setup and acceptance of the system.
What does the Snellius system look like?
Like Cartesius, Snellius will also be a heterogeneous system, with thin nodes, fat nodes, high-memory nodes and a number of nodes with GPU accelerators. Snellius will be delivered in several phases, so the growth of Snellius will follow the anticipated growth in usage of Snellius. The growth phases are as follows.
Phase 1 (Q3 2021)
The hardware installed in this phase provides a peak compute performance of 3.2 PFLOP/s (CPU) + 2.8 PFLOP/s (GPU).
|Type||Amount||Technical Details||Memory/core (GiB)||Total #cores|
AMD Rome 7H12@2.6GHz processor, dual socket,
|fat nodes||72||AMD Rome 7H12@2.6GHz processor, dual socket, |
|high-memory nodes||2||AMD Rome 7H12@2.6GHz processor, dual socket, |
|high-memory nodes||2||AMD Rome 7H12@2.6GHz processor, dual socket, |
Intel Ice Lake 8360Y@2.4GHz processor, dual socket,
4x NVIDIA A100 Tensor Core GPU
Phase 2 (Q3 2022)
An extension will be added with more CPU-only thin nodes (future generation AMD EPYC processors, 2 GB per core), with a peak performance of 5.1 PFLOP/s.
Phase 3 (Q3 2023)
There are three options for this extension:
- CPU thin nodes (same future generation AMD EPYC processors, aggregate: 2.4 PFLOP/s), or
- GPU nodes (future generation NVIDIA GPUs, aggregate: 10.3 PFLOP/s), or
- Storage (the amount still needs to be determined)
The choice will be made 1.5 years after the start of production of Phase 1 and will be based on actual usage and demand of the system.
When Phase 3 is complete Snellius will have a total performance (CPU+GPU) in the range 13.6 - 21.5 PFLOP/s. This corresponds to roughly 7 - 11.5 times the total peak compute performance of Cartesius.
I still have an active account on Cartesius, what will happen with this account and the budget?
Active accounts will be migrated to Snellius, with a cutoff date for being active to be published soon. The remains of the budget which you had on Cartesius will be transferred to Snellius. A conversion factor from Cartesius budget to Snellius budget has yet to be determined.
Newly granted projects, that have a start date after 1 August 2021 will receive new accounts only on Snellius. We will not create a Cartesius account for granted projects which start after 1 August 2021. If the new grant is a continuation of a previous project, the new budget will be accessible on Cartesius along with an already existing account.
How do I get all my data from Cartesius to Snellius?
SURF will migrate relevant user data, the exact data migration schedule and included set of data will be published here soon.
In general, the data migration includes the following:
/home/<user>from accounts that were active on, or after, the cutoff date mentioned above
- Project spaces that have a (co-)owner that has a valid login, again dependent on being active on a cutoff date as defined above
- Tape archive does not need to be migrated and will still be accessible from Snellius
All data that is not included in the above will not be migrated to Snellius and will become unavailable as soon as Cartesius is taken offline.
Files that reside on scratch filesystems of Cartesius will not be migrated to Snellius
When is the data migrated? What is the Freeze Date?
Somewhere in the second half of August (the exact date is yet to be published), the project spaces on Cartesius will be frozen and migrated to Snellius. For the time being we will refer to this date as the Freeze Date. We will not do any incremental migration of files on project spaces that were generated after this Freeze Date.
Data cleanup and preparation by users
To keep the data migration from Cartesius to Snellius to manageable proportions, we kindly ask the cooperation of all users.
Minimize the content of your
/home directories and of your project space(s) as much as possible:
- Clean up your
/homedirectory and project space as much as possible.
- Remove obsolete files and directories.
- Move files from project space, that you would have transferred to local storage anyway, as soon as possible to this local storage.
- If you have access to the tape archive, please compress and move all data to the tape archive that you will not immediately need after migration. You can restore them at a later time on Snellius from the tape archive.
- Make sure that you don't have links in your
/homefolder that reference storage outside the
/homefolder, as these links will be broken after the migration to Snellius.
- Make sure that you move all legacy files from Huygens that you want to keep outside of the
~/HUYGENSlocation, i.e. copy the contents of
~/HUYGENSto a different location in your home directory.
- If you intend to upload large input datasets to project space, consider postponing this operation until the transition to Snellius is complete.
How can I get access to Snellius?
The procedure for obtaining access to Snellius will be similar to the one for Cartesius. For large applications, you have to submit a proposal through isaac.nwo.nl and for small, pilot, applications you can apply via the SURF access portal (more information on the latter can be found here). Of course, the definitions of what is considered big and small will be adapted to reflect the increased capacity of Snellius.
How does the Snellius software environment compare to the Cartesius environment?
Snellius will use the same type of modules environment for providing software packages as used on Cartesius. We will do our best to port the software from the 2020 modules environment on Cartesius to the new 2021 modules environment on Snellius. This implies that Snellius will host newer versions of the software that is currently available on Cartesius.
We have already started building a new 2021 modules environment on Cartesius. This environment currently contains the most important core libraries and development tools. We will continue to enhance the 2021 modules-environment on Cartesius by frequently adding more packages. Please note, the 2021 modules environment on Cartesius is subject to changes. This environment is opened to users for testing only. Thus, users can already try to rebuild their software on Cartesius, test new versions of libraries, and adapt their current workflow to the upcoming new system. This can reduce the effort of setting up your environment on Snellius after the migration.
On Snellius we will use a GNU-based toolchain (foss) as the main one. However, for advanced users, we will provide a complete intel toolchain as a module in the new 2021 modules-environment.
2019 and pre2019 module environments
We will not migrate the 2019 and pre2019 modules environments to Snellius.
Custom built software and locally-installed modules
Users that use in-house developed software, or more generally that build software themselves on Cartesius, will have to rebuild that software on Snellius. The same applies for locally installed modules, where you will have to rebuild these modules on Snellius.
To facilitate the transition from Cartesius to Snellius for users who have locally installed modules, we intend to install the intel/2020a and foss/2020a toolchains on Snellius. Please note, we will provide these toolchains only for compatibility with the previous modules-environment on Cartesius. We will not install any software system-wide on Snellius using these toolchains.
Similarities and Differences between Cartesius and Snellius
|Intel Sandy Bridge, Ivy Bridge, Haswell, Broadwell, Knights Landing||AMD 7H12 (Rome), 64 cores/socket, 2.6GHz|
|GPU architecture||NVIDIA Kepler - K40||NVIDIA Ampere - A100|
|Node types||thin nodes, fat nodes, GPU nodes||thin nodes, fat nodes, high-memory nodes, GPU nodes|
|Number of nodes (total cores/GPUs)|
Thin nodes: 1620 (38,880 cores)
Broadwell nodes: 177 (5,664 cores)
Fat nodes: 32 (1,024 cores)
GPU nodes: 66 (132 GPUs, 1,056 cores)
Thin nodes: 504 (64,512 cores)
Fat nodes: 72 (9,216 cores)
High-memory nodes: 4 (512 cores)
GPU nodes: 36 (144 GPUs, 2,592 cores)
|Cores per node|
Thin nodes: 24 (2S x 12 cores/socket)
Broadwell nodes: 32 (2S x 16 cores/socket)
Fat nodes: 32 (4S x 8 cores/socket)
GPU nodes: 2 GPUs/node + 16 CPU cores (2S x 8 cores/socket)
Thin nodes: 128 (2S x 64 cores/socket)
Fat nodes: 128 (2S x 64 cores/socket)
High-memory nodes: 128 (2S x 64 cores/socket)
GPU nodes: 4 GPUs/node + 72 CPU cores (2S x 36 cores/socket
|Memory per node|
Thin nodes: 64 GB (2.66 GB/core)
Broadwell nodes: 64 GB (2GB/core)
Fat nodes: 256 GB (8 GB/core)
GPU nodes: 96GB (6GB/node, 48 GB/GPU)
Thin nodes: 256GB (2 GB/core)
Fat nodes: 1TB (8GB/core)
High-memory nodes: 4/8 TB (32-64GB/core)
GPU nodes: 512GB (7.11 GB/core, 128 GB/GPU)
|Interconnect||Infiniband FDR (56Gbps), pruned||Infiniband HDR100 (100Gbps), fat tree|
Home filesystem: 180TB
Parallel filesystem: Lustre, 7.7PB
Home filesystem: 720 TB
Parallel filesystem: Spectrum Scale (GPFS), 12.4PB
NVMe parallel filesystem: Spectrum Scale (GPFS), 200TB
Fat nodes include 6.4 TB of NVMe local storage
Software and usage characteristics
|Node usage||Exclusive (jobs take full nodes)||Shared (jobs can share nodes) and exclusive|
|Operating system||CentOS7||CentOS8/Rocky Linux/RHEL8|
|Supported compiler suites||Intel, GNU, PGI (NVIDIA)||Intel, GNU, PGI (NVIDIA), AMD|
|Accounting||1 SBU/core-hour||1 SBU/core-hour|
|Pilot application limits||500,000 SBUs||1,000,000 SBUs (tentatively)|
|Modules environment||pre2019, 2019, 2020, (part of) 2021||2021|
|Supported toolchains||foss and intel; versions 2018b and 2020a|
foss and intel; versions 2020a and 2021a
|Toolchains used for the software installation||foss, intel||foss|