h1. User instructions for Dione cluster

University of Turku

Åbo Akademi

Jussi Salmi (jussi.salmi@utu.fi)

h2. 1. Resources

h3. 1.1. Computation nodes

<pre>

PARTITION NODES NODELIST  MEMORY

normal    36    di[1-36]  192GB

gpu       6     di[37-42] 384GB

</pre>

Dione has 6 GPU-nodes where the user can perform calculation which benefits from very fast and parallel number crunching. This includes e.g. neural nets. The 36 other nodes are general purpose processors. Please use 'normal' or 'gpu' partitions when starting jobs to keep the gpu resources separate. The nodes are connected via a fast network, Infiniband, enabling MPI (Message Passing Interface) usage in the cluster. In addition, the cluster is connected to the EGI-grid (European Grid Infrastructure) and NORDUGRID which are allowed to use a part of the computational resources. The website

https://p55cc.utu.fi/

Contains information on the cluster, a cluster monitor and provides instructions on getting access and using the cluster.

h3. 1.2. Disk space

The system has an NFS4 file system with 100TB capacity on the home partition. The system is not backed up anywhere, so the user must handle backups himself/herself.

h3. 1.3. Software

The system uses the SLURM workload manager (Simple Linux Utility for Resource Management) for scheduling the jobs.

The cluster uses the module-system for loading software modules with different version for execution.

h2. 2. Executing jobs in the cluster

The user may not execute jobs on the login node. All jobs must be dispatched to the cluster by using SLURM commands. Normally a script is used to define the jobs and the parameters for SLURM. There is a large number of parameters and environment variables that can be used to define how the jobs should be executed, please look at the SLURM manual for a complete list.

A typical script for starting the jobs can look as follows (name:batch-submit.job):

<pre>

#!/bin/bash

#SBATCH --job-name=test

#SBATCH -o result.txt

#SBATCH --workdir=<Workdir path>

#SBATCH -c 1

#SBATCH -t 10:00

#SBATCH --mem=10M

#SBATCH --partition=all

module purge # Purge modules for a clean start

module load <desired modules if needed> # You can either inherit module environment, or insert one here

srun <executable>

srun sleep 60

</pre>

The script is run with

sbatch batch-submit.job

The script defines several parameters that will be used for the job.

<pre>

--job-name          defines the name

-o result.txt       redirects the standard output to results.txt

--workdir           defines the working directory

-c 1                sets the number of cpus per task to 1

-t 10:00            the time limit of the task is set to 10 minutes. After that the process is stopped

--mem=10M           the memory required for the task is 10MB.

--partition=normal  use the 'normal' partition. Please use 'normal' or 'gpu' partitions to keep the gpu resources separate. 'all' uses all partitions.

</pre>

srun starts a task. When starting the task SLURM gives it a job id which can be used to track it’s execution with e.g. the squeue command.

h2. 3. The module system

Many of the software packages in Dione require you to load the kernel modules prior to using the software. Different versions of the software can be used with module.

<pre>

module avail Show available modules

module list Show loaded modules

module unload <module> Unload a module

module load <module> Load a module

module load <module>/10.0 Load version 10.0 of <module>

module purge unload all modules

</pre>

h2. 4. Useful commands in SLURM

<pre>

sinfo shows the current status of the cluster.

sinfo -p gpu Shows the status of the GPU-partition

sinfo -O all Shows a comprehensive status report node per node

sstat <job id> Shows information on your job

squeue The status of the job queue

squeue -u <username> Show only your jobs

srun <command> Dispatch jobs to the scheduler

sbatch <script> Run a script defining jobs to be run

scontrol Control your jobs in many aspects

scontrol show job <job id> Show details about the job

scontrol -u <username> Show only a certain users jobs

scancel <job id> Cancel a job

scancel -u <username> Cancel all your jobs

</pre>

h2. 5. Further information

Further information can be asked from the administrators (fgi-admins@lists.utu.fi).