This document primarily presents a quick start guide to the usage of the Gina Cody
+ This document primarily presents a quick start guide to the usage of the Gina Cody
School of Engineering and Computer Science compute server farm called “Speed” – the
GCS ENCS Speed cluster, managed by HPC/NAG of GCS ENCS, Concordia University,
Montreal, Canada.
@@ -39,70 +39,76 @@ This document contains basic information required to use “Speed” as well as tips and tricks,
+ This document contains basic information required to use “Speed” as well as tips and tricks,
examples, and references to projects and papers that have used Speed. User contributions of sample
jobs and/or references are welcome. Details are sent to the hpc-ml mailing list.
-
+
+
We receive support from the rest of AITS teams, such as NAG, SAG, FIS, and DOG.
-
+ We receive support from the rest of AITS teams, such as NAG, SAG, FIS, and DOG.
+
-
+ Prepare them for big clusters: Prepare them for big clusters:
-
+
+
We have a great number of open-source software available and installed on Speed – various Python,
+ We have a great number of open-source software available and installed on Speed – various Python,
CUDA versions, C++/Java compilers, OpenGL, OpenFOAM, OpenCV, TensorFlow, OpenMPI,
-OpenISS, MARF [15], etc. There are also a number of commercial packages, subject to
-licensing contributions, available, such as MATLAB [5, 14], Abaqus [1], Ansys, Fluent [2],
+OpenISS, MARF [18], etc. There are also a number of commercial packages, subject to
+licensing contributions, available, such as MATLAB [7, 17], Abaqus [1], Ansys, Fluent [2],
etc.
- To see the packages available, run ls -al /encs/pkg/ on speed.encs.
- In particular, there are over 2200 programs available in /encs/bin and /encs/pkg under Scientific
+ To see the packages available, run ls -al /encs/pkg/ on speed.encs.
+ In particular, there are over 2200 programs available in /encs/bin and /encs/pkg under Scientific
Linux 7 (EL7).
Popular concrete examples: Popular concrete examples: Popular examples mentioned (loaded with, module): Popular examples mentioned (loaded with, module):
+
+
+
+
After reviewing the “What Speed is” (Section 1.4) and “What Speed is Not” (Section 1.5), request
+ After reviewing the “What Speed is” (Section 1.4) and “What Speed is Not” (Section 1.5), request
access to the “Speed” cluster by emailing: rt-ex-hpc AT encs.concordia.ca. Faculty
and staff may request the access directly. Students must include the following in their
message:
-
-
-
+
In these instructions, anything bracketed like so, <>, indicates a label/value to be replaced (the entire
+ In these instructions, anything bracketed like so, <>, indicates a label/value to be replaced (the entire
bracketed term needs replacement).
-
+
Before getting started, please review the “What Speed is” (Section 1.4) and “What Speed is Not”
+ Before getting started, please review the “What Speed is” (Section 1.4) and “What Speed is Not”
(Section 1.5). Once your GCS ENCS account has been granted access to “Speed”, use
your GCS ENCS account credentials to create an SSH connection to speed (an alias for
speed-submit.encs.concordia.ca).
-
+
Requirements to create connections to Speed:
+ Requirements to create connections to Speed:
+
+
+
Open up a terminal window and type in the following SSH command being sure to replace
+ Open up a terminal window and type in the following SSH command being sure to replace
<ENCSusername> with your ENCS account’s username.
@@ -291,18 +298,18 @@
- All users are expected to have a basic understanding of Linux and its commonly used
+
+ All users are expected to have a basic understanding of Linux and its commonly used
commands.
-
+
After creating an SSH connection to “Speed”, you will need to source the “Altair Grid Engine
+ After creating an SSH connection to “Speed”, you will need to source the “Altair Grid Engine
(AGE)” scheduler’s settings file. Sourcing the settings file will set the environment variables required
to execute scheduler commands.
- Based on the UNIX shell type, choose one of the following commands to source the settings
+ Based on the UNIX shell type, choose one of the following commands to source the settings
file.
- csh/tcsh:
+ csh/tcsh:
@@ -310,8 +317,8 @@
- Bourne shell/bash:
+
+ Bourne shell/bash:
@@ -319,8 +326,8 @@
- In order to set up the default ENCS bash shell, executing the following command is also
+
+ In order to set up the default ENCS bash shell, executing the following command is also
required:
@@ -329,10 +336,10 @@
- To verify that you have access to the scheduler commands execute qstat -f -u "*". If an error is
+
+ To verify that you have access to the scheduler commands execute qstat -f -u "*". If an error is
returned, attempt sourcing the settings file again.
- The next step is to copy a job template to your home directory and to set up your cluster-specific
+ The next step is to copy a job template to your home directory and to set up your cluster-specific
storage. Execute the following command from within your home directory. (To move to your home
directory, type cd at the Linux prompt and press Enter.)
@@ -342,15 +349,15 @@
- Tip: Add the source command to your shell-startup script.
- Tip: the default shell for GCS ENCS users is tcsh. If you would like to use bash, please contact
+
+ Tip: Add the source command to your shell-startup script.
+ Tip: the default shell for GCS ENCS users is tcsh. If you would like to use bash, please contact
rt-ex-hpc AT encs.concordia.ca.
- For new ENCS Users, and/or those who don’t have a shell-startup script, based on your shell
+ For new ENCS Users, and/or those who don’t have a shell-startup script, based on your shell
type use one of the following commands to copy a start up script from nul-uge’s. home directory to
your home directory. (To move to your home directory, type cd at the Linux prompt and press
Enter.)
- csh/tcsh:
+ csh/tcsh:
@@ -358,8 +365,8 @@
- Bourne shell/bash:
+
+ Bourne shell/bash:
@@ -367,11 +374,11 @@
- Users who already have a shell-startup script, use a text editor, such as vim or emacs, to add the
+
+ Users who already have a shell-startup script, use a text editor, such as vim or emacs, to add the
source request to your existing shell-startup environment (i.e., to the .tcshrc file in your home
directory).
- csh/tcsh: Sample .tcshrc file:
+ csh/tcsh: Sample .tcshrc file:
@@ -382,8 +389,8 @@
- Bourne shell/bash: Sample .bashrc file:
+
+ Bourne shell/bash: Sample .bashrc file:
@@ -395,34 +402,34 @@
- Note that you will need to either log out and back in, or execute a new shell, for the environment
+
+ Note that you will need to either log out and back in, or execute a new shell, for the environment
changes in the updated .tcshrc or .bashrc file to be applied (important).
-
+
Preparing your job for submission is fairly straightforward. Editing a copy of the template.sh you
+ Preparing your job for submission is fairly straightforward. Editing a copy of the template.sh you
moved into your home directory during Section 2.1.2 is a good place to start. You can also use a job
script example from our GitHub’s (https://github.com/NAG-DevOps/speed-hpc) “src” directory
and base your job on it.
- Job scripts are broken into four main sections:
Contents
1.1 Resources
-
1.2 Team
-
1.3 What Speed Comprises
-
1.4 What Speed Is Ideal For
-
1.5 What Speed Is Not
-
1.6 Available Software
-
1.7 Requesting Access
-
2 Job Management
-
2.1 Getting Started
-
2.1.1 SSH Connections
-
2.1.2 Environment Set Up
-
2.2 Job Submission Basics
-
2.2.1 Directives
-
2.2.2 Module Loads
-
2.2.3 User Scripting
-
2.3 Sample Job Script
-
2.4 Common Job Management Commands Summary
-
2.5 Advanced qsub Options
-
2.6 Array Jobs
-
-
-
-
2.7 Requesting Multiple Cores (i.e., Multithreading Jobs)
-
2.8 Interactive Jobs
-
2.9 Scheduler Environment Variables
-
2.10 SSH Keys For MPI
-
2.11 Creating Virtual Environments
-
2.11.1 Anaconda
-
2.12 Example Job Script: Fluent
-
2.13 Example Job: efficientdet
-
2.14 Java Jobs
-
2.15 Scheduling On The GPU Nodes
-
2.15.1 CUDA
-
2.15.2 Special Notes for sending CUDA jobs to the GPU Queue
-
3 Conclusion
-
3.1 Important Limitations
-
3.2 Tips/Tricks
-
3.3 Use Cases
-
A History
-
A.1 Acknowledgments
-
A.2 Phase 3
-
A.3 Phase 2
-
A.4 Phase 1
-
B Frequently Asked Questions
-
B.1 How to use the “bash shell” on Speed?
-
B.1.1 How do I set bash as my login shell?
-
B.1.2 How do I move into a bash shell on Speed?
-
B.1.3 How do I run scripts written in bash on Speed?
-
B.2 How to resolve“Disk quota exceeded” errors?
-
B.2.1 Probably Cause
-
B.2.2 Possible Solutions
-
B.2.3 Example of setting working directories for COMSOL
-
C Sister Facilities
-
Annotated Bibliography
+ 1 Introduction
+
1.1 Resources
+
1.2 Team
+
1.3 What Speed Comprises
+
1.4 What Speed Is Ideal For
+
1.5 What Speed Is Not
+
1.6 Available Software
+
1.7 Requesting Access
+
2 Job Management
+
2.1 Getting Started
+
2.1.1 SSH Connections
+
2.1.2 Environment Set Up
+
2.2 Job Submission Basics
+
2.2.1 Directives
+
2.2.2 Module Loads
+
2.2.3 User Scripting
+
2.3 Sample Job Script
+
2.4 Common Job Management Commands Summary
+
2.5 Advanced qsub Options
+
2.6 Array Jobs
+
+
+
+
2.7 Requesting Multiple Cores (i.e., Multithreading Jobs)
+
2.8 Interactive Jobs
+
2.9 Scheduler Environment Variables
+
2.10 SSH Keys For MPI
+
2.11 Creating Virtual Environments
+
2.11.1 Anaconda
+
2.12 Example Job Script: Fluent
+
2.13 Example Job: efficientdet
+
2.14 Java Jobs
+
2.15 Scheduling On The GPU Nodes
+
2.15.1 CUDA
+
2.15.2 Special Notes for sending CUDA jobs to the GPU Queue
+
3 Conclusion
+
3.1 Important Limitations
+
3.2 Tips/Tricks
+
3.3 Use Cases
+
A History
+
A.1 Acknowledgments
+
A.2 Phase 3
+
A.3 Phase 2
+
A.4 Phase 1
+
B Frequently Asked Questions
+
B.1 Where do I learn about Linux?
+
B.2 How to use the “bash shell” on Speed?
+
B.2.1 How do I set bash as my login shell?
+
B.2.2 How do I move into a bash shell on Speed?
+
B.2.3 How do I run scripts written in bash on Speed?
+
B.3 How to resolve“Disk quota exceeded” errors?
+
B.3.1 Probable Cause
+
B.3.2 Possible Solutions
+
B.3.3 Example of setting working directories for COMSOL
+
B.3.4 Example of setting working directories for Python Modules
+
B.4 How do I check my job’s status?
+
B.5 Why is my job pending when nodes are empty?
+
B.5.1 Disabled nodes
+
B.5.2 Error in job submit request.
+
C Sister Facilities
+
Annotated Bibliography
1 Introduction
-1.1 Resources
@@ -115,16 +121,16 @@
-1.1
1.2 Team
-
-1.2
1.3 What Speed Comprises
-
-1.4 What Speed Is Ideal For
+ 1.4 What Speed Is Ideal For
-
-
1.5 What Speed Is Not
+1.5 What Speed Is Not
-1.5
1.6 Available Software
-
-
@@ -224,7 +231,7 @@ 1.6
1.6
1.7 Requesting Access
-
-1.7
2 Job Management
-2.1 Getting Started
-2.1.1 SSH Connections
-
-2.1.1
ssh <ENCSusername>@speed.encs.concordia.ca
-2.1.2 Environment Set Up
-2.
source /local/pkg/uge-8.6.3/root/default/common/settings.csh
-2.
. /local/pkg/uge-8.6.3/root/default/common/settings.sh
-2.
printenv ORGANIZATION | grep -qw ENCS || . /encs/Share/bash/profile
-2.
cp /home/n/nul-uge/template.sh . && mkdir /speed-scratch/$USER
-2.
cp /home/n/nul-uge/.tcshrc .
-2.
cp /home/n/nul-uge/.bashrc .
-2.
source /local/pkg/uge-8.6.3/root/default/common/settings.csh
endif
-
2.
printenv ORGANIZATION | grep -qw ENCS || . /encs/Share/bash/profile
fi
-
2.2 Job Submission Basics
-
Job scripts are broken into four main sections:
+
Directives are comments included at the beginning of a job script that set the shell and the options for +
Directives are comments included at the beginning of a job script that set the shell and the options for the job scheduler. -
The shebang directive is always the first line of a script. In your job script, this directive sets +
The shebang directive is always the first line of a script. In your job script, this directive sets which shell your script’s commands will run in. On “Speed”, we recommend that your script use a shell from the /encs/bin directory. -
To use the tcsh shell, start your script with: #!/encs/bin/tcsh -
For bash, start with: #!/encs/bin/bash -
Directives that start with "#$", set the options for the cluster’s “Altair Grid Engine (AGE)” +
To use the tcsh shell, start your script with: #!/encs/bin/tcsh +
For bash, start with: #!/encs/bin/bash +
Directives that start with "#$", set the options for the cluster’s “Altair Grid Engine (AGE)” scheduler. The script template, template.sh, provides the essentials: @@ -435,15 +442,15 @@
-
Replace, <jobname>, with the name that you want your cluster job to have; -cwd, makes the +
+
Replace, <jobname>, with the name that you want your cluster job to have; -cwd, makes the current working directory the “job working directory”, and your standard output file will appear here; -m bea, provides e-mail notifications (begin/end/abort); replace, <corecount>, with the degree of (multithreaded) parallelism (i.e., cores) you attach to your job (up to 32), be sure to delete or comment out the #$ -pe smp parameter if it is not relevant; replace, <memory>, with the value (in GB), that you want your job’s memory space to be (up to 500), and all jobs MUST have a memory-space assignment. -
If you are unsure about memory footprints, err on assigning a generous memory space to your job +
If you are unsure about memory footprints, err on assigning a generous memory space to your job so that it does not get prematurely terminated (the value given to h_vmem is a hard memory ceiling). You can refine h_vmem values for future jobs by monitoring the size of a job’s active memory space on speed-submit with: @@ -454,17 +461,17 @@
-
Memory-footprint values are also provided for completed jobs in the final e-mail notification (as, +
+
Memory-footprint values are also provided for completed jobs in the final e-mail notification (as, “Max vmem”). -
Jobs that request a low-memory footprint are more likely to load on a busy cluster. -
+
Jobs that request a low-memory footprint are more likely to load on a busy cluster. +
As your job will run on a compute or GPU “Speed” node, and not the submit node, any software that +
As your job will run on a compute or GPU “Speed” node, and not the submit node, any software that is needed must be loaded by the job script. Software is loaded within the script just as it would be from the command line. -
To see a list of which modules are available, execute the following from the command line on +
To see a list of which modules are available, execute the following from the command line on speed-submit. @@ -473,8 +480,8 @@
module avail-
-
To list for a particular program (matlab, for example): +
+
To list for a particular program (matlab, for example): @@ -482,8 +489,8 @@
module -t avail matlab-
-
Which, of course, can be shortened to match all that start with a particular letter: +
+
Which, of course, can be shortened to match all that start with a particular letter: @@ -491,8 +498,8 @@
module -t avail m-
-
Insert the following in your script to load the matlab/R2020a) module: +
+
Insert the following in your script to load the matlab/R2020a) module: @@ -500,9 +507,9 @@
module load matlab/R2020a/default-
-
Use, unload, in place of, load, to remove a module from active use. -
To list loaded modules: +
+
Use, unload, in place of, load, to remove a module from active use. +
To list loaded modules: @@ -510,8 +517,8 @@
module list-
-
To purge all software in your working environment: +
+
To purge all software in your working environment: @@ -519,28 +526,28 @@
module purge-
-
Typically, only the module load command will be used in your script. -
+
+
Typically, only the module load command will be used in your script. +
The last part the job script is the scripting that will be executed by the job. This part of +
The last part the job script is the scripting that will be executed by the job. This part of the job script includes all commands required to set up and execute the task your script has been written to do. Any Linux command can be used at this step. This section can be a simple call to an executable or a complex loop which iterates through a series of commands. -
Every software program has a unique execution framework. It is the responsibility of the script’s +
Every software program has a unique execution framework. It is the responsibility of the script’s author (e.g., you) to know what is required for the software used in your script by reviewing the software’s documentation. Regardless of which software your script calls, your script should be written so that the software knows the location of the input and output files as well as the degree of parallelism. Note that the cluster-specific environment variable, NSLOTS, resolves to the value provided to the scheduler in the -pe smp option. -
Jobs which touch data-input and data-output files more than once, should make use of TMPDIR, a +
Jobs which touch data-input and data-output files more than once, should make use of TMPDIR, a scheduler-provided working space almost 1 TB in size. TMPDIR is created when a job starts, and exists on the local disk of the compute node executing your job. Using TMPDIR results in faster I/O operations than those to and from shared storage (which is provided over NFS). -
An sample job script using TMPDIR is available at /home/n/nul-uge/templateTMPDIR.sh: the job +
An sample job script using TMPDIR is available at /home/n/nul-uge/templateTMPDIR.sh: the job is instructed to change to $TMPDIR, to make the new directory input, to copy data from $SGE_O_WORKDIR/references/ to input/ ($SGE_O_WORKDIR represents the current working directory), to make the new directory results, to execute the program (which takes input from @@ -548,10 +555,10 @@
+
Now, let’s look at a basic job script, tcsh.sh in Figure 1 (you can copy it from our GitHub page or +
Now, let’s look at a basic job script, tcsh.sh in Figure 1 (you can copy it from our GitHub page or from /home/n/nul-uge).
#!/encs/bin/tcsh +#!/encs/bin/tcsh #$ -N qsub-test #$ -cwd @@ -578,7 +585,7 @@2.3
The first line is the shell declaration (also know as a shebang) and sets the shell to tcsh. The lines +
The first line is the shell declaration (also know as a shebang) and sets the shell to tcsh. The lines that begin with #$ are directives for the scheduler.
The script then: +
The script then:
The scheduler command, qsub, is used to submit (non-interactive) jobs. From an ssh session on +
The scheduler command, qsub, is used to submit (non-interactive) jobs. From an ssh session on speed-submit, submit this job with qsub ./tcsh.sh. You will see, "Your job X ("qsub-test") has been submitted". The command, qstat, can be used to look at the status of the cluster: qstat -f -u "*". You will see something like this: @@ -652,25 +659,25 @@
+
-
Remember that you only have 30 seconds before the job is essentially over, so if you do not see a +
Remember that you only have 30 seconds before the job is essentially over, so if you do not see a similar output, either adjust the sleep time in the script, or execute the qstat statement more quickly. The qstat output listed above shows you that your job is running on node speed-05, that it has a job number of 144, that it was started at 16:39:30 on 12/03/2018, and that it is a single-core job (the default). -
Once the job finishes, there will be a new file in the directory that the job was started from, with +
Once the job finishes, there will be a new file in the directory that the job was started from, with the syntax of, "job name".o"job number", so in this example the file is, qsub test.o144. This file represents the standard output (and error, if there is any) of the job in question. If you look at the contents of your newly created file, you will see that it contains the output of the, module list command. Important information is often written to this file. -
Congratulations on your first job! +
Congratulations on your first job!
Here are useful job-management commands: +
Here are useful job-management commands:
+
In addition to the basic qsub options presented earlier, there are a few additional options that are +
In addition to the basic qsub options presented earlier, there are a few additional options that are generally useful:
+
Array jobs are those that start a batch job or a parallel job multiple times. Each iteration of the job +
Array jobs are those that start a batch job or a parallel job multiple times. Each iteration of the job array is called a task and receives a unique job ID. -
To submit an array job, use the t option of the qsub command as follows: +
To submit an array job, use the t option of the qsub command as follows: @@ -737,15 +744,15 @@
qsub -t n[-m[:s]] <batch_script>-
-
-t Option Syntax:
++
-t Option Syntax:
Examples:
+Examples:
Output files for Array Jobs: -
The default and output and error-files are job_name.[o|e]job_id and
job_name.[o|e]job_id.task_id. This means that Speed creates an output and an error-file for each
+
Output files for Array Jobs: +
The default and output and error-files are job_name.[o|e]job_id and
job_name.[o|e]job_id.task_id. This means that Speed creates an output and an error-file for each
task generated by the array-job as well as one for the super-ordinate array-job. To alter this behavior
use the -o and -e option of qsub.
-
For more details about Array Job options, please review the manual pages for qsub by executing +
For more details about Array Job options, please review the manual pages for qsub by executing the following at the command line on speed-submit man qsub. -
+
For jobs that can take advantage of multiple machine cores, up to 32 cores (per job) can be requested +
For jobs that can take advantage of multiple machine cores, up to 32 cores (per job) can be requested in your script with: @@ -772,26 +779,26 @@
-
Do not request more cores than you think will be useful, as larger-core jobs +
+
Do not request more cores than you think will be useful, as larger-core jobs are more difficult to schedule. On the flip side, though, if you are going to be running a program that scales out to the maximum single-machine core count available, please (please) request 32 cores, to avoid node oversubscription (i.e., to avoid overloading the CPUs). -
Core count associated with a job appears under, “states”, in the, qstat -f -u "*", +
Core count associated with a job appears under, “states”, in the, qstat -f -u "*", output. -
+
Job sessions can be interactive, instead of batch (script) based. Such sessions can be useful for testing +
Job sessions can be interactive, instead of batch (script) based. Such sessions can be useful for testing and optimising code and resource requirements prior to batch submission. To request an interactive job session, use, qlogin [options], similarly to a qsub command-line job (e.g., qlogin -N qlogin-test -l h_vmem=1G). Note that the options that are available for qsub are not necessarily available for qlogin, notably, -cwd, and, -v. -
+
The scheduler presents a number of environment variables that can be used in your jobs. Three of the +
The scheduler presents a number of environment variables that can be used in your jobs. Three of the more useful are TMPDIR, SGE_O_WORKDIR, and NSLOTS:
In Figure 2 is a sample script, using all three. +
In Figure 2 is a sample script, using all three.
#!/encs/bin/tcsh #$ -N envs #$ -cwd @@ -836,7 +843,7 @@2.10 SSH Keys For MPI
-Some programs effect their parallel processing via MPI (which is a communication protocol). An +
Some programs effect their parallel processing via MPI (which is a communication protocol). An example of such software is Fluent. MPI needs to have ‘passwordless login’ set up, which means SSH keys. In your NFS-mounted home directory:
@@ -851,15 +858,15 @@2.10
- Set file permissions of authorized_keys to 600; of your NFS-mounted home to 700 (note that you likely will not have to do anything here, as most people will have those permissions by default).
+
The following documentation is specific to the Speed HPC Facility at the Gina Cody School of +
The following documentation is specific to the Speed HPC Facility at the Gina Cody School of Engineering and Computer Science. -
+
To create an anaconda environment in your speed-scratch directory, use the prefix option when +
To create an anaconda environment in your speed-scratch directory, use the prefix option when executing conda create. For example, to create an anaconda environment for ai_user, execute the following at the command line: @@ -869,11 +876,11 @@
-
Note: Without the prefix option, the conda create command creates the environment in +
+
Note: Without the prefix option, the conda create command creates the environment in texttta_user’s home directory by default.
-List Environments. To view your conda environments, type: conda info --envs @@ -885,9 +892,9 @@
+
-
Activate an Environment. Activate the environment speedscratcha_usermyconda as follows @@ -896,7 +903,7 @@
After activating your environment, add pip to your environment by using +
After activating your environment, add pip to your environment by using @@ -904,10 +911,10 @@
This will install pip and pip’s dependencies, including python, into the environment. -
Important Note: pip (and pip3) are used to install modules from the python distribution while +
This will install pip and pip’s dependencies, including python, into the environment. +
Important Note: pip (and pip3) are used to install modules from the python distribution while conda install installs modules from anaconda’s repository. -
+
#!/encs/bin/tcsh #$ -N flu10000 #$ -cwd @@ -939,7 +946,7 @@-The job script in Figure 3 runs Fluent in parallel over 32 cores. Of note, we have requested e-mail +
The job script in Figure 3 runs Fluent in parallel over 32 cores. Of note, we have requested e-mail notifications (-m), are defining the parallel environment for, fluent, with, -sgepe smp (very important), and are setting $TMPDIR as the in-job location for the “moment” rfile.out file (in-job, because the last line of the script copies everything from $TMPDIR to a directory in the user’s @@ -949,7 +956,7 @@
2.13 Example Job: efficientdet
-The following steps describing how to create an efficientdet environment on Speed, were submitted by +
The following steps describing how to create an efficientdet environment on Speed, were submitted by a member of Dr. Amer’s research group.
@@ -978,24 +985,24 @@
-
-
+
+
2.14 Java Jobs
-Jobs that call java have a memory overhead, which needs to be taken into account when assigning a +
Jobs that call java have a memory overhead, which needs to be taken into account when assigning a value to h_vmem. Even the most basic java call, java -Xmx1G -version, will need to have, -l h_vmem=5G, with the 4-GB difference representing the memory overhead. Note that this memory overhead grows proportionally with the value of -Xmx. To give you an idea, when -Xmx has a value of 100G, h_vmem has to be at least 106G; for 200G, at least 211G; for 300G, at least 314G. -
+
2.15 Scheduling On The GPU Nodes
-The primary cluster has two GPU nodes, each with six Tesla (CUDA-compatible) P6 cards: each card +
The primary cluster has two GPU nodes, each with six Tesla (CUDA-compatible) P6 cards: each card has 2048 cores and 16GB of RAM. Though note that the P6 is mainly a single-precision card, so unless you need the GPU double precision, double-precision calculations will be faster on a CPU node. -
Job scripts for the GPU queue differ in that they do not need these statements: +
Job scripts for the GPU queue differ in that they do not need these statements: @@ -1004,8 +1011,8 @@
-
But do need this statement, which attaches either a single GPU, or, two GPUs, to the +
+
But do need this statement, which attaches either a single GPU, or, two GPUs, to the job: @@ -1014,11 +1021,11 @@
#$ -l gpu=[1|2]
-
Single-GPU jobs are granted 5 CPU cores and 80GB of system memory, and dual-GPU jobs are +
+
Single-GPU jobs are granted 5 CPU cores and 80GB of system memory, and dual-GPU jobs are granted 10 CPU cores and 160GB of system memory. A total of four GPUs can be actively attached to any one user at any given time. -
Once that your job script is ready, you can submit it to the GPU queue with: +
Once that your job script is ready, you can submit it to the GPU queue with: @@ -1026,8 +1033,8 @@
-
And you can query nvidia-smi on the node that is running your job with: +
+
And you can query nvidia-smi on the node that is running your job with: @@ -1035,8 +1042,8 @@
-
Status of the GPU queue can be queried with: +
+
Status of the GPU queue can be queried with: @@ -1044,16 +1051,16 @@
-
Very important note regarding TensorFlow and PyTorch: if you are planning to run TensorFlow +
+
Very important note regarding TensorFlow and PyTorch: if you are planning to run TensorFlow
and/or PyTorch multi-GPU jobs, do not use the tf.distribute and/or
torch.nn.DataParallel functions, as they will crash the compute node (100% certainty). This
appears to be the current hardware’s architecture’s defect. The workaround is to either manually
effect GPU parallelisation (TensorFlow has an example on how to do this), or to run on a single
GPU.
-
Important -
Users without permission to use the GPU nodes can submit jobs to the g.q queue but those jobs +
Important +
Users without permission to use the GPU nodes can submit jobs to the g.q queue but those jobs will hang and never run. -
There are two GPUs in both speed-05 and speed-17, and one in speed-19. Their availability is +
There are two GPUs in both speed-05 and speed-17, and one in speed-19. Their availability is seen with, qstat -F g (note the capital): @@ -1077,8 +1084,8 @@
This status demonstrates that all five are available (i.e., have not been requested as resources). To +
+
This status demonstrates that all five are available (i.e., have not been requested as resources). To specifically request a GPU node, add, -l g=[#GPUs], to your qsub (statement/script) or qlogin (statement) request. For example, qsub -l h_vmem=1G -l g=1 ./count.sh. You will see that this job has been assigned to one of the GPU nodes: @@ -1098,13 +1105,13 @@
And that there are no more GPUs available on that node (hc:gpu=0). Note that no more than two +
+
And that there are no more GPUs available on that node (hc:gpu=0). Note that no more than two GPUs can be requested for any one job. -
+
When calling CUDA within job scripts, it is important to create a link to the desired CUDA libraries and +
When calling CUDA within job scripts, it is important to create a link to the desired CUDA libraries and set the runtime link path to the same libraries. For example, to use the cuda-11.5 libraries, specify the following in your Makefile. @@ -1114,16 +1121,16 @@
-L/encs/pkg/cuda-11.5/root/lib64 -Wl,-rpath,/encs/pkg/cuda-11.5/root/lib64-
-
In your job script, specify the version of gcc to use prior to calling cuda. For example: module
+
+ In your job script, specify the version of gcc to use prior to calling cuda. For example: module
load gcc/8.4 or module load gcc/9.3
-
+
It is not possible to create a qlogin session on to a node in the GPU Queue (g.q). As direct logins
+ It is not possible to create a qlogin session on to a node in the GPU Queue (g.q). As direct logins
to these nodes is not available, jobs must be submitted to the GPU Queue in order to compile and
link.
- We have several versions of CUDA installed in:
+ We have several versions of CUDA installed in:
@@ -1133,17 +1140,17 @@ 2.15.2 Special Notes for sending CUDA jobs to the GPU Queue
-
-
For CUDA to compile properly for the GPU queue, edit your Makefile replacing usrlocalcuda +
+
For CUDA to compile properly for the GPU queue, edit your Makefile replacing usrlocalcuda with one of the above. -
+
The cluster is, “first come, first served”, until it fills, and then job position in the queue is +
The cluster is, “first come, first served”, until it fills, and then job position in the queue is based upon past usage. The scheduler does attempt to fill gaps, though, so sometimes a single-core job of lower priority will schedule before a multi-core job of higher priority, for example. -
+
Scripts can live in your NFS-provided home, but any substantial data need to be in your +
Scripts can live in your NFS-provided home, but any substantial data need to be in your cluster-specific directory (located at /speed-scratch/<ENCSusername>/). -
NFS is great for acute activity, but is not ideal for chronic activity. Any data that a +
NFS is great for acute activity, but is not ideal for chronic activity. Any data that a job will read more than once should be copied at the start to the scratch disk of a compute node using $TMPDIR (and, perhaps, $SGE_O_WORKDIR), any intermediary job data should be produced in $TMPDIR, and once a job is near to finishing, those data should @@ -1174,7 +1181,7 @@
+
+
HPC Committee’s initial batch about 6 students (end of 2019):
+HPC Committee’s initial batch about 6 students (end of 2019):
NAG’s MAC spoofer analyzer [9, 8], such as https://github.com/smokhov/atsm/tree/master/examples/flucid +
NAG’s MAC spoofer analyzer [12, 11], such as https://github.com/smokhov/atsm/tree/master/examples/flucid
S4 LAB/GIPSY R&D Group’s:
+S4 LAB/GIPSY R&D Group’s:
+ planetary rover mobility. International Journal of Robotics Research, 39(7):797–811, 2020. + https://doi.org/10.1177/0278364920913945 +
+
+
Phase 3 had 4 vidpro nodes added from Dr. Amer totalling 6x P6 and 6x V100 GPUs +
Phase 3 had 4 vidpro nodes added from Dr. Amer totalling 6x P6 and 6x V100 GPUs added. - - - -
+
Phase 2 saw 6x NVIDIA Tesla P6 added and 8x more compute nodes. The P6s replaced 4x of FirePro +
Phase 2 saw 6x NVIDIA Tesla P6 added and 8x more compute nodes. The P6s replaced 4x of FirePro S7150. -
+
Phase 1 of Speed was of the following configuration: +
Phase 1 of Speed was of the following configuration:
+
-
-This section describes how to use the “bash shell” on Speed. Review subsubsection 2.1.2 to ensure -that your bash enviroment is set up. -
+ + + +
-
In order to set your login shell to bash on Speed, your login shell on all GCS servers must be changed +
All Speed users are expected to have a basic understanding of Linux and its commonly used +commands. +
+
+Software Carpentry provides free resources to learn software, including a workshop on the Unix shell. +https://software-carpentry.org/lessons/ +
+
+There are a number of Udemy courses, including free ones, that will assist you in learning Linux. +Active Concordia faculty, staff and students have access to Udemy courses such as Linux Mastery: +Master the Linux Command Line in 11.5 Hours is a good starting point for beginners. Visit +https://www.concordia.ca/it/services/udemy.html to learn how Concordians may access +Udemy. +
+
+This section describes how to use the “bash shell” on Speed. Review Section 2.1.2 to ensure that your +bash environment is set up. +
+
+In order to set your login shell to bash on Speed, your login shell on all GCS servers must be changed to bash. To make this change, create a ticket with the Service Desk (or email help at concordia.ca) to request that bash become your default login shell for your ENCS user account on all GCS servers. - - - -
+
-
To move to the bash shell, type bash at the command prompt. For example: +
To move to the bash shell, type bash at the command prompt. For example: @@ -1304,45 +1343,45 @@
-
Note how the command prompt changed from [speed-27] [/home/a/a_user] > to bash-4.4$ +
+
Note how the command prompt changed from [speed-27] [/home/a/a_user] > to bash-4.4$ after entering the bash shell. -
+
-
To execute bash scripts on Speed: +
To execute bash scripts on Speed:
The Speed GitHub contains a sample bash job script. -
+
The Speed GitHub contains a sample bash job script. +
-
+
-
The ‘‘Disk quota exceeded’’ Error occurs when your application has run out of disk space to write +
The ‘‘Disk quota exceeded’’ Error occurs when your application has run out of disk space to write to. On Speed this error can be returned when:
+
-
+
The use local disk space is generally recommended for IO intensive operations. However, as the +
The use local disk space is generally recommended for IO intensive operations. However, as the size of /tmp on speed nodes is 1GB it can be necessary for scripts to store temporary data elsewhere. Review the documentation for each module called within your script to determine how to set working directories for that application. The basic steps for this solution are: @@ -1352,70 +1391,155 @@
Create a working directory in speed-scratch for output files. For example, this command - creates subdirectory called output in a_user’s speed-scratch directory: +
Create a working directory in speed-scratch for output files. For example, this + command will create a subdirectory called output in your speed-scratch + directory:
- mkdir -m 750 /speed-scratch/a_user/output
+ mkdir -m 750 /speed-scratch/$USER/output
- +
To create a subdirectory for recovery files: +
To create a subdirectory for recovery files:
- mkdir -m 750 /speed-scratch/a_user/recovery + mkdir -m 750 /speed-scratch/$USER/recovery-
+
+
In the above example, $USER is an environment variable containing your ENCS username. +
-
Create directories for recovery, temporary, and configuration files. For example, to create these - directories for a_user: +
Create directories for recovery, temporary, and configuration files. For example, to create these + directories for your encs user account:
- mkdir -m 750 -p /speed-scratch/a_user/comsol/{recovery,tmp,config}
+ mkdir -m 750 -p /speed-scratch/$USER/comsol/{recovery,tmp,config}
- +
Add the following command switches to the COMSOL command to use the directories created - for a_user above: +
Add the following command switches to the COMSOL command to use the directories created + above:
- -recoverydir /speed-scratch/a_user/comsol/recovery - -tmpdir /speed-scratch/a_user/comsol/tmp - -configuration/speed-scratch/a_user/comsol/config + -recoverydir /speed-scratch/$USER/comsol/recovery + -tmpdir /speed-scratch/$USER/comsol/tmp + -configuration/speed-scratch/$USER/comsol/config ++
In the above example, $USER is an environment variable containing your ENCS username. +
+
+By default when adding a python module the /tmp directory is set as the temporary repository for +files downloads. The size of the /tmp directory on speed-submit is too small for pytorch. To add a +python module
+Create your own tmp directory in your speed-scratch direcrtory + + + +
++ mkdir /speed-scratch/$USER/tmp-
+
+
+Use the tmp direcrtory you created + + + +
++ setenv TMPDIR /speed-scratch/$USER/tmp ++
+
In the above example, $USER is an environment variable containing your ENCS username. +
-
Below is a list of resources and facilities similar to Speed at various capacities. Depending on your +
When a job with a job id of 1234 is running, the status of that job can be tracked using +‘qstat -j 1234‘. Likewise, if the job is pending, the ‘qstat -j 1234‘ command will report as to +why the job is not scheduled or running. Once the job has finished, or has been killed, the qacct +command must be used to query the job’s status, e.g., ‘qaact -j [jobid]‘. +
+
++
+It is possible that a (or a number of) the Speed nodes are disabled. Nodes are disabled if they require +maintenance. To verify if Speed nodes are disabled, request the current list of disabled nodes from +qstat. + + + +
++qstat -f -qs d +queuename qtype resv/used/tot. load_avg arch states +--------------------------------------------------------------------------------- +g.q@speed-05.encs.concordia.ca BIP 0/0/32 0.27 lx-amd64 d +--------------------------------------------------------------------------------- +s.q@speed-07.encs.concordia.ca BIP 0/0/32 0.01 lx-amd64 d +--------------------------------------------------------------------------------- +s.q@speed-10.encs.concordia.ca BIP 0/0/32 0.01 lx-amd64 d +--------------------------------------------------------------------------------- +s.q@speed-16.encs.concordia.ca BIP 0/0/32 0.02 lx-amd64 d +--------------------------------------------------------------------------------- +s.q@speed-19.encs.concordia.ca BIP 0/0/32 0.03 lx-amd64 d +--------------------------------------------------------------------------------- +s.q@speed-24.encs.concordia.ca BIP 0/0/32 0.01 lx-amd64 d +--------------------------------------------------------------------------------- +s.q@speed-36.encs.concordia.ca BIP 0/0/32 0.03 lx-amd64 d ++
+
Note how the all of the Speed nodes in the above list have a state of d, or disabled. +
Your job will run once the maintenance has been completed and the disabled nodes have been +enabled. +
+It is possible that your job is pending, because the job requested resources that are not available +within Speed. To verify why pending job with job id 1234 is not running, execute ‘qstat -j 1234‘ +and review the messages in the scheduling info: section. +
+
+Below is a list of resources and facilities similar to Speed at various capacities. Depending on your research group and needs, they might be available to you. They are not managed by HPC/NAG of AITS, so contact their respective representatives.
@@ -1460,63 +1587,84 @@ C http://aosabook.org.
- [5] MathWorks. MATLAB. [online], 2000–2012. http://www.mathworks.com/products/matlab/. + [5] Goutam Yelluru Gopal and Maria Amer. Mobile vision transformer-based visual object + tracking. In 34th British Machine Vision Conference (BMVC), Aberdeen, UK, November 2023. + https://arxiv.org/abs/2309.05829 and https://github.com/goutamyg/MVT.
- [6] Serguei A. Mokhov. The use of machine learning with signal- and NLP processing + [6] Goutam Yelluru Gopal and Maria Amer. Separable self and mixed attention transformers + for efficient object tracking. In IEEE/CVF Winter Conference on Applications of Computer + Vision (WACV), Waikoloa, Hawaii, January 2024. https://arxiv.org/abs/2309.03979 and + https://github.com/goutamyg/SMAT. +
++ [7] MathWorks. MATLAB. [online], 2000–2012. http://www.mathworks.com/products/matlab/. +
++ [8] Serguei Mokhov, Jonathan Llewellyn, Carlos Alarcon Meza, Tariq Daradkeh, and Gillian + Roper. The use of containers in OpenGL, ML and HPC for teaching and research support. + In ACM SIGGRAPH 2023 Posters, SIGGRAPH ’23, New York, NY, USA, 2023. ACM. + https://doi.org/10.1145/3588028.3603676. +
++ [9] Serguei A. Mokhov. The use of machine learning with signal- and NLP processing of source code to fingerprint, detect, and classify vulnerabilities and weaknesses with MARFCAT. Technical Report NIST SP 500-283, NIST, October 2011. Report: + + + http://www.nist.gov/manuscript-publication-search.cfm?pub_id=909407, online e-print at http://arxiv.org/abs/1010.2511.
- [7] Serguei A. Mokhov. Intensional Cyberforensics. PhD thesis, Department of Computer Science + [10] Serguei A. Mokhov. Intensional Cyberforensics. PhD thesis, Department of Computer Science and Software Engineering, Concordia University, Montreal, Canada, September 2013. Online at http://arxiv.org/abs/1312.0466.
- [8] Serguei A. Mokhov, Michael J. Assels, Joey Paquet, and Mourad Debbabi. Automating MAC + [11] Serguei A. Mokhov, Michael J. Assels, Joey Paquet, and Mourad Debbabi. Automating MAC spoofer evidence gathering and encoding for investigations. In Frederic Cuppens et al., editors, Proceedings of The 7th International Symposium on Foundations & Practice of Security (FPS’14), LNCS 8930, pages 168–183. Springer, November 2014. Full paper.
- [9] Serguei A. Mokhov, Michael J. Assels, Joey Paquet, and Mourad Debbabi. Toward automated + [12] Serguei A. Mokhov, Michael J. Assels, Joey Paquet, and Mourad Debbabi. Toward automated MAC spoofer investigations. In Proceedings of C3S2E’14, pages 179–184. ACM, August 2014. Short paper. - - -
- [10] Serguei A. Mokhov and Scott Bunnell. Speed server farm: + [13] Serguei A. Mokhov and Scott Bunnell. Speed server farm: Gina Cody School of ENCS HPC facility. [online], 2018–2019. https://docs.google.com/presentation/d/1bWbGQvYsuJ4U2WsfLYp8S3yb4i7OdU7QDn3l_Q9mYis.
- [11] Serguei A. Mokhov, Joey Paquet, and Mourad Debbabi. The use of NLP techniques in static + [14] Serguei A. Mokhov, Joey Paquet, and Mourad Debbabi. The use of NLP techniques in static code analysis to detect weaknesses and vulnerabilities. In Maria Sokolova and Peter van Beek, editors, Proceedings of Canadian Conference on AI’14, volume 8436 of LNAI, pages 326–332. Springer, May 2014. Short paper.
- [12] Parna Niksirat, Adriana Daca, and Krzysztof Skonieczny. The effects of reduced-gravity on - planetary rover mobility. International Journal of Robotics Research, 39(7):797–811, 2020. + [15] Parna Niksirat, Adriana Daca, and Krzysztof Skonieczny. The effects of reduced-gravity + on planetary rover mobility. International Journal of Robotics Research, 39(7):797–811, 2020. + https://doi.org/10.1177/0278364920913945.
- [13] Chet Ramey. The Bourne-Again Shell. In Brown and Wilson [4]. + [16] Chet Ramey. The Bourne-Again Shell. In Brown and Wilson [4]. http://aosabook.org/en/bash.html.
- [14] Rob Schreiber. MATLAB. Scholarpedia, 2(6):2929, 2007. + [17] Rob Schreiber. MATLAB. Scholarpedia, 2(6):2929, 2007. http://www.scholarpedia.org/article/MATLAB.
- [15] The MARF Research and Development Group. The Modular Audio Recognition + [18] The MARF Research and Development Group. The Modular Audio Recognition Framework and its Applications. [online], 2002–2014. http://marf.sf.net and http://arxiv.org/abs/0905.1235, last viewed May 2015.