From b80461989cb1872654e5bee929fb83a739fa0df3 Mon Sep 17 00:00:00 2001 From: Serguei Mokhov Date: Mon, 16 Oct 2023 00:34:59 -0400 Subject: [PATCH] [manual][HTML] release 6.6 --- doc/web/index.html | 985 +++++++++++++++++++++++---------------- doc/web/speed-manual.css | 1 + 2 files changed, 588 insertions(+), 398 deletions(-) diff --git a/doc/web/index.html b/doc/web/index.html index 58d1454..b05bcba 100644 --- a/doc/web/index.html +++ b/doc/web/index.html @@ -24,14 +24,14 @@

Speed: The GCS ENCS Cluster


Concordia University
Montreal, Quebec, Canada
rt-ex-hpc~AT~encs.concordia.ca

-
Version 6.6-dev-07
+
Version 6.6 (final GE version)

∗The group acknowledges the initial manual version VI produced by Dr. Scott Bunnell while with us.

Abstract

-

This document primarily presents a quick start guide to the usage of the Gina Cody +

This document primarily presents a quick start guide to the usage of the Gina Cody School of Engineering and Computer Science compute server farm called “Speed” – the GCS ENCS Speed cluster, managed by HPC/NAG of GCS ENCS, Concordia University, Montreal, Canada. @@ -39,76 +39,78 @@

Contents

- 1 Introduction -
 1.1 Resources -
 1.2 Team -
 1.3 What Speed Comprises -
 1.4 What Speed Is Ideal For -
 1.5 What Speed Is Not -
 1.6 Available Software -
 1.7 Requesting Access -
2 Job Management -
 2.1 Getting Started -
  2.1.1 SSH Connections -
  2.1.2 Environment Set Up -
 2.2 Job Submission Basics -
  2.2.1 Directives -
  2.2.2 Module Loads -
  2.2.3 User Scripting -
 2.3 Sample Job Script -
 2.4 Common Job Management Commands Summary -
 2.5 Advanced qsub Options -
 2.6 Array Jobs - - - -
 2.7 Requesting Multiple Cores (i.e., Multithreading Jobs) -
 2.8 Interactive Jobs -
 2.9 Scheduler Environment Variables -
 2.10 SSH Keys For MPI -
 2.11 Creating Virtual Environments -
  2.11.1 Anaconda -
 2.12 Example Job Script: Fluent -
 2.13 Example Job: efficientdet -
 2.14 Java Jobs -
 2.15 Scheduling On The GPU Nodes -
  2.15.1 CUDA -
  2.15.2 Special Notes for sending CUDA jobs to the GPU Queue -
3 Conclusion -
 3.1 Important Limitations -
 3.2 Tips/Tricks -
 3.3 Use Cases -
A History -
 A.1 Acknowledgments -
 A.2 Phase 3 -
 A.3 Phase 2 -
 A.4 Phase 1 -
B Frequently Asked Questions -
 B.1 Where do I learn about Linux? -
 B.2 How to use the “bash shell” on Speed? -
  B.2.1 How do I set bash as my login shell? -
  B.2.2 How do I move into a bash shell on Speed? -
  B.2.3 How do I run scripts written in bash on Speed? -
 B.3 How to resolve“Disk quota exceeded” errors? -
  B.3.1 Probable Cause -
  B.3.2 Possible Solutions -
  B.3.3 Example of setting working directories for COMSOL -
  B.3.4 Example of setting working directories for Python Modules -
 B.4 How do I check my job’s status? -
 B.5 Why is my job pending when nodes are empty? -
  B.5.1 Disabled nodes -
  B.5.2 Error in job submit request. -
C Sister Facilities -
Annotated Bibliography +  1 Introduction +
  1.1 Resources +
  1.2 Team +
  1.3 What Speed Comprises +
  1.4 What Speed Is Ideal For +
  1.5 What Speed Is Not +
  1.6 Available Software +
  1.7 Requesting Access +
 2 Job Management +
  2.1 Getting Started +
   2.1.1 SSH Connections +
   2.1.2 Environment Set Up +
  2.2 Job Submission Basics +
   2.2.1 Directives +
   2.2.2 Module Loads +
   2.2.3 User Scripting +
  2.3 Sample Job Script +
  2.4 Common Job Management Commands Summary +
  2.5 Advanced qsub Options +
  2.6 Array Jobs + + + +
  2.7 Requesting Multiple Cores (i.e., Multithreading Jobs) +
  2.8 Interactive Jobs +
  2.9 Scheduler Environment Variables +
  2.10 SSH Keys For MPI +
  2.11 Creating Virtual Environments +
   2.11.1 Anaconda +
  2.12 Example Job Script: Fluent +
  2.13 Example Job: efficientdet +
  2.14 Java Jobs +
  2.15 Scheduling On The GPU Nodes +
   2.15.1 CUDA +
   2.15.2 Special Notes for sending CUDA jobs to the GPU Queue +
   2.15.3 OpenISS Examples +
  2.16 Singularity Containers +
 3 Conclusion +
  3.1 Important Limitations +
  3.2 Tips/Tricks +
  3.3 Use Cases +
 A History +
  A.1 Acknowledgments +
  A.2 Phase 3 +
  A.3 Phase 2 +
  A.4 Phase 1 +
 B Frequently Asked Questions +
  B.1 Where do I learn about Linux? +
  B.2 How to use the “bash shell” on Speed? +
   B.2.1 How do I set bash as my login shell? +
   B.2.2 How do I move into a bash shell on Speed? +
   B.2.3 How do I run scripts written in bash on Speed? +
  B.3 How to resolve “Disk quota exceeded” errors? +
   B.3.1 Probable Cause +
   B.3.2 Possible Solutions +
   B.3.3 Example of setting working directories for COMSOL +
   B.3.4 Example of setting working directories for Python Modules +
  B.4 How do I check my job’s status? +
  B.5 Why is my job pending when nodes are empty? +
   B.5.1 Disabled nodes +
   B.5.2 Error in job submit request. +
 C Sister Facilities +
 Annotated Bibliography

1 Introduction

-

This document contains basic information required to use “Speed” as well as tips and tricks, +

This document contains basic information required to use “Speed” as well as tips and tricks, examples, and references to projects and papers that have used Speed. User contributions of sample jobs and/or references are welcome. Details are sent to the hpc-ml mailing list. -

+

1.1 Resources

-

+

1.2 Team

@@ -138,8 +140,8 @@

1.2
  • Tariq Daradkeh, PhD, IT Instructional Specialist, Information Technology
    • -

    We receive support from the rest of AITS teams, such as NAG, SAG, FIS, and DOG. -

    +

    We receive support from the rest of AITS teams, such as NAG, SAG, FIS, and DOG. +

    1.3 What Speed Comprises

    -

    -

    References

    +

    +

    References

    @@ -1598,66 +1770,83 @@

    C https://github.com/goutamyg/SMAT.

    - [7]   MathWorks. MATLAB. [online], 2000–2012. http://www.mathworks.com/products/matlab/. + [7]   Haotao Lai. An OpenISS framework + specialization for deep learning-based person re-identification. Master’s thesis, Department of + Computer Science and Software Engineering, Concordia University, Montreal, Canada, August + 2019. https://spectrum.library.concordia.ca/id/eprint/985788/.

    - [8]   Serguei Mokhov, Jonathan Llewellyn, Carlos Alarcon Meza, Tariq Daradkeh, and Gillian + [8]   Haotao Lai et al. OpenISS keras-yolo3 v0.1.0, June 2021. + https://github.com/OpenISS/openiss-yolov3. +

    +

    + [9]   Haotao Lai et al. Openiss person re-identification baseline v0.1.1, June 2021. + https://github.com/OpenISS/openiss-reid-tfk. + + + +

    +

    + [10]   MathWorks. MATLAB. [online], 2000–2012. http://www.mathworks.com/products/matlab/. +

    +

    + [11]   Serguei Mokhov, Jonathan Llewellyn, Carlos Alarcon Meza, Tariq Daradkeh, and Gillian Roper. The use of containers in OpenGL, ML and HPC for teaching and research support. In ACM SIGGRAPH 2023 Posters, SIGGRAPH ’23, New York, NY, USA, 2023. ACM. https://doi.org/10.1145/3588028.3603676.

    - [9]   Serguei A. Mokhov. The use of machine learning with signal- and NLP processing + [12]   Serguei A. Mokhov. The use of machine learning with signal- and NLP processing of source code to fingerprint, detect, and classify vulnerabilities and weaknesses with MARFCAT. Technical Report NIST SP 500-283, NIST, October 2011. Report: - - - http://www.nist.gov/manuscript-publication-search.cfm?pub_id=909407, online e-print at http://arxiv.org/abs/1010.2511.

    - [10]   Serguei A. Mokhov. Intensional Cyberforensics. PhD thesis, Department of Computer Science + [13]   Serguei A. Mokhov. Intensional Cyberforensics. PhD thesis, Department of Computer Science and Software Engineering, Concordia University, Montreal, Canada, September 2013. Online at http://arxiv.org/abs/1312.0466.

    - [11]   Serguei A. Mokhov, Michael J. Assels, Joey Paquet, and Mourad Debbabi. Automating MAC + [14]   Serguei A. Mokhov, Michael J. Assels, Joey Paquet, and Mourad Debbabi. Automating MAC spoofer evidence gathering and encoding for investigations. In Frederic Cuppens et al., editors, Proceedings of The 7th International Symposium on Foundations & Practice of Security (FPS’14), LNCS 8930, pages 168–183. Springer, November 2014. Full paper.

    - [12]   Serguei A. Mokhov, Michael J. Assels, Joey Paquet, and Mourad Debbabi. Toward automated + [15]   Serguei A. Mokhov, Michael J. Assels, Joey Paquet, and Mourad Debbabi. Toward automated MAC spoofer investigations. In Proceedings of C3S2E’14, pages 179–184. ACM, August 2014. Short paper.

    - [13]   Serguei A. Mokhov and Scott Bunnell. Speed server farm: + [16]   Serguei A. Mokhov and Scott Bunnell. Speed server farm: Gina Cody School of ENCS HPC facility. [online], 2018–2019. https://docs.google.com/presentation/d/1bWbGQvYsuJ4U2WsfLYp8S3yb4i7OdU7QDn3l_Q9mYis.

    - [14]   Serguei A. Mokhov, Joey Paquet, and Mourad Debbabi. The use of NLP techniques in static + [17]   Serguei A. Mokhov, Joey Paquet, and Mourad Debbabi. The use of NLP techniques in static code analysis to detect weaknesses and vulnerabilities. In Maria Sokolova and Peter van Beek, editors, Proceedings of Canadian Conference on AI’14, volume 8436 of LNAI, pages 326–332. Springer, May 2014. Short paper.

    - [15]   Parna Niksirat, Adriana Daca, and Krzysztof Skonieczny. The effects of reduced-gravity + [18]   Parna Niksirat, Adriana Daca, and Krzysztof Skonieczny. The effects of reduced-gravity on planetary rover mobility. International Journal of Robotics Research, 39(7):797–811, 2020. https://doi.org/10.1177/0278364920913945.

    + + +

    - [16]   Chet Ramey. The Bourne-Again Shell. In Brown and Wilson [4]. + [19]   Chet Ramey. The Bourne-Again Shell. In Brown and Wilson [4]. http://aosabook.org/en/bash.html.

    - [17]   Rob Schreiber. MATLAB. Scholarpedia, 2(6):2929, 2007. + [20]   Rob Schreiber. MATLAB. Scholarpedia, 2(6):2929, 2007. http://www.scholarpedia.org/article/MATLAB.

    - [18]   The MARF Research and Development Group. The Modular Audio Recognition + [21]   The MARF Research and Development Group. The Modular Audio Recognition Framework and its Applications. [online], 2002–2014. http://marf.sf.net and http://arxiv.org/abs/0905.1235, last viewed May 2015.

    diff --git a/doc/web/speed-manual.css b/doc/web/speed-manual.css index d00465d..6d423e9 100644 --- a/doc/web/speed-manual.css +++ b/doc/web/speed-manual.css @@ -140,6 +140,7 @@ table.pmatrix {width:100%;} span.bar-css {text-decoration:overline;} img.cdots{vertical-align:middle;} .partToc a, .partToc, .likepartToc a, .likepartToc {line-height: 200%; font-weight:bold; font-size:110%;} +.chapterToc a, .chapterToc, .likechapterToc a, .likechapterToc, .appendixToc a, .appendixToc {line-height: 200%; font-weight:bold;} .index-item, .index-subitem, .index-subsubitem {display:block} div.caption {text-indent:-2em; margin-left:3em; margin-right:1em; text-align:left;} div.caption span.id{font-weight: bold; white-space: nowrap; }