-
Notifications
You must be signed in to change notification settings - Fork 0
/
how-to-containerize-an-application-with-singularity.html
188 lines (149 loc) · 16.5 KB
/
how-to-containerize-an-application-with-singularity.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
<!doctype html>
<html lang="en">
<head>
<!-- Required meta tags -->
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<title> How-to: Containerize an application with Singularity | Computational Approaches to Biological Problems
</title>
<link rel="canonical" href="./how-to-containerize-an-application-with-singularity.html">
<link rel="stylesheet" href="./theme/css/bootstrap.min.css">
<link rel="stylesheet" href="./theme/css/fontawesome.min.css">
<link rel="stylesheet" href="./theme/css/pygments/default.min.css">
<link rel="stylesheet" href="./theme/css/theme.css">
<meta name="description" content="In this article I'm going to cover the benefits of using containerized applications. Specifically, I'm going to focus on using Singularity which is the more secure alternative to Docker (which many people have heard of). Not only is Singularity more secure, it was designed with high-performance computing in mind. This …">
</head>
<body>
<header class="header">
<div class="container">
<div class="row">
<div class="col-sm-4">
<a href="./">
<img class="img-fluid rounded" src=./images/LucasBoatwrightHeadShot-161x215.jpeg alt="Computational Approaches to Biological Problems">
</a>
</div>
<div class="col-sm-8">
<h1 class="title"><a href="./">Computational Approaches to Biological Problems</a></h1>
<p class="text-muted">and other biological inklings</p>
<ul class="list-inline">
<li class="list-inline-item"><a href="./index.html">Home</a></li>
<li class="list-inline-item"><a href="./archives.html">posts</a></li>
<li class="list-inline-item"><a href="https://github.com/jlboat">GitHub</a></li>
<li class="list-inline-item"><a href="https://scholar.google.com/citations?hl=en&user=q5G2bJEAAAAJ&view_op=list_works&gmla=AJsN-F4gfKvkjBIT6Nv8Xy97IIUHuhFi9jF8x5ntZveyPGo_xO2ws-_5NjCw-_9z9CD2mWJnlbL1DeMd-0l-IUNYPsQ1NR4wu8s_l6YjZYlsb9mTQ1d8TX8">Pubs</a></li>
<li class="list-inline-item text-muted">|</li>
<li class="list-inline-item"><a href="./pages/about-me.html">About me</a></li>
</ul>
</div>
</div> </div>
</header>
<div class="main">
<div class="container">
<h1> How-to: Containerize an application with Singularity
</h1>
<hr>
<article class="article">
<header>
<ul class="list-inline">
<li class="list-inline-item text-muted" title="2018-10-19T00:00:00-05:00">
<i class="fas fa-clock"></i>
Fri 19 October 2018
</li>
<li class="list-inline-item">
<i class="fas fa-folder-open"></i>
<a href="./category/how-to.html">How-to</a>
</li>
<li class="list-inline-item">
<i class="fas fa-user"></i>
<a href="./author/j-lucas-boatwright.html">J. Lucas Boatwright</a> </li>
<li class="list-inline-item">
<i class="fas fa-tag"></i>
<a href="./tag/bioinformatics.html">#bioinformatics</a>, <a href="./tag/education.html">#education</a>, <a href="./tag/singularity.html">#singularity</a>, <a href="./tag/containers.html">#containers</a> </li>
</ul>
</header>
<div class="content">
<p>In this article I'm going to cover the benefits of using containerized applications. Specifically, I'm going to focus on using Singularity which is the more secure alternative to Docker (which many people have heard of).</p>
<p>Not only is Singularity more secure, it was designed with high-performance computing in mind. This means that containers you make locally may also be executed on your HPC system. This is generally not the case for Docker containers.</p>
<p>So, why should you use Singularity/containers?</p>
<p>Containers are for those of us that have ever: </p>
<div class="highlight"><pre><span></span><code><span class="mi">1</span><span class="p">.</span> <span class="n">installed</span> <span class="n">a</span> <span class="n">program</span> <span class="k">only</span> <span class="k">to</span> <span class="n">find</span> <span class="n">that</span> <span class="n">there</span> <span class="k">are</span> <span class="n">a</span> <span class="n">stream</span> <span class="k">of</span> <span class="n">dependency</span> <span class="n">issues</span>
<span class="mi">2</span><span class="p">.</span> <span class="n">tried</span> <span class="k">to</span> <span class="n">reproduce</span> <span class="n">someone</span> <span class="k">else</span><span class="err">'</span><span class="n">s</span> <span class="n">research</span> <span class="k">from</span> <span class="n">deprecated</span> <span class="n">software</span>
<span class="mi">3</span><span class="p">.</span> <span class="n">had</span> <span class="k">to</span> <span class="k">start</span> <span class="n">up</span> <span class="n">a</span> <span class="n">virtual</span> <span class="n">machine</span> <span class="k">every</span> <span class="k">time</span> <span class="n">you</span> <span class="n">need</span> <span class="k">to</span> <span class="n">run</span> <span class="n">specialized</span> <span class="n">software</span>
<span class="mi">4</span><span class="p">.</span> <span class="n">wanted</span> <span class="n">a</span> <span class="n">portable</span> <span class="n">environment</span>
</code></pre></div>
<p>As a bioinformatician, all of these problems have occurred for me at some point. So, I highly recommend the use of containers for resolving all of these problems. </p>
<p>To start, you'll need to install Singularity. It's not complicated but it's also not a simple 'apt-get install' either. So, I recommend you follow their <a href="http://singularity.lbl.gov/docs-installation">installation guide</a> as this will vary depending on your operating system.</p>
<p>Singularity containers are built from a Singularity Recipe. These have the following format (slightly simplified):</p>
<div class="highlight"><pre><span></span><code><span class="o">*</span><span class="n">Header</span><span class="o">*</span> <span class="o">-</span> <span class="n">describes</span> <span class="n">the</span> <span class="n">core</span> <span class="n">operating</span> <span class="n">system</span>
<span class="n">Sections</span> <span class="o">-</span> <span class="n">identified</span> <span class="n">below</span>
<span class="nf">%help</span> <span class="o">-</span> <span class="n">prints</span> <span class="n">helpful</span> <span class="n">informations</span>
<span class="nf">%setup</span> <span class="o">-</span> <span class="n">commands</span> <span class="n">executed</span> <span class="n">on</span> <span class="n">the</span> <span class="n">host</span> <span class="n">system</span> <span class="n">outside</span> <span class="n">of</span> <span class="n">the</span> <span class="n">container</span> <span class="n">after</span> <span class="n">the</span> <span class="n">base</span> <span class="n">OS</span> <span class="n">has</span> <span class="n">been</span> <span class="n">installed</span>
<span class="nf">%files</span> <span class="o">-</span> <span class="n">copy</span> <span class="n">files</span> <span class="n">from</span> <span class="n">your</span> <span class="n">host</span> <span class="n">system</span> <span class="n">into</span> <span class="n">the</span> <span class="n">container</span> <span class="p">(</span><span class="n">Files</span> <span class="n">are</span> <span class="n">copied</span> <span class="n">before</span> <span class="n">any</span> <span class="nf">%post</span> <span class="n">commands</span> <span class="n">are</span> <span class="n">run</span><span class="p">)</span>
<span class="nf">%labels</span> <span class="o">-</span> <span class="n">To</span> <span class="n">store</span> <span class="n">metadata</span> <span class="n">with</span> <span class="n">your</span> <span class="n">container</span> <span class="p">(</span><span class="n">stored</span> <span class="n">at</span> <span class="o">/</span><span class="p">.</span><span class="n">singularity</span><span class="p">.</span><span class="n">d</span><span class="o">/</span><span class="n">labels</span><span class="p">.</span><span class="n">json</span><span class="p">)</span>
<span class="nf">%environment</span> <span class="o">-</span> <span class="n">environment</span> <span class="n">variables</span> <span class="n">which</span> <span class="n">are</span> <span class="n">sourced</span> <span class="n">at</span> <span class="n">runtime</span> <span class="n">and</span> <span class="n">not</span> <span class="n">at</span> <span class="n">build</span> <span class="n">time</span>
<span class="nf">%post</span> <span class="o">-</span> <span class="n">commands</span> <span class="n">executed</span> <span class="n">within</span> <span class="n">the</span> <span class="n">container</span> <span class="n">after</span> <span class="n">the</span> <span class="n">base</span> <span class="n">OS</span> <span class="n">has</span> <span class="n">been</span> <span class="n">installed</span> <span class="n">at</span> <span class="n">build</span> <span class="n">time</span>
<span class="nf">%runscript</span> <span class="o">-</span> <span class="n">commands</span> <span class="n">executed</span> <span class="n">at</span> <span class="n">runtime</span>
<span class="nf">%test</span> <span class="o">-</span> <span class="n">commands</span> <span class="n">run</span> <span class="n">at</span> <span class="n">the</span> <span class="n">very</span> <span class="n">end</span> <span class="n">of</span> <span class="n">the</span> <span class="n">build</span> <span class="n">process</span> <span class="k">for</span> <span class="n">build</span> <span class="n">validation</span>
</code></pre></div>
<p>Honestly, I rarely use all of them. You can largely get away with just using %post and %runscript along with the required Header. </p>
<p>Now, I realize that all of this information is available on Singularity's webpage and with more details. So, rather than reiterate, I'm going to provide some working examples here (and you can find more <a href="https://github.com/jlboat/SingularityRecipes">here</a> and I also recommend <a href="https://github.com/BioContainers/containers">BioContainers</a>):</p>
<p>How to containerize a simple python package via pip:</p>
<div class="highlight"><pre><span></span><code><span class="nl">Bootstrap</span><span class="p">:</span> <span class="n">docker</span>
<span class="nl">From</span><span class="p">:</span> <span class="nl">python</span><span class="p">:</span><span class="mi">2</span>
<span class="nf">%post</span>
<span class="n">apt</span><span class="o">-</span><span class="n">get</span> <span class="n">update</span> <span class="o">--</span><span class="n">fix</span><span class="o">-</span><span class="n">missing</span> <span class="o">&&</span> <span class="n">apt</span><span class="o">-</span><span class="n">get</span> <span class="n">install</span> <span class="o">-</span><span class="n">y</span> <span class="n">python</span><span class="o">-</span><span class="n">pip</span>
<span class="n">pip</span> <span class="n">install</span> <span class="n">multiqc</span>
<span class="nf">%runscript</span>
<span class="n">exec</span> <span class="n">multiqc</span> <span class="s">"$@"</span>
</code></pre></div>
<p>How to containerize a GitHub application:</p>
<div class="highlight"><pre><span></span><code><span class="nl">Bootstrap</span><span class="p">:</span> <span class="n">docker</span>
<span class="nl">From</span><span class="p">:</span> <span class="nl">debian</span><span class="p">:</span><span class="n">latest</span>
<span class="nf">%post</span>
<span class="n">apt</span><span class="o">-</span><span class="n">get</span> <span class="n">update</span> <span class="o">--</span><span class="n">fix</span><span class="o">-</span><span class="n">missing</span> <span class="o">&&</span> <span class="n">apt</span><span class="o">-</span><span class="n">get</span> <span class="n">install</span> <span class="o">-</span><span class="n">y</span> <span class="n">git</span> <span class="n">make</span> <span class="n">g</span><span class="o">++</span> <span class="n">libz</span><span class="o">-</span><span class="n">dev</span>
<span class="n">git</span> <span class="n">clone</span> <span class="nl">https</span><span class="p">:</span><span class="c1">//github.com/gpertea/stringtie</span>
<span class="n">cd</span> <span class="n">stringtie</span>
<span class="n">make</span> <span class="n">release</span>
<span class="nf">%runscript</span>
<span class="n">exec</span> <span class="o">/</span><span class="n">stringtie</span><span class="o">/</span><span class="n">stringtie</span> <span class="s">"$@"</span>
</code></pre></div>
<p>How to containerize a miniconda environment (environment.yml can be from an existing environment or made from scratch):</p>
<div class="highlight"><pre><span></span><code><span class="nl">Bootstrap</span><span class="p">:</span> <span class="n">docker</span>
<span class="nl">From</span><span class="p">:</span> <span class="n">continuumio</span><span class="o">/</span><span class="n">miniconda3</span>
<span class="nf">%files</span>
<span class="n">environment</span><span class="p">.</span><span class="n">yml</span>
<span class="nf">%environment</span>
<span class="n">PATH</span><span class="o">=/</span><span class="n">opt</span><span class="o">/</span><span class="n">conda</span><span class="o">/</span><span class="n">envs</span><span class="o">/</span><span class="err">$</span><span class="p">(</span><span class="n">head</span> <span class="o">-</span><span class="mi">1</span> <span class="n">environment</span><span class="p">.</span><span class="n">yml</span> <span class="o">|</span> <span class="n">cut</span> <span class="o">-</span><span class="n">d</span><span class="sc">' '</span> <span class="o">-</span><span class="n">f2</span><span class="p">)</span><span class="o">/</span><span class="nl">bin</span><span class="p">:</span><span class="n">$PATH</span>
<span class="nf">%post</span>
<span class="n">echo</span> <span class="s">". /opt/conda/etc/profile.d/conda.sh"</span> <span class="o">>></span> <span class="o">~/</span><span class="p">.</span><span class="n">bashrc</span>
<span class="n">echo</span> <span class="s">"source activate $(head -1 environment.yml | cut -d' ' -f2)"</span> <span class="o">></span> <span class="o">~/</span><span class="p">.</span><span class="n">bashrc</span>
<span class="o">/</span><span class="n">opt</span><span class="o">/</span><span class="n">conda</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">conda</span> <span class="n">env</span> <span class="n">create</span> <span class="o">-</span><span class="n">f</span> <span class="n">environment</span><span class="p">.</span><span class="n">yml</span>
<span class="nf">%runscript</span>
<span class="n">exec</span> <span class="s">"$@"</span>
</code></pre></div>
<p>The way I set up my containers, they're typically run similar to how the installed program should run. For example:</p>
<div class="highlight"><pre><span></span><code><span class="c1"># Print the help information for FASTQC</span>
singularity run fastqc.simg -h
</code></pre></div>
<p>One may also put executables on the PATH variable (as in my conda example above) or use the Singularity %apps option to have multiple subcommands (as in the bioscripts example under the <a href="https://github.com/jlboat/SingularityRecipes">SingularityRecipes repo</a> on my GitHub page.</p>
<p>One last recommendation, one program per container is much more modular for long-term maintenance. I hope this helps get you started with your own Singularity containers!</p>
</div>
</article>
</div>
</div>
<footer class="footer">
<div class="container">
<div class="row">
<ul class="col-sm-6 list-inline">
<li class="list-inline-item"><a href="./authors.html">Authors</a></li>
<li class="list-inline-item"><a href="./archives.html">Archives</a></li>
<li class="list-inline-item"><a href="./categories.html">Categories</a></li>
<li class="list-inline-item"><a href="./tags.html">Tags</a></li>
</ul>
<p class="col-sm-6 text-sm-right text-muted">
Generated by <a href="https://github.com/getpelican/pelican" target="_blank">Pelican</a>
/ <a href="https://github.com/nairobilug/pelican-alchemy" target="_blank">✨</a>
</p>
</div> </div>
</footer>
</body>
</html>