Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NF: Add Environment class, with initial Native/Docker implementations #516

Merged
merged 39 commits into from
Jan 29, 2024
Merged
Changes from 1 commit
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
9406a9c
NF: Add Environment class, with initial Native/Docker implementations
effigies Mar 22, 2022
9f11b85
TEST: Add basic test for native environment
effigies Mar 31, 2022
58b4a34
FIX: import and arg
effigies Mar 31, 2022
39874d4
removing resetting self.output_ to None (not sure why it was needed)
djarecka Jun 15, 2022
b6b8130
implementing execute for the Docker env; changes to the ShellCommandT…
djarecka Jan 19, 2023
5922c47
updating docker tests; fixing issue with working directory for docker
djarecka Jan 30, 2023
dcac97e
adding need docker to the tests
djarecka Feb 27, 2023
4f21181
FIX: Bad rebase
effigies Aug 29, 2023
42a0588
RF: Rewrite _check_input with FileSets
effigies Aug 29, 2023
3dbc054
FIX: Missing argument
effigies Aug 29, 2023
81abb09
FIX: Get mode
effigies Aug 29, 2023
f495366
TEST: Update tests to be typing-friendly
effigies Aug 29, 2023
377692a
Merge branch 'rf/environments' of https://github.com/nipype/pydra int…
djarecka Sep 16, 2023
2eb88da
Update pydra/engine/task.py
djarecka Sep 17, 2023
ff3f5d0
Update pydra/engine/task.py
djarecka Sep 17, 2023
6889577
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 17, 2023
25f14a2
Merge branch 'rf/environments' of https://github.com/nipype/pydra int…
djarecka Sep 17, 2023
e4644a5
removing temporary old env, some renaming
djarecka Sep 17, 2023
fa240d9
Merge pull request #705 from djarecka/env
djarecka Sep 17, 2023
aaa399f
adding environment to run_el for slurm worker
djarecka Sep 18, 2023
7d39b4d
cleaning DockerTask remains
djarecka Sep 19, 2023
4d2098b
Merge pull request #706 from djarecka/env
djarecka Sep 19, 2023
bd76c3d
adding the Singularity environment class
djarecka Oct 5, 2023
9cc0904
Merge pull request #711 from djarecka/env
djarecka Oct 5, 2023
979bb60
removing psi plugin from one test
djarecka Oct 28, 2023
2dd5603
creating Container class
djarecka Nov 2, 2023
161635b
cleaning: removing ContainerTask and ContainerSpec
djarecka Nov 3, 2023
8d60dd1
adding xarg to env command, changing output_cpath to root used in the…
djarecka Nov 3, 2023
58038f5
Merge pull request #718 from djarecka/env
djarecka Nov 3, 2023
9673908
Update pydra/engine/environments.py
djarecka Nov 15, 2023
89de9e6
Update pydra/engine/environments.py
djarecka Nov 15, 2023
bfaa681
Update pydra/engine/environments.py
djarecka Nov 15, 2023
575785e
Update pydra/engine/tests/test_specs.py
djarecka Nov 15, 2023
3d54253
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 15, 2023
73e6856
Update pydra/engine/environments.py
djarecka Nov 15, 2023
d1793cb
small fix
djarecka Nov 15, 2023
89c1e27
Merge pull request #721 from djarecka/env
djarecka Nov 15, 2023
be5a870
small edits to the core
djarecka Nov 15, 2023
0b0c71b
Merge pull request #722 from djarecka/env
djarecka Nov 15, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
NF: Add Environment class, with initial Native/Docker implementations
effigies committed Sep 7, 2023
commit 9406a9c234882a47d5f92335690ab04c88d78cca
64 changes: 64 additions & 0 deletions pydra/engine/environments.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
from .helpers import execute


class Environment:
effigies marked this conversation as resolved.
Show resolved Hide resolved
def setup(self):
pass

def execute(self, task):
raise NotImplementedError

def teardown(self):
pass


class Native(Environment):
def execute(self, task):
args = task.render_arguments_in_root("/")
keys = ["return_code", "stdout", "stderr"]
values = execute(args, strip=task.strip)
output = dict(zip(keys, values))
if output["return_code"]:
if output["stderr"]:
raise RuntimeError(output["stderr"])
else:
raise RuntimeError(output["stdout"])
return output


class Docker(Environment):
def __init__(self, image, tag="latest"):
self.image = image
self.tag = tag

@staticmethod
def bind(loc, mode="ro"):
# XXX Failure mode: {loc} overwrites a critical directory in image
djarecka marked this conversation as resolved.
Show resolved Hide resolved
# To fix, we'll need to update any args within loc to a new location
# such as /mnt/pydra/loc
return f"{loc}:{loc}:{mode}"

def execute(self, task):
# XXX Need to mount all input locations
docker_img = f"{self.image}:{self.tag}"
# Renders arguments where `File`s have an additional prefix
args = task.render_arguments_in_root("/mnt/pydra")
# Skips over any inputs in task.cache_dir
# Needs to include `out_file`s when not relative to working dir
# Possibly a `TargetFile` type to distinguish between `File` and `str`?
mounts = task.get_inputs_in_root(root="/mnt/pydra")

docker_args = ["docker", "run", "-v", self.bind(task.cache_dir, "rw")]
docker_args.extend(flatten(["-v", self.bind(mount)] for mount in mounts))
keys = ["return_code", "stdout", "stderr"]
values = execute(docker_args + [docker_img] + args, strip=task.strip)
output = dict(zip(keys, values))
if output["return_code"]:
if output["stderr"]:
raise RuntimeError(output["stderr"])
else:
raise RuntimeError(output["stdout"])
# Any outputs that have been created with a re-rooted path need
# to be de-rooted
task.finalize_outputs("/mnt/pydra")
return output