Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] CLI-227: add implicit context_mount to Image docker methods #2582

Closed
wants to merge 34 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
a650752
init
kramstrom Nov 28, 2024
7019599
patterns
kramstrom Dec 3, 2024
57fdb88
Merge branch 'main' of github.com:modal-labs/modal-client into kramst…
kramstrom Dec 3, 2024
13ca6a1
abs path
kramstrom Dec 3, 2024
e6adb96
Merge branch 'main' into kramstrom/cli-227-remove-context_mount-from
kramstrom Dec 4, 2024
c956397
Merge branch 'main' of github.com:modal-labs/modal-client into kramst…
kramstrom Dec 16, 2024
e861d5f
Merge branch 'main' of github.com:modal-labs/modal-client into kramst…
kramstrom Dec 16, 2024
034693e
add back context_mount
kramstrom Dec 16, 2024
bf61dc2
Merge branch 'main' of github.com:modal-labs/modal-client into kramst…
kramstrom Dec 16, 2024
b8d3e14
add back
kramstrom Dec 16, 2024
75346c1
update
kramstrom Dec 16, 2024
7e6546b
Merge branch 'main' of github.com:modal-labs/modal-client into kramst…
kramstrom Dec 17, 2024
b013546
dockerignore find
kramstrom Dec 17, 2024
b2cb8ce
use dockerignore
kramstrom Dec 18, 2024
21ec1e7
dockerfile match port
kramstrom Dec 18, 2024
d7c9e76
fix test
kramstrom Dec 18, 2024
24e4b57
Merge branch 'main' into kramstrom/cli-227-remove-context_mount-from
kramstrom Dec 18, 2024
37475c3
remove
kramstrom Dec 18, 2024
8c0b412
move
kramstrom Dec 18, 2024
ed19a20
renaming
kramstrom Dec 18, 2024
e67a264
remove
kramstrom Dec 18, 2024
e5c1545
ignore
kramstrom Dec 19, 2024
a4048fb
remove docker_copy_match
kramstrom Dec 19, 2024
b6b3d51
Merge branch 'main' of github.com:modal-labs/modal-client into kramst…
kramstrom Dec 19, 2024
b7ff573
auto dockerignore
kramstrom Dec 20, 2024
6ac1400
remove autodockerignore
kramstrom Dec 20, 2024
c44a6ea
name
kramstrom Dec 20, 2024
baac945
Merge branch 'main' of github.com:modal-labs/modal-client into kramst…
kramstrom Dec 20, 2024
97685a8
autodockerignore
kramstrom Dec 20, 2024
b52fdbb
move to inside load
kramstrom Dec 20, 2024
aa8a5b0
types
kramstrom Dec 20, 2024
fa89c54
move repr inside
kramstrom Dec 20, 2024
e83725c
fix
kramstrom Dec 20, 2024
41b55d2
fix . pattern
kramstrom Dec 20, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions modal/_utils/docker_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -88,3 +88,24 @@ def valid_dockerignore_file(fp):
possible_locations.append(context_directory / generic_name)

return next((e for e in possible_locations if valid_dockerignore_file(e)), None)


class _AutoDockerIgnore:
_custom_repr: Optional[str] = None

def with_repr(self, custom_repr) -> "_AutoDockerIgnore":
# use to give an instance of a matcher a custom name - useful for visualizing default values in signatures
self._custom_repr = custom_repr
return self

def __repr__(self) -> str:
if self._custom_repr:
return self._custom_repr

return super().__repr__()

def __call__(self, _: Path) -> bool:
return True


AUTO_DOCKERIGNORE = _AutoDockerIgnore().with_repr(f"{__name__}.AUTO_DOCKERIGNORE")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's all this repr magic used for?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved the repr inside the class, but it's just a placeholder value for type hints & docs to make it simpler to understand what will happen if you don't pass in a custom ignore pattern-list or callable --> then we default to trying to find a matching dockerignore to your dockerfile (or just a generic one if you're using docker_commands) -- same as we have for add_local_python_source

Screenshot 2024-12-20 at 13 49 59

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TLDR for why it's needed (this was my suggestion)

It's so that we can use these as default values in signatures and still emit valid python signatures in our type stubs, since things like inspect.signature(...) will just use the repr of all default values, and can't figure out stuff like import path of the specific constant we use as a default.

It's definitely a bit of a hack, but not something users should see. An alternative would be to do a bunch of ast magic to extract the signatures.

19 changes: 12 additions & 7 deletions modal/image.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,12 @@
from ._utils.async_utils import synchronize_api
from ._utils.blob_utils import MAX_OBJECT_SIZE_BYTES
from ._utils.deprecation import deprecation_error, deprecation_warning
from ._utils.docker_utils import extract_copy_command_patterns, find_dockerignore_file
from ._utils.docker_utils import (
AUTO_DOCKERIGNORE,
_AutoDockerIgnore,
extract_copy_command_patterns,
find_dockerignore_file,
)
from ._utils.function_utils import FunctionInfo
from ._utils.grpc_utils import RETRYABLE_GRPC_STATUS_CODES, retry_transient_errors
from ._utils.pattern_utils import read_ignorefile
Expand Down Expand Up @@ -1181,7 +1186,7 @@ def dockerfile_commands(
# modal.Mount with local files to supply as build context for COPY commands
context_mount: Optional[_Mount] = None,
force_build: bool = False, # Ignore cached builds, similar to 'docker build --no-cache'
ignore: Optional[Union[Sequence[str], Callable[[Path], bool]]] = None,
ignore: Union[Sequence[str], Callable[[Path], bool]] = AUTO_DOCKERIGNORE,
) -> "_Image":
"""Extend an image with arbitrary Dockerfile-like commands."""
cmds = _flatten_str_args("dockerfile_commands", "dockerfile_commands", dockerfile_commands)
Expand All @@ -1195,7 +1200,7 @@ def dockerfile_commands(
+ " without this flag and can be ignored with the `ignore` argument."
+ " Defaults to using .dockerignore files.",
)
if ignore is not None:
if not isinstance(ignore, _AutoDockerIgnore):
raise InvalidError("Cannot set both `context_mount` and `ignore`")

def wrapper_context_mount_function():
Expand All @@ -1204,7 +1209,7 @@ def wrapper_context_mount_function():
context_mount_function = wrapper_context_mount_function

else:
if ignore is None:
if isinstance(ignore, _AutoDockerIgnore):
dockerignore_fp = find_dockerignore_file(Path.cwd())
if dockerignore_fp is not None:
with open(dockerignore_fp) as f:
Expand Down Expand Up @@ -1592,7 +1597,7 @@ def from_dockerfile(
add_python: Optional[str] = None,
# If ignore is set to None
# it will look for a dockerignore file to get ignore patterns
ignore: Optional[Union[Sequence[str], Callable[[Path], bool]]] = None,
ignore: Union[Sequence[str], Callable[[Path], bool]] = AUTO_DOCKERIGNORE,
) -> "_Image":
"""Build a Modal image from a local Dockerfile.

Expand All @@ -1616,15 +1621,15 @@ def from_dockerfile(
+ " without this flag and can be ignored with the `ignore` argument."
+ " Defaults to using .dockerignore files.",
)
if ignore is not None:
if not isinstance(ignore, _AutoDockerIgnore):
raise InvalidError("Cannot set both `context_mount` and `ignore`")

def wrapper_context_mount_function():
return context_mount

context_mount_function = wrapper_context_mount_function
else:
if ignore is None:
if isinstance(ignore, _AutoDockerIgnore):
dockerignore_fp = find_dockerignore_file(Path.cwd())
if dockerignore_fp is not None:
with open(dockerignore_fp) as f:
Expand Down
Loading