diff --git a/CHANGELOG.md b/CHANGELOG.md index a6a05a9c44..08a46ac9c2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,6 +20,11 @@ instructions, because git commits are used to generate release notes: + +## v16.1.1 (2023-08-29) + +- 💥[Bugfix] Apply "fix mysql crash after upgrade to Palm" from 16.1.0 to `tutor k8s` deployments, as well. Users previously running `tutor k8s` with `RUN_MYSQL: true`, with any version between 16.0.0 and 16.1.0 including, might have to fix their data manually. For users running `tutor local`, this change has no effect, as the underlying issue was already fixed in 16.1.0. For users running `tutor k8s` with `RUN_MYSQL: false`, this change is also a no-op. (by @fghaas) + ## v16.1.0 (2023-08-16) diff --git a/changelog.d/20230818_112124_kyle_buildkit.md b/changelog.d/20230818_112124_kyle_buildkit.md new file mode 100644 index 0000000000..7084b29041 --- /dev/null +++ b/changelog.d/20230818_112124_kyle_buildkit.md @@ -0,0 +1,2 @@ +- [Deprecation] The template variable ``is_buildkit_enabled``, which now always returns True, is deprecated. Plugin authors should assume BuildKit is enabled and remove the variable from their templates (by @kdmccormick). +- 💥[Deprecation] Tutor no longer supports the legacy Docker builder, which was previously available by setting ``DOCKER_BUILDKIT=0`` in the host environment. Going forward, Tutor will always use BuildKit (a.k.a. ``docker buildx`` in Docker v19-v22, or just ``docker build`` in Docker v23). This transition will improve build performance and should be seamless for Tutor users who are running a supported Docker version (by @kdmccormick). diff --git a/docs/install.rst b/docs/install.rst index 0c39f1e2eb..f328ac841f 100644 --- a/docs/install.rst +++ b/docs/install.rst @@ -9,7 +9,7 @@ Requirements ------------ * Supported OS: Tutor runs on any 64-bit, UNIX-based OS. It was also reported to work on Windows (with `WSL 2 `__). -* Architecture: support for ARM64 is a work-in-progress. See `this issue `__. +* Architecture: Both AMD64 and ARM64 are supported. * Required software: - `Docker `__: v20.10.15+ @@ -114,7 +114,7 @@ Upgrading to a new Open edX release Major Open edX releases are published twice a year, in June and December, by the Open edX `Build/Test/Release working group `__. When a new Open edX release comes out, Tutor gets a major version bump (see :ref:`versioning`). Such an upgrade typically includes multiple breaking changes. Any upgrade is final because downgrading is not supported. Thus, when upgrading your platform from one major version to the next, it is strongly recommended to do the following: 1. Read the changes listed in the `CHANGELOG.md `__ file. Breaking changes are identified by a "💥". -2. Perform a backup. On a local installation, this is typically done with:: +2. Perform a backup (see the :ref:`backup tutorial `). On a local installation, this is typically done with:: tutor local stop sudo rsync -avr "$(tutor config printroot)"/ /tmp/tutor-backup/ diff --git a/docs/intro.rst b/docs/intro.rst index bbef116f8d..81d59e0a17 100644 --- a/docs/intro.rst +++ b/docs/intro.rst @@ -51,7 +51,7 @@ Tutor simplifies the deployment of Open edX by: :width: 500px :align: center -Because Docker containers are becoming an industry-wide standard, that means that with Tutor it becomes possible to run Open edX anywhere: for now, Tutor supports deploying on a local server, with `docker-compose `_, and in a large cluster, with `Kubernetes `_. But in the future, Tutor may support other deployment platforms. +Because Docker containers are becoming an industry-wide standard, that means that with Tutor it becomes possible to run Open edX anywhere: for now, Tutor supports deploying on a local server, with `docker compose `_, and in a large cluster, with `Kubernetes `_. But in the future, Tutor may support other deployment platforms. Where can I try Open edX and Tutor? ----------------------------------- @@ -101,7 +101,7 @@ You can now take advantage of the Tutor-powered CLI (item #3) to bootstrap your tutor local launch -Under the hood, Tutor simply runs ``docker-compose`` and ``docker`` commands to launch your platform. These commands are printed in the standard output, such that you are free to replicate the same behaviour by simply copying/pasting the same commands. +Under the hood, Tutor simply runs ``docker compose`` and ``docker`` commands to launch your platform. These commands are printed in the standard output, such that you are free to replicate the same behaviour by simply copying/pasting the same commands. How do I navigate Tutor's command-line interface? ------------------------------------------------- diff --git a/docs/k8s.rst b/docs/k8s.rst index a7a80ad693..3d934d2c5f 100644 --- a/docs/k8s.rst +++ b/docs/k8s.rst @@ -119,7 +119,7 @@ Common tasks Executing commands inside service pods ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The Tutor and plugin documentation usually often instructions to execute some ``tutor local run ...`` commands. These commands are only valid when running Tutor locally with docker-compose, and will not work on Kubernetes. Instead, you should run ``tutor k8s exec ...`` commands. Arguments and options should be identical. +The Tutor and plugin documentation usually often instructions to execute some ``tutor local run ...`` commands. These commands are only valid when running Tutor locally with docker compose, and will not work on Kubernetes. Instead, you should run ``tutor k8s exec ...`` commands. Arguments and options should be identical. For instance, to run a Python shell in the lms container, run:: diff --git a/docs/local.rst b/docs/local.rst index 62cb431d89..7f3ebcb307 100644 --- a/docs/local.rst +++ b/docs/local.rst @@ -6,7 +6,7 @@ Local deployment This method is for deploying Open edX locally on a single server, where docker images are orchestrated with `docker-compose `_. .. note:: - Tutor is compatible with the ``docker compose`` subcommand. However, this support is still in beta and we're not sure it will behave the same as the previous ``docker-compose`` command. So ``docker-compose`` will be preferred, unless you set an environment variable ``TUTOR_USE_COMPOSE_SUBCOMMAND`` to enforce using ``docker compose``. + As of v16.0.0, Tutor now uses the ``docker compose`` subcommand instead of the separate ``docker-compose`` command. .. _tutor_root: diff --git a/docs/tutorials/datamigration.rst b/docs/tutorials/datamigration.rst index 95bfb86065..8434eeec79 100644 --- a/docs/tutorials/datamigration.rst +++ b/docs/tutorials/datamigration.rst @@ -1,3 +1,5 @@ +.. _backup_tutorial: + Making backups and migrating data --------------------------------- @@ -10,7 +12,7 @@ With Tutor, all data are stored in a single folder. This means that it's extreme 3. Transfer the configuration, environment, and platform data from server 1 to server 2:: - rsync -avr "$(tutor config printroot)/" username@server2:/tmp/tutor/ + sudo -avr "$(tutor config printroot)/" username@server2:/tmp/tutor/ 4. On server 2, move the data to the right location:: diff --git a/requirements/base.txt b/requirements/base.txt index 228782bb3c..dff55c9f19 100644 --- a/requirements/base.txt +++ b/requirements/base.txt @@ -1,5 +1,5 @@ # -# This file is autogenerated by pip-compile with Python 3.10 +# This file is autogenerated by pip-compile with Python 3.8 # by the following command: # # pip-compile requirements/base.in @@ -12,26 +12,28 @@ certifi==2023.7.22 # via # kubernetes # requests -charset-normalizer==3.1.0 +charset-normalizer==3.2.0 # via requests -click==8.1.3 +click==8.1.7 # via -r requirements/base.in -google-auth==2.19.1 +google-auth==2.22.0 # via kubernetes idna==3.4 # via requests jinja2==3.1.2 # via -r requirements/base.in -kubernetes==26.1.0 +kubernetes==27.2.0 # via -r requirements/base.in markupsafe==2.1.3 # via jinja2 -mypy==1.3.0 +mypy==1.5.1 # via -r requirements/base.in mypy-extensions==1.0.0 # via mypy oauthlib==3.2.2 - # via requests-oauthlib + # via + # kubernetes + # requests-oauthlib pyasn1==0.5.0 # via # pyasn1-modules @@ -42,7 +44,7 @@ pycryptodome==3.18.0 # via -r requirements/base.in python-dateutil==2.8.2 # via kubernetes -pyyaml==6.0 +pyyaml==6.0.1 # via # -r requirements/base.in # kubernetes @@ -61,7 +63,7 @@ six==1.16.0 # python-dateutil tomli==2.0.1 # via mypy -typing-extensions==4.6.3 +typing-extensions==4.7.1 # via # -r requirements/base.in # mypy @@ -70,8 +72,5 @@ urllib3==1.26.16 # google-auth # kubernetes # requests -websocket-client==1.5.2 +websocket-client==1.6.2 # via kubernetes - -# The following packages are considered to be unsafe in a requirements file: -# setuptools diff --git a/requirements/dev.in b/requirements/dev.in index 8dc0c3d1f8..c53771c419 100644 --- a/requirements/dev.in +++ b/requirements/dev.in @@ -7,10 +7,6 @@ pyinstaller scriv twine -# doc requirement is lagging behind -# https://github.com/readthedocs/sphinx_rtd_theme/issues/1323 -docutils<0.19 - # Types packages types-docutils types-PyYAML diff --git a/requirements/dev.txt b/requirements/dev.txt index c2400af9fa..abb13c7448 100644 --- a/requirements/dev.txt +++ b/requirements/dev.txt @@ -1,5 +1,5 @@ # -# This file is autogenerated by pip-compile with Python 3.10 +# This file is autogenerated by pip-compile with Python 3.8 # by the following command: # # pip-compile requirements/dev.in @@ -8,11 +8,11 @@ altgraph==0.17.3 # via pyinstaller appdirs==1.4.4 # via -r requirements/base.txt -astroid==2.15.5 +astroid==2.15.6 # via pylint attrs==23.1.0 # via scriv -black==23.3.0 +black==23.7.0 # via -r requirements/dev.in bleach==6.0.0 # via readme-renderer @@ -29,11 +29,11 @@ certifi==2023.7.22 # requests cffi==1.15.1 # via cryptography -charset-normalizer==3.1.0 +charset-normalizer==3.2.0 # via # -r requirements/base.txt # requests -click==8.1.3 +click==8.1.7 # via # -r requirements/base.txt # black @@ -42,17 +42,15 @@ click==8.1.3 # scriv click-log==0.4.0 # via scriv -coverage==7.2.7 +coverage==7.3.0 # via -r requirements/dev.in cryptography==41.0.3 # via secretstorage -dill==0.3.6 +dill==0.3.7 # via pylint -docutils==0.18.1 - # via - # -r requirements/dev.in - # readme-renderer -google-auth==2.19.1 +docutils==0.20.1 + # via readme-renderer +google-auth==2.22.0 # via # -r requirements/base.txt # kubernetes @@ -60,13 +58,15 @@ idna==3.4 # via # -r requirements/base.txt # requests -importlib-metadata==6.6.0 +importlib-metadata==6.8.0 # via # keyring # twine +importlib-resources==6.0.1 + # via keyring isort==5.12.0 # via pylint -jaraco-classes==3.2.3 +jaraco-classes==3.3.0 # via keyring jeepney==0.8.0 # via @@ -76,13 +76,13 @@ jinja2==3.1.2 # via # -r requirements/base.txt # scriv -keyring==23.13.1 +keyring==24.2.0 # via twine -kubernetes==26.1.0 +kubernetes==27.2.0 # via -r requirements/base.txt lazy-object-proxy==1.9.0 # via astroid -markdown-it-py==2.2.0 +markdown-it-py==3.0.0 # via rich markupsafe==2.1.3 # via @@ -92,9 +92,9 @@ mccabe==0.7.0 # via pylint mdurl==0.1.2 # via markdown-it-py -more-itertools==9.1.0 +more-itertools==10.1.0 # via jaraco-classes -mypy==1.3.0 +mypy==1.5.1 # via -r requirements/base.txt mypy-extensions==1.0.0 # via @@ -104,18 +104,19 @@ mypy-extensions==1.0.0 oauthlib==3.2.2 # via # -r requirements/base.txt + # kubernetes # requests-oauthlib packaging==23.1 # via # black # build -pathspec==0.11.1 +pathspec==0.11.2 # via black -pip-tools==6.13.0 +pip-tools==7.3.0 # via -r requirements/dev.in pkginfo==1.9.6 # via twine -platformdirs==3.5.1 +platformdirs==3.10.0 # via # black # pylint @@ -132,15 +133,15 @@ pycparser==2.21 # via cffi pycryptodome==3.18.0 # via -r requirements/base.txt -pygments==2.15.1 +pygments==2.16.1 # via # readme-renderer # rich -pyinstaller==5.11.0 +pyinstaller==5.13.1 # via -r requirements/dev.in -pyinstaller-hooks-contrib==2023.3 +pyinstaller-hooks-contrib==2023.7 # via pyinstaller -pylint==2.17.4 +pylint==2.17.5 # via -r requirements/dev.in pyproject-hooks==1.0.0 # via build @@ -148,11 +149,11 @@ python-dateutil==2.8.2 # via # -r requirements/base.txt # kubernetes -pyyaml==6.0 +pyyaml==6.0.1 # via # -r requirements/base.txt # kubernetes -readme-renderer==37.3 +readme-renderer==41.0 # via twine requests==2.31.0 # via @@ -170,7 +171,7 @@ requests-toolbelt==1.0.0 # via twine rfc3986==2.0.0 # via twine -rich==13.4.1 +rich==13.5.2 # via twine rsa==4.9 # via @@ -193,23 +194,27 @@ tomli==2.0.1 # black # build # mypy + # pip-tools # pylint # pyproject-hooks -tomlkit==0.11.8 +tomlkit==0.12.1 # via pylint twine==4.0.2 # via -r requirements/dev.in -types-docutils==0.20.0.1 +types-docutils==0.20.0.3 # via -r requirements/dev.in -types-pyyaml==6.0.12.10 +types-pyyaml==6.0.12.11 # via -r requirements/dev.in -types-setuptools==67.8.0.0 +types-setuptools==68.1.0.0 # via -r requirements/dev.in -typing-extensions==4.6.3 +typing-extensions==4.7.1 # via # -r requirements/base.txt # astroid + # black # mypy + # pylint + # rich urllib3==1.26.16 # via # -r requirements/base.txt @@ -219,16 +224,18 @@ urllib3==1.26.16 # twine webencodings==0.5.1 # via bleach -websocket-client==1.5.2 +websocket-client==1.6.2 # via # -r requirements/base.txt # kubernetes -wheel==0.40.0 +wheel==0.41.2 # via pip-tools wrapt==1.15.0 # via astroid -zipp==3.15.0 - # via importlib-metadata +zipp==3.16.2 + # via + # importlib-metadata + # importlib-resources # The following packages are considered to be unsafe in a requirements file: # pip diff --git a/requirements/docs.in b/requirements/docs.in index 79c02e20e3..1732d9b76f 100644 --- a/requirements/docs.in +++ b/requirements/docs.in @@ -1,4 +1,6 @@ -r base.txt -sphinx +# Python 3.8 support was dropped in 7.2.0 +# https://github.com/sphinx-doc/sphinx/pull/11511 +sphinx<7.2.0 sphinx-rtd-theme sphinx-click diff --git a/requirements/docs.txt b/requirements/docs.txt index cfa5d7195b..78d4debb08 100644 --- a/requirements/docs.txt +++ b/requirements/docs.txt @@ -1,5 +1,5 @@ # -# This file is autogenerated by pip-compile with Python 3.10 +# This file is autogenerated by pip-compile with Python 3.8 # by the following command: # # pip-compile requirements/docs.in @@ -19,11 +19,11 @@ certifi==2023.7.22 # -r requirements/base.txt # kubernetes # requests -charset-normalizer==3.1.0 +charset-normalizer==3.2.0 # via # -r requirements/base.txt # requests -click==8.1.3 +click==8.1.7 # via # -r requirements/base.txt # sphinx-click @@ -32,7 +32,7 @@ docutils==0.18.1 # sphinx # sphinx-click # sphinx-rtd-theme -google-auth==2.19.1 +google-auth==2.22.0 # via # -r requirements/base.txt # kubernetes @@ -42,17 +42,19 @@ idna==3.4 # requests imagesize==1.4.1 # via sphinx +importlib-metadata==6.8.0 + # via sphinx jinja2==3.1.2 # via # -r requirements/base.txt # sphinx -kubernetes==26.1.0 +kubernetes==27.2.0 # via -r requirements/base.txt markupsafe==2.1.3 # via # -r requirements/base.txt # jinja2 -mypy==1.3.0 +mypy==1.5.1 # via -r requirements/base.txt mypy-extensions==1.0.0 # via @@ -61,6 +63,7 @@ mypy-extensions==1.0.0 oauthlib==3.2.2 # via # -r requirements/base.txt + # kubernetes # requests-oauthlib packaging==23.1 # via sphinx @@ -75,13 +78,15 @@ pyasn1-modules==0.3.0 # google-auth pycryptodome==3.18.0 # via -r requirements/base.txt -pygments==2.15.1 +pygments==2.16.1 # via sphinx python-dateutil==2.8.2 # via # -r requirements/base.txt # kubernetes -pyyaml==6.0 +pytz==2023.3 + # via babel +pyyaml==6.0.1 # via # -r requirements/base.txt # kubernetes @@ -107,15 +112,15 @@ six==1.16.0 # python-dateutil snowballstemmer==2.2.0 # via sphinx -sphinx==6.2.1 +sphinx==7.1.2 # via # -r requirements/docs.in # sphinx-click # sphinx-rtd-theme # sphinxcontrib-jquery -sphinx-click==4.4.0 +sphinx-click==5.0.1 # via -r requirements/docs.in -sphinx-rtd-theme==1.2.1 +sphinx-rtd-theme==1.3.0 # via -r requirements/docs.in sphinxcontrib-applehelp==1.0.4 # via sphinx @@ -135,7 +140,7 @@ tomli==2.0.1 # via # -r requirements/base.txt # mypy -typing-extensions==4.6.3 +typing-extensions==4.7.1 # via # -r requirements/base.txt # mypy @@ -145,10 +150,9 @@ urllib3==1.26.16 # google-auth # kubernetes # requests -websocket-client==1.5.2 +websocket-client==1.6.2 # via # -r requirements/base.txt # kubernetes - -# The following packages are considered to be unsafe in a requirements file: -# setuptools +zipp==3.16.2 + # via importlib-metadata diff --git a/tests/commands/test_images.py b/tests/commands/test_images.py index 7b0957790d..61c4ccaeb2 100644 --- a/tests/commands/test_images.py +++ b/tests/commands/test_images.py @@ -128,10 +128,8 @@ def test_images_build_plugin_with_args(self, image_build: Mock) -> None: "service1", ] with temporary_root() as root: - utils.is_buildkit_enabled.cache_clear() - with patch.object(utils, "is_buildkit_enabled", return_value=False): - self.invoke_in_root(root, ["config", "save"]) - result = self.invoke_in_root(root, build_args) + self.invoke_in_root(root, ["config", "save"]) + result = self.invoke_in_root(root, build_args) self.assertIsNone(result.exception) self.assertEqual(0, result.exit_code) image_build.assert_called() diff --git a/tutor/__about__.py b/tutor/__about__.py index 7998b83929..40ffd7040e 100644 --- a/tutor/__about__.py +++ b/tutor/__about__.py @@ -2,7 +2,7 @@ # Increment this version number to trigger a new release. See # docs/tutor.html#versioning for information on the versioning scheme. -__version__ = "16.1.0" +__version__ = "16.1.1" # The version suffix will be appended to the actual version, separated by a # dash. Use this suffix to differentiate between the actual released version and diff --git a/tutor/commands/cli.py b/tutor/commands/cli.py index c8e552a2c9..e434a05e7a 100755 --- a/tutor/commands/cli.py +++ b/tutor/commands/cli.py @@ -32,7 +32,7 @@ def main() -> None: sys.exit(1) -class TutorCli(click.MultiCommand): +class TutorCli(click.Group): """ Dynamically load subcommands at runtime. @@ -43,26 +43,14 @@ class TutorCli(click.MultiCommand): IS_ROOT_READY = False - @classmethod - def iter_commands(cls, ctx: click.Context) -> t.Iterator[click.Command]: - """ - Return the list of subcommands (click.Command). - """ - cls.ensure_plugins_enabled(ctx) - yield from hooks.Filters.CLI_COMMANDS.iterate() - - @classmethod - def ensure_plugins_enabled(cls, ctx: click.Context) -> None: + def get_command( + self, ctx: click.Context, cmd_name: str + ) -> t.Optional[click.Command]: """ - We enable plugins as soon as possible to have access to commands. + This is run when passing a command from the CLI. E.g: tutor config ... """ - if not "root" in ctx.params: - # When generating docs, this function is called with empty args. - # That's ok, we just ignore it. - return - if not cls.IS_ROOT_READY: - hooks.Actions.PROJECT_ROOT_READY.do(ctx.params["root"]) - cls.IS_ROOT_READY = True + self.ensure_plugins_enabled(ctx) + return super().get_command(ctx, cmd_name=cmd_name) def list_commands(self, ctx: click.Context) -> list[str]: """ @@ -70,20 +58,22 @@ def list_commands(self, ctx: click.Context) -> list[str]: - shell autocompletion: tutor - print help: tutor, tutor -h """ - return sorted( - [command.name or "" for command in self.iter_commands(ctx)] - ) + self.ensure_plugins_enabled(ctx) + return super().list_commands(ctx) - def get_command( - self, ctx: click.Context, cmd_name: str - ) -> t.Optional[click.Command]: + def ensure_plugins_enabled(self, ctx: click.Context) -> None: """ - This is run when passing a command from the CLI. E.g: tutor config ... + We enable plugins as soon as possible to have access to commands. """ - for command in self.iter_commands(ctx): - if cmd_name == command.name: - return command - return None + if not "root" in ctx.params: + # When generating docs, this function is called with empty args. + # That's ok, we just ignore it. + return + if not self.IS_ROOT_READY: + hooks.Actions.PROJECT_ROOT_READY.do(ctx.params["root"]) + self.IS_ROOT_READY = True + for cmd in hooks.Filters.CLI_COMMANDS.iterate(): + self.add_command(cmd) @click.group( diff --git a/tutor/commands/compose.py b/tutor/commands/compose.py index baa1ca0016..76aab73664 100644 --- a/tutor/commands/compose.py +++ b/tutor/commands/compose.py @@ -20,6 +20,7 @@ from tutor.exceptions import TutorError from tutor.tasks import BaseComposeTaskRunner from tutor.types import Config +from tutor.utils import execute as execute_shell class ComposeTaskRunner(BaseComposeTaskRunner): @@ -114,6 +115,10 @@ def launch( click.echo(fmt.title("Docker image updates")) context.invoke(dc_command, command="pull") + if bindmount.get_mounts(config): + click.echo(fmt.title("Copying build artifacts into bind-mounted directories")) + context.invoke(copyartifacts) + click.echo(fmt.title("Starting the platform in detached mode")) context.invoke(start, detach=True) @@ -272,7 +277,7 @@ def reboot(context: click.Context, detach: bool, services: list[str]) -> None: @click.command( short_help="Restart some components from a running platform.", help="""Specify 'openedx' to restart the lms, cms and workers, or 'all' to -restart all services. Note that this performs a 'docker-compose restart', so new images +restart all services. Note that this performs a 'docker compose restart', so new images may not be taken into account. It is useful for reloading settings, for instance. To fully stop the platform, use the 'reboot' command.""", ) @@ -302,8 +307,8 @@ def do() -> None: @click.command( short_help="Run a command in a new container", help=( - "Run a command in a new container. This is a wrapper around `docker-compose run`. Any option or argument passed" - " to this command will be forwarded to docker-compose. Thus, you may use `-v` or `-p` to mount volumes and" + "Run a command in a new container. This is a wrapper around `docker compose run`. Any option or argument passed" + " to this command will be forwarded to docker compose. Thus, you may use `-v` or `-p` to mount volumes and" " expose ports." ), context_settings={"ignore_unknown_options": True}, @@ -365,10 +370,84 @@ def copyfrom( ) +@click.command( + help="TODO describe" +) +@click.argument( + "mount_paths", + metavar="mount_path", + nargs=-1, + type=click.Path(dir_okay=True, file_okay=False, resolve_path=True), +) +@click.pass_obj +def copyartifacts(context: BaseComposeContext, mount_paths: list[Path]) -> None: + """ + TODO write docstring + """ + config = tutor_config.load(context.root) + host_mount_paths: list[str] = [ + os.path.abspath(os.path.expanduser(mount_path)) + for mount_path + in mount_paths or bindmount.get_mounts(config) + ] + + # Sort out the (source, target) pairs by service name so that we can + # work one-at-a-time later. + copies_by_service: dict[str, tuple[str, str]] = {} + container_mounts_by_service: dict[str, list[str]] = {} + for host_mount_path in host_mount_paths: + mount_name = os.path.basename(host_mount_path) + for service, container_mount_path in hooks.Filters.COMPOSE_MOUNTS.iterate(mount_name): + for path_in_mount in hooks.Filters.COMPOSE_MOUNT_ARTIFACTS.iterate(mount_name, service): + source = f"{container_mount_path}/{path_in_mount}" + target = f"{host_mount_path}/{path_in_mount}" + copies_by_service.setdefault(service, []).append((source, target)) + container_mounts_by_service.setdefault(service, []).append(container_mount_path) + + container_name = "tutor_mounts_populate_temp" # TODO: improve name? + runner = context.job_runner(config) + + # For each service: create a temporary container, do the copy operations, and then kill the container. + for service, copies in copies_by_service.items(): + execute_shell("docker", "rm", "-f", container_name) + runner.docker_compose( + "run", + "--rm", + "--no-deps", + "--user=0", + # Recall that these artifact source directories may actually be bind-mounted into + # into the service from the host, which would prevent us from copying the original + # image's artifacts! + # To work around this, we shadow each relevant bind-mount with a fresh anonymous volume, + # which Docker populates with the image's original contents. + # The volume creation takes a bit of time, so we avoid using this shadowing strategy + # on any bind-mounts that aren't relevant to the operation. + *( + f"--volume={container_mount_path}" + for container_mount_path in set(container_mounts_by_service[service]) + ), + # Give the container a predetermined name so we can refer to it later. + "--name", + container_name, + # Run in the backround. + "--detach", + service, + # Rather than starting the service's real command, save some CPU by just sleeping + # until killed. + "sleep", + "infinity", + ) + for source, target in copies: + execute_shell("rm", "-rf", target) # Wipe any existing artifact. + execute_shell("sh", "-c", f'mkdir -p "$(dirname "{target}")"') # Ensure parent dirs exist. + execute_shell("docker", "cp", f"{container_name}:{source}", target) # Actually do the copy. + execute_shell("sh", "-c", f"docker kill '{container_name}' || true") + + @click.command( short_help="Run a command in a running container", help=( - "Run a command in a running container. This is a wrapper around `docker-compose exec`. Any option or argument" + "Run a command in a running container. This is a wrapper around `docker compose exec`. Any option or argument" " passed to this command will be forwarded to docker-compose. Thus, you may use `-e` to manually define" " environment variables." ), @@ -383,7 +462,7 @@ def execute(context: click.Context, args: list[str]) -> None: @click.command( short_help="View output from containers", - help="View output from containers. This is a wrapper around `docker-compose logs`.", + help="View output from containers. This is a wrapper around `docker compose logs`.", ) @click.option("-f", "--follow", is_flag=True, help="Follow log output") @click.option("--tail", type=int, help="Number of lines to show from each container") @@ -406,10 +485,10 @@ def status(context: click.Context) -> None: @click.command( - short_help="Direct interface to docker-compose.", + short_help="Direct interface to docker compose.", help=( - "Direct interface to docker-compose. This is a wrapper around `docker-compose`. Most commands, options and" - " arguments passed to this command will be forwarded as-is to docker-compose." + "Direct interface to docker compose. This is a wrapper around `docker compose`. Most commands, options and" + " arguments passed to this command will be forwarded as-is to docker compose." ), context_settings={"ignore_unknown_options": True}, name="dc", @@ -447,6 +526,27 @@ def _mount_edx_platform( return volumes +@hooks.Filters.COMPOSE_MOUNT_ARTIFACTS.add() +def _populate_edx_platform( + paths_to_copy: list[str], mount_name: str, service: str +) -> list[tuple[str, str]]: + """ + TODO describe + """ + if mount_name == "edx-platform" and service == "lms": + paths_to_copy += [ + "Open_edX.egg-info", + "node_modules", + "lms/static/css", + "lms/static/certificates/css", + "cms/static/css", + "common/static/bundles", + "common/static/common/js/vendor", + "common/static/common/css/vendor", + ] + return paths_to_copy + + @hooks.Filters.APP_PUBLIC_HOSTS.add() def _edx_platform_public_hosts( hosts: list[str], context_name: t.Literal["local", "dev"] @@ -471,6 +571,7 @@ def add_commands(command_group: click.Group) -> None: command_group.add_command(dc_command) command_group.add_command(run) command_group.add_command(copyfrom) + command_group.add_command(copyartifacts) command_group.add_command(execute) command_group.add_command(logs) command_group.add_command(status) diff --git a/tutor/commands/images.py b/tutor/commands/images.py index d21e2801ca..e8b43bbd2c 100644 --- a/tutor/commands/images.py +++ b/tutor/commands/images.py @@ -34,7 +34,7 @@ def _add_core_images_to_build( image, os.path.join("build", image), tutor_config.get_typed(config, tag, str), - (), + ("--target=production",), ) ) @@ -156,7 +156,7 @@ def images_command() -> None: # Export image to docker. This is necessary to make the image available to docker-compose. # The `--load` option is a shorthand for `--output=type=docker`. default="type=docker", - help="Same as `docker build --output=...`. This option will only be used when BuildKit is enabled.", + help="Same as `docker build --output=...`.", ) @click.option( "-a", @@ -211,7 +211,7 @@ def build( command_args += ["--add-host", add_host] if target: command_args += ["--target", target] - if utils.is_buildkit_enabled() and docker_output: + if docker_output: command_args.append(f"--output={docker_output}") if docker_args: command_args += docker_args @@ -223,27 +223,21 @@ def build( image_build_args = [*command_args, *custom_args] # Registry cache - if utils.is_buildkit_enabled(): - if not no_registry_cache: - image_build_args.append( - f"--cache-from=type=registry,ref={tag}-cache" - ) - if cache_to_registry: - image_build_args.append( - f"--cache-to=type=registry,mode=max,ref={tag}-cache" - ) + if not no_registry_cache: + image_build_args.append( + f"--cache-from=type=registry,ref={tag}-cache" + ) + if cache_to_registry: + image_build_args.append( + f"--cache-to=type=registry,mode=max,ref={tag}-cache" + ) # Build contexts for host_path, stage_name in build_contexts.get(name, []): - if utils.is_buildkit_enabled(): - fmt.echo_info( - f"Adding {host_path} to the build context '{stage_name}' of image '{image}'" - ) - image_build_args.append(f"--build-context={stage_name}={host_path}") - else: - fmt.echo_alert( - f"Unable to add {host_path} to the build context '{stage_name}' of image '{host_path}' because BuildKit is disabled." - ) + fmt.echo_info( + f"Adding {host_path} to the build context '{stage_name}' of image '{image}'" + ) + image_build_args.append(f"--build-context={stage_name}={host_path}") # Build images.build( diff --git a/tutor/commands/jobs.py b/tutor/commands/jobs.py index 562e78d454..f31cef08a4 100644 --- a/tutor/commands/jobs.py +++ b/tutor/commands/jobs.py @@ -42,15 +42,6 @@ def _add_core_init_tasks() -> None: ("mysql", env.read_core_template_file("jobs", "init", "mysql.sh")) ) with hooks.Contexts.app("lms").enter(): - hooks.Filters.CLI_DO_INIT_TASKS.add_item( - ( - "lms", - env.read_core_template_file("jobs", "init", "mounted-edx-platform.sh"), - ), - # If edx-platform is mounted, then we may need to perform some setup - # before other initialization scripts can be run. - priority=priorities.HIGH, - ) hooks.Filters.CLI_DO_INIT_TASKS.add_item( ("lms", env.read_core_template_file("jobs", "init", "lms.sh")) ) diff --git a/tutor/env.py b/tutor/env.py index 980bb77349..3183b1b37f 100644 --- a/tutor/env.py +++ b/tutor/env.py @@ -54,7 +54,7 @@ def _prepare_environment() -> None: ("HOST_USER_ID", utils.get_user_id()), ("TUTOR_APP", __app__.replace("-", "_")), ("TUTOR_VERSION", __version__), - ("is_buildkit_enabled", utils.is_buildkit_enabled), + ("is_buildkit_enabled", lambda: True), # Will be removed soon. ], ) diff --git a/tutor/hooks/catalog.py b/tutor/hooks/catalog.py index e1eac6d71f..7ceaa616e1 100644 --- a/tutor/hooks/catalog.py +++ b/tutor/hooks/catalog.py @@ -50,7 +50,7 @@ def run_this_on_start(root, config, name): For more information about how actions work, check out the :py:class:`tutor.core.hooks.Action` API. """ - #: Triggered whenever a "docker-compose start", "up" or "restart" command is executed. + #: Triggered whenever a "docker compose start", "up" or "restart" command is executed. #: #: :parameter str root: project root. #: :parameter dict config: project configuration. @@ -222,6 +222,32 @@ def your_filter_callback(some_data): #: conditionally add mounts. COMPOSE_MOUNTS: Filter[list[tuple[str, str]], [str]] = Filter() + #: Relative paths of build artifacts, to be copied into a host-mounted folder from a service image. + #: + #: Docker images contain many build artifacts, such as generated assets and packaging metadata, which + #: must exist at runtime in order for services to run properly. When a user bind-mounts a directory + #: from their host machine, there is no guarantee that the host directory will contain those essential + #: artifacts, and regenerating them from source may be time consuming. To remedy this, Tutor provides + #: the ``copyartifacts`` command, which efficiently copies the necessary artifacts from the original image + #: into the host directory. This command is also automatically run as part of ``launch``. + #: + #: The ``COMPOSE_MOUNT_ARTIFACTS`` filter tells Tutor which artifacts must be copied from which + #: service for any given host-mounted directory. By default, for edx-platform, this includes + #: several directories such as ``node_modules`` and ``lms/static/css``, to be copied from the lms + #: service's image. + #: + #: Note that any given artifact should only be specified once in this filter. If an artifact + #: exists on an image used by multiple services, choose one of those service for it to be + #: copied from. + #: + #: :parameter artifacts list[str]: files or directories considered build artifacts for the given + #: host-mounted folder in the given service. Paths must be relative to the mount directory. + #: :parameter str mount_name: basename of a host-mounted folder. + #: :parameter service: name of a service from which artifackts will be copied. The ``COMPOSE_MOUNTS`` + #: filter should map ``mount_name`` somewhere in this container; that mount location will be used + #: as the source for ``artifacts``. + COMPOSE_MOUNT_ARTIFACTS: Filter[list[str], [str, str]] = Filter() + #: Declare new default configuration settings that don't necessarily have to be saved in the user #: ``config.yml`` file. Default settings may be overridden with ``tutor config save --set=...``, in which #: case they will automatically be added to ``config.yml``. @@ -244,8 +270,7 @@ def your_filter_callback(some_data): #: names must be prefixed with the plugin name in all-caps. CONFIG_UNIQUE: Filter[list[tuple[str, Any]], []] = Filter() - #: Use this filter to modify the ``docker build`` command. For instance, to replace - #: the ``build`` subcommand by ``buildx build``. + #: Use this filter to modify the ``docker build`` command. #: #: :parameter list[str] command: the full build command, including options and #: arguments. Note that these arguments do not include the leading ``docker`` command. @@ -328,7 +353,7 @@ def your_filter_callback(some_data): #: - ``HOST_USER_ID``: the numerical ID of the user on the host. #: - ``TUTOR_APP``: the app name ("tutor" by default), used to determine the dev/local project names. #: - ``TUTOR_VERSION``: the current version of Tutor. - #: - ``is_buildkit_enabled``: a boolean function that indicates whether BuildKit is available on the host. + #: - ``is_buildkit_enabled``: a deprecated function which always returns ``True`` now. Will be removed before Quince. #: - ``iter_values_named``: a function to iterate on variables that start or end with a given string. #: - ``iter_mounts``: a function that yields compose-compatible bind-mounts for any given service. #: - ``patch``: a function to incorporate extra content into a template. diff --git a/tutor/images.py b/tutor/images.py index 26f80b1824..66f2b0b1c9 100644 --- a/tutor/images.py +++ b/tutor/images.py @@ -4,8 +4,7 @@ def build(path: str, tag: str, *args: str) -> None: fmt.echo_info(f"Building image {tag}") build_command = ["build", f"--tag={tag}", *args, path] - if utils.is_buildkit_enabled(): - build_command.insert(0, "buildx") + build_command.insert(0, "buildx") command = hooks.Filters.DOCKER_BUILD_COMMAND.apply(build_command) utils.docker(*command) diff --git a/tutor/templates/build/openedx/Dockerfile b/tutor/templates/build/openedx/Dockerfile index 8df5eb8aec..a07939cd5f 100644 --- a/tutor/templates/build/openedx/Dockerfile +++ b/tutor/templates/build/openedx/Dockerfile @@ -1,30 +1,46 @@ -{% if is_buildkit_enabled() %}# syntax=docker/dockerfile:1.4{% endif %} -###### Minimal image with base system requirements for most stages +# syntax=docker/dockerfile:1.4 + + +########################################################################################################################### +############## MINIMAL +############## Minimal image with base system requirements for most stages +########################################################################################################################### FROM docker.io/ubuntu:20.04 as minimal LABEL maintainer="Overhang.io " ENV DEBIAN_FRONTEND=noninteractive -RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/var/cache/apt,sharing=locked \ - --mount=type=cache,target=/var/lib/apt,sharing=locked{% endif %} \ +RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ + --mount=type=cache,target=/var/lib/apt,sharing=locked \ apt update && \ apt install -y build-essential curl git language-pack-en ENV LC_ALL en_US.UTF-8 {{ patch("openedx-dockerfile-minimal") }} -###### Install python with pyenv in /opt/pyenv and create virtualenv in /openedx/venv + +########################################################################################################################### +############## PYTHON +############## Install python with pyenv in /opt/pyenv and create virtualenv in /openedx/venv +########################################################################################################################### FROM minimal as python + # https://github.com/pyenv/pyenv/wiki/Common-build-problems#prerequisites -RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/var/cache/apt,sharing=locked \ - --mount=type=cache,target=/var/lib/apt,sharing=locked {% endif %}apt update && \ +RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ + --mount=type=cache,target=/var/lib/apt,sharing=locked \ + apt update && \ apt install -y libssl-dev zlib1g-dev libbz2-dev \ libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \ - xz-utils tk-dev libffi-dev liblzma-dev python-openssl git + xz-utils tk-dev libffi-dev liblzma-dev python-openssl git \ + software-properties-common libmysqlclient-dev libxmlsec1-dev libgeos-dev + +ARG PYTHON_VERSION=3.8.15 +ENV PYENV_ROOT /opt/pyenv +ENV PATH /openedx/venv/bin:${PATH} +ENV VIRTUAL_ENV /openedx/venv/ +ENV XDG_CACHE_HOME /openedx/.cache # Install pyenv # https://www.python.org/downloads/ # https://github.com/pyenv/pyenv/releases -ARG PYTHON_VERSION=3.8.15 -ENV PYENV_ROOT /opt/pyenv RUN git clone https://github.com/pyenv/pyenv $PYENV_ROOT --branch v2.3.17 --depth 1 # Install Python @@ -33,8 +49,21 @@ RUN $PYENV_ROOT/bin/pyenv install $PYTHON_VERSION # Create virtualenv RUN $PYENV_ROOT/versions/$PYTHON_VERSION/bin/python -m venv /openedx/venv -###### Checkout edx-platform code +# Install the right version of pip/setuptools +RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ + pip install \ + # https://pypi.org/project/setuptools/ + # https://pypi.org/project/pip/ + # https://pypi.org/project/wheel/ + setuptools==67.6.1 pip==23.0.1. wheel==0.40.0 + + +########################################################################################################################### +############## CODE +########################################################################################################################### FROM minimal as code + +###### Checkout edx-platform code ARG EDX_PLATFORM_REPOSITORY={{ EDX_PLATFORM_REPOSITORY }} ARG EDX_PLATFORM_VERSION={{ EDX_PLATFORM_VERSION }} RUN mkdir -p /openedx/edx-platform && \ @@ -55,12 +84,24 @@ RUN git config --global user.email "tutor@overhang.io" \ {# Example: RUN curl -fsSL https://github.com/openedx/edx-platform/commit/.patch | git am #} {{ patch("openedx-dockerfile-post-git-checkout") }} -##### Empty layer with just the repo at the root. -# This is useful when overriding the build context with a host repo: -# docker build --build-context edx-platform=/path/to/edx-platform +# TODO: Temporary +# Apply these edx-platform changes which have not yet merged. +RUN curl -fsSL https://github.com/openedx/edx-platform/pull/32835.patch | git am + + +########################################################################################################################### +############## EDX-PLATFORM +############## Empty layer with just the repo at the root. +############## This is useful when overriding the build context with a host repo: +############## docker build --build-context edx-platform=/path/to/edx-platform +########################################################################################################################### FROM scratch as edx-platform COPY --from=code /openedx/edx-platform / + +########################################################################################################################### +############## LOCALES +########################################################################################################################### ###### Download extra locales to /openedx/locale/contrib/locale FROM minimal as locales ARG OPENEDX_I18N_VERSION={{ OPENEDX_COMMON_VERSION }} @@ -71,32 +112,126 @@ RUN cd /tmp \ && mv openedx-i18n-*/edx-platform/locale /openedx/locale/contrib \ && rm -rf openedx-i18n* -###### Install python requirements in virtualenv -FROM python as python-requirements -ENV PATH /openedx/venv/bin:${PATH} -ENV VIRTUAL_ENV /openedx/venv/ -ENV XDG_CACHE_HOME /openedx/.cache -RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/var/cache/apt,sharing=locked \ - --mount=type=cache,target=/var/lib/apt,sharing=locked {% endif %}apt update \ - && apt install -y software-properties-common libmysqlclient-dev libxmlsec1-dev libgeos-dev -# Install the right version of pip/setuptools -RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/openedx/.cache/pip,sharing=shared {% endif %}pip install \ - # https://pypi.org/project/setuptools/ - # https://pypi.org/project/pip/ - # https://pypi.org/project/wheel/ - setuptools==67.6.1 pip==23.0.1. wheel==0.40.0 +########################################################################################################################### +############## FRONTEND REQUIREMENTS +########################################################################################################################### +FROM python as frontend-requirements + +WORKDIR /openedx/edx-platform + +# Install python reqs for setting up node & building assets +COPY --link --from=edx-platform /requirements/edx/assets.txt requirements/edx/assets.txt +RUN pip install -r requirements/edx/assets.txt + +# Set up node env +ENV PATH /openedx/nodeenv/bin:/openedx/venv/bin:${PATH} +RUN nodeenv /openedx/nodeenv --node=16.14.0 --prebuilt + +# Install nodejs requirements +ARG NPM_REGISTRY={{ NPM_REGISTRY }} +COPY --link --from=edx-platform /package.json package.json +COPY --link --from=edx-platform /package-lock.json package-lock.json +COPY --link --from=edx-platform /scripts/copy-node-modules.sh scripts/copy-node-modules.sh +RUN --mount=type=cache,target=/root/.npm,sharing=shared \ + npm clean-install --no-audit --registry=$NPM_REGISTRY + +# Set up static root. +# By default, CMS static root is assumed to be $STUDIO_ROOT_LMS/studio. +ENV STATIC_ROOT_LMS=/openedx/staticfiles +RUN mkdir -p "$STATIC_ROOT_LMS" + + +########################################################################################################################### +############## BUNDLES +########################################################################################################################### + +FROM frontend-requirements as bundles + +COPY --link --from=edx-platform /.babelrc .babelrc +COPY --link --from=edx-platform /webpack.common.config.js webpack.common.config.js +COPY --link --from=edx-platform /webpack-config/file-lists.js webpack-config/file-lists.js +COPY --link --from=edx-platform /webpack.dev.config.js webpack.dev.config.js +COPY --link --from=edx-platform /webpack.prod.config.js webpack.prod.config.js +COPY --link --from=edx-platform /webpack.builtinblocks.config.js webpack.builtinblocks.config.js +COPY --link --from=edx-platform /cms/djangoapps/pipeline_js/js/xmodule.js cms/djangoapps/pipeline_js/js/xmodule.js +COPY --link --from=edx-platform /cms/static cms/static +COPY --link --from=edx-platform /cms/templates/ cms/templates/ +COPY --link --from=edx-platform /common/static/js/ common/static/js +COPY --link --from=edx-platform /common/static/common common/static/common +COPY --link --from=edx-platform /lms/djangoapps/discussion/static lms/djangoapps/discussion/static +COPY --link --from=edx-platform /lms/djangoapps/instructor/static lms/djangoapps/instructor/static +COPY --link --from=edx-platform /lms/djangoapps/support/static/support lms/djangoapps/support/static/support +COPY --link --from=edx-platform /lms/djangoapps/teams/static lms/djangoapps/teams/static +COPY --link --from=edx-platform /lms/static/ lms/static/ +COPY --link --from=edx-platform /lms/templates/ lms/templates/ +COPY --link --from=edx-platform /openedx/features/announcements/static openedx/features/announcements/static +COPY --link --from=edx-platform /openedx/features/course_bookmarks/static openedx/features/course_bookmarks/static +COPY --link --from=edx-platform /openedx/features/course_experience/static openedx/features/course_experience/static +COPY --link --from=edx-platform /openedx/features/course_search/static openedx/features/course_search/static +COPY --link --from=edx-platform /openedx/features/learner_profile/static openedx/features/learner_profile/static +COPY --link --from=edx-platform /xmodule/assets xmodule/assets +COPY --link --from=edx-platform /xmodule/js xmodule/js + +COPY --link --from=frontend-requirements /openedx/edx-platform/common/static/common/js/vendor common/static/common/js/vendor +COPY --link --from=frontend-requirements /openedx/edx-platform/common/static/common/css/vendor common/static/common/css/vendor + +FROM bundles as bundles-production +RUN npm run webpack + +FROM bundles as bundles-development +RUN npm run webpack-dev + + +########################################################################################################################### +############## CSS +########################################################################################################################### + +FROM frontend-requirements as css + +ENV PATH ./node_modules/.bin:${PATH} + +COPY --link --from=edx-platform /scripts/compile_sass.py scripts/compile_sass.py +COPY --link --from=edx-platform /common/static common/static +COPY --link --from=edx-platform /lms/static/sass lms/static/sass +COPY --link --from=edx-platform /lms/static/sass/partials lms/static/sass/partials +COPY --link --from=edx-platform /lms/static/certificates/sass lms/static/certificates/sass +COPY --link --from=edx-platform /cms/static/sass cms/static/sass +COPY --link --from=edx-platform /cms/static/sass/partials cms/static/sass/partials +COPY --link --from=edx-platform /xmodule/assets xmodule/assets +COPY --link --from=edx-platform --chown=app:app /lms/static/css/vendor lms/static/css/vendor + +FROM css as css-production +# Compile default CSS +RUN npm run compile-sass -- --skip-themes +# Compile themed CSS +ENV EDX_PLATFORM_THEME_DIRS=/openedx/themes +COPY --link --chown=app:app ./themes/ /openedx/themes/ +RUN npm run compile-sass -- --skip-default + +FROM css as css-development +# Compile default CSS +RUN npm run compile-sass -- --skip-themes --env=dev +# Compile themed CSS +ENV EDX_PLATFORM_THEME_DIRS=/openedx/themes +COPY --link --chown=app:app ./themes/ /openedx/themes/ +RUN npm run compile-sass -- --skip-default --env=dev + + +########################################################################################################################### +############## REQUIREMENTS (both frontend + backend) +########################################################################################################################### +FROM frontend-requirements as requirements # Install base requirements -{% if not is_buildkit_enabled() %} -COPY --from=edx-platform /requirements/edx/base.txt /openedx/edx-platform/requirements/edx/base.txt -{% endif %} -RUN {% if is_buildkit_enabled() %}--mount=type=bind,from=edx-platform,source=/requirements/edx/base.txt,target=/openedx/edx-platform/requirements/edx/base.txt \ - --mount=type=cache,target=/openedx/.cache/pip,sharing=shared {% endif %}pip install -r /openedx/edx-platform/requirements/edx/base.txt +COPY --link --from=edx-platform /requirements/edx/base.txt /openedx/edx-platform/requirements/edx/base.txt +RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ + pip install -r /openedx/edx-platform/requirements/edx/base.txt # Install extra requirements -RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/openedx/.cache/pip,sharing=shared {% endif %}pip install \ +RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ + pip install \ # Use redis as a django cache https://pypi.org/project/django-redis/ django-redis==5.2.0 \ # uwsgi server https://pypi.org/project/uWSGI/ @@ -105,42 +240,40 @@ RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/openedx/.cache/pip, {{ patch("openedx-dockerfile-post-python-requirements") }} # Install private requirements: this is useful for installing custom xblocks. -COPY ./requirements/ /openedx/requirements -RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/openedx/.cache/pip,sharing=shared {% endif %}cd /openedx/requirements/ \ - && touch ./private.txt \ - && pip install -r ./private.txt +COPY --link ./requirements/ /openedx/requirements +RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ + cd /openedx/requirements/ \ + && touch ./private.txt \ + && pip install -r ./private.txt -{% for extra_requirements in OPENEDX_EXTRA_PIP_REQUIREMENTS %}RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/openedx/.cache/pip,sharing=shared {% endif %}pip install '{{ extra_requirements }}' +{% for extra_requirements in OPENEDX_EXTRA_PIP_REQUIREMENTS %}RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared pip install '{{ extra_requirements }}' {% endfor %} -###### Install nodejs with nodeenv in /openedx/nodeenv -FROM python as nodejs-requirements -ENV PATH /openedx/nodeenv/bin:/openedx/venv/bin:${PATH} +FROM requirements as requirements-production +# To make prod requirements, we simply take the general requirements, and remove everything related to +# libsass (which is installed via assets.txt) in order to shrink the final image. We don't need to +# compile Sass in production. +# Now, what we *could* have done was base this stage on minimal, thus avoiding assets.txt altogether. +# However, this would mean that we'd have to install libsass *twice*: once for frontend-requirements and +# once for requirements. Since libsass takes so long to compile, that doesn't seem worth it. +RUN rm -r /openedx/venv/lib/python3.8/site-packages/*sass* -# Install nodeenv with the version provided by edx-platform -# https://github.com/openedx/edx-platform/blob/master/requirements/edx/base.txt -# https://github.com/pyenv/pyenv/releases -RUN pip install nodeenv==1.7.0 -RUN nodeenv /openedx/nodeenv --node=16.14.0 --prebuilt +FROM requirements as requirements-development +# Install dev requirements. +COPY --link --from=edx-platform /requirements/edx/development.txt /openedx/edx-platform/requirements/edx/development.txt +RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ + pip install -r /openedx/edx-platform/requirements/edx/development.txt -# Install nodejs requirements -ARG NPM_REGISTRY={{ NPM_REGISTRY }} -WORKDIR /openedx/edx-platform -{% if not is_buildkit_enabled() %} -COPY --from=edx-platform /package.json /openedx/edx-platform/package.json -COPY --from=edx-platform /package-lock.json /openedx/edx-platform/package-lock.json -{% endif %} -RUN {% if is_buildkit_enabled() %}--mount=type=bind,from=edx-platform,source=/package.json,target=/openedx/edx-platform/package.json \ - --mount=type=bind,from=edx-platform,source=/package-lock.json,target=/openedx/edx-platform/package-lock.json \ - --mount=type=bind,from=edx-platform,source=/scripts/copy-node-modules.sh,target=/openedx/edx-platform/scripts/copy-node-modules.sh \ - --mount=type=cache,target=/root/.npm,sharing=shared {% endif %}npm clean-install --no-audit --registry=$NPM_REGISTRY - -###### Production image with system and python requirements -FROM minimal as production + +########################################################################################################################### +############## APPLICATION (base for final production and development images) +########################################################################################################################### +FROM minimal as application # Install system requirements -RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/var/cache/apt,sharing=locked \ - --mount=type=cache,target=/var/lib/apt,sharing=locked {% endif %}apt update \ +RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ + --mount=type=cache,target=/var/lib/apt,sharing=locked \ + apt update \ && apt install -y gettext gfortran graphviz graphviz-dev libffi-dev libfreetype6-dev libgeos-dev libjpeg8-dev liblapack-dev libmysqlclient-dev libpng-dev libsqlite3-dev libxmlsec1-dev lynx mysql-client ntp pkg-config rdfind # From then on, run as unprivileged "app" user @@ -150,18 +283,15 @@ RUN if [ "$APP_USER_ID" = 0 ]; then echo "app user may not be root" && false; fi RUN useradd --home-dir /openedx --create-home --shell /bin/bash --uid ${APP_USER_ID} app USER ${APP_USER_ID} -# https://hub.docker.com/r/powerman/dockerize/tags -COPY {% if is_buildkit_enabled() %}--link {% endif %}--from=docker.io/powerman/dockerize:0.19.0 /usr/local/bin/dockerize /usr/local/bin/dockerize -COPY --chown=app:app --from=edx-platform / /openedx/edx-platform -COPY --chown=app:app --from=locales /openedx/locale /openedx/locale -COPY --chown=app:app --from=python /opt/pyenv /opt/pyenv -COPY --chown=app:app --from=python-requirements /openedx/venv /openedx/venv -COPY --chown=app:app --from=python-requirements /openedx/requirements /openedx/requirements -COPY --chown=app:app --from=nodejs-requirements /openedx/nodeenv /openedx/nodeenv -COPY --chown=app:app --from=nodejs-requirements /openedx/edx-platform/node_modules /openedx/node_modules - -# Symlink node_modules such that we can bind-mount the edx-platform repository -RUN ln -s /openedx/node_modules /openedx/edx-platform/node_modules +# Copy in backend code and requirements. +# Dockerize tags: https://hub.docker.com/r/powerman/dockerize/tags +COPY --link --from=docker.io/powerman/dockerize:0.19.0 /usr/local/bin/dockerize /usr/local/bin/dockerize +COPY --link --chown=app:app --from=edx-platform / /openedx/edx-platform +COPY --link --chown=app:app ./themes/ /openedx/themes/ +COPY --link --chown=app:app --from=locales /openedx/locale /openedx/locale +COPY --link --chown=app:app --from=python /opt/pyenv /opt/pyenv +COPY --link --chown=app:app --from=requirements-production /openedx/venv /openedx/venv +COPY --link --chown=app:app --from=requirements-production /openedx/requirements /openedx/requirements ENV PATH /openedx/venv/bin:./node_modules/.bin:/openedx/nodeenv/bin:${PATH} ENV VIRTUAL_ENV /openedx/venv/ @@ -187,6 +317,20 @@ COPY --chown=app:app ./locale/ /openedx/locale/user/locale/ RUN cd /openedx/locale/user && \ django-admin compilemessages -v1 +# Copy in frontend requirements. +COPY --link --chown=app:app --from=frontend-requirements \ + /openedx/nodeenv \ + /openedx/nodeenv +COPY --link --chown=app:app --from=frontend-requirements \ + /openedx/edx-platform/node_modules \ + /openedx/edx-platform/node_modules +COPY --link --chown=app:app --from=frontend-requirements \ + /openedx/edx-platform/common/static/common/js/vendor \ + /openedx/edx-platform/common/static/common/js/vendor +COPY --link --chown=app:app --from=frontend-requirements \ + /openedx/edx-platform/common/static/common/css/vendor \ + /openedx/edx-platform/common/static/common/css/vendor + # Compile i18n strings: in some cases, js locales are not properly compiled out of the box # and we need to do a pass ourselves. Also, we need to compile the djangojs.js files for # the downloaded locales. @@ -194,77 +338,85 @@ RUN ./manage.py lms --settings=tutor.i18n compilejsi18n RUN ./manage.py cms --settings=tutor.i18n compilejsi18n # Copy scripts -COPY --chown=app:app ./bin /openedx/bin +COPY --link --chown=app:app ./bin /openedx/bin RUN chmod a+x /openedx/bin/* ENV PATH /openedx/bin:${PATH} -{{ patch("openedx-dockerfile-pre-assets") }} - -# Collect production assets. By default, only assets from the default theme -# will be processed. This makes the docker image lighter and faster to build. -# Only the custom themes added to /openedx/themes will be compiled. -# Here, we don't run "paver update_assets" which is slow, compiles all themes -# and requires a complex settings file. Instead, we decompose the commands -# and run each one individually to collect the production static assets to -# /openedx/staticfiles. -ENV NO_PYTHON_UNINSTALL 1 -ENV NO_PREREQ_INSTALL 1 -# We need to rely on a separate openedx-assets command to accelerate asset processing. -# For instance, we don't want to run all steps of asset collection every time the theme -# is modified. -RUN openedx-assets xmodule \ - && openedx-assets npm \ - && openedx-assets webpack --env=prod \ - && openedx-assets common -COPY --chown=app:app ./themes/ /openedx/themes/ -RUN openedx-assets themes \ - && openedx-assets collect --settings=tutor.assets \ - # De-duplicate static assets with symlinks - && rdfind -makesymlinks true -followsymlinks true /openedx/staticfiles/ - # Create a data directory, which might be used (or not) RUN mkdir /openedx/data -# If this "canary" file is missing from a container, then that indicates that a -# local edx-platform was bind-mounted into that container, thus overwriting the -# canary. This information is useful during edx-platform initialisation. -RUN echo \ - "This copy of edx-platform was built into a Docker image." \ - > bindmount-canary - # service variant is "lms" or "cms" ENV SERVICE_VARIANT lms -ENV DJANGO_SETTINGS_MODULE lms.envs.tutor.production {{ patch("openedx-dockerfile") }} EXPOSE 8000 -###### Intermediate image with dev/test dependencies -FROM production as development + +########################################################################################################################### +############## PRODUCTION +########################################################################################################################### +FROM application as production + +# Copy in and collect production static assets. +COPY --link --chown=app:app --from=bundles-production /openedx/staticfiles /openedx/staticfiles +COPY --link --chown=app:app --from=bundles-production /openedx/edx-platform/common/static/bundles common/static/bundles +COPY --link --chown=app:app --from=css-production /openedx/edx-platform/lms/static/css lms/static/css +COPY --link --chown=app:app --from=css-production /openedx/edx-platform/lms/static/certificates/css lms/static/certificates/css +COPY --link --chown=app:app --from=css-production /openedx/edx-platform/cms/static/css cms/static/css +RUN ./manage.py lms collectstatic --noinput --settings=tutor.assets +RUN ./manage.py cms collectstatic --noinput --settings=tutor.assets + +# De-dupe static assets with synlinks +RUN rdfind -makesymlinks true -followsymlinks true /openedx/staticfiles/ + +# Default amount of uWSGI processes +ENV UWSGI_WORKERS=2 + +# Copy the default uWSGI configuration +COPY --chown=app:app settings/uwsgi.ini . + +# Default django settings +ENV DJANGO_SETTINGS_MODULE lms.envs.tutor.production + +# Run server +CMD uwsgi uwsgi.ini + +{{ patch("openedx-dockerfile-final") }} + + +########################################################################################################################### +############## DEVELOPMENT +########################################################################################################################### +FROM application as development # Install useful system requirements (as root) USER root -RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/var/cache/apt,sharing=locked \ - --mount=type=cache,target=/var/lib/apt,sharing=locked {% endif %}apt update && \ +RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ + --mount=type=cache,target=/var/lib/apt,sharing=locked \ + apt update && \ apt install -y vim iputils-ping dnsutils telnet USER app -# Install dev python requirements -RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/openedx/.cache/pip,sharing=shared {% endif %}pip install -r requirements/edx/development.txt +# Copy in dev python requirements +COPY --link --chown=app:app --from=requirements-development /openedx/venv /openedx/venv + # https://pypi.org/project/ipdb/ # https://pypi.org/project/ipython -RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/openedx/.cache/pip,sharing=shared {% endif %}pip install ipdb==0.13.13 ipython==8.12.0 +RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ + pip install ipdb==0.13.13 ipython==8.12.0 # Add ipdb as default PYTHONBREAKPOINT ENV PYTHONBREAKPOINT=ipdb.set_trace -# Recompile static assets: in development mode all static assets are stored in edx-platform, -# and the location of these files is stored in webpack-stats.json. If we don't recompile -# static assets, then production assets will be served instead. -RUN rm -r /openedx/staticfiles && \ - mkdir /openedx/staticfiles && \ - openedx-assets webpack --env=dev +# Copy in development static assets. +# In development mode, edx-platform expects the files to be in the repo, and pointed to by webpack-stats.json. +COPY --link --chown=app:app --from=bundles-development /openedx/staticfiles/webpack-stats.json /openedx/staticfiles/webpack-stats.json +COPY --link --chown=app:app --from=bundles-development /openedx/staticfiles/studio/webpack-stats.json /openedx/staticfiles/studio/webpack-stats.json +COPY --link --chown=app:app --from=bundles-development /openedx/edx-platform/common/static/bundles common/static/bundles +COPY --link --chown=app:app --from=css-development /openedx/edx-platform/lms/static/css lms/static/css +COPY --link --chown=app:app --from=css-development /openedx/edx-platform/lms/static/certificates/css lms/static/certificates/css +COPY --link --chown=app:app --from=css-development /openedx/edx-platform/cms/static/css cms/static/css {{ patch("openedx-dev-dockerfile-post-python-requirements") }} @@ -272,18 +424,3 @@ RUN rm -r /openedx/staticfiles && \ ENV DJANGO_SETTINGS_MODULE lms.envs.tutor.development CMD ./manage.py $SERVICE_VARIANT runserver 0.0.0.0:8000 - -###### Final image with production cmd -FROM production as final - -# Default amount of uWSGI processes -ENV UWSGI_WORKERS=2 - -# Copy the default uWSGI configuration -COPY --chown=app:app settings/uwsgi.ini . - -# Run server -CMD uwsgi uwsgi.ini - -{{ patch("openedx-dockerfile-final") }} - diff --git a/tutor/templates/jobs/init/mounted-edx-platform.sh b/tutor/templates/jobs/init/mounted-edx-platform.sh deleted file mode 100644 index 9516654ff5..0000000000 --- a/tutor/templates/jobs/init/mounted-edx-platform.sh +++ /dev/null @@ -1,26 +0,0 @@ -# When a new local copy of edx-platform is bind-mounted, certain build -# artifacts from the openedx image's edx-platform directory are lost. -# We regenerate them here. - -if [ -f /openedx/edx-platform/bindmount-canary ] ; then - # If this file exists, then edx-platform has not been bind-mounted, - # so no build artifacts need to be regenerated. - echo "Using edx-platform from image (not bind-mount)." - echo "No extra setup is required." - exit -fi - -echo "Performing additional setup for bind-mounted edx-platform." -set -x # Echo out executed lines - -# Regenerate Open_edX.egg-info -pip install -e . - -# Regenerate node_modules -npm clean-install - -# Regenerate static assets. -openedx-assets build --env=dev - -set -x -echo "Done setting up bind-mounted edx-platform." diff --git a/tutor/templates/k8s/deployments.yml b/tutor/templates/k8s/deployments.yml index ee017246ed..ff833ad1ca 100644 --- a/tutor/templates/k8s/deployments.yml +++ b/tutor/templates/k8s/deployments.yml @@ -392,7 +392,7 @@ spec: containers: - name: mysql image: {{ DOCKER_IMAGE_MYSQL }} - args: ["mysqld", "--character-set-server=utf8", "--collation-server=utf8_general_ci"] + args: ["mysqld", "--character-set-server=utf8mb3", "--collation-server=utf8mb3_general_ci"] env: - name: MYSQL_ROOT_PASSWORD value: "{{ MYSQL_ROOT_PASSWORD }}" diff --git a/tutor/templates/local/docker-compose.jobs.yml b/tutor/templates/local/docker-compose.jobs.yml index c70fa23f59..984e085e71 100644 --- a/tutor/templates/local/docker-compose.jobs.yml +++ b/tutor/templates/local/docker-compose.jobs.yml @@ -1,8 +1,8 @@ # Tutor provides the `tutor MODE do JOB ...` CLI as a consistent way to execute jobs -# across the dev, local, and k8s modes. To support jobs in the docker-compose modes +# across the dev, local, and k8s modes. To support jobs in the docker compose modes # (dev and local), we must define a `-job` variant service in which jobs could be run. -# When `tutor local do JOB ...` is invoked, we `docker-compose run` each of JOB's +# When `tutor local do JOB ...` is invoked, we `docker compose run` each of JOB's # tasks against the appropriate `-job` services, as defined here. # When `tutor dev do JOB ...` is invoked, we do the same, but also include any # compose overrides in ../dev/docker-compose.jobs.yml. diff --git a/tutor/utils.py b/tutor/utils.py index 59adee4377..2d8d3e561e 100644 --- a/tutor/utils.py +++ b/tutor/utils.py @@ -173,25 +173,6 @@ def docker(*command: str) -> int: return execute("docker", *command) -@lru_cache(maxsize=None) -def is_buildkit_enabled() -> bool: - """ - A helper function to determine whether we can run `docker buildx` with BuildKit. - """ - # First, we respect the DOCKER_BUILDKIT environment variable - enabled_by_env = { - "1": True, - "0": False, - }.get(os.environ.get("DOCKER_BUILDKIT", "")) - if enabled_by_env is not None: - return enabled_by_env - try: - subprocess.run(["docker", "buildx", "version"], capture_output=True, check=True) - return True - except subprocess.CalledProcessError: - return False - - def docker_compose(*command: str) -> int: return execute("docker", "compose", *command)