From 758b7d0f73cc1753bf37b80d90ae0f91a13aa108 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?R=C3=A9gis=20Behmo?= Date: Mon, 28 Aug 2023 09:42:54 +0200 Subject: [PATCH 01/14] docs: backup with `sudo` See: https://discuss.overhang.io/t/copying-tutor-gives-permission-denied/3500 --- docs/install.rst | 2 +- docs/tutorials/datamigration.rst | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/install.rst b/docs/install.rst index 0c39f1e2ebe..eb821010272 100644 --- a/docs/install.rst +++ b/docs/install.rst @@ -114,7 +114,7 @@ Upgrading to a new Open edX release Major Open edX releases are published twice a year, in June and December, by the Open edX `Build/Test/Release working group `__. When a new Open edX release comes out, Tutor gets a major version bump (see :ref:`versioning`). Such an upgrade typically includes multiple breaking changes. Any upgrade is final because downgrading is not supported. Thus, when upgrading your platform from one major version to the next, it is strongly recommended to do the following: 1. Read the changes listed in the `CHANGELOG.md `__ file. Breaking changes are identified by a "💥". -2. Perform a backup. On a local installation, this is typically done with:: +2. Perform a backup (see the :ref:`backup tutorial `). On a local installation, this is typically done with:: tutor local stop sudo rsync -avr "$(tutor config printroot)"/ /tmp/tutor-backup/ diff --git a/docs/tutorials/datamigration.rst b/docs/tutorials/datamigration.rst index 95bfb860658..8434eeec79e 100644 --- a/docs/tutorials/datamigration.rst +++ b/docs/tutorials/datamigration.rst @@ -1,3 +1,5 @@ +.. _backup_tutorial: + Making backups and migrating data --------------------------------- @@ -10,7 +12,7 @@ With Tutor, all data are stored in a single folder. This means that it's extreme 3. Transfer the configuration, environment, and platform data from server 1 to server 2:: - rsync -avr "$(tutor config printroot)/" username@server2:/tmp/tutor/ + sudo -avr "$(tutor config printroot)/" username@server2:/tmp/tutor/ 4. On server 2, move the data to the right location:: From 97b999b51927f11869b686f1257a58927009c9e1 Mon Sep 17 00:00:00 2001 From: "Kyle D. McCormick" Date: Mon, 10 Jul 2023 08:58:24 -0400 Subject: [PATCH 02/14] docs: tutor uses `docker compose` now, not `docker-compose` --- docs/intro.rst | 4 ++-- docs/k8s.rst | 2 +- docs/local.rst | 2 +- tutor/commands/compose.py | 16 ++++++++-------- tutor/hooks/catalog.py | 2 +- tutor/templates/local/docker-compose.jobs.yml | 4 ++-- 6 files changed, 15 insertions(+), 15 deletions(-) diff --git a/docs/intro.rst b/docs/intro.rst index bbef116f8d8..81d59e0a17b 100644 --- a/docs/intro.rst +++ b/docs/intro.rst @@ -51,7 +51,7 @@ Tutor simplifies the deployment of Open edX by: :width: 500px :align: center -Because Docker containers are becoming an industry-wide standard, that means that with Tutor it becomes possible to run Open edX anywhere: for now, Tutor supports deploying on a local server, with `docker-compose `_, and in a large cluster, with `Kubernetes `_. But in the future, Tutor may support other deployment platforms. +Because Docker containers are becoming an industry-wide standard, that means that with Tutor it becomes possible to run Open edX anywhere: for now, Tutor supports deploying on a local server, with `docker compose `_, and in a large cluster, with `Kubernetes `_. But in the future, Tutor may support other deployment platforms. Where can I try Open edX and Tutor? ----------------------------------- @@ -101,7 +101,7 @@ You can now take advantage of the Tutor-powered CLI (item #3) to bootstrap your tutor local launch -Under the hood, Tutor simply runs ``docker-compose`` and ``docker`` commands to launch your platform. These commands are printed in the standard output, such that you are free to replicate the same behaviour by simply copying/pasting the same commands. +Under the hood, Tutor simply runs ``docker compose`` and ``docker`` commands to launch your platform. These commands are printed in the standard output, such that you are free to replicate the same behaviour by simply copying/pasting the same commands. How do I navigate Tutor's command-line interface? ------------------------------------------------- diff --git a/docs/k8s.rst b/docs/k8s.rst index a7a80ad6934..3d934d2c5f9 100644 --- a/docs/k8s.rst +++ b/docs/k8s.rst @@ -119,7 +119,7 @@ Common tasks Executing commands inside service pods ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The Tutor and plugin documentation usually often instructions to execute some ``tutor local run ...`` commands. These commands are only valid when running Tutor locally with docker-compose, and will not work on Kubernetes. Instead, you should run ``tutor k8s exec ...`` commands. Arguments and options should be identical. +The Tutor and plugin documentation usually often instructions to execute some ``tutor local run ...`` commands. These commands are only valid when running Tutor locally with docker compose, and will not work on Kubernetes. Instead, you should run ``tutor k8s exec ...`` commands. Arguments and options should be identical. For instance, to run a Python shell in the lms container, run:: diff --git a/docs/local.rst b/docs/local.rst index 62cb431d894..7f3ebcb3075 100644 --- a/docs/local.rst +++ b/docs/local.rst @@ -6,7 +6,7 @@ Local deployment This method is for deploying Open edX locally on a single server, where docker images are orchestrated with `docker-compose `_. .. note:: - Tutor is compatible with the ``docker compose`` subcommand. However, this support is still in beta and we're not sure it will behave the same as the previous ``docker-compose`` command. So ``docker-compose`` will be preferred, unless you set an environment variable ``TUTOR_USE_COMPOSE_SUBCOMMAND`` to enforce using ``docker compose``. + As of v16.0.0, Tutor now uses the ``docker compose`` subcommand instead of the separate ``docker-compose`` command. .. _tutor_root: diff --git a/tutor/commands/compose.py b/tutor/commands/compose.py index baa1ca00169..0f19e3a5e3d 100644 --- a/tutor/commands/compose.py +++ b/tutor/commands/compose.py @@ -272,7 +272,7 @@ def reboot(context: click.Context, detach: bool, services: list[str]) -> None: @click.command( short_help="Restart some components from a running platform.", help="""Specify 'openedx' to restart the lms, cms and workers, or 'all' to -restart all services. Note that this performs a 'docker-compose restart', so new images +restart all services. Note that this performs a 'docker compose restart', so new images may not be taken into account. It is useful for reloading settings, for instance. To fully stop the platform, use the 'reboot' command.""", ) @@ -302,8 +302,8 @@ def do() -> None: @click.command( short_help="Run a command in a new container", help=( - "Run a command in a new container. This is a wrapper around `docker-compose run`. Any option or argument passed" - " to this command will be forwarded to docker-compose. Thus, you may use `-v` or `-p` to mount volumes and" + "Run a command in a new container. This is a wrapper around `docker compose run`. Any option or argument passed" + " to this command will be forwarded to docker compose. Thus, you may use `-v` or `-p` to mount volumes and" " expose ports." ), context_settings={"ignore_unknown_options": True}, @@ -368,7 +368,7 @@ def copyfrom( @click.command( short_help="Run a command in a running container", help=( - "Run a command in a running container. This is a wrapper around `docker-compose exec`. Any option or argument" + "Run a command in a running container. This is a wrapper around `docker compose exec`. Any option or argument" " passed to this command will be forwarded to docker-compose. Thus, you may use `-e` to manually define" " environment variables." ), @@ -383,7 +383,7 @@ def execute(context: click.Context, args: list[str]) -> None: @click.command( short_help="View output from containers", - help="View output from containers. This is a wrapper around `docker-compose logs`.", + help="View output from containers. This is a wrapper around `docker compose logs`.", ) @click.option("-f", "--follow", is_flag=True, help="Follow log output") @click.option("--tail", type=int, help="Number of lines to show from each container") @@ -406,10 +406,10 @@ def status(context: click.Context) -> None: @click.command( - short_help="Direct interface to docker-compose.", + short_help="Direct interface to docker compose.", help=( - "Direct interface to docker-compose. This is a wrapper around `docker-compose`. Most commands, options and" - " arguments passed to this command will be forwarded as-is to docker-compose." + "Direct interface to docker compose. This is a wrapper around `docker compose`. Most commands, options and" + " arguments passed to this command will be forwarded as-is to docker compose." ), context_settings={"ignore_unknown_options": True}, name="dc", diff --git a/tutor/hooks/catalog.py b/tutor/hooks/catalog.py index e1eac6d71f7..2f628102d6b 100644 --- a/tutor/hooks/catalog.py +++ b/tutor/hooks/catalog.py @@ -50,7 +50,7 @@ def run_this_on_start(root, config, name): For more information about how actions work, check out the :py:class:`tutor.core.hooks.Action` API. """ - #: Triggered whenever a "docker-compose start", "up" or "restart" command is executed. + #: Triggered whenever a "docker compose start", "up" or "restart" command is executed. #: #: :parameter str root: project root. #: :parameter dict config: project configuration. diff --git a/tutor/templates/local/docker-compose.jobs.yml b/tutor/templates/local/docker-compose.jobs.yml index c70fa23f594..984e085e712 100644 --- a/tutor/templates/local/docker-compose.jobs.yml +++ b/tutor/templates/local/docker-compose.jobs.yml @@ -1,8 +1,8 @@ # Tutor provides the `tutor MODE do JOB ...` CLI as a consistent way to execute jobs -# across the dev, local, and k8s modes. To support jobs in the docker-compose modes +# across the dev, local, and k8s modes. To support jobs in the docker compose modes # (dev and local), we must define a `-job` variant service in which jobs could be run. -# When `tutor local do JOB ...` is invoked, we `docker-compose run` each of JOB's +# When `tutor local do JOB ...` is invoked, we `docker compose run` each of JOB's # tasks against the appropriate `-job` services, as defined here. # When `tutor dev do JOB ...` is invoked, we do the same, but also include any # compose overrides in ../dev/docker-compose.jobs.yml. From ea79dfb85dea3b5b5d029af0307e2ad0da45f9cc Mon Sep 17 00:00:00 2001 From: "Kyle D. McCormick" Date: Fri, 18 Aug 2023 11:55:53 -0400 Subject: [PATCH 03/14] docs: update README to reflect official ARM64 support --- docs/install.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/install.rst b/docs/install.rst index eb821010272..d46f6dd8516 100644 --- a/docs/install.rst +++ b/docs/install.rst @@ -9,7 +9,7 @@ Requirements ------------ * Supported OS: Tutor runs on any 64-bit, UNIX-based OS. It was also reported to work on Windows (with `WSL 2 `__). -* Architecture: support for ARM64 is a work-in-progress. See `this issue `__. +* Architecture: Both AMD64 and ARM64 are supported. * Required software: - `Docker `__: v20.10.15+ @@ -17,7 +17,6 @@ Requirements .. warning:: Do not attempt to simply run ``apt-get install docker docker-compose`` on older Ubuntu platforms, such as 16.04 (Xenial), as you will get older versions of these utilities. - * Ports 80 and 443 should be open. If other web services run on these ports, check the tutorial on :ref:`how to setup a web proxy `. * Hardware: From edd19fc0d068dbc9cab6ad9e3a33704c8aa2cf35 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?R=C3=A9gis=20Behmo?= Date: Mon, 28 Aug 2023 09:48:16 +0200 Subject: [PATCH 04/14] docs: fix build --- docs/install.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/install.rst b/docs/install.rst index d46f6dd8516..f328ac841f7 100644 --- a/docs/install.rst +++ b/docs/install.rst @@ -17,6 +17,7 @@ Requirements .. warning:: Do not attempt to simply run ``apt-get install docker docker-compose`` on older Ubuntu platforms, such as 16.04 (Xenial), as you will get older versions of these utilities. + * Ports 80 and 443 should be open. If other web services run on these ports, check the tutorial on :ref:`how to setup a web proxy `. * Hardware: From cfb786ab2d242a511eb58a10c1b6432c4947afbe Mon Sep 17 00:00:00 2001 From: Florian Haas Date: Thu, 24 Aug 2023 10:36:19 +0200 Subject: [PATCH 05/14] fix: Apply mysqld character set fix to Kubernetes deployment This is a follow-up fix to #819, where the corresponding change was added to the mysqld invocation in the "tutor local" (that is, docker-compose) deployment method, but omitted from its "tutor k8s" equivalent. --- changelog.d/20230824_104119_fghaas_utf8mb3_k8s.md | 1 + tutor/templates/k8s/deployments.yml | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) create mode 100644 changelog.d/20230824_104119_fghaas_utf8mb3_k8s.md diff --git a/changelog.d/20230824_104119_fghaas_utf8mb3_k8s.md b/changelog.d/20230824_104119_fghaas_utf8mb3_k8s.md new file mode 100644 index 00000000000..b7320653918 --- /dev/null +++ b/changelog.d/20230824_104119_fghaas_utf8mb3_k8s.md @@ -0,0 +1 @@ +- 💥[Bugfix] Apply "fix mysql crash after upgrade to Palm" from 16.1.0 to `tutor k8s` deployments, as well. Users previously running `tutor k8s` with `RUN_MYSQL: true`, with any version between 16.0.0 and 16.1.0 including, might have to fix their data manually. For users running `tutor local`, this change has no effect, as the underlying issue was already fixed in 16.1.0. For users running `tutor k8s` with `RUN_MYSQL: false`, this change is also a no-op. (by @fghaas) diff --git a/tutor/templates/k8s/deployments.yml b/tutor/templates/k8s/deployments.yml index ee017246ed6..ff833ad1ca7 100644 --- a/tutor/templates/k8s/deployments.yml +++ b/tutor/templates/k8s/deployments.yml @@ -392,7 +392,7 @@ spec: containers: - name: mysql image: {{ DOCKER_IMAGE_MYSQL }} - args: ["mysqld", "--character-set-server=utf8", "--collation-server=utf8_general_ci"] + args: ["mysqld", "--character-set-server=utf8mb3", "--collation-server=utf8mb3_general_ci"] env: - name: MYSQL_ROOT_PASSWORD value: "{{ MYSQL_ROOT_PASSWORD }}" From 748340f751b61438eade8eb8838d566d112e892f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?R=C3=A9gis=20Behmo?= Date: Tue, 29 Aug 2023 11:43:24 +0200 Subject: [PATCH 06/14] v16.1.1 --- CHANGELOG.md | 5 +++++ changelog.d/20230824_104119_fghaas_utf8mb3_k8s.md | 1 - tutor/__about__.py | 2 +- 3 files changed, 6 insertions(+), 2 deletions(-) delete mode 100644 changelog.d/20230824_104119_fghaas_utf8mb3_k8s.md diff --git a/CHANGELOG.md b/CHANGELOG.md index a6a05a9c441..08a46ac9c2f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,6 +20,11 @@ instructions, because git commits are used to generate release notes: + +## v16.1.1 (2023-08-29) + +- 💥[Bugfix] Apply "fix mysql crash after upgrade to Palm" from 16.1.0 to `tutor k8s` deployments, as well. Users previously running `tutor k8s` with `RUN_MYSQL: true`, with any version between 16.0.0 and 16.1.0 including, might have to fix their data manually. For users running `tutor local`, this change has no effect, as the underlying issue was already fixed in 16.1.0. For users running `tutor k8s` with `RUN_MYSQL: false`, this change is also a no-op. (by @fghaas) + ## v16.1.0 (2023-08-16) diff --git a/changelog.d/20230824_104119_fghaas_utf8mb3_k8s.md b/changelog.d/20230824_104119_fghaas_utf8mb3_k8s.md deleted file mode 100644 index b7320653918..00000000000 --- a/changelog.d/20230824_104119_fghaas_utf8mb3_k8s.md +++ /dev/null @@ -1 +0,0 @@ -- 💥[Bugfix] Apply "fix mysql crash after upgrade to Palm" from 16.1.0 to `tutor k8s` deployments, as well. Users previously running `tutor k8s` with `RUN_MYSQL: true`, with any version between 16.0.0 and 16.1.0 including, might have to fix their data manually. For users running `tutor local`, this change has no effect, as the underlying issue was already fixed in 16.1.0. For users running `tutor k8s` with `RUN_MYSQL: false`, this change is also a no-op. (by @fghaas) diff --git a/tutor/__about__.py b/tutor/__about__.py index 45e43fc494e..0066407232d 100644 --- a/tutor/__about__.py +++ b/tutor/__about__.py @@ -2,7 +2,7 @@ # Increment this version number to trigger a new release. See # docs/tutor.html#versioning for information on the versioning scheme. -__version__ = "16.1.0" +__version__ = "16.1.1" # The version suffix will be appended to the actual version, separated by a # dash. Use this suffix to differentiate between the actual released version and From 7e443ac40d2129cca38facaafa8bfd59d577e965 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?R=C3=A9gis=20Behmo?= Date: Tue, 29 Aug 2023 15:53:27 +0200 Subject: [PATCH 07/14] chore: upgrade reqs Now that sphinx_rtd support docutils>=0.19 we can drop that max version requirement. But we need to limit sphinx max version because they removed python 3.8 support before EOL. --- requirements/base.txt | 25 +++++++------ requirements/dev.in | 4 --- requirements/dev.txt | 83 +++++++++++++++++++++++-------------------- requirements/docs.in | 4 ++- requirements/docs.txt | 36 ++++++++++--------- 5 files changed, 80 insertions(+), 72 deletions(-) diff --git a/requirements/base.txt b/requirements/base.txt index 228782bb3ce..dff55c9f19e 100644 --- a/requirements/base.txt +++ b/requirements/base.txt @@ -1,5 +1,5 @@ # -# This file is autogenerated by pip-compile with Python 3.10 +# This file is autogenerated by pip-compile with Python 3.8 # by the following command: # # pip-compile requirements/base.in @@ -12,26 +12,28 @@ certifi==2023.7.22 # via # kubernetes # requests -charset-normalizer==3.1.0 +charset-normalizer==3.2.0 # via requests -click==8.1.3 +click==8.1.7 # via -r requirements/base.in -google-auth==2.19.1 +google-auth==2.22.0 # via kubernetes idna==3.4 # via requests jinja2==3.1.2 # via -r requirements/base.in -kubernetes==26.1.0 +kubernetes==27.2.0 # via -r requirements/base.in markupsafe==2.1.3 # via jinja2 -mypy==1.3.0 +mypy==1.5.1 # via -r requirements/base.in mypy-extensions==1.0.0 # via mypy oauthlib==3.2.2 - # via requests-oauthlib + # via + # kubernetes + # requests-oauthlib pyasn1==0.5.0 # via # pyasn1-modules @@ -42,7 +44,7 @@ pycryptodome==3.18.0 # via -r requirements/base.in python-dateutil==2.8.2 # via kubernetes -pyyaml==6.0 +pyyaml==6.0.1 # via # -r requirements/base.in # kubernetes @@ -61,7 +63,7 @@ six==1.16.0 # python-dateutil tomli==2.0.1 # via mypy -typing-extensions==4.6.3 +typing-extensions==4.7.1 # via # -r requirements/base.in # mypy @@ -70,8 +72,5 @@ urllib3==1.26.16 # google-auth # kubernetes # requests -websocket-client==1.5.2 +websocket-client==1.6.2 # via kubernetes - -# The following packages are considered to be unsafe in a requirements file: -# setuptools diff --git a/requirements/dev.in b/requirements/dev.in index 8dc0c3d1f82..c53771c4193 100644 --- a/requirements/dev.in +++ b/requirements/dev.in @@ -7,10 +7,6 @@ pyinstaller scriv twine -# doc requirement is lagging behind -# https://github.com/readthedocs/sphinx_rtd_theme/issues/1323 -docutils<0.19 - # Types packages types-docutils types-PyYAML diff --git a/requirements/dev.txt b/requirements/dev.txt index c2400af9fac..abb13c7448c 100644 --- a/requirements/dev.txt +++ b/requirements/dev.txt @@ -1,5 +1,5 @@ # -# This file is autogenerated by pip-compile with Python 3.10 +# This file is autogenerated by pip-compile with Python 3.8 # by the following command: # # pip-compile requirements/dev.in @@ -8,11 +8,11 @@ altgraph==0.17.3 # via pyinstaller appdirs==1.4.4 # via -r requirements/base.txt -astroid==2.15.5 +astroid==2.15.6 # via pylint attrs==23.1.0 # via scriv -black==23.3.0 +black==23.7.0 # via -r requirements/dev.in bleach==6.0.0 # via readme-renderer @@ -29,11 +29,11 @@ certifi==2023.7.22 # requests cffi==1.15.1 # via cryptography -charset-normalizer==3.1.0 +charset-normalizer==3.2.0 # via # -r requirements/base.txt # requests -click==8.1.3 +click==8.1.7 # via # -r requirements/base.txt # black @@ -42,17 +42,15 @@ click==8.1.3 # scriv click-log==0.4.0 # via scriv -coverage==7.2.7 +coverage==7.3.0 # via -r requirements/dev.in cryptography==41.0.3 # via secretstorage -dill==0.3.6 +dill==0.3.7 # via pylint -docutils==0.18.1 - # via - # -r requirements/dev.in - # readme-renderer -google-auth==2.19.1 +docutils==0.20.1 + # via readme-renderer +google-auth==2.22.0 # via # -r requirements/base.txt # kubernetes @@ -60,13 +58,15 @@ idna==3.4 # via # -r requirements/base.txt # requests -importlib-metadata==6.6.0 +importlib-metadata==6.8.0 # via # keyring # twine +importlib-resources==6.0.1 + # via keyring isort==5.12.0 # via pylint -jaraco-classes==3.2.3 +jaraco-classes==3.3.0 # via keyring jeepney==0.8.0 # via @@ -76,13 +76,13 @@ jinja2==3.1.2 # via # -r requirements/base.txt # scriv -keyring==23.13.1 +keyring==24.2.0 # via twine -kubernetes==26.1.0 +kubernetes==27.2.0 # via -r requirements/base.txt lazy-object-proxy==1.9.0 # via astroid -markdown-it-py==2.2.0 +markdown-it-py==3.0.0 # via rich markupsafe==2.1.3 # via @@ -92,9 +92,9 @@ mccabe==0.7.0 # via pylint mdurl==0.1.2 # via markdown-it-py -more-itertools==9.1.0 +more-itertools==10.1.0 # via jaraco-classes -mypy==1.3.0 +mypy==1.5.1 # via -r requirements/base.txt mypy-extensions==1.0.0 # via @@ -104,18 +104,19 @@ mypy-extensions==1.0.0 oauthlib==3.2.2 # via # -r requirements/base.txt + # kubernetes # requests-oauthlib packaging==23.1 # via # black # build -pathspec==0.11.1 +pathspec==0.11.2 # via black -pip-tools==6.13.0 +pip-tools==7.3.0 # via -r requirements/dev.in pkginfo==1.9.6 # via twine -platformdirs==3.5.1 +platformdirs==3.10.0 # via # black # pylint @@ -132,15 +133,15 @@ pycparser==2.21 # via cffi pycryptodome==3.18.0 # via -r requirements/base.txt -pygments==2.15.1 +pygments==2.16.1 # via # readme-renderer # rich -pyinstaller==5.11.0 +pyinstaller==5.13.1 # via -r requirements/dev.in -pyinstaller-hooks-contrib==2023.3 +pyinstaller-hooks-contrib==2023.7 # via pyinstaller -pylint==2.17.4 +pylint==2.17.5 # via -r requirements/dev.in pyproject-hooks==1.0.0 # via build @@ -148,11 +149,11 @@ python-dateutil==2.8.2 # via # -r requirements/base.txt # kubernetes -pyyaml==6.0 +pyyaml==6.0.1 # via # -r requirements/base.txt # kubernetes -readme-renderer==37.3 +readme-renderer==41.0 # via twine requests==2.31.0 # via @@ -170,7 +171,7 @@ requests-toolbelt==1.0.0 # via twine rfc3986==2.0.0 # via twine -rich==13.4.1 +rich==13.5.2 # via twine rsa==4.9 # via @@ -193,23 +194,27 @@ tomli==2.0.1 # black # build # mypy + # pip-tools # pylint # pyproject-hooks -tomlkit==0.11.8 +tomlkit==0.12.1 # via pylint twine==4.0.2 # via -r requirements/dev.in -types-docutils==0.20.0.1 +types-docutils==0.20.0.3 # via -r requirements/dev.in -types-pyyaml==6.0.12.10 +types-pyyaml==6.0.12.11 # via -r requirements/dev.in -types-setuptools==67.8.0.0 +types-setuptools==68.1.0.0 # via -r requirements/dev.in -typing-extensions==4.6.3 +typing-extensions==4.7.1 # via # -r requirements/base.txt # astroid + # black # mypy + # pylint + # rich urllib3==1.26.16 # via # -r requirements/base.txt @@ -219,16 +224,18 @@ urllib3==1.26.16 # twine webencodings==0.5.1 # via bleach -websocket-client==1.5.2 +websocket-client==1.6.2 # via # -r requirements/base.txt # kubernetes -wheel==0.40.0 +wheel==0.41.2 # via pip-tools wrapt==1.15.0 # via astroid -zipp==3.15.0 - # via importlib-metadata +zipp==3.16.2 + # via + # importlib-metadata + # importlib-resources # The following packages are considered to be unsafe in a requirements file: # pip diff --git a/requirements/docs.in b/requirements/docs.in index 79c02e20e35..1732d9b76f1 100644 --- a/requirements/docs.in +++ b/requirements/docs.in @@ -1,4 +1,6 @@ -r base.txt -sphinx +# Python 3.8 support was dropped in 7.2.0 +# https://github.com/sphinx-doc/sphinx/pull/11511 +sphinx<7.2.0 sphinx-rtd-theme sphinx-click diff --git a/requirements/docs.txt b/requirements/docs.txt index cfa5d7195b5..78d4debb081 100644 --- a/requirements/docs.txt +++ b/requirements/docs.txt @@ -1,5 +1,5 @@ # -# This file is autogenerated by pip-compile with Python 3.10 +# This file is autogenerated by pip-compile with Python 3.8 # by the following command: # # pip-compile requirements/docs.in @@ -19,11 +19,11 @@ certifi==2023.7.22 # -r requirements/base.txt # kubernetes # requests -charset-normalizer==3.1.0 +charset-normalizer==3.2.0 # via # -r requirements/base.txt # requests -click==8.1.3 +click==8.1.7 # via # -r requirements/base.txt # sphinx-click @@ -32,7 +32,7 @@ docutils==0.18.1 # sphinx # sphinx-click # sphinx-rtd-theme -google-auth==2.19.1 +google-auth==2.22.0 # via # -r requirements/base.txt # kubernetes @@ -42,17 +42,19 @@ idna==3.4 # requests imagesize==1.4.1 # via sphinx +importlib-metadata==6.8.0 + # via sphinx jinja2==3.1.2 # via # -r requirements/base.txt # sphinx -kubernetes==26.1.0 +kubernetes==27.2.0 # via -r requirements/base.txt markupsafe==2.1.3 # via # -r requirements/base.txt # jinja2 -mypy==1.3.0 +mypy==1.5.1 # via -r requirements/base.txt mypy-extensions==1.0.0 # via @@ -61,6 +63,7 @@ mypy-extensions==1.0.0 oauthlib==3.2.2 # via # -r requirements/base.txt + # kubernetes # requests-oauthlib packaging==23.1 # via sphinx @@ -75,13 +78,15 @@ pyasn1-modules==0.3.0 # google-auth pycryptodome==3.18.0 # via -r requirements/base.txt -pygments==2.15.1 +pygments==2.16.1 # via sphinx python-dateutil==2.8.2 # via # -r requirements/base.txt # kubernetes -pyyaml==6.0 +pytz==2023.3 + # via babel +pyyaml==6.0.1 # via # -r requirements/base.txt # kubernetes @@ -107,15 +112,15 @@ six==1.16.0 # python-dateutil snowballstemmer==2.2.0 # via sphinx -sphinx==6.2.1 +sphinx==7.1.2 # via # -r requirements/docs.in # sphinx-click # sphinx-rtd-theme # sphinxcontrib-jquery -sphinx-click==4.4.0 +sphinx-click==5.0.1 # via -r requirements/docs.in -sphinx-rtd-theme==1.2.1 +sphinx-rtd-theme==1.3.0 # via -r requirements/docs.in sphinxcontrib-applehelp==1.0.4 # via sphinx @@ -135,7 +140,7 @@ tomli==2.0.1 # via # -r requirements/base.txt # mypy -typing-extensions==4.6.3 +typing-extensions==4.7.1 # via # -r requirements/base.txt # mypy @@ -145,10 +150,9 @@ urllib3==1.26.16 # google-auth # kubernetes # requests -websocket-client==1.5.2 +websocket-client==1.6.2 # via # -r requirements/base.txt # kubernetes - -# The following packages are considered to be unsafe in a requirements file: -# setuptools +zipp==3.16.2 + # via importlib-metadata From 39d3eaf8019f32b00f19d58de432864aad206a60 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?R=C3=A9gis=20Behmo?= Date: Tue, 29 Aug 2023 16:21:43 +0200 Subject: [PATCH 08/14] fix: type tests Type tests were broken following the upgrade of click. We take the opportunity to simplify the TutorCli implementation. --- tutor/commands/cli.py | 50 +++++++++++++++++-------------------------- 1 file changed, 20 insertions(+), 30 deletions(-) diff --git a/tutor/commands/cli.py b/tutor/commands/cli.py index c8e552a2c9c..e434a05e7aa 100755 --- a/tutor/commands/cli.py +++ b/tutor/commands/cli.py @@ -32,7 +32,7 @@ def main() -> None: sys.exit(1) -class TutorCli(click.MultiCommand): +class TutorCli(click.Group): """ Dynamically load subcommands at runtime. @@ -43,26 +43,14 @@ class TutorCli(click.MultiCommand): IS_ROOT_READY = False - @classmethod - def iter_commands(cls, ctx: click.Context) -> t.Iterator[click.Command]: - """ - Return the list of subcommands (click.Command). - """ - cls.ensure_plugins_enabled(ctx) - yield from hooks.Filters.CLI_COMMANDS.iterate() - - @classmethod - def ensure_plugins_enabled(cls, ctx: click.Context) -> None: + def get_command( + self, ctx: click.Context, cmd_name: str + ) -> t.Optional[click.Command]: """ - We enable plugins as soon as possible to have access to commands. + This is run when passing a command from the CLI. E.g: tutor config ... """ - if not "root" in ctx.params: - # When generating docs, this function is called with empty args. - # That's ok, we just ignore it. - return - if not cls.IS_ROOT_READY: - hooks.Actions.PROJECT_ROOT_READY.do(ctx.params["root"]) - cls.IS_ROOT_READY = True + self.ensure_plugins_enabled(ctx) + return super().get_command(ctx, cmd_name=cmd_name) def list_commands(self, ctx: click.Context) -> list[str]: """ @@ -70,20 +58,22 @@ def list_commands(self, ctx: click.Context) -> list[str]: - shell autocompletion: tutor - print help: tutor, tutor -h """ - return sorted( - [command.name or "" for command in self.iter_commands(ctx)] - ) + self.ensure_plugins_enabled(ctx) + return super().list_commands(ctx) - def get_command( - self, ctx: click.Context, cmd_name: str - ) -> t.Optional[click.Command]: + def ensure_plugins_enabled(self, ctx: click.Context) -> None: """ - This is run when passing a command from the CLI. E.g: tutor config ... + We enable plugins as soon as possible to have access to commands. """ - for command in self.iter_commands(ctx): - if cmd_name == command.name: - return command - return None + if not "root" in ctx.params: + # When generating docs, this function is called with empty args. + # That's ok, we just ignore it. + return + if not self.IS_ROOT_READY: + hooks.Actions.PROJECT_ROOT_READY.do(ctx.params["root"]) + self.IS_ROOT_READY = True + for cmd in hooks.Filters.CLI_COMMANDS.iterate(): + self.add_command(cmd) @click.group( From 56c4ac895cdb9c35bbbb2d2054dc4492831a8227 Mon Sep 17 00:00:00 2001 From: "Kyle D. McCormick" Date: Fri, 18 Aug 2023 10:36:13 -0400 Subject: [PATCH 09/14] feat: `tutor mounts populate` TODOs: * describe commit * address TODOs in code * should it be `tutor dev populate-mounts` instead? * add to `tutor dev launch`? * circulate a TEP? Part of: TODO link ticket --- tutor/commands/jobs.py | 9 -- tutor/commands/mounts.py | 102 +++++++++++++++++- tutor/hooks/catalog.py | 11 ++ .../jobs/init/mounted-edx-platform.sh | 26 ----- 4 files changed, 112 insertions(+), 36 deletions(-) delete mode 100644 tutor/templates/jobs/init/mounted-edx-platform.sh diff --git a/tutor/commands/jobs.py b/tutor/commands/jobs.py index 562e78d4549..f31cef08a44 100644 --- a/tutor/commands/jobs.py +++ b/tutor/commands/jobs.py @@ -42,15 +42,6 @@ def _add_core_init_tasks() -> None: ("mysql", env.read_core_template_file("jobs", "init", "mysql.sh")) ) with hooks.Contexts.app("lms").enter(): - hooks.Filters.CLI_DO_INIT_TASKS.add_item( - ( - "lms", - env.read_core_template_file("jobs", "init", "mounted-edx-platform.sh"), - ), - # If edx-platform is mounted, then we may need to perform some setup - # before other initialization scripts can be run. - priority=priorities.HIGH, - ) hooks.Filters.CLI_DO_INIT_TASKS.add_item( ("lms", env.read_core_template_file("jobs", "init", "lms.sh")) ) diff --git a/tutor/commands/mounts.py b/tutor/commands/mounts.py index c97075521b4..0dcbf73cddb 100644 --- a/tutor/commands/mounts.py +++ b/tutor/commands/mounts.py @@ -1,6 +1,7 @@ from __future__ import annotations import os +from collections import defaultdict import click import yaml @@ -10,7 +11,13 @@ from tutor import exceptions, fmt, hooks from tutor.commands.config import save as config_save from tutor.commands.context import Context +from tutor.commands.images import ( + find_images_to_build, + find_remote_image_tags, + ImageNotFoundError, +) from tutor.commands.params import ConfigLoaderParam +from tutor.utils import execute as execute_shell class MountParamType(ConfigLoaderParam): @@ -73,8 +80,9 @@ def mounts_list(context: Context) -> None: @click.command(name="add") @click.argument("mounts", metavar="mount", type=click.Path(), nargs=-1) +@click.option("-p", "--populate", is_flag=True, help="Populate mount after adding it") @click.pass_context -def mounts_add(context: click.Context, mounts: list[str]) -> None: +def mounts_add(context: click.Context, mounts: list[str], populate: bool) -> None: """ Add a bind-mounted folder @@ -98,6 +106,8 @@ def mounts_add(context: click.Context, mounts: list[str]) -> None: explicit form. """ new_mounts = [] + implicit_mounts = [] + for mount in mounts: if not bindmount.parse_explicit_mount(mount): # Path is implicit: check that this path is valid @@ -105,11 +115,15 @@ def mounts_add(context: click.Context, mounts: list[str]) -> None: mount = os.path.abspath(os.path.expanduser(mount)) if not os.path.exists(mount): raise exceptions.TutorError(f"Path {mount} does not exist on the host") + implicit_mounts.append(mount) new_mounts.append(mount) fmt.echo_info(f"Adding bind-mount: {mount}") context.invoke(config_save, append_vars=[("MOUNTS", mount) for mount in new_mounts]) + if populate: + context.invoke(mounts_populate, mounts=implicit_mounts) + @click.command(name="remove") @click.argument("mounts", metavar="mount", type=MountParamType(), nargs=-1) @@ -133,6 +147,92 @@ def mounts_remove(context: click.Context, mounts: list[str]) -> None: ) +@click.command(name="populate", help="TODO document command") +@click.argument("mounts", metavar="mount", type=str, nargs=-1) +@click.pass_obj +def mounts_populate(context, mounts: str) -> None: + """ + TODO document command + """ + container_name = "tutor_mounts_populate_temp" # TODO: improve name? + config = tutor_config.load(context.root) + paths_to_copy_by_image: dict[str, tuple[str, str]] = defaultdict(list) + + if not mounts: + mounts = bindmount.get_mounts(config) + + for mount in mounts: + mount_items: list[tuple[str, str, str]] = bindmount.parse_mount(mount) + if not mount_items: + raise exceptions.TutorError(f"No mount for {mount}") + _service, mount_host_path, _container_path = mount_items[ + 0 + ] # [0] is arbitrary, as all host_paths should be equal + mount_expanded = os.path.abspath(os.path.expanduser(mount)) + mount_name = os.path.basename(mount_expanded) + for ( + image, + path_on_image, + path_in_host_mount, + ) in hooks.Filters.COMPOSE_MOUNT_POPULATORS.iterate(mount_name): + paths_to_copy_by_image[image].append( + (path_on_image, f"{mount_expanded}/{path_in_host_mount}") + ) + for image_name, paths_to_copy in paths_to_copy_by_image.items(): + image_tag = _get_image_tag(config, image_name) + execute_shell("docker", "rm", "-f", container_name) + execute_shell("docker", "create", "--name", container_name, image_tag) + for path_on_image, path_on_host in paths_to_copy: + fmt.echo_info(f"Populating {path_on_host} from {image_name}") + execute_shell("sh", "-c", f'mkdir -p "$(dirname "{path_on_host}")"') + execute_shell("rm", "-rf", path_on_host) + execute_shell( + "docker", "cp", f"{container_name}:{path_on_image}", path_on_host + ) + execute_shell("docker", "rm", "-f", container_name) + + +def _get_image_tag(config: Config, image_name: str) -> str: + """ + Translate from a Tutor/plugin-defined image name to a specific Docker image tag. + + Searches for image_name in IMAGES_PULL then IMAGES_BUILD. + Raises ImageNotFoundError if no match. + """ + try: + return next( + find_remote_image_tags(config, hooks.Filters.IMAGES_PULL, image_name) + ) + except ImageNotFoundError: + _name, _path, tag, _args = next(find_images_to_build(config, image_name)) + return tag + + +@hooks.Filters.COMPOSE_MOUNT_POPULATORS.add() +def _populate_edx_platform_generated_dirs( + populators: list[tuple[str, str, str]], mount_name: str +) -> list[str]: + """ + TODO write docstring + """ + if mount_name == "edx-platform": + populators += [ + ("openedx-dev", f"/openedx/edx-platform/{generated_dir}", generated_dir) + for generated_dir in [ + "Open_edX.egg-info", + "node_modules", + "lms/static/css", + "lms/static/certificates/css", + "cms/static/css", + "common/static/bundles", + "common/static/common/js/vendor", + "common/static/common/css/vendor", + ] + ] + return populators + + mounts_command.add_command(mounts_list) mounts_command.add_command(mounts_add) mounts_command.add_command(mounts_remove) +mounts_command.add_command(mounts_populate) diff --git a/tutor/hooks/catalog.py b/tutor/hooks/catalog.py index 2f628102d6b..6d6601ffa70 100644 --- a/tutor/hooks/catalog.py +++ b/tutor/hooks/catalog.py @@ -222,6 +222,17 @@ def your_filter_callback(some_data): #: conditionally add mounts. COMPOSE_MOUNTS: Filter[list[tuple[str, str]], [str]] = Filter() + #: TODO describe + #: + #: TODO show example + #: + #: :parameter list[tuple[str, str, str]] populators: each item is a + #: ``(image_name, path_on_image, path_in_host_mount)`` tuple. TODO finish describing. + #: :parameter str name: basename of the host-mounted folder. In the example above, + #: this is "edx-platform". When implementing this filter you should check this name to + #: conditionally add populators for this folder. + COMPOSE_MOUNT_POPULATORS: Filter[list[tuple[str, str, str], [str]]] = Filter() + #: Declare new default configuration settings that don't necessarily have to be saved in the user #: ``config.yml`` file. Default settings may be overridden with ``tutor config save --set=...``, in which #: case they will automatically be added to ``config.yml``. diff --git a/tutor/templates/jobs/init/mounted-edx-platform.sh b/tutor/templates/jobs/init/mounted-edx-platform.sh deleted file mode 100644 index 9516654ff58..00000000000 --- a/tutor/templates/jobs/init/mounted-edx-platform.sh +++ /dev/null @@ -1,26 +0,0 @@ -# When a new local copy of edx-platform is bind-mounted, certain build -# artifacts from the openedx image's edx-platform directory are lost. -# We regenerate them here. - -if [ -f /openedx/edx-platform/bindmount-canary ] ; then - # If this file exists, then edx-platform has not been bind-mounted, - # so no build artifacts need to be regenerated. - echo "Using edx-platform from image (not bind-mount)." - echo "No extra setup is required." - exit -fi - -echo "Performing additional setup for bind-mounted edx-platform." -set -x # Echo out executed lines - -# Regenerate Open_edX.egg-info -pip install -e . - -# Regenerate node_modules -npm clean-install - -# Regenerate static assets. -openedx-assets build --env=dev - -set -x -echo "Done setting up bind-mounted edx-platform." From 8a7f0b5850493e29e1124c64b11dabfc3d213269 Mon Sep 17 00:00:00 2001 From: "Kyle D. McCormick" Date: Thu, 31 Aug 2023 15:31:57 -0400 Subject: [PATCH 10/14] revert: feat: `tutor mounts populate` This reverts commit 56c4ac895cdb9c35bbbb2d2054dc4492831a8227. --- tutor/commands/jobs.py | 9 ++ tutor/commands/mounts.py | 102 +----------------- tutor/hooks/catalog.py | 11 -- .../jobs/init/mounted-edx-platform.sh | 26 +++++ 4 files changed, 36 insertions(+), 112 deletions(-) create mode 100644 tutor/templates/jobs/init/mounted-edx-platform.sh diff --git a/tutor/commands/jobs.py b/tutor/commands/jobs.py index f31cef08a44..562e78d4549 100644 --- a/tutor/commands/jobs.py +++ b/tutor/commands/jobs.py @@ -42,6 +42,15 @@ def _add_core_init_tasks() -> None: ("mysql", env.read_core_template_file("jobs", "init", "mysql.sh")) ) with hooks.Contexts.app("lms").enter(): + hooks.Filters.CLI_DO_INIT_TASKS.add_item( + ( + "lms", + env.read_core_template_file("jobs", "init", "mounted-edx-platform.sh"), + ), + # If edx-platform is mounted, then we may need to perform some setup + # before other initialization scripts can be run. + priority=priorities.HIGH, + ) hooks.Filters.CLI_DO_INIT_TASKS.add_item( ("lms", env.read_core_template_file("jobs", "init", "lms.sh")) ) diff --git a/tutor/commands/mounts.py b/tutor/commands/mounts.py index 0dcbf73cddb..c97075521b4 100644 --- a/tutor/commands/mounts.py +++ b/tutor/commands/mounts.py @@ -1,7 +1,6 @@ from __future__ import annotations import os -from collections import defaultdict import click import yaml @@ -11,13 +10,7 @@ from tutor import exceptions, fmt, hooks from tutor.commands.config import save as config_save from tutor.commands.context import Context -from tutor.commands.images import ( - find_images_to_build, - find_remote_image_tags, - ImageNotFoundError, -) from tutor.commands.params import ConfigLoaderParam -from tutor.utils import execute as execute_shell class MountParamType(ConfigLoaderParam): @@ -80,9 +73,8 @@ def mounts_list(context: Context) -> None: @click.command(name="add") @click.argument("mounts", metavar="mount", type=click.Path(), nargs=-1) -@click.option("-p", "--populate", is_flag=True, help="Populate mount after adding it") @click.pass_context -def mounts_add(context: click.Context, mounts: list[str], populate: bool) -> None: +def mounts_add(context: click.Context, mounts: list[str]) -> None: """ Add a bind-mounted folder @@ -106,8 +98,6 @@ def mounts_add(context: click.Context, mounts: list[str], populate: bool) -> Non explicit form. """ new_mounts = [] - implicit_mounts = [] - for mount in mounts: if not bindmount.parse_explicit_mount(mount): # Path is implicit: check that this path is valid @@ -115,15 +105,11 @@ def mounts_add(context: click.Context, mounts: list[str], populate: bool) -> Non mount = os.path.abspath(os.path.expanduser(mount)) if not os.path.exists(mount): raise exceptions.TutorError(f"Path {mount} does not exist on the host") - implicit_mounts.append(mount) new_mounts.append(mount) fmt.echo_info(f"Adding bind-mount: {mount}") context.invoke(config_save, append_vars=[("MOUNTS", mount) for mount in new_mounts]) - if populate: - context.invoke(mounts_populate, mounts=implicit_mounts) - @click.command(name="remove") @click.argument("mounts", metavar="mount", type=MountParamType(), nargs=-1) @@ -147,92 +133,6 @@ def mounts_remove(context: click.Context, mounts: list[str]) -> None: ) -@click.command(name="populate", help="TODO document command") -@click.argument("mounts", metavar="mount", type=str, nargs=-1) -@click.pass_obj -def mounts_populate(context, mounts: str) -> None: - """ - TODO document command - """ - container_name = "tutor_mounts_populate_temp" # TODO: improve name? - config = tutor_config.load(context.root) - paths_to_copy_by_image: dict[str, tuple[str, str]] = defaultdict(list) - - if not mounts: - mounts = bindmount.get_mounts(config) - - for mount in mounts: - mount_items: list[tuple[str, str, str]] = bindmount.parse_mount(mount) - if not mount_items: - raise exceptions.TutorError(f"No mount for {mount}") - _service, mount_host_path, _container_path = mount_items[ - 0 - ] # [0] is arbitrary, as all host_paths should be equal - mount_expanded = os.path.abspath(os.path.expanduser(mount)) - mount_name = os.path.basename(mount_expanded) - for ( - image, - path_on_image, - path_in_host_mount, - ) in hooks.Filters.COMPOSE_MOUNT_POPULATORS.iterate(mount_name): - paths_to_copy_by_image[image].append( - (path_on_image, f"{mount_expanded}/{path_in_host_mount}") - ) - for image_name, paths_to_copy in paths_to_copy_by_image.items(): - image_tag = _get_image_tag(config, image_name) - execute_shell("docker", "rm", "-f", container_name) - execute_shell("docker", "create", "--name", container_name, image_tag) - for path_on_image, path_on_host in paths_to_copy: - fmt.echo_info(f"Populating {path_on_host} from {image_name}") - execute_shell("sh", "-c", f'mkdir -p "$(dirname "{path_on_host}")"') - execute_shell("rm", "-rf", path_on_host) - execute_shell( - "docker", "cp", f"{container_name}:{path_on_image}", path_on_host - ) - execute_shell("docker", "rm", "-f", container_name) - - -def _get_image_tag(config: Config, image_name: str) -> str: - """ - Translate from a Tutor/plugin-defined image name to a specific Docker image tag. - - Searches for image_name in IMAGES_PULL then IMAGES_BUILD. - Raises ImageNotFoundError if no match. - """ - try: - return next( - find_remote_image_tags(config, hooks.Filters.IMAGES_PULL, image_name) - ) - except ImageNotFoundError: - _name, _path, tag, _args = next(find_images_to_build(config, image_name)) - return tag - - -@hooks.Filters.COMPOSE_MOUNT_POPULATORS.add() -def _populate_edx_platform_generated_dirs( - populators: list[tuple[str, str, str]], mount_name: str -) -> list[str]: - """ - TODO write docstring - """ - if mount_name == "edx-platform": - populators += [ - ("openedx-dev", f"/openedx/edx-platform/{generated_dir}", generated_dir) - for generated_dir in [ - "Open_edX.egg-info", - "node_modules", - "lms/static/css", - "lms/static/certificates/css", - "cms/static/css", - "common/static/bundles", - "common/static/common/js/vendor", - "common/static/common/css/vendor", - ] - ] - return populators - - mounts_command.add_command(mounts_list) mounts_command.add_command(mounts_add) mounts_command.add_command(mounts_remove) -mounts_command.add_command(mounts_populate) diff --git a/tutor/hooks/catalog.py b/tutor/hooks/catalog.py index 6d6601ffa70..2f628102d6b 100644 --- a/tutor/hooks/catalog.py +++ b/tutor/hooks/catalog.py @@ -222,17 +222,6 @@ def your_filter_callback(some_data): #: conditionally add mounts. COMPOSE_MOUNTS: Filter[list[tuple[str, str]], [str]] = Filter() - #: TODO describe - #: - #: TODO show example - #: - #: :parameter list[tuple[str, str, str]] populators: each item is a - #: ``(image_name, path_on_image, path_in_host_mount)`` tuple. TODO finish describing. - #: :parameter str name: basename of the host-mounted folder. In the example above, - #: this is "edx-platform". When implementing this filter you should check this name to - #: conditionally add populators for this folder. - COMPOSE_MOUNT_POPULATORS: Filter[list[tuple[str, str, str], [str]]] = Filter() - #: Declare new default configuration settings that don't necessarily have to be saved in the user #: ``config.yml`` file. Default settings may be overridden with ``tutor config save --set=...``, in which #: case they will automatically be added to ``config.yml``. diff --git a/tutor/templates/jobs/init/mounted-edx-platform.sh b/tutor/templates/jobs/init/mounted-edx-platform.sh new file mode 100644 index 00000000000..9516654ff58 --- /dev/null +++ b/tutor/templates/jobs/init/mounted-edx-platform.sh @@ -0,0 +1,26 @@ +# When a new local copy of edx-platform is bind-mounted, certain build +# artifacts from the openedx image's edx-platform directory are lost. +# We regenerate them here. + +if [ -f /openedx/edx-platform/bindmount-canary ] ; then + # If this file exists, then edx-platform has not been bind-mounted, + # so no build artifacts need to be regenerated. + echo "Using edx-platform from image (not bind-mount)." + echo "No extra setup is required." + exit +fi + +echo "Performing additional setup for bind-mounted edx-platform." +set -x # Echo out executed lines + +# Regenerate Open_edX.egg-info +pip install -e . + +# Regenerate node_modules +npm clean-install + +# Regenerate static assets. +openedx-assets build --env=dev + +set -x +echo "Done setting up bind-mounted edx-platform." From ace9d949490d09106444aa0ac0ec7043f6a8402d Mon Sep 17 00:00:00 2001 From: "Kyle D. McCormick" Date: Fri, 18 Aug 2023 11:33:41 -0400 Subject: [PATCH 11/14] feat!: assume BuildKit is available BuildKit replaces and improves the legacy Docker builder, which was deprecated back in Feb 2023. Assuming BuildKit allows us to simplify the Dockerfile and makes future build performance improvements easier. The Docker versions which Tutor recommends (v20+) all come with BuildKit, so As follow-up work, we will need to remove `is_buildkit_enabled` from the official plugins templates. Relevant discussion: https://github.com/overhangio/tutor/pull/868#issuecomment-1657961059 --- changelog.d/20230818_112124_kyle_buildkit.md | 2 + tests/commands/test_images.py | 6 +- tutor/commands/images.py | 34 +++++------ tutor/env.py | 2 +- tutor/hooks/catalog.py | 5 +- tutor/images.py | 3 +- tutor/templates/build/openedx/Dockerfile | 64 +++++++++++--------- tutor/utils.py | 19 ------ 8 files changed, 57 insertions(+), 78 deletions(-) create mode 100644 changelog.d/20230818_112124_kyle_buildkit.md diff --git a/changelog.d/20230818_112124_kyle_buildkit.md b/changelog.d/20230818_112124_kyle_buildkit.md new file mode 100644 index 00000000000..7084b290415 --- /dev/null +++ b/changelog.d/20230818_112124_kyle_buildkit.md @@ -0,0 +1,2 @@ +- [Deprecation] The template variable ``is_buildkit_enabled``, which now always returns True, is deprecated. Plugin authors should assume BuildKit is enabled and remove the variable from their templates (by @kdmccormick). +- 💥[Deprecation] Tutor no longer supports the legacy Docker builder, which was previously available by setting ``DOCKER_BUILDKIT=0`` in the host environment. Going forward, Tutor will always use BuildKit (a.k.a. ``docker buildx`` in Docker v19-v22, or just ``docker build`` in Docker v23). This transition will improve build performance and should be seamless for Tutor users who are running a supported Docker version (by @kdmccormick). diff --git a/tests/commands/test_images.py b/tests/commands/test_images.py index 7b0957790d9..61c4ccaeb28 100644 --- a/tests/commands/test_images.py +++ b/tests/commands/test_images.py @@ -128,10 +128,8 @@ def test_images_build_plugin_with_args(self, image_build: Mock) -> None: "service1", ] with temporary_root() as root: - utils.is_buildkit_enabled.cache_clear() - with patch.object(utils, "is_buildkit_enabled", return_value=False): - self.invoke_in_root(root, ["config", "save"]) - result = self.invoke_in_root(root, build_args) + self.invoke_in_root(root, ["config", "save"]) + result = self.invoke_in_root(root, build_args) self.assertIsNone(result.exception) self.assertEqual(0, result.exit_code) image_build.assert_called() diff --git a/tutor/commands/images.py b/tutor/commands/images.py index d21e2801cab..d5469f9263f 100644 --- a/tutor/commands/images.py +++ b/tutor/commands/images.py @@ -156,7 +156,7 @@ def images_command() -> None: # Export image to docker. This is necessary to make the image available to docker-compose. # The `--load` option is a shorthand for `--output=type=docker`. default="type=docker", - help="Same as `docker build --output=...`. This option will only be used when BuildKit is enabled.", + help="Same as `docker build --output=...`.", ) @click.option( "-a", @@ -211,7 +211,7 @@ def build( command_args += ["--add-host", add_host] if target: command_args += ["--target", target] - if utils.is_buildkit_enabled() and docker_output: + if docker_output: command_args.append(f"--output={docker_output}") if docker_args: command_args += docker_args @@ -223,27 +223,21 @@ def build( image_build_args = [*command_args, *custom_args] # Registry cache - if utils.is_buildkit_enabled(): - if not no_registry_cache: - image_build_args.append( - f"--cache-from=type=registry,ref={tag}-cache" - ) - if cache_to_registry: - image_build_args.append( - f"--cache-to=type=registry,mode=max,ref={tag}-cache" - ) + if not no_registry_cache: + image_build_args.append( + f"--cache-from=type=registry,ref={tag}-cache" + ) + if cache_to_registry: + image_build_args.append( + f"--cache-to=type=registry,mode=max,ref={tag}-cache" + ) # Build contexts for host_path, stage_name in build_contexts.get(name, []): - if utils.is_buildkit_enabled(): - fmt.echo_info( - f"Adding {host_path} to the build context '{stage_name}' of image '{image}'" - ) - image_build_args.append(f"--build-context={stage_name}={host_path}") - else: - fmt.echo_alert( - f"Unable to add {host_path} to the build context '{stage_name}' of image '{host_path}' because BuildKit is disabled." - ) + fmt.echo_info( + f"Adding {host_path} to the build context '{stage_name}' of image '{image}'" + ) + image_build_args.append(f"--build-context={stage_name}={host_path}") # Build images.build( diff --git a/tutor/env.py b/tutor/env.py index 980bb773490..3183b1b37fe 100644 --- a/tutor/env.py +++ b/tutor/env.py @@ -54,7 +54,7 @@ def _prepare_environment() -> None: ("HOST_USER_ID", utils.get_user_id()), ("TUTOR_APP", __app__.replace("-", "_")), ("TUTOR_VERSION", __version__), - ("is_buildkit_enabled", utils.is_buildkit_enabled), + ("is_buildkit_enabled", lambda: True), # Will be removed soon. ], ) diff --git a/tutor/hooks/catalog.py b/tutor/hooks/catalog.py index 2f628102d6b..dde0f4a4cf5 100644 --- a/tutor/hooks/catalog.py +++ b/tutor/hooks/catalog.py @@ -244,8 +244,7 @@ def your_filter_callback(some_data): #: names must be prefixed with the plugin name in all-caps. CONFIG_UNIQUE: Filter[list[tuple[str, Any]], []] = Filter() - #: Use this filter to modify the ``docker build`` command. For instance, to replace - #: the ``build`` subcommand by ``buildx build``. + #: Use this filter to modify the ``docker build`` command. #: #: :parameter list[str] command: the full build command, including options and #: arguments. Note that these arguments do not include the leading ``docker`` command. @@ -328,7 +327,7 @@ def your_filter_callback(some_data): #: - ``HOST_USER_ID``: the numerical ID of the user on the host. #: - ``TUTOR_APP``: the app name ("tutor" by default), used to determine the dev/local project names. #: - ``TUTOR_VERSION``: the current version of Tutor. - #: - ``is_buildkit_enabled``: a boolean function that indicates whether BuildKit is available on the host. + #: - ``is_buildkit_enabled``: a deprecated function which always returns ``True`` now. Will be removed before Quince. #: - ``iter_values_named``: a function to iterate on variables that start or end with a given string. #: - ``iter_mounts``: a function that yields compose-compatible bind-mounts for any given service. #: - ``patch``: a function to incorporate extra content into a template. diff --git a/tutor/images.py b/tutor/images.py index 26f80b1824e..66f2b0b1c96 100644 --- a/tutor/images.py +++ b/tutor/images.py @@ -4,8 +4,7 @@ def build(path: str, tag: str, *args: str) -> None: fmt.echo_info(f"Building image {tag}") build_command = ["build", f"--tag={tag}", *args, path] - if utils.is_buildkit_enabled(): - build_command.insert(0, "buildx") + build_command.insert(0, "buildx") command = hooks.Filters.DOCKER_BUILD_COMMAND.apply(build_command) utils.docker(*command) diff --git a/tutor/templates/build/openedx/Dockerfile b/tutor/templates/build/openedx/Dockerfile index 8df5eb8aeca..e58b1064722 100644 --- a/tutor/templates/build/openedx/Dockerfile +++ b/tutor/templates/build/openedx/Dockerfile @@ -1,11 +1,11 @@ -{% if is_buildkit_enabled() %}# syntax=docker/dockerfile:1.4{% endif %} +# syntax=docker/dockerfile:1.4 ###### Minimal image with base system requirements for most stages FROM docker.io/ubuntu:20.04 as minimal LABEL maintainer="Overhang.io " ENV DEBIAN_FRONTEND=noninteractive -RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/var/cache/apt,sharing=locked \ - --mount=type=cache,target=/var/lib/apt,sharing=locked{% endif %} \ +RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ + --mount=type=cache,target=/var/lib/apt,sharing=locked \ apt update && \ apt install -y build-essential curl git language-pack-en ENV LC_ALL en_US.UTF-8 @@ -14,8 +14,9 @@ ENV LC_ALL en_US.UTF-8 ###### Install python with pyenv in /opt/pyenv and create virtualenv in /openedx/venv FROM minimal as python # https://github.com/pyenv/pyenv/wiki/Common-build-problems#prerequisites -RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/var/cache/apt,sharing=locked \ - --mount=type=cache,target=/var/lib/apt,sharing=locked {% endif %}apt update && \ +RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ + --mount=type=cache,target=/var/lib/apt,sharing=locked \ + apt update && \ apt install -y libssl-dev zlib1g-dev libbz2-dev \ libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \ xz-utils tk-dev libffi-dev liblzma-dev python-openssl git @@ -77,26 +78,27 @@ ENV PATH /openedx/venv/bin:${PATH} ENV VIRTUAL_ENV /openedx/venv/ ENV XDG_CACHE_HOME /openedx/.cache -RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/var/cache/apt,sharing=locked \ - --mount=type=cache,target=/var/lib/apt,sharing=locked {% endif %}apt update \ +RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ + --mount=type=cache,target=/var/lib/apt,sharing=locked \ + apt update \ && apt install -y software-properties-common libmysqlclient-dev libxmlsec1-dev libgeos-dev # Install the right version of pip/setuptools -RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/openedx/.cache/pip,sharing=shared {% endif %}pip install \ +RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ + pip install \ # https://pypi.org/project/setuptools/ # https://pypi.org/project/pip/ # https://pypi.org/project/wheel/ setuptools==67.6.1 pip==23.0.1. wheel==0.40.0 # Install base requirements -{% if not is_buildkit_enabled() %} -COPY --from=edx-platform /requirements/edx/base.txt /openedx/edx-platform/requirements/edx/base.txt -{% endif %} -RUN {% if is_buildkit_enabled() %}--mount=type=bind,from=edx-platform,source=/requirements/edx/base.txt,target=/openedx/edx-platform/requirements/edx/base.txt \ - --mount=type=cache,target=/openedx/.cache/pip,sharing=shared {% endif %}pip install -r /openedx/edx-platform/requirements/edx/base.txt +RUN --mount=type=bind,from=edx-platform,source=/requirements/edx/base.txt,target=/openedx/edx-platform/requirements/edx/base.txt \ + --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ + pip install -r /openedx/edx-platform/requirements/edx/base.txt # Install extra requirements -RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/openedx/.cache/pip,sharing=shared {% endif %}pip install \ +RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ + pip install \ # Use redis as a django cache https://pypi.org/project/django-redis/ django-redis==5.2.0 \ # uwsgi server https://pypi.org/project/uWSGI/ @@ -106,11 +108,14 @@ RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/openedx/.cache/pip, # Install private requirements: this is useful for installing custom xblocks. COPY ./requirements/ /openedx/requirements -RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/openedx/.cache/pip,sharing=shared {% endif %}cd /openedx/requirements/ \ +RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ + cd /openedx/requirements/ \ && touch ./private.txt \ && pip install -r ./private.txt -{% for extra_requirements in OPENEDX_EXTRA_PIP_REQUIREMENTS %}RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/openedx/.cache/pip,sharing=shared {% endif %}pip install '{{ extra_requirements }}' +{% for extra_requirements in OPENEDX_EXTRA_PIP_REQUIREMENTS %} +RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ + pip install '{{ extra_requirements }}' {% endfor %} ###### Install nodejs with nodeenv in /openedx/nodeenv @@ -126,21 +131,19 @@ RUN nodeenv /openedx/nodeenv --node=16.14.0 --prebuilt # Install nodejs requirements ARG NPM_REGISTRY={{ NPM_REGISTRY }} WORKDIR /openedx/edx-platform -{% if not is_buildkit_enabled() %} -COPY --from=edx-platform /package.json /openedx/edx-platform/package.json -COPY --from=edx-platform /package-lock.json /openedx/edx-platform/package-lock.json -{% endif %} -RUN {% if is_buildkit_enabled() %}--mount=type=bind,from=edx-platform,source=/package.json,target=/openedx/edx-platform/package.json \ +RUN --mount=type=bind,from=edx-platform,source=/package.json,target=/openedx/edx-platform/package.json \ --mount=type=bind,from=edx-platform,source=/package-lock.json,target=/openedx/edx-platform/package-lock.json \ --mount=type=bind,from=edx-platform,source=/scripts/copy-node-modules.sh,target=/openedx/edx-platform/scripts/copy-node-modules.sh \ - --mount=type=cache,target=/root/.npm,sharing=shared {% endif %}npm clean-install --no-audit --registry=$NPM_REGISTRY + --mount=type=cache,target=/root/.npm,sharing=shared \ + npm clean-install --no-audit --registry=$NPM_REGISTRY ###### Production image with system and python requirements FROM minimal as production # Install system requirements -RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/var/cache/apt,sharing=locked \ - --mount=type=cache,target=/var/lib/apt,sharing=locked {% endif %}apt update \ +RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ + --mount=type=cache,target=/var/lib/apt,sharing=locked \ + apt update \ && apt install -y gettext gfortran graphviz graphviz-dev libffi-dev libfreetype6-dev libgeos-dev libjpeg8-dev liblapack-dev libmysqlclient-dev libpng-dev libsqlite3-dev libxmlsec1-dev lynx mysql-client ntp pkg-config rdfind # From then on, run as unprivileged "app" user @@ -151,7 +154,7 @@ RUN useradd --home-dir /openedx --create-home --shell /bin/bash --uid ${APP_USER USER ${APP_USER_ID} # https://hub.docker.com/r/powerman/dockerize/tags -COPY {% if is_buildkit_enabled() %}--link {% endif %}--from=docker.io/powerman/dockerize:0.19.0 /usr/local/bin/dockerize /usr/local/bin/dockerize +COPY --link --from=docker.io/powerman/dockerize:0.19.0 /usr/local/bin/dockerize /usr/local/bin/dockerize COPY --chown=app:app --from=edx-platform / /openedx/edx-platform COPY --chown=app:app --from=locales /openedx/locale /openedx/locale COPY --chown=app:app --from=python /opt/pyenv /opt/pyenv @@ -245,16 +248,19 @@ FROM production as development # Install useful system requirements (as root) USER root -RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/var/cache/apt,sharing=locked \ - --mount=type=cache,target=/var/lib/apt,sharing=locked {% endif %}apt update && \ +RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ + --mount=type=cache,target=/var/lib/apt,sharing=locked \ + apt update && \ apt install -y vim iputils-ping dnsutils telnet USER app # Install dev python requirements -RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/openedx/.cache/pip,sharing=shared {% endif %}pip install -r requirements/edx/development.txt +RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ + pip install -r requirements/edx/development.txt # https://pypi.org/project/ipdb/ # https://pypi.org/project/ipython -RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/openedx/.cache/pip,sharing=shared {% endif %}pip install ipdb==0.13.13 ipython==8.12.0 +RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ + pip install ipdb==0.13.13 ipython==8.12.0 # Add ipdb as default PYTHONBREAKPOINT ENV PYTHONBREAKPOINT=ipdb.set_trace diff --git a/tutor/utils.py b/tutor/utils.py index 59adee43778..2d8d3e561e6 100644 --- a/tutor/utils.py +++ b/tutor/utils.py @@ -173,25 +173,6 @@ def docker(*command: str) -> int: return execute("docker", *command) -@lru_cache(maxsize=None) -def is_buildkit_enabled() -> bool: - """ - A helper function to determine whether we can run `docker buildx` with BuildKit. - """ - # First, we respect the DOCKER_BUILDKIT environment variable - enabled_by_env = { - "1": True, - "0": False, - }.get(os.environ.get("DOCKER_BUILDKIT", "")) - if enabled_by_env is not None: - return enabled_by_env - try: - subprocess.run(["docker", "buildx", "version"], capture_output=True, check=True) - return True - except subprocess.CalledProcessError: - return False - - def docker_compose(*command: str) -> int: return execute("docker", "compose", *command) From f9638ca2cdfb6eccf48b488c4be59cc7dc420c8e Mon Sep 17 00:00:00 2001 From: "Kyle D. McCormick" Date: Fri, 18 Aug 2023 10:34:27 -0400 Subject: [PATCH 12/14] build: take advantage of upstream static asset build improvements TODOs: * See if we can reduce the huge mount blocks a bit. * Merge edx-platform asset folder changes instead of patching in a PR. * Add changelog entry. * Test more thoroughly. * Circulate a TEP or some other form of proposal? * Deprecate patches that no longer exist or have changed. Part of: https://github.com/openedx/wg-developer-experience/issues/166 --- tutor/commands/images.py | 2 +- tutor/templates/build/openedx/Dockerfile | 448 +++++++++++++++++------ 2 files changed, 329 insertions(+), 121 deletions(-) diff --git a/tutor/commands/images.py b/tutor/commands/images.py index d5469f9263f..e8b43bbd2c8 100644 --- a/tutor/commands/images.py +++ b/tutor/commands/images.py @@ -34,7 +34,7 @@ def _add_core_images_to_build( image, os.path.join("build", image), tutor_config.get_typed(config, tag, str), - (), + ("--target=production",), ) ) diff --git a/tutor/templates/build/openedx/Dockerfile b/tutor/templates/build/openedx/Dockerfile index e58b1064722..22ed8e73fa2 100644 --- a/tutor/templates/build/openedx/Dockerfile +++ b/tutor/templates/build/openedx/Dockerfile @@ -1,5 +1,10 @@ # syntax=docker/dockerfile:1.4 -###### Minimal image with base system requirements for most stages + + +########################################################################################################################### +############## MINIMAL +############## Minimal image with base system requirements for most stages +########################################################################################################################### FROM docker.io/ubuntu:20.04 as minimal LABEL maintainer="Overhang.io " @@ -11,21 +16,31 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ ENV LC_ALL en_US.UTF-8 {{ patch("openedx-dockerfile-minimal") }} -###### Install python with pyenv in /opt/pyenv and create virtualenv in /openedx/venv + +########################################################################################################################### +############## PYTHON +############## Install python with pyenv in /opt/pyenv and create virtualenv in /openedx/venv +########################################################################################################################### FROM minimal as python + # https://github.com/pyenv/pyenv/wiki/Common-build-problems#prerequisites RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ --mount=type=cache,target=/var/lib/apt,sharing=locked \ apt update && \ apt install -y libssl-dev zlib1g-dev libbz2-dev \ libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \ - xz-utils tk-dev libffi-dev liblzma-dev python-openssl git + xz-utils tk-dev libffi-dev liblzma-dev python-openssl git \ + software-properties-common libmysqlclient-dev libxmlsec1-dev libgeos-dev + +ARG PYTHON_VERSION=3.8.15 +ENV PYENV_ROOT /opt/pyenv +ENV PATH /openedx/venv/bin:${PATH} +ENV VIRTUAL_ENV /openedx/venv/ +ENV XDG_CACHE_HOME /openedx/.cache # Install pyenv # https://www.python.org/downloads/ # https://github.com/pyenv/pyenv/releases -ARG PYTHON_VERSION=3.8.15 -ENV PYENV_ROOT /opt/pyenv RUN git clone https://github.com/pyenv/pyenv $PYENV_ROOT --branch v2.3.17 --depth 1 # Install Python @@ -34,8 +49,21 @@ RUN $PYENV_ROOT/bin/pyenv install $PYTHON_VERSION # Create virtualenv RUN $PYENV_ROOT/versions/$PYTHON_VERSION/bin/python -m venv /openedx/venv -###### Checkout edx-platform code +# Install the right version of pip/setuptools +RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ + pip install \ + # https://pypi.org/project/setuptools/ + # https://pypi.org/project/pip/ + # https://pypi.org/project/wheel/ + setuptools==67.6.1 pip==23.0.1. wheel==0.40.0 + + +########################################################################################################################### +############## CODE +########################################################################################################################### FROM minimal as code + +###### Checkout edx-platform code ARG EDX_PLATFORM_REPOSITORY={{ EDX_PLATFORM_REPOSITORY }} ARG EDX_PLATFORM_VERSION={{ EDX_PLATFORM_VERSION }} RUN mkdir -p /openedx/edx-platform && \ @@ -56,12 +84,24 @@ RUN git config --global user.email "tutor@overhang.io" \ {# Example: RUN curl -fsSL https://github.com/openedx/edx-platform/commit/.patch | git am #} {{ patch("openedx-dockerfile-post-git-checkout") }} -##### Empty layer with just the repo at the root. -# This is useful when overriding the build context with a host repo: -# docker build --build-context edx-platform=/path/to/edx-platform +# TODO: Temporary +# Apply these edx-platform changes which have not yet merged. +RUN curl -fsSL https://github.com/openedx/edx-platform/pull/32835.patch | git am + + +########################################################################################################################### +############## EDX-PLATFORM +############## Empty layer with just the repo at the root. +############## This is useful when overriding the build context with a host repo: +############## docker build --build-context edx-platform=/path/to/edx-platform +########################################################################################################################### FROM scratch as edx-platform COPY --from=code /openedx/edx-platform / + +########################################################################################################################### +############## LOCALES +########################################################################################################################### ###### Download extra locales to /openedx/locale/contrib/locale FROM minimal as locales ARG OPENEDX_I18N_VERSION={{ OPENEDX_COMMON_VERSION }} @@ -72,24 +112,185 @@ RUN cd /tmp \ && mv openedx-i18n-*/edx-platform/locale /openedx/locale/contrib \ && rm -rf openedx-i18n* -###### Install python requirements in virtualenv -FROM python as python-requirements -ENV PATH /openedx/venv/bin:${PATH} -ENV VIRTUAL_ENV /openedx/venv/ -ENV XDG_CACHE_HOME /openedx/.cache -RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ - --mount=type=cache,target=/var/lib/apt,sharing=locked \ - apt update \ - && apt install -y software-properties-common libmysqlclient-dev libxmlsec1-dev libgeos-dev -# Install the right version of pip/setuptools -RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ - pip install \ - # https://pypi.org/project/setuptools/ - # https://pypi.org/project/pip/ - # https://pypi.org/project/wheel/ - setuptools==67.6.1 pip==23.0.1. wheel==0.40.0 +########################################################################################################################### +############## FRONTEND REQUIREMENTS +########################################################################################################################### +FROM python as requirements-frontend + +WORKDIR /openedx/edx-platform + +# Install python reqs for building assets +RUN --mount=type=bind,from=edx-platform,source=/requirements/edx/assets.txt,target=requirements/edx/assets.txt \ + pip install -r requirements/edx/assets.txt + +# Set up node env +ENV PATH /openedx/nodeenv/bin:/openedx/venv/bin:${PATH} +RUN nodeenv /openedx/nodeenv --node=16.14.0 --prebuilt + +# Install nodejs requirements +ARG NPM_REGISTRY={{ NPM_REGISTRY }} +RUN --mount=type=bind,from=edx-platform,source=/package.json,target=package.json \ + --mount=type=bind,from=edx-platform,source=/package-lock.json,target=package-lock.json \ + --mount=type=bind,from=edx-platform,source=/scripts/copy-node-modules.sh,target=scripts/copy-node-modules.sh \ + --mount=type=cache,target=/root/.npm,sharing=shared \ + npm clean-install --no-audit --registry=$NPM_REGISTRY + +# Set up static root. +# By default, CMS static root is assumed to be $STUDIO_ROOT_LMS/studio. +ENV STATIC_ROOT_LMS=/openedx/staticfiles +RUN mkdir -p "$STATIC_ROOT_LMS" + + +########################################################################################################################### +############## BUNDLES: PRODUCTION +########################################################################################################################### +FROM requirements-frontend as bundles-production + +# Run webpack +RUN --mount=type=bind,from=edx-platform,source=/package.json,target=package.json \ + --mount=type=bind,from=edx-platform,source=/package-lock.json,target=/openedx/edx-platform/package-lock.json \ + --mount=type=bind,from=edx-platform,source=/.babelrc,target=.babelrc \ + --mount=type=bind,from=edx-platform,source=/webpack.common.config.js,target=webpack.common.config.js \ + --mount=type=bind,from=edx-platform,source=/webpack-config/file-lists.js,target=webpack-config/file-lists.js \ + --mount=type=bind,from=edx-platform,source=/webpack.dev.config.js,target=webpack.dev.config.js \ + --mount=type=bind,from=edx-platform,source=/webpack.prod.config.js,target=webpack.prod.config.js \ + --mount=type=bind,from=edx-platform,source=/webpack.builtinblocks.config.js,target=webpack.builtinblocks.config.js \ + --mount=type=bind,from=edx-platform,source=/cms/djangoapps/pipeline_js/js/xmodule.js,target=cms/djangoapps/pipeline_js/js/xmodule.js \ + --mount=type=bind,from=edx-platform,source=/cms/static,target=cms/static \ + --mount=type=bind,from=edx-platform,source=/cms/templates/,target=cms/templates/ \ + --mount=type=bind,from=edx-platform,source=/common/static/js/,target=common/static/js \ + --mount=type=bind,from=edx-platform,source=/common/static/common,target=common/static/common,rw \ + --mount=type=bind,from=requirements-frontend,source=/openedx/edx-platform/common/static/common/js/vendor,target=common/static/common/js/vendor \ + --mount=type=bind,from=requirements-frontend,source=/openedx/edx-platform/common/static/common/css/vendor,target=common/static/common/css/vendor \ + --mount=type=bind,from=edx-platform,source=/lms/djangoapps/discussion/static,target=lms/djangoapps/discussion/static \ + --mount=type=bind,from=edx-platform,source=/lms/djangoapps/instructor/static,target=lms/djangoapps/instructor/static \ + --mount=type=bind,from=edx-platform,source=/lms/djangoapps/support/static/support,target=lms/djangoapps/support/static/support \ + --mount=type=bind,from=edx-platform,source=/lms/djangoapps/teams/static,target=lms/djangoapps/teams/static \ + --mount=type=bind,from=edx-platform,source=/lms/static/,target=lms/static/ \ + --mount=type=bind,from=edx-platform,source=/lms/templates/,target=lms/templates/ \ + --mount=type=bind,from=edx-platform,source=/openedx/features/announcements/static,target=openedx/features/announcements/static \ + --mount=type=bind,from=edx-platform,source=/openedx/features/course_bookmarks/static,target=openedx/features/course_bookmarks/static \ + --mount=type=bind,from=edx-platform,source=/openedx/features/course_experience/static,target=openedx/features/course_experience/static \ + --mount=type=bind,from=edx-platform,source=/openedx/features/course_search/static,target=openedx/features/course_search/static \ + --mount=type=bind,from=edx-platform,source=/openedx/features/learner_profile/static,target=openedx/features/learner_profile/static \ + --mount=type=bind,from=edx-platform,source=/xmodule/assets,target=xmodule/assets \ + --mount=type=bind,from=edx-platform,source=/xmodule/js,target=xmodule/js \ + npm run webpack + + +########################################################################################################################### +############## BUNDLES: DEVELOPMENT +########################################################################################################################### +FROM requirements-frontend as bundles-development + +# Run webpack +RUN --mount=type=bind,from=edx-platform,source=/package.json,target=package.json \ + --mount=type=bind,from=edx-platform,source=/package-lock.json,target=/openedx/edx-platform/package-lock.json \ + --mount=type=bind,from=edx-platform,source=/.babelrc,target=.babelrc \ + --mount=type=bind,from=edx-platform,source=/webpack.common.config.js,target=webpack.common.config.js \ + --mount=type=bind,from=edx-platform,source=/webpack-config/file-lists.js,target=webpack-config/file-lists.js \ + --mount=type=bind,from=edx-platform,source=/webpack.dev.config.js,target=webpack.dev.config.js \ + --mount=type=bind,from=edx-platform,source=/webpack.prod.config.js,target=webpack.prod.config.js \ + --mount=type=bind,from=edx-platform,source=/webpack.builtinblocks.config.js,target=webpack.builtinblocks.config.js \ + --mount=type=bind,from=edx-platform,source=/cms/djangoapps/pipeline_js/js/xmodule.js,target=cms/djangoapps/pipeline_js/js/xmodule.js \ + --mount=type=bind,from=edx-platform,source=/cms/static,target=cms/static \ + --mount=type=bind,from=edx-platform,source=/cms/templates/,target=cms/templates/ \ + --mount=type=bind,from=edx-platform,source=/common/static/js/,target=common/static/js \ + --mount=type=bind,from=edx-platform,source=/common/static/common,target=common/static/common,rw \ + --mount=type=bind,from=requirements-frontend,source=/openedx/edx-platform/common/static/common/js/vendor,target=common/static/common/js/vendor \ + --mount=type=bind,from=requirements-frontend,source=/openedx/edx-platform/common/static/common/css/vendor,target=common/static/common/css/vendor \ + --mount=type=bind,from=edx-platform,source=/lms/djangoapps/discussion/static,target=lms/djangoapps/discussion/static \ + --mount=type=bind,from=edx-platform,source=/lms/djangoapps/instructor/static,target=lms/djangoapps/instructor/static \ + --mount=type=bind,from=edx-platform,source=/lms/djangoapps/support/static/support,target=lms/djangoapps/support/static/support \ + --mount=type=bind,from=edx-platform,source=/lms/djangoapps/teams/static,target=lms/djangoapps/teams/static \ + --mount=type=bind,from=edx-platform,source=/lms/static/,target=lms/static/ \ + --mount=type=bind,from=edx-platform,source=/lms/templates/,target=lms/templates/ \ + --mount=type=bind,from=edx-platform,source=/openedx/features/announcements/static,target=openedx/features/announcements/static \ + --mount=type=bind,from=edx-platform,source=/openedx/features/course_bookmarks/static,target=openedx/features/course_bookmarks/static \ + --mount=type=bind,from=edx-platform,source=/openedx/features/course_experience/static,target=openedx/features/course_experience/static \ + --mount=type=bind,from=edx-platform,source=/openedx/features/course_search/static,target=openedx/features/course_search/static \ + --mount=type=bind,from=edx-platform,source=/openedx/features/learner_profile/static,target=openedx/features/learner_profile/static \ + --mount=type=bind,from=edx-platform,source=/xmodule/assets,target=xmodule/assets \ + --mount=type=bind,from=edx-platform,source=/xmodule/js,target=xmodule/js \ + npm run webpack-dev + + +########################################################################################################################### +############## CSS: PRODUCTION +########################################################################################################################### +FROM requirements-frontend as css-production + +ENV PATH ./node_modules/.bin:${PATH} + +# Compile default CSS +RUN --mount=type=bind,from=edx-platform,source=/package.json,target=package.json \ + --mount=type=bind,from=edx-platform,source=/scripts/compile_sass.py,target=scripts/compile_sass.py \ + --mount=type=bind,from=edx-platform,source=/common/static,target=common/static \ + --mount=type=bind,from=edx-platform,source=/lms/static/sass,target=lms/static/sass \ + --mount=type=bind,from=edx-platform,source=/lms/static/sass/partials,target=lms/static/sass/partials \ + --mount=type=bind,from=edx-platform,source=/lms/static/certificates/sass,target=lms/static/certificates/sass \ + --mount=type=bind,from=edx-platform,source=/cms/static/sass,target=cms/static/sass \ + --mount=type=bind,from=edx-platform,source=/cms/static/sass/partials,target=cms/static/sass/partials \ + --mount=type=bind,from=edx-platform,source=/xmodule/assets,target=xmodule/assets \ + npm run compile-sass -- --skip-themes + +# Compile themed CSS +ENV EDX_PLATFORM_THEME_DIRS=/openedx/themes +COPY --link --chown=app:app ./themes/ /openedx/themes/ +RUN --mount=type=bind,from=edx-platform,source=/package.json,target=package.json \ + --mount=type=bind,from=edx-platform,source=/scripts/compile_sass.py,target=scripts/compile_sass.py \ + --mount=type=bind,from=edx-platform,source=/common/static,target=common/static \ + --mount=type=bind,from=edx-platform,source=/lms/static/sass,target=lms/static/sass \ + --mount=type=bind,from=edx-platform,source=/lms/static/sass/partials,target=lms/static/sass/partials \ + --mount=type=bind,from=edx-platform,source=/lms/static/certificates/sass,target=lms/static/certificates/sass \ + --mount=type=bind,from=edx-platform,source=/cms/static/sass,target=cms/static/sass \ + --mount=type=bind,from=edx-platform,source=/cms/static/sass/partials,target=cms/static/sass/partials \ + --mount=type=bind,from=edx-platform,source=/xmodule/assets,target=xmodule/assets \ + npm run compile-sass -- --skip-default + + +########################################################################################################################### +############## CSS: DEVELOPMENT +########################################################################################################################### +FROM requirements-frontend as css-development + +ENV PATH ./node_modules/.bin:${PATH} + +COPY --link --from=edx-platform --chown=app:app /lms/static/css/vendor lms/static/css/vendor + +# Compile default CSS +RUN --mount=type=bind,from=edx-platform,source=/package.json,target=package.json \ + --mount=type=bind,from=edx-platform,source=/scripts/compile_sass.py,target=scripts/compile_sass.py \ + --mount=type=bind,from=edx-platform,source=/common/static,target=common/static \ + --mount=type=bind,from=edx-platform,source=/lms/static/sass,target=lms/static/sass \ + --mount=type=bind,from=edx-platform,source=/lms/static/sass/partials,target=lms/static/sass/partials \ + --mount=type=bind,from=edx-platform,source=/lms/static/certificates/sass,target=lms/static/certificates/sass \ + --mount=type=bind,from=edx-platform,source=/cms/static/sass,target=cms/static/sass \ + --mount=type=bind,from=edx-platform,source=/cms/static/sass/partials,target=cms/static/sass/partials \ + --mount=type=bind,from=edx-platform,source=/xmodule/assets,target=xmodule/assets \ + npm run compile-sass -- --skip-themes --env=dev + +# Compile themed CSS +ENV EDX_PLATFORM_THEME_DIRS=/openedx/themes +COPY --link --chown=app:app ./themes/ /openedx/themes/ +RUN --mount=type=bind,from=edx-platform,source=/package.json,target=package.json \ + --mount=type=bind,from=edx-platform,source=/scripts/compile_sass.py,target=scripts/compile_sass.py \ + --mount=type=bind,from=edx-platform,source=/common/static,target=common/static \ + --mount=type=bind,from=edx-platform,source=/lms/static/sass,target=lms/static/sass \ + --mount=type=bind,from=edx-platform,source=/lms/static/sass/partials,target=lms/static/sass/partials \ + --mount=type=bind,from=edx-platform,source=/lms/static/certificates/sass,target=lms/static/certificates/sass \ + --mount=type=bind,from=edx-platform,source=/cms/static/sass,target=cms/static/sass \ + --mount=type=bind,from=edx-platform,source=/cms/static/sass/partials,target=cms/static/sass/partials \ + --mount=type=bind,from=edx-platform,source=/xmodule/assets,target=xmodule/assets \ + npm run compile-sass -- --skip-default --env=dev + + +########################################################################################################################### +############## APPLICATION REQUIREMENTS +########################################################################################################################### +FROM requirements-frontend as requirements-application # Install base requirements RUN --mount=type=bind,from=edx-platform,source=/requirements/edx/base.txt,target=/openedx/edx-platform/requirements/edx/base.txt \ @@ -107,38 +308,44 @@ RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ {{ patch("openedx-dockerfile-post-python-requirements") }} # Install private requirements: this is useful for installing custom xblocks. -COPY ./requirements/ /openedx/requirements +COPY --link ./requirements/ /openedx/requirements RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ - cd /openedx/requirements/ \ - && touch ./private.txt \ - && pip install -r ./private.txt + cd /openedx/requirements/ \ + && touch ./private.txt \ + && pip install -r ./private.txt -{% for extra_requirements in OPENEDX_EXTRA_PIP_REQUIREMENTS %} -RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ - pip install '{{ extra_requirements }}' +{% for extra_requirements in OPENEDX_EXTRA_PIP_REQUIREMENTS %}RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared pip install '{{ extra_requirements }}' {% endfor %} -###### Install nodejs with nodeenv in /openedx/nodeenv -FROM python as nodejs-requirements -ENV PATH /openedx/nodeenv/bin:/openedx/venv/bin:${PATH} -# Install nodeenv with the version provided by edx-platform -# https://github.com/openedx/edx-platform/blob/master/requirements/edx/base.txt -# https://github.com/pyenv/pyenv/releases -RUN pip install nodeenv==1.7.0 -RUN nodeenv /openedx/nodeenv --node=16.14.0 --prebuilt +########################################################################################################################### +############## PRODUCTION REQUIREMENTS +############## Simply take the general requirements, and remove everything related to libsass. +############## Alternatively, we could have based requirements-application on minimal, thus keeping libsass out of it, +############## BUT that would have required requirements-development to recompile libsass, which is slow. +########################################################################################################################### +FROM requirements-application as requirements-production + +RUN rm -r /openedx/venv/lib/python3.8/site-packages/*sass* + + +########################################################################################################################### +############## DEVELOPMENT REQUIREMENTS +########################################################################################################################### +FROM requirements-application as requirements-development + +# Install dev requirements +RUN --mount=type=bind,from=edx-platform,source=/requirements/edx/development.txt,target=/openedx/edx-platform/requirements/edx/development.txt \ + --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ + pip install -r /openedx/edx-platform/requirements/edx/development.txt -# Install nodejs requirements -ARG NPM_REGISTRY={{ NPM_REGISTRY }} -WORKDIR /openedx/edx-platform -RUN --mount=type=bind,from=edx-platform,source=/package.json,target=/openedx/edx-platform/package.json \ - --mount=type=bind,from=edx-platform,source=/package-lock.json,target=/openedx/edx-platform/package-lock.json \ - --mount=type=bind,from=edx-platform,source=/scripts/copy-node-modules.sh,target=/openedx/edx-platform/scripts/copy-node-modules.sh \ - --mount=type=cache,target=/root/.npm,sharing=shared \ - npm clean-install --no-audit --registry=$NPM_REGISTRY -###### Production image with system and python requirements -FROM minimal as production +########################################################################################################################### +############## APPLICATION +############## Application image with system and python requirements. +############## Basis for final prod and dev images. +########################################################################################################################### +FROM minimal as application # Install system requirements RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ @@ -153,18 +360,15 @@ RUN if [ "$APP_USER_ID" = 0 ]; then echo "app user may not be root" && false; fi RUN useradd --home-dir /openedx --create-home --shell /bin/bash --uid ${APP_USER_ID} app USER ${APP_USER_ID} -# https://hub.docker.com/r/powerman/dockerize/tags +# Copy in backend code and requirements. +# Dockerize tags: https://hub.docker.com/r/powerman/dockerize/tags COPY --link --from=docker.io/powerman/dockerize:0.19.0 /usr/local/bin/dockerize /usr/local/bin/dockerize -COPY --chown=app:app --from=edx-platform / /openedx/edx-platform -COPY --chown=app:app --from=locales /openedx/locale /openedx/locale -COPY --chown=app:app --from=python /opt/pyenv /opt/pyenv -COPY --chown=app:app --from=python-requirements /openedx/venv /openedx/venv -COPY --chown=app:app --from=python-requirements /openedx/requirements /openedx/requirements -COPY --chown=app:app --from=nodejs-requirements /openedx/nodeenv /openedx/nodeenv -COPY --chown=app:app --from=nodejs-requirements /openedx/edx-platform/node_modules /openedx/node_modules - -# Symlink node_modules such that we can bind-mount the edx-platform repository -RUN ln -s /openedx/node_modules /openedx/edx-platform/node_modules +COPY --link --chown=app:app --from=edx-platform / /openedx/edx-platform +COPY --link --chown=app:app ./themes/ /openedx/themes/ +COPY --link --chown=app:app --from=locales /openedx/locale /openedx/locale +COPY --link --chown=app:app --from=python /opt/pyenv /opt/pyenv +COPY --link --chown=app:app --from=requirements-production /openedx/venv /openedx/venv +COPY --link --chown=app:app --from=requirements-production /openedx/requirements /openedx/requirements ENV PATH /openedx/venv/bin:./node_modules/.bin:/openedx/nodeenv/bin:${PATH} ENV VIRTUAL_ENV /openedx/venv/ @@ -190,6 +394,20 @@ COPY --chown=app:app ./locale/ /openedx/locale/user/locale/ RUN cd /openedx/locale/user && \ django-admin compilemessages -v1 +# Copy in frontend requirements. +COPY --link --chown=app:app --from=requirements-frontend \ + /openedx/nodeenv \ + /openedx/nodeenv +COPY --link --chown=app:app --from=requirements-frontend \ + /openedx/edx-platform/node_modules \ + /openedx/edx-platform/node_modules +COPY --link --chown=app:app --from=requirements-frontend \ + /openedx/edx-platform/common/static/common/js/vendor \ + /openedx/edx-platform/common/static/common/js/vendor +COPY --link --chown=app:app --from=requirements-frontend \ + /openedx/edx-platform/common/static/common/css/vendor \ + /openedx/edx-platform/common/static/common/css/vendor + # Compile i18n strings: in some cases, js locales are not properly compiled out of the box # and we need to do a pass ourselves. Also, we need to compile the djangojs.js files for # the downloaded locales. @@ -197,54 +415,57 @@ RUN ./manage.py lms --settings=tutor.i18n compilejsi18n RUN ./manage.py cms --settings=tutor.i18n compilejsi18n # Copy scripts -COPY --chown=app:app ./bin /openedx/bin +COPY --link --chown=app:app ./bin /openedx/bin RUN chmod a+x /openedx/bin/* ENV PATH /openedx/bin:${PATH} -{{ patch("openedx-dockerfile-pre-assets") }} - -# Collect production assets. By default, only assets from the default theme -# will be processed. This makes the docker image lighter and faster to build. -# Only the custom themes added to /openedx/themes will be compiled. -# Here, we don't run "paver update_assets" which is slow, compiles all themes -# and requires a complex settings file. Instead, we decompose the commands -# and run each one individually to collect the production static assets to -# /openedx/staticfiles. -ENV NO_PYTHON_UNINSTALL 1 -ENV NO_PREREQ_INSTALL 1 -# We need to rely on a separate openedx-assets command to accelerate asset processing. -# For instance, we don't want to run all steps of asset collection every time the theme -# is modified. -RUN openedx-assets xmodule \ - && openedx-assets npm \ - && openedx-assets webpack --env=prod \ - && openedx-assets common -COPY --chown=app:app ./themes/ /openedx/themes/ -RUN openedx-assets themes \ - && openedx-assets collect --settings=tutor.assets \ - # De-duplicate static assets with symlinks - && rdfind -makesymlinks true -followsymlinks true /openedx/staticfiles/ - # Create a data directory, which might be used (or not) RUN mkdir /openedx/data -# If this "canary" file is missing from a container, then that indicates that a -# local edx-platform was bind-mounted into that container, thus overwriting the -# canary. This information is useful during edx-platform initialisation. -RUN echo \ - "This copy of edx-platform was built into a Docker image." \ - > bindmount-canary - # service variant is "lms" or "cms" ENV SERVICE_VARIANT lms -ENV DJANGO_SETTINGS_MODULE lms.envs.tutor.production {{ patch("openedx-dockerfile") }} EXPOSE 8000 -###### Intermediate image with dev/test dependencies -FROM production as development + +########################################################################################################################### +############## PRODUCTION +########################################################################################################################### +FROM application as production + +# Copy in and collect production static assets. +COPY --link --chown=app:app --from=bundles-production /openedx/staticfiles /openedx/staticfiles +COPY --link --chown=app:app --from=bundles-production /openedx/edx-platform/common/static/bundles common/static/bundles +COPY --link --chown=app:app --from=css-production /openedx/edx-platform/lms/static/css lms/static/css +COPY --link --chown=app:app --from=css-production /openedx/edx-platform/lms/static/certificates/css lms/static/certificates/css +COPY --link --chown=app:app --from=css-production /openedx/edx-platform/cms/static/css cms/static/css +RUN ./manage.py lms collectstatic --noinput --settings=tutor.assets +RUN ./manage.py cms collectstatic --noinput --settings=tutor.assets + +# De-dupe static assets with synlinks +RUN rdfind -makesymlinks true -followsymlinks true /openedx/staticfiles/ + +# Default amount of uWSGI processes +ENV UWSGI_WORKERS=2 + +# Copy the default uWSGI configuration +COPY --chown=app:app settings/uwsgi.ini . + +# Default django settings +ENV DJANGO_SETTINGS_MODULE lms.envs.tutor.production + +# Run server +CMD uwsgi uwsgi.ini + +{{ patch("openedx-dockerfile-final") }} + + +########################################################################################################################### +############## DEVELOPMENT +########################################################################################################################### +FROM application as development # Install useful system requirements (as root) USER root @@ -254,9 +475,9 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ apt install -y vim iputils-ping dnsutils telnet USER app -# Install dev python requirements -RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ - pip install -r requirements/edx/development.txt +# Copy in dev python requirements +COPY --link --chown=app:app --from=requirements-development /openedx/venv /openedx/venv + # https://pypi.org/project/ipdb/ # https://pypi.org/project/ipython RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ @@ -265,12 +486,14 @@ RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ # Add ipdb as default PYTHONBREAKPOINT ENV PYTHONBREAKPOINT=ipdb.set_trace -# Recompile static assets: in development mode all static assets are stored in edx-platform, -# and the location of these files is stored in webpack-stats.json. If we don't recompile -# static assets, then production assets will be served instead. -RUN rm -r /openedx/staticfiles && \ - mkdir /openedx/staticfiles && \ - openedx-assets webpack --env=dev +# Copy in development static assets. +# In development mode, edx-platform expects the files to be in the repo, and pointed to by webpack-stats.json. +COPY --link --chown=app:app --from=bundles-development /openedx/staticfiles/webpack-stats.json /openedx/staticfiles/webpack-stats.json +COPY --link --chown=app:app --from=bundles-development /openedx/staticfiles/studio/webpack-stats.json /openedx/staticfiles/studio/webpack-stats.json +COPY --link --chown=app:app --from=bundles-development /openedx/edx-platform/common/static/bundles common/static/bundles +COPY --link --chown=app:app --from=css-development /openedx/edx-platform/lms/static/css lms/static/css +COPY --link --chown=app:app --from=css-development /openedx/edx-platform/lms/static/certificates/css lms/static/certificates/css +COPY --link --chown=app:app --from=css-development /openedx/edx-platform/cms/static/css cms/static/css {{ patch("openedx-dev-dockerfile-post-python-requirements") }} @@ -278,18 +501,3 @@ RUN rm -r /openedx/staticfiles && \ ENV DJANGO_SETTINGS_MODULE lms.envs.tutor.development CMD ./manage.py $SERVICE_VARIANT runserver 0.0.0.0:8000 - -###### Final image with production cmd -FROM production as final - -# Default amount of uWSGI processes -ENV UWSGI_WORKERS=2 - -# Copy the default uWSGI configuration -COPY --chown=app:app settings/uwsgi.ini . - -# Run server -CMD uwsgi uwsgi.ini - -{{ patch("openedx-dockerfile-final") }} - From 780d6f6de2e2245ce62a2912620072abb2fd4055 Mon Sep 17 00:00:00 2001 From: "Kyle D. McCormick" Date: Thu, 31 Aug 2023 11:33:20 -0400 Subject: [PATCH 13/14] build: squash: use 'COPY --from' rather than 'RUN --mount' (WIP) --- tutor/templates/build/openedx/Dockerfile | 239 ++++++++--------------- 1 file changed, 81 insertions(+), 158 deletions(-) diff --git a/tutor/templates/build/openedx/Dockerfile b/tutor/templates/build/openedx/Dockerfile index 22ed8e73fa2..a07939cd5fc 100644 --- a/tutor/templates/build/openedx/Dockerfile +++ b/tutor/templates/build/openedx/Dockerfile @@ -117,13 +117,13 @@ RUN cd /tmp \ ########################################################################################################################### ############## FRONTEND REQUIREMENTS ########################################################################################################################### -FROM python as requirements-frontend +FROM python as frontend-requirements WORKDIR /openedx/edx-platform -# Install python reqs for building assets -RUN --mount=type=bind,from=edx-platform,source=/requirements/edx/assets.txt,target=requirements/edx/assets.txt \ - pip install -r requirements/edx/assets.txt +# Install python reqs for setting up node & building assets +COPY --link --from=edx-platform /requirements/edx/assets.txt requirements/edx/assets.txt +RUN pip install -r requirements/edx/assets.txt # Set up node env ENV PATH /openedx/nodeenv/bin:/openedx/venv/bin:${PATH} @@ -131,10 +131,10 @@ RUN nodeenv /openedx/nodeenv --node=16.14.0 --prebuilt # Install nodejs requirements ARG NPM_REGISTRY={{ NPM_REGISTRY }} -RUN --mount=type=bind,from=edx-platform,source=/package.json,target=package.json \ - --mount=type=bind,from=edx-platform,source=/package-lock.json,target=package-lock.json \ - --mount=type=bind,from=edx-platform,source=/scripts/copy-node-modules.sh,target=scripts/copy-node-modules.sh \ - --mount=type=cache,target=/root/.npm,sharing=shared \ +COPY --link --from=edx-platform /package.json package.json +COPY --link --from=edx-platform /package-lock.json package-lock.json +COPY --link --from=edx-platform /scripts/copy-node-modules.sh scripts/copy-node-modules.sh +RUN --mount=type=cache,target=/root/.npm,sharing=shared \ npm clean-install --no-audit --registry=$NPM_REGISTRY # Set up static root. @@ -144,157 +144,89 @@ RUN mkdir -p "$STATIC_ROOT_LMS" ########################################################################################################################### -############## BUNDLES: PRODUCTION +############## BUNDLES ########################################################################################################################### -FROM requirements-frontend as bundles-production - -# Run webpack -RUN --mount=type=bind,from=edx-platform,source=/package.json,target=package.json \ - --mount=type=bind,from=edx-platform,source=/package-lock.json,target=/openedx/edx-platform/package-lock.json \ - --mount=type=bind,from=edx-platform,source=/.babelrc,target=.babelrc \ - --mount=type=bind,from=edx-platform,source=/webpack.common.config.js,target=webpack.common.config.js \ - --mount=type=bind,from=edx-platform,source=/webpack-config/file-lists.js,target=webpack-config/file-lists.js \ - --mount=type=bind,from=edx-platform,source=/webpack.dev.config.js,target=webpack.dev.config.js \ - --mount=type=bind,from=edx-platform,source=/webpack.prod.config.js,target=webpack.prod.config.js \ - --mount=type=bind,from=edx-platform,source=/webpack.builtinblocks.config.js,target=webpack.builtinblocks.config.js \ - --mount=type=bind,from=edx-platform,source=/cms/djangoapps/pipeline_js/js/xmodule.js,target=cms/djangoapps/pipeline_js/js/xmodule.js \ - --mount=type=bind,from=edx-platform,source=/cms/static,target=cms/static \ - --mount=type=bind,from=edx-platform,source=/cms/templates/,target=cms/templates/ \ - --mount=type=bind,from=edx-platform,source=/common/static/js/,target=common/static/js \ - --mount=type=bind,from=edx-platform,source=/common/static/common,target=common/static/common,rw \ - --mount=type=bind,from=requirements-frontend,source=/openedx/edx-platform/common/static/common/js/vendor,target=common/static/common/js/vendor \ - --mount=type=bind,from=requirements-frontend,source=/openedx/edx-platform/common/static/common/css/vendor,target=common/static/common/css/vendor \ - --mount=type=bind,from=edx-platform,source=/lms/djangoapps/discussion/static,target=lms/djangoapps/discussion/static \ - --mount=type=bind,from=edx-platform,source=/lms/djangoapps/instructor/static,target=lms/djangoapps/instructor/static \ - --mount=type=bind,from=edx-platform,source=/lms/djangoapps/support/static/support,target=lms/djangoapps/support/static/support \ - --mount=type=bind,from=edx-platform,source=/lms/djangoapps/teams/static,target=lms/djangoapps/teams/static \ - --mount=type=bind,from=edx-platform,source=/lms/static/,target=lms/static/ \ - --mount=type=bind,from=edx-platform,source=/lms/templates/,target=lms/templates/ \ - --mount=type=bind,from=edx-platform,source=/openedx/features/announcements/static,target=openedx/features/announcements/static \ - --mount=type=bind,from=edx-platform,source=/openedx/features/course_bookmarks/static,target=openedx/features/course_bookmarks/static \ - --mount=type=bind,from=edx-platform,source=/openedx/features/course_experience/static,target=openedx/features/course_experience/static \ - --mount=type=bind,from=edx-platform,source=/openedx/features/course_search/static,target=openedx/features/course_search/static \ - --mount=type=bind,from=edx-platform,source=/openedx/features/learner_profile/static,target=openedx/features/learner_profile/static \ - --mount=type=bind,from=edx-platform,source=/xmodule/assets,target=xmodule/assets \ - --mount=type=bind,from=edx-platform,source=/xmodule/js,target=xmodule/js \ - npm run webpack +FROM frontend-requirements as bundles -########################################################################################################################### -############## BUNDLES: DEVELOPMENT -########################################################################################################################### -FROM requirements-frontend as bundles-development - -# Run webpack -RUN --mount=type=bind,from=edx-platform,source=/package.json,target=package.json \ - --mount=type=bind,from=edx-platform,source=/package-lock.json,target=/openedx/edx-platform/package-lock.json \ - --mount=type=bind,from=edx-platform,source=/.babelrc,target=.babelrc \ - --mount=type=bind,from=edx-platform,source=/webpack.common.config.js,target=webpack.common.config.js \ - --mount=type=bind,from=edx-platform,source=/webpack-config/file-lists.js,target=webpack-config/file-lists.js \ - --mount=type=bind,from=edx-platform,source=/webpack.dev.config.js,target=webpack.dev.config.js \ - --mount=type=bind,from=edx-platform,source=/webpack.prod.config.js,target=webpack.prod.config.js \ - --mount=type=bind,from=edx-platform,source=/webpack.builtinblocks.config.js,target=webpack.builtinblocks.config.js \ - --mount=type=bind,from=edx-platform,source=/cms/djangoapps/pipeline_js/js/xmodule.js,target=cms/djangoapps/pipeline_js/js/xmodule.js \ - --mount=type=bind,from=edx-platform,source=/cms/static,target=cms/static \ - --mount=type=bind,from=edx-platform,source=/cms/templates/,target=cms/templates/ \ - --mount=type=bind,from=edx-platform,source=/common/static/js/,target=common/static/js \ - --mount=type=bind,from=edx-platform,source=/common/static/common,target=common/static/common,rw \ - --mount=type=bind,from=requirements-frontend,source=/openedx/edx-platform/common/static/common/js/vendor,target=common/static/common/js/vendor \ - --mount=type=bind,from=requirements-frontend,source=/openedx/edx-platform/common/static/common/css/vendor,target=common/static/common/css/vendor \ - --mount=type=bind,from=edx-platform,source=/lms/djangoapps/discussion/static,target=lms/djangoapps/discussion/static \ - --mount=type=bind,from=edx-platform,source=/lms/djangoapps/instructor/static,target=lms/djangoapps/instructor/static \ - --mount=type=bind,from=edx-platform,source=/lms/djangoapps/support/static/support,target=lms/djangoapps/support/static/support \ - --mount=type=bind,from=edx-platform,source=/lms/djangoapps/teams/static,target=lms/djangoapps/teams/static \ - --mount=type=bind,from=edx-platform,source=/lms/static/,target=lms/static/ \ - --mount=type=bind,from=edx-platform,source=/lms/templates/,target=lms/templates/ \ - --mount=type=bind,from=edx-platform,source=/openedx/features/announcements/static,target=openedx/features/announcements/static \ - --mount=type=bind,from=edx-platform,source=/openedx/features/course_bookmarks/static,target=openedx/features/course_bookmarks/static \ - --mount=type=bind,from=edx-platform,source=/openedx/features/course_experience/static,target=openedx/features/course_experience/static \ - --mount=type=bind,from=edx-platform,source=/openedx/features/course_search/static,target=openedx/features/course_search/static \ - --mount=type=bind,from=edx-platform,source=/openedx/features/learner_profile/static,target=openedx/features/learner_profile/static \ - --mount=type=bind,from=edx-platform,source=/xmodule/assets,target=xmodule/assets \ - --mount=type=bind,from=edx-platform,source=/xmodule/js,target=xmodule/js \ - npm run webpack-dev +COPY --link --from=edx-platform /.babelrc .babelrc +COPY --link --from=edx-platform /webpack.common.config.js webpack.common.config.js +COPY --link --from=edx-platform /webpack-config/file-lists.js webpack-config/file-lists.js +COPY --link --from=edx-platform /webpack.dev.config.js webpack.dev.config.js +COPY --link --from=edx-platform /webpack.prod.config.js webpack.prod.config.js +COPY --link --from=edx-platform /webpack.builtinblocks.config.js webpack.builtinblocks.config.js +COPY --link --from=edx-platform /cms/djangoapps/pipeline_js/js/xmodule.js cms/djangoapps/pipeline_js/js/xmodule.js +COPY --link --from=edx-platform /cms/static cms/static +COPY --link --from=edx-platform /cms/templates/ cms/templates/ +COPY --link --from=edx-platform /common/static/js/ common/static/js +COPY --link --from=edx-platform /common/static/common common/static/common +COPY --link --from=edx-platform /lms/djangoapps/discussion/static lms/djangoapps/discussion/static +COPY --link --from=edx-platform /lms/djangoapps/instructor/static lms/djangoapps/instructor/static +COPY --link --from=edx-platform /lms/djangoapps/support/static/support lms/djangoapps/support/static/support +COPY --link --from=edx-platform /lms/djangoapps/teams/static lms/djangoapps/teams/static +COPY --link --from=edx-platform /lms/static/ lms/static/ +COPY --link --from=edx-platform /lms/templates/ lms/templates/ +COPY --link --from=edx-platform /openedx/features/announcements/static openedx/features/announcements/static +COPY --link --from=edx-platform /openedx/features/course_bookmarks/static openedx/features/course_bookmarks/static +COPY --link --from=edx-platform /openedx/features/course_experience/static openedx/features/course_experience/static +COPY --link --from=edx-platform /openedx/features/course_search/static openedx/features/course_search/static +COPY --link --from=edx-platform /openedx/features/learner_profile/static openedx/features/learner_profile/static +COPY --link --from=edx-platform /xmodule/assets xmodule/assets +COPY --link --from=edx-platform /xmodule/js xmodule/js +COPY --link --from=frontend-requirements /openedx/edx-platform/common/static/common/js/vendor common/static/common/js/vendor +COPY --link --from=frontend-requirements /openedx/edx-platform/common/static/common/css/vendor common/static/common/css/vendor -########################################################################################################################### -############## CSS: PRODUCTION -########################################################################################################################### -FROM requirements-frontend as css-production +FROM bundles as bundles-production +RUN npm run webpack -ENV PATH ./node_modules/.bin:${PATH} - -# Compile default CSS -RUN --mount=type=bind,from=edx-platform,source=/package.json,target=package.json \ - --mount=type=bind,from=edx-platform,source=/scripts/compile_sass.py,target=scripts/compile_sass.py \ - --mount=type=bind,from=edx-platform,source=/common/static,target=common/static \ - --mount=type=bind,from=edx-platform,source=/lms/static/sass,target=lms/static/sass \ - --mount=type=bind,from=edx-platform,source=/lms/static/sass/partials,target=lms/static/sass/partials \ - --mount=type=bind,from=edx-platform,source=/lms/static/certificates/sass,target=lms/static/certificates/sass \ - --mount=type=bind,from=edx-platform,source=/cms/static/sass,target=cms/static/sass \ - --mount=type=bind,from=edx-platform,source=/cms/static/sass/partials,target=cms/static/sass/partials \ - --mount=type=bind,from=edx-platform,source=/xmodule/assets,target=xmodule/assets \ - npm run compile-sass -- --skip-themes - -# Compile themed CSS -ENV EDX_PLATFORM_THEME_DIRS=/openedx/themes -COPY --link --chown=app:app ./themes/ /openedx/themes/ -RUN --mount=type=bind,from=edx-platform,source=/package.json,target=package.json \ - --mount=type=bind,from=edx-platform,source=/scripts/compile_sass.py,target=scripts/compile_sass.py \ - --mount=type=bind,from=edx-platform,source=/common/static,target=common/static \ - --mount=type=bind,from=edx-platform,source=/lms/static/sass,target=lms/static/sass \ - --mount=type=bind,from=edx-platform,source=/lms/static/sass/partials,target=lms/static/sass/partials \ - --mount=type=bind,from=edx-platform,source=/lms/static/certificates/sass,target=lms/static/certificates/sass \ - --mount=type=bind,from=edx-platform,source=/cms/static/sass,target=cms/static/sass \ - --mount=type=bind,from=edx-platform,source=/cms/static/sass/partials,target=cms/static/sass/partials \ - --mount=type=bind,from=edx-platform,source=/xmodule/assets,target=xmodule/assets \ - npm run compile-sass -- --skip-default +FROM bundles as bundles-development +RUN npm run webpack-dev ########################################################################################################################### -############## CSS: DEVELOPMENT +############## CSS ########################################################################################################################### -FROM requirements-frontend as css-development + +FROM frontend-requirements as css ENV PATH ./node_modules/.bin:${PATH} +COPY --link --from=edx-platform /scripts/compile_sass.py scripts/compile_sass.py +COPY --link --from=edx-platform /common/static common/static +COPY --link --from=edx-platform /lms/static/sass lms/static/sass +COPY --link --from=edx-platform /lms/static/sass/partials lms/static/sass/partials +COPY --link --from=edx-platform /lms/static/certificates/sass lms/static/certificates/sass +COPY --link --from=edx-platform /cms/static/sass cms/static/sass +COPY --link --from=edx-platform /cms/static/sass/partials cms/static/sass/partials +COPY --link --from=edx-platform /xmodule/assets xmodule/assets COPY --link --from=edx-platform --chown=app:app /lms/static/css/vendor lms/static/css/vendor +FROM css as css-production # Compile default CSS -RUN --mount=type=bind,from=edx-platform,source=/package.json,target=package.json \ - --mount=type=bind,from=edx-platform,source=/scripts/compile_sass.py,target=scripts/compile_sass.py \ - --mount=type=bind,from=edx-platform,source=/common/static,target=common/static \ - --mount=type=bind,from=edx-platform,source=/lms/static/sass,target=lms/static/sass \ - --mount=type=bind,from=edx-platform,source=/lms/static/sass/partials,target=lms/static/sass/partials \ - --mount=type=bind,from=edx-platform,source=/lms/static/certificates/sass,target=lms/static/certificates/sass \ - --mount=type=bind,from=edx-platform,source=/cms/static/sass,target=cms/static/sass \ - --mount=type=bind,from=edx-platform,source=/cms/static/sass/partials,target=cms/static/sass/partials \ - --mount=type=bind,from=edx-platform,source=/xmodule/assets,target=xmodule/assets \ - npm run compile-sass -- --skip-themes --env=dev +RUN npm run compile-sass -- --skip-themes +# Compile themed CSS +ENV EDX_PLATFORM_THEME_DIRS=/openedx/themes +COPY --link --chown=app:app ./themes/ /openedx/themes/ +RUN npm run compile-sass -- --skip-default +FROM css as css-development +# Compile default CSS +RUN npm run compile-sass -- --skip-themes --env=dev # Compile themed CSS ENV EDX_PLATFORM_THEME_DIRS=/openedx/themes COPY --link --chown=app:app ./themes/ /openedx/themes/ -RUN --mount=type=bind,from=edx-platform,source=/package.json,target=package.json \ - --mount=type=bind,from=edx-platform,source=/scripts/compile_sass.py,target=scripts/compile_sass.py \ - --mount=type=bind,from=edx-platform,source=/common/static,target=common/static \ - --mount=type=bind,from=edx-platform,source=/lms/static/sass,target=lms/static/sass \ - --mount=type=bind,from=edx-platform,source=/lms/static/sass/partials,target=lms/static/sass/partials \ - --mount=type=bind,from=edx-platform,source=/lms/static/certificates/sass,target=lms/static/certificates/sass \ - --mount=type=bind,from=edx-platform,source=/cms/static/sass,target=cms/static/sass \ - --mount=type=bind,from=edx-platform,source=/cms/static/sass/partials,target=cms/static/sass/partials \ - --mount=type=bind,from=edx-platform,source=/xmodule/assets,target=xmodule/assets \ - npm run compile-sass -- --skip-default --env=dev +RUN npm run compile-sass -- --skip-default --env=dev ########################################################################################################################### -############## APPLICATION REQUIREMENTS +############## REQUIREMENTS (both frontend + backend) ########################################################################################################################### -FROM requirements-frontend as requirements-application +FROM frontend-requirements as requirements # Install base requirements -RUN --mount=type=bind,from=edx-platform,source=/requirements/edx/base.txt,target=/openedx/edx-platform/requirements/edx/base.txt \ - --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ +COPY --link --from=edx-platform /requirements/edx/base.txt /openedx/edx-platform/requirements/edx/base.txt +RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ pip install -r /openedx/edx-platform/requirements/edx/base.txt # Install extra requirements @@ -317,33 +249,24 @@ RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ {% for extra_requirements in OPENEDX_EXTRA_PIP_REQUIREMENTS %}RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared pip install '{{ extra_requirements }}' {% endfor %} - -########################################################################################################################### -############## PRODUCTION REQUIREMENTS -############## Simply take the general requirements, and remove everything related to libsass. -############## Alternatively, we could have based requirements-application on minimal, thus keeping libsass out of it, -############## BUT that would have required requirements-development to recompile libsass, which is slow. -########################################################################################################################### -FROM requirements-application as requirements-production - +FROM requirements as requirements-production +# To make prod requirements, we simply take the general requirements, and remove everything related to +# libsass (which is installed via assets.txt) in order to shrink the final image. We don't need to +# compile Sass in production. +# Now, what we *could* have done was base this stage on minimal, thus avoiding assets.txt altogether. +# However, this would mean that we'd have to install libsass *twice*: once for frontend-requirements and +# once for requirements. Since libsass takes so long to compile, that doesn't seem worth it. RUN rm -r /openedx/venv/lib/python3.8/site-packages/*sass* - -########################################################################################################################### -############## DEVELOPMENT REQUIREMENTS -########################################################################################################################### -FROM requirements-application as requirements-development - -# Install dev requirements -RUN --mount=type=bind,from=edx-platform,source=/requirements/edx/development.txt,target=/openedx/edx-platform/requirements/edx/development.txt \ - --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ +FROM requirements as requirements-development +# Install dev requirements. +COPY --link --from=edx-platform /requirements/edx/development.txt /openedx/edx-platform/requirements/edx/development.txt +RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ pip install -r /openedx/edx-platform/requirements/edx/development.txt ########################################################################################################################### -############## APPLICATION -############## Application image with system and python requirements. -############## Basis for final prod and dev images. +############## APPLICATION (base for final production and development images) ########################################################################################################################### FROM minimal as application @@ -395,16 +318,16 @@ RUN cd /openedx/locale/user && \ django-admin compilemessages -v1 # Copy in frontend requirements. -COPY --link --chown=app:app --from=requirements-frontend \ +COPY --link --chown=app:app --from=frontend-requirements \ /openedx/nodeenv \ /openedx/nodeenv -COPY --link --chown=app:app --from=requirements-frontend \ +COPY --link --chown=app:app --from=frontend-requirements \ /openedx/edx-platform/node_modules \ /openedx/edx-platform/node_modules -COPY --link --chown=app:app --from=requirements-frontend \ +COPY --link --chown=app:app --from=frontend-requirements \ /openedx/edx-platform/common/static/common/js/vendor \ /openedx/edx-platform/common/static/common/js/vendor -COPY --link --chown=app:app --from=requirements-frontend \ +COPY --link --chown=app:app --from=frontend-requirements \ /openedx/edx-platform/common/static/common/css/vendor \ /openedx/edx-platform/common/static/common/css/vendor From bb59612d0f0e60afe749bda86a060817744b7f76 Mon Sep 17 00:00:00 2001 From: "Kyle D. McCormick" Date: Thu, 31 Aug 2023 16:17:08 -0400 Subject: [PATCH 14/14] feat: `tutor dev/local copyartifacts` TODO: * Changelog entry * Comments * Linting * Tests * TEP --- tutor/commands/compose.py | 101 ++++++++++++++++++ tutor/commands/jobs.py | 9 -- tutor/hooks/catalog.py | 26 +++++ .../jobs/init/mounted-edx-platform.sh | 26 ----- 4 files changed, 127 insertions(+), 35 deletions(-) delete mode 100644 tutor/templates/jobs/init/mounted-edx-platform.sh diff --git a/tutor/commands/compose.py b/tutor/commands/compose.py index 0f19e3a5e3d..76aab736641 100644 --- a/tutor/commands/compose.py +++ b/tutor/commands/compose.py @@ -20,6 +20,7 @@ from tutor.exceptions import TutorError from tutor.tasks import BaseComposeTaskRunner from tutor.types import Config +from tutor.utils import execute as execute_shell class ComposeTaskRunner(BaseComposeTaskRunner): @@ -114,6 +115,10 @@ def launch( click.echo(fmt.title("Docker image updates")) context.invoke(dc_command, command="pull") + if bindmount.get_mounts(config): + click.echo(fmt.title("Copying build artifacts into bind-mounted directories")) + context.invoke(copyartifacts) + click.echo(fmt.title("Starting the platform in detached mode")) context.invoke(start, detach=True) @@ -365,6 +370,80 @@ def copyfrom( ) +@click.command( + help="TODO describe" +) +@click.argument( + "mount_paths", + metavar="mount_path", + nargs=-1, + type=click.Path(dir_okay=True, file_okay=False, resolve_path=True), +) +@click.pass_obj +def copyartifacts(context: BaseComposeContext, mount_paths: list[Path]) -> None: + """ + TODO write docstring + """ + config = tutor_config.load(context.root) + host_mount_paths: list[str] = [ + os.path.abspath(os.path.expanduser(mount_path)) + for mount_path + in mount_paths or bindmount.get_mounts(config) + ] + + # Sort out the (source, target) pairs by service name so that we can + # work one-at-a-time later. + copies_by_service: dict[str, tuple[str, str]] = {} + container_mounts_by_service: dict[str, list[str]] = {} + for host_mount_path in host_mount_paths: + mount_name = os.path.basename(host_mount_path) + for service, container_mount_path in hooks.Filters.COMPOSE_MOUNTS.iterate(mount_name): + for path_in_mount in hooks.Filters.COMPOSE_MOUNT_ARTIFACTS.iterate(mount_name, service): + source = f"{container_mount_path}/{path_in_mount}" + target = f"{host_mount_path}/{path_in_mount}" + copies_by_service.setdefault(service, []).append((source, target)) + container_mounts_by_service.setdefault(service, []).append(container_mount_path) + + container_name = "tutor_mounts_populate_temp" # TODO: improve name? + runner = context.job_runner(config) + + # For each service: create a temporary container, do the copy operations, and then kill the container. + for service, copies in copies_by_service.items(): + execute_shell("docker", "rm", "-f", container_name) + runner.docker_compose( + "run", + "--rm", + "--no-deps", + "--user=0", + # Recall that these artifact source directories may actually be bind-mounted into + # into the service from the host, which would prevent us from copying the original + # image's artifacts! + # To work around this, we shadow each relevant bind-mount with a fresh anonymous volume, + # which Docker populates with the image's original contents. + # The volume creation takes a bit of time, so we avoid using this shadowing strategy + # on any bind-mounts that aren't relevant to the operation. + *( + f"--volume={container_mount_path}" + for container_mount_path in set(container_mounts_by_service[service]) + ), + # Give the container a predetermined name so we can refer to it later. + "--name", + container_name, + # Run in the backround. + "--detach", + service, + # Rather than starting the service's real command, save some CPU by just sleeping + # until killed. + "sleep", + "infinity", + ) + for source, target in copies: + execute_shell("rm", "-rf", target) # Wipe any existing artifact. + execute_shell("sh", "-c", f'mkdir -p "$(dirname "{target}")"') # Ensure parent dirs exist. + execute_shell("docker", "cp", f"{container_name}:{source}", target) # Actually do the copy. + execute_shell("sh", "-c", f"docker kill '{container_name}' || true") + + @click.command( short_help="Run a command in a running container", help=( @@ -447,6 +526,27 @@ def _mount_edx_platform( return volumes +@hooks.Filters.COMPOSE_MOUNT_ARTIFACTS.add() +def _populate_edx_platform( + paths_to_copy: list[str], mount_name: str, service: str +) -> list[tuple[str, str]]: + """ + TODO describe + """ + if mount_name == "edx-platform" and service == "lms": + paths_to_copy += [ + "Open_edX.egg-info", + "node_modules", + "lms/static/css", + "lms/static/certificates/css", + "cms/static/css", + "common/static/bundles", + "common/static/common/js/vendor", + "common/static/common/css/vendor", + ] + return paths_to_copy + + @hooks.Filters.APP_PUBLIC_HOSTS.add() def _edx_platform_public_hosts( hosts: list[str], context_name: t.Literal["local", "dev"] @@ -471,6 +571,7 @@ def add_commands(command_group: click.Group) -> None: command_group.add_command(dc_command) command_group.add_command(run) command_group.add_command(copyfrom) + command_group.add_command(copyartifacts) command_group.add_command(execute) command_group.add_command(logs) command_group.add_command(status) diff --git a/tutor/commands/jobs.py b/tutor/commands/jobs.py index 562e78d4549..f31cef08a44 100644 --- a/tutor/commands/jobs.py +++ b/tutor/commands/jobs.py @@ -42,15 +42,6 @@ def _add_core_init_tasks() -> None: ("mysql", env.read_core_template_file("jobs", "init", "mysql.sh")) ) with hooks.Contexts.app("lms").enter(): - hooks.Filters.CLI_DO_INIT_TASKS.add_item( - ( - "lms", - env.read_core_template_file("jobs", "init", "mounted-edx-platform.sh"), - ), - # If edx-platform is mounted, then we may need to perform some setup - # before other initialization scripts can be run. - priority=priorities.HIGH, - ) hooks.Filters.CLI_DO_INIT_TASKS.add_item( ("lms", env.read_core_template_file("jobs", "init", "lms.sh")) ) diff --git a/tutor/hooks/catalog.py b/tutor/hooks/catalog.py index dde0f4a4cf5..7ceaa616e15 100644 --- a/tutor/hooks/catalog.py +++ b/tutor/hooks/catalog.py @@ -222,6 +222,32 @@ def your_filter_callback(some_data): #: conditionally add mounts. COMPOSE_MOUNTS: Filter[list[tuple[str, str]], [str]] = Filter() + #: Relative paths of build artifacts, to be copied into a host-mounted folder from a service image. + #: + #: Docker images contain many build artifacts, such as generated assets and packaging metadata, which + #: must exist at runtime in order for services to run properly. When a user bind-mounts a directory + #: from their host machine, there is no guarantee that the host directory will contain those essential + #: artifacts, and regenerating them from source may be time consuming. To remedy this, Tutor provides + #: the ``copyartifacts`` command, which efficiently copies the necessary artifacts from the original image + #: into the host directory. This command is also automatically run as part of ``launch``. + #: + #: The ``COMPOSE_MOUNT_ARTIFACTS`` filter tells Tutor which artifacts must be copied from which + #: service for any given host-mounted directory. By default, for edx-platform, this includes + #: several directories such as ``node_modules`` and ``lms/static/css``, to be copied from the lms + #: service's image. + #: + #: Note that any given artifact should only be specified once in this filter. If an artifact + #: exists on an image used by multiple services, choose one of those service for it to be + #: copied from. + #: + #: :parameter artifacts list[str]: files or directories considered build artifacts for the given + #: host-mounted folder in the given service. Paths must be relative to the mount directory. + #: :parameter str mount_name: basename of a host-mounted folder. + #: :parameter service: name of a service from which artifackts will be copied. The ``COMPOSE_MOUNTS`` + #: filter should map ``mount_name`` somewhere in this container; that mount location will be used + #: as the source for ``artifacts``. + COMPOSE_MOUNT_ARTIFACTS: Filter[list[str], [str, str]] = Filter() + #: Declare new default configuration settings that don't necessarily have to be saved in the user #: ``config.yml`` file. Default settings may be overridden with ``tutor config save --set=...``, in which #: case they will automatically be added to ``config.yml``. diff --git a/tutor/templates/jobs/init/mounted-edx-platform.sh b/tutor/templates/jobs/init/mounted-edx-platform.sh deleted file mode 100644 index 9516654ff58..00000000000 --- a/tutor/templates/jobs/init/mounted-edx-platform.sh +++ /dev/null @@ -1,26 +0,0 @@ -# When a new local copy of edx-platform is bind-mounted, certain build -# artifacts from the openedx image's edx-platform directory are lost. -# We regenerate them here. - -if [ -f /openedx/edx-platform/bindmount-canary ] ; then - # If this file exists, then edx-platform has not been bind-mounted, - # so no build artifacts need to be regenerated. - echo "Using edx-platform from image (not bind-mount)." - echo "No extra setup is required." - exit -fi - -echo "Performing additional setup for bind-mounted edx-platform." -set -x # Echo out executed lines - -# Regenerate Open_edX.egg-info -pip install -e . - -# Regenerate node_modules -npm clean-install - -# Regenerate static assets. -openedx-assets build --env=dev - -set -x -echo "Done setting up bind-mounted edx-platform."