From aa34c23a28998b6de6965ae148138e3066ee444c Mon Sep 17 00:00:00 2001 From: Feynman Zhou Date: Tue, 10 Jan 2023 11:58:38 +0800 Subject: [PATCH 01/37] doc: add ORAS CLI proposal doc Signed-off-by: Feynman Zhou --- docs/proposals/compatibility-mode.md | 61 ++++++++++++++++++++++++++++ 1 file changed, 61 insertions(+) create mode 100644 docs/proposals/compatibility-mode.md diff --git a/docs/proposals/compatibility-mode.md b/docs/proposals/compatibility-mode.md new file mode 100644 index 000000000..be63cf26b --- /dev/null +++ b/docs/proposals/compatibility-mode.md @@ -0,0 +1,61 @@ +# Proposal: Provide a compatibility mode for ORAS + +This document is for adding a proposal to enable compatibility mode for ORAS CLI and solve [issue #720](https://github.com/oras-project/oras/issues/720). + +## Background + +OCI group announced the release of v1.1 for [Image-spec](https://github.com/opencontainers/image-spec/blob/main/artifact.md) and [Distribution-spec](https://github.com/opencontainers/distribution-spec) in Sep 2022. It supports the OCI artifact manifest and provides a new referrers discovery API that allows artifact references. + +Since 0.16.0, ORAS supports pushing OCI artifact manifest to OCI v1.0 compliant registries. However, the new manifest type may not be supported on the consumer side (e.g. self-crafted scripts) or in those OCI v1.0 compliant registries. To enable ORAS to work with popular registries, it provides backward compatibility which supports two types of manifest and allows fallback to upload the OCI image manifest to OCI v1.0 compliant registries or enabled OCI image manifest storage. + +## Challenge + +However, the ORAS fallback may fail as there is no deterministic way to confirm if a registry supporting OCI artifact manifest and no consistent error response. On the other hand, users may want to force-push a specific manifest type to a registry for some reason. + +![This table is captured from [https://toddysm.com/2023/01/05/oci-artifct-manifests-oci-referrers-api-and-their-support-across-registries-part-1/](https://toddysm.com/2023/01/05/oci-artifct-manifests-oci-referrers-api-and-their-support-across-registries-part-1/)](Proposal%20Provide%20Compatibility%20mode%20for%20ORAS%20a1e29245ea014b10a9711bae0d04db3c/Untitled.png) + +This table is captured from [https://toddysm.com/2023/01/05/oci-artifct-manifests-oci-referrers-api-and-their-support-across-registries-part-1/](https://toddysm.com/2023/01/05/oci-artifct-manifests-oci-referrers-api-and-their-support-across-registries-part-1/) + +The current workaround for enabling the compatibility mode is to specify a `--config` flag when using `oras push`. Since the OCI Artifact Manifest does not have a `config`, it will push an OCI image manifest instead. It is not user-friendly and is a bit hacky. It would be better if we can provide a compatibility mode for switching the manifest uploading behavior. + +## Goals + +- Enable ORAS to work with more registries +- Provide different options to allow users to customize the behaviors of uploading manifest to the registry +- Provide an easy-to-use and secure user experience when switching the behaviors + +## Solution + +Adding a new flag `--compatibility` under CLI commands `oras push` and `oras attach` with different variables to configure the behaviors of uploading manifest. We will only use `oras push` to demonstrate the examples below. + +### Use case A + +If users want to force-push the OCI Artifact Manifest to registries whether they are compliant with OCI v1.0 or v1.1, using `--compatibility artifact-manifest` will only upload OCI Image Manifest to registries. Users might choose it for security requirements, such as pushing a signature to a registry without changing its digest. + +```bash +oras push localhost:5000/hello-artifact:v1 --artifact-type sbom/example --compatibility artifact-manifest sbom.json +``` + +### Use case B + +If users want to force-push the OCI Image Manifest to registries whether they are compliant with OCI v1.0 or v1.1, using `--compatibility image-manifest` will only upload OCI Image Manifest to registries. This option is helpful when users have concerns to use OCI Artifact Manifest or migrate content to OCI v1.0 compliant registries. + +```bash +oras push localhost:5000/hello-artifact:v1 --artifact-type sbom/example --compatibility image-manifest sbom.json +``` + +### Use case C + +Disable backward compatibility and only upload OCI Artifact Manifest to a registry. This flag `--compatibility min` will be commonly used with the OCI v1.1 compliant registries or registries that support storing OCI Artifact Manifest, such as Zot, Azure Container Registry, and Docker Hub. This is the most strict option for the behavior of uploading manifest. Users might choose it for security requirements. + +```bash +oras push localhost:5000/hello-artifact:v1 --artifact-type sbom/example --compatibility min sbom.json +``` + +### Use case D + +If users want to upload manifest to OCI v1.0 compliant registries like Harbor, GitHub Container Registry, etc, using `--compatibility max` will upload OCI Image Manifest first and Referrals tag schema. This option enables maximum backward compatibility for registries. If it uploads fails, it will try to upload OCI Artifact Manifest. If it still uploads fails, which means ORAS is not compatible with the registry. + +```bash +oras push localhost:5000/hello-artifact:v1 --artifact-type sbom/example --compatibility max sbom.json +``` \ No newline at end of file From 5e058388cac34857ced90f01740931ace333d94a Mon Sep 17 00:00:00 2001 From: Feynman Zhou Date: Tue, 10 Jan 2023 12:13:35 +0800 Subject: [PATCH 02/37] doc: add ORAS CLI proposal doc Signed-off-by: Feynman Zhou --- docs/proposals/compatibility-mode.md | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/docs/proposals/compatibility-mode.md b/docs/proposals/compatibility-mode.md index be63cf26b..26a02a88c 100644 --- a/docs/proposals/compatibility-mode.md +++ b/docs/proposals/compatibility-mode.md @@ -10,13 +10,10 @@ Since 0.16.0, ORAS supports pushing OCI artifact manifest to OCI v1.0 compliant ## Challenge -However, the ORAS fallback may fail as there is no deterministic way to confirm if a registry supporting OCI artifact manifest and no consistent error response. On the other hand, users may want to force-push a specific manifest type to a registry for some reason. +However, the ORAS fallback may fail as there is no deterministic way to confirm if a registry supporting OCI artifact manifest and no consistent error response. +You can find the testing result of the implementation result for OCI Spec from this [blog](https://toddysm.com/2023/01/05/oci-artifct-manifests-oci-referrers-api-and-their-support-across-registries-part-1/). On the other hand, users may want to force-push a specific manifest type to a registry for some reason. -![This table is captured from [https://toddysm.com/2023/01/05/oci-artifct-manifests-oci-referrers-api-and-their-support-across-registries-part-1/](https://toddysm.com/2023/01/05/oci-artifct-manifests-oci-referrers-api-and-their-support-across-registries-part-1/)](Proposal%20Provide%20Compatibility%20mode%20for%20ORAS%20a1e29245ea014b10a9711bae0d04db3c/Untitled.png) - -This table is captured from [https://toddysm.com/2023/01/05/oci-artifct-manifests-oci-referrers-api-and-their-support-across-registries-part-1/](https://toddysm.com/2023/01/05/oci-artifct-manifests-oci-referrers-api-and-their-support-across-registries-part-1/) - -The current workaround for enabling the compatibility mode is to specify a `--config` flag when using `oras push`. Since the OCI Artifact Manifest does not have a `config`, it will push an OCI image manifest instead. It is not user-friendly and is a bit hacky. It would be better if we can provide a compatibility mode for switching the manifest uploading behavior. +The current workaround for enabling a kind of compatibility mode is to specify a `--config` flag when using `oras push`. Since the OCI Artifact Manifest does not have a `config`, it will push an OCI image manifest instead. It is not user-friendly and is a bit hacky. It would be better if we can provide a compatibility mode to easily customize and switch the manifest uploading behavior, and enable users to handle the incompatibility problem when using ORAS across different registries. ## Goals From cce1f66b1474c4e489ec1be560faf18242367888 Mon Sep 17 00:00:00 2001 From: Feynman Zhou Date: Tue, 10 Jan 2023 18:30:07 +0800 Subject: [PATCH 03/37] doc: update the compatibility mode proposal Signed-off-by: Feynman Zhou --- docs/proposals/compatibility-mode.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/proposals/compatibility-mode.md b/docs/proposals/compatibility-mode.md index 26a02a88c..ba4bd5069 100644 --- a/docs/proposals/compatibility-mode.md +++ b/docs/proposals/compatibility-mode.md @@ -13,7 +13,7 @@ Since 0.16.0, ORAS supports pushing OCI artifact manifest to OCI v1.0 compliant However, the ORAS fallback may fail as there is no deterministic way to confirm if a registry supporting OCI artifact manifest and no consistent error response. You can find the testing result of the implementation result for OCI Spec from this [blog](https://toddysm.com/2023/01/05/oci-artifct-manifests-oci-referrers-api-and-their-support-across-registries-part-1/). On the other hand, users may want to force-push a specific manifest type to a registry for some reason. -The current workaround for enabling a kind of compatibility mode is to specify a `--config` flag when using `oras push`. Since the OCI Artifact Manifest does not have a `config`, it will push an OCI image manifest instead. It is not user-friendly and is a bit hacky. It would be better if we can provide a compatibility mode to easily customize and switch the manifest uploading behavior, and enable users to handle the incompatibility problem when using ORAS across different registries. +The current workaround for enabling a kind of compatibility mode is to specify a `--config` flag when using `oras push`. Since the OCI artifact manifest does not have a `config`, it will push an OCI image manifest instead. It is not user-friendly and is a bit hacky. It would be better if we can provide a compatibility mode to easily customize and switch the manifest uploading behavior, and enable users to handle the incompatibility problem when using ORAS across different registries. ## Goals @@ -27,7 +27,7 @@ Adding a new flag `--compatibility` under CLI commands `oras push` and `oras att ### Use case A -If users want to force-push the OCI Artifact Manifest to registries whether they are compliant with OCI v1.0 or v1.1, using `--compatibility artifact-manifest` will only upload OCI Image Manifest to registries. Users might choose it for security requirements, such as pushing a signature to a registry without changing its digest. +If users want to force-push the OCI artifact manifest to registries whether they are compliant with OCI v1.0 or v1.1, using `--compatibility artifact-manifest` will only upload OCI image manifest to registries. Users might choose it for security requirements, such as pushing a signature to a registry without changing its digest. ```bash oras push localhost:5000/hello-artifact:v1 --artifact-type sbom/example --compatibility artifact-manifest sbom.json @@ -35,7 +35,7 @@ oras push localhost:5000/hello-artifact:v1 --artifact-type sbom/example --compat ### Use case B -If users want to force-push the OCI Image Manifest to registries whether they are compliant with OCI v1.0 or v1.1, using `--compatibility image-manifest` will only upload OCI Image Manifest to registries. This option is helpful when users have concerns to use OCI Artifact Manifest or migrate content to OCI v1.0 compliant registries. +If users want to force-push the OCI image manifest to registries whether they are compliant with OCI v1.0 or v1.1, using `--compatibility image-manifest` will only upload OCI image manifest to registries. This option is helpful when users have concerns to use OCI artifact manifest or migrate content to OCI v1.0 compliant registries. ```bash oras push localhost:5000/hello-artifact:v1 --artifact-type sbom/example --compatibility image-manifest sbom.json @@ -43,7 +43,7 @@ oras push localhost:5000/hello-artifact:v1 --artifact-type sbom/example --compat ### Use case C -Disable backward compatibility and only upload OCI Artifact Manifest to a registry. This flag `--compatibility min` will be commonly used with the OCI v1.1 compliant registries or registries that support storing OCI Artifact Manifest, such as Zot, Azure Container Registry, and Docker Hub. This is the most strict option for the behavior of uploading manifest. Users might choose it for security requirements. +Disable backward compatibility and only upload OCI artifact manifest to a registry. This flag `--compatibility min` will be commonly used with the OCI v1.1 compliant registries or registries that support storing OCI artifact manifest, such as Zot, Azure Container Registry, and Docker Hub. ORAS will push OCI artifact manifest only and no further actions for maintaining artifact references in registries. This is the most strict option for the behavior of uploading manifest. Users might choose it for security requirements. ```bash oras push localhost:5000/hello-artifact:v1 --artifact-type sbom/example --compatibility min sbom.json @@ -51,8 +51,10 @@ oras push localhost:5000/hello-artifact:v1 --artifact-type sbom/example --compat ### Use case D -If users want to upload manifest to OCI v1.0 compliant registries like Harbor, GitHub Container Registry, etc, using `--compatibility max` will upload OCI Image Manifest first and Referrals tag schema. This option enables maximum backward compatibility for registries. If it uploads fails, it will try to upload OCI Artifact Manifest. If it still uploads fails, which means ORAS is not compatible with the registry. +If users want to upload manifest to OCI v1.0 compliant registries like Harbor, GitHub Container Registry, etc, using `--compatibility max` will push OCI image manifest and then push Referrals tag schema if the `subject` field of the OCI image manifest is not empty. This option enables maximum backward compatibility for registries. It allows `oras push` or `oras attach` to work with the OCI v1.0 compliant registries even though the registries return non-404 response code. ```bash oras push localhost:5000/hello-artifact:v1 --artifact-type sbom/example --compatibility max sbom.json -``` \ No newline at end of file +``` + + From 0066516df1ba38f3e904fea292d24d3549fea816 Mon Sep 17 00:00:00 2001 From: Feynman Zhou Date: Sat, 28 Jan 2023 12:36:13 +0800 Subject: [PATCH 04/37] update compatibility doc Signed-off-by: Feynman Zhou --- docs/proposals/compatibility-mode.md | 74 +++++++++++++++++++--------- 1 file changed, 52 insertions(+), 22 deletions(-) diff --git a/docs/proposals/compatibility-mode.md b/docs/proposals/compatibility-mode.md index ba4bd5069..74784c1e6 100644 --- a/docs/proposals/compatibility-mode.md +++ b/docs/proposals/compatibility-mode.md @@ -1,60 +1,90 @@ -# Proposal: Provide a compatibility mode for ORAS +# Compatibility mode for ORAS -This document is for adding a proposal to enable compatibility mode for ORAS CLI and solve [issue #720](https://github.com/oras-project/oras/issues/720). +This document is for adding a document to elaborate the compatibility mode for ORAS CLI which was prosed in [issue #720](https://github.com/oras-project/oras/issues/720). ## Background OCI group announced the release of v1.1 for [Image-spec](https://github.com/opencontainers/image-spec/blob/main/artifact.md) and [Distribution-spec](https://github.com/opencontainers/distribution-spec) in Sep 2022. It supports the OCI artifact manifest and provides a new referrers discovery API that allows artifact references. -Since 0.16.0, ORAS supports pushing OCI artifact manifest to OCI v1.0 compliant registries. However, the new manifest type may not be supported on the consumer side (e.g. self-crafted scripts) or in those OCI v1.0 compliant registries. To enable ORAS to work with popular registries, it provides backward compatibility which supports two types of manifest and allows fallback to upload the OCI image manifest to OCI v1.0 compliant registries or enabled OCI image manifest storage. +Since 0.16.0, ORAS supports pushing OCI artifact manifest to OCI v1.0 compliant registries. However, the new manifest type may not be supported on the consumer side (e.g. self-crafted scripts) or in those OCI v1.0 compliant registries. To enable ORAS to work with popular registries, it provides backward compatibility which supports two types of manifest and allows fallback to upload the OCI image manifest to OCI v1.0 compliant registries or those which enabled OCI image manifest storage. ## Challenge -However, the ORAS fallback may fail as there is no deterministic way to confirm if a registry supporting OCI artifact manifest and no consistent error response. +However, ORAS fallback may fail as there is no deterministic way to confirm if a registry supporting OCI artifact manifest and no consistent error response. + You can find the testing result of the implementation result for OCI Spec from this [blog](https://toddysm.com/2023/01/05/oci-artifct-manifests-oci-referrers-api-and-their-support-across-registries-part-1/). On the other hand, users may want to force-push a specific manifest type to a registry for some reason. The current workaround for enabling a kind of compatibility mode is to specify a `--config` flag when using `oras push`. Since the OCI artifact manifest does not have a `config`, it will push an OCI image manifest instead. It is not user-friendly and is a bit hacky. It would be better if we can provide a compatibility mode to easily customize and switch the manifest uploading behavior, and enable users to handle the incompatibility problem when using ORAS across different registries. ## Goals -- Enable ORAS to work with more registries -- Provide different options to allow users to customize the behaviors of uploading manifest to the registry +- Provide different options to allow users to customize the manifest uploading behavior to the registry - Provide an easy-to-use and secure user experience when switching the behaviors +- Enable ORAS to work with more registries flexibly ## Solution -Adding a new flag `--compatibility` under CLI commands `oras push` and `oras attach` with different variables to configure the behaviors of uploading manifest. We will only use `oras push` to demonstrate the examples below. +Adding new flags to `oras push` and `oras attach` respectively with different variables to configure the manifest uploading behaviors. -### Use case A +- Adding a flag `--image-spec` to `oras push` and `oras attach` to force uploading a specific manifest type to registry +- Adding a flag `--distribution-spec` to `oras attach` to configure compatibility with registry -If users want to force-push the OCI artifact manifest to registries whether they are compliant with OCI v1.0 or v1.1, using `--compatibility artifact-manifest` will only upload OCI image manifest to registries. Users might choose it for security requirements, such as pushing a signature to a registry without changing its digest. +### Force uploading a specific manifest type using a flag `--image-spec` -```bash -oras push localhost:5000/hello-artifact:v1 --artifact-type sbom/example --compatibility artifact-manifest sbom.json -``` +It follows `--image-spec -` to enable configuration of using which spec version and manifest type. Currently, it only supports specifying v1.1 as the spec version. -### Use case B +| registry support | v1.1-artifact | v1.1-image | +| :-------------------------------------- | ----------------- | -------------- | +| OCI spec 1.0 | no | yes[^footnote] | +| OCI spec 1.1 without referrers API | yes | yes | +| OCI spec 1.1 with referrers API support | yes | yes | -If users want to force-push the OCI image manifest to registries whether they are compliant with OCI v1.0 or v1.1, using `--compatibility image-manifest` will only upload OCI image manifest to registries. This option is helpful when users have concerns to use OCI artifact manifest or migrate content to OCI v1.0 compliant registries. +> [^footnote]: It only works when the registry returns error code 404 for [referrers API](https://github.com/opencontainers/distribution-spec/blob/v1.1.0-rc1/spec.md#listing-referrers). + +If users want to force pushing a specific version of OCI artifact manifest to a registry, they can use `--image-spec v1.1-artifact`. An OCI artifact manifest will be packed and uploaded. Users might choose it for security requirements, such as pushing a signature to a registry without changing its digest. For example: ```bash -oras push localhost:5000/hello-artifact:v1 --artifact-type sbom/example --compatibility image-manifest sbom.json +oras push localhost:5000/hello-artifact:v1 \ +--image-spec v1.1-artifact +--artifact-type sbom/example \ +sbom.json ``` -### Use case C - -Disable backward compatibility and only upload OCI artifact manifest to a registry. This flag `--compatibility min` will be commonly used with the OCI v1.1 compliant registries or registries that support storing OCI artifact manifest, such as Zot, Azure Container Registry, and Docker Hub. ORAS will push OCI artifact manifest only and no further actions for maintaining artifact references in registries. This is the most strict option for the behavior of uploading manifest. Users might choose it for security requirements. +If users want to force pushing an OCI image manifest, no matter whether the registry is compliant with the OCI Spec v1.0 or v1.1, using `--image-spec v1.1-image` will only upload the OCI image manifest to a registry. This option is useful when users have concerns to use OCI artifact manifest or need to migrate content to OCI v1.0 compliant registry. For example: ```bash -oras push localhost:5000/hello-artifact:v1 --artifact-type sbom/example --compatibility min sbom.json +oras push localhost:5000/hello-artifact:v1 \ +--image-spec v1.1-image \ +--artifact-type sbom/example \ +sbom.json \ ``` -### Use case D +### Configure compatibility with OCI registry using a flag `--distribution-spec` -If users want to upload manifest to OCI v1.0 compliant registries like Harbor, GitHub Container Registry, etc, using `--compatibility max` will push OCI image manifest and then push Referrals tag schema if the `subject` field of the OCI image manifest is not empty. This option enables maximum backward compatibility for registries. It allows `oras push` or `oras attach` to work with the OCI v1.0 compliant registries even though the registries return non-404 response code. +Based on the Referrers API status in the registry, users can use flag `--distribution-spec --