master-22d45ae August, 25 2020

UPDATES

Issues addressed in this release:
issue: #368 Title: add Code of Conduct to README
issue: #317 Title: [Process] Adopt CNCF's Code of conduct
issue: #366 Title: Add OCI compliant container runtime test
issue: #327 Title: [Platform] Hardware test: Test if Container Runtime is OCI compliant
issue: #364 Title: Don't publish release if github keys aren't set
issue: #360 Title: Add node failure resilience test cncf/cnf-conformance#322
issue: #322 Title: [Platform] Proof of Concept - Resilience test: Perform chaos resilience with node failure
issue: #362 Title: make wait_for_install wait on all replicas
issue: #326 Title: Maint/upgrade crystal to 0.35.1 logging refactor
issue: #363 Title: #362 release manager spec tests now disabled for pull requests
issue: #356 Title: Fix chaos-mesh version through constant and update repository URL
issue: #347 Title: Add emojis to tests that are missing emojis
issue: #325 Title: Maint/upgrade crystal to 0.35.1 311
issue: #332 Title: Add linter and usage doc
issue: #331 Title: #301 curl now supports new cnf-conformance binary names
issue: #301 Title: [Install] Add curl install option

Artifact info:

• Commit: master-22d45ae
• SHA256SUM: c0bfbc94c52389c8d9d76f73aa879f8af43111fc0b6dde3cb432fd15d32bb53d cnf-conformance-master-22d45ae.tar.gz

master-0d613f8 August, 26 2020

UPDATES

Issues addressed in this release:
issue: #327 Title: [Platform] Hardware test: Test if Container Runtime is OCI compliant
issue: #371 Title: Add missing spec test tag for platform:oci_compliant
issue: #367 Title: #365 Add support for running platform tests without a workload cnf
issue: #365 Title: Fix platform tests so they can be run independent of a workload cnf
issue: #368 Title: add Code of Conduct to README
issue: #317 Title: [Process] Adopt CNCF's Code of conduct
issue: #366 Title: Add OCI compliant container runtime test
issue: #364 Title: Don't publish release if github keys aren't set
issue: #360 Title: Add node failure resilience test cncf/cnf-conformance#322
issue: #322 Title: [Platform] Proof of Concept - Resilience test: Perform chaos resilience with node failure
issue: #362 Title: make wait_for_install wait on all replicas
issue: #326 Title: Maint/upgrade crystal to 0.35.1 logging refactor
issue: #363 Title: #362 release manager spec tests now disabled for pull requests
issue: #356 Title: Fix chaos-mesh version through constant and update repository URL
issue: #347 Title: Add emojis to tests that are missing emojis
issue: #325 Title: Maint/upgrade crystal to 0.35.1 311
issue: #332 Title: Add linter and usage doc
issue: #331 Title: #301 curl now supports new cnf-conformance binary names
issue: #301 Title: [Install] Add curl install option

Artifact info:

• Commit: master-0d613f8
• SHA256SUM: fa5efab47792d58a31645b7b0f868403d03b877392c6c15647136b28dee297c2 cnf-conformance-master-0d613f8.tar.gz

master-56fe968 August, 24 2020

UPDATES

Issues addressed in this release:
issue: #366 Title: Add OCI compliant container runtime test
issue: #327 Title: [Platform] Hardware test: Test if Container Runtime is OCI compliant
issue: #364 Title: Don't publish release if github keys aren't set
issue: #360 Title: Add node failure resilience test cncf/cnf-conformance#322
issue: #322 Title: [Platform] Proof of Concept - Resilience test: Perform chaos resilience with node failure
issue: #362 Title: make wait_for_install wait on all replicas
issue: #326 Title: Maint/upgrade crystal to 0.35.1 logging refactor
issue: #363 Title: #362 release manager spec tests now disabled for pull requests
issue: #356 Title: Fix chaos-mesh version through constant and update repository URL
issue: #347 Title: Add emojis to tests that are missing emojis
issue: #325 Title: Maint/upgrade crystal to 0.35.1 311
issue: #332 Title: Add linter and usage doc
issue: #331 Title: #301 curl now supports new cnf-conformance binary names
issue: #301 Title: [Install] Add curl install option

Artifact info:

• Commit: master-56fe968
• SHA256SUM: 620d42f83226577efc9ea4fdb854a39df2ae721438a0a6ba6e07f319038bbc8f cnf-conformance-master-56fe968.tar.gz

master-ffa67f9 August, 19 2020

UPDATES

Issues addressed in this release:
issue: #364 Title: Don't publish release if github keys aren't set
issue: #360 Title: Add node failure resilience test cncf/cnf-conformance#322
issue: #322 Title: [Platform] Proof of Concept - Resilience test: Perform chaos resilience with node failure
issue: #362 Title: make wait_for_install wait on all replicas
issue: #326 Title: Maint/upgrade crystal to 0.35.1 logging refactor
issue: #363 Title: #362 release manager spec tests now disabled for pull requests
issue: #356 Title: Fix chaos-mesh version through constant and update repository URL
issue: #347 Title: Add emojis to tests that are missing emojis
issue: #325 Title: Maint/upgrade crystal to 0.35.1 311
issue: #332 Title: Add linter and usage doc
issue: #331 Title: #301 curl now supports new cnf-conformance binary names
issue: #301 Title: [Install] Add curl install option

Artifact info:

• Commit: master-ffa67f9
• SHA256SUM: 986ad30321b33813fecc730b337ffa89d9b41232734e5c4ffb1bea23404ce209 cnf-conformance-master-ffa67f9.tar.gz

master-673c872 August, 18 2020

UPDATES

Issues addressed in this release:
issue: #360 Title: Add node failure resilience test cncf/cnf-conformance#322
issue: #322 Title: [Platform] Proof of Concept - Resilience test: Perform chaos resilience with node failure
issue: #362 Title: make wait_for_install wait on all replicas
issue: #326 Title: Maint/upgrade crystal to 0.35.1 logging refactor
issue: #363 Title: #362 release manager spec tests now disabled for pull requests
issue: #356 Title: Fix chaos-mesh version through constant and update repository URL
issue: #347 Title: Add emojis to tests that are missing emojis
issue: #325 Title: Maint/upgrade crystal to 0.35.1 311
issue: #332 Title: Add linter and usage doc
issue: #331 Title: #301 curl now supports new cnf-conformance binary names
issue: #301 Title: [Install] Add curl install option

Artifact info:

• Commit: master-673c872
• SHA256SUM: 121a348f671eac6f969525ede4eda4edb3a7248d645f3ce77b71aeb570780a79 cnf-conformance-master-673c872.tar.gz

v0.0.9 August, 04 2020

UPDATES

Issues Addressed in this release:
issue: #301 Title: [Install] Add curl install option
issue: #323 Title: Curl install add unsupported shell message cncf/cnf-conformance#301
issue: #321 Title: Feature/automated release manager 306
issue: #306 Title: [Feature] Automated Release Manager
issue: #319 Title: Add spec for Curl install#301
issue: #316 Title: Feature/automated release manager 306
issue: #318 Title: #206 documentation update
issue: #206 Title: Proof of Concept: Scripted/helper for using an ephemeral dev environment
issue: #315 Title: Add Curl install option cncf/cnf-conformance#301
issue: #314 Title: Feature/automated release manager 306
issue: #313 Title: Ephemeral dev env#206 Static build
issue: #308 Title: move verbose logging stuff to a seperate logger
issue: #309 Title: Ephemeral dev env#206 Documentation
issue: #280 Title: [Refactor] Improve output by refactoring all console messaging to use standard logging
issue: #305 Title: Version command#284 Uploading corrupted files
issue: #284 Title: [Feature] Add version command
issue: #303 Title: #284 asset upload now compares local hash with remote master hash
issue: #302 Title: Add version command
issue: #300 Title: Feature/280 log level change aka logging refactor p3
issue: #298 Title: Ephemeral Dev Tool cncf/cnf-conformance#206
issue: #291 Title: logging refactor part 2
issue: #281 Title: Logging Refactor
issue: #290 Title: Update how and when results are saved so a repeated warning for overwriting results.yml does not occur
issue: #282 Title: [Refactor] Update how and when results are saved so a repeated warning for overwriting results.yml does not occur
issue: #289 Title: Upsert now stop double counting of test scores
issue: #159 Title: Statelessness test: Does the CNF use a non-cloud native data store: hostPath volumes found in runtime deployment manifest?
issue: #286 Title: update MAINTAINERS adding agentpoyo
issue: #287 Title: Pin ChaosMesh version to v0.8.0
issue: #277 Title: Add hostPath volumes test. cncf/cnf-conformance#159
issue: #272 Title: update static build command
issue: #276 Title: updating README, Test Categories and Install Guide
issue: #274 Title: Add missing spec tests & fix bug with warning message for yml validator cncf/cnf-conformance#201
issue: #201 Title: [Example CNFs] Create a yml validator that fails an example CNF with incomplete config files
issue: #273 Title: Create CONTRIBUTING.md
issue: #275 Title: #158 container kill documentation
issue: #262 Title: chaos mesh container kill test re: cncf/cnf-conformance#158
issue: #158 Title: Resilience test: Does CNF pass the Container Kill Chaos Mesh Experiment 🗡️💀♻️?
issue: #270 Title: Updated points.yml to only have implemented tests enabled
issue: #271 Title: update TUG to monthly
issue: #261 Title: Chaosmesh update
issue: #260 Title: Add Pod CPU Hog Test #231
issue: #231 Title: Resilience test: Does CNF pass the Pod CPU Hog Litmus Chaos Experiment?
issue: #257 Title: Adding 📶☠️ emoji to chaos network test #157
issue: #157 Title: Resilience test: Does CNF pass the Pod Network Loss Litmus Chaos Experiment?
issue: #255 Title: Deployment wait output to make defaults quieter
issue: #253 Title: need kube-multus whitelisted because beta test clusters have multus i…
issue: #249 Title: Add Network Loss Chaos Test lfn-cnti/certification#157
issue: #232 Title: [Beta Test] Beta Test Instructions - Phase 1
issue: #181 Title: [CNF Developer UX] display the maximum number of points with the final score, aka 99/100
issue: #251 Title: [BUG] in Beta QA, max score is too high for the actual tests that are running
issue: #250 Title: Whitelisting kube-multus in CoreDNS example
issue: #238 Title: [Beta Test] Create clusters for beta testing
issue: #242 Title: #239 updated shard.yml to ensure correct version of crystal is used
issue: #239 Title: BUG: crystal spec fails when cnf-conformance repo directory is named differently
issue: #245 Title: Display the maximum number of points with the final score, aka 99/100
issue: #241 Title: Bugfix: microservice test doesn't clean up envoy
issue: #240 Title: Bugfix: microservice test doesn't clean up envoy
issue: #234 Title: re: #201 add validate_config task
issue: #230 Title: All tests should work with a cnf-config= argument or run on all cnfs in the cnfs directory #228
issue: #228 Title: All tests should work with a cnf-config= argument or run on all cnfs in the cnfs directory
issue: #225 Title: Update KinD documentation
issue: #216 Title: [Example CNFs] Create a yml validator that fails an example CNF with incomplete config files
issue: #226 Title: Update Halite version to 0.10.5
issue: #219 Title: [CNF Developer UX] Documenation: Support passing cnf configuration to all and cleanup commands to simplify CLI usage
issue: #213 Title: [CNF Developer UX] Support passing cnf configuration to all and cleanup commands to simplify CLI usage
issue: #227 Title: Fix bugs with reasonable_startup, rolling_update & hardcoded ip tests.
issue: #222 Title: #221 helm_deploy now uses a prefix for testing the deployment
issue: #221 Title: Helm_deploy test should use a temporary release name
issue: #218 Title: [CNF Developer UX] Support passing cnf configuration to all and cleanup commands to simplify CLI usage

Artifact info:

• Commit: v0.0.9
• SHA256SUM: 909830e123b0a880b32caa6b7e73c4a895c9574263af62cad5e27aed48a970fb cnf-conformance-v0.0.9.tar.gz

master-3517309 July, 30 2020

UPDATES

Issues Addressed in this release:
issue: #321 Title: Feature/automated release manager 306
issue: #306 Title: [Feature] Automated Release Manager
issue: #319 Title: Add spec for Curl install#301
issue: #301 Title: [Install] Add curl install option
issue: #316 Title: Feature/automated release manager 306
issue: #318 Title: #206 documentation update
issue: #206 Title: Proof of Concept: Scripted/helper for using an ephemeral dev environment
issue: #315 Title: Add Curl install option cncf/cnf-conformance#301
issue: #314 Title: Feature/automated release manager 306
issue: #313 Title: Ephemeral dev env#206 Static build
issue: #308 Title: move verbose logging stuff to a seperate logger
issue: #309 Title: Ephemeral dev env#206 Documentation
issue: #280 Title: [Refactor] Improve output by refactoring all console messaging to use standard logging
issue: #305 Title: Version command#284 Uploading corrupted files
issue: #284 Title: [Feature] Add version command
issue: #303 Title: #284 asset upload now compares local hash with remote master hash
issue: #302 Title: Add version command
issue: #300 Title: Feature/280 log level change aka logging refactor p3
issue: #298 Title: Ephemeral Dev Tool cncf/cnf-conformance#206
issue: #291 Title: logging refactor part 2
issue: #281 Title: Logging Refactor
issue: #290 Title: Update how and when results are saved so a repeated warning for overwriting results.yml does not occur
issue: #282 Title: [Refactor] Update how and when results are saved so a repeated warning for overwriting results.yml does not occur
issue: #289 Title: Upsert now stop double counting of test scores
issue: #159 Title: Statelessness test: Does the CNF use a non-cloud native data store: hostPath volumes found in runtime deployment manifest?
issue: #286 Title: update MAINTAINERS adding agentpoyo
issue: #287 Title: Pin ChaosMesh version to v0.8.0
issue: #277 Title: Add hostPath volumes test. cncf/cnf-conformance#159
issue: #272 Title: update static build command
issue: #276 Title: updating README, Test Categories and Install Guide
issue: #274 Title: Add missing spec tests & fix bug with warning message for yml validator cncf/cnf-conformance#201
issue: #201 Title: [Example CNFs] Create a yml validator that fails an example CNF with incomplete config files
issue: #273 Title: Create CONTRIBUTING.md
issue: #275 Title: #158 container kill documentation
issue: #262 Title: chaos mesh container kill test re: cncf/cnf-conformance#158
issue: #158 Title: Resilience test: Does CNF pass the Container Kill Chaos Mesh Experiment 🗡️💀♻️?
issue: #270 Title: Updated points.yml to only have implemented tests enabled
issue: #271 Title: update TUG to monthly
issue: #261 Title: Chaosmesh update
issue: #260 Title: Add Pod CPU Hog Test #231
issue: #231 Title: Resilience test: Does CNF pass the Pod CPU Hog Litmus Chaos Experiment?
issue: #257 Title: Adding 📶☠️ emoji to chaos network test #157
issue: #157 Title: Resilience test: Does CNF pass the Pod Network Loss Litmus Chaos Experiment?
issue: #255 Title: Deployment wait output to make defaults quieter
issue: #253 Title: need kube-multus whitelisted because beta test clusters have multus i…
issue: #249 Title: Add Network Loss Chaos Test lfn-cnti/certification#157
issue: #232 Title: [Beta Test] Beta Test Instructions - Phase 1
issue: #181 Title: [CNF Developer UX] display the maximum number of points with the final score, aka 99/100
issue: #251 Title: [BUG] in Beta QA, max score is too high for the actual tests that are running
issue: #250 Title: Whitelisting kube-multus in CoreDNS example
issue: #238 Title: [Beta Test] Create clusters for beta testing
issue: #242 Title: #239 updated shard.yml to ensure correct version of crystal is used
issue: #239 Title: BUG: crystal spec fails when cnf-conformance repo directory is named differently
issue: #245 Title: Display the maximum number of points with the final score, aka 99/100
issue: #241 Title: Bugfix: microservice test doesn't clean up envoy
issue: #240 Title: Bugfix: microservice test doesn't clean up envoy
issue: #234 Title: re: #201 add validate_config task
issue: #230 Title: All tests should work with a cnf-config= argument or run on all cnfs in the cnfs directory #228
issue: #228 Title: All tests should work with a cnf-config= argument or run on all cnfs in the cnfs directory
issue: #225 Title: Update KinD documentation
issue: #216 Title: [Example CNFs] Create a yml validator that fails an example CNF with incomplete config files
issue: #226 Title: Update Halite version to 0.10.5
issue: #219 Title: [CNF Developer UX] Documenation: Support passing cnf configuration to all and cleanup commands to simplify CLI usage
issue: #213 Title: [CNF Developer UX] Support passing cnf configuration to all and cleanup commands to simplify CLI usage
issue: #227 Title: Fix bugs with reasonable_startup, rolling_update & hardcoded ip tests.
issue: #148 Title: [CNFs] Test CNF Testbed's VPP IP Forwarder as an example CNF with the CNF Conformance Test Suite
issue: #222 Title: #221 helm_deploy now uses a prefix for testing the deployment
issue: #221 Title: Helm_deploy test should use a temporary release name
issue: #218 Title: [CNF Developer UX] Support passing cnf configuration to all and cleanup commands to simplify CLI usage

Artifact info:

• Commit: master-3517309
• SHA256SUM: d6460e8ebe886722e13030003b1a257f4e3a6687750351a7f81ac7280d325b2f cnf-conformance-master-3517309.tar.gz

v0.7.2-beta1 June, 4th 2020

UPDATES

Bugfixes:

• Default scoring and points don't show unimplemented tests
• Timeout for startup time test increased

Other minor improvements

Artifact info:

• Commit: ede0a47
• MD5SUM: 78827add4f1c79009339f7270716e73c cnf-conformance-v0.7.2-amd64-static

v0.7.1-beta1 June, 4th 2020

UPDATE

• minor fixes
• reduce debug output by default

Artifact info

• Commit: db34bd3
• MD5SUM: 3d82d98cd0681a9bf303a1ce02bf9959 cnf-conformance

v0.7.0-beta1 June, 3rd 2020

UPDATES for v0.7.0 Beta 1

Workload Tests:

• Resilience category

New feature, bugfixes, and updates:

• display the maximum number of points with the final score, aka 99/100
• New cnf-config option for all commands
• Crystal and library version issues
• Helm deploy uses a temporary release name
• Regex for kubectl version fixed

Artifact info:

• commit: 3e633a7
• md5sum: 7a995fac49033662d048a34b7f7b4210 cnf-conformance
• architecture: amd64