From 363eebe82b7b0b2e279e40a54873abc074f5378e Mon Sep 17 00:00:00 2001 From: Jose Luis Urien Date: Thu, 24 Oct 2024 17:39:56 +0200 Subject: [PATCH 1/5] Common artifacts for testing error scenarios for device and phoneNumber --- artifacts/testing/C01-device-errors.feature | 118 ++++++++++++++++++ .../testing/C02-phoneNumber-errors.feature | 74 +++++++++++ 2 files changed, 192 insertions(+) create mode 100644 artifacts/testing/C01-device-errors.feature create mode 100644 artifacts/testing/C02-phoneNumber-errors.feature diff --git a/artifacts/testing/C01-device-errors.feature b/artifacts/testing/C01-device-errors.feature new file mode 100644 index 00000000..838ab323 --- /dev/null +++ b/artifacts/testing/C01-device-errors.feature @@ -0,0 +1,118 @@ +Feature: CAMARA Common Artifact C01 - Test scenarios for device errors + + Common error scenarios for POST operations with device as input either in the request + body or implied from the access. + + Artifact parameters (to be replaced by values according to the API operation): + + - {{feature_identifier}} + + This is not a complete feature but a collection of scenarios that can be applied + with minor modifications to test plans. + + These scenarios assume that other properties not explicitly mentioned in the scenario + are set by default to a valid value. This can be specified in the feature Background. + + # Error scenarios for management of input parameter device + + # If the access token identifies a device, error 422 UNNECESSARY_DEVICE may be returned instead + @{{feature_identifier}}_C01.01_device_empty + Scenario: The device value is an empty object + Given the header "Authorization" is set to a valid access which does not identifiy a single device + And the request body property "$.device" is set to: {} + When the HTTP "POST" request is sent + Then the response status code is 400 + And the response property "$.status" is 400 + And the response property "$.code" is "INVALID_ARGUMENT" + And the response property "$.message" contains a user friendly text + + + # If the access token identifies a device, error 422 UNNECESSARY_DEVICE may be returned instead + @{{feature_identifier}}_C01.02_device_identifiers_not_schema_compliant + Scenario Outline: Some device identifier value does not comply with the schema + Given the header "Authorization" is set to a valid access which does not identifiy a single device + And the request body property "" does not comply with the OAS schema at "" + When the HTTP "POST" request is sent + Then the response status code is 400 + And the response property "$.status" is 400 + And the response property "$.code" is "INVALID_ARGUMENT" + And the response property "$.message" contains a user friendly text + + Examples: + | device_identifier | oas_spec_schema | + | $.device.phoneNumber | /components/schemas/PhoneNumber | + | $.device.ipv4Address | /components/schemas/NetworkAccessIdentifier | + | $.device.ipv6Address | /components/schemas/DeviceIpv4Addr | + | $.device.networkIdentifier | /components/schemas/DeviceIpv6Address | + + + # This scenario may happen e.g. with 2-legged access tokens, which do not identify a single device. + @{{feature_identifier}}_C01.03_device_not_found + Scenario: Some identifier cannot be matched to a device + Given the header "Authorization" is set to a valid access which does not identifiy a single device + And the request body property "$.device" is compliant with the schema but does not identify a valid device + When the HTTP "POST" request is sent + Then the response status code is 404 + And the response property "$.status" is 404 + And the response property "$.code" is "IDENTIFIER_NOT_FOUND" + And the response property "$.message" contains a user friendly text + + + @{{feature_identifier}}_C02.04_unnecessary_device + Scenario: Device not to be included when can be deducted from the access token + Given the header "Authorization" is set to a valid access token identifying a device + And the request body property "$.device" is set to a valid device + When the HTTP "POST" request is sent + Then the response status code is 403 + And the response property "$.status" is 422 + And the response property "$.code" is "UNNECESSARY_IDENTIFIER" + And the response property "$.message" contains a user friendly text + + + @{{feature_identifier}}_C01.05_unidentifiable_device + Scenario: Device not included and cannot be deducted from the access token + Given the header "Authorization" is set to a valid access which does not identifiy a single device + And the request body property "$.device" is not included + When the HTTP "POST" request is sent + Then the response status code is 422 + And the response property "$.status" is 422 + And the response property "$.code" is "MISSING_IDENTIFIER" + And the response property "$.message" contains a user friendly text + + + # For r1.x APIs, networkAccessIdentifier is never supported + @{{feature_identifier}}_C01.06_device_identifiers_unsupported + Scenario: None of the provided device identifiers is supported by the implementation + Given that some type of device identifiers are not supported by the implementation + And the request body property "$.device" only includes device identifiers not supported by the implementation + When the HTTP "POST" request is sent + Then the response status code is 422 + And the response property "$.status" is 422 + And the response property "$.code" is "UNSUPPORTED_IDENTIFIER" + And the response property "$.message" contains a user friendly text + + + # When the service is only offered to certain type of devices or subcriptions, e.g. IoT, , B2C, etc + @{{feature_identifier}}_C01.07_device_not_supported + Scenario: Service not available for the device + Given that the service is not available for all devices commercialized by the operator + And a valid device, identified by the token or provided in the request body, for which the service is not applicable + When the HTTP "POST" request is sent + Then the response status code is 422 + And the response property "$.status" is 422 + And the response property "$.code" is "SERVICE_NOT_APPLICABLE" + And the response property "$.message" contains a user friendly text + + + # Several identifiers provided but they do not identify the same device + # This scenario is under discussion, as it may reveal undesired information or even substitute the Number Verification API functionality + @{{feature_identifier}}_C01.08_device_identifiers_mismatch + Scenario: Device identifiers mismatch + Given the header "Authorization" is set to a valid access which does not identifiy a single device + And at least 2 types of device identifiers are supported by the implementation + And the request body property "$.device" includes several identifiers, each of them identifying a valid but different device + When the HTTP "POST" request is sent + Then the response status code is 422 + And the response property "$.status" is 422 + And the response property "$.code" is "IDENTIFIER_MISMATCH" + And the response property "$.message" contains a user friendly text diff --git a/artifacts/testing/C02-phoneNumber-errors.feature b/artifacts/testing/C02-phoneNumber-errors.feature new file mode 100644 index 00000000..d43dfb97 --- /dev/null +++ b/artifacts/testing/C02-phoneNumber-errors.feature @@ -0,0 +1,74 @@ +Feature: CAMARA Common Artifact C02 - Test scenarios for phoneNumber errors + + Common error scenarios for POST operations with phoneNumber as input either in the request + body or implied from the access. + + Artifact parameters (to be replaced by values according to the API operation): + + - {{feature_identifier}} + + This is not a complete feature but a collection of scenarios that can be applied + with minor modifications to test plans. + + These scenarios assume that other properties not explicitly mentioned in the scenario + are set by default to a valid value. This can be specified in the feature Background. + + + # Error scenarios for management of input parameter phoneNumber + + # If the access token identifies a phone number, error 422 UNNECESSARY_DEVICE may be returned instead + @{{feature_identifier}}_C02.01_phone_number_not_schema_compliant + Scenario: Phone number value does not comply with the schema + Given the header "Authorization" is set to a valid access which does not identifiy a single phone number + And the request body property "$.phoneNumber" does not comply with the OAS schema at "/components/schemas/PhoneNumber" + When the HTTP "POST" request is sent + Then the response status code is 400 + And the response property "$.status" is 400 + And the response property "$.code" is "INVALID_ARGUMENT" + And the response property "$.message" contains a user friendly text + + + # This scenario may happen e.g. with 2-legged access tokens, which do not identify a single phone number. + @{{feature_identifier}}_C02.02_phone_number_not_found + Scenario: Phone number not found + Given the header "Authorization" is set to a valid access which does not identifiy a single phone number + And the request body property "$.phoneNumber" is compliant with the schema but does not identify a valid phone number + When the HTTP "POST" request is sent + Then the response status code is 404 + And the response property "$.status" is 404 + And the response property "$.code" is "IDENTIFIER_NOT_FOUND" + And the response property "$.message" contains a user friendly text + + + @{{feature_identifier}}_C02.03_unnecessary_phone_number + Scenario: Phone number not to included when can be deducted from the access token + Given the header "Authorization" is set to a valid access token identifying a phone number + And the request body property "$.phoneNumber" is set to a valid phone number + When the HTTP "POST" request is sent + Then the response status code is 403 + And the response property "$.status" is 422 + And the response property "$.code" is "UNNECESSARY_IDENTIFIER" + And the response property "$.message" contains a user friendly text + + + @{{feature_identifier}}_C02.04_unidentifiable_device + Scenario: Phone number not included and cannot be deducted from the access token + Given the header "Authorization" is set to a valid access which does not identifiy a single phone number + And the request body property "$.phoneNumber" is not included + When the HTTP "POST" request is sent + Then the response status code is 422 + And the response property "$.status" is 422 + And the response property "$.code" is "MISSING_IDENTIFIER" + And the response property "$.message" contains a user friendly text + + + # When the service is only offered to certain type of subcriptions, e.g. IoT, , B2C, etc + @{{feature_identifier}}_C02.05_phone_number_not_supported + Scenario: Service not available for the phone number + Given that the service is not available for all phone numbers commercialized by the operator + And a valid phone number, identified by the token or provided in the request body, for which the service is not applicable + When the HTTP "POST" request is sent + Then the response status code is 422 + And the response property "$.status" is 422 + And the response property "$.code" is "SERVICE_NOT_APPLICABLE" + And the response property "$.message" contains a user friendly text From a1bee21d94d024024ca5ed5dae78a5b16a9d10c0 Mon Sep 17 00:00:00 2001 From: Jose Luis Urien Date: Fri, 22 Nov 2024 19:59:16 +0100 Subject: [PATCH 2/5] Comments addressed --- artifacts/testing/C01-device-errors.feature | 26 +++++++++---------- .../testing/C02-phoneNumber-errors.feature | 14 +++++----- 2 files changed, 18 insertions(+), 22 deletions(-) diff --git a/artifacts/testing/C01-device-errors.feature b/artifacts/testing/C01-device-errors.feature index 838ab323..02d7314c 100644 --- a/artifacts/testing/C01-device-errors.feature +++ b/artifacts/testing/C01-device-errors.feature @@ -15,10 +15,9 @@ Feature: CAMARA Common Artifact C01 - Test scenarios for device errors # Error scenarios for management of input parameter device - # If the access token identifies a device, error 422 UNNECESSARY_DEVICE may be returned instead @{{feature_identifier}}_C01.01_device_empty Scenario: The device value is an empty object - Given the header "Authorization" is set to a valid access which does not identifiy a single device + Given the header "Authorization" is set to a valid access which does not identify a single device And the request body property "$.device" is set to: {} When the HTTP "POST" request is sent Then the response status code is 400 @@ -27,10 +26,9 @@ Feature: CAMARA Common Artifact C01 - Test scenarios for device errors And the response property "$.message" contains a user friendly text - # If the access token identifies a device, error 422 UNNECESSARY_DEVICE may be returned instead @{{feature_identifier}}_C01.02_device_identifiers_not_schema_compliant Scenario Outline: Some device identifier value does not comply with the schema - Given the header "Authorization" is set to a valid access which does not identifiy a single device + Given the header "Authorization" is set to a valid access which does not identify a single device And the request body property "" does not comply with the OAS schema at "" When the HTTP "POST" request is sent Then the response status code is 400 @@ -41,15 +39,15 @@ Feature: CAMARA Common Artifact C01 - Test scenarios for device errors Examples: | device_identifier | oas_spec_schema | | $.device.phoneNumber | /components/schemas/PhoneNumber | - | $.device.ipv4Address | /components/schemas/NetworkAccessIdentifier | - | $.device.ipv6Address | /components/schemas/DeviceIpv4Addr | - | $.device.networkIdentifier | /components/schemas/DeviceIpv6Address | + | $.device.ipv4Address | /components/schemas/DeviceIpv4Addr | + | $.device.ipv6Address | /components/schemas/DeviceIpv6Address | + | $.device.networkIdentifier | /components/schemas/NetworkAccessIdentifier | # This scenario may happen e.g. with 2-legged access tokens, which do not identify a single device. @{{feature_identifier}}_C01.03_device_not_found Scenario: Some identifier cannot be matched to a device - Given the header "Authorization" is set to a valid access which does not identifiy a single device + Given the header "Authorization" is set to a valid access which does not identify a single device And the request body property "$.device" is compliant with the schema but does not identify a valid device When the HTTP "POST" request is sent Then the response status code is 404 @@ -63,15 +61,15 @@ Feature: CAMARA Common Artifact C01 - Test scenarios for device errors Given the header "Authorization" is set to a valid access token identifying a device And the request body property "$.device" is set to a valid device When the HTTP "POST" request is sent - Then the response status code is 403 + Then the response status code is 422 And the response property "$.status" is 422 And the response property "$.code" is "UNNECESSARY_IDENTIFIER" And the response property "$.message" contains a user friendly text - @{{feature_identifier}}_C01.05_unidentifiable_device + @{{feature_identifier}}_C01.05_missing_device Scenario: Device not included and cannot be deducted from the access token - Given the header "Authorization" is set to a valid access which does not identifiy a single device + Given the header "Authorization" is set to a valid access which does not identify a single device And the request body property "$.device" is not included When the HTTP "POST" request is sent Then the response status code is 422 @@ -81,7 +79,7 @@ Feature: CAMARA Common Artifact C01 - Test scenarios for device errors # For r1.x APIs, networkAccessIdentifier is never supported - @{{feature_identifier}}_C01.06_device_identifiers_unsupported + @{{feature_identifier}}_C01.06_unsupported_device Scenario: None of the provided device identifiers is supported by the implementation Given that some type of device identifiers are not supported by the implementation And the request body property "$.device" only includes device identifiers not supported by the implementation @@ -92,7 +90,7 @@ Feature: CAMARA Common Artifact C01 - Test scenarios for device errors And the response property "$.message" contains a user friendly text - # When the service is only offered to certain type of devices or subcriptions, e.g. IoT, , B2C, etc + # When the service is only offered to certain type of devices or subscriptions, e.g. IoT, , B2C, etc @{{feature_identifier}}_C01.07_device_not_supported Scenario: Service not available for the device Given that the service is not available for all devices commercialized by the operator @@ -108,7 +106,7 @@ Feature: CAMARA Common Artifact C01 - Test scenarios for device errors # This scenario is under discussion, as it may reveal undesired information or even substitute the Number Verification API functionality @{{feature_identifier}}_C01.08_device_identifiers_mismatch Scenario: Device identifiers mismatch - Given the header "Authorization" is set to a valid access which does not identifiy a single device + Given the header "Authorization" is set to a valid access which does not identify a single device And at least 2 types of device identifiers are supported by the implementation And the request body property "$.device" includes several identifiers, each of them identifying a valid but different device When the HTTP "POST" request is sent diff --git a/artifacts/testing/C02-phoneNumber-errors.feature b/artifacts/testing/C02-phoneNumber-errors.feature index d43dfb97..eeb4280a 100644 --- a/artifacts/testing/C02-phoneNumber-errors.feature +++ b/artifacts/testing/C02-phoneNumber-errors.feature @@ -16,10 +16,9 @@ Feature: CAMARA Common Artifact C02 - Test scenarios for phoneNumber errors # Error scenarios for management of input parameter phoneNumber - # If the access token identifies a phone number, error 422 UNNECESSARY_DEVICE may be returned instead @{{feature_identifier}}_C02.01_phone_number_not_schema_compliant Scenario: Phone number value does not comply with the schema - Given the header "Authorization" is set to a valid access which does not identifiy a single phone number + Given the header "Authorization" is set to a valid access which does not identify a single phone number And the request body property "$.phoneNumber" does not comply with the OAS schema at "/components/schemas/PhoneNumber" When the HTTP "POST" request is sent Then the response status code is 400 @@ -28,10 +27,9 @@ Feature: CAMARA Common Artifact C02 - Test scenarios for phoneNumber errors And the response property "$.message" contains a user friendly text - # This scenario may happen e.g. with 2-legged access tokens, which do not identify a single phone number. @{{feature_identifier}}_C02.02_phone_number_not_found Scenario: Phone number not found - Given the header "Authorization" is set to a valid access which does not identifiy a single phone number + Given the header "Authorization" is set to a valid access which does not identify a single phone number And the request body property "$.phoneNumber" is compliant with the schema but does not identify a valid phone number When the HTTP "POST" request is sent Then the response status code is 404 @@ -45,15 +43,15 @@ Feature: CAMARA Common Artifact C02 - Test scenarios for phoneNumber errors Given the header "Authorization" is set to a valid access token identifying a phone number And the request body property "$.phoneNumber" is set to a valid phone number When the HTTP "POST" request is sent - Then the response status code is 403 + Then the response status code is 422 And the response property "$.status" is 422 And the response property "$.code" is "UNNECESSARY_IDENTIFIER" And the response property "$.message" contains a user friendly text - @{{feature_identifier}}_C02.04_unidentifiable_device + @{{feature_identifier}}_C02.04_missing_phone_number Scenario: Phone number not included and cannot be deducted from the access token - Given the header "Authorization" is set to a valid access which does not identifiy a single phone number + Given the header "Authorization" is set to a valid access which does not identify a single phone number And the request body property "$.phoneNumber" is not included When the HTTP "POST" request is sent Then the response status code is 422 @@ -62,7 +60,7 @@ Feature: CAMARA Common Artifact C02 - Test scenarios for phoneNumber errors And the response property "$.message" contains a user friendly text - # When the service is only offered to certain type of subcriptions, e.g. IoT, , B2C, etc + # When the service is only offered to certain type of subscriptions, e.g. IoT, , B2C, etc @{{feature_identifier}}_C02.05_phone_number_not_supported Scenario: Service not available for the phone number Given that the service is not available for all phone numbers commercialized by the operator From 9f05b73f628c969b3b32dbd7b551a4c362b2714b Mon Sep 17 00:00:00 2001 From: Jose Luis Urien Date: Wed, 4 Dec 2024 17:53:30 +0100 Subject: [PATCH 3/5] Extended clarifications --- artifacts/testing/C01-device-errors.feature | 15 ++++++++------- .../testing/C02-phoneNumber-errors.feature | 18 +++++++++--------- 2 files changed, 17 insertions(+), 16 deletions(-) diff --git a/artifacts/testing/C01-device-errors.feature b/artifacts/testing/C01-device-errors.feature index 02d7314c..16e25ce8 100644 --- a/artifacts/testing/C01-device-errors.feature +++ b/artifacts/testing/C01-device-errors.feature @@ -3,15 +3,16 @@ Feature: CAMARA Common Artifact C01 - Test scenarios for device errors Common error scenarios for POST operations with device as input either in the request body or implied from the access. - Artifact parameters (to be replaced by values according to the API operation): - - - {{feature_identifier}} - - This is not a complete feature but a collection of scenarios that can be applied - with minor modifications to test plans. + NOTES: + * This is not a complete feature but a collection of scenarios that can be applied with minor + modifications to test plans. Test plans would have to copy and adapt the scenarios as part of + their own feature files, along with other scenarios - These scenarios assume that other properties not explicitly mentioned in the scenario + * These scenarios assume that other properties not explicitly mentioned in the scenario are set by default to a valid value. This can be specified in the feature Background. + + * {{feature_identifier}} has to be substituted to the value corresponding to the feature file where + these scenarios are included. # Error scenarios for management of input parameter device diff --git a/artifacts/testing/C02-phoneNumber-errors.feature b/artifacts/testing/C02-phoneNumber-errors.feature index eeb4280a..a0352fea 100644 --- a/artifacts/testing/C02-phoneNumber-errors.feature +++ b/artifacts/testing/C02-phoneNumber-errors.feature @@ -1,18 +1,18 @@ Feature: CAMARA Common Artifact C02 - Test scenarios for phoneNumber errors Common error scenarios for POST operations with phoneNumber as input either in the request - body or implied from the access. - - Artifact parameters (to be replaced by values according to the API operation): + body or implied from the access - - {{feature_identifier}} - - This is not a complete feature but a collection of scenarios that can be applied - with minor modifications to test plans. + NOTES: + * This is not a complete feature but a collection of scenarios that can be applied with minor + modifications to test plans. Test plans would have to copy and adapt the scenarios as part of + their own feature files, along with other scenarios - These scenarios assume that other properties not explicitly mentioned in the scenario + * These scenarios assume that other properties not explicitly mentioned in the scenario are set by default to a valid value. This can be specified in the feature Background. - + + * {{feature_identifier}} has to be substituted to the value corresponding to the feature file where + these scenarios are included. # Error scenarios for management of input parameter phoneNumber From 91183a59e02327d1f9158c556fe75c89edd27090 Mon Sep 17 00:00:00 2001 From: Jose Luis Urien Date: Thu, 5 Dec 2024 11:52:56 +0100 Subject: [PATCH 4/5] Update artifacts/testing/C02-phoneNumber-errors.feature MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Pedro Díez García --- artifacts/testing/C02-phoneNumber-errors.feature | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/artifacts/testing/C02-phoneNumber-errors.feature b/artifacts/testing/C02-phoneNumber-errors.feature index a0352fea..b967e00c 100644 --- a/artifacts/testing/C02-phoneNumber-errors.feature +++ b/artifacts/testing/C02-phoneNumber-errors.feature @@ -4,7 +4,8 @@ Feature: CAMARA Common Artifact C02 - Test scenarios for phoneNumber errors body or implied from the access NOTES: - * This is not a complete feature but a collection of scenarios that can be applied with minor + * This is not a complete feature but a collection of scenarios that can be applied with minor + modifications to test plans. Test plans would have to copy and adapt the scenarios as part of their own feature files, along with other scenarios From be3f2013712f9c1228191ff9efa9d0d661457060 Mon Sep 17 00:00:00 2001 From: Jose Luis Urien Date: Thu, 5 Dec 2024 11:53:04 +0100 Subject: [PATCH 5/5] Update artifacts/testing/C01-device-errors.feature MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Pedro Díez García --- artifacts/testing/C01-device-errors.feature | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/artifacts/testing/C01-device-errors.feature b/artifacts/testing/C01-device-errors.feature index 16e25ce8..ce112b68 100644 --- a/artifacts/testing/C01-device-errors.feature +++ b/artifacts/testing/C01-device-errors.feature @@ -4,7 +4,8 @@ Feature: CAMARA Common Artifact C01 - Test scenarios for device errors body or implied from the access. NOTES: - * This is not a complete feature but a collection of scenarios that can be applied with minor + * This is not a complete feature but a collection of scenarios that can be applied with minor + modifications to test plans. Test plans would have to copy and adapt the scenarios as part of their own feature files, along with other scenarios