Invert the filter

Usually a filter function only includes elements if the filter condition evaluates to true. Let's do that. Also, update various docs and examples. Signed-off-by: Nic Cope <[email protected]>
crossplane-contrib · Feb 18, 2024 · fa8c11a · fa8c11a
1 parent 3e66ea5
commit fa8c11a
Show file tree

Hide file tree

Showing 9 changed files with 185 additions and 49 deletions.
diff --git a/README.md b/README.md
@@ -1,8 +1,19 @@
 # function-cel-filter
 [![CI](https://github.com/negz/function-cel-filter/actions/workflows/ci.yml/badge.svg)](https://github.com/negz/function-cel-filter/actions/workflows/ci.yml)
 
-A [composition function][functions] that can filter desired composed resources
-produced by previous functions in the pipeline using CEL expressions.
+A [composition function][functions] that [filters][filter] matching composed
+resources using [CEL expressions][cel].
+
+Each filter:
+
+* Matches composed resources by name using a regular expression.
+* Specifies whether resources should be included using a CEL expression.
+
+If a filter's CEL expression evaluates to true, Crossplane creates the matching
+composed resources.
+
+Filters only apply to matching composed resources. The function doesn't filter
+composed resources that don't match a filter. 
 
 ```yaml
 apiVersion: apiextensions.crossplane.io/v1
@@ -12,29 +23,50 @@ metadata:
 spec:
   compositeTypeRef:
     apiVersion: example.crossplane.io/v1
-    kind: XR
+    kind: NoSQL
   mode: Pipeline
   pipeline:
-  # TODO(negz): Replace me with function-dummy.
-  - step: produce-composed-resources
+  - step: patch-and-transform
     functionRef:
-      name: some-composition-function
+      name: function-patch-and-transform
+    input:
+      apiVersion: pt.fn.crossplane.io/v1beta1
+      kind: Resources
+      resources:
+      - name: table
+        base:
+          apiVersion: dynamodb.aws.upbound.io/v1beta1
+          kind: Table
+          metadata:
+            name: crossplane-quickstart-database
+          spec:
+            forProvider:
+              region: "us-east-2"
+              writeCapacity: 1
+              readCapacity: 1
+              attribute:
+                - name: S3ID
+                  type: S
+              hashKey: S3ID
+      - name: bucket
+        base:
+          apiVersion: s3.aws.upbound.io/v1beta1
+          kind: Bucket
+          spec:
+            forProvider:
+              region: us-east-2
   - step: filter-composed-resources
     functionRef:
       name: function-cel-filter
     input:
       apiVersion: cel.fn.crossplane.io/v1beta1
       kind: Filters
       filters:
-      # Remove the composed resource named a-desired-composed-resource
-      # from the function pipeline if the XR has spec.widgets == 42.
-      - name: a-desired-composed-resource
-        expression: observed.composed.resource.spec.widgets == 42
+      # Only create the bucket if the XR's spec.export field is set to "S3".
+      - name: bucket
+        expression: observed.composite.resource.spec.export == "S3"
 ```
 
 [functions]: https://docs.crossplane.io/latest/concepts/composition-functions
-[go]: https://go.dev
-[function guide]: https://docs.crossplane.io/knowledge-base/guides/write-a-composition-function-in-go
-[package docs]: https://pkg.go.dev/github.com/crossplane/function-sdk-go
-[docker]: https://www.docker.com
-[cli]: https://docs.crossplane.io/latest/cli
+[cel]: https://github.com/google/cel-spec
+[filter]: https://en.wikipedia.org/wiki/Filter_(higher-order_function)
diff --git a/example/README.md b/example/README.md
@@ -13,13 +13,32 @@ $ go run . --insecure --debug
 $ crossplane beta render xr.yaml composition.yaml functions.yaml -r
 ---
 apiVersion: example.crossplane.io/v1
-kind: XR
+kind: NoSQL
 metadata:
   name: example-xr
 ---
-apiVersion: render.crossplane.io/v1beta1
-kind: Result
-message: I was run with input "Hello world"!
-severity: SEVERITY_NORMAL
-step: run-the-template
+apiVersion: dynamodb.aws.upbound.io/v1beta1
+kind: Table
+metadata:
+  annotations:
+    crossplane.io/composition-resource-name: table
+  generateName: example-xr-
+  labels:
+    crossplane.io/composite: example-xr
+  ownerReferences:
+  - apiVersion: example.crossplane.io/v1
+    blockOwnerDeletion: true
+    controller: true
+    kind: NoSQL
+    name: example-xr
+    uid: ""
+spec:
+  forProvider:
+    attribute:
+    - name: S3ID
+      type: S
+    hashKey: S3ID
+    readCapacity: 1
+    region: us-east-2
+    writeCapacity: 1
 ```
diff --git a/example/composition.yaml b/example/composition.yaml
@@ -5,21 +5,45 @@ metadata:
 spec:
   compositeTypeRef:
     apiVersion: example.crossplane.io/v1
-    kind: XR
+    kind: NoSQL
   mode: Pipeline
   pipeline:
-  # TODO(negz): Replace me with function-dummy.
-  - step: produce-composed-resources
+  - step: patch-and-transform
     functionRef:
-      name: some-composition-function
+      name: function-patch-and-transform
+    input:
+      apiVersion: pt.fn.crossplane.io/v1beta1
+      kind: Resources
+      resources:
+      - name: table
+        base:
+          apiVersion: dynamodb.aws.upbound.io/v1beta1
+          kind: Table
+          metadata:
+            name: crossplane-quickstart-database
+          spec:
+            forProvider:
+              region: "us-east-2"
+              writeCapacity: 1
+              readCapacity: 1
+              attribute:
+                - name: S3ID
+                  type: S
+              hashKey: S3ID
+      - name: bucket
+        base:
+          apiVersion: s3.aws.upbound.io/v1beta1
+          kind: Bucket
+          spec:
+            forProvider:
+              region: us-east-2
   - step: filter-composed-resources
     functionRef:
       name: function-cel-filter
     input:
       apiVersion: cel.fn.crossplane.io/v1beta1
       kind: Filters
       filters:
-      # Remove the desired composed resource named a-desired-composed-resource
-      # from the function pipeline if the XR has spec.widgets == 42.
-      - name: a-desired-composed-resource
-        expression: observed.composed.resource.spec.widgets == 42
+      # Only create the bucket if the XR's spec.export field is set to "S3".
+      - name: bucket
+        expression: observed.composite.resource.spec.export == "S3"
diff --git a/example/functions.yaml b/example/functions.yaml
@@ -2,10 +2,18 @@
 apiVersion: pkg.crossplane.io/v1beta1
 kind: Function
 metadata:
-  name: function-template-go
+  name: function-patch-and-transform
+spec:
+  package: xpkg.upbound.io/crossplane-contrib/function-patch-and-transform:v0.3.0
+
+---
+apiVersion: pkg.crossplane.io/v1beta1
+kind: Function
+metadata:
+  name: function-cel-filter
   annotations:
     # This tells crossplane beta render to connect to the function locally.
     render.crossplane.io/runtime: Development
 spec:
   # This is ignored when using the Development runtime.
-  package: function-template-go
+  package: function-cel-filter
diff --git a/example/xr.yaml b/example/xr.yaml
@@ -1,6 +1,8 @@
 # Replace this with your XR!
 apiVersion: example.crossplane.io/v1
-kind: XR
+kind: NoSQL
 metadata:
   name: example-xr
-spec: {}
+spec:
+  # Change this to "S3" to include the bucket.
+  export: "Disabled"
diff --git a/fn.go b/fn.go
@@ -61,12 +61,12 @@ func (f *Function) RunFunction(_ context.Context, req *fnv1beta1.RunFunctionRequ
 
 		// Evaluate this filter's CEL expression.
 		expr = in.Filters[i].Expression
-		filter, err := Evaluate(f.env, req, in.Filters[i].Expression)
+		include, err := Evaluate(f.env, req, in.Filters[i].Expression)
 		if err != nil {
 			response.Fatal(rsp, errors.Wrapf(err, "cannot evaluate CEL expression %q for filter %d", expr, i))
 			return rsp, nil
 		}
-		celexps[i] = filter
+		celexps[i] = include
 	}
 
 	for name := range rsp.GetDesired().GetResources() {
@@ -82,12 +82,12 @@ func (f *Function) RunFunction(_ context.Context, req *fnv1beta1.RunFunctionRequ
 				continue
 			}
 
-			if filter := celexps[i]; !filter {
-				log.Debug("Not filtering desired composed resource: CEL expression did not evaluate to true")
+			if include := celexps[i]; include {
+				log.Debug("Not filtering desired composed resource: CEL expression evaluated to true")
 				continue
 			}
 
-			log.Info("Filtering desired composed resource")
+			log.Info("Filtering desired composed resource: CEL expression evaluated to false")
 			delete(rsp.GetDesired().GetResources(), name)
 		}
 	}

diff --git a/fn_test.go b/fn_test.go
@@ -49,32 +49,42 @@ func TestRunFunction(t *testing.T) {
 			},
 		},
 		"BasicFilter": {
-			reason: "The function should filter a resource if the name matches and the CEL expression evaluates to true",
+			reason: "If the filter name matches a resource, it should only be included if the CEL expression evaluates to true",
 			args: args{
 				req: &fnv1beta1.RunFunctionRequest{
 					Meta: &fnv1beta1.RequestMeta{Tag: "hello"},
+					// The first filter matches the resources but it evaluates
+					// to true, so it won't filter the resources. However the
+					// second filter will, because it also matches the resources
+					// and evaluates to false.
 					Input: resource.MustStructJSON(`{
 						"apiVersion": "filters.cel.crossplane.io/v1beta1",
 						"kind": "Filters",
 						"filters": [
 							{
-								"name": "matching-resource",
-								"expression": "observed.composite.resource.spec.widgets == 42"
+								"name": "matching-.*",
+								"expression": "observed.composite.resource.spec.watchers == 42"
+							},
+							{
+								"name": "matching-.*",
+								"expression": "observed.composite.resource.spec.widgets == 88"
 							}
 						]
 					}`),
 					Observed: &fnv1beta1.State{
 						Composite: &fnv1beta1.Resource{
 							Resource: resource.MustStructJSON(`{
 								"spec": {
+									"watchers": 42,
 									"widgets": 42
 								}
 							}`),
 						},
 					},
 					Desired: &fnv1beta1.State{
 						Resources: map[string]*fnv1beta1.Resource{
-							"matching-resource":     {},
+							"matching-resource-a":   {},
+							"matching-resource-b":   {},
 							"non-matching-resource": {},
 						},
 					},
@@ -85,7 +95,8 @@ func TestRunFunction(t *testing.T) {
 					Meta: &fnv1beta1.ResponseMeta{Tag: "hello", Ttl: durationpb.New(response.DefaultTTL)},
 					Desired: &fnv1beta1.State{
 						Resources: map[string]*fnv1beta1.Resource{
-							// matching-resource was filtered.
+							// matching-resource-a was filtered.
+							// matching-resource-b was filtered.
 							"non-matching-resource": {},
 						},
 					},

diff --git a/input/v1beta1/input.go b/input/v1beta1/input.go
@@ -30,10 +30,27 @@ type Filters struct {
 // A Filter can be used to filter a desired composed resource produced by a
 // previous function in the pipeline.
 type Filter struct {
-	// Name of the desired composed resource this filter should match. Supports
-	// regular expressions. Only the first matching filter will apply.
+	// Name of the desired composed resource(s) this filter should match.
+	//
+	// Use regular expressions to match multiple resources. Expressions are
+	// automatically prefixed with ^ and suffixed with $. For example 'buck.*'
+	// becomes '^buck.*$'. See https://github.com/google/re2/wiki/Syntax.
 	Name string `json:"name"`
 
 	// Expression is a CEL expression. See https://github.com/google/cel-spec.
+	// The following top-level variables are available to the expression:
+	//
+	// * observed
+	// * desired
+	// * context
+	//
+	// Example expressions:
+	//
+	// * observed.composite.resource.spec.widgets == 42
+	// * observed.resources['composed'].connection_details['user'] == 'admin'
+	// * desired.resources['composed'].resource.spec.widgets == 42
+	//
+	// See the RunFunctionRequest protobuf message for schema details.
+	// https://buf.build/crossplane/crossplane/docs/main:apiextensions.fn.proto.v1beta1
 	Expression string `json:"expression"`
 }
diff --git a/package/input/cel.fn.crossplane.io_filters.yaml b/package/input/cel.fn.crossplane.io_filters.yaml
@@ -19,7 +19,7 @@ spec:
   - name: v1beta1
     schema:
       openAPIV3Schema:
-        description: Filters can be used to provide input to this Function.
+        description: Filters can be used to filter desired composed resources.
         properties:
           apiVersion:
             description: |-
@@ -41,12 +41,35 @@ spec:
                 previous function in the pipeline.
               properties:
                 expression:
-                  description: Expression is a CEL expression. See https://github.com/google/cel-spec.
+                  description: |-
+                    Expression is a CEL expression. See https://github.com/google/cel-spec.
+                    The following top-level variables are available to the expression:
+
+
+                    * observed
+                    * desired
+                    * context
+
+
+                    Example expressions:
+
+
+                    * observed.composite.resource.spec.widgets == 42
+                    * observed.resources['composed'].connection_details['user'] == 'admin'
+                    * desired.resources['composed'].resource.spec.widgets == 42
+
+
+                    See the RunFunctionRequest protobuf message for schema details.
+                    https://buf.build/crossplane/crossplane/docs/main:apiextensions.fn.proto.v1beta1
                   type: string
                 name:
                   description: |-
-                    Name of the desired composed resource this filter should match. Supports
-                    regular expressions. Only the first matching filter will apply.
+                    Name of the desired composed resource(s) this filter should match.
+
+
+                    Use regular expressions to match multiple resources. Expressions are
+                    automatically prefixed with ^ and suffixed with $. For example 'buck.*'
+                    becomes '^buck.*$'. See https://github.com/google/re2/wiki/Syntax.
                   type: string
               required:
               - expression