Some fields are lost when a field.set processor is used

[hariso]

Bug description

When the field.set processor is used, then some fields can be lost, like in the pipeline below that uses a generator source, field.set to add a new field, and a file destination.

When the field.set processor is not present, then the generated records have the fields as expected.

Steps to reproduce

1. Run Conduit with this pipeline:

version: "2.2"
pipelines:
  - id: example-pipeline
    status: running
    name: "generator-to-file"
    connectors:
      - id: example-source
        type: source
        plugin: "generator"
        settings:
          format.options.airline: 'string'
          format.options.scheduledDeparture: 'time'
          format.type: 'structured'
          rate: 1
      - id: example-destination
        type: destination
        plugin: "file"
        settings:
          path: './destination.txt'

2. Check contents of destination.txt with tail -f destination.txt | jq. All the records will look similar to this:

{
      "airline": "demicanon",
      "scheduledDeparture": "2024-11-05T13:57:19.436001Z"
}

3. Stop Conduit, change the pipeline include a field.set processor, like below:

version: "2.2"
pipelines:
  - id: example-pipeline
    status: running
    name: "generator-to-file"
    processors:
      - id: add-city
        plugin: "field.set"
        settings:
          field: ".Payload.After.city"
          value: "Bedrock"
    connectors:
      - id: example-source
        type: source
        plugin: "generator"
        settings:
          format.options.airline: 'string'
          format.options.scheduledDeparture: 'time'
          format.type: 'structured'
          rate: 1
      - id: example-destination
        type: destination
        plugin: "file"
        settings:
          path: './destination.txt'

4. Run Conduit. Check contents of destination.txt with tail -f destination.txt | jq. All the records will look similar to this:

{
      "airline": "rhombogenous",
      "scheduledDeparture": "1970-01-01T00:00:00.000007Z"
    }

Expected result:

Field scheduledDeparture has a non-zero value, city is present and set to Bedrock.

Actual result:

Field scheduledDeparture has a zero value, city is not present.

Version

Commit 4c3c375

[lovromazgon]

I can't really reproduce it, this is the record I get in the output:

{
  "position": "MjE=",
  "operation": "create",
  "metadata": {
    "conduit.source.connector.id": "example-pipeline:example-source",
    "opencdc.createdAt": "1730813554815653000",
    "opencdc.payload.schema.subject": "example-pipeline:example-source:payload",
    "opencdc.payload.schema.version": "2"
  },
  "key": "eXVjYQ==",
  "payload": {
    "before": null,
    "after": {
      "airline": "cuadra",
      "city": "Bedrock",
      "scheduledDeparture": "2024-11-05T13:32:34.815657Z"
    }
  }
}

[hariso]

Yeah, I changed the steps a bit, thinking they could be simplified. I'll update the steps in a bit to what they were (and they consistently reproduce this behavior). It looks like the issue is in our schema registry and the way we generate IDs.