Unmarshing DynamoStream events

[maxm450]

Describe the bug

TS types seems to cause an issue when unmarshing a dynamoStream

Argument of type '{ [key: string]: AWSLambda.AttributeValue; }' is not assignable to parameter of type '{ [key: string]: import("/node_modules/@aws-sdk/client-dynamodb/dist/types/models/models_0").AttributeValue; }'.

Your environment

SDK version number

@aws-sdk/[email protected]

Steps to reproduce

import { DynamoDBStreamEvent } from 'aws-lambda'
import { unmarshall } from "@aws-sdk/util-dynamodb"

export const handler =  async (event) => {
  for (const record of event.Records) {
    if (record.dynamodb?.NewImage) {
      const streamItem = record.dynamodb?.NewImage
      const item = unmarshall(streamItem)
      console.log(item)
    }
  }
}

Observed behavior

receiving typing issue for AttributeValue type.

Expected behavior

should be able to unmarshall a Dynamo object, wherever the object comes from a stream event or a getItem action from the DynamoClient

[monken]

unmarshall will also not convert binary attributes properly. The DynamoDB stream will provide a base64 encoded presentation of the binary attribute and unmarshall does not base64-decode the value.

[sekhavati]

I'm still seeing this issue in 3.34.0.

Interestingly there are no type errors if my package only contains @aws-sdk/util-dynamodb. But as soon as I install @aws-sdk/client-dynamodb the aforementioned TypeScript errors pop up.

From my brief testing with unmarshall it behaved as expected prior to installing @aws-sdk/client-dynamodb , so as a workaround I've set a @ts-ignore on the problematic line of code for now.

[ajredniwja]

Hey everyone, apologies the issue fell out of queue. A lot of changes and features were added to the latest package. Is it still an persisting issue with the latest version. I was able to use unmarshall correctly used: https://www.npmjs.com/package/@aws-sdk/lib-dynamodb.

[faridnsh]

Yes, this is still an issue as the types are not compatible. More info about it in DefinitelyTyped/DefinitelyTyped#51331

[nikiitat]

Any updates on this one?
I think this is also related to: DefinitelyTyped/DefinitelyTyped#51331

[obones]

unmarshall will also not convert binary attributes properly. The DynamoDB stream will provide a base64 encoded presentation of the binary attribute and unmarshall does not base64-decode the value.

I'm facing the same behavior here and I'm not sure what to do here.
Should this be reported in a separate issue?

[jimmyn]

Any update on this? I'm facing the same issue with the latest version

[zshzbh]

Hey everyone,

I want to provide this workaround so you can decode the binary attributes.

import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import { unmarshall } from "@aws-sdk/util-dynamodb";

const dynamoClient = new DynamoDBClient({ region: "us-west-2" });

function customUnmarshall(item) {
  const unmarshalled = unmarshall(item);
  
  for (const [key, value] of Object.entries(unmarshalled)) {
    if (item[key] && item[key].B) {
      // Decode the base64 string to a UTF-8 string
      unmarshalled[key] = Buffer.from(item[key].B, 'base64').toString('utf-8');
    }
  }
  
  return unmarshalled;
}

export const handler = async (event) => {
  for (const record of event.Records) {
    if (record.dynamodb?.NewImage) {
      const streamItem = record.dynamodb.NewImage;
      const item = customUnmarshall(streamItem);
      console.log("Unmarshalled item (including decoded binary attributes):", item);
      
      for (const [key, value] of Object.entries(item)) {
        if (typeof value === 'string' && item[key] && item[key].B) {
          console.log(`Decoded binary attribute ${key}:`, value);
        }
      }
    }
  }
};

[zshzbh]

Using the workaround above, I used the following test case of DynamoDB object and it's working :

{
  "Records": [
    {
      "eventID": "1",
      "eventName": "INSERT",
      "eventVersion": "1.0",
      "eventSource": "aws:dynamodb",
      "awsRegion": "us-west-2",
      "dynamodb": {
        "Keys": {
          "Id": {
            "N": "101"
          }
        },
        "NewImage": {
          "Id": {
            "N": "101"
          },
          "Name": {
            "S": "John Doe"
          },
          "BinaryData": {
            "B": "SGVsbG8gV29ybGQ="
          }
        },
        "SequenceNumber": "111",
        "SizeBytes": 26,
        "StreamViewType": "NEW_AND_OLD_IMAGES"
      },
      "eventSourceARN": "arn:aws:dynamodb:us-west-2:123456789012:table/YourTableName/stream/2023-05-11T00:00:00.000"
    },
    {
      "eventID": "2",
      "eventName": "MODIFY",
      "eventVersion": "1.0",
      "eventSource": "aws:dynamodb",
      "awsRegion": "us-west-2",
      "dynamodb": {
        "Keys": {
          "Id": {
            "N": "102"
          }
        },
        "NewImage": {
          "Id": {
            "N": "102"
          },
          "Name": {
            "S": "Jane Smith"
          },
          "BinaryData": {
            "B": "SGVsbG8gQWdhaW4="
          }
        },
        "SequenceNumber": "222",
        "SizeBytes": 28,
        "StreamViewType": "NEW_AND_OLD_IMAGES"
      },
      "eventSourceARN": "arn:aws:dynamodb:us-west-2:123456789012:table/YourTableName/stream/2023-05-11T00:00:00.000"
    }
  ]
}

This is the log I got "

START RequestId: 1c248220-6a2f-49e0-a15b-0d32ac45d714 Version: $LATEST
2024-10-29T16:03:16.831Z	1c248220-6a2f-49e0-a15b-0d32ac45d714	INFO	Unmarshalled item (including decoded binary attributes): { Id: 101, Name: 'John Doe', BinaryData: 'Hello World' }
2024-10-29T16:03:16.833Z	1c248220-6a2f-49e0-a15b-0d32ac45d714	INFO	Unmarshalled item (including decoded binary attributes): { Id: 102, Name: 'Jane Smith', BinaryData: 'Hello Again' }
END RequestId: 1c248220-6a2f-49e0-a15b-0d32ac45d714
REPORT RequestId: 1c248220-6a2f-49e0-a15b-0d32ac45d714	Duration: 303.34 ms	Billed Duration: 304 ms	Memory Size: 128 MB	Max Memory Used: 87 MB	

@jimmyn,
Could you please help me understand what issue you are facing now? Do you have any minimal code reproduction
and error msg?

Thanks!

[zshzbh]

Just talked with the team, it seems the issue is caused by the type mismatch.

Please vote or reply if you are encountering this issue with aws-lambda pkg installed

[jimmyn]

Just talked with the team, it seems the issue is caused by the type mismatch.

Please vote or reply if you are encountering this issue with aws-lambda pkg installed

Yes, this is exactly what happens.