AWS.DynamoDB.DocumentClient with DAX service scan & query are missing pagination usage/implementation

[BloodShop]

1. Trying to figure out how to imlement the pagination of scan and query using AWS.DynamoDB.DocumentClient with amazon-dax-client, cause for now this ain't returning the lastEvaludatedKey on the response.

2. Is there any easier way to decorate the DynamoDBClient with DAX so I won't need to change in each use case of DynamoDBClient to AWS.DynamoDB.DocumentClient instance?

setup:

import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import AWS from "aws-sdk";
import { CONFIG } from "../config";
const AmazonDaxClient = require('amazon-dax-client');

const daxClient = new AmazonDaxClient({
  endpoints: [CONFIG.DAX_ENDPOINT],
  maxRetries: 1,
});

export const ddbDocumentClient: AWS.DynamoDB.DocumentClient = new AWS.DynamoDB.DocumentClient({
  service: daxClient,
});

export default new DynamoDBClient({});

The scan operation:

import { AttributeValue, DocumentClient } from "aws-sdk/clients/dynamodb";
import { DB_TABLES } from "../constants/consts";
import { SomeEntity } from "../types";
import { ddbDocumentClient } from "./dynamo";

export const getAccessoryCentersDAL = async (): Promise<Array<SomeEntity>> => {
  let lastEvaluatedKey: Record<string, AttributeValue> | undefined;
  const itemsResult: Array<SomeEntity> = [];

  do {
    const params: DocumentClient.ScanInput = {
      TableName: DB_TABLES.SOME_TABLE,
      ExclusiveStartKey: lastEvaluatedKey,
    } as const;

    const { Items, LastEvaluatedKey } = await ddbDocumentClient.scan(params).promise();

    if (Items?.[0]) {
      itemsResult.concat(Items as Array<SomeEntity>);
    }

    lastEvaluatedKey = LastEvaluatedKey;
  } while (lastEvaluatedKey);

  return itemsResult;
};

Any suggetions?

[aBurmeseDev]

Hi @BloodShop - thanks for reaching out.

It appears that your questions revolve around services like DDB and DAX, rather than the AWS SDK itself. No worries though as I can internally communicate with the respective service teams to address your concerns.

Additionally, I noticed that your code utilizes the AWS SDK for JavaScript v2, which is currently in maintenance mode. It would be advisable to consider upgrading to the latest aws-sdk-js-v3 version.

[BloodShop]

Thank you for your help @aBurmeseDev . Please if you may provide example for best practices using daxClient for replacing ddbClient with node (Typescript would be great)

[aBurmeseDev]

I submitted an internal ticket to service team (ref: P157736270) for guidance and will post back when I hear back.

[aBurmeseDev]

Hi @BloodShop - I just heard back from service team member and please see their response addressed to your questions below.

1. Trying to figure out how to imlement the pagination of scan and query using AWS.DynamoDB.DocumentClient with amazon-dax-client, cause for now this ain't returning the lastEvaludatedKey on the response.

itemsResult.concat(Items as Array<SomeEntity>); line doesn’t actually modify itemsResult because concat() returns a new array, but it's not reassigned back to itemsResult. I believe you should use push() instead.

Test with concat():

items = []
for (var i=0 ; i<5 ; i++) {
    items.concat(i);
}
console.log(items)

### Output:

[]

Test with push():

items = []
for (var i=0 ; i<5 ; i++) {
    items.push(i);
}
console.log(items)

### Output:

[0, 1, 2, 3, 4]

Please let me know if you continue to see issues. If yes, we'll need logs from your side on what response you are receiving from DAX.

2. Is there any easier way to decorate the DynamoDBClient with DAX so I won't need to change in each use case of DynamoDBClient to AWS.DynamoDB.DocumentClient instance?

DynamoDBClient from "@aws-sdk/client-dynamodb" belongs to AWS JS SDK v3 but DAX doesn't support it yet. This would be addressed once DAX supports AWS JS SDK v3.

[BloodShop]

Hey @aBurmeseDev , thanks for your answer.

1. The example with the concat isn't the problem ofc, it was just a fast code generation.
   The thing is the there is no lastEvaluatedKey retuned in the response, so I cannot handle all the scan/query entities.

2. I haven't found any "average" documentation that shows how to implement dax with pagination using scan/query commands. On the other hand dynamoDbClient works amazing with the lastEvaluatedKey.

[aBurmeseDev]

@BloodShop - here's further response from service team:

DynamoDb returns LastEvaluateKey if we add explicit Limit in the params or if the response size exceed 1MB.

Refer below doc to understand more about the API contract and page limits as DAX always maintain the same contract -
https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Scan.html
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Scan.html

You can try adding Limit to your request params and check if you are able to get LastEvaluatedKey or not.
If not, then please share us the logs on what response you are receiving from DAX.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.