How to iterate through multiple pages of an API call

[wheaton3000]

I'm trying to create a Mirth process to consume a backlog of data from an API. This API uses pagination and only 100 records can be obtained at a time. So for a specific time range, there could be 1000s of pages to iterate through. I currently have a process setup to go through one page at a time but it it way too time consuming to manually edit the Mirth channel for each page of records.

In the source transformer I am generating the JSON request (end, start and trackingId are defined in a different part of this code):

var str = JSON.stringify({
    'content': {
	    	'pageNumber': 0,
		'endDate': end + '',
		'pageSize': 100,
		'startDate': start + ''
    },
     'trackingId': trackingId + ''
});

channelMap.put("varJSON",str);

The first destination I have is an HTTP sender to send the generated JSON to the API. In this destination the Content is: ${varJSON}

The response I have for this destination is:

var str = JSON.stringify(msg);

channelMap.put("varChannel",channelName);
channelMap.put('varJSONResponseLen',msg.length);
channelMap.put('varTotalElements',msg['content']['totalElements']);

channelMap.put('varJSONResponse',JSON.stringify(msg['content']['notifications']));

The second destination is a Channel Writer that has ${varJSONResponse} as the template. This destination/channel formats the data a bit and then inserts it into a database, so I don't think anything needs to change with this channel.

[pacmano1]

A few options - you can store the next page index (so to speak) in the globalChannelMap (not the channelMap) and source that value in your JSON from that value. In the deploy script you can initialize that value.

You can also send a message back in to your current channel with the next JSON to return that has the new index.

[pacmano1]

oh.. obviously you can just call the endpoint natively in javascript/java and loop there.

[wheaton3000]

thanks for all of your suggestions. i'm not that proficient in Mirth and programming in general, so out of these suggestions, which would be the easiest to implement and where do I start?

[kirbykn2]

To efficiently consume the backlog of data from the API with pagination in Mirth Connect, you can automate the process by implementing a loop to handle the pagination within the Mirth channel. Here's a high-level approach you can take to achieve this: Step 1: Initialize Variables In the source transformer, you should initialize the pageNumber variable to start at 0 and store the API response details like totalElements in a global map. javascript Copy code var pageNumber = 0;var continuePaging = true; // Set this to true initially to start the loop globalMap.put('totalElements', 0); globalMap.put('processedElements', 0); Step 2: Update the JSON Request with Pagination Modify the JSON request to include the pageNumber and update it dynamically with each iteration. javascript Copy code var str = JSON.stringify({ 'content': { 'pageNumber': pageNumber, 'endDate': end + '', 'pageSize': 100, 'startDate': start + '' }, 'trackingId': trackingId + '' }); channelMap.put("varJSON", str); Step 3: Send the HTTP Request and Handle Response In the HTTP sender destination, after receiving the response, update the variables related to pagination and check if there are more pages to process. javascript Copy code var response = JSON.parse(msg); // Assuming the API response is in JSON format var totalElements = response['content']['totalElements'];var processedElements = globalMap.get('processedElements') + response['content']['notifications'].length; globalMap.put('totalElements', totalElements); globalMap.put('processedElements', processedElements); if (processedElements < totalElements) { pageNumber++; continuePaging = true; } else { continuePaging = false; } globalMap.put('pageNumber', pageNumber); globalMap.put('continuePaging', continuePaging); channelMap.put('varJSONResponse', JSON.stringify(response['content']['notifications'])); Step 4: Loop Until All Pages are Processed Add a filter or a step to repeat the HTTP sender destination if there are more pages to process. - Use a JavaScript filter to check if continuePaging is true. javascript Copy code return globalMap.get('continuePaging') == true; - If true, loop back to the HTTP sender. If false, the loop terminates, and the process can move to the next task. Step 5: Handle the Results Your existing Channel Writer destination should handle the processing of each batch of notifications as it currently does. Step 6: Configure the Loop Mechanism Ensure your Mirth channel's looping is correctly configured to reprocess until all pages are fetched. This approach allows your Mirth channel to automatically iterate through all pages without manual intervention. The pageNumber is automatically incremented, and the loop continues until all records are fetched. https://chatgpt.com/g/g-XG1Dzv32o-interop-guru/c/dd8543bb-1ff2-47e2-b004-395f9baed966

…

On Thu, Aug 1, 2024 at 4:47 PM wheaton3000 ***@***.***> wrote: thanks for all of your suggestions. i'm not that proficient in Mirth and programming in general, so out of these suggestions, which would be the easiest to implement and where do I start? — Reply to this email directly, view it on GitHub <#6274 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APRXWD63S4NYTUFPDH7RQ4TZPKNF5AVCNFSM6AAAAABL2ZCFG6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMRRGY2TSNI> . You are receiving this because you are subscribed to this thread.Message ID: <nextgenhealthcare/connect/repo-discussions/6274/comments/10216595@ github.com>

-- Best, Kirby Knight | 231.735.4650 | ***@***.***

[wheaton3000]

When I first tried this, it was complaining about 'pageNumber' not being defined in step 3. I defined it in a few places and the error went away, but it's still only processing 1 page.

[kirbykn2]

This was untested and the link to the ChatGPT conversation is in my last post.

…

On Fri, Aug 2, 2024 at 4:59 PM wheaton3000 ***@***.***> wrote: When I first tried this, it was complaining about 'pageNumber' not being defined in step 3. I defined it in a few places and the error went away, but it's still only processing 1 page. — Reply to this email directly, view it on GitHub <#6274 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APRXWD7OOR57PHXDDMAM35DZPPXMFAVCNFSM6AAAAABL2ZCFG6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMRSG42DSNY> . You are receiving this because you commented.Message ID: ***@***.*** com>

-- Best, Kirby Knight | 231.735.4650 | ***@***.***