How does the Message Storage slider actually affect destination chains?

[Coooz]

Hi,

From the NextGen Connect user guide I learn that destination queueing is not supported in Raw, Metadata and Disabled storage modes. I wanted to test this since I am asked to investigate how we can keep the database as small as possible - but still workable.
So I made a simple channel with four File Writer destinations: the first three write and append to a certain file; the fourth reads the file and logs its contents.
Test 1: If all destinations are part of the same chain, it works as described.
Test 2: If the fourth destination does not wait for the third, I get the expected error that the file does not exist.
Test 3: If I make the fourth destination wait for the third again and set the message storage slider to Raw, I would also expect the error that the file does not exist, since destination queueing is not supported. This is not the case however: I get the Test 1 result instead.

So, what exactly is meant by "destination queueing is not supported" here?
I use Mirth Connect 3.11.
Thanks in advance!

[michaeljeffreycollins]

HI Coooz,
Destination Chaining means order of operations and concurrent operations.

Test 1: They 4 destinations are write, write, write, read. Of course it works
Test 2: Destination 1 and 4 trigger at the same time. Read and write. The file isn't there and you get the expected failure.
Test 3: I don't even understand your sentence. RAW has nothing on Destination chaining.

RAW vs Production vs Development takes into account the durable message processing and metadata storage on the message and process status. It doesn't even play into your scenario.

Mirth receives a message ->
1.Stores the data dump as bare text.
2. Saves off attachments.
3. Stores the XML Transformation from whatever message type it
4. Stores the encoded XML based on Transformations ( If it doesn't get filtered ).
5. Encode and Store the message as the Outgoing message type.
6. Possibly reattach the Attachments
7. Repeat entire process 1 -> 6 FOR EACH DESTINATION

RAW storage setting gets rid of a bunch of the database writes so you don't have an audit of each step of transformation of the message.

Now you specifically reference destination queuing isn't happening. No process occurs on D4 until D3 is done with chaining. There is no queue used here. But let me give you an example where Destination queuing would matter.

HL7 ADT feed to System X. This can even happen on a 1 Destination channel. System X is down. If D queues are turned on and that option is selected, the channel will continue to attempt to process the message and following messages so that when System X comes back up they will file to the system. But if set to RAW and no D queues exist when a message fails to send. That is it. You have to manually reprocess that message at a later time when the system comes back up. There is no wait and try again.

Was this helpful? I feel like I am not addressing your real question. The whole destination queue is tangent to RAW vs Prod vs Dev storage settings.

[Coooz]

@tonygermano
That is a valuable addition indeed. I appreciate it.

[Coooz]

I wouldn't have asked my first question if the process was called "Message queueing" or "Message queueing (Destination side)" instead of "Destination queueing". No actual destination is ever queued and I can't even picture what that would look like; the closest I get to a destination queue is a destination chain.

[tonygermano]

Halted means the channel is forcibly stopped before it is ready. If you try to stop a channel while it is in the middle of processing, it will go to a stopping state. Until the channel reaches a stopped state, you will be presented with the option to halt the channel, which will attempt to interrupt the thread that is still processing. A server crash could also stop a channel while in the middle of processing a message.

I think the messages affected by this would be in a RECEIVED state, as opposed to QUEUED if the message had been intentionally put into a destination queue. When the channel restarts, message recovery will attempt to reprocess those messages that never finished, even if you don't have the queue enabled.

The way that most destination connectors are implemented is that they always return a SENT or QUEUED status after attempting to send the message. If you do not have the queue enabled for that connector, then the QUEUED statuses are converted to ERROR. this is what would happen in the scenario where remote System X is down and you do not have the destination queue enabled. The message will error because it was unable to send. Any reprocessing attempts would need to be done manually. If you are not at least storing the Raw content, then reprocessing is impossible. If you manually reprocess, then the entire message (including the source connector) needs to reprocess.

[michaeljeffreycollins]

Ok, first there is Source Queuing. If Source queuing is turned on you can set Mirth to automaticly send a CA/AA response as soon as the machine receives the message. Destination queuing works on each of the destination connectors and can be turned on/off per connector. If a given system is down it can mess up the entire processing chain. But when queuing is turned on destinations don't finish processing before the queue continues. But when the system comes back up the process will finally complete.

Halt is an interrupt command sent to a channel that is hung. They non functioning states of a channel are not deployed, Deployed + stopped, and deployed + pause.

Durable message processing deals with saving off states and transformations of messages and responses so in the case of a system crash Mirth can pick up at step 24 out of 35 (arbitrary #s here as an example) instead of having to start all over back at step 1.

Your primary goal is a super small database? How often and how soon do you erase message content, message metadata, and vacuum the database?

[Coooz]

Thank you, @tonygermano and @michaeljeffreycollins.
This is all much clearer now. I can continue from here.

@michaeljeffreycollins: "Your primary goal is a super small database? How often and how soon do you erase message content, message metadata, and vacuum the database?"
Metadata and content should be kept for five days, and with that condition in mind I was asked to explore the possibilities here. We will probably shrink the database every now and then if it gets too big.