Getting Connection Closed on New Connection Error when it was previously not generated

[ctruo]

Context

• OS, version, SKU and CPU architecture used: Windows Server 2022 Datacenter
• Application's .NET Target Framework : .NET Framework 4.6.2
• Device: Laptop
• SDK version used:
  Microsoft.Azure.Devices.Client 1.42.0
  Microsoft.Azure.Devices 1.39.0
  Microsoft.Azure.Devices.Shared 1.30.3

Description of the issue

We have an application that streams data to IoT Hub on a scheduled frequency. We have an update data functionality that allows us to send our backfilled data during a specified time frame. Currently with Microsoft.Azure.Devices.Client 1.42.0, our update data functionality is suddenly breaking every time we call it and we are seeing the 2 following exceptions generated:

_Type: IotHubException,
_Message: error(condition:com.microsoft:connection-closed-on-new-connection,description:,info:[]),
_Source: 'Microsoft.Azure.Devices.Client',

_Type: AmqpException,
_Message: ,
_Source: '',

The way our current update data functionality works is it creates a new connection while the scheduled frequency stream's connection is still existing. This may be the root of our issues, but we are surprised because everything was working fine when we last tested on 3/13/2024 and no changes were made since. Strangely, this behavior is not consistent across our IoT Hubs. Below is a table of the behavior for each of our IoT Hubs.

Tier	Creation Date	Behavior
Standard	11/21/2023	Fails
Standard	01/03/2024	Fails
Standard	03/06/2024	Fails
Standard	4/11/2024	Works
Standard	4/11/2024	Works
Basic	10/25/2023	Works

The Standard tier IoT Hubs created on 4/11/2024 were made to test when we found out the issue in our application. What is puzzling to us is that these 2 new IoT Hubs and our older Basic IoT Hub work fine, but our existing Standard tier IoT Hubs are running into this issue.

For now we have reverted Microsoft.Azure.Devices.Client to 1.18.1 which does not cause this issue, but there are side effect errors I discuss below.

Something we're also noticing in both 1.42.0 and 1.18.1 is when we connect using a connection string with a device in it (HostName=xxxxx.azure-devices.net;DeviceId=xxxxx;SharedAccessKey=xxxxx), our application will eventually generate exceptions after an inconsistent time and stop publishing data to our "older" Standard tier targets. For some reason we are using MQTT WebSocket Only for this specific connection case. I'm thinking it is because we have multiple device connections at the same and I believe MQTT is only supposed to have one. More clarification is appreciated.

The exceptions generated differ between these two versions:
v1.42.0:

_Type: IotHubCommunicationException,
_Message: Transient network error occurred, please retry.,
_Source: 'Microsoft.Azure.Devices.Client',

_Type: SocketException,
_Message: An existing connection was forcibly closed by the remote host,
_Source: 'mscorlib',

v1.18.1

_Type: TimeoutException,
_Message: Operation timeout expired.,
_Source: 'Microsoft.Azure.Devices.Client',

Any information and help would be greatly appreciated. Thanks!

Console log of the issue

Logs were collected when invoking the update data functionality that caused the exception to throw. Comparison of logs show no exceptions in v1.18.1.
Logs for v1.42.0 (exceptions are near the end): v1.42.0.txt
Logs for v1.18.1: v1.18.1.txt

[andyk-ms]

Confirming the version you used:
1.18.1 -> https://github.com/Azure/azure-iot-sdk-csharp/releases/tag/2018-10-9
1.42.0 -> https://github.com/Azure/azure-iot-sdk-csharp/releases/tag/2023-05-01
these are the versions in questions, correct?

In the log for 1.42.0, the error listed is AMQP but you seems to discuss a MQTT error. For the AMQP in the log, it indicates a retry-able error, do you have retry logic to handle this? Please elaborate on the error handling.
2024-04-16T02:03:39.7255428 [Microsoft-Azure-Devices-Device-Client-Enter] (RetryStrategyAdapter#31617841, RetryStrategyAdapter.ShouldRetry, (0, Microsoft.Azure.Devices.Client.Exceptions.IotHubCommunicationException: Exception of type 'Microsoft.Azure.Devices.Client.Exceptions.IotHubCommunicationException' was thrown.)).
_Type: AmqpException,
_Message: ,
_Source: '',
_HResult: [ -2146233088 ]

For the package, would you be able to update to the latest 1.42.3 with fixes to re-test?
https://github.com/Azure/azure-iot-sdk-csharp/releases/tag/2024-03-28

[ctruo]

Hi Andy. Thanks for your reply. Yes those are the versions.

We currently do not have retry logic to handle this. We only have a simple try-catch block. I can try to upgrade to 1.42.3 and test, but we do have some constraints when it comes to upgrades (sorry I cannot elaborate further). Ideally we would like to not have an upgrade, but if it is necessary we will see.

MQTT is only used when we connect using a connection string with DeviceId in it (HostName=xxxxx.azure.devices.net;DeviceId=xxxxx;SharedAccessKey=xxxxx). If we use a shared access level connection string with no DeviceId, AMQP is used so the logs you are seeing is that. We've changed the MQTT transport to AMQP for the time being.