Application not pushing logs to SEQ post restart

[KalyS77]

Ours is a .NET 4.8 Windows Service which uses LurgleLogging to push log data into File and Seq. It was all working fine until last day it was stopped. Post starting the windows service application is not pushing logs into SEQ. We also have a Asp.Net web application which uses the same API key as the Windows service to ingest logs into SEQ and it works perfectly well even after the restart.

Any clue as to where we have to look for this would be helpful. There is an error logged in the SEQ ingest logs as below. But not sure whether that is related as all the other applications are able to push the logs to the SEQ.

Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: S. Path '', line 0, position 0.
at Newtonsoft.Json.JsonTextReader.ParseValue()
at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
at Seq.Server.Features.Apps.Inputs.InputChannel.Send(Input input, String clef)

[nblumhardt]

Hi! Not sure whether my email reply yesterday reached you, so copying it below:

Based on the ingestion log entries being unrelated to the HTTP API, it's most likely a client-side logging issue we need to track down, in the app itself.

The best debugging tactic will be to enable Serilog's SelfLog, which should still show most logging issues generated through Lurgle Logging.

This would go in the app's Program.cs right at the start of execution:

var file = File.CreateText(...);
Serilog.Debugging.SelfLog.Enable(TextWriter.Synchronized(file));

Where the text file goes would depend on your deployment environment. You can send the diagnostic messages somewhere else using the callback version instead:

Serilog.Debugging.SelfLog.Enable(message => /* do something to record message */);

I'd suggest only enabling this for a few minutes, just in case message volumes cause a lot of data to be created.

Best regards,
Nick

[KalyS77]

Thanks Nick, We are trying this. Thank you! ***@***.*** Sr. Application Developer | Architecture & Development | Unisys Mortgage Processing Unisys Australia | M : +61 481-262-914 | E : ***@***.******@***.***> 1G Homebush Bay Dr, Rhodes NSW 2138 ***@***.***<http://www.unisys.com/> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. [Grey_LI]<http://www.linkedin.com/company/unisys> [Grey_TW] <http://twitter.com/unisyscorp> [Grey_YT] <http://www.youtube.com/theunisyschannel> [Grey_FB] <http://www.facebook.com/unisyscorp> [Grey_Vimeo] <https://vimeo.com/unisys> [Grey_UB] <http://blogs.unisys.com/> From: Nicholas Blumhardt ***@***.***> Sent: Wednesday, 11 September 2024 10:11 AM To: datalust/seq-tickets ***@***.***> Cc: Kalyadan, Sathyajith ***@***.***>; Author ***@***.***> Subject: Re: [datalust/seq-tickets] Application not pushing logs to SEQ post restart (Discussion #2270) Hi! Not sure whether my email reply yesterday reached you, so copying it below: Based on the ingestion log entries being unrelated to the HTTP API, it's most likely a client-side logging issue we need to track down, in the app itself. The best Hi! Not sure whether my email reply yesterday reached you, so copying it below: Based on the ingestion log entries being unrelated to the HTTP API, it's most likely a client-side logging issue we need to track down, in the app itself. The best debugging tactic will be to enable Serilog's SelfLog, which should still show most logging issues generated through Lurgle Logging. This would go in the app's Program.cs right at the start of execution: var file = File.CreateText(...); Serilog.Debugging.SelfLog.Enable(TextWriter.Synchronized(file)); Where the text file goes would depend on your deployment environment. You can send the diagnostic messages somewhere else using the callback version instead: Serilog.Debugging.SelfLog.Enable(message => /* do something to record message */); I'd suggest only enabling this for a few minutes, just in case message volumes cause a lot of data to be created. Best regards, Nick — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/datalust/seq-tickets/discussions/2270*discussioncomment-10608049__;Iw!!JZxO7R1e!ul19Sj6riXSeQG0nXK-VmRFjwaPw9PWTiFCiqgQQgDa1urcT4r32IZiUn6x54z5ED9jENTEL4JrirJtCGM-L-MAsgsJdhW_O$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/BLFOEVMPBRZWFPVWBPVZDCLZV6DBJAVCNFSM6AAAAABN6DHMJOVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANRQHAYDIOI__;!!JZxO7R1e!ul19Sj6riXSeQG0nXK-VmRFjwaPw9PWTiFCiqgQQgDa1urcT4r32IZiUn6x54z5ED9jENTEL4JrirJtCGM-L-MAsgvTHAopv$>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>

[KalyS77]

Hello Nick, We had enabled Serilog Self log, but that did not capture anything. * But what we have observer is that there is an exception occurring right at the point where the Serilog Logger is instantiated. Please find the code snippet at which it happens and the exception below. * When we comment the .WriteTo.Seq part of the Logger creation, it works without exception. * Please note that this happens only in our PROD environment and the below code work seamlessly to ingest logs into SEQ from our UAT environment. Serilog library version: 3.0.1.0 Serilog.Sinks.Seq library version: 5.2.2.0 Code Snippet: Serilog.Log.Logger = new LoggerConfiguration() .WriteTo.File(Path.Combine(Config.logFolder, string.Format(LogNameDate, Config.seqSelfLogFileName + "-appLog", DateTime.Now.ToString(DateIso)))) .WriteTo.Seq(Config.logSeqServer, apiKey: Config.logSeqApiKey, restrictedToMinimumLevel: Serilog.Events.LogEventLevel.Verbose ) .CreateLogger(); Serilog.Log.Information("Hello, {Name}! - Starting {AppName}", Environment.UserName, Config.appName); Exception: Exception Message : "The inner handler has not been assigned." Inner Exception : null StackTrace : " at System.Net.Http.DelegatingHandler.SetOperationStarted() at System.Net.Http.DelegatingHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at Serilog.SeqLoggerConfigurationExtensions.Seq(LoggerSinkConfiguration loggerSinkConfiguration, String serverUrl, LogEventLevel restrictedToMinimumLevel, Int32 batchPostingLimit, Nullable`1 period, String apiKey, String bufferBaseFilename, Nullable`1 bufferSizeLimitBytes, Nullable`1 eventBodyLimitBytes, LoggingLevelSwitch controlLevelSwitch, HttpMessageHandler messageHandler, Nullable`1 retainedInvalidPayloadsLimitBytes, Int32 queueSizeLimit) at PaperlessWatcher.wsPaperlessWatcher.OnStart(String[] args) Your thoughts please. Thank you! ***@***.*** Sr. Application Developer | Architecture & Development | Unisys Mortgage Processing Unisys Australia | M : +61 481-262-914 | E : ***@***.******@***.***> 1G Homebush Bay Dr, Rhodes NSW 2138 ***@***.***<http://www.unisys.com/> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. [Grey_LI]<http://www.linkedin.com/company/unisys> [Grey_TW] <http://twitter.com/unisyscorp> [Grey_YT] <http://www.youtube.com/theunisyschannel> [Grey_FB] <http://www.facebook.com/unisyscorp> [Grey_Vimeo] <https://vimeo.com/unisys> [Grey_UB] <http://blogs.unisys.com/> From: Kalyadan, Sathyajith Sent: Thursday, 12 September 2024 1:40 PM To: datalust/seq-tickets ***@***.***>; datalust/seq-tickets ***@***.***> Cc: Author ***@***.***> Subject: RE: [datalust/seq-tickets] Application not pushing logs to SEQ post restart (Discussion #2270) Thanks Nick, We are trying this. Thank you! ***@***.*** Sr. Application Developer | Architecture & Development | Unisys Mortgage Processing Unisys Australia | M : +61 481-262-914 | E : ***@***.******@***.***> 1G Homebush Bay Dr, Rhodes NSW 2138 ***@***.***<http://www.unisys.com/> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. [Grey_LI]<http://www.linkedin.com/company/unisys> [Grey_TW] <http://twitter.com/unisyscorp> [Grey_YT] <http://www.youtube.com/theunisyschannel> [Grey_FB] <http://www.facebook.com/unisyscorp> [Grey_Vimeo] <https://vimeo.com/unisys> [Grey_UB] <http://blogs.unisys.com/> From: Nicholas Blumhardt ***@***.******@***.***>> Sent: Wednesday, 11 September 2024 10:11 AM To: datalust/seq-tickets ***@***.******@***.***>> Cc: Kalyadan, Sathyajith ***@***.******@***.***>>; Author ***@***.******@***.***>> Subject: Re: [datalust/seq-tickets] Application not pushing logs to SEQ post restart (Discussion #2270) Hi! Not sure whether my email reply yesterday reached you, so copying it below: Based on the ingestion log entries being unrelated to the HTTP API, it's most likely a client-side logging issue we need to track down, in the app itself. The best Hi! Not sure whether my email reply yesterday reached you, so copying it below: Based on the ingestion log entries being unrelated to the HTTP API, it's most likely a client-side logging issue we need to track down, in the app itself. The best debugging tactic will be to enable Serilog's SelfLog, which should still show most logging issues generated through Lurgle Logging. This would go in the app's Program.cs right at the start of execution: var file = File.CreateText(...); Serilog.Debugging.SelfLog.Enable(TextWriter.Synchronized(file)); Where the text file goes would depend on your deployment environment. You can send the diagnostic messages somewhere else using the callback version instead: Serilog.Debugging.SelfLog.Enable(message => /* do something to record message */); I'd suggest only enabling this for a few minutes, just in case message volumes cause a lot of data to be created. Best regards, Nick — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/datalust/seq-tickets/discussions/2270*discussioncomment-10608049__;Iw!!JZxO7R1e!ul19Sj6riXSeQG0nXK-VmRFjwaPw9PWTiFCiqgQQgDa1urcT4r32IZiUn6x54z5ED9jENTEL4JrirJtCGM-L-MAsgsJdhW_O$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/BLFOEVMPBRZWFPVWBPVZDCLZV6DBJAVCNFSM6AAAAABN6DHMJOVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANRQHAYDIOI__;!!JZxO7R1e!ul19Sj6riXSeQG0nXK-VmRFjwaPw9PWTiFCiqgQQgDa1urcT4r32IZiUn6x54z5ED9jENTEL4JrirJtCGM-L-MAsgvTHAopv$>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>

[nblumhardt]

Thanks for your follow-up.

Those package versions are quite some way behind (the Seq sink one especially); is it possible to update them to Serilog 4.0.1 and Serilog.Sinks.Seq 8.0.0? This would rule out some possible package compatibility issues.

The Serilog.Sinks.Seq package itself doesn't use DelegatingHandler (the type throwing the exception). I can see that there's some handler functionality in Lurgle.Logging but I'm not sure it'd result in a DelegatingHandler being used. Is it possible your own logging config code is plugging this in, or perhaps an older Lurgle.Logging version uses this?

Hope this helps,
Nick

[KalyS77]

Hello Nick, Lurgle.Logging’s latest version uses this old version of the Serilog and Seq Sink. But what we are unable to understand is that, we use the same library in our Web application code for logging which is running on a different server. That application is able to write into SEQ seamlessly even now. In fact the windows service application for which the logging is broken was working perfectly fine until it was stopped on 08 Sep 2024. It started to behave like this (Not pushing logs to SEQ) post the restart? There was nothing changed for the application at this time, before or after. Thank you! ***@***.*** Sr. Application Developer | Architecture & Development | Unisys Mortgage Processing Unisys Australia | M : +61 481-262-914 | E : ***@***.******@***.***> 1G Homebush Bay Dr, Rhodes NSW 2138 ***@***.***<http://www.unisys.com/> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. [Grey_LI]<http://www.linkedin.com/company/unisys> [Grey_TW] <http://twitter.com/unisyscorp> [Grey_YT] <http://www.youtube.com/theunisyschannel> [Grey_FB] <http://www.facebook.com/unisyscorp> [Grey_Vimeo] <https://vimeo.com/unisys> [Grey_UB] <http://blogs.unisys.com/> From: Nicholas Blumhardt ***@***.***> Sent: Friday, 13 September 2024 11:09 AM To: datalust/seq-tickets ***@***.***> Cc: Kalyadan, Sathyajith ***@***.***>; Author ***@***.***> Subject: Re: [datalust/seq-tickets] Application not pushing logs to SEQ post restart (Discussion #2270) Thanks for your follow-up. Those package versions are quite some way behind (the Seq sink one especially); is it possible to update them to Serilog 4. 0. 1 and Serilog. Sinks. Seq 8. 0. 0? This would rule out some possible package compatibility issues.  Thanks for your follow-up. Those package versions are quite some way behind (the Seq sink one especially); is it possible to update them to Serilog 4.0.1 and Serilog.Sinks.Seq 8.0.0? This would rule out some possible package compatibility issues. The Serilog.Sinks.Seq package itself doesn't use DelegatingHandler (the type throwing the exception). I can see that there's some handler functionality in Lurgle.Logging<https://urldefense.com/v3/__https:/github.com/MattMofDoom/Lurgle.Logging/blob/dev/Lurgle.Logging/Classes/SeqClient.cs__;!!JZxO7R1e!plMr8j-4YwiWntItDoNIlgp5y1cyLHT-zC6AiGBwI2xvNI_wP_xjqd3Oh7S-jKrTjA93MqMfeswWGhYqUpt59eQ23uknqjzr$> but I'm not sure it'd result in a DelegatingHandler being used. Is it possible your own logging config code is plugging this in, or perhaps an older Lurgle.Logging version uses this? Hope this helps, Nick — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/datalust/seq-tickets/discussions/2270*discussioncomment-10631981__;Iw!!JZxO7R1e!plMr8j-4YwiWntItDoNIlgp5y1cyLHT-zC6AiGBwI2xvNI_wP_xjqd3Oh7S-jKrTjA93MqMfeswWGhYqUpt59eQ23u2qFAa9$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/BLFOEVKD2VKZ5DOO26JMUADZWI3JLAVCNFSM6AAAAABN6DHMJOVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANRTGE4TQMI__;!!JZxO7R1e!plMr8j-4YwiWntItDoNIlgp5y1cyLHT-zC6AiGBwI2xvNI_wP_xjqd3Oh7S-jKrTjA93MqMfeswWGhYqUpt59eQ23qGU28dK$>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>

[nblumhardt]

Hi @KalyS77, thanks for your reply.

Unfortunately there's not a single straightforward way to diagnose this kind of issue. I'd recommend trying the usual box of tricks: redeploy the application, diff the contents of the deployment directory against the working machine, etc.

It's possible something changed in the GAC on the application's host machine, e.g. because of a patch or another application being installed or auto-updating. Or perhaps some changes on disk weren't picked up until the restart? This is the kind of area I'd be exploring.

You might also find that restarting the working app triggers the same problem to show up there?

I'd also consider either requesting or proposing an update to Lurgle.Logging that pulls in the latest versions of its dependencies, or if that isn't possible, pulling a copy of Lurgle.Logging into your solution and updating it there. Being able to update your logging dependencies to recent versions will be important if you want to pull in bug fixes, updated protocol support, etc.

Hope this helps,
Nick

[KalyS77]

Thanks Nick, We are already trying the below methods you suggested, we will nail it anyhow. Will update you / contact you in case we identify anything in the direction of SEQ. Appreciate your help. Thank you! ***@***.*** Sr. Application Developer | Architecture & Development | Unisys Mortgage Processing Unisys Australia | M : +61 481-262-914 | E : ***@***.******@***.***> 1G Homebush Bay Dr, Rhodes NSW 2138 ***@***.***<http://www.unisys.com/> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. [Grey_LI]<http://www.linkedin.com/company/unisys> [Grey_TW] <http://twitter.com/unisyscorp> [Grey_YT] <http://www.youtube.com/theunisyschannel> [Grey_FB] <http://www.facebook.com/unisyscorp> [Grey_Vimeo] <https://vimeo.com/unisys> [Grey_UB] <http://blogs.unisys.com/> From: Nicholas Blumhardt ***@***.***> Sent: Friday, 13 September 2024 2:27 PM To: datalust/seq-tickets ***@***.***> Cc: Kalyadan, Sathyajith ***@***.***>; Mention ***@***.***> Subject: Re: [datalust/seq-tickets] Application not pushing logs to SEQ post restart (Discussion #2270) Hi @KalyS77, thanks for your reply. Unfortunately there's not a single straightforward way to diagnose this kind of issue. I'd recommend trying the usual box of tricks: redeploy the application, diff the contents of the deployment directory Hi @KalyS77<https://urldefense.com/v3/__https:/github.com/KalyS77__;!!JZxO7R1e!ql_i6JV6Dt3_LVeg_tlM96Nd4TDiey6JgjtgHelaK-9NO4WnxvKuuJp5AgQUfcexyr5F2N94U3xIhplBlkl3y4ljibkYJsRM$>, thanks for your reply. Unfortunately there's not a single straightforward way to diagnose this kind of issue. I'd recommend trying the usual box of tricks: redeploy the application, diff the contents of the deployment directory against the working machine, etc. It's possible something changed in the GAC on the application's host machine, e.g. because of a patch or another application being installed or auto-updating. Or perhaps some changes on disk weren't picked up until the restart? This is the kind of area I'd be exploring. You might also find that restarting the working app triggers the same problem to show up there? I'd also consider either requesting or proposing an update to Lurgle.Logging that pulls in the latest versions of its dependencies, or if that isn't possible, pulling a copy of Lurgle.Logging into your solution and updating it there. Being able to update your logging dependencies to recent versions will be important if you want to pull in bug fixes, updated protocol support, etc. Hope this helps, Nick — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/datalust/seq-tickets/discussions/2270*discussioncomment-10633047__;Iw!!JZxO7R1e!ql_i6JV6Dt3_LVeg_tlM96Nd4TDiey6JgjtgHelaK-9NO4WnxvKuuJp5AgQUfcexyr5F2N94U3xIhplBlkl3y4ljiYC5mIZy$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/BLFOEVMQCUM34Z67BGSENBDZWJSTBAVCNFSM6AAAAABN6DHMJOVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANRTGMYDINY__;!!JZxO7R1e!ql_i6JV6Dt3_LVeg_tlM96Nd4TDiey6JgjtgHelaK-9NO4WnxvKuuJp5AgQUfcexyr5F2N94U3xIhplBlkl3y4ljibgGXgbw$>. You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>>