Error: snappy: corrupt input (unsupported chunk length: 0)

[ibrahimnamdar]

Hello,

We recently migrated our SEQ logs from a Windows Server to Linux (ubuntu 22.04) Docker container in Azure.
We are getting the following error on SEQ Log Indexing Process:

{"@t":"2024-11-25T23:51:05.3159073Z","@sp":"8505a9f7897d7c4d","@tr":"8b6237624825ea9f082059edcc8667cc","@mt":"Indexing with {Allocated:0.000} ms allowance","@r":["600000.000"],"@l":"Error","@x":"Flare.Ffi.Result.FlareException: Flare native storage failed (IOError), the task encountered an error in a worker.\n caused by: error evaluating the next record to index\n caused by: snappy: error snappy reading frame header\n caused by: snappy: corrupt input (unsupported chunk length: 0)\n at Seq.Engine.Storage.StorageEngine.QueryInternal(Cursor cursor, NativeCancellationToken nativeCancel, Object[] sharedColumnBuffer, CancellationToken cancel)+MoveNext()\n at Seq.Engine.Events.EventStore.QueryInternal(IEnumerable1 rowset, Boolean disableReadRateLimit)+MoveNext()\n at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable1 source, Boolean& found)\n at System.Linq.Enumerable.Single[TSource](IEnumerable`1 source)\n at Seq.Engine.Queries.DataStore.WriteNextIndexSet(CancellationToken cancel)\n at Seq.Server.Features.Indexing.EventStoreIndexer.Index(TimeSpan allocatedIndexingTime, CancellationToken cancel)","Allocated":600000,"@st":"2024-11-25T23:50:58.0895946Z","SourceContext":"Seq.Server.Features.Indexing.EventStoreIndexer"}

We tried the flaretl repair --metadata /data/Stream command but no luck.

Any ideas?

Kind regards,
Ibrahim

[nblumhardt]

Hi Ibrahim, thanks for your message.

If the new install is hosted in an Azure Web App, or using the non-premium Azure Files backing storage, this issue will likely continue to appear; let me know if that's the case.

Because this error is in the stored event data, rather than metadata, you'll need to drop the --metadata flag from the flaretl repair command-line.

I'm not 100% certain the repair process will cover this case, but if it produces output that should at least identify the problem chunk, and at that point we should be able to provide further steps to sort it out.

[ibrahimnamdar]

Hi Nicholas,

Thank you for the answer.

The new install is on a Linux (ubuntu 22.04) VM with Premium SSD v2 LRS disks attached.

Here are the results of the commands:

1. flaretl repair /data/Stream
No storage problems were found
2. flaretl repair /data/Stream/stream.flare
No storage problems were found

It seems like flaretl repair didn't find anything.
Anything else I can try?

[ibrahimnamdar]

Thank you @KodrAus
I'll be waiting for your response

[KodrAus]

Thanks @ibrahimnamdar. One last thing I need is your UTC offset. Those file names encode UTC timestamp ranges, which when decoded look like this:

stream-ls.txt

The offending file is the one(s) that overlap the time range from your screenshot, which will be in your local offset.

[ibrahimnamdar]

Thanks for decoding @KodrAus
My local timezone is UTC +1. Do I delete the file that corresponds to the failure time?

[nblumhardt]

Hi @ibrahimnamdar;

Moving it to an offline location would be safer than deleting, but yes 👍 - here's the process:

1. Stop the Seq container (important)
2. Move the problem .span file to a separate storage location
3. Either through an init script, or by mounting the storage volume to a temporary container instance invoked with the shell argument*, run flaretl repair --metadata /data/Stream/stream.flare
4. Start the Seq container

(* not sure you'll have encountered this, but it would look something like docker run --rm -it -v (host data path):/data datalust/seq shell)

[ibrahimnamdar]

Thank you @nblumhardt and @KodrAus 🙏,

I moved the problem file out of the Stream folder and ran flaretl repair /data/Stream cmd via an init script.
Indexing was successful after the script ran.

[KodrAus]

Thanks @ibrahimnamdar. For what it’s worth, we’ve since improved these error messages to include the path of the file they encountered a corruption in to make them easier to identify in future releases. This is extremely uncommon though, especially on the kind of file you found one in, so hopefully won’t come up again.

[ibrahimnamdar]

Yeah, not sure how the corruption happened but it probably has to do with our log file migration.
Seeing the corrupted file name in SEQ internal logs would definitely lead me to move the file and see what happens (on a test VM of course).

Thanks for the improvement and support