Getting Error: Flare native storage failed (InternalError), data file "/data/Stream/stream.08dbb03a9d7f1400.9ad67f18e4fb4cf6ac4c24e46f2bb5a2.tick" is missing.

[ankitkaushik94]

Hi Team,

We've seq running on kubernetes as a Pod. But the seq pod is crashing again and again due to which Seq Application is not coming and showing error "503 Service Temporarily Unavailable".

When I checked the logs of the seq pod it is throwing below error. Could you please help in resolving the error.
PFB log snippet:

────────────────────────────────────────
Seq ♦ Machine data, for humans.
─────────── © 2023 Datalust Pty Ltd ────

Running as server; press Ctrl+C to exit.

[09:02:02 INF] Seq 2023.4.10067 running on OS Linux 5.4.0-162-generic #179-Ubuntu SMP Mon Aug 14 08:51:31 UTC 2023
[09:02:03 INF] Opening event store at /data/Stream/stream.flare
Application startup exception: Autofac.Core.DependencyResolutionException: An exception was thrown while activating λ:Seq.Cluster.IClusterNode -> λ:Seq.Server.StorageSubsystem.
---> Flare.Ffi.Result.FlareException: Flare native storage failed (InternalError), data file "/data/Stream/stream.08dbb03a9d7f1400.9ad67f18e4fb4cf6ac4c24e46f2bb5a2.tick" is missing.
at Flare.Ffi.Result.FlareResult.EnsureSuccess()
at Flare.Ffi.Result.FlareResult.HandleNonSuccessWith(NonSuccessResultHandling handling)
at Flare.Ffi.Bindings.StoreOpen(IntPtr path, UIntPtr pathLen, IntPtr limits, StoreHandle& store, NonSuccessResultHandling nonSuccessResultHandling)
at Flare.Store.OpenStandalone(String path, StoreLimits limits)
at Seq.Engine.Storage.StorageEngine..ctor(Boolean standalone, String dataFile, PerformanceCounters sharedCounters, String nodeName, Nullable1 initialRole, String listenUri, CertificateBundle listenTlsCertificate, String nextPeerUri, String authenticationKey, StorageEngineLimits limits) at Seq.Engine.Storage.StorageEngine..ctor(String dataFile, PerformanceCounters sharedCounters, StorageEngineLimits limits) at Seq.Engine.Cluster.StorageNodeCoordinator..ctor(String streamPath, StorageNodeConfiguration configuration, BroadcastChannel broadcast, IFilesystemCertificateStore certificateStore, PerformanceCounters sharedCounters, StorageNodeLimits limits) at Seq.Cluster.ClusterNode..ctor(SeqConfig config, String nodeStateFilePath, String streamPath, WorkerPool workerPool, BroadcastChannel broadcast, IPortableEncryption encryption, IFilesystemCertificateStore certificateStore, PerformanceCounters sharedCounters) at Seq.Server.StorageSubsystem.Initialize(IComponentContext ctx, SeqConfig config, StorageLayout layout, PerformanceCounters sharedCounters) at Seq.Server.SeqServerModule.<Load>b__9_10(IComponentContext c) at Autofac.RegistrationExtensions.<>c__DisplayClass41_01.b__0(IComponentContext c, IEnumerable1 p) at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass0_01.b__0(IComponentContext c, IEnumerable1 p) at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable1 parameters)
at Autofac.Core.Activators.Delegate.DelegateActivator.b__2_0(ResolveRequestContext ctxt, Action1 next) at Autofac.Core.Resolving.Middleware.DelegateMiddleware.Execute(ResolveRequestContext context, Action1 next)
at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.b__1(ResolveRequestContext ctxt)
at Autofac.Core.Resolving.Middleware.DisposalTrackingMiddleware.Execute(ResolveRequestContext context, Action1 next) at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt) at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action1 next)
--- End of inner exception stack trace ---
at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action1 next) at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt) at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext ctxt) at Autofac.Core.Resolving.Middleware.RegistrationPipelineInvokeMiddleware.Execute(ResolveRequestContext context, Action1 next)
at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.b__1(ResolveRequestContext ctxt)
at Autofac.Core.Resolving.Middleware.SharingMiddleware.Execute(ResolveRequestContext context, Action1 next) at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt) at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt) at Autofac.Core.Resolving.Middleware.CircularDependencyDetectorMiddleware.Execute(ResolveRequestContext context, Action1 next)
at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.b__1(ResolveRequestContext ctxt)
at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext ctxt)
at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, ResolveRequest request)
at Autofac.Core.Resolving.ResolveOperation.ExecuteOperation(ResolveRequest request)
at Autofac.Core.Resolving.ResolveOperation.Execute(ResolveRequest request)
at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest request)
at Autofac.Core.Container.ResolveComponent(ResolveRequest request)
at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable1 parameters, Object& instance) at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable1 parameters)
at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType, IEnumerable`1 parameters)
at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType)
at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
at Microsoft.Extensions.Internal.ActivatorUtilities.ConstructorMatcher.CreateInstance(IServiceProvider provider)
at Microsoft.Extensions.Internal.ActivatorUtilities.CreateInstance(IServiceProvider provider, Type instanceType, Object[] parameters)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_0.b__0(RequestDelegate next)
at Microsoft.AspNetCore.Builder.ApplicationBuilder.Build()
at Microsoft.AspNetCore.Hosting.WebHost.BuildApplication()

Best,
Ankit Kaushik

[KodrAus]

Hi @ankitkaushik94 👋

Thanks for reaching out! This error can happen if you had an unexpected restart on some network filesystems or underlying storage corruption. You can get yourself back to good by running the following command in a container attached to your Seq instance:

flaretl repair --metadata /data/Stream

The easiest way to run this in Kubernetes is the create a Seq init script that will run that command once on startup.

Hope that helps! If you run into any more trouble please feel free to reach out to us via email through the Support menu in Seq.

[ankitkaushik94]

Thanks @KodrAus.
It worked and removed the problematic file.

[SSIvanov19]

Hi,
I'm facing a similar problem where my stream.flare file is corrupted and has been deleted. When I try to run the following command:

"C:\Program Files\Seq\Native\flaretl.exe" repair "C:\ProgramData\Seq\Stream\stream.flare"

I get the following output:

The command could not be executed: the command failed while opening the metadata store; failed to open a metadata store; a frame in the key-value store could not be read; invalid frame identifier: `0`

Any help would be appreciated!

[KodrAus]

Hi @SSIvanov19. What kind of storage are you running Seq on? It sounds like your metadata store has itself also become corrupted, which makes recovery trickier. The easiest way to get back up and running will be to rename your /data/Stream directory into /data/StreamBackup so when Seq starts again it will create a new /data/Stream directory to work off. If you're running on storage that's prone to corruption though, it's likely you'll hit this problem again at some point.

Depending on how much data you have, you can try recover your existing logs by setting up a new Seq container and using flaretl export /data/Stream/x.span, where x.span is a .span file in your Stream directory. This command will output line-delimited JSON which you can write to a file, like x.json, as a backup, or then re-import into your new instance via /seqsvr/Client/seqcli ingest --json -i x.json.