failed to open the key-value store at: "/data/Stream/stream.0.metadata"

[porya0ras]

hi
i have problem in seq
i use azure web app and docker to run seq service
after 3 month i have this error

2023-11-20T11:47:03.476504199Z ──────────────────────────────────────── 2023-11-20T11:47:03.476653000Z Seq ♦ Machine data, for humans. 2023-11-20T11:47:03.491035917Z ─────────── © 2023 Datalust Pty Ltd ──── 2023-11-20T11:47:03.491109417Z 2023-11-20T11:47:03.491117017Z Running as server; press Ctrl+C to exit. 2023-11-20T11:47:03.491121117Z 2023-11-20T11:47:03.981310087Z [11:47:03 INF] Seq 2023.4.10219 running on OS Linux 5.15.116.1-1.cm2 #1 SMP Wed Jun 21 23:17:09 UTC 2023 2023-11-20T11:47:04.556121642Z [11:47:04 INF] Opening event store at /data/Stream/stream.flare 2023-11-20T11:53:02.785149752Z Application startup exception: Autofac.Core.DependencyResolutionException: An exception was thrown while activating λ:Seq.Cluster.IClusterNode -> λ:Seq.Server.StorageSubsystem. 2023-11-20T11:53:02.785192553Z ---> Flare.Ffi.Result.FlareException: Flare native storage failed (IOError), error attempting to open a metadata store. 2023-11-20T11:53:02.785199353Z caused by: failed to open a metadata store 2023-11-20T11:53:02.785203353Z caused by: failed to open the key-value store at: "/data/Stream/stream.0.metadata" 2023-11-20T11:53:02.785207653Z caused by: I/O error at path: "/data/Stream/stream.0.metadata" 2023-11-20T11:53:02.785212153Z caused by: memory map must have a non-zero length 2023-11-20T11:53:02.785215753Z at Flare.Ffi.Result.FlareResult.EnsureSuccess() 2023-11-20T11:53:02.785219653Z at Flare.Ffi.Result.FlareResult.HandleNonSuccessWith(NonSuccessResultHandling handling) 2023-11-20T11:53:02.785223753Z at Flare.Ffi.Bindings.StoreOpen(IntPtr path, UIntPtr pathLen, IntPtr limits, StoreHandle& store, NonSuccessResultHandling nonSuccessResultHandling) 2023-11-20T11:53:02.785228253Z at Flare.Store.OpenStandalone(String path, StoreLimits limits) 2023-11-20T11:53:02.785236353Z at Seq.Engine.Storage.StorageEngine..ctor(Boolean standalone, String dataFile, PerformanceCounters sharedCounters, String nodeName, Nullable1 initialRole, String listenUri, CertificateBundle listenTlsCertificate, String nextPeerUri, String authenticationKey, StorageEngineLimits limits)
2023-11-20T11:53:02.785241153Z at Seq.Engine.Storage.StorageEngine..ctor(String dataFile, PerformanceCounters sharedCounters, StorageEngineLimits limits)
2023-11-20T11:53:02.785244853Z at Seq.Engine.Cluster.StorageNodeCoordinator..ctor(String streamPath, StorageNodeConfiguration configuration, BroadcastChannel broadcast, IFilesystemCertificateStore certificateStore, PerformanceCounters sharedCounters, StorageNodeLimits limits)
2023-11-20T11:53:02.785249053Z at Seq.Cluster.ClusterNode..ctor(SeqConfig config, String nodeStateFilePath, String streamPath, WorkerPool workerPool, BroadcastChannel broadcast, IPortableEncryption encryption, IFilesystemCertificateStore certificateStore, PerformanceCounters sharedCounters)
2023-11-20T11:53:02.785253753Z at Seq.Server.StorageSubsystem.Initialize(IComponentContext ctx, SeqConfig config, StorageLayout layout, PerformanceCounters sharedCounters)
2023-11-20T11:53:02.785259453Z at Seq.Server.SeqServerModule.b__9_10(IComponentContext c)
2023-11-20T11:53:02.785276953Z at Autofac.RegistrationExtensions.<>c__DisplayClass41_01.<Register>b__0(IComponentContext c, IEnumerable1 p)
2023-11-20T11:53:02.785281353Z at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass0_01.<ForDelegate>b__0(IComponentContext c, IEnumerable1 p)
2023-11-20T11:53:02.785285153Z at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable1 parameters) 2023-11-20T11:53:02.785289053Z at Autofac.Core.Activators.Delegate.DelegateActivator.<ConfigurePipeline>b__2_0(ResolveRequestContext ctxt, Action1 next)
2023-11-20T11:53:02.785292853Z at Autofac.Core.Resolving.Middleware.DelegateMiddleware.Execute(ResolveRequestContext context, Action1 next) 2023-11-20T11:53:02.785296753Z at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt) 2023-11-20T11:53:02.785300953Z at Autofac.Core.Resolving.Middleware.DisposalTrackingMiddleware.Execute(ResolveRequestContext context, Action1 next)
2023-11-20T11:53:02.785304554Z at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.b__1(ResolveRequestContext ctxt)
2023-11-20T11:53:02.785308254Z at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action1 next) 2023-11-20T11:53:02.785312154Z --- End of inner exception stack trace --- 2023-11-20T11:53:02.785316154Z at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action1 next)
2023-11-20T11:53:02.785320054Z at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.b__1(ResolveRequestContext ctxt)
2023-11-20T11:53:02.785324154Z at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext ctxt)
2023-11-20T11:53:02.785328054Z at Autofac.Core.Resolving.Middleware.RegistrationPipelineInvokeMiddleware.Execute(ResolveRequestContext context, Action1 next) 2023-11-20T11:53:02.785332054Z at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt) 2023-11-20T11:53:02.785336154Z at Autofac.Core.Resolving.Middleware.SharingMiddleware.Execute(ResolveRequestContext context, Action1 next)
2023-11-20T11:53:02.785340554Z at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.b__1(ResolveRequestContext ctxt)
2023-11-20T11:53:02.785345854Z at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.b__1(ResolveRequestContext ctxt)
2023-11-20T11:53:02.785349454Z at Autofac.Core.Resolving.Middleware.CircularDependencyDetectorMiddleware.Execute(ResolveRequestContext context, Action1 next) 2023-11-20T11:53:02.785353054Z at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt) 2023-11-20T11:53:02.785356954Z at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext ctxt) 2023-11-20T11:53:02.785366054Z at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, ResolveRequest request) 2023-11-20T11:53:02.785370654Z at Autofac.Core.Resolving.ResolveOperation.ExecuteOperation(ResolveRequest request) 2023-11-20T11:53:02.785374654Z at Autofac.Core.Resolving.ResolveOperation.Execute(ResolveRequest request) 2023-11-20T11:53:02.785378854Z at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest request) 2023-11-20T11:53:02.785383054Z at Autofac.Core.Container.ResolveComponent(ResolveRequest request) 2023-11-20T11:53:02.785386754Z at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable1 parameters, Object& instance)
2023-11-20T11:53:02.785390954Z at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable1 parameters) 2023-11-20T11:53:02.785394754Z at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType, IEnumerable1 parameters)
2023-11-20T11:53:02.785398754Z at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType)
2023-11-20T11:53:02.785402654Z at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
2023-11-20T11:53:02.785407054Z at Microsoft.Extensions.Internal.ActivatorUtilities.ConstructorMatcher.CreateInstance(IServiceProvider provider)
2023-11-20T11:53:02.785410954Z at Microsoft.Extensions.Internal.ActivatorUtilities.CreateInstance(IServiceProvider provider, Type instanceType, Object[] parameters)
2023-11-20T11:53:02.785414854Z at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_0.b__0(RequestDelegate next)
2023-11-20T11:53:02.785418954Z at Microsoft.AspNetCore.Builder.ApplicationBuilder.Build()
2023-11-20T11:53:02.785422754Z at Microsoft.AspNetCore.Hosting.WebHost.BuildApplication()
2023-11-20T11:53:02.811748351Z [11:53:02 FTL] Application startup exception
2023-11-20T11:53:02.811791852Z Autofac.Core.DependencyResolutionException: An exception was thrown while activating λ:Seq.Cluster.IClusterNode -> λ:Seq.Server.StorageSubsystem.
2023-11-20T11:53:02.811799052Z ---> Flare.Ffi.Result.FlareException: Flare native storage failed (IOError), error attempting to open a metadata store.
2023-11-20T11:53:02.811803552Z caused by: failed to open a metadata store
2023-11-20T11:53:02.811807252Z caused by: failed to open the key-value store at: "/data/Stream/stream.0.metadata"
2023-11-20T11:53:02.811811252Z caused by: I/O error at path: "/data/Stream/stream.0.metadata"
2023-11-20T11:53:02.811815352Z caused by: memory map must have a non-zero length
2023-11-20T11:53:02.811819352Z at Flare.Ffi.Result.FlareResult.EnsureSuccess()
2023-11-20T11:53:02.811823752Z at Flare.Ffi.Result.FlareResult.HandleNonSuccessWith(NonSuccessResultHandling handling)
2023-11-20T11:53:02.811842052Z at Flare.Ffi.Bindings.StoreOpen(IntPtr path, UIntPtr pathLen, IntPtr limits, StoreHandle& store, NonSuccessResultHandling nonSuccessResultHandling)
2023-11-20T11:53:02.811846452Z at Flare.Store.OpenStandalone(String path, StoreLimits limits)
2023-11-20T11:53:02.811850552Z at Seq.Engine.Storage.StorageEngine..ctor(Boolean standalone, String dataFile, PerformanceCounters sharedCounters, String nodeName, Nullable1 initialRole, String listenUri, CertificateBundle listenTlsCertificate, String nextPeerUri, String authenticationKey, StorageEngineLimits limits) 2023-11-20T11:53:02.811854952Z at Seq.Engine.Storage.StorageEngine..ctor(String dataFile, PerformanceCounters sharedCounters, StorageEngineLimits limits) 2023-11-20T11:53:02.811859152Z at Seq.Engine.Cluster.StorageNodeCoordinator..ctor(String streamPath, StorageNodeConfiguration configuration, BroadcastChannel broadcast, IFilesystemCertificateStore certificateStore, PerformanceCounters sharedCounters, StorageNodeLimits limits) 2023-11-20T11:53:02.811863952Z at Seq.Cluster.ClusterNode..ctor(SeqConfig config, String nodeStateFilePath, String streamPath, WorkerPool workerPool, BroadcastChannel broadcast, IPortableEncryption encryption, IFilesystemCertificateStore certificateStore, PerformanceCounters sharedCounters) 2023-11-20T11:53:02.811869052Z at Seq.Server.StorageSubsystem.Initialize(IComponentContext ctx, SeqConfig config, StorageLayout layout, PerformanceCounters sharedCounters) 2023-11-20T11:53:02.811874952Z at Seq.Server.SeqServerModule.<Load>b__9_10(IComponentContext c) 2023-11-20T11:53:02.811879352Z at Autofac.RegistrationExtensions.<>c__DisplayClass41_01.b__0(IComponentContext c, IEnumerable1 p) 2023-11-20T11:53:02.811883752Z at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass0_01.b__0(IComponentContext c, IEnumerable1 p) 2023-11-20T11:53:02.811888352Z at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable1 parameters)
2023-11-20T11:53:02.811892152Z at Autofac.Core.Activators.Delegate.DelegateActivator.b__2_0(ResolveRequestContext ctxt, Action1 next) 2023-11-20T11:53:02.811896152Z at Autofac.Core.Resolving.Middleware.DelegateMiddleware.Execute(ResolveRequestContext context, Action1 next)
2023-11-20T11:53:02.811899752Z at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.b__1(ResolveRequestContext ctxt)
2023-11-20T11:53:02.811903752Z at Autofac.Core.Resolving.Middleware.DisposalTrackingMiddleware.Execute(ResolveRequestContext context, Action1 next) 2023-11-20T11:53:02.811907852Z at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt) 2023-11-20T11:53:02.811912152Z at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action1 next)
2023-11-20T11:53:02.811916352Z --- End of inner exception stack trace ---
2023-11-20T11:53:02.811920252Z at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action1 next) 2023-11-20T11:53:02.811924653Z at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt) 2023-11-20T11:53:02.811933153Z at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext ctxt) 2023-11-20T11:53:02.811937453Z at Autofac.Core.Resolving.Middleware.RegistrationPipelineInvokeMiddleware.Execute(ResolveRequestContext context, Action1 next)
2023-11-20T11:53:02.811942153Z at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.b__1(ResolveRequestContext ctxt)
2023-11-20T11:53:02.811946853Z at Autofac.Core.Resolving.Middleware.SharingMiddleware.Execute(ResolveRequestContext context, Action1 next) 2023-11-20T11:53:02.811951253Z at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt) 2023-11-20T11:53:02.811956853Z at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt) 2023-11-20T11:53:02.811961353Z at Autofac.Core.Resolving.Middleware.CircularDependencyDetectorMiddleware.Execute(ResolveRequestContext context, Action1 next)
2023-11-20T11:53:02.811965753Z at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.b__1(ResolveRequestContext ctxt)
2023-11-20T11:53:02.811970153Z at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext ctxt)
2023-11-20T11:53:02.811974353Z at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, ResolveRequest request)
2023-11-20T11:53:02.811978953Z at Autofac.Core.Resolving.ResolveOperation.ExecuteOperation(ResolveRequest request)
2023-11-20T11:53:02.811983353Z at Autofac.Core.Resolving.ResolveOperation.Execute(ResolveRequest request)
2023-11-20T11:53:02.811987653Z at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest request)
2023-11-20T11:53:02.811991953Z at Autofac.Core.Container.ResolveComponent(ResolveRequest request)
2023-11-20T11:53:02.811996153Z at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable1 parameters, Object& instance) 2023-11-20T11:53:02.812000453Z at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable1 parameters)
2023-11-20T11:53:02.812004553Z at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType, IEnumerable1 parameters) 2023-11-20T11:53:02.812008753Z at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType) 2023-11-20T11:53:02.812012953Z at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType) 2023-11-20T11:53:02.812016553Z at Microsoft.Extensions.Internal.ActivatorUtilities.ConstructorMatcher.CreateInstance(IServiceProvider provider) 2023-11-20T11:53:02.812020153Z at Microsoft.Extensions.Internal.ActivatorUtilities.CreateInstance(IServiceProvider provider, Type instanceType, Object[] parameters) 2023-11-20T11:53:02.812027353Z at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_0.<UseMiddleware>b__0(RequestDelegate next) 2023-11-20T11:53:02.812031253Z at Microsoft.AspNetCore.Builder.ApplicationBuilder.Build() 2023-11-20T11:53:02.812034453Z at Microsoft.AspNetCore.Hosting.WebHost.BuildApplication() 2023-11-20T11:53:02.816889690Z [11:53:02 FTL] Error running the server application 2023-11-20T11:53:02.816920990Z Autofac.Core.DependencyResolutionException: An exception was thrown while activating λ:Seq.Cluster.IClusterNode -> λ:Seq.Server.StorageSubsystem. 2023-11-20T11:53:02.816927790Z ---> Flare.Ffi.Result.FlareException: Flare native storage failed (IOError), error attempting to open a metadata store. 2023-11-20T11:53:02.816931990Z caused by: failed to open a metadata store 2023-11-20T11:53:02.816935890Z caused by: failed to open the key-value store at: "/data/Stream/stream.0.metadata" 2023-11-20T11:53:02.816939990Z caused by: I/O error at path: "/data/Stream/stream.0.metadata" 2023-11-20T11:53:02.816944090Z caused by: memory map must have a non-zero length 2023-11-20T11:53:02.816947990Z at Flare.Ffi.Result.FlareResult.EnsureSuccess() 2023-11-20T11:53:02.816951890Z at Flare.Ffi.Result.FlareResult.HandleNonSuccessWith(NonSuccessResultHandling handling) 2023-11-20T11:53:02.816956390Z at Flare.Ffi.Bindings.StoreOpen(IntPtr path, UIntPtr pathLen, IntPtr limits, StoreHandle& store, NonSuccessResultHandling nonSuccessResultHandling) 2023-11-20T11:53:02.816960990Z at Flare.Store.OpenStandalone(String path, StoreLimits limits) 2023-11-20T11:53:02.816964990Z at Seq.Engine.Storage.StorageEngine..ctor(Boolean standalone, String dataFile, PerformanceCounters sharedCounters, String nodeName, Nullable1 initialRole, String listenUri, CertificateBundle listenTlsCertificate, String nextPeerUri, String authenticationKey, StorageEngineLimits limits)
2023-11-20T11:53:02.816969790Z at Seq.Engine.Storage.StorageEngine..ctor(String dataFile, PerformanceCounters sharedCounters, StorageEngineLimits limits)
2023-11-20T11:53:02.816973790Z at Seq.Engine.Cluster.StorageNodeCoordinator..ctor(String streamPath, StorageNodeConfiguration configuration, BroadcastChannel broadcast, IFilesystemCertificateStore certificateStore, PerformanceCounters sharedCounters, StorageNodeLimits limits)
2023-11-20T11:53:02.816978490Z at Seq.Cluster.ClusterNode..ctor(SeqConfig config, String nodeStateFilePath, String streamPath, WorkerPool workerPool, BroadcastChannel broadcast, IPortableEncryption encryption, IFilesystemCertificateStore certificateStore, PerformanceCounters sharedCounters)
2023-11-20T11:53:02.816982590Z at Seq.Server.StorageSubsystem.Initialize(IComponentContext ctx, SeqConfig config, StorageLayout layout, PerformanceCounters sharedCounters)
2023-11-20T11:53:02.816987390Z at Seq.Server.SeqServerModule.b__9_10(IComponentContext c)
2023-11-20T11:53:02.816991490Z at Autofac.RegistrationExtensions.<>c__DisplayClass41_01.<Register>b__0(IComponentContext c, IEnumerable1 p)
2023-11-20T11:53:02.816995890Z at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass0_01.<ForDelegate>b__0(IComponentContext c, IEnumerable1 p)
2023-11-20T11:53:02.817011891Z at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable1 parameters) 2023-11-20T11:53:02.817016091Z at Autofac.Core.Activators.Delegate.DelegateActivator.<ConfigurePipeline>b__2_0(ResolveRequestContext ctxt, Action1 next)
2023-11-20T11:53:02.817020091Z at Autofac.Core.Resolving.Middleware.DelegateMiddleware.Execute(ResolveRequestContext context, Action1 next) 2023-11-20T11:53:02.817024191Z at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt) 2023-11-20T11:53:02.817028291Z at Autofac.Core.Resolving.Middleware.DisposalTrackingMiddleware.Execute(ResolveRequestContext context, Action1 next)
2023-11-20T11:53:02.817032291Z at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.b__1(ResolveRequestContext ctxt)
2023-11-20T11:53:02.817035991Z at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action1 next) 2023-11-20T11:53:02.817040491Z --- End of inner exception stack trace --- 2023-11-20T11:53:02.817044191Z at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action1 next)
2023-11-20T11:53:02.817048491Z at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.b__1(ResolveRequestContext ctxt)
2023-11-20T11:53:02.817052191Z at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext ctxt)
2023-11-20T11:53:02.817058891Z at Autofac.Core.Resolving.Middleware.RegistrationPipelineInvokeMiddleware.Execute(ResolveRequestContext context, Action1 next) 2023-11-20T11:53:02.817062491Z at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt) 2023-11-20T11:53:02.817066091Z at Autofac.Core.Resolving.Middleware.SharingMiddleware.Execute(ResolveRequestContext context, Action1 next)
2023-11-20T11:53:02.817069591Z at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.b__1(ResolveRequestContext ctxt)
2023-11-20T11:53:02.817073791Z at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.b__1(ResolveRequestContext ctxt)
2023-11-20T11:53:02.817077691Z at Autofac.Core.Resolving.Middleware.CircularDependencyDetectorMiddleware.Execute(ResolveRequestContext context, Action1 next) 2023-11-20T11:53:02.817081191Z at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt) 2023-11-20T11:53:02.817084991Z at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext ctxt) 2023-11-20T11:53:02.817088491Z at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, ResolveRequest request) 2023-11-20T11:53:02.817091991Z at Autofac.Core.Resolving.ResolveOperation.ExecuteOperation(ResolveRequest request) 2023-11-20T11:53:02.817100291Z at Autofac.Core.Resolving.ResolveOperation.Execute(ResolveRequest request) 2023-11-20T11:53:02.817103991Z at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest request) 2023-11-20T11:53:02.817107391Z at Autofac.Core.Container.ResolveComponent(ResolveRequest request) 2023-11-20T11:53:02.817110491Z at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable1 parameters, Object& instance)
2023-11-20T11:53:02.817113991Z at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable1 parameters) 2023-11-20T11:53:02.817117391Z at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType, IEnumerable1 parameters)
2023-11-20T11:53:02.817120891Z at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType)
2023-11-20T11:53:02.817124291Z at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
2023-11-20T11:53:02.817127691Z at Microsoft.Extensions.Internal.ActivatorUtilities.ConstructorMatcher.CreateInstance(IServiceProvider provider)
2023-11-20T11:53:02.817131191Z at Microsoft.Extensions.Internal.ActivatorUtilities.CreateInstance(IServiceProvider provider, Type instanceType, Object[] parameters)
2023-11-20T11:53:02.817134891Z at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_0.b__0(RequestDelegate next)
2023-11-20T11:53:02.817138791Z at Microsoft.AspNetCore.Builder.ApplicationBuilder.Build()
2023-11-20T11:53:02.817142992Z at Microsoft.AspNetCore.Hosting.WebHost.BuildApplication()
2023-11-20T11:53:02.817147192Z at Microsoft.AspNetCore.Hosting.WebHost.StartAsync(CancellationToken cancellationToken)
2023-11-20T11:53:02.817151092Z at Microsoft.AspNetCore.Hosting.WebHost.Start()
2023-11-20T11:53:02.817155092Z at Seq.ServiceProcess.ServerService.Start()
2023-11-20T11:53:02.818377301Z [11:53:02 FTL] Failed to start server
2023-11-20T11:53:02.818399801Z Autofac.Core.DependencyResolutionException: An exception was thrown while activating λ:Seq.Cluster.IClusterNode -> λ:Seq.Server.StorageSubsystem.
2023-11-20T11:53:02.818407101Z ---> Flare.Ffi.Result.FlareException: Flare native storage failed (IOError), error attempting to open a metadata store.
2023-11-20T11:53:02.818411301Z caused by: failed to open a metadata store
2023-11-20T11:53:02.818415301Z caused by: failed to open the key-value store at: "/data/Stream/stream.0.metadata"
2023-11-20T11:53:02.818419501Z caused by: I/O error at path: "/data/Stream/stream.0.metadata"
2023-11-20T11:53:02.818424001Z caused by: memory map must have a non-zero length
2023-11-20T11:53:02.818427501Z at Flare.Ffi.Result.FlareResult.EnsureSuccess()
2023-11-20T11:53:02.818431401Z at Flare.Ffi.Result.FlareResult.HandleNonSuccessWith(NonSuccessResultHandling handling)
2023-11-20T11:53:02.818435601Z at Flare.Ffi.Bindings.StoreOpen(IntPtr path, UIntPtr pathLen, IntPtr limits, StoreHandle& store, NonSuccessResultHandling nonSuccessResultHandling)
2023-11-20T11:53:02.818448901Z at Flare.Store.OpenStandalone(String path, StoreLimits limits)
2023-11-20T11:53:02.818453101Z at Seq.Engine.Storage.StorageEngine..ctor(Boolean standalone, String dataFile, PerformanceCounters sharedCounters, String nodeName, Nullable1 initialRole, String listenUri, CertificateBundle listenTlsCertificate, String nextPeerUri, String authenticationKey, StorageEngineLimits limits) 2023-11-20T11:53:02.818457901Z at Seq.Engine.Storage.StorageEngine..ctor(String dataFile, PerformanceCounters sharedCounters, StorageEngineLimits limits) 2023-11-20T11:53:02.818462701Z at Seq.Engine.Cluster.StorageNodeCoordinator..ctor(String streamPath, StorageNodeConfiguration configuration, BroadcastChannel broadcast, IFilesystemCertificateStore certificateStore, PerformanceCounters sharedCounters, StorageNodeLimits limits) 2023-11-20T11:53:02.818467701Z at Seq.Cluster.ClusterNode..ctor(SeqConfig config, String nodeStateFilePath, String streamPath, WorkerPool workerPool, BroadcastChannel broadcast, IPortableEncryption encryption, IFilesystemCertificateStore certificateStore, PerformanceCounters sharedCounters) 2023-11-20T11:53:02.818472201Z at Seq.Server.StorageSubsystem.Initialize(IComponentContext ctx, SeqConfig config, StorageLayout layout, PerformanceCounters sharedCounters) 2023-11-20T11:53:02.818479401Z at Seq.Server.SeqServerModule.<Load>b__9_10(IComponentContext c) 2023-11-20T11:53:02.818484402Z at Autofac.RegistrationExtensions.<>c__DisplayClass41_01.b__0(IComponentContext c, IEnumerable1 p) 2023-11-20T11:53:02.818488702Z at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass0_01.b__0(IComponentContext c, IEnumerable1 p) 2023-11-20T11:53:02.818492702Z at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable1 parameters)
2023-11-20T11:53:02.818497202Z at Autofac.Core.Activators.Delegate.DelegateActivator.b__2_0(ResolveRequestContext ctxt, Action1 next) 2023-11-20T11:53:02.818502202Z at Autofac.Core.Resolving.Middleware.DelegateMiddleware.Execute(ResolveRequestContext context, Action1 next)
2023-11-20T11:53:02.818506402Z at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.b__1(ResolveRequestContext ctxt)
2023-11-20T11:53:02.818510502Z at Autofac.Core.Resolving.Middleware.DisposalTrackingMiddleware.Execute(ResolveRequestContext context, Action1 next) 2023-11-20T11:53:02.818514602Z at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt) 2023-11-20T11:53:02.818518702Z at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action1 next)
2023-11-20T11:53:02.818522802Z --- End of inner exception stack trace ---
2023-11-20T11:53:02.818527002Z at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action1 next) 2023-11-20T11:53:02.818531802Z at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt) 2023-11-20T11:53:02.818540402Z at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext ctxt) 2023-11-20T11:53:02.818554702Z at Autofac.Core.Resolving.Middleware.RegistrationPipelineInvokeMiddleware.Execute(ResolveRequestContext context, Action1 next)
2023-11-20T11:53:02.818560002Z at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.b__1(ResolveRequestContext ctxt)
2023-11-20T11:53:02.818564602Z at Autofac.Core.Resolving.Middleware.SharingMiddleware.Execute(ResolveRequestContext context, Action1 next) 2023-11-20T11:53:02.818568502Z at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt) 2023-11-20T11:53:02.818572702Z at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt) 2023-11-20T11:53:02.818577502Z at Autofac.Core.Resolving.Middleware.CircularDependencyDetectorMiddleware.Execute(ResolveRequestContext context, Action1 next)
2023-11-20T11:53:02.818581702Z at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.b__1(ResolveRequestContext ctxt)
2023-11-20T11:53:02.818585402Z at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext ctxt)
2023-11-20T11:53:02.818589302Z at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, ResolveRequest request)
2023-11-20T11:53:02.818593502Z at Autofac.Core.Resolving.ResolveOperation.ExecuteOperation(ResolveRequest request)
2023-11-20T11:53:02.818598602Z at Autofac.Core.Resolving.ResolveOperation.Execute(ResolveRequest request)
2023-11-20T11:53:02.818602602Z at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest request)
2023-11-20T11:53:02.818606202Z at Autofac.Core.Container.ResolveComponent(ResolveRequest request)
2023-11-20T11:53:02.818610302Z at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable1 parameters, Object& instance) 2023-11-20T11:53:02.818614903Z at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable1 parameters)
2023-11-20T11:53:02.818619103Z at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType, IEnumerable1 parameters) 2023-11-20T11:53:02.818622803Z at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType) 2023-11-20T11:53:02.818626103Z at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType) 2023-11-20T11:53:02.818635703Z at Microsoft.Extensions.Internal.ActivatorUtilities.ConstructorMatcher.CreateInstance(IServiceProvider provider) 2023-11-20T11:53:02.818640403Z at Microsoft.Extensions.Internal.ActivatorUtilities.CreateInstance(IServiceProvider provider, Type instanceType, Object[] parameters) 2023-11-20T11:53:02.818644403Z at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_0.<UseMiddleware>b__0(RequestDelegate next) 2023-11-20T11:53:02.818652503Z at Microsoft.AspNetCore.Builder.ApplicationBuilder.Build() 2023-11-20T11:53:02.818655903Z at Microsoft.AspNetCore.Hosting.WebHost.BuildApplication() 2023-11-20T11:53:02.818659703Z at Microsoft.AspNetCore.Hosting.WebHost.StartAsync(CancellationToken cancellationToken) 2023-11-20T11:53:02.818663703Z at Microsoft.AspNetCore.Hosting.WebHost.Start() 2023-11-20T11:53:02.818667303Z at Seq.ServiceProcess.ServerService.Start() 2023-11-20T11:53:02.818670803Z at Seq.Cli.RunCommand.RunStandardIOAsync(Func1 bootstrap, TextWriter cout)

2023-11-20T12:07:56.662Z INFO - Logging is not enabled for this container.
Please use https://aka.ms/linux-diagnostics to enable logging to see container logs here.
2023-11-20T12:07:57.005Z INFO - Initiating warmup request to container seq_0_c8034c25 for site seq
2023-11-20T12:08:03.829Z ERROR - Container seq_0_c8034c25 for site seq has exited, failing site start
2023-11-20T12:08:03.836Z ERROR - Container seq_0_c8034c25 didn't respond to HTTP pings on port: 80, failing site start. See container logs for debugging.
2023-11-20T12:08:03.839Z INFO - Stopping site seq because it failed during startup.
2023-11-20T12:12:22.642Z INFO - 2023.4 Pulling from datalust/seq
2023-11-20T12:12:22.707Z INFO - Digest: sha256:3049922a8a44e90faef8d0bf8cb7cd4d7223ecb55533a0995f2ce9ddf6fe193b
2023-11-20T12:12:22.708Z INFO - Status: Image is up to date for datalust/seq:2023.4
2023-11-20T12:12:22.720Z INFO - Pull Image successful, Time taken: 0 Minutes and 0 Seconds
2023-11-20T12:12:22.728Z INFO - Starting container for site
2023-11-20T12:12:22.729Z INFO - docker run -d --expose=80 --name seq_0_8a6993b1 -e WEBSITES_PORT=80:5341 -e WEBSITE_SITE_NAME=seq -e WEBSITE_AUTH_ENABLED=False -e PORT=80 -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=seq.azurewebsites.net -e WEBSITE_INSTANCE_ID=17f66a494d7210fa13de0e1affd4c02041bf8cf4fada674821980cc88a710a76 -e WEBSITE_USE_DIAGNOSTIC_SERVER=False datalust/seq:2023.4

2023-11-20T12:12:22.729Z INFO - Logging is not enabled for this container.
Please use https://aka.ms/linux-diagnostics to enable logging to see container logs here.
2023-11-20T12:12:23.744Z INFO - Initiating warmup request to container seq_0_8a6993b1 for site seq
2023-11-20T12:12:39.132Z ERROR - Container seq_0_8a6993b1 for site seq has exited, failing site start
2023-11-20T12:12:39.136Z ERROR - Container seq_0_8a6993b1 didn't respond to HTTP pings on port: 80, failing site start. See container logs for debugging.
2023-11-20T12:12:39.138Z INFO - Stopping site seq because it failed during startup.
2023-11-20T12:16:53.284Z INFO - 2023.4 Pulling from datalust/seq
2023-11-20T12:16:53.324Z INFO - Digest: sha256:3049922a8a44e90faef8d0bf8cb7cd4d7223ecb55533a0995f2ce9ddf6fe193b
2023-11-20T12:16:53.327Z INFO - Status: Image is up to date for datalust/seq:2023.4
2023-11-20T12:16:53.336Z INFO - Pull Image successful, Time taken: 0 Minutes and 0 Seconds
2023-11-20T12:16:53.345Z INFO - Starting container for site
2023-11-20T12:16:53.345Z INFO - docker run -d --expose=80 --name seq_0_524a2806 -e WEBSITES_PORT=80:5341 -e WEBSITE_SITE_NAME=seq -e WEBSITE_AUTH_ENABLED=False -e PORT=80 -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=seq.azurewebsites.net -e WEBSITE_INSTANCE_ID=17f66a494d7210fa13de0e1affd4c02041bf8cf4fada674821980cc88a710a76 -e WEBSITE_USE_DIAGNOSTIC_SERVER=False datalust/seq:2023.4

2023-11-20T12:16:53.346Z INFO - Logging is not enabled for this container.
Please use https://aka.ms/linux-diagnostics to enable logging to see container logs here.
2023-11-20T12:16:53.692Z INFO - Initiating warmup request to container seq_0_524a2806 for site seq
2023-11-20T12:17:01.144Z ERROR - Container seq_0_524a2806 for site seq has exited, failing site start
2023-11-20T12:17:01.149Z ERROR - Container seq_0_524a2806 didn't respond to HTTP pings on port: 80, failing site start. See container logs for debugging.
2023-11-20T12:17:01.151Z INFO - Stopping site seq because it failed during startup.
2023-11-20T12:21:24.128Z INFO - 2023.4 Pulling from datalust/seq
2023-11-20T12:21:24.176Z INFO - Digest: sha256:3049922a8a44e90faef8d0bf8cb7cd4d7223ecb55533a0995f2ce9ddf6fe193b
2023-11-20T12:21:24.176Z INFO - Status: Image is up to date for datalust/seq:2023.4
2023-11-20T12:21:24.188Z INFO - Pull Image successful, Time taken: 0 Minutes and 0 Seconds
2023-11-20T12:21:24.198Z INFO - Starting container for site
2023-11-20T12:21:24.198Z INFO - docker run -d --expose=80 --name seq_0_6310e60f -e WEBSITES_PORT=80:5341 -e WEBSITE_SITE_NAME=seq -e WEBSITE_AUTH_ENABLED=False -e PORT=80 -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=seq.azurewebsites.net -e WEBSITE_INSTANCE_ID=17f66a494d7210fa13de0e1affd4c02041bf8cf4fada674821980cc88a710a76 -e WEBSITE_USE_DIAGNOSTIC_SERVER=False datalust/seq:2023.4

2023-11-20T12:21:24.199Z INFO - Logging is not enabled for this container.
Please use https://aka.ms/linux-diagnostics to enable logging to see container logs here.
2023-11-20T12:21:25.511Z INFO - Initiating warmup request to container seq_0_6310e60f for site seq
2023-11-20T12:21:40.835Z ERROR - Container seq_0_6310e60f for site seq has exited, failing site start
2023-11-20T12:21:40.839Z ERROR - Container seq_0_6310e60f didn't respond to HTTP pings on port: 80, failing site start. See container logs for debugging.
2023-11-20T12:21:40.842Z INFO - Stopping site seq because it failed during startup.
`
and not start.

[nblumhardt]

Hi @porya0ras, thanks for your message.

Azure Web Apps don't support the features Seq needs to run reliably, and aren't a supported deployment platform: https://docs.datalust.co/docs/using-azure-websites-or-app-service

Your best bet is to grab a backup of your instance's configuration from the Backups/ folder under the /data mount, and to restore this to another instance.

To get this one running again, though there will be data loss, I think you can move the Stream/ folder from the /data mount to another location, and then restart the web app (all of the log data is in the Stream/ folder so won't appear in Seq after the restart).

If you have a current, supported Seq license, and want to attempt recovering the data in the (moved) Stream/ folder, please reach out via [email protected]; it's not certain that anything can be recovered, but if it's a mission-critical instance we can take a look.

Hope this helps,
Nick

[tormodfj]

For what it's worth, here is how I resolved this issue when it occurred to me.

Root Cause

Warning

Azure Websites, Container Apps, and App Service are unsupported

These services permit container instances to run concurrently, which will cause storage corruption when hosting Seq.

Ref.: https://docs.datalust.co/docs/using-azure-websites-or-app-service

In my particular case, one instance attempted to delete /Stream/stream.0.metadata while another instance had a lock on it. This caused a deadlock where the file was marked for deletion (making it unreadable), but unsuccessful in actually deleting it because of the other instance's lock.

Resolving the Issue

This is how I managed to resolve my issue, YMMV. The procedure assumes that Seq's data volume is an Azure SMB File Share. Also note that the order in which you perform the operations is important.

1. Stop the web-app that hosts Seq.
2. Create a snapshot of the file share, and verify that /Stream/stream.0.metadata is present and readable in the snapshot. (You will not be able to restore that file yet, because the original file is still locked and marked for deletion.)
3. Unlock the original file using e.g. the Az PowerShell module as described below 👇🏻. This will, in fact, result in the original file being deleted, since it was marked for deletion upon deadlocking. This is why we created the snapshot first.
4. Restore /Stream/stream.0.metadata from the snapshot.
5. Restart the web-app that hosts Seq.

Unlocking /Stream/stream.0.metadata

I followed this recipe: https://infra.engineer/azure/65-azure-clearing-the-lock-on-a-file-within-an-azure-file-share

In case that site should some day disappear, here is the procedure:

1. Install the Az PowerShell module if not already installed:
   Install-Module -Name Az -AllowClobber
2. Connect to the account/subscription where your storage account is located:
   Connect-AzAccount
3. Create a storage context for the target account:
   $Context = New-AzStorageContext -StorageAccountName "StorageAccountName" -StorageAccountKey "StorageAccessKey"
4. Close all file handles to /data/Stream/stream.0.metadata
Close-AzStorageFileHandle -Context $Context -ShareName "FileShareName" -Path 'Stream/stream.0.metadata' -CloseAll

At this point, stream.0.metadata should no longer be present in the Stream directory, since it was marked for deletion.