Event store partitioning for hot/cold storage and documentation on al…

…l the new 7.25 event store optimizations. Closes GH-770. Closes GH-3321

docs/.vitepress/config.mts

@@ -164,6 +164,8 @@ const config: UserConfig<DefaultTheme.Config> = {
             { text: 'Querying Events', link: '/events/querying' },
             { text: 'Metadata', link: '/events/metadata' },
             { text: 'Archiving Streams', link: '/events/archiving' },
+            { text: 'Optimizing Performance', link: '/events/optimizing' },
+
             {
               text: 'Projections Overview', link: '/events/projections/', collapsed: true, items: [
                 {

docs/configuration/storeoptions.md

@@ -211,7 +211,7 @@ public class ConfiguresItself
     }
 }
 ```
-<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Configuration/DocumentMappingTests.cs#L801-L813' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_configuremarten-generic' title='Start of snippet'>anchor</a></sup>
+<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Configuration/DocumentMappingTests.cs#L808-L820' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_configuremarten-generic' title='Start of snippet'>anchor</a></sup>
 <!-- endSnippet -->
 
 The `DocumentMapping` type is the core configuration class representing how a document type is persisted or
@@ -235,7 +235,7 @@ public class ConfiguresItselfSpecifically
     }
 }
 ```
-<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Configuration/DocumentMappingTests.cs#L815-L828' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_configuremarten-specifically' title='Start of snippet'>anchor</a></sup>
+<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Configuration/DocumentMappingTests.cs#L822-L835' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_configuremarten-specifically' title='Start of snippet'>anchor</a></sup>
 <!-- endSnippet -->
 
 ## Document Policies

docs/diagnostics.md

@@ -4,6 +4,37 @@ So far, Marten has diagnostics, command logging, and unit of work life cycle tra
 
 For information on accessing and previewing the database schema objects generated by Marten, see [Marten and Postgres Schema](/schema/)
 
+## Disabling Npgsql Logging <Badge type="tip" text="7.0" />
+
+The built in Npgsql logging is turned on by default in Marten, so to disable that logging so you
+can actually glean some value from your logs without blowing up the storage costs for your logging
+provider, use this flag:
+
+<!-- snippet: sample_disabling_npgsql_logging -->
+<a id='snippet-sample_disabling_npgsql_logging'></a>
+```cs
+var builder = Host.CreateDefaultBuilder();
+builder.ConfigureServices(services =>
+{
+    services.AddMarten(opts =>
+    {
+        opts.Connection(ConnectionSource.ConnectionString);
+
+        // Disable the absurdly verbose Npgsql logging
+        opts.DisableNpgsqlLogging = true;
+
+        opts.Events.AppendMode = EventAppendMode.Quick;
+        opts.Events.UseIdentityMapForInlineAggregates = true;
+
+        opts.Projections.Add<DaemonTests.TestingSupport.TripProjection>(ProjectionLifecycle.Inline);
+    });
+});
+```
+<sup><a href='https://github.com/JasperFx/marten/blob/master/src/EventAppenderPerfTester/Program.cs#L8-L27' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_disabling_npgsql_logging' title='Start of snippet'>anchor</a></sup>
+<!-- endSnippet -->
+
+The Marten team will be considering reversing the default for this behavior in Marten 8.
+
 ## Listening for Document Store Events
 
 ::: tip INFO
@@ -508,7 +539,7 @@ The `IMartenLogger` can be swapped out on any `IQuerySession` or `IDocumentSessi
 // session to pipe Marten logging to the xUnit.Net output
 theSession.Logger = new TestOutputMartenLogger(_output);
 ```
-<sup><a href='https://github.com/JasperFx/marten/blob/master/src/EventSourcingTests/archiving_events.cs#L231-L237' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_replacing_logger_per_session' title='Start of snippet'>anchor</a></sup>
+<sup><a href='https://github.com/JasperFx/marten/blob/master/src/EventSourcingTests/archiving_events.cs#L301-L307' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_replacing_logger_per_session' title='Start of snippet'>anchor</a></sup>
 <!-- endSnippet -->
 
 ## Previewing the PostgreSQL Query Plan

docs/documents/identity.md

@@ -337,7 +337,7 @@ public struct Task2Id
     public static Task2Id From(Guid value) => new Task2Id(value);
 }
 ```
-<sup><a href='https://github.com/JasperFx/marten/blob/master/src/ValueTypeTests/TestingTypes.cs#L29-L50' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_valid_strong_typed_identifiers' title='Start of snippet'>anchor</a></sup>
+<sup><a href='https://github.com/JasperFx/marten/blob/master/src/ValueTypeTests/TestingTypes.cs#L30-L51' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_valid_strong_typed_identifiers' title='Start of snippet'>anchor</a></sup>
 <!-- endSnippet -->
 
 In _all_ cases, the type name will have to be suffixed with "Id" (and it's case sensitive) to be considered by Marten to be

docs/documents/storage.md

@@ -55,7 +55,7 @@ public class Customer
     [Identity] public string Name { get; set; }
 }
 ```
-<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Configuration/DocumentMappingTests.cs#L830-L838' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_using_databaseschemaname_attribute' title='Start of snippet'>anchor</a></sup>
+<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Configuration/DocumentMappingTests.cs#L837-L845' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_using_databaseschemaname_attribute' title='Start of snippet'>anchor</a></sup>
 <!-- endSnippet -->
 
 ## Type Aliases

docs/documents/storing.md

@@ -81,7 +81,7 @@ using (var session = theStore.LightweightSession())
     session.SaveChanges();
 }
 ```
-<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Writing/document_inserts.cs#L74-L82' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_sample-document-insertonly' title='Start of snippet'>anchor</a></sup>
+<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Writing/document_inserts.cs#L75-L83' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_sample-document-insertonly' title='Start of snippet'>anchor</a></sup>
 <!-- endSnippet -->
 
 ## Bulk Loading
@@ -151,7 +151,7 @@ await store.BulkInsertDocumentsAsync(data, BulkInsertMode.InsertsOnly);
 // being loaded
 await store.BulkInsertDocumentsAsync(data, BulkInsertMode.OverwriteExisting);
 ```
-<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Writing/bulk_loading.cs#L302-L321' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_bulkinsertmode_usages' title='Start of snippet'>anchor</a></sup>
+<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Writing/bulk_loading.cs#L329-L348' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_bulkinsertmode_usages' title='Start of snippet'>anchor</a></sup>
 <!-- endSnippet -->
 
 The bulk insert feature can also be used with multi-tenanted documents, but in that
@@ -173,5 +173,5 @@ using var store = DocumentStore.For(opts =>
 // If multi-tenanted
 await store.BulkInsertDocumentsAsync("a tenant id", data);
 ```
-<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Writing/bulk_loading.cs#L326-L340' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_multitenancywithbulkinsert' title='Start of snippet'>anchor</a></sup>
+<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Writing/bulk_loading.cs#L353-L367' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_multitenancywithbulkinsert' title='Start of snippet'>anchor</a></sup>
 <!-- endSnippet -->

docs/events/appending.md

@@ -1,8 +1,10 @@
 # Appending Events
 
 ::: tip
-Marten V5.4 introduced the new `FetchForWriting()` and `IEventStream` models that streamline the workflow of capturing events against
-an aggregated "write" model.
+For CQRS style command handlers that append events to an existing event stream, the Marten team very
+strongly recommends the [FetchForWriting](/scenarios/command_handler_workflow) API. This API is used underneath
+the Wolverine [Aggregate Handler Workflow](https://wolverinefx.net/guide/durability/marten/event-sourcing.html) that is probably the very simplest possible way to build command handlers
+with Marten event sourcing today.
 :::
 
 With Marten, events are captured and appended to logical "streams" of events. Marten provides
@@ -18,21 +20,55 @@ The event data is persisted to two tables:
 Events can be captured by either starting a new stream or by appending events to an existing stream. In addition, Marten has some tricks up its sleeve for dealing
 with concurrency issues that may result from multiple transactions trying to simultaneously append events to the same stream.
 
-## "Rich" vs "Quick" Appends <Badge type="tip" text="7.21" />
+## "Rich" vs "Quick" Appends <Badge type="tip" text="7.25" />
+
+::: tip
+Long story short, the new "Quick" model appears to provide much better performance and scalability.
+:::
 
 Before diving into starting new event streams or appending events to existing streams, just know that there are two different
 modes of event appending you can use with Marten:
 
-snippet: sample_configuring_event_append_mode
+<!-- snippet: sample_configuring_event_append_mode -->
+<a id='snippet-sample_configuring_event_append_mode'></a>
+```cs
+var builder = Host.CreateApplicationBuilder();
+builder.Services.AddMarten(opts =>
+    {
+        // This is the default Marten behavior from 4.0 on
+        opts.Events.AppendMode = EventAppendMode.Rich;
+
+        // Lighter weight mode that should result in better
+        // performance, but with a loss of available metadata
+        // within inline projections
+        opts.Events.AppendMode = EventAppendMode.Quick;
+    })
+    .UseNpgsqlDataSource();
+```
+<sup><a href='https://github.com/JasperFx/marten/blob/master/src/EventSourcingTests/QuickAppend/Examples.cs#L12-L27' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_configuring_event_append_mode' title='Start of snippet'>anchor</a></sup>
+<!-- endSnippet -->
 
 The classic `Rich` mode will append events in a two step process where the local session will first determine all possible
 metadata for the events about to be appended such that inline projections can use event versions and the global event sequence
 numbers at the time that the inline projections are created. 
 
+::: warning
+If you are using `Inline` projections with the "Quick" mode, just be aware that you will not have access to the final
+event sequence or stream version at the time the projections are built. Marten _is_ able to set the stream version into
+a single stream projection document built `Inline`, but that's done on the server side. Just be warned.
+:::
+
 The newer `Quick` mode eschews version and sequence metadata in favor of performing the event append and stream creation
 operations with minimal overhead. The improved performance comes at the cost of not having the `IEvent.Version` and `IEvent.Sequence`
 information available at the time that inline projections are executed.
 
+From initial load testing, the "Quick" mode appears to lead to a 40-50% time reduction Marten's process of appending
+events. Your results will vary of course. Maybe more importantly, the "Quick" mode seems to make a large positive
+in the functioning of the asynchronous projections and subscriptions by preventing the event "skipping" issue that
+can happen with the "Rich" mode when a system becomes slow under heavy loads. Lastly, the Marten team believes that the
+"Quick" mode can alleviate concurrency issues from trying to append events to the same stream without utilizing optimistic
+or exclusive locking on the stream.
+
 If using inline projections for a single stream (`SingleStreamProjection` or _snapshots_) and the `Quick` mode, the Marten team
 highly recommends using the `IRevisioned` interface on your projected aggregate documents so that Marten can "move" the version
 set by the database operations to the version of the projected documents loaded from the database later. Mapping a custom member
@@ -167,3 +203,4 @@ in the event store sequence due to failed transactions. Marten V4 introduced sup
 event sequence numbers that failed in a Marten transaction. This is done strictly to improve the functioning of the [async daemon](/events/projections/async-daemon) that looks for gaps in the event sequence to "know" how
 far it's safe to process asynchronous projections. If you see event rows in your database of type "tombstone", it's representative of failed transactions (maybe from optimistic concurrency violations,
 transient network issues, timeouts, etc.).
+

docs/events/archiving.md

@@ -1,9 +1,24 @@
 # Archiving Event Streams
 
-New in Marten V4 is the ability to mark an event stream and all of its events as "archived." While
-in the future this may have serious optimization benefits when Marten is able to utilize
-Postgresql sharding, today it's metadata and default filtering in the Linq querying against event
-data as well as asynchronous projections inside of the [async daemon](/events/projections/async-daemon).
+Like most (all?) event stores, Marten is designed around the idea of the events being persisted to a single file, immutable
+log of events. All the same though, there are going to be problem domains where certain event streams become obsolete. Maybe
+because a workflow is completed, maybe through time based expiry rules, or maybe because a customer or user is removed
+from the system. To help optimize Marten's event store usage, you can take advantage of the stream archiving to 
+mark events as archived on a stream by stream basis. 
+
+::: warning
+You can obviously use pure SQL to modify the events persisted by Marten. While that might be valuable in some cases,
+we urge you to be cautious about doing so.
+:::
+
+The impact of archiving an event stream is:
+
+* In the "classic" usage of Marten, the relevant stream and event rows are marked with an `is_archived = TRUE`
+* With the "opt in" table partitioning model for "hot/cold" storage described in the next section, the stream and event rows are
+  moved to the archived partition tables for streams and events
+* The [async daemon](/events/projections/async-daemon) subsystem process that processes projections and subscriptions in a background process automatically ignores
+  archived events -- but that can be modified on a per projection/subscription basis
+* Archived events are excluded by default from any event data queries through the LINQ support in Marten
 
 To mark a stream as archived, it's just this syntax:
 
@@ -16,13 +31,18 @@ public async Task SampleArchive(IDocumentSession session, string streamId)
     await session.SaveChangesAsync();
 }
 ```
-<sup><a href='https://github.com/JasperFx/marten/blob/master/src/EventSourcingTests/archiving_events.cs#L27-L35' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_archive_stream_usage' title='Start of snippet'>anchor</a></sup>
+<sup><a href='https://github.com/JasperFx/marten/blob/master/src/EventSourcingTests/archiving_events.cs#L28-L36' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_archive_stream_usage' title='Start of snippet'>anchor</a></sup>
 <!-- endSnippet -->
 
 As in all cases with an `IDocumentSession`, you need to call `SaveChanges()` to commit the
 unit of work.
 
-The `mt_events` and `mt_streams` tables now both have a boolean column named `is_archived`.
+::: tip
+At this point, you will also have to manually delete any projected aggregates based on the event streams being
+archived if that is desirable
+:::
+
+The `mt_events` and `mt_streams` tables both have a boolean column named `is_archived`.
 
 Archived events are filtered out of all event Linq queries by default. But of course, there's a way
 to query for archived events with the `IsArchived` property of `IEvent` as shown below:
@@ -35,7 +55,7 @@ var events = await theSession.Events
     .Where(x => x.IsArchived)
     .ToListAsync();
 ```
-<sup><a href='https://github.com/JasperFx/marten/blob/master/src/EventSourcingTests/archiving_events.cs#L166-L173' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_querying_for_archived_events' title='Start of snippet'>anchor</a></sup>
+<sup><a href='https://github.com/JasperFx/marten/blob/master/src/EventSourcingTests/archiving_events.cs#L228-L235' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_querying_for_archived_events' title='Start of snippet'>anchor</a></sup>
 <!-- endSnippet -->
 
 You can also query for all events both archived and not archived with `MaybeArchived()`
@@ -47,5 +67,45 @@ like so:
 var events = await theSession.Events.QueryAllRawEvents()
     .Where(x => x.MaybeArchived()).ToListAsync();
 ```
-<sup><a href='https://github.com/JasperFx/marten/blob/master/src/EventSourcingTests/archiving_events.cs#L197-L202' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_query_for_maybe_archived_events' title='Start of snippet'>anchor</a></sup>
+<sup><a href='https://github.com/JasperFx/marten/blob/master/src/EventSourcingTests/archiving_events.cs#L263-L268' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_query_for_maybe_archived_events' title='Start of snippet'>anchor</a></sup>
+<!-- endSnippet -->
+
+## Hot/Cold Storage Partitioning <Badge type="tip" text="7.25" />
+
+::: warning
+This option will only be beneficial if you are being aggressive about marking obsolete, old, or expired event data
+as archived.
+:::
+
+Want your system using Marten to scale and perform even better than it already does? If you're leveraging
+event archiving in your application workflow, you can possibly derive some significant performance and scalability
+improvements by opting into using PostgreSQL native table partitioning on the event and event stream data
+to partition the "hot" (active) and "cold" (archived) events into separate partition tables. 
+
+The long and short of this option is that it keeps the active `mt_streams` and `mt_events` tables smaller, which pretty
+well always results in better performance over time.
+
+The simple flag for this option is:
+
+<!-- snippet: sample_turn_on_stream_archival_partitioning -->
+<a id='snippet-sample_turn_on_stream_archival_partitioning'></a>
+```cs
+var builder = Host.CreateApplicationBuilder();
+builder.Services.AddMarten(opts =>
+{
+    opts.Connection("some connection string");
+
+    // Turn on the PostgreSQL table partitioning for
+    // hot/cold storage on archived events
+    opts.Events.UseArchivedStreamPartitioning = true;
+});
+```
+<sup><a href='https://github.com/JasperFx/marten/blob/master/src/EventSourcingTests/Examples/Optimizations.cs#L13-L25' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_turn_on_stream_archival_partitioning' title='Start of snippet'>anchor</a></sup>
 <!-- endSnippet -->
+
+::: warning
+If you are turning this option on to an existing system, you may want to run the database schema migration script
+by hand rather than trying to let Marten do it automatically. The data migration from non-partitioned to partitioned
+will probably require system downtime because it actually has to copy the old table data, drop the old table, create the new 
+table, copy all the existing data from the temp table to the new partitioned table, and finally drop the temporary table.
+:::