Skip to content

Commit

Permalink
The Big Linq Overhaul of 2023!
Browse files Browse the repository at this point in the history
Revamped Linq baseline support. Closes GH-2684

Ability to use compiled queries even when the generated SQL uses containment operators or JSONPath queries. Closes GH-2702

WIP: Refactored the Contains() query logic to centralize it a bit

Updated to Weasel 6.2, changed the shape of ICompiledQueryAwareFilter to get ready for containment filters

Added some test helper for detecting with a Linq query uses a CTE prior to improving child collection queries

Cleaning out some ununsed ContainmentWhereFragment code

WIP: refactored compiled queries quite a bit to get ready to handle JSONPath & containment operators in compiled queries

Revert "tweaking github builds"

This reverts commit fdfe09022191923242ca802f0c8449709ec90f1b.

maybe fixed some references to code snippets

tweaking github builds

removing older postgres versions

removing older Postgres versions

Fixed a likely merge problem w/ the duplicated field doc page

Fixed a combination of soft deleted documents being part of an Include() operation

Few more fixes for LINQ overhaul regressions

Bug fixing for a couple v7 merge issues

doc snippet change

Fixed new parsers to the v7 linq model
  • Loading branch information
jeremydmiller committed Nov 23, 2023
1 parent e9bea56 commit 22d698b
Show file tree
Hide file tree
Showing 561 changed files with 10,640 additions and 56,398 deletions.
12 changes: 0 additions & 12 deletions benchmarks/1.3/DocumentActions-report-default.md
Original file line number Diff line number Diff line change
@@ -1,12 +0,0 @@

BenchmarkDotNet=v0.10.1, OS=Microsoft Windows NT 6.1.7601 Service Pack 1
Processor=Intel(R) Core(TM) i7-4980HQ CPU 2.80GHz, ProcessorCount=4
Frequency=10000000 Hz, Resolution=100.0000 ns, Timer=UNKNOWN
[Host] : Clr 4.0.30319.42000, 64bit RyuJIT-v4.6.1076.0
Job-DNINOD : Clr 4.0.30319.42000, 64bit RyuJIT-v4.6.1076.0

WarmupCount=2 Gen 0=287.5000 Allocated=4.62 MB

Method | Mean | StdDev |
---------------- |----------- |---------- |
InsertDocuments | 43.7957 ms | 2.6786 ms |
16 changes: 0 additions & 16 deletions benchmarks/2.0.baseline/results/LinqActions-report-github.md
Original file line number Diff line number Diff line change
@@ -1,16 +0,0 @@
``` ini

BenchmarkDotNet=v0.10.1, OS=Microsoft Windows NT 6.1.7601 Service Pack 1
Processor=Intel(R) Core(TM) i7-4980HQ CPU 2.80GHz, ProcessorCount=4
Frequency=10000000 Hz, Resolution=100.0000 ns, Timer=UNKNOWN
[Host] : Clr 4.0.30319.42000, 64bit RyuJIT-v4.6.1076.0
Job-DNINOD : Clr 4.0.30319.42000, 64bit RyuJIT-v4.6.1076.0

WarmupCount=2

```
Method | Mean | StdDev | Gen 0 | Gen 1 | Allocated |
------------------ |---------------- |-------------- |---------- |-------- |---------- |
CreateLinqCommand | 189.7593 us | 2.3160 us | - | - | 42.64 kB |
RunLinqQuery | 124,248.2148 us | 1,785.5957 us | 4975.0000 | - | 34.13 MB |
CompiledQueries | 29,998.1235 us | 719.6757 us | 958.3333 | 83.3333 | 8.71 MB |
8 changes: 7 additions & 1 deletion build/build.cs
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,12 @@ private static async Task Main(string[] args)

Target("test-event-sourcing", DependsOn("compile-event-sourcing-tests"), () =>
RunTests("EventSourcingTests"));

Target("compile-linq-tests", DependsOn("clean"), () =>
Run("dotnet", $"build src/LinqTests/LinqTests.csproj --framework {_framework} --configuration {configuration}"));

Target("test-linq", DependsOn("compile-linq-tests"), () =>
RunTests("LinqTests"));

Target("test-codegen", () =>
{
Expand All @@ -112,7 +118,7 @@ private static async Task Main(string[] args)
Target("test-plv8", DependsOn("compile", "compile-plv8"), () =>
RunTests("Marten.PLv8.Testing"));

Target("test", DependsOn("test-base-lib", "test-core", "test-document-db", "test-event-sourcing", "test-cli", "test-codegen"));
Target("test", DependsOn("test-base-lib", "test-core", "test-document-db", "test-event-sourcing", "test-cli", "test-linq", "test-codegen"));

Target("test-extension-libs-without-plv8", DependsOn("test-noda-time", "test-aspnetcore"));

Expand Down
10 changes: 5 additions & 5 deletions docs/configuration/storeoptions.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ public class OrganizationRegistry: MartenRegistry
}
}
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Configuration/MartenRegistryTests.cs#L149-L160' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_organizationregistry' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Configuration/MartenRegistryTests.cs#L153-L164' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_organizationregistry' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

To apply your new `MartenRegistry`, just include it when you bootstrap the `IDocumentStore` as in this example:
Expand All @@ -109,7 +109,7 @@ var store = DocumentStore.For(opts =>
opts.Connection(ConnectionSource.ConnectionString);
});
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Configuration/MartenRegistryTests.cs#L182-L191' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_including_a_custom_martenregistry' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Configuration/MartenRegistryTests.cs#L186-L195' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_including_a_custom_martenregistry' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

Do note that you could happily use multiple `MartenRegistry` classes in larger applications if that is advantageous.
Expand All @@ -131,7 +131,7 @@ var store = DocumentStore.For(opts =>
.For<User>().Duplicate(x => x.UserName);
});
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Configuration/MartenRegistryTests.cs#L164-L176' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_using_storeoptions_schema' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Configuration/MartenRegistryTests.cs#L168-L180' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_using_storeoptions_schema' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

## Custom Attributes
Expand Down Expand Up @@ -202,7 +202,7 @@ public class ConfiguresItself
}
}
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Configuration/DocumentMappingTests.cs#L126-L138' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_configuremarten-generic' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Configuration/DocumentMappingTests.cs#L788-L800' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_configuremarten-generic' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

The `DocumentMapping` type is the core configuration class representing how a document type is persisted or
Expand All @@ -226,7 +226,7 @@ public class ConfiguresItselfSpecifically
}
}
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Configuration/DocumentMappingTests.cs#L140-L153' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_configuremarten-specifically' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Configuration/DocumentMappingTests.cs#L802-L815' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_configuremarten-specifically' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

## Document Policies
Expand Down
8 changes: 4 additions & 4 deletions docs/documents/deletes.md
Original file line number Diff line number Diff line change
Expand Up @@ -203,7 +203,7 @@ public void query_soft_deleted_docs()
.ToList().Single().UserName.ShouldBe("foo");
}
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Deleting/soft_deletes.cs#L275-L301' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_query_soft_deleted_docs' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Deleting/soft_deletes.cs#L280-L306' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_query_soft_deleted_docs' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

The SQL generated for the first call to `Query<User>()` above would be:
Expand Down Expand Up @@ -247,7 +247,7 @@ public void query_maybe_soft_deleted_docs()
.ShouldHaveTheSameElementsAs("bar", "baz", "foo");
}
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Deleting/soft_deletes.cs#L303-L331' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_query_maybe_soft_deleted_docs' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Deleting/soft_deletes.cs#L308-L336' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_query_maybe_soft_deleted_docs' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

### Fetching Only Deleted Documents
Expand Down Expand Up @@ -285,7 +285,7 @@ public void query_is_soft_deleted_docs()
.Single().ShouldBe("bar");
}
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Deleting/soft_deletes.cs#L333-L361' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_query_is_soft_deleted_docs' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Deleting/soft_deletes.cs#L338-L366' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_query_is_soft_deleted_docs' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

### Fetching Documents Deleted Before or After a Specific Time
Expand Down Expand Up @@ -319,7 +319,7 @@ public void query_is_soft_deleted_since_docs()
.ToList().ShouldHaveTheSameElementsAs("jack");
}
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Deleting/soft_deletes.cs#L363-L387' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_query_soft_deleted_since' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Deleting/soft_deletes.cs#L368-L392' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_query_soft_deleted_since' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

_Neither `DeletedSince` nor `DeletedBefore` are inclusive searches as shown_below:
Expand Down
13 changes: 8 additions & 5 deletions docs/documents/hierarchies.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ public class BrainySmurf: PapaSmurf
{
}
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Reading/Linq/query_with_inheritance.cs#L12-L42' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_smurfs-hierarchy' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/LinqTests/Acceptance/query_with_inheritance.cs#L12-L42' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_smurfs-hierarchy' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

If you wish to query over one of hierarchy classes and be able to get all of its documents as well as its subclasses,
Expand All @@ -98,8 +98,9 @@ first you will need to map the hierarchy like so:
<!-- snippet: sample_add-subclass-hierarchy -->
<a id='snippet-sample_add-subclass-hierarchy'></a>
```cs
public query_with_inheritance()
public query_with_inheritance(ITestOutputHelper output)
{
_output = output;
StoreOptions(_ =>
{
_.Schema.For<ISmurf>()
Expand All @@ -119,7 +120,7 @@ public query_with_inheritance()
});
}
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Reading/Linq/query_with_inheritance.cs#L86-L110' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_add-subclass-hierarchy' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/LinqTests/Acceptance/query_with_inheritance.cs#L91-L116' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_add-subclass-hierarchy' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

Note that if you wish to use aliases on certain subclasses, you could pass a `MappedType`, which contains the type to map
Expand All @@ -138,7 +139,7 @@ _.Schema.For<ISmurf>()
typeof(BrainySmurf)
);
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Reading/Linq/query_with_inheritance.cs#L50-L61' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_add-subclass-hierarchy-with-aliases' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/LinqTests/Acceptance/query_with_inheritance.cs#L53-L64' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_add-subclass-hierarchy-with-aliases' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

Now you can query the "complex" hierarchy in the following ways:
Expand Down Expand Up @@ -169,6 +170,8 @@ public void get_all_subclasses_of_a_subclass2()

theSession.SaveChanges();

theSession.Logger = new TestOutputMartenLogger(_output);

theSession.Query<PapaSmurf>().Count().ShouldBe(2);
}

Expand Down Expand Up @@ -232,5 +235,5 @@ public void get_all_subclasses_of_an_interface()
theSession.Query<IPapaSmurf>().Count().ShouldBe(3);
}
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Reading/Linq/query_with_inheritance.cs#L144-L234' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_query-subclass-hierarchy' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/LinqTests/Acceptance/query_with_inheritance.cs#L150-L242' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_query-subclass-hierarchy' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->
149 changes: 0 additions & 149 deletions docs/documents/indexing/foreign-keys.md
Original file line number Diff line number Diff line change
@@ -1,149 +0,0 @@
# Foreign Keys

Marten **is** built on top of a relational database, so why not take advantage of those abilities
where they still add value? In this case, Marten allows for a special kind of "Searchable" column
that also adds a foreign key constraint to enforce referential integrity between document types.

One of our sample document types in Marten is the `Issue` class that has
a couple properties that link to the id's of related `User` documents:

<!-- snippet: sample_Issue -->
<a id='snippet-sample_issue'></a>
```cs
public class Issue
{
public Issue()
{
Id = Guid.NewGuid();
}

public string[] Tags { get; set; }

public Guid Id { get; set; }

public string Title { get; set; }

public int Number { get; set; }

public Guid? AssigneeId { get; set; }

public Guid? ReporterId { get; set; }

public Guid? BugId { get; set; }
public string Status { get; set; }
}
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/Marten.Testing/Documents/Issue.cs#L5-L29' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_issue' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

If I want to enforce referential integrity between the `Issue` document and the `User` documents,
I can use this syntax shown below to configure Marten:

<!-- snippet: sample_configure-foreign-key -->
<a id='snippet-sample_configure-foreign-key'></a>
```cs
var store = DocumentStore
.For(_ =>
{
_.Connection("some database connection");

// In the following line of code, I'm setting
// up a foreign key relationship to the User document
_.Schema.For<Issue>().ForeignKey<User>(x => x.AssigneeId);
});
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/Marten.Testing/Examples/ForeignKeyExamples.cs#L11-L21' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_configure-foreign-key' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

With the configuration above, Marten will make an `assignee_id` field in the database table and build a
foreign key constraint to the `User` document like so:

```sql
ALTER TABLE public.mt_doc_issue
ADD CONSTRAINT mt_doc_issue_assignee_id_fkey FOREIGN KEY (assignee_id)
REFERENCES public.mt_doc_user (id);

CREATE INDEX mt_doc_issue_idx_assignee_id ON public.mt_doc_issue ("assignee_id");
```

And some other things you probably want to know about how this works internally:

Marten is smart enough to order the "upsert" operations to make the dependent documents be updated last.
In the `Issue` referencing `User` example above, this means that if you create a new `User` and a new
`Issue` in the same session, when you call `IDocumentSession.SaveChanges()/SaveChangesAsync()`, Marten will know
to save the new user first so that the issue will not fail with referential integrity violations.

## Foreign Keys to non-Marten tables

Marten can also create a foreign key to tables that are not managed by Marten. Continuing the our sample
of `Issue`, we can create a foreign key from our `Issue` to our external bug tracking system:

<!-- snippet: sample_configure-external-foreign-key -->
<a id='snippet-sample_configure-external-foreign-key'></a>
```cs
var store = DocumentStore
.For(_ =>
{
_.Connection("some database connection");

// Here we create a foreign key to table that is not
// created or managed by marten
_.Schema.For<Issue>().ForeignKey(i => i.BugId, "bugtracker", "bugs", "id");
});
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/Marten.Testing/Examples/ForeignKeyExamples.cs#L29-L39' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_configure-external-foreign-key' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

With the configuration above, Marten will generate a foreign key constraint from the `Issue` to a table in the
`bug-tracker` schema called `bugs` on the `id` column. The constraint would be defined as:

```sql
ALTER TABLE public.mt_doc_issue
ADD CONSTRAINT mt_doc_issue_bug_id_fkey FOREIGN KEY (bug_id)
REFERENCES bug-tracker.bugs (id);
```

## Cascading deletes

Marten can also cascade deletes on the foreign keys that it creates. The `ForeignKeyDefinition` has a
`CascadeDeletes` property that indicates whether the foreign key should enable cascading deletes. One way
to enable this is to use a configuration function like:

<!-- snippet: sample_cascade_deletes_with_config_func -->
<a id='snippet-sample_cascade_deletes_with_config_func'></a>
```cs
var store = DocumentStore
.For(_ =>
{
_.Connection("some database connection");

_.Schema.For<Issue>().ForeignKey<User>(x => x.AssigneeId, fkd => fkd.OnDelete = CascadeAction.Cascade);
});
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/Marten.Testing/Examples/ForeignKeyExamples.cs#L44-L52' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_cascade_deletes_with_config_func' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

## Configuring with Attributes

You can optionally configure properties or fields as foreign key relationships with the `[ForeignKey]` attribute:

<!-- snippet: sample_issue-with-fk-attribute -->
<a id='snippet-sample_issue-with-fk-attribute'></a>
```cs
public class Issue
{
public Issue()
{
Id = Guid.NewGuid();
}

public Guid Id { get; set; }

[ForeignKey(typeof(User))]
public Guid UserId { get; set; }

public Guid? OtherUserId { get; set; }
}
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/ForeignKeys/configuring_foreign_key_fields.cs#L68-L84' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_issue-with-fk-attribute' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->
3 changes: 2 additions & 1 deletion docs/documents/indexing/metadata-indexes.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,10 @@ an index on the document's `mt_last_modified` metadata column either using `Inde
[IndexedLastModified]
public class Customer
{
public Guid Id { get; set; }
}
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Metadata/last_modified_queries.cs#L26-L31' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_index-last-modified-via-attribute' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Metadata/last_modified_queries.cs#L26-L32' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_index-last-modified-via-attribute' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

Or by using the fluent interface:
Expand Down
2 changes: 1 addition & 1 deletion docs/documents/metadata.md
Original file line number Diff line number Diff line change
Expand Up @@ -215,7 +215,7 @@ public async Task sample_usage(IQuerySession session)
.ToListAsync();
}
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Metadata/last_modified_queries.cs#L34-L48' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_last_modified_queries' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Metadata/last_modified_queries.cs#L35-L49' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_last_modified_queries' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

## Indexing
Expand Down
Loading

0 comments on commit 22d698b

Please sign in to comment.