Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FEATURE: Subscription Engine #5321

Draft
wants to merge 113 commits into
base: 9.0
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
113 commits
Select commit Hold shift + click to select a range
2172f09
WIP
bwaidelich Oct 24, 2024
13b1073
Merge branch '9.0' into feature/4746-rework-catchup-mechanism-3
bwaidelich Oct 24, 2024
d75f174
Merge branch '9.0' into feature/4746-rework-catchup-mechanism-3
bwaidelich Oct 24, 2024
937f84d
Merge branch '9.0' into feature/4746-rework-catchup-mechanism-3
bwaidelich Oct 26, 2024
03fb7ca
Merge branch '9.0' into feature/4746-rework-catchup-mechanism-3
bwaidelich Oct 29, 2024
dcb4be2
Merge branch '9.0' into feature/4746-rework-catchup-mechanism-3
bwaidelich Nov 3, 2024
fafe080
Merge branch '9.0' into feature/4746-rework-catchup-mechanism-3
bwaidelich Nov 5, 2024
29ba908
WIP
bwaidelich Nov 5, 2024
207179b
Tweak type comments
bwaidelich Nov 6, 2024
a127706
Wiring...
bwaidelich Nov 13, 2024
765b84f
Merge branch '9.0' into feature/4746-rework-catchup-mechanism-3
bwaidelich Nov 13, 2024
5b035c1
first (almost) working version
bwaidelich Nov 14, 2024
aca0f9b
Fix CatchUpHooks
bwaidelich Nov 14, 2024
7fea53e
Replace ProjectionService
bwaidelich Nov 14, 2024
ec08d64
remove `EventPersister` and much more :)
bwaidelich Nov 14, 2024
9fef068
Remove `RunSubscriptionEventStore`
bwaidelich Nov 14, 2024
46ad9d1
Improve error handling (WIP)
bwaidelich Nov 14, 2024
5ea8fae
Merge branch '9.0' into feature/4746-rework-catchup-mechanism-3
bwaidelich Nov 17, 2024
88cc600
Fix `ContentRepositoryFactory` constructor
bwaidelich Nov 17, 2024
e3f85ac
Improve error handling during `SubscriptionEngine::setup()`
bwaidelich Nov 17, 2024
c43a2ea
Allow to reset subscriptions
bwaidelich Nov 18, 2024
40e8d35
Re-implement `cr:status` CLI command
bwaidelich Nov 18, 2024
3744fd5
Fix `test_parallel` cr settings
bwaidelich Nov 18, 2024
c0fbfe1
Fix `test_parallel` cr settings 2/2
bwaidelich Nov 18, 2024
f5ff7d6
Tweak subscription engine setup/reset from tests
bwaidelich Nov 18, 2024
f8a8b5b
Fix behat tests ?
bwaidelich Nov 18, 2024
1562435
Disable `EventExportProcessor` test
bwaidelich Nov 18, 2024
2902dc2
Revert "Disable `EventExportProcessor` test"
bwaidelich Nov 18, 2024
0fe05b5
TASK: Fix phpstan
mhsdesign Nov 20, 2024
6e5b565
Merge remote-tracking branch 'origin/9.0' into feature/4746-rework-ca…
mhsdesign Nov 20, 2024
8967ead
TASK: Add sanity check assertion after setting node properties a coup…
mhsdesign Nov 20, 2024
2b5d338
TASK: Introduce dedicated `contentRepositoryLogger`
mhsdesign Nov 20, 2024
2d3a136
Fix parallel tests and publish events on correct stream
mhsdesign Nov 20, 2024
46dc510
TASK: Simplify `normalizeEvents`
mhsdesign Nov 20, 2024
0680533
Merge remote-tracking branch 'origin/9.0' into feature/4746-rework-ca…
mhsdesign Nov 20, 2024
de92d79
TASK: Test that locking and concurrent writing works under heavy load
mhsdesign Nov 20, 2024
f58cefa
TASK: Improve exception thrown if subscriber failed
mhsdesign Nov 20, 2024
e15ae79
TASK: Fix of-by-one error in catchup
mhsdesign Nov 20, 2024
c7df820
TASK: Improve logging and simplify information on debug
mhsdesign Nov 20, 2024
0bd129a
TASK: Remove obsolete ContentRepositorySubscribersFactoryInterface
mhsdesign Nov 20, 2024
1dfaa2c
TASK: Remove generic `EventHandlerInterface` for now, everything is a…
mhsdesign Nov 20, 2024
441b035
TASK: Remove other generic subscription concepts not required for pro…
mhsdesign Nov 20, 2024
9c6ba75
TASK: Remove subscription groups and filtering except for status and ids
mhsdesign Nov 21, 2024
0ac8751
TASK: Remove `$skipBooting` because its an odd signature and unused
mhsdesign Nov 20, 2024
9be3308
TASK: Remove removal of subscriptions
mhsdesign Nov 20, 2024
53790b9
TASK: Remove sqlite support for `DoctrineSubscriptionStore` as db loc…
mhsdesign Nov 20, 2024
481f173
TASK: Inline `ProjectionEventHandler` to `Subscriber` and make it a `…
mhsdesign Nov 21, 2024
b9569c5
TASK: Introduce tests for subscription booting, active and error state
mhsdesign Nov 21, 2024
427829e
TASK: Allow to replace setting of cr registry via `injectSettings`
mhsdesign Nov 21, 2024
715ec2e
TASK: Subscription engine test new and detached status
mhsdesign Nov 21, 2024
512e3c4
TASK: Speedup tests by using truncate
mhsdesign Nov 21, 2024
408ceb2
TASK: Subscription engine test filtering by subscription id
mhsdesign Nov 21, 2024
b71165c
TASK: Subscription engine test `filteringReset`
mhsdesign Nov 22, 2024
c8f8c6a
TASK: introduce `DebugEventProjection` for testing to assert each eve…
mhsdesign Nov 22, 2024
91046e6
TASK: introduce test that projection is rollback'd in case of error
mhsdesign Nov 22, 2024
f624ac6
TASK: test catchup hooks on failure
mhsdesign Nov 22, 2024
0197017
TASK: Introduce test to assert that projection keeps events previousl…
mhsdesign Nov 23, 2024
3b6ca26
TASK: Improve catchup rollback test
mhsdesign Nov 23, 2024
794be11
TASK: Split up mighty `SubscriptionEngineTest`
mhsdesign Nov 23, 2024
752d434
TASK: Simplify AbstractSubscriptionEngineTestCase by moving out speci…
mhsdesign Nov 23, 2024
c4da7fe
TASK: Test error behaviour for onBeforeCatchUp and onAfterCatchUp
mhsdesign Nov 23, 2024
c8cb0a2
TASK: Add test for happy catchup hooks
mhsdesign Nov 23, 2024
dfe5eb3
TASK: Adjust subscription test exceptions to do no retry
mhsdesign Nov 23, 2024
a32bfb2
TASK: Remove `retry_attempt` from subscriptions
mhsdesign Nov 23, 2024
836c347
TASK: Rename factory back to `$additionalSubscriberFactories`
mhsdesign Nov 23, 2024
830b5cc
BUGFIX: `discoverDetachedSubscriptions` did not persist changes
mhsdesign Nov 23, 2024
3d43183
Merge pull request #5375 from mhsdesign/task/radical-cleanup-for-subs…
mhsdesign Nov 23, 2024
6f43825
TASK: Fix phpstan
mhsdesign Nov 23, 2024
60d4f8c
BUGFIX: Reintroduce catchup hooks for all projections
mhsdesign Nov 23, 2024
5ef4ab1
TASK: Adjust to doctrine deprecations in DoctrineSubscriptionStore
mhsdesign Nov 23, 2024
c7cb75b
TASK: Minor code adjustments
mhsdesign Nov 23, 2024
a80639b
TASK: Ensure that the content graph projection is not part of the gen…
mhsdesign Nov 23, 2024
e7acfaa
BUGFIX: ProjectionErrorTest::fixFailedProjection reset error on reset
mhsdesign Nov 23, 2024
fdeec75
TASK: Assertions that setup and boot do not retry failed projections
mhsdesign Nov 23, 2024
31913f5
BUGFIX: SubscriptionDetachedStatusTest::projectionIsDetachedIfConfigu…
mhsdesign Nov 23, 2024
c5ea757
TASK: Throw `CatchUpFailed` exception in case onBeforeCatchUp or onAf…
mhsdesign Nov 23, 2024
794eaf2
TASK: Use save points to rollback projections during transaction on f…
mhsdesign Nov 23, 2024
7852f61
TASK: Handle `TableNotFoundException` gracefully in `subscriptionStat…
mhsdesign Nov 23, 2024
d582766
TASK: Move back to `subscriptionStatuses` test. We need to make sure …
mhsdesign Nov 23, 2024
5dfa592
TASK: Inline `discoverDetachedSubscriptions`
mhsdesign Nov 23, 2024
d5715c7
TASK: Do not discover new subscriptions during catchup
mhsdesign Nov 23, 2024
37a4e47
TASK: Introduce further tests to assert behaviour for catchup and setup
mhsdesign Nov 24, 2024
a8f246b
BUGFIX: Setup must re-setup active projections for migrations
mhsdesign Nov 24, 2024
73e1097
BUGFIX: Setup should reattach detached projections if possible, and m…
mhsdesign Nov 24, 2024
6726d73
FEATURE: Setup marks failed projections to be booted again
mhsdesign Nov 24, 2024
f46077e
FEATURE: Introduce `ContentRepositoryMaintainer` and restore cr:proje…
mhsdesign Nov 24, 2024
bace8ff
TASK: Rename `ProjectionStatus` and introduce `ProjectionSubscription…
mhsdesign Nov 24, 2024
8ff0f61
TASK: Introduce `DetachedSubscriptionStatus` as the projection setup …
mhsdesign Nov 24, 2024
9675572
TASK: Inline `pruneAllWorkspacesAndContentStreamsFromEventStream` int…
mhsdesign Nov 24, 2024
655ac3c
TASK: Reimplement 40e8d35e09ee690406c6a9cfc823c775d4ee3b51
mhsdesign Nov 24, 2024
e235e69
TASK: Document new `ContentRepositoryMaintainer`
mhsdesign Nov 24, 2024
611ca37
TASK: Rename `ProjectionSetupStatus` back to `ProjectionStatus`
mhsdesign Nov 25, 2024
0b8a3b5
TASK: Rename `SubscriptionStatuses` to `SubscriptionStatusCollection`
mhsdesign Nov 25, 2024
51f0cf6
TASK: Leave warning hint for why we do a replay
mhsdesign Nov 25, 2024
a2a2411
TASK: Warn in `catchupProjection` if projection is not ready to be ca…
mhsdesign Nov 25, 2024
63d1589
TASK: Document `catchupProjection` correctly
mhsdesign Nov 25, 2024
2297c14
TASK: Reintroduce `ContentRepositoryStatus` object and expose current…
mhsdesign Nov 25, 2024
1220f82
TASK: Swap Projection and Setup in output so that Setup comes first
mhsdesign Nov 25, 2024
800fd53
WIP: Introduce `cr:reactivateSubscription`
mhsdesign Nov 26, 2024
a0c9f90
TASK: Dont crash on status when the event store is not setup
mhsdesign Nov 27, 2024
8c079d9
TASK: Split projection replay into separate SubscriptionCommandContro…
mhsdesign Nov 27, 2024
4c65d81
TASK: Status also shows new subscriptions even if they are not persis…
mhsdesign Nov 27, 2024
8c9c0e8
TASK: Refine todos
mhsdesign Nov 27, 2024
4424483
TASK: Declare SubscriptionEngine and friends as internal
mhsdesign Nov 27, 2024
baa5e4a
TASK: Add error code to `SubscriptionEngineAlreadyProcessingException`
mhsdesign Nov 27, 2024
66e54bc
TASK: Allow cr registry to implement internal subscription store beca…
mhsdesign Nov 27, 2024
51d39e5
TASK: Rename to `SubscriptionReplayProcessor`
mhsdesign Nov 27, 2024
b2c1a29
TASK: Improve legacy projectionReplayCommand stub
mhsdesign Nov 27, 2024
d84c2a4
TASK: Add test for Subscription & Cr Commands (and thus CRMaintainer)
mhsdesign Nov 27, 2024
fd768da
Merge pull request #5378 from mhsdesign/feature/content-repository-ma…
mhsdesign Nov 27, 2024
ac425ff
TASK: Remove `SubscriptionManager` and make subscriptions immutable
mhsdesign Nov 27, 2024
eb0d792
FEATURE: Implement `reactivateSubscription`
mhsdesign Nov 27, 2024
dc5ff10
TASK: Move transactional logic _on_ projection as it does not belong …
mhsdesign Nov 27, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
use Neos\ContentGraph\DoctrineDbalAdapter\Domain\Repository\DimensionSpacePointsRepository;
use Neos\ContentGraph\DoctrineDbalAdapter\Domain\Repository\NodeFactory;
use Neos\ContentGraph\DoctrineDbalAdapter\Domain\Repository\ProjectionContentGraph;
use Neos\ContentRepository\Core\Factory\ProjectionFactoryDependencies;
use Neos\ContentRepository\Core\Factory\SubscriberFactoryDependencies;
use Neos\ContentRepository\Core\Projection\ContentGraph\ContentGraphProjectionFactoryInterface;

/**
Expand All @@ -24,8 +24,7 @@ public function __construct(
}

public function build(
ProjectionFactoryDependencies $projectionFactoryDependencies,
array $options,
SubscriberFactoryDependencies $projectionFactoryDependencies,
): DoctrineDbalContentGraphProjection {
$tableNames = ContentGraphTableNames::create(
$projectionFactoryDependencies->contentRepositoryId
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
use Doctrine\DBAL\Connection;
use Neos\ContentGraph\PostgreSQLAdapter\Domain\Projection\HypergraphProjection;
use Neos\ContentGraph\PostgreSQLAdapter\Domain\Repository\NodeFactory;
use Neos\ContentRepository\Core\Factory\ProjectionFactoryDependencies;
use Neos\ContentRepository\Core\Factory\SubscriberFactoryDependencies;
use Neos\ContentRepository\Core\Projection\ContentGraph\ContentGraphProjectionFactoryInterface;
use Neos\ContentRepository\Core\SharedModel\ContentRepository\ContentRepositoryId;

Expand All @@ -28,8 +28,7 @@ public static function graphProjectionTableNamePrefix(
}

public function build(
ProjectionFactoryDependencies $projectionFactoryDependencies,
array $options,
SubscriberFactoryDependencies $projectionFactoryDependencies,
): HypergraphProjection {
$tableNamePrefix = self::graphProjectionTableNamePrefix(
$projectionFactoryDependencies->contentRepositoryId
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,8 @@
use Neos\ContentRepository\BehavioralTests\ProjectionRaceConditionTester\Dto\TraceEntries;
use Neos\ContentRepository\BehavioralTests\ProjectionRaceConditionTester\Dto\TraceEntryType;
use Neos\ContentRepository\Core\EventStore\EventInterface;
use Neos\ContentRepository\Core\Projection\CatchUpHookInterface;
use Neos\ContentRepository\Core\Projection\CatchUpHook\CatchUpHookInterface;
use Neos\ContentRepository\Core\Subscription\SubscriptionStatus;
use Neos\EventStore\Model\EventEnvelope;
use Neos\Flow\Annotations as Flow;

Expand Down Expand Up @@ -73,7 +74,7 @@
*
* When {@see onBeforeEvent} is called, we know that we are inside applyEvent() in the diagram above,
* thus we know the lock *HAS* been acquired.
* When {@see onBeforeBatchCompleted}is called, we know the lock will be released directly afterwards.
* When {@see onAfterCatchUp}is called, we know the lock will be released directly afterwards.
*
* We track these timings across processes in a single Redis Stream. Because Redis is single-threaded,
* we can be sure that we observe the correct, total order of interleavings *across multiple processes*
Expand Down Expand Up @@ -107,7 +108,7 @@ final class RaceTrackerCatchUpHook implements CatchUpHookInterface
protected $configuration;
private bool $inCriticalSection = false;

public function onBeforeCatchUp(): void
public function onBeforeCatchUp(SubscriptionStatus $subscriptionStatus): void
{
RedisInterleavingLogger::connect($this->configuration['redis']['host'], $this->configuration['redis']['port']);
}
Expand All @@ -126,16 +127,12 @@ public function onAfterEvent(EventInterface $eventInstance, EventEnvelope $event
{
}

public function onBeforeBatchCompleted(): void
public function onAfterCatchUp(): void
{
// we only want to track relevant lock release calls (i.e. if we were in the event processing loop before)
if ($this->inCriticalSection) {
$this->inCriticalSection = false;
RedisInterleavingLogger::trace(TraceEntryType::LockWillBeReleasedIfItWasAcquiredBefore);
}
}

public function onAfterCatchUp(): void
{
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,9 @@

namespace Neos\ContentRepository\BehavioralTests\ProjectionRaceConditionTester;

use Neos\ContentRepository\Core\Projection\CatchUpHookFactoryDependencies;
use Neos\ContentRepository\Core\Projection\CatchUpHookFactoryInterface;
use Neos\ContentRepository\Core\Projection\CatchUpHookInterface;
use Neos\ContentRepository\Core\Projection\CatchUpHook\CatchUpHookFactoryDependencies;
use Neos\ContentRepository\Core\Projection\CatchUpHook\CatchUpHookFactoryInterface;
use Neos\ContentRepository\Core\Projection\CatchUpHook\CatchUpHookInterface;
use Neos\ContentRepository\Core\Projection\ContentGraph\ContentGraphReadModelInterface;

/**
Expand Down
Loading
Loading