Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FEATURE: Subscription Engine #5321

Draft
wants to merge 113 commits into
base: 9.0
Choose a base branch
from

Commits on Oct 24, 2024

  1. WIP

    Related: #4746
    bwaidelich committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    2172f09 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    13b1073 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    d75f174 View commit details
    Browse the repository at this point in the history

Commits on Oct 26, 2024

  1. Merge branch '9.0' into feature/4746-rework-catchup-mechanism-3

    # Conflicts:
    #	Neos.ContentRepository.Core/Classes/ContentRepository.php
    bwaidelich committed Oct 26, 2024
    Configuration menu
    Copy the full SHA
    937f84d View commit details
    Browse the repository at this point in the history

Commits on Oct 29, 2024

  1. Configuration menu
    Copy the full SHA
    03fb7ca View commit details
    Browse the repository at this point in the history

Commits on Nov 3, 2024

  1. Configuration menu
    Copy the full SHA
    dcb4be2 View commit details
    Browse the repository at this point in the history

Commits on Nov 5, 2024

  1. Configuration menu
    Copy the full SHA
    fafe080 View commit details
    Browse the repository at this point in the history
  2. WIP

    bwaidelich committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    29ba908 View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2024

  1. Tweak type comments

    bwaidelich committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    207179b View commit details
    Browse the repository at this point in the history

Commits on Nov 13, 2024

  1. Wiring...

    bwaidelich committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    a127706 View commit details
    Browse the repository at this point in the history
  2. Merge branch '9.0' into feature/4746-rework-catchup-mechanism-3

    # Conflicts:
    #	Neos.ContentRepository.Core/Classes/ContentRepository.php
    #	Neos.ContentRepository.Core/Classes/Factory/ContentRepositoryFactory.php
    #	Neos.ContentRepositoryRegistry/Classes/ContentRepositoryRegistry.php
    #	Neos.ContentRepositoryRegistry/Classes/Service/ProjectionService.php
    #	Neos.ContentRepositoryRegistry/Classes/Service/ProjectionServiceFactory.php
    #	phpstan-baseline.neon
    bwaidelich committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    765b84f View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2024

  1. Configuration menu
    Copy the full SHA
    5b035c1 View commit details
    Browse the repository at this point in the history
  2. Fix CatchUpHooks

    bwaidelich committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    aca0f9b View commit details
    Browse the repository at this point in the history
  3. Replace ProjectionService

    bwaidelich committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    7fea53e View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    ec08d64 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    9fef068 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    46ad9d1 View commit details
    Browse the repository at this point in the history

Commits on Nov 17, 2024

  1. Merge branch '9.0' into feature/4746-rework-catchup-mechanism-3

    # Conflicts:
    #	Neos.ContentRepository.Core/Classes/ContentRepository.php
    #	Neos.ContentRepository.Core/Classes/Factory/ContentRepositoryFactory.php
    #	Neos.ContentRepositoryRegistry/Classes/ContentRepositoryRegistry.php
    bwaidelich committed Nov 17, 2024
    Configuration menu
    Copy the full SHA
    5ea8fae View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    88cc600 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    e3f85ac View commit details
    Browse the repository at this point in the history

Commits on Nov 18, 2024

  1. Allow to reset subscriptions

    and implement progress callback (for progress bars)
    bwaidelich committed Nov 18, 2024
    Configuration menu
    Copy the full SHA
    c43a2ea View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    40e8d35 View commit details
    Browse the repository at this point in the history
  3. Fix test_parallel cr settings

    type annotation tweaks
    bwaidelich committed Nov 18, 2024
    Configuration menu
    Copy the full SHA
    3744fd5 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    c0fbfe1 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    f5ff7d6 View commit details
    Browse the repository at this point in the history
  6. Fix behat tests ?

    ...probably not
    bwaidelich committed Nov 18, 2024
    Configuration menu
    Copy the full SHA
    f8a8b5b View commit details
    Browse the repository at this point in the history
  7. Disable EventExportProcessor test

    to see wether other tests succeed
    bwaidelich committed Nov 18, 2024
    Configuration menu
    Copy the full SHA
    1562435 View commit details
    Browse the repository at this point in the history
  8. Revert "Disable EventExportProcessor test"

    This reverts commit 1562435.
    bwaidelich committed Nov 18, 2024
    Configuration menu
    Copy the full SHA
    2902dc2 View commit details
    Browse the repository at this point in the history

Commits on Nov 20, 2024

  1. TASK: Fix phpstan

    mhsdesign committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    0fe05b5 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    6e5b565 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    8967ead View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    2b5d338 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    2d3a136 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    46dc510 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    0680533 View commit details
    Browse the repository at this point in the history
  8. TASK: Test that locking and concurrent writing works under heavy load

    Attempting to write to the content repository from two threads previously would cause likely an exception in Neos 9.0:
    
    > Failed to acquire checkpoint lock for subscriber "Neos\ContentGraph\DoctrineDbalAdapter\DoctrineDbalContentGraphProjection" because it is acquired already
    
    Now `forUpdate` is used in `DoctrineSubscriptionStore::findByCriteria` which solves this.
    
    and if commented out throws as expected:
    
    > An exception occurred while executing a query: SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction.
    mhsdesign committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    de92d79 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    f58cefa View commit details
    Browse the repository at this point in the history
  10. TASK: Fix of-by-one error in catchup

    Without advancing via `->next()` we will attempt to handle one event twice, and then run into the log: Which makes sense as the subscriber _was_ on that sequence number before and we are interested in the next events.
    
    > subscription "contentGraph" is farther than the current position (5053 >= 5053)
    
    Also simplifies `lowestPosition`
    mhsdesign committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    e15ae79 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    c7df820 View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    0bd129a View commit details
    Browse the repository at this point in the history
  13. TASK: Remove generic EventHandlerInterface for now, everything is a…

    … projection (can be later reintroduced)
    mhsdesign committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    1dfaa2c View commit details
    Browse the repository at this point in the history
  14. TASK: Remove other generic subscription concepts not required for pro…

    …jections
    
    - retry strategy
    - runmode
    - pausing, finishing
    - InMemorySubscriptionStore
    mhsdesign committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    441b035 View commit details
    Browse the repository at this point in the history

Commits on Nov 21, 2024

  1. TASK: Remove subscription groups and filtering except for status and ids

    Followup for remove other generic subscription concepts not required for projections
    mhsdesign committed Nov 21, 2024
    Configuration menu
    Copy the full SHA
    9c6ba75 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    0ac8751 View commit details
    Browse the repository at this point in the history
  3. TASK: Remove removal of subscriptions

    Detached subscriptions cannot be removed for now
    mhsdesign committed Nov 21, 2024
    Configuration menu
    Copy the full SHA
    9be3308 View commit details
    Browse the repository at this point in the history
  4. TASK: Remove sqlite support for DoctrineSubscriptionStore as db loc…

    …king is untested there
    mhsdesign committed Nov 21, 2024
    Configuration menu
    Copy the full SHA
    53790b9 View commit details
    Browse the repository at this point in the history
  5. TASK: Inline ProjectionEventHandler to Subscriber and make it a `…

    …SubscriberForProjection`
    
    Simplification to make the projection use case first level
    See #5375 (comment)
    mhsdesign committed Nov 21, 2024
    Configuration menu
    Copy the full SHA
    481f173 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    b9569c5 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    427829e View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    715ec2e View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    512e3c4 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    408ceb2 View commit details
    Browse the repository at this point in the history

Commits on Nov 22, 2024

  1. TASK: Subscription engine test filteringReset

    and extract commitExampleContentStreamEvent
    mhsdesign committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    b71165c View commit details
    Browse the repository at this point in the history
  2. TASK: introduce DebugEventProjection for testing to assert each eve…

    …nt is only handled once and further introspection
    mhsdesign committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    c8f8c6a View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    91046e6 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    f624ac6 View commit details
    Browse the repository at this point in the history

Commits on Nov 23, 2024

  1. TASK: Introduce test to assert that projection keeps events previousl…

    …y applied in the same batch that did not cause errors
    mhsdesign committed Nov 23, 2024
    Configuration menu
    Copy the full SHA
    0197017 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    3b6ca26 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    794be11 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    752d434 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    c4da7fe View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    c8cb0a2 View commit details
    Browse the repository at this point in the history
  7. TASK: Adjust subscription test exceptions to do no retry

    No retry is simpler at first and its unlikely that a projection will fix itself.
    mhsdesign committed Nov 23, 2024
    Configuration menu
    Copy the full SHA
    dfe5eb3 View commit details
    Browse the repository at this point in the history
  8. TASK: Remove retry_attempt from subscriptions

    #5321 (comment)
    
    > Anyways, I think with the removed retry strategy we should just get rid of the automatic retry altogether right now.
      It's quite unlikely that a retry suddenly works without other changes. So I'd be fully OK if it was only possible to manually retry failed subscriptions for 9.0
    mhsdesign committed Nov 23, 2024
    Configuration menu
    Copy the full SHA
    a32bfb2 View commit details
    Browse the repository at this point in the history
  9. TASK: Rename factory back to $additionalSubscriberFactories

    > there's a mismatch between variable name "additionalProjectionsFactory" and type "ContentRepositorySubsciberFactory"
    > the distinction between projection and subscription makes sense even if the only supported type of subscription target projections (for now)
    
    #5375 (review)
    mhsdesign committed Nov 23, 2024
    Configuration menu
    Copy the full SHA
    836c347 View commit details
    Browse the repository at this point in the history
  10. BUGFIX: discoverDetachedSubscriptions did not persist changes

    which lead to it being invoked on aktive
    
    >  Subscriber "Vendor.Package:FakeProjection" could not invoke onBeforeCatchUp: Subscriber with the subscription id "Vendor.Package:FakeProjection" not found.
    mhsdesign committed Nov 23, 2024
    Configuration menu
    Copy the full SHA
    830b5cc View commit details
    Browse the repository at this point in the history
  11. Merge pull request #5375 from mhsdesign/task/radical-cleanup-for-subs…

    …cription-pr
    
    TASK: Radical cleanup for subscription pr to simplify and get into 9.0
    mhsdesign authored Nov 23, 2024
    Configuration menu
    Copy the full SHA
    3d43183 View commit details
    Browse the repository at this point in the history
  12. TASK: Fix phpstan

    mhsdesign committed Nov 23, 2024
    Configuration menu
    Copy the full SHA
    6f43825 View commit details
    Browse the repository at this point in the history
  13. BUGFIX: Reintroduce catchup hooks for all projections

    Also allows the `Vendor.Package:FakeCatchupHook` to be picked up for testing
    mhsdesign committed Nov 23, 2024
    Configuration menu
    Copy the full SHA
    60d4f8c View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    5ef4ab1 View commit details
    Browse the repository at this point in the history
  15. Configuration menu
    Copy the full SHA
    c7cb75b View commit details
    Browse the repository at this point in the history
  16. TASK: Ensure that the content graph projection is not part of the gen…

    …eric `ProjectionStates`
    mhsdesign committed Nov 23, 2024
    Configuration menu
    Copy the full SHA
    a80639b View commit details
    Browse the repository at this point in the history
  17. Configuration menu
    Copy the full SHA
    e7acfaa View commit details
    Browse the repository at this point in the history
  18. TASK: Assertions that setup and boot do not retry failed projections

    Readded assertions as retry was removed for now ... and should probably NOT do anything if setup is called!
    mhsdesign committed Nov 23, 2024
    Configuration menu
    Copy the full SHA
    fdeec75 View commit details
    Browse the repository at this point in the history
  19. BUGFIX: SubscriptionDetachedStatusTest::projectionIsDetachedIfConfigu…

    …rationIsRemoved
    
    > InvalidArgumentException: Subscriber with the subscription id "Vendor.Package:FakeProjection" not found.
    
    The subscriber is detached, so the state is not calculate-able at this point!
    mhsdesign committed Nov 23, 2024
    Configuration menu
    Copy the full SHA
    31913f5 View commit details
    Browse the repository at this point in the history
  20. TASK: Throw CatchUpFailed exception in case onBeforeCatchUp or onAf…

    …terCatchUp failes
    
    as we consider it a critical developer error
    
    For `onBeforeCatchUp` we could probably wrap a savepoint and roll it back and skip also the projection, but errors in `onAfterCatchUp` would then analog also need to rollback only the one projection where it was registered. This is not possible and too complex.
    
    see \Neos\ContentRepository\BehavioralTests\Tests\Functional\Subscription\CatchUpHookErrorTest::error_onBeforeCatchUp_abortsCatchup
    mhsdesign committed Nov 23, 2024
    Configuration menu
    Copy the full SHA
    c5ea757 View commit details
    Browse the repository at this point in the history
  21. TASK: Use save points to rollback projections during transaction on f…

    …ailure
    
    https://neos-project.slack.com/archives/C04PYL8H3/p1732318989845619
    
    We dont want to rollback the main transaction, as other projections still need to be processed, the previously working events need to be applied, and we want to set the ERROR state of the projection
    mhsdesign committed Nov 23, 2024
    Configuration menu
    Copy the full SHA
    794eaf2 View commit details
    Browse the repository at this point in the history
  22. Configuration menu
    Copy the full SHA
    7852f61 View commit details
    Browse the repository at this point in the history
  23. TASK: Move back to subscriptionStatuses test. We need to make sure …

    …the tables are dropped
    mhsdesign committed Nov 23, 2024
    Configuration menu
    Copy the full SHA
    d582766 View commit details
    Browse the repository at this point in the history
  24. TASK: Inline discoverDetachedSubscriptions

    To reduce additional sql query and lock, and do it in the main transaction
    mhsdesign committed Nov 23, 2024
    Configuration menu
    Copy the full SHA
    5dfa592 View commit details
    Browse the repository at this point in the history
  25. TASK: Do not discover new subscriptions during catchup

    We do not expect any changes during runtime. Setup and status should handle this case.
    mhsdesign committed Nov 23, 2024
    Configuration menu
    Copy the full SHA
    d5715c7 View commit details
    Browse the repository at this point in the history

Commits on Nov 24, 2024

  1. TASK: Introduce further tests to assert behaviour for catchup and setup

    and failing test for setupIsInvokedForPreviouslyActiveSubscribers
    mhsdesign committed Nov 24, 2024
    Configuration menu
    Copy the full SHA
    37a4e47 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    a8f246b View commit details
    Browse the repository at this point in the history
  3. BUGFIX: Setup should reattach detached projections if possible, and m…

    …ark detached ones as detached.
    mhsdesign committed Nov 24, 2024
    Configuration menu
    Copy the full SHA
    73e1097 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    6726d73 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    f46077e View commit details
    Browse the repository at this point in the history
  6. TASK: Rename ProjectionStatus and introduce `ProjectionSubscription…

    …Status`
    
    Like the `ProjectionSubscription` projections subscribers for projections will have an explicit state: `ProjectionSubscriptionStatus`. If its extended to allow other subscribers another status type should be introduced.
    
    replayRequired todo remove we cannot figurea that out in the status after all!
    mhsdesign committed Nov 24, 2024
    Configuration menu
    Copy the full SHA
    bace8ff View commit details
    Browse the repository at this point in the history
  7. TASK: Introduce DetachedSubscriptionStatus as the projection setup …

    …status cannot be calculated
    
    ... and when extending the system to support multiple subscribers, we cannot know their original classification but have to use a special empty placeholder like: `DetachedSubscriptionStatus`
    
    This also makes the `$projectionStatus === null` detached case more explicit when using status.
    mhsdesign committed Nov 24, 2024
    Configuration menu
    Copy the full SHA
    8ff0f61 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    9675572 View commit details
    Browse the repository at this point in the history
  9. TASK: Reimplement 40e8d35

    Under consideration of the new `ProjectionSubscriptionStatus`
    mhsdesign committed Nov 24, 2024
    Configuration menu
    Copy the full SHA
    655ac3c View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    e235e69 View commit details
    Browse the repository at this point in the history

Commits on Nov 25, 2024

  1. Configuration menu
    Copy the full SHA
    611ca37 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    0b8a3b5 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    51f0cf6 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    a2a2411 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    63d1589 View commit details
    Browse the repository at this point in the history
  6. TASK: Reintroduce ContentRepositoryStatus object and expose current…

    … event store position for debugging and status information
    mhsdesign committed Nov 25, 2024
    Configuration menu
    Copy the full SHA
    2297c14 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    1220f82 View commit details
    Browse the repository at this point in the history

Commits on Nov 26, 2024

  1. WIP: Introduce cr:reactivateSubscription

    in hindsight that well revert that setup updates the error state or detached state
    mhsdesign committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    800fd53 View commit details
    Browse the repository at this point in the history

Commits on Nov 27, 2024

  1. Configuration menu
    Copy the full SHA
    a0c9f90 View commit details
    Browse the repository at this point in the history
  2. TASK: Split projection replay into separate SubscriptionCommandContro…

    …ller
    
    this allows us to keep the namings short and precise instead of introducing `cr:subscriptionreplayall`
    
    see also: https://neos-project.slack.com/archives/C04PYL8H3/p1732629147379509
    mhsdesign committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    8c079d9 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    4c65d81 View commit details
    Browse the repository at this point in the history
  4. TASK: Refine todos

    skipBooting was removed via 0ac8751
    mhsdesign committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    8c9c0e8 View commit details
    Browse the repository at this point in the history
  5. TASK: Declare SubscriptionEngine and friends as internal

    only things in `SubscriptionStatusCollection` are API
    mhsdesign committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    4424483 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    baa5e4a View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    66e54bc View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    51d39e5 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    b2c1a29 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    d84c2a4 View commit details
    Browse the repository at this point in the history
  11. Merge pull request #5378 from mhsdesign/feature/content-repository-ma…

    …intainer
    
    FEATURE: Introduce `ContentRepositoryMaintainer` and restore projection cli commands
    mhsdesign authored Nov 27, 2024
    Configuration menu
    Copy the full SHA
    fd768da View commit details
    Browse the repository at this point in the history
  12. TASK: Remove SubscriptionManager and make subscriptions immutable

    This removes any magic from the code flow and makes transactions, locking, and update more explicit and easier to follow.
    mhsdesign committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    ac425ff View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    eb0d792 View commit details
    Browse the repository at this point in the history
  14. TASK: Move transactional logic _on_ projection as it does not belong …

    …to the subscription store
    
    ... which technically only coincidentally uses the same connection and dbal instance
    
    see #5321 (comment)
    mhsdesign committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    dc5ff10 View commit details
    Browse the repository at this point in the history