Juggler Refactor - Epic

[kjdelisle]

Overview

This epic is meant to encapsulate all of the tasks required to update the loopback-datasource-juggler to v4.0 including

1. Monorepo: [EPIC] Monorepo for [email protected] and connectors #890

   • connectors + juggler + dependencies like loopback-filters
   • a different repo than loopback-next to keep lerna bootstrap fast enough
   • preserve original repos - we will keep LB-3.x codebase there

2. Migrate juggler to typescript: [Juggler] Migrate the Juggler to TypeScript #891

3. Migrate individual connectors to typescript too: [Juggler] Migrate Connectors to TypeScript #892

4. Drop callback APIs, use Promises only: [Juggler] Drop callbacks, use Promises only #896

5. Spike: Remove data-access APIs we don't want to support anymore, both from juggler
   and connectors, e.g. updateOrCreate, findOrCreate, etc. [Juggler] Spike: simplify DataAccessObject API  #897

6. Spike: what to do with EventEmitters (Observables?) [Juggler] Spike: EventEmitters in the new world of async/await #898

7. Semver-major release of everything (alpha pre-release or preferably a 0.1.0 release if we change names from loopback-* to @loopback/*)

[bajtos]

I was thinking about this Epic recently, and I would like to propose a different approach. My expectation is that the approach will allow us to work in smaller increments and keep frequent releases. This way we can keep shipping useful improvements to our users. Even if a time comes that this refactor has to be put on hold because of other priorities, then no unfinished work will be wasted.

1. Enhance juggler (DAO, KVAO) to support promise-based connectors functions. I.e. allow the connectors to implement methods like create as async functions. This should be fully backwards-compatible.
2. Rework base connectors in loopback-connector to use ES6 classes and implement DAO/KVAO methods as async functions returning promise (let's call this style ES2017). If there are any other callback-based functions in loopback-connector, then rework them to async functions too. As part of this change, we need to update all tests from callbacks to async/await style. This is a breaking change.
3. Once we have a new version of loopback-connector, upgrade few connectors to leverage this new version. We can start e.g. with mongodb and mysql. Rewrite all tests to async/await style too. This is a breaking change, because ES2017-based connectors are not compatible with juggler v3.
4. With this change in place, we can start looking into ways how to make CrudRepositoryImpl a working repository that can be used instead of juggler-based DefaultCrudRepository. The idea is to call connector methods directly from CrudRepositoryImpl, bypassing any juggler layers.
5. Ideally, I'd like CrudRepositoryImpl to be a drop-in replacement for DefaultCrudRepository, at least for typical use cases. To verify this aspect, and to document any differences, we should have a shared test suite that will be executed against both repository implementations and using different connectors. I have already started to work on such suite as part of "Inclusion of related models", see Create a shared Repository test suite and run it against different connectors #3079.

Missing piece:

• How and when to update the built-in memory connector from current ES5 style to ES2017. If we make the changes in loopback-datasource-juggler, would it require a semver-major release?
• Migrate connectors to monorepo? I think it make more sense to keep connectors in loopback-next because we want to test them together with @loopback/repository changes. If we are going to migrate them, them maybe we should do that as part of steps 2 & 3 (upgrade to ES2017)?

Next steps:

• Upgrade all other connectors to ES2017 style.
• Migrate connectors to TypeScript
• Find a new home for the memory connector - extract it into a standalone npm package.

Eventually, when all above is done:

• Switch our code (examples, tests, etc.) to use CrudRepositoryImpl instead of DefaultCrudRepository, update all documentation.
• Soft-deprecate DefaultCrudRepository (only in docs/api-docs).
• Move juggler to Active LTS mode.

@raymondfeng @strongloop/loopback-maintainers @strongloop/loopback-next Thoughts?

[stale]

This issue has been marked stale because it has not seen activity within six months. If you believe this to be in error, please contact one of the code owners, listed in the CODEOWNERS file at the top-level of this repository. This issue will be closed within 30 days of being stale.

[hacksparrow]

From my work on #3456, I noted that the mechanism for creating the model object that's returned from the DAO methods are different for most methods. This was quite unexpected. Implementing new features or bug fixes that center around the model object will require changes for each DAO method.

Ideally, the model object should be generated by a single pipeline, which would be invoked by all the DAO methods that deal with data retrieval.

Take a look at the differences in how the model data is created in the callback function of these DAO methods:

create: https://github.com/strongloop/loopback-datasource-juggler/blob/261dd1c865e8b197882a161b909b90b0e6c3c2a3/lib/dao.js#L383
save: https://github.com/strongloop/loopback-datasource-juggler/blob/261dd1c865e8b197882a161b909b90b0e6c3c2a3/lib/dao.js#L2137
replaceById: https://github.com/strongloop/loopback-datasource-juggler/blob/261dd1c865e8b197882a161b909b90b0e6c3c2a3/lib/dao.js#L2700
find: https://github.com/strongloop/loopback-datasource-juggler/blob/261dd1c865e8b197882a161b909b90b0e6c3c2a3/lib/dao.js#L1622

[stale]

This issue has been marked stale because it has not seen activity within six months. If you believe this to be in error, please contact one of the code owners, listed in the CODEOWNERS file at the top-level of this repository. This issue will be closed within 30 days of being stale.

[stale]

This issue has been closed due to continued inactivity. Thank you for your understanding. If you believe this to be in error, please contact one of the code owners, listed in the CODEOWNERS file at the top-level of this repository.