feat: Upstream Sync with Content Library Blocks

[kdmccormick]

Description

This is the majority of the backend work needed for the Content Libraries Beta launch in Sumac.

The Python API

The Python API is entirely defined in cms/lib/xblock/upstream_sync.py, and features:

• A new UpstreamSyncMixin which is added to all CMS blocks, including:

  • upstream (e.g., the usage key of the library block)
  • upstream_version (e.g., the last-synced library block version number)
  • upstream_display_name (e.g., the library-defined title of the block)
  • upstream_max_attempts (e.g., the library-defined max-attempts for the block)
  • get_customizable_fields(...), allowing blocks to set up further upstream_* fields.

• The UpstreamLink data object, which holds metadata on a block's relationship with its upstream, with classmethods for fetching the metadata.

• Three public API functions:

  • sync_from_upstream (e.g., update a block from its library block)
  • fetch_customizable_fields (e.g., update the library-defined defaults, without updating visible content).
  • decline_sync (e.g., decline an update from a library block)

The HTTP API

It is defined and documented here: https://github.com/kdmccormick/edx-platform/blob/kdmccormick/upstream-proto/cms/djangoapps/contentstore/rest_api/v2/views/downstreams.py#L2

Temporary UI

This PR includes @bradenmacdonald 's basic, dev-only UI from this PR: kdmccormick#16 . We will improve this UI before the Sumac launch, but having it landed will aid with testing and further development over the next week.

We display an icon when content comes from a content library (see on the left):

Or if the link is broken, it says that:

And when an update is available, it displays a working "Update" button

It does not yet have a modal to allow you to accept/reject/preview the changes.

Known issues

Definitely to be fixed before Sumac:

• Problems pasted from a library to a course are "converted" the advanced editor, even if they've only been edited with the visual editor.
• These UI, its icons, and copytext are temporary
• We sync from draft versions instead of published versions. See: feat: Allow specifying a version when loading a v2 XBlock #35626

Might be fixed before Sumac, definitely before Teak:

• List of customizable fields is super minimal right now (just display_name, max_attempts).
• API docs need to be moved from a docstring to decorators so they show up in the /api-docs/ UI. See Better API docs for /api/contentstore/v2/downstreams #35653

Supporting Information

ADR: https://github.com/kdmccormick/edx-platform/blob/kdmccormick/upstream-proto/docs/decisions/0020-upstream-block.rst

Testing

Basic setup (do this first)

• Create a (v2) content library with a component
• Copy the component from a library to a course unit
• Ensure: the little "Sourced from a library" icon shows up, but "Update Available" does NOT show up.

Edit + Update

• Edit the library component.
• Refresh the course unit page.
  • Ensure: "Update Available" shows up
• Click "Update Available" to sync from the library
  • Ensure: Page reloads, with updated library block. "Update Available" button is gone.

Overrides

• Edit the library component including updates to both:
  • a customizable field, like the title or max_attempts
  • a non-customizable field, like the problem/html data, or the video URL
• Refresh the course unit page
  • Ensure: "Update Available" shows up
• Edit the course-local instance of your library component, including an edit to both a customizable and a non-customizable field (ideally the same ones you edited above)
• Click "Update Available" to sync from the library
  • Ensure: Page reloads, with updated library block. "Update Available" button is gone.
  • Ensure: Your course-local edit to the customizable field remains intact.
  • Ensure: Your course-local edit to the non-customizable is blown away, and instead you see the library-level edit manifest

"Restore to library default" (Sumac version)

In Sumac, we won't have full "Restore a field to its library default" functionality. We'll have a halfway version instead. More details. Here's how to test the halfway version, using display_name (although any customizable field should work) and a linked library/course component pair:

• In your library: Set your component's title to My Upstream Title 1
• In your course: Click "Update Available"
  • Ensure that the course-local title becomes My Upstream Title 1.
• In your course: Set your component's title to My Downstream Title
• In your library: Now set your component's title to My Upstream Title 2
• In your course: Click "Update Available"
  • Ensure that the course-local title remains My Downstream Title.
• In your course: Set your component's title to My Upstream Title 2 (this is our halfway "restore default" mechanism)
• In your library: Now set your component's title to My Upstream Title 3
• In your course: Click "Update Available"
  • Ensure that the course-local title becomes My Upstream Title 3.

Export/Import

• Edit the library component including updates to both:
  • a customizable field, like the title or max_attempts
  • a non-customizable field, like the problem/html data, or the video URL
• Refresh the course unit page
  • Ensure: "Update Available" shows up
• Export your course to OLX
• Import the OLX back into the same course run, and navigate to your library-linked component.
  • Ensure: "Update Available" is shown, and clicking it works as expected.
• Now import the OLX into a different course run on the same instance, and navigate to your library-linked component in that course run.
  • Ensure: "Update Available" is shown, and clicking it works as expected.
• Now import the OLX into a different instance, and navigate to your library-linked component.
  • Ensure: The content says it is has a broken library link.

Merge Deadline

ASAP

[kdmccormick]

@bradenmacdonald So I've got this draft version of upstream sync working, implemented with XBlock fields and handlers. Using the XBlock fields and handlers for this was nice because it gave me persistence, URL routing, and OLX serialization for free.

But now I'm wondering if this is bad approach because we'll need to support Units and Subsections in libraries next, and we've agreed that those structural blocks should stop being XBlocks eventually.

The alternative I see is to:

• use Django models to store upstream links, upstream defaults, and upstream overrides,
• define REST/Python APIs for managing them, and
• handle OLX serialization as special case

Thoughts?

[kdmccormick]

use Django models to store upstream links, upstream defaults, and upstream overrides,

Okay, I started prototyping this, and it's easier said than done. Particularly, if you imagine a model like this, there's a question of whether we FK to entities or entityverisons:

class UpstreamDownstreamLink(models.Model):
    """
    Link a piece of content (the downstream) to a source for updates (the upstream).
    """
    # Pretty straightforward -- FK to a particular upstream version, and then update the
    # version whenever sync happens.
    upstream = models.ForeignKey(PublishableEntityVersion)

    # Which one of these is right?? Neither?

    # This one would apply to all versions of the downstream component, so it would play weird
    # draft/publish, and would fail if we ever added an "undo" button to studio.
    downstream = models.OneToOneField(PublishableEntity)

    # This seems more correct, but would require updating every time a new version of the downstream
    # component is created. Perhaps this would work fine as a cache, but it seems wrong as the authoritative
    # data source.
    downstream = models.OneToOneField(PublishableEntityVersion)

It seems to me, then, that we do want to persist upstream metadata using XBlocks fields, so that version management works as we expect. Still, I'm open to feedback on whether we should use XBlock handlers vs. build a separate REST API for getting/settings links and syncing from upstream.

[open-craft-grove]

Sandbox deployment failed 💥
Please check the settings and requirements.
Retry deployment by pushing a new commit or updating the requirements/settings in the pull request's description.
📜 Failure Logs
ℹ️ Grove Config, Tutor Config, Tutor Requirements

[bradenmacdonald]

@kdmccormick I think it's fine to implement this with fields and handlers for now; as you said, it gives you a lot for free. It even gives us most of the Unit and Subsection functionality for free, I think. Yes, we want to implement those differently in the future but we aren't going to be able to make that change anytime soon, and I think we can worry about how library sync works in that case after we have the new design for how that stuff works. Right now it's mostly an unknown.

e.g. I have an idea for an "outline" object that encodes the course hierarchy and library content and much more, and if we implemented it that way, we'd have to re-implement even your django model based prototype. Or previously I had proposed this compositor architecture which is different yet again. So I think it's not worth planning for that too much until we've got an ADR and buy-in about what the future direction is for the non-XBlock way to represent hierarchy.

[open-craft-grove]

Sandbox deployment successful 🚀
🎓 LMS
📝 Studio
ℹ️ Grove Config, Tutor Config, Tutor Requirements

[open-craft-grove]

Sandbox deployment successful 🚀
🎓 LMS
📝 Studio
ℹ️ Grove Config, Tutor Config, Tutor Requirements

[open-craft-grove]

Sandbox deployment successful 🚀
🎓 LMS
📝 Studio
ℹ️ Grove Config, Tutor Config, Tutor Requirements

[open-craft-grove]

Sandbox deployment successful 🚀
🎓 LMS
📝 Studio
ℹ️ Grove Config, Tutor Config, Tutor Requirements

[kdmccormick]

@ormsbee Ready for another pass

[ormsbee]

Just a minor question about the API endpoint in the docstring (whether it's intended to be /api/contentstore/v2/...). Please consider the remaining logging request as optional.

[bradenmacdonald]

At some point, we'd like to be able to give an upstream ID and get the list of all its downstreams. I'm assuming we'll need some kind of new data model to track that before we're able to implement something like that in the future, right?

[kdmccormick]

Yep. Indeed, we'll need something like a DownstreamUpstreamLink table, which the upstream_sync API writes to and keeps updated.

[bradenmacdonald]

Awesome work, @kdmccormick and very nice clean code 💯

I didn't test import-export but I tested various other things and it was working as advertised:

[bradenmacdonald]

(Not asking for changes now.) Personally, I've really come to appreciate a clear separation between 403 and 404 errors for things like this. Makes debugging, messaging to users, and supporting users much easier. "Leaking" what IDs exist or do not exist in a course is not really a big security concern, and would only apply to users who somehow have permission to read some parts of the course but not other parts - a tiny subset of user requests.

[bradenmacdonald]

They do now :) (mostly)

[kdmccormick]

Nice! One of us can do a follow-up right after this merges, then.

[bradenmacdonald]

It could be nicer in the future to move UpstreamSyncMixin to its own file? It seems fairly separable from everything else in this file.

[kdmccormick]

Yeah, I'm also thinking that might be necessary if we're to make these fields available in the LMS (for analytics) without exposing the whole upstream_sync API to LMS.

I'll leave it for now, but I will likely do this refactoring when it becomes necessary.

[edx-pipeline-bot]

2U Release Notice: This PR has been deployed to the edX staging environment in preparation for a release to production.

[edx-pipeline-bot]

2U Release Notice: This PR has been deployed to the edX production environment.

[edx-pipeline-bot]

2U Release Notice: This PR has been deployed to the edX production environment.

[edx-pipeline-bot]

2U Release Notice: This PR has been deployed to the edX staging environment in preparation for a release to production.

[edx-pipeline-bot]

2U Release Notice: This PR has been deployed to the edX production environment.

[edx-pipeline-bot]

2U Release Notice: This PR has been deployed to the edX production environment.