[DEPR]: Replace cs_comments_service

[regisb]

Proposal Date

2024-04-16

Target Ticket Acceptance Date

2024-10-16

Earliest Open edX Named Release Without This Functionality

Sumac - 2024-10

Rationale

The Open edX forum is an essential feature of the platform. In technical terms, it is a web application that is developed in the cs_comments_service repository. This application is implemented in the Ruby programming language. Only a few Open edX contributors have expertise in Ruby: most developers are skilled in Python and NodeJS. This makes it extremely difficult to implement new features and extend the forum.

Moreover, the cs_comments_service application currently stores data in MongoDB, which is neither a consistent or a relational data storage solution -- as opposed to MySQL. This results both in operational and architecture complexity, as some data needs to be synced up between the two data stores (notably for user data).

Removal

The cs_comments_service repository will be removed. Note howerver that the tutor-forum plugin will be maintained, though it will be considerably simplified.

Replacement

The Ruby application will be replaced by a Django app, currently being developed in the following repository: https://github.com/edly-io/forum/ (soon to be migrated to the openedx organization, as per this public-engineering issue).

Deprecation

The cs_comments_service repository will include a deprecation notice in its README file. Migration will be transparent for most users running Tutor, as the tutor-forum plugin will smoothly transition from running the former ruby app to installing the new forum application in the edx-platform runtime.

Migration

Strategy

Let's call the legacy cs_comments_service "forum v1" and the new forum application "forum v2".

In forum v1, edx-platform communicates with the Ruby application via an HTTP rest API. All calls are made from the backend, Client-side API calls are made to edx-platform, which then forwards parts of the requests to forum v1.

In forum v2, these API calls take the form of a native Python API. This is made possible by the fact that the forum v2 application is installed in the same runtime as edx-platform.

This means that the diagram above will remain the same in the transition from v1 to v2. The only difference is that in step 3, communication will change from an HTTP API call to a native Python function call.

For storage, forum v2 will implement two different backends: one for MySQL (in the form of Django models) and one for MongoDB. The MongoDB backend is only preserved to ease the transition from v1 to v2; it is expected that it will be eventually removed.

Opt-out

Forum v2 will make use of two CourseWaffleFlag (source code). For every course, platform administrators will have the choice to:

1. Send HTTP requests to the forum v1 app or native API calls to the forum v2 app.
2. If calling the forum v2 app, use either the legacy MongoDB backend or the new MySQL backend.
3. Run a dedicated migration command that will copy the course data from MongoDB to the new MySQL models.

By default, users running Tutor (and the tutor-forum plugin) will automatically transition from v1 to v2 and the MySQL models. In that sense, the transition is opt-out rather than opt-in.

Proposed timeline

• July - October 23rd 2024: implementation of forum v2
• October 23rd 2024: Forum v1 marked for deprecation: no new change accepted in the cs_comments_service repository.
• October 23rd - December 9th 2024: Sumac testing, fixing issues in forum v2.
• December 9th 2024: Sumac release
  • By default, Tutor will run forum v2. The storage backend will depend on whether or not it's a new tutor installation
    • Existing platforms will keep using MongoDB: instructions to migrate will be printed on tutor ... launch and be available in the tutor-forum README.
    • New platforms will default to the MySQL backend.
  • It will be possible to override this default behavior on a course-per-course basis.
• December 2024 - April 2025: all platforms are expected to migrate from forum v1 to forum v2, with the MySQL storage backend.
• June 2025: Teak release:
  • the forum v1 integration is removed from edx-platform
  • the mongodb backend is removed from forum v2

Additional Info

• This project was first announced in April 2024 here: https://discuss.openedx.org/t/axim-funded-contribution-proposal-forum-rewrite-from-ruby-mongodb-to-python-mysql/12788
• DEPR Announcement: https://discuss.openedx.org/t/deprecation-removal-forum-cs-comments-service-437/14056
• Then the technical implementation was discussed here: https://docs.google.com/document/d/1oa-L-ETJkXIPEOxLLOlcENE49o3kSh_HNVp-5gouwAc/edit
• Implementation of the forum v2 app started in July 2024 and project was managed publicly here: https://github.com/orgs/edly-io/projects/1/views/1
• DEPR announcement: https://discuss.openedx.org/t/deprecation-removal-forum-cs-comments-service-437/14056
• For reference, here are the guidelines for deprecation announcements: https://open-edx-proposals.readthedocs.io/en/latest/processes/oep-0021-proc-deprecation.html

Task List

• Migrate the edly-io/forum repository to the openedx organisation: [GH Request] Move edly-io/forum repository to the openedx organization axim-engineering#1272
• Merge the following pull request in edx-platform: [WIP] feat: call python methods from forum v2 edx-platform#35490
• Update the tutor-forum plugin to replace forum v1 by forum v2
• Document the opt-out strategy in forum v2
• Testing:
  • Forum v2 and MySQL storage: tested during the Sumac testing period
  • Migration from MongoDB to MySQL: coordinate with 2U to test the correctness of the migration scripts.

[kdmccormick]

It sounds like the "Earliest Open edX Named Release Without This Functionality" will actually be Teak, not Sumac?

[regisb]

@kdmccormick Yes, I wasn't sure to indicate Sumac or Teak in this field. We are proposing that cs_comments_service be removed in Teak but, technically speaking, Sumac will be the first release to ship with the replacement of cs_comments_service by default. In the end I decided to announce the change for Sumac to avoid taking people by surprise.

[ormsbee]

@hurtstotouchfire: Do you folks have time to give some feedback on this proposal?

Thank you.

[regisb]

I'll recap here the decisions that were reached in the BTR working group meeting that happened yesterday:

• Forum v2 support will be merged in edx-platform in Sumac.
• Platforms not running Tutor will keep running forum v1 by default.
• Platform running Tutor will run forum v2, but with different data backends:
  • Existing platforms will keep using MongoDB: instructions to migrate will be printed on tutor ... launch and be available in the tutor-forum README.
  • New platforms will default to the MySQL backend.
• Platforms running Tutor and which need to stick to forum v1 will have to create their own Tutor plugin, likely based on top of this version.
• Legacy forum v1-related code will be removed from edx-platform before Teak.
• Legacy MongoDB related code will be removed from forum v2 before Teak, with the exception of the migration script, which will remain available in Teak.

[feanil]

I've updated the timeline in the description to match this recap.

[jristau1984]

Hello! Can a separate DEPR ticket be created for the MongoDB portion of what is listed in the description here? It feels like something independent (at least from a timeline perspective) of cs_comments_service. Thanks!

[kbuchanan-2u]

Hi guys, it does not look like Forums V2 is actually merged and working for Sumac at this point. Can we update the DEPR timelines to reflect this?

[regisb]

@jristau1984 You mean to say that the migration of data from mongodb to mysql should be migrated to a separate DEPR ticket? What for? Would you like to have a different timeline?

@kbuchanan-2u Forum v2 was merged and is working in Sumac. It was rollbacked by 2U in master, but the release branch is doing just fine.

[ayub02]

@regisb Would you be able to share the test results for this work? From product perspective, I'm curious to know which forum features and workflows the team has considered.

[regisb]

I assume you mean Q&A testing, right @ayub02? (Forum v2 has extensive unit and integration testing, but I suppose you are not referring to those)

The following scenarios were tested:

• Integration of edx-platform with forum v2:
  • With mysql backend
  • With mongodb backend
  • With both mongodb+mysql backends for different courses
  • With Elasticsearch as a search backend
  • With Meilisearch as a search backend
• Compatibility of forum v1 in edx-platform:
  • By using forum v1 exclusively
  • By using forum v2 exclusively
  • By using forum v1 and v2 for different courses
• Data migration from mongodb to mysql: making sure that data is preserved and compatible with forum v2.

Those scenarios were tested in the context of a Tutor deployment. As a consequence, some differences with the 2U architecture were not detected, including:

• The presence of incompatible mongodb indices
• Different MongoDB hostnames

Our team was composed exclusively of developers and did not include Q&A members, so these results were not properly collected in a testing sheet. Still, I can guarantee that these tests were conducted.

Do you have such a test sheet that you could share with us? It would be quite useful both to us and to the build/test/release working group, which is also testing the forum for every major Open edX release.

[ayub02]

@regisb yes i was asking about Q&A testing. Thank you for sharing the results. My question is more about user acceptance testing (UAT). Forums unfortunately has one of the most complex workflows in edX. Here is a sheet that we have used in the past to test discussion MFE after completion of BD-38. https://docs.google.com/spreadsheets/d/1dFZdYkzrUnKTjrQZl7MIqoGu8Y24u0-SmSqsrrPWAi4/edit?usp=sharing

Maybe your team and run these tests with a backend configuration(s) that makes sense (you and @asadazam93 can probably discuss that).