feat: replace Elasticsearch by Meilisearch

[regisb]

With this change, we get rid of Elasticsearch across all of Tutor. Instead, we run Meilisearch, which is much more lightweight in terms of memory usage. Obviously, this is a (very) breaking change. Indexing commands will be run during init, such that search should work as before.

After the edx-search PR is merged and the dependency is upgraded in edx-platform, we should remove the manual RUN pip install ... command.

[pomegranited]

@regisb This is looking great -- just a couple of minor issues when I used this in my tutor dev env, along with these open PRs:

• feat: add Meilisearch-compatible search engine openedx/edx-search#162
• [sumac] feat: migrate from Ruby to forum v2 Python app tutor-forum#48
• feat: enable meilisearch in the Authoring MFE if enabled via tutor tutor-mfe#228 -- minor change required to MFE_CONFIG

Am deploying a sandbox to test the k8s setup, will report back with any issues there.

[pomegranited]

Should gate these too (though not sure about which variable to use to do this?)

Suggested change

	./manage.py cms reindex_studio --experimental
	./manage.py cms reindex_course --active
	{% if ACTIVATE_MEILISEARCH %}
	./manage.py cms reindex_studio --experimental
	./manage.py cms reindex_course --active
	{% endif %}

[regisb]

No, I don't think that's right. We assume that meilisearch is running, somewhere. Even if RUN_MEILISEARCH is false, it just means that meilisearch is running on a remote host. So we still need to run the migration commands.

[pomegranited]

Oh yes, you're correct!

There's an issue with reindex_course --setup referencing elasticsearch directly (openedx/edx-platform#35743), but that's not Tutor's problem :)

[regisb]

I suggest we run ./manage.py lms shell -c "import search.meilisearch; search.meilisearch.create_indexes()" in init/lms.sh. What do you think?

[bradenmacdonald]

Also keep in mind that until we resolve openedx/modular-learning#235 (which is in progress), we should avoid running ./manage.py cms reindex_studio --experimental on any instances with thousands of courses or more.

[bradenmacdonald]

As for re-indexing courses, I haven't looked into it but it does sound like we're doing it more than is necessary.

[regisb]

Is there a function that would only create and configure the indexes (without deleting them first), such that we could call this function at every launch? Pretty much like a ./manage.py migrate command.

[bradenmacdonald]

It doesn't exist yet, but we could add that. A --create-only flag would be trivial.

The edge case is when upgrading to a newer commit/release and the index parameters have changed; in that case, we need to replace the index and the new index will be empty unless we also re-index all the content. This could be done asynchronously though; it also doesn't replace the existing index until the reindexing is complete, so there's no downtime.

[regisb]

I think we do need a lightweight, idempotent index creation command that works in just all cases. This can take different forms: a single python function, a management command, or a set of migrations. If we can get this in Sumac then that's great, if not then we should probably think about it for later releases.

[bradenmacdonald]

I added a proposal to openedx/modular-learning#235 for a new --init flag that will achieve this.

[pomegranited]

👍 once comments 1 & 2 are addressed.

I tested this by deploying and testing a k8s sandbox configured with openedx/edx-platform#35689 . The sandbox demonstrates a functional meilisearch deployment, with published course content search indexed for students, and draft content for courses + libraries v2 indexed in Authoring MFE for Content Authors. Note that this deployment required fixes from overhangio/tutor-mfe#231 + openedx/edx-platform#35743.

However, I'm unable to create new posts using tutor-forum@v2 (overhangio/tutor-forum#48), so I wasn't able to test this with the new edx-search Meilisearch engine. Happy to leave the sandbox running to test updated versions of tutor-forum@v2 and https://github.com/edly-io/edx-platform.git@edly/forumv.

• I tested this using the sandbox configured by DO NOT MERGE -- testing Meilisearch in tutor core openedx/edx-platform#35689
• I read through the code
• I checked for accessibility issues N/A
• Includes documentation -- upgrade documentation will be done as part of a separate PR.
• User-facing strings are extracted for translation N/A

[regisb]

Again, thanks for the quick and thorough review Jill. Meilisearch is not supposed to work with forum v2 just yet, this would be pending some work on my side. I'd like to complete this PR before I move on to the integration of meilisearch in forum v2.