Issue with Question Publishing Process: Need Assistance

[rajnishdargan]

Hi everyone,

I'm encountering an issue with the question publishing process.

Background:
Our team recently completed the setup of Sunbird in AWS using the Sunbird installer. During the setup, we noticed that the Question and Questionset schemas were missing from our schema path. We manually uploaded the missing schemas and resolved the issue on our end. The APIs are now functioning correctly.

Steps I followed:

Question Creation:
I successfully created a question with public visibility using the question/v5/create API.

Question Review:
I reviewed the question using the question/v5/review/{do_id} API, and everything worked fine.

Question Publishing:
When I published the question using the question/v5/publish/{do_id} API, the response returned a success status (200).

The Issue:
After publishing, when I attempt to read the question using the question/v5/read/{do_id} API, the status still shows as "Review" instead of "Live," which is unexpected.

My Question:
Which service or job is responsible for handling question publishing in this setup?
Any insights or suggestions would be greatly appreciated!

@HarishGangula @maheshkumargangula @pallakartheekreddy @coolbung @snehal0904 @AbhilashKD

[krgauraw]

@rajnishdargan async-questionset-publish flink job is responsible for publish activity. Please check if the job is deployed and running.

Source Code Ref: https://github.com/Sunbird-inQuiry/data-pipeline/tree/master/publish-pipeline/async-questionset-publish

[rajnishdargan]

@krgauraw
async-questionset-publish is giving following error when we hit the question or questionset publish API.

ERROR org.apache.flink.core.fs.FileSystem                          [] - Failed to load a file system via services
java.util.ServiceConfigurationError: org.apache.flink.core.fs.FileSystemFactory: Provider org.apache.flink.fs.s3hadoop.S3AFileSystemFactory could not be instantiated

sunbird-async-questionset-publish-jobmanager-95f889cdd-72w4l-1725369018876873820.log

Also one observation is that search indexer job is CrashLoopBackOff status and restarting.
Could you please guide what we might have missed or what could be the root cause of error ?

cc @HarishGangula @maheshkumargangula @pallakartheekreddy @coolbung @snehal0904 @AbhilashKD

[krgauraw]

Hi @rajnishdargan if you can share task-manager log of both flink job, i will be able to guide you.

[AbhilashKD]

@krgauraw @pallakartheekreddy
sunbird-ed-installer-Updated-for-AWS.zip

I have made changes to the helm charts to deploy the knowlg and InQuiry on the AWS EKS Cluster for that i have updated the global-cloud-values.yaml file located at the location /terraform/azure/template/
And made changes in the helm charts of the Inquirybb and Knowlgbb located at location in the zip /helmcharts/
You can compare my provided helm charts with the umbrella charts : https://github.com/project-sunbird/sunbird-ed-installer.git branch : release-7.0.0 to know the changes made for deploying the platform in AWS EKS.
I have attached the zip file i am currently using for deployment of the platforms.

Please Note : After this Changes Question and Content Publish is working fine. But Questionset publish is getting skipped, can somebody please guide how to fix the questionset publish issue.
@rajnishdargan can you provide logs for their better understandings ?

[rajnishdargan]

When reviewing the logs, I noticed something suspicious in the questionset-republish-job-manager. Some warnings are appearing, such as:

2024-09-24 05:46:54,206 WARN  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Could not properly dispose of the private states in the pending checkpoint 1119 of job db736ae4357c8023fd8984761f6927c5.

2024-09-24 05:47:54,200 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 1120 (type=CHECKPOINT) @ 1727156874199 for job db736ae4357c8023fd8984761f6927c5.
2024-09-24 05:47:54,205 WARN  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Error while processing AcknowledgeCheckpoint message.
org.apache.flink.runtime.checkpoint.CheckpointException: Could not finalize the pending checkpoint 1120. Failure reason: Failure to finalize checkpoint.

Additionally, in the questionset-republish-task-manager logs, it indicates that the questionset publish is being skipped:
2024-09-24 05:44:03,429 WARN org.sunbird.job.questionset.republish.function.PublishEventRouter [] - Republish Event skipped for identifier: do_21414834543222784013 | objectType: QuestionSet | pkgVersion

Here are the complete logs:
sunbird-async-questionset-publish-jobmanager-79857bb75d-ml9jk-1727156764675173628.log
sunbird-async-questionset-publish-taskmanager-84c49fbfdf-v8zb2-1727156712446868703.log
sunbird-questionset-republish-jobmanager-5d45768d9-c6rns-1727156917915371365.log
sunbird-questionset-republish-taskmanager-6675757fd5-8jfsh-1727156853479736351.log

@kumargauraw @pallakartheekreddy @amitpriyadarshi @gandham-santhosh, could you please guide on how do we fix the questionset publish issue?

@AbhilashKD @coolbung @snehal0904

[rajnishdargan]

@pallakartheekreddy @krgauraw @amitpriyadarshi @gandham-santhosh,

Could you please take a look at my question regarding the questionset publish issue? Question publish and content publish are working fine, but for the questionset publish, the API returns a 200 success response, yet the status of the questionset does not change from Review to Live.

@AbhilashKD @coolbung @snehal0904 @itsvick

[AbhilashKD]

@pallakartheekreddy
Can you share the image name and tag of the print services as your provided helm chart is variablised :
https://github.com/project-sunbird/sunbird-devops/tree/b5797a4039a45c8f2c813ef343529b8a4212a009/kubernetes/helm_charts/core/print
other settings i have added in global-values.yaml file and changed the deployment.yaml file accordingly. Just need the Image name for deployment of the print service

@rajnishdargan

[pallakartheekreddy]

@AbhilashKD sunbirded.azurecr.io/print-service:release-3.0.0_RC1

[AbhilashKD]

@pallakartheekreddy
Thanks.
Can you provide sample config map file needed for print service?
Means which variables need to add in the config map of the print service as i am facing issue as below :
sunbird-print-service-1727952961961914590.log

[aimansharief]

@AbhilashKD
Can you provide the following environment variables to the print-service

service_threads: "2"
sunbird_azure_account_name: ""
sunbird_azure_account_key: ""
sunbird_azure_container_name: ""

sunbird_pvt_azure_account_name: ""
sunbird_pvt_azure_account_key: ""
sunbird_pvt_azure_container_name: ""

[AbhilashKD]

@aimansharief
Thanks
For now its working