migrate Annual Reports to BI Airflow

[drjova]

The Annual Reports feature needs to be migrated to BI Airflow. This task involves moving three tables to the new system and ensuring that the migration is seamless. The existing dashboards should be connected to the new database post-migration.

Scope:

Migration to Airflow:

Migrate the three tables defined in the models.py to BI Airflow.
Ensure that the data pipeline is established correctly in Airflow.

Dashboard Integration:

Update the existing dashboards to connect to the new database once the migration is complete.
Ensure the transparency of the migration process, minimizing downtime or disruptions.

Tasks:

• Data Migration (We can create two DAGS):

  • Migrate the get_subjects data to Airflow.
  • Migrate the get_publications task to Airflow.

• Dashboard Integration

  • Update the existing dashboards to fetch data from the new BI Airflow database.
  • Test the dashboards to ensure they are functioning correctly post-migration.

• Deploy to Production

Acceptance Criteria:

• All three tables are successfully migrated to BI Airflow.
• get_subjects and get_publications tasks are running correctly in Airflow.
• Existing dashboards are updated to connect to the new database and are functioning without issues.
• The migration process is seamless with minimal impact on current operations.

References:

• Annual Reports code
• Models.py File