Pagination on the Sinequa sql.engine Api

[CarsonDavis]

Description

Implementation Considerations

Deliverable

Dependencies

depends on #

Acceptance Criteria

Preview Give feedback

1. Criteria 1

[saifrk]

-Docker Configuration Changes: Updated CPU and memory limits and reservations to the celeryworker service in Docker Compose
-SQL Query Functionality Enhancements:
-Pagination in SQL Queries: Modified the sql_query method to support pagination, which helps in handling large datasets by fetching
data in chunks of 5000 records at a time.
-Dynamic SQL Construction: Introduced SKIP and COUNT clauses dynamically into the SQL queries based on pagination
-Data Processing: Added a method process_and_update_data to handle batch data processing and updating operations
-Error Handling: Improved error handling in data processing to manage exceptions more effectively, specifically focusing on key
errors and general exceptions.
-Updated the fetch_and_replace_full_text function to perform Integrated Fetching and Processing, and also the soft_time_limit of the celery task has been increased.