Skip to content

Commit

Permalink
jobs/downloads/process_log: Decrease batch size
Browse files Browse the repository at this point in the history
On our production servers we've been seeing warnings like:

> Failed to fill temp_downloads table: error encoding message to server: value too large to transmit

It looks like this error is returned by `tokio-postgres`, which can't handle the amount of data we throw at it when using a batch size of 10k rows. This commit decreases the batch size in the hope that this makes the background job succeed. If not, we might have to further decrease it.

A potentially better solution for this would be to use a `COPY` query, but that wasn't available yet in `diesel` when the code was originally written.
  • Loading branch information
Turbo87 committed Jul 31, 2024
1 parent e669bdb commit c8978d2
Showing 1 changed file with 4 additions and 5 deletions.
9 changes: 4 additions & 5 deletions src/worker/jobs/downloads/process_log.rs
Original file line number Diff line number Diff line change
Expand Up @@ -268,11 +268,10 @@ fn create_temp_downloads_table(conn: &mut impl Conn) -> QueryResult<usize> {
fields(message = "INSERT INTO temp_downloads ...")
)]
fn fill_temp_downloads_table(downloads: DownloadsMap, conn: &mut impl Conn) -> QueryResult<()> {
// Postgres has a limit of 65,535 parameters per query, so we have to
// insert the downloads in batches. Since we fill four columns per
// [NewDownload] we can only insert 16,383 rows at a time. To be safe we
// use a maximum batch size of 10,000.
const MAX_BATCH_SIZE: usize = 10_000;
// `tokio-postgres` has a limit on the size of values it can send to the
// database. To avoid hitting this limit, we insert the downloads in
// batches.
const MAX_BATCH_SIZE: usize = 5_000;

let map = downloads
.into_vec()
Expand Down

0 comments on commit c8978d2

Please sign in to comment.