Migrate auctions to new database table #3067

sunce86 · 2024-10-17T15:44:02Z

Description

Data migration is controlled by configuration parameter, so it can be enabled on one by one network.

Data migration is done in a background task. Separate batched transactions are executed until all data from solver_competition table is moved to new auction table.

How to test

Manually on sepolia staging, then rest.

m-lord-renkse · 2024-10-17T16:19:07Z

crates/autopilot/src/infra/persistence/mod.rs

@@ -802,6 +802,92 @@ impl Persistence {
        ex.commit().await?;
        Ok(())
    }
+
+    pub async fn populate_historic_auctions(&self) -> Result<(), DatabaseError> {


is it possible to test this beforehand in a unit test or e2e test?

We could even set up a snapshot of the actual DB and and run a local DB migration with that to see how it will behave with real data.

m-lord-renkse · 2024-10-17T16:20:01Z

crates/autopilot/src/infra/persistence/mod.rs

+            ex = self.postgres.pool.begin().await?;
+
+            // update the current auction id
+            current_auction_id = competitions.last().unwrap().id;


what happens if this process get interrupted in the middle of it?

Another point for a separate script, which could store the counter on disk.

m-lord-renkse · 2024-10-17T16:21:17Z

crates/autopilot/src/run.rs

@@ -449,6 +449,11 @@ pub async fn run(args: Arguments) {
            .instrument(tracing::info_span!("order_events_cleaner")),
    );

+    if args.migrate_auctions {
+        let persistence_clone = persistence.clone();
+        tokio::spawn(async move { persistence_clone.populate_historic_auctions().await });


when we spawn this I assume we are still populating both tables, therefore the migration will never end, right? 🤔

MartinquaXD

Rather than injecting this code into this code base which relies a restart to run wouldn't it be more convenient to build a separate small CLI tool for that? That way this code can be run / tested independently from the services.

MartinquaXD · 2024-10-18T06:57:47Z

crates/autopilot/src/infra/persistence/mod.rs

+                        .clone(),
+                };
+
+                if let Err(err) = database::auction::save(&mut ex, auction).await {


Is there are reason we only populate one of the new tables. AFAICS we can populate the proposed_solutions and proposed_trade_executions (at least partially).
I think it might be good to migrate as much as data as possible if we are considering deleting the old table entirely some time in the future.

We have a separate issue for proposed solutions migration: #3056

The migration of solver competition is a bit more complicated and I planned doing it in a separate step -> smaller PRs

squadgazzz

I’ll second for a separate script. Not sure why we need this in the codebase. I understand this is required only to populate the new table with historical data, so it is like a one-shot operation.

sunce86 · 2024-10-18T13:02:32Z

Looks like there is a consensus this is not a right approach. Let me summarize the options:

SQL migration file - Tried first with this, but writing a proper SQL query to convert from data types saved as json into strongly typed new tables was a nightmare to get it right (spent a lot of hours just to get the Auction part right).
Same as (1) but execute SQL queries manually on each database. This would give us more certainty that performance is not problematic, but writing the queries is still painful.
Do it through Rust code. Gives certainty that types are converted properly and much easier to write the code. But has this cons you mentioned - not super elegant and needs a cleanup PR after. But still requires the least engineering time.
A separate tool - probably the most elegant way, but it requires more time - a separate repo with all dependencies, infrastructure PR to setup the tool for execution etc. (or am i missing some shortcut here?)

squadgazzz · 2024-10-18T18:05:28Z

Looks like there is a consensus this is not a right approach. Let me summarize the options:

SQL migration file - Tried first with this, but writing a proper SQL query to convert from data types saved as json into strongly typed new tables was a nightmare to get it right (spent a lot of hours just to get the Auction part right).

Same as (1) but execute SQL queries manually on each database. This would give us more certainty that performance is not problematic, but writing the queries is still painful.

Do it through Rust code. Gives certainty that types are converted properly and much easier to write the code. But has this cons you mentioned - not super elegant and needs a cleanup PR after. But still requires the least engineering time.

A separate tool - probably the most elegant way, but it requires more time - a separate repo with all dependencies, infrastructure PR to setup the tool for execution etc. (or am i missing some shortcut here?)

Why not a simple Python script? Too time-consuming? Chatgpt should work well with generating something from the existing resources.

MartinquaXD · 2024-10-23T06:46:15Z

With a separate tool you don't necessarily have to set up everything to have it automatically be spawned by kubernetes. You could have the tool and still run it manually inside the cluster using k9s.
If you decide to use a python script like @squadgazzz suggested you'd only have to install python in a pod and copy the script into it (easy with k9s). If you decide to go for the rust approach you can cross-compile the binary for linux on your machine and copy that onto the pod.
Cross compiling and copying stuff on and off a pod is covered in this notion doc.

sunce86 · 2024-10-24T14:56:27Z

Closing as it will be executed via external tool

sunce86 added 2 commits October 17, 2024 17:35

Migrate auctions

a0f59df

small nit

be898f0

sunce86 self-assigned this Oct 17, 2024

sunce86 requested a review from a team as a code owner October 17, 2024 15:44

m-lord-renkse reviewed Oct 17, 2024

View reviewed changes

MartinquaXD reviewed Oct 18, 2024

View reviewed changes

squadgazzz reviewed Oct 18, 2024

View reviewed changes

sunce86 closed this Oct 24, 2024

github-actions bot locked and limited conversation to collaborators Oct 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate auctions to new database table #3067

Migrate auctions to new database table #3067

sunce86 commented Oct 17, 2024

m-lord-renkse Oct 17, 2024

MartinquaXD Oct 18, 2024

m-lord-renkse Oct 17, 2024

squadgazzz Oct 18, 2024

m-lord-renkse Oct 17, 2024

MartinquaXD left a comment

MartinquaXD Oct 18, 2024

sunce86 Oct 18, 2024

squadgazzz left a comment

sunce86 commented Oct 18, 2024

squadgazzz commented Oct 18, 2024

MartinquaXD commented Oct 23, 2024

sunce86 commented Oct 24, 2024

Migrate auctions to new database table #3067

Migrate auctions to new database table #3067

Conversation

sunce86 commented Oct 17, 2024

Description

How to test

m-lord-renkse Oct 17, 2024

Choose a reason for hiding this comment

MartinquaXD Oct 18, 2024

Choose a reason for hiding this comment

m-lord-renkse Oct 17, 2024

Choose a reason for hiding this comment

squadgazzz Oct 18, 2024

Choose a reason for hiding this comment

m-lord-renkse Oct 17, 2024

Choose a reason for hiding this comment

MartinquaXD left a comment

Choose a reason for hiding this comment

MartinquaXD Oct 18, 2024

Choose a reason for hiding this comment

sunce86 Oct 18, 2024

Choose a reason for hiding this comment

squadgazzz left a comment

Choose a reason for hiding this comment

sunce86 commented Oct 18, 2024

squadgazzz commented Oct 18, 2024

MartinquaXD commented Oct 23, 2024

sunce86 commented Oct 24, 2024