Feature/backups samples with restore

[prunge-helix]

Added restore sample.

The restore database name is generated similarly to the test database name with a different prefix and similar truncation logic to prevent > 30 character errors.

I have noticed there's some race condition happening with the DB. Sometimes it works, sometimes getting an error:

snippet_test.go:299: got output "deletebackup failed with rpc error: code = FailedPrecondition desc = Backup projects/appdev-soda-spanner-staging/instances/test-instance/backups/my-backup cannot be deleted at the moment because it is still in use by 1 databases for pending restore activity. Please try deleting the backup once the restore or post-restore optimize operations have completed on these databases. The databases using this backup are displayed in the UI in the Backups table `In use by` column, and returned by the GetBackup API."; want it to contain "Deleted backup [my-backup]"

My best guess is that the optimizing stage is sometimes still ongoing after the restore operation completes which requires the backup to still be present. Do we need to wait for the database state to exit the 'optimizing' state?

[prunge-helix]

I'm also getting that same error with deleting a backup from the node samples as well. Pretty sure I had the node samples working at one point so maybe this has changed? Or maybe it's always been a race and I've never hit it before.

[jeanbza]

Could you add tbpg@ to this review?

[skuruppu]

Could you add tbpg@ to this review?

Please also add @larkee to the repo.

[prunge-helix]

Could you add tbpg@ to this review?

Please also add @larkee to the repo.

@jsuresh @ashvds would you be do the add/invite? I don't have enough permissions to do this.

[ashvds]

They both have invites

Could you add tbpg@ to this review?

Please also add @larkee to the repo.

@jsuresh @ashvds would you be do the add/invite? I don't have enough permissions to do this.

[larkee]

My best guess is that the optimizing stage is sometimes still ongoing after the restore operation completes which requires the backup to still be present. Do we need to wait for the database state to exit the 'optimizing' state?

Sorry, I totally missed this. Yes, you need to wait for the database to finish optimizing before you can delete the backup.

[prunge-helix]

Sorry, I totally missed this. Yes, you need to wait for the database to finish optimizing before you can delete the backup.

Added poll loop in the test harness to wait for the restored database to come out of the READY_OPTIMIZING state. de7f5a0

[larkee]

LGTM - minor nits