spanner: code samples for backups #1296

prunge-helix · 2020-03-20T01:14:12Z

Samples and tests for the new Cloud Spanner backups functionality.

Original PRs:
helix-collective/golang-samples#1
helix-collective/golang-samples#2

…nning samples.

…e samples.

…te database) don't have them.

…r names.

…times.

Co-Authored-By: skuruppu <[email protected]>

…mples-with-restore # Conflicts: # spanner/spanner_snippets/snippet_test.go

prunge-helix · 2020-03-25T23:33:11Z

@googlebot I consent.

skuruppu

Thanks for the fixes @hengfengli and @ashvds

spanner/spanner_snippets/snippet_test.go

skuruppu · 2020-03-25T23:48:54Z

Is there any way we can speed up the tests? It takes ~35 minutes to run all of the tests in the repo right now (funny enough, Spanner tests are one of the slower ones). So, I'd prefer not to add another 8+ minutes. Some tests for other APIs "fire and forget/cancel."

Yeah we have the same issue in the other language repos as well :( CreateBackup and RestoreBackup take the longest time. To clean up the backups/databases created during the test, we had some code to wait for the DB to be optimized, but seems like that is no longer a requirement.

I think the minimum requirements for the samples are to show how to create a backup and restore from it. We can fire and forget but that means we won't be able to cleanup some of the resources created during the tests, the Go samples won't be showing users how to wait for a long-running operation to complete and they will be inconsistent with those for other languages.

@tbpg would this be a major issue to add ~15 mins? I realize that's quite a bit of time but I'm not sure how to get around it given that it takes so long to do the backup operations :(

tbpg · 2020-03-26T00:06:58Z

I think if we make the change for Go, we should do it across all of the languages.

I talked with @hengfengli offline -- we're going to change the test to be E2E and increase the timeout. Then as part of the style guide update, which we should file an issue for, we can look into how we can speed up the tests.

Would it help speed up the tests if we added more nodes?

skuruppu · 2020-03-26T01:13:45Z

I think if we make the change for Go, we should do it across all of the languages.

I'm happy to consider that. But it will have to be after the launch though since it's a bit fiddly to change all the samples now.

I talked with @hengfengli offline -- we're going to change the test to be E2E and increase the timeout. Then as part of the style guide update, which we should file an issue for, we can look into how we can speed up the tests.

I filed #1315.

Would it help speed up the tests if we added more nodes?

Unfortunately, it won't help :(

hengfengli · 2020-03-26T03:31:44Z

Kokoro CI doesn't pass due to flaky storage tests:

googleapi: Error 429: The project exceeded the rate limit for creating and deleting buckets., rateLimitExceeded

hengfengli · 2020-03-26T06:00:51Z

@tbpg Tried a couple of times. Still failed due to the flaky storage tests.

tbpg · 2020-03-26T13:33:43Z

Will you merge the latest changes? Looks like I don't have permission. We can merge and fix the storage issue separately.

prunge-helix and others added 30 commits December 3, 2019 10:02

Allow API endpoint to be configured with environment variable when ru…

033ec18

…nning samples.

Trim the database name to 30 characters if the generated name goes over.

704d1cb

Add create backup sample.

66ae8bc

Delete any backups that belong to the test database before running th…

faaf1d7

…e samples.

Example reading metadata from the backup operation.

f749d67

Remove check for database name since the other tests (apart from crea…

b9b3d83

…te database) don't have them.

Sample for deleting a backup.

dea2dbc

Sample for updating a backup expiry time.

041dc86

Sample for list backups of a database.

84c7990

Whitespace fix.

c0ecd51

Fix up the database name trimmed message to show both before and afte…

18df171

…r names.

Sample for restoring backup.

e55626b

Check for errors in the restore operation.

617696f

Whitespace fix.

8ad08f2

Use backup info from the restore response.

157eb2b

Use restore info from the restore response.

0bd56a5

Include create time and use backup attributes for output.

4e6cb21

Add list backups example that uses pagination.

a25c3ea

Add example that lists backups by backup name.

453d6fb

Add example that small backups by filtering by state and backup size.

4394417

Add example that lists new backups by filtering by create and expire …

cd36bdc

…times.

Fix whitespace.

843e3b1

dbFragment variable rename.

193e957

Update spanner/spanner_snippets/snippet.go

f0f1a60

Co-Authored-By: skuruppu <[email protected]>

Add 30 days to existing backup expiry for update sample.

4c03568

Add list backup operations sample.

f1e65e6

Add list database operations sample.

2b66743

Add cancel backup sample.

b2eb15b

Fix to make backup cancellation actually work.

aefa353

Merge branch 'feature/backups-samples-part-1' into feature/backups-sa…

ab2c744

…mples-with-restore # Conflicts: # spanner/spanner_snippets/snippet_test.go

skuruppu approved these changes Mar 25, 2020

View reviewed changes

Sync go mod with master.

353fc21

skuruppu reviewed Mar 25, 2020

View reviewed changes

spanner/spanner_snippets/snippet_test.go Outdated Show resolved Hide resolved

Update backup tests to E2E and the total test timeout to 60 mins.

3f0b3bb