1s1t for Redshift

[evantahler]

• Redshift gets the Dv2 treatment
• Also include price data in perf dashboard(s)
• Speed and Cost fit perf constraints above across many sizes of sync
• manage rollout (3 beta users)

---

SqlGenerator
CatalogParser
Normalization into container (but we need to roll it out)
V1->V2 migrator
Make it work

---

migration (v1-->v2)
full refresh
incremental

[pranasziaukas]

manage rollout (3 beta users)

Is it still possible to participate in the upcoming beta rollout @evantahler? cc @cynthiaxyin

I filled some form in the summer stating that we'd be interested in trying out the Redshift destination V2.

[cynthiaxyin]

@pranasziaukas Thanks for your interest! Yes, it is still possible and you can track the progress of this project on this issue. #31575 is likely where we will add updates about the beta rollout. cc @evantahler

[cynthiaxyin]

Redshift Destinations V2 weekly update:

✅ What’s been completed:

• draft PR being collaborated on
• Cynthia, Ed, and Gireesh met to walk through the code and generate a shortlist of TODO's
• 23 beta users contacted with a "coming soon" email

📚 Learnings and hurdles:

• beta users are excited!
• some of the existing destinations code structure could be better organized
  • Gireesh documented some of the current flow

🚫 Blockers:

• folks got sick

🌱 Looking ahead:

• more collaboration and progress on the code (SqlGenerator, migration code, etc.)
• Cynthia working on documenting our process for future implementations of Destinations V2

[cynthiaxyin]

Redshift Destinations V2 weekly update:

✅ What’s been completed:

• lots of progress on the Dv2 implementation!
  • RedshiftSqlGenerator and migration code are complete
  • Redshift Dv2 and T&D test suites are all passing

📚 Learnings and hurdles:

• the Dv2 test suites are pretty high-effort to set up, but they pay off very quickly in terms of leverage

🚫 Blockers:

• minimal dev time last week due to hack days + holidays

🌱 Looking ahead:

• move Redshift standard inserts onto the async framework to reduce duplicate/throwaway work
• plug T&D into the rest of the connector (standard inserts and S3 staging)
• release the first beta image once one loading method is working (targeting next week)
• beta user group will receive comms and be added to a private Slack channel when the first beta image is released

[cynthiaxyin]

Redshift Destinations V2 weekly update:

✅ What’s been completed:

• moved Redshift standard inserts onto the async framework
• Dv2 is ready for beta testing (both standard inserts and S3 staging) with green build and passing tests

📚 Learnings and hurdles:

• preliminary performance testing shows Dv2 on par with Dv1 speed
• Metabase is missing data around Redshift loading methods

🚫 Blockers:

• none

🌱 Looking ahead:

• investigating memory consumption footprint after switching Dv1 standard inserts to async
• final PR review and performance testing
• release the beta!
  • beta user group will receive comms and be added to a private Slack channel

[cynthiaxyin]

Redshift Destinations V2 weekly update:

✅ What’s been completed:

• launched the beta! 🎉
• invited ~20 users to try it out #ext-destinations-v2-redshift-beta

📚 Learnings and hurdles:

• beta user has already given us valuable feedback around data types, permissions, and optimizations

🚫 Blockers:

• none

🌱 Looking ahead:

• continue sourcing feedback from beta users (goal is 3, ideally covering both loading methods)
• road to GA and certification
• further improvements as we prepare for Postgres and MySQL V2

[cynthiaxyin]

Redshift Destinations V2 weekly update:

✅ What’s been completed:

• PR campaign via many tools to source more beta testers
• identified additional improvements around error reporting and timeouts

📚 Learnings and hurdles:

• refined our speed measurements which look good

🚫 Blockers:

• waiting for 2 more beta testers to give us feedback

🌱 Looking ahead:

• fix outstanding bugs
• tentative timeline
  • 1st week of Jan - GA
  • middle of Jan - set up Redshift cost dashboard
  • Feb 1 - certification
  • Mar 1 - breaking change deadline

[cynthiaxyin]

Redshift Destinations V2 weekly update:

✅ What’s been completed:

• ~3 more beta users gave feedback

📚 Learnings and hurdles:

• OSS users need to select Raw data (JSON) as their Normalization method under the Transformation tab
• beta users have helped us discover error cases to address

🚫 Blockers:

• none

🌱 Looking ahead:

• fix remaining bugs for GA
• write up summary of beta program
• tentative timeline
  • 2nd week of Jan - GA
  • 2nd half of Jan - set up Redshift cost dashboard
  • Feb 1 - certification
  • Mar 1 - breaking change deadline

[cynthiaxyin]

Redshift Destinations V2 weekly update:

✅ What’s been completed:

• completed bug fixes/tasks for GA

📚 Learnings and hurdles:

• reflections on beta program

🚫 Blockers:

• none

🌱 Looking ahead:

• 3 remaining bugs/tasks, aiming to GA this week
• subsequent timeline
  • 2nd half of Jan - set up Redshift cost dashboard
  • Feb 1 - certification
  • Mar 1 - breaking change deadline

[cynthiaxyin]

Redshift Destinations V2 weekly update:

✅ What’s been completed:

• bug fixes (1, 2)
• GA code changes are ready

📚 Learnings and hurdles:

• more beta user feedback

🚫 Blockers:

• CI issues
• OC

🌱 Looking ahead:

• this week
  • one open bug
  • breaking change release doc
  • GA by end of week
• then
  • 2nd half of Jan - set up Redshift cost dashboard
  • mid Feb - certification
  • mid March - breaking change deadline

[cynthiaxyin]

Redshift Destinations V2 weekly update:

✅ What’s been completed:

• we GA'd! 🎉
• breaking change release doc

📚 Learnings and hurdles:

• use_1s1t_format continues to rear its head
• also the CDK

🚫 Blockers:

• n/a

🌱 Looking ahead:

• Redshift cost dashboard
• certification!
• breaking change deadline March 15

[cynthiaxyin]

onward to certification! airbytehq/airbyte-internal-issues#2386