CalData - Benefits Recommender collaboration next steps

[ian-r-rose]

The CalData team had a productive meeting with the CalInnovate benefits recommender team yesterday to discuss their data needs and how to move forward with our collaboration. Here are the notes from that meeting. This morning, as part of CalData DSE's spring planning we sketched out a proposal for next steps over our upcoming sprint (2023-01-09 through 2023-01-20), which this issue intends to capture. The below points could spin out to separate issues as necessary.

1. Investigate the needs around data update frequency

@sarah-letson and @jpetrucione are frequent consumers of the Looker dashboards, and have needs around data freshness and reliability. In particular, @sarah-letson asked if we could increase the data loading frequency from the current six-hour cadence. We can! In order to better serve the needs of the project, I'd like to understand:

• What business processes are currently hampered by having data that is (up to) six hours out-of-date? If we are doing rapid experiments on short timescales, how quickly do we need to be able to analyze results?
• What would an ideal load frequency be?

2. Investigate needs for different filters and aggregations of the benefits widget data.

@mediajunkie was interested in having more aggregated views of the data easily available. For instance, we might want to break down a Clicks/Views ratio by several different facets:

• Experiment variation
• Source site
• Referred program
• ...?

Such aggregation may be possible but painful in Looker, and doing some pre-modeling in dbt could be a more natural fit.

• Discuss with @reganking and @mediajunkie what a helpful aggregate view of the data might be.
• Discuss with @reganking whether he is interested in dbt rapid onboarding to get his hands dirty with some data modeling.

3. Investigate pain points around analyzing and joining data that exists today

Our understanding of the datasets that exist today are:

1. The benefits widget data from DynamoDB
2. Google Analytics Data for EDD (and possibly also the referred-to sites)
3. Information about the different experiments that are conducted (is this in Airtable? @reganking can you confirm?)

It sounds like one pain point is around designing a schema for the experiment variations such that it can be easily joined with widget data to get descriptive statistics of experiment results. Some follow-up things we want to learn:

• Is it useful to load the experiment description data from Airtable (or similar) into our data warehouse for better data modeling?
• Can we help design a schema for the experiment description data such that

4. Determine how to leverage the currently-collected Google Analytics data

A longer-term analysis of this project might include tracking of user journeys from benefits site -> recommender -> benefits site -> completed application -> approved application through Google Analytics data. That would likely require some significant planning and engineering in partnership with downstream agencies. However, we did learn about a query parameter put in the URL of the referral links, and thought it might be fruitful to look for that in our data:

• Confirm with engineering where the query parameter is and how to identify it (perhaps @aaronhans?)
• Look for the query parameter in our GA4 datasets, do some light modeling to get some descriptive statistics.
• If we cannot find that in GA4, determine why not and make recommendations to CDT or our downstream partners
• If we can find it in GA4, can we tell a self-consistent story between the source (benefits widget) and landing page?

[aaronhans]

We currently use query parameters in our links to BenefitsCal.com and getCalFresh.org. We don't have direct Google Analytics to either of these accounts. BenefitsCal doesn't use GA (I think they use amplitude instead). These partners provided the query parameters we should use for them to track inbound links. We depend on them sharing stats with us periodically. We receive a spreadsheet with the status of applications started from inbound traffic from BenefitsCal. We haven't received data from getCalFresh.org yet but expect it soon.

[ian-r-rose]

Thanks @aaronhans -- what about the referral links to things that do fall under the ca.gov domain (WIC, water assistance...)? Or do I misunderstand the construction of the links?

[aaronhans]

We aren't using any tracking parameters for any other links like WIC, LIHEAP, LIHWAP.

We also have noopener & noreferrer set on the offsite a tags so in those cases the partner site shouldn't be able to identify the source of our traffic.

@reganking can elaborate but I believe our partners like WIC, LIHEAP, LIWHAP aren't likely able to share data with us. The water programs have different instructions for each county which sometimes end up with a phone number to call instead of a web form.

[ian-r-rose]

So, would it be fair to say that we currently do not have a way to identify traffic coming from the recommender widget on any of the ca.gov sites (even with Google Analytics data)?

[aaronhans]

Correct Ian.

cc @reganking

[reganking]

Some background on recommender / GA use and rationale. Early on, @aaronhans, @britt-allen, and I clarified what metrics and dimensions we would like from the recommender interactions, but it was clear that adding the weighty GA library would be an issue for placement pages that don't have GA loaded. We agreed that we would, for now, resort to recommender-only views/clicks data and work on getting downstream data reports, as Aaron mentioned. I'm receiving CalFresh & CalWORKs weekly and 90-delayed monthly reports, BenefitsCal CBO-style weekly applications data (gold!), and working on downstream data flows for CSD (LIH(E|W)P), WIC, and CalEITC (just now beginning to enter tax season where CalEITC promotions start).

This is great. I'm unfamiliar with "CBO-style". How manually-generated are these reports? Are they structured tabular data (e.g., excel, csv)? If so, should we do the legwork to start including them in the data warehouse for our modeling needs? - Ian

I shared the BenefitsCal weekly report with you: @ian-r-rose and @britt-allen Since this report provides the most detailed "downstream" referral partner data, it makes sense to view this with respect to supporting an application flow diagram. - Regan

To dbt; yes, excited to get hooked up and collaborating. Have some basic dbt knowledge, so onboarding should be easy. See last note below regarding GA keyword analysis.

Awesome. We're planning to do take advantage of some rapid onboarding training from our contract with dbt (@britt-allen has been spearheading that), though we may wait until one or two more of our team are hired/onboard (February?). But if you're interested in that I'm sure you'd be welcome. In the meantime, since you already have some familiarity, here is our dbt repo. We can get you a seat in our dbt cloud account, and I also put some effort into documenting a local workflow if you feel like getting a local Python environment set up (I prefer this to the cloud UI). - Ian

Thanks Ian ;) I will check it out and would also like to train with the group. - Regan

To No 3, experiments; Yes, Christian has this setup in AirTable and I'm VERY interested in discussing integration options (methodologically and for descriptive statistics reporting). I also have tests draft model I can share and discuss. AirTable may not be the best solution for integration. I've pulled the 30+ experiments into a flat list

We're discussing with @kimberlyhicksodi different ways we might engage on experiment design, so I'd imagine that we may have some recommendations on how to set up the experiment schema to be maximally useful. In the meantime, we'd be interested in looking at an extract of the current table if you have it handy. Airtable may be possible (I haven't kicked the tires on how easy it is to jailbreak that data), but it's likely there are easier systems to integrate. - Ian

Here's our worksheet for BR Experiments based on the list in AirTable. We created the original list prior to launching the widget and have, thus far, tested #1 (in usertesting.com protocol; results not linked), #20-30, with last test #21 (head-to-head) ending yesterday. These are not all really experiments. I have added the uniform parameters to each experiment in the list for my own assessment needs. We do not currently have: a formal protocol for prioritization experiments, determining the impact of experiment outcomes, or reporting of experiments. I look forward to discussing this with @kimberlyhicksodi - Regan

and to update for statistical priorities (e.g., null-hypothesis validation and required confidence levels).

Can you elaborate on what you mean here? Do you already have models or columns for these, or are you looking forward to what we would like to do? - Ian

Only the parameters added in the worksheet, and this is for assessment only at this point. Here's a draft data model and our AirTable Experiment pipeline data definition. The model captures entity relationships I'm interested in with respect to scaling UX multi-variate tests (e.g., language, content, position in a layout, design style, etc.). - Regan

To No 4, given the current constraints, we're focused on the last item you listed (with edit "can" to "can't"): "If we can't find it in GA4, can we tell a self-consistent story between the source (benefits widget) and landing page?" Would love h ear your take on the self-consistent story ;)

I didn't mean anything too fancy by "self-consistent". Was mostly hoping we could say something like: "The number of GA4 views with XX query parameter is within 5% of the number of widget clicks referring to that page". But I guess that is not possible with the current setup. - Ian

That would be great, of course. Capture of gA from referral partners CSD (LIH(W|E)P) and CDPH (WIC) is a goal. Discussions this week hopefully can move this forward a bit. The WIC referral basically points to a ZIP code entry / phone number but with noreferrer set, we're limited. gA data also from search console and trends could help us fill in the gap understanding what the size of our placement page audiences are compared to those searching in California for benefits in general. - Regan

Two or three gA exceptions;

• We have WIC GA data (just no online app/form) ... https://myfamily.wic.ca.gov/)) (UA-31125811-41 under eServices Analytics No 9). Take note of @aaronhans comments above about use of noreferrer ... we could use a datatime proxy maybe?

We could try to match based on timestamp. It would be circumstantial, but I'd view this as largely a gut-check at this point. I'm curious whether you or @aaronhans think that would be a useful exercise. - Ian

Well, WIC clicks already give us the amount of traffic we're generating. Our discussion with WIC on Friday should help us understand where they see their program going, in terms of gA measurement in general. They've only been operational with a new team within the last month, so this is a really an ideal discussion partner. - Regan

• CSD is showing willingness to give us access to their GA4 data (this is maybe where we can focus and build a "success case" showing how CSD excelled ... I'm setting up a meeting to discuss with Rob McAndrews if someone on your team wants to join.

Do we already have this? If not, perhaps we can get CDT to ask them to add the statewide GA tag. - Ian

No. We're meeting with Rob McAndrews team to discuss. They are willing. We can discuss adding the statewide. - Regan

• Other downstream tools coming into play include Amplitude and Code for America custom source report, etc. How to align these with GA reporting? We may need to capture baselines manually.

I'd love to see early previews of these reports if you have any. - Ian

Me too ;) I can try to get an Amplitude report from BenefitsCal. I opted for an excel weekly report. CfA promised us access to their custom dashboard. - Regan

I still think there's also a place for context with respect to Search Console data using the existing keyword analysis. We might collab on a dbt project duplicating what we previously had on Syntasa. This could help us better define our target population searching for benefits.

Do you have any decks or other materials on the keyword analysis? I haven't seen it yet. - Ian

I can walk you though what I did here in the BR enhanced keyword analysis. - Regan

There's also some geo-data we're capturing in weekly BenefitsCal CalFresh reports (Excel spreadsheet) that we could explore.

Very interesting, that would be great to see as well! - Ian

Let's look at the BenefitsCal weekly report I shared with you. It's only capturing counties. - Regan

[ian-r-rose]

Closing as complete, pending new activity we may revisit, but nothing is planned right now

[reganking]

Thanks @ian-r-rose and @britt-allen. We're re-grouping as we leave the benefits pilot. We will select a couple top priority items to look at together. And, give you time to choose what makes sense for CalData. Most of the GA issues are resolved so this should be more interesting to look at in the next round.