Variable view rewrite & consistent data source coloring

[frostyfan109]

• Rewrite/cleanup variable view w/context so that we can support filters and other features that weren't otherwise feasible
• Add infinite scroll to variable view
• Fix study data source (e.g., HEAL studies vs Non-HEAL studies) coloring
• Group by study/collapse into variable list
• Add filters:
  • Subsearch filter
  • Data source filtering
  • Sort by score/data source, with ascending/descending options

[joshua-seals]

RUN npm clean-install is causing the builds to fail, just fyi, is this a custom command or do you mean RUN npm ci

[gaurav]

I can't review the many code changes in this PR, but the end results look great! Some thoughts:

1. I think the subsearch filter is a bit too fuzzy: I tried filtering with "heart" and ended up matching a ton of "healing"s and one "part". But I LOVE that the substring search includes the variable source! So I think you're definitely heading in the right direction here.
2. I would personally have "Group by study" be the default, but I'm very curious what other Dug folks think -- I wonder if we could get you to demo this PR at one of our Tuesday 3pm meetings or at another time? I think we could get a lot of useful feedback on the UI that way.
3. Ideally the counts in the boxes (e.g. "HEAL Studies (4)") should change when you change the histogram -- but maybe that's too complicated? I like how the text above it changes from "Showing X of Y variables" to "Showing X of Y studies" when you turn on "Group by study" mode -- maybe the box could also change from "HEAL Studies (4 studies)" to "HEAL Studies (10 variables)" or something?
4. These results will be a lot more readable if we were to include the form/module name in these search results or if we had a study page to link to, but we don't have either yet :( But someday!

[frostyfan109]

RUN npm clean-install is causing the builds to fail, just fyi, is this a custom command or do you mean RUN npm ci

npm ci is just an alias of clean-install as far as I know. I think we just need to rebuild the package lock

[frostyfan109]

I can't review the many code changes in this PR, but the end results look great! Some thoughts:

1. I think the subsearch filter is a bit too fuzzy: I tried filtering with "heart" and ended up matching a ton of "healing"s and one "part". But I LOVE that the substring search includes the variable source! So I think you're definitely heading in the right direction here.
2. I would personally have "Group by study" be the default, but I'm very curious what other Dug folks think -- I wonder if we could get you to demo this PR at one of our Tuesday 3pm meetings or at another time? I think we could get a lot of useful feedback on the UI that way.
3. Ideally the counts in the boxes (e.g. "HEAL Studies (4)") should change when you change the histogram -- but maybe that's too complicated? I like how the text above it changes from "Showing X of Y variables" to "Showing X of Y studies" when you turn on "Group by study" mode -- maybe the box could also change from "HEAL Studies (4 studies)" to "HEAL Studies (10 variables)" or something?
4. These results will be a lot more readable if we were to include the form/module name in these search results or if we had a study page to link to, but we don't have either yet :( But someday!

(1) Good note about the fuzziness. I was noticing something similar happening with another term (interim vs interval) so I looked into the code, and it turns out Lunr stems each term before making it into the index, which ultimately results in the fuzziness being a bit more forgiving than perhaps intended. I think we could turn down the fuzziness by 1 edit distance.

(2) I agree, group by study as default makes sense as that's the appearance of variable view prior to this PR, so it's what people expect. I was wondering if you had any feedback about the placement of the button? I'm not super happy with putting it in the filter menu but couldn't find a better position for it.

(3) That's an oversight on my part, I'll change it so the counts reflect the amounts actually being displayed.

(4) Ping me when these changes are implemented :)

[frostyfan109]

Regarding fuzziness, we were by default using 1 LD if < 5 chars, otherwise 2 LD. I've fixed it to use 1 edit distance (LD) regardless of term length.

[hina-shah]

Thanks for all the work @frostyfan109 ! A few cleanup things, but other than that I think we should be okay.

[hina-shah]

(3) That's an oversight on my part, I'll change it so the counts reflect the amounts actually being displayed.
Was this implemented?

[frostyfan109]

(3) That's an oversight on my part, I'll change it so the counts reflect the amounts actually being displayed.
Was this implemented?

I'm not sure, but I don't see it mentioned in newer commits so probably not. I'll check

[frostyfan109]

@hina-shah can you approve?