User research report Sprint 7, May 2021

User research report

Sprint 7, May 2021

Executive summary

We conducted research to explore these design hypotheses around the concept of electronic proposal creation and personalized dashboards for NRM G&A users. To explore these hypotheses, we created these mock ups that focus on a refined dashboard concept, based on learnings from sprint Hen of Woods research, and an electronic proposal creation process. We observed 6 program managers using these mocks to complete tasks and we asked them open-ended questions to learn how they’d feel about more collaboration with Grants Management Specialists early in the proposal phase.

Many of the refinements to the dashboard concept meet the Program Manager needs identified in sprint Hen of the Woods research, and we also got some ideas on how to make it even better. There’s a fair amount of work to be done to improve the proposal creation form, but we proved our initial hypothesis, and people liked the more progressive approach.

New user needs statements

Here are the new user needs statements that we identified in rough priority order, based on what we think needs to be part of the MVP, what should come next, and what can wait until later.

MVP: Version 1 (V1)

As someone creating a new proposal, I need a way to connect relevant docs to my proposal for others to review, so that reviewers will be able to tell whether the proposal is accurate, complete, and fundable. #241
As someone creating a new proposal, I need to understand whether the opportunity has a workplan and has been vetted by budget before I invest any effort into entering data. #237
As someone creating a new proposal, I need to understand that the proposed start date is the start of the agreement, not the work, so that my proposal will be accepted.
As someone creating a new proposal, I need any applicable data from workplan to be auto populated so that I can avoid data entry errors and the need to call other people to get the information. #185
As a user, I need an understandable way to filter proposals and agreements by dates that are relevant to me so that I can find what I need quickly.
As a proposal creator, I need to understand the questions I’m being asked and know how to enter the information I need to provide so that I can submit a new proposal correctly and completely on the first try. #248
As a Program Manager, I need to understand the expectation that I will have been in contact with my cooperator before I get started creating a new proposal so that I can be sure a proposed work is viable to avoid wasting effort creating a proposal for a project that can’t happen. #235
As a Program Manager, I’d like to be reminded to discuss a new proposal with a Grants Management Specialist before I start entering data into the system, so that I can be sure the proposed work is viable to avoid wasting effort entering data for a project that can’t happen. #236
Agreement team - As someone creating a new proposal, I need to be able to add more than one budget approver to a new proposal so that I can be sure everyone who needs to will understand the proposal’s progress. #238
Agreement team - As someone creating a new proposal, I need to be able to add more than one grants management specialist to a new proposal so that I can be sure everyone who needs to be notified will be notified re: the proposal’s progress. #238

Agreement team - As someone creating a new proposal, I need to be able to add cooperator contacts to a new proposal so that I can be sure everyone who needs to be notified will be notified re: the proposal’s progress. #238
As someone creating a new proposal, I need the people I’m assigning to the team to have the appropriate training and permissions to be able to carry out the roles to which I’m assigning them so that I don’t inadvertently assign someone to a task they don’t have the training or authority to do. #240
As someone creating a new proposal, I need in-context guidance on how to write about the different components that are required as part of a project description so that I can be sure my proposal will be accepted.
As a user, I need to better understand who is on the “team” that I can assign to tasks and/or automatically notify of a proposal or agreement’s progress.
As someone creating a new proposal, I’d like basic information about the proposal to be persistent on every page so that I don’t lose track of what I’m working on.

Later

As a user, I’d like to be able to tailor the parameters that will trigger alerts so that I can be sure I’ll be alerted in time to do what I’m responsible for.
As a user, I want to be able to filter my list of agreements by what has an alert, so that I can easily see the records that need the most urgent attention.
As a Program Manager, I’d like a direct path from creating a new proposal to filling out the appropriate instrument online so that I can feel confident I’m using the correct instrument and avoid duplicate data entry down the road.
As a G&A user, I’d like a direct path from my dashboard to a proposal, grant, or agreement’s linked documents (financial plan, SOW), so that I can find the information I need quickly.

Research questions and insights

Here are the research questions we sought to answer and what we learned:

Electronic proposal creation

How do Program Managers feel about the progressive disclosure approach to entering proposal data?

Participants' reactions to the progressive disclosure approach were generally positive! Several folks commented that the approach would be helpful for new PMs in particular.

“This seems a lot easier than the NRM process I just went through, so that’s great.” - Sparrow

“For now it wouldn’t change anything, but if I were starting out it would be so much easier to understand. If you don’t have a user guide in front of you right now it’s so easy to mess up.” - Chickadee

“Yes, and it’s asking the same questions as NRM but easier to use so folks should be prepared.” - Cardinal

“I’m so excited, it’s so intuitive if it could be like this.” - Warbler

“The biggest advantage I see is that it’s a lot more intuitive than the current process.” - Goldfinch

Do we have the minimum fields for creating new proposals right?

For the most part, yes, but there are a few things that participants felt were out of place or missing.

What information is out of place?

There are opportunities to improve how the questions and form fields are ordered and grouped. Folks expected to link workplan up front, felt that it was missing from the first page, and were then surprised when it appeared later in the process. A benefit of linking workplan up front is that it can pre populate some of the project overview and commitment request data.

Participants commented that pre-populating fields with workplan data would be helpful, despite our previous research finding that workplan data is not always good quality, and workplan pre populated fields often need to be edited.

A linkage to workplan early on in the process provides additional information that folks rely on to feel confident submitting a new proposal—it conveys that the proposed project has been vetted by budget, and folks want to know that a proposed budget amount has been vetted before they invest time in entering data.

“[Knowing there’s a workplan ID] gives assurance that the money is lining up.” - Chickadee

“I’d probably want to know G&A perspective if it’s already in workplan or planned for, line officer approval, something signifying “yes, there’s been some vetting” - Cardinal

Participant comments upon seeing workplan ID alongside adding commitments:

“I like seeing the workplan link here. Also might be nice to have it on the overview page” - Warbler

“This would be more useful earlier on. A lot of that information would fill in.“ - Chickadee

Other information that appeared out of place to participants was the partner/cooperator name and the total amount of funds being requested—people expected these details to be part of the first step in proposal creation.

“I don’t think I’m seeing the cooperator. That’s always where I start. I'd expect that to be right up here by the title.“ - Sparrow

What information or form fields are missing?

Several people commented that it's important to be able to point reviewers to or revisit draft SOWs and financial plans as a part of submitting a draft proposal and felt that an attachment capability was missing from our mocks.

People also need the ability to add more people to the “agreement team” than our mocks allowed, such as more than one budget approver, more than one grants management specialist, cooperator contacts, and “other” people who might need to stay informed about a proposal’s progress.

When it came to requesting commitments, one person said they’d need the ability to associate more than one workplan ID with their proposal, and another person commented that they may need more job code fields saying “anywhere from one to five [job codes] is typical.”

Is there any information we’re asking program managers for that they won’t know at the time of proposal creation?

Yes! There were a few questions that stood out as confusing or difficult for participants to answer at the time they’re creating a new proposal.

On the overview page:

Project description could be more structured:
- “I wonder about project description: I would have a Q for GMS about what needs to go in there. It seems too open ended. I would want to know how much I need to stick to particular set of language vs. off the top of my head “here’s what I think it will do” - Sparrow
- “Questions like this - the quality of the response are based on the experience of the person entering the information. If they work with agreement instruments frequently or have relationships, they’ll have info to get the ball rolling but if they’re new or do it infrequently, ? I’m not sure you would get the information necessary to start the process.” - Cardinal
Proposed start date tripped people up:
- “Proposed start date of agreement? Sometimes that is hard to switch in my mind.” - Warbler
- “Proposed start date is always hard. I always consider it a guesstimate. I’d wonder if I put a May date, but it's approved April, is it a problem to get started in April?” - Sparrow
- “Start date also concerns me. I can see it happening more frequently than not that the start date is Monday, May 10 [next Monday]. They have tomorrow to work on it. - Cardinal
Will this be covered by FFA stumped people:
- “I would put a little more description on “will this be covered by FFA”... a little more definition there. Could be confusing. We’re giving federal funding to the partner no matter what, so what makes this different?” - Goldfinch
- “Don’t know what Federal Financial Assistance means. Don’t know how I'd answer that question. I’d have to ask someone.” - Loon
Answering Advanced allowed? may be better left to grants management specialists:
- “The advance allowed question, that’s assuming the person doing this knows what they’re doing. I almost want to keep that out of there unless there’s been a conversation with the G&A specialist to do that.” - Goldfinch

On the partnership details page

New cooperator is not clear whether it’s new to the PM, the forest, or to USFS altogether. One PM explained why they wouldn’t be able to know the answer.
- “Is this a new cooperator, I would know to say “no” but if someone else on a different forest was going to start working with them, how much do they need to know about MY work with them. You should be able to say “I think it's a new cooperator…?” - Sparrow
DUNS # might throw people off initially, but they’ll probably take to it quickly:
- “I think finding a DUNS # could be a new concept to some folks, but if there was a help guide or question mark that someone could click on to find DUNS #, or telling them to get it from the cooperator would be helpful.” - Cardinal

On the Agreement team page:

People don’t always know the grants management specialist when they’re creating a proposal, but they said that it’s generally not an issue.
- “You might not know the Grants Management Specialist when you start working on it.” - Warbler
- "Budget approver, I think I know who I would put, but GMS I don't always know. So maybe that would be part of emailing the person I typically work with. In some cases I don't know who’s going to be assigned, so I can’t always answer that myself. GMS, I feel like it's often multiple people. Esp. if there are trainees. Same w/ budget, We had this bunch of GAOA projects where there was a budget person and multiple backups.“ - Sparrow
People generally know a budget contact, but they aren’t always clear who the budget approver for a given proposal is.
- “BA approver would be helpful with a gray explanation text - budget person on the unit that the agreement is with, vs who the funds come from. That would be helpful because the person making this choice would be able to answer that based on that direction.” - Loon

On the commitments page:

Being asked to manually enter Budget object code and Budget org code may confuse most people:
- “It’s not used outside of budget. Most program managers have no idea what that means…” - Chickadee
- “Budget object code never makes sense to me so I always have to phone a friend… Budget org code is also confusing. I never know what to put.” - Sparrow
Being asked to manually enter Fiscal year and budget fiscal year probably will too:
- “...that would be confusing to people who aren’t heavy into budget. If you could populate all of that from the workplan and just have budget org code and job code, the rest is superfluous from this end.” - Chickadee

How do Program Managers feel about entering commitment data at the time they're creating a proposal?

Generally fine. We asked folks if they’d feel comfortable submitting the information to budget and the answer was yes. However, there was some information on the commitments page that a few folks felt should be auto populated from workplan; otherwise they’d need help to fill it in with confidence.

Do Program Managers rely on data from workplan as they go about creating new proposals and commitments?

Yes! A few people commented that they’d need data from workplan to be able to request a commitment and would prefer that workplan auto populate the information. Two folks described relying on screen shots of workplan that their budget contact sends to them to request commitments.

One person suggested that the amount, BLI, fiscal year and budget fiscal year would be particularly helpful to have workplan auto populate.

Q: Is there info from workplan you need to request a commitment?

A: Only the amount and the BLI. I typically know those already, but I confirm that they match [workplan] before I put them in NRM.“ - Sparrow

What are people’s needs around saving their progress? Where and how would it be intuitive for them to do this?

Most people correctly assumed that their information would be saved as they went and also appreciated the ability to save at each step.

What do Program Managers think happens after they submit?

People assumed that submitting the proposal would move it forward for budget to review and approve. People also assumed that the system would automatically notify the budget approver and grants management specialist of the proposal’s progress and what they need to do.

“I’m guessing that people get emails. G&A folks, budget folks get emails that they have something to do. Maybe I would get an email saying that I, notifying me of the progress I guess. I’m guessing all three of us can have access to the same page and go in and do their portion of it.” - Warbler

“I would expect that my role would be done for a while and the next task in the workflow would be for [Budget Approver] to give the budget approval. Then G&A would do the review…” - Sparrow

“What would be better than how it works now, is if it automatically generated a notification to the three individuals instead of having to paste in the information of the budget approver.” - Cardinal

How do Program Managers feel about creating agreements in the system?

Several people commented that it would be useful for our proposal creation form to then generate an agreement instrument template that they could fill out in the system, which is the intended future direction.

“I think I love it. At least the agreement form itself... if I just had a choice like turbo tax… ‘will your cooperator be doing publications Y/N’ and if no, you don't need these provisions.” - Sparrow

“I would like to see the actual documents in the system, like the draft instrument… It would be really helpful if once it got budget approved if the whole package went to whoever it needed to.” - Chickadee

How do Program Managers feel about involving Grants Specialists earlier in the process?

Pretty good! Most of the folks we talked to commented that they would typically have connected with their grants management specialists prior to creating a new proposal and that a system that helped drive such early collaboration has the potential to save PM and GMS time.

One person felt that our new proposal creation form might reduce their need to “bug” grants management specialists to figure out what kind of agreement they need. Other participants hypothesized that using our form to involve GMS early in the process would save GMS time by improving the quality of proposal submissions and ensuring that they're routed correctly, while a second participant offered the counterpoint that they wouldn’t know how to start a new proposal had they not talked to their GMS first:

“That would make your job a bit easier to triage and weed out. I know you get a lot of silliness submitted and you’re very patient given those realities. If there is a way from a biz efficiency standpoint to weed things out and channeled to the direction they need to go, that totally makes sense.” - Cardinal

“If I haven’t talked to my grant specialist about an instrument I don't know where I’d start. If I have, I'd go to create a proposal and start it that way.” - Chickadee

While participants were generally enthusiastic about the idea for early collaboration with GMS, one person raised a concern that it might create an unnecessary stumbling block or delay for very experienced PMs who don’t need support from GMS in identifying the right type of instrument; they also thought that having a GMS assigned early on might not be feasible in practice, given the frequency of staffing changes.

Proposal Creation | Recommendations

General to proposal creation process

Provide a way for people to link proposals and agreements to related documents or pdfs.
Make workplan linkage a first step in proposal creation.
Auto populate fields with workplan data
Explore a persistent header that includes workplan ID and cooperator name
Add messaging that makes it clear to proposal creators that they should discuss the potential project with (a) a GMS and (b) the cooperator before they enter proposal data.
Further explore generating an agreement template from the proposal.

Project overview

Include cooperator name in project overview
Include total amount of funds being requested in project overview
Break project description down into a few form fields that guide people to provide the specific types of information needed to “build” an acceptable project description.
Provide link to help text to explain proposed start date fields
Add data validation on proposed start date field so that start date can’t be within 45 days of today’s date
Consider removing “will this be covered by FFA?” Q, but first explore how it will impact agreement creation and downstream approval flow.
Consider removing “advance allowed” Q, but first explore how it will impact agreement creation and downstream approval flow.

When adding partnership details

Remove “new cooperator” question

When creating commitments

Considering allowing the ability to link to multiple workplans.
Consider allowing the ability to add more than one job code.
Make the job code field(s) a combo box to help people quickly find the correct job code

When setting up the agreement team

Provide the ability to add more people to the “team”(multiple budget contacts, multiple grants management specialists, as well as “other” folks you want to stay in the loop)
Include an explanation on what “budget approver” means to clarify that it is the person in the region where the agreement is being done, not where the work will be carried out.

Speculative features

When someone submits a scope of work it is sent to a GMS to advise on agreement type and also automatically creates an Outlook meeting for the PM and GMS to discuss.

Dashboard

Do our refinements to the dashboard better support users’ ability to focus and act on information that is relevant to them?

Yes! While a few folks were a bit disoriented upon arriving at the dashboard, which looks significantly different than the current system, they quickly recovered and correctly interpreted what the page was for. In appendix b, we list Program Manager needs identified in the last round of research that are now met in the new design.

What people loved

People loved being able to see so much information about their agreements at a glance. In particular, people were excited to see a proposal or agreement’s status, where something was in the approval process, and which steps were done vs. in progress. People also liked the addition of information on the agreement amount, remaining funds, and when an agreement would expire.

“If it was in NRM like this, that would be very helpful. When someone comes in the first thing they’re trying to figure out is what’s going on with all these agreements. To see that would be very helpful…coming from the perspective of someone who hasn’t done this… someone can just log in and see statuses, I think that's a good change.” - Goldfinch

“I really like that it shows remaining funds there. That makes it so nice and easy not to have to go in and search and the start and end date. That's another thing I'm often checking--what's the exact end date.” - Sparrow

Filters

People liked the filters on the dashboard, but there’s room for improvement. The ability to filter by status was particularly exciting to some. A few people suggested that a filter for whether an agreement had an alert would be helpful. The way we labeled our start and end date filters wasn’t clear.

“Start of the agreement or the actual work? Probably the agreement is what I would want. Sometimes we do an agreement but don’t start work, but we keep track of that... I don’t really understand the ‘start between.’” - Warbler

“This is a lot more familiar way to filter. The way you filter in NRM is counterintuitive.” - Loon

Alerts

The alert icon was easily recognizable to folks as something they’d want to act on. People loved being able to see alerts on invoices are due and commented that reports due is also something they’d expect to see. Several people felt that the alert on whether the funds were low would be helpful, but we learned that whether a “low funds” alert was warranted depends on a few factors including the expected burn rate.

“When does low funds trigger? Might need to be a customizable trigger because it depends on what is being spent and how fast. My two employees' invoices are between 2-3k/month. Because a mod takes awhile to get through, I’d want to know at least 3 months ahead of time. 2 months at 3k, the trigger would be at 6k. Maybe that can be set by the user. “ - Loon

Several people also commented that the “create proposal” button was clear and well placed.

“I like the create proposal button, that's easy to see and find. That's great.” - Sparrow

What was missing

Folks wanted to be able to see a link to the official copy of the agreement that was sent to the partner from their dashboard as well as a way to attach reports.

One person wondered how they could “pay” an invoice from the dashboard. While our mocks included alerts for “needs invoice review” that are intended to take folks to where they would review and accept invoices, the word “review” did not resonate--renaming this alert to “pay invoice” may bring it more in line with how PMs think of their work.

What was confusing

There were some things we included on the dashboard that did not make sense to people. “Manage team” was poorly understood and “Watch” was a complete miss.

Watching

“Watching” initially stumped most of our participants when we asked them what they thought it was for. Eventually half of the folks we asked correctly described that the “watch” feature was a way they could hide or bring records to the forefront of their attention:

Watching - not sure what that is. Maybe if you set it to not watching, it’s going along and you don’t need to have it at forefront of dashboard but if you set it to watching, the ones will show up at the top” - Loon

What’s the watching drop down? I don’t know? Something you’re following, you could deselect and not follow but be able to search for. - Cardinal

“maybe that makes them go away, and that would be something I would not be watching. - Sparrow

One participant described wanting to be able to do what we’d intended the “watch” feature to provide, but they did not make the connection between their desire and the “watch” feature on the page. The word watch is clearly not communicating what we want it to; our Goldfinch participant described it as “ominous,” adding, “who’s watching what?.”

Manage Team

Similarly, the term “manage” in the “manage team” menu did not resonate with the program managers we talked to. Several described that they personally don’t feel that it was their role to “manage” a team or “assign” tasks or team members, but rather they’d appreciate an ability to alert people and let them know when there’s something for them to do or pay attention to.

"It looks like its’ from the group lead assigning. I’m more of the customer. I don’t manage the team, I’m sort of part of that team but not really.” - Chickadee

“I see there’s a manage team button. Maybe the way someone else on the forest could add me, and that would be really nice... What does “assign tasks” mean? … I don’t know, as a PM I don't assign too many tasks. I'm not sure. I work with someone who’s a backup on agreements, and this would be a way to say “take a look at this while I’m gone” - Sparrow

“I’m part of a directorate of other PMs who manage agreements on behalf of my region or forests. [to be able to] access who is on tap for what, from a management perspective that would be helpful but also backfilling, transition management. Anything that can help memorialize and leave breadcrumbs would be helpful.” - Cardinal

People also questioned who was on the “team”in this context.

“I would question, what is the team? Assign tasks to others? Manage team, view team? Are you referring to the cooperator or who is the team? Lots of times it is just one person’s agreement.” - Warbler

“I’m not sure if that refers to the team that is the managers of this agreement or the team / contractor.” - Loon

Dashboard | Recommendations

Refine “manage team” assignment feature and copy; explore alternatives to “manage” and “assign” language
Provide help desk that explains who is on the “team”
Better explain start and end date filters
Explore letting people filter by alert
Come up for a different way to label and represent “watching” functionality
Change invoice alert language to “pay”
Explore customization around alert parameters

Appendix: Design Hypotheses

H1: We believe that an easy to use interface for creating new proposals electronically will reduce data entry errors and make it easier for Program Managers to submit complete, accurate, valid information needed to execute a new agreement. We’ll know we’re right when we observe usability test participants complete their tasks without errors.

H2: We believe that giving people an individualized view into grants and agreements that are relevant to them will make it easier and faster for them to address issues that are waiting on their action. We’ll know we’re right when participants in usability tests (1) are able to get to their next tasks directly without pogo sticking between different pages and/or views and (2) can accurately describe the things that need their attention, what kind of attention is needed and can quickly and correctly identify their next step in performing the needed action.

H3: We believe that involving grants management specialists earlier in the proposal approval process will help prevent inappropriately scoped projects and speed time to award. We’ll know we’re right when usability test participants react positively to the proposed change.

Appendix B: Program Manager dashboard needs statements

Need statement	Tested	Met
As a G&A user, I need to see at a glance whether a proposal, grant, or agreement that I’m assigned to requires action by me or by someone else so that I can either complete the action or raise it to someone else’s attention	✔️	✔️
As a G&A user, I need to see at a glance, which of my grants or agreements are nearing a milestone so that I can ensure reports, invoices, and closeouts are handled on time.	✔️	✔️
As a G&A user, I need to be able to easily review and approve final invoices and reports in order to ensure a timely closeout process.	X	--
As a Program Manager / Budget Approver / Grant Technician / Grants Specialist, I need direct paths to complete actions specific to my role on a particular proposal, grant, or agreement when my action is required to move it forward.	✔️	✔️
As a G&A user, I need to be able to see proposals, grants, and agreements that I’m not assigned to but that I’m interested in so that I can cover for my colleagues when they’re gone, and make sure things don’t fall through the cracks.	✔️	X
As a G&A user, I need to be able to assign a G&A user as needed (e.g. when I’m on leave or on assignment) so that I can ensure my work is covered (e.g. invoices and budgets are approved) while I am out.	✔️	X
As a G&A user, I need to be able to see a proposal, grant, or agreement’s status, or where it’s at in the process, so that I can decide whether and how to act.	✔️	✔️
As a G&A user, I need to see proposals, grants, and agreements that are created by me so that I can focus my attention and make sure they move forward.	X	--
As a G&A user, I need to see grants, and agreements that I’m assigned to so that I can focus my attention and make sure they move forward.	✔️	✔️
As a G&A user, I need to be able to distinguish at a glance items that are due soon from those that are overdue so that I can be sure I address what’s most urgent first.	X	--
As a G&A user, I need to be able to filter so that I can see proposals and agreements by various states of pending in a single view.	✔️	✔️
As a Program Manager / Budget Approver / Grant Technician / Grants Specialist, I need tasks specific to my role on a particular proposal or agreement to be written in language that I understand.	✔️	?
As a G&A user, I need to be able to indicate which proposals, grants, and agreements that I’m not assigned to that I’d like to be notified about so that I can cover for my colleagues when they’re gone, and make sure things don’t fall through the cracks.	✔️	X
As a G&A user, I need to be able to modify my selections re: proposals, grants, and agreements that I’m not assigned to but that I’d like to be notified about so that I can remove them from my list when they’re no longer relevant to me.	✔️	X
As a G&A user, I need the information on my region, unit, and forest that’s embedded in agreement numbers to be scannable so that I can quickly find what I need.	X	--

NRM-Grants-Agreements Path Analysis

Home

How we work

Tech

Design

User research