Forms.gov
In early 2022 I worked with the Office of Evaluation Sciences to bring equitable access to federal benefits and services for all Americans by designing an integrated testing service for forms.gov.
federal forms are an opportunity to improve the american people’s access to federal benefits and services
Millions of Americans are unable to benefit from government programs that help pay for food, housing, health care, and other basic living expenses simply because they cannot complete the application process.
Consider the launch of HealthCare.gov in October 2013. The administration set a target of an 80% success rate for people using the site to sign up for health insurance.
That 80 percent goal, high as it was for a launch of that size, meant that tens of thousands of Americans were still unable to enroll successfully.
Gaps like this are not limited to health insurance. When federal forms are prohibitively complex and burdensome, entire populations cannot get help with affordable housing, financial aid, grants and loans, military programs and benefits, retirement, social security, and unemployment.
Meet the Office of Evaluation sciences
OES is a government agency made of behavioral scientists that designs and conducts evaluations of existing programs and evidence-based program changes across the federal government.
Their projects range widely in scope, from understanding the impact of language on levels of enrollment in low-income medicare programs to a pilot program focused on increasing access to family planning services in Nigeria.
In 2021, the General Services Administration (GSA) identified federal forms as an opportunity to improve the American people’s access to federal benefits and services.
OES and I worked together to explore the feasibility of integrating coordinating testing into Form.io - the software platform for the government’s form digitization process.
How work began
The start of my work focused intensively on understanding the perspectives of the behavioral scientists at OES. I researched what they did, what they needed, and what they valued.
Although my background is in anthropology - a behavioral science - and my career centers on the application of research, I did not make any assumptions about what or how OES researchers think. Instead, I used a series of workshops, interviews, and collaborative exercises to inform an ethnographic analysis of the behavioral scientists at OES as individuals and as a group.
The outcome was a collection of narratives and personas. These established a shared understanding among the team about the people for whom we were solving problems.
How do they run experiments?
Scientific validity is a core requirement for all of OES’s experiments.
While hunches may inspire and even guide research, the final determinations have to be based on data. OES researchers need to justify conclusions in ways that are quantifiable and their results must be repeatable for their recommendations to be scientifically valid.
test parameters
An OES researcher needs to set basic parameters defining how an experiment is conducted so that it balances research priorities with external constraints like budget and time.
variants
OES researchers create multiple versions of a given form to gather quantifiable evidence showing why one form design is more likely than another to produce the intended outcome for federal agencies.
randomization
Randomization involves using chance to assign individuals with some probability to one of a predetermined number of groups. OES researchers use randomization to ensure that the process for assigning participants to variants is free of unconscious bias.
distribution
Just because a sample was randomly generated does not mean that it is a representative sample. Clusters and outliers can still appear in random sets and skew results.
OES researchers need ways to specify the distribution within a sample and ensure that the distribution remains constant even when the sample size changes.
data analysis
In order for a researcher evaluate the outcome of an experiment, they must be able to analyze the data the experiment produced. OES seeks to run experiments at a scale that makes manual analysis impractical. The solution is to get data out of the form delivery platform and into analysis tools that give the researchers the ability to work with sample sizes numbering in the millions.
What is important to a researcher?
Up to this point in the project, the focus of my effort had been on discovery, observation, and recording. I knew what was required for an OES researcher to run a test that produced scientifically valid results. What I did not know was what was most important for research and what was most important for determining feasibility.
I had all the ingredients but still needed to find the right recipe.
I established the foundation with a series of workshops and exercises designed to unpack assumptions by having the researchers make difficult choices about features and functionality. The goal of these was not to revisit the scope of the project but rather to inform the design and development timeline by uncovering dependencies and opportunities for improvement.
The experiments vary, but scientific rigor is a constant.
All of this ethnographic work revolved around understanding the OES definition of feasible. The word served as a gateway to understanding the role the agency plays within the federal government and the perspectives of the people who work to support that role.
The key finding from my research was that the experiments the agency runs vary in terms of format, parameters, and goals, but scientific rigor is a constant. If conclusions are not quantifiable or results are not repeatable, then the researchers’ conclusions are not scientifically valid.
The Outcome
I took these and other insights from discovery and landscape analysis and synthesized them into a set of design proposals. The designs were distinguished by the granularity of control they provided over experiments and how they defined the experiment management process.
At one end of the spectrum lived designs that prioritized autonomy for researchers by delivering rich, digital interfaces. On the other end lived designs that took more of a service-oriented approach.
User Experience Design
The UxD solutions prioritized autonomy for the researchers. They varied in their combinations of features, but all adopted a digital interface as the primary mediator between the researcher and the functionality required for managing an experiment.
Service Design
The solutions that were oriented toward service design took a different approach. For these, the management of an experiment was mediated entirely through an outside contractor.
testing federal forms at scale is feasible
1. We can design digital forms to improve equitable access to federal benefits.
2. We can integrate scientifically valid testing into form digitization.
3. We can minimize the impact the digitization of forms has on the accuracy, efficiency, and burden of information collection.
4. We can do all of these things in a way that is scalable, repeatable, and low-cost.
All people deserve equitable access to government programs, services, and benefits. When the information collection process places too high of a burden on people, entire populations can go without healthcare, housing, or even the ability to vote.
My work with OES addressed these burdens on a national level. I delivered a set of solutions based on basic principles of human-centered research and tailored to the particular needs of behavioral scientists. They all demonstrated how testing of digital federal forms is feasible, practical, and in line with the requirements of behavioral scientists who value scientific rigor.