What is one thing the country’s largest private foundation has in common with nearly every other foundation in the U.S.? The need for feedback to ensure it is implementing the smartest version of its strategy.
In fall 2022 the Bill and Melinda Gates Foundation pivoted its education strategy to make math its top priority, with the goal of ensuring that all students, particularly Black, Latino, and students from low-income backgrounds, graduate from high school equipped with the necessary math skills for success in higher education, the workforce, and life. In making this shift, the Foundation took listening to heart.
Yesterday, CEP’s YouthTruth Initiative released a new report, Making Sense of Learning Math: Insights from the Student Experience. This report is the culmination of a 15-month project, the Math Learning and Identity Project, funded by the Gates Foundation. The insights from nearly 90,000 high school students shared in this report, detailed below, bring much-needed first-person perspectives to bear in considering how public education — and those that fund in this area and related areas — must grapple with mathematics teaching and learning in a changing economy and society.
In addition to the specific insights from students about math learning and identity, the process behind generating these insights holds important considerations for philanthropy about feedback and listening practices that apply much more broadly, including:
- How can perceptual feedback from the people funders seek to serve inform decision-making for strategy improvement over time?
- How does listening play out in the funder-grantee relationship and how can it strengthen funder-grantee collaborations?
- How might AI enhance philanthropy’s ability to listen more rapidly to those it seeks to serve, and what cautions should we keep in mind?
Feedback for Strategy Improvement
Through the Math Learning and Identity Project, CEP’s YouthTruth Initiative partnered with the Gates Foundation to bring student perceptions about learning math front and center as the foundation iterated on its new K-12 education strategy. The project stemmed from the foundation’s desire to learn from and incorporate direct feedback from those they seek to serve into strategy development — and YouthTruth’s expertise in gathering and making meaning of such student feedback. In particular, as part of its measurement, learning, and evaluation plan, the Foundation was interested in understanding students’ math experiences in terms of their sense of belonging, care and support from teachers, and relevance to their interests and real-world contexts.
As part of the project, YouthTruth presented summary data and statistical analysis showing both overall student feedback and statistically significant differences in student experiences by demographic group to the Gates K-12 team at a staff retreat. Program officer Adam Goldfarb reflected that “some of the data confirmed our working hypotheses.” He cited seeing a positive relationship between student experiences in math and when students took Algebra I as an example that validated the team’s expectations. “We value being able to incorporate student perspectives into our discussions and will continue to do so as a result of this work.” The student feedback gathered in this project, in combination with inputs from other projects and grantees “will help ensure that the K-12 team, leadership, and partners have access to relevant information about the strategy and its context to inform decision-making for strategy improvement over time,” said Goldfarb.
The Art of Listening in the Grantee-Funder Relationship
Like any strong relationship, the funder-grantee relationship rests on a foundation of good listening and open communication. This funder-grantee relationship required listening — in all directions — to set the project up for success.
In conceptualizing the project and co-designing a set of student survey questions, each party listened deeply to the other’s context and perspective. YouthTruth had to understand the baseline data that the Gates Foundation brought to the table from pilot surveys, the goals the foundation sought to accomplish through this project, and the constructs they wanted to measure to support their research questions. The foundation likewise listened intently to YouthTruth’s input on survey design, constructing strong and age-appropriate survey items, and creating wrap-around elements of the project to enrich our collective understanding of the data through structured workshops with students.
Throughout the project, we maintained open lines of communication in monthly check-ins between our Gates Foundation program officer and YouthTruth project leads to share updates, any changes from the initial plan, and lessons learned. As the project progressed, the Foundation expanded the listening tent beyond the program officer-grantee relationship and invited YouthTruth to share insights from the project with their broader team.
Finally, as the project evolved with an opportunity to participate in a cohort of grantees exploring AI tools to advance math education and equitable outcomes, we enjoyed the privilege of contributing to, listening to, and learning from shared challenges and new ideas — both as relayed by our program officer in monthly check-ins and in a learning cohort meeting.
Employing AI in Feedback Practice
Artificial intelligence (AI) has the potential to be transformative in many arenas, including as a listening tool. In fall 2023 the Gates Foundation launched a pilot AI initiative, providing funds to a small group of existing partners, including YouthTruth, to test new AI ideas in support of equitable K-12 mathematics outcomes for students who are Black, Latino, and from low-income backgrounds. In this project, YouthTruth piloted the use of generative AI research tools to derive insights effectively and efficiently from large datasets of students’ open-ended survey comments.
After investigating a dozen AI research tools, YouthTruth ultimately generated summaries highlighting students’ experiences using a natural language processing tool. This AI tool captured insights from two open-ended survey responses from 66,484 students in a dramatically more efficient manner than we had ever experienced with previous methods of qualitative analysis. These insights were then explored alongside the quantitative student survey data and workshop insights gathered in the Math Learning and Identity Project and inform the key findings in the Making Sense of Learning Math: Insights from the Student Experience report.
Employing an AI research tool to efficiently analyze perceptual feedback from a very large student survey dataset was an eye-opening experience for our team at YouthTruth. However, the many questions we encountered along the way remind us that we must remain alert and not simply awestruck by AI’s promise. There is a preponderance of AI tools available, but which are the right fit for the intended purpose? How variable is the quality? Which tools operate in an open versus gated environment, and what privacy considerations are at play? What biases and equity concerns must we navigate? Upon selecting a suitable tool, what does it take to get users up to speed and to structure our data and inquiry appropriately?
Ultimately, we were reminded in this process that for our use case, good AI is, for now, expensive. And while the AI research tool we used dramatically increased the efficiency of our qualitative analysis, there was a learning curve that required an up-front investment. Funders can help lower the barrier to entry and ensure that quality AI tools are accessible to a broad range of nonprofits and that learning is shared.
Key Findings from the Math Learning and Identity Project
In addition to informing the Gates Foundation’s strategy development and iteration, the listening practices described above resulted in YouthTruth’s new report, Making Sense of Learning Math: Insights from the Student Experience, which publicly shares insights from nearly 90,000 high school students about their everyday experiences learning math. Key findings include:
1. School Math Versus Real Math: Many high school students perceive “school math” or “education math” as lacking inherent value and as disconnected from their practical needs. Students express a strong desire to acquire “real” math skills that will empower them to pursue the future they envision for themselves.
2. Determined Math Learners: Students who have strong math identities and positive math learning experiences describe their relationships with their teachers as an important source of their determination to learn math.
3. (Un)interesting Math Problems: Students explain that their desire to learn math is inhibited by uninteresting work that dampens their intrinsic motivation to acquire math skills.
Math is a subject that sparks seemingly endless debates in universities and think tanks, among leaders of industry, and in government, popularly referred to as the “math wars.” But these debates often gloss over how students themselves experience math. We hope these insights directly from students — and the advice students share in the report — inspire those with the power to shape math education to create a more meaningful, engaging, and relevant experience that empowers all students to succeed.
Beyond mathematics per se, these findings invite us to consider the purpose of public education and how those investing in education can help ensure that what we are teaching remains relevant to real life after high school in a changing society and economy. They likewise implore us to remember and invest in the power of relationships and the balance of intrinsic versus extrinsic motivation in the K-12 education experience.
Zooming out from the education ecosystem altogether, the listening practices underpinning this project and these findings are evergreen and apply well beyond those working and funding in education.
Perceptual feedback from those we seek to help is an integral part of developing, iterating, and improving philanthropic strategy across many domains. Seeking out the perspectives of, and truly listening to, the people funders seek to serve helps check assumptions, make course corrections in a timely manner, and understand the way strategies are playing out in a much more nuanced way. Listening is a key ingredient in any strong funder-grantee relationship. Making space for sufficiently open and frequent communication, honoring the unique contributions and expertise of each party, and creating a relationship that allows for honesty and transparency leads to deeper collaboration and impact. And AI has a place in feedback practice. When used well, and in the right contexts, AI tools can complement and enhance other forms of listening to help funders better understand the needs and aspirations of those they seek to serve.
Jen Vorse Wilka is executive director of YouthTruth, an initiaitve of CEP. Find her on LinkedIn.
