Author: Beatrice Grillo

A team of scientists asked citizens to evaluate social impact and select which research to support. Here’s what they found.

 

Diletta Di Marco, PhD Student in Management Engineering – Innovation and Public Policy 

Science strives to improve the conditions of humanity and nature. But it is not always clear how to identify the research that serves the most pressing needs. For a long time, the direction of science has been chosen by professional scientists alone, through peer reviews, but new initiatives of participated democracy are trying to second the desire of citizens to take an active role in important decisions about science. For example, a Danish local government has asked citizens to choose which medical research projects should be funded by voting online.[1] Also, the Canadian Fathom Fund has chosen to top up funding to scientists that display their project on crowdfunding platforms and collect at least 25% of their budgeted costs online.[2] 

In a world facing unprecedented social, environmental, and economic challenges, the main idea of these initiatives is to involve those most affected by the problems and their consequences – the citizens themselves.

While scientists, research organizations, and research funders are experimenting new ways of actively collaborating with citizens, one concern is that what constitutes a high social impact is problematic and subjective. Moreover, the mechanisms used to actively engage citizens in the agenda-setting process can create biases or grant undue influence to wealthy or powerful groups.

For all these reasons, assessing the impact of research is an exciting area for professional scientists, funding agencies and policymakers, who are keen to identify new criteria for judging the sustainability and value of research, in addition to traditional ones which are more centred around prerequisites like age, gender, previous experiences in research, and previous project experience in the same area of research.

In an attempt to investigate this important but under-explored area, a research team of our School of Management has studied how the public evaluates social impact and choses to grant or deny support to scientific research. The team consists of Chiara Franzoni and Diletta Di Marco from Politecnico di Milano, in collaboration with Henry Sauermann from ESMT Berlin.

The team selected four real research proposals that were actively raising funds on the platform Experiment.com. The projects were in very different domains, ranging from environmental studies on the diffusion of otters in Florida, to social studies on sexual orientation and pay-gaps, to curing Alzheimer’s disease, and Covid-19. They recruited more than 2,300 citizens on Amazon Mechanical Turk and asked for their assessment of one of the four projects in terms of the three criteria normally used in research evaluations: i) social impact, ii) scientific merit, and iii) team qualifications.
They then asked the citizens whether or not they had a direct interest or experience in the problem that the research was trying to solve (e.g. a family member affected by Alzheimer’s disease when evaluating a project studying a cure for Alzheimer’s), and finally elicited the citizens’ opinions on whether or not the project should be funded. They did so under two different voting mechanisms: i) as a simple free-of-charge recommendation to fund or not to fund the project (costless vote) and ii) as a small direct donation to the project (costly vote), which the evaluators could do by choosing not to cash in a $1 bonus given by the team. At the end of the day, the team then devoted the donated bonuses to real research projects.
They later analysed the responses with statistical and econometric modelling and with qualitative coding of the textual responses.

Their analyses showed three key results:

1. Firstly, citizens placed a strong emphasis on social impact. They were more likely to support a project if they assessed social impact to be high, even if they assessed scientific merit or team qualifications to be low. A complementary analysis of opinions provided in the form of open-ended responses corroborated this view. Citizens tended to focus on the perceived importance of the problem (e.g. size of the affected population, problem severity) and paid less attention to the project’s ability to solve the problem.
2. Secondly, the voting system adopted substantially affected the composition of those who voted. Costly voting shifted the crowd’s composition towards people with higher levels of education and income. This suggests that mechanisms that impose even a small personal cost trade off the intended benefits of inclusion and representativeness when involving citizens.
   
3. Thirdly, citizens who had a personal interest in the problem addressed by the project were more likely to vote in favour of the project, irrespective of using a costless or costly voting mechanism. However, they did not seem to overestimate the project’s social impact expectations. This suggests that crowdsourcing may give more power to interest groups and members of the public with personal interests in the research. At the same time, even citizens with a personal interest in the project seemed to be able to provide unbiased assessments of social impact if asked to do so independently from expressing their support for the project itself.

The findings of this broad research project contribute to advancing the academic debate in different areas, like the management of online communities (by shedding light on the link between voting mechanisms and self-selection and the literature that compares crowd and expert contributions with science funding).
More importantly, they have an immediate practical use for policy makers, funding agencies and interest groups that strive to promote participated democracy.

Considering that traditional research grant mechanisms and review mechanisms focus on things that could go wrong and pay too little attention to potential gains, these results suggest that citizens’ evaluations of social impact are not necessarily “better“, but they may provide a different and potentially complementary perspective.

 

[1] https://www.sdu.dk/da/forskning/forskningsformidling/citizenscience/afviklede+cs-projekter/et+sundere+syddanmark Accessed November 15, 2021.

[2] https://fathom.fund/ Accessed November 15, 2021.