impact analysis Archives - Politecnico di Milano School of Management

Conversation with:
Federico Caniato, Full Professor of Supply Chain & Procurement Management at School of Management, Politecnico di Milano
Stefano Magistretti, Assistant Professor of Agile Innovation at School of Management, Politecnico di Milano

Universities are increasingly engaged in demonstrating the impact of their research. What is the impact of research?

The impact of research is crucial not only for the Politecnico di Milano, but for the entire Italian university system, and more in general for universities worldwide. It is not easy to define what research impact is. We can say that the impact of research encompasses all the results, implications, and consequences resulting from scientific research activities aimed at generating knowledge, but they are also expected to provide concrete benefits. In our school, we have defined research impact in three progressive levels of maturity: dissemination, adoption, and benefits. Dissemination is the spread of the results and findings among the relevant stakeholders, adoption is the use of the research results by the stakeholders, and benefits are the consequences of this adoption.

Why is impact so important for research?

Research is often accused of being self-referential, i.e. ‘speaking’ only to members of the academic community without providing a significant contribution to society at large. Instead, research can have a much broader and more significant impact than expected. Therefore, it is crucial to illustrate such impacts to a broader audience, requiring researchers to learn to assess and share the value of their work with multiple stakeholders.

What is the approach to impact assessment in the School of Management?

In 2017, we started a journey in the School of Management to develop a culture of research impact assessment. This journey saw a reflection on the assessment framework, the development of a method, and the collection and analysis of the research impact assessments. We started by combing the literature for impact assessments, interviewing experts, and interacting with our international advisory board to define our framework. The framework comprises the three levels of maturity (i.e. dissemination, adoption, and benefits) and five stakeholder domains (i.e. institutions, enterprises, students and faculty, citizens, and the academic community). The second step was the adoption of the framework. This initially began in 2019 with a set of 16 pilot projects, which then extended to a more extensive set of projects (42 in 2020; 43 in 2021).

The conventional idea of ‘impact’ makes sense in a linear model: changes or discoveries in science and research are expected to cause changes in society, but impact assessment frameworks are usually far more complex, can you tell us why?

The research impact assessment is more complex because the impact is not linear. Some elements impact one stakeholder, causing indirect effects on other stakeholders. For example, research results adopted by public institutions may benefit citizens, or the results disseminated to students may be adopted later, when the students are professionals within companies. Thus, the impact network is intertwined. Seeing the link among the domains and level of maturity, and how an initiative might influence other areas of impact requires a framework that tries to bring everything together. Let’s take an example. If you publish an academic paper, there is diffusion within the academic community, but if you share it in class, there is also an impact on students; if you use it in corporate education, that novel piece of research can become the seed for a potential company project. So from a single action — dissemination of research among the academic community — you might have an impact over multiple stakeholders on different levels.

How much of this impact analysis must be made ex ante, while planning the activity, and how much ex post?

The impact assessment is a helpful tool in every moment of a research project. We saw colleagues adopting it when writing proposals for an EU project or internal research initiative. This is because the impact is both ex-ante and ex-post. The most important thing is to envision potential impact ex-ante, which helps to set the expectations and objective of the project. Ex-post assessment instead aims to measure the results obtained in terms of impact, monitor the results of the planned activities, and demonstrate the actual achievements. Thus, there is not just a single moment for impact analysis; it is always a good to measure it before, during, and after the research initiative.

Is the impact ‘native’ or built over time? Do we need our PhD candidates to be ‘natural-born impacters’ or is it an orientation that can be encouraged and sustained over time?

The impact culture is not native. It is something that PhD candidates and researchers in general should be trained in. Indeed, some impacts are easy to design and achieve, but impacts of a higher level are more challenging and require careful consideration, so it is important to build impact over time. Indeed, it is difficult to gain everything with a single new research programme. As for PhD candidates, it is probably something that we should share with them and encourage them to reflect on. This is something we started at the last AiIG (Associazione Italiana Ingegneria Gestionali) Summer School held by the Politecnico di Bari in September 2021, where we shared the framework with more than 50 Italian PhD candidates and asked them to apply it to their PhD research. The PhD candidates were positively surprised about the unexpected outcomes of this assessment exercise. Disseminating the culture of research impact assessment is something we need to do at every level.

The Politecnico di Milano School of Management has been promoting a culture of assessment and improvement of research impact on institutions, companies, students, professors, citizens and academic communities since 2016

Federico Caniato, Full Professor of Supply Chain and Procurement Management at School of Management, Politecnico di Milano
Stefano Magistretti, Assistant Professor of Innovation and Design Management at School of Management, Politecnico di Milano

The contribution made by universities to society is being called into question more and more often nowadays. They are therefore increasingly asked to measure and demonstrate this contribution, which is often described as an “impact”. The traditional approach consisted in identifying three major missions: research, training and the so-called “third mission”, a broad term encompassing interactions with society as a whole, such as technology transfer, cultural promotion and external communications. However, there is a limitation to this approach, as it risks seeing the three missions as separate activities, each with its own rules and metrics.

The School of Management has been working on this subject since 2016 with a more integrated approach. Rather than viewing the three missions as separate entities, research is seen as an engine able to generate impact on multiple domains, not only on the academic community and students, but on society in general. We promote a culture of research impact assessment and improvement for this reason, in line with our mission:

“to contribute to the collective good through a critical understanding of the opportunities offered by innovation. We accomplish this mission by creating and sharing knowledge through high quality teaching, exceptional research and active engagement with the community”.

When we embarked on this journey, we aimed, first and foremost, to encourage all our colleagues to reflect on the broader impact of their research projects. In the early years, we focused on stimulating critical thinking and encouraging the creation of an impact measurement culture. In the beginning, we did not assume that all projects, from the simplest to the most complex and the shortest to the longest, would have an impact on different areas of our school’s mission. However, this culture of measurement was, and still is to this day, fundamental in assessing and demonstrating the impact on many domains and not only on the most traditional indicators (e.g. number of academic publications and number of publications in newspapers).

We therefore felt the need to develop our own model to guide impact assessment throughout the SoM, which would enable us to pursue the following objectives:

Raising awareness throughout our entire community
Learning to assess the impact of research
Encouraging all our colleagues to plan, conduct and disseminate research aimed at having a measurable impact
Improving the capacity to account for the impact generated
Recognising the results of research conducted by the SoM
Publishing research results both within the SoM and externally

To meet these objectives, we built a model, inspired by the scientific literature, identifying five domains and three levels of maturity of research impact.
Impact is measured in the following five domains:

Institutions
Companies
Students and professors
Citizens
The academic community

The impact on each domain is then measured on a three-level increasing maturity scale:

Communication of research results
Adoption of research results
Benefits obtained through adoption

This model was deliberately designed to be general, so that it could be adapted to the various themes and types of research conducted at the SoM. Precise indicators need to be identified for each domain and maturity level, and they should be quantitative wherever possible (e.g. number of participants at events, number of journal articles published, number of academic conferences organised), thus enabling us to measure and demonstrate impact. The indicators chosen should be consistent with the nature of each individual research project.

The model was tested first of all by a few colleagues who assessed 16 projects according to these dimensions in 2019. This enabled us to evaluate the soundness and validity of the model and identify many useful metrics for the various domains and different maturity levels.

In 2020, we invested in engaging everyone at the SoM in performing this important exercise, thereby broadening involvement and carrying out the Research Impact Assessment for 42 different projects conducted within the SoM, with at least one for each line of research of the SoM.
It was primarily an opportunity for training and reflection on the topic across the school, with sessions organised to give those involved a chance to exchange views and discuss the process.
A booklet was compiled with a summary and the main findings, featuring a wealth and variety of impacts. It will serve at first as a tool for internal communication to raise awareness and gather best practices.

Our work continues. We have already started gathering data for 2021, with a view to updating the information on the 42 projects and expanding participation even further. Our hope is that this exercise will make it increasingly possible not only to measure impact retrospectively, but also to plan the impact of research projects from the very outset. We also hope that this assessment will eventually cover all domains and reach the highest possible level of maturity, in other words, that of real benefits.

Tag: impact analysis

The challenge of pursuing impact in research