Digital Policy: Impact Measurement Is Often Impossible
ResearchZEW and Agora Point Out Shortcomings in Evaluation of Digital Programmes
It is often impossible to assess with certainty whether the billions in tax revenue spent on the federal government’s digitalisation are used effectively. Although there are some signs of target- and performance-oriented budget management in key digital programmes, the approach has not yet been consistently implemented. These are the findings of a ZEW study commissioned by Agora Digitale Transformation. Ten programmes with a total budget of around 6.6 billion euros were examined. On average, they scored 3.1 out of a possible five points in terms of the prerequisites for target- and performance-oriented management. Significant weaknesses are particularly evident in the areas of measurable success indicators, data availability and performance evaluation.
“Successful digital policy should not be judged by how much money is spent, but by the impact achieved. While many federal programmes set out transparent objectives, they often lack the tools to measure reliably whether these objectives are being attained,” explains Professor Friedrich Heinemann, head of the ZEW Research Unit “Corporation Taxation and Public Finance”.
“The objectives should be set out in the budget itself. After all, if no clear targets are set when the funds are allocated, this will also undermine the focus on impact during implementation,” emphasises Stefan Heumann, Managing Director of Agora Digitale Transformation.
Programme size is not the decisive factor
Impact monitoring works best in broadband roll-out (3.9 points), the Sovereign Tech Fund (3.8 points) and the digitalisation programme for the public health system (3.6 points). These three programmes are united by a clear logic of impact: They set out precise targets, underpin these with quantifiable indicators and use insights from ongoing implementation to make adjustments.
For example, the broadband roll-out programme, which aims to achieve nationwide fibre-optic coverage by 2030, has a clearly measurable target. The Sovereign Tech Fund focuses on the resilience of critical open-source components, while the programme in the public health sector assesses the digital maturity of public health authorities. It is worth noting that the Sovereign Tech Fund, with a budget of 17 million euros, is by far the smallest programme examined. Impact depends less on the size of the budget than on how well it is managed.
Measurable targets and better data are often lacking
Success is hardest to measure in the Smart Cities pilot project, the IPCEI Cloud initiative and the funding scheme for the gaming industry. The causes range from a lack of targets and a confusing funding and governance structure to inadequate programme evaluations.
“A common pattern emerges across all ten programmes: They often lack interfaces to data management systems and a suitable digital data basis for causally determining actual impacts,” says Heinemann.
About the methodology
The researchers assess the digital programmes according to six management categories: the precision and performance-orientation of the targets; the use of SMART indicators; data availability and digital infrastructure; the culture of evaluation; feedback and learning processes; and governance and accountability. The assessment was carried out on a five-point scale. The study explicitly does not examine how successful a programme is at achieving its objectives, but rather whether the conditions for target- and performance-oriented management are in place.
