Resource Automation + AI Generative AI

Consult Evaluation: Scottish Government’s Non-surgical cosmetic procedures consultation

An evaluation report assessing the performance of “Consult,” an AI-powered tool developed by the UK government’s Incubator for AI (i.AI), to analyse public consultation responses for the Scottish Government.

Organization: Government of Scotland Department for Science, Innovation and Technology (DSIT)

Authors: Lucy Makinson, James Lowe, Alex Moore, Michael Pryse-Davies, Harry Banton, Katie French, Charlotte Ryall, Nina Menezes, Rachael Robinson, Jude Webb, Rishav Punatar

Published Year: 2024

View Source

Download Resource

This report evaluates how accurately Consult generated and mapped themes from responses to a live Scottish Government consultation on non-surgical cosmetic procedures.

Using quantitative metrics (including a multi-label F1 score of 0.76) and user research, the evaluation found that Consult’s performance closely matched human reviewers and had negligible impact on overall theme rankings, while significantly reducing analysis time (median review time of 23 seconds per response). The report also identifies areas for improvement, including refining the theme sign-off process, better detecting missing themes, and strengthening reviewer trust while maintaining a “human-in-the-loop” model.