5 Oct – 11 Oct 2026

When good AI answers need to hold up in practice

This week points to a shared question: How do we know AI helps with the task we actually need to perform? We look at municipal expectations, staff learning, supplier dependencies and agents' access to data. The week's concept is AI evaluation, with an illustrative municipal example.

Edition · Selection as of 9 October 2026Permanent link to this edition

You are reading an earlier edition. Dates and information reflect that edition. Read the latest

Selected on Friday 9 October. The news is from this week; the knowledge section includes both new methods and a clearly dated older NIST source. Surveys, policy working papers and vendor experience are distinguished from measured municipal impact. Suggestions for your practice are our editorial judgement.

The most important developments

The week’s news

Local Government Denmark · survey · 6 Oct

Municipal AI expectations are not measured benefits

KL's Technology Radar draws on 586 responses from managers and staff in June. AI-assisted casework systems lead expectations of future value at 24%; RPA leads already perceived value at 20%. These are respondents' assessments, not shares of municipalities or documented savings.

For your practice: Choose a specific trial and measure quality, time and costs locally.

Read the original source

Google Research · research report · 7 Oct

Better work with AI does not automatically mean lasting learning

Google Research reports a three-month randomised study involving 133 patent lawyers at 11 firms. AI-assisted writing quality improved; improvement in subsequent unaided tasks varied with experience. The study is Google-supported and concerns patent work, not Danish public administration.

For your practice: Track both quality with AI and staff skills without it.

Read the original source

KL · policy working paper · 8 Oct

Digital choice needs an ability to switch

A Committee of the Regions working paper seeks easier supplier switching under the EU's proposed cloud and AI regulation. KL stresses service continuity. This is policy work, not adopted law.

For your practice: Check data portability and dependencies in AI procurement.

Read the original source

ICO · call for evidence · 8 Oct

ICO asks about AI agents' data-protection risks

The UK data-protection authority is gathering evidence on AI agents until 20 November. Themes include security, transparency and accountability. The exercise informs future UK guidance, not new Danish rules.

For your practice: Map agent access and people's ability to intervene.

Read the original source

Perspectives to take into your work

Beyond the network

NVIDIA · technical methods · 8 Oct

Fewer tools can make a data agent easier to inspect

NVIDIA shares KDD Cup experience with few tools and logs of intermediate steps. Repeated attempts and tuning can increase costs and overfit the benchmark. Competition experience is not evidence of municipal reliability.

For your practice: Test on new tasks and review both answers and workflow.

Read the original source

UK AI Security Institute · evaluation tool · 7 Oct

Look at the workflow behind an agent's answer

AISI's Transect helps inspect long agent logs using categories and links back to specific runs. An AI-based judge can itself make mistakes. The tool supports analysis; it is not a security guarantee.

For your practice: Use logs to inspect failures and unexpected actions. Have people check consequential judgements.

Read the original source

NIST · methods research from February · 17 Feb

Older source: What does the test score actually tell us?

NIST AI 800-3 distinguishes performance on a particular test from estimated performance on similar new tasks. The report studies statistical methods that clarify uncertainty and assumptions. A high benchmark score alone does not promise good local practice.

For your practice: Describe which tasks the test covers and which conclusions it cannot support.

Read the original source

Make room in the calendar

Coming up

19 Oct09:00–15:30

AI Alliancen · KL House · Weidekampsgade 10, Copenhagen

AI Alliancen meets at KL

Network meeting for members and invited guests. No open registration; attendance is agreed with the network. The agenda will follow on the meeting page.

19 Oct14:00–15:00

VIA University College · Online · Microsoft Teams

Get ready to use AI in public administration

Free external public-administration webinar, open to anyone interested. Registration by 19 October requires consent to VIA emails about continuing education, events and professional inspiration. Danish-language event.

27 Oct09:00–10:00

KL · Videncenter Meetup · Online · Microsoft Teams

Municipal collaboration on data and new technology

Requires a Meetup subscription: DKK 2,000 annually for a municipal employee or DKK 8,000 for the municipality, excluding VAT. Staff at subscribing municipalities attend without an additional participant charge. Check KL's terms and registration. Danish-language event.

Times are Danish local time. External events are organised by the named providers. Check their current programme and terms. See the full calendar

The week’s concept / Explained properly

AI evaluation

AI evaluation is a systematic examination of how an AI system performs specified tasks against criteria chosen in advance. It should help us decide where the system can be used and what we still do not know.

A good demonstration shows one successful run. An evaluation also examines routine, difficult and unexpected situations. It therefore starts with the task: Should AI help make letters understandable, find information or use a tool? These are different questions requiring different tests.

A test score concerns the tasks and conditions examined. NIST distinguishes performance on a fixed test set from estimated performance on similar new tasks. The latter requires assumptions and an account of uncertainty. A small difference between two systems is not automatically a sound basis for choosing one.

A language-model evaluation can be described through a set of tasks, a way of solving them and a way of assessing the result; AISI's Inspect uses this structure. For an agent, we should also inspect the workflow: Which information was read, which tools were used and where could a person stop the work?

An AI model can help judge answers, but its judgements may also be wrong. Consequential assessments should be checked by people with relevant expertise. Record the model version, instructions, sources and tools so changes can be examined again. We are evaluating a system in its actual context, not just a model name.

The tasks

Choose examples resembling the work, including situations in which the system should ask for help. Hold back some tasks while improving the solution.

The criteria

Agree what a good answer must satisfy: correct content, clear language and preserved deadlines. Report serious errors separately; an average must not hide them.

The test run

Run the same bounded test on the versions you want to compare. Save inputs, answers and relevant logs. Examine variation and record time and costs.

The assessment

Inspect specific failures and uncertainty. Describe what the test shows, what it does not cover and which tasks still require human oversight.

Illustrative municipal example

Can AI make a citizen letter easier to understand?

An illustrative municipal team wants to trial AI to help staff rewrite language. The system must not make administrative decisions or send letters automatically. The numbers below illustrate a small initial test; they are not results from an actual municipality.

Create 30 entirely fictional letters without personal data. Include short and difficult letters, missing information, amounts, deadlines and text trying to make the model change its task.
Use 20 letters for development and keep 10 new letters out of ongoing tuning. Agree requirements before seeing answers: preserve meaning and do not change or invent amounts or deadlines.
Test the selected versions on the same held-back letters, preferably several times. Record inputs, outputs, model and instruction versions, time and costs. Ask two specialists to assess answers without knowing the system's name.
Compare readability, factual errors and disagreements between assessors. One changed amount is a specific failure to investigate; good answers elsewhere do not make it disappear.
Adjust the solution and then try different new examples. Describe limitations and retain staff review before use. A small test may justify the next investigation, but cannot prove that all future letters will be correct.

The pitfall is turning an average score into a general approval. Examine types of failure, task coverage and uncertainty. Evaluation does not replace a separate assessment of legal basis, data protection, security or accountability.

Sources behind the explanation

The explanation draws on the sources below. The municipal example is our own illustration, not a reported implementation or measured result.

Editorial selection and explanation: AI Alliancen · Prepared with AI assistance and source checks. Public-sector applications are editorial suggestions, not measured effects or legal approval.