Building Agentic Systems for Cultural Archives

IBM commissioned this work in partnership with the GRAMMY Museum for a permanent installation about the connections that run through recorded music. The brief was deceptively simple: let a visitor pick a piece of music information — a track, an artist, a genre — and find their way to related content.

There are several reasonable ways to build a system like that, and the obvious answer was a recommender. Recommenders are now the default response to “what relates to this?” They power the next song, the next film, the next product, and almost every platform large enough to predict what someone might want next. For a music archive, that logic seemed useful at first, until the system began to show us the difference between proximity and meaning.

The Limits of Latent Space

The first build represented music as vectors in latent space. We embedded ten-second audio chunks through both CLAP and MERT to capture timbral and emotional character. We transcribed lyrics with Whisper and re-embedded the transcripts with a sentence transformer. Each artist’s biography, era, and origin went through the same sentence transformer. Then we ran KNN across the resulting latent spaces and let similarity build a graph.

From Retrieval to Interpretation

We stopped asking the system what is nearest and started asking how they are related. That second question is structurally different. It requires reading several sources, weighing them against each other, writing a paragraph, and being prepared to be wrong about it in public. None of those is operations a vector index performs. They are operations a person performs, and the closest software analogue we have to a person performing them today is an agent.

The problem also changed shape. It stopped being a Cartesian problem: vectors, distances, KNN. It became a data-aggregation problem. The task was to populate a graph from text by gathering verifiable facts from named online sources, then turning those facts into edges. The graph stopped being learned and became authored. Every edge carries a label, a direction, and a weight put there by something we can interrogate.

A Small Editorial System

The system that replaced the recommender looks like an editorial department rather than a model. Five agents, each running on IBM watsonx against the Granite 4H Small family, sit atop a Neo4j graph. The graph is the archive that the editorial team works from.

The choice of model was a real constraint. Granite 4H Small is a small, on-premise-friendly model, far below the generalist frontier in raw capability. That constraint pushed the design toward small, isolated, testable functions; at this size, reading, writing, evaluating, and rewriting needed tighter prompts and stricter output schemas. The authored graph also had to carry much of the model's reasoning. Most agentic write-ups quietly assume frontier-scale capability; this one does not.

The team has a particular shape: there is no orchestrator agent at the top of the system delegating to sub-agents. The five agents are isolated from each other. Each has its own context, tools, prompt, and output schema. They are arranged on a pipeline rather than wired into a hierarchy. The orchestration is the pipeline; the intelligence stays local to the role, closer to a small editorial process than to a swarm.

The research agent goes out and reads. It queries a set of trusted music data sources through tool calls, then sanitises the responses. It returns atomic research facts tagged by category: biography, history, style, influence, and impact. This is what tells a visitor where an artist made their name, or how a genre took shape in a particular decade.

The Audit Loop

A public cultural system has a low tolerance for confident drift. The evaluate and refine agents handle that work before a paragraph reaches the audience.

The evaluate agent reads what the describe agent wrote and scores it against the research log. It uses five rubrics: factuality, style, specificity, context use, and grounding. Each score sits on a one-to-five scale, with reasoning recorded per criterion. It does not rewrite; its only job is to flag what is wrong and explain why. The evaluator is calibrated in our eval suite against a golden-standard set of human-written samples. Its rubric is anchored to a curator’s judgement rather than the model’s own self-assessment.

The refine agent takes the full description and evaluation, then rewrites. It is not trying to improve the prose; its brief is narrower: correct what was flagged, strip what exceeds the evidence. It records its own reasoning as a diff, so the changes are traceable back to specific evaluation criteria.

Handing the Interpretation Back

The four agents described so far operate outside the visitor loop. They populate the graph, write the descriptions, and audit them. The report agent is the exception: it runs at the table, in real time, and is the only one that responds to a specific visitor’s behaviour. It reads the traversal as a behavioural signature, nodes visited, order, dwell category, and graph paths between interests, and returns a title, a short paragraph, and one onward recommendation.

Closing the experience this way makes the session history visible. The visitor leaves the table with a short account of the path they made through the graph, rather than another generic recommendation.

When the System Becomes the Work

For most of the last few years, agentic tools have sat on the studio’s side of the deliverable. They helped texture a render, allowed creative teams to quickly draft prototypes, and tested our code. This project sits on the other side. The deliverable is the agentic system.

That changes the design work. The client is not only buying output; they are buying behaviour over time. Roles, prompts, schemas, review states, and failure visibility become part of the experience layer. The system has to keep working when the people who designed it are no longer in the room.

For cultural institutions, brands, and platforms, capability alone will not be enough. Agents need roles, restraint, and an observable means of correction. That is where the work starts to look less like model selection and more like editorial design.