1.
The humanities are essential to how humans reason about values, institutions, history, and culture. They are also particularly difficult domains in which to evaluate artificial intelligence.
A language model can possess all the relevant information and still fail at humanities reasoning: for example, it can miss the distinction an argument turns on, treat a contested interpretation as if it were settled, or acknowledge competing values while reasoning as though only one matters. Many humanities constructs are interpretation-dependent, normatively loaded, and value-plural. The reasoning target is therefore not always a single correct answer, but a constrained space of defensible interpretations and arguments. That does not mean “anything goes.” It means that quality must be judged against disciplinary standards while preserving reasonable disagreement.
For expertise-intensive humanities reasoning, evaluator expertise is therefore part of construct validity. But expertise alone is not sufficient. Reliable judgment also depends on domain match, appropriate task design, clearly articulated grounds and provenance, and appropriate treatment of uncertainty and disagreement. Bentham Research is building the human-feedback measurement infrastructure required to turn such judgments into expert-warranted signal for AI evaluation and training.
That infrastructure must be designed around the way humanities expertise is actually exercised. Much expert knowledge becomes visible only in the course of substantive scholarly inquiry. Published scholarship records arguments, interpretations, and conclusions; scholarly interaction reveals how expertise operates when reasoning must be challenged, defended, qualified, or revised. Bentham turns those interactions into structured, validated, and training-worthy reasoning data, while automation translates them into AI-compatible form. Scarce expert attention can thus be concentrated where human judgment is most informative.
Making Bentham intrinsically useful to scholars is part of the same approach. A personalised AI research environment increases effective expert availability by giving scholars a reason to continue using Bentham. Each interaction improves Bentham’s evolving map of scholars, literatures, model capabilities, and model weaknesses, allowing expert attention to be directed towards the questions where it is most valuable — both for scholarship and for improving AI.
The result is a continuing feedback loop between frontier AI systems and expert scholarly communities: the humanities improve AI, and AI supports the humanities. Where this feedback informs safety, alignment, post-training, or consequential model assessment, the quality of the measurement system itself becomes part of what makes the resulting evaluation and training data trustworthy.
2.
Humanities reasoning will not be improved by testing isolated capabilities alone. AI systems must be able to connect interpretation, evidence, concepts, objections, counterarguments, and conclusions across extended chains of reasoning while remaining responsive to the standards of the relevant discipline.
Bentham is therefore interested in evaluations that capture substantive scholarly reasoning rather than isolated tasks. The aim is to understand whether frontier systems can sustain good judgment across the kinds of problems humanities scholars actually work on, and to identify where those chains of reasoning break down.
This requires assessment across several dimensions: factual knowledge, conceptual precision, engagement with relevant literature, treatment of competing interpretations, quality of argument, sensitivity to scope and context, and the handling of reasonable disagreement. Strength on one dimension does not guarantee strength on another, and a failure early in the chain can distort everything that follows.
As models become more capable, the evaluation problem also changes. Weaknesses that were once obvious become less frequent, while subtler failures emerge deeper in the reasoning process. Bentham’s role is to keep the evaluation frontier moving with the models: identifying where expert judgment is still required, where automated signals are becoming reliable, and where new forms of assessment are needed.
Over time, repeated evaluation across different literatures, disciplines, and model generations can produce a progressively richer picture of what high-quality humanities reasoning requires and where frontier systems continue to fall short.
In short, Bentham is building an evaluation layer that connects multiple forms of evidence about model reasoning into increasingly realistic and demanding assessments of AI performance in the humanities. The long-term objective is a verification infrastructure capable of supporting AI systems that can reason more reliably about consequential human affairs — particularly where values, institutions, interpretation, responsibility, and competing human interests are at stake.
3.
In the limit, this emerging infrastructure can radically expand the amount of expert humanities reasoning available both to AI systems and to scholars themselves.
The shape of humanities research will change as a result. Scholars will work in increasingly personalised AI research environments that can engage with large literatures, surface relevant arguments and disagreements, track the development of questions over time, and connect individual inquiry to a much wider scholarly network. A single scholar will be able to investigate more questions, follow more branches of an argument, and engage more deeply with peers than is practical today.
Today, humanities expertise is scarce and fragmented. It is distributed across universities, disciplines, languages, intellectual traditions, and cultures, much of it difficult to discover or access outside the communities in which it was produced. Only a small fraction of this expertise reaches the development and evaluation of frontier AI systems. At the same time, scholars themselves are constrained by attention: there are far more arguments to examine, literatures to follow, and intellectually worthwhile questions to pursue than any individual or research community can address.
AI can change that constraint while keeping the scholar at the centre. The opportunity is to move from scarce, episodic access to expert judgment towards a global scholarly infrastructure in which human expertise can be applied far more effectively. Scholars remain the source of disciplinary judgment, while AI expands the range of material they can interrogate, surfaces related work and points of disagreement, connects scholars, and directs attention towards questions where expert reasoning is most informative.
The global character of that infrastructure is essential. Different scholarly traditions bring different histories, concepts, interpretive frameworks, and accounts of value to the same questions. An AI genuinely aligned with human values should not reduce those differences to a single dominant perspective. It should be able to represent value pluralism faithfully, understand the traditions from which those differences arise, distinguish reasonable disagreement from error, and treat human plurality itself as a source of intellectual and moral value.
To get there, the infrastructure connecting scholars and AI systems will need to be built for continuous interaction rather than one-off annotation tasks. Bentham is building towards that future: a global scholarly network in which diverse intellectual traditions help shape increasingly capable and genuinely pluralistic AI systems, and those systems in turn expand what humanities scholars around the world are able to achieve.
Our aim is to bring scholars’ expertise into model development in a form that preserves what makes it valuable — including its context, plurality, and grounds of reasonable disagreement — and to use the resulting advances to expand what scholars themselves can investigate.
4.
As AI systems become more capable and participate in increasingly consequential human affairs, they will encounter problems that cannot be solved by factual knowledge or technical capability alone. Judgments about harm, responsibility, authority, rights, obligations, and competing human interests depend on forms of reasoning that the humanities have examined for centuries.
Humanities reasoning thus has a material role in improving AI alignment and safety. An aligned system must be able to reason when values conflict, distinguish principled disagreement from error, understand how conclusions depend on assumptions and context, and navigate competing moral and intellectual traditions when making judgments. The quality of the human feedback used to evaluate and improve these capabilities consequently matters. For Bentham, expert oversight, domain match, multidimensional evaluation, provenance, and explicit treatment of uncertainty and disagreement are part of the evidential standard for using human judgment to shape model behaviour in these domains.
Bentham’s contribution is to build the scholarly and technical infrastructure through which these forms of human judgment can inform frontier AI. The ambition is to enable AI systems to engage more faithfully with the complexity of human values while remaining answerable to expert human judgment where the stakes are highest.