TL;DR: A fixed procedure can force the language model to consider the opposing view, but it doesn't guarantee that it will calculate. In a test where a 3-step calculation was spread across 3 sources a general-purpose agent didn't do the calculation at all. I think the issue isn't with the multiplication per se, it's with working out which numbers you're meant to multiply. Therefore, I suggest removing this choice from the model: the code should try all multiplication and division sequences whose units end in a decision unit, compare the results to the thresholds, and have "no discovery" as the default.
Epistemic status: The full design hasn't been built yet. What exists is just a small prototype of the enumeration step, which I tested on a single test case, and the code is attached. However, it is based on evidence that structured methodologies improve judgment accuracy. In geopolitical forecasting competitions, a training session in structured thinking lasting less than an hour increased the accuracy of predictions by 6% to 11% compared to the control group (Chang et al., 2016). I launched a Manifund campaign to finance this work myself.
Sijal is an AI reasoning engine for institutional decisions. The outcome of the Sijal exercise, conducted around controversial decisions, is a sealed decision document, structured in accordance with the Toulmin model and ICD 203, that summarizes the exercise's different phases and results. In Sijal each judge rules individually and one of the participants must defend tenaciously the opposite position. Through structured discussion Sijal emulates institutional reasoning.
Sijal actually works, and when the voting was restarted on more than 30 controversial questions, some of which were posed in Arabic and English, the same rulings came out.
However, the document records what everyone said, but it doesn't extract the one crucial point that the session proved but no one else mentioned. This article is about that step.
To test this step I built up a consistent set of figures and inserted a single discovery. In the session, the food basket was presented as the cheapest option for a family without gas. But three different sources, when put together tell a very different story. The family cook 21 meals a week, each meal requires 1.2kg of firewood and a kg of firewood is five shekels. So the family require 126 shekels worth of firewood a week to cook a basket that costs 120 shekels. The cheapest basket can't be cooked.
A general-purpose AI agent was asked to extract the session's discovery and failed in four ways:
These are all procedural failures and each has a solution that can be engineered. The most serious is the first one: the agent didn't make a mistake in the multiplication, it simply didn't know which numbers went together. So I don't ask the model to make that choice. I start from the decision unit and work backward, in effect doing a reverse dimensional analysis, trying out every possible sequence and coding it up.
Some evidence:
Any failure to gather relevant information will of course limit how well downstream tasks can link that information together.
Therefore, the design begins with a deterministic collection step before any inference: each number is extracted as a triad of value, unit of measurement, and measured property, and each piece is linked to its source. The idea is not new; it is borrowed from other fields that have already explored it.
| Methodology | Origin and Year | Concept |
|---|---|---|
| Wigmore chart | John Henry Wigmore, 1913 | Deconstructing a case into numbered evidentiary points, linked to the claims they support or weaken |
| Provenance tracking (W3C PROV) | W3C Recommendation, 2013 | Entities, activities, agents, and the "wasDerivedFrom" relationship |
| Quantity extraction & unit normalization | UCUM standard and libraries like pint | Representing each number as a triplet: value, standardized unit, and the property being measured |
How I would implement this:
Given a combination lock, one would of course attempt all possibilities, 000-999. No guarantee of being any quicker, but obviously much more certain.
In the test case, all units divide out except for the decision unit:
| Quantity | Value | Unit |
|---|---|---|
| Meals per week | 21 | meal/week |
| Firewood per meal | 1.2 | kg/meal |
| Firewood price | 5 | shekel/kg |
| Chain result | 126 | shekel/week |
| Basket price (weekly basket) | 120 | shekel/week |
126 shekel/week > 120 shekel/week
I hacked together a quick prototype in Python using the pint package. Enumerating all possible sequences and filtering out those that don't have units that make sense, we're left with 7 sequences (out of 230 total from 6 quantities). Requiring a sequence to cost more than 120 shekels leaves us with two: the planted discovery (126 shekels) and a decoy (203 shekels). The decoy snuck through because the households without gas statistic is unitless (just a percentage), so dividing by it doesn't change the units. The quantities were entered by hand, so extraction isn't tested here.
Discovery may also be made by looking for contradictions, i.e., performing a deterministic (numerical) match, then doing linguistic inferences (checked with two models) and filling an analysis of competing hypotheses (ACH) matrix to identify the unknown factor. This approach is less solid than the one presented in this article. Similarly, discovery of second order effects, i.e., trying to understand what would happen (changes in behaviours for each of the parties involved) if the judgement was right, given that each arrow relies on a fact. That's why their targets in the evaluation below are lower.
Program-of-Thoughts (PoT) does well at improving reasoning performance (on average about 12% higher performance than CoT in eight math and finance datasets, and further improved with self-consistency; TMLR 2023) by decomposing the task into 1) reasoning part (solved by LLM) and 2) calculation part (solved by external calculator). In the above case, however, what went wrong was not the calculation part but the reasoning part: LLM failed to recognize the necessity of calculation in the first place. Therefore, rather than letting LLM generate a program, we exhaustively generate (in Python) all possible calculations suggested by the analysis unit and ask LLM to interpret the results.
We propose several filters to retain only substantively new insights. The first three ideas are borrowed from the patent domain because patent law has a long history of dealing with the question of whether something qualifies as a discovery.
If we still have more than one insight after applying the filters, we can rank them using lexicographic order with respect to the following attributes (inspired by Ronald Howard's definition of the value of information as that which can potentially change decisions):
If it's still a tie, we can ask three LLMs to compare the insights in pairs (changing the order in which insights are presented to avoid bias) and aggregate the results using Bradley-Terry.
Finally, we ask LLM to write the insight in a specific format, to facilitate mechanical verification:
We can mechanically verify the writeup as follows:
Evaluation set composition:
Success metrics:
All hyperparameters (insight ranking method, thresholds for insight filters, etc.) should be pre-specified and documented ex ante, for ease of reproducibility (we can also attach a hash that encapsulates the insight, relevant search terms, relevant data extracts, models and parameters used to arrive at the insight, etc. to each reported insight to facilitate ex post auditing by other people).
Planted insights could also include insights that nearly qualify but don't quite (because they're already made in the analyst report, don't have big enough implications, are too broad in scope, are already well-known in public sources, etc.) to check that the agent doesn't report those.
Finally, we can evaluate on actual analyst reports and data rooms. Presumably, if the agent flags an insight on an actual report, we'd want to have another expert look at the report to establish ground truth about whether the insight was actually there, so this should only be a very small fraction of the evaluation set (I'd start with 10%).
Little so far:
No grants, no investment, nothing beyond that. I have two applications currently pending for Sijal: a BlueDot rapid grant and an application to the Epistemics Fund via Lightcone Commons, with a decision on the latter due mid-December 2026. I'll update this post if either of those comes through.
I'm especially curious whether anyone thinks the eval I've described would go poorly in practice (for either of the two reasons above), and what would be a better eval.
The Manifund page for Sijal's project (campaign open until Dec 30) is here if you'd like to follow along or consider supporting!
I'm Abdalrahman Alyazouri, a software developer based in Gaza, Palestine. I've been programming professionally for the last 8 years (mostly in Python), and since April 2026 I've been working full-time on Sijal and Saql.
Some of the ideas behind Sijal and Saql came out of the following chain of realizations:
Contact: [email protected]