The AI would act as a representation scientist, not an illustrator. A multimodal foundation model interprets domain, task and audience, proposes candidate encodings and explains the perceptual trade-offs of each. Unlike an image model trained to imitate finished charts, a representation foundation model would be trained on inference: given evidence, a task and a candidate view, it must predict which conclusions a person can correctly draw, which errors become likely and which information has become unreachable. A constraint solver rejects views that violate units, cardinality, accessibility, privacy or known graphical-perception limits. Program synthesis compiles high-level intent into a declarative scene grammar; a differentiable renderer searches the design space against legibility, information retention and cognitive-load objectives. Critic agents attack each view for misleading baselines, aggregation bias, colour failure, occlusion, causal overclaiming and narrative omission. A provenance model traces every generated sentence, shape, sound and transition to data and code. A semantic checksum travels with the scene and fails when an export, summary or modality change no longer supports the authorised claims. The system learns from interaction without silently personalising truth: adaptation changes the route into the evidence, never the underlying claim. When several views support incompatible interpretations, it preserves the disagreement and designs the next observation needed to separate them.
Reality Syntax
“Can data become a world we can enter, question and remember?”
One dataset.
Four ways of knowing.
This is not four different stories. It is one evidence object moving through four inspectable representations. Change the lens; the underlying observations remain fixed.
The beginning
Reality Syntax begins with a refusal: data does not naturally want to become a dashboard. A dataset is evidence collected through instruments, categories and choices; a chart is one temporary projection of that evidence into human perception. Yet most systems freeze the projection, hide the transformation and reward the fastest-looking answer. I imagine a representation operating system that can compile the same evidence into visual, spatial, sonic, tactile, narrative and conversational forms—each adapted to the question, the person, the room and the consequences of being wrong. The representation would remain alive: touching a mark could reveal its source; changing an assumption could reshape the entire scene; uncertainty could acquire movement, texture or sound; and a blind analyst could navigate the same structure through spatial audio and haptics. The ambition is not to decorate data. It is to create a new medium in which evidence can be perceived from several valid positions without losing the line back to reality.
Modern organisations have more data but a narrower sensory language for understanding it. Tables and charts flatten time, causality, uncertainty and provenance into pixels. Dashboards optimise monitoring but encourage metric fixation; generative systems can produce persuasive graphics whose encodings, filters and invented annotations are hard to audit. Analysts make hundreds of consequential choices—collection, cleaning, aggregation, exclusion, scale, colour, baseline, model and narrative—yet the viewer sees only the final surface. Large models answer questions fluently while concealing which records, transformations and counterfactual views would change the conclusion. Spatial computing often turns three dimensions into spectacle rather than insight. Accessibility remains an afterthought, forcing different people to receive reduced versions of the same evidence. The deeper problem is that representation is treated as an output file instead of a reversible, testable and plural computation.
What it could become
Reality Syntax would be a compiler and runtime for evidence. A semantic data fabric first records entities, measurements, units, time, uncertainty, lineage, permissions and contested definitions. An intent layer captures what someone is trying to compare, detect, explain or decide, together with their perceptual needs and the cost of error. A representation compiler then generates a family of views rather than one chart: a compact analytical surface, a temporal story, a navigable spatial field, a sonified structure, a tactile map or a collaborative room-scale model. Every mark carries an executable provenance path back through transformations to authorised sources. A contrast engine deliberately produces rival encodings and asks which claims survive them. A consequence simulator lets the viewer change assumptions and watch uncertainty propagate. A memory layer preserves the path by which a group reached an insight, including discarded views and unresolved disagreement. The result is a signed evidence scene: portable across screen, headset, room, paper, audio and assistive technology without becoming a different truth in each medium.
For whom
- Scientists navigating high-dimensional and uncertain systems
- Public institutions explaining consequential evidence
- Newsrooms and educators creating inspectable data stories
- Clinical, climate and infrastructure teams working under uncertainty
- Blind, low-vision and neurodivergent people underserved by visual-first tools
- Decision rooms that need a shared, challengeable model rather than another dashboard
Core capabilities
- A declarative grammar spanning visual, spatial, sonic, tactile and narrative encodings
- Executable provenance from every perceptual mark to source and transformation
- AI-generated view families constrained by data types, tasks and perceptual evidence
- Uncertainty propagation represented as a first-class channel rather than a footnote
- Cross-modal equivalence tests for accessibility and semantic fidelity
- Collaborative spatial canvases with shared attention, annotation and dissent
- Adversarial re-representation that exposes conclusions dependent on one framing
- Signed evidence scenes that replay the complete path from question to claim
The project becomes meaningful only when a new technical possibility is translated into a clear human advantage, an experience people can understand, and a system capable of earning trust over time.
Intelligence and mathematics
At its core is a typed algebra of representation. Data transformations, statistical models and perceptual encodings form a composable program whose semantics can be checked before rendering. Information theory measures what each projection preserves or discards; rate-distortion theory frames representation as controlled compression in which acceptable loss depends on the decision. Topological data analysis preserves shape across dimensions, while manifold learning creates navigable local neighbourhoods without presenting distance as objective truth. Optimal transport links corresponding structures across modalities and time. Bayesian models propagate epistemic and aleatoric uncertainty through every transformation. Causal graphs separate intervention from association. Sheaf-theoretic consistency tests ask whether locally valid views can be assembled into one coherent global account; a failure becomes visible evidence of incompatible definitions, missing joins or genuine disagreement. A lattice of views orders representations by the questions they can answer, making it possible to calculate when a simplified view has crossed from compression into semantic loss. Multi-objective optimisation searches a Pareto surface across accuracy, accessibility, memorability, cognitive effort, privacy and emotional force. Formal perceptual contracts specify the inferences a scene promises to preserve, and proof-carrying exports allow another system to verify those promises. The radical possibility is a representation that can prove not that it is beautiful, but which conclusions it preserves, which assumptions they depend on and which realities it makes impossible to see.
How it might live
I would begin with high-consequence evidence rooms rather than general business dashboards: climate adaptation, scientific review, clinical research, public budgets or infrastructure planning. The first product could compile one governed dataset into an inspectable family of interactive views and an evidence scene that teams can replay, challenge and sign. A second layer would provide cross-format publishing so the same semantic object becomes a web exploration, briefing, accessible audio experience and room-scale model. Enterprise workspaces, domain grammars and secure rendering could finance an open representation standard and accessibility test suite. Long-term, Reality Syntax could become the protocol between data systems, generative models and human perception—the layer that decides how machine-scale evidence is allowed to enter collective understanding.
For me, a venture is more than an interesting technology. It needs a narrow first user, a repeated problem, a distribution path, a credible advantage and a reason to improve as more people use it. I would test those conditions before deciding whether this idea should become a company, a product, an open technology or an ongoing research programme.
Rules for making it real
- 01
A representation is an argument, never a neutral window.
- 02
Every mark, sound and sentence must reveal its provenance on demand.
- 03
Uncertainty is data and must survive the journey to perception.
- 04
Generate multiple valid views before recommending one.
- 05
Accessibility is a source of invention, not a reduced fallback mode.
- 06
Adapt the interface to the person without personalising the facts.
- 07
Keep observation, model output, inference and speculation visibly distinct.
- 08
Preserve discarded views and dissent when they matter to the decision.
- 09
Never use immersion to manufacture certainty or emotional compliance.
- 10
The viewer must always be able to leave the representation and inspect the evidence.
From question to company
- 01Frame
Define a minimal cross-modal grammar for data, uncertainty, provenance and interaction.
- 02Prototype
Compile one dataset into equivalent visual, audio and tactile analytical experiences.
- 03Prove
Build a provenance debugger that traces any mark back to source, transformation and author choice.
- 04Build
Create an adversarial view generator that searches for truthful representations producing different conclusions.
- 05Advance
Evaluate inference accuracy, memory and trust with diverse users rather than aesthetic preference alone.
- 06Advance
Prototype a collaborative evidence room spanning browser, spatial display and assistive interfaces.
- 07Advance
Add signed evidence scenes with replayable decision histories and dissent.
- 08Advance
Test the system in one high-consequence public or scientific decision process.
- 09Advance
Publish an open conformance suite for semantic equivalence across modalities.
- 10Advance
Develop governance for which optimisations a representation compiler is never allowed to perform.
What could go wrong
Serious imagination includes the possibility that an idea should change radically—or should not exist. These are the tensions the project would need to resolve:
- Generative fluency making an unsupported view feel inevitable.
- Immersion increasing emotional conviction faster than understanding.
- Optimisation for memorability quietly distorting statistical truth.
- Personal adaptation becoming personalised persuasion.
- Cross-modal translations preserving data but changing the inference people draw.
- Provenance interfaces becoming so complex that only specialists can audit them.
- Three-dimensional displays adding occlusion and cognitive load without insight.
- Synthetic examples or annotations being mistaken for observations.
- Accessibility claims being made without co-design and evaluation with disabled people.
- A proprietary representation grammar becoming a gatekeeper for public evidence.
- Sensitive source data leaking through derived views or interaction histories.
- The system producing endless perspectives and preventing a necessary decision.
Built from state-of-the-art research, not visual spectacle
Reality Syntax extends active research in declarative visualization, scalable interaction, uncertainty communication, accessible description and immersive analytics. Its more radical claims—especially cross-modal equivalence and proof-carrying representations—would require new standards, controlled studies and co-design with disabled people before they could be treated as reliable.