# OBSERVER Format Specification **Version 0.1** The OBSERVER Format is an open standard for reporting the results of quantitative analysis to AI agents, LLMs, and human readers. It arranges an analysis into a short hierarchy of files that answers one question first, "What do I need to know?", and then lets any reader descend, as far as their task requires, into the formulas and raw data that produced the answer. The OBSERVER Format is an openly acknowledged variant of the Open Knowledge Format (OKF) put forward by Google. It keeps OKF's core conventions (plain markdown, YAML frontmatter, no required tooling, progressive disclosure, permissive consumption) and specializes them for one kind of knowledge: computed metrics and the evidence behind them. If you can cat a file, you can read an OBSERVER Folder. If you can run Python, you can check one. ## Contents - [Motivation](#1-motivation) - [Terminology](#2-terminology) - [Folder structure](#3-folder-structure) - [Common conventions](#4-common-conventions) - [1.0_INDEX.md: the contract](#5-10_indexmd-the-contract) - [1.1_LOG.md: the update history](#6-11_logmd-the-update-history) - [Observation objects](#7-observation-objects) - [2.0_VALENCE.md: the highest-value information](#8-20_valencemd-the-highest-value-information) - [3.0_PERCEPTION.md: all computed metrics](#9-30_perceptionmd-all-computed-metrics) - [4.0_RAW_DATA: the input layer](#10-40_raw_data-the-input-layer) - [Trust, verification, and declared opacity](#11-trust-verification-and-declared-opacity) - [Relationship to OKF](#12-relationship-to-okf) - [Complexity: the Total Space of Information (TSI)](#13-complexity-the-total-space-of-information-tsi) - [How a consumer reads an OBSERVER Folder (informative)](#14-how-a-consumer-reads-an-observer-folder-informative) - [Conformance](#15-conformance) - [Versioning](#16-versioning) - [Appendix A: Worked example](#appendix-a-worked-example) The key words MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY are to be interpreted as described in RFC 2119. ## 1. Motivation Agents and LLMs are increasingly the first consumers of a quantitative report. A person asks a question; an agent finds the analysis, reads it, and summarizes it before any human sees the source. That arrangement creates two problems a conventional report does not solve: - **Efficiency.** A report written for humans buries its most important numbers inside narrative, tables, and appendices. An agent spends tokens and attention reading everything to find the few values that matter, and may still choose the wrong ones. - **Trust.** An agent that repeats a number has no structured way to show where it came from. The reader downstream must either accept the figure on faith or redo the analysis from scratch. The OBSERVER Format addresses both with a single structural idea: the most important information is placed first, and every piece of it carries its own means of verification. High-value results sit at the top of the hierarchy in a compact, predictable format. Each result carries a trust field (how it was computed) and a verify field (which raw inputs it was computed from). A reader can stop at the top, or descend one layer at a time until the result is confirmed to the degree the task demands. This is what the specification calls white-boxing: making a quantitative analysis transparent enough that black-box AI systems can consume it efficiently and accurately, and so that humans can audit what those systems relayed. ### Goals - Define a folder format that producers (analysts, analytical engines, agents) can write the output of any quantitative analysis into. - Define how consumers (agents, LLMs, deterministic code, people) should read it, so that the highest-value information is reached at the lowest cost. - Require that every reported value be traceable to its formula and its inputs, or that any limit on that traceability be stated explicitly. - Remain compatible with OKF so that OBSERVER Folders can live inside, and be read by tools built for, OKF bundles. ### Non-goals - Prescribing which metrics, statistics, or models an analysis uses. - Prescribing the method used to decide what is important. The format requires that the method be declared, not that it be any particular method. - Replacing data formats (CSV, Parquet, JSON) or computation formats (SQL, notebooks, dbt). OBSERVER references them. - Defining storage, serving, or execution infrastructure. ## 2. Terminology - **OBSERVER Format:** The standard defined by this document: the file structure, fields, and rules an OBSERVER Folder follows. - **OBSERVER Folder (or Observer):** A self-contained directory holding one quantitative analysis in the structure defined here. The unit of distribution. - **Raw reality:** The complete input data set of an analysis, stored in the 4.0_RAW_DATA layer. - **Perception:** The full set of metric values the analysis computed from raw reality, stored in `3.0_PERCEPTION.md`. - **Valence:** The subset of perception that stands out when compared with the rest of perception, with past perception, or both. Stored in `2.0_VALENCE.md`. High-valence information is the answer to "What do I need to know?" - **Contract:** The rules an individual Observer declares for computing perception and selecting valence. Stored in `1.0_INDEX.md` frontmatter. - **Metric:** A named quantity the analysis computes, defined in the contract with a formula and a statement of utility, and optionally a citation. - **Subject:** The entity a metric value describes (a store, a ticker, a patient cohort, a region). Optional when a metric has only one subject. - **Observation object:** One computed metric value together with its comparison results, trust field, and verify field. The basic unit of both `3.0_PERCEPTION.md` and `2.0_VALENCE.md`. - **Relative observation (relative_ob):** A contract-defined comparison of a metric value against other values, for example "z-score against all subjects at the current date" or "z-score against this subject's own history." - **Relative value:** A contract-defined score, computed from an object's relative observations, that determines whether the object is elevated from perception to valence. - **Elevation:** The act of promoting an observation object from `3.0_PERCEPTION.md` to `2.0_VALENCE.md` under the contract's rule. - **Trust field:** The part of an observation object that states how the value was computed: the formula in Python, or an explicit statement that the formula is proprietary. - **Verify field:** The part of an observation object that points to the exact rows and columns of raw data used to compute the value, with notes on any data gaps and how they were handled. - **Disclosure:** Whether a formula or data source is open or proprietary. Proprietary disclosure is permitted; undisclosed opacity is not. - **White-box depth:** How far a consumer has descended into the hierarchy to confirm a value (§14). - **Layer:** A numbered level of the hierarchy (1 contract, 2 valence, 3 perception, 4 raw data), shown by each file's numeric prefix. - **Navigation block:** The navigation frontmatter every .md file carries, stating its layer and pointing to related layers (§3.4.2). - **Entry file:** `README.md`, `AGENTS.md`, or `CLAUDE.md`; an identical short map that directs a reader to `1.0_INDEX.md` (§3.4.3). - **Raw data reading notice:** The standard statement limiting full reading of layer 4 to explicitly requested or mandated verification (§3.4.4). - **Total Space of Information (TSI):** A two-dimensional measure of an Observer's scale, rows by columns, reported at each layer of the hierarchy (§13). ## 3. Folder structure An OBSERVER Folder is a directory with exactly this layout: ``` OBSERVER/ # The folder. One analysis. 1.0_INDEX.md # Layer 1. The contract: purpose, folder map, metric and valence definitions. 1.1_LOG.md # Layer 1 history. Relevant to recurring and one-off reports alike. 2.0_VALENCE.md # Layer 2. The highest-value information. 3.0_PERCEPTION.md # Layer 3. All computed metrics. 4.0_RAW_DATA. # Layer 4. The raw data the computations ran on (.csv, .json, .parquet, ...). # Absent when the raw data is referenced or withheld (§10.3). AGENTS.md # Entry file. Identical to README.md. README.md # Entry file. Identical to AGENTS.md. CLAUDE.md # Entry file. Identical to AGENTS.md. references/ # Optional. Supporting code and documents (§3.1). ``` The numbered files form a hierarchy that is read top-down. Each layer is a compression of the layer below it, and each layer carries pointers into the layer below it: ``` OBSERVER/ │ ├── README.md / AGENTS.md /CLAUDE.md Entry files: a short map that points to 1.0_INDEX.md. │ ├── 1.0_INDEX.md What this Observer measures, and the rules it uses. │ ├── 1.1_LOG.md When it ran and what changed. │ │ [Efficient perception of high-value information, with citations │ that can be confirmed within the hierarchy] ├── 2.0_VALENCE.md 'trust' and 'verify' fields on every object │ │ [Optional white-box depth 1: 'trust' formulas checked for │ consistency with 1.0_INDEX.md; optionally confirmed against the │ cited source or outside references] ├── 3.0_PERCEPTION.md 'trust' and 'verify' fields on every object │ │ [Optional white-box depth 2: 'verify' inputs confirmed to exist; │ optionally, the perception computation replicated] └── 4.0_RAW_DATA. Rows and columns that support or refute any 'verify' map ``` ### 3.1 Reserved filenames The following names are reserved at the root of an OBSERVER Folder. Filenames are case-sensitive. | Name | Required | Purpose | |---|---|---| | `1.0_INDEX.md` | Yes | Layer 1. The contract. See §5. | | `1.1_LOG.md` | Yes | Layer 1 history. See §6. | | `2.0_VALENCE.md` | Yes | Layer 2. High-valence observation objects. See §8. | | `3.0_PERCEPTION.md` | Yes | Layer 3. All observation objects. See §9. | | `4.0_RAW_DATA.` | When availability is included | Layer 4. Input data. See §10. Absent when the data is referenced or withheld; `1.0_INDEX.md` then declares where it is or why it is not shared (§10.3). | | `4.0_RAW_DATA/` | No | Directory form of layer 4, for analyses with more than one table (§10.1). | | `references/` | Optional | Supporting material: longer Python modules, source documents (see below). | | `README.md` | Recommended | Entry file. See §3.4.3. | | `AGENTS.md` | Recommended | Entry file, identical to `README.md`. See §3.4.3. | | `CLAUDE.md` | Optional | Entry file alias for harnesses that read this name. See §3.4.5. | A producer MAY add other files, most commonly a `references/` directory holding longer Python modules or source documents, following the OKF convention. Every .md file under `references/` MUST carry type: OBSERVER Reference and a navigation block (§3.4.2), so that a reader who lands on one by search or retrieval is directed back into the hierarchy like any other. Consumers MUST ignore files they do not recognize. ### 3.2 Naming the folder The folder SHOULD be named OBSERVER when it stands alone, or `_OBSERVER` (for example weekly_sales_OBSERVER) when several Observers sit side by side. A folder's identity comes from the `observer_id` in its `1.0_INDEX.md`, not from the directory name. ### 3.3 Distribution An OBSERVER Folder MAY be distributed as a git repository (recommended, since history and diffs make recurring reports auditable), a zip or tar archive, or a subdirectory of a larger repository or OKF bundle (§12.3). ### 3.4 Entry and navigation An OBSERVER Folder is only efficient if it is read in order: contract first, then valence, then perception, then raw data only as deeply as the task requires. No agent or LLM can be assumed to know this order. OKF is a young format and its reading conventions are not yet learned behavior, and agents enter a folder in different ways: by listing a directory, by searching for a keyword, by loading an instruction file, or by retrieving a single file or chunk with no knowledge of the rest. OBSERVER therefore reinforces the reading order through several independent mechanisms, so that an agent is directed to the right place whichever way it arrives. Each mechanism covers a different point of entry: | Mechanism | Covers an agent that... | Status | |---|---|---| | Layer prefixes (§3.4.1) | lists the directory | REQUIRED | | Navigation frontmatter (§3.4.2) | lands on any single file, by search or retrieval | REQUIRED | | Raw data reading notice (§3.4.4) | is about to open the raw data | REQUIRED | | `README.md` + `AGENTS.md` pair (§3.4.3) | opens the conventional entry file, or has its harness load it | RECOMMENDED | | `CLAUDE.md`, Agent Skill, MCP server (§3.4.5) | uses a harness with its own conventions | OPTIONAL | Navigation text in every mechanism SHOULD be written as a plain description of where the reader is and where related information lives (for example "Layer 3 of 4. The contract is in `1.0_INDEX.md`."), not as an imperative command (for example "YOU MUST READ INDEX FIRST"). Many agent harnesses deliberately treat file content as data rather than as instructions and are wary of imperative text inside files, which resembles prompt injection. Descriptive navigation works with that caution rather than against it. #### 3.4.1 Layer prefixes Every hierarchy file carries a numeric prefix giving its layer, so that any alphabetical directory listing presents the files in reading order. Without prefixes, a listing sorts the files as INDEX, LOG, PERCEPTION, RAW_DATA, VALENCE, which places the most important file last. The prefix scheme is `._`: | Prefix | File | Meaning | |---|---|---| | `1.0_` | `1.0_INDEX.md` | Layer 1 | | `1.1_` | `1.1_LOG.md` | Layer 1, supplementary: the history of layer 1 | | `2.0_` | `2.0_VALENCE.md` | Layer 2 | | `3.0_` | `3.0_PERCEPTION.md` | Layer 3 | | `4.0_` | `4.0_RAW_DATA.` | Layer 4 | Two details of the scheme are deliberate: - **Every primary layer carries .0.** In byte-order sorting (used by most tools and by Python's sorted), the character . sorts before _. A file named `1.1_LOG.md` would therefore sort before `1_INDEX.md`. Writing `1.0_INDEX.md` puts both names on the same footing, and the pair sorts correctly under byte-order, locale-aware, and natural ("version") sorting alike. (A letter suffix such as `1a_LOG.md` sorts correctly under byte order but not under natural sorting, which is why it is not used.) - **The log is 1.1, not a layer of its own.** It records the history of the contract and of each report, so it belongs with layer 1 and is read, if at all, immediately after it. Entry files (`README.md`, `AGENTS.md`) are unprefixed because the tools that look for them search by exact name. They sort after the numbered files, so a directory listing still begins with `1.0_INDEX.md`. #### 3.4.2 Navigation frontmatter Every .md file in an OBSERVER Folder MUST carry a navigation block in its frontmatter, immediately after type. Frontmatter is the first thing in the file, so this is the first thing any reader sees, however they arrived. ```yaml navigation: layer: "3 of 4" # REQUIRED. This file's position ("entry" for entry files). entry: 1.0_INDEX.md # REQUIRED. Always 1.0_INDEX.md. up: 2.0_VALENCE.md # REQUIRED except in 1.0_INDEX.md and entry files. down: 4.0_RAW_DATA.csv # REQUIRED except in the lowest layer present. note: > # REQUIRED. One or two descriptive sentences. Layer 3 of 4: every computed value. The contract is in 1.0_INDEX.md; the ranked highest-value results are in 2.0_VALENCE.md. ``` - layer, entry, up, and down are for programmatic traversal. note is for a model or person reading the file as text; it restates the same position in a sentence. - `1.1_LOG.md` uses layer: "1.1 (history of layer 1)", up: `1.0_INDEX.md`, and no down. - A .md file under `references/` uses layer: reference, up: `1.0_INDEX.md`, and no down. Its note says what the file supports, for example "Helper functions called by the trust formulas of `wow_revenue_change`. The contract is in `1.0_INDEX.md`." - Every file that entry, up, or down names MUST exist in the folder, with paths written relative to the folder root (so a file under `references/` still writes entry: `1.0_INDEX.md`), so that navigation never leads a reader to a dead end. - When raw data is included, the note of `3.0_PERCEPTION.md` MUST include the raw data reading notice (§3.4.4), since layer 4 is the next step down. When raw data is referenced or withheld, `3.0_PERCEPTION.md` is the lowest layer present and has no down; its note SHOULD say that the raw data is not in the folder and that `raw_data` in `1.0_INDEX.md` states where it is or why it is not shared. #### 3.4.3 Entry files: README.md and AGENTS.md `README.md` is the file people, GitHub, and many agents open first in any directory. `AGENTS.md` is an open convention, supported by many coding agents, for instructions an agent loads when it works in a directory; nested `AGENTS.md` files are supported, with the nearest one taking precedence. Providing both covers the two most common conventional entry points. A producer SHOULD provide both files. When both are present: - They MUST be byte-identical, so neither can drift from the other. - Each MUST state that the two files are identical copies, so a reader who has opened one knows the other adds nothing. - Each MUST carry the frontmatter of §3.4.2 with type: OBSERVER Entry and layer: entry, which also keeps the folder a conformant OKF bundle (§12.3). The body SHOULD be short: what the Observer is, the reading order, and the raw data notice. The template is: ```markdown --- type: OBSERVER Entry navigation: layer: entry entry: 1.0_INDEX.md down: 1.0_INDEX.md note: > This folder is an OBSERVER Folder. Its contract and folder map are in 1.0_INDEX.md. README.md and AGENTS.md are identical copies; reading one makes the other unnecessary. --- # This folder is an OBSERVER Folder: a quantitative analysis arranged so the most important results come first, each with its formula and its raw inputs attached. Specification: . ## Reading order 1. `1.0_INDEX.md`: what this Observer measures and the rules that decide which results matter. Its frontmatter alone is enough to judge relevance. 2. `2.0_VALENCE.md`: the highest-value results, ranked. For most tasks, reading can stop here. 3. `3.0_PERCEPTION.md`: every computed value, for context. 4. `4.0_RAW_DATA.`: rows × columns. Trust verification with select columns and rows only. Full data set only intended for full reading when verification of values reported in 2.0_VALENCE.md and/or 3.0_PERCEPTION.md is explicitly requested by the user or programmatically mandated by the assigned workflow. (When the raw data is referenced or withheld, item 4 instead reads: "The raw data is not in this folder; `raw_data` in `1.0_INDEX.md` states where it is or why it is not shared.") `1.1_LOG.md` holds the report history and any changes to the rules. README.md and AGENTS.md in this folder are identical copies. Reading one makes the other unnecessary. ``` #### 3.4.4 Raw data reading notice The raw data layer is the costliest to read and is rarely needed in full. The following notice, with the actual file name and size filled in, is the raw data reading notice: > `4.0_RAW_DATA.`: `` rows × `` columns. Trust verification with select columns and rows only. Full data set only intended for full reading when verification of values reported in `2.0_VALENCE.md` and/or `3.0_PERCEPTION.md` is explicitly requested by the user or programmatically mandated by the assigned workflow. The notice covers both ordinary use, where a value is checked against the few rows and columns its verify field names, and cases where full verification is warranted, for example when the Observer supports a mission-critical decision and the workflow requires the entire data set to be checked. When raw data is included, the notice MUST appear in: - `1.0_INDEX.md`, as `raw_data.reading_notice` (§5.4); - the `navigation.note` of `3.0_PERCEPTION.md` (§3.4.2); - the entry files, when present (§3.4.3). When no raw data is in the folder, the declaration lives in `1.0_INDEX.md` alone. For referenced data, whose rows a consumer can still fetch, the notice MUST appear as `raw_data.reading_notice`, naming the URI in place of the file name. For withheld data there is nothing to read, and `raw_data.withheld_reason` takes the notice's place. In the directory form (§10.1), the notice's first sentence describes the directory rather than one file, and the identification of each table is left to the objects that use it: > `4.0_RAW_DATA/`: `` files; each object's `verify.source` names the file it was computed from, and `raw_data.files` gives each file's rows and columns. Trust verification with select columns and rows only. Full data set only intended for full reading when verification of values reported in `2.0_VALENCE.md` and/or `3.0_PERCEPTION.md` is explicitly requested by the user or programmatically mandated by the assigned workflow. A CSV file cannot carry the notice itself. Where the raw data format supports embedded metadata (Parquet key-value metadata, or a JSON wrapper object), the producer MAY also embed the notice there. #### 3.4.5 Optional harness-specific entry points These mechanisms help particular agents or environments. They MAY be added alongside the required and recommended ones and MUST NOT replace them. - **CLAUDE.md.** Some harnesses, notably Claude Code, read `CLAUDE.md` rather than `AGENTS.md`. A producer targeting those harnesses MAY add a `CLAUDE.md` identical to the entry pair, and the identical-copies statement then names all three files. - **Agent Skill.** A reusable skill that teaches the reading protocol of §14 to any agent runtime that supports skills. - **MCP server.** A server exposing tools such as `observer_index`, `observer_valence`, and observer_verify(id). This is the only mechanism that enforces reading order, because the agent reaches each layer only through the tool for it. It is also the most work to build and maintain. ## 4. Common conventions ### 4.1 File format Every .md file in an OBSERVER Folder is UTF-8 markdown that begins with a YAML frontmatter block delimited by --- lines, as in OKF. Every frontmatter block MUST contain a type field with the value given for that file below, followed by the navigation block of §3.4.2. The type field makes each file a conformant OKF concept (§12). | File | type value | |---|---| | `1.0_INDEX.md` | OBSERVER Index | | `1.1_LOG.md` | OBSERVER Log | | `2.0_VALENCE.md` | OBSERVER Valence | | `3.0_PERCEPTION.md` | OBSERVER Perception | | `README.md`, `AGENTS.md`, `CLAUDE.md` | OBSERVER Entry | | `references/**/*.md` | OBSERVER Reference | ### 4.2 Keys Field names are lowercase `snake_case`. Producers MAY add their own keys anywhere; consumers MUST NOT reject a file because of unknown keys and SHOULD preserve them when rewriting a file. ### 4.3 Dates and times `report_date` values are ISO 8601 dates (2026-09-20) or datetimes with an explicit UTC offset (2026-09-20T23:00:00Z). Timestamps inherited from OKF (`generated.at`, `verified[].at`, `stale_after`) are always datetimes with an explicit offset. ### 4.4 Actors Wherever an identity is recorded, OBSERVER uses the OKF actor convention: `/` for agents and tools (observer_engine/1.4.0), `human:` for people, and `process:` for automated processes. ### 4.5 Python in trust fields Open formulas are written in Python so they can be replicated directly. A formula: - SHOULD be a single self-contained function whose name matches the metric or rule it computes. - SHOULD name its arguments after the 4.0_RAW_DATA columns (or upstream metrics) they receive. - MUST NOT perform I/O, network access, or randomness without a declared seed. - MAY use only the standard library and the packages declared in `1.0_INDEX.md` under `runtime.dependencies`. Longer implementations MAY live in a file (for example `references/metrics.py`); the trust field then carries the function and a resource path to the file. ## 5. 1.0_INDEX.md: the contract `1.0_INDEX.md` is the first file any consumer reads. It states what the Observer is for, maps the folder, and declares every rule that governs the rest of it. An agent SHOULD be able to decide from `1.0_INDEX.md` frontmatter alone whether this Observer is relevant to its task. All contract content lives in YAML frontmatter at the head of the file. The body is a short human-readable statement of purpose and a folder map. ### 5.1 Frontmatter fields ```yaml --- type: OBSERVER Index # REQUIRED navigation: { ... } # REQUIRED. See §3.4.2. observer_version: "0.1" # REQUIRED. Spec version targeted (§16). observer_id: # REQUIRED title: # REQUIRED purpose: # REQUIRED. What question this Observer answers. report_date: # REQUIRED. Date of the current report. cadence: # Recommended. transparency: # REQUIRED. See §11.4. runtime: # Recommended when any formula is open. language: python version: ">=3.10" dependencies: [, ...] raw_data: { ... } # REQUIRED. See §5.4. metrics: [ ... ] # REQUIRED. See §5.2. valence: { ... } # REQUIRED. See §5.3. complexity: { ... } # Recommended. See §13. generated: { by: , at: } # Recommended (OKF §5.2). verified: [ { by: , at: } ]# Optional (OKF §5.2). stale_after: # Optional (OKF §5.5). sources: [ ... ] # Optional (OKF §5.1). --- ``` ### 5.2 metrics: the degrees of freedom of perception metrics lists every metric that appears in `3.0_PERCEPTION.md`. Together these entries are the full set of formulas that make up the perception layer, so an agent can see at a glance what kind of information the Observer contains. ```yaml metrics: - metric: # REQUIRED. The name used in observation objects. description: # REQUIRED. What the value means, with units. formula: | # REQUIRED. Python (open), or a proprietary statement. disclosure: open # REQUIRED. open | proprietary. No default. utility: # REQUIRED. Why this metric is part of perception. citation: # Optional; REQUIRED when disclosure is proprietary. See below. inputs: [, ...] # REQUIRED. 4.0_RAW_DATA columns or upstream metrics. ``` - formula is the authoritative definition. Every observation object for this metric echoes it in its trust field (§7.3), and the two MUST match. - utility states why the metric is included: what decision or question it informs. - citation is the producer's choice for an open formula. The trust and verify fields are the format's mechanisms of scrutiny; a citation adds credibility on top of them. It is most useful where readers may not know the method: an industry report that relies on Shannon entropy, for example, can cite Shannon (1948), Bell System Technical Journal 27(3), so that professionals who have never met the measure can see that it is established. A citation may be an academic or standard reference (for example Welford (1962), Technometrics 4(3)), or original for a formula introduced by the producer. - For a proprietary formula, citation is REQUIRED and takes the form proprietary, citable as "``" (§11.3): it is how the opacity is declared. ### 5.3 valence: the rules of elevation valence declares how the Observer decides that a perceived value is important. It has three parts: the relative observations, the relative value that combines them, and the elevation rule. ```yaml valence: relative_obs: # REQUIRED. One or more. - id: # REQUIRED. Referenced by observation objects. name: # REQUIRED. comparison: # REQUIRED. population: # REQUIRED. What the value is compared against. # For history: window, gap handling, minimum count (§7.4). formula: | # REQUIRED. Python, or proprietary statement. disclosure: open # REQUIRED. open | proprietary. citation: # Optional; REQUIRED when proprietary (§5.2). relative_value: # REQUIRED. description: # REQUIRED. formula: | # REQUIRED. Combines relative_obs into one score. disclosure: open # REQUIRED. open | proprietary. elevation: # REQUIRED. rule: # REQUIRED. E.g. "relative_value >= 2.0". order: # REQUIRED. How 2.0_VALENCE.md is ranked. max_objects: # Optional cap on 2.0_VALENCE.md size. ``` - Relative observations are the comparisons that make a value stand out. The elementary case is the value that moved furthest, in either direction, compared with all other values now (`cross_sectional`) or with its own past (temporal). Producers MAY use any comparison method (z-scores, percentile ranks, log-ratios in the manner of the Weber-Fechner law, model residuals) as long as it is declared here. - Relative value reduces an object's relative observations to one number used for elevation and ranking. - Elevation is the rule that decides membership in `2.0_VALENCE.md`. A consumer MUST be able to tell, from `1.0_INDEX.md` alone, why any object in `2.0_VALENCE.md` is there. ### 5.4 raw_data: the data dictionary ```yaml raw_data: files: - path: 4.0_RAW_DATA.csv # REQUIRED. Path within the folder, or URI. format: csv # REQUIRED. csv | json | jsonl | parquet | ... rows: # REQUIRED. key: [, ...] # REQUIRED. Columns that uniquely identify a row. columns: # REQUIRED. - { name: , type: , description: } availability: included # included | referenced | withheld. See §10.3. reading_notice: > # REQUIRED unless withheld. The raw data reading notice (§3.4.4). sources: [ ... ] # Optional. Where the raw data came from (OKF §5.1). ``` The key columns are what verify fields use to select rows (§7.4). ### 5.5 Body The body SHOULD contain a one-paragraph statement of purpose and a folder map in OKF index form: ```markdown # Weekly Store Sales Observer Surfaces the stores whose weekly revenue or ticket size moved most unusually this week, relative to other stores and to their own history. # Folder * [Valence](2.0_VALENCE.md) - The objects elevated this report, ranked. * [Perception](3.0_PERCEPTION.md) - Every computed metric value. * [Log](1.1_LOG.md) - Report history and contract changes. * [Raw data](4.0_RAW_DATA.csv) - Input rows; see `raw_data` for the dictionary and reading notice. ``` ## 6. 1.1_LOG.md: the update history `1.1_LOG.md` records every report the Observer has produced and every change to its contract. It follows the OKF log format (OKF §9): date headings in YYYY-MM-DD form, newest first, each followed by a list of entries. ```markdown --- type: OBSERVER Log navigation: layer: "1.1 (history of layer 1)" entry: 1.0_INDEX.md up: 1.0_INDEX.md note: > Report history and contract changes for this Observer. The current contract is in 1.0_INDEX.md. --- # Observer Log ## 2026-09-20 * **Report**: 24 perception objects; 2 elevated to valence. * **Data**: 12 rows added to 4.0_RAW_DATA.csv (week ending 2026-09-20); no rows removed; columns unchanged (5). ## 2026-09-06 * **Contract**: Elevation threshold raised from 1.5 to 2.0 to reduce noise. ``` Conventional leading words are `**Report**`, `**Data**`, `**Contract**`, `**Correction**`, and `**Initialization**`. A `**Data**` entry SHOULD state how the raw data changed along every dimension: rows added and removed, columns added and removed, and, in the directory form, files added and removed, with counts, saying so explicitly where a dimension did not change. A reader can then see how the input layer expanded or contracted without comparing files. A change to any formula, relative observation, relative value, or elevation rule in `1.0_INDEX.md` MUST be recorded with a `**Contract**` entry. A change in how importance is judged changes what a reader is told is important, so it is a trust-relevant event and must never be silent. For a one-off report, `1.1_LOG.md` MAY hold a single `**Initialization**` or `**Report**` entry. ## 7. Observation objects `3.0_PERCEPTION.md` and `2.0_VALENCE.md` are both lists of observation objects, and both use the same object structure. An agent parses one shape across every value in every Observer. ### 7.1 Placement in the file After the file's frontmatter, each object appears in the body under a level-two heading holding its id, followed by a single fenced YAML block: ````markdown ## wow_revenue_change/S07/2026-09-20 ```yaml id: wow_revenue_change/S07/2026-09-20 metric: wow_revenue_change ... ``` ```` Headings let a reader or a text search jump to one object without parsing the whole file; the YAML block keeps each object compact and machine-readable. ### 7.2 Fields ```yaml id: # REQUIRED. Unique within its file (see below). metric: # REQUIRED. A metric declared in 1.0_INDEX.md. subject: # Recommended. The entity the value describes. report_date: # REQUIRED. The date the value describes. value: # REQUIRED. The reported value. unit: # REQUIRED. What the value is measured in; for a # unitless value, say what it is (count, ratio, label). observations: # REQUIRED. One entry per relative_ob in the contract. - { rel_ob: , value: , note: } relative_value: # REQUIRED. The contract's relative value; null when it cannot be computed. elevated: # REQUIRED in 3.0_PERCEPTION.md. trust: { ... } # REQUIRED. See §7.3. verify: { ... } # REQUIRED. See §7.4. ``` - id MUST be unique within its file. It is deliberately not unique across the folder: an object in `2.0_VALENCE.md` carries the same id as the `3.0_PERCEPTION.md` object it was elevated from (§8). - id SHOULD follow the form `//`, which is stable and self-describing. Positional identifiers (such as "row 14" or "item 3") MUST NOT be used, because rewriting or re-sorting the file would silently change what they point at. - observations holds the result of each relative observation declared in `1.0_INDEX.md`. These are the comparisons that do or do not surface the value to `2.0_VALENCE.md`. - `relative_value` and elevated make the elevation decision explicit and checkable against `valence.elevation.rule`. ### 7.3 The trust field The trust field answers "How was this number computed?" ```yaml trust: disclosure: open # open | proprietary formula: | # REQUIRED when open. Echoes 1.0_INDEX.md exactly. def wow_revenue_change(revenue_t, revenue_t_minus_1): return (revenue_t - revenue_t_minus_1) / revenue_t_minus_1 resource: references/metrics.py # Optional. Longer implementation. ``` For a proprietary formula: ```yaml trust: disclosure: proprietary statement: 'Proprietary, citable as "Acme Momentum Score v3 (Acme Research, 2026)".' track_record: # Optional. See §11.3. ``` ### 7.4 The verify field The verify field answers "Which inputs was this number computed from?" ```yaml verify: source: 4.0_RAW_DATA.csv # REQUIRED. The file declared in raw_data that # this value was computed from (§7.4). rows: # REQUIRED. Selector on the key columns. store_id: S07 week_ending: [2026-09-13, 2026-09-20] columns: [revenue] # REQUIRED. Columns the formula read. args: # Conditional. Formula argument → data. revenue_t: { column: revenue, rows: { week_ending: 2026-09-20 } } revenue_t_minus_1: { column: revenue, rows: { week_ending: 2026-09-13 } } notes: # Optional. Gaps, imputation, exclusions. ``` - source identifies the table. In the directory form (§10.1) it is the file's path within the directory, for example `4.0_RAW_DATA/transactions.csv`, and it is the only place a reader learns which table an object was computed from; the reading notice describes the directory as a whole (§3.4.4). - rows MUST select rows by the key columns declared in `raw_data`, not by position. A value in the selector may be a single value, a list of values, or a range written `{ from: , to: }`. - columns lists every column the formula read, from every file it read. - args maps each argument of the trust formula to the data it receives, so a consumer never has to guess which row feeds which argument. Each entry names one column, which MUST be listed in columns, and MAY give a source and a rows selector: - source defaults to the object's `verify.source`. When an argument's data comes from another table, as when a value is computed from a join, the entry names that file. It MUST be declared in `raw_data.files`, and the entry's column MUST be in that file's column dictionary. - An entry's rows selects on the key columns of its own source. Keys it shares with `verify.rows` narrow it: their values MUST lie within those `verify.rows` gives, and a shared key the entry does not name is inherited unchanged, so the two can never disagree. Keys of `verify.rows` that are not key columns of the entry's source do not apply to it. - An entry that selects one row passes a single value. An entry that selects several (through a list or a `{ from, to }` range) passes a list of values, ordered by the key columns in their declared order. - args MUST be present, and MUST name exactly the formula's arguments, when the mapping is not evident: when any argument is not named after a column in columns, when `verify.rows` selects more than one row, or when the formula reads from more than one file. It MAY be omitted otherwise, in which case each argument receives the column of the same name from the single row selected. - A value computed from a join, for example revenue per head from a weekly sales table and a monthly staffing table: ```yaml verify: source: 4.0_RAW_DATA/sales.csv # key: [store_id, week_ending] rows: { store_id: S07, week_ending: 2026-09-20 } columns: [revenue, headcount] args: revenue: { column: revenue } headcount: { column: headcount, source: 4.0_RAW_DATA/staff.csv, rows: { month: 2026-09 } } # key: [store_id, month] ``` headcount inherits `store_id`: S07 from `verify.rows`; `week_ending` is not a key of `staff.csv`, so it does not apply. - notes MUST describe any missing, imputed, excluded, or corrected inputs, and the method used to handle them. A value computed around a data gap without saying so is not conformant. Relative observations compare a value to other perception values, not directly to raw data. They are therefore verified one layer up: a consumer recomputes a `rel_ob` from the `3.0_PERCEPTION.md` objects named by its population in `1.0_INDEX.md`. Where a relative observation needs values that are not in `3.0_PERCEPTION.md`, most often a subject's own history, a consumer rebuilds them by running the metric's formula over the earlier rows of 4.0_RAW_DATA and then runs the relative observation's formula on the result. So that this is always possible: - 4.0_RAW_DATA MUST retain every row the comparison's window reaches, not only the rows named in `verify.rows`. When raw data is referenced or withheld (§10.3), those rows MUST be part of the referenced or withheld data set. - The relative observation's population MUST state the window precisely: its length, how gaps within it are handled, and the minimum number of values required for the comparison to be computed. - The object's `verify.notes` SHOULD name the history rows the comparison used, and MUST name any that were missing (§7.4). Whether to carry replication this far is the consumer's decision (§14); the producer's obligation is that it can be done from the folder alone. ## 8. 2.0_VALENCE.md: the highest-value information `2.0_VALENCE.md` contains the objects that the contract's elevation rule selected: the values showing the most asymmetric behavior relative to the rest of perception. It is the primary information object of an Observer, the file an agent is directed to read and derive its output from. ```yaml --- type: OBSERVER Valence navigation: { ... } # REQUIRED. See §3.4.2. observer_id: # REQUIRED. report_date: # REQUIRED. elevation_rule: # REQUIRED. Echoes 1.0_INDEX.md valence.elevation.rule. object_count: # REQUIRED. May be 0. max_objects: # REQUIRED when the contract declares max_objects. Echoes it. elevated_count: # REQUIRED when the contract declares max_objects. # Objects in 3.0_PERCEPTION.md with elevated: true. generated: { by: , at: } --- ``` Each object in `2.0_VALENCE.md` uses the observation object structure of §7 with two additional fields: ```yaml rank: # REQUIRED. 1 = highest relative value. perception_ref: # REQUIRED. The matching object id in 3.0_PERCEPTION.md. ``` Rules: - `2.0_VALENCE.md` MUST be a subset of `3.0_PERCEPTION.md`. Every object MUST have a `perception_ref` to an object with elevated: true, and the value, observations, `relative_value`, trust, and verify fields MUST match that object. - Every object in `3.0_PERCEPTION.md` with elevated: true MUST appear in `2.0_VALENCE.md`, unless `max_objects` caps the list. - When the contract declares `valence.elevation.max_objects`, the frontmatter MUST state the cap with two keys: `max_objects`, echoing the contract, and `elevated_count`, the number of objects in `3.0_PERCEPTION.md` with elevated: true. `object_count` MUST then equal the smaller of the two, and the objects kept MUST be the highest-ranked elevated objects. The list was truncated exactly when `elevated_count` exceeds `object_count`. - Objects are ordered by rank. Objects whose `relative_value` is null rank last. - The format sets no minimum or maximum number of objects. How many the elevation rule selects is the producer's choice; `max_objects` is an optional cap the producer may apply. - An empty `2.0_VALENCE.md` (`object_count`: 0) is valid and meaningful: nothing in this report met the contract's threshold for importance. - Because most readers stop at this layer, a compact `2.0_VALENCE.md` serves the hierarchy best, and a producer SHOULD prefer a rule that keeps it readable in full at low token cost where the analysis allows. Compactness is a goal, not a limit: when every object meets the rule, as can happen when elevation follows a classification that every subject satisfies, every object belongs here. - The body MAY begin with a single sentence per object in plain language before the objects themselves, provided each sentence links to its object heading. ## 9. 3.0_PERCEPTION.md: all computed metrics `3.0_PERCEPTION.md` contains every metric value the analysis computed for the current report. Every value here is a candidate for elevation to `2.0_VALENCE.md`. This layer supplies the context around high-valence values, and it is where an agent looks for supporting detail its task needs beyond the headline. ```yaml --- type: OBSERVER Perception navigation: { ... } # REQUIRED. Includes the raw data reading notice (§3.4.2). observer_id: # REQUIRED. Matches 1.0_INDEX.md. report_date: # REQUIRED. Matches 1.0_INDEX.md. object_count: # Recommended. generated: { by: , at: } --- ``` Rules: - Every metric declared in `1.0_INDEX.md` MUST have at least one object in `3.0_PERCEPTION.md`. - Every object MUST carry elevated, set according to the contract's elevation rule. - Objects SHOULD appear in a single list ordered by `relative_value`, descending, across all metrics, with objects whose `relative_value` is null at the bottom, so that a partial read still reaches the most interesting values first. ## 10. 4.0_RAW_DATA: the input layer 4.0_RAW_DATA holds the structured data from which every value in `3.0_PERCEPTION.md` and `2.0_VALENCE.md` was calculated. It is the transparent input layer, vital when an analysis informs a mission-critical or high-value decision, and it lets an agent or person replicate a value, extend the analysis, or compare reported values against their own standard. ### 10.1 Form - A single file named `4.0_RAW_DATA.`, in any common structured format (.csv, .json, .jsonl, .parquet), or - A `4.0_RAW_DATA/` directory holding several such files when the analysis joins more than one table. Each file MUST be declared in `1.0_INDEX.md` under `raw_data.files`. ### 10.2 Labeling Column names in 4.0_RAW_DATA MUST match the names used in `raw_data.columns` and in every `verify.columns`, so that any verify map can be resolved mechanically. Every row MUST be uniquely identifiable by the declared key columns. ### 10.3 Availability Raw data cannot always be shipped. `raw_data.availability` declares which case applies. When the data is referenced or withheld, no 4.0_RAW_DATA file or directory is present, `3.0_PERCEPTION.md` is the lowest layer, and `1.0_INDEX.md` carries the whole declaration (§3.4.4): - **included:** the data is in the folder. The default. - **referenced:** the data is too large to include and is available at a stated URI. Each `raw_data.files[].path` is that URI, and the producer SHOULD provide a checksum (sha256) so a consumer can confirm it is reading the same data. - **withheld:** the data cannot be shared (licensing, privacy, or confidentiality). The producer MUST state the reason in `raw_data.withheld_reason`, MUST still provide the column dictionary, and MUST still write verify fields, so that someone with access can resolve them. A withheld data layer makes the Observer's transparency `declared_opacity` (§11.4). ## 11. Trust, verification, and declared opacity ### 11.1 The principle Any time information is assigned high valence, it must be accompanied by the means through which trust in it can be verified. In an OBSERVER Folder this is structural: no observation object is valid without a trust field and a verify field, and no object reaches `2.0_VALENCE.md` except by a rule declared in `1.0_INDEX.md`. ### 11.2 Trust and verify together The two fields answer different questions and are both required: | Field | Question it answers | Checked against | |---|---|---| | trust | How was this value computed? | `1.0_INDEX.md` formula; outside citation | | verify | What was it computed from? | 4.0_RAW_DATA rows and columns | A reader who confirms both, and reruns the formula on the selected inputs, has replicated the value. ### 11.3 Declared opacity Some analyses rely on proprietary methods whose formulas are kept from public view. OBSERVER permits this, on one condition: the opacity MUST be stated explicitly. A proprietary formula is written as disclosure: proprietary with a statement of how it may be cited, in both `1.0_INDEX.md` and every trust field that uses it. It is through this explicit statement of opacity that other methods of trust become possible. A reader who cannot inspect a formula can still judge it by its record: how often values it elevated in the past proved to matter. Producers of proprietary Observers SHOULD provide a `track_record` (a summary, or a path to an OKF concept or file) giving historical hit rates of elevated values against later outcomes. What is never permitted is undisclosed opacity: a value presented as if its derivation were available when it is not. ### 11.4 Transparency levels `1.0_INDEX.md` declares one of two transparency levels: - **open:** every formula and relative observation is disclosed, and raw data is included or referenced. The Observer is fully white-boxed. - **declared_opacity:** at least one formula is proprietary or the raw data is withheld, and every such case is explicitly declared. ### 11.5 Relation to OKF trust fields OBSERVER's trust and verify fields describe each value. OKF's generated and verified fields describe who wrote and who confirmed a document. Both are useful and they do not overlap. Producers SHOULD record generated on every file and MAY record verified when a human or process has confirmed the report. A consumer derives an OKF trust tier (unverified, machine-confirmed, human-reviewed) from verified as OKF §5.3 describes. ## 12. Relationship to OKF The OBSERVER Format is an acknowledged variant of the Open Knowledge Format (OKF), version 0.2, put forward by Google. OKF defines a general format for knowledge bundles; OBSERVER specializes it for reporting quantitative analysis. Where this specification is silent, OKF's conventions are the intended default. ### 12.1 What OBSERVER inherits - Markdown files with YAML frontmatter as the only required technology. - A required type field on every concept document. - Progressive disclosure: an index that lets a reader see what is available before opening anything else. - The log format, the actor convention, ISO 8601 timestamps. - The provenance, trust, and lifecycle families: sources, generated, verified, status, `stale_after`. - Permissive consumption: unknown keys are preserved, never rejected. - Keyed rather than positional references, for the same reason OKF gives: agents rewrite these files, and positions shift. ### 12.2 Where OBSERVER differs | Aspect | OKF | OBSERVER | |---|---|---| | Structure | Free directory tree chosen by the producer | Fixed five-part hierarchy | | Unit | One concept per file | Many observation objects per file | | Index | `index.md`, optional, frontmatter-free | `1.0_INDEX.md`, required, carries the contract in frontmatter | | Log | `log.md`, optional | `1.1_LOG.md`, required; contract changes must be logged | | Ordering of content | Not prescribed | Prescribed by importance: valence before perception before raw data | | Reading order | Progressive disclosure through `index.md` | Reinforced by layer prefixes, navigation frontmatter, and entry files (§3.4) | | Computation | Attested Computation concept, optional | trust formula on every value, required | | Inputs | sources at the document level | verify on every value, to the row and column | | Opacity | Not addressed | Permitted only when declared (§11.3) | ### 12.3 Compatibility OBSERVER's reserved filenames are prefixed and uppercase, so they do not collide with OKF's lowercase `index.md` and `log.md`. Because every OBSERVER .md file, entry files included, carries a type, an OBSERVER Folder is also a conformant OKF v0.2 bundle, and it MAY be placed as a subdirectory inside a larger OKF bundle. An OKF consumer that knows nothing about OBSERVER will read its files as generic concepts; a consumer that recognizes the OBSERVER `*` types can apply the reading protocol in §14. An OBSERVER metric MAY also link to an OKF Attested Computation concept (OKF §10) from its `1.0_INDEX.md` entry, for producers who want run-level attestation in addition to OBSERVER's value-level verification. ## 13. Complexity: the Total Space of Information (TSI) This section is provisional and will be refined alongside the OBSERVER white paper. Observers can be placed within a simple space of complexity, which lets their scale be compared. This helps in designing agentic workflows, in assessing LLM performance at different scales of data, and, most practically, it gives a reader a headline figure for the compute budget that full verification would require. A reader can then weigh that cost against the value of the decision the analysis will inform. The Total Space of Information is two-dimensional: - **p (x-axis):** the parameter space, the number of columns or fields. - **n (y-axis):** the instance depth, the number of rows or objects. An Observer compresses an n × p raw space into a smaller n′ × p′ space at each step. TSI is reported at four points: | Layer | n | p | |---|---|---| | raw | Rows in 4.0_RAW_DATA | Columns in 4.0_RAW_DATA | | compute | Instances over which computation ran | Input columns used + metrics + relative observations + relative value | | perception | Objects in `3.0_PERCEPTION.md` | Metrics + relative observations + relative value | | valence | Objects in `2.0_VALENCE.md` | Relative observations that promoted them | ```yaml complexity: raw: { n: 120, p: 5 } compute: { n: 120, p: 9 } perception: { n: 24, p: 5 } valence: { n: 2, p: 2 } ``` In some Observers the compute and perception layers have the same magnitude; in many they differ, because the analysis computes intermediate values that are not reported. Where they differ, the gap tells a reader how much computation stands behind the reported perception, which is useful when checking the Python in trust fields. ## 14. How a consumer reads an OBSERVER Folder (informative) This section is informative, not normative. It describes the reading protocol the format is designed for. The entry files of §3.4.3 carry a condensed version of it, and the same text is suitable for an Agent Skill (§3.4.5). **Step 0: Orientation.** Whatever file an agent opens first, its navigation frontmatter states where it sits and points to `1.0_INDEX.md`. An agent that begins in a directory listing sees `1.0_INDEX.md` first; one that begins at `README.md` or `AGENTS.md` reads a short map ending at the same place. **Step 1: Relevance.** Read the frontmatter of `1.0_INDEX.md`: purpose, metrics, `report_date`, transparency, and complexity. Decide whether this Observer bears on the task. If not, stop. The cost is one small YAML block. **Step 2: Valence (white-box depth 0).** Read `2.0_VALENCE.md` in full. This is the answer to "What do I need to know?" For most tasks, this is where reading ends. When relaying a value, cite its id, so a downstream reader can find it. **Step 3: Context.** If the task needs more than the headline, for example how one subject compares with the others, read the relevant objects in `3.0_PERCEPTION.md`, using the object headings to jump directly. **Step 4: White-box depth 1, trust.** Confirm that each relied-upon object's `trust.formula` matches its metric's formula in `1.0_INDEX.md`, and that `relative_value` and elevated follow from the declared rules. Optionally, confirm the formula against its citation or other outside sources. For proprietary formulas, read the statement and any `track_record`. **Step 5: White-box depth 2, verify.** Confirm that the rows and columns named in verify exist in 4.0_RAW_DATA. Optionally, replicate the value by running the trust formula on those inputs. Read `verify.notes` for known data limitations. Following the raw data reading notice (§3.4.4), read only the selected rows and columns unless the user explicitly requests, or the workflow programmatically mandates, verification against the full data set. **Choosing a depth.** The right depth depends on what the answer will be used for. A casual question may stop at depth 0. A decision with financial, medical, or safety consequences warrants depth 2. The complexity block tells a consumer in advance roughly what depth 2 will cost. **Gating.** A consumer SHOULD warn when now >= `stale_after`, when `2.0_VALENCE.md` and `3.0_PERCEPTION.md` disagree, or when a trust formula does not match `1.0_INDEX.md`, and SHOULD say so rather than silently relaying the value. ## 15. Conformance An OBSERVER Folder is conformant with OBSERVER Format version 0.1 if: 1. `1.0_INDEX.md`, `1.1_LOG.md`, `2.0_VALENCE.md`, and `3.0_PERCEPTION.md` exist at the folder root under exactly these names, each with parseable YAML frontmatter carrying the type value in §4.1 and a navigation block conforming to §3.4.2. 2. `1.0_INDEX.md` contains every field marked REQUIRED in §5. 3. A 4.0_RAW_DATA file or directory is present, or `raw_data.availability` is referenced or withheld with the information §10.3 requires. 4. Every observation object in `3.0_PERCEPTION.md` and `2.0_VALENCE.md` has an id unique within its file and every field marked REQUIRED in §7, including trust and verify. 5. Every metric and `rel_ob` referenced by an object is declared in `1.0_INDEX.md`. 6. `2.0_VALENCE.md` is a subset of `3.0_PERCEPTION.md` as described in §8. 7. Every proprietary formula and every withholding of data is declared, and transparency reflects it (§11.4). 8. Every contract change is recorded in `1.1_LOG.md` (§6). 9. The raw data reading notice appears everywhere §3.4.4 requires. 10. Entry files, when present, are byte-identical to one another and state that they are identical copies (§3.4.3). Consumers: - MUST NOT reject a folder because of unknown keys, unknown files, or broken links. - SHOULD surface, rather than silently drop, a failed consistency check (§14, Gating). - SHOULD attempt best-effort reading of a non-conformant folder and report which rules it fails. ## 16. Versioning This document specifies OBSERVER Format version 0.1. Versions follow `.`: a minor version adds backward-compatible optional fields or conventions; a major version may rename required fields or reserved files. A folder declares its target version with `observer_version` in `1.0_INDEX.md`. Consumers that do not recognize the version SHOULD read on a best-effort basis. ## Appendix A: Worked example A weekly Observer over sales at twelve retail stores. The raw data holds ten weeks of revenue and transaction counts per store. The Observer computes two metrics per store and elevates the values that stand out against other stores this week or against the store's own recent history. (Statistics shown in observations are illustrative.) ``` weekly_sales_OBSERVER/ 1.0_INDEX.md 1.1_LOG.md 2.0_VALENCE.md 3.0_PERCEPTION.md 4.0_RAW_DATA.csv AGENTS.md README.md ``` ### 1.0_INDEX.md ```markdown --- type: OBSERVER Index navigation: layer: "1 of 4" entry: 1.0_INDEX.md down: 2.0_VALENCE.md note: > Layer 1 of 4: the contract for this Observer. The ranked highest-value results are in 2.0_VALENCE.md; report history is in 1.1_LOG.md. observer_version: "0.1" observer_id: weekly_sales title: Weekly Store Sales Observer purpose: > Identify the stores whose weekly revenue change or average ticket moved most unusually this week, relative to other stores and to their own trailing history. report_date: 2026-09-20 cadence: weekly transparency: open runtime: { language: python, version: ">=3.10", dependencies: [numpy] } raw_data: files: - path: 4.0_RAW_DATA.csv format: csv rows: 120 key: [store_id, week_ending] columns: - { name: week_ending, type: date, description: Last day of the sales week. } - { name: store_id, type: string, description: Store identifier, S01-S12. } - { name: region, type: string, description: Sales region. } - { name: revenue, type: number, description: Gross revenue for the week, USD. } - { name: transactions, type: integer, description: Completed transactions for the week. } availability: included reading_notice: > 4.0_RAW_DATA.csv: 120 rows × 5 columns. Trust verification with select columns and rows only. Full data set only intended for full reading when verification of values reported in 2.0_VALENCE.md and/or 3.0_PERCEPTION.md is explicitly requested by the user or programmatically mandated by the assigned workflow. metrics: - metric: wow_revenue_change description: Week-over-week change in revenue, as a fraction. formula: | def wow_revenue_change(revenue_t, revenue_t_minus_1): return (revenue_t - revenue_t_minus_1) / revenue_t_minus_1 disclosure: open utility: Detects abrupt shifts in store demand. citation: original inputs: [revenue] - metric: avg_ticket description: Average revenue per transaction, USD. formula: | def avg_ticket(revenue, transactions): return revenue / transactions disclosure: open utility: Separates changes in basket size from changes in footfall. citation: original inputs: [revenue, transactions] valence: relative_obs: - id: rel_ob_1 name: Cross-sectional z-score comparison: cross_sectional population: All 12 stores, same metric, same report_date. formula: | import numpy as np def rel_ob_1(value, peer_values): return (value - np.mean(peer_values)) / np.std(peer_values, ddof=1) disclosure: open citation: Standard score (z-score). - id: rel_ob_2 name: Temporal z-score comparison: temporal population: > Same store, same metric, the 8 report dates immediately before report_date. Missing weeks are not filled and do not extend the window; at least 6 values are required. formula: | import numpy as np def rel_ob_2(value, history): return (value - np.mean(history)) / np.std(history, ddof=1) disclosure: open citation: Standard score (z-score). relative_value: description: The larger absolute z-score of the two relative observations. formula: | def relative_value(rel_ob_1, rel_ob_2): return max(abs(rel_ob_1), abs(rel_ob_2)) disclosure: open elevation: rule: relative_value >= 2.0 order: descending complexity: raw: { n: 120, p: 5 } compute: { n: 120, p: 9 } perception: { n: 24, p: 5 } valence: { n: 2, p: 2 } generated: { by: observer_engine/0.1.0, at: 2026-09-21T06:00:00Z } stale_after: 2026-09-28T06:00:00Z --- # Weekly Store Sales Observer Surfaces the stores whose weekly revenue or ticket size moved most unusually this week, relative to other stores and to their own history. # Folder * [Valence](2.0_VALENCE.md) - Objects elevated this report, ranked. * [Perception](3.0_PERCEPTION.md) - All 24 computed values. * [Log](1.1_LOG.md) - Report history and contract changes. * [Raw data](4.0_RAW_DATA.csv) - 120 rows × 5 columns; select rows via `verify` fields (see `raw_data.reading_notice`). ``` ### 2.0_VALENCE.md ````markdown --- type: OBSERVER Valence navigation: layer: "2 of 4" entry: 1.0_INDEX.md up: 1.0_INDEX.md down: 3.0_PERCEPTION.md note: > Layer 2 of 4: the highest-value results, ranked. The rules that selected them are in 1.0_INDEX.md; every computed value is in 3.0_PERCEPTION.md. observer_id: weekly_sales report_date: 2026-09-20 elevation_rule: relative_value >= 2.0 object_count: 2 generated: { by: observer_engine/0.1.0, at: 2026-09-21T06:00:00Z } --- 1. [S03 average ticket rose to $45.00](#avg_tickets032026-09-20), far above its own recent range. 2. [S07 revenue fell 26.5% week over week](#wow_revenue_changes072026-09-20), the sharpest drop of any store. ## avg_ticket/S03/2026-09-20 ```yaml rank: 1 perception_ref: avg_ticket/S03/2026-09-20 id: avg_ticket/S03/2026-09-20 metric: avg_ticket subject: S03 report_date: 2026-09-20 value: 45.00 unit: USD observations: - { rel_ob: rel_ob_1, value: 2.21 } - { rel_ob: rel_ob_2, value: 3.40 } relative_value: 3.40 trust: disclosure: open formula: | def avg_ticket(revenue, transactions): return revenue / transactions verify: source: 4.0_RAW_DATA.csv rows: { store_id: S03, week_ending: 2026-09-20 } columns: [revenue, transactions] ``` ## wow_revenue_change/S07/2026-09-20 ```yaml rank: 2 perception_ref: wow_revenue_change/S07/2026-09-20 id: wow_revenue_change/S07/2026-09-20 metric: wow_revenue_change subject: S07 report_date: 2026-09-20 value: -0.2648 unit: fraction observations: - { rel_ob: rel_ob_1, value: -3.12 } - { rel_ob: rel_ob_2, value: -2.87 } relative_value: 3.12 trust: disclosure: open formula: | def wow_revenue_change(revenue_t, revenue_t_minus_1): return (revenue_t - revenue_t_minus_1) / revenue_t_minus_1 verify: source: 4.0_RAW_DATA.csv rows: { store_id: S07, week_ending: [2026-09-13, 2026-09-20] } columns: [revenue] args: revenue_t: { column: revenue, rows: { week_ending: 2026-09-20 } } revenue_t_minus_1: { column: revenue, rows: { week_ending: 2026-09-13 } } notes: > Temporal comparison (rel_ob_2) uses S07 rows for weeks ending 2026-07-19 through 2026-09-13. No gaps. ``` ```` ### 3.0_PERCEPTION.md (excerpt) ````markdown --- type: OBSERVER Perception navigation: layer: "3 of 4" entry: 1.0_INDEX.md up: 2.0_VALENCE.md down: 4.0_RAW_DATA.csv note: > Layer 3 of 4: every computed value. The contract is in 1.0_INDEX.md; the ranked highest-value results are in 2.0_VALENCE.md. 4.0_RAW_DATA.csv: 120 rows × 5 columns. Trust verification with select columns and rows only. Full data set only intended for full reading when verification of values reported in 2.0_VALENCE.md and/or 3.0_PERCEPTION.md is explicitly requested by the user or programmatically mandated by the assigned workflow. observer_id: weekly_sales report_date: 2026-09-20 object_count: 24 generated: { by: observer_engine/0.1.0, at: 2026-09-21T06:00:00Z } --- ## wow_revenue_change/S07/2026-09-20 ```yaml id: wow_revenue_change/S07/2026-09-20 metric: wow_revenue_change subject: S07 report_date: 2026-09-20 value: -0.2648 unit: fraction observations: - { rel_ob: rel_ob_1, value: -3.12 } - { rel_ob: rel_ob_2, value: -2.87 } relative_value: 3.12 elevated: true trust: disclosure: open formula: | def wow_revenue_change(revenue_t, revenue_t_minus_1): return (revenue_t - revenue_t_minus_1) / revenue_t_minus_1 verify: source: 4.0_RAW_DATA.csv rows: { store_id: S07, week_ending: [2026-09-13, 2026-09-20] } columns: [revenue] args: revenue_t: { column: revenue, rows: { week_ending: 2026-09-20 } } revenue_t_minus_1: { column: revenue, rows: { week_ending: 2026-09-13 } } notes: > Temporal comparison (rel_ob_2) uses S07 rows for weeks ending 2026-07-19 through 2026-09-13. No gaps. ``` ## wow_revenue_change/S01/2026-09-20 ```yaml id: wow_revenue_change/S01/2026-09-20 metric: wow_revenue_change subject: S01 report_date: 2026-09-20 value: 0.0210 unit: fraction observations: - { rel_ob: rel_ob_1, value: 0.44 } - { rel_ob: rel_ob_2, value: 0.61 } relative_value: 0.61 elevated: false trust: disclosure: open formula: | def wow_revenue_change(revenue_t, revenue_t_minus_1): return (revenue_t - revenue_t_minus_1) / revenue_t_minus_1 verify: source: 4.0_RAW_DATA.csv rows: { store_id: S01, week_ending: [2026-09-13, 2026-09-20] } columns: [revenue] args: revenue_t: { column: revenue, rows: { week_ending: 2026-09-20 } } revenue_t_minus_1: { column: revenue, rows: { week_ending: 2026-09-13 } } ``` ```` ### 4.0_RAW_DATA.csv (excerpt) ```csv week_ending,store_id,region,revenue,transactions 2026-09-13,S03,West,66120,1740 2026-09-13,S07,Central,84200,2105 2026-09-20,S03,West,72450,1610 2026-09-20,S07,Central,61900,1590 ``` A reader verifying rank 2 at depth 2 selects the two S07 rows, reads revenue, and runs the trust formula: (61900 - 84200) / 84200 = -0.2648. The reported value is confirmed. ### 1.1_LOG.md ```markdown --- type: OBSERVER Log navigation: layer: "1.1 (history of layer 1)" entry: 1.0_INDEX.md up: 1.0_INDEX.md note: > Report history and contract changes for this Observer. The current contract is in 1.0_INDEX.md. --- # Observer Log ## 2026-09-20 * **Report**: 24 perception objects; 2 elevated to valence. * **Data**: 12 rows added (week ending 2026-09-20); no rows removed; columns unchanged (5). ## 2026-07-19 * **Initialization**: Observer created with 12 stores and two metrics. ``` ### README.md and AGENTS.md (identical) ```markdown --- type: OBSERVER Entry navigation: layer: entry entry: 1.0_INDEX.md down: 1.0_INDEX.md note: > This folder is an OBSERVER Folder. Its contract and folder map are in 1.0_INDEX.md. README.md and AGENTS.md are identical copies; reading one makes the other unnecessary. --- # Weekly Store Sales Observer This folder is an OBSERVER Folder: a quantitative analysis arranged so the most important results come first, each with its formula and its raw inputs attached. Specification: . ## Reading order 1. `1.0_INDEX.md`: what this Observer measures and the rules that decide which results matter. Its frontmatter alone is enough to judge relevance. 2. `2.0_VALENCE.md`: the highest-value results, ranked. For most tasks, reading can stop here. 3. `3.0_PERCEPTION.md`: every computed value, for context. 4. `4.0_RAW_DATA.csv`: 120 rows × 5 columns. Trust verification with select columns and rows only. Full data set only intended for full reading when verification of values reported in 2.0_VALENCE.md and/or 3.0_PERCEPTION.md is explicitly requested by the user or programmatically mandated by the assigned workflow. `1.1_LOG.md` holds the report history and any changes to the rules. README.md and AGENTS.md in this folder are identical copies. Reading one makes the other unnecessary. ``` The OBSERVER Format was originally developed by Ian Carleton and is released as an open standard. It welcomes further development from stakeholders at every level. For communication and clarification, contact software@inc-research.com.