Pitfalls
Nine ways to produce a folder that passes conformance and still misleads a reader. Interrogate your instance of analysis for foreseeable misalignment with the OBSERVER standard's transparency goals, most failure modes will surface for consumers of OBSERVER structures through the embedded trust and verify mechanisms. This failure of trust and verify reflects poorly on the producer of the OBSERVER structure and should be avoided.
1. Forward-filling a gap
The defect. A discontinued index kept "reporting" a slowly decaying year-over-year change for months after its source stopped publishing, because the build carried the last level forward and divided a frozen numerator by a moving denominator. Nothing looked broken. The number simply drifted toward zero, and was quoted.
The rule. Never fill a gap — not forward, not backward, not by interpolation — unless the analysis is explicitly about imputation, in which case the method is a declared metric of its own. A missing month is a blank row. Every window function must be gap-aware: shifting twelve rows of a complete monthly index is correct only if absent months are present as empty rows.
The check. Find a series with a known gap and confirm the derived value around it is blank, not smooth.
2. Rebasing to "the first month available"
The defect. An index anchored at whatever the earliest date in the window happened to be. Every time the window grew, the whole history changed, and last month's published chart no longer matched this month's.
The rule. Declare the anchor as a constant — base='2018-01' — inside the published formula. If you must move it, that is a **Contract** entry in the log and a new metric name, not a silent shift.
3. Publishing a formula that is not the one you ran
The defect. Someone edits an engine function and not the formula string, or the string and not the function. The folder now documents a computation nobody performed, and it will validate perfectly.
The rule. Gate the build. Before writing anything, evaluate every published formula against the raw table and require agreement with the engine's value to a stated tolerance, across the whole range you publish, not just the report date. Raise rather than write. Then ship the shared helper module the formulas call, copied verbatim from the engine's own source, so the folder is runnable on its own.
The check. Change a coefficient in a published formula string and confirm the build fails, then revert that same coefficient to correct and conformant value.
4. Inferring the inventory from what is on disk
The defect. A fetch quietly failed, the series vanished from the input directory, and the report was published with one fewer input and no mention of it. The analysis "worked" because nothing declared what should have been there.
The rule. Declare what you expect — the series, the subjects, the date range — separately from what you have, and fail closed when a required input is missing, empty or stale. This belongs in the build, not in the folder: a report that would be missing an input is not published.
5. Explaining a gap in the wrong place
The defect. A known source outage was mentioned once, in the folder's introduction. Objects whose inputs actually lacked that month carried no note, and objects unaffected by it were read as suspect.
The rule. Put the gap in verify.notes of exactly the objects whose inputs lack the data. A primary-source citation for the gap (the publisher's notice of an outage, say) is optional, and worth adding where it helps a reader trust the explanation. Remember the horizon: a twelve-month change is affected for twelve months after the missing month, not only in it.
6. Positional references
The defect. A verify field that pointed at "rows 100–112". The file was re-sorted, and the reference silently pointed at different data.
The rule. Select rows by the declared key columns, always. The same applies to object ids: <metric>/<subject>/<report_date>, never an index.
7. Refitting a model in place
The defect. A model was refitted on all available data each run, and its "out-of-sample" performance was reported from the same rows it had just been fitted on. The published skill was an artifact of the refit.
The rule. Freeze the model. Fit once, on a window that ends before the first period you publish, and keep the training boundary clear of the forecast horizon — a six-month target leaks six months. The fitted coefficients belong in the published formula as constants, which is what makes each value replicable. A refit is a new, announced model version with a Contract log entry, never an edit.
8. Direction conventions left implicit
The defect. Two exchange rate series were quoted in opposite directions — foreign-per-USD and USD-per-foreign — and were combined without inverting one. The published figures had the wrong sign for years, and nobody noticed, because "the dollar strengthened" and "the dollar weakened" are both plausible sentences.
The rule. Fix one convention per family of metrics, state it in the metric's description and utility ("positive = stronger dollar", "positive = wages winning"), and normalize at the edge of the pipeline where the data arrives. Sign conventions are the cheapest thing to document and the most expensive thing to get wrong.
9. Non-deterministic builds
The defect. A generated.at of now() meant every rebuild produced a different folder, so nobody could tell a real change from a rerun, and a "rebuild and diff" check was useless.
The rule. generated.at follows OKF: it records when the build ran, so it legitimately changes on every rebuild. Capture it once per build, write the same value into every file, and exclude it when diffing one build against another. Everything else must be derived from the inputs and sorted with an explicit key. Rebuilding from the same inputs must produce the same bytes apart from generated.at, which is what makes a diff of the published folder a meaningful audit trail.
Two smaller ones
Zero instead of null. When a comparison cannot be computed, relative_value is null, not 0. A zero sorts as "perfectly ordinary" and is indistinguishable from a real result.
A valence layer that is a field dump. Opening sentences in 2.0_VALENCE.md are optional, but where a producer includes them they are what most readers will quote. Write them as findings — what moved, how much, against what — not as a restatement of the YAML beneath.