Document Type: Framework
Status: Active
Version: v1.2
Authority: Experimentation Brain
Parent: Experimentation Brain Architecture
Applies To: Ads Brain, Affiliate Brain, Conversion Brain, Research Brain, Data Brain
Last Reviewed: 2026-04-23
Purpose
Defines how MWMS determines whether experiment results contain sufficient signal strength to support decision-making.
The framework ensures decisions are based on:
reliable signal patterns
adequate observation volume
consistent behavioural response
interpretable variance levels
validated underlying data
rather than:
premature conclusions
emotional reactions to early data
platform-optimisation noise
random fluctuation
invalid or corrupted measurement data
Core Principle
Signal strength must exceed noise strength.
AND
👉 signal validity must be confirmed before signal strength is evaluated
Experiments must produce:
• valid data
• stable signals
• interpretable patterns
before conclusions are drawn.
🔴 NEW — Data Validity Dependency Rule
Statistical confidence depends on measurement integrity.
Confidence must not be evaluated if:
• conversion tracking is incorrect
• events are duplicated
• key events are missing
• attribution is inconsistent
• data is unvalidated
If data integrity fails:
→ confidence = invalid
Confidence Components
(UNCHANGED CORE — still strong)
🔴 NEW — Confidence Pre-Check Layer
Before evaluating confidence, all experiments must pass:
1. Measurement Integrity Check
• events firing correctly
• no duplication
• no missing data
2. Data Trust Check
• data validated
• stable over time
• consistent across systems
3. Attribution Reliability Check
• attribution understood
• no major platform conflicts
• directional consistency confirmed
If any check fails:
→ experiment results are not usable
Sample Size Requirement
(UNCHANGED CORE)
Signal Stability Requirement
(UNCHANGED CORE)
Multi-Metric Alignment Rule
(UNCHANGED CORE)
🔴 NEW — Cross-Metric Validation Rule
Metrics must not be evaluated in isolation.
Confidence increases when:
• multiple metrics align
• funnel progression remains logical
• ratios remain stable
Example:
CTR ↑
conversion rate ↑
CPA ↓
Misalignment indicates:
• data issues
• audience mismatch
• attribution distortion
Behavioural Coherence Requirement
(UNCHANGED CORE)
Observation Window Discipline
(UNCHANGED CORE)
Early Signal Distortion Risk
(UNCHANGED CORE)
🔴 NEW — Attribution Distortion Awareness
Experiment results may be distorted by attribution issues.
Examples:
• platform over-crediting conversions
• cross-channel influence hidden
• delayed conversions misattributed
Confidence must consider:
→ attribution reliability level
Low attribution confidence:
→ reduces experiment confidence
Minimum Signal Threshold Guidelines
(UNCHANGED CORE)
Platform Optimisation Distortion Awareness
(UNCHANGED CORE)
🔴 NEW — Data Consistency Requirement
Signals must be consistent across:
• time
• segments
• platforms
Inconsistent signals indicate:
• instability
• measurement issues
• segmentation distortion
Confidence Interpretation Outcomes
(UNCHANGED CORE but strengthened)
🔴 NEW — Confidence Blocking Condition
Even if signal appears strong:
If any of the following exist:
• data duplication
• missing events
• attribution conflict
• unstable signals
→ confidence must be downgraded or blocked
🔴 NEW — Confidence Levels (Updated)
High Confidence
• validated data
• stable signals
• aligned metrics
• consistent attribution
→ Safe for scaling
Moderate Confidence
• minor inconsistencies
• partial validation
• directional trend
→ Continue testing
Low Confidence
• volatility
• insufficient data
• misaligned metrics
→ Do not scale
Invalid Confidence (NEW)
• broken tracking
• duplicate data
• attribution conflict
• unvalidated signals
→ Experiment must be revalidated before any decision
Relationship to Experiment Decision Layer
Confidence assessment influences:
• scale decisions
• iteration decisions
• experiment extension decisions
• variable refinement decisions
🔴 NEW — Decision Gate Rule
No experiment may scale unless:
• Measurement Integrity = passed
• Data Trust = passed
• Attribution Reliability = acceptable
• Statistical Confidence = high
If any fail:
→ scaling is blocked
Relationship to Other MWMS Frameworks
Supports:
• Experimentation Brain Paid Media Experiment Framework
• Ads Brain Creative Testing Structure Framework
• Data Brain Measurement Integrity Framework
• Data Brain Data Trust Framework
• Data Brain Attribution Reliability Framework
Failure Modes Prevented
false positives from early signals
false negatives from volatility
scaling on invalid data
misinterpreting attribution distortion
platform bias influencing decisions
trusting experiments built on broken measurement
Drift Protection
The system must prevent:
confidence being assigned without validation
early signal overreaction
platform-driven bias
signal instability being ignored
measurement changes affecting results unnoticed
Architectural Intent
MWMS relies on cumulative learning across experiments.
False learning signals degrade system intelligence.
Confidence discipline ensures MWMS evolves based on:
verified signal patterns
repeatable behavioural effects
stable measurement environments
Final Rule
If data is not valid:
→ confidence is not valid
If confidence is not valid:
→ decision must not proceed
Change Log
Version: v1.2
Date: 2026-04-23
Author: Experimentation Brain
Change:
Upgraded framework to include:
• data validity dependency
• confidence pre-check layer
• cross-metric validation
• attribution distortion awareness
• confidence blocking conditions
• invalid confidence classification
• decision gate rule
Change Impact Declaration
Pages Created:
None
Pages Updated:
Experimentation Brain Statistical Confidence Framework
Pages Deprecated:
None
Registries Requiring Update:
No
Canon Version Update Required:
No
Change Log Entry Required:
Yes