Having controls is not the same as having effective controls. CommunityPay measures whether its automated enforcement mechanisms actually produce appropriate human responses. This page describes the control effectiveness measurement system — what it measures, how, and why it matters.
The Measurement Gap
Most accounting software (and most SOC reports) answer the question: "Do controls exist?"
CommunityPay answers a harder question: "When controls fire, do people respond appropriately?"
This distinction matters because:
- A control that fires but is never acknowledged is effectively absent
- A control that fires and is always overridden may indicate the control is miscalibrated
- A control that fires and is resolved within SLA indicates operational governance
What Gets Measured
Response Time Metrics
When an escalation is triggered, the clock starts. The control effectiveness service measures, over acknowledged escalations:
| Metric | Definition |
|---|---|
| Mean response time | Average time from escalation to first acknowledgment |
| Median response time | 50th percentile response time (less affected by outliers) |
| p95 response time | 95th percentile — captures the tail (slow responses) |
These metrics are computed from escalation event records. Each event records when it was triggered and when it was acknowledged; response time is the difference.
Acknowledgment Rate
The share of escalations that receive an explicit acknowledgment from the responsible party.
Acknowledgment Rate = Acknowledged or Resolved Escalations / Total Escalations
A declining acknowledgment rate is itself a risk signal — it suggests that responsible parties may be ignoring governance notifications.
SLA Compliance
An escalation is owed an acknowledgment within 48 hours of its notification. A notified escalation counts as met when it was acknowledged within 48 hours, or when it is still unacknowledged and the 48 hours have not run out. It counts as a breach when it was acknowledged later, or is still unacknowledged after 48 hours:
SLA Compliance = Met Escalations / (Met + Breached Escalations)
A period in which no escalation was notified has no rate.
Control Coverage
For each exclusion triggered in the period, the service counts whether it produced a notification, whether it produced an escalation, and whether it ended by decision, lifted or overturned. Notification coverage, escalation coverage, and the resolution rate follow. A period in which no exclusion was triggered has no coverage rate and no resolution rate. It does not have perfect ones.
Override Pattern Analysis
Overrides are legitimate — they exist for situations where guards correctly identify a condition but authorized personnel determine the operation should proceed anyway. However, override patterns reveal governance health:
By scope:
| Scope | Meaning |
|---|---|
| CLOSED_PERIOD | Post into a closed period, within a named date range and fiscal year |
| FUND_SEG | Move cash across a fund boundary, for one named fund |
| YEAR_CLOSE | Close a fiscal year that does not balance |
| INTEGRITY_GATE | A recordable scope for proceeding past an integrity block. No control consults it today, because integrity findings do not stop payments |
| VENDOR_RISK | Pay a vendor with a critical compliance failure. Bound to one bill and one amount, and used up by one posting |
| DISBURSE_COMP | Pay outside the state disbursement rules. Bound to one bill and one amount, and used up by one posting |
By use:
| Measure | What it counts |
|---|---|
| Overrides authorized | Every override written in the period, grouped by scope |
| Times used | Every recorded use, grouped by the scope of the override used |
| Active and expired | How many are still usable, and how many are not: past their expiry, or deactivated, which includes an override already used up |
| Override rate | Recorded uses divided by all journal entries dated in the period, in any status |
| Most-cited reasons | The five justification texts written most often |
Writing an override takes a fresh authenticator code. An override records the control it suspends, the person who authorized it, the time, and a written reason of at least twenty characters. It records no authorization tier, so the register cannot be split into board-authorized and administrative overrides; what it holds is which control was suspended, how often, and on whose authority. For VENDOR_RISK and DISBURSE_COMP — the two that stand between association money and an external bank account — it also names the one bill and one amount the override authorizes, and the entry that used it up.
Escalation Chaining
When an alert is not acknowledged, it escalates up a chain of rules. Each escalation rule names its recipient and its waits. By default a rule fires after three days without action, and the next rule in the chain fires two days after that.
The report counts escalations whose status is chained, beside total escalations. A high share of chained escalations suggests that first responders are not engaging with governance alerts, which is itself a control effectiveness concern.
Measurement Sources
All control effectiveness metrics are derived from recorded source data:
| Source | Mutability | What It Provides |
|---|---|---|
| EscalationEvent | Identity fixed at creation; status and the notification, acknowledgment, resolution, and chaining fields are written as it progresses | Alert creation, notification, acknowledgment, resolution timestamps |
| ExclusionNotificationEvent | Immutable | Notification delivery, success/failure |
| ExclusionStatusHistory | Append-only | Exclusion status transitions with timestamps |
| AuditOverride | Written once with its reason; its active and expiry state change | Override scope, who authorized it and when, the bill and amount for a bill-bound override |
| AuditOverrideUsage | Immutable | Every recorded use of an override |
| EnforcementDecision | Immutable | Guard results, decision outcomes |
A CPA reviewing control effectiveness metrics can verify them against the raw event data.
Governance Controls Attestation Integration
The Governance Controls Attestation (GCA) reports control pass rate, override, exclusion, and vendor standing statistics for a period, with an approval-latency figure: how long an item waited at an approval level against that level's threshold. That figure is not the 48-hour escalation rate above. Its overall assessment is determined as follows:
| Assessment | Criteria |
|---|---|
| STRONG | Controls pass rate of 95% or more, and no high-severity risk flag |
| ADEQUATE | Controls pass rate of 85% or more |
| NEEDS_IMPROVEMENT | Controls pass rate of 70% or more |
| DEFICIENT | Any critical risk flag, checked first, or a controls pass rate below 70% |
| NOT MEASURED | No control activity was recorded in the period and no critical risk flag was raised |
A period with no control activity earns neither STRONG nor ADEQUATE; it is NOT MEASURED unless a critical risk flag makes it DEFICIENT. Boards receive quantified metrics derived from source records, not a subjective assessment.
Effectiveness Score
The dashboard also reports four component scores and an overall score. The override score falls linearly from 100 at no overrides to 0 at a 10 percent override rate. The response score is 60 percent acknowledgment rate and 40 percent SLA rate. The coverage score is the notification coverage. The control plane score starts at 100, 70, or 30 for a green, yellow, or red health status, less 10 points for each breached threshold. The overall score weights override discipline at 20 percent, response at 30 percent, coverage at 30 percent, and the control plane at 20 percent.
A component with nothing to measure is dropped, not scored. Dropping one re-weights the rest, so the overall score is issued only while the measured components carry more than half of the weight.
Control Plane Health
In addition to governance-level metrics, CommunityPay records a control plane health snapshot for each association:
| Metric | What It Measures |
|---|---|
| Controls total, active, and shadow | How many controls are in the catalog, and how many enforce |
| Orphan count | Journal entries without a decision (must be 0) |
| Persist errors | Decisions that failed to persist in the window |
| Block count and error decisions | Decisions by outcome in the window |
| Flow and guard breakdown | Decisions by flow, and runs, passes, and fails by guard |
| Health status and threshold breaches | Green, yellow, or red, and the names of the thresholds breached |
These technical metrics show whether guards are being invoked, whether decisions are persisting, and whether any entry escaped the enforcement path.
What This System Answers
The control effectiveness measurement system answers questions that sit above the technical control layer:
- "Are our controls monitored?" — Response time and acknowledgment metrics show whether alerts reach people and people respond
- "Are issues resolved promptly?" — SLA compliance shows whether escalations are acknowledged within 48 hours
- "Are overrides appropriate?" — Override pattern analysis shows which controls were suspended, how often, and on whose authority
- "Is governance improving or degrading?" — Daily trends in escalations, response rate, and override use show the direction of governance health
- "Can we show all of this?" — All metrics are derived from source records that an auditor can independently verify
CARI Integration
The CARI Enforcement Integrity sub-score, which carries a 15% weight in the composite CARI score, reads two signals measured near these: the enforcement block rate and approval SLA breaches over the trailing twelve months. The escalation response and override measures on this page do not enter the CARI score.
For published methodology and component weights, see CARI Methodology and Scoring Framework.
How CommunityPay Enforces This
- Response time metrics captured: mean, median, and p95 for escalation acknowledgment
- Acknowledgment rate tracked: percentage of alerts acknowledged by responsible parties
- SLA compliance measured: percentage of notified escalations acknowledged within 48 hours
- Override patterns analyzed: by scope (closed period, fund segregation, year-end close, integrity gate, vendor risk, disbursement compliance) and by how often each is used
- Escalation chaining tracked: how often escalations propagate up the chain
- All metrics derived from recorded source data — escalation events, notification events, override records and their uses — with each event's identity fixed at creation
- A period with nothing to measure reports no rate, never a perfect one
- Writing an override takes a fresh authenticator code, and a vendor or disbursement override is bound to one bill and one amount