Incident Response Playbook

Severity Classification · Escalation Paths · Runbooks · Communication · Post-Incident Review

Operational reference for the on-call team, DevOps engineers, compliance officers, and executive leadership during production incidents. Every person on the escalation chain must have read this document before their first on-call rotation.

The platform is operated by Groovy Company, Inc. The trading venue is CEDEX. The qualified-custody and onboarding anchor is Empire Stock Transfer. Production runbooks cover all three production modules: Equities (Module 1), Real Estate (Module 2), and CORECM — Carbon Ore, Rare Earth, and Critical Minerals (Module 3).

This document describes what must happen at each phase of incident response, why each step is required, and how each step is verified. Specific commands, scripts, and runbook automation are maintained in the platform's internal operations runbooks and are accessible to authorized on-call personnel under documented chain-of-custody.

Classification: Internal — authorized personnel only.

Incident Command Structure
Severity Classification
Detection and Alerting
General Response Procedure
Runbook: Custody Discrepancy (P0)
Runbook: Smart Contract Exploit (P0)
Runbook: Oracle Failure (P1/P2)
Runbook: Module 2 NAV Oracle Failure (P1)
Runbook: Module 3 Classification Oracle Failure (P2)
Runbook: RPC Infrastructure Degradation (P1/P2)
Runbook: Circuit Breaker Activation (P2)
Runbook: Regulatory Freeze — Control 42 (P0)
Runbook: Module 3 Federal-Action Freeze (P0)
Runbook: Key Compromise (P0)
Communication Protocol
Post-Incident Review
Escalation Contact Sheet
Tabletop Exercise Schedule

1. Incident Command Structure

1.1 Roles

RolePersonResponsibility

Incident Commander (IC)

Chief Technology Officer (Frank Yglesias)

Full authority during declared incident. Decides severity, coordinates response, authorizes containment actions.

Technical Lead

On-call senior engineer

Root cause investigation, remediation execution, technical diagnostics

Compliance Lead

Legal Counsel (JDT Legal)

Regulatory notification decisions, SAR filing assessment, Control 42 authorization, federal-action coordination (Module 3)

Empire Liaison

Empire Stock Transfer (Patrick Mokros, Founder)

Empire coordination — custody issues, onboarding impacts, qualified-custodian functions

Communications

Designated by IC at incident declaration

External communications — status page, investor notifications, issuer notifications

DevOps Lead

Platform DevOps lead

Infrastructure operations, oracle relay services, monitoring

1.2 War Room Activation

Upon any P0 or P1 incident declaration:

PagerDuty fires alert to IC, Technical Lead, and Compliance Lead.
IC acknowledges within 5 minutes (P0) or 15 minutes (P1).
IC opens a dedicated incident channel for the responding team.
All responders join the channel within 15 minutes of declaration.
IC assigns roles, opens the relevant runbook, and announces severity.
Status updates posted to the incident channel every 15 minutes (P0) or 30 minutes (P1).

1.3 Decision Authority

ActionAuthorityAdditional Approval

Declare P0 or P1

IC, or on-call engineer (provisional)

IC confirms within 15 minutes

Activate Control 42 (regulatory freeze)

Legal Counsel plus 3-of-5 multi-signature

None — immediate execution

Emergency program upgrade

IC plus 5-of-9 multi-signature

Certora re-verification required post-execution

Notify SEC or FinCEN

Legal Counsel

IC concurrence

Public disclosure

IC plus Communications role

Legal Counsel review

Rollback to previous program version

IC plus 5-of-9 multi-signature

Post-execution audit attestation

Coordinate Module 3 federal-action freeze

Legal Counsel plus IC

Empire notification mandatory

If the IC is unreachable within 15 minutes of a P0 page, the on-call senior engineer assumes command provisionally and is responsible for engaging Legal Counsel and the Empire liaison while continuing escalation attempts to the IC.

2. Severity Classification

SeverityDefinitionResponse SLAEscalationExamples

P0 — Active Exploit or Critical Failure

Funds at risk, control bypass confirmed, custody discrepancy, key compromise, federal-action requirement

15 minutes

IC plus all leads plus multi-signature holders

Custody discrepancy detected (Error 6001); Transfer Hook bypass; unauthorized fund extraction; key compromise; federal-action freeze required (Module 3)

P1 — Service Degradation

Trading halted or severely degraded; oracle stale beyond threshold; CEDEX API down

1 hour

IC plus Technical Lead plus DevOps

Custody oracle stale beyond 5 slots; OFAC oracle stale beyond 48 hours; CEDEX 5xx error rate above 10%; RPC failover triggered; NAV oracle stale (Module 2)

P2 — Partial Failure

Non-critical oracle degraded; single service impacted; monitoring alert

4 hours

On-call engineer

AML provider timeout (cached serving); TWAP stale beyond 5 minutes; EDGAR batch failed; Classification oracle stale (Module 3); single relay crash

P3 — Monitoring Alert

Threshold warning; unusual pattern; non-exploitable anomaly

24 hours

On-call engineer

Volume spike detection; latency increase; disk space warning; certificate expiry approaching

Severity Escalation Rules

ConditionEscalation

P2 unresolved after 4 hours

Escalate to P1

P1 unresolved after 4 hours

Escalate to P0

Any P0 involving funds

IC plus Legal Counsel plus Empire plus all multi-signature holders

IC unreachable within 15 minutes

On-call senior engineer assumes provisional command

Module 3 federal action received

Automatic P0 regardless of operational impact

3. Detection and Alerting

3.1 Automated Detection

MonitorToolCheck IntervalP0 TriggerP1 TriggerP2 Trigger

Custody oracle freshness

Datadog

10 seconds

Discrepancy detected

Slot age above 5

Slot age above 3

OFAC oracle staleness

Datadog

1 minute

—

Age above 48 hours

Age above 24 hours

AML provider availability

Datadog

1 minute

—

Both providers down beyond 30 minutes

One provider down

CEDEX API error rate

Datadog

30 seconds

—

5xx rate above 10% for 5 minutes

5xx rate above 5%

RPC latency

Datadog

10 seconds

—

Primary plus fallback both down

Primary down (failover active)

Transfer Hook error spike

Datadog

1 minute

Unexpected Error 6001 (custody)

Error rate above 5%

Error rate above 1%

Global Pool balance

Custom

Every block

Balance decreased

—

Multi-signature transactions

Datadog

Real-time

Unauthorized upgrade attempt

—

Solana network

Helius

30 seconds

—

Network halt beyond 5 minutes

TPS below 500

NAV oracle freshness (per Module 2 mint)

Datadog

1 minute

—

Reappraisal age beyond nav_reappraisal_max_age_secs

Reappraisal age above 50% of nav_reappraisal_max_age_secs

Classification oracle freshness (per Module 3 mint)

Datadog

5 minutes

Federal-action notification received

Classification age above 48 hours

Classification age above 12 hours

Module 3 federal-action feed

Custom

5 minutes

New EO, Section 232 action, or DPA Title III order matching basin

—

3.2 Alert Routing

SeverityPagerDuty PolicyNotification

Critical-severity policy: phone call plus SMS plus dedicated channel plus email

IC plus all leads plus multi-signature holders

High-severity policy: SMS plus dedicated channel plus email

IC plus Technical Lead plus DevOps Lead

Medium-severity policy: dedicated channel plus email

On-call engineer

Low-severity policy: dedicated channel only

On-call engineer (next business day)

3.3 Manual Reporting

Anyone can declare an incident through any of the following channels:

PagerDuty manual trigger
Dedicated incident channel with structured command
Email to incidents@rwatokens.net
Direct phone call to the IC line (P0 only — when other channels are insufficient)

4. General Response Procedure

Every incident follows this seven-step framework. Specific runbooks (Sections 5 through 14) provide detailed procedures for common incident types.

Step 1 — Detect

Automated alert fires or manual report received. On-call engineer acknowledges within the SLA for the declared severity.

Step 2 — Triage (first 15 minutes)

Confirm the incident is real (not a false positive from a known monitoring artifact).
Classify severity (P0 / P1 / P2 / P3).
Identify affected systems, mints, and wallets.
Estimate blast radius — how many users or issuers are affected, across which modules.
Open the incident channel and page the IC if severity is P0 or P1.

Step 3 — Contain (immediate)

Stop the bleeding — prevent further damage.
Activate Control 42 (regulatory freeze) if funds are at risk.
Disable affected services if necessary.
Preserve evidence — capture logs, on-chain state snapshots, oracle attestation history.
Notify Empire Stock Transfer if the incident is custody-related.

Step 4 — Investigate

Conduct full on-chain forensics — transaction trace from suspected wallet or mint.
Audit oracle state — attestation history, slot ages, ed25519 verification records.
Analyze logs — relay services, RPC, CEDEX backend.
Review smart-contract behavior to identify any vulnerability or bypass path.
Reconstruct the timeline (who, what, when, how).

Step 5 — Remediate

Class of IssueRemediation

Program vulnerability

Emergency upgrade with 5-of-9 multi-signature plus Certora re-verification

Oracle issue

Rotate keys, verify backup consensus

Key compromise

Emergency key rotation with appropriate multi-signature threshold

Infrastructure

Restore, fail over, scale

External (Solana network, Helius)

Wait, communicate, monitor recovery

Step 6 — Recover

Verify the fix is in place via on-chain confirmation.
Confirm 2-of-3 oracle consensus on integrity (custody-affected incidents).
Lift Control 42 freeze if it was activated.
Resume CEDEX trading.
Verify monitoring shows healthy state across all dashboards.
Notify stakeholders that the incident is resolved.

Step 7 — Post-Mortem (within 72 hours)

Conduct root cause analysis, reconstruct the timeline, review control effectiveness, verify remediation, identify process improvements, and prepare any required public disclosure.

5. Runbook: Custody Discrepancy (P0)

Trigger: CustodyOracle detects that the Empire-attested Common Class B share balance does not match the on-chain ST22 token supply. Error 6001 returned on every transfer for the affected mint. Applies across all modules — custody integrity is the foundational invariant for the platform.

Severity: P0 — automatic. No triage required.

Immediate Actions (0 to 15 minutes)

Confirm the discrepancy is real. Query the platform's custody oracle endpoint for the affected mint. Verify that discrepancy_detected is true and capture the reported common_b_balance and token_supply values along with the delta.
Confirm transfers are halted. Control 1 automatically rejects every transfer for this mint with Error 6001. No manual containment action is required. Verify by attempting a test transfer and confirming the rejection.
Notify Empire Stock Transfer within the 15-minute SLA. Contact Empire's compliance hotline by phone and follow up via email. Provide the mint address, the oracle's reported balance and supply, and the delta.
Notify the IC if not already paged via PagerDuty, and open the incident channel.
Preserve evidence. Snapshot the CustodyOracle account state, record the current Solana slot and timestamp, and capture the Datadog custody dashboard at the moment of detection.

Investigation (15 to 60 minutes)

Determine which of the five possible root causes applies:

Possible CauseIndicatorResolution Path

Empire custody change not yet reflected on chain

Recent Empire-side custody event with relay lag

Wait for relay to catch up; verify cadence within next slot

Unauthorized minting

Recent mint instruction not authorized by Empire

Escalate to Legal Counsel; SEC notification assessment required

Token burn without corresponding custody withdrawal

Burn instruction on-chain not matched by Empire-side withdrawal

Should never happen by design; escalate as P0 architectural issue

Oracle relay malfunction (Ed25519 signature issue)

Relay logs show signature failures

Restart relay; rotate keys if signature problem persists

Empire API returning incorrect data

Empire confirms wrong data was served

Empire corrects records; relay pushes update

Coordinate directly with Empire to reconcile actual share counts against the Master Securityholder File. Empire is the authoritative source for Common Class B balances; any reconciliation defers to Empire's records.

Conduct on-chain forensics by examining the recent transaction history for the affected mint, looking for unexpected mint or burn instructions and any program upgrade transactions on the Transfer Hook.

Resolution

Once Empire and the platform agree on the correct values, push an updated attestation through the custody relay.
Verify 2-of-3 oracle consensus: primary (Empire Ed25519 attestation) matches supply; secondary (platform verification node) confirms the match; tertiary (most recent quarterly audit record) is consistent.
Confirm that discrepancy_detected returns to false and a test transfer succeeds.
Verify monitoring dashboards return to normal and notify stakeholders that the incident is resolved.

Post-Resolution

Conduct the post-mortem within 72 hours. If the SAR criteria are met (suspicious activity at or above the regulatory threshold), Legal Counsel assesses filing. If the event is material, SEC notification is initiated within 24 hours of confirmation per the platform's compliance policy.

6. Runbook: Smart Contract Exploit (P0)

Trigger: Unauthorized state change, fund extraction, control bypass, or unexpected error pattern indicating an exploitable vulnerability.

Severity: P0 — immediate.

Immediate Actions (0 to 15 minutes)

Activate Control 42 — regulatory freeze on the affected mint(s). Requires Legal Counsel authorization plus 3-of-5 multi-signature; executes immediately with no timelock.
If the exploit is cross-mint — affecting multiple issuers — freeze all mints simultaneously. This is the platform's nuclear option and is reserved for systemic exploits where the scope is not yet contained.
Preserve on-chain evidence. Record all transactions from the suspected attacker wallet or wallets, snapshot all affected account states, and record the Solana slot and block hash at detection.
Notify the IC, Legal Counsel, Empire Stock Transfer, and all multi-signature holders simultaneously through PagerDuty critical-severity policy.

Investigation

Identify the attack vector by determining:

Which of the 42 controls was bypassed.
Which program instruction was exploited.
Whether the cause was a program-logic flaw, oracle manipulation, account confusion, or operational misconfiguration.

Identify all affected wallets and mints by tracing transactions from the attacker wallet or wallets and determining the total funds at risk or extracted.

Determine whether a program upgrade is needed:

If a program-logic vulnerability: emergency upgrade is required.
If an oracle vulnerability: rotate keys and fix the relay; no program upgrade required.
If an operational issue: fix configuration; no program upgrade required.

Remediation

For program upgrades the procedure is:

Build the patched version with the toolchain matching the audited release.
Run all six Certora invariants on the patched bytecode — every invariant must pass.
Create the emergency governance proposal.
Collect 5-of-9 multi-signature signatures for the upgrade authority.
Deploy the upgrade.
Verify on-chain that the patched program is active and the bytecode hash matches the audited replacement.

Recovery

Lift the Control 42 freeze only after the patch is deployed, verified, and monitoring is healthy. Lifting requires Legal Counsel authorization plus 3-of-5 multi-signature.

Monitor closely for 48 hours after the freeze is lifted: enhanced alerting thresholds active, manual review of the first 100 post-resume transactions.

7. Runbook: Oracle Failure (P1/P2)

7.1 Custody Oracle Stale (P1)

Trigger: Custody oracle slot age above 5. Transfers automatically halted with Error 6002 across all modules.

Check the relay service. Confirm whether the custody-relay process is running. If stopped, restart it. If running but stale, examine recent logs for connectivity errors, signature failures, or rate-limit events.
Check the Empire API. Verify Empire's API health endpoint. A 5xx response indicates an Empire-side outage; contact Empire compliance immediately. A 200 response with stale data indicates an Empire backend issue; also contact Empire.
Check RPC connectivity. Confirm the primary RPC (Helius dedicated cluster) is reachable. If the primary is down, verify the platform has failed over to the Triton fallback.
Restart the relay with a fresh connection and monitor that the oracle slot age drops to 0 or 1 within the next two slots.
If unresolved after 30 minutes, escalate to P0 and prepare communications for an extended trading halt.

7.2 OFAC Oracle Stale (P1 if above 48 hours; P2 if above 24 hours)

Check the OFAC indexer. Confirm the indexer process is running and review recent logs.
Check the U.S. Treasury OFAC API. A response failure indicates the public Treasury endpoint is unavailable. The platform's cached SDN list remains valid for up to 48 hours; trading continues during cache validity.
Restart the indexer and verify the oracle's is_fresh status returns to healthy.
If staleness exceeds 48 hours, all transfers halt with Error 6005. Escalate to P1, prepare communications, and verify Empire is informed.

7.3 AML Provider Failure (P2)

Identify which provider or providers are failing by reviewing the AML bridge logs for timeout or error patterns from Chainalysis KYT and TRM Labs.
If one provider is down, the system continues using the surviving provider's score. P2 — monitor for resolution.
If both providers are down, cached scores serve wallets with scores under 6 hours old. New wallets cannot trade because no AML score is available; transfers reject with Error 6006. Escalate to P1 if both providers remain down beyond 30 minutes.
Check API key validity. Both providers' credentials are stored in the platform's secrets manager. If credentials have expired, rotate them and restart the AML bridge.

7.4 TWAP Oracle Stale (P2)

Check the Pyth Network feed. Pyth's status page indicates whether the issue is platform-specific or Solana-network-wide.
Check the TWAP consumer service. Confirm it is running and review recent logs.
Understand the impact. After 5 minutes of staleness, the TWAP-based price-impact circuit breaker is disabled — trades execute without price-impact protection. The other 41 controls remain active. P2 alert is appropriate for short-term degradation.
Restart the TWAP consumer and verify Pyth feed consumption resumes.

8. Runbook: Module 2 NAV Oracle Failure (P1)

Trigger: NAVOracle for a Module 2 mint has not received a fresh appraisal within nav_reappraisal_max_age_secs. The affected mint is paused on the AMM until a fresh appraisal arrives. Other Module 2 mints and all other modules are unaffected.

Severity: P1.

Immediate Actions

Confirm the staleness. Query the platform's NAV oracle endpoint for the affected mint. Verify the last_reappraisal timestamp, the next_reappraisal timestamp, and the current age against nav_reappraisal_max_age_secs.
Identify the property and the appraiser of record. The platform's compliance database maps each Module 2 mint to its single-asset entity, the property identifier, and the licensed appraiser engaged for the reappraisal cycle.
Contact the appraiser. The appraiser may be running late on a scheduled cycle or may have submitted to the wrong endpoint. The appraiser's response determines whether the issue is operational (delivery delay) or substantive (no completed appraisal).
Notify Empire Stock Transfer. Empire is the qualified custodian for the underlying Common Class B shares of the single-asset entity. Empire is informed of any NAV oracle disruption that affects an issuance under their custody.

Investigation

Possible CauseResolution Path

Appraiser delivery delay

Appraiser submits the completed appraisal to the NAV relay; relay pushes to chain

NAV relay service failure

Restart the NAV relay; verify the appraiser endpoint is reachable

Appraiser API authentication failure

Rotate API credentials; verify mTLS configuration

Appraiser engagement lapsed

Engage new appraiser; document in compliance database; reissue offering documentation if cadence change

Property under dispute (legal hold, title issue)

Coordinate with Legal Counsel; the mint may need a formal hold rather than a stale-NAV pause

Resolution

Once the new appraisal is received, the NAV relay verifies the appraiser's signature, validates the value falls within reasonable bounds against the prior appraisal, and pushes the update on chain.
Confirm the NAVOracle freshness drops below the warning threshold.
Confirm the affected mint resumes AMM trading.
Notify the issuer and affected investors that trading has resumed.

Post-Resolution

The post-mortem evaluates whether the cadence configured for this property is adequate. If the property type or market conditions warrant tighter cadence, an updated offering disclosure may be required.

If the NAV deviation between the new appraisal and the prior appraisal exceeds nav_deviation_max_bps, the NAV-deviation circuit breaker (Section 11) may also activate, requiring separate handling.

9. Runbook: Module 3 Classification Oracle Failure (P2)

Trigger: ClassificationOracle for a Module 3 mint has not received a fresh USGS or DOE classification update within the configured classification_max_age_secs (default 24 hours). Transfers continue but enhanced compliance review is triggered for all transfers on the affected mint.

Severity: P2 (escalates to P1 if classification age exceeds 48 hours; P0 if a federal-action notification arrives during staleness — see Section 13).

Immediate Actions

Confirm the staleness. Query the platform's classification oracle endpoint for the affected mint. Verify the USGS class, DOE status, federal-action status, and last-refresh timestamp.
Check the federal feeds. USGS Critical Minerals List and DOE Critical Materials Strategy feeds are the primary inputs. A federal-side outage (rare) means the platform's cached classification continues to govern.
Check the Classification relay service. Confirm the relay is running and review recent logs.
Notify the Compliance Lead. Module 3 classifications are regulated metadata; staleness affects the platform's compliance posture for the affected basin asset.

Investigation

Possible CauseResolution Path

USGS or DOE feed temporary unavailability

Wait for federal endpoint recovery; cached classification valid through cache window

Classification relay service crash

Restart the relay; verify federal endpoints are reachable

Classification source change (USGS reclassifies the mineral)

Compliance Lead reviews; if the mineral is removed from the Critical Minerals List, broader review of the basin's federal status is required

Federal action affecting the basin (EO, Section 232 order, DPA Title III order)

This is a P0 — see Section 13

Resolution

Once the federal feeds return current data, the relay pushes the updated classification on chain.
Confirm classification oracle freshness drops below the warning threshold.
Enhanced compliance review automatically de-activates.
The post-mortem documents the cause, notes any pattern of federal-feed reliability issues, and updates the classification cadence if warranted.

What This Runbook Does Not Cover

This runbook handles operational staleness only. If the federal classification has substantively changed (the mineral has been added to or removed from a critical list, or a federal action has been issued affecting the basin), the appropriate response is the federal-action freeze runbook in Section 13.

10. Runbook: RPC Infrastructure Degradation (P1/P2)

Trigger: Primary RPC (Helius dedicated cluster) latency above 500ms, error rate above 5%, or total failure.

Verify automatic failover is active. The SDK and relay services automatically switch to the Triton fallback. A P2 alert is generated on failover trigger. Confirm the platform's health endpoint reports the fallback is serving.
Check Helius status. Helius publishes a public status page. If an incident is in progress, monitor for resolution. If no incident is published, contact Helius support directly.
If both primary and fallback are down (P1 escalation): all oracle relays will fail because no RPC path is available. The custody oracle will go stale within minutes and transfers will halt automatically. From the platform's perspective, this is effectively a Solana-connectivity issue. Communications: "Infrastructure maintenance — trading paused."
Recovery. Monitor Helius and Triton status. When primary RPC recovers, verify all relay services reconnect and the custody oracle returns to fresh status. Verify the CEDEX order book is in sync. Resume normal operations.

11. Runbook: Circuit Breaker Activation (P2)

Trigger: Automated circuit breaker triggered. Trading is halted or restricted for a specific mint.

11.1 Price Halt (above 10% move in 5 minutes)

Verify the trigger is legitimate. Check the CEDEX trading history: was there genuine volume causing the move? Check for coordinated activity across multiple wallets and timing patterns. Check whether a public news event justifies the price movement.
No manual action is required. The 15-minute cooldown elapses and the circuit breaker resets automatically when the TWAP normalizes. Monitor the first trade after cooldown to confirm successful resumption.
If repeated triggers occur (more than three in 24 hours), investigate for manipulation. Review cross-wallet clustering. The compliance team assesses for SAR criteria.

11.2 Volume Halt (above 30% daily sell by single wallet)

Identify the wallet. Datadog query on Transfer Hook errors with code 6037 produces the wallet address.
No manual action is required. The wallet is automatically halted for 24 hours. Other wallets and other mints are unaffected.
Investigation. The compliance team determines within 24 hours whether the activity represents a legitimate institutional rebalance or coordinated activity warranting escalation.

11.3 Module 2 NAV Deviation Breaker (Real Estate)

Trigger. On-chain price for a Module 2 mint has deviated outside nav_deviation_max_bps from the on-chain NAV. The mint is paused on the AMM.
Verify. Query the NAV oracle endpoint and confirm the deviation magnitude.
Investigation. Either the NAV is stale (Section 8) or genuine market pricing has diverged from appraised NAV. Coordinate with the issuer and appraiser.
Recovery. Resumes upon receipt of a fresh appraisal that brings the on-chain price within the deviation band, or upon governance-approved adjustment to the deviation threshold.

12. Runbook: Regulatory Freeze — Control 42 (P0)

Trigger: Legal Counsel directs an immediate halt of all transfers on a specific mint or platform-wide. This is the most severe containment action and is used only when legally required or when it is the appropriate response to an active exploit.

Activation

Legal Counsel authorization is documented in writing in the incident channel before execution. Verbal authorization may be used only when written authorization will follow within the same hour.
Multi-signature execution (3-of-5) with no timelock — executes immediately. Three of the five emergency-authority signers coordinate the freeze instruction.
Verify the freeze is active by attempting a test transfer (should return Error 6042) and by confirming SecurityConfig.is_paused is true.
Notify stakeholders in this order: Empire Stock Transfer (immediate), affected issuer (within 1 hour), affected investors via the CEDEX portal notification (within 1 hour), public status page ("Trading halted for [SYMBOL] — regulatory review").

Triggers for Control 42

TriggerAuthoritySpeed

SEC enforcement action against issuer

Legal Counsel

Immediate

Court order or subpoena requiring halt

Legal Counsel

Immediate

FinCEN SAR investigation requiring halt

Legal Counsel

Immediate

OFAC emergency SDN designation

Automated (Control 39) plus Legal Counsel confirmation

Automatic

Active exploit (while investigation proceeds)

IC plus Legal Counsel

Immediate

Issuer bankruptcy filing

Automated (Control 35)

Automatic — protective conversion

Deactivation

The Control 42 freeze is lifted only when:

Legal Counsel authorizes the lift in writing.
Root cause is resolved.
3-of-5 multi-signature executes the unfreeze instruction.
A test transfer succeeds.
Monitoring confirms normal operations.
Stakeholder notification: "Trading resumed."

13. Runbook: Module 3 Federal-Action Freeze (P0)

Trigger: A federal action has been issued that affects a basin asset underlying a Module 3 mint. Federal actions include Executive Orders affecting strategic minerals, Section 232 tariff or restriction orders, Defense Production Act Title III orders, Department of Energy program changes affecting eligibility, and any court order or regulatory directive specifically naming the basin or mineral classification.

Severity: P0 — automatic. No triage required. Federal-action notifications go directly to Legal Counsel and the IC.

Immediate Actions (0 to 15 minutes)

Capture the federal action. Record the issuing agency, action type, effective date, and scope (specific basin, specific mineral class, or broader category).
Identify affected mints. The compliance database maps each Module 3 mint to its basin identifier, mineral class, and federal program participation. Determine which mints are within the action's scope.
Activate Control 42 on each affected mint (Section 12 procedure). Federal-action freezes use the same Control 42 mechanism — Legal Counsel authorization plus 3-of-5 multi-signature, no timelock.
Notify Empire Stock Transfer within the 15-minute SLA. Empire holds Common Class B shares for the underlying single-asset entity and coordinates any custodial response that may be required.
Notify affected issuers (the basin operators) and affected investors.
Update the Classification oracle with the federal-action status so that any subsequent compliance check on the affected mints reflects the action.

Investigation

The investigation phase for federal-action freezes is led by Legal Counsel rather than the Technical Lead, because the action's scope, duration, and required response are legal questions:

DeterminationPath

Does the action restrict transfer of the underlying basin asset?

Legal review of the action text; may require coordination with issuer's outside counsel

Does the action affect federal program eligibility (such as DOE coal-derived REE recovery)?

Compliance database is updated with new eligibility state

Is the action time-bounded (e.g., 90-day Section 232 review) or open-ended (e.g., new EO)?

Determines remediation path

Is the action subject to legal challenge, and is challenge being initiated?

Coordinate with issuer; freeze remains until resolution

Resolution

The freeze remains active for the full duration that the federal action prohibits transfer or while substantial uncertainty about transfer status persists. Resolution requires:

Federal action resolution. The action expires, is rescinded, the basin is exempted, or the issuer obtains an authorized variance.
Legal Counsel authorization for the lift, in writing.
3-of-5 multi-signature executes the unfreeze.
Classification oracle updated to reflect the new federal status.
Stakeholder notification that trading has resumed.

Post-Resolution

The post-mortem documents the federal action, the platform's response timeline, the duration of the freeze, the impact on issuers and investors, and any process improvements identified. If the federal action revealed a gap in the platform's federal-action monitoring, the gap is remediated before the next incident.

14. Runbook: Key Compromise (P0)

Trigger: Suspected or confirmed compromise of any signing key — multi-signature signer, oracle relay, or Empire attestation key.

14.1 Multi-Signature Signer Key Compromise

Assess severity. How many keys are compromised? Is the attacker below threshold (the upgrade-authority requires 5-of-9; the parameter and emergency authorities require 3-of-5)? Has any unauthorized transaction been submitted on chain?
If below threshold (1 to 4 keys for upgrade authority; 1 to 2 keys for parameter or emergency authority): the attacker cannot execute. Immediate risk is low. Rotate the compromised keys via governance proposal. Monitor for unauthorized multi-signature proposals.
If at or above threshold: this is a critical P0. Check whether any program upgrade is currently pending in the timelock window. If yes, cancel the pending upgrade immediately (a 2-of-5 cancellation is sufficient during the timelock window). Activate Control 42 across all mints as a precautionary measure. Initiate emergency governance to rotate all affected keys. Conduct a physical-security review of all key-holder locations and chain-of-custody.

14.2 Key Rotation Procedure

Generate new keys on fresh Ledger Enterprise devices.
Initiate the governance proposal that registers new keys and deauthorizes old keys.
Execute the proposal through the standard timelock and multi-signature thresholds for the affected authority. Even in emergencies, the timelock remains in force unless an emergency override has been authorized through the platform's governance bylaws.
Verify the rotation by confirming old-key transactions are rejected and new-key transactions are accepted.

14.3 Oracle Relay Key Compromise

Impact assessment. An attacker with a compromised relay key could submit false oracle data. The Ed25519 payload format includes the slot, so stale data is rejected at the program level. The 2-of-3 oracle consensus catches manipulated data when only one source is compromised.
Rotate immediately. Generate a new key in the platform's HSM. Update the oracle program to register the new relay authority. Restart the relay service with the new key. The old key is deauthorized.
Audit. Review the last 24 hours of oracle updates for anomalies.

14.4 Empire Attestation Key Compromise

Impact assessment. This is the most critical key compromise scenario because Empire's Ed25519 attestation drives the platform's foundational 1:1 backing verification.
Immediate. Halt all transfers — the custody oracle is no longer trusted. The IC activates Control 42 across all mints simultaneously.
Coordinate with Empire. Empire revokes the compromised key, generates a new Ed25519 keypair on Empire's HSM, and communicates the new public key through documented chain-of-custody.
On-chain update. Initiate a governance proposal that updates the Empire public key on every CustodyOracle account. The proposal proceeds through the standard 48-hour timelock unless an emergency override has been authorized. 5-of-9 multi-signature executes the rotation.
Resume. New attestations are verified on chain. The Control 42 freeze is lifted across all mints after 2-of-3 oracle consensus confirms integrity.

15. Communication Protocol

15.1 Internal Communication

ChannelUseDuring Incident

Dedicated incident channel

Real-time coordination among responders

All responders join. Status updates every 15 minutes (P0) or 30 minutes (P1).

PagerDuty

Alerting and acknowledgment

On-call engineer acknowledges within SLA.

Encrypted messaging

Multi-signature coordination

Signer coordination for emergency actions.

Phone

IC direct line

P0 only — when other channels are insufficient.

15.2 External Communication

AudienceChannelTimingContent

Status page

Automated plus manual

Within 15 minutes of P0 or P1

System status, affected services, ETA

Institutional investors

CEDEX portal notification

Within 1 hour of P0

Incident summary, impact, ETA

Affected issuers

Direct email plus phone

Within 1 hour of P0

Issuer-specific impact, action required

Empire Stock Transfer

Compliance hotline

Within 15 minutes of custody-related P0

Custody status, discrepancy details

Public and community

Social media plus blog

After containment — never during active response

Post-incident summary

SEC or FinCEN

Formal notification

Per Legal Counsel determination

If material event or SAR threshold met

Federal agency (Module 3)

Counsel-to-counsel via outside SEC counsel

Per Legal Counsel determination

If federal action coordination is appropriate

15.3 Status Page Template

The standard status page entry includes timestamp, brief description, impact statement, affected ST22 tokens or "all", and the next update commitment. Updates are issued at the cadence required by severity. The final entry is a Resolved status with root cause summary, total duration, and a commitment to publish a full post-incident report within 72 hours.

15.4 Communication Rules

Never speculate about root cause in external communications.
Never share technical details of an exploit before the patch is deployed.
Never promise specific resolution times — use "investigating" and "ETA when known."
Always coordinate external messaging through the Communications role.
Always have Legal Counsel review any SEC or FinCEN notification before sending.
For Module 3 federal-action incidents, all external communications are reviewed by Legal Counsel before publication regardless of audience.

16. Post-Incident Review

16.1 Timeline

MilestoneDeadline

Post-incident review meeting

72 hours after resolution

Draft post-mortem document

5 business days

Action items assigned

5 business days

Action items tracked in the platform's project-tracking system

5 business days

Public disclosure (if warranted)

30 calendar days

16.2 Post-Mortem Structure

The post-mortem document is structured into the following sections:

Summary. Severity, duration, impact, one-to-two-sentence root cause statement.
Timeline. Time-stamped sequence of events from initial alert through resolution.
Root Cause Analysis. Detailed technical or operational explanation.
What Went Well. Specific actions, controls, or decisions that limited the impact.
What Went Wrong. Specific actions, controls, or decisions that allowed the incident to occur or extended its duration.
Action Items. Numbered list with owner, priority, and deadline.
Regulatory Implications. SAR filing assessment, SEC notification assessment, Empire notification confirmation, federal-action assessment for Module 3 incidents.
Lessons Learned. Key takeaways for the team and for the platform's runbooks.

16.3 Blameless Culture

Post-incident reviews focus on systems and processes — not individuals. The goal is to understand what happened, why it happened, and how to prevent recurrence. If human error contributed, the question is "why did the system allow this error to propagate?" — not "who made the mistake?"

17. Escalation Contact Sheet

RolePrimaryBackupContact Method

Incident Commander

Chief Technology Officer (Frank Yglesias)

On-call senior engineer (provisional command)

PagerDuty → phone → encrypted messaging

Technical Lead

On-call engineer

Senior engineer (rotation backup)

PagerDuty → dedicated channel

Compliance Lead

Legal Counsel (JDT Legal)

IC (interim coverage only)

Phone → email

Empire Liaison

Empire Stock Transfer (Patrick Mokros, Founder)

Empire compliance hotline

Phone → email

Communications

Designated by IC at incident declaration

Dedicated channel → email

DevOps

Platform DevOps Lead

On-call engineer

PagerDuty → dedicated channel

Helius Support

Helius dedicated support channel

Helius escalation email

Dedicated channel → email

PagerDuty Services

ServiceEscalation PolicyOn-Call Rotation

Custody Oracle

Critical-severity policy (P0)

24/7 weekly rotation

CEDEX API

High-severity policy (P1)

24/7 weekly rotation

Oracle Services (general)

Medium-severity policy (P2)

Business hours plus on-call

Module 2 NAV Oracle

High-severity policy (P1)

24/7 weekly rotation

Module 3 Classification Oracle

Medium-severity policy (P2)

Business hours plus on-call

Module 3 Federal-Action Feed

Critical-severity policy (P0)

24/7 weekly rotation

Infrastructure

Medium-severity policy (P2)

Business hours plus on-call

18. Tabletop Exercise Schedule

Quarterly tabletop exercises ensure the team can execute runbooks under pressure. Exercises rotate through high-impact runbooks across all three modules.

QuarterScenarioModule CoverageParticipantsObjective

Q3 2026

Custody discrepancy (Section 5)

All modules

Full team

Validate detection → containment → Empire coordination → resolution

Q4 2026

Smart contract exploit (Section 6)

All modules

IC plus engineers plus Legal

Validate Control 42 activation → emergency upgrade → Certora re-verification

Q1 2027

Module 2 NAV oracle failure (Section 8)

Module 2

IC plus engineers plus Legal plus appraiser representative

Validate appraiser coordination → relay restart → mint resume

Q2 2027

Module 3 federal-action freeze (Section 13)

Module 3

IC plus Legal plus Empire plus issuer representative

Validate federal-action capture → Control 42 activation → Empire coordination → multi-stakeholder communication

Q3 2027

Multi-signature key compromise (Section 14)

All modules

IC plus multi-signature holders

Validate key rotation → pending upgrade cancellation → enhanced monitoring

Q4 2027

Full oracle cascade failure

All modules

Full team

Validate fail-safe cascade → communication protocol → recovery sequence

Exercise Format

Scenario briefing (10 minutes). Facilitator presents the incident scenario.
Response execution (45 minutes). Team follows the relevant runbook in real time. The facilitator injects new information as the exercise progresses to mirror real-incident dynamics.
Debrief (30 minutes). What worked, what did not, and what runbook updates are required.
Runbook updates are completed within 5 business days following the exercise.

Solana Blockchain Foundation — Why Solana, Token-2022, Transfer Hook, module-specific foundations
Architecture Decisions — ADR-001 through ADR-012 with full alternatives analysis
Security Model — Threat model, key management, formal verification, module-specific threat surfaces, bug bounty
Infrastructure Overview — Cloud architecture, environment separation, blockchain infrastructure
Network Configuration — Program addresses, RPC endpoints, oracle PDAs, multi-signature addresses, production parameters
Deployment Guide — Build, deploy, verify, upgrade, rollback procedures
Smart Contract Reference — Program instruction specifications, account schemas, error codes
Oracle Integration Guide — Oracle relay service architecture, fail-safe cascade, module-specific oracles
Compliance Integration Guide — SAR filing obligations, regulatory notifications, federal-action coordination

RWA Tokens · Incident Response Playbook · Groovy Company, Inc. Classification: Internal — authorized personnel only

PreviousDeployment Guide

Last updated 1 month ago

Was this helpful?

Good evening

Incident Response Playbook