diff --git a/aws/proposals/dice-decennial/multi-region-prerequisites-checklist.md b/aws/proposals/dice-decennial/multi-region-prerequisites-checklist.md new file mode 100755 index 00000000..dfd5c7c4 --- /dev/null +++ b/aws/proposals/dice-decennial/multi-region-prerequisites-checklist.md @@ -0,0 +1,229 @@ +# Multi-Region Architecture Analysis — Prerequisites & Discovery Checklist + +**Purpose:** Before any component-level dependency mapping begins, this checklist captures the business, organizational, and boundary-setting information needed to scope a multi-region (geo-partitioned, write-available-on-failover) architecture for the identified application suites. Organized by **Why / What / When / Where / Who** — deliberately light on *how*, since implementation approach follows from these answers, not the reverse. + +--- + +## Checklist + +### Why — Business Drivers & Risk Tolerance +- [ ] [Confirm the primary driver: latency vs. resiliency vs. compliance](#confirm-the-primary-driver) +- [ ] [Establish RTO per application](#establish-rto-per-application) +- [ ] [Establish RPO per application, including failure-boundary write loss tolerance](#establish-rpo-per-application) +- [ ] [Quantify cost of downtime / degraded write availability](#quantify-cost-of-downtime) +- [ ] [Identify regulatory, contractual, or data residency mandates](#identify-regulatory-mandates) +- [ ] [Determine if this must satisfy an audit, insurance, or customer-contract requirement](#audit-and-contract-requirements) + +### What — Scope, Data, and Application Boundaries +- [ ] [Confirm exactly which 3 applications are in scope](#confirm-applications-in-scope) +- [ ] [Confirm the partition model: is data cleanly regional, and by what key](#confirm-partition-model) +- [ ] [Identify the traveling-user / roaming-access requirement in full](#traveling-user-requirement) +- [ ] [Classify every component: stateless vs. stateful](#classify-components) +- [ ] [Identify singleton or leader-election behavior](#identify-singleton-behavior) +- [ ] [Inventory MSK topics and producer/consumer idempotency](#inventory-msk-topics) +- [ ] [Map inter-service and inter-suite dependencies](#map-dependencies) +- [ ] [Identify shared/cross-partition data, if any](#shared-cross-partition-data) +- [ ] [Inventory supporting AWS resources requiring cross-region parity](#inventory-supporting-resources) + +### When — Timing, Triggers, and Sequencing +- [ ] [Define what triggers a failover decision](#define-failover-triggers) +- [ ] [Define what triggers a failback decision, and who schedules it](#define-failback-triggers) +- [ ] [Establish acceptable detection time before failover begins](#detection-time) +- [ ] [Set expectations for degraded-mode duration](#degraded-mode-duration) +- [ ] [Define testing cadence (game days) before and after go-live](#testing-cadence) +- [ ] [Set a re-evaluation checkpoint for the entire design](#re-evaluation-checkpoint) + +### Where — Regions, Boundaries, and Routing +- [ ] [Select the two target regions and validate service parity](#select-target-regions) +- [ ] [Define the geographic partition boundary and how it's drawn](#define-partition-boundary) +- [ ] [Decide routing mechanism: geolocation vs. geoproximity vs. identity-based](#routing-mechanism) +- [ ] [Design the write-target indirection layer](#write-target-indirection) +- [ ] [Plan non-overlapping network addressing across regions](#network-addressing) +- [ ] [Confirm data residency/jurisdiction implications per region](#data-residency-per-region) + +### Who — Ownership, Skills, and Decision Rights +- [ ] [Assign an accountable owner for the overall multi-region program](#accountable-owner) +- [ ] [Identify who has authority to declare a failover](#failover-authority) +- [ ] [Identify who has authority to approve failback](#failback-authority) +- [ ] [Assess current team skill coverage against required expertise](#skill-assessment) +- [ ] [Identify skill gaps and closure plan (hire, train, contract)](#skill-gap-closure) +- [ ] [Identify who owns cost monitoring and reports on it](#cost-ownership) +- [ ] [Identify who owns changed-requirement / pivot scenarios](#pivot-ownership) + +### Cross-Cutting — Cost, Complexity, and the Unknown +- [ ] [Build a three-scenario cost model before committing](#cost-model) +- [ ] [Document fencing and split-brain prevention approach](#fencing-and-split-brain) +- [ ] [Document reconciliation approach for failback](#failback-reconciliation) +- [ ] [Record architecture decisions (ADRs) as they're made](#architecture-decision-records) +- [ ] [Explicitly acknowledge and plan for out-of-control requirement changes](#changed-requirements-planning) +- [ ] [Select and apply dependency-mapping tooling before deep design work](#dependency-mapping-tooling) + +--- + +## Detailed Sections + +### Why + +#### Confirm the primary driver +State explicitly, per application, whether the primary goal is **latency reduction** (nationwide user base, geo-split for performance), **resiliency** (surviving regional failure), or **compliance** (data residency mandate) — or some combination. This conversation has already established latency is primary for the three in-scope apps, with resiliency as a close secondary requirement given the write-during-failover need. Getting this on record prevents scope drift later, since the "right" architecture differs depending on which driver dominates. + +#### Establish RTO per application +For each of the three applications: how long can it be down or degraded before business impact becomes unacceptable? This number directly determines whether automated failover is required or a manual/human-approved process is sufficient. + +#### Establish RPO per application +How much data loss is tolerable at the exact moment of an unplanned regional failure? Given write-during-failover is a confirmed requirement, this must be stated as a real number (e.g., "sub-second, best-effort, non-zero") and formally accepted by stakeholders — not left as an assumed "zero." + +#### Quantify cost of downtime +Put a dollar or business-impact figure on an hour of full outage and an hour of degraded (read-only, or partial-region) operation. This number is what all subsequent cost-of-resiliency tradeoffs get measured against. + +#### Identify regulatory mandates +Determine whether any current or anticipated regulation (state data residency laws, federal contract requirements, industry-specific compliance) constrains where data for these three applications may legally live. This may override a convenience-based East/West split. + +#### Audit and contract requirements +Separately from regulation, check whether any customer contract, SLA, or insurance policy already commits your organization to a specific uptime or DR posture that this design must satisfy or can be used to justify. + +--- + +### What + +#### Confirm applications in scope +Get explicit, named agreement on which three applications require geographic split. Confirm this list won't silently grow — subsequent design decisions (partition key, routing indirection, IaC structure) should be built to extend, but the initial effort and cost estimate should be scoped to exactly these three. + +#### Confirm partition model +For each application, confirm data is genuinely regional and identify the specific partition key (state of residence, account origination region, etc.). Confirm the key is stable over the record's lifetime — a key that can change (e.g., a user relocates permanently) needs an explicit re-partitioning process defined now, not discovered later. + +#### Traveling-user requirement +This has already been identified as a hard requirement: a user whose data lives in one partition must retain full read/write access when physically located in, or rerouted to, the other region. Document this precisely per application, since the degree of "must work" (read-only acceptable vs. full write required) has now been confirmed as **write required** for at least one case — confirm this is true for all three or varies by app. + +#### Classify components +For every ECS service in each application: is it stateless (safe to run identically in both regions) or stateful (session data, in-memory caches, local state)? Stateless components are comparatively simple to make active-active; stateful ones need individual design attention. + +#### Identify singleton behavior +Flag any component that assumes it is the only instance performing an action — scheduled jobs, leader-elected processes, anything writing without expecting concurrent writers. These need explicit multi-region handling (e.g., only run in the currently-primary region for that partition) or they will produce duplicate or conflicting work during failover. + +#### Inventory MSK topics +List every topic per application: producers, consumers, throughput, and whether consumers are idempotent. This determines which topics need cross-region replication (MSK Replicator/MirrorMaker 2) to support the traveling-user write requirement, and which can remain region-local. + +#### Map dependencies +Build the actual service-to-service and service-to-data call graph per application suite (see [Dependency-Mapping Tooling](#dependency-mapping-tooling)). This is a prerequisite for knowing which components can be safely partitioned independently and which are coupled tightly enough that they must move together. + +#### Shared cross-partition data +Identify any data that is *not* cleanly regional — reference data, shared configuration, cross-user features (if any exist even in a mostly-partitioned app). This data needs its own replication or single-source-of-truth decision, separate from the partitioned data model. + +#### Inventory supporting resources +Beyond ECS/Aurora/MSK: Secrets Manager entries, Parameter Store values, IAM roles/policies, S3 buckets, and any other resource each application depends on. All of these need a plan for existing correctly in both regions. + +--- + +### When + +#### Define failover triggers +Specify precisely what condition initiates a failover — a health check failure, a manual declaration, a specific error rate/latency threshold. Ambiguity here is where real incidents go wrong; this should be a written, numeric definition, not a judgment call made live. + +#### Define failback triggers +Specify separately what condition allows failback to begin, and confirm (as discussed) that this is a **planned, scheduled event** requiring human approval — not automatic. State who signs off. + +#### Detection time +Given Route 53 health check minimums and DNS TTL behavior, establish the realistic floor for how fast a failure can be detected and traffic rerouted. Compare this against the RTO from the Why section to confirm feasibility. + +#### Degraded-mode duration +Set expectations for how long the suite can reasonably run in a promoted/failed-over state before failback becomes urgent — this affects both cost (running failover capacity longer than expected) and risk (longer divergence windows before reconciliation). + +#### Testing cadence +Define how often failover and failback will be rehearsed via game days, both before go-live (to validate the design) and on an ongoing basis afterward (to catch drift). Failback specifically should not go untested — it is the step most commonly skipped and the one most likely to fail under real conditions. + +#### Re-evaluation checkpoint +Set a fixed point (e.g., 6 or 12 months post-launch) to revisit whether the original assumptions (traffic split, partition key stability, driver priority) still hold, independent of any forced pivot event. + +--- + +### Where + +#### Select target regions +Confirm the two AWS regions and validate that every AWS service in use (Aurora Global Database, MSK Replicator, the specific ECS/networking features required) is available and at feature parity in both. Don't assume parity — verify it. + +#### Define partition boundary +Determine precisely how the US is split — by state list, by geoproximity bias, or another method — and who owns adjusting that boundary if real traffic patterns don't match the assumed 50/50 split. + +#### Routing mechanism +Decide between Route 53 Geolocation routing (explicit boundary control) and Geoproximity routing (tunable bias) for anonymous/first-touch traffic, and confirm the identity-aware override needed for authenticated users whose home partition differs from their physical location. + +#### Write-target indirection +Design the mechanism (private hosted zone alias, service discovery, config lookup) that lets application code resolve "which region is currently the writer for this user's partition" as a single, centrally-updated source of truth — not a value baked into per-service configuration. + +#### Network addressing +Plan non-overlapping CIDR ranges across the two regions now, before either region's VPC is built out further, to avoid a costly re-addressing exercise later. + +#### Data residency per region +Cross-check the chosen regions against the regulatory findings from the Why section — confirm neither region introduces a jurisdictional conflict with where specific users' data is legally permitted to reside. + +--- + +### Who + +#### Accountable owner +Name a single accountable owner for the multi-region program across all three applications — someone empowered to make cross-team tradeoff calls (cost vs. resiliency vs. timeline), since this work touches infrastructure, application, and data teams simultaneously. + +#### Failover authority +Document who (or what automated system, with what override authority) is permitted to declare a regional failover in a live incident. Given the fencing/split-brain risk discussed earlier, this should be a short, pre-agreed list, not open to ad hoc decision during an incident. + +#### Failback authority +Document separately who approves failback — this should be a deliberate, likely more senior or more cross-functional sign-off than failover authority, given failback carries the higher reconciliation risk. + +#### Skill assessment +See [Skill Sets & Expertise Required](#skill-sets--expertise-required) below for the full breakdown. At minimum, assess current team familiarity with: Aurora Global Database operations, MSK cross-region replication, Route 53 advanced routing policies, and incident response for split-brain/partial-failure scenarios. + +#### Skill gap closure +For any gap identified above, decide explicitly: hire, train existing staff, or bring in short-term contractor/partner expertise (e.g., an AWS Solutions Architect engagement or a formal Well-Architected Review) to close it before go-live rather than during a live incident. + +#### Cost ownership +Assign ownership for ongoing monitoring of cross-region data transfer, replication, and standby capacity costs — these are recurring, not one-time, and can drift upward silently as traffic grows. + +#### Pivot ownership +Assign ownership for periodically reassessing whether the two-region, geography-based design still fits — this is the person or team responsible for raising a flag if a regulatory, business, or technical change (see [Changed-Requirements Planning](#changed-requirements-planning)) is on the horizon. + +--- + +### Cross-Cutting + +#### Cost model +Build a three-scenario model before committing engineering effort: (1) current single-region baseline, (2) partitioned active-active without cross-region standby (i.e., no traveling-user write support), (3) full mutual active-standby with promotion/failback as now required. Include compute, Aurora Global Database replica costs, MSK replication costs, and cross-region data transfer — the gap between scenarios 2 and 3 is usually larger than initially assumed and should be visible to whoever approves the budget. + +#### Fencing and split-brain prevention +Document explicitly how the old writer region is prevented from accepting writes once the other region is promoted — network isolation, application-level rejection, or both. This must be tested under a partial-failure condition (East reachable by some clients but not by monitoring), not just a clean shutdown. + +#### Failback reconciliation +Document the specific procedure for treating the recovered region as a new replica, re-establishing replication from the promoted region, and the criteria for when a second, planned failover back to the original region is safe to execute. + +#### Architecture decision records +Capture every non-obvious decision (partition key choice, routing method, failover/failback authority, boundary definition) as a written ADR at the time it's made. This is what makes a future forced redesign tractable rather than requiring archaeological reconstruction of "why did we do it this way." + +#### Changed-requirements planning +Explicitly acknowledge, in whatever plan goes to stakeholders, that this design is built for the *currently known* requirement set and is not claimed to be future-proof against an out-of-control change — a new data residency law, an acquisition, a forced third region, or a partition boundary that turns out to be wrong in practice. State the specific mitigations already built in (abstracted partition key, region-parameterized IaC, ADRs) as what reduces — not eliminates — the cost of a future pivot. + +#### Dependency-mapping tooling +Before deep application-level design begins, instrument the three applications (Java/Python/Node.js on ECS) with distributed tracing to produce the real service dependency graph, rather than relying on assumed or documented dependencies: +- **AWS X-Ray** or **OpenTelemetry via AWS Distro for OpenTelemetry (ADOT)**, run as an ECS sidecar, exported to X-Ray or CloudWatch Application Signals for automatic service-map generation. +- **VPC Flow Logs + Athena** to catch network-level dependencies not captured by application-level tracing. +- Static dependency tools (Maven/Gradle dependency trees for Java, `pipdeptree` for Python, `madge`/`npm ls` for Node.js) as a supplement, not a replacement, for runtime tracing. + +--- + +## Skill Sets & Expertise Required + +This analysis and the resulting build require a broader and deeper skill set than standard single-region operations. Assess current team coverage against each of the following before finalizing timeline or ownership. + +| Area | Skill Needed | Approximate Level | Why It's Required | +|---|---|---|---| +| **Database (Aurora/Postgres)** | Aurora Global Database architecture, managed and manual failover, replication lag behavior, promotion/failback procedures | Senior/Expert — this is the highest-risk technical area in the design | Incorrect promotion or failback handling is the most likely source of data loss or a prolonged outage | +| **Streaming (MSK/Kafka)** | MSK Replicator or MirrorMaker 2 configuration, consumer idempotency design, offset management across regions | Senior | Replication misconfiguration causes silent duplicate or out-of-order processing, often undetected until far downstream | +| **Networking** | Multi-region VPC design, non-overlapping CIDR planning, Transit Gateway/inter-region peering, Route 53 advanced routing policies (geolocation, geoproximity, failover) | Senior | Routing and addressing mistakes are expensive to unwind after services are live | +| **Application Engineering (Java/Python/Node.js)** | Identifying and removing single-region assumptions (hardcoded ARNs, region-specific SDK calls), implementing write-target indirection, idempotent event handling | Mid–Senior across each language's service owners | Every hidden regional assumption becomes a production incident during an actual failover | +| **Infrastructure as Code** | Region-parameterized Terraform/Terragrunt modules (not copy-pasted per-region stacks) | Senior | Directly determines how expensive it is to add a third region or redraw the partition boundary later | +| **Observability/Distributed Tracing** | OpenTelemetry/ADOT instrumentation, X-Ray or CloudWatch Application Signals, building and reading service dependency graphs | Mid–Senior | This is the prerequisite deliverable requested for the next phase of this analysis | +| **Incident Response / SRE** | Designing and running failover/failback game days, writing and rehearsing runbooks, fencing/split-brain mitigation | Senior — ideally with prior multi-region operational experience specifically, not just multi-AZ | Multi-AZ incident response experience does not fully transfer; cross-region promotion and reconciliation is a materially different operational skill | +| **Data Modeling** | Partition key design, evaluating key stability, planning for re-partitioning if a key selection proves wrong | Senior, with product/domain input | A poorly chosen partition key is very costly to change after data exists in production | +| **Compliance/Legal Liaison** | Interpreting data residency and regulatory constraints as they apply to the chosen regions and partition boundary | Not an engineering skill, but must be represented — legal/compliance stakeholder, not just engineering judgment | Engineering cannot self-certify regulatory compliance; this needs a named non-engineering owner | +| **Program/Technical Ownership** | Cross-team coordination across data, application, and infrastructure groups; budget and tradeoff authority | Senior/Staff+ or equivalent program lead | This spans more organizational surface area than a typical infrastructure project and needs a single accountable owner to avoid decisions stalling between teams | + +**Overall assessment guidance:** if the team has strong multi-AZ operational experience but no prior multi-region production experience, budget explicit time and/or external expertise (an AWS Solutions Architect engagement, a formal Well-Architected Review, or a short-term specialist contractor) for the database failover/failback and MSK replication design specifically — these are the two areas where a skill gap is most likely to translate directly into a production incident rather than just a delay.