Skip to content

Commit

Permalink
docs: update generalized architecture to reflect flat delta-overlay t…
Browse files Browse the repository at this point in the history
…emplate model

- Template repos are flat: just .tf.j2 + tf-run.data, no nested layer/region dirs
- LAYER and REGION_DIR are Proposer env vars; files are written to the correct
  path at copy time, not encoded in template directory structure
- Remove lambda/templates/{product_type}/ tree (templates live in the template repo)
- Layer-level remote_state.yml built by Lambda Pydantic model extra_files()
  from validated SC form inputs, not stored in template repo
- Pydantic model example updated with account_alias field + extra_files() method
- Onboarding checklist updated: no skeleton clone, no lambda/templates/ step
  • Loading branch information
Dave Arnold committed May 20, 2026
1 parent a16101c commit 6728094
Showing 1 changed file with 67 additions and 22 deletions.
89 changes: 67 additions & 22 deletions docs/generalized-terraform-product-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,32 +87,54 @@ The following components require **no changes** to support new product types:

### 1. Template repo on GHE

Create a new repo under `SCT-Engineering/` (e.g. `template-s3-bucket`) that follows
the standard account repo directory layout. This repo is cloned by the executor
CodeBuild build and serves as the starting point for rendered files.
Create a new repo under `SCT-Engineering/` (e.g. `template-s3-bucket`) containing
**only the workload-specific files** — nothing else.

The template repo must contain:
- Standard `.tf-control`, `.tf-control.tfrc`, `region.tf`, `credentials.d/`, `variables.d/`
- Layer directories (`common/`, `infrastructure/`, `vpc/`) as applicable
- `remote_state.yml` stubs that the Proposer build will populate
Account repos already carry all standard scaffolding from initial setup:
`.tf-control`, `.tf-control.tfrc`, `region.tf`, `credentials.d/`, `variables.d/`,
and layer-level `remote_state.yml` files with account-specific values.
Duplicating any of that in a template repo would overwrite working values with
stubs and make the template non-reusable across accounts.

A minimal template repo looks like:

```
template-s3-bucket/
├── INF.s3-standard.tf.j2 # S3 bucket + policy resources
├── tf-run.data # REMOTE-STATE + tf-directory-setup + ALL steps
└── .sc-automation.yml.j2 # optional; Proposer writes a default if absent
```

The files are **flat**. `LAYER` and `REGION_DIR` are env vars already known to the
Proposer build — it writes the rendered files into `${ACCOUNT_REPO}/${LAYER}/${REGION_DIR}/`
at copy time. There is no reason to encode layer or region as directory structure
inside the template repo.

If the target layer does not yet exist in the account repo, the Lambda Pydantic
model constructs the layer-level `remote_state.yml` from SC form inputs
(`account_id`, `account_alias`, `bucket`, `profile`, `region`) and passes it to
the Proposer via `EXTRA_FILES`. The template repo never carries this file.

### 2. Jinja2 templates

Add a new subdirectory under `lambda/templates/{product_type}/` containing the
`.tf.j2` and `.hcl.j2` files that are rendered by the Proposer build before being
committed to the new repo branch.
Jinja2 template files (`.tf.j2`) live **in the template repo itself** — flat, alongside
`tf-run.data`. There is no separate `lambda/templates/` directory tree. The Proposer
build clones the template repo and renders every `.j2` file it finds, writing the
result (minus the `.j2` extension) into `${LAYER}/${REGION_DIR}/` in the account repo.

Example for an S3 product:

```
lambda/templates/
├── eks_cluster/ # existing
│ ├── infrastructure/west/cluster.tf.j2
│ └── ...
├── s3_bucket/ # new
│ ├── infrastructure/west/s3.tf.j2
│ └── ...
└── {future_product}/ # pattern
template-s3-bucket/
├── INF.s3-standard.tf.j2 # rendered → infrastructure/west/INF.s3-standard.tf
├── tf-run.data
└── .sc-automation.yml.j2
```

Subdirectory variants (e.g. `standard/` vs `encrypted/`) are supported via the
`source_path` parameter — the Proposer copies only the named subdirectory's contents,
stripped of the subdirectory prefix.

### 3. Pydantic config model

Add a new model in `lambda/models/{product_type}.py`:
Expand All @@ -123,17 +145,38 @@ class S3BucketConfig(BaseModel):
bucket_name: str
account_name: str
aws_account_id: str
account_alias: str # used to build remote_state.yml profile field
environment: Literal["dev", "test", "prod"]
layer: str = "infrastructure"
region_dir: str = "west"
aws_region: str = "us-gov-west-1"
versioning_enabled: bool = True
lifecycle_days: int = 90
team: str
workload: str
tier: str
partition: str = "gov"

def extra_files(self) -> dict[str, str]:
"""Layer-level remote_state.yml — only needed if layer doesn't exist yet."""
return {
f"{self.layer}/remote_state.yml": render_remote_state_yml(
directory=self.layer,
account_id=self.aws_account_id,
account_alias=self.account_alias,
bucket=f"inf-tfstate-{self.aws_account_id}",
bucket_region="us-gov-east-1",
profile=f"{self.aws_account_id}-{self.account_alias}",
region=self.aws_region,
aws_environment="gov",
)
}
```

The model enforces required fields and default values before any CodeBuild build is started.
The model enforces required fields and default values before any CodeBuild build
is started. The `extra_files()` method produces the layer-level `remote_state.yml`
from validated inputs — account-specific values stay in the Lambda model, not in
the template repo.

### 4. Lambda dispatcher

Expand Down Expand Up @@ -195,9 +238,11 @@ existing EKS product config.
The following checklist can be handed to a product team or platform engineer to
onboard any new Terraform workload without Lambda or CodeBuild changes:

- [ ] Create `SCT-Engineering/template-{product_type}` repo from the standard account repo skeleton
- [ ] Add `lambda/templates/{product_type}/` with Jinja2 templates for each rendered file
- [ ] Add `lambda/models/{product_type}.py` with a Pydantic model defining required inputs
- [ ] Create `SCT-Engineering/template-{product_type}` containing **only** the workload
delta: flat `.tf.j2` file(s) + `tf-run.data` (+ optional `.sc-automation.yml.j2`)
- [ ] Add `lambda/models/{product_type}.py` with a Pydantic model defining required
inputs and an `extra_files()` method that builds the layer-level `remote_state.yml`
from validated SC form inputs
- [ ] Register the handler in `lambda/app.py` `PRODUCT_HANDLERS` table
- [ ] Create `service-catalog/{product_type}-product-template.yaml` CFN template
- [ ] Add census config YAML and SC portfolio registration in `terraform-service-catalog-census`
Expand Down

0 comments on commit 6728094

Please sign in to comment.