diff --git a/docs/generalized-terraform-product-architecture.md b/docs/generalized-terraform-product-architecture.md index 71b9772..944c6ac 100644 --- a/docs/generalized-terraform-product-architecture.md +++ b/docs/generalized-terraform-product-architecture.md @@ -87,32 +87,54 @@ The following components require **no changes** to support new product types: ### 1. Template repo on GHE -Create a new repo under `SCT-Engineering/` (e.g. `template-s3-bucket`) that follows -the standard account repo directory layout. This repo is cloned by the executor -CodeBuild build and serves as the starting point for rendered files. +Create a new repo under `SCT-Engineering/` (e.g. `template-s3-bucket`) containing +**only the workload-specific files** — nothing else. -The template repo must contain: -- Standard `.tf-control`, `.tf-control.tfrc`, `region.tf`, `credentials.d/`, `variables.d/` -- Layer directories (`common/`, `infrastructure/`, `vpc/`) as applicable -- `remote_state.yml` stubs that the Proposer build will populate +Account repos already carry all standard scaffolding from initial setup: +`.tf-control`, `.tf-control.tfrc`, `region.tf`, `credentials.d/`, `variables.d/`, +and layer-level `remote_state.yml` files with account-specific values. +Duplicating any of that in a template repo would overwrite working values with +stubs and make the template non-reusable across accounts. + +A minimal template repo looks like: + +``` +template-s3-bucket/ +├── INF.s3-standard.tf.j2 # S3 bucket + policy resources +├── tf-run.data # REMOTE-STATE + tf-directory-setup + ALL steps +└── .sc-automation.yml.j2 # optional; Proposer writes a default if absent +``` + +The files are **flat**. `LAYER` and `REGION_DIR` are env vars already known to the +Proposer build — it writes the rendered files into `${ACCOUNT_REPO}/${LAYER}/${REGION_DIR}/` +at copy time. There is no reason to encode layer or region as directory structure +inside the template repo. + +If the target layer does not yet exist in the account repo, the Lambda Pydantic +model constructs the layer-level `remote_state.yml` from SC form inputs +(`account_id`, `account_alias`, `bucket`, `profile`, `region`) and passes it to +the Proposer via `EXTRA_FILES`. The template repo never carries this file. ### 2. Jinja2 templates -Add a new subdirectory under `lambda/templates/{product_type}/` containing the -`.tf.j2` and `.hcl.j2` files that are rendered by the Proposer build before being -committed to the new repo branch. +Jinja2 template files (`.tf.j2`) live **in the template repo itself** — flat, alongside +`tf-run.data`. There is no separate `lambda/templates/` directory tree. The Proposer +build clones the template repo and renders every `.j2` file it finds, writing the +result (minus the `.j2` extension) into `${LAYER}/${REGION_DIR}/` in the account repo. + +Example for an S3 product: ``` -lambda/templates/ -├── eks_cluster/ # existing -│ ├── infrastructure/west/cluster.tf.j2 -│ └── ... -├── s3_bucket/ # new -│ ├── infrastructure/west/s3.tf.j2 -│ └── ... -└── {future_product}/ # pattern +template-s3-bucket/ +├── INF.s3-standard.tf.j2 # rendered → infrastructure/west/INF.s3-standard.tf +├── tf-run.data +└── .sc-automation.yml.j2 ``` +Subdirectory variants (e.g. `standard/` vs `encrypted/`) are supported via the +`source_path` parameter — the Proposer copies only the named subdirectory's contents, +stripped of the subdirectory prefix. + ### 3. Pydantic config model Add a new model in `lambda/models/{product_type}.py`: @@ -123,7 +145,10 @@ class S3BucketConfig(BaseModel): bucket_name: str account_name: str aws_account_id: str + account_alias: str # used to build remote_state.yml profile field environment: Literal["dev", "test", "prod"] + layer: str = "infrastructure" + region_dir: str = "west" aws_region: str = "us-gov-west-1" versioning_enabled: bool = True lifecycle_days: int = 90 @@ -131,9 +156,27 @@ class S3BucketConfig(BaseModel): workload: str tier: str partition: str = "gov" + + def extra_files(self) -> dict[str, str]: + """Layer-level remote_state.yml — only needed if layer doesn't exist yet.""" + return { + f"{self.layer}/remote_state.yml": render_remote_state_yml( + directory=self.layer, + account_id=self.aws_account_id, + account_alias=self.account_alias, + bucket=f"inf-tfstate-{self.aws_account_id}", + bucket_region="us-gov-east-1", + profile=f"{self.aws_account_id}-{self.account_alias}", + region=self.aws_region, + aws_environment="gov", + ) + } ``` -The model enforces required fields and default values before any CodeBuild build is started. +The model enforces required fields and default values before any CodeBuild build +is started. The `extra_files()` method produces the layer-level `remote_state.yml` +from validated inputs — account-specific values stay in the Lambda model, not in +the template repo. ### 4. Lambda dispatcher @@ -195,9 +238,11 @@ existing EKS product config. The following checklist can be handed to a product team or platform engineer to onboard any new Terraform workload without Lambda or CodeBuild changes: -- [ ] Create `SCT-Engineering/template-{product_type}` repo from the standard account repo skeleton -- [ ] Add `lambda/templates/{product_type}/` with Jinja2 templates for each rendered file -- [ ] Add `lambda/models/{product_type}.py` with a Pydantic model defining required inputs +- [ ] Create `SCT-Engineering/template-{product_type}` containing **only** the workload + delta: flat `.tf.j2` file(s) + `tf-run.data` (+ optional `.sc-automation.yml.j2`) +- [ ] Add `lambda/models/{product_type}.py` with a Pydantic model defining required + inputs and an `extra_files()` method that builds the layer-level `remote_state.yml` + from validated SC form inputs - [ ] Register the handler in `lambda/app.py` `PRODUCT_HANDLERS` table - [ ] Create `service-catalog/{product_type}-product-template.yaml` CFN template - [ ] Add census config YAML and SC portfolio registration in `terraform-service-catalog-census`