diff --git a/docs/generalized-terraform-product-architecture.md b/docs/generalized-terraform-product-architecture.md index 944c6ac..4f8d663 100644 --- a/docs/generalized-terraform-product-architecture.md +++ b/docs/generalized-terraform-product-architecture.md @@ -135,71 +135,129 @@ Subdirectory variants (e.g. `standard/` vs `encrypted/`) are supported via the `source_path` parameter — the Proposer copies only the named subdirectory's contents, stripped of the subdirectory prefix. -### 3. Pydantic config model +### 3. `handler.py` in the template repo -Add a new model in `lambda/models/{product_type}.py`: +`handler.py` lives **at the root of the template repo** alongside the Jinja2 templates. +It is the single place that defines everything the Lambda needs to know about the product: +required inputs, defaults, and any computed `EXTRA_FILES`. No files inside the Lambda +repository are created or modified. + +#### Contract + +| Symbol | Type | Purpose | +|--------|------|---------| +| `PRODUCT_TYPE` | `str` | Unique key; must match the `product_type` field in the CFN Properties block | +| `handle(props: dict) -> dict` | callable | Receives normalized CFN props; returns (possibly modified) props ready for `TfRunRequest` | + +`handle()` is called before `TfRunRequest` is constructed. It should: +- Apply product-specific defaults (`layer`, `region_dir`, `template_repo`, …) +- Validate required inputs (via a Pydantic model or plain assertions) +- Inject computed `extra_files` entries (e.g. a layer-level `remote_state.yml`) + +#### Example — complete `handler.py` for an S3 bucket product ```python -class S3BucketConfig(BaseModel): - """Input model for S3 bucket SC product.""" +# template-s3-bucket/handler.py +from __future__ import annotations +from pydantic import BaseModel +from typing import Literal + +PRODUCT_TYPE = "s3_bucket" + + +class _Config(BaseModel): bucket_name: str account_name: str aws_account_id: str - account_alias: str # used to build remote_state.yml profile field + account_alias: str environment: Literal["dev", "test", "prod"] layer: str = "infrastructure" region_dir: str = "west" aws_region: str = "us-gov-west-1" - versioning_enabled: bool = True - lifecycle_days: int = 90 team: str workload: str tier: str - partition: str = "gov" - - def extra_files(self) -> dict[str, str]: - """Layer-level remote_state.yml — only needed if layer doesn't exist yet.""" - return { - f"{self.layer}/remote_state.yml": render_remote_state_yml( - directory=self.layer, - account_id=self.aws_account_id, - account_alias=self.account_alias, - bucket=f"inf-tfstate-{self.aws_account_id}", - bucket_region="us-gov-east-1", - profile=f"{self.aws_account_id}-{self.account_alias}", - region=self.aws_region, - aws_environment="gov", - ) - } + + +def handle(props: dict) -> dict: + cfg = _Config(**{k: v for k, v in props.items() if k in _Config.model_fields}) + props.setdefault("layer", cfg.layer) + props.setdefault("region_dir", cfg.region_dir) + props.setdefault("template_repo", "template-s3-bucket") + # Inject layer-level remote_state.yml if the layer is new + props.setdefault("extra_files", {}) + props["extra_files"].setdefault( + f"{cfg.layer}/remote_state.yml", + _render_remote_state(cfg), + ) + return props + + +def _render_remote_state(cfg: _Config) -> str: + return ( + f"directory: \"{cfg.layer}\"\n" + f"profile: \"{cfg.aws_account_id}-{cfg.account_alias}\"\n" + f"bucket: \"inf-tfstate-{cfg.aws_account_id}\"\n" + f"bucket_region: \"us-gov-east-1\"\n" + f"region: \"{cfg.aws_region}\"\n" + f"account_id: \"{cfg.aws_account_id}\"\n" + f"account_alias: \"{cfg.account_alias}\"\n" + f"aws_environment: \"gov\"\n" + ) ``` -The model enforces required fields and default values before any CodeBuild build -is started. The `extra_files()` method produces the layer-level `remote_state.yml` -from validated inputs — account-specific values stay in the Lambda model, not in -the template repo. +Because `handler.py` is versioned in the template repo, the Pydantic model and defaults +evolve alongside the templates — no Lambda redeploy required. + +### 4. Lambda dispatcher — runtime fetch from template repo -### 4. Lambda dispatcher +The Lambda has no `handlers/` directory and no handler registry. Instead, it fetches +`handler.py` directly from the template repo via the GHE API at request time and loads +it dynamically. The template repo is identified by the `template_repo` field already +present in the CFN Properties. -A single routing table maps `product_type` to the correct handler: +``` +lambda/ +└── app.py ← one-time change: fetch + exec handler.py, then call handle(props) + no lambda/handlers/ directory, no lambda/models/ directory +``` + +#### How it works (design intent for `app.py`) ```python -PRODUCT_HANDLERS = { - "eks_cluster": handle_eks, - "s3_bucket": handle_s3, - # future: "rds_postgres": handle_rds -} - -def handle_create(props: dict): - product_type = props.get("product_type", "eks_cluster") # default: backward-compat - handler = PRODUCT_HANDLERS.get(product_type) - if not handler: - raise ValueError(f"Unknown product_type: {product_type}") - return handler(props) +# 1. Read template_repo from CFN props (before TfRunRequest is constructed) +template_repo = normalized.get("template_repo") # e.g. "template-s3-bucket" +if not template_repo: + raise ValueError("template_repo is required") + +# 2. Fetch handler.py from GHE via the raw contents API +github_org = os.environ.get("GITHUB_ORG_NAME", "SCT-Engineering") +github_api = os.environ.get("GITHUB_API", "https://github.e.it.census.gov/api/v3") +handler_url = f"{github_api}/repos/{github_org}/{template_repo}/contents/handler.py" +# ...fetch with Authorization header, base64-decode the content... + +# 3. Load the module dynamically +import types, importlib +mod = types.ModuleType("_handler") +exec(compile(handler_source, "handler.py", "exec"), mod.__dict__) + +# 4. Validate the contract and dispatch +if not (callable(getattr(mod, "handle", None)) and getattr(mod, "PRODUCT_TYPE", None)): + raise ValueError(f"{template_repo}/handler.py must define PRODUCT_TYPE and handle()") +normalized = mod.handle(normalized) +tf_req = TfRunRequest(**normalized) ``` -This is a **one-time change** to `lambda/app.py`. After it is in place, adding a new -product type requires only a new entry in the table and a new handler function — no -other Lambda changes. +#### Security boundary + +The Lambda only fetches `handler.py` from repos whose name is in an allow-list prefix +(`template-*` within `SCT-Engineering`). The GHE token used has **read-only** scope on +template repos, so a compromised template repo cannot write to account repos via this +path. Handler execution is the only place arbitrary code runs — this is intentional and +auditable (every template repo change is a PR in the SCT-Engineering org). + +**Adding a new product type requires only creating a new template repo with a `handler.py`. +No Lambda code changes, no Lambda redeployment, no registry entries.** ### 5. CloudFormation product template @@ -238,12 +296,12 @@ existing EKS product config. The following checklist can be handed to a product team or platform engineer to onboard any new Terraform workload without Lambda or CodeBuild changes: -- [ ] Create `SCT-Engineering/template-{product_type}` containing **only** the workload - delta: flat `.tf.j2` file(s) + `tf-run.data` (+ optional `.sc-automation.yml.j2`) -- [ ] Add `lambda/models/{product_type}.py` with a Pydantic model defining required - inputs and an `extra_files()` method that builds the layer-level `remote_state.yml` - from validated SC form inputs -- [ ] Register the handler in `lambda/app.py` `PRODUCT_HANDLERS` table +- [ ] Create `SCT-Engineering/template-{product_type}` containing: + - `handler.py` — `PRODUCT_TYPE`, Pydantic model, `handle()` function + - flat `.tf.j2` file(s) rendered by the Proposer + - `tf-run.data` + - `.sc-automation.yml.j2` (optional; Proposer writes a default if absent) + **No files in the Lambda repository need to be created or modified.** - [ ] Create `service-catalog/{product_type}-product-template.yaml` CFN template - [ ] Add census config YAML and SC portfolio registration in `terraform-service-catalog-census` - [ ] Test end-to-end via `scripts/test_service_catalog.py` with the new product type diff --git a/docs/service-catalog-census-integration.md b/docs/service-catalog-census-integration.md index 3502c7a..575be1a 100644 --- a/docs/service-catalog-census-integration.md +++ b/docs/service-catalog-census-integration.md @@ -27,15 +27,18 @@ roles), or census-managed (portfolios, products, constraints) — and handled ac ``` sc-lambda-ghactions/ ← Lambda + CodeBuild buildspecs + SC product templates -├── lambda/app.py ← Lambda handler (dispatcher by product_type) -├── lambda/models/{product_type}.py ← Pydantic input models per product type -├── lambda/templates/{product_type}/ ← Jinja2 HCL templates per product type +├── lambda/app.py ← Lambda handler (fetches + runs handler.py from template repo at runtime) ├── service-catalog/{product_type}-product-template.yaml ← CFN product template └── deploy/ ← Terraform: Lambda, ECR, IAM, Function URL +SCT-Engineering/template-{product_type}/ ← one repo per product type; fully self-contained +├── handler.py ← PRODUCT_TYPE + Pydantic model + handle() +├── {workload}.tf.j2 ← Jinja2 HCL templates (flat) +├── tf-run.data ← tf-run steps +└── .sc-automation.yml.j2 ← optional webhook config template + terraform-sc-fleet/ ← Fleet operations manifest (all managed workloads) packer-pipeline/ ← Container build CLI -template-{product_type}/ ← Template repos (one per product type) ``` ### `terraform-service-catalog-census` (census repo) diff --git a/docs/template-management.md b/docs/template-management.md index 4213fdd..ddb8038 100644 --- a/docs/template-management.md +++ b/docs/template-management.md @@ -238,7 +238,7 @@ creates the initial PR if it does not already exist on `main`. ```yaml # .sc-automation.yml -product_type: eks_cluster # Must match a registered PRODUCT_HANDLERS key +product_type: eks_cluster # Must match PRODUCT_TYPE in template repo's handler.py executor_project: sc-executor # CodeBuild project name for the Executor build dry_run: true # If true, Executor runs tf plan only (no apply) template_repo: SCT-Engineering/template-eks-cluster # Source template repo @@ -302,11 +302,11 @@ changes and the post-apply file diff is empty, the commit step is skipped. Checklist when onboarding a new product type: -- [ ] Create `SCT-Engineering/template-{product_type}` containing **only** the workload - delta: `{layer}/{workspace}/{workload}.tf.j2` + `tf-run.data` -- [ ] Add a Pydantic model in `lambda/models/{product_type}.py` that validates - product-specific inputs and builds `TEMPLATE_VARS` + any `EXTRA_FILES` - (e.g. layer-level `remote_state.yml` if the target layer may not exist yet) -- [ ] Register the handler in `lambda/app.py` `PRODUCT_HANDLERS` table +- [ ] Create `SCT-Engineering/template-{product_type}` containing: + - `handler.py` — `PRODUCT_TYPE`, Pydantic model, `handle()` function + - flat `.tf.j2` file(s) (rendered into `${LAYER}/${REGION_DIR}/` by the Proposer) + - `tf-run.data` + - `.sc-automation.yml.j2` (optional) + **No files in the Lambda repository need to be created or modified.** - [ ] Create a CFN product template in `service-catalog/{product_type}-product-template.yaml` - [ ] Add the product to `terraform-service-catalog-census` (see [service-catalog-census-integration.md](service-catalog-census-integration.md))