docs: handler.py lives in template repo; Lambda fetches at runtime

docs/generalized-terraform-product-architecture.md

@@ -135,71 +135,129 @@ Subdirectory variants (e.g. `standard/` vs `encrypted/`) are supported via the
 `source_path` parameter — the Proposer copies only the named subdirectory's contents,
 stripped of the subdirectory prefix.
 
-### 3. Pydantic config model
+### 3. `handler.py` in the template repo
 
-Add a new model in `lambda/models/{product_type}.py`:
+`handler.py` lives **at the root of the template repo** alongside the Jinja2 templates.
+It is the single place that defines everything the Lambda needs to know about the product:
+required inputs, defaults, and any computed `EXTRA_FILES`. No files inside the Lambda
+repository are created or modified.
+
+#### Contract
+
+| Symbol | Type | Purpose |
+|--------|------|---------|
+| `PRODUCT_TYPE` | `str` | Unique key; must match the `product_type` field in the CFN Properties block |
+| `handle(props: dict) -> dict` | callable | Receives normalized CFN props; returns (possibly modified) props ready for `TfRunRequest` |
+
+`handle()` is called before `TfRunRequest` is constructed. It should:
+- Apply product-specific defaults (`layer`, `region_dir`, `template_repo`, …)
+- Validate required inputs (via a Pydantic model or plain assertions)
+- Inject computed `extra_files` entries (e.g. a layer-level `remote_state.yml`)
+
+#### Example — complete `handler.py` for an S3 bucket product
 
 ```python
-class S3BucketConfig(BaseModel):
-    """Input model for S3 bucket SC product."""
+# template-s3-bucket/handler.py
+from __future__ import annotations
+from pydantic import BaseModel
+from typing import Literal
+
+PRODUCT_TYPE = "s3_bucket"
+
+
+class _Config(BaseModel):
     bucket_name: str
     account_name: str
     aws_account_id: str
-    account_alias: str          # used to build remote_state.yml profile field
+    account_alias: str
     environment: Literal["dev", "test", "prod"]
     layer: str = "infrastructure"
     region_dir: str = "west"
     aws_region: str = "us-gov-west-1"
-    versioning_enabled: bool = True
-    lifecycle_days: int = 90
     team: str
     workload: str
     tier: str
-    partition: str = "gov"
-
-    def extra_files(self) -> dict[str, str]:
-        """Layer-level remote_state.yml — only needed if layer doesn't exist yet."""
-        return {
-            f"{self.layer}/remote_state.yml": render_remote_state_yml(
-                directory=self.layer,
-                account_id=self.aws_account_id,
-                account_alias=self.account_alias,
-                bucket=f"inf-tfstate-{self.aws_account_id}",
-                bucket_region="us-gov-east-1",
-                profile=f"{self.aws_account_id}-{self.account_alias}",
-                region=self.aws_region,
-                aws_environment="gov",
-            )
-        }
+
+
+def handle(props: dict) -> dict:
+    cfg = _Config(**{k: v for k, v in props.items() if k in _Config.model_fields})
+    props.setdefault("layer",        cfg.layer)
+    props.setdefault("region_dir",   cfg.region_dir)
+    props.setdefault("template_repo", "template-s3-bucket")
+    # Inject layer-level remote_state.yml if the layer is new
+    props.setdefault("extra_files", {})
+    props["extra_files"].setdefault(
+        f"{cfg.layer}/remote_state.yml",
+        _render_remote_state(cfg),
+    )
+    return props
+
+
+def _render_remote_state(cfg: _Config) -> str:
+    return (
+        f"directory: \"{cfg.layer}\"\n"
+        f"profile: \"{cfg.aws_account_id}-{cfg.account_alias}\"\n"
+        f"bucket: \"inf-tfstate-{cfg.aws_account_id}\"\n"
+        f"bucket_region: \"us-gov-east-1\"\n"
+        f"region: \"{cfg.aws_region}\"\n"
+        f"account_id: \"{cfg.aws_account_id}\"\n"
+        f"account_alias: \"{cfg.account_alias}\"\n"
+        f"aws_environment: \"gov\"\n"
+    )
 ```
 
-The model enforces required fields and default values before any CodeBuild build
-is started. The `extra_files()` method produces the layer-level `remote_state.yml`
-from validated inputs — account-specific values stay in the Lambda model, not in
-the template repo.
+Because `handler.py` is versioned in the template repo, the Pydantic model and defaults
+evolve alongside the templates — no Lambda redeploy required.
+
+### 4. Lambda dispatcher — runtime fetch from template repo
 
-### 4. Lambda dispatcher
+The Lambda has no `handlers/` directory and no handler registry. Instead, it fetches
+`handler.py` directly from the template repo via the GHE API at request time and loads
+it dynamically. The template repo is identified by the `template_repo` field already
+present in the CFN Properties.
 
-A single routing table maps `product_type` to the correct handler:
+```
+lambda/
+└── app.py     ← one-time change: fetch + exec handler.py, then call handle(props)
+                  no lambda/handlers/ directory, no lambda/models/ directory
+```
+
+#### How it works (design intent for `app.py`)
 
 ```python
-PRODUCT_HANDLERS = {
-    "eks_cluster":  handle_eks,
-    "s3_bucket":    handle_s3,
-    # future: "rds_postgres": handle_rds
-}
-
-def handle_create(props: dict):
-    product_type = props.get("product_type", "eks_cluster")  # default: backward-compat
-    handler = PRODUCT_HANDLERS.get(product_type)
-    if not handler:
-        raise ValueError(f"Unknown product_type: {product_type}")
-    return handler(props)
+# 1. Read template_repo from CFN props (before TfRunRequest is constructed)
+template_repo = normalized.get("template_repo")   # e.g. "template-s3-bucket"
+if not template_repo:
+    raise ValueError("template_repo is required")
+
+# 2. Fetch handler.py from GHE via the raw contents API
+github_org  = os.environ.get("GITHUB_ORG_NAME", "SCT-Engineering")
+github_api  = os.environ.get("GITHUB_API", "https://github.e.it.census.gov/api/v3")
+handler_url = f"{github_api}/repos/{github_org}/{template_repo}/contents/handler.py"
+# ...fetch with Authorization header, base64-decode the content...
+
+# 3. Load the module dynamically
+import types, importlib
+mod = types.ModuleType("_handler")
+exec(compile(handler_source, "handler.py", "exec"), mod.__dict__)
+
+# 4. Validate the contract and dispatch
+if not (callable(getattr(mod, "handle", None)) and getattr(mod, "PRODUCT_TYPE", None)):
+    raise ValueError(f"{template_repo}/handler.py must define PRODUCT_TYPE and handle()")
+normalized = mod.handle(normalized)
+tf_req = TfRunRequest(**normalized)
 ```
 
-This is a **one-time change** to `lambda/app.py`. After it is in place, adding a new
-product type requires only a new entry in the table and a new handler function — no
-other Lambda changes.
+#### Security boundary
+
+The Lambda only fetches `handler.py` from repos whose name is in an allow-list prefix
+(`template-*` within `SCT-Engineering`). The GHE token used has **read-only** scope on
+template repos, so a compromised template repo cannot write to account repos via this
+path. Handler execution is the only place arbitrary code runs — this is intentional and
+auditable (every template repo change is a PR in the SCT-Engineering org).
+
+**Adding a new product type requires only creating a new template repo with a `handler.py`.
+No Lambda code changes, no Lambda redeployment, no registry entries.**
 
 ### 5. CloudFormation product template
 
@@ -238,12 +296,12 @@ existing EKS product config.
 The following checklist can be handed to a product team or platform engineer to
 onboard any new Terraform workload without Lambda or CodeBuild changes:
 
-- [ ] Create `SCT-Engineering/template-{product_type}` containing **only** the workload
-      delta: flat `.tf.j2` file(s) + `tf-run.data` (+ optional `.sc-automation.yml.j2`)
-- [ ] Add `lambda/models/{product_type}.py` with a Pydantic model defining required
-      inputs and an `extra_files()` method that builds the layer-level `remote_state.yml`
-      from validated SC form inputs
-- [ ] Register the handler in `lambda/app.py` `PRODUCT_HANDLERS` table
+- [ ] Create `SCT-Engineering/template-{product_type}` containing:
+      - `handler.py` — `PRODUCT_TYPE`, Pydantic model, `handle()` function
+      - flat `.tf.j2` file(s) rendered by the Proposer
+      - `tf-run.data`
+      - `.sc-automation.yml.j2` (optional; Proposer writes a default if absent)
+      **No files in the Lambda repository need to be created or modified.**
 - [ ] Create `service-catalog/{product_type}-product-template.yaml` CFN template
 - [ ] Add census config YAML and SC portfolio registration in `terraform-service-catalog-census`
 - [ ] Test end-to-end via `scripts/test_service_catalog.py` with the new product type

docs/service-catalog-census-integration.md

@@ -27,15 +27,18 @@ roles), or census-managed (portfolios, products, constraints) — and handled ac
 
 ```
 sc-lambda-ghactions/                     ← Lambda + CodeBuild buildspecs + SC product templates
-├── lambda/app.py                        ← Lambda handler (dispatcher by product_type)
-├── lambda/models/{product_type}.py      ← Pydantic input models per product type
-├── lambda/templates/{product_type}/     ← Jinja2 HCL templates per product type
+├── lambda/app.py                        ← Lambda handler (fetches + runs handler.py from template repo at runtime)
 ├── service-catalog/{product_type}-product-template.yaml  ← CFN product template
 └── deploy/                              ← Terraform: Lambda, ECR, IAM, Function URL
 
+SCT-Engineering/template-{product_type}/ ← one repo per product type; fully self-contained
+├── handler.py                           ← PRODUCT_TYPE + Pydantic model + handle()
+├── {workload}.tf.j2                     ← Jinja2 HCL templates (flat)
+├── tf-run.data                          ← tf-run steps
+└── .sc-automation.yml.j2               ← optional webhook config template
+
 terraform-sc-fleet/                      ← Fleet operations manifest (all managed workloads)
 packer-pipeline/                         ← Container build CLI
-template-{product_type}/                 ← Template repos (one per product type)
 ```
 
 ### `terraform-service-catalog-census` (census repo)

docs/template-management.md

@@ -238,7 +238,7 @@ creates the initial PR if it does not already exist on `main`.
 
 ```yaml
 # .sc-automation.yml
-product_type: eks_cluster          # Must match a registered PRODUCT_HANDLERS key
+product_type: eks_cluster          # Must match PRODUCT_TYPE in template repo's handler.py
 executor_project: sc-executor      # CodeBuild project name for the Executor build
 dry_run: true                      # If true, Executor runs tf plan only (no apply)
 template_repo: SCT-Engineering/template-eks-cluster   # Source template repo
@@ -302,11 +302,11 @@ changes and the post-apply file diff is empty, the commit step is skipped.
 
 Checklist when onboarding a new product type:
 
-- [ ] Create `SCT-Engineering/template-{product_type}` containing **only** the workload
-      delta: `{layer}/{workspace}/{workload}.tf.j2` + `tf-run.data`
-- [ ] Add a Pydantic model in `lambda/models/{product_type}.py` that validates
-      product-specific inputs and builds `TEMPLATE_VARS` + any `EXTRA_FILES`
-      (e.g. layer-level `remote_state.yml` if the target layer may not exist yet)
-- [ ] Register the handler in `lambda/app.py` `PRODUCT_HANDLERS` table
+- [ ] Create `SCT-Engineering/template-{product_type}` containing:
+      - `handler.py` — `PRODUCT_TYPE`, Pydantic model, `handle()` function
+      - flat `.tf.j2` file(s) (rendered into `${LAYER}/${REGION_DIR}/` by the Proposer)
+      - `tf-run.data`
+      - `.sc-automation.yml.j2` (optional)
+      **No files in the Lambda repository need to be created or modified.**
 - [ ] Create a CFN product template in `service-catalog/{product_type}-product-template.yaml`
 - [ ] Add the product to `terraform-service-catalog-census` (see [service-catalog-census-integration.md](service-catalog-census-integration.md))