From 77f9a49d8417579f183028328321299209306a89 Mon Sep 17 00:00:00 2001 From: Dave Arnold Date: Wed, 20 May 2026 13:32:47 -0400 Subject: [PATCH] feat: Proposer generates all workspace files (REMOTE-STATE + tf-directory-setup.py) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - buildspec-proposer.yml: install tf-directory-setup.py from terraform/support in INSTALL phase; add python-dateutil + pyyaml pip deps. BUILD phase: after template rendering, run Python bootstrap step that: 1. Processes REMOTE-STATE directives in tf-run.data files — derives workspace remote_state.yml from layer-level file (identical to tf-run.sh behavior) 2. Runs tf-directory-setup.py --link none in each workspace with remote_state.yml — generates remote_state.backend.tf + .tf.s3/.local/.none variant files + symlink - buildspec-executor.yml: add note that REMOTE-STATE and tf-directory-setup.py steps are idempotent — files already exist from Proposer PR, no new files created - docs/HOW-IT-WORKS.md: expand BUILD phase step 5 to document the full file generation sequence including REMOTE-STATE and tf-directory-setup.py; add rationale explaining why all generation must happen in the Proposer - docs/template-management.md: fix template repo structure diagram — workspace remote_state.yml.j2 files removed (wrong); layer-level remote_state.yml.j2 shown; workspace tf-run.data with REMOTE-STATE directive shown; add layout rules for auto.tfvars profile/region requirement and .j2 source file handling. Expand Proposer Build steps to cover REMOTE-STATE + tf-directory-setup.py. Add principle callout: PR diff is the complete truth. --- buildspec-executor.yml | 7 ++++ buildspec-proposer.yml | 71 +++++++++++++++++++++++++++++++++++-- docs/HOW-IT-WORKS.md | 23 ++++++++++-- docs/template-management.md | 45 +++++++++++++++++------ 4 files changed, 130 insertions(+), 16 deletions(-) diff --git a/buildspec-executor.yml b/buildspec-executor.yml index 8bf5620..e9d6629 100644 --- a/buildspec-executor.yml +++ b/buildspec-executor.yml @@ -111,6 +111,13 @@ phases: # --- Run Terraform in target layer/region directory --- # tf-run auto-proceeds on non-TTY stdin (read -t timeout defaults to "y") + # + # NOTE on file-generating tf-run.data directives: + # REMOTE-STATE — generates workspace remote_state.yml from parent + # COMMAND tf-directory-setup.py — generates remote_state.backend.tf + variant files + # The Proposer already ran both of these and committed the results in the PR. + # When tf-run hits these steps here they are idempotent: they overwrite files + # that already exist with identical content. No new files are created at apply time. - cd "${LAYER}/${REGION_DIR}" - | if [ "${DRY_RUN}" = "true" ]; then diff --git a/buildspec-proposer.yml b/buildspec-proposer.yml index ddccbaa..624710b 100644 --- a/buildspec-proposer.yml +++ b/buildspec-proposer.yml @@ -47,8 +47,13 @@ phases: - aws s3 cp "$CENSUS_CA_S3" /etc/pki/ca-trust/source/anchors/census-ca.pem - update-ca-trust extract - # --- Python deps for template rendering --- - - pip3 install --quiet jinja2 + # --- tf-directory-setup.py (generates remote_state.backend.tf + variant files) --- + # Must be available in Proposer because ALL file generation happens here, not in Executor. + - cp /tmp/tf-support/local-app/tf-directory-setup/tf-directory-setup.py /usr/local/bin/tf-directory-setup.py + - chmod +x /usr/local/bin/tf-directory-setup.py + + # --- Python deps for template rendering + tf-directory-setup.py --- + - pip3 install --quiet jinja2 python-dateutil pyyaml # --- gh CLI (from S3, version pinned in terraform/support) --- - aws s3 cp "${GH_CLI_S3_PREFIX}/gh_${GH_VERSION}_linux_amd64.tar.gz" /tmp/gh.tar.gz @@ -122,6 +127,68 @@ phases: print(f'Wrote {len(files)} extra file(s)') " + # --- Bootstrap workspace state files (REMOTE-STATE + tf-directory-setup.py) --- + # tf-run.sh's REMOTE-STATE directive generates workspace remote_state.yml at apply time. + # tf-run.data COMMAND steps run tf-directory-setup.py to generate remote_state.backend.tf + # and the three variant files (.tf.s3 / .tf.local / .tf.none) + activate the symlink. + # + # ALL of this must happen in the Proposer so every generated file appears in the PR diff. + # The Executor must not silently create files; it inherits what the PR committed. + - | + python3 - <<'PYEOF' + import os, re, subprocess, sys, pathlib + + repo_root = pathlib.Path('.') + + for tfrun_data in sorted(repo_root.rglob('tf-run.data')): + ws_dir = tfrun_data.parent + # Skip .git internals + if any(p.startswith('.git') for p in ws_dir.parts): + continue + + content = tfrun_data.read_text() + lines = [l.strip() for l in content.splitlines() if l.strip() and not l.startswith('#')] + + # ── Step 1: REMOTE-STATE ────────────────────────────────────────────────── + # Mirrors tf-run.sh: read ../remote_state.yml, append /{workspace_name} to + # the directory field, write workspace-level remote_state.yml. + if any(l.startswith('REMOTE-STATE') for l in lines): + parent_rs = ws_dir.parent / 'remote_state.yml' + if not parent_rs.exists(): + print(f'WARNING: {ws_dir}: REMOTE-STATE in tf-run.data but no ' + f'parent remote_state.yml found — skipping', flush=True) + continue + parent_text = parent_rs.read_text() + subdir = ws_dir.name + # Replicate: sed -E s#(^directory.*)\"\'#\1/{subdir}\" + ws_rs_text = re.sub( + r'^(directory\s*:\s*")([^"]+)(")', + lambda m: m.group(1) + m.group(2).rstrip('/') + '/' + subdir + m.group(3), + parent_text, count=1, flags=re.MULTILINE + ) + ws_rs = ws_dir / 'remote_state.yml' + ws_rs.write_text(ws_rs_text) + print(f'REMOTE-STATE: wrote {ws_rs} (directory += /{subdir})', flush=True) + + # ── Step 2: tf-directory-setup.py ──────────────────────────────────────── + # Run whenever the workspace has a remote_state.yml (just written or from + # the template). Generates remote_state.backend.tf + 3 variant files. + # --link none: initial state; the Executor will re-link to s3 after first apply. + rs_file = ws_dir / 'remote_state.yml' + if rs_file.exists(): + result = subprocess.run( + [sys.executable, '/usr/local/bin/tf-directory-setup.py', '--link', 'none'], + cwd=str(ws_dir), capture_output=True, text=True + ) + print(result.stdout, end='', flush=True) + if result.returncode != 0: + print(f'ERROR: tf-directory-setup.py failed in {ws_dir}:\n{result.stderr}', + file=sys.stderr, flush=True) + sys.exit(result.returncode) + + print('Bootstrap complete.', flush=True) + PYEOF + # --- Commit and push --- - git add -A - | diff --git a/docs/HOW-IT-WORKS.md b/docs/HOW-IT-WORKS.md index bc42ba3..5bb5b5b 100644 --- a/docs/HOW-IT-WORKS.md +++ b/docs/HOW-IT-WORKS.md @@ -246,9 +246,26 @@ TEMPLATE_REPO, TEMPLATE_VARS, EXTRA_FILES, GITHUB_TOKEN - Copy rendered + non-template files into account repo at same relative paths 4. If `EXTRA_FILES` is non-empty: - Parse the JSON dict; write each `path → content` entry directly (overrides templates) -5. `git add -A && git commit -m "feat: sc-automation propose" --allow-empty` -6. `git push origin ${GIT_BRANCH} --force-with-lease` -7. `gh pr create --base main --head ${GIT_BRANCH} --title "..." --body "..."` (idempotent — skips if PR already exists) +5. **Bootstrap workspace state files** (all file generation must be in the Proposer PR): + - For every `tf-run.data` containing a `REMOTE-STATE` directive: + - Read `../remote_state.yml` (layer-level) and append `/{workspace_name}` to the `directory` field + - Write the result as `remote_state.yml` in the workspace directory + - This mirrors exactly what `tf-run.sh`'s `REMOTE-STATE` handler does at apply time + - For every workspace directory that now has a `remote_state.yml`: + - Run `tf-directory-setup.py --link none` to generate: + - `remote_state.backend.tf` — the S3 backend block + - `remote_state.{dir}.tf.s3` — production variant + - `remote_state.{dir}.tf.local` — local-state variant + - `remote_state.{dir}.tf.none` — empty stub (activated by `--link none`) + - Symlink `remote_state.{dir}.tf → remote_state.{dir}.tf.none` (bootstrap state) + - `--link none` is the correct bootstrap choice: state does not exist yet; the Executor will re-link to `.s3` after the first successful apply + > **Why here and not in the Executor?** `tf-run.sh` generates these files at apply time via + > `REMOTE-STATE` directive and `COMMAND tf-directory-setup.py` steps. If the Executor generates + > them, they are invisible to reviewers. By running this in the Proposer, every generated file + > appears in the PR diff and is subject to human review before any infrastructure changes. +6. `git add -A && git commit -m "feat: sc-automation propose" --allow-empty` +7. `git push origin ${GIT_BRANCH} --force-with-lease` +8. `gh pr create --base main --head ${GIT_BRANCH} --title "..." --body "..."` (idempotent — skips if PR already exists) ### 6. CodeBuild - POST_BUILD phase diff --git a/docs/template-management.md b/docs/template-management.md index 5a20ec4..3cbcb93 100644 --- a/docs/template-management.md +++ b/docs/template-management.md @@ -94,19 +94,24 @@ template-{product_type}/ ├── variables.d/ │ ├── variables.common.tf │ └── variables.tfstate.tf +│ └── {region}.variables.common.auto.tfvars.j2 # ← must emit profile + region keys ├── infrastructure/ +│ ├── remote_state.yml.j2 # ← layer-level; Proposer renders to remote_state.yml │ ├── east/ -│ │ ├── remote_state.yml.j2 # ← Jinja2: rendered by Proposer +│ │ ├── tf-run.data # ← must contain REMOTE-STATE directive │ │ └── {workload}.tf.j2 # ← Jinja2: rendered by Proposer │ └── west/ -│ ├── remote_state.yml.j2 -│ └── {workload}.tf.j2 +│ ├── tf-run.data # ← must contain REMOTE-STATE directive +│ └── {workload}.tf.j2 # ← Jinja2: rendered by Proposer └── README.md ``` -Files ending in `.j2` are Jinja2 templates. The Proposer CodeBuild build renders -them using the product inputs and commits the rendered result (without the `.j2` -extension) to the new account repo branch. +**Key layout rules:** + +- `remote_state.yml.j2` lives at the **layer level** (`infrastructure/`, `common/`, `vpc/`), **not** inside workspace subdirectories. The Proposer's REMOTE-STATE processor derives each workspace's `remote_state.yml` from the layer-level file by appending `/{workspace_name}` to the `directory` field — identical to what `tf-run.sh` does at apply time. +- Each workspace directory (`east/`, `west/`, `global/`) **must** include a `tf-run.data` file with a `REMOTE-STATE` directive so the Proposer knows to generate its `remote_state.yml`. +- The `.auto.tfvars.j2` file must render `profile = "..."` and `region = "..."` entries at the top level — `tf-run.sh` auto-discovers profile and region by grepping `*.tfvars`, so these values must be present for placeholder substitution (`%%REGION%%`, `%%PROFILE%%`, etc.) to work correctly. +- Files ending in `.j2` are Jinja2 templates. The Proposer renders them using the product input variables and commits the result (without the `.j2` extension) to the work branch. The `.j2` source files are **not** committed. --- @@ -139,11 +144,29 @@ The Proposer CodeBuild build (started by the Lambda via `codebuild:StartBuild`) 1. Clone the template repo (full repo or `source_path` subdirectory) 2. For each `.j2` file found: - Render it using `jinja2.Environment` with the product input variables - - Write the rendered output alongside the source file (without `.j2` extension) - - Remove the `.j2` source file from the working tree -3. Add rendered `remote_state.yml` files using actual account/bucket values -4. Write `.sc-automation.yml` to the repo root if it does not already exist on `main` -5. Commit all rendered files to a new branch (`proposal/{timestamp}`) and open a PR + - Write the rendered output to the same relative path (without `.j2` extension) +3. Write any `EXTRA_FILES` entries (direct path → content map; overrides template output) +4. **REMOTE-STATE processing** — for every `tf-run.data` with a `REMOTE-STATE` directive: + - Read the layer-level `remote_state.yml` (e.g. `infrastructure/remote_state.yml`) + - Append `/{workspace_basename}` to the `directory` field via regex substitution + - Write the result as `remote_state.yml` in the workspace directory (e.g. `infrastructure/west/remote_state.yml`) + - This is the same transformation `tf-run.sh` performs at apply time for the `REMOTE-STATE` directive +5. **`tf-directory-setup.py` bootstrap** — for every workspace directory that now has a `remote_state.yml`: + - Run `tf-directory-setup.py --link none` to generate: + - `remote_state.backend.tf` — the S3 backend configuration block + - `remote_state.{dir}.tf.s3` — S3-backed remote state variant + - `remote_state.{dir}.tf.local` — local state file variant + - `remote_state.{dir}.tf.none` — empty no-op stub (active on first propose) + - Symlink `remote_state.{dir}.tf → remote_state.{dir}.tf.none` + - `--link none` is the correct bootstrap value: Terraform state does not exist yet for a new workspace + - After a successful `tf apply` in the Executor, the `tf-run.data` `COMMAND tf-directory-setup.py --link s3` step re-links to `.s3` +6. Write `.sc-automation.yml` to the repo root if it does not already exist on `main` +7. Commit all files (rendered templates + generated state bootstrap files) to a work branch and open a PR + +> **Principle: the PR diff is the complete truth.** Every file the Executor will find at +> apply time must already be committed in the Proposer PR. Neither `REMOTE-STATE` nor +> `tf-directory-setup.py` should create new files during `tf-run apply` — those steps become +> idempotent re-generations of files already in the repo. The PR is reviewed by a platform engineer before merging. On merge, the webhook handler reads `.sc-automation.yml` and automatically starts the executor CodeBuild build.