feat: Proposer generates all workspace files (REMOTE-STATE + tf-direc…

…tory-setup.py)

- buildspec-proposer.yml: install tf-directory-setup.py from terraform/support in
  INSTALL phase; add python-dateutil + pyyaml pip deps.
  BUILD phase: after template rendering, run Python bootstrap step that:
    1. Processes REMOTE-STATE directives in tf-run.data files — derives workspace
       remote_state.yml from layer-level file (identical to tf-run.sh behavior)
    2. Runs tf-directory-setup.py --link none in each workspace with remote_state.yml —
       generates remote_state.backend.tf + .tf.s3/.local/.none variant files + symlink

- buildspec-executor.yml: add note that REMOTE-STATE and tf-directory-setup.py
  steps are idempotent — files already exist from Proposer PR, no new files created

- docs/HOW-IT-WORKS.md: expand BUILD phase step 5 to document the full file
  generation sequence including REMOTE-STATE and tf-directory-setup.py; add
  rationale explaining why all generation must happen in the Proposer

- docs/template-management.md: fix template repo structure diagram — workspace
  remote_state.yml.j2 files removed (wrong); layer-level remote_state.yml.j2 shown;
  workspace tf-run.data with REMOTE-STATE directive shown; add layout rules for
  auto.tfvars profile/region requirement and .j2 source file handling.
  Expand Proposer Build steps to cover REMOTE-STATE + tf-directory-setup.py.
  Add principle callout: PR diff is the complete truth.

buildspec-executor.yml

@@ -111,6 +111,13 @@ phases:
 
       # --- Run Terraform in target layer/region directory ---
       # tf-run auto-proceeds on non-TTY stdin (read -t timeout defaults to "y")
+      #
+      # NOTE on file-generating tf-run.data directives:
+      #   REMOTE-STATE   — generates workspace remote_state.yml from parent
+      #   COMMAND tf-directory-setup.py — generates remote_state.backend.tf + variant files
+      # The Proposer already ran both of these and committed the results in the PR.
+      # When tf-run hits these steps here they are idempotent: they overwrite files
+      # that already exist with identical content. No new files are created at apply time.
       - cd "${LAYER}/${REGION_DIR}"
       - |
         if [ "${DRY_RUN}" = "true" ]; then

buildspec-proposer.yml

@@ -47,8 +47,13 @@ phases:
       - aws s3 cp "$CENSUS_CA_S3" /etc/pki/ca-trust/source/anchors/census-ca.pem
       - update-ca-trust extract
 
-      # --- Python deps for template rendering ---
-      - pip3 install --quiet jinja2
+      # --- tf-directory-setup.py (generates remote_state.backend.tf + variant files) ---
+      # Must be available in Proposer because ALL file generation happens here, not in Executor.
+      - cp /tmp/tf-support/local-app/tf-directory-setup/tf-directory-setup.py /usr/local/bin/tf-directory-setup.py
+      - chmod +x /usr/local/bin/tf-directory-setup.py
+
+      # --- Python deps for template rendering + tf-directory-setup.py ---
+      - pip3 install --quiet jinja2 python-dateutil pyyaml
 
       # --- gh CLI (from S3, version pinned in terraform/support) ---
       - aws s3 cp "${GH_CLI_S3_PREFIX}/gh_${GH_VERSION}_linux_amd64.tar.gz" /tmp/gh.tar.gz
@@ -122,6 +127,68 @@ phases:
         print(f'Wrote {len(files)} extra file(s)')
         "
 
+      # --- Bootstrap workspace state files (REMOTE-STATE + tf-directory-setup.py) ---
+      # tf-run.sh's REMOTE-STATE directive generates workspace remote_state.yml at apply time.
+      # tf-run.data COMMAND steps run tf-directory-setup.py to generate remote_state.backend.tf
+      # and the three variant files (.tf.s3 / .tf.local / .tf.none) + activate the symlink.
+      #
+      # ALL of this must happen in the Proposer so every generated file appears in the PR diff.
+      # The Executor must not silently create files; it inherits what the PR committed.
+      - |
+        python3 - <<'PYEOF'
+        import os, re, subprocess, sys, pathlib
+
+        repo_root = pathlib.Path('.')
+
+        for tfrun_data in sorted(repo_root.rglob('tf-run.data')):
+            ws_dir = tfrun_data.parent
+            # Skip .git internals
+            if any(p.startswith('.git') for p in ws_dir.parts):
+                continue
+
+            content = tfrun_data.read_text()
+            lines = [l.strip() for l in content.splitlines() if l.strip() and not l.startswith('#')]
+
+            # ── Step 1: REMOTE-STATE ──────────────────────────────────────────────────
+            # Mirrors tf-run.sh: read ../remote_state.yml, append /{workspace_name} to
+            # the directory field, write workspace-level remote_state.yml.
+            if any(l.startswith('REMOTE-STATE') for l in lines):
+                parent_rs = ws_dir.parent / 'remote_state.yml'
+                if not parent_rs.exists():
+                    print(f'WARNING: {ws_dir}: REMOTE-STATE in tf-run.data but no '
+                          f'parent remote_state.yml found — skipping', flush=True)
+                    continue
+                parent_text = parent_rs.read_text()
+                subdir = ws_dir.name
+                # Replicate: sed -E s#(^directory.*)\"\'#\1/{subdir}\"
+                ws_rs_text = re.sub(
+                    r'^(directory\s*:\s*")([^"]+)(")',
+                    lambda m: m.group(1) + m.group(2).rstrip('/') + '/' + subdir + m.group(3),
+                    parent_text, count=1, flags=re.MULTILINE
+                )
+                ws_rs = ws_dir / 'remote_state.yml'
+                ws_rs.write_text(ws_rs_text)
+                print(f'REMOTE-STATE: wrote {ws_rs}  (directory += /{subdir})', flush=True)
+
+            # ── Step 2: tf-directory-setup.py ────────────────────────────────────────
+            # Run whenever the workspace has a remote_state.yml (just written or from
+            # the template). Generates remote_state.backend.tf + 3 variant files.
+            # --link none: initial state; the Executor will re-link to s3 after first apply.
+            rs_file = ws_dir / 'remote_state.yml'
+            if rs_file.exists():
+                result = subprocess.run(
+                    [sys.executable, '/usr/local/bin/tf-directory-setup.py', '--link', 'none'],
+                    cwd=str(ws_dir), capture_output=True, text=True
+                )
+                print(result.stdout, end='', flush=True)
+                if result.returncode != 0:
+                    print(f'ERROR: tf-directory-setup.py failed in {ws_dir}:\n{result.stderr}',
+                          file=sys.stderr, flush=True)
+                    sys.exit(result.returncode)
+
+        print('Bootstrap complete.', flush=True)
+        PYEOF
+
       # --- Commit and push ---
       - git add -A
       - |

docs/HOW-IT-WORKS.md

@@ -246,9 +246,26 @@ TEMPLATE_REPO, TEMPLATE_VARS, EXTRA_FILES, GITHUB_TOKEN
    - Copy rendered + non-template files into account repo at same relative paths
 4. If `EXTRA_FILES` is non-empty:
    - Parse the JSON dict; write each `path → content` entry directly (overrides templates)
-5. `git add -A && git commit -m "feat: sc-automation propose" --allow-empty`
-6. `git push origin ${GIT_BRANCH} --force-with-lease`
-7. `gh pr create --base main --head ${GIT_BRANCH} --title "..." --body "..."` (idempotent — skips if PR already exists)
+5. **Bootstrap workspace state files** (all file generation must be in the Proposer PR):
+   - For every `tf-run.data` containing a `REMOTE-STATE` directive:
+     - Read `../remote_state.yml` (layer-level) and append `/{workspace_name}` to the `directory` field
+     - Write the result as `remote_state.yml` in the workspace directory
+     - This mirrors exactly what `tf-run.sh`'s `REMOTE-STATE` handler does at apply time
+   - For every workspace directory that now has a `remote_state.yml`:
+     - Run `tf-directory-setup.py --link none` to generate:
+       - `remote_state.backend.tf` — the S3 backend block
+       - `remote_state.{dir}.tf.s3` — production variant
+       - `remote_state.{dir}.tf.local` — local-state variant
+       - `remote_state.{dir}.tf.none` — empty stub (activated by `--link none`)
+       - Symlink `remote_state.{dir}.tf → remote_state.{dir}.tf.none` (bootstrap state)
+     - `--link none` is the correct bootstrap choice: state does not exist yet; the Executor will re-link to `.s3` after the first successful apply
+   > **Why here and not in the Executor?** `tf-run.sh` generates these files at apply time via
+   > `REMOTE-STATE` directive and `COMMAND tf-directory-setup.py` steps. If the Executor generates
+   > them, they are invisible to reviewers. By running this in the Proposer, every generated file
+   > appears in the PR diff and is subject to human review before any infrastructure changes.
+6. `git add -A && git commit -m "feat: sc-automation propose" --allow-empty`
+7. `git push origin ${GIT_BRANCH} --force-with-lease`
+8. `gh pr create --base main --head ${GIT_BRANCH} --title "..." --body "..."` (idempotent — skips if PR already exists)
 
 ### 6. CodeBuild - POST_BUILD phase
 

docs/template-management.md

@@ -94,19 +94,24 @@ template-{product_type}/
 ├── variables.d/
 │   ├── variables.common.tf
 │   └── variables.tfstate.tf
+│   └── {region}.variables.common.auto.tfvars.j2  # ← must emit profile + region keys
 ├── infrastructure/
+│   ├── remote_state.yml.j2              # ← layer-level; Proposer renders to remote_state.yml
 │   ├── east/
-│   │   ├── remote_state.yml.j2          # ← Jinja2: rendered by Proposer
+│   │   ├── tf-run.data                  # ← must contain REMOTE-STATE directive
 │   │   └── {workload}.tf.j2             # ← Jinja2: rendered by Proposer
 │   └── west/
-│       ├── remote_state.yml.j2
-│       └── {workload}.tf.j2
+│       ├── tf-run.data                  # ← must contain REMOTE-STATE directive
+│       └── {workload}.tf.j2             # ← Jinja2: rendered by Proposer
 └── README.md
 ```
 
-Files ending in `.j2` are Jinja2 templates. The Proposer CodeBuild build renders
-them using the product inputs and commits the rendered result (without the `.j2`
-extension) to the new account repo branch.
+**Key layout rules:**
+
+- `remote_state.yml.j2` lives at the **layer level** (`infrastructure/`, `common/`, `vpc/`), **not** inside workspace subdirectories.  The Proposer's REMOTE-STATE processor derives each workspace's `remote_state.yml` from the layer-level file by appending `/{workspace_name}` to the `directory` field — identical to what `tf-run.sh` does at apply time.
+- Each workspace directory (`east/`, `west/`, `global/`) **must** include a `tf-run.data` file with a `REMOTE-STATE` directive so the Proposer knows to generate its `remote_state.yml`.
+- The `.auto.tfvars.j2` file must render `profile = "..."` and `region = "..."` entries at the top level — `tf-run.sh` auto-discovers profile and region by grepping `*.tfvars`, so these values must be present for placeholder substitution (`%%REGION%%`, `%%PROFILE%%`, etc.) to work correctly.
+- Files ending in `.j2` are Jinja2 templates. The Proposer renders them using the product input variables and commits the result (without the `.j2` extension) to the work branch. The `.j2` source files are **not** committed.
 
 ---
 
@@ -139,11 +144,29 @@ The Proposer CodeBuild build (started by the Lambda via `codebuild:StartBuild`)
 1. Clone the template repo (full repo or `source_path` subdirectory)
 2. For each `.j2` file found:
    - Render it using `jinja2.Environment` with the product input variables
-   - Write the rendered output alongside the source file (without `.j2` extension)
-   - Remove the `.j2` source file from the working tree
-3. Add rendered `remote_state.yml` files using actual account/bucket values
-4. Write `.sc-automation.yml` to the repo root if it does not already exist on `main`
-5. Commit all rendered files to a new branch (`proposal/{timestamp}`) and open a PR
+   - Write the rendered output to the same relative path (without `.j2` extension)
+3. Write any `EXTRA_FILES` entries (direct path → content map; overrides template output)
+4. **REMOTE-STATE processing** — for every `tf-run.data` with a `REMOTE-STATE` directive:
+   - Read the layer-level `remote_state.yml` (e.g. `infrastructure/remote_state.yml`)
+   - Append `/{workspace_basename}` to the `directory` field via regex substitution
+   - Write the result as `remote_state.yml` in the workspace directory (e.g. `infrastructure/west/remote_state.yml`)
+   - This is the same transformation `tf-run.sh` performs at apply time for the `REMOTE-STATE` directive
+5. **`tf-directory-setup.py` bootstrap** — for every workspace directory that now has a `remote_state.yml`:
+   - Run `tf-directory-setup.py --link none` to generate:
+     - `remote_state.backend.tf` — the S3 backend configuration block
+     - `remote_state.{dir}.tf.s3` — S3-backed remote state variant
+     - `remote_state.{dir}.tf.local` — local state file variant
+     - `remote_state.{dir}.tf.none` — empty no-op stub (active on first propose)
+     - Symlink `remote_state.{dir}.tf → remote_state.{dir}.tf.none`
+   - `--link none` is the correct bootstrap value: Terraform state does not exist yet for a new workspace
+   - After a successful `tf apply` in the Executor, the `tf-run.data` `COMMAND tf-directory-setup.py --link s3` step re-links to `.s3`
+6. Write `.sc-automation.yml` to the repo root if it does not already exist on `main`
+7. Commit all files (rendered templates + generated state bootstrap files) to a work branch and open a PR
+
+> **Principle: the PR diff is the complete truth.** Every file the Executor will find at
+> apply time must already be committed in the Proposer PR. Neither `REMOTE-STATE` nor
+> `tf-directory-setup.py` should create new files during `tf-run apply` — those steps become
+> idempotent re-generations of files already in the repo.
 
 The PR is reviewed by a platform engineer before merging. On merge, the webhook
 handler reads `.sc-automation.yml` and automatically starts the executor CodeBuild build.