diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 898c348..99d33bd 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -174,26 +174,20 @@ Properties: ## Rebuilding the Lambda Image -When `template_automation/app.py` or other Lambda source files change: +When `template_automation/app.py` or other Lambda source files change, use `packer-pipeline`: ```bash -# 1. Zip source and upload to S3 cd lambda-template-repo-generator -zip -r ~/tmp/lambda-source.zip . -x "*.git*" -x "design-docs/*" -x "__pycache__/*" -x "*.pyc" -x "deploy/.terraform/*" -x "deploy/terraform.tfstate*" -UUID=$(python3 -c "import uuid; print(uuid.uuid4())") source ~/aws-creds -aws s3 cp ~/tmp/lambda-source.zip \ - "s3://csvd-packer-pipeline-builds/packer-builds/eks-terragrunt-repo-generator/source/${UUID}/repo.zip" \ - --region us-gov-west-1 +packer-pipeline --config csvd_config_packer.hcl +``` + +This handles zipping the source, uploading to S3, and triggering the +`eks-terragrunt-repo-generator-builder` CodeBuild project automatically. -# 2. Start the packer CodeBuild build -aws codebuild start-build \ - --project-name eks-terragrunt-repo-generator-builder \ - --region us-gov-west-1 \ - --source-type-override S3 \ - --source-location-override "csvd-packer-pipeline-builds/packer-builds/eks-terragrunt-repo-generator/source/${UUID}/repo.zip" +After the build completes (SUCCEEDED), force the Lambda to pull the new image: -# 3. After build SUCCEEDED, force Lambda to pull the new image +```bash aws lambda update-function-code \ --function-name eks-terragrunt-repo-gen-template-automation \ --image-uri "229685449397.dkr.ecr.us-gov-west-1.amazonaws.com/eks-terragrunt-repo-generator/lambda:latest" \ @@ -214,6 +208,40 @@ python scripts/cleanup_test_repos.py --- +## Python & CLI Automation Standards + +All automation scripts in this project are written in **Python 3**. Use the following libraries +as the standard stack — do not introduce alternatives without good reason: + +| Purpose | Library | +|---------|---------| +| Data validation / config models | `pydantic` (v2) | +| Rich terminal output / progress | `rich` | +| CLI argument parsing | `typer` (preferred) or `argparse` | +| AWS API calls | `boto3` | +| YAML config files | `pyyaml` | +| HTTP calls | `httpx` or `requests` | + +### `AWS_DEFAULT_REGION` — always required + +The account is in `us-gov-west-1`. Many boto3 calls and the AWS CLI silently fail or +target the wrong region if `AWS_DEFAULT_REGION` is not set. + +**Always export it before any AWS CLI or boto3 script:** + +```bash +export AWS_DEFAULT_REGION=us-gov-west-1 +source ~/aws-creds +``` + +### SC Template Parameters + +`aws_account_id` and `aws_region` are **not** on the SC product form — the CFN template +resolves them automatically via `!Sub "${AWS::AccountId}"` and `!Sub "${AWS::Region}"` +before the Lambda is called. Do not add them back as user-facing parameters. + +--- + ## What NOT to Do - ❌ Do not rewrite repo creation logic in Lambda Python — all repo creation runs in CodeBuild via `terraform-eks-deployment` @@ -221,6 +249,8 @@ python scripts/cleanup_test_repos.py - ✅ DO use `CSVD/terraform-github-repo` (https://github.e.it.census.gov/CSVD/terraform-github-repo) — internal module, supports `template_repo` + `managed_extra_files` - ❌ Do not pass `vpc_id` to the Lambda — use `vpc_name` - ❌ Do not re-add `LambdaFunctionArn` as a CFN parameter — use `!Sub "arn:..."` directly +- ❌ Do not re-add `AWSAccountId` or `AwsRegion` as SC product form parameters — use `!Sub` auto-resolution - ❌ Do not use SSH-based module sources (`git::ssh://`) — Census proxy blocks SSH host key exchange; use HTTPS - ❌ Do not write temp files or command output to `/tmp` — use `~/tmp` (i.e. `/home/a/arnol377/tmp`) instead - ❌ Do not use the `terraform` command directly — always use the `tf` alias (e.g. `tf plan`, `tf apply`, `tf init`) +- ❌ Do not run AWS CLI or boto3 without first exporting `AWS_DEFAULT_REGION=us-gov-west-1` diff --git a/CLOUDFORMATION_CUSTOM_RESOURCE_MIGRATION.md b/CLOUDFORMATION_CUSTOM_RESOURCE_MIGRATION.md index dcd2476..3f79744 100644 --- a/CLOUDFORMATION_CUSTOM_RESOURCE_MIGRATION.md +++ b/CLOUDFORMATION_CUSTOM_RESOURCE_MIGRATION.md @@ -229,7 +229,8 @@ Sees outputs: 1. **Deploy Lambda** with updated code: ```bash cd /home/a/arnol377/git/lambda-template-repo-generator - packer-pipeline build --config config_packer.hcl + source ~/aws-creds + packer-pipeline --config csvd_config_packer.hcl ``` 2. **Update Infrastructure**: diff --git a/DEPLOYMENT.md b/DEPLOYMENT.md index 9c73e58..ca69b3b 100644 --- a/DEPLOYMENT.md +++ b/DEPLOYMENT.md @@ -64,7 +64,7 @@ terraform apply -var-file=varfiles/default.tfvars cd /path/to/lambda-template-repo-generator # Build container image via CodeBuild (waits for completion, ~4 minutes) -packer-pipeline --config config_packer.hcl --wait +packer-pipeline --config csvd_config_packer.hcl ``` This will: @@ -259,7 +259,7 @@ When you change Lambda code in `template_automation/`: ```bash # 1. Build new container -packer-pipeline --config config_packer.hcl --wait +packer-pipeline --config csvd_config_packer.hcl # 2. Update Lambda to new image aws lambda update-function-code \ diff --git a/csvd_config_packer.hcl b/csvd_config_packer.hcl index 5c64bb6..4db971a 100644 --- a/csvd_config_packer.hcl +++ b/csvd_config_packer.hcl @@ -109,92 +109,3 @@ packer_pipeline { } ] } - - // Required parameters - packer_template_file = "packer.pkr.hcl" // Relative path within the repo to the Packer template - s3_bucket = "csvd-packer-pipeline-builds" // S3 bucket for artifacts (derived from environment_name) - assets_bucket = "csvd-packer-pipeline-assets" // S3 bucket containing tool assets (derived from environment_name) - codebuild_project_name = "eks-terragrunt-repo-generator-builder" // Name for the CodeBuild project - - // Tools configuration - tools = [ - { - name = "packer" - version = "1.10.3" - zip_path = "packer_1.10.3_linux_amd64.zip" - binary_name = "packer" - install_path = "/usr/local/bin" - } - ] - - // AWS Account Configuration - account_number = "229685449397" // AWS account number - partition = "aws-us-gov" // AWS partition (aws or aws-us-gov) - - // Role management - create_role = false // Role already exists; provide ARN directly - codebuild_role_arn = "arn:aws-us-gov:iam::229685449397:role/CodeBuildPackerRole-eks-terragrunt-repo-generator-builder" - - // Region and partition configuration - aws_region = "us-gov-west-1" // AWS region - gov_cloud = true // Explicitly set GovCloud partition - - // Optional parameters with defaults - s3_key_prefix = "packer-builds/eks-terragrunt-repo-generator" // Prefix for S3 keys - compute_type = "BUILD_GENERAL1_MEDIUM" // CodeBuild compute type - image = "aws/codebuild/amazonlinux2-x86_64-standard:3.0" // CodeBuild image - buildspec_template = "buildspec.yml.j2" // Buildspec template file - - // Post-build commands to push Docker image to ECR - additional_post_build_commands = "docker push ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${ECR_REPOSITORY}:${IMAGE_TAG}" - - // Exclude patterns for zip creation - exclude_dirs = [ - "design-docs", - "docs", - "dist", - "events", - "scripts", - "tests" - ] - - // VPC Configuration with the specified details - vpc_config { - vpc_id = "vpc-00576a396ec570b94" // Specified VPC - subnet_ids = ["subnet-0b1992a84536c581b"] // Subnet ID - security_group_ids = ["sg-0641c697588b9aa6b"] // Security group ID - } - - // Environment variables for the CodeBuild environment - environment_variables = { - REPOSITORY_NAME = "eks-terragrunt-repo-generator-lambda" - ECR_REPOSITORY = "eks-terragrunt-repo-generator/lambda" - AWS_ACCOUNT_ID = "229685449397" - IMAGE_TAG = "latest" - HTTP_PROXY = "http://proxy.tco.census.gov:3128" - HTTPS_PROXY = "http://proxy.tco.census.gov:3128" - // NO_PROXY: things that should bypass the proxy entirely - // - 169.254.* = EC2 metadata / ECS task credentials - // - .s3.*amazonaws.com = S3 via VPC gateway endpoint - // - .dkr.ecr / .ecr = private ECR via internal routing - // - sts / logs = AWS service endpoints via internal routing - // - github.e.it / nexus = internal census hosts - // Everything else (pypi.org, files.pythonhosted.org, public.ecr.aws) goes through proxy - NO_PROXY = "169.254.169.254,169.254.170.2,.s3.us-gov-west-1.amazonaws.com,.s3.amazonaws.com,.s3-fips.us-gov-west-1.amazonaws.com,.dkr.ecr.us-gov-west-1.amazonaws.com,.ecr.us-gov-west-1.amazonaws.com,sts.us-gov-west-1.amazonaws.com,logs.us-gov-west-1.amazonaws.com,github.e.it.census.gov,nexus.it.census.gov" - ECR_REGISTRY = "229685449397.dkr.ecr.us-gov-west-1.amazonaws.com" // ECR registry URL - } - - // ECR Image Cloning Configuration - ecr_registry_name = "eks-terragrunt-repo-generator" // ECR registry prefix for cloned images - - ecr_clone_images = [ - { - name = "lambda-python" - tag = "3.11" - source_registry = "public.ecr.aws" - source_image = "lambda/python" - source_tag = "3.11" - enabled = true - } - ] -} \ No newline at end of file diff --git a/deploy/.terraform_commits b/deploy/.terraform_commits index feb8b91..e1a72c3 100644 --- a/deploy/.terraform_commits +++ b/deploy/.terraform_commits @@ -160,5 +160,23 @@ "commit_message": "fix: use PAT (ghe-runner/github-token) for Terraform GitHub provider in CodeBuild\n\nThe standard github_token (/eks-cluster-deployment/github_token) is a GitHub\nApp installation token (ghs_ prefix) which cannot access /api/v3/user. This\nendpoint is always called by the CSVD terraform-github-repo module's\ndata.github_user.current resource.\n\nChanges:\n- app.py: check TF_GITHUB_TOKEN_SECRET_NAME env var first for CodeBuild token;\n falls back to GITHUB_TOKEN_SECRET_NAME if not set\n- deploy/main.tf: add TF_GITHUB_TOKEN_SECRET_NAME=ghe-runner/github-token env var\n- deploy/main.tf: add IAM policy granting Lambda access to ghe-runner/github-token", "author": "Your Name", "timestamp": "2026-04-07T13:10:20.067727" + }, + { + "commit_hash": "e6547ed0a07eaddd227ba8ab7b278f03e4896a91", + "commit_message": "docs: add ECA demo script with talking points and Q&A prep", + "author": "Your Name", + "timestamp": "2026-04-20T13:33:25.979480" + }, + { + "commit_hash": "e6547ed0a07eaddd227ba8ab7b278f03e4896a91", + "commit_message": "docs: add ECA demo script with talking points and Q&A prep", + "author": "Your Name", + "timestamp": "2026-04-21T15:45:48.422507" + }, + { + "commit_hash": "e6547ed0a07eaddd227ba8ab7b278f03e4896a91", + "commit_message": "docs: add ECA demo script with talking points and Q&A prep", + "author": "Your Name", + "timestamp": "2026-04-21T16:04:00.320816" } ] \ No newline at end of file diff --git a/deploy/terraform.tfstate b/deploy/terraform.tfstate index 8c4f88c..4bca3e1 100644 --- a/deploy/terraform.tfstate +++ b/deploy/terraform.tfstate @@ -1,7 +1,7 @@ { "version": 4, "terraform_version": "1.9.1", - "serial": 210, + "serial": 217, "lineage": "637f189b-ce2c-766c-35d1-8b43eb7ae216", "outputs": { "api_endpoint": { @@ -56,6 +56,38 @@ } ] }, + { + "mode": "data", + "type": "aws_iam_role", + "name": "codebuild", + "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]", + "instances": [ + { + "index_key": 0, + "schema_version": 0, + "attributes": { + "arn": "arn:aws-us-gov:iam::229685449397:role/CodeBuildPackerRole-eks-terragrunt-repo-generator-builder", + "assume_role_policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"codebuild.amazonaws.com\"},\"Action\":\"sts:AssumeRole\"}]}", + "create_date": "2026-02-20T19:54:55Z", + "description": "IAM role for CodeBuild Packer project: eks-terragrunt-repo-generator-builder", + "id": "CodeBuildPackerRole-eks-terragrunt-repo-generator-builder", + "max_session_duration": 3600, + "name": "CodeBuildPackerRole-eks-terragrunt-repo-generator-builder", + "path": "/", + "permissions_boundary": "", + "role_last_used": [ + { + "last_used_date": "2026-04-21T19:59:20Z", + "region": "us-gov-west-1" + } + ], + "tags": {}, + "unique_id": "AROATK6SR2K2WWTGTHZSR" + }, + "sensitive_attributes": [] + } + ] + }, { "mode": "data", "type": "aws_partition", @@ -92,6 +124,98 @@ } ] }, + { + "mode": "data", + "type": "aws_security_group", + "name": "lambda", + "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]", + "instances": [ + { + "schema_version": 0, + "attributes": { + "arn": "arn:aws-us-gov:ec2:us-gov-west-1:229685449397:security-group/sg-0641c697588b9aa6b", + "description": "Linux Common Base Security Group", + "filter": [ + { + "name": "group-name", + "values": [ + "it-linux-base" + ] + } + ], + "id": "sg-0641c697588b9aa6b", + "name": "it-linux-base", + "tags": { + "CostAllocation": "csvd:infrastructure", + "Environment": "dev", + "Name": "sg-it-linux-base", + "boc:created_by": "terraform", + "boc:tf_module_name": "aws-vpc-setup/security-groups", + "boc:tf_module_version": "1.1.0", + "boc:vpc:info": "vpc-00576a396ec570b94 vpc2-csvd-dev" + }, + "timeouts": null, + "vpc_id": "vpc-00576a396ec570b94" + }, + "sensitive_attributes": [] + } + ] + }, + { + "mode": "data", + "type": "aws_subnet", + "name": "lambda", + "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]", + "instances": [ + { + "schema_version": 0, + "attributes": { + "arn": "arn:aws-us-gov:ec2:us-gov-west-1:229685449397:subnet/subnet-0b1992a84536c581b", + "assign_ipv6_address_on_creation": false, + "availability_zone": "us-gov-west-1a", + "availability_zone_id": "usgw1-az1", + "available_ip_address_count": 49, + "cidr_block": "10.252.192.0/26", + "customer_owned_ipv4_pool": "", + "default_for_az": false, + "enable_dns64": false, + "enable_lni_at_device_index": 0, + "enable_resource_name_dns_a_record_on_launch": false, + "enable_resource_name_dns_aaaa_record_on_launch": false, + "filter": [ + { + "name": "tag:Name", + "values": [ + "vpc2-csvd-dev-endpoints-us-gov-west-1a" + ] + } + ], + "id": "subnet-0b1992a84536c581b", + "ipv6_cidr_block": "", + "ipv6_cidr_block_association_id": "", + "ipv6_native": false, + "map_customer_owned_ip_on_launch": false, + "map_public_ip_on_launch": false, + "outpost_arn": "", + "owner_id": "229685449397", + "private_dns_hostname_type_on_launch": "ip-name", + "state": "available", + "tags": { + "CostAllocation": "csvd:infrastructure", + "Environment": "dev", + "Name": "vpc2-csvd-dev-endpoints-us-gov-west-1a", + "boc:created_by": "terraform", + "boc:tf_module_name": "aws-vpc-setup/subnets", + "boc:tf_module_version": "2.11.1", + "boc:vpc:subnet_label": "endpoints" + }, + "timeouts": null, + "vpc_id": "vpc-00576a396ec570b94" + }, + "sensitive_attributes": [] + } + ] + }, { "mode": "managed", "type": "aws_codebuild_project", @@ -177,7 +301,7 @@ { "auth": [], "build_status_config": [], - "buildspec": "version: 0.2\n# buildspec.yml — terraform-eks-deployment / eks-terragrunt-repo-creator\n#\n# This buildspec is used by the CodeBuild project that is triggered by the\n# Lambda function (eks-terragrunt-repo-gen-template-automation) to create an\n# EKS cluster GitHub repository.\n#\n# Required environment variables (injected by the Lambda as overrides):\n# TF_VAR_name — cluster / repo name\n# TF_VAR_environment — environment (dev / nonprod / prod)\n# TF_VAR_region — AWS region (e.g. us-gov-west-1)\n# TF_VAR_cluster_config — JSON object with account_name, aws_account_id, etc.\n# TF_VAR_finops — JSON object with finops project_name / project_number\n# GITHUB_TOKEN — GitHub PAT (passed from Lambda's Secrets Manager read)\n# GITHUB_OWNER — GitHub org (default: SCT-Engineering)\n# GITHUB_BASE_URL — GHE base URL (e.g. https://github.e.it.census.gov)\n\nenv:\n variables:\n TF_VERSION: \"1.9.1\"\n ASSETS_BUCKET: \"csvd-packer-pipeline-assets\"\n REPO_HOST: \"github.e.it.census.gov\"\n REPO_ORG: \"SCT-Engineering\"\n REPO_NAME: \"terraform-eks-deployment\"\n REPO_BRANCH: \"fix/eca-copilot-instructions-and-callnotes\" # update to main once merged\n # Disable TLS verification for Census GHE (Census CA cert not trusted by default)\n GIT_SSL_NO_VERIFY: \"true\"\n TF_CLI_ARGS: \"-no-color\"\n # Census proxy — required for registry.terraform.io provider downloads\n HTTPS_PROXY: \"http://proxy.tco.census.gov:3128\"\n HTTP_PROXY: \"http://proxy.tco.census.gov:3128\"\n # Exclude AWS-internal endpoints and Census GHE from the proxy\n NO_PROXY: \"169.254.169.254,169.254.170.2,s3.us-gov-west-1.amazonaws.com,s3.amazonaws.com,.amazonaws.com,.us-gov-west-1.amazonaws.com,github.e.it.census.gov\"\n\nphases:\n install:\n commands:\n # ── Install Census Bureau CA certificate ──────────────────────────────\n # The Census GHE TLS cert is issued by the Census Bureau CA which is not\n # trusted by the CodeBuild Amazon Linux 2 trust store by default.\n - |\n aws s3 cp \"s3://${ASSETS_BUCKET}/certs/census-ca.pem\" \\\n /etc/pki/ca-trust/source/anchors/census-ca.pem 2\u003e/dev/null \\\n \u0026\u0026 update-ca-trust \\\n \u0026\u0026 echo \"Census CA cert installed\" \\\n || echo \"WARNING: could not install Census CA cert (continuing anyway)\"\n\n # ── Install Terraform ─────────────────────────────────────────────────\n - |\n if ! command -v terraform \u0026\u003e/dev/null; then\n TF_ZIP=\"terraform_${TF_VERSION}_linux_amd64.zip\"\n echo \"Installing Terraform ${TF_VERSION}...\"\n aws s3 cp \"s3://${ASSETS_BUCKET}/terraform/${TF_ZIP}\" /tmp/${TF_ZIP} 2\u003e/dev/null \\\n || curl -fsSL \"https://releases.hashicorp.com/terraform/${TF_VERSION}/${TF_ZIP}\" -o /tmp/${TF_ZIP}\n unzip -oq /tmp/${TF_ZIP} -d /usr/local/bin/\n chmod +x /usr/local/bin/terraform\n rm /tmp/${TF_ZIP}\n fi\n - terraform version\n\n # ── Clone terraform-eks-deployment ───────────────────────────────────\n - |\n git config --global credential.helper \\\n \"!f() { echo username=x-access-token; echo password=${GITHUB_TOKEN}; }; f\"\n git clone --depth 1 --branch \"${REPO_BRANCH}\" \\\n \"https://${REPO_HOST}/${REPO_ORG}/${REPO_NAME}.git\" \\\n /tmp/eks-deploy\n - echo \"Cloned ${REPO_ORG}/${REPO_NAME} @ $(git -C /tmp/eks-deploy rev-parse --short HEAD)\"\n\n build:\n commands:\n - cd /tmp/eks-deploy\n - echo \"=== terraform init ===\"\n - terraform init -no-color\n - echo \"=== terraform apply ===\"\n - terraform apply -auto-approve -no-color\n\n post_build:\n commands:\n - |\n if [ \"${CODEBUILD_BUILD_SUCCEEDING}\" = \"0\" ]; then\n echo \"Build FAILED — check logs above\"\n else\n echo \"Build SUCCEEDED — repository created\"\n fi\n", + "buildspec": "version: 0.2\n# buildspec-eks-repo-creator.yml\n#\n# Used by the CodeBuild project eks-terragrunt-repo-creator, which is triggered\n# by the Lambda (eks-terragrunt-repo-gen-template-automation) to create an EKS\n# cluster GitHub repository.\n#\n# Required environment variables (injected by the Lambda as overrides):\n# TF_VAR_name — cluster / repo name\n# TF_VAR_environment — environment (dev / nonprod / prod)\n# TF_VAR_region — AWS region (e.g. us-gov-west-1)\n# TF_VAR_cluster_config — JSON object with account_name, aws_account_id, etc.\n# TF_VAR_finops — JSON object with finops project_name / project_number\n# GITHUB_TOKEN — GitHub PAT (passed from Lambda's Secrets Manager read)\n# GITHUB_OWNER — GitHub org (default: SCT-Engineering)\n# GITHUB_BASE_URL — GHE base URL (e.g. https://github.e.it.census.gov)\n\nenv:\n variables:\n TF_VERSION: \"1.9.1\"\n ASSETS_BUCKET: \"csvd-packer-pipeline-assets\"\n REPO_HOST: \"github.e.it.census.gov\"\n REPO_ORG: \"SCT-Engineering\"\n REPO_NAME: \"terraform-eks-deployment\"\n REPO_BRANCH: \"test_cluster\" # PR #16 — switch back to main after merge\n # Disable TLS verification for Census GHE (Census CA cert not trusted by default)\n GIT_SSL_NO_VERIFY: \"true\"\n TF_VAR_run_in_codebuild: \"true\"\n TF_CLI_ARGS: \"-no-color\"\n # Census proxy — required for registry.terraform.io provider downloads\n HTTPS_PROXY: \"http://proxy.tco.census.gov:3128\"\n HTTP_PROXY: \"http://proxy.tco.census.gov:3128\"\n # Exclude AWS-internal endpoints and Census GHE from the proxy\n NO_PROXY: \"169.254.169.254,169.254.170.2,s3.us-gov-west-1.amazonaws.com,s3.amazonaws.com,.amazonaws.com,.us-gov-west-1.amazonaws.com,github.e.it.census.gov\"\n\nphases:\n install:\n commands:\n # ── Install Census Bureau CA certificate ──────────────────────────────\n # The Census GHE TLS cert is issued by the Census Bureau CA which is not\n # trusted by the CodeBuild Amazon Linux 2 trust store by default.\n - |\n aws s3 cp \"s3://${ASSETS_BUCKET}/certs/census-ca.pem\" \\\n /etc/pki/ca-trust/source/anchors/census-ca.pem 2\u003e/dev/null \\\n \u0026\u0026 update-ca-trust \\\n \u0026\u0026 echo \"Census CA cert installed\" \\\n || echo \"WARNING: could not install Census CA cert (continuing anyway)\"\n\n # ── Install Terraform ─────────────────────────────────────────────────\n - |\n if ! command -v terraform \u0026\u003e/dev/null; then\n TF_ZIP=\"terraform_${TF_VERSION}_linux_amd64.zip\"\n echo \"Installing Terraform ${TF_VERSION}...\"\n aws s3 cp \"s3://${ASSETS_BUCKET}/terraform/${TF_ZIP}\" /tmp/${TF_ZIP} 2\u003e/dev/null \\\n || curl -fsSL \"https://releases.hashicorp.com/terraform/${TF_VERSION}/${TF_ZIP}\" -o /tmp/${TF_ZIP}\n unzip -oq /tmp/${TF_ZIP} -d /usr/local/bin/\n chmod +x /usr/local/bin/terraform\n rm /tmp/${TF_ZIP}\n fi\n - terraform version\n\n # ── Install Python dependencies for post-apply scripts ───────────────\n - pip3 install --quiet httpx rich\n\n # ── Clone terraform-eks-deployment ───────────────────────────────────\n - |\n git config --global credential.helper \\\n \"!f() { echo username=x-access-token; echo password=${GITHUB_TOKEN}; }; f\"\n git clone --depth 1 --branch \"${REPO_BRANCH}\" \\\n \"https://${REPO_HOST}/${REPO_ORG}/${REPO_NAME}.git\" \\\n /tmp/eks-deploy\n - echo \"Cloned ${REPO_ORG}/${REPO_NAME} @ $(git -C /tmp/eks-deploy rev-parse --short HEAD)\"\n\n build:\n commands:\n - cd /tmp/eks-deploy\n - echo \"=== terraform init ===\"\n - terraform init -no-color\n - echo \"=== terraform apply ===\"\n - terraform apply -auto-approve -no-color\n\n post_build:\n commands:\n - |\n if [ \"${CODEBUILD_BUILD_SUCCEEDING}\" = \"0\" ]; then\n echo \"Build FAILED — check logs above\"\n else\n echo \"Build SUCCEEDED — repository created\"\n fi\n", "git_clone_depth": 0, "git_submodules_config": [], "insecure_ssl": false, @@ -210,7 +334,13 @@ ] }, "sensitive_attributes": [], - "private": "bnVsbA==" + "private": "bnVsbA==", + "dependencies": [ + "aws_iam_role.codebuild", + "data.aws_iam_role.codebuild", + "data.aws_security_group.lambda", + "data.aws_subnet.lambda" + ] } ] }, @@ -278,6 +408,10 @@ "private": "bnVsbA==", "dependencies": [ "aws_codebuild_project.eks_repo_creator", + "aws_iam_role.codebuild", + "data.aws_iam_role.codebuild", + "data.aws_security_group.lambda", + "data.aws_subnet.lambda", "module.eks_terragrunt_repo_generator.aws_iam_role.lambda", "module.eks_terragrunt_repo_generator.data.aws_partition.current" ] @@ -307,6 +441,8 @@ "data.aws_caller_identity.current", "data.aws_partition.current", "data.aws_region.current", + "data.aws_security_group.lambda", + "data.aws_subnet.lambda", "module.eks_terragrunt_repo_generator.aws_cloudwatch_log_group.lambda", "module.eks_terragrunt_repo_generator.aws_iam_role.lambda", "module.eks_terragrunt_repo_generator.aws_iam_role_policy_attachment.lambda_logs", @@ -335,6 +471,7 @@ "private": "bnVsbA==", "dependencies": [ "data.aws_caller_identity.current", + "data.aws_partition.current", "module.eks_terragrunt_repo_generator.aws_iam_role.lambda", "module.eks_terragrunt_repo_generator.data.aws_partition.current" ] @@ -368,7 +505,7 @@ "content_encoding": "", "content_language": "", "content_type": "application/octet-stream", - "etag": "8b5997769f7c04a86468f9b1d41e03c3", + "etag": "938ed808a2e81afe41c6662a83c54e97", "force_destroy": false, "id": "eks-terragrunt-repo-creator/v2.0/product-template.yaml", "key": "eks-terragrunt-repo-creator/v2.0/product-template.yaml", @@ -573,89 +710,6 @@ } ] }, - { - "mode": "managed", - "type": "aws_vpc_endpoint", - "name": "codebuild", - "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]", - "instances": [ - { - "index_key": 0, - "schema_version": 0, - "attributes": { - "arn": "arn:aws-us-gov:ec2:us-gov-west-1:229685449397:vpc-endpoint/vpce-0e2a877f8ef17a404", - "auto_accept": null, - "cidr_blocks": [], - "dns_entry": [ - { - "dns_name": "vpce-0e2a877f8ef17a404-bx1tfjab.codebuild.us-gov-west-1.vpce.amazonaws.com", - "hosted_zone_id": "Z12529ZODG2B6H" - }, - { - "dns_name": "vpce-0e2a877f8ef17a404-bx1tfjab-us-gov-west-1a.codebuild.us-gov-west-1.vpce.amazonaws.com", - "hosted_zone_id": "Z12529ZODG2B6H" - }, - { - "dns_name": "codebuild.us-gov-west-1.amazonaws.com", - "hosted_zone_id": "Z0032114FCSKWRU3K1CW" - } - ], - "dns_options": [ - { - "dns_record_ip_type": "ipv4", - "private_dns_only_for_inbound_resolver_endpoint": false - } - ], - "id": "vpce-0e2a877f8ef17a404", - "ip_address_type": "ipv4", - "network_interface_ids": [ - "eni-08b50691ecd508c05" - ], - "owner_id": "229685449397", - "policy": "{\"Statement\":[{\"Action\":\"*\",\"Effect\":\"Allow\",\"Principal\":\"*\",\"Resource\":\"*\"}]}", - "prefix_list_id": null, - "private_dns_enabled": true, - "requester_managed": false, - "resource_configuration_arn": "", - "route_table_ids": [], - "security_group_ids": [ - "sg-0641c697588b9aa6b" - ], - "service_name": "com.amazonaws.us-gov-west-1.codebuild", - "service_network_arn": "", - "service_region": "", - "state": "available", - "subnet_configuration": [ - { - "ipv4": "10.252.192.37", - "ipv6": "", - "subnet_id": "subnet-0b1992a84536c581b" - } - ], - "subnet_ids": [ - "subnet-0b1992a84536c581b" - ], - "tags": { - "Environment": "production", - "ManagedBy": "Terraform", - "Name": "eks-terragrunt-codebuild-endpoint", - "Purpose": "EKSTerragruntRepoGenerator" - }, - "tags_all": { - "Environment": "production", - "ManagedBy": "Terraform", - "Name": "eks-terragrunt-codebuild-endpoint", - "Purpose": "EKSTerragruntRepoGenerator" - }, - "timeouts": null, - "vpc_endpoint_type": "Interface", - "vpc_id": "vpc-00576a396ec570b94" - }, - "sensitive_attributes": [], - "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjo2MDAwMDAwMDAwMDAsImRlbGV0ZSI6NjAwMDAwMDAwMDAwLCJ1cGRhdGUiOjYwMDAwMDAwMDAwMH19" - } - ] - }, { "module": "module.eks_terragrunt_repo_generator", "mode": "data", @@ -836,6 +890,8 @@ "dependencies": [ "data.aws_caller_identity.current", "data.aws_region.current", + "data.aws_security_group.lambda", + "data.aws_subnet.lambda", "module.eks_terragrunt_repo_generator.aws_apigatewayv2_api.this", "module.eks_terragrunt_repo_generator.aws_cloudwatch_log_group.lambda", "module.eks_terragrunt_repo_generator.aws_iam_role.lambda", @@ -875,6 +931,8 @@ "dependencies": [ "data.aws_caller_identity.current", "data.aws_region.current", + "data.aws_security_group.lambda", + "data.aws_subnet.lambda", "module.eks_terragrunt_repo_generator.aws_apigatewayv2_api.this", "module.eks_terragrunt_repo_generator.aws_apigatewayv2_integration.this", "module.eks_terragrunt_repo_generator.aws_cloudwatch_log_group.lambda", @@ -1184,7 +1242,7 @@ "x86_64" ], "arn": "arn:aws-us-gov:lambda:us-gov-west-1:229685449397:function:eks-terragrunt-repo-gen-template-automation", - "code_sha256": "d49d016b3984ba4e9d860e4ec1f9511a4e7eaa1cbe240fd9dbfb36dea069fbdf", + "code_sha256": "3c63038c4f6605ccdcc191a285c01ab7a7c286da0939abe3808fa79b4d7683c6", "code_signing_config_arn": null, "dead_letter_config": [], "description": "", @@ -1217,7 +1275,7 @@ "image_uri": "229685449397.dkr.ecr.us-gov-west-1.amazonaws.com/eks-terragrunt-repo-generator/lambda:latest", "invoke_arn": "arn:aws-us-gov:apigateway:us-gov-west-1:lambda:path/2015-03-31/functions/arn:aws-us-gov:lambda:us-gov-west-1:229685449397:function:eks-terragrunt-repo-gen-template-automation/invocations", "kms_key_arn": "", - "last_modified": "2026-04-07T17:09:55.000+0000", + "last_modified": "2026-04-20T18:41:52.000+0000", "layers": [], "logging_config": [ { @@ -1307,6 +1365,8 @@ "dependencies": [ "data.aws_caller_identity.current", "data.aws_region.current", + "data.aws_security_group.lambda", + "data.aws_subnet.lambda", "module.eks_terragrunt_repo_generator.aws_cloudwatch_log_group.lambda", "module.eks_terragrunt_repo_generator.aws_iam_role.lambda", "module.eks_terragrunt_repo_generator.aws_iam_role_policy_attachment.lambda_logs", @@ -1343,6 +1403,8 @@ "dependencies": [ "data.aws_caller_identity.current", "data.aws_region.current", + "data.aws_security_group.lambda", + "data.aws_subnet.lambda", "module.eks_terragrunt_repo_generator.aws_apigatewayv2_api.this", "module.eks_terragrunt_repo_generator.aws_cloudwatch_log_group.lambda", "module.eks_terragrunt_repo_generator.aws_iam_role.lambda", @@ -1381,6 +1443,8 @@ "dependencies": [ "data.aws_caller_identity.current", "data.aws_region.current", + "data.aws_security_group.lambda", + "data.aws_subnet.lambda", "module.eks_terragrunt_repo_generator.aws_cloudwatch_log_group.lambda", "module.eks_terragrunt_repo_generator.aws_iam_role.lambda", "module.eks_terragrunt_repo_generator.aws_iam_role_policy_attachment.lambda_logs", @@ -1668,13 +1732,13 @@ [ { "type": "get_attr", - "value": "value" + "value": "value_wo" } ], [ { "type": "get_attr", - "value": "value_wo" + "value": "value" } ] ], diff --git a/deploy/terraform.tfstate.backup b/deploy/terraform.tfstate.backup index 8569f2e..3a7c1b5 100644 --- a/deploy/terraform.tfstate.backup +++ b/deploy/terraform.tfstate.backup @@ -1,7 +1,7 @@ { "version": 4, "terraform_version": "1.9.1", - "serial": 209, + "serial": 215, "lineage": "637f189b-ce2c-766c-35d1-8b43eb7ae216", "outputs": { "api_endpoint": { @@ -56,6 +56,38 @@ } ] }, + { + "mode": "data", + "type": "aws_iam_role", + "name": "codebuild", + "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]", + "instances": [ + { + "index_key": 0, + "schema_version": 0, + "attributes": { + "arn": "arn:aws-us-gov:iam::229685449397:role/CodeBuildPackerRole-eks-terragrunt-repo-generator-builder", + "assume_role_policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"codebuild.amazonaws.com\"},\"Action\":\"sts:AssumeRole\"}]}", + "create_date": "2026-02-20T19:54:55Z", + "description": "IAM role for CodeBuild Packer project: eks-terragrunt-repo-generator-builder", + "id": "CodeBuildPackerRole-eks-terragrunt-repo-generator-builder", + "max_session_duration": 3600, + "name": "CodeBuildPackerRole-eks-terragrunt-repo-generator-builder", + "path": "/", + "permissions_boundary": "", + "role_last_used": [ + { + "last_used_date": "2026-04-20T18:51:33Z", + "region": "us-gov-west-1" + } + ], + "tags": {}, + "unique_id": "AROATK6SR2K2WWTGTHZSR" + }, + "sensitive_attributes": [] + } + ] + }, { "mode": "data", "type": "aws_partition", @@ -92,6 +124,98 @@ } ] }, + { + "mode": "data", + "type": "aws_security_group", + "name": "lambda", + "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]", + "instances": [ + { + "schema_version": 0, + "attributes": { + "arn": "arn:aws-us-gov:ec2:us-gov-west-1:229685449397:security-group/sg-0641c697588b9aa6b", + "description": "Linux Common Base Security Group", + "filter": [ + { + "name": "group-name", + "values": [ + "it-linux-base" + ] + } + ], + "id": "sg-0641c697588b9aa6b", + "name": "it-linux-base", + "tags": { + "CostAllocation": "csvd:infrastructure", + "Environment": "dev", + "Name": "sg-it-linux-base", + "boc:created_by": "terraform", + "boc:tf_module_name": "aws-vpc-setup/security-groups", + "boc:tf_module_version": "1.1.0", + "boc:vpc:info": "vpc-00576a396ec570b94 vpc2-csvd-dev" + }, + "timeouts": null, + "vpc_id": "vpc-00576a396ec570b94" + }, + "sensitive_attributes": [] + } + ] + }, + { + "mode": "data", + "type": "aws_subnet", + "name": "lambda", + "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]", + "instances": [ + { + "schema_version": 0, + "attributes": { + "arn": "arn:aws-us-gov:ec2:us-gov-west-1:229685449397:subnet/subnet-0b1992a84536c581b", + "assign_ipv6_address_on_creation": false, + "availability_zone": "us-gov-west-1a", + "availability_zone_id": "usgw1-az1", + "available_ip_address_count": 49, + "cidr_block": "10.252.192.0/26", + "customer_owned_ipv4_pool": "", + "default_for_az": false, + "enable_dns64": false, + "enable_lni_at_device_index": 0, + "enable_resource_name_dns_a_record_on_launch": false, + "enable_resource_name_dns_aaaa_record_on_launch": false, + "filter": [ + { + "name": "tag:Name", + "values": [ + "vpc2-csvd-dev-endpoints-us-gov-west-1a" + ] + } + ], + "id": "subnet-0b1992a84536c581b", + "ipv6_cidr_block": "", + "ipv6_cidr_block_association_id": "", + "ipv6_native": false, + "map_customer_owned_ip_on_launch": false, + "map_public_ip_on_launch": false, + "outpost_arn": "", + "owner_id": "229685449397", + "private_dns_hostname_type_on_launch": "ip-name", + "state": "available", + "tags": { + "CostAllocation": "csvd:infrastructure", + "Environment": "dev", + "Name": "vpc2-csvd-dev-endpoints-us-gov-west-1a", + "boc:created_by": "terraform", + "boc:tf_module_name": "aws-vpc-setup/subnets", + "boc:tf_module_version": "2.11.1", + "boc:vpc:subnet_label": "endpoints" + }, + "timeouts": null, + "vpc_id": "vpc-00576a396ec570b94" + }, + "sensitive_attributes": [] + } + ] + }, { "mode": "managed", "type": "aws_codebuild_project", @@ -177,7 +301,7 @@ { "auth": [], "build_status_config": [], - "buildspec": "version: 0.2\n# buildspec.yml — terraform-eks-deployment / eks-terragrunt-repo-creator\n#\n# This buildspec is used by the CodeBuild project that is triggered by the\n# Lambda function (eks-terragrunt-repo-gen-template-automation) to create an\n# EKS cluster GitHub repository.\n#\n# Required environment variables (injected by the Lambda as overrides):\n# TF_VAR_name — cluster / repo name\n# TF_VAR_environment — environment (dev / nonprod / prod)\n# TF_VAR_region — AWS region (e.g. us-gov-west-1)\n# TF_VAR_cluster_config — JSON object with account_name, aws_account_id, etc.\n# TF_VAR_finops — JSON object with finops project_name / project_number\n# GITHUB_TOKEN — GitHub PAT (passed from Lambda's Secrets Manager read)\n# GITHUB_OWNER — GitHub org (default: SCT-Engineering)\n# GITHUB_BASE_URL — GHE base URL (e.g. https://github.e.it.census.gov)\n\nenv:\n variables:\n TF_VERSION: \"1.9.1\"\n ASSETS_BUCKET: \"csvd-packer-pipeline-assets\"\n REPO_HOST: \"github.e.it.census.gov\"\n REPO_ORG: \"SCT-Engineering\"\n REPO_NAME: \"terraform-eks-deployment\"\n REPO_BRANCH: \"fix/eca-copilot-instructions-and-callnotes\" # update to main once merged\n # Disable TLS verification for Census GHE (Census CA cert not trusted by default)\n GIT_SSL_NO_VERIFY: \"true\"\n TF_CLI_ARGS: \"-no-color\"\n # Census proxy — required for registry.terraform.io provider downloads\n HTTPS_PROXY: \"http://proxy.tco.census.gov:3128\"\n HTTP_PROXY: \"http://proxy.tco.census.gov:3128\"\n # Exclude AWS-internal endpoints and Census GHE from the proxy\n NO_PROXY: \"169.254.169.254,169.254.170.2,s3.us-gov-west-1.amazonaws.com,s3.amazonaws.com,.amazonaws.com,.us-gov-west-1.amazonaws.com,github.e.it.census.gov\"\n\nphases:\n install:\n commands:\n # ── Install Census Bureau CA certificate ──────────────────────────────\n # The Census GHE TLS cert is issued by the Census Bureau CA which is not\n # trusted by the CodeBuild Amazon Linux 2 trust store by default.\n - |\n aws s3 cp \"s3://${ASSETS_BUCKET}/certs/census-ca.pem\" \\\n /etc/pki/ca-trust/source/anchors/census-ca.pem 2\u003e/dev/null \\\n \u0026\u0026 update-ca-trust \\\n \u0026\u0026 echo \"Census CA cert installed\" \\\n || echo \"WARNING: could not install Census CA cert (continuing anyway)\"\n\n # ── Install Terraform ─────────────────────────────────────────────────\n - |\n if ! command -v terraform \u0026\u003e/dev/null; then\n TF_ZIP=\"terraform_${TF_VERSION}_linux_amd64.zip\"\n echo \"Installing Terraform ${TF_VERSION}...\"\n aws s3 cp \"s3://${ASSETS_BUCKET}/terraform/${TF_ZIP}\" /tmp/${TF_ZIP} 2\u003e/dev/null \\\n || curl -fsSL \"https://releases.hashicorp.com/terraform/${TF_VERSION}/${TF_ZIP}\" -o /tmp/${TF_ZIP}\n unzip -oq /tmp/${TF_ZIP} -d /usr/local/bin/\n chmod +x /usr/local/bin/terraform\n rm /tmp/${TF_ZIP}\n fi\n - terraform version\n\n # ── Clone terraform-eks-deployment ───────────────────────────────────\n - |\n git config --global credential.helper \\\n \"!f() { echo username=x-access-token; echo password=${GITHUB_TOKEN}; }; f\"\n git clone --depth 1 --branch \"${REPO_BRANCH}\" \\\n \"https://${REPO_HOST}/${REPO_ORG}/${REPO_NAME}.git\" \\\n /tmp/eks-deploy\n - echo \"Cloned ${REPO_ORG}/${REPO_NAME} @ $(git -C /tmp/eks-deploy rev-parse --short HEAD)\"\n\n build:\n commands:\n - cd /tmp/eks-deploy\n - echo \"=== terraform init ===\"\n - terraform init -no-color\n - echo \"=== terraform apply ===\"\n - terraform apply -auto-approve -no-color\n\n post_build:\n commands:\n - |\n if [ \"${CODEBUILD_BUILD_SUCCEEDING}\" = \"0\" ]; then\n echo \"Build FAILED — check logs above\"\n else\n echo \"Build SUCCEEDED — repository created\"\n fi\n", + "buildspec": "version: 0.2\n# buildspec-eks-repo-creator.yml\n#\n# Used by the CodeBuild project eks-terragrunt-repo-creator, which is triggered\n# by the Lambda (eks-terragrunt-repo-gen-template-automation) to create an EKS\n# cluster GitHub repository.\n#\n# Required environment variables (injected by the Lambda as overrides):\n# TF_VAR_name — cluster / repo name\n# TF_VAR_environment — environment (dev / nonprod / prod)\n# TF_VAR_region — AWS region (e.g. us-gov-west-1)\n# TF_VAR_cluster_config — JSON object with account_name, aws_account_id, etc.\n# TF_VAR_finops — JSON object with finops project_name / project_number\n# GITHUB_TOKEN — GitHub PAT (passed from Lambda's Secrets Manager read)\n# GITHUB_OWNER — GitHub org (default: SCT-Engineering)\n# GITHUB_BASE_URL — GHE base URL (e.g. https://github.e.it.census.gov)\n\nenv:\n variables:\n TF_VERSION: \"1.9.1\"\n ASSETS_BUCKET: \"csvd-packer-pipeline-assets\"\n REPO_HOST: \"github.e.it.census.gov\"\n REPO_ORG: \"SCT-Engineering\"\n REPO_NAME: \"terraform-eks-deployment\"\n REPO_BRANCH: \"main\"\n # Disable TLS verification for Census GHE (Census CA cert not trusted by default)\n GIT_SSL_NO_VERIFY: \"true\"\n TF_VAR_run_in_codebuild: \"true\"\n TF_CLI_ARGS: \"-no-color\"\n # Census proxy — required for registry.terraform.io provider downloads\n HTTPS_PROXY: \"http://proxy.tco.census.gov:3128\"\n HTTP_PROXY: \"http://proxy.tco.census.gov:3128\"\n # Exclude AWS-internal endpoints and Census GHE from the proxy\n NO_PROXY: \"169.254.169.254,169.254.170.2,s3.us-gov-west-1.amazonaws.com,s3.amazonaws.com,.amazonaws.com,.us-gov-west-1.amazonaws.com,github.e.it.census.gov\"\n\nphases:\n install:\n commands:\n # ── Install Census Bureau CA certificate ──────────────────────────────\n # The Census GHE TLS cert is issued by the Census Bureau CA which is not\n # trusted by the CodeBuild Amazon Linux 2 trust store by default.\n - |\n aws s3 cp \"s3://${ASSETS_BUCKET}/certs/census-ca.pem\" \\\n /etc/pki/ca-trust/source/anchors/census-ca.pem 2\u003e/dev/null \\\n \u0026\u0026 update-ca-trust \\\n \u0026\u0026 echo \"Census CA cert installed\" \\\n || echo \"WARNING: could not install Census CA cert (continuing anyway)\"\n\n # ── Install Terraform ─────────────────────────────────────────────────\n - |\n if ! command -v terraform \u0026\u003e/dev/null; then\n TF_ZIP=\"terraform_${TF_VERSION}_linux_amd64.zip\"\n echo \"Installing Terraform ${TF_VERSION}...\"\n aws s3 cp \"s3://${ASSETS_BUCKET}/terraform/${TF_ZIP}\" /tmp/${TF_ZIP} 2\u003e/dev/null \\\n || curl -fsSL \"https://releases.hashicorp.com/terraform/${TF_VERSION}/${TF_ZIP}\" -o /tmp/${TF_ZIP}\n unzip -oq /tmp/${TF_ZIP} -d /usr/local/bin/\n chmod +x /usr/local/bin/terraform\n rm /tmp/${TF_ZIP}\n fi\n - terraform version\n\n # ── Install Python dependencies for post-apply scripts ───────────────\n - pip3 install --quiet httpx rich\n\n # ── Clone terraform-eks-deployment ───────────────────────────────────\n - |\n git config --global credential.helper \\\n \"!f() { echo username=x-access-token; echo password=${GITHUB_TOKEN}; }; f\"\n git clone --depth 1 --branch \"${REPO_BRANCH}\" \\\n \"https://${REPO_HOST}/${REPO_ORG}/${REPO_NAME}.git\" \\\n /tmp/eks-deploy\n - echo \"Cloned ${REPO_ORG}/${REPO_NAME} @ $(git -C /tmp/eks-deploy rev-parse --short HEAD)\"\n\n build:\n commands:\n - cd /tmp/eks-deploy\n - echo \"=== terraform init ===\"\n - terraform init -no-color\n - echo \"=== terraform apply ===\"\n - terraform apply -auto-approve -no-color\n\n post_build:\n commands:\n - |\n if [ \"${CODEBUILD_BUILD_SUCCEEDING}\" = \"0\" ]; then\n echo \"Build FAILED — check logs above\"\n else\n echo \"Build SUCCEEDED — repository created\"\n fi\n", "git_clone_depth": 0, "git_submodules_config": [], "insecure_ssl": false, @@ -210,7 +334,13 @@ ] }, "sensitive_attributes": [], - "private": "bnVsbA==" + "private": "bnVsbA==", + "dependencies": [ + "aws_iam_role.codebuild", + "data.aws_iam_role.codebuild", + "data.aws_security_group.lambda", + "data.aws_subnet.lambda" + ] } ] }, @@ -278,6 +408,10 @@ "private": "bnVsbA==", "dependencies": [ "aws_codebuild_project.eks_repo_creator", + "aws_iam_role.codebuild", + "data.aws_iam_role.codebuild", + "data.aws_security_group.lambda", + "data.aws_subnet.lambda", "module.eks_terragrunt_repo_generator.aws_iam_role.lambda", "module.eks_terragrunt_repo_generator.data.aws_partition.current" ] @@ -307,6 +441,8 @@ "data.aws_caller_identity.current", "data.aws_partition.current", "data.aws_region.current", + "data.aws_security_group.lambda", + "data.aws_subnet.lambda", "module.eks_terragrunt_repo_generator.aws_cloudwatch_log_group.lambda", "module.eks_terragrunt_repo_generator.aws_iam_role.lambda", "module.eks_terragrunt_repo_generator.aws_iam_role_policy_attachment.lambda_logs", @@ -335,6 +471,7 @@ "private": "bnVsbA==", "dependencies": [ "data.aws_caller_identity.current", + "data.aws_partition.current", "module.eks_terragrunt_repo_generator.aws_iam_role.lambda", "module.eks_terragrunt_repo_generator.data.aws_partition.current" ] @@ -368,7 +505,7 @@ "content_encoding": "", "content_language": "", "content_type": "application/octet-stream", - "etag": "8b5997769f7c04a86468f9b1d41e03c3", + "etag": "938ed808a2e81afe41c6662a83c54e97", "force_destroy": false, "id": "eks-terragrunt-repo-creator/v2.0/product-template.yaml", "key": "eks-terragrunt-repo-creator/v2.0/product-template.yaml", @@ -573,89 +710,6 @@ } ] }, - { - "mode": "managed", - "type": "aws_vpc_endpoint", - "name": "codebuild", - "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]", - "instances": [ - { - "index_key": 0, - "schema_version": 0, - "attributes": { - "arn": "arn:aws-us-gov:ec2:us-gov-west-1:229685449397:vpc-endpoint/vpce-0e2a877f8ef17a404", - "auto_accept": null, - "cidr_blocks": [], - "dns_entry": [ - { - "dns_name": "vpce-0e2a877f8ef17a404-bx1tfjab.codebuild.us-gov-west-1.vpce.amazonaws.com", - "hosted_zone_id": "Z12529ZODG2B6H" - }, - { - "dns_name": "vpce-0e2a877f8ef17a404-bx1tfjab-us-gov-west-1a.codebuild.us-gov-west-1.vpce.amazonaws.com", - "hosted_zone_id": "Z12529ZODG2B6H" - }, - { - "dns_name": "codebuild.us-gov-west-1.amazonaws.com", - "hosted_zone_id": "Z0032114FCSKWRU3K1CW" - } - ], - "dns_options": [ - { - "dns_record_ip_type": "ipv4", - "private_dns_only_for_inbound_resolver_endpoint": false - } - ], - "id": "vpce-0e2a877f8ef17a404", - "ip_address_type": "ipv4", - "network_interface_ids": [ - "eni-08b50691ecd508c05" - ], - "owner_id": "229685449397", - "policy": "{\"Statement\":[{\"Action\":\"*\",\"Effect\":\"Allow\",\"Principal\":\"*\",\"Resource\":\"*\"}]}", - "prefix_list_id": null, - "private_dns_enabled": true, - "requester_managed": false, - "resource_configuration_arn": "", - "route_table_ids": [], - "security_group_ids": [ - "sg-0641c697588b9aa6b" - ], - "service_name": "com.amazonaws.us-gov-west-1.codebuild", - "service_network_arn": "", - "service_region": "", - "state": "available", - "subnet_configuration": [ - { - "ipv4": "10.252.192.37", - "ipv6": "", - "subnet_id": "subnet-0b1992a84536c581b" - } - ], - "subnet_ids": [ - "subnet-0b1992a84536c581b" - ], - "tags": { - "Environment": "production", - "ManagedBy": "Terraform", - "Name": "eks-terragrunt-codebuild-endpoint", - "Purpose": "EKSTerragruntRepoGenerator" - }, - "tags_all": { - "Environment": "production", - "ManagedBy": "Terraform", - "Name": "eks-terragrunt-codebuild-endpoint", - "Purpose": "EKSTerragruntRepoGenerator" - }, - "timeouts": null, - "vpc_endpoint_type": "Interface", - "vpc_id": "vpc-00576a396ec570b94" - }, - "sensitive_attributes": [], - "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjo2MDAwMDAwMDAwMDAsImRlbGV0ZSI6NjAwMDAwMDAwMDAwLCJ1cGRhdGUiOjYwMDAwMDAwMDAwMH19" - } - ] - }, { "module": "module.eks_terragrunt_repo_generator", "mode": "data", @@ -836,6 +890,8 @@ "dependencies": [ "data.aws_caller_identity.current", "data.aws_region.current", + "data.aws_security_group.lambda", + "data.aws_subnet.lambda", "module.eks_terragrunt_repo_generator.aws_apigatewayv2_api.this", "module.eks_terragrunt_repo_generator.aws_cloudwatch_log_group.lambda", "module.eks_terragrunt_repo_generator.aws_iam_role.lambda", @@ -875,6 +931,8 @@ "dependencies": [ "data.aws_caller_identity.current", "data.aws_region.current", + "data.aws_security_group.lambda", + "data.aws_subnet.lambda", "module.eks_terragrunt_repo_generator.aws_apigatewayv2_api.this", "module.eks_terragrunt_repo_generator.aws_apigatewayv2_integration.this", "module.eks_terragrunt_repo_generator.aws_cloudwatch_log_group.lambda", @@ -992,6 +1050,10 @@ "name": "eks-repo-creator-codebuild-access", "policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Action\":[\"codebuild:StartBuild\",\"codebuild:BatchGetBuilds\"],\"Effect\":\"Allow\",\"Resource\":\"arn:aws-us-gov:codebuild:us-gov-west-1:229685449397:project/eks-terragrunt-repo-creator\",\"Sid\":\"StartAndPollBuild\"}]}" }, + { + "name": "eks-repo-creator-tf-github-token-access", + "policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Action\":[\"secretsmanager:GetSecretValue\"],\"Effect\":\"Allow\",\"Resource\":\"arn:aws-us-gov:secretsmanager:us-gov-west-1:229685449397:secret:ghe-runner/github-token-*\",\"Sid\":\"ReadTFGitHubToken\"}]}" + }, { "name": "eks-terragrunt-repo-gen-kms-access-policy", "policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Action\":[\"kms:Decrypt\",\"kms:DescribeKey\"],\"Effect\":\"Allow\",\"Resource\":[\"*\"]}]}" @@ -1180,7 +1242,7 @@ "x86_64" ], "arn": "arn:aws-us-gov:lambda:us-gov-west-1:229685449397:function:eks-terragrunt-repo-gen-template-automation", - "code_sha256": "d49d016b3984ba4e9d860e4ec1f9511a4e7eaa1cbe240fd9dbfb36dea069fbdf", + "code_sha256": "3c63038c4f6605ccdcc191a285c01ab7a7c286da0939abe3808fa79b4d7683c6", "code_signing_config_arn": null, "dead_letter_config": [], "description": "", @@ -1213,7 +1275,7 @@ "image_uri": "229685449397.dkr.ecr.us-gov-west-1.amazonaws.com/eks-terragrunt-repo-generator/lambda:latest", "invoke_arn": "arn:aws-us-gov:apigateway:us-gov-west-1:lambda:path/2015-03-31/functions/arn:aws-us-gov:lambda:us-gov-west-1:229685449397:function:eks-terragrunt-repo-gen-template-automation/invocations", "kms_key_arn": "", - "last_modified": "2026-04-07T17:09:55.000+0000", + "last_modified": "2026-04-20T18:41:52.000+0000", "layers": [], "logging_config": [ { @@ -1303,6 +1365,8 @@ "dependencies": [ "data.aws_caller_identity.current", "data.aws_region.current", + "data.aws_security_group.lambda", + "data.aws_subnet.lambda", "module.eks_terragrunt_repo_generator.aws_cloudwatch_log_group.lambda", "module.eks_terragrunt_repo_generator.aws_iam_role.lambda", "module.eks_terragrunt_repo_generator.aws_iam_role_policy_attachment.lambda_logs", @@ -1339,6 +1403,8 @@ "dependencies": [ "data.aws_caller_identity.current", "data.aws_region.current", + "data.aws_security_group.lambda", + "data.aws_subnet.lambda", "module.eks_terragrunt_repo_generator.aws_apigatewayv2_api.this", "module.eks_terragrunt_repo_generator.aws_cloudwatch_log_group.lambda", "module.eks_terragrunt_repo_generator.aws_iam_role.lambda", @@ -1377,6 +1443,8 @@ "dependencies": [ "data.aws_caller_identity.current", "data.aws_region.current", + "data.aws_security_group.lambda", + "data.aws_subnet.lambda", "module.eks_terragrunt_repo_generator.aws_cloudwatch_log_group.lambda", "module.eks_terragrunt_repo_generator.aws_iam_role.lambda", "module.eks_terragrunt_repo_generator.aws_iam_role_policy_attachment.lambda_logs", @@ -1429,13 +1497,13 @@ [ { "type": "get_attr", - "value": "value_wo" + "value": "value" } ], [ { "type": "get_attr", - "value": "value" + "value": "value_wo" } ] ], @@ -1476,13 +1544,13 @@ [ { "type": "get_attr", - "value": "value" + "value": "value_wo" } ], [ { "type": "get_attr", - "value": "value_wo" + "value": "value" } ] ], @@ -1523,13 +1591,13 @@ [ { "type": "get_attr", - "value": "value" + "value": "value_wo" } ], [ { "type": "get_attr", - "value": "value_wo" + "value": "value" } ] ], @@ -1570,13 +1638,13 @@ [ { "type": "get_attr", - "value": "value_wo" + "value": "value" } ], [ { "type": "get_attr", - "value": "value" + "value": "value_wo" } ] ], @@ -1711,13 +1779,13 @@ [ { "type": "get_attr", - "value": "value_wo" + "value": "value" } ], [ { "type": "get_attr", - "value": "value" + "value": "value_wo" } ] ], diff --git a/docs/DEMO_SCRIPT.md b/docs/DEMO_SCRIPT.md index 62ca86e..46be309 100644 --- a/docs/DEMO_SCRIPT.md +++ b/docs/DEMO_SCRIPT.md @@ -18,17 +18,24 @@ GitHub repository in under 5 minutes, by filling out a form in the AWS console. | Parameter | Demo Value | |-----------|------------| -| Repository Name (`project_name`) | `demo-eks-cluster-XX` | +| Repository Name | `demo-eks-cluster-XX` | | EKS Cluster Name | *(leave blank — defaults to repo name)* | | Environment | `dev` | -| AWS Region | `us-gov-west-1` | | Account Name | `csvd-dev-ew` | -| AWS Account ID | `229685449397` | -| Environment Abbr | `dev` | +| Environment Abbreviation | `dev` | | VPC Name | `csvd-dev-ew-vpc-01` | | VPC Domain Name | `dev.inf.csp1.census.gov` | | Owning Team | `tf-module-admins` | +| Cluster Mailing List | *(optional — leave blank)* | +| Organization Path | `census:ocio:csvd` *(default)* | +| FinOps Project Name | *(optional — leave blank)* | +| FinOps Project Number | *(optional — leave blank)* | | Creator Username | *(your GHE username for admin access)* | +| Additional Tags (JSON) | `{}` *(default)* | + +> **Note:** AWS Account ID and AWS Region are not on the form — they are +> automatically resolved by CloudFormation (`!Sub "${AWS::AccountId}"` / +> `!Sub "${AWS::Region}"`) in the provisioner's account before the Lambda runs. --- @@ -124,26 +131,32 @@ Service Catalog console **Cluster Configuration:** > "Repository name becomes the cluster name by default. We use lowercase hyphens only -> because this name flows into Kubernetes resource names. Environment is dev — this +> because this name flows into Kubernetes resource names. Environment is `dev` — this > controls which Terragrunt environment directory gets created." **Account Configuration:** -> "These three fields — account name, account ID, and environment abbreviation — get -> stamped into multiple HCL files. In the old world, you'd copy-paste these by hand -> into each file. Here, you type them once." +> "Account Name is the human-readable AWS account identifier — like `csvd-dev-ew`. Environment +> Abbreviation is the short label that appears in file paths and resource names. Notice +> that AWS Account ID and Region are *not* on this form — CloudFormation resolves those +> automatically from the account you're logged into." **VPC Configuration:** -> "VPC Name and VPC Domain Name. Note: this is the VPC name string, not a VPC ID. -> The Terragrunt configuration uses names for readability and to avoid account-specific -> IDs hardcoded in source control." - -**Contact & FinOps:** -> "Team information and cost allocation tags. These get embedded in the cluster -> configuration and inherited by all Terraform runs against this cluster." - -**Creator Username (optional):** -> "If you provide your GitHub username here, you'll get direct admin access to the -> created repo in addition to the owning team." +> "VPC Name is the name-tag string for the VPC the cluster will live in — not a VPC ID. +> The Terragrunt configuration uses names for readability. VPC Domain Name is the DNS +> domain for that network." + +**Contact & Organization:** +> "Owning Team is the GitHub team that gets admin access to the created repo. +> Cluster Mailing List is optional — leave blank if you don't have one. +> Organization Path defaults to `census:ocio:csvd`." + +**FinOps:** +> "FinOps Project Name and Number are optional cost-allocation fields. Leave blank +> if you don't have them yet — they can be added to the config manually later." + +**Optional Metadata:** +> "Creator Username grants your GHE user direct admin access in addition to the team. +> Additional Tags accepts a JSON object of extra resource tags — the default `{}` is fine." ### Step 4: Launch and watch *Click "Launch" → switch to the CloudFormation tab* diff --git a/docs/SC-TEMPLATE-FIX-PLAN.md b/docs/SC-TEMPLATE-FIX-PLAN.md new file mode 100644 index 0000000..77f53bd --- /dev/null +++ b/docs/SC-TEMPLATE-FIX-PLAN.md @@ -0,0 +1,86 @@ +# Plan: Fix Live SC Product Template + Lambda Cleanup + +**Date:** 2026-04-20 +**Status:** Complete + +The live `2-0-0.yaml` in S3 is broken — it uses `vpc_id` instead of `vpc_name` +(which the Lambda requires), omits `account_name` entirely, and relies on a DynamoDB +lookup that was never implemented and cannot work given the centralized Lambda +architecture (Lambda runs in csvd-dev and cannot reach provisioner-account resources). + +Additionally, the Lambda has dead `is_eks_deployment` conditional logic — the Lambda +is EKS-only by design, and the "false" branch just raises an error, so the check +provides no value and should be removed in favour of proper Pydantic field validation. + +--- + +## Steps + +- [x] **1. Clean up `CloudFormationResourceInput` in `template_automation/app.py`** + - Remove the `is_eks_deployment` property entirely + - Make previously-optional EKS fields required (`str = Field(...)`) for: + `cluster_name`, `account_name`, `aws_account_id`, `vpc_name`, `vpc_domain_name`, + `environment`, `environment_abbr`, `aws_region` + - Remove the `if not cfn_input.is_eks_deployment: raise ValueError(...)` block in + `lambda_handler` — Pydantic validation will surface missing fields with a precise + error naming exactly which field(s) are absent + - Keep `cluster_mailing_list`, `organization_path`, `finops_project_name`, + `finops_project_number`, `creator_username` as optional + +- [x] **2. Update `terraform-service-catalog-census/templates/products/eks-terragrunt-repo/2-0-0.yaml`** + - Remove `AWSAccountId` and `AwsRegion` as user-facing `Parameters` + - Pass them in the `Properties` block via `!Sub "${AWS::AccountId}"` and + `!Sub "${AWS::Region}"` — resolved client-side by CFN in the provisioner's account + - Replace `VpcId` (`Type: AWS::EC2::VPC::Id`) with `VpcName` (`Type: String`) + - Add back `AccountName` as a required `String` parameter + - Restore all other fields from the canonical local template: `EnvironmentAbbr`, + `VpcDomainName`, `OwningTeam`, `ClusterMailingList`, `OrganizationPath`, + `FinOpsProjectName`, `FinOpsProjectNumber`, `CreatorUsername`, `AdditionalTags` + - Pass all fields to the Custom Resource `Properties` block in `snake_case` + +- [x] **3. Sync `lambda-template-repo-generator/service-catalog/product-template.yaml`** + - Apply the same `aws_account_id`/`aws_region` substitution change + - This is the canonical reference copy — must stay identical to the census template + +- [x] **4. Deploy the updated template via `terraform-service-catalog-census`** + - `cd terraform-service-catalog-census/non-prod/csvd-dev/west/service-catalog` + - `tf apply` + - Confirm a new provisioning artifact version is created and set as the active version + +- [x] **5. Rebuild and redeploy the Lambda** + - `cd lambda-template-repo-generator && source ~/aws-creds && packer-pipeline --config csvd_config_packer.hcl` + - After build SUCCEEDED: `aws lambda update-function-code --function-name eks-terragrunt-repo-gen-template-automation --image-uri 229685449397.dkr.ecr.us-gov-west-1.amazonaws.com/eks-terragrunt-repo-generator/lambda:latest --region us-gov-west-1` + +- [x] **6. Update the demo script** + - Update `docs/DEMO_SCRIPT.md` pre-stage parameter table to match the corrected form + - Update the Step 3 walkthrough to reflect removed fields and correct field names + +--- + +## Verification + +- Launch the SC product from a non-csvd-dev account and confirm the form shows correct fields +- Verify `aws_account_id` in CFN stack events shows the provisioner's account ID (not csvd-dev's) +- Confirm Lambda CloudWatch logs show Pydantic validation passing and CodeBuild starting +- Confirm `is_eks` log line is gone from Lambda logs +- Full end-to-end: `python scripts/test_service_catalog.py sc-e2e-test-$(date +%Y%m%d-%H%M)` +- Clean up: `python scripts/cleanup_test_repos.py` + +--- + +## Decisions + +- **No DynamoDB lookup:** Architecturally incompatible with centralized Lambda. Lambda runs + in csvd-dev and cannot reach provisioner-account resources (DynamoDB, SSM, EC2) without + cross-account role assumption. The `service-catalog-projects` table was also never designed + for EKS records. +- **`aws_account_id` and `aws_region` dropped from form:** Resolved client-side by CFN + (`!Sub "${AWS::AccountId}"` / `!Sub "${AWS::Region}"`) before the Lambda is invoked — + gives the provisioner's correct values, not csvd-dev's. +- **All other fields remain user-supplied:** `account_name`, `vpc_name`, `vpc_domain_name` + etc. have no CFN pseudo-parameter equivalent and no reachable lookup source from csvd-dev. +- **FinOps fields kept as optional inputs:** Ambiguous per-account — one account can have + many FinOps projects, so no reliable auto-resolution strategy exists today. +- **`is_eks_deployment` removed:** Lambda is EKS-only by design. The false-branch only raised + a `ValueError` with no alternative code path. Pydantic required fields give cleaner, more + precise validation errors at the point of invocation. diff --git a/scripts/test_service_catalog.py b/scripts/test_service_catalog.py index 9229ad7..c0fc2f8 100755 --- a/scripts/test_service_catalog.py +++ b/scripts/test_service_catalog.py @@ -49,7 +49,7 @@ # Service Catalog product details SC_PRODUCT_NAME = "eks-terragrunt-eks-repo-creator" -SC_ARTIFACT_NAME = "v2.0" +SC_ARTIFACT_NAME = "v2.1" # GitHub / EKS defaults used as provisioning parameters GITHUB_API = "https://github.e.it.census.gov" @@ -60,9 +60,9 @@ DEFAULT_PARAMS: dict[str, str] = { "OwningTeam": "tf-module-admins", "Environment": "dev", - "AwsRegion": REGION, + # AwsRegion and AWSAccountId are intentionally omitted — the CFN template + # resolves them automatically via !Sub "${AWS::Region}" / "${AWS::AccountId}" "AccountName": "csvd-dev-ew", - "AWSAccountId": ACCOUNT_ID, "EnvironmentAbbr": "dev", "VpcName": "csvd-dev-vpc", "VpcDomainName": "dev.inf.csp1.census.gov",