Lambda Template Repository Generator

Overview

This Lambda function automates the creation and configuration of new GitHub repositories from templates using AWS Service Catalog and CloudFormation Custom Resources. When a user launches the product in Service Catalog, CloudFormation invokes this Lambda, which creates a new repository, clones a template, renders configuration files, and opens a pull request — all in a single provisioning operation.

The system supports two deployment modes:

Generic mode — writes a single config.json with all CloudFormation parameters
EKS deployment mode — renders a full Terragrunt/HCL file hierarchy (8 files) using Jinja2 templates, providing a production-ready EKS cluster configuration repository

Key Features

CloudFormation Native — works as a CloudFormation Custom Resource (Custom::GitHubRepository)
Service Catalog Integration — users launch from the AWS console, get outputs (repo URL, PR URL)
EKS Config Rendering — renders 8 Terragrunt HCL files (root.hcl, account.hcl, cluster.hcl, etc.) with 60+ version pins
Atomic Multi-File Commits — uses the Git tree API to write all files in a single commit
Input Validation — Pydantic v1 models for parameter validation
Multi-Provider — supports both GitHub Enterprise and GitLab
Automatic PR Creation — opens a pull request with configuration ready for review

Repository Structure

lambda-template-repo-generator/
├── main.tf                        # Step 1: Terraform for ECR repository
├── variables.tf                   # ECR Terraform variables
├── backend.tf                     # Terraform state backend
├── varfiles/                      # Terraform/Packer variable files
├── config_packer.hcl              # Step 2: Packer Pipeline configuration
├── packer.pkr.hcl                 # Packer template (container image build)
├── Dockerfile                     # Reference only (build uses Packer)
├── deploy/                        # Step 3: Lambda + Service Catalog deployment
│   ├── main.tf                    #   Uses terraform-aws-template-automation module
│   ├── variables.tf               #   Deployment variables
│   ├── service_catalog.tf         #   Portfolio, product, constraints, launch role
│   └── terraform.tfvars           #   Deployment configuration
├── service-catalog/               # CloudFormation product template
│   └── product-template.yaml      #   EKS-enabled Service Catalog product definition
├── cloudformation-template.yaml   # Generic (non-EKS) product template example
├── template_automation/           # Lambda source code (Python package)
│   ├── app.py                     #   Lambda handler (CloudFormation events)
│   ├── eks_config.py              #   EKS deployment Pydantic models + renderer
│   ├── github_provider.py         #   GitHub API provider (tree API, PRs, branches)
│   ├── gitlab_provider.py         #   GitLab API provider
│   ├── repository_provider.py     #   Abstract provider interface
│   ├── models.py                  #   Shared Pydantic models
│   ├── template_manager.py        #   Jinja2 template rendering
│   └── templates/
│       └── eks/                   #   8 Jinja2 templates for EKS Terragrunt files
│           ├── root.hcl.j2
│           ├── account.hcl.j2
│           ├── region.hcl.j2
│           ├── vpc.hcl.j2
│           ├── cluster.hcl.j2
│           ├── common-variables.hcl.j2
│           ├── default-versions.hcl.j2
│           └── README.md.j2
├── scripts/                       # Operational / test scripts
│   ├── test_workflow.py           #   End-to-end Lambda workflow tester
│   ├── test_service_catalog.py    #   End-to-end Service Catalog product tester
│   ├── check_github_permissions.py#   GitHub PAT permissions audit (11 checks)
│   ├── cleanup_test_repos.py      #   Delete temp-test-repo-* repos from GHE
│   ├── validate_github_token.py   #   Token retrieval + validation
│   └── lambda_setup.py            #   Container setup helper
├── tests/                         # Unit and integration tests
│   ├── test_app.py
│   ├── test_github_client.py
│   └── integration/
├── events/                        # Test event payloads
│   └── cloudformation-create-event.json
├── requirements.txt               # Python dependencies
└── Makefile                       # Development commands

Quick Start

Launching the Product in Service Catalog (End-User)

Sign in to the AWS GovCloud Console → navigate to Service Catalog
Click Products in the left sidebar
Find eks-terragrunt-eks-repo-creator and click Launch product

Fill in the parameters:

Parameter	Required	Example	Description
ProjectName	Yes	`my-eks-cluster`	Repository name (lowercase, hyphens)
ClusterName	No	`my-cluster`	EKS cluster name (defaults to ProjectName)
Environment	Yes	`dev`	`dev`, `test`, or `prod`
AwsRegion	Yes	`us-gov-west-1`	Target AWS region
AccountName	Yes	`csvd-dev-ew`	AWS account name
AWSAccountId	Yes	`229685449397`	12-digit account ID
EnvironmentAbbr	Yes	`dev`	Short environment name
VpcName	Yes	`csvd-dev-vpc`	VPC name for the cluster
VpcDomainName	Yes	`dev.inf.csp1.census.gov`	VPC domain
OwningTeam	No	`tf-module-admins`	GitHub team with admin access
ClusterMailingList	No	`team@census.gov`	Notification email
OrganizationPath	No	`census:ocio:csvd`	Org hierarchy path
FinOpsProjectName	No	`EKS Platform`	FinOps project
FinOpsProjectNumber	No	`12345`	FinOps number

Click Launch product
Wait for the stack to reach CREATE_COMPLETE (typically 30–60 seconds)
Click the provisioned product → Outputs tab to find:
- RepositoryUrl — the new GitHub repository
- PullRequestUrl — PR with all configuration files ready for review

What Gets Created

When the product launches with EKS parameters, the Lambda:

Creates a new internal repository in the SCT-Engineering GitHub org
Clones the template-eks-cluster template into it
Renders 8 Terragrunt/HCL configuration files from Jinja2 templates:
- root.hcl — Terragrunt root config with remote state and provider generation
- account.hcl — Account-level locals (account name, ID, org path)
- region.hcl — Region-level locals
- vpc.hcl — VPC configuration (name, domain)
- cluster.hcl — Cluster-specific settings (name, mailing list, FinOps tags)
- common-variables.hcl — Shared Terraform variable definitions
- default-versions.hcl — 60+ pinned versions for all EKS add-ons
- README.md — Auto-generated documentation
Writes all files atomically in a single Git commit (via Git tree API)
Also writes a config.json with all raw parameters for backward compatibility
Opens a pull request from repo-init → main
Assigns the owning team as repository admins

Build & Deploy Workflow

Deployment is a three-step process:

Step 1: Create ECR Repo          Step 2: Build Container           Step 3: Deploy Lambda + Service Catalog
────────────────────────          ─────────────────────────         ────────────────────────────────────────
  terraform apply                   packer-pipeline                   terraform apply
  (root main.tf)                    (config_packer.hcl)               (deploy/ directory)
       │                                  │                                 │
       ▼                                  ▼                                 ▼
  ECR Repository                   CodeBuild → Packer →              Lambda Function
  (container storage)              Docker → ECR push                 Service Catalog Portfolio
                                                                     Service Catalog Product
                                                                     Launch Role + Constraints
                                                                     IAM, CloudWatch, API Gateway

Step 1: Create the ECR Repository

cd lambda-template-repo-generator/
terraform init
terraform apply -var-file=varfiles/default.tfvars

Creates the ECR repository at 229685449397.dkr.ecr.us-gov-west-1.amazonaws.com/eks-terragrunt-repo-generator/lambda.

Step 2: Build the Container Image

The packer-pipeline tool builds the Lambda container via CodeBuild:

# Build and push to ECR (waits for completion)
packer-pipeline --config config_packer.hcl --wait

This:

Zips the repository (91 files) and uploads to S3
Clones the base image public.ecr.aws/lambda/python:3.11 to private ECR
Creates/updates CodeBuild project eks-terragrunt-repo-generator-builder
CodeBuild runs the Packer template which:
- Installs Python dependencies from requirements.txt
- Copies template_automation/ package into /var/task
- Installs Census Bureau Root CA certificate for SSL
- Tags and pushes the image to ECR

After the build, update the Lambda to use the new image:

aws lambda update-function-code \
  --function-name eks-terragrunt-repo-gen-template-automation \
  --image-uri 229685449397.dkr.ecr.us-gov-west-1.amazonaws.com/eks-terragrunt-repo-generator/lambda:latest \
  --region us-gov-west-1

Step 3: Deploy Lambda + Service Catalog

cd deploy/
# Edit terraform.tfvars with your values (see deploy/terraform.tfvars for reference)
terraform init
terraform apply

This deploys:

Lambda Function — container image from ECR, VPC-attached, 512 MB, 5 min timeout
IAM Role & Policies — SSM, Secrets Manager, KMS, CloudWatch, VPC
Service Catalog Portfolio & Product — with launch role and template constraint
Launch Constraint — dedicated role for CloudFormation to invoke Lambda
Template Constraint — locks LambdaFunctionArn to the deployed function ARN
API Gateway — HTTP API with POST /template endpoint (alternative invocation)
CloudWatch Log Group — /aws/lambda/eks-terragrunt-repo-gen-template-automation

Testing

End-to-End Test Script (`scripts/test_workflow.py`)

The test script validates the entire stack without using Service Catalog:

# Generic (non-EKS) test — creates "workflow-test-<timestamp>" repo
python scripts/test_workflow.py

# EKS deployment test — creates "eks-test-<timestamp>" repo with full HCL rendering
python scripts/test_workflow.py --eks

# Custom repo name
python scripts/test_workflow.py my-test-repo
python scripts/test_workflow.py --eks my-eks-test

The script runs 6 sequential checks:

Step	What It Does
1. AWS Credentials	Validates STS identity and account
2. Lambda Env Vars	Reads Lambda configuration, checks required vars
3. Secrets Manager	Retrieves GitHub token, validates format
4. GitHub API	Tests authentication against GHE `/user` endpoint
5. Lambda Invoke	Sends async CloudFormation Custom Resource event
6. CloudWatch Tail	Tails logs for 120s, watches for SUCCESS/FAILED

Output includes a Rich-formatted summary table and pass/fail panel with repo and PR URLs.

Service Catalog Test Script (`scripts/test_service_catalog.py`)

Tests the complete Service Catalog → CloudFormation → Lambda → GitHub path:

# Full SC product test (auto-named sc-test-<timestamp>, auto-cleanup)
python scripts/test_service_catalog.py

# Custom repo name, no cleanup
python scripts/test_service_catalog.py --no-cleanup my-sc-test

# Terminate an existing provisioned product
python scripts/test_service_catalog.py --terminate my-sc-test

# Longer timeout (default is 600s)
python scripts/test_service_catalog.py --timeout 900

Step	What It Does
1. AWS Credentials	Validates STS identity and account
2. SC Product Lookup	Finds product and provisioning artifact by name
3. Provision Product	Launches the SC product with all 15 EKS parameters
4. Poll CFN Stack	Streams CloudFormation events until terminal state
5. GitHub Verification	Confirms the repository exists and has an open PR
6. Cleanup	Terminates the provisioned product (skippable with `--no-cleanup`)

GitHub Permissions Checker

Validates that the PAT has all required scopes before deployment:

python scripts/check_github_permissions.py
python scripts/check_github_permissions.py --token ghp_... --org MyOrg

Checks 11 API operations: /user, /user/orgs, /orgs/{org}, membership, repo listing, repo creation, template access, branch listing, git ref creation, PR creation, and team permissions.

Direct Lambda Invocation

aws lambda invoke \
  --function-name eks-terragrunt-repo-gen-template-automation \
  --payload file://events/cloudformation-create-event.json \
  --region us-gov-west-1 \
  response.json

CloudFormation Stack Test

aws cloudformation create-stack \
  --stack-name test-eks-repo \
  --template-body file://service-catalog/product-template.yaml \
  --parameters \
    ParameterKey=ProjectName,ParameterValue=test-eks-$(date +%s) \
    ParameterKey=AccountName,ParameterValue=csvd-dev-ew \
    ParameterKey=AWSAccountId,ParameterValue=229685449397 \
    ParameterKey=EnvironmentAbbr,ParameterValue=dev \
    ParameterKey=VpcName,ParameterValue=csvd-dev-vpc \
    ParameterKey=VpcDomainName,ParameterValue=dev.inf.csp1.census.gov \
  --region us-gov-west-1

Runtime Architecture

┌──────────────────────────────────────────────────────────────────────┐
│  User → AWS Service Catalog UI → "Launch Product"                    │
│           ↓                                                          │
│  CloudFormation creates stack                                        │
│           ↓                                                          │
│  Custom::GitHubRepository resource invokes Lambda                    │
│           ↓                                                          │
│  Lambda (template_automation.app.lambda_handler)                     │
│    ├─ Validates CloudFormation parameters (Pydantic)                 │
│    ├─ Creates repository in GitHub (internal visibility)             │
│    ├─ Clones template-eks-cluster contents                           │
│    ├─ Creates repo-init branch                                       │
│    ├─ Detects EKS deployment? ─┬─ YES: render 8 HCL files + config  │
│    │                           └─ NO:  write single config.json      │
│    ├─ Atomic commit via Git tree API                                 │
│    ├─ Opens PR: repo-init → main                                     │
│    ├─ Sets team permissions (admin)                                  │
│    └─ Returns SUCCESS + URLs to CloudFormation                       │
│           ↓                                                          │
│  CloudFormation completes → Outputs: RepositoryUrl, PullRequestUrl   │
└──────────────────────────────────────────────────────────────────────┘

Lambda Environment Variables

Variable	Description	Example
`GITHUB_API`	GitHub Enterprise API URL	`https://github.e.it.census.gov`
`GITHUB_TOKEN_SECRET_NAME`	Secrets Manager path for PAT	`/eks-cluster-deployment/github_token`
`GITHUB_ORG_NAME`	GitHub organization	`SCT-Engineering`
`TEMPLATE_REPO_NAME`	Template repo to clone from	`template-eks-cluster`
`REPO_VISIBILITY`	Repo visibility (private/internal/public)	`internal`
`VERIFY_SSL`	SSL verification (`true`/`false`)	`false` (until CA cert baked in)
`PARAM_STORE_PREFIX`	SSM parameter path prefix	`/eks-terragrunt-repo-gen`
`TEMPLATE_CONFIG_FILE`	Config filename (optional)	`config.json`

EKS Deployment Detection

The Lambda automatically detects EKS deployments when the incoming parameters include all of:

cluster_name
account_name
aws_account_id
vpc_name
vpc_domain_name

When detected, it uses eks_config.py to build a fully-hydrated EKSDeploymentConfig Pydantic model and renders all 8 Jinja2 templates into HCL files.

Development

Install Dependencies

pip install -r requirements.txt
pip install -e .

Run Tests

# Unit tests
make test-unit

# Integration tests (requires GitHub token)
make test-integration

# All tests
make test

Clean Up Test Repositories

python scripts/cleanup_test_repos.py
# or
make clean-test-repos

Monitoring

# Tail Lambda logs
aws logs tail /aws/lambda/eks-terragrunt-repo-gen-template-automation \
  --follow --region us-gov-west-1

# Filter for errors
aws logs filter-log-events \
  --log-group-name /aws/lambda/eks-terragrunt-repo-gen-template-automation \
  --filter-pattern "ERROR" --region us-gov-west-1

Related Repositories

Repository	Purpose
packer-pipeline	Build tool — packages code, runs CodeBuild, pushes to ECR
terraform-aws-template-automation	Terraform module — deploys Lambda, IAM, API Gateway, SSM
template-eks-cluster	Template repo — cloned as starting point for new EKS repos
terraform-eks-deployment	Reference — Terragrunt structure this Lambda replicates

Documentation Index

Document	Description
DEPLOYMENT.md	Step-by-step deployment guide
CLOUDFORMATION_CUSTOM_RESOURCE_GUIDE.md	How Custom Resources work, Service Catalog setup, troubleshooting
CLOUDFORMATION_CUSTOM_RESOURCE_MIGRATION.md	Migration notes from EventBridge to Custom Resources
PACKER_PIPELINE_EXPLANATION.md	Resolving the `terraform_version` variable issue
PACKER_UPDATES.md	Packer/container build configuration changes
design-docs/README.md	System architecture and implementation phases
design-docs/CUSTOM_TEMPLATES.MD	Subdirectory and multi-template patterns
docs/SERVICE_CATALOG_RESOLUTION.md	S3 tag-based access fix for Service Catalog

CSVD/lambda-template-repo-generator

About

Resources

Stars

Watchers

Forks

Releases

Languages