GitHub Actions ECS Fargate Runners

Infrastructure as Code (Terraform) for deploying self-hosted GitHub Actions runners on AWS ECS Fargate.

Overview

This repository manages the deployment of organization-level self-hosted GitHub Actions runners using AWS ECS Fargate. Runners are deployed per AWS account and automatically register with your GitHub Enterprise organization, providing secure, scalable, and cost-effective CI/CD execution environments.

Runner Model: Runners are persistent, long-running containers that stay active 24/7, continuously polling GitHub for jobs. They are not ephemeral — the same runner handles multiple workflow jobs over its lifetime.

Key Features

Persistent Runners: Long-running containers that stay online and handle multiple jobs
Serverless Infrastructure: ECS Fargate eliminates server management overhead
Account-Based Isolation: Each AWS account has its own dedicated runners
Automatic IAM Authentication: ECS Task Roles provide seamless AWS access
Multi-Label Support: Runners tagged with account ID, name, region, and more
Proxy-Enabled: Pre-configured for enterprise proxy environments
CloudWatch Integration: Centralized logging, monitoring, dashboard, and SNS alerting
Automated Token Refresh: Lambda (optional) keeps registration tokens fresh for task restarts
ECR Mirroring: Mirrors public runner image into private ECR by default (enable_ecr_clone = true)
Scalable: Adjust runner count based on workload demands

Architecture

Runner Lifecycle Model

Runners are persistent, long-running containers that operate continuously:

Startup: Container starts, reads registration token from Secrets Manager, registers with GitHub
Active State: Runner stays online indefinitely, polling GitHub for workflow jobs
Job Execution: When a job arrives, runner executes it and returns to polling
Restart: Only stops when task fails, is manually terminated, or service is updated
Auto-Recovery: If a task dies, ECS automatically starts a replacement (requires valid token)

GitHub Enterprise (github.e.it.census.gov)
            |
            | (Token Authentication via GITHUB_TOKEN)
            v
    ECS Cluster (per account/region)
    {ecs_cluster_name}-{region}   e.g. ecs-ghe-runners-us-gov-west-1
            |
            +-- Fargate Task (Persistent Runner 1)
            |   +-- Image: {account_id}.dkr.ecr.{region}.amazonaws.com/github-runners/github-runner:{version}
            |   +-- Lifecycle: Long-running (24/7)
            |   +-- Registration: One-time at startup (token from Secrets Manager)
            |   +-- IAM Task Role (AWS Auth for workflows)
            |   +-- Labels: Account ID, Name, Region, self-hosted, etc.
            |   +-- Logs -> CloudWatch
            |
            +-- Fargate Task (Persistent Runner 2)
            +-- Fargate Task (Persistent Runner N)

    Lambda Token Refresh (every 30 min, when enable_lambda_token_refresh = true)
            |
            +---> AWS Secrets Manager
                  (Keeps registration token fresh for ECS task restarts)

    Monitoring (always deployed)
            +-- SNS Topic:            github-runner-critical-alerts-{account_id}-{region}
            +-- CloudWatch Alarm:     runners < 50% capacity  (critical)
            +-- CloudWatch Alarm:     all runners down        (emergency)
            +-- CloudWatch Dashboard: github-runners-{aws_account}

Network Architecture:

Deployment: Private subnets (no public IP by default)
Internet Access: Corporate proxy (proxy.tco.census.gov:3128)
VPC Endpoints: Optional (ECR, S3, EC2) for AWS service access
Security Groups: Restrict egress to necessary services

Prerequisites

AWS Requirements

AWS Account Access

You need an AWS account with appropriate IAM permissions to deploy the infrastructure. The deploying user/role requires:

Terraform Deployment Permissions:

IAM: Create and manage IAM roles, policies, and policy attachments
ECS: Create and manage ECS clusters, services, and task definitions
CloudWatch Logs: Create and manage log groups and dashboards
Secrets Manager: Create and manage secrets for GitHub tokens
EC2 (VPC): Query VPC resources and optionally create VPC endpoints
ECR: Pull and push container images (ECR mirroring is enabled by default)
SNS: Create topics and email subscriptions for alerting
Lambda: Deploy token refresh Lambda function (when enable_lambda_token_refresh = true)
EventBridge: Create scheduled events for Lambda token refresh
S3/DynamoDB: Access Terraform state backend

See AWS_PERMISSIONS.md for detailed permission policies and example IAM policy documents.

Infrastructure Requirements

VPC with private subnets (runners deploy without public IPs by default)
Security groups allowing egress to:
- GitHub Enterprise (github.e.it.census.gov on HTTPS/443)
- Corporate proxy (proxy.tco.census.gov:3128)
- AWS services (S3, DynamoDB, Secrets Manager) via VPC endpoints or internet gateway
(Optional) VPC endpoints for ECR, S3, EC2 to reduce data transfer costs and improve security

GitHub Requirements

GitHub Enterprise organization admin access
GitHub Personal Access Token (PAT) — the primary authentication method
- Set via environment variable: export GITHUB_TOKEN="your-token"
- Required scopes: admin:org, repo
- Used by the Terraform GitHub provider and, when enabled, the Lambda token refresh function
- Never commit this token to version control or store it in .tfvars files
(Optional) GitHub App authentication — see GITHUB_APP_SETUP.md
- Variables github_app_id, github_app_installation_id, and github_app_pem_file all default to null

Tools Required

Terraform >= 1.0
- Local shell: use the tf script vendored in the repo at codebuild/tf. It wraps terraform and automatically sets TF_DATA_DIR, TF_CLI_ARGS_* var-file injection, and loads workspace-specific JSON env files — all things the plain terraform binary does not do on its own.
- CodeBuild: installs codebuild/tf from the repo checkout during the install phase (see Automation (CodeBuild)).
AWS CLI configured
Python 3 + pip (required for Lambda package build when enable_lambda_token_refresh = true)
Access to the organization's Terraform state backend

Quick Start

1. Clone the Repository

git clone git@github.e.it.census.gov:CSVD/ghe-runners.git
cd ghe-runner

2. Export Required Environment Variables

# Required: GitHub PAT used by Terraform GitHub provider and Lambda token refresh
export GITHUB_TOKEN="ghp_your_personal_access_token"

# AWS credentials (refresh via awscreds if expired)
awscreds

3. Configure Variables

Edit default.auto.tfvars or create a workspace-specific .auto.tfvars file:

# GitHub Configuration (token is set via GITHUB_TOKEN env var — not here)
repo_org   = "SCT-Engineering"                         # Your GitHub organization
server_url = "https://github.e.it.census.gov"         # GitHub Enterprise URL

# ECS Cluster
ecs_cluster_name   = "ecs-ghe-runners"                # Base name; region is appended automatically
create_ecs_cluster = true                             # Set false to use an existing cluster

# AWS
aws_account     = "csvd-dev-ew"                       # Account identifier (also used as runner label)
namespace       = "csvd"                              # Namespace for resource naming
vpc_id          = "vpc-0abc123def456"                 # VPC ID
subnets         = ["subnet-abc123", "subnet-def456"]  # Private subnet IDs
security_groups = ["sg-xyz789"]                       # Security group IDs

# Runner image (mirrored from public.ecr.aws/h1g9x7n8/ into private ECR)
image_name       = "github-runner"
image_version    = "1.67.0"
enable_ecr_clone = true   # default: true — mirrors image to private ECR at apply time

# Runner count
desired_count = 1

# Monitoring (required)
alert_email = "your.email@census.gov"   # Receives CloudWatch alarm SNS notifications

# Optional: automated token refresh Lambda
enable_lambda_token_refresh = false   # Set true to prevent token-expiry deadlocks

4. Initialize Terraform

tf init

5. Create or Select Workspace

# Create new workspace
tf workspace new csvd-dev-ew

# Or select existing workspace
tf workspace select default

6. Deploy Runners

# Review planned changes
tf plan

# Apply configuration
tf apply

Important: Always use the tf script (codebuild/tf, also installed to your PATH as tf) instead of terraform directly in your local shell. It sets TF_DATA_DIR, injects the workspace-specific var-file via TF_CLI_ARGS_*, and loads env vars from the JSON file — plain terraform will not do this automatically.

7. Confirm SNS Email Subscription

After the first deployment, check your inbox for an AWS SNS subscription confirmation email and click Confirm subscription to activate alerting.

8. Verify Deployment

Check ECS:

aws ecs list-tasks --cluster ecs-ghe-runners-us-gov-west-1

Check GitHub:

Navigate to https://github.e.it.census.gov/organizations/SCT-Engineering/settings/actions/runners
Verify runners appear with status "Idle"
Confirm runner labels match configuration

Automation (CodeBuild)

The codebuild/ directory contains a Terraform module that deploys a scheduled CodeBuild project (ghe-runner-daily-{workspace}) that runs tf apply daily. The tf script is vendored in the repo at codebuild/tf and copied to /usr/local/bin/tf during the install phase, so all workspace-aware behavior (TF_DATA_DIR, var-file injection, JSON env loading) is handled automatically. To publish changes back to the canonical gist, run make sync-gist from codebuild/.

How Workspace Selection Works in CodeBuild

The tf script is installed from the repo checkout at build time:

Responsibility	How it's handled
`TF_DATA_DIR`, `TF_CLI_ARGS_*`, JSON env vars	Vendored `codebuild/tf` handles everything automatically
Target workspace	`TF_WORKSPACE` env var injected by CodeBuild at build time
Backend config	`pre_build` selects `backend-configs/${TF_WORKSPACE}.tf` if present
GitHub PAT	`GITHUB_TOKEN` (from Secrets Manager) exported as `TF_VAR_github_token`

The tf binary resolver falls back to terraform in PATH, which is the binary installed earlier in install — no extra configuration required.

CodeBuild Job Phases

Phase	What happens
`install`	Downloads Terraform `${TF_VERSION}`; copies `codebuild/tf` → `/usr/local/bin/tf`; warms provider cache from S3; installs `requests`
`pre_build`	Exports `TF_VAR_github_token`; runs `tf init` (with optional backend-config); runs `tf workspace select`
`build`	`tf apply -auto-approve`; invokes Lambda token refresh (so token is fresh before containers start); forces new ECS deployment
`post_build`	Logs completion timestamp and active workspace

Deploying the CodeBuild Module

The CodeBuild infrastructure lives in codebuild/ and is managed separately from the runner infrastructure.

Step 1: Add workspace files (if targeting a new workspace)

varfiles/{workspace}.tfvars         # required — same vars as for local tf apply
varfiles/{workspace}.json           # optional — extra env vars loaded by tf script
backend-configs/{workspace}.tf      # optional — workspace-specific backend config

Step 2: Deploy the CodeBuild project

cd codebuild/
tf init
tf workspace select -or-create default   # CodeBuild itself uses the local tf alias
tf apply -var tf_workspace=default \
         -var github_token_secret_arn=arn:aws-us-gov:secretsmanager:...

Step 3: Verify

# Trigger a manual build to test
aws codebuild start-build \
  --project-name ghe-runner-daily-default \
  --environment-variables-override name=TF_WORKSPACE,value=default,type=PLAINTEXT

Targeting a Different Workspace Without Redeploying

You can override TF_WORKSPACE at build invocation time without changing the CodeBuild project's default:

aws codebuild start-build \
  --project-name ghe-runner-daily-default \
  --environment-variables-override name=TF_WORKSPACE,value=sct-engineering,type=PLAINTEXT

This works because the buildspec reads $TF_WORKSPACE at runtime — terraform workspace select -or-create "$TF_WORKSPACE" will create the workspace if it does not yet exist.

GitHub PAT in CodeBuild

The PAT is stored in AWS Secrets Manager and injected as GITHUB_TOKEN by CodeBuild at build start. The buildspec then maps it to TF_VAR_github_token so the Terraform github_token variable is populated without ever writing the token to disk or a var-file:

- export TF_VAR_github_token="$GITHUB_TOKEN"

The Terraform GitHub provider reads GITHUB_TOKEN automatically; the Lambda variable (var.github_token) is satisfied by TF_VAR_github_token.

Configuration Reference

Required Variables

Variable	Description	Example
`ecs_cluster_name`	Base ECS cluster name (region is appended automatically)	`"ecs-ghe-runners"`
`repo_org`	GitHub organization name	`"SCT-Engineering"`
`namespace`	Namespace for resource naming	`"csvd"`
`aws_account`	AWS account identifier (also used as a runner label)	`"csvd-dev-ew"`
`vpc_id`	VPC ID for runner and Lambda deployment	`"vpc-0abc123"`
`image_name`	GitHub runner container image name	`"github-runner"`
`image_version`	Container image version tag	`"1.67.0"`
`alert_email`	Email for CloudWatch alarm notifications	`"team@census.gov"`

Authentication — set via environment variable, not in .tfvars:

Variable	Description	How to Set
`github_token`	GitHub PAT for Terraform provider and Lambda token refresh	`export GITHUB_TOKEN="ghp_..."`

Optional Variables

Variable	Description	Default
`subnets`	List of private subnet IDs for runners and Lambda	`[]`
`security_groups`	Security group IDs for runners and Lambda	`[]`
`desired_count`	Number of runner instances	`3`
`create_ecs_cluster`	Create a new ECS cluster (false = use existing)	`false`
`assign_public_ip`	Assign public IP to runner tasks	`false`
`create_vpc_endpoint`	Create Interface VPC endpoints for AWS services	`false`
`enable_ecr_clone`	Mirror runner image from public ECR to private ECR	`true`
`enable_lambda_token_refresh`	Deploy Lambda for automatic 30-min token refresh	`false`
`server_url`	GitHub Enterprise URL for runner registration	`""`
`base_url`	GitHub base URL for Terraform provider	`"https://github.e.it.census.gov"`
`cluster_size`	ECS cluster capacity setting	`1`
`certs`	S3 `{ bucket, key }` for CA certificate to inject into runners	`null`
`github_app_id`	GitHub App ID (optional alternative to PAT)	`null`
`github_app_installation_id`	GitHub App Installation ID	`null`
`github_app_pem_file`	Path to GitHub App private key `.pem`	`null`

Environment Variables (Set in ECS Task Definition)

HTTP_PROXY  = "http://proxy.tco.census.gov:3128"
HTTPS_PROXY = "http://proxy.tco.census.gov:3128"
NO_PROXY    = "169.254.170.2,.census.gov,169.254.169.254,148.129.0.0/16,10.0.0.0/8,<github_ips>,172.16.0/12,.eks.amazonaws.com,.s3.amazonaws.com,.amazonaws.com,.gcr.io,.pkg.dev"

Note: NO_PROXY is built dynamically at plan time. GitHub Enterprise IP addresses are resolved via DNS (data.dns_a_record_set.github) and included automatically.

Runner Image

By default (enable_ecr_clone = true), the runner image is mirrored from the public ECR registry into your account's private ECR during tf apply using the HappyPathway/ecr-clone/aws Terraform module:

Setting	Value
Source registry	`public.ecr.aws`
Source image	`h1g9x7n8/github-runner:{version}`
Private ECR repository	`github-runners/github-runner`
Image URI (ECR clone enabled)	`{account_id}.dkr.ecr.{region}.amazonaws.com/github-runners/github-runner:{version}`
Image URI (ECR clone disabled)	`public.ecr.aws/h1g9x7n8/github-runner:{version}`

When you update image_version, the new image is automatically mirrored to private ECR during the next tf apply.

Runner Labels

Each runner is automatically configured with the following labels:

Label	Source	Example
`self-hosted`	Fixed	`self-hosted`
`ecs`	Fixed	`ecs`
`github-runner`	Fixed	`github-runner`
`{aws_account}`	`var.aws_account`	`csvd-dev-ew`
`{org_name_lowercase}`	`lower(var.repo_org)`	`sct-engineering`
`{account_id}`	AWS caller identity	`229685449397`
`{account_id}-{region}`	AWS caller identity + region	`229685449397-us-gov-west-1`
`{region}`	AWS region	`us-gov-west-1`
`ecs-github-runner`	Fixed	`ecs-github-runner`
`ubuntu-latest`	Fixed (compatibility)	`ubuntu-latest`

Most Common Usage:

jobs:
  build:
    runs-on: ["self-hosted", "229685449397"]  # Target specific AWS account by ID

Runner Group

The runner group is automatically created in GitHub and named after the AWS account ID (e.g., 229685449397). Group configuration:

allows_public_repositories = true
restricted_to_workflows = false
visibility = "all"

Network Configuration

Proxy Settings

Runners are pre-configured to route traffic through the corporate proxy:

HTTP/HTTPS Proxy: http://proxy.tco.census.gov:3128
No Proxy: Internal Census Bureau networks, AWS services, and GitHub Enterprise IPs (resolved dynamically at plan time)

VPC Endpoints (Optional)

When create_vpc_endpoint = true, the following Interface endpoints are created in the runner VPC:

ECR API: com.amazonaws.{region}.ecr.api
ECR Docker: com.amazonaws.{region}.ecr.dkr
S3: com.amazonaws.{region}.s3
EC2: com.amazonaws.{region}.ec2

Benefits:

Reduced data transfer costs
Improved security (no internet egress required for AWS services)
Faster access to AWS services from runner tasks

IAM Permissions

The runner infrastructure uses two distinct IAM roles:

ECS Task Role

Assumed by the running container — grants GitHub Actions workflows AWS permissions.

Role Name Pattern: {repo_org}-task-role (e.g., SCT-Engineering-task-role)

Default Policy (iam_policy/admin.json) — full admin, configured via github_runner_permissions_arn:

{
  "Statement": [{ "Action": "*", "Effect": "Allow", "Resource": "*" }],
  "Version": "2012-10-17"
}

Automatically Attached Permissions:

Secrets Manager: Read GitHub registration token from /github-runners/{namespace}/*
S3: Access CA certificate (when certs variable is configured)

Customization: To use a least-privilege policy instead of the default admin policy:

github_runner_permissions_arn = "arn:aws:iam::{account_id}:policy/custom-runner-policy"

Security Note: The admin policy is intentionally permissive to support Terraform's dynamic provisioning. Security is enforced through repository access controls, pull request reviews, branch protection rules, and CloudTrail audit logging. See AWS_PERMISSIONS.md and SECURITY_REVIEW.md.

ECS Execution Role

Used by ECS to manage the container lifecycle (image pull, log write, secret retrieval).

Role Name Pattern: {repo_org}-task-execution-role

Managed Policy: arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy

Monitoring & Logging

All monitoring resources are deployed automatically as part of this Terraform configuration. No additional setup is needed beyond confirming the initial SNS email subscription.

CloudWatch Log Group

Property	Value
Log group pattern	`/ecs-ghe-runners/{workspace}-{account_id}-{region}`
Retention	90 days

# Tail runner logs
aws logs tail /ecs-ghe-runners/default-229685449397-us-gov-west-1 --follow

SNS Alerting

Resource	Name Pattern
SNS topic	`github-runner-critical-alerts-{account_id}-{region}`
Email subscription	`var.alert_email`

After first tf apply, confirm the SNS subscription from the AWS email sent to alert_email.

CloudWatch Alarms

Alarm	Trigger	Severity
`github-runners-critical-capacity-{aws_account}`	`RunningTaskCount` (ECS/ContainerInsights) < 50% of `desired_count` for 10 min	Critical
`github-runners-emergency-all-down-{aws_account}`	`RunningTaskCount` = 0 for 1 min	Emergency
`{lambda_function_name}-errors`	Lambda error count > 1 (only when Lambda enabled)	Warning

Both runner alarms also send OK notifications upon recovery.

CloudWatch Dashboard

A dashboard named github-runners-{aws_account} is created automatically and includes:

Runner count over time (Average / Maximum / Minimum, with desired-count and 50% threshold annotations)
ECS CPU and Memory utilization
Critical alarm status panel
Recent error events (CloudWatch Logs Insights query)

Access the dashboard:

https://console.aws.amazon.com/cloudwatch/home?region=us-gov-west-1#dashboards:name=github-runners-{aws_account}

Automated Token Refresh (Lambda)

The Lambda token refresh is deployed and controlled by enable_lambda_token_refresh (default: false). Enable it for production deployments to prevent runner recovery deadlocks.

Critical: Token Expiry Deadlock Risk

⚠️ GitHub registration tokens expire after ~1 hour. If all runners go down AND the token in Secrets Manager has expired, ECS cannot auto-recover — new tasks need a valid token to register. The Lambda prevents this by refreshing the token every 30 minutes.

How It Works

EventBridge triggers the Lambda every 30 minutes
Lambda calls the GitHub API (api/v3/orgs/{org}/actions/runners/registration-token) using GITHUB_TOKEN
Fresh registration token is written to Secrets Manager at the path ECS tasks use on startup
Whenever a task is restarted for any reason, a valid token is available

Enable Lambda Token Refresh

# In default.auto.tfvars
enable_lambda_token_refresh = true

tf apply

Lambda Reference

Property	Value
Function name	`github-runner-token-refresh-{aws_account}`
Runtime	Python 3.11
Timeout	60 seconds
Schedule	Every 30 minutes (EventBridge)
VPC	Deployed in the same `subnets` + `security_groups` as runners
Authentication	`GITHUB_TOKEN` env var (value of `var.github_token`, set at deploy time)
Log group	`/aws/lambda/github-runner-token-refresh-{aws_account}` (7-day retention)
Secret path prefix	`/github-runners/{lower(repo_org)}-{account_id}-{region}`

Important: The Lambda runs inside the VPC. It requires outbound internet access through the corporate proxy to reach the GitHub API. Ensure the subnets and security groups allow egress to github.e.it.census.gov:443 via the proxy.

Invoke Lambda Manually

aws lambda invoke \
  --function-name github-runner-token-refresh-csvd-dev-ew \
  --log-type Tail \
  /tmp/lambda-output.json && cat /tmp/lambda-output.json

View Lambda Logs

aws logs tail /aws/lambda/github-runner-token-refresh-csvd-dev-ew --follow

Troubleshooting

Runner Not Appearing in GitHub

Symptoms: ECS tasks running, but runners not visible in GitHub organization settings

Check CloudWatch logs for authentication errors (look for 401, failed, token)
Verify GITHUB_TOKEN is valid and has admin:org scope
If enable_lambda_token_refresh = false, the token in Secrets Manager may be expired (> 1 h)
Verify security group allows outbound HTTPS (443) via the proxy to github.e.it.census.gov

Runner Shows as Offline

Symptoms: Runner appears in GitHub but status is "Offline"

Check ECS task status — verify task is in RUNNING state
Review CloudWatch logs for errors
Verify proxy connectivity from the VPC

Authentication Failures in Workflows

Symptoms: Workflows fail with AWS permission errors

Verify ECS Task Role has required permissions ({repo_org}-task-role)
Confirm the correct IAM policy is attached
Review CloudTrail for AccessDenied events
Confirm the runner is in the correct AWS account

ECR Pull Failures

Symptoms: Tasks fail with CannotPullContainerError

# Verify image exists in private ECR (when enable_ecr_clone = true)
aws ecr describe-images \
  --repository-name github-runners/github-runner \
  --image-ids imageTag=1.67.0

If enable_ecr_clone = false, verify the public ECR image is accessible from the VPC.

High Memory/CPU Usage

Review the github-runners-{aws_account} CloudWatch dashboard
Consider increasing desired_count for more parallel capacity
Optimize workflow jobs (caching, parallelization)

Workflow Jobs Stuck in Queue

Symptoms: Jobs stay in "Waiting for a runner to pick up this job" state

Verify runners are online and idle in GitHub UI
Check runner labels match the runs-on value in the workflow
Increase desired_count if all runners are occupied

Scaling Runners

Via Terraform (Persistent)

Update desired_count in default.auto.tfvars:

desired_count = 5

tf apply

Via AWS CLI (Temporary)

# Scale up
aws ecs update-service \
  --cluster ecs-ghe-runners-us-gov-west-1 \
  --service SCT-Engineering \
  --desired-count 5

# Scale down
aws ecs update-service \
  --cluster ecs-ghe-runners-us-gov-west-1 \
  --service SCT-Engineering \
  --desired-count 2

CLI changes are temporary and will be overwritten by the next tf apply.

Warning: Scaling down terminates runner tasks. Active jobs may be cancelled unless they complete before ECS stops the task.

Emergency Scale to Zero

# Stop all runners immediately
aws ecs update-service \
  --cluster ecs-ghe-runners-us-gov-west-1 \
  --service SCT-Engineering \
  --desired-count 0

# Restore
aws ecs update-service \
  --cluster ecs-ghe-runners-us-gov-west-1 \
  --service SCT-Engineering \
  --desired-count 3

Operations

Refreshing Runners

Use the Terraform taint method when runners stop responding, fail to register, or tokens have expired.

Why Manual GitHub Cleanup is Required First:

Orphaned runner registrations prevent new runners from registering with the same labels
Runner groups that contain offline runners cannot be deleted automatically

Step 1: Clean Up GitHub

Go to: https://github.e.it.census.gov/organizations/SCT-Engineering/settings/actions/runners
For each offline/stale runner: click the runner name → Remove runner → Confirm
Go to: https://github.e.it.census.gov/organizations/SCT-Engineering/settings/actions/runner-groups
Find the runner group named after the AWS account ID → Delete runner group → Confirm

Step 2: Taint Resources and Apply

# Force recreation of the GitHub runner group
tf taint 'module.github-runner.github_actions_runner_group.runner_group'

# Force recreation of the ECS service (all runner tasks restarted)
tf taint 'module.github-runner.module.ecs_service.aws_ecs_service.github_runner'

# Force fresh token retrieval from GitHub
tf taint 'module.github-runner.aws_secretsmanager_secret_version.secret'

# Review and apply
tf plan
tf apply

Step 3: Verify Recovery

# Watch ECS task count recover
watch -n 5 'aws ecs describe-services \
  --cluster ecs-ghe-runners-us-gov-west-1 \
  --services SCT-Engineering \
  --query "services[0].{Desired:desiredCount,Running:runningCount,Pending:pendingCount}"'

# Stream runner registration events
aws logs tail /ecs-ghe-runners/default-229685449397-us-gov-west-1 --follow

Expected Timeline: ~10–15 minutes for full recovery.

Check Runner Status

aws ecs describe-services \
  --cluster ecs-ghe-runners-us-gov-west-1 \
  --services SCT-Engineering \
  --query 'services[0].{Desired:desiredCount,Running:runningCount,Pending:pendingCount,Deployments:deployments[*].{Status:status,Running:runningCount,Desired:desiredCount}}'

Healthy state:

{
  "Desired": 3,
  "Running": 3,
  "Pending": 0,
  "Deployments": [{ "Status": "PRIMARY", "Running": 3, "Desired": 3 }]
}

View Recent Runner Activity

aws logs tail /ecs-ghe-runners/default-229685449397-us-gov-west-1 \
  --since 1h \
  --filter-pattern "Job|Running|Completed|error|fail"

Handling Expired Tokens (Without Lambda)

If enable_lambda_token_refresh = false and the Secrets Manager token has expired:

# Fetch a fresh token (requires valid GITHUB_TOKEN env var)
NEW_TOKEN=$(curl -s -X POST \
  -H "Authorization: token $GITHUB_TOKEN" \
  -H "Accept: application/vnd.github.v3+json" \
  "https://github.e.it.census.gov/api/v3/orgs/SCT-Engineering/actions/runners/registration-token" \
  | jq -r '.token')

# Update the secret (find the secret name in AWS Console or from tf outputs)
aws secretsmanager update-secret \
  --secret-id /github-runners/sct-engineering-229685449397-us-gov-west-1/... \
  --secret-string "$NEW_TOKEN"

Then follow the taint-and-apply procedure above to restart the ECS service.

Handling Stuck ECS Tasks

# Diagnose task stop reason
aws ecs describe-tasks \
  --cluster ecs-ghe-runners-us-gov-west-1 \
  --tasks $(aws ecs list-tasks \
    --cluster ecs-ghe-runners-us-gov-west-1 \
    --query 'taskArns[0]' --output text) \
  --query 'tasks[0].{Status:lastStatus,Reason:stoppedReason,Container:containers[0].reason}'

Common stop reasons:

Reason	Likely Cause
`CannotPullContainerError`	ECR access issue or invalid `image_version`
`ResourceInitializationError`	Network or security group misconfiguration
`TaskFailedToStart`	Task role or execution role permission issue

Disaster Recovery

See RUNBOOK.md for complete incident response procedures covering:

Lambda token refresh failing
Runners at 50% capacity
All runners down (EMERGENCY)

Terraform State Recovery

# Re-import ECS cluster (when create_ecs_cluster = true)
tf import aws_ecs_cluster.github-runner[0] ecs-ghe-runners-us-gov-west-1

# Verify state
tf plan

# Apply drift corrections
tf apply

Maintenance

Updating Runner Version

Update image_version in default.auto.tfvars:
```
image_version = "1.68.0"
```
Apply:
```
tf apply
```

When enable_ecr_clone = true, the new image is automatically mirrored to private ECR during apply. ECS performs a rolling update, replacing one task at a time.

Updating Configuration

All configuration changes must be made via Terraform:

Modify variables in default.auto.tfvars
tf plan — preview changes
tf apply — apply changes

Never modify resources directly in the AWS Console — changes will be overwritten by the next tf apply.

Terraform State Backend

Property	Value
Backend type	S3 (configured in `backend.tf` and `backend-configs/`)
State locking	DynamoDB
Workspace isolation	One state file per workspace

Security Best Practices

Use Private Subnets: Deploy runners without public IPs
Never Commit GITHUB_TOKEN: Always set via environment variable, never in .tfvars
Minimize IAM Permissions: Replace the default admin task role policy with least-privilege permissions
Rotate GitHub PAT: Periodically rotate the PAT and redeploy (tf apply) to update Lambda env vars
Enable VPC Endpoints: Set create_vpc_endpoint = true to reduce internet egress
Respond to Alarms: Act on SNS alerting emails promptly to prevent runner outages
Update Runner Images: Keep image_version current
Restrict Security Groups: Allow only necessary outbound traffic

Cost Optimization

Fargate Pricing Factors:

CPU and memory allocation per task
Continuous run duration (runners are 24/7)
Number of concurrent tasks

Optimization Strategies:

Set desired_count to match actual workload (start at 1 for low-traffic accounts)
Right-size task CPU/memory using CloudWatch utilization metrics from the dashboard
Enable VPC endpoints (create_vpc_endpoint = true) to reduce data transfer costs
Use workflow job timeouts to prevent runaway jobs
Use caching strategies in workflows to reduce execution time per job

Related Documentation

Emergency Runbook — Incident response procedures
Monitoring Implementation Plan — Monitoring design details
Monitoring Deployment Summary — Deployed monitoring resources
AWS Permissions Documentation — Complete IAM requirements and examples
Security Review Guide — Summary for security team reviews and approvals
GitHub App Setup — Optional GitHub App authentication configuration
GitHub App Migration Guide — Migrating from PAT to GitHub App
AWS ECS Fargate Documentation
GitHub Actions Self-Hosted Runners

Support

For assistance:

Infrastructure Issues: Contact the infrastructure team
Workflow Issues: Review workflow-specific documentation
GitHub Issues: Open an issue in this repository
AWS Issues: Check CloudWatch logs, alarms, and the ECS console

Contributing

Fork the repository
Create a feature branch (git checkout -b feature/improvement)
Make your changes
Test in a dev environment using tf plan / tf apply
Submit a pull request with a detailed description
Ensure Terraform formatting (tf fmt)
Update documentation as needed

License

Internal use only — Census Bureau CSVD organization.

CSVD/ghe-runners

About

Topics

Resources

Security policy

Stars

Watchers

Forks

Releases

Contributors 3

Languages