This repository uses GitHub App authentication instead of Personal Access Tokens. The GitHub App private key (PEM file) provides administrative access to your GitHub organization. Improper handling can lead to:

• Unauthorized access to private repositories
• Ability to modify organization settings
• Creation/deletion of runners and runner groups
• Access to organization secrets

✅ DO:

• Store PEM files with restrictive permissions (chmod 600)
• Keep PEM files outside of version control
• Use absolute paths in Terraform variables
• Store in a secure directory (e.g., ~/.github-apps/)
• Use AWS Secrets Manager for Lambda production deployments
• Rotate keys every 6-12 months

❌ DO NOT:

• Commit PEM files to version control
• Share PEM files via email, Slack, or messaging
• Store PEM files in shared network drives
• Use the same key across multiple environments
• Give PEM files world-readable permissions

# Create secure directory
mkdir -p ~/.github-apps
chmod 700 ~/.github-apps

# Move downloaded PEM file
mv ~/Downloads/your-app.*.private-key.pem ~/.github-apps/runner-mgmt.pem

# Set restrictive permissions
chmod 600 ~/.github-apps/runner-mgmt.pem

# Verify permissions
ls -la ~/.github-apps/
# Should show: -rw------- (only owner can read/write)

GitHub App authentication requires three values:

1. App ID - Can be stored in .tfvars (not sensitive)
2. Installation ID - Can be stored in .tfvars (not sensitive)
3. PEM file path - Reference to secure file location

# csvd-229685449397-us-gov-east-1.auto.tfvars
github_app_id              = "123456"                              # Safe to commit
github_app_installation_id = "12345678"                            # Safe to commit
github_app_pem_file        = "~/.github-apps/runner-mgmt.pem"      # Path only, safe to commit

# Note: The actual PEM file content is never in version control

# Set environment variables
export TF_VAR_github_app_id="123456"
export TF_VAR_github_app_installation_id="12345678"
export TF_VAR_github_app_pem_file="~/.github-apps/runner-mgmt.pem"

# Run Terraform
terraform plan
terraform apply

# Unset when done (optional, as these are not sensitive)
unset TF_VAR_github_app_id
unset TF_VAR_github_app_installation_id
unset TF_VAR_github_app_pem_file

GitHub Actions:

- name: Terraform Apply
  env:
    TF_VAR_github_app_id: ${{ vars.GITHUB_APP_ID }}
    TF_VAR_github_app_installation_id: ${{ vars.GITHUB_APP_INSTALLATION_ID }}
    TF_VAR_github_app_pem_file: /tmp/app-key.pem
  run: |
    # Retrieve PEM from secrets and write to file
    echo "${{ secrets.GITHUB_APP_PEM }}" > /tmp/app-key.pem
    chmod 600 /tmp/app-key.pem
    
    terraform apply -auto-approve
    
    # Clean up
    rm /tmp/app-key.pem

AWS CodeBuild:

env:
  variables:
    TF_VAR_github_app_id: "123456"
    TF_VAR_github_app_installation_id: "12345678"
  parameter-store:
    TF_VAR_github_app_pem_file: /github-apps/runner-management/pem-path
  secrets-manager:
    GITHUB_APP_PEM: github-apps/runner-management:pem-content

DO NOT commit PEM files:

# ❌ NEVER include PEM file content in Terraform
variable "github_app_pem_file" {
  default = <<-EOT
    -----BEGIN RSA PRIVATE KEY-----
    MIIEpAIBAAKCAQEA...  # NEVER DO THIS!
    -----END RSA PRIVATE KEY-----
  EOT
}

# ❌ NEVER commit PEM files
# runner-mgmt.pem          # NEVER DO THIS!
# *.private-key.pem        # NEVER DO THIS!

Ensure your .gitignore includes:

# GitHub App Private Keys
*.pem
*.private-key.pem
.github-apps/

# Terraform
.terraform/
.terraform.lock.hcl
terraform.tfstate
terraform.tfstate.backup

# Lambda Artifacts
lambda/package/
lambda/*.zip

# Sensitive files (if any)
*secret*.tfvars
*credentials*.tfvars

Your GitHub App must have these permissions:

Repository Permissions:

• Administration: Read & write (for managing self-hosted runners)
• Actions: Read & write (for generating registration tokens)

Organization Permissions:

• Self-hosted runners: Read & write

See GITHUB_APP_SETUP.md for complete GitHub App creation and configuration instructions.

The Lambda function authenticates using GitHub App credentials and:

1. Call GitHub API to generate fresh registration tokens
2. Update AWS Secrets Manager with new registration tokens

Lambda Security Features:

• Environment variables encrypted at rest by AWS
• IAM role restricts Secrets Manager access to specific secrets
• CloudWatch logs do NOT log the GitHub token (only API responses)
• Function executes in VPC (if configured)

Lambda IAM Permissions:

{
  "Effect": "Allow",
  "Action": [
    "secretsmanager:UpdateSecret",
    "secretsmanager:GetSecretValue",
    "secretsmanager:PutSecretValue"
  ],
  "Resource": "arn:aws:secretsmanager:region:account:secret:/github-runners/*"
}

Terraform authenticates to AWS using one of:

1. IAM Role (Recommended for EC2/ECS/Lambda)

   # No configuration needed - automatic

2. AWS CLI Profile

   export AWS_PROFILE=your-profile
   terraform apply

3. Environment Variables

   export AWS_ACCESS_KEY_ID="your-key"
   export AWS_SECRET_ACCESS_KEY="your-secret"
   export AWS_SESSION_TOKEN="your-token"  # If using temporary credentials
   terraform apply

Security Best Practices:

• ✅ Use IAM roles whenever possible
• ✅ Use temporary credentials (STS) for human access
• ✅ Enable MFA for AWS CLI access
• ❌ Never commit AWS credentials to version control

Runner registration tokens are stored in AWS Secrets Manager:

Secret Path: /github-runners/{namespace}/{hostname}-{random_id}

Access Control:

• Only ECS tasks with specific IAM role can read
• Lambda function can read and update
• Encrypted at rest using AWS KMS
• Encrypted in transit using TLS

Secret Rotation:

• Lambda automatically refreshes registration tokens every 30 minutes
• No manual intervention required
• Old tokens are overwritten (no retention needed)

Monitor for security events:

# Lambda token refresh logs
aws logs tail /aws/lambda/github-runner-token-refresh-{account} --follow

# ECS runner logs (check for auth failures)
aws logs tail /ecs-ghe-runners/{workspace}-{account-id}-{region} \
  --filter-pattern "401\|403\|authentication\|unauthorized"

Monitor AWS API calls:

# Check Secrets Manager access
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceType,AttributeValue=AWS::SecretsManager::Secret \
  --max-results 50

# Check Lambda invocations
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceType,AttributeValue=AWS::Lambda::Function \
  --max-results 50

Monitor GitHub organization activity:

1. Navigate to Organization Settings → Audit log
2. Filter for:
   • Runner registration/removal events
   • OAuth application usage
   • Token generation events

Immediate Actions:

1. Revoke the compromised token in GitHub settings
2. Generate a new token
3. Update CI/CD secrets or environment variables
4. Run terraform apply with new token
5. Review GitHub audit logs for unauthorized activity
6. Check ECS runner logs for suspicious job executions

Immediate Actions:

1. Rotate AWS access keys immediately
2. Review CloudTrail logs for unauthorized API calls
3. Check for unauthorized resource creation
4. Update IAM policies if needed
5. Consider using AWS GuardDuty for threat detection

Risk Level: Low (tokens expire in ~1 hour)

Actions:

1. Wait for token to expire naturally
2. Lambda will refresh with new token automatically
3. Review runner logs for unauthorized registrations
4. Check GitHub for unexpected runners

• GitHub PAT: Highly Sensitive - Treat as password
• AWS Credentials: Highly Sensitive - Treat as password
• Registration Token: Sensitive - Short-lived (1 hour)
• Runner Logs: May contain sensitive build artifacts

• CloudWatch Logs: 7 days (Lambda), 90 days (ECS runners)
• Secrets Manager: Current version only
• Terraform State: Stored in S3 with versioning enabled

• ✅ GitHub tokens: Encrypted in Lambda environment variables (AWS KMS)
• ✅ Registration tokens: Encrypted in Secrets Manager (AWS KMS)
• ✅ Terraform state: Encrypted in S3 (SSE-S3 or SSE-KMS)
• ✅ CloudWatch logs: Encrypted at rest
• ✅ Data in transit: TLS 1.2+ for all communications

For security concerns or questions:

1. Review this document
2. Check AWS Security Best Practices
3. Review GitHub Security documentation
4. Contact the infrastructure security team