From 24b6515833cde21e49bc8509ba78059a20ad7cbb Mon Sep 17 00:00:00 2001 From: "Matthew C. Morgan" Date: Wed, 30 Apr 2025 12:23:31 -0400 Subject: [PATCH] add proposal doc for backing up ghe to s3 --- aws/proposals/github-concerns/README.md | 139 ++++++++++++++++++++++++ 1 file changed, 139 insertions(+) diff --git a/aws/proposals/github-concerns/README.md b/aws/proposals/github-concerns/README.md index e69de29b..6e6cec7e 100644 --- a/aws/proposals/github-concerns/README.md +++ b/aws/proposals/github-concerns/README.md @@ -0,0 +1,139 @@ +# GitHub Enterprise Backup to S3 + +This proposal outlines a plan to safely export all GitHub Enterprise content (code repositories, issues, wikis, releases, projects, etc.) to AWS S3 storage in the event of a lapse in funding for our GitHub Enterprise licenses. + +## Why + +During a government funding lapse, as documented in [aws/documentation/gov-funding-lapse/README.md](../documentation/gov-funding-lapse/README.md), there's a risk that the Census Bureau may be unable to maintain payments for GitHub Enterprise licenses. If access to GitHub Enterprise is suspended due to non-payment, all code repositories, issues, wikis, releases, projects, and other critical development assets could become temporarily inaccessible, severely impacting development activities and knowledge management once funding is restored. + +Creating a backup on AWS S3 provides a safety net that ensures: +1. Continuity of access to critical code and documentation during funding lapses +2. Protection against data loss or temporary service suspensions +3. Ability to continue some development activities in an alternative environment if necessary +4. Quick restoration of normal operations after funding is restored + +## What + +We propose implementing a backup process that will: + +1. Export the following GitHub Enterprise assets to S3: + * Git repositories (code, commits, branches, tags) + * Issues and pull requests (including comments and metadata) + * Wikis + * Releases and assets + * Project boards + * GitHub Pages content + * Organization and team metadata + +2. Use AWS services to: + * Store exports in a secure S3 bucket with versioning enabled + * Implement lifecycle policies to manage storage costs + * Encrypt data at rest using AWS KMS + * Maintain access logs for all backup operations + +3. Create a documentation and recovery process for: + * Accessing backed-up content during GitHub outages + * Restoring repositories if needed + * Continuing development using the backups during an extended outage + +## Where + +The backup infrastructure will be deployed in AWS with the following components: + +1. Primary storage bucket: + * Region: us-gov-east-1 (GovCloud East) + * Bucket naming format: `v-s3-ghe-backup-{org}-{env}` + +2. Redundant backup bucket (optional): + * Region: us-gov-west-1 (GovCloud West) + * Cross-region replication from primary bucket + +3. AWS accounts: + * Primary: ent-gov-operations-prod + * Potential backup hosting in lab-gov-operations-nonprod for dev/testing + +## When + +This capability is needed within the next 30 days to prepare for any potential funding lapses that might occur in the next fiscal year. The implementation timeline includes: + +1. Week 1: Design and approval of backup architecture +2. Week 1-2: Development of backup scripts and infrastructure +3. Week 2: Testing in lab environment +4. Week 2: Documentation and production deployment +5. Week 3-4: Validation and team training + +## Who + +This initiative will be led by the Secure Cloud Team (SCT) with support from: + +1. Primary stakeholders: + * GitHub Enterprise administrators (ADSD) + * Cloud Infrastructure team (SCT/CSVD) + * Security operations team (CSVD Sec/OIS) + * Development team leads who depend on GitHub (SCT/CSVD/ADSD) + +2. Users: + * All development teams currently using GitHub Enterprise + * Operations personnel who may need to restore content during outages + * Security teams who need to maintain audit trails + +## How + +The backup process will be implemented using the following approach: + +1. Automation scripts: + ```python + # Example GitHub backup script pseudocode + from datetime import datetime, timezone + import github + import boto3 + + timestamp = datetime.now(timezone.utc).isoformat() + s3_client = boto3.client('s3') + + # Connect to GitHub using token or app authentication + gh = github.Github(auth_token) + + # For each org in our GitHub Enterprise + for org in gh.get_organizations(): + # Export repositories, issues, wikis, etc. + # Upload to S3 with appropriate prefixes + ``` + +2. Infrastructure as Code: + * AWS S3 bucket creation via Terraform + * IAM roles and policies for secure access + +3. Monitoring and auditing: + * CloudWatch alarms for backup failures + * S3 access logging + +4. Recovery process: + * Documentation for accessing backups + * Scripts for bulk restoration if needed + +## Tagging + +In accordance with the [government funding lapse guidelines](../documentation/gov-funding-lapse/README.md), we'll use the following tags: + +* `gfl_shutdown_excluded`: `true` - These backups should continue even during funding lapses +* `backup_source`: `github_enterprise` +* `data_classification`: `internal` + +## Links + +Product Documentation Links: +* [GitHub Enterprise API](https://docs.github.com/en/enterprise-server/rest) +* [AWS S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/what-is-s3.html) +* [AWS S3 Cross-Region Replication](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html) +* [GitHub Enterprise Backup Utilities](https://github.com/github/backup-utils) + +Internal Links: +* [Government Funding Lapse Plans](../documentation/gov-funding-lapse/README.md) +* [Cloud Architecture Outline](../documentation/architecture-outline.md) +* [AWS Account Naming](../documentation/account-setup/README.md) + +# CHANGELOG + +- 1.0.0 -- 2024-06-15 + - initial proposal