-
Notifications
You must be signed in to change notification settings - Fork 12
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
add proposal doc for backing up ghe to s3 (#344)
- Loading branch information
Showing
1 changed file
with
139 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,139 @@ | ||
| # GitHub Enterprise Backup to S3 | ||
|
|
||
| This proposal outlines a plan to safely export all GitHub Enterprise content (code repositories, issues, wikis, releases, projects, etc.) to AWS S3 storage in the event of a lapse in funding for our GitHub Enterprise licenses. | ||
|
|
||
| ## Why | ||
|
|
||
| During a government funding lapse, as documented in [aws/documentation/gov-funding-lapse/README.md](../documentation/gov-funding-lapse/README.md), there's a risk that the Census Bureau may be unable to maintain payments for GitHub Enterprise licenses. If access to GitHub Enterprise is suspended due to non-payment, all code repositories, issues, wikis, releases, projects, and other critical development assets could become temporarily inaccessible, severely impacting development activities and knowledge management once funding is restored. | ||
|
|
||
| Creating a backup on AWS S3 provides a safety net that ensures: | ||
| 1. Continuity of access to critical code and documentation during funding lapses | ||
| 2. Protection against data loss or temporary service suspensions | ||
| 3. Ability to continue some development activities in an alternative environment if necessary | ||
| 4. Quick restoration of normal operations after funding is restored | ||
|
|
||
| ## What | ||
|
|
||
| We propose implementing a backup process that will: | ||
|
|
||
| 1. Export the following GitHub Enterprise assets to S3: | ||
| * Git repositories (code, commits, branches, tags) | ||
| * Issues and pull requests (including comments and metadata) | ||
| * Wikis | ||
| * Releases and assets | ||
| * Project boards | ||
| * GitHub Pages content | ||
| * Organization and team metadata | ||
|
|
||
| 2. Use AWS services to: | ||
| * Store exports in a secure S3 bucket with versioning enabled | ||
| * Implement lifecycle policies to manage storage costs | ||
| * Encrypt data at rest using AWS KMS | ||
| * Maintain access logs for all backup operations | ||
|
|
||
| 3. Create a documentation and recovery process for: | ||
| * Accessing backed-up content during GitHub outages | ||
| * Restoring repositories if needed | ||
| * Continuing development using the backups during an extended outage | ||
|
|
||
| ## Where | ||
|
|
||
| The backup infrastructure will be deployed in AWS with the following components: | ||
|
|
||
| 1. Primary storage bucket: | ||
| * Region: us-gov-east-1 (GovCloud East) | ||
| * Bucket naming format: `v-s3-ghe-backup-{org}-{env}` | ||
|
|
||
| 2. Redundant backup bucket (optional): | ||
| * Region: us-gov-west-1 (GovCloud West) | ||
| * Cross-region replication from primary bucket | ||
|
|
||
| 3. AWS accounts: | ||
| * Primary: ent-gov-operations-prod | ||
| * Potential backup hosting in lab-gov-operations-nonprod for dev/testing | ||
|
|
||
| ## When | ||
|
|
||
| This capability is needed within the next 30 days to prepare for any potential funding lapses that might occur in the next fiscal year. The implementation timeline includes: | ||
|
|
||
| 1. Week 1: Design and approval of backup architecture | ||
| 2. Week 1-2: Development of backup scripts and infrastructure | ||
| 3. Week 2: Testing in lab environment | ||
| 4. Week 2: Documentation and production deployment | ||
| 5. Week 3-4: Validation and team training | ||
|
|
||
| ## Who | ||
|
|
||
| This initiative will be led by the Secure Cloud Team (SCT) with support from: | ||
|
|
||
| 1. Primary stakeholders: | ||
| * GitHub Enterprise administrators (ADSD) | ||
| * Cloud Infrastructure team (SCT/CSVD) | ||
| * Security operations team (CSVD Sec/OIS) | ||
| * Development team leads who depend on GitHub (SCT/CSVD/ADSD) | ||
|
|
||
| 2. Users: | ||
| * All development teams currently using GitHub Enterprise | ||
| * Operations personnel who may need to restore content during outages | ||
| * Security teams who need to maintain audit trails | ||
|
|
||
| ## How | ||
|
|
||
| The backup process will be implemented using the following approach: | ||
|
|
||
| 1. Automation scripts: | ||
| ```python | ||
| # Example GitHub backup script pseudocode | ||
| from datetime import datetime, timezone | ||
| import github | ||
| import boto3 | ||
|
|
||
| timestamp = datetime.now(timezone.utc).isoformat() | ||
| s3_client = boto3.client('s3') | ||
|
|
||
| # Connect to GitHub using token or app authentication | ||
| gh = github.Github(auth_token) | ||
|
|
||
| # For each org in our GitHub Enterprise | ||
| for org in gh.get_organizations(): | ||
| # Export repositories, issues, wikis, etc. | ||
| # Upload to S3 with appropriate prefixes | ||
| ``` | ||
|
|
||
| 2. Infrastructure as Code: | ||
| * AWS S3 bucket creation via Terraform | ||
| * IAM roles and policies for secure access | ||
|
|
||
| 3. Monitoring and auditing: | ||
| * CloudWatch alarms for backup failures | ||
| * S3 access logging | ||
|
|
||
| 4. Recovery process: | ||
| * Documentation for accessing backups | ||
| * Scripts for bulk restoration if needed | ||
|
|
||
| ## Tagging | ||
|
|
||
| In accordance with the [government funding lapse guidelines](../documentation/gov-funding-lapse/README.md), we'll use the following tags: | ||
|
|
||
| * `gfl_shutdown_excluded`: `true` - These backups should continue even during funding lapses | ||
| * `backup_source`: `github_enterprise` | ||
| * `data_classification`: `internal` | ||
|
|
||
| ## Links | ||
|
|
||
| Product Documentation Links: | ||
| * [GitHub Enterprise API](https://docs.github.com/en/enterprise-server/rest) | ||
| * [AWS S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/what-is-s3.html) | ||
| * [AWS S3 Cross-Region Replication](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html) | ||
| * [GitHub Enterprise Backup Utilities](https://github.com/github/backup-utils) | ||
|
|
||
| Internal Links: | ||
| * [Government Funding Lapse Plans](../documentation/gov-funding-lapse/README.md) | ||
| * [Cloud Architecture Outline](../documentation/architecture-outline.md) | ||
| * [AWS Account Naming](../documentation/account-setup/README.md) | ||
|
|
||
| # CHANGELOG | ||
|
|
||
| - 1.0.0 -- 2024-06-15 | ||
| - initial proposal |