Skip to content

add proposal doc for backing up ghe to s3 #344

Merged
merged 1 commit into from
May 2, 2025
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
139 changes: 139 additions & 0 deletions aws/proposals/github-concerns/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
# GitHub Enterprise Backup to S3

This proposal outlines a plan to safely export all GitHub Enterprise content (code repositories, issues, wikis, releases, projects, etc.) to AWS S3 storage in the event of a lapse in funding for our GitHub Enterprise licenses.

## Why

During a government funding lapse, as documented in [aws/documentation/gov-funding-lapse/README.md](../documentation/gov-funding-lapse/README.md), there's a risk that the Census Bureau may be unable to maintain payments for GitHub Enterprise licenses. If access to GitHub Enterprise is suspended due to non-payment, all code repositories, issues, wikis, releases, projects, and other critical development assets could become temporarily inaccessible, severely impacting development activities and knowledge management once funding is restored.

Creating a backup on AWS S3 provides a safety net that ensures:
1. Continuity of access to critical code and documentation during funding lapses
2. Protection against data loss or temporary service suspensions
3. Ability to continue some development activities in an alternative environment if necessary
4. Quick restoration of normal operations after funding is restored

## What

We propose implementing a backup process that will:

1. Export the following GitHub Enterprise assets to S3:
* Git repositories (code, commits, branches, tags)
* Issues and pull requests (including comments and metadata)
* Wikis
* Releases and assets
* Project boards
* GitHub Pages content
* Organization and team metadata

2. Use AWS services to:
* Store exports in a secure S3 bucket with versioning enabled
* Implement lifecycle policies to manage storage costs
* Encrypt data at rest using AWS KMS
* Maintain access logs for all backup operations

3. Create a documentation and recovery process for:
* Accessing backed-up content during GitHub outages
* Restoring repositories if needed
* Continuing development using the backups during an extended outage

## Where

The backup infrastructure will be deployed in AWS with the following components:

1. Primary storage bucket:
* Region: us-gov-east-1 (GovCloud East)
* Bucket naming format: `v-s3-ghe-backup-{org}-{env}`

2. Redundant backup bucket (optional):
* Region: us-gov-west-1 (GovCloud West)
* Cross-region replication from primary bucket

3. AWS accounts:
* Primary: ent-gov-operations-prod
* Potential backup hosting in lab-gov-operations-nonprod for dev/testing

## When

This capability is needed within the next 30 days to prepare for any potential funding lapses that might occur in the next fiscal year. The implementation timeline includes:

1. Week 1: Design and approval of backup architecture
2. Week 1-2: Development of backup scripts and infrastructure
3. Week 2: Testing in lab environment
4. Week 2: Documentation and production deployment
5. Week 3-4: Validation and team training

## Who

This initiative will be led by the Secure Cloud Team (SCT) with support from:

1. Primary stakeholders:
* GitHub Enterprise administrators (ADSD)
* Cloud Infrastructure team (SCT/CSVD)
* Security operations team (CSVD Sec/OIS)
* Development team leads who depend on GitHub (SCT/CSVD/ADSD)

2. Users:
* All development teams currently using GitHub Enterprise
* Operations personnel who may need to restore content during outages
* Security teams who need to maintain audit trails

## How

The backup process will be implemented using the following approach:

1. Automation scripts:
```python
# Example GitHub backup script pseudocode
from datetime import datetime, timezone
import github
import boto3

timestamp = datetime.now(timezone.utc).isoformat()
s3_client = boto3.client('s3')

# Connect to GitHub using token or app authentication
gh = github.Github(auth_token)

# For each org in our GitHub Enterprise
for org in gh.get_organizations():
# Export repositories, issues, wikis, etc.
# Upload to S3 with appropriate prefixes
```

2. Infrastructure as Code:
* AWS S3 bucket creation via Terraform
* IAM roles and policies for secure access

3. Monitoring and auditing:
* CloudWatch alarms for backup failures
* S3 access logging

4. Recovery process:
* Documentation for accessing backups
* Scripts for bulk restoration if needed

## Tagging

In accordance with the [government funding lapse guidelines](../documentation/gov-funding-lapse/README.md), we'll use the following tags:

* `gfl_shutdown_excluded`: `true` - These backups should continue even during funding lapses
* `backup_source`: `github_enterprise`
* `data_classification`: `internal`

## Links

Product Documentation Links:
* [GitHub Enterprise API](https://docs.github.com/en/enterprise-server/rest)
* [AWS S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/what-is-s3.html)
* [AWS S3 Cross-Region Replication](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html)
* [GitHub Enterprise Backup Utilities](https://github.com/github/backup-utils)

Internal Links:
* [Government Funding Lapse Plans](../documentation/gov-funding-lapse/README.md)
* [Cloud Architecture Outline](../documentation/architecture-outline.md)
* [AWS Account Naming](../documentation/account-setup/README.md)

# CHANGELOG

- 1.0.0 -- 2024-06-15
- initial proposal