From 0cd519735fe81d19e24da041a534985da82f6a81 Mon Sep 17 00:00:00 2001 From: Your Name Date: Thu, 22 May 2025 12:55:57 -0400 Subject: [PATCH] Add GitHub to GitLab migration plan with AWS CodeBuild integration documentation --- docs/gitlab-migration.md | 436 ++++++++++++++++++++++ template-automation-lambda.code-workspace | 3 + 2 files changed, 439 insertions(+) create mode 100644 docs/gitlab-migration.md diff --git a/docs/gitlab-migration.md b/docs/gitlab-migration.md new file mode 100644 index 0000000..fe0e516 --- /dev/null +++ b/docs/gitlab-migration.md @@ -0,0 +1,436 @@ +# GitHub to GitLab Migration Plan with AWS CodeBuild Integration + +## Overview + +This document outlines the migration plan for transitioning our template automation system from GitHub to GitLab while simultaneously moving CI/CD operations from GitHub Actions to AWS CodeBuild. This migration will affect all repositories in our template automation system. + +> **Note**: A separate team will handle the actual repository migration to GitLab. This plan focuses on adapting our Lambda function to work with GitLab's API and migrating CI/CD pipelines to AWS CodeBuild. + +## Current Architecture + +Our template automation system consists of four interconnected repositories: + +1. **template-eks-cluster**: Template repository for EKS clusters +2. **template-automation-lambda**: Lambda function code that handles template repository creation +3. **terraform-aws-template-automation**: Terraform modules for the automation infrastructure +4. **template-repos-lambda-deployment**: Terraform code that deploys the Lambda function + +The system currently uses GitHub for source control and GitHub Actions for CI/CD. The Lambda function interacts with the GitHub API to create repositories from templates. + +## Migration Goals + +1. ~~Move all repositories from GitHub to GitLab~~ (Being handled by a separate team) +2. Migrate CI/CD from GitHub Actions to AWS CodeBuild +3. Update the Lambda function to interact with GitLab's API instead of GitHub's +4. Ensure consistent and reliable operation after migration + +## Migration Plan + +### ~~Phase 1: Repository Migration (Week 1)~~ (Being handled by a separate team) + +### Phase 1: Lambda Function Adaptation (Week 1-2) + +#### 1.1 Update Lambda Code + +In the `template-automation-lambda` repository: + +- Replace `PyGithub` with `python-gitlab` in `requirements.txt` +- Refactor the GitHub API interaction code in `app.py` and related modules: + - Update authentication mechanism to use GitLab tokens + - Replace GitHub-specific API calls with equivalent GitLab API calls + - Update error handling for GitLab API responses +- Update configuration management for GitLab-specific parameters + +#### 1.2 Key Code Changes Required + +1. **Authentication**: + ```python + # GitHub (current) + from github import Github + g = Github(github_token) + + # GitLab (new) + import gitlab + gl = gitlab.Gitlab(gitlab_url, private_token=gitlab_token) + ``` + +2. **Repository Operations**: + ```python + # GitHub (current) + org = g.get_organization(org_name) + template_repo = org.get_repo(template_repo_name) + new_repo = org.create_repo_from_template( + name=project_name, + template_repository_id=template_repo.id + ) + + # GitLab (new) + group = gl.groups.get(group_id) + # Option 1: Fork from template + template_project = gl.projects.get(template_project_path) + new_project = template_project.forks.create({ + 'name': project_name, + 'namespace_id': group.id + }) + # Option 2: Create new project and populate + new_project = gl.projects.create({ + 'name': project_name, + 'namespace_id': group.id + }) + # Then clone template locally and push to new project + ``` + +3. **File Operations**: + ```python + # GitHub (current) + repo.create_file(path, message, content, branch) + + # GitLab (new) + project.files.create({ + 'file_path': path, + 'branch': branch, + 'content': content, + 'commit_message': message + }) + ``` + +4. **Pull/Merge Requests**: + ```python + # GitHub (current) + repo.create_pull(title, body, base_branch, head_branch) + + # GitLab (new) + project.mergerequests.create({ + 'source_branch': source_branch, + 'target_branch': target_branch, + 'title': title, + 'description': description + }) + ``` + +#### 1.3 Configuration Updates + +Update the Lambda environment variables and SSM parameters in `template-repos-lambda-deployment`: + +- Replace `GITHUB_API_URL` with `GITLAB_API_URL` +- Replace `GITHUB_ORG_NAME` with `GITLAB_GROUP_ID` or similar +- Update token references and secret management + +### Phase 2: CI/CD Migration to AWS CodeBuild (Week 3) + +#### 2.1 Define CodeBuild Projects + +For each repository that uses CI/CD: + +1. Create `buildspec.yml` files for each workflow currently in GitHub Actions +2. Create AWS CodeBuild projects using the AWS Management Console or Terraform +3. Configure necessary IAM roles and permissions for CodeBuild projects + +#### 2.2 GitHub Actions to CodeBuild Conversion + +For the `initialize.yml` GitHub Actions workflow in `template-eks-cluster`: + +```yaml +# buildspec.yml for repository initialization +version: 0.2 + +env: + variables: + CONFIG_PATH: "config.json" + parameter-store: + # Add any parameters needed from SSM + +phases: + install: + runtime-versions: + python: 3.10 + commands: + - echo Setting up environment... + - pip install --upgrade pip + - pip install ansible + - if [ -f requirements.txt ]; then pip install -r requirements.txt; fi + + pre_build: + commands: + - echo Verifying config file exists... + - if [ ! -f "$CONFIG_PATH" ]; then echo "Error: Config file not found!"; exit 1; fi + - cat "$CONFIG_PATH" + + build: + commands: + - echo Running Ansible playbook... + - ansible-playbook ansible/generate_hcl_files.yml -e "config_file=$CONFIG_PATH" + + post_build: + commands: + - echo Configuration complete + - git config --global user.email "codebuild@example.com" + - git config --global user.name "AWS CodeBuild" + - git add -A + - git diff --staged --quiet || git commit -m "Initialize repository structure from template" + - git push origin HEAD:repo-init || true + +artifacts: + files: + - '**/*' +``` + +#### 2.3 AWS CodeBuild Triggers + +Set up triggers for CodeBuild projects: + +1. **Option 1 - GitLab Webhooks**: + - Configure GitLab webhooks to trigger AWS Lambda functions + - Lambda functions invoke CodeBuild projects via AWS SDK + +2. **Option 2 - AWS CodeStar**: + - Use AWS CodeStar connections to connect to GitLab + - Configure CodeBuild to monitor GitLab repositories directly + +3. **Option 3 - Custom Integration**: + - Use AWS EventBridge to orchestrate workflow + - Create Lambda functions to poll GitLab API for changes + +#### 2.4 Update Makefile and Ansible Templates + +Update the `Makefile.j2` template to work with CodeBuild instead of GitHub Actions: + +```makefile +# ...existing code... +deploy-to-pipeline: + # ...existing code... + # Replace GitHub Actions references with CodeBuild + aws codebuild start-build --project-name $(CLUSTER_NAME)-init --source-version $(BRANCH) --region $(AWS_REGION) --profile $(AWS_PROFILE) + # ...existing code... +``` + +### Phase 3: Testing and Validation (Week 4) + +#### 3.1 Test Repository Creation Flow + +1. Test the updated Lambda function with GitLab integration +2. Verify template repository cloning works correctly +3. Confirm file creation, commits, and merge requests function as expected +4. Validate that the end-to-end flow creates properly configured EKS cluster repositories + +#### 3.2 Test CI/CD Pipeline + +1. Test CodeBuild projects with various triggers +2. Verify build and deployment processes complete successfully +3. Validate that Ansible automation still functions correctly +4. Test error scenarios and failure recovery + +#### 3.3 Performance and Security Validation + +1. Compare performance metrics between GitHub and GitLab APIs +2. Validate IAM permissions and security configurations +3. Ensure secrets and tokens are properly secured +4. Verify all access controls are correctly implemented + +### Phase 4: Cutover and Documentation (Week 5) + +#### 4.1 Production Cutover + +1. Freeze changes to GitHub repositories +2. Perform final synchronization to GitLab +3. Update all DNS entries, links, and references to point to GitLab +4. Activate GitLab webhooks and AWS CodeBuild triggers +5. Begin using GitLab as the primary source code repository + +#### 4.2 Documentation Updates + +1. Update all README files with GitLab URLs and CodeBuild references +2. Create internal documentation for the new workflow +3. Update diagrams and architecture documents +4. Provide training materials for team members + +#### 4.3 Monitoring and Support + +1. Implement monitoring for GitLab API interactions +2. Set up alerting for CodeBuild job failures +3. Establish support process for issues during transition period +4. Schedule post-migration review after 2 weeks + +## Resource Requirements + +- **Personnel**: + - DevOps engineers (2) + - Software developers (2) + - Project manager (1) + +- **Tools**: + - GitLab API access + - AWS CodeBuild projects + - AWS IAM roles and policies + - GitLab API tokens + - AWS Lambda resources + +## Rollback Plan + +If critical issues are encountered during migration: + +1. Keep GitHub repositories active until migration is complete +2. Maintain dual-write capability during testing phase +3. Prepare scripts to restore GitHub Actions workflows if needed +4. Document specific rollback procedures for each component + +## Timeline + +- **Week 1-2**: Lambda Function Adaptation +- **Week 3**: CI/CD Migration to AWS CodeBuild +- **Week 4**: Testing and Validation +- **Week 5**: Cutover and Documentation + +## Jira Tasks Breakdown + +### Epic: GitHub to GitLab Migration with AWS CodeBuild Integration + +> **Note**: Repository migration tasks (MIGRATE-1 through MIGRATE-7) are being handled by a separate team and are therefore excluded from this plan. + +#### Phase 1: Lambda Function Adaptation + +1. **MIGRATE-8**: Update Lambda Dependencies + - Replace PyGithub with python-gitlab in requirements.txt + - Update Dockerfile if needed + - Estimate: 2 hours + - Assignee: Developer + +2. **MIGRATE-9**: Refactor Authentication Code + - Update token retrieval and authentication logic + - Update error handling for GitLab API + - Estimate: 4 hours + - Assignee: Developer + +3. **MIGRATE-10**: Refactor Repository Operations + - Update repository creation, cloning, and templating logic + - Test with GitLab API + - Estimate: 1 day + - Assignee: Developer + +4. **MIGRATE-11**: Refactor File Operations + - Update file creation, modification, and commit logic + - Test with GitLab API + - Estimate: 4 hours + - Assignee: Developer + +5. **MIGRATE-12**: Refactor Merge Request Logic + - Update pull request creation to use GitLab merge requests + - Test with GitLab API + - Estimate: 4 hours + - Assignee: Developer + +6. **MIGRATE-13**: Update Lambda Configuration + - Update environment variables in template-repos-lambda-deployment + - Update SSM parameters + - Estimate: 2 hours + - Assignee: Developer + +7. **MIGRATE-14**: Update Lambda IAM Permissions + - Review and update IAM permissions for GitLab integration + - Estimate: 2 hours + - Assignee: Developer + +#### Phase 2: CI/CD Migration to AWS CodeBuild + +8. **MIGRATE-15**: Create CodeBuild IAM Roles + - Define IAM roles for CodeBuild projects + - Set up permissions for GitLab integration + - Estimate: 4 hours + - Assignee: Developer + +9. **MIGRATE-16**: Convert Initialize Workflow to CodeBuild + - Create buildspec.yml for repository initialization + - Set up CodeBuild project for template-eks-cluster + - Test workflow conversion + - Estimate: 6 hours + - Assignee: Developer + +10. **MIGRATE-17**: Implement GitLab to CodeBuild Triggers + - Set up webhook mechanism from GitLab to AWS + - Configure trigger Lambda function if needed + - Estimate: 1 day + - Assignee: Developer + +11. **MIGRATE-18**: Update Makefile and Ansible Templates + - Modify Makefile.j2 to work with CodeBuild + - Update Ansible templates if needed + - Estimate: 4 hours + - Assignee: Developer + +12. **MIGRATE-19**: Create Pipeline Infrastructure as Code + - Develop Terraform configuration for CodeBuild/CodePipeline + - Integrate with existing template-repos-lambda-deployment + - Estimate: 1 day + - Assignee: Developer + +#### Phase 3: Testing and Validation + +13. **MIGRATE-20**: Unit Test Lambda Function + - Create/update unit tests for GitLab integration + - Run tests against GitLab development instance + - Estimate: 6 hours + - Assignee: Developer + +14. **MIGRATE-21**: Integration Test End-to-End Flow + - Test template repository creation flow + - Verify configuration and initialization works + - Estimate: 1 day + - Assignee: Developer + +15. **MIGRATE-22**: Test CodeBuild Pipelines + - Verify CodeBuild triggered by GitLab changes + - Test various scenarios and triggers + - Estimate: 4 hours + - Assignee: Developer + +16. **MIGRATE-23**: Security Validation + - Review IAM permissions and security configurations + - Verify token and secret management + - Estimate: 4 hours + - Assignee: Developer + +#### Phase 4: Cutover and Documentation + +17. **MIGRATE-24**: Finalize Documentation Updates + - Update all README files with GitLab references + - Create internal documentation for new workflow + - Update architecture diagrams + - Estimate: 1 day + - Assignee: Developer + +18. **MIGRATE-25**: Production Cutover Planning + - Develop detailed cutover plan + - Schedule cutover window + - Prepare announcement and communication + - Estimate: 4 hours + - Assignee: Developer + +19. **MIGRATE-26**: Execute Production Cutover + - Coordinate with repository migration team + - Update Lambda configurations + - Switch to CodeBuild pipelines + - Estimate: 4 hours + - Assignee: Developer + +20. **MIGRATE-27**: Post-Migration Support + - Monitor system for issues + - Provide support for user questions + - Resolve any migration issues + - Estimate: 1 day + - Assignee: Developer + +21. **MIGRATE-28**: Post-Migration Review + - Conduct review meeting + - Document lessons learned + - Close out migration project + - Estimate: 2 hours + - Assignee: Developer + +## Conclusion + +This migration plan provides a structured approach to adapting our Lambda function to work with GitLab and migrating CI/CD operations to AWS CodeBuild. While the repository migration is being handled by a separate team, our focus remains on ensuring the Lambda function and CI/CD pipelines continue to operate reliably with the new GitLab infrastructure. + +The most critical aspects of this migration are: +1. Adapting the Lambda function to work with GitLab's API +2. Ensuring CodeBuild projects correctly replace GitHub Actions workflows +3. Maintaining proper security configurations throughout the migration \ No newline at end of file diff --git a/template-automation-lambda.code-workspace b/template-automation-lambda.code-workspace index 0940f3e..05373c7 100644 --- a/template-automation-lambda.code-workspace +++ b/template-automation-lambda.code-workspace @@ -15,6 +15,9 @@ { "name": "template-repos-lambda-deployment", "path": "../template-repos-lambda-deployment" + }, + { + "path": "../module-decomposition" } ], "settings": {}