From 02c4151dd1d8ba759dbb6c248e58df1b0de686c4 Mon Sep 17 00:00:00 2001 From: Delong Yang Date: Thu, 10 Apr 2025 10:54:49 -0400 Subject: [PATCH] Rewrite README.md, fix wrong API URL output --- .pre-commit-config.yaml | 1 - README.md | 150 ++++++++++++++++++++++++++++------------ template.yaml | 4 +- 3 files changed, 106 insertions(+), 49 deletions(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 5d4601c..4d07f26 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -5,7 +5,6 @@ repos: hooks: - id: trailing-whitespace - id: end-of-file-fixer - # - id: check-yaml - id: check-added-large-files - repo: https://github.com/psf/black rev: 24.4.0 diff --git a/README.md b/README.md index 166de79..cb08bf7 100644 --- a/README.md +++ b/README.md @@ -2,72 +2,130 @@ ## Description -This repository hosts `eks-automation-lambda` automation work at `Census`. +This repository contains source code and supporting files for a serverless application that you can deploy with the SAM CLI. +The application uses a Lambda function to process JSON input data and create a new GitHub repo for **Census EKS CI/CD pipeline**. -## Dependencies -- `Anaconda` or a `bare bone Python 3` - create a `Remedy ticket` to have it available in `software center` -- `virtualenv` - must be installed outside of the `requirements.txt` install -- requirements.txt -- pre-commit hook +## Getting Started -## Project Setup +First of all, you need access to an AWS account with adequate permission to which the resources will be deployed. +You also need to create an [`AWS CLI` profile](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html#getting-started-quickstart-new). -### Install `virtualenv` +A [GitHub Personal Access Token (PAT)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) +is required to access the Census GitHub Enterprise Server. +The `PAT` must be securely stored in `AWS Systems Manager Parameter Store`. The parameter name must match the value of the +"SECRET_NAME" constant defined in the `eks_automation/app.py` file. -```sh -pip install virtualenv +To access the Census GitHub Enterprise Server, a VPC with private subnets connected to the server must also be attached. +The VPC configuration is set in the `template.yaml` file. Change the `Subnet IDs` and `Security Group IDs` as needed. -# below is the output from a successful install -$ virtualenv --version -virtualenv 20.25.0 from C:\Users\{your username}\AppData\Local\anaconda3\Lib\site-packages\virtualenv\__init__.py -``` +You may also want to adjust other settings (API Usage Plan, tags, etc.) in the `template.yaml` file. -## Create and activate `virtual environment` +## Prerequites -```sh -virtualenv .venv +- git +- python3.11 +- pip +- pre-commit +- AWS CLI +- SAM CLI -# activate env (windows) -.venv/Scripts/activate.ps1 (PowerShell) -.venv/Scripts/activate.bat (Command Prompt) +You may need to submit a support ticket to request the installation of these tools on your laptop. -# activate env (linux) -source .venv/bin/activate +### Installing -# install dependencies -pip3 install -r requirements.txt -# deactivate env -deactivate -``` +- Clone this repository: -### Install pre-commit + ```sh + git clone git@github.e.it.census.gov:SCT-Engineering/eks-automation-lambda.git + ``` -Run the command below to install `pre-commit hooks` listed in the `.pre-commit-config.yaml`. +- After cloning, access the folder and install `pre-commit hooks` listed in the `.pre-commit-config.yaml`: -```sh -pre-commit install -``` + ```sh + cd eks-automation-lambda + pre-commit install + ``` -### Tidy Up (manual linting) +## Deploy/Test the application -`Pre-commit` does this automatically. This script is used to `lint / format python resources manually`. Run `tidy.sh` to `lint` and `format` code. This project uses `pylint` and `black`. Below is an example output from a successful run of this script. +- Create an `AWS S3 bucket`: -```sh --------------------------------------------------------------------- -Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00) + ```sh + aws s3api create-bucket --bucket eks-automation-lambda-s3-bucket \ + --create-bucket-configuration LocationConstraint=us-gov-east-1 \ + --region us-gov-east-1 \ + --profile 229685449397-csvd-dev-gov + ``` -reformatted main.py + The `bucket name` must match the one specified in the `samconfig.toml` file. + Please adjust the profile name and region accordingly. +- Download [`git-lambda-layer`](https://github.com/lambci/git-lambda-layer/blob/master/lambda2/layer.zip) `zip` file. +- Upload `git-lambda-layer` to the newly created `AWS S3 bucket`: -All done! ✨ 🍰 ✨ -1 files reformatted, 1 files left unchanged. -``` + ```sh + aws s3 cp {download-folder}/layer.zip s3://eks-automation-lambda-s3-bucket/ --profile 229685449397-csvd-dev-gov + ``` -## NOTES +- Build the application: -- A GitHub Personal Access Token (PAT) is required to access the Census GitHub Enterprise Server. The PAT should be securely stored in AWS Systems Manager Parameter Store. -- This lambda function relies on [`git-lambda-layer`](https://github.com/lambci/git-lambda-layer), which must be uploaded to the S3 bucket specified in samconfig.toml prior to deployment. -- The REST API for this Lambda function is configured to be accessed using an API key. -- To access the Census GitHub Enterprise Server, a VPC with private subnets connected to the server must be attached. + ```sh + sam build + ``` + +- Deploy the application: + + ```sh + sam deploy --profile 229685449397-csvd-dev-gov + ``` + + Save the `API Gateway endpoint URL` listed in the output. You will need this URL for testing. + +- Test: + + Get the `API Key`: + + ```sh + aws apigateway get-api-keys --query 'items[?contains(name, `eks-`)].value' --include-values --output text --profile 229685449397-csvd-dev-gov + ``` + + ```sh + curl -X POST -H "X-API-Key: {API Key}" https://{API Gateway endpoint URL} -d ' + { + "attrs": { + "account_name": "lab-dev-ew", + "aws_region": "us-gov-east-1", + "cluster_mailing_list": "matthew.c.morgan@census.gov", + "cluster_name": "csvd-platform-lab-mcm", + "eks_instance_disk_size": 100, + "eks_ng_desired_size": 2, + "eks_ng_max_size": 10, + "eks_ng_min_size": 2, + "environment": "development", + "environment_abbr": "dev", + "organization": "census:ocio:csvd", + "finops_project_name": "csvd_platformbaseline", + "finops_project_number": "fs0000000078", + "finops_project_role": "csvd_platformbaseline_app", + "vpc_domain_name": "dev.lab.csp2.census.gov", + "vpc_name": "vpc3-lab-dev" + }, + "tags" : { + "slim:schedule": "8:00-17:00" + } + } + ' + ``` + + Replace `{API Key}` with the result of the last command and `{API Gateway endpoint URL}` with the value saved from the `sam deploy` command output. + +## Resources + +- [AWS Serverless Application Model](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/what-is-sam.html) +- [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) +- [Git Lambda Layer](https://github.com/lambci/git-lambda-layer/) +- [AWS API Gateway](https://docs.aws.amazon.com/apigateway/latest/developerguide/welcome.html) +- [PyGithub](https://pygithub.readthedocs.io/en/stable/introduction.html) +- [GitPython](https://gitpython.readthedocs.io/en/stable/) diff --git a/template.yaml b/template.yaml index 2067247..c022b1d 100644 --- a/template.yaml +++ b/template.yaml @@ -49,7 +49,7 @@ Resources: Properties: LayerName: git-lambda-layer Description: Git Lambda Layer - ContentUri: s3://eks-automation-s3-bucket/layer.zip + ContentUri: s3://eks-automation-lambda-s3-bucket/layer.zip CompatibleRuntimes: - python3.9 - python3.10 @@ -106,7 +106,7 @@ Resources: Outputs: EKSAutomationApi: Description: "API Gateway endpoint URL for Prod stage for EKS Automation function" - Value: !Sub "https://${EKSAutomationApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/automation/" + Value: !Sub "https://${EKSAutomationApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/EKSAutomation/" EKSAutomationFunction: Description: "EKS Automation Lambda Function ARN" Value: !GetAtt EKSAutomationFunction.Arn