Skip to content

Commit

Permalink
[Major] Code Freeze PR (#23)
Browse files Browse the repository at this point in the history
* yep

* set back to normal

* missed tempo

* change branch ref to test provider-resolution

* fix min vals

* 2 is the lowest

* docs and keycloak

* use default for eks again

* tempo and kiali updates while working on keycloak

* almost

* no v

* cleanup

* namespaces

* use main

* fmt

* namespace changes

* update internal url ref

* fmt

* versions

* more wip:

* keycloak wip

* update prom internal url input value

* test changes on prom

* deleted old cluster platform-eng-eks-test and created new cluster platform-eng-eks-srn

* testing more autoscaling stuffs

* wip

* wip

* use my eks

* otel added

* ordering

* test branch

* removed a few folders from workspace

* reset branches to default

* missed one

* fmt

* use client id and secret

* fix service name regex violation

* updates

* update from lukes pr

* disable gatekeeper

* updated

* update eks module renaming

* updated vars

* revert dashboard stuff

* update inputs

* fix profile

* latest

* updated eks module version and sorted namespaces

* update to pass account and regino

* back to functional

* initial arcgis module

* fix ordering

* add skip modules ability

* add enterprise ecr stuff

* fmt

* wip

* cross account pulls

* updating readme (#22)

* updating readme

* Add initial README.md with project overview, structure, and usage instructions

* Revise README.md for clarity and organization, enhancing descriptions of repository contents and usage instructions.

* full cluster

* clean up ws

* fix profile and update versions

* update ordering is istio is after otel

* match the version in eecr

* add otel

* add image versions passthrough

* fmt

* move eecr data item to provider

* pass ecr_info

* dupe

* add gatekeeper

* add gatekeeper

* add gatekeeper to workspace

* update gatekeeper stuff

* fmt

* add keycloak_public_url

* add discovery_url

* fix enabled_modules ref

* fix discovery_url source

* add mock value to support run-all destroy after keycloak has been destroyed

* update input_vars.hcl contents

* use the right gatekeeper

* fmt

* add release versions map

* add module_versions and dynamic lookup for release_version

* rename release date

* update karpenter version, fmt

* fix(eks-gatekeeper): update inputs after module change

* wip

* update versions after merges

* updated gatekeeper for plan

* move buildspecs to buildspecs dir

* python3.9 on this image

* refactor pipeline stuff

* min

* turn pipeline back on

* more wip

* update

* updated from tfmod-pipeline

* add and delete for less git churn

* add and delete for less git churn

* add prefixes

* add prefixes into locals context

* add prefixes into root

* add aws-logs output

* add log tailing

* update source strings to use https

* central settings for codebuild

* ssh stuff

* wip

* back to ssh

* back to https now that network works

* ssh again

* https again

* use a more recent image

* add downloader

* updates to state mgmt

* testing

* fmt

* state issues

* update to gitlab

* wip

* back to github

* getting sboms

* skip if in codebuild

* cleanup/cruft removal

* new cluster updates

---------

Co-authored-by: Srini Nangunuri <srinivasa.nangunuri@census.gov>
Co-authored-by: David John Arnold Jr <david.j.arnold.jr@census.gov>
  • Loading branch information
3 people committed Jun 26, 2025
1 parent 636bfad commit 78adde4
Show file tree
Hide file tree
Showing 89 changed files with 1,825 additions and 3,738 deletions.
24 changes: 0 additions & 24 deletions .checkov.yml

This file was deleted.

42 changes: 31 additions & 11 deletions .github/platform-tg-infra.code-workspace
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,22 @@
"name": "tfmod-cert-mgr",
"path": "../../tfmod-cert-mgr"
},
{
"name": "tfmod-config-job",
"path": "../../tfmod-config-job"
},
{
"name": "tfmod-cribl",
"path": "../../tfmod-cribl"
},
{
"name": "tfmod-custom-iam-role-for-service-account-eks",
"path": "../../tfmod-custom-iam-role-for-service-account-eks"
},
{
"name": "tfmod-downloader",
"path": "../../tfmod-downloader"
},
{
"name": "tfmod-eks",
"path": "../../tfmod-eks"
Expand All @@ -21,8 +37,12 @@
"path": "../../tfmod-eks-dns"
},
{
"name": "tfmod-gogatekeeper",
"path": "../../tfmod-gogatekeeper"
"name": "tfmod-esri-arcgis",
"path": "../../tfmod-esri-arcgis"
},
{
"name": "tfmod-gatekeeper",
"path": "../../tfmod-gatekeeper"
},
{
"name": "tfmod-grafana",
Expand Down Expand Up @@ -65,23 +85,23 @@
"path": "../../tfmod-open-telemetry"
},
{
"name": "tfmod-prometheus",
"path": "../../tfmod-prometheus"
"name": "tfmod-pipeline",
"path": "../../tfmod-pipeline"
},
{
"name": "tfmod-tempo",
"path": "../../tfmod-tempo"
"name": "tfmod-prometheus",
"path": "../../tfmod-prometheus"
},
{
"name": "terraform-aws-eks",
"path": "../../terraform-aws-eks"
"name": "tfmod-postgresql",
"path": "../../tfmod-postgresql"
},
{
"name": "terragrunt",
"path": "../../terragrunt"
"name": "tfmod-tempo",
"path": "../../tfmod-tempo"
},
{
"path": "../../tfmod-config-job"
"path": "../../repo-setup"
}
]
}
108 changes: 107 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
.PHONY: help init validate plan fmt check clean
.PHONY: help config init validate plan fmt check clean deploy-to-pipeline tail parse
# Default values
ENV ?= development
REGION_DIR ?= us-gov-east-1
CLUSTER_DIR ?= csvd-platform-lab-mcm

help:
@echo "Available targets:"
Expand All @@ -8,7 +12,62 @@ help:
@echo " fmt - Format HCL files"
@echo " check - Run all checks (format, validate, plan)"
@echo " clean - Clean up Terragrunt cache and temporary files"
@echo " deploy-to-pipeline - Zip and upload to S3 to trigger CodePipeline"
@echo " tail - Tail the logs of the CodeBuild project"

# Shared configuration target that exports all variables
config:
@echo "Loading configuration..."

# Detect configuration files
$(eval ACCOUNT_HCL=lab/$(ENV)/account.hcl)
$(eval REGION_HCL=lab/$(ENV)/$(REGION_DIR)/region.hcl)
$(eval CLUSTER_HCL=lab/$(ENV)/$(REGION_DIR)/vpc/$(CLUSTER_DIR)/cluster.hcl)

@if [ ! -f "$(ACCOUNT_HCL)" ]; then echo "Error: $(ACCOUNT_HCL) not found"; exit 1; fi
@if [ ! -f "$(REGION_HCL)" ]; then echo "Error: $(REGION_HCL) not found"; exit 1; fi
@if [ ! -f "$(CLUSTER_HCL)" ]; then echo "Error: $(CLUSTER_HCL) not found"; exit 1; fi

# Extract values from HCL files
$(eval AWS_ACCOUNT_ID=$(shell grep -oP 'aws_account_id\s*=\s*"\K[^"]+' $(ACCOUNT_HCL)))
$(eval ACCOUNT_NAME=$(shell grep -oP 'account_name\s*=\s*"\K[^"]+' $(ACCOUNT_HCL)))
$(eval AWS_PROFILE=$(shell echo $(AWS_ACCOUNT_ID)-$(shell echo $(ACCOUNT_NAME) | sed 's/-ew/-gov/')))
$(eval AWS_REGION=$(shell grep -oP 'aws_region\s*=\s*"\K[^"]+' $(REGION_HCL)))
$(eval CLUSTER_NAME=$(shell grep -oP 'cluster_name\s*=\s*"\K[^"]+' $(CLUSTER_HCL)))

# Calculate derived values
$(eval REGION_SHORT=$(shell echo $(AWS_REGION) | sed 's/\([a-z]\)[a-z]*-/\1/g'))
$(eval S3_BUCKET=v-s3-eks-$(CLUSTER_NAME)-artifacts-$(AWS_ACCOUNT_ID)-$(REGION_SHORT))
$(eval OBJECT_KEY=clusters/$(CLUSTER_NAME)/platform-tg-infra.zip)
$(eval PIPELINE_NAME=eks-$(CLUSTER_NAME)-codepipeline)
$(eval PIPELINE_URL=https://console.amazonaws-us-gov.com/codesuite/codepipeline/pipelines/$(PIPELINE_NAME)/view?region=$(AWS_REGION))

@echo "Using configuration:"
@echo " AWS_ACCOUNT_ID: $(AWS_ACCOUNT_ID)"
@echo " ACCOUNT_NAME: $(ACCOUNT_NAME)"
@echo " AWS_PROFILE: $(AWS_PROFILE)"
@echo " AWS_REGION: $(AWS_REGION)"
@echo " CLUSTER_NAME: $(CLUSTER_NAME)"
@echo " S3_BUCKET: $(S3_BUCKET)"

@if [ -z "$(AWS_ACCOUNT_ID)" ] || [ -z "$(AWS_PROFILE)" ] || [ -z "$(AWS_REGION)" ] || [ -z "$(CLUSTER_NAME)" ]; then \
echo "Error: Failed to extract all required variables from HCL files"; \
exit 1; \
fi

# Export variables for child processes
$(eval export AWS_ACCOUNT_ID)
$(eval export ACCOUNT_NAME)
$(eval export AWS_PROFILE)
$(eval export AWS_REGION)
$(eval export CLUSTER_NAME)
$(eval export REGION_SHORT)
$(eval export S3_BUCKET)
$(eval export OBJECT_KEY)
$(eval export PIPELINE_NAME)
$(eval export PIPELINE_URL)

# Basic terragrunt operations
init:
@echo "Initializing Terragrunt configurations..."
terragrunt run-all init
Expand All @@ -33,3 +92,50 @@ clean:
find . -type d -name ".terragrunt-cache" -exec rm -rf {} +
find . -type f -name ".terraform.lock.hcl" -delete
find . -type f -name "terragrunt-debug.tfvars.json" -delete

# Pipeline operations that depend on shared config
deploy-to-pipeline: config
@echo "Preparing to deploy to pipeline..."

@echo "Copy buildspecs from tfmod-pipeline"
mkdir -p ./buildspecs
cp -r ../tfmod-pipeline/buildspecs/* ./buildspecs

@echo "Creating zip file..."
zip -r platform-tg-infra.zip . -x "*.git*" "*.github*" "*.terragrunt-cache*" "*.terraform*"

rm -rf ./buildspecs

@echo "Stopping any active pipeline executions for $(PIPELINE_NAME)..."
$(eval PIPELINE_EXECUTIONS=$(shell aws codepipeline list-pipeline-executions --pipeline-name $(PIPELINE_NAME) --region $(AWS_REGION) --profile $(AWS_PROFILE) --query "pipelineExecutionSummaries[?status=='InProgress'].pipelineExecutionId" --output text))
@if [ -n "$(PIPELINE_EXECUTIONS)" ]; then \
echo "Found active pipeline executions: $(PIPELINE_EXECUTIONS)"; \
for EXECUTION_ID in $(PIPELINE_EXECUTIONS); do \
echo "Stopping execution $$EXECUTION_ID..."; \
aws codepipeline stop-pipeline-execution --pipeline-name $(PIPELINE_NAME) --pipeline-execution-id $$EXECUTION_ID --region $(AWS_REGION) --profile $(AWS_PROFILE) --abandon || echo "Warning: Failed to stop execution $$EXECUTION_ID"; \
done; \
echo "Waiting for pipeline executions to stop (10 seconds)..."; \
sleep 10; \
else \
echo "No active pipeline executions found."; \
fi

@echo "Uploading to S3 bucket $(S3_BUCKET)..."
aws s3 cp platform-tg-infra.zip s3://$(S3_BUCKET)/$(OBJECT_KEY) --profile $(AWS_PROFILE) --sse aws:kms
@echo "Upload complete. Pipeline should trigger automatically."
rm -f platform-tg-infra.zip

@echo "Pipeline URL: $(PIPELINE_URL)"
@echo "You can access the pipeline directly at the URL above."
@echo "Logs will start once CodeBuild runs... this can take a few minutes..."
$(MAKE) tail

# Improved tail action using shared config
tail: config
@echo "Tailing Pipeline Logs for cluster $(CLUSTER_NAME):"
aws logs tail /aws/codebuild/$(CLUSTER_NAME) \
--format short \
--follow \
--since 1m \
--region $(AWS_REGION) \
--profile $(AWS_PROFILE)
168 changes: 114 additions & 54 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,55 +1,115 @@
## How to setup and run terragrunt scripts for EKS related modules in a LAB account

### Lab Account request and setup:
- Open a REMEDY ticket for creating an account in LAB environment, preferably with t3-admin role.
- LAB account url:https://pssvlab.tco.census.gov/PSS/
- Make a note that the LAB account password is different from laptop password
- Once you have successful log proceed to next steps below
### Access LAB jumphost (bromine):
Refer this page for additional help on sso credentials: https://github.e.it.census.gov/terraform/support/tree/master/docs/how-to/aws-sso

1. Goto LAB workspace:https://clients.amazonworkspaces.com/
2. Hit the web access login on the top right corner
3. Enter the following registration code: FRosu+FMEXNZ and click Register
4. Use your jbid and password (use lab password)
5. Open reflection client and ssh connect to bromine.cto.census.gov
6. On bromine, sso login to lab-gov as:
$ aws-sso-login.sh lab-gov
7. On a browser goto auth-dev.census.gov
8. Use PIV card option to login
9. Copy the link from step 6 and paste it on the portal and authenticate
10. Go back to bromine and doubleclick at an empty space, you should get successful login lab env

### Environment Setup:
- Set your profile as a default profile by exporting AWS_PROFILE variable
$ export AWS_PROFILE="224384469011-lab-dev-gov.inf-admin-t3"
$ aws sts get-caller-identity <to verify you are getting the credential from 224384469011>
- Run any aws commands to make sure you are getting responses from the account your profile is set to:
$ aws s3 ls

### Terraform/Terragrunt binaries and versions:
The following binaries used:
- Terraform version: v1.7.5
- Terragrunt version: v0.55.21
These versions can be found on bromine.cto.census.gov host at:
/app/terraform/bin/terr* folder
### Run the Terragrunt script:
- Cd to specifc folder (example: cd eks) and Run terragrunt plan
$ terragrunt plan
- Verify the plan output and make sure there are no errors
- Run terragrunt apply
$ terragrunt apply
- Verify apply completes successfully and verify the resources on AWS Console.

### Accessing the cluster:
$ aws eks --region us-gov-east-1 update-kubeconfig --name platform-eng-eks-test
$ kubectl config use-context arn:aws-us-gov:eks:us-gov-east-1:224384469011:cluster/platform-eng-eks-test
$ kubectl config get-contexts

### Run few kubectl commands to verify you are accessing the cluster
$ kubectl cluster-info
$ kubectl get pods -A
$ kubectl get ns
# Platform Infrastructure with Terragrunt

This repository helps manage cloud infrastructure using Terragrunt. It is designed to make managing infrastructure easier and more organized. Terragrunt is used to handle infrastructure as code for different environments.

## What's in This Repository

- `/configs` - Files for setting up things like node groups and resource limits
- `/docs` - Guides and rules for how to set up and manage the infrastructure
- `/lab` - Settings for testing and development environments
- `/monitoring` - Tools for keeping an eye on the system
- `/tests` - Tests to make sure everything works as expected

## Documentation

You can find detailed guides in the `/docs` folder:

- [Architecture](docs/ARCHITECTURE.md) - How the system is built
- [Documentation Standards](docs/DOCUMENTATION_STANDARDS.md) - How to write good documentation
- [Infrastructure Standards](docs/INFRASTRUCTURE_STANDARDS.md) - Rules for setting up infrastructure
- [Module Dependencies](docs/MODULE_DEPENDENCIES.md) - How different parts depend on each other
- [Module Standards](docs/MODULE_STANDARDS.md) - Rules for creating modules
- [Observability Standards](docs/OBSERVABILITY_STANDARDS.md) - How to monitor the system
- [Security Audit Checklist](docs/SECURITY_AUDIT_CHECKLIST.md) - Steps to check for security issues
- [Security Baseline](docs/SECURITY_BASELINE.md) - Basic security requirements
- [Testing Standards](docs/TESTING_STANDARDS.md) - Rules for testing
- [Version Control](docs/VERSION_CONTROL.md) - How to manage code versions

## What You Need

- Terraform v1.0.0 or newer
- Terragrunt v0.36.0 or newer
- AWS CLI set up with the right permissions
- Access to AWS resources

## How to Get Started

### Using the Makefile

This repository has a Makefile with helpful commands:

```bash
make help # See available commands
make init # Set up Terragrunt
make validate # Check if everything is set up correctly
make plan # Preview changes
make fmt # Format files
make check # Run all checks
make clean # Clean up temporary files
```

### Manual Terragrunt Commands

Go to the folder with the Terragrunt configuration you want to use:

```bash
cd lab/environment/component

# Set up Terragrunt
terragrunt init

# Preview changes
terragrunt plan

# Apply changes
terragrunt apply

# Remove resources
terragrunt destroy
```

### Running All Configurations

Run commands for all Terragrunt configurations:

```bash
# Set up everything
terragrunt run-all init

# Preview all changes
terragrunt run-all plan

# Apply all changes
terragrunt run-all apply
```

## Configuration

Terragrunt configurations are organized like this:

1. Main settings for each environment
2. Specific settings for different parts of the system
3. Overrides for special cases

Check the environment folders for more details.

## Testing

The `/tests` folder has tools to check if everything works. To run tests:

```bash
cd tests
./run_tests.sh
```

## How to Contribute

1. Make a copy of this repository
2. Create a new branch for your changes
3. Make your updates
4. Run `make check` to ensure everything is correct
5. Submit a pull request

## License

Copyright © 2025 Your Organization. All rights reserved.
Loading

0 comments on commit 78adde4

Please sign in to comment.