Skip to content

Commit

Permalink
Helm Refactor + autoscaling (#11)
Browse files Browse the repository at this point in the history
* autoscaling

* use bitnami image:

* lower requests

* refactor chart values

* cleanup

* kill the canary

* remove canary

* unified config

* fix values

* template error

* remove extras

* add both storage configs

* add some back

* add back required images

* add cleanup_on_fail

* add replace true

* add path_prefix

* give more mem to backend and write

* sort

* update requests resources

* set timeout to 10m

* more values

* more values

* shorter timeout again

* fix config parsing

* fix schema

* add s3 prefix

* no thanos

* more testing

* delete store fix

* fix volumes

* update s3

* guess

* less is more

* increase requests
  • Loading branch information
morga471 committed Mar 17, 2025
1 parent f77d9c1 commit b8a5c25
Show file tree
Hide file tree
Showing 8 changed files with 316 additions and 285 deletions.
101 changes: 101 additions & 0 deletions .github/workflows/terragrunt-cicd.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
name: 'Terraform Module CI'

on:
push:
branches:
- main
paths:
- '**/*.hcl'
- '**/*.tf'
pull_request:
branches:
- main
paths:
- '**/*.hcl'
- '**/*.tf'

permissions:
contents: read
pull-requests: write

jobs:
validate:
name: 'Validate Module'
runs-on: self-hosted

steps:
- name: Checkout
uses: actions/checkout@v3

- name: Setup Terraform
uses: hashicorp/setup-terraform@v2
with:
terraform_version: 1.5.0

- name: Terraform Init
run: |
terraform init -backend=false
- name: Terraform Format
run: |
terraform fmt -check
- name: Terraform Validate
run: |
terraform validate
- name: Run tflint
uses: terraform-linters/setup-tflint@v3
if: github.event_name == 'pull_request'

- name: Lint Terraform
if: github.event_name == 'pull_request'
run: |
tflint --format compact
release:
name: 'Create Release'
needs: validate
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
runs-on: self-hosted
permissions:
contents: write

steps:
- name: Checkout
uses: actions/checkout@v3
with:
fetch-depth: 0
token: ${{ secrets.GITHUB_TOKEN }}

- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: '3.9'

- name: Install Commitizen
run: |
pip install commitizen
- name: Configure Git
run: |
git config --local user.email "action@github.com"
git config --local user.name "GitHub Action"
- name: Bump Version and Generate Changelog
id: cz
run: |
cz bump --yes
echo "new_version=$(cz version --project)" >> $GITHUB_OUTPUT
echo "changelog=$(cz changelog --dry-run)" >> $GITHUB_OUTPUT
- name: Create Release
uses: actions/create-release@v1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
tag_name: v${{ steps.cz.outputs.new_version }}
release_name: Release v${{ steps.cz.outputs.new_version }}
draft: false
prerelease: false
body: ${{ steps.cz.outputs.changelog }}
8 changes: 2 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,32 +25,29 @@ to loki.

| Name | Version |
|------|---------|
| <a name="provider_aws"></a> [aws](#provider\_aws) | 5.87.0 |
| <a name="provider_aws"></a> [aws](#provider\_aws) | 5.89.0 |
| <a name="provider_helm"></a> [helm](#provider\_helm) | 2.17.0 |
| <a name="provider_kubernetes"></a> [kubernetes](#provider\_kubernetes) | 2.35.1 |

## Modules

| Name | Source | Version |
|------|--------|---------|
| <a name="module_images"></a> [images](#module\_images) | git@github.e.it.census.gov:terraform-modules/aws-ecr-copy-images.git/ | tf-upgrade |
| <a name="module_loki_irsa_role"></a> [loki\_irsa\_role](#module\_loki\_irsa\_role) | git@github.e.it.census.gov:SCT-Engineering/tfmod-custom-iam-role-for-service-account-eks.git | n/a |
| <a name="module_loki_irsa_role"></a> [loki\_irsa\_role](#module\_loki\_irsa\_role) | git@github.e.it.census.gov:SCT-Engineering/tfmod-custom-iam-role-for-service-account-eks.git// | main |
| <a name="module_loki_s3"></a> [loki\_s3](#module\_loki\_s3) | git@github.e.it.census.gov:terraform-modules/aws-s3.git//standard | tf-upgrade |

## Resources

| Name | Type |
|------|------|
| [helm_release.loki](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource |
| [kubernetes_namespace.ns](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/namespace) | resource |
| [aws_caller_identity.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/caller_identity) | data source |
| [aws_s3_bucket.s3_server_access_logs](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/s3_bucket) | data source |

## Inputs

| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| <a name="input_canary_tag"></a> [canary\_tag](#input\_canary\_tag) | The tag of the grafana/loki-canary image to use. | `string` | `"3.0.0"` | no |
| <a name="input_cluster_name"></a> [cluster\_name](#input\_cluster\_name) | EKS cluster name name component used through out the EKS cluster describing its purpose (ex: dice-dev) | `string` | n/a | yes |
| <a name="input_enterprise_logs_provisioner_tag"></a> [enterprise\_logs\_provisioner\_tag](#input\_enterprise\_logs\_provisioner\_tag) | The version of the grafana/enterprise-logs-provisioner image to use. | `string` | `"v1.7.0"` | no |
| <a name="input_exporter_tag"></a> [exporter\_tag](#input\_exporter\_tag) | The version of prom/memcached-exporter to use for the gateway. | `string` | `"v0.14.4"` | no |
Expand All @@ -64,7 +61,6 @@ to loki.
| <a name="input_region"></a> [region](#input\_region) | The region holding these resources (for the s3 bucket.) | `string` | n/a | yes |
| <a name="input_rwo_storage_class"></a> [rwo\_storage\_class](#input\_rwo\_storage\_class) | Specify the storage class for read/write/once persistent volumes. | `string` | `"gp3-encrypted"` | no |
| <a name="input_sidecar_tag"></a> [sidecar\_tag](#input\_sidecar\_tag) | The version of kiwigrid/k8s-sidecar to use for the gateway. | `string` | `"1.27.4"` | no |
| <a name="input_tag_costallocation"></a> [tag\_costallocation](#input\_tag\_costallocation) | Tag CostAllocation (default) | `string` | `"csvd:infrastructure"` | no |
| <a name="input_tags"></a> [tags](#input\_tags) | Additional tags to add to resources created in AWS (s3 bucket, ...) | `map(string)` | `{}` | no |

## Outputs
Expand Down
28 changes: 9 additions & 19 deletions copy_images.tf
Original file line number Diff line number Diff line change
@@ -1,37 +1,27 @@
locals {
exporter_key = format("%v#%v", "prom/memcached-exporter", var.exporter_tag)
gateway_key = format("%v#%v", "grafana/nginx-unprivileged", var.gateway_tag)
loki_key = format("%v#%v", "grafana/loki", var.loki_tag)
canary_key = format("%v#%v", "grafana/loki-canary", var.canary_tag)
memcached_key = format("%v#%v", "memcached", var.memcached_tag)
provisioner_key = format("%v#%v", "grafana/enterprise-logs-provisioner", var.enterprise_logs_provisioner_tag)
gateway_key = format("%v#%v", "grafana/nginx-unprivileged", var.gateway_tag)
sidecar_key = format("%v#%v", "kiwigrid/k8s-sidecar", var.sidecar_tag)
memcached_key = format("%v#%v", "memcached", var.memcached_tag)
exporter_key = format("%v#%v", "prom/memcached-exporter", var.exporter_tag)

image_config = [
{
enabled = true
dest_path = null
name = "grafana/loki"
source_image = "grafana/loki"
source_registry = "docker.io"
source_image = "bitnami/grafana-loki"
source_registry = "public.ecr.aws"
source_tag = var.loki_tag
tag = var.loki_tag
},
{
enabled = true
dest_path = null
name = "grafana/loki-canary"
source_image = "grafana/loki-canary"
source_registry = "docker.io"
source_tag = var.canary_tag
tag = var.canary_tag
},
{
enabled = true
dest_path = null
name = "memcached"
source_image = "memcached"
source_registry = "docker.io"
source_image = "bitnami/memcached"
source_registry = "public.ecr.aws"
source_tag = var.memcached_tag
tag = var.memcached_tag
},
Expand Down Expand Up @@ -66,8 +56,8 @@ locals {
enabled = true
dest_path = null
name = "grafana/nginx-unprivileged"
source_image = "nginxinc/nginx-unprivileged"
source_registry = "docker.io"
source_image = "nginx/nginx-unprivileged"
source_registry = "public.ecr.aws"
source_tag = var.gateway_tag
tag = var.gateway_tag
},
Expand Down
Loading

0 comments on commit b8a5c25

Please sign in to comment.