No kubectl (#16)

* convert to helm charts * update pathing * cruft and tags * jsonit * tags * formatting * fmt * comment * tags2 * add json * fix fmt * formatting v2 * quotes * escape the slashes * fmt3 * fmt4 * subnet selectors * fmt4 * quotes * more quotes * remove values defaults * remove problem lines * add subnetSelectorTerms again * ergh * index * use the tools * Name? * pass in same * wip * add if around labels * lints * data structures * cleanup * fmt * index on securityGroup * indexes * swap role for instanceProfile * .name * comment out bad default value * not a map * not a map * not a map * not a map * remove commented value * weird tflint errors * plugins * retry * parsing errors * update values * add budgets * update nodepool.yaml * set limits * more values * fmt * comment likely problem * update values * more * even more * undo * redo * undo * dont set default limits * more reqs * update default limits * update default limits * tags maybe * dont look up the value guess it * add instance profile again because govcloud * restore tflint config
SCT-Engineering · Feb 13, 2025 · 22b267c · 22b267c
1 parent 3626a9c
commit 22b267c
Show file tree

Hide file tree

Showing 12 changed files with 492 additions and 140 deletions.
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -26,7 +26,7 @@ repos:
       args: [--markdown-linebreak-ext=md]
       exclude: CHANGELOG.md
 
-    - id: check-yaml
+    # - id: check-yaml
     - id: check-merge-conflict
     - id: check-executables-have-shebangs
 
@@ -50,7 +50,7 @@ repos:
 
 # Terraform Hooks
 - repo: https://github.com/antonbabenko/pre-commit-terraform
-  rev: v1.96.1 # Get the latest from: https://github.com/antonbabenko/pre-commit-terraform/releases
+  rev: v1.97.3 # Get the latest from: https://github.com/antonbabenko/pre-commit-terraform/releases
   hooks:
     - id: terraform_fmt
       args:

diff --git a/README.md b/README.md
@@ -20,15 +20,13 @@ The module deploys Karpenter needed AWS resources, namely in IAM. It copies the
 | <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.13 |
 | <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 5.14.0 |
 | <a name="requirement_helm"></a> [helm](#requirement\_helm) | >= 2.11.0 |
-| <a name="requirement_kubernetes"></a> [kubernetes](#requirement\_kubernetes) | >= 2.23.0 |
 
 ## Providers
 
 | Name | Version |
 |------|---------|
 | <a name="provider_aws"></a> [aws](#provider\_aws) | 5.84.0 |
 | <a name="provider_helm"></a> [helm](#provider\_helm) | 2.17.0 |
-| <a name="provider_kubernetes"></a> [kubernetes](#provider\_kubernetes) | 2.35.1 |
 
 ## Modules
 
@@ -41,10 +39,9 @@ The module deploys Karpenter needed AWS resources, namely in IAM. It copies the
 
 | Name | Type |
 |------|------|
-| [aws_iam_instance_profile.karpenter_node_instance_profile](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_instance_profile) | resource |
+| [aws_iam_instance_profile.karpenter_node](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_instance_profile) | resource |
 | [helm_release.karpenter](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource |
-| [kubernetes_manifest.karpenter_node_class](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/manifest) | resource |
-| [kubernetes_manifest.karpenter_node_pool](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/manifest) | resource |
+| [helm_release.nodepool_resources](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource |
 | [aws_arn.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/arn) | data source |
 | [aws_caller_identity.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/caller_identity) | data source |
 | [aws_ecr_image.karpenter_image](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ecr_image) | data source |

diff --git a/karpenter-resources/Chart.yaml b/karpenter-resources/Chart.yaml
@@ -0,0 +1,6 @@
+apiVersion: v2
+name: karpenter-resources
+description: Helm chart for configuring custom resources for Karpenter on the cluster
+version: 0.0.2
+appVersion: 0.0.2
+icon: https://repository-images.githubusercontent.com/278480393/dab059c8-caa1-4b55-aaa7-3d30e47a5616
diff --git a/karpenter-resources/README.md b/karpenter-resources/README.md
@@ -0,0 +1,71 @@
+# Karpenter Resources Helm Chart
+
+## Overview
+
+This Helm chart is an abstraction layer designed for deploying various configurations of Karpenter nodes in a Kubernetes cluster managed by EKS. It integrates seamlessly with Terraform, allowing users to define different node pools and settings for their Kubernetes cluster.
+
+## Prerequisites
+
+- Helm 3.x or later installed
+- Terraform installed
+- Access to an AWS EKS cluster
+
+## Configuration
+
+The chart is configured to be used with Terraform. Here is an example of how you might define Helm releases for different Karpenter configurations in your Terraform files using EKS Blueprints add-ons:
+
+```hcl
+module "eks_blueprints_addons" {
+    # ... other configurations ...
+    helm_releases = {
+    karpenter-resources-default = {
+      name        = "default"
+      description = "A Helm chart for default node pool"
+      chart       = "${path.module}/helm-values/karpenter-resources"
+      values = [
+        <<-EOT
+      clusterName: ${module.eks.cluster_name}
+      ec2NodeClass:
+        karpenterRole: ${split("/", module.eks_blueprints_addons.karpenter.node_iam_role_arn)[1]}
+      nodePool:
+        labels:
+          - provisioner: default
+          - workload: rayhead
+      EOT
+      ]
+    }
+    }
+}
+```
+
+## Testing the Chart with Helm Template
+
+To review the Kubernetes manifests that will be generated by the Helm chart based on your configuration, you can use the `helm template` command. This is especially useful for validating your Terraform configurations before applying them.
+
+1. **Generate the Manifests**
+
+    Navigate to the directory where your Helm chart is located.
+
+    ```sh
+    cd path/to/helm-chart
+    ```
+
+2. **Run Helm Template**
+
+    Use the `helm template` command with your custom values. For example:
+
+    ```sh
+    helm template my-release-name . --values values.yaml
+    ```
+
+    Replace `my-release-name` with a name for your release, and `values.yaml` with the path to your custom values file.
+
+    To test specific configurations defined in your Terraform file, you can create a temporary values file with the configuration snippet from your Terraform definition:
+
+    ```sh
+    echo '<your Terraform values here>' > temp-values.yaml
+    helm template my-release-name . --values temp-values.yaml
+    rm temp-values.yaml
+    ```
+
+    This will output the Kubernetes manifests to your terminal, allowing you to review them.
diff --git a/karpenter-resources/templates/_helpers.tpl b/karpenter-resources/templates/_helpers.tpl
@@ -0,0 +1,125 @@
+{{/*
+Expand the name of the chart.
+*/}}
+{{- define "karpenter-node-pool.name" -}}
+{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
+{{- end }}
+
+{{/*
+Create a default fully qualified app name.
+We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
+If release name contains chart name it will be used as a full name.
+*/}}
+{{- define "karpenter-node-pool.fullname" -}}
+{{- if .Values.fullnameOverride }}
+{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
+{{- else }}
+{{- $name := default .Chart.Name .Values.nameOverride }}
+{{- if contains $name .Release.Name }}
+{{- .Release.Name | trunc 63 | trimSuffix "-" }}
+{{- else }}
+{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
+{{- end }}
+{{- end }}
+{{- end }}
+
+{{/*
+Create chart name and version as used by the chart label.
+*/}}
+{{- define "karpenter-node-pool.chart" -}}
+{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
+{{- end }}
+
+{{/*
+Common labels
+*/}}
+{{- define "karpenter-node-pool.labels" -}}
+helm.sh/chart: {{ include "karpenter-node-pool.chart" . }}
+{{ include "karpenter-node-pool.selectorLabels" . }}
+{{- if .Chart.AppVersion }}
+app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
+{{- end }}
+app.kubernetes.io/managed-by: {{ .Release.Service }}
+{{- end }}
+
+{{/*
+Selector labels
+*/}}
+{{- define "karpenter-node-pool.selectorLabels" -}}
+app.kubernetes.io/name: {{ include "karpenter-node-pool.name" . }}
+app.kubernetes.io/instance: {{ .Release.Name }}
+{{- end }}
+
+{{/*
+Create the name of the service account to use
+*/}}
+{{- define "karpenter-node-pool.serviceAccountName" -}}
+{{- if .Values.serviceAccount.create }}
+{{- default (include "karpenter-node-pool.fullname" .) .Values.serviceAccount.name }}
+{{- else }}
+{{- default "default" .Values.serviceAccount.name }}
+{{- end }}
+{{- end }}
+
+{{/*
+Expand the name of the chart.
+*/}}
+{{- define "karpenter-node-class.name" -}}
+{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
+{{- end }}
+
+{{/*
+Create a default fully qualified app name.
+We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
+If release name contains chart name it will be used as a full name.
+*/}}
+{{- define "karpenter-node-class.fullname" -}}
+{{- if .Values.fullnameOverride }}
+{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
+{{- else }}
+{{- $name := default .Chart.Name .Values.nameOverride }}
+{{- if contains $name .Release.Name }}
+{{- .Release.Name | trunc 63 | trimSuffix "-" }}
+{{- else }}
+{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
+{{- end }}
+{{- end }}
+{{- end }}
+
+{{/*
+Create chart name and version as used by the chart label.
+*/}}
+{{- define "karpenter-node-class.chart" -}}
+{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
+{{- end }}
+
+{{/*
+Common labels
+*/}}
+{{- define "karpenter-node-class.labels" -}}
+helm.sh/chart: {{ include "karpenter-node-class.chart" . }}
+{{ include "karpenter-node-class.selectorLabels" . }}
+{{- if .Chart.AppVersion }}
+app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
+{{- end }}
+app.kubernetes.io/managed-by: {{ .Release.Service }}
+{{- end }}
+
+{{/*
+Selector labels
+*/}}
+{{- define "karpenter-node-class.selectorLabels" -}}
+app.kubernetes.io/name: {{ include "karpenter-node-class.name" . }}
+app.kubernetes.io/instance: {{ .Release.Name }}
+{{- end }}
+
+{{/*
+Create the name of the service account to use
+*/}}
+{{- define "karpenter-node-class.serviceAccountName" -}}
+{{- if .Values.serviceAccount.create }}
+{{- default (include "karpenter-node-class.fullname" .) .Values.serviceAccount.name }}
+{{- else }}
+{{- default "default" .Values.serviceAccount.name }}
+{{- end }}
+{{- end }}
diff --git a/karpenter-resources/templates/node-class.yaml b/karpenter-resources/templates/node-class.yaml
@@ -0,0 +1,73 @@
+{{- if .Values.ec2NodeClass.enabled }}
+apiVersion: karpenter.k8s.aws/v1
+kind: EC2NodeClass
+metadata:
+  name: {{ .Values.nodePool.nodeClassRef.name }}
+spec:
+  {{- if .Values.ec2NodeClass.amiFamily }}
+  amiFamily: {{ .Values.ec2NodeClass.amiFamily }}
+  {{- end }}
+  {{- if .Values.ec2NodeClass.amiSelectorTerms }}
+  amiSelectorTerms:
+    {{- toYaml .Values.ec2NodeClass.amiSelectorTerms | nindent 4 }}
+  {{- end }}
+  {{- if .Values.ec2NodeClass.subnetSelectorTerms }}
+  subnetSelectorTerms:
+    {{- range .Values.ec2NodeClass.subnetSelectorTerms }}
+    - {{- if .tags }}
+      tags:
+        {{- range $key, $value := .tags }}
+        {{ $key }}: {{ $value | quote }}
+        {{- end }}
+      {{- end }}
+      {{- if .id }}
+      id: {{ .id }}
+      {{- end }}
+    {{- end }}
+  {{- end }}
+  {{- if .Values.ec2NodeClass.securityGroupSelectorTerms }}
+  securityGroupSelectorTerms:
+    {{- range .Values.ec2NodeClass.securityGroupSelectorTerms }}
+    - {{- if .tags }}
+      tags:
+        {{- range $key, $value := .tags }}
+        {{ $key }}: {{ $value | quote }}
+        {{- end }}
+      {{- end }}
+      {{- if .id }}
+      id: {{ .id }}
+      {{- end }}
+      {{- if .name }}
+      name: {{ .name }}
+      {{- end }}
+    {{- end }}
+  {{- end }}
+  instanceProfile: {{ .Values.ec2NodeClass.karpenterRole }}
+  {{- if .Values.ec2NodeClass.tags }}
+  tags:
+    {{- range $key, $value := .Values.ec2NodeClass.tags }}
+    {{ $key }}: {{ $value | quote }}
+    {{- end }}
+  {{- end }}
+  metadataOptions:
+    httpEndpoint: {{ .Values.ec2NodeClass.metadataOptions.httpEndpoint }}
+    {{- if .Values.ec2NodeClass.metadataOptions.httpProtocolIPv6 }}
+    httpProtocolIPv6: {{ .Values.ec2NodeClass.metadataOptions.httpProtocolIPv6 }}
+    {{- end }}
+    {{- if .Values.ec2NodeClass.metadataOptions.httpPutResponseHopLimit }}
+    httpPutResponseHopLimit: {{ .Values.ec2NodeClass.metadataOptions.httpPutResponseHopLimit }}
+    {{- end }}
+    httpTokens: {{ .Values.ec2NodeClass.metadataOptions.httpTokens }}
+  {{- if .Values.ec2NodeClass.blockDeviceMappings }}
+  blockDeviceMappings:
+    {{- toYaml .Values.ec2NodeClass.blockDeviceMappings | nindent 4 }}
+  {{- end }}
+  detailedMonitoring: {{ .Values.ec2NodeClass.detailedMonitoring }}
+  {{- if .Values.ec2NodeClass.instanceStorePolicy }}
+  instanceStorePolicy: {{ .Values.ec2NodeClass.instanceStorePolicy }}
+  {{- end }}
+  {{- if .Values.ec2NodeClass.userData }}
+  userData: |
+    {{- .Values.ec2NodeClass.userData | nindent 4 }}
+  {{- end }}
+{{- end }}
diff --git a/karpenter-resources/templates/node-pool.yaml b/karpenter-resources/templates/node-pool.yaml
@@ -0,0 +1,57 @@
+{{- if .Values.nodePool.enabled }}
+apiVersion: karpenter.sh/v1
+kind: NodePool
+metadata:
+  name: {{ .Values.nodePool.name }}
+spec:
+  template:
+    metadata:
+      labels:
+        NodePool: {{ .Values.nodePool.name }}
+        NodeGroupType: {{ .Values.nodePool.name }}
+        {{- if .Values.nodePool.labels }}
+        {{- with .Values.nodePool.labels }}
+        {{- range $key, $value := . }}
+        {{ $key }}: {{ $value | quote }}
+        {{- end }}
+        {{- end }}
+        {{- end }}
+    spec:
+      nodeClassRef:
+        name: {{ .Values.nodePool.nodeClassRef.name }}
+        group: {{ .Values.nodePool.nodeClassRef.group }}
+        kind: {{ .Values.nodePool.nodeClassRef.kind }}
+      {{- with .Values.nodePool.taints }}
+      taints:
+      {{- toYaml . | nindent 8 }}
+      {{- end }}
+      {{- with .Values.nodePool.requirements }}
+      requirements:
+      {{- toYaml . | nindent 8 }}
+      {{- end }}
+      {{- if .Values.nodePool.expireAfter }}
+      expireAfter: {{ .Values.nodePool.expireAfter }}
+      {{- end }}
+      {{- if .Values.nodePool.terminationGracePeriod }}
+      terminationGracePeriod: {{ .Values.nodePool.terminationGracePeriod }}
+      {{- end }}
+  disruption:
+    consolidationPolicy: {{ .Values.nodePool.disruption.consolidationPolicy }}
+    consolidateAfter: {{ .Values.nodePool.disruption.consolidateAfter }}
+    {{- with .Values.nodePool.disruption.budgets }}
+    budgets:
+    {{- toYaml . | nindent 4 }}
+    {{- end }}
+  {{- with .Values.nodePool.limits }}
+  limits:
+    {{- if .cpu }}
+    cpu: {{ .cpu }}
+    {{- end }}
+    {{- if .memory }}
+    memory: {{ .memory }}
+    {{- end }}
+  {{- end }}
+  {{- if .Values.nodePool.weight }}
+  weight: {{ .Values.nodePool.weight }}
+  {{- end }}
+{{- end }}