Skip to content

Commit

Permalink
Merge branch 'tf-upgrade' of github.e.it.census.gov:terraform-modules…
Browse files Browse the repository at this point in the history 
…/aws-eks into tf-upgrade
  • Loading branch information
@badra001
badra001 committed Sep 7, 2023
2 parents c164165 + 97b72c2 commit dbcec37
Show file tree
Hide file tree
Showing 12 changed files with 789 additions and 57 deletions.
20 changes: 20 additions & 0 deletions examples/extras/cloudwatch-agent/.tf-control
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# .tf-control
# allows for setting a specific command to be used for tf-* commands under this git repo
# see tf-control.sh help for more info

TFCONTROL_VERSION="1.0.5"

TFCOMMAND="terraform_latest"
# TF_CLI_CONFIG_FILE=PATH-TO-FILE/.tf-control.tfrc
# TFARGS=""
# TFNOLOG=""
# TFNOCOLOR=""

# use the following to force a specific version. An upgrade of an existing 0.12.31 to 1.x
# needs you to cycle through 0.13.17, 0.14.11, and then latest (0.15.5 not needed). Other
# steps in between. See https://github.e.it.census.gov/terraform/support/tree/master/docs/how-to/terraform-upgrade for details
#
#TFCOMMAND="terraform_0.12.31"
#TFCOMMAND="terraform_0.13.7"
#TFCOMMAND="terraform_0.14.11"
#TFCOMMAND="terraform_0.15.5"
24 changes: 24 additions & 0 deletions examples/extras/cloudwatch-agent/.tf-control.tfrc
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
TFCONTROL_VERSION="1.0.5"

# https://www.terraform.io/docs/cli/config/config-file.html
plugin_cache_dir = "/data/terraform/terraform.d/plugin-cache"
#disable_checkpoint = true

provider_installation {
# filesystem_mirror {
# path = "/apps/terraform/terraform.d/providers"
# include = [ "*/*/*" ]
# }
filesystem_mirror {
path = "/data/terraform/terraform.d/providers"
include = [ "*/*/*" ]
}
# filesystem_mirror {
# path = "/apps/terraform/terraform.d/providers"
# include = [ "external.terraform.census.gov/*/*" ]
# }
direct {
include = [ "*/*/*" ]
}
}

127 changes: 127 additions & 0 deletions examples/extras/cloudwatch-agent/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
# Extras :: cloudwatch-agent

The configuration in this dierectory will deploy cloudwatch-agent and fluentbit, to be used for EKS Container Insights.

# Links

* AWS Docs
* https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-prerequisites.html
* https://aws.amazon.com/blogs/opensource/centralized-container-logging-fluent-bit/
* https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-use-kubelet.html
* https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-logs-FluentBit.html
* https://aws.github.io/eks-charts"
* Cloudwatch Agnet
* https://github.com/aws/eks-charts/tree/master/stable/aws-cloudwatch-metrics
* Fluent Bit
* https://github.com/aws/aws-for-fluent-bit
* https://github.com/aws/eks-charts/tree/master/stable/aws-for-fluent-bit

# Supported Versions

This configuration has been tested and validated on EKS versions

* 1.24
* 1.25

# Configuration

This uses a helm chart, an IRSA role, and pulls the latest images at the time of creating this module.
Look in the `variables.*.auto.tfvars` files for the version numbers.

# Installation

You will need the latest copy of the `aws-eks` module, using the `tf-upgrade` branch. This requires the use of
Terraform 1.x, and as it is deployed in a subdirectory, it should work without issue.

## Step 1: Get aws-eks repo

If you do not have the `aws-eks` repo, clone it in the branch `tf-upgrade`.

```script
# go to your TF repository directory
cd $PATH_TO_TERRAFORM
git clone git@github.e.it.census.gov:terraform-modules/aws-eks.git -b tf-upgrade
cd aws-eks
export EKS_SOURCE=$(pwd)
```

If you already have the repo, go into the directory, checkout the branch and refresh it.

```script
# go to your TF repository directory
cd $PATH_TO_TERRAFORM
cd aws-eks
git checkout tf-upgrade
git pull origin tf-upgrade
export EKS_SOURCE=$(pwd)
```

## Step 2: Copy code

Go into the `common-services` directory of the EKS cluster where you wish to deploy this. Make a directory, `cloudwatch-agent`, and then
rsync the code. Please use rsync, not copy. There is a directory, and there may be softlinks. You'll work in a new branch. An example is below:

```script
cd $PATH_TO_TERRAFORM
cd 107742151971-do2-govcloud/vpc/east/vpc5/apps/eks-ditd-gups-stage/common-services
mkdir cloudwatch-agent
cd cloudwatch-agent
git checkout -b add-cloudwatch-agent
rsync -avRWH $EKS_SOURCE/examples/extras/cloudwatch-agent/./ ./
```

## Step 3: Plan

There is no configuration needed. All relevant details are pulled from the parent directories. You do need EKS cluster access,
so be sure you are running with a user who has K8S RBAC access.

```script
tf-run plan
tf-plan summary
# add to git
git add .
git commit -m 'add cloudwatch, fluentbit' .
git push
# submit PR with plan summary and plan log
```

## Step 4: Apply

Once the PR is merged, apply, and finalize the directory.

```script
tf-run apply
```

Make sure it started up:

```console
% kubectl --kubeconfig setup/kube.config get pods -n aws-cloudwatch
NAME READY STATUS RESTARTS AGE
aws-cloudwatch-metrics-8jlwh 1/1 Running 0 24h
aws-cloudwatch-metrics-8jxqs 1/1 Running 0 24h
aws-cloudwatch-metrics-k668c 1/1 Running 0 24h
fluent-bit-aws-for-fluent-bit-6bvgk 1/1 Running 0 24h
fluent-bit-aws-for-fluent-bit-b4hk5 1/1 Running 0 24h
fluent-bit-aws-for-fluent-bit-chx46 1/1 Running 0 24h
```

All should be running. If any errors, or not running, look at `events` and `logs`.

Then, check AWS CloudWatch Logs. There will be four log as follows: /aws/containerinsights/{clustername}/{name}
where {name} is

* performance
* host
* applications
* dataplane

The Container Insight dashboard should also show performance data for the cluster, though it may take some
time to appear.


# CHANGELOG

* 1.0.0 -- 2023-08-24
- initial
129 changes: 73 additions & 56 deletions examples/extras/cloudwatch-agent/cloudwatch-agent.tf
View file
Original file line number Diff line number Diff line change
@@ -1,37 +1,4 @@
# https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-prerequisites.html
# https://registry.terraform.io/modules/bailey84j/cloudwatch-agent/kubernetes/latest
# cannot let this create the role, as it tries to attache a policy that does not exist
# we need this policy: arn:aws-us-gov:iam::aws:policy/CloudWatchAgentServerPolicy

# need to hack the module for now
# main.tf
## container {
## name = "${var.name}-agent"
## # image = "amazon/${var.image_name}:${var.image_version}"
## image = "${var.image_name}:${var.image_version}"
#
# want to replace this with var.image_repository
#
# also want to do the same for the role, to allow a different managed role(s) besides
# the appsync one to be used

module "cloudwatch-agent" {
source = "bailey84j/cloudwatch-agent/kubernetes"
version = "1.0.1"

eks_cluster_name = var.cluster_name
create_namespace = false
image_name = split(":", local.cloudwatch_agent_images_output["cloudwatch-agent"].dest_full_path)[0]
image_version = local.cloudwatch_agent_images_output["cloudwatch-agent"].tag
create_iam_role = false
iam_role_arn = module.role_cloudwatch-agent.iam_role_arn

tags = merge(
local.base_tags,
local.common_tags,
var.application_tags,
)
}

data "aws_iam_policy" "policy_cloudwatch-agent" {
name = "CloudWatchAgentServerPolicy"
Expand All @@ -41,7 +8,7 @@ module "role_cloudwatch-agent" {
source = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"

role_description = "EKS IAM Role for ${var.cluster_name} for service account ${var.cloudwatch_agent_namespace}:${var.cloudwatch_agent_name}"
role_name = format("%v%v-irsa__%v", local._prefixes["eks"], var.cluster_name, var.cloudwatch_agent_name)
role_name = format("%v%v-irsa__%v", local._prefixes["eks-role"], var.cluster_name, var.cloudwatch_agent_name)

role_policy_arns = {
policy = data.aws_iam_policy.policy_cloudwatch-agent.arn
Expand All @@ -65,28 +32,6 @@ module "role_cloudwatch-agent" {
)
}

## module "role_cloudwatch-agent" {
## source = "git@github.e.it.census.gov:terraform-modules/aws-iam-role.git?ref=tf-upgrade"
##
## role_description = "EKS IAM Role for ${var.cluster_name} for service account ${var.cloudwatch_agent_namespace}:${var.cloudwatch_agent_name}"
## role_name = format("%v%v-irsa__%v", local._prefixes["eks"], var.cluster_name, var.cloudwatch_agent_name)
## enable_ldap_creation = false
## assume_policy_document = data.aws_iam_policy_document.assume_role_cloudwatch-agent.json
## attached_policies = [aws_iam_policy.policy_cloudwatch-agent.arn]
##
## tags = merge(
## local.base_tags,
## local.common_tags,
## var.tags,
## var.application_tags,
## {
## "eks:namespace" = var.namespace
## "eks:user" = var.name
## }
## )
## }


locals {
cloudwatch_agent_images_output = { for k, v in module.images_cloudwatch-agent.images : v.name => v }
}
Expand All @@ -104,3 +49,75 @@ module "images_cloudwatch-agent" {
var.application_tags,
)
}

resource "aws_cloudwatch_log_group" "cloudwatch_agent_logs" {
for_each = toset(var.cloudwatch_agent_log_names)
name = format("/aws/containerinsights/%v/%v", var.cluster_name, each.key)
retention_in_days = var.cloudwatch_agent_log_retention_days
tags = merge(
local.base_tags,
local.common_tags,
var.application_tags,
)
}

resource "kubernetes_namespace" "cloudwatch-agent" {
metadata {
name = var.cloudwatch_agent_namespace
}
}

# chart
# https://github.com/aws/eks-charts/tree/master/stable/aws-cloudwatch-metrics
resource "helm_release" "cloudwatch-agent" {
chart = "aws-cloudwatch-metrics"
name = "aws-cloudwatch-metrics"
namespace = var.cloudwatch_agent_namespace
repository = var.cloudwatch_agent_charts["cloudwatch-agent"].use_remote ? var.cloudwatch_agent_charts["cloudwatch-agent"].repository : "${path.module}/charts"
version = var.cloudwatch_agent_charts["cloudwatch-agent"].use_remote ? var.cloudwatch_agent_charts["cloudwatch-agent"].version : null
depends_on = [kubernetes_namespace.cloudwatch-agent, module.images_cloudwatch-agent]
set {
name = "image.repository"
value = split(":", local.cloudwatch_agent_images_output["cloudwatch-agent"].dest_full_path)[0]
}
set {
name = "image.tag"
value = local.cloudwatch_agent_images_output["cloudwatch-agent"].tag
}
set {
name = "clusterName"
value = var.cluster_name
}
set {
name = "serviceAccount.name"
value = var.cloudwatch_agent_name
}
set {
name = "serviceAccount.create"
value = "true"
}
set {
name = "serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
value = module.role_cloudwatch-agent.iam_role_arn
}
timeout = 300
}
data "aws_iam_policy_document" "cloudwatch_agent_policy_extra" {
statement {
sid = "DescribeVolumes"
effect = "Allow"
actions = ["ec2:DescribeVolumes"]
resources = ["*"]
}
}

resource "aws_iam_role_policy" "cloudwatch_agent_policy_extra" {
name = "extra"
role = module.role_cloudwatch-agent.iam_role_name

policy = data.aws_iam_policy_document.cloudwatch_agent_policy_extra.json
}

0 comments on commit dbcec37

Please sign in to comment.