Skip to content

Commit

Permalink
move more examples, add more readmes
Browse files Browse the repository at this point in the history
  • Loading branch information
@badra001
badra001 committed Nov 10, 2021
1 parent 398f9f1 commit 5279dcd
Show file tree
Hide file tree
Showing 14 changed files with 328 additions and 110 deletions.
4 changes: 4 additions & 0 deletions examples/established-cluster-examples/empty/locals.tf
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
locals {
region = var.region
}

1 change: 1 addition & 0 deletions examples/established-cluster-examples/empty/prefixes.tf
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
../prefixes.tf
5 changes: 5 additions & 0 deletions examples/established-cluster-examples/empty/test.tf
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
data "aws_ebs_default_kms_key" "current" {}

data "aws_kms_key" "ebs_key" {
key_id = data.aws_ebs_default_kms_key.current.key_arn
}
1 change: 1 addition & 0 deletions examples/established-cluster-examples/empty/version.tf
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
../version.tf
100 changes: 98 additions & 2 deletions examples/full-cluster/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ For creating a service which uses load balancers (ELB, ALB, or NLB), the last ta
to the subnet(s) for load balancing. A separate set of subnets exist for load balacning, with a name including `private-lb`.


## Copy Variable Settings
## Copy Variable Settings when in a submodule repo

We need the `variables.vpc.tf` and `variables.vpc.auto.tfvars` from the main repository. These are not to be modified in
this submodule.
Expand All @@ -89,7 +89,14 @@ done

Replace {region} and {number} and {APPNAME} with the correct values.

## Copy includes.d
## Link Variable Settings when in the main account repo

Link these files from the `vpc/{region}/vpc{number}/` dirctory:

* variables.vpc.tf
* variables.vpc.auto.tfvars

## Copy includes.d when in a submodule repo

This makes a copy of the entire `MAIN/includes.d` structure in the submodule, for use as soft links to bring in
application variables for tagging.
Expand All @@ -103,6 +110,95 @@ rsync -avRWH $MAINTOP/./includes.d ./

Replace {APPNAME} with the correct value.

## Links includes.d when in the main account repo

If thre is an existing `MAIN/includes.d/` path for the specific application variables you wish to apply,
make a link to it as appropraite.

## Update the settings.auto.tfvars file

Set the appropriate values in the `settings.auto.tfvars` file. An example starter file is at `settings.auto.tfvars.example`.

Here is a sample file:

```hcl
cluster_name = "org-project-env
cluster_version = "1.21"
region = "us-gov-east-1"
domain = "org-project-env.env.domain.census.gov"
eks_instance_disk_size = 40
eks_vpc_name = "*vpcshortname*"
eks_instance_type = "t3.xlarge"
eks_ng_desire_size = 3
eks_ng_max_size = 15
eks_ng_min_size = 3
```

You need to change these values:

* cluster_name: put in the proper org, project, and environment. Cluster names should not be replicated across the environment.
These are tracked in the repo cloud-information/aws/documentation/containers/ (fix link).
* region: include the correct region. This really is a duplicate of the `region` variable, so it may be removed in the future.
* domain: this is the domain name of the clsuter, consisting of the cluster name and the proper domain name for the environment/VPC.
* eks_vpc_name: replace *vpcshortname* with the appropriate vpc name. This is used to find the vpc ID. This will be fixed at a later date.

All the others are subject to your configuration. They are a good starting point, but can vary.

# Terraform Automated Setup

A `tf-run.data` file exists here, so the simplest way to implemnt is with the `tf-run.sh` script.

* copy the `remote_state.yml` from the parent and update `directory` to be the current directory
* run the tf-run.sh

```console
% tf-run.sh apply
```

* example of the `tf-run.sh` steps

This is part of a larger cluster configuration, so at the end of the run it indicates another directory
to visit when done.

```console
% tf-run.sh list
* running action=plan
* START: tf-run.sh v1.1.2 start=1636562594 end= logfile=logs/run.plan.20211110.1636562594.log (not-created)
* reading from tf-run.data
* read 22 entries from tf-run.data
> list
** START: start=1636562594
* 1 COMMENT> make sure the private-lb subnet and container subnets are tagged properly (see README.md)
* 2 STOP> then continue with at step 3
* 3 COMMAND> tf-directory-setup.py -l none -f
* 4 COMMAND> setup-new-directory.sh
* 5 COMMAND> tf-init -upgrade
* 6 POLICY> (*.tf) aws_iam_policy.nlb-policy aws_iam_policy.cloudwatch-policy aws_iam_policy.cluster-admin-policy aws_iam_policy.cluster-admin_assume_policy
* 6 tf-plan -target=aws_iam_policy.nlb-policy -target=aws_iam_policy.cloudwatch-policy -target=aws_iam_policy.cluster-admin-policy -target=aws_iam_policy.cluster-admin_assume_policy
* 7 COMMENT> EC2 key pairs
* 8 tf-plan -target=null_resource.generate_keypair
* 9 tf-plan -target=aws_key_pair.cluster_keypair
* 10 COMMAND> tf-directory-setup.py -l s3
* 11 COMMENT> be sure to add the setup/ec2-ssh-eks-{cluster} to git-secret, git-secret hide, add the setup/*secret and setup/*pub got git, and commit the entirety of the change
* 12 tf-plan
* 13 COMMENT> setup the includes.d/parent_rs.tf according to the includes.d/README
* 14 STOP>
* 15 COMMENT> cd aws-auth and tf-run.sh apply
* 16 STOP>
* 17 COMMENT> cd efs and tf-run.sh apply
* 18 STOP>
* 19 COMMENT> cd irsa-roles and tf-run.sh apply
* 20 STOP>
* 21 COMMENT> cd common-services and tf-run.sh apply
* 22 STOP>
** END: start=1636562594 end=1636562594 elapsed=0 logfile=logs/run.plan.20211110.1636562594.log (not-created)
```

It is highly recommended to use the `tf-run.sh` approach. This has a number of stopping points along the way with comments telling you what to do.
It also directs you to the subdirectories to visit to complete the setup.

# Terraform Manual Setup

## Initialize Cluster Main

We need to setup the main directory for the cluster. Be sure `remote_state.yml` is correct. Then:
Expand Down
132 changes: 57 additions & 75 deletions examples/full-cluster/common-services/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,84 +1,66 @@
```console
% kubectl -n kube-system get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
aws-load-balancer-controller-54fdf64896-jzwsr 1/1 Running 0 23h 10.194.26.74 ip-10-194-26-252.ec2.internal <none> <none>
aws-load-balancer-controller-54fdf64896-qqt6d 1/1 Running 0 23h 10.194.24.242 ip-10-194-24-49.ec2.internal <none> <none>
aws-node-29kmc 1/1 Running 0 7d1h 10.194.24.90 ip-10-194-24-90.ec2.internal <none> <none>
aws-node-6d8ls 1/1 Running 1 7d1h 10.194.25.120 ip-10-194-25-120.ec2.internal <none> <none>
aws-node-6vrbg 1/1 Running 1 7d1h 10.194.26.252 ip-10-194-26-252.ec2.internal <none> <none>
aws-node-ldgxc 1/1 Running 1 7d1h 10.194.24.49 ip-10-194-24-49.ec2.internal <none> <none>
coredns-65bfc5645f-g86rx 1/1 Running 0 7d2h 10.194.24.207 ip-10-194-24-90.ec2.internal <none> <none>
coredns-65bfc5645f-xj9rl 1/1 Running 0 7d2h 10.194.24.69 ip-10-194-24-90.ec2.internal <none> <none>
efs-csi-controller-65fb886fd4-7slw6 3/3 Running 0 2d21h 10.194.24.90 ip-10-194-24-90.ec2.internal <none> <none>
efs-csi-controller-65fb886fd4-vcf9l 3/3 Running 0 2d21h 10.194.25.120 ip-10-194-25-120.ec2.internal <none> <none>
efs-csi-node-6t6v6 3/3 Running 0 2d21h 10.194.25.120 ip-10-194-25-120.ec2.internal <none> <none>
efs-csi-node-kxqfb 3/3 Running 0 2d21h 10.194.24.49 ip-10-194-24-49.ec2.internal <none> <none>
efs-csi-node-p8hzn 3/3 Running 0 2d21h 10.194.26.252 ip-10-194-26-252.ec2.internal <none> <none>
efs-csi-node-xxq9h 3/3 Running 0 2d21h 10.194.24.90 ip-10-194-24-90.ec2.internal <none> <none>
kube-proxy-78n7f 1/1 Running 0 7d1h 10.194.24.90 ip-10-194-24-90.ec2.internal <none> <none>
kube-proxy-cms7c 1/1 Running 0 7d1h 10.194.24.49 ip-10-194-24-49.ec2.internal <none> <none>
kube-proxy-h2t6n 1/1 Running 0 7d1h 10.194.26.252 ip-10-194-26-252.ec2.internal <none> <none>
kube-proxy-jkxnz 1/1 Running 0 7d1h 10.194.25.120 ip-10-194-25-120.ec2.internal <none> <none>
```
# common-services

This is a directory where the common services are setup:

* cert-manager
* istio service mesh
* metrics-server

## Setup Steps

First, copy the `remote_state.yml` from the parent and update `directory` to be the current directory.

## Terraform Automated

A `tf-run.data` file exists here, so the simplest way to implemnt is with the `tf-run.sh` script.

* copy the `remote_state.yml` from the parent and update `directory` to be the current directory
* run the tf-run.sh

```console
% kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
cert-manager cert-manager-7fcbc79fc5-xwt4s 1/1 Running 0 51m 10.194.24.138 ip-10-194-24-49.ec2.internal <none> <none>
cert-manager cert-manager-cainjector-6b7f4575f4-xpgnc 1/1 Running 0 51m 10.194.24.56 ip-10-194-24-49.ec2.internal <none> <none>
cert-manager cert-manager-webhook-6cd54b96fc-rvld4 1/1 Running 0 51m 10.194.24.170 ip-10-194-24-90.ec2.internal <none> <none>
istio-system istio-egressgateway-7fcc58ddf7-dtx25 1/1 Running 0 95m 10.194.26.120 ip-10-194-26-252.ec2.internal <none> <none>
istio-system istio-ingressgateway-75f76c546b-vx2v6 1/1 Running 0 95m 10.194.24.8 ip-10-194-24-90.ec2.internal <none> <none>
istio-system istiod-85b6f86f94-vqfj2 1/1 Running 0 95m 10.194.25.155 ip-10-194-25-120.ec2.internal <none> <none>
kube-system aws-load-balancer-controller-54fdf64896-jzwsr 1/1 Running 0 23h 10.194.26.74 ip-10-194-26-252.ec2.internal <none> <none>
kube-system aws-load-balancer-controller-54fdf64896-qqt6d 1/1 Running 0 23h 10.194.24.242 ip-10-194-24-49.ec2.internal <none> <none>
kube-system aws-node-29kmc 1/1 Running 0 7d1h 10.194.24.90 ip-10-194-24-90.ec2.internal <none> <none>
kube-system aws-node-6d8ls 1/1 Running 1 7d1h 10.194.25.120 ip-10-194-25-120.ec2.internal <none> <none>
kube-system aws-node-6vrbg 1/1 Running 1 7d1h 10.194.26.252 ip-10-194-26-252.ec2.internal <none> <none>
kube-system aws-node-ldgxc 1/1 Running 1 7d1h 10.194.24.49 ip-10-194-24-49.ec2.internal <none> <none>
kube-system coredns-65bfc5645f-g86rx 1/1 Running 0 7d2h 10.194.24.207 ip-10-194-24-90.ec2.internal <none> <none>
kube-system coredns-65bfc5645f-xj9rl 1/1 Running 0 7d2h 10.194.24.69 ip-10-194-24-90.ec2.internal <none> <none>
kube-system efs-csi-controller-65fb886fd4-7slw6 3/3 Running 0 2d21h 10.194.24.90 ip-10-194-24-90.ec2.internal <none> <none>
kube-system efs-csi-controller-65fb886fd4-vcf9l 3/3 Running 0 2d21h 10.194.25.120 ip-10-194-25-120.ec2.internal <none> <none>
kube-system efs-csi-node-6t6v6 3/3 Running 0 2d21h 10.194.25.120 ip-10-194-25-120.ec2.internal <none> <none>
kube-system efs-csi-node-kxqfb 3/3 Running 0 2d21h 10.194.24.49 ip-10-194-24-49.ec2.internal <none> <none>
kube-system efs-csi-node-p8hzn 3/3 Running 0 2d21h 10.194.26.252 ip-10-194-26-252.ec2.internal <none> <none>
kube-system efs-csi-node-xxq9h 3/3 Running 0 2d21h 10.194.24.90 ip-10-194-24-90.ec2.internal <none> <none>
kube-system kube-proxy-78n7f 1/1 Running 0 7d1h 10.194.24.90 ip-10-194-24-90.ec2.internal <none> <none>
kube-system kube-proxy-cms7c 1/1 Running 0 7d1h 10.194.24.49 ip-10-194-24-49.ec2.internal <none> <none>
kube-system kube-proxy-h2t6n 1/1 Running 0 7d1h 10.194.26.252 ip-10-194-26-252.ec2.internal <none> <none>
kube-system kube-proxy-jkxnz 1/1 Running 0 7d1h 10.194.25.120 ip-10-194-25-120.ec2.internal <none> <none>
operators istio-operator-7cc8974d48-f2j2m 1/1 Running 0 14h 10.194.26.211 ip-10-194-26-252.ec2.internal <none> <none>
sample-alb sample-alb-8744f54f9-7w4cj 1/1 Running 0 23h 10.194.25.67 ip-10-194-25-120.ec2.internal <none> <none>
sample-alb sample-alb-8744f54f9-gs8f5 1/1 Running 0 23h 10.194.24.147 ip-10-194-24-49.ec2.internal <none> <none>
sample-alb sample-alb-8744f54f9-v6kgr 1/1 Running 0 23h 10.194.26.168 ip-10-194-26-252.ec2.internal <none> <none>
sample-elb sample-elb-69786b5f7d-d7nb4 1/1 Running 0 2d21h 10.194.26.178 ip-10-194-26-252.ec2.internal <none> <none>
sample-elb sample-elb-69786b5f7d-mw7jb 1/1 Running 0 2d21h 10.194.24.193 ip-10-194-24-49.ec2.internal <none> <none>
sample-elb sample-elb-69786b5f7d-tqz2s 1/1 Running 0 2d21h 10.194.25.96 ip-10-194-25-120.ec2.internal <none> <none>
sample-nlb sample-nlb-6cd5769dfb-n8dmd 1/1 Running 0 2d21h 10.194.25.198 ip-10-194-25-120.ec2.internal <none> <none>
sample-nlb sample-nlb-6cd5769dfb-qw8n4 1/1 Running 0 2d21h 10.194.24.132 ip-10-194-24-49.ec2.internal <none> <none>
sample-nlb sample-nlb-6cd5769dfb-t2nhp 1/1 Running 0 2d21h 10.194.26.18 ip-10-194-26-252.ec2.internal <none> <none>
% tf-run.sh apply
```

* example of the `tf-run.sh` steps

This is part of a larger cluster configuration, so at the end of the run it indicates another directory
to visit when done.

```console
% kubectl -n istio-system get secret | grep -iE "ca-secret|tls"
istio-ca-secret istio.io/ca-root 5 7d2h
nginx-cert kubernetes.io/tls 3 6d20h
root-secret kubernetes.io/tls 3 7d14h
% tf-run.sh list
* running action=plan
* START: tf-run.sh v1.1.2 start=1636563207 end= logfile=logs/run.plan.20211110.1636563207.log (not-created)
* reading from tf-run.data
* read 23 entries from tf-run.data
> list
** START: start=1636563207
* 1 COMMAND> tf-directory-setup.py -l none -f
* 2 COMMAND> setup-new-directory.sh
* 3 COMMAND> tf-init -upgrade
* 4 tf-plan -target=tls_private_key.ca
* 5 tf-plan -target=tls_cert_request.ca
* 6 tf-plan -target=null_resource.ca_root_cert
* 7 tf-plan -target=null_resource.ca_files
* 8 tf-plan -target=null_resource.ca_cert
* 9 tf-plan -target=local_file.ca_bundle_cert
* 10 COMMAND> tf-directory-setup.py -l s3
* 11 COMMENT> submit certs/*csr using command ouptut listed in apply to TCO for signing
* 12 STOP> once that is availabile, change cert_download to true
* 13 COMMAND> terraform taint null_resource.ca_cert
* 14 tf-plan -target=null_resource.ca_root_cert
* 15 tf-plan -target=null_resource.ca_files
* 16 tf-plan -target=null_resource.ca_cert
* 17 COMMENT> second run is to complete the steps
* 18 tf-plan -target=null_resource.ca_root_cert
* 19 tf-plan -target=null_resource.ca_files
* 20 tf-plan -target=null_resource.ca_cert
* 21 tf-plan
* 22 COMMENT> run: git-secret add certs/*.key; git-secret hide
* 23 COMMENT> be sure to add all files to git, and be sure to commit -a to get .gitsecret/ changes
** END: start=1636563207 end=1636563207 elapsed=0 logfile=logs/run.plan.20211110.1636563207.log (not-created)
```

kubectl get pods --all-namespaces -o wide|grep -i cert
cert-manager cert-manager-7fcbc79fc5-xwt4s 1/1 Running 0 7d22h 10.194.24.138 ip-10-194-24-49.ec2.internal <none> <none>
cert-manager cert-manager-cainjector-6b7f4575f4-xpgnc 1/1 Running 0 7d22h 10.194.24.56 ip-10-194-24-49.ec2.internal <none> <none>
cert-manager cert-manager-webhook-6cd54b96fc-rvld4 1/1 Running 0 7d22h 10.194.24.170 ip-10-194-24-90.ec2.internal <none> <none>
It is highly recommended to use the `tf-run.sh` approach.

$ kubectl -n cert-manager get secrets
NAME TYPE
DATA AGE
ca-key-pair Opaque
2 5m2s
...
$ kubectl get clusterissuer
NAME READY AGE
clusterissuer True 5m36s
## Terraform Manual

0 comments on commit 5279dcd

Please sign in to comment.