Skip to content

Commit

Permalink
* 2.11.0 -- 2024-12-11
Browse files Browse the repository at this point in the history 
  - flowlog
    - added field enable_s3 so we can stop creating S3 flow logs (and go only to data firehose)
    - remove splunk resources, as splunk is gone from our environment
  • Loading branch information
@badra001
badra001 committed Dec 11, 2024
1 parent db4f59a commit e403cc9
Show file tree
Hide file tree
Showing 5 changed files with 174 additions and 44 deletions.
5 changes: 5 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -430,3 +430,8 @@
* 2.10.8 -- 2024-11-18
- vpc-interface-endpoint
- add zone_name, zone_id to output

* 2.11.0 -- 2024-12-11
- flowlog
- added field enable_s3 so we can stop creating S3 flow logs (and go only to data firehose)
- remove splunk resources, as splunk is gone from our environment
2 changes: 1 addition & 1 deletion common/version.tf
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
locals {
_module_version = "2.10.8"
_module_version = "2.11.0"
_module_names = {
"_main_" = "aws-vpc-setup"

Expand Down
116 changes: 116 additions & 0 deletions flowlogs/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -114,3 +114,119 @@ No modules.
| Name | Description |
|------|-------------|
| <a name="output_kinesis_flowlog_arn"></a> [kinesis\_flowlog\_arn](#output\_kinesis\_flowlog\_arn) | VPC Flowlog Kinesis stream ARN |

<!-- BEGIN_TF_DOCS -->
# About aws-vpc-setup :: flowlogs

This submodule creates VPC flow logs. It creates one for private and one for public subnets. Run this once
per VPC. You will need to run the flow logs role once before (aws-vpc-setup/flowlogs-role).

This also creates cloudwatch logs and cloudwatch streams (Kinesis). It no longer creates a configuration file in `setup/`
to be used in Splunk. Distribute this configuration to the splunk team for flowlog ingestion, as we have
removed Splunk from our environment.

```
# setup/aws_kinesis_tasks.lgs-vpc2-dice-dev.conf
[ma6-gov-flowlogs-us-gov-east-1_lg-vpc2-dice-dev]
account = ma6-gov
format = CloudWatchLogs
index = aws_vpc_flow_logs
region = us-gov-east-1
sourcetype = aws:cloudwatchlogs:vpcflow
init_stream_position = LATEST
stream_names = lgs-vpc2-dice-dev
```

# Usage

```hcl
module "flowlogs" {
source = "git@github.e.it.census.gov:terraform-modules/aws-vpc-setup.git//flowlogs"
vpc_id = var.vpc_id
vpc_full_name = var.vpc_full_name
account_alias = "ma6-gov"
public_subnets_ids = module.subnets.public_subnets_ids
private_subnets_ids = module.subnets.private_subnets_ids
flowlog_bucket_arn = data.terraform_remote_state.common.infrastructure_east.flowlogs_arn
flowlog_role_arn = data.terraform_remote_state.common.outputs.role_flowlogs_arn
## optional
# public_subnet_ids = [ for s in module.subnets.public_subnets_ids : s.id ]
# private_subnet_ids = [ for s in module.subnets.private_subnets_ids : s.id ]
# use_flowlog_custom_format = true
# flowlog_custom_format = "${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status} ${flow-direction} ${traffic-path}"
# enable_s3 = false
tags = {}
}
```

## Requirements

| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.13 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.66.0 |
| <a name="requirement_ldap"></a> [ldap](#requirement\_ldap) | >= 0.5.4 |
| <a name="requirement_local"></a> [local](#requirement\_local) | >= 1.0.0 |
| <a name="requirement_null"></a> [null](#requirement\_null) | >= 3.0 |
| <a name="requirement_random"></a> [random](#requirement\_random) | >= 3.0 |
| <a name="requirement_template"></a> [template](#requirement\_template) | >= 2.0 |

## Providers

| Name | Version |
|------|---------|
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 3.66.0 |

## Modules

No modules.

## Resources

| Name | Type |
|------|------|
| [aws_cloudwatch_log_group.flowlog](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_log_group) | resource |
| [aws_cloudwatch_log_subscription_filter.flowlog](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_log_subscription_filter) | resource |
| [aws_flow_log.flowlog](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/flow_log) | resource |
| [aws_flow_log.flowlog_cloudwatch](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/flow_log) | resource |
| [aws_flow_log.flowlog_public](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/flow_log) | resource |
| [aws_kinesis_stream.flowlog](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/kinesis_stream) | resource |
| [aws_arn.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/arn) | data source |
| [aws_caller_identity.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/caller_identity) | data source |
| [aws_iam_account_alias.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_account_alias) | data source |
| [aws_region.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/region) | data source |

## Inputs

| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| <a name="input_account_alias"></a> [account\_alias](#input\_account\_alias) | AWS Account Alias (default: will pull from current account\_alias) | `string` | `""` | no |
| <a name="input_account_id"></a> [account\_id](#input\_account\_id) | AWS Account ID (default: will pull from current user) | `string` | `""` | no |
| <a name="input_enable_kinesis_stream"></a> [enable\_kinesis\_stream](#input\_enable\_kinesis\_stream) | Flag to enable or disable creation of kineis stream for Splunk | `bool` | `false` | no |
| <a name="input_enable_s3"></a> [enable\_s3](#input\_enable\_s3) | Flag to enable or disable creation of flowlog to S3 | `bool` | `false` | no |
| <a name="input_flowlog_bucket_arn"></a> [flowlog\_bucket\_arn](#input\_flowlog\_bucket\_arn) | S3 Bucket to hold the VPC flowlogs | `string` | n/a | yes |
| <a name="input_flowlog_custom_format"></a> [flowlog\_custom\_format](#input\_flowlog\_custom\_format) | If use\_flowlog\_custom\_format defined, use the custom format listed here. See https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html | `string` | `"${account-id} ${vpc-id} ${subnet-id} ${interface-id} ${start} ${end} ${pkt-srcaddr} ${srcaddr} ${srcport} ${pkt-dstaddr} ${dstaddr} ${dstport} ${protocol} ${packets} ${bytes} ${tcp-flags} ${log-status} ${action} ${type} ${flow-direction} ${traffic-path}"` | no |
| <a name="input_flowlog_role_arn"></a> [flowlog\_role\_arn](#input\_flowlog\_role\_arn) | IAM Role with proper permissions to allow writing VPC flowlogs to cloudwatch logs and streamss | `string` | n/a | yes |
| <a name="input_override_prefixes"></a> [override\_prefixes](#input\_override\_prefixes) | Override built-in prefixes by component. This should be used primarily for common infrastructure things | `map(string)` | `{}` | no |
| <a name="input_private_subnet_ids"></a> [private\_subnet\_ids](#input\_private\_subnet\_ids) | List of private subnet IDs (not objects) | `list(string)` | `[]` | no |
| <a name="input_private_subnets_ids"></a> [private\_subnets\_ids](#input\_private\_subnets\_ids) | List of private subnet objects including: subnet, label, availability\_zone, id | <pre>list(object({<br> subnet = string<br> label = string<br> availability_zone = string<br> id = string<br> tags = optional(map(string))<br> }))</pre> | `[]` | no |
| <a name="input_public_subnet_ids"></a> [public\_subnet\_ids](#input\_public\_subnet\_ids) | List of public subnet IDs (not objects) | `list(string)` | `[]` | no |
| <a name="input_public_subnets_ids"></a> [public\_subnets\_ids](#input\_public\_subnets\_ids) | List of public subnet objects including: subnet, label, availability\_zone, id | <pre>list(object({<br> subnet = string<br> label = string<br> availability_zone = string<br> id = string<br> tags = optional(map(string))<br> }))</pre> | `[]` | no |
| <a name="input_retention_in_days"></a> [retention\_in\_days](#input\_retention\_in\_days) | Number of days to keep cloudwatch logs (default is 180). See the documentation for available values. | `number` | `180` | no |
| <a name="input_tags"></a> [tags](#input\_tags) | AWS Tags to apply to appropriate resources (S3, KMS). Do not include safeguard tags here, use the data\_safeguard field for such things. | `map(string)` | `{}` | no |
| <a name="input_use_flowlog_custom_format"></a> [use\_flowlog\_custom\_format](#input\_use\_flowlog\_custom\_format) | Flag to control the use of a custom format. See https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html | `bool` | `false` | no |
| <a name="input_vpc_environment"></a> [vpc\_environment](#input\_vpc\_environment) | VPC environment purpose (infrastructure, common, shared, dev, stage, ite, prod) | `string` | `null` | no |
| <a name="input_vpc_full_name"></a> [vpc\_full\_name](#input\_vpc\_full\_name) | VPC full name component (vpc{index}-{vpc\_name}) | `string` | `null` | no |
| <a name="input_vpc_id"></a> [vpc\_id](#input\_vpc\_id) | VPC ID | `string` | n/a | yes |
| <a name="input_vpc_index"></a> [vpc\_index](#input\_vpc\_index) | VPC index number (integer starting at 1) | `number` | `null` | no |
| <a name="input_vpc_name"></a> [vpc\_name](#input\_vpc\_name) | VPC name component used through the VPC descrbing its purpose (ex: dice-dev) | `string` | `null` | no |
| <a name="input_vpc_short_name"></a> [vpc\_short\_name](#input\_vpc\_short\_name) | VPC short name component (vpc{index}) | `string` | `null` | no |

## Outputs

| Name | Description |
|------|-------------|
| <a name="output_kinesis_flowlog_arn"></a> [kinesis\_flowlog\_arn](#output\_kinesis\_flowlog\_arn) | VPC Flowlog Kinesis stream ARN |
<!-- END_TF_DOCS -->
89 changes: 46 additions & 43 deletions flowlogs/main.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,9 @@
* This submodule creates VPC flow logs. It creates one for private and one for public subnets. Run this once
* per VPC. You will need to run the flow logs role once before (aws-vpc-setup/flowlogs-role).
*
* This also creates cloudwatch logs and cloudwatch streams (Kinesis), and drops a configuration file in `setup/`
* to be used in Splunk. Distribute this configuration to the splunk team for flowlog ingestion.
* This also creates cloudwatch logs and cloudwatch streams (Kinesis). It no longer creates a configuration file in `setup/`
* to be used in Splunk. Distribute this configuration to the splunk team for flowlog ingestion, as we have
* removed Splunk from our environment.
*
* ```
* # setup/aws_kinesis_tasks.lgs-vpc2-dice-dev.conf
Expand Down Expand Up @@ -37,6 +38,7 @@
* # private_subnet_ids = [ for s in module.subnets.private_subnets_ids : s.id ]
* # use_flowlog_custom_format = true
* # flowlog_custom_format = "${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status} ${flow-direction} ${traffic-path}"
* # enable_s3 = false
*
* tags = {}
* }
Expand Down Expand Up @@ -65,7 +67,7 @@ locals {
# use s3 for flow logs, create two, one for the whole vpc, and one for just public
#---
resource "aws_flow_log" "flowlog_public" {
for_each = toset(local.public_ids)
for_each = var.enable_s3 ? toset(local.public_ids) : toset([])
log_destination = format("%v/%v-%v/", var.flowlog_bucket_arn, var.vpc_full_name, "public")
log_destination_type = "s3"
log_format = var.use_flowlog_custom_format ? var.flowlog_custom_format : null
Expand All @@ -82,6 +84,7 @@ resource "aws_flow_log" "flowlog_public" {

# whole VPC
resource "aws_flow_log" "flowlog" {
count = var.enable_s3 ? 1 : 0
log_destination = format("%v/%v/", var.flowlog_bucket_arn, var.vpc_full_name)
log_destination_type = "s3"
log_format = var.use_flowlog_custom_format ? var.flowlog_custom_format : null
Expand Down Expand Up @@ -149,43 +152,43 @@ resource "aws_cloudwatch_log_subscription_filter" "flowlog" {
distribution = "ByLogStream"
}

#---
# generate splunk inputs file
#---
data "template_file" "splunk_flowlog" {
template = file("${path.module}/templates/aws_kinesis_tasks.conf.tpl")
vars = {
account_id = local.account_id
account_alias = local.account_alias
region = local.region
flowlog_name = aws_cloudwatch_log_group.flowlog.name
flowlog_stream_name = local.flowlog_stream_name
vpc_full_name = var.vpc_full_name
}
}

resource "null_resource" "splunk_flowlog" {
triggers = {
filename = format("aws_kinesis_tasks.%v-%v.%v.%v.conf", local.account_id, local.account_alias, local.region, local.flowlog_stream_name)
directory = format("%v/setup", path.root)
}
provisioner "local-exec" {
command = "test -d ${self.triggers.directory} || mkdir ${self.triggers.directory}"
}

# provisioner "local-exec" {
# working_dir = path.root
# command = "test -d setup || mkdir setup"
# }
# provisioner "local-exec" {
# working_dir = "${path.root}/setup"
# command = "echo '${data.template_file.splunk_flowlog_tasks_flowlog.rendered}' > aws_kinesis_tasks.${local.flowlog_stream_name}.conf"
# }
}

resource "local_file" "splunk_flowlog" {
count = var.enable_kinesis_stream ? 1 : 0
content = data.template_file.splunk_flowlog.rendered
file_permission = "0644"
filename = format("%v/%v", null_resource.splunk_flowlog.triggers.directory, null_resource.splunk_flowlog.triggers.filename)
}
## #---
## # generate splunk inputs file
## #---
## data "template_file" "splunk_flowlog" {
## template = file("${path.module}/templates/aws_kinesis_tasks.conf.tpl")
## vars = {
## account_id = local.account_id
## account_alias = local.account_alias
## region = local.region
## flowlog_name = aws_cloudwatch_log_group.flowlog.name
## flowlog_stream_name = local.flowlog_stream_name
## vpc_full_name = var.vpc_full_name
## }
## }
##
## resource "null_resource" "splunk_flowlog" {
## triggers = {
## filename = format("aws_kinesis_tasks.%v-%v.%v.%v.conf", local.account_id, local.account_alias, local.region, local.flowlog_stream_name)
## directory = format("%v/setup", path.root)
## }
## provisioner "local-exec" {
## command = "test -d ${self.triggers.directory} || mkdir ${self.triggers.directory}"
## }
##
## # provisioner "local-exec" {
## # working_dir = path.root
## # command = "test -d setup || mkdir setup"
## # }
## # provisioner "local-exec" {
## # working_dir = "${path.root}/setup"
## # command = "echo '${data.template_file.splunk_flowlog_tasks_flowlog.rendered}' > aws_kinesis_tasks.${local.flowlog_stream_name}.conf"
## # }
## }
##
## resource "local_file" "splunk_flowlog" {
## count = var.enable_kinesis_stream ? 1 : 0
## content = data.template_file.splunk_flowlog.rendered
## file_permission = "0644"
## filename = format("%v/%v", null_resource.splunk_flowlog.triggers.directory, null_resource.splunk_flowlog.triggers.filename)
## }
6 changes: 6 additions & 0 deletions flowlogs/variables.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -55,3 +55,9 @@ variable "flowlog_custom_format" {
# default = "${version} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status}"
default = "$${account-id} $${vpc-id} $${subnet-id} $${interface-id} $${start} $${end} $${pkt-srcaddr} $${srcaddr} $${srcport} $${pkt-dstaddr} $${dstaddr} $${dstport} $${protocol} $${packets} $${bytes} $${tcp-flags} $${log-status} $${action} $${type} $${flow-direction} $${traffic-path}"
}

variable "enable_s3" {
description = "Flag to enable or disable creation of flowlog to S3"
type = bool
default = false
}

0 comments on commit e403cc9

Please sign in to comment.