complete refactor, add s3

README.md

@@ -1,45 +1,90 @@
 # AWS Service Catalog Terraform Module
 
-This module provides a standardized way to provision and manage AWS Service Catalog products using Terraform.
+This module provides simplified Terraform wrappers for provisioning AWS Service Catalog products with strongly-typed variables, validation, and consistent patterns.
+
+## Overview
+
+The module uses a layered architecture:
+- **product module** - Base module handling Service Catalog provisioning (internal use)
+- **ec2 module** - EC2-specific wrapper with instance parameters and VPC networking
+- **s3 module** - S3-specific wrapper with bucket parameters and data classification
+
+Each wrapper module provides:
+- Product-specific variables with validation
+- Parameter mapping to CloudFormation templates
+- Consistent tagging strategy
+- Comprehensive documentation and examples
 
 ## Features
 
-- **Product Provisioning**: Provision EC2, RDS, ECS, and S3 products from existing Service Catalog portfolios
-- **Lifecycle Management**: Update and terminate provisioned products
-- **Service Actions**: Execute service actions (start, stop, reboot, etc.) on provisioned products
-- **Automatic Discovery**: Lookup portfolios and products by naming patterns
-- **Stack Outputs**: Access CloudFormation stack outputs from provisioned products
+- ✅ **EC2 Provisioning** - Provision EC2 instances with automated VPC/subnet discovery
+- ✅ **S3 Provisioning** - Provision S3 buckets with title data classification
+- ✅ **Lifecycle Management** - Create, update, and destroy provisioned products
+- ✅ **Stack Outputs** - Access CloudFormation outputs from provisioned products
+- ✅ **Variable Validation** - Built-in validation for parameters (instance types, bucket names, etc.)
+- ✅ **Comprehensive Examples** - Working examples with terraform.tfvars.example files
 
 ## Module Structure
 
-- `ec2/` - Provision EC2 instances from Service Catalog
-- `ec2-actions/` - Execute service actions on EC2 provisioned products
-- `rds/` - Provision RDS databases from Service Catalog
-- `rds-actions/` - Execute service actions on RDS provisioned products
-- `ecs/` - Provision ECS services from Service Catalog
-- `ecs-actions/` - Execute service actions on ECS provisioned products
-- `s3/` - Provision S3 buckets from Service Catalog
-- `s3-actions/` - Execute service actions on S3 provisioned products
-- `common/` - Shared components and utilities
+```
+aws-servicecatalog/
+├── modules/
+│   ├── product/         # Base Service Catalog provisioning module (internal)
+│   ├── ec2/            # EC2 instance provisioning wrapper
+│   └── s3/             # S3 bucket provisioning wrapper
+└── examples/
+    ├── ec2/linux/      # EC2 Linux instance example
+    └── s3/             # S3 bucket example
+```
+
+### Currently Implemented
+
+- ✅ **modules/ec2/** - Provision EC2 instances from Service Catalog
+- ✅ **modules/s3/** - Provision S3 buckets from Service Catalog
+- ✅ **modules/product/** - Base module for Service Catalog provisioning (internal use only)
+
+### Planned/Future Enhancements
+
+- ⏳ **RDS module** - Provision RDS databases
+- ⏳ **ECS module** - Provision ECS services
+- ⏳ **Service actions** - Execute service actions (start, stop, reboot)
 
 ## Usage
 
-See individual submodule README files for detailed usage examples.
+See individual module README files for detailed documentation:
+- [EC2 Module Documentation](modules/ec2/README.md)
+- [S3 Module Documentation](modules/s3/README.md)
+- [Product Module Documentation](modules/product/README.md)
 
-### Basic EC2 Example
+### EC2 Example
 
 ```hcl
 module "ec2_instance" {
-  source = "path/to/aws-servicecatalog/ec2"
+  source = "path/to/aws-servicecatalog/modules/ec2"
 
+  # Product identity
   provisioned_product_name = "my-web-server"
-  portfolio_name_pattern   = "edl-portfolio"
-  product_name_pattern     = "linux-product"
 
+  # EC2-specific parameters
+  instance_type   = "t3.small"
+  os_name         = "RHEL9"
+  requires_backup = "yes"
+  power_schedule  = "Always_On"
+
+  # Networking (auto-discovers VPC/subnets by name tag)
+  vpc_name     = "my-vpc"
+  subnets_name = "*-apps-*"
+
+  # Project/creator information
+  project_name   = "myorg_myapp_prod-123456789012"
+  creator        = "jsmith123"
+  contact_email  = "jsmith@example.com"
+  inc_poc_email  = "team@example.com"
+
+  # Additional parameters
   parameters = {
-    InstanceType = "t3.medium"
-    KeyName      = "my-key"
-    SubnetId     = "subnet-12345"
+    NameTag            = "web-server-01"
+    SecurityGroupNames = "it-linux-base,web-servers"
   }
 
   tags = {
@@ -49,7 +94,63 @@ module "ec2_instance" {
 }
 ```
 
+### S3 Example
+
+```hcl
+module "s3_bucket" {
+  source = "path/to/aws-servicecatalog/modules/s3"
+
+  # Product identity
+  provisioned_product_name = "my-data-bucket"
+
+  # S3-specific parameters
+  bucket_name              = "my-data-lake"
+  enable_bucket_versioning = false
+  title_data               = ["no_title"]
+  project_role             = "data-engineering"
+
+  # Project/creator information
+  project_name   = "myorg_myapp_prod-123456789012"
+  creator        = "jsmith123"
+  contact_email  = "jsmith@example.com"
+  inc_poc_email  = "team@example.com"
+
+  tags = {
+    Environment = "production"
+    DataClass   = "internal"
+  }
+}
+```
+
 ## Requirements
 
 - Terraform >= 1.0
-- AWS Provider >= 5.0
+- AWS Provider >= 6.0
+
+## Examples
+
+Complete working examples with terraform.tfvars.example files are available in:
+- [examples/ec2/linux/](examples/ec2/linux/) - EC2 Linux instance provisioning
+- [examples/s3/](examples/s3/) - S3 bucket provisioning
+
+## Architecture
+
+This module follows a base + specialized wrapper pattern:
+
+1. **product module** provides generic Service Catalog provisioning
+2. **ec2/s3 modules** add product-specific parameters and data sources
+3. Each wrapper handles parameter mapping to CloudFormation format
+4. Networking data sources are only in modules that need them (EC2, not S3)
+
+## Contributing
+
+When adding new product modules:
+1. Create a new wrapper module under `modules/`
+2. Call the `product` module with product-specific parameters
+3. Add product-specific data sources if needed (e.g., VPC lookups for EC2)
+4. Create comprehensive README with examples
+5. Add terraform.tfvars.example to examples/
+
+## License
+
+Internal use only.

examples/ec2/linux/census-rhel-instance.tf

@@ -4,56 +4,46 @@
 # Aligns with the 2-0-2.yaml CloudFormation product template.
 
 module "ec2_instance" {
-  source = "../../../ec2"
+  source = "../../../modules/ec2"
 
+  # === Product Identity ===
   provisioned_product_name = var.provisioned_product_name
-  portfolio_name_pattern   = var.portfolio_name_pattern
-  product_name_pattern     = var.product_name_pattern
 
-  # CloudFormation parameters passed to the Service Catalog product template
+  # === Required EC2-Specific Parameters ===
+  instance_type   = var.instance_type
+  os_name         = var.os_name
+  requires_backup = var.requires_backup
+  power_schedule  = var.power_schedule
+
+  # === Required Project/Creator Parameters ===
+  project_name    = var.project_name
+  creator         = var.creator_jbid
+  contact_email   = var.contact_email
+  inc_poc_email   = var.incident_poc_email
+
+  # === Network Configuration (looks up VPC/subnets by name tag) ===
+  vpc_name           = var.vpc_name
+  subnets_name       = var.subnets_name
+  availability_zones = var.availability_zones
+
+  # === Additional CloudFormation Parameters (optional) ===
+  # For parameters not exposed as module variables, pass them here
   parameters = {
-    # === Required Census Project Parameters ===
-    ProjectName    = var.project_name
-    VpcId          = var.vpc_id
-    AZName         = var.availability_zone
-    NameTag        = var.instance_hostname
-    InstanceType   = var.instance_type
-    OSName         = var.os_name
-    Creator        = var.creator_jbid
-    ContactEmail   = var.contact_email
-    IncPocEmail    = var.incident_poc_email
-
-    # === Optional EC2 Configuration ===
-    pDescription      = var.instance_description
-    VolumeAppsSize    = tostring(var.volume_apps_size)
-    SecurityGroupNames = join(",", var.security_group_names)
-
-    # === Extra Volumes (optional) ===
-    # Uncomment below and populate var.extra_volumes to add additional volumes:
-    # Volume1Mount  = try(var.extra_volumes[0].mount, "")
-    # Volume1Size   = try(tostring(var.extra_volumes[0].size), "0")
-    # Volume2Mount  = try(var.extra_volumes[1].mount, "")
-    # Volume2Size   = try(tostring(var.extra_volumes[1].size), "0")
-    # Volume3Mount  = try(var.extra_volumes[2].mount, "")
-    # Volume3Size   = try(tostring(var.extra_volumes[2].size), "0")
-
-    # === Compliance and Operational Tags ===
-    FISMAID        = var.fisma_id
-    TitleData      = join(",", var.title_data_types)
-    RequiresBackup = var.requires_backup
-    PowerSchedule  = var.power_schedule
-    ProjectRole    = var.project_role
-    MapMigrated    = var.map_migrated
+    NameTag             = var.instance_hostname
+    pDescription        = var.instance_description
+    VolumeAppsSize      = tostring(var.volume_apps_size)
+    SecurityGroupNames  = join(",", var.security_group_names)
+    # ProjectRole and FISMAID will be passed from module automatically
+    TitleData           = join(",", var.title_data_types)
+    MapMigrated         = var.map_migrated
   }
 
-  enable_deletion_protection = var.enable_deletion_protection
-
+  # === Tags ===
   tags = merge(
     {
       Environment    = var.environment
       ManagedBy      = "Terraform"
       ServiceCatalog = "true"
-      Creator        = var.creator_jbid
     },
     var.additional_tags
   )

examples/ec2/linux/main.tf

@@ -1,13 +1 @@
-# terraform {
-#   required_version = ">= 1.0"
-#   required_providers {
-#     aws = {
-#       source  = "hashicorp/aws"
-#       version = ">= 5.0"
-#     }
-#   }
-# }
-
-# provider "aws" {
-#   region = var.region
-# }
+# Module is defined in census-rhel-instance.tf

examples/ec2/linux/outputs.tf

@@ -13,12 +13,6 @@ output "provisioned_product_status" {
   value       = module.ec2_instance.provisioned_product_status
 }
 
-output "stack_outputs" {
-  description = "CloudFormation stack outputs from the provisioned product (instance details, IPs, etc.)"
-  value       = module.ec2_instance.stack_outputs
-  sensitive   = true
-}
-
 output "instance_configuration_summary" {
   description = "Summary of the EC2 instance configuration"
   value = {
@@ -31,6 +25,5 @@ output "instance_configuration_summary" {
     security_groups   = var.security_group_names
     backup_enabled    = var.requires_backup == "yes"
     power_schedule    = var.power_schedule
-    deletion_protected = var.enable_deletion_protection
   }
 }

examples/ec2/linux/terraform.tfvars.example

@@ -8,35 +8,43 @@
 region      = "us-east-1"
 environment = "Production"
 
-# Service Catalog Configuration (typically don't need to change these)
+# === SERVICE CATALOG CONFIGURATION ===
+
+# Name for the provisioned product (unique identifier in your account)
 provisioned_product_name = "census-rhel-instance"
-portfolio_name_pattern   = "csvd-portfolio"
-product_name_pattern     = "linux-product"
 
 # === EC2 INSTANCE CONFIGURATION ===
 
 # Instance Type - Options: t3.nano, t3.micro, t3.small (default), t3.medium, c5.large, m5.large, etc.
 instance_type = "t3a.small"
 
-# Operating System - Options: RHEL9 (default), RHEL8
+# Operating System - Options: RHEL8, RHEL9 (default)
 os_name = "RHEL9"
 
-# Instance Hostname - Must be 3-16 characters, lowercase alphanumeric with hyphens/underscores
+# Instance Hostname (local name, no domain) - Must be 3-16 characters, lowercase alphanumeric with hyphens/underscores
 instance_hostname = "app-mcm-01"
 
 # Instance Description
 instance_description = "Census RHEL instance provisioned via Terraform"
 
 # === NETWORK CONFIGURATION ===
 
-# VPC ID (REQUIRED - must be in the same account as the project)
-vpc_id = "vpc-1234567890abcdef0"
+# VPC Configuration - Choose ONE approach:
+# Approach 1: Use vpc_name to auto-discover by name tag (recommended)
+vpc_name = "csvd-vpc"
+
+# Approach 2 (ALTERNATIVE): Use vpc_id directly
+# vpc_id = "vpc-1234567890abcdef0"
+
+# Subnet Configuration - Name pattern for subnet auto-discovery
+# Default pattern "*-apps-*" matches subnets tagged with names like "csvd-apps-primary"
+subnets_name = "*-apps-*"
 
-# Availability Zone (REQUIRED)
-availability_zone = "us-east-1a"
+# Availability Zones - Leave empty to use all available, or specify list for filtering
+# Examples: ["us-east-1a"], ["us-east-1a", "us-east-1b"]
+availability_zones = []
 
-# Security Groups (it-linux-base is required and included by default)
-# Add additional security groups as needed
+# Security Groups - List of security group names to attach (it-linux-base is automatically included)
 security_group_names = ["it-linux-base", "ois-scanner"]
 
 # === PROJECT CONFIGURATION ===
@@ -82,7 +90,8 @@ incident_poc_email = "matthew.c.morgan@census.gov"
 fisma_id = "OCIO_CSVD"
 
 # Title Data Types - Data classification for this instance
-# Options: no_title, title_2, title_5, title_13, title_14, title_21, title_25, title_26, etc.
+# Options: no_title, title_2, title_5, title_13, title_14, title_21, title_25, title_26, 
+#          title_26_pending, title_health_data, title_pii, title_pub_1075
 title_data_types = ["no_title"]
 
 # Backup Requirement - Whether to add instance to enterprise backup
@@ -98,6 +107,12 @@ requires_backup = "no"
 #   - Weekday_Business_Hours_9-5
 #   - Weekday_Core_Hours_7-7
 #   - Weekday_Hours_0600-1900
+power_schedule = "Always_On"
+
+# MAP Migrated - MAP program migration tag for cost credits (5-25% discount)
+# Example: "migMSDNVLIJDZ: On-Prem Server, Storage, Networking"
+map_migrated = ""
+
 power_schedule = "Weekday_Core_Hours_7-7"
 
 # MAP Migration Tag - For AWS Migration Accelerator Program credits