From 4a6501d926fa0bcdd06a725b15f6932cd00434b7 Mon Sep 17 00:00:00 2001 From: Braxton Richard Lollis Date: Wed, 4 Sep 2024 15:45:38 -0600 Subject: [PATCH] Update README.md Changes from Davids Feedback, please review. --- README.md | 37 +++++++++++++++++++++---------------- 1 file changed, 21 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index 5790038..2b7f917 100644 --- a/README.md +++ b/README.md @@ -14,15 +14,15 @@ ________________________________________ 2. Pipeline Flow Source Phase: 1. Pull Configurations: -o The pipeline pulls Terraform configurations from the aws-image-pipeline repository. -o It fetches Packer configurations from the linux-image-pipeline repository. +o The pipeline configuration is initiated by the aws-image-pipeline repository, which calls the necessary Terraform modules responsible for building the pipelines. The actual configurations are pulled from other specified repositories. +o The pipeline fetches Packer configurations from multiple repositories, including linux-image-pipeline, docker-image-pipeline, and windows-image-pipeline. o The Ansible playbooks are retrieved from image-pipeline-ansible-playbooks. o The Goss tests are pulled from image-pipeline-goss-testing. Build Phase: 1. Launch the EC2 Instance: -o Packer, defined in the build.pkr.hcl file, starts the build process by launching an EC2 instance using a pre-defined AMI. -o The instance type, subnet, security groups, and SSH user credentials are sourced from AWS Parameter Store. +o Packer launches an EC2 instance using an AMI specified in the terraform-aws-image-pipeline module, rather than a "predefined" AMI. +o Instance credentials are securely stored in AWS Secrets Manager, not in Parameter Store. Both AWS Parameter Store and Secrets Manager are populated by the terraform-aws-image-pipeline module based on the provided parameters. 2. Instance Configuration: o Ansible playbooks from the image-pipeline-ansible-playbooks repo are run to configure the instance. o Example configurations include installing required packages, setting up services, and applying security hardening measures. @@ -37,7 +37,7 @@ o Tests check whether all services are running, the necessary packages are insta o Tests Passed:  If the Goss tests pass, no further action is required regarding the AMI. The AMI is not destroyed, and no action is taken to remove it. o Tests Failed or Troubleshooting Enabled: - If tests fail, the EC2 instance used for testing is destroyed via Terraform. + The EC2 instance used for testing is destroyed via Terraform after the tests are completed, regardless of whether they pass or fail.  If troubleshooting is enabled or the tests fail, the AMI is deregistered from AWS using the following commands: false || /bin/terraform destroy -var project_name=rhel-image-pipeline-demo -var goss_directory=${CODEBUILD_SRC_DIR_SourceGossOutput} -auto-approve @@ -72,9 +72,10 @@ provisioner "ansible" { Steps to Update or Add Playbooks: 1. Modify or add your playbooks to the image-pipeline-ansible-playbooks repo. -2. Ensure the correct playbook name is referenced in AWS Parameter Store under the parameter for playbook (e.g., /image-pipeline//playbook). -3. Update the rhel.tf (or equivalent) to point to the correct playbook by modifying the playbook parameter: -playbook = "your-new-playbook.yaml" +2. Update the playbook parameter in the Terraform configuration (e.g., [rhel.tf] or equivalent) to point to the correct playbook, by modifying the playbook parameter: playbook = "your-new-playbook.yaml". +3. The Terraform configuration will automatically update the value in AWS Parameter Store when applying the changes, ensuring the correct playbook is referenced during the pipeline execution. + +Note: This is the only step required when pointing to new playbooks. There is no need to manually modify AWS Secrets Manager or Parameter Store, as the Terraform configuration handles these updates automatically. ________________________________________ 2. How to Add/Change/Update Goss Tests If you need to add or update the Goss tests used to validate the AMI, follow these steps: @@ -82,7 +83,12 @@ Where Do Goss Tests Go? • Goss tests are stored in the image-pipeline-goss-testing repository. • The relevant test files are located in the goss-files/ directory, where you can add or modify .yaml files to include new tests. How Are They Integrated? -• The Goss tests are integrated through the linux.tf file in the image-pipeline-goss-testing repository. +• The Goss tests are integrated through the relevant Terraform configuration files, depending on the pipeline. +• For Linux pipelines, they are integrated via the [linux.tf]) file in the image-pipeline-goss-testing repository. +• For Windows pipelines, Goss tests are integrated via the [windows.tf] file in the image-pipeline-goss-testing repository. +For example, in the [windows.tf]) file: +• The goss_profile parameter is used to specify the set of tests to run during the validation phase: goss_profile = "windows-base-test". +• The Goss repository (goss_repo) is referenced, and the appropriate Goss tests are pulled from this repository during the pipeline execution. • The goss_profile is passed in as a variable from the rhel.tf in the aws-image-pipeline repository and is tied to a specific set of Goss tests in the goss-files/ directory. goss_profile = "rhel-base-test" @@ -92,8 +98,10 @@ Steps to Update or Add Goss Tests: goss_profile = "your-new-test" 3. The pipeline will automatically reference this new set of tests during the test phase when creating a new AMI. Consideration: Parking Goss Tests in the Ansible Playbooks -• While Goss tests are currently separated from the Ansible playbooks, we did consider parking Goss tests as part of the playbooks for simplicity. However, separating the two ensures that testing is isolated from configuration, providing clearer validation steps and better modularity. -• This separation also allows more flexibility in testing different configurations without modifying the Ansible playbooks directly. +• While Goss tests are currently separated from the Ansible playbooks, we did consider parking Goss tests as part of the playbooks for simplicity. However, separating the two provides several key benefits: +• Isolation of Testing and Configuration: By keeping Goss tests separate, we ensure that testing is isolated from configuration. This provides clearer validation steps and better modularity in the pipeline. +• Flexibility in Testing: This separation allows more flexibility in testing different configurations without the need to modify the Ansible playbooks directly. +• Separation of Concerns: It also enables different teams to control the Goss and Packer repositories independently. For example, security teams can design and • maintain Goss tests to ensure their specific concerns are addressed, without needing to alter the configurations managed by other teams. This division of responsibilities enhances security and ensures that each team can focus on their core expertise. ________________________________________ 3. How to Add a New Pipeline Type (e.g., ARM Amazon Linux or ARM Windows Instance) If you want to create a new pipeline type, such as one for ARM-based Amazon Linux or Windows instances, you need to follow these steps: @@ -101,10 +109,7 @@ Steps to Add a New Pipeline Type: 1. Create a New Packer Configuration: o In the linux-image-pipeline repository, create a new Packer configuration file, such as build-arm-linux.pkr.hcl or build-arm-windows.pkr.hcl. o In this new file, define the necessary source AMI for ARM architectures and the required configurations for the instance type (e.g., a1.medium for ARM): - -source_ami = "ami-arm-linux" # Replace with actual ARM Linux AMI ID -instance_type = "a1.medium" # Instance type for ARM - +Note: Its not always necessary, but It introduces a step to evaluate whether the existing Packer templates can be adapted for the new pipeline type. This approach ensures that unnecessary duplication of configurations is avoided. 2. Update the Ansible Playbooks (if necessary): o Update the Ansible playbooks in the image-pipeline-ansible-playbooks repo to account for any differences in ARM architecture, if applicable. @@ -126,4 +131,4 @@ module "arm-linux" { 5. Push to the Repository: o Once you’ve made these changes, push your new files to the relevant repositories (e.g., aws-image-pipeline, linux-image-pipeline, image-pipeline-ansible-playbooks, and image-pipeline-goss-testing). o Ensure the new pipeline type is registered by updating the AWS CodeBuild configuration, so it triggers the new pipeline on push. - +Note: You do not need to modify the AWS CodeBuild configuration when adding a new pipeline type. CodeBuild serves as a shell for automation, and the focus should be on updating Ansible playbooks and Goss tests. The infrastructure for running the pipeline remains unchanged.