Updated README.md

README.md

@@ -1,2 +1,172 @@
+
 # aws-image-pipeline
 Terraform Workspace for creating and managing AWS Image Pipelines.
+RHEL Pipeline: End-to-End Flow Breakdown
+This README describes the end-to-end process for the RHEL pipeline, covering the source, build, and test phases, and how the pipeline integrates with AWS services, Packer, Ansible, and Goss.
+Overview
+The RHEL pipeline is designed to automate the creation, configuration, and testing of an Amazon Machine Image (AMI) for Red Hat Enterprise Linux (RHEL) instances. It utilizes multiple repositories and tools such as Packer, Terraform, Ansible, and Goss to build and validate images. Below is a breakdown of each step and the repositories involved.
+________________________________________
+1. **Repositories Involved**
+
+•	aws-image-pipeline: Provides Terraform configurations and defines the infrastructure pipeline (e.g., EC2 instances, security groups).
+
+•	linux-image-pipeline: Contains the Packer configurations responsible for building the AMIs.
+
+•	image-pipeline-ansible-playbooks: Hosts the Ansible playbooks that are used to configure the instances during the Packer build.
+
+•	image-pipeline-goss-testing: Contains Goss test definitions used to validate the AMI configurations during the test phase.
+________________________________________
+2. **Pipeline Flow**
+Source Phase:
+
+Pull Configurations:
+
+•	The pipeline configuration is initiated by the aws-image-pipeline repository, which calls the necessary Terraform modules responsible for building the pipelines. The actual configurations are pulled from other specified repositories.
+
+•	The pipeline fetches Packer configurations from multiple repositories, including linux-image-pipeline, docker-image-pipeline, and windows-image-pipeline.
+
+•	The Ansible playbooks are retrieved from image-pipeline-ansible-playbooks.
+
+•	The Goss tests are pulled from image-pipeline-goss-testing.
+
+**Build Phase:**
+1.	Launch the EC2 Instance:
+
+•	Packer launches an EC2 instance using an AMI specified in the terraform-aws-image-pipeline module, rather than a "predefined" AMI.
+
+•	Instance credentials are securely stored in AWS Secrets Manager, not in Parameter Store. Both AWS Parameter Store and Secrets Manager are populated by the terraform-aws-image-pipeline module based on the provided parameters.
+
+2.	Instance Configuration:
+
+•	Ansible playbooks from the image-pipeline-ansible-playbooks repo are run to configure the instance.
+
+•	Example configurations include installing required packages, setting up services, and applying security hardening measures.
+
+3.	Capture AMI:
+
+•	After the instance is successfully configured by Ansible, Packer captures the configured instance as an AMI.
+
+•	The AMI ID is stored in tf_ami_id.txt and uploaded to the AWS Parameter Store for later use.
+
+**Test Phase:**
+
+1.	Run Goss Tests:
+
+•	Once the AMI is built, Goss tests defined in the image-pipeline-goss-testing repo are run on the newly created AMI to validate the configuration.
+
+•	Tests check whether all services are running, the necessary packages are installed, and the configurations meet security compliance.
+
+**Post-Build Steps: If tests pass or fail, the following happens:**
+
+Tests Passed:
+
+- If the Goss tests pass, no further action is required regarding the AMI. The AMI is not destroyed, and no action is taken to remove it.
+
+- Tests Failed or Troubleshooting Enabled:
+
+- The EC2 instance used for testing is destroyed via Terraform after the tests are completed, regardless of whether they pass or fail.
+
+- If troubleshooting is enabled or the tests fail, the AMI is deregistered from AWS using the following commands:
+
+~~~
+false || /bin/terraform destroy -var project_name=rhel-image-pipeline-demo -var goss_directory=${CODEBUILD_SRC_DIR_SourceGossOutput} -auto-approve
+
+false || test -f tf_ami_id.txt && aws ec2 deregister-image --image-id `cat tf_ami_id.txt` --region $AWS_REGION || echo "Tests passed, no AMI to deregister"
+~~~
+- The AMI is not immediately deleted but is marked for deregistration using aws ec2 deregister-image, which removes it from being available for future EC2 launches.
+
+**Additional Information for Pipeline Management**
+
+This section addresses specific questions that a consumer of these repositories might have when interacting with or modifying the pipeline. It will guide users on how to update Ansible playbooks, manage Goss tests, and add new pipeline types.
+________________________________________
+1. How to Add/Change/Update Ansible Playbooks
+
+If you need to modify or add Ansible playbooks that run during the AMI build process, follow these steps:
+
+Where Do Playbooks Go?
+
+•	Playbooks should be stored in the image-pipeline-ansible-playbooks repository.
+
+•	Playbooks are structured within this repository, and you should add your playbooks here if you need to update the configurations applied to the AMI.
+
+How Are They Integrated?
+
+•	The playbooks are integrated into the pipeline through the build.pkr.hcl file in the linux-image-pipeline repository.
+
+•	Within the build.pkr.hcl, the Ansible provisioner is used to execute the playbooks during the AMI build phase:
+
+~~~
+provisioner "ansible" {
+    command           = "/root/.local/bin/ansible-playbook"
+    playbook_file     = "${var.ansible_dir}/${data.amazon-parameterstore.playbook.value}"
+    roles_path        = "${var.ansible_dir}/roles"
+    ansible_env_vars  = ["ANSIBLE_STDOUT_CALLBACK=yaml", "ANSIBLE_NOCOLOR=True"]
+    user              = data.amazon-parameterstore.ssh_user.value
+~~~
+
+**Steps to Update or Add Playbooks:**
+1.	Modify or add your playbooks to the image-pipeline-ansible-playbooks repo.
+2.  Update the playbook parameter in the Terraform configuration (e.g., [rhel.tf] or equivalent) to point to the correct playbook, by modifying the playbook parameter: playbook = "your-new-playbook.yaml".
+3.  The Terraform configuration will automatically update the value in AWS Parameter Store when applying the changes, ensuring the correct playbook is referenced during the pipeline execution.
+
+Note: This is the only step required when pointing to new playbooks. There is no need to manually modify AWS Secrets Manager or Parameter Store, as the Terraform configuration handles these updates automatically.
+________________________________________
+2. How to Add/Change/Update Goss Tests
+If you need to add or update the Goss tests used to validate the AMI, follow these steps:
+
+Where Do Goss Tests Go?
+- Goss tests are stored in the image-pipeline-goss-testing repository.
+- The relevant test files are located in the goss-files/ directory, where you can add or modify .yaml files to include new tests.
+How Are They Integrated?
+- The Goss tests are integrated through the relevant Terraform configuration files, depending on the pipeline.
+- For Linux pipelines, they are integrated via the [linux.tf]) file in the image-pipeline-goss-testing repository.
+- For Windows pipelines, Goss tests are integrated via the [windows.tf] file in the image-pipeline-goss-testing repository.    
+For example, in the [windows.tf]) file:
+- The goss_profile parameter is used to specify the set of tests to run during the validation phase: goss_profile = "windows-base-test".
+- The Goss repository (goss_repo) is referenced, and the appropriate Goss tests are pulled from this repository during the pipeline execution.
+- The goss_profile is passed in as a variable from the rhel.tf in the aws-image-pipeline repository and is tied to a specific set of Goss tests in the goss-files/ directory.
+goss_profile = "rhel-base-test"
+
+**Steps to Update or Add Goss Tests:**
+1. Add or update your .yaml files in the image-pipeline-goss-testing/goss-files/ directory.
+2. Ensure the goss_profile in rhel.tf matches the name of the new or updated test file:
+goss_profile = "your-new-test".
+
+3. The pipeline will automatically reference this new set of tests during the test phase when creating a new AMI.
+Consideration: Parking Goss Tests in the Ansible Playbooks
+- While Goss tests are currently separated from the Ansible playbooks, we did consider parking Goss tests as part of the playbooks for simplicity. However, separating the two provides several key benefits:
+- Isolation of Testing and Configuration: By keeping Goss tests separate, we ensure that testing is isolated from configuration. This provides clearer validation steps and better modularity in the pipeline.
+- Flexibility in Testing: This separation allows more flexibility in testing different configurations without the need to modify the Ansible playbooks directly.
+- Separation of Concerns: It also enables different teams to control the Goss and Packer repositories independently. For example, security teams can design and •   maintain Goss tests to ensure their specific concerns are addressed, without needing to alter the configurations managed by other teams. This division of responsibilities enhances security and ensures that each team can focus on their core expertise.
+________________________________________
+3. How to Add a New Pipeline Type (e.g., ARM Amazon Linux or ARM Windows Instance)
+- If you want to create a new pipeline type, such as one for ARM-based Amazon Linux or Windows instances, you need to follow these steps:
+**Steps to Add a New Pipeline Type:**
+1.	Create a New Packer Configuration:
+- In the linux-image-pipeline repository, create a new Packer configuration file, such as build-arm-linux.pkr.hcl or build-arm-windows.pkr.hcl.
+- In this new file, define the necessary source AMI for ARM architectures and the required configurations for the instance type (e.g., a1.medium for ARM):
+Note: Its not always necessary, but It introduces a step to evaluate whether the existing Packer templates can be adapted for the new pipeline type. This approach ensures that unnecessary duplication of configurations is avoided.
+
+2.	Update the Ansible Playbooks (if necessary):
+- Update the Ansible playbooks in the image-pipeline-ansible-playbooks repo to account for any differences in ARM architecture, if applicable.
+3.	Update the Goss Tests (if necessary):
+- Modify or create Goss tests in the image-pipeline-goss-testing repo that validate ARM-specific configurations, such as hardware compatibility or package installations.
+4.	Update the Terraform Configuration:
+- In the aws-image-pipeline repository, create a new Terraform configuration file, such as arm-linux.tf or arm-windows.tf.
+- Reference the new Packer file and adjust any required parameters (e.g., instance type, security groups, etc.).
+
+module "arm-linux" {
+  source           = "HappyPathway/image-pipeline/aws"
+  project_name     = "arm-linux-image-pipeline"
+  playbook         = "arm-linux-playbook.yaml"
+  instance_type    = "a1.medium"
+  source_ami       = "ami-arm-linux"  # Replace with actual ARM Linux AMI ID
+  goss_profile     = "arm-linux-test"
+}
+
+5.	Push to the Repository:
+- Once you’ve made these changes, push your new files to the relevant repositories (e.g., aws-image-pipeline, linux-image-pipeline, image-pipeline-ansible-playbooks, and image-pipeline-goss-testing).
+- Ensure the new pipeline type is registered by updating the AWS CodeBuild configuration, so it triggers the new pipeline on push.
+Note: You do not need to modify the AWS CodeBuild configuration when adding a new pipeline type. CodeBuild serves as a shell for automation, and the focus should be on updating Ansible playbooks and Goss tests. The infrastructure for running the pipeline remains unchanged.
+
+