Updated README.md

README.md

@@ -1,2 +1,129 @@
 # aws-image-pipeline
 Terraform Workspace for creating and managing AWS Image Pipelines.
+RHEL Pipeline: End-to-End Flow Breakdown
+This README describes the end-to-end process for the RHEL pipeline, covering the source, build, and test phases, and how the pipeline integrates with AWS services, Packer, Ansible, and Goss.
+Overview
+The RHEL pipeline is designed to automate the creation, configuration, and testing of an Amazon Machine Image (AMI) for Red Hat Enterprise Linux (RHEL) instances. It utilizes multiple repositories and tools such as Packer, Terraform, Ansible, and Goss to build and validate images. Below is a breakdown of each step and the repositories involved.
+________________________________________
+1. Repositories Involved
+•	aws-image-pipeline: Provides Terraform configurations and defines the infrastructure pipeline (e.g., EC2 instances, security groups).
+•	linux-image-pipeline: Contains the Packer configurations responsible for building the AMIs.
+•	image-pipeline-ansible-playbooks: Hosts the Ansible playbooks that are used to configure the instances during the Packer build.
+•	image-pipeline-goss-testing: Contains Goss test definitions used to validate the AMI configurations during the test phase.
+________________________________________
+2. Pipeline Flow
+Source Phase:
+1.	Pull Configurations:
+o	The pipeline pulls Terraform configurations from the aws-image-pipeline repository.
+o	It fetches Packer configurations from the linux-image-pipeline repository.
+o	The Ansible playbooks are retrieved from image-pipeline-ansible-playbooks.
+o	The Goss tests are pulled from image-pipeline-goss-testing.
+
+Build Phase:
+1.	Launch the EC2 Instance:
+o	Packer, defined in the build.pkr.hcl file, starts the build process by launching an EC2 instance using a pre-defined AMI.
+o	The instance type, subnet, security groups, and SSH user credentials are sourced from AWS Parameter Store.
+2.	Instance Configuration:
+o	Ansible playbooks from the image-pipeline-ansible-playbooks repo are run to configure the instance.
+o	Example configurations include installing required packages, setting up services, and applying security hardening measures.
+3.	Capture AMI:
+o	After the instance is successfully configured by Ansible, Packer captures the configured instance as an AMI.
+o	The AMI ID is stored in tf_ami_id.txt and uploaded to the AWS Parameter Store for later use.
+Test Phase:
+1.	Run Goss Tests:
+o	Once the AMI is built, Goss tests defined in the image-pipeline-goss-testing repo are run on the newly created AMI to validate the configuration.
+o	Tests check whether all services are running, the necessary packages are installed, and the configurations meet security compliance.
+2.	Post-Build Steps: If tests pass or fail, the following happens:
+o	Tests Passed:
+	If the Goss tests pass, no further action is required regarding the AMI. The AMI is not destroyed, and no action is taken to remove it.
+o	Tests Failed or Troubleshooting Enabled:
+	If tests fail, the EC2 instance used for testing is destroyed via Terraform.
+	If troubleshooting is enabled or the tests fail, the AMI is deregistered from AWS using the following commands:
+false || /bin/terraform destroy -var project_name=rhel-image-pipeline-demo -var goss_directory=${CODEBUILD_SRC_DIR_SourceGossOutput} -auto-approve
+
+false || test -f tf_ami_id.txt && aws ec2 deregister-image --image-id `cat tf_ami_id.txt` --region $AWS_REGION || echo "Tests passed, no AMI to deregister"
+
+	The AMI is not immediately deleted but is marked for deregistration using aws ec2 deregister-image, which removes it from being available for future EC2 launches.
+
+
+
+
+
+
+Additional Information for Pipeline Management
+This section addresses specific questions that a consumer of these repositories might have when interacting with or modifying the pipeline. It will guide users on how to update Ansible playbooks, manage Goss tests, and add new pipeline types.
+________________________________________
+1. How to Add/Change/Update Ansible Playbooks
+If you need to modify or add Ansible playbooks that run during the AMI build process, follow these steps:
+Where Do Playbooks Go?
+•	Playbooks should be stored in the image-pipeline-ansible-playbooks repository.
+•	Playbooks are structured within this repository, and you should add your playbooks here if you need to update the configurations applied to the AMI.
+How Are They Integrated?
+•	The playbooks are integrated into the pipeline through the build.pkr.hcl file in the linux-image-pipeline repository.
+•	Within the build.pkr.hcl, the Ansible provisioner is used to execute the playbooks during the AMI build phase:
+
+provisioner "ansible" {
+    command           = "/root/.local/bin/ansible-playbook"
+    playbook_file     = "${var.ansible_dir}/${data.amazon-parameterstore.playbook.value}"
+    roles_path        = "${var.ansible_dir}/roles"
+    ansible_env_vars  = ["ANSIBLE_STDOUT_CALLBACK=yaml", "ANSIBLE_NOCOLOR=True"]
+    user              = data.amazon-parameterstore.ssh_user.value
+
+
+Steps to Update or Add Playbooks:
+1.	Modify or add your playbooks to the image-pipeline-ansible-playbooks repo.
+2.	Ensure the correct playbook name is referenced in AWS Parameter Store under the parameter for playbook (e.g., /image-pipeline/<project_name>/playbook).
+3.	Update the rhel.tf (or equivalent) to point to the correct playbook by modifying the playbook parameter:
+playbook = "your-new-playbook.yaml"
+________________________________________
+2. How to Add/Change/Update Goss Tests
+If you need to add or update the Goss tests used to validate the AMI, follow these steps:
+Where Do Goss Tests Go?
+•	Goss tests are stored in the image-pipeline-goss-testing repository.
+•	The relevant test files are located in the goss-files/ directory, where you can add or modify .yaml files to include new tests.
+How Are They Integrated?
+•	The Goss tests are integrated through the linux.tf file in the image-pipeline-goss-testing repository.
+•	The goss_profile is passed in as a variable from the rhel.tf in the aws-image-pipeline repository and is tied to a specific set of Goss tests in the goss-files/ directory.
+goss_profile = "rhel-base-test"
+
+Steps to Update or Add Goss Tests:
+1.	Add or update your .yaml files in the image-pipeline-goss-testing/goss-files/ directory.
+2.	Ensure the goss_profile in rhel.tf matches the name of the new or updated test file:
+goss_profile = "your-new-test"
+3.	The pipeline will automatically reference this new set of tests during the test phase when creating a new AMI.
+Consideration: Parking Goss Tests in the Ansible Playbooks
+•	While Goss tests are currently separated from the Ansible playbooks, we did consider parking Goss tests as part of the playbooks for simplicity. However, separating the two ensures that testing is isolated from configuration, providing clearer validation steps and better modularity.
+•	This separation also allows more flexibility in testing different configurations without modifying the Ansible playbooks directly.
+________________________________________
+3. How to Add a New Pipeline Type (e.g., ARM Amazon Linux or ARM Windows Instance)
+If you want to create a new pipeline type, such as one for ARM-based Amazon Linux or Windows instances, you need to follow these steps:
+Steps to Add a New Pipeline Type:
+1.	Create a New Packer Configuration:
+o	In the linux-image-pipeline repository, create a new Packer configuration file, such as build-arm-linux.pkr.hcl or build-arm-windows.pkr.hcl.
+o	In this new file, define the necessary source AMI for ARM architectures and the required configurations for the instance type (e.g., a1.medium for ARM):
+
+source_ami         = "ami-arm-linux"  # Replace with actual ARM Linux AMI ID
+instance_type      = "a1.medium"      # Instance type for ARM
+
+
+2.	Update the Ansible Playbooks (if necessary):
+o	Update the Ansible playbooks in the image-pipeline-ansible-playbooks repo to account for any differences in ARM architecture, if applicable.
+3.	Update the Goss Tests (if necessary):
+o	Modify or create Goss tests in the image-pipeline-goss-testing repo that validate ARM-specific configurations, such as hardware compatibility or package installations.
+4.	Update the Terraform Configuration:
+o	In the aws-image-pipeline repository, create a new Terraform configuration file, such as arm-linux.tf or arm-windows.tf.
+o	Reference the new Packer file and adjust any required parameters (e.g., instance type, security groups, etc.).
+
+module "arm-linux" {
+  source           = "HappyPathway/image-pipeline/aws"
+  project_name     = "arm-linux-image-pipeline"
+  playbook         = "arm-linux-playbook.yaml"
+  instance_type    = "a1.medium"
+  source_ami       = "ami-arm-linux"  # Replace with actual ARM Linux AMI ID
+  goss_profile     = "arm-linux-test"
+}
+
+5.	Push to the Repository:
+o	Once you’ve made these changes, push your new files to the relevant repositories (e.g., aws-image-pipeline, linux-image-pipeline, image-pipeline-ansible-playbooks, and image-pipeline-goss-testing).
+o	Ensure the new pipeline type is registered by updating the AWS CodeBuild configuration, so it triggers the new pipeline on push.
+