diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 16517607..cd786b56 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -11,7 +11,7 @@ repos: - id: isort - repo: https://github.com/pycqa/flake8 - rev: 7.1.2 + rev: 7.2.0 hooks: - id: flake8 additional_dependencies: [flake8-docstrings] diff --git a/README.prowler b/README.prowler index 33067082..1732e658 100644 --- a/README.prowler +++ b/README.prowler @@ -1,4 +1,3 @@ git clone https://github.com/toniblyx/prowler.git git submodule add https://github.com/toniblyx/prowler.git prowler - diff --git a/docs/BOOTSTRAP.md b/docs/BOOTSTRAP.md index cc81d423..21e2b097 100644 --- a/docs/BOOTSTRAP.md +++ b/docs/BOOTSTRAP.md @@ -3,7 +3,7 @@ Do all setup from (GETTING-STARTED.md). ```code -git clone +git clone git-secret reveal ``` @@ -13,7 +13,7 @@ etc. We need to create an account in which to bootstap one's own user. Generally when set up, we have an a-USER account, and access keys and setup is done under that user, and access -keys mapped to the specific profile. Since all the accounts are created under +keys mapped to the specific profile. Since all the accounts are created under Terraform, it's not possible to create the same user that is running the configurations. More details coming. @@ -22,10 +22,10 @@ More details coming. Some configuration files have dependencies on other parts of the configuration in other directories. We've collected base setup files into common and infrastructure, -where common are IAM things (users, groups, policies, roles) and infrastructure are +where common are IAM things (users, groups, policies, roles) and infrastructure are buckets, config, cloudtrail, and others. -## 1. Common +## 1. Common We need to start with an empty remote_state.* for common and infrastructure @@ -38,14 +38,14 @@ cd common ln -sf remote_state.common.tf.none remote_state.common.tf terraform init -terraform plan -terraform apply +terraform plan +terraform apply ``` We are now in a state with a number of resources setup, and we can move on to the next step. -## 2. Infrastructure +## 2. Infrastructure Now that we have a state, we'll link to it @@ -61,8 +61,8 @@ cd ../infrastructure terraform init -terraform plan -terraform apply +terraform plan +terraform apply ``` After this is completed, we now have an S3 bukcet for state, so we'll start using that. @@ -115,7 +115,7 @@ Do you want to copy existing state to the new backend? Enter a value: ``` -## 3. Common +## 3. Common We'll now go into common and activate the remote state. @@ -132,7 +132,7 @@ Let's pull in the rest of the stage2 configurations and activate them. ```code mv .stage2/config.tf .stage2/flowlog.tf ./ -terraform plan +terraform plan terraform apply ``` diff --git a/docs/GETTING-STARTED.md b/docs/GETTING-STARTED.md index ba233a1e..84da3447 100644 --- a/docs/GETTING-STARTED.md +++ b/docs/GETTING-STARTED.md @@ -2,7 +2,7 @@ We have changed from a RHEL7 to a RHEL8 server in order to support some newer things needed for EKS. While RHEL7 will still work for most things, it will not work for any EKS deployments. This is because -a utility callsed `skopeo` is needed, and the RHEL7 version of it is missing login capability. This +a utility callsed `skopeo` is needed, and the RHEL7 version of it is missing login capability. This capability exists in RHEL8. RHEL 8 Linux Server minimum @@ -29,7 +29,7 @@ These are delivered through the OS (yum, dnf) This is delivered through either the OS, via download, or the Terraform application package: -* awscli v2 +* awscli v2 These are delivered through the Terraform application package. @@ -87,7 +87,7 @@ aws_secret_access_key={secret_access_key} region = us-gov-west-1 ``` -This should be reflective of your IAM account `a-{username}` in the `g-inf-cloud-admin` group, so that +This should be reflective of your IAM account `a-{username}` in the `g-inf-cloud-admin` group, so that the proper permissions are available to run the various terraform. At some point, there may be a specific set of terraform permissions scoped out and narrowed down for more granular control. @@ -139,4 +139,3 @@ ln -sf ../outputs.common.tf ./ ln -sf ../variables.common.auto.tfvars ./ ln -sf ../variables.common.tf ./ ``` - diff --git a/docs/GPG.md b/docs/GPG.md index eee8fa78..5a996479 100644 --- a/docs/GPG.md +++ b/docs/GPG.md @@ -11,7 +11,7 @@ Setup the `/etc/sysconfig/rngd` file as follows: ``` # /etc/sysconfig/rngd -EXTRAOPTIONS="-n 0 +EXTRAOPTIONS="-n 0 ``` Then enable and start @@ -173,24 +173,24 @@ GPG_EMAIL="other-name@census.gov" GPG_USERNAME="other" setup-gpg.sh To set up GPG keys for a service account: -1. Submit LDAP Service Account Remedy ticket to have an enterprise service account created, if not already done. +1. Submit LDAP Service Account Remedy ticket to have an enterprise service account created, if not already done. * Include in the ticket details that the service account needs to be able to authenticate with Enterprise Github and must have POSIX/Unix attributes enabled so Linux based OS can recognize it. -2. Once the Service Account is created, submit an Application Account Remedy ticket to add the service account to your GitHub repositories. - * In the ticket details, include the service account name and the repositories to which it will require access. Once the service account is added, you will need to log into GitHub using the service account name/password in a PrivateTab window using https://id-provider.tco.census.gov/nidp/saml2/sso. -3. Once the Service Account is created, submit a ticket for sudo access to the service account from the iebcloud server. - * Include the service account name in the ticket and what it is being used for. -4. From the iebcloud server, sudo into the the service account. +2. Once the Service Account is created, submit an Application Account Remedy ticket to add the service account to your GitHub repositories. + * In the ticket details, include the service account name and the repositories to which it will require access. Once the service account is added, you will need to log into GitHub using the service account name/password in a PrivateTab window using https://id-provider.tco.census.gov/nidp/saml2/sso. +3. Once the Service Account is created, submit a ticket for sudo access to the service account from the iebcloud server. + * Include the service account name in the ticket and what it is being used for. +4. From the iebcloud server, sudo into the the service account. * Run `sudo su - {service_account_name}` where **{service_account_name}** is the service account name. 5. As the service account, create a **gpg-files** directory in the service account home direcotry and copy the the [setup-gpg.sh](../local-app/bin/setup-gpg.sh) file into it or refer to it at `/apps/terraform/bin/setup-gpg.sh` if present. 6. Run the gpg script as the service account. - * Run the command `GPG_EMAIL={example_email_distro} setup-gpg.sh` where **{example_email_distro}** is a Census email distribution list corresponding to users who manage the service account. *Do not use an individual user's email address.* + * Run the command `GPG_EMAIL={example_email_distro} setup-gpg.sh` where **{example_email_distro}** is a Census email distribution list corresponding to users who manage the service account. *Do not use an individual user's email address.* 9. Verify the resulting **{service_account_name}.gpg.asc** file maps to exactly ONE pub, sub, and **{service_account_name}** uid with corresponding **{GPG_EMAIL}** email distribution. If the following command return more than one of those things, there are too many keys in this file and it won't work. - * Run the command `gpg {service_account_name}.gpg.asc`. + * Run the command `gpg {service_account_name}.gpg.asc`. **Example** - + ```console - % gpg edl-svc-tf-deploy.gpg.asc + % gpg edl-svc-tf-deploy.gpg.asc gpg: WARNING: no command supplied. Trying to guess what you mean ... pub rsa4096 2024-05-06 [SCEA] XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX diff --git a/docs/api-reference.md b/docs/api-reference.md index 51516768..478da607 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -275,10 +275,10 @@ from tf_upgrade.reporters.base import BaseReporter class CustomReporter(BaseReporter): def __init__(self): super().__init__() - + def add_section(self, title, content): # Custom implementation - + def generate_report(self): # Custom implementation return "Generated report" @@ -357,11 +357,11 @@ class CustomUpgrader(BaseUpgrader): source_version="0.12", target_version="0.13" ) - + def pre_upgrade_check(self, directory): # Check if directory is ready for upgrade return True - + def upgrade(self, directory): # Implement upgrade logic return self.upgrade_result(success=True) @@ -439,28 +439,28 @@ class CustomV013Upgrader(BaseUpgrader): target_version="0.13" ) self.transformer = HCLTransformer() - + # Add custom rules for Census patterns self.transformer.add_rule( - name="census_module_source", + name="census_module_source", pattern=r'source\s+=\s+"git::https://github\.com/census-bureau-internal/([^"]+)\.git\?ref=([^"]+)"', replacement=lambda m: f'source = "git::https://github.com/census-bureau-internal/{m.group(1)}.git?ref=tf-upgrade"' ) - + def upgrade(self, directory): # Create backup self.create_backup(directory) - + try: # Apply custom transformations self.transformer.transform_directory(directory, "*.tf") - + # Run standard 0.13upgrade command self.terraform_runner.run_command( directory=directory, command=["0.13upgrade", "-yes"] ) - + # Validate the upgraded configuration validation = self.terraform_runner.validate(directory) if not validation.success: @@ -468,9 +468,9 @@ class CustomV013Upgrader(BaseUpgrader): success=False, error=f"Validation failed: {validation.stderr}" ) - + return self.upgrade_result(success=True) - + except Exception as e: # Restore from backup in case of error self.restore_backup(directory) @@ -483,7 +483,7 @@ The tool uses custom exceptions for better error handling: ```python from tf_upgrade.exceptions import ( - UpgradeError, + UpgradeError, ValidationError, TerraformExecutionError, BackupError @@ -594,7 +594,7 @@ def custom_scan(directory, custom_option): """Perform a custom scan of Terraform configurations.""" click.echo(f"Scanning {directory} with {custom_option}") # Custom scan implementation - + @cli_group.command("custom-upgrade") @click.argument("directory", type=click.Path(exists=True)) @click.option("--special-handling", is_flag=True, help="Use special handling") diff --git a/docs/app-setup.md b/docs/app-setup.md index 4cc18928..f7678594 100644 --- a/docs/app-setup.md +++ b/docs/app-setup.md @@ -42,7 +42,7 @@ cd my-app-name Then: -1. copy remote-state.yml from the parent directory +1. copy remote-state.yml from the parent directory 1. update the `directory:` line to include the new app directory at the end ``` directory: "common/app/my-app-name" diff --git a/docs/app-setup/.terraform-docs.yml b/docs/app-setup/.terraform-docs.yml index 8391b9d3..5738e398 100644 --- a/docs/app-setup/.terraform-docs.yml +++ b/docs/app-setup/.terraform-docs.yml @@ -5,7 +5,7 @@ footer-from: "" sections: ## hide: [] - show: + show: - data-sources - header - footer @@ -15,7 +15,7 @@ sections: - providers - requirements - resources - + output: file: README.md mode: inject @@ -27,11 +27,11 @@ output: ## output-values: ## enabled: false ## from: "" -## +## ## sort: ## enabled: true ## by: name -## +## ## settings: ## anchor: true ## color: true diff --git a/docs/census-examples.md b/docs/census-examples.md index 3050e616..a5950394 100644 --- a/docs/census-examples.md +++ b/docs/census-examples.md @@ -54,7 +54,7 @@ For EDL modules, follow this version reference pattern after upgrading to Terraf ```terraform module "edl_processing" { source = "git::https://github.com/census-bureau-internal/terraform-aws-edl-workflow.git?ref=tf-upgrade" - + // Rest of module configuration } ``` @@ -73,7 +73,7 @@ resource "aws_security_group" "app_sg" { name = "${var.environment}-${var.app_name}-sg" description = "Security group for ${var.app_name}" vpc_id = var.vpc_id - + tags = { "boc:application" = var.app_name "boc:environment" = var.environment @@ -86,7 +86,7 @@ resource "aws_security_group" "app_sg" { name = "${var.environment}-${var.app_name}-sg" description = "Security group for ${var.app_name}" vpc_id = var.vpc_id - + tags = { "boc:application" = var.app_name "boc:environment" = var.environment @@ -105,30 +105,30 @@ Census Bureau IAM roles follow a standard pattern. Here's how they're upgraded: # Before upgrade (0.12.x) module "app_role" { source = "git::https://github.com/census-bureau-internal/terraform-aws-iam-role.git?ref=v0.5.0" - + name = "${var.environment}-${var.app_name}-role" assume_role_policy = data.aws_iam_policy_document.assume_role.json - + policy_arns = [ "arn:aws-us-gov:iam::aws:policy/AmazonS3ReadOnlyAccess", aws_iam_policy.custom_policy.arn ] - + tags = var.tags } # After upgrade (1.x) module "app_role" { source = "git::https://github.com/census-bureau-internal/terraform-aws-iam-role.git?ref=v1.0.0" - + name = "${var.environment}-${var.app_name}-role" assume_role_policy = data.aws_iam_policy_document.assume_role.json - + policy_arns = [ "arn:aws-us-gov:iam::aws:policy/AmazonS3ReadOnlyAccess", aws_iam_policy.custom_policy.arn ] - + tags = var.tags } ``` @@ -203,7 +203,7 @@ terraform { backend "s3" { # These values are populated from environment variables } - + required_version = ">= 1.0.0" required_providers { aws = { diff --git a/docs/code-architecture.md b/docs/code-architecture.md index 9b464110..cf074e7f 100644 --- a/docs/code-architecture.md +++ b/docs/code-architecture.md @@ -124,7 +124,7 @@ class Command(ABC): @abstractmethod def execute(self, context): pass - + class ScanCommand(Command): def execute(self, context): # Scanning logic @@ -139,7 +139,7 @@ class BaseUpgrader(ABC): @abstractmethod def upgrade(self, directory): pass - + class V013Upgrader(BaseUpgrader): def upgrade(self, directory): # 0.13 specific upgrade logic @@ -157,7 +157,7 @@ class BaseUpgrader(ABC): self.transform_configuration(directory) self.post_validation(directory) self.report_results(directory) - + @abstractmethod def transform_configuration(self, directory): pass @@ -171,13 +171,13 @@ Progress reporting uses the Observer pattern: class ProgressSubject(ABC): def __init__(self): self._observers = [] - + def attach(self, observer): self._observers.append(observer) - + def detach(self, observer): self._observers.remove(observer) - + def notify(self, message, progress_pct): for observer in self._observers: observer.update(message, progress_pct) @@ -211,11 +211,11 @@ class FileUtils: @staticmethod def read_file(path): # Common file reading logic - + @staticmethod def write_file(path, content): # Common file writing logic - + @staticmethod def backup_file(path, backup_dir): # Common backup logic @@ -230,7 +230,7 @@ class TransformationLibrary: @staticmethod def replace_provider_syntax(content): # Common provider transformation logic - + @staticmethod def update_interpolation_syntax(content): # Common interpolation transformation logic @@ -244,13 +244,13 @@ A base Reporter class with common functionality: class BaseReporter(ABC): def __init__(self): self.results = {} - + def add_result(self, key, value): self.results[key] = value - + def get_result(self, key, default=None): return self.results.get(key, default) - + @abstractmethod def generate_report(self): pass @@ -275,10 +275,10 @@ class ConfigurationManager: def __init__(self, config_file=None): self.config = self._load_config(config_file) self.defaults = self._load_defaults() - + def get(self, key, default=None): # Get configuration with fallback to defaults - + def set(self, key, value): # Set and persist configuration ``` diff --git a/docs/github-workflow.md b/docs/github-workflow.md index 4e7b6a3e..d7cbf5e9 100644 --- a/docs/github-workflow.md +++ b/docs/github-workflow.md @@ -223,7 +223,7 @@ To begin using the GitHub integration: ```bash # Set GitHub token (recommended to use environment variable) export GITHUB_TOKEN=your_personal_access_token - + # Or configure in config file tf-upgrade config set github.token "your_personal_access_token" ``` @@ -245,7 +245,7 @@ To begin using the GitHub integration: ```bash # List tickets make github-list - + # Claim a directory make github-claim DIR=/path/to/terraform/config ``` diff --git a/docs/how-to/account-provisioning-ansible/README.md b/docs/how-to/account-provisioning-ansible/README.md index d1900581..ac00b673 100644 --- a/docs/how-to/account-provisioning-ansible/README.md +++ b/docs/how-to/account-provisioning-ansible/README.md @@ -10,8 +10,8 @@ It requires you have completed some prerequisites. ```console # on iebcloud, make sure you have a workspace % test -d /data/terraform/workspaces/$USER/terraform/ || mkdir -p /data/terraform/workspaces/$USER/terraform/ - - % cd /data/terraform/workspaces/$USER/terraform/ + + % cd /data/terraform/workspaces/$USER/terraform/ % git clone git@github.e.it.census.gov:terraform/support.git % cd support % git pull origin master @@ -21,8 +21,8 @@ It requires you have completed some prerequisites. ## Steps References: -* [Ansible setup for Baseline of Account](../account-provisioning-ansible/) -* [Ansible Details](../../../local-app/aws-account-setup/ansible/README.md) +* [Ansible setup for Baseline of Account](../account-provisioning-ansible/) +* [Ansible Details](../../../local-app/aws-account-setup/ansible/README.md) * [Running Ansible (old)](../../../local-app/aws-account-setup/ansible/Ansible.md) Steps @@ -57,7 +57,7 @@ and account alias. This example will reference the account *ma12-gov*. It will create a log file `setup.{account_alias}.{short_hostname}.{timestamp}.log`. -You must execute from the master branch, and make sure you have the latest code before each deployment, as changes to +You must execute from the master branch, and make sure you have the latest code before each deployment, as changes to documentation, Ansible roles, etc. need to be picked up. ```console @@ -70,10 +70,10 @@ documentation, Ansible roles, etc. need to be picked up. # lots of output, sample recap below PLAY RECAP **************************************************************************************************************************** -ma12-gov.cloud ansible_host=localhost : ok=183 changed=80 unreachable=0 failed=0 skipped=49 rescued=0 ignored=0 +ma12-gov.cloud ansible_host=localhost : ok=183 changed=80 unreachable=0 failed=0 skipped=49 rescued=0 ignored=0 -Thursday 11 August 2022 12:41:46 -0400 (0:00:00.051) 0:02:20.889 ******* -=============================================================================== +Thursday 11 August 2022 12:41:46 -0400 (0:00:00.051) 0:02:20.889 ******* +=============================================================================== setup-variables -------------------------------------------------------- 22.38s setup-directories ------------------------------------------------------ 20.61s inf-common ------------------------------------------------------------- 19.04s @@ -87,10 +87,10 @@ inf-vpc ----------------------------------------------------------------- 5.79s setup-git-secret -------------------------------------------------------- 5.64s setup-gpg --------------------------------------------------------------- 5.02s gather_facts ------------------------------------------------------------ 2.91s -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ total ----------------------------------------------------------------- 140.81s -Thursday 11 August 2022 12:41:46 -0400 (0:00:00.051) 0:02:20.888 ******* -=============================================================================== +Thursday 11 August 2022 12:41:46 -0400 (0:00:00.051) 0:02:20.888 ******* +=============================================================================== inf-infrastructure : deploy per-region infrastructure setup configs ----------------------------------------------------------- 11.12s inf-common : deploy common setup configs --------------------------------------------------------------------------------------- 8.85s setup-provider-configs : Copy provider_configs --------------------------------------------------------------------------------- 4.94s @@ -123,13 +123,13 @@ at the rest of the log. Finally, we need to move this out of `/tmp` and into your workspace. ```console -% cd /data/terraform/workspaces/$USER/terraform/ +% cd /data/terraform/workspaces/$USER/terraform/ % mv /tmp/{account_id}-{account_alias} ./ ``` Here is an example: ```console -% cd /data/terraform/workspaces/$USER/terraform/ +% cd /data/terraform/workspaces/$USER/terraform/ % mv /tmp/412295344020-ma12-gov ./ ``` diff --git a/docs/how-to/account-provisioning-execute/README.md b/docs/how-to/account-provisioning-execute/README.md index d29018ee..aec5cd47 100644 --- a/docs/how-to/account-provisioning-execute/README.md +++ b/docs/how-to/account-provisioning-execute/README.md @@ -5,7 +5,7 @@ It requires you have completed some prerequisites. ## Prerequisites 1. [Create New AWS Accounts](https://github.e.it.census.gov/terraform/cloud-information/tree/master/aws/documentation/account-setup) -1. An IAM `bootstrap` account, or cross-account configuration with Administrator permissions using the standard *{account_id}-{account_alias}* +1. An IAM `bootstrap` account, or cross-account configuration with Administrator permissions using the standard *{account_id}-{account_alias}* profile. We will call this _bootstrap access_. 1. [Ansible setup for Baseline of Account](../account-provisioning-ansible/) @@ -68,7 +68,7 @@ To start, we must be in the directory for the account git repository set up in t on with the *ma12-gov* account as our example. ```console -% cd /data/terraform/workspaces/$USER/terraform/ +% cd /data/terraform/workspaces/$USER/terraform/ % mv /tmp/412295344020-ma12-gov ./ ``` @@ -179,7 +179,7 @@ There is no need to run any git commits just yet. ## Steps: common -In this directory we create a lot of resources, policies, users, roles, groups, which form the basis of use within all of the other +In this directory we create a lot of resources, policies, users, roles, groups, which form the basis of use within all of the other directories across the repository. You should have come from the first infrastructure setup. You should be back at the git root (top of the repo). The first command here will send you there if not. @@ -198,7 +198,7 @@ Next, we can do the apply (optionally, with tag:from-infrastructure): ```console % tf-run apply tag:from-infrastrucure # OR -$ tf-run apply +$ tf-run apply ``` We'll finish this up for now, commit and push, and then back up to the top. diff --git a/docs/how-to/account-provisioning-organization/README.md b/docs/how-to/account-provisioning-organization/README.md index 1a1e0b7a..5ad41e54 100644 --- a/docs/how-to/account-provisioning-organization/README.md +++ b/docs/how-to/account-provisioning-organization/README.md @@ -10,7 +10,7 @@ It requires you have completed some prerequisites. ## Steps -1. [Add to appropriate AWS Organization](../account-provisioning-organization/) +1. [Add to appropriate AWS Organization](../account-provisioning-organization/) * EW in censusaws * GovCloud in ma5-gov 1. [Add to Organization](https://github.e.it.census.gov/terraform/cloud-information/blob/master/aws/documentation/account-setup/add-to-org.md) diff --git a/docs/how-to/account-provisioning-submodule-repo/README.md b/docs/how-to/account-provisioning-submodule-repo/README.md index 7fe64fcb..f3161072 100644 --- a/docs/how-to/account-provisioning-submodule-repo/README.md +++ b/docs/how-to/account-provisioning-submodule-repo/README.md @@ -346,7 +346,7 @@ Back to the main repository. ## Step 3.1: Finalize git repo settings Now, we are back in the submodule directory. Change the `settings.auto.tfvars` value of `github_initial_commit` to `true` -or add the line to the end of the file if it doesn't exist and then apply the rest of the configuration. +or add the line to the end of the file if it doesn't exist and then apply the rest of the configuration. This is how it should look as added in the file: github_initial_commit = true ```shell @@ -651,7 +651,7 @@ git push origin add-tf-control-files In the GUI for the submodule repository, go to settings, and then make sure this box is checked. -- [x] Automatically delete head branches +- [x] Automatically delete head branches Deleted branches will still be able to be restored. # Conclusion diff --git a/docs/how-to/account-provisioning/README.md b/docs/how-to/account-provisioning/README.md index 16e3c277..09bb77ae 100644 --- a/docs/how-to/account-provisioning/README.md +++ b/docs/how-to/account-provisioning/README.md @@ -39,6 +39,6 @@ of branch, edit, commit, push, PR, discuss, merge. * EW: vpc/east-1, vpc/east-2, vpc/west-1, vpc/west-2 1. vpc/unused (EW only) * vpc/unused/* -1. [Add to appropriate AWS Organization](../account-provisioning-organization/) +1. [Add to appropriate AWS Organization](../account-provisioning-organization/) * EW in censusaws * GovCloud in ma5-gov diff --git a/docs/how-to/app-setup/README.md b/docs/how-to/app-setup/README.md index f8d34f79..77609c86 100644 --- a/docs/how-to/app-setup/README.md +++ b/docs/how-to/app-setup/README.md @@ -69,7 +69,7 @@ The [init](https://github.e.it.census.gov/terraform/support/blob/master/local-ap * region.tf * tf-run.data -and the apply will +and the apply will * generate `remote_state.yml` * create links to parent directory, `includes.d` files as needed, links to credentials, variables, and provider files. @@ -142,14 +142,14 @@ Note that apply statements are to be done after review and merging of the code. ```shell # create policy(ies) first -tf-plan -target=aws_iam_policy.policy_app -tf-apply -target=aws_iam_policy.policy_app +tf-plan -target=aws_iam_policy.policy_app +tf-apply -target=aws_iam_policy.policy_app # create LDIF marker file tf-plan -tf-apply +tf-apply # create ldap object (role) tf-plan -tf-apply +tf-apply ``` The targeted apply creates the policy(ies). Include all policies you define in the configuration as targets. @@ -201,4 +201,3 @@ Any errors saying something like "branch is not fully merged" need to be investi If any of the terraform steps or other commands fail, please submit an issue through the github UI. Select the Bug Report, and enter the requested information. This will help quickly get to resolution, and will help others who may run into similar issues to check the closed issues log before submitting their own issue. - diff --git a/docs/how-to/aws-eks-change-node-group/README.md b/docs/how-to/aws-eks-change-node-group/README.md index 5081f0b2..b18abb64 100644 --- a/docs/how-to/aws-eks-change-node-group/README.md +++ b/docs/how-to/aws-eks-change-node-group/README.md @@ -27,7 +27,7 @@ tf-apply # CHANGELOG -* 1.0.0 -- 2022-10-27 +* 1.0.0 -- 2022-10-27 * initial -* 1.0.1 -- 2024-05-08 +* 1.0.1 -- 2024-05-08 * added description of changing instance size diff --git a/docs/how-to/aws-eks-destroy/README.md b/docs/how-to/aws-eks-destroy/README.md index 15446b84..c1fa875d 100644 --- a/docs/how-to/aws-eks-destroy/README.md +++ b/docs/how-to/aws-eks-destroy/README.md @@ -2,7 +2,7 @@ # purpose - Remove eks cluster and all its created resources. # Assumption: - - All steps + - All steps Steps: # Assumption: - All steps below are executed from root directory of the EKS cluster. @@ -19,12 +19,12 @@ ALL 3) List all stored secrets in git and removed them. git-secret list|grep NAME_OF_CLUSTER - + git-secret remove FILE_NAMES_RETURNED_BY_ABOVE_COMMAND - + verify files are removed from git-secret. git-secret list|grep NAME_OF_CLUSTER - + 4) cd cluster-roles cp ../tf-run.destroy.data . tf-init @@ -32,14 +32,14 @@ ALL 5) cd cd ../common-services cp ../tf-run.destroy.data . - - Removed key_algorithm = "RSA" from "tls_cert_request" "ca" resource in ca-cert.tf file to avoid an error thrown by terraform. - + + Removed key_algorithm = "RSA" from "tls_cert_request" "ca" resource in ca-cert.tf file to avoid an error thrown by terraform. + resource "tls_cert_request" "ca" { #key_algorithm = "RSA" tf-init tf-destroy - + 6) cd ../irsa-roles/cluster-autoscaler/ cp ../../tf-run.destroy.data . tf-init @@ -212,7 +212,7 @@ tf-run clean rm -rf .terraform ``` -Now, you'll commit all the changes (LOTS of deleted files). It may have some additions. Make sure to +Now, you'll commit all the changes (LOTS of deleted files). It may have some additions. Make sure to commit with `a` as you have a git-secret change. ```script diff --git a/docs/how-to/aws-rename-account/README.md b/docs/how-to/aws-rename-account/README.md index d4baf9a1..e9d9a5d9 100644 --- a/docs/how-to/aws-rename-account/README.md +++ b/docs/how-to/aws-rename-account/README.md @@ -38,17 +38,17 @@ git pull git checkout -b update-dapps-common cd inventory/ -vi inventory.yml +vi inventory.yml [ make your changes ] cd group_vars/ cd adsd-dapps-dev-ew -vi main.yml +vi main.yml [ make your changes ] cd .. cd adsd-dapps-dev-gov/ -vi main.yml +vi main.yml [ make your changes ] cd .. @@ -59,11 +59,10 @@ git add . git status git commit -m "update adsd-dapps-dev to adsd-dapps-common" git branch -git push -u origin update-dapps-common +git push -u origin update-dapps-common PR ``` 4. update the organization configurations (EW, GovCloud) 5. change the git repository for each - diff --git a/docs/how-to/aws-setup-cloudforms/README.md b/docs/how-to/aws-setup-cloudforms/README.md index 4ac90605..1912a59b 100644 --- a/docs/how-to/aws-setup-cloudforms/README.md +++ b/docs/how-to/aws-setup-cloudforms/README.md @@ -1,6 +1,6 @@ # How To Setup and Enable CloudForms Integration -* enable INF*tf +* enable INF*tf * apply policy, service account * create access keys * subscribe CF SQS to default Config SNS (with TF, perhaps) @@ -10,7 +10,7 @@ * Clone or pull from master the baseline repo, and reveal secrets \# on iebcloud, make sure you have a workspace - * cd /data/terraform/workspaces/$USER/terraform/ + * cd /data/terraform/workspaces/$USER/terraform/ * git clone git@github.e.it.census.gov:terraform/ACCOUNTNUMBER-maXX-gov.git * git pull origin master * git-secret reveal -f diff --git a/docs/how-to/aws-sso/README.md b/docs/how-to/aws-sso/README.md index ea1cbd03..ae8788b1 100644 --- a/docs/how-to/aws-sso/README.md +++ b/docs/how-to/aws-sso/README.md @@ -22,8 +22,8 @@ A more detailed conversion document is available [here](convert.md). If you are ## Restore Prior Credentials -To return the saved credentials from the -[steps above](#save-old--aws-configs) +To return the saved credentials from the +[steps above](#save-old--aws-configs) , execute the following: ```console @@ -33,8 +33,8 @@ To return the saved credentials from the % mv config.save config ``` -You may also leave them in place and use -[environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) +You may also leave them in place and use +[environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) to reference the saved files: ```console @@ -42,22 +42,22 @@ to reference the saved files: % setenv AWS_SHARED_CREDENTIALS_FILE $HOME/.aws/credentials.save ``` -You are free, of course, to come up with another method by which to switch -between these. Keep in mind though that once SSO is fully in place, IAM -accounts will be removed, and there will not be much need for the +You are free, of course, to come up with another method by which to switch +between these. Keep in mind though that once SSO is fully in place, IAM +accounts will be removed, and there will not be much need for the `credentials` file. `config` will contain those profiles generated -though the `refresh-profile.sh` script as well as any additional assume-role +though the `refresh-profile.sh` script as well as any additional assume-role based profiles created (say, for EKS). # SSO Configuration ## Obtain the SSO URL for the cloud you need to work in -Use the below redirector link to get you to the proper SSO for console access. -The links are different for GovCloud and Commercial and each underlying SSO +Use the below redirector link to get you to the proper SSO for console access. +The links are different for GovCloud and Commercial and each underlying SSO link may change so using these will make sure it continues to work. -* Redirector Links +* Redirector Links * Enterprise GovCloud [ent-gov](http://r.census.gov/a/aws/sso/gov) * Enterprise Commercial/EW [ent-ew](http://r.census.gov/a/aws/sso) * Lab GovCloud [lab-gov](http://r.census.gov/a/aws/sso/lab-gov) @@ -76,10 +76,10 @@ The scripts listed here assume you are on `iebcloud.csvd.census.gov`. ### Configure Login Profiles -Create a new `$HOME/.aws/config` (after saving above) or add this to the -beginning of the file, one time only, to simplify your daily login. +Create a new `$HOME/.aws/config` (after saving above) or add this to the +beginning of the file, one time only, to simplify your daily login. -Below is an example as of this writing. Please use the current, appropriate +Below is an example as of this writing. Please use the current, appropriate SSO URL's you obtained in the steps above. ```script @@ -124,7 +124,7 @@ at https://github.e.it.census.gov/terraform/support/blob/master/docs/how-to/aws- An example: ```console -% aws-sso-login.sh +% aws-sso-login.sh * checking for profile ent-gov * logging into profile ent-gov Browser will not be automatically opened. @@ -150,10 +150,10 @@ Here's a screen recording of what this should look like: - As shown above, navigate to the URL with the code autofilled - At the Authorize Request screen, click the Allow button. - After you have authenticated successfully through your browser, close the browser - tab and return to your terminal window. -- You should now be authenticated for any account in the cloud you passed to the + tab and return to your terminal window. +- You should now be authenticated for any account in the cloud you passed to the sso command. -- A token is created in your `$HOME/.aws/sso/cache` directory that will +- A token is created in your `$HOME/.aws/sso/cache` directory that will authenticate you into any account in the corresponding cloud for up to the time configured for the specific permission set(s), which does vary. - **You should not need to re-authenticate through your browser for any of @@ -161,9 +161,9 @@ Here's a screen recording of what this should look like: ## Run or refresh profiles -- ent-gov -- ent-ew -- lab-gov +- ent-gov +- ent-ew +- lab-gov Run the command `refresh-profile.sh` with one of the above arguments. If you omit the argument it defaults to `ent-gov`. If you are already logged into the specific AWS SSO organization, you will not be promptd diff --git a/docs/how-to/aws-sso/convert.md b/docs/how-to/aws-sso/convert.md index 378672e1..efda30f4 100644 --- a/docs/how-to/aws-sso/convert.md +++ b/docs/how-to/aws-sso/convert.md @@ -59,7 +59,7 @@ though we are working to extend that to 16. ## Create base profiles -Given you have a near empty configuration file, you'll need to run the `refresh-profile.sh` to populate the +Given you have a near empty configuration file, you'll need to run the `refresh-profile.sh` to populate the profiles. You will also need to run this script each time we add new accounts or if you have been granted access to additional permissionsets (or removed, too). diff --git a/docs/how-to/aws-sso/edl-project-group.md b/docs/how-to/aws-sso/edl-project-group.md index 031a1ae2..30629787 100644 --- a/docs/how-to/aws-sso/edl-project-group.md +++ b/docs/how-to/aws-sso/edl-project-group.md @@ -25,7 +25,7 @@ Next, create a new project group directory from a template directory. This sets ```script cd infrastructure/global/sso/permissionsets/edl-projects -export PROJECT=nnnnnnn # enter the correct project number +export PROJECT=nnnnnnn # enter the correct project number mkdir edl-u-$PROJECT rsync -avRWH TEMPLATE/./ edl-u-$PROJECT cd edl-u-$PROJECT diff --git a/docs/how-to/aws-sso/manage-user.md b/docs/how-to/aws-sso/manage-user.md index a757300c..ad1b4918 100644 --- a/docs/how-to/aws-sso/manage-user.md +++ b/docs/how-to/aws-sso/manage-user.md @@ -11,7 +11,7 @@ The AWS account and repositories for adding users are in the AWS IAM Identity Ce | Organization | Description | Location | |--------------|-------------|----------| | ent-gov | GovCloud | [252903981224-ma5-gov](https://github.e.it.census.gov/terraform/252903981224-ma5-gov/tree/master/infrastructure/global/sso) | -| ent-ew | Commercial | [109223337795-censusaws](https://github.e.it.census.gov/terraform/109223337795-censusaws/tree/master/infrastructure/global/sso) | +| ent-ew | Commercial | [109223337795-censusaws](https://github.e.it.census.gov/terraform/109223337795-censusaws/tree/master/infrastructure/global/sso) | | lab-gov | Lab GovCloud | [243219719746-lab-gov-management-nonprod](https://github.e.it.census.gov/terraform/243219719746-lab-gov-management-nonprod/tree/master/infrastructure/global/sso) | ## Adding a new user to IDC diff --git a/docs/how-to/aws-sso/native-sso-setup.md b/docs/how-to/aws-sso/native-sso-setup.md index 9aa57a02..1d76a750 100644 --- a/docs/how-to/aws-sso/native-sso-setup.md +++ b/docs/how-to/aws-sso/native-sso-setup.md @@ -165,7 +165,7 @@ You can now use this profile to access the account: # Additional settings -If your permissionset allows you to assume a role, you can link back to the SSO profile to set it up. For example, to assume an EKS +If your permissionset allows you to assume a role, you can link back to the SSO profile to set it up. For example, to assume an EKS cluster admin role with the SSO profile I setup above, I would do something like this: ```console @@ -184,7 +184,7 @@ aws --profile my-account-eks whoami "Arn": "arn:aws-us-gov:sts::412187151792:assumed-role/r-eks-csvd-datadog-poc-cluster-admin/badra001" } ``` - + # Links * [AWS CLI Installation](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) diff --git a/docs/how-to/aws-sso/requirements.md b/docs/how-to/aws-sso/requirements.md index dac5a9ee..74d84a88 100644 --- a/docs/how-to/aws-sso/requirements.md +++ b/docs/how-to/aws-sso/requirements.md @@ -52,7 +52,7 @@ ent-gov (primary one), and lab-gov (not addressed here, nor visable from the ent # How +it affects specific permissions and/or policies which are to be applied to the configuration. --> # Additional Information diff --git a/docs/how-to/aws-vpc-setup/README.md b/docs/how-to/aws-vpc-setup/README.md index ca8a47eb..9b1a30a0 100644 --- a/docs/how-to/aws-vpc-setup/README.md +++ b/docs/how-to/aws-vpc-setup/README.md @@ -2,17 +2,17 @@ This section describe how to handle VPC configurations and setup. -* [Setup for using a shared VPC](shared-vpc.md) +* [Setup for using a shared VPC](shared-vpc.md) This describes the step to share a VPC from the `network-prod` account into another account. This is our primary means of setting up VPC access in accounts -* [Setup for shared VPC Endpoints](shared-vpc-endpoints.md) +* [Setup for shared VPC Endpoints](shared-vpc-endpoints.md) This describes how to convert an existing VPC configuration with local VPC-specific VPC endpoints to use the common shared VPC endpoints defined in the `ent-gov-network-prod` account. -* [Create a VPC in an account](local-vpc.md) +* [Create a VPC in an account](local-vpc.md) This describes the steps for creating a VPC in an account. We are moving towards shared VPCs so this will no longer be the primary VPC activity -* [Transit Gateway post-migration](tgw-post-migration.md) +* [Transit Gateway post-migration](tgw-post-migration.md) Now that we have fully migrated all VPCs to Transit Gateway, we must go through each VPC and destroy the VPN, peers, and the Cisco ASR on-prem VPN configuration (TCO task) diff --git a/docs/how-to/aws-vpc-setup/local-vpc.md b/docs/how-to/aws-vpc-setup/local-vpc.md index aec4ceec..cd5dfdaf 100644 --- a/docs/how-to/aws-vpc-setup/local-vpc.md +++ b/docs/how-to/aws-vpc-setup/local-vpc.md @@ -23,7 +23,7 @@ and then use `tf-run` to perform the setup. The steps are ## Directory Setup -In your repo, you will create a a directory under the git root at `vpc/{region}/vpc{number}` where {region} is +In your repo, you will create a a directory under the git root at `vpc/{region}/vpc{number}` where {region} is * GovCloud * east diff --git a/docs/how-to/aws-vpc-setup/shared-vpc-endpoints.md b/docs/how-to/aws-vpc-setup/shared-vpc-endpoints.md index 72d64954..eeae7727 100644 --- a/docs/how-to/aws-vpc-setup/shared-vpc-endpoints.md +++ b/docs/how-to/aws-vpc-setup/shared-vpc-endpoints.md @@ -94,7 +94,7 @@ cd ACCOUNT-REPO-DIR git grep -Ei -A 4 -n 'vpc_endpoint|sourcevpce' ``` -Look through the results for files NOT `vpc-endpoints.tf` or the directory `vpc-endpoints/`. You may need to validate that removing those will not cause issues, +Look through the results for files NOT `vpc-endpoints.tf` or the directory `vpc-endpoints/`. You may need to validate that removing those will not cause issues, which involves a partial setup of the shared endpoints. Contact @badra001 for assistance if this is the case. We have had policies for ECR using the local VPC endpoint that conitnued to work without issue after changing over to the @@ -184,19 +184,19 @@ git commit -m'convert vpc-endpoints to shared endpoints' . There are a few outputs from the code to help you identify what specific endpoints and zones were shared. -* vpc_endpoints_ssm +* vpc_endpoints_ssm This shows you the System Manager Parameter store paths (from the network-prod account) for each of the shared endpoints. -* vpc_endpoints_ssm_ids +* vpc_endpoints_ssm_ids This shows the VPC endpoint IDs, in the network-prod account, per VPC endpoint service name. -* vpc_endpoints_ssm_zone_ids +* vpc_endpoints_ssm_zone_ids This shows the associated Route53 PHZ IDs, in the network-prod account, per VPC endpoint service name. Here are few snippets -## vpc_endpoints_ssm +## vpc_endpoints_ssm ```hcl vpc_endpoints_ssm = { @@ -221,7 +221,7 @@ vpc_endpoints_ssm = { ``` -## vpc_endpoints_ssm_ids +## vpc_endpoints_ssm_ids ```hcl vpc_endpoints_ssm_ids = { @@ -241,7 +241,7 @@ vpc_endpoints_ssm_ids = { } ``` -## vpc_endpoints_ssm_zone_ids +## vpc_endpoints_ssm_zone_ids ```hcl vpc_endpoints_ssm_zone_ids = { diff --git a/docs/how-to/aws-vpc-setup/shared-vpc.md b/docs/how-to/aws-vpc-setup/shared-vpc.md index 6b063b9a..b8136a4b 100644 --- a/docs/how-to/aws-vpc-setup/shared-vpc.md +++ b/docs/how-to/aws-vpc-setup/shared-vpc.md @@ -4,7 +4,7 @@ We are provisioning VPC access in accounts through the use of centrally defined organziation and/or internal/external configuration (internal vs dmz), to be used across the enterprise. This will allow better utilization of the address space and easier setup in new accounts. -To share a VPC in an account, there are a few setup steps. +To share a VPC in an account, there are a few setup steps. * network-prod, dmz-network-prod, or lab-network-nonprod (based on what environment you are using) * add account or OU to shared VPC setup @@ -57,7 +57,7 @@ server cluster communications, and will not be a valid routable address on our n will want/need something more specific, such as `dev.adsd.csp1.census.gov`. There are two ways to handle this 1. If this is the first account which will use the zone , follow the steps for creating the dummy VPC above, in order to create the local PHZ. 1. If this zone exists in some other account, the `vpcN/apps/dns` setup will need to include code to reference this remote PHZ. Get the -code from [here](https://github.e.it.census.gov/terraform-modules/aws-vpc-setup/tree/tf-upgrade/examples/vpc-apps-dns-remote-zone). The remote +code from [here](https://github.e.it.census.gov/terraform-modules/aws-vpc-setup/tree/tf-upgrade/examples/vpc-apps-dns-remote-zone). The remote account will need the `common/apps/remote-roles` setup from the [remote-roles example code](https://github.e.it.census.gov/terraform-modules/aws-vpc-setup/tree/tf-upgrade/examples/common-apps-remote-roles) 1. See the relevant README in each of these two refrenced examples for setup. 1. For deployment of EC2 instances (say, with CloudForms), the domain name used for the servers should not come from the default DHCP option. It should be specified in the creation, again, something like @@ -124,9 +124,9 @@ the `tf-cli version` command shows a version less than 1.x. | VPC{n} | Label | Purpose | |--------|-------|---------| | vpc1 | vpc1-gen-common | Common applications accessible from any environment (services and shared, possibly, but they belong in an infrastructure account) | -| vpc2 | vpc2-gen-dev | Development | +| vpc2 | vpc2-gen-dev | Development | | vpc3 | vpc3-gen-qa | QA (Quality Assurance; a subset of test) | -| vpc4 | vpc4-gen-uat | UAT (User Acceptance Test; a subset of test) | +| vpc4 | vpc4-gen-uat | UAT (User Acceptance Test; a subset of test) | | vpc5 | vpc5-gen-ite | ITE (Integrated Test Environment; a subset of test) | | vpc6 | vpc6-gen-stage | Stage (performance, stress test) | | vpc8 | vpc8-gen-test | Test (not one of QA, UAT, or ITE) | @@ -156,7 +156,7 @@ the `tf-cli version` command shows a version less than 1.x. | VPC{n} | Label | Purpose | |--------|-------|---------| | vpc8 | vpc8-dmz-prod | Production | - + ### lab-gov internal (lab-gov-network-nonprod) There is no production or staging function in the lab. @@ -165,8 +165,8 @@ There is no production or staging function in the lab. |--------|-------|---------| | vpc1 | vpc1-lab-services | Lab services and "enterprise" shared applications | | vpc2 | vpc2-lab-common | Common applications accessible from any environment (services and shared, possibly, but they belong in an infrastructure account) | -| vpc3 | vpc3-lab-dev | Development | -| vpc4 | vpc4-lab-test | Test | +| vpc3 | vpc3-lab-dev | Development | +| vpc4 | vpc4-lab-test | Test | ## network account @@ -183,14 +183,14 @@ Go into that git repository. * 273715889907-ent-gov-dmz-network-prod * lab-gov internal lab-network-nonprod * 269244441389-lab-gov-network-nonprod - + Then, go to the directory with desired region and selected VPC number. You **must** use one from the table above, as the files are copied from the network account in a later step. There is no variation permitted here. ```script cd vpc-shared/west/general cd vpc5 # ite -``` +``` * edit `variables.share.auto.tfvars` * add account or OU @@ -208,7 +208,7 @@ share_account_list = [ ``` ```console -% tf-plan +% tf-plan . . % tf-plan summary @@ -334,7 +334,7 @@ from the same `terraform-modules/aws-vpc-setup` repo (and tf-upgrade branc). ```console % cd TARGET-ACCOUNT-REPO % cd vpc/west -% VPC=vpc5 +% VPC=vpc5 % mkdir $VPC % cd $VPC diff --git a/docs/how-to/aws-vpc-setup/tgw-post-migration.md b/docs/how-to/aws-vpc-setup/tgw-post-migration.md index 93a62fa0..65d810f2 100644 --- a/docs/how-to/aws-vpc-setup/tgw-post-migration.md +++ b/docs/how-to/aws-vpc-setup/tgw-post-migration.md @@ -318,7 +318,7 @@ push, and do a PR. # CHANGELOG * 1.0.0 - - created + - created * 1.0.1 -- 2023-07-17 - add details for creating vpn-configs if they did not exist * 1.0.2 -- 2023-07-19 diff --git a/docs/how-to/git/add-user-to-git-secret.md b/docs/how-to/git/add-user-to-git-secret.md index ac8cca64..89bf0d59 100644 --- a/docs/how-to/git/add-user-to-git-secret.md +++ b/docs/how-to/git/add-user-to-git-secret.md @@ -3,7 +3,7 @@ This describes the steps to add a user's GPG key to the main `support` keyring, as well as how to add it to the `git-secret` setup in each account. -The key creation and addition to the support repo is a one time activity per user. After that, adding the user to an +The key creation and addition to the support repo is a one time activity per user. After that, adding the user to an account's `git-secret` setup is straight foward. ## User Creates the key diff --git a/docs/how-to/git/workflow.md b/docs/how-to/git/workflow.md index b1c61a89..aa81d437 100644 --- a/docs/how-to/git/workflow.md +++ b/docs/how-to/git/workflow.md @@ -143,7 +143,7 @@ may be listed in the PR comments, if something different). ## Merged Code After merge, you will make sure you move back to the master branch and apply your changes. Sometimes, there will be additional -files generate by your code. You want to be sure to capture those, and then complete a post-apply PR. In this case, you +files generate by your code. You want to be sure to capture those, and then complete a post-apply PR. In this case, you may stay in the branch, but pull in master: ```script diff --git a/docs/how-to/review-pull-request/README.md b/docs/how-to/review-pull-request/README.md index b7fe50dd..3d9bcb0f 100644 --- a/docs/how-to/review-pull-request/README.md +++ b/docs/how-to/review-pull-request/README.md @@ -14,7 +14,7 @@ - (respond to comments) - (wait for merge) - apply -1. All communications about the PR need to stay in the PR. People looking at it later will need that context. You can chat someone to +1. All communications about the PR need to stay in the PR. People looking at it later will need that context. You can chat someone to go look at something, but keep comments in the PR. Comment at the bottom, or inline in sections of code. 1. Ask questions if you don't understand (you can @USERNAME someone in the comment boxes) 1. Use markdown in commenting diff --git a/docs/how-to/review-pull-request/reviewer-idc.md b/docs/how-to/review-pull-request/reviewer-idc.md index 56f6cc18..19bd430a 100644 --- a/docs/how-to/review-pull-request/reviewer-idc.md +++ b/docs/how-to/review-pull-request/reviewer-idc.md @@ -32,7 +32,7 @@ submitters failing to pull the master branch before starting. They may also be has been removed from our directory but is still in this configuration. There may be attribute changes (case changes to the email address and username do cause issues because the username is case sensitive in IDC). -There is a script called `./check-users.sh` which will read the `users.csv` file and report any issues. If this +There is a script called `./check-users.sh` which will read the `users.csv` file and report any issues. If this results in output, these users often need to be removed from the file. You may need to look them up in LDAP to see what is going wrong. @@ -42,8 +42,8 @@ The base directory we are talking about in the appropriate repository is `infras to this as `$BASE` in this document. - [ ] To add a user, the JBID must appear in `$BASE/users.csv`. This may be a two-part PR, or it could be all included -in one. Plan output for the group-specific change(s) will fail if the user is not already present in IDC, so -it is more common to see two PRs. Be sure that the full context of the request is included in the PR. That is, +in one. Plan output for the group-specific change(s) will fail if the user is not already present in IDC, so +it is more common to see two PRs. Be sure that the full context of the request is included in the PR. That is, even though it is two-step process (one for the user, one for the user in the group), the details should be complete to know what the ultimate change is going to be (i.e., add user X to group Y). - [ ] When there are two PRs (one for user in IDC, one for user in group), the second one should reference diff --git a/docs/how-to/ssh-agent-setup/README.md b/docs/how-to/ssh-agent-setup/README.md index 980214ed..3473343f 100644 --- a/docs/how-to/ssh-agent-setup/README.md +++ b/docs/how-to/ssh-agent-setup/README.md @@ -104,7 +104,7 @@ You can add the keys in the `.bashrc` above (assuming they are not there) as lis % ssh-add $HOME/.ssh/filename # add main id_* key -% ssh-add +% ssh-add # clear agent % ssh-add -D diff --git a/docs/how-to/submit-pull-request/README.md b/docs/how-to/submit-pull-request/README.md index 11586245..a674d1ab 100644 --- a/docs/how-to/submit-pull-request/README.md +++ b/docs/how-to/submit-pull-request/README.md @@ -29,11 +29,11 @@ Edit or create your code. Follow these tips (this should be its own page!): * no hardcoding of resource names, region, account ids, ARNs, etc., unless completely unavoidable * use `data` resources to get things like a VPC ID, subnet ids, etc. * keep your code DRY: Don't Repeat Yourself. This means using loops and data structures vs copy/paste of the same code with different names -* use Terraform 1.x for new directories. This is the default setup for accounts created after MA10, and it can be setup per directory. +* use Terraform 1.x for new directories. This is the default setup for accounts created after MA10, and it can be setup per directory. See [tf-init upgrade](https://github.e.it.census.gov/terraform/support/tree/master/local-app/tf-run#init-upgrade) * use Terraform 0.13+ interpolation (i.e., `aws_vpc.vpc.id` vs `${aws_vpc.vpc.id}`) * use `for_each` instead of `count` when dealing with multiple resources. `count` is still good for true/false type resource generation -* be sure to use the `tf-run.data` file to setup the flow needed for the code. `tf-run init` gives you a basic starting point. Things like +* be sure to use the `tf-run.data` file to setup the flow needed for the code. `tf-run init` gives you a basic starting point. Things like extra links, POLICY, and so forth need to be here. The goal is a complete run of all the code in steps necessary through `tf-run`. * run `tf-fmt` to nicely format the code before commiting * run `tf-plan` to see changes, and if there are a lot of changes, `tf-plan summary` to see a nice listing of what is going on @@ -96,7 +96,7 @@ through the PR if needed), and then commit, push, and add a new plan log (if app code from the master branch. -## Screenshots +## Screenshots ![Alt text](images/submit-pr-1.png?raw=true) ![Alt text](images/submit-pr-2.png?raw=true) diff --git a/docs/how-to/terraform-code-tips/README.md b/docs/how-to/terraform-code-tips/README.md index 56084542..8bca5a4d 100644 --- a/docs/how-to/terraform-code-tips/README.md +++ b/docs/how-to/terraform-code-tips/README.md @@ -23,7 +23,7 @@ 1. All communicatiosn about the PR need to stay in the PR. Comment at the bottom, or inline in sections of code. 1. Use markdown in commenting 1. Add output from a `tf-plan` (upload log) -1. Follow the PR template +1. Follow the PR template - [ ] Description: present and has some content. A bulleted list is fine. - [ ] Purpose: present and has some content. Slightly longer than description - [ ] Screenshots: It is rare you will add something here. diff --git a/docs/how-to/terraform-code-tips/enterprise-ldap-changes.md b/docs/how-to/terraform-code-tips/enterprise-ldap-changes.md index 8cd3b252..40f0f6fb 100644 --- a/docs/how-to/terraform-code-tips/enterprise-ldap-changes.md +++ b/docs/how-to/terraform-code-tips/enterprise-ldap-changes.md @@ -21,10 +21,10 @@ The server replacements are as folows: # Affected Modules -* aws-iam-role [tf-0.12](#aws-iam-role--terraform-0-12) [tf-1.x](#aws-iam-role--terraform-1-x) -This has been upgraded in both the master branch and tf-upgrade branch to handle the issue. The documentation is also updated. - * https://github.e.it.census.gov/terraform-modules/aws-iam-role/releases/tag/1.4.2 - * https://github.e.it.census.gov/terraform-modules/aws-iam-role/releases/tag/2.3.2 +* aws-iam-role [tf-0.12](#aws-iam-role--terraform-0-12) [tf-1.x](#aws-iam-role--terraform-1-x) +This has been upgraded in both the master branch and tf-upgrade branch to handle the issue. The documentation is also updated. + * https://github.e.it.census.gov/terraform-modules/aws-iam-role/releases/tag/1.4.2 + * https://github.e.it.census.gov/terraform-modules/aws-iam-role/releases/tag/2.3.2 * ldap-get-attribute Not updated yet diff --git a/docs/how-to/terraform-directory-removal/README.md b/docs/how-to/terraform-directory-removal/README.md index 1d86a9cd..b110ce63 100644 --- a/docs/how-to/terraform-directory-removal/README.md +++ b/docs/how-to/terraform-directory-removal/README.md @@ -19,7 +19,7 @@ The steps are: 1. another commit for the move/archive of the directory 1. submit a final PR completing the change -## Sample +## Sample This assumes you are working within the directory under which you desire to destroy all resources and archive. @@ -54,7 +54,7 @@ git push ``` Now submit a PR. In the body of the PR, describe that you are destroying and archiving the directory. Please -include a reference to a Remedy or Jira ticket from which this was requested and/or approved. Paste in the +include a reference to a Remedy or Jira ticket from which this was requested and/or approved. Paste in the `tf-destroy summary` output (surrounded with markdown 3 backticks at the top and bottom). ### Destroy @@ -68,7 +68,7 @@ tf-run destroy ``` You may need to reach out for help if you have any failures in this step. Make sure all resources have been destroyed -berfore continuing. +berfore continuing. ``` tf-state list @@ -118,7 +118,7 @@ remote state and lock, otherwise very odd things happen. manage-remote-state.sh delete ``` -This gets a copy of the rmeote state and the lock entry, and stores it under `logs/DATESTAMP`. This is useful in case we need to +This gets a copy of the rmeote state and the lock entry, and stores it under `logs/DATESTAMP`. This is useful in case we need to examine what it used to look like. ### Clean generated files diff --git a/docs/how-to/terraform-getting-started/README.md b/docs/how-to/terraform-getting-started/README.md index fbf339d6..9e9566ed 100644 --- a/docs/how-to/terraform-getting-started/README.md +++ b/docs/how-to/terraform-getting-started/README.md @@ -19,7 +19,7 @@ Follow the documentation at [SSH key](https://github.e.it.census.gov/terraform/s Follow the documentation at [GPG key](https://github.e.it.census.gov/terraform/support/blob/master/docs/GETTING-STARTED.md#create-gpg-key) -## Working directory +## Working directory To start, we must be in the directory for the account git repository. The base directory for Terraform usage is `/data/terraform/workspaces`. Each user gets a directory created here under their username. Currently, you must request this of one of: Roy Ashley, Derryle Gogel, or Don Badrak. This is not an automated @@ -30,8 +30,8 @@ would use a different diectory name. ```console # first time: -% mkdir /data/terraform/workspaces/$USER/terraform/ -% cd /data/terraform/workspaces/$USER/terraform/ +% mkdir /data/terraform/workspaces/$USER/terraform/ +% cd /data/terraform/workspaces/$USER/terraform/ ``` You will then be able to clone the needed repositories in this location. diff --git a/docs/how-to/terraform-upgrade/tf-upgrade-fix-one.sh b/docs/how-to/terraform-upgrade/tf-upgrade-fix-one.sh index 5c218c0d..2ff0dac4 100755 --- a/docs/how-to/terraform-upgrade/tf-upgrade-fix-one.sh +++ b/docs/how-to/terraform-upgrade/tf-upgrade-fix-one.sh @@ -1,3 +1,2 @@ sed -i -e 's/one(\(.*\))/try(\1,null)/' $@ sed -i -e 's/\[\*\]/[0]/g' $@ - diff --git a/docs/how-to/terraform-upgrade/upgrade-code.md b/docs/how-to/terraform-upgrade/upgrade-code.md index 9a082d62..740fa928 100644 --- a/docs/how-to/terraform-upgrade/upgrade-code.md +++ b/docs/how-to/terraform-upgrade/upgrade-code.md @@ -8,7 +8,7 @@ we run Terraform. Here is the [reference documentation](https://github.e.it.cens # Changes Needed -If you have deployed a new account from Ansible, all of the code is in place already for Terraform 1.x. If not, will need to +If you have deployed a new account from Ansible, all of the code is in place already for Terraform 1.x. If not, will need to make a few changes. `tf-run` is setup so that it will detect the Terraform version by using a config file located either in diff --git a/docs/how-to/terraform-upgrade/upgrade-state.md b/docs/how-to/terraform-upgrade/upgrade-state.md index ae08e5af..88d06d9f 100644 --- a/docs/how-to/terraform-upgrade/upgrade-state.md +++ b/docs/how-to/terraform-upgrade/upgrade-state.md @@ -134,7 +134,7 @@ Answer `yes`. This will setup a `versions.tf` file, which will will update/repl Next we check to see if any module calls need to be updated. ```script -tf-run check +tf-run check ``` If this returns anything, please fix the module calls to update the `source` statement. This usually means adding `?ref=tf-upgrade` @@ -265,7 +265,7 @@ Comment the line for 0.14.11 in `.tf-control`: This leaces the first occurence ## Reconfigure 1.x -Next, we reconfigure for 1.x. +Next, we reconfigure for 1.x. ```script tf-init -reconfigure @@ -383,7 +383,7 @@ Wait until after you have done a successful `tf-apply` before any further `tf-in ## ECS task changes You may see during one of the upgrade steps a change to an ECS task, most often due to a change -in the task definition external to Terraform. +in the task definition external to Terraform. ```script # aws_ecs_service.app will be updated in-place @@ -424,7 +424,7 @@ resource "aws_ecs_service" "app" { security_groups = [local.sg_web_id] assign_public_ip = false } - + propagate_tags = "TASK_DEFINITION" } @@ -451,10 +451,10 @@ This was added in 0.14, so we will remove it temporarily. ``` Error: Invalid type specification - + on .terraform/modules/ec2_project/variables.tf line 212, in variable "volumes": 212: size = optional(number, 10) - + Keyword "optional" is not a valid type constructor. ``` @@ -473,24 +473,24 @@ For RDS, we need to replace 2 providers in Reconfigure 0.14 step ```hcl tf-state replace-provider registry.terraform.io/-/aws registry.terraform.io/hashicorp/aws -tf-state replace-provider registry.terraform.io/-/random registry.terraform.io/hashicorp/random +tf-state replace-provider registry.terraform.io/-/random registry.terraform.io/hashicorp/random ``` During the Reconfigure 0.14 step, we might see below errror. ```script Error: Unsupported argument - + on .terraform/modules/db/modules/db_instance/main.tf line 34, in resource "aws_db_instance" "this": 34: name = var.name - + An argument named "name" is not expected here. - + Error: Unsupported argument - + on .terraform/modules/db/modules/db_instance/main.tf line 140, in resource "aws_db_instance" "this_mssql": 140: name = var.name - + An argument named "name" is not expected here. ``` diff --git a/docs/how-to/terraform-upgrade/upgrade.vpc-code.sh b/docs/how-to/terraform-upgrade/upgrade.vpc-code.sh index 9526d258..e38c652f 100755 --- a/docs/how-to/terraform-upgrade/upgrade.vpc-code.sh +++ b/docs/how-to/terraform-upgrade/upgrade.vpc-code.sh @@ -2,7 +2,7 @@ VERSION="1.0.9" THIS=$(basename $0 .sh) - + ## VPCEXAMPLE="$HOME/terraform/terraform-modules/aws-vpc-setup/examples/full-setup-tf-upgrade" ## if [ ! -d "$VPCEXAMPLE" ] ## then diff --git a/docs/onboarding.md b/docs/onboarding.md index a5116433..ca2e81a4 100644 --- a/docs/onboarding.md +++ b/docs/onboarding.md @@ -1,4 +1,4 @@ -# Onboarding a Person Into AWS +# Onboarding a Person Into AWS ## Common Steps @@ -8,7 +8,7 @@ 1. Enter required information, specifically access required 1. Ask CSVD cloud team for clarification on any items 1. Once SRM approved by DC and CSVD: - 1. SAML: The ticket is forwarded to TCO for adding to the group + 1. SAML: The ticket is forwarded to TCO for adding to the group 1. IAM: Terraform code is created to add the user, and they are contacted and provided their access key 1. Need software on desired source system (Linux or Windows) 1. aws cli v2 diff --git a/docs/shared-terraform.md b/docs/shared-terraform.md index f5dfd8ea..8949bded 100644 --- a/docs/shared-terraform.md +++ b/docs/shared-terraform.md @@ -13,16 +13,16 @@ This setup allow for splitting the repo into different parts to allow other non- to the terraform code setup, while ensuring cloud admins maintain the required access to all code (and remote state) used to deploy to cloud. -* {primary_repository}_apps-{identifier} - * ex: 079788916859-do2-cat_apps-adsd-eks - * {primary_repository} = {account_id}-{account_alias} - * ex: 079788916859-do2-cat - * {identifier} = {org-or-program}-{project-or-component} - * ex: adsd-eks - * {org-or-program} = organization abbreviation or program name - * ex: adsd | dice | das - * {project-or-component} = project abbreviation or component name - * ex: eks | mojo | centurion +* {primary_repository}_apps-{identifier} + * ex: 079788916859-do2-cat_apps-adsd-eks + * {primary_repository} = {account_id}-{account_alias} + * ex: 079788916859-do2-cat + * {identifier} = {org-or-program}-{project-or-component} + * ex: adsd-eks + * {org-or-program} = organization abbreviation or program name + * ex: adsd | dice | das + * {project-or-component} = project abbreviation or component name + * ex: eks | mojo | centurion Two teams in github will be created to represent access to the new repository. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 8d0e7cdb..2b22022e 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -25,14 +25,14 @@ This guide helps diagnose and resolve common issues encountered when using the T unzip terraform_0.13.7_linux_amd64.zip -d /tmp sudo mv /tmp/terraform /usr/local/bin/terraform-0.13 ``` - + Repeat for other versions (0.14.x, 0.15.x, 1.x) as needed. 2. Verify Python version: ```bash python3 --version ``` - + Ensure you have Python 3.8 or newer. If not, install it through your package manager. ### Python Package Issues @@ -113,7 +113,7 @@ This guide helps diagnose and resolve common issues encountered when using the T [profile account1] role_arn = arn:aws:iam::123456789012:role/YourRole source_profile = base-profile - + [profile account2] role_arn = arn:aws:iam::210987654321:role/YourRole source_profile = base-profile @@ -211,7 +211,7 @@ This guide helps diagnose and resolve common issues encountered when using the T ```bash # Manually upgrade to 0.13 first make upgrade-version VERSION=0.13 DIR=/path/to/your/terraform/repo/module1 - + # Then continue to subsequent versions make upgrade-version VERSION=0.14 DIR=/path/to/your/terraform/repo/module1 ``` @@ -232,7 +232,7 @@ This guide helps diagnose and resolve common issues encountered when using the T source = "git::https://github.com/example/module.git?ref=v2.0.0" } ``` - + 3. Use the Census Bureau's compatible module versions for common modules. ## Git-Related Issues diff --git a/eks-tools/eksctl/get.sh b/eks-tools/eksctl/get.sh index 55001cfd..5cf485b1 100755 --- a/eks-tools/eksctl/get.sh +++ b/eks-tools/eksctl/get.sh @@ -4,7 +4,7 @@ ARG=$1 THIS=$(basename $0) BINNAME="eksctl" -URL="https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" +URL="https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" #SHA_URL="" if [ -z $VERSION ] @@ -37,7 +37,7 @@ then # fi # if [[ $status == 0 ]] && [[ $shastatus == 0 ]] - if [[ $status == 0 ]] + if [[ $status == 0 ]] then ## reformat checksum file # cat $shafile | awk '{print $1 " " $2}' > $shafile.tmp diff --git a/eks-tools/kubectl/get.sh b/eks-tools/kubectl/get.sh index 6346bb65..1518bb54 100755 --- a/eks-tools/kubectl/get.sh +++ b/eks-tools/kubectl/get.sh @@ -34,7 +34,7 @@ then else shastatus=0 fi - + if [[ $status == 0 ]] && [[ $shastatus == 0 ]] then # reformat checksum file diff --git a/git-secret.PATCH/git-secret.diffs b/git-secret.PATCH/git-secret.diffs index 24de2f5f..fa195dc5 100644 --- a/git-secret.PATCH/git-secret.diffs +++ b/git-secret.PATCH/git-secret.diffs @@ -6,6 +6,5 @@ local dest_file - dest_file="$(echo "$parms" | gawk -v RS="'" -v FS="'" 'END{ gsub(/^\s+/,""); print $1 }' | sed -e 's/^ *//')" + dest_file="$(echo "$parms" | gawk -v RS="'" -v FS="'" 'END{ gsub(/^\s+/,""); print $1 }')" - + _temporary_file - diff --git a/git-secret.PATCH/git-secret.dist b/git-secret.PATCH/git-secret.dist index a0eefafb..dbcad735 100755 --- a/git-secret.PATCH/git-secret.dist +++ b/git-secret.PATCH/git-secret.dist @@ -36,7 +36,7 @@ function __get_octal_perms_freebsd { filename=$1 local perms perms=$(stat -f "%04OLp" "$filename") - # perms is a string like '0644'. + # perms is a string like '0644'. # In the "%04OLp': # the '04' means 4 digits, 0 padded. So we get 0644, not 644. # the 'O' means Octal. @@ -70,18 +70,18 @@ function __temp_file_linux { local filename # man mktemp on CentOS 7: # mktemp [OPTION]... [TEMPLATE] - # ... + # ... # -p DIR, --tmpdir[=DIR] - # interpret TEMPLATE relative to DIR; if DIR is not specified, - # use $TMPDIR if set, else /tmp. With this option, TEMPLATE - # must not be an absolute name; unlike with -t, TEMPLATE may + # interpret TEMPLATE relative to DIR; if DIR is not specified, + # use $TMPDIR if set, else /tmp. With this option, TEMPLATE + # must not be an absolute name; unlike with -t, TEMPLATE may # contain slashes, but mktemp creates only the final component # ... - # -t interpret TEMPLATE as a single file name component, - # relative to a directory: $TMPDIR, if set; else the directory + # -t interpret TEMPLATE as a single file name component, + # relative to a directory: $TMPDIR, if set; else the directory # specified via -p; else /tmp [deprecated] - filename=$(mktemp -p "${TMPDIR}" _git_secret.XXXXXX ) + filename=$(mktemp -p "${TMPDIR}" _git_secret.XXXXXX ) # makes a filename like /$TMPDIR/_git_secret.ONIHo echo "$filename" } @@ -128,7 +128,7 @@ function __replace_in_file_osx { function __temp_file_osx { local filename - # man mktemp on OSX: + # man mktemp on OSX: # ... # "If the -t prefix option is given, mktemp will generate a template string # based on the prefix and the _CS_DARWIN_USER_TEMP_DIR configuration vari- @@ -136,8 +136,8 @@ function __temp_file_osx { # available are TMPDIR and /tmp." # we use /usr/bin/mktemp in case there's another mktemp available. See #485 - filename=$(/usr/bin/mktemp -t _git_secret ) - # On OSX this can make a filename like + filename=$(/usr/bin/mktemp -t _git_secret ) + # On OSX this can make a filename like # '/var/folders/nz/vv4_91234569k3tkvyszvwg90009gn/T/_git_secret.HhvUPlUI' echo "$filename"; } diff --git a/git-secret.PATCH/git-secret.fixed b/git-secret.PATCH/git-secret.fixed index a7eff53d..8651b13d 100755 --- a/git-secret.PATCH/git-secret.fixed +++ b/git-secret.PATCH/git-secret.fixed @@ -36,7 +36,7 @@ function __get_octal_perms_freebsd { filename=$1 local perms perms=$(stat -f "%04OLp" "$filename") - # perms is a string like '0644'. + # perms is a string like '0644'. # In the "%04OLp': # the '04' means 4 digits, 0 padded. So we get 0644, not 644. # the 'O' means Octal. @@ -70,18 +70,18 @@ function __temp_file_linux { local filename # man mktemp on CentOS 7: # mktemp [OPTION]... [TEMPLATE] - # ... + # ... # -p DIR, --tmpdir[=DIR] - # interpret TEMPLATE relative to DIR; if DIR is not specified, - # use $TMPDIR if set, else /tmp. With this option, TEMPLATE - # must not be an absolute name; unlike with -t, TEMPLATE may + # interpret TEMPLATE relative to DIR; if DIR is not specified, + # use $TMPDIR if set, else /tmp. With this option, TEMPLATE + # must not be an absolute name; unlike with -t, TEMPLATE may # contain slashes, but mktemp creates only the final component # ... - # -t interpret TEMPLATE as a single file name component, - # relative to a directory: $TMPDIR, if set; else the directory + # -t interpret TEMPLATE as a single file name component, + # relative to a directory: $TMPDIR, if set; else the directory # specified via -p; else /tmp [deprecated] - filename=$(mktemp -p "${TMPDIR}" _git_secret.XXXXXX ) + filename=$(mktemp -p "${TMPDIR}" _git_secret.XXXXXX ) # makes a filename like /$TMPDIR/_git_secret.ONIHo echo "$filename" } @@ -128,7 +128,7 @@ function __replace_in_file_osx { function __temp_file_osx { local filename - # man mktemp on OSX: + # man mktemp on OSX: # ... # "If the -t prefix option is given, mktemp will generate a template string # based on the prefix and the _CS_DARWIN_USER_TEMP_DIR configuration vari- @@ -136,8 +136,8 @@ function __temp_file_osx { # available are TMPDIR and /tmp." # we use /usr/bin/mktemp in case there's another mktemp available. See #485 - filename=$(/usr/bin/mktemp -t _git_secret ) - # On OSX this can make a filename like + filename=$(/usr/bin/mktemp -t _git_secret ) + # On OSX this can make a filename like # '/var/folders/nz/vv4_91234569k3tkvyszvwg90009gn/T/_git_secret.HhvUPlUI' echo "$filename"; } diff --git a/git-xargs-releases/README.md b/git-xargs-releases/README.md index d5937494..36b9879f 100644 --- a/git-xargs-releases/README.md +++ b/git-xargs-releases/README.md @@ -6,5 +6,3 @@ * https://blog.gruntwork.io/introducing-git-xargs-an-open-source-tool-to-update-multiple-github-repos-753f9f3675ec * github * https://github.com/gruntwork-io/git-xargs - - diff --git a/github-cli-releases/ghe b/github-cli-releases/ghe index 91f9ab81..a83d1256 100755 --- a/github-cli-releases/ghe +++ b/github-cli-releases/ghe @@ -2,4 +2,3 @@ export GH_HOST=github.e.it.census.gov gh $@ - diff --git a/helm-releases/get.sh b/helm-releases/get.sh index c0e57b39..b9d8fb66 100755 --- a/helm-releases/get.sh +++ b/helm-releases/get.sh @@ -1,7 +1,7 @@ #!/bin/bash get_latest_release() { - curl -k --silent "https://api.github.com/repos/helm/helm/releases/latest" | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/' + curl -k --silent "https://api.github.com/repos/helm/helm/releases/latest" | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/' } ARG=$1 @@ -109,4 +109,3 @@ then umask 022 cp ${PACKAGE}_${version} $BINDIR/ && ln -sf ${PACKAGE}_${version} $BINDIR/$PACKAGE fi - diff --git a/init/git-secret/ibekw001.gpg.asc b/init/git-secret/ibekw001.gpg.asc new file mode 100644 index 00000000..0d49d60e --- /dev/null +++ b/init/git-secret/ibekw001.gpg.asc @@ -0,0 +1,64 @@ +-----BEGIN PGP PUBLIC KEY BLOCK----- + +mQINBGTSbvsBEACzVFlm+I3TPiBYryEKWAV3XSg5caVmQ8aLBYw4ehMGIwp7a5pl +J9AZcfPVGXGeaHAxtPwigAJTn04WWdgNzb+SEDD9KigviSSCGGaW5yumsmdGo6OV +XxJIc+zllPAWfzran0CNtDbPxbncOzpgnGUQKhUNG/NQevR0bmvVxWt3Uz+4Ro6q +QsSG8dt6c7bqBCstRIyYwfYglYeL2YMU5sLwnsFsl5hYAzby8bLDP3dNqEuJG82x +wylTp3vKJxxs8IeNVAEmxdmj8VZiB/bnkl3jT8jirJRKqCvONtcYjLFgpcK1GZ6t +b1FQSTUZIJHCSnI++GrObWfEDpN5jPTTuwf1+W0eogZNsYMD369+DPxL9nNpW+I1 +FfnCqq6FLdbAg9wflrxXMSHZNSjj7k6zxHqx0gMTbD74eU84wiiWewzqX6LnrNuP +dO167L3e3lob9zcoDBcYdVN+q78iEuYKF509bj+pt9KXyx9FpMC598/QqTMbNJDM +fnnqjDHcL3pflrjcq/7g4ci/7MGAqfz8aA6qwpCNiC43/5EP1U58UEm3MgQdJ7jB +tyfwfgdqmWvzNoMphhA5gUgD9VpND1j9IGWe2jR2JDbk+kiegKqPiXu7otlhQwQF +KqYUrSuqWHbUB2HkMOEUl+eGqZ+U8qei8NhdBIwVzV18ot6LyDO2ca9NcQARAQAB +tDNQYXRyaWNrIE9iaW9tYSBJYmVrd2UgPHBhdHJpY2suby5pYmVrd2VAY2Vuc3Vz +Lmdvdj6JAk4EEwEIADgWIQRR5UI4wLq+IF2ElOnnZC1htiF1agUCZNJu+wIbLwUL +CQgHAgYVCgkICwIEFgIDAQIeAQIXgAAKCRDnZC1htiF1aoTZD/95s/0VgXoJnH8L +vN3hGu8UWn4OPdc+coOwc7h2YBB5kOxpVh2a2tcOOg9auyr0y2hjLwvL2lItAmWB +M3NSlu29ABr/OxlgjyUylM0Ds8EVneisg9E/Cx7e1qkGOIwufto7wdV2DRaNq8w4 +EkCbom4tjuEReMar8CKvhO4D6zIungFio+sLNgukQZuSxwSLkQVFBvNBOBcD6KVh +9IeZQb0RQ/j5ySYmY5wZg2sfgRHh8sZnMXMkWt5U8hx6Gdonq6jUs0AUP6/L3r/o +NU9gi8VkCauoxRASprfnGIm3yCD5TqQkCDNRnJQXdYYd4iPAebJkHOeVefHgrQ5A +HZ4YkKStIBprA+0zkynGlsLXzOx2XFLbyem74Rccl0NYbjWc0hB6uLRz2SrkUesc +5o8ZEX4G9SEUurMcFQDM+UMVESwtpnhBM2QRUsQGPv4Jb+aJoEYZwLLH01NgcE/v +igHsfw0FwfFLCYfkDQvg/t7CGho0QtL+sP0bCVdlxWLO8aVVNgQkRnvcE/xLX0t4 +D+y2ilAbs9K5ScMnmf4fiLRQXGPrv24v/0Lh9a5nbR/a1UIatbIislkl5+R18Rn9 +W9PKmJBW7PGwGWU1+LcO7dM17TWjmHEcdtuzOU8m+nsYCjo6xQFZQ92Nlh5/1EoA +GJqBTu984yQI8o31LDRpGr+nwrMdzbkCDQRk0m77ARAAqLcooZDTntSzFT4S17m5 +e/dcPyfpR+NtTovvJnM0ETk+kkPvaS5/UvVHJhuKn8lDaFuNXidu2f//m5ARGt7q +6VQv6WjozcEumI37s1pDKS73MZOPdg8llyboKq4ZrILxS+M1JqaRRH/WapzAl1n+ +0ZIx0ZiQ9oR2/YfMyXnUSW0c3mMB4eyzRQYsJFom6ckmaVEBDRtuAXNLqDNgYo7u +++Q6GI8oohWcIyk5xVBLi26IlbV2r1j60CLs+u0b7GjEZTV9fhgmFB54I4RmO19H +6uTpEM0k9x2YBV8Kxpza4QkI43LHFkzSNJ5w7d/E+Bf/zpeOe7n0EgW5ezFtqShw +8+G9yrVitIrHJ21allN/kfdLzNmLULrOsOy9wqdlZ5MjOLRBhd96pfo820Z3BMkP +dLTy67Ytn/YS9mDVc/Bzevg7fimPqrxNW3AKxbycFokyXrTCb8s5BHnzT5SoqVFB +TgV/LC6hivnE9h5OjeG37Ac8J1B2QI7obB9eLW3la1h3vs+qTyc5TMOM2VEwZTV1 +8a49XLNOJSFyZvAmMedrTGouTigIiE6L4oXzivplnNsGHjJ3mYbCROPYojejpBhS +m54Nu6ePK28xIENNQxoUHKjN0YU6/+Fe/2O7UK03rTfkJBHIevzH3IitMGi1ETW9 +1X/3cpLNwzpYV1HDF7xBS+UAEQEAAYkEbAQYAQgAIBYhBFHlQjjAur4gXYSU6edk +LWG2IXVqBQJk0m77AhsuAkAJEOdkLWG2IXVqwXQgBBkBCAAdFiEEt6cbQSm8v5Kg +Da0NjYwAYZP50v4FAmTSbvsACgkQjYwAYZP50v599A/+P0KnR6F27z6b11E07MmQ +mNOibo5CqsVWzdj7kWxaY1IflrFhBp+xGYRjvuSyMvh8gN9mQ57pYDqwXkYP2noU +tmbEtKOGdPMGrvntMivUH7fM9qg0Sdux85wZDzJlHFH4hEl8uTYZDO2rC8Re/Pl/ +r/xRZgznbcSE/x0jyM0dbRdk337kXUqinsc2Vjt2YFJohuUyucrqr31p0X3Og/wj +XWBfYD9FIGbnanH4hx7XQ/9r8N8u00LmW7ven0E3QqERvTlewlULzkOZdFiwqabs +e2A411itzC+s4qd/+qL2uaNU261oUhtckgCXMVjeN4JLiGb6d8N70Btsx2sALOvQ +jVk6StB1bDLJeqvgA4AkYuHlWUleQX+WeKG4QmoE+hEBmQoEZh2rSInXMvR/iAZr +yi2XGl3WvjzPqbCz+MN8bVZttj2rOca3X3T0exw0q3dkOPcpPhs6fquNDvxfrogq +n2NravAMcU0U/pl9SBjSnShi7yhDalnGjeS0ct6kMQyMJmFisG7J/jEePmYK5vjl +1w5UkQQcf7Gs1iWg6mh8VLxFwW3xw7LgI7P7KvSvMbFsmYwrKrLo+OFUA4oZlaK/ +4BI+BoRpoCJtWQe2+3ArwxoaaH0KP+58pLiy3eITJQpIeVoUqsJJ/PQAoxtjnesx +cy0FPC7DQ00802M+zJ8CzZbwSw/8CcKCDjBTWHGCnSREGJ27IVeM1z6ctAMsm9zm +E8Z6qaJQuIHpZNyWNLT9UxYMOSrQG2+HL8JVgRS9ubQGWMfEbPhhbN4HS8HGoOao +vo3zkmK7QNO6Se7Wc2l2wnCKm2nfH3RTxJaMVr79u3cpT0cYMo6YKIzwwR7I+P6c ++2FFfq3iTYsSgaBzhdQmMourolO/6u2Lc1mr9rH5Dd4fOCUWruASOr3dtCjCMmh6 +swSzydN0anpzMV9i58iXzOVlzC7KJywgF1K6os5YiBlPc8scx33Ma+UkSyEAmA8Q +kag5B+0T4eB7QhtSSVF9s6vm+tmUpQ/MqhWLPWRkRWaXXXv30HtOdXZTOeiDYJzp +TJE1MCn1zHaaYO+cZW3sYFQIVxtOY7fPV18BWzCDeoZTXRQIpug7nZxq2pyH3ggs +6OACpOODB6vQBHEGAyM4eVQNCmln5sUE5LqCKn9g2ifvGRrTagE38qOJ5ZyL/uuD +kTzNAAUwb0WbizjKIdIcY5OosHUiGmkfsM/tWx3M1AX2B51ktIZw2qP8DP2PAE6C +anoiRG05vtDb7teLOgWU+ypP+LcXzm/4S/abEK6hZcDeWQ9n74WvnZbmf4g3P8fG +eFIqEMST6B3qRI83mknM9WupAXj4SbPsprV55+wYeSyy9IlnwuCraxCp77LtPXRI +kdEtwC0= +=Qvj9 +-----END PGP PUBLIC KEY BLOCK----- diff --git a/init/gpg-setup/INF.gpg-setup.md b/init/gpg-setup/INF.gpg-setup.md index 693bffd6..f76936e0 100644 --- a/init/gpg-setup/INF.gpg-setup.md +++ b/init/gpg-setup/INF.gpg-setup.md @@ -18,7 +18,7 @@ in the support repo. ```shell cd init/gpg-setup tf-init -tf-plan +tf-plan tf-apply ``` diff --git a/init/gpg-setup/tf-terraform-setup.gpg.b64 b/init/gpg-setup/tf-terraform-setup.gpg.b64 index c76455c0..101404e7 100644 --- a/init/gpg-setup/tf-terraform-setup.gpg.b64 +++ b/init/gpg-setup/tf-terraform-setup.gpg.b64 @@ -1 +1 @@ -mQINBGL0Ix0BEADxKaN5o8kBgAQ2WUtsg9L/tq13XwVjXuSBBrIZ0HTaryUaqCZjCABFpky/WkG9R0PkHDLXFWmc6g/VvVi8AwU+GLztPrPbOsVfjjllxxz28JHxRMSpeetplhlplCg7ztLhpKkFKdcAzXeQkhaQkYLUWaR0w6WH7ARU3VZCn6UHIO8mmrTFzgZbalr3I+baylq8Y4pi+gie45lJXR0KCVb6ViUBwCjSKUL9nNloxZPhDkGpI+MYpp7UxBCdYaFo+Q8lYgfcQg19HTCPCICW6tGVNKUIglybx0f0V1R5r21piZhYVVVuvOQoIhz6OL2dl24m/pc+8no+3hkRlNkwdhmdQ/w8/p61uCqWRokyQ/Id2cswfM5eJDU4rgHZ17FGuzAZx2qsuc9sx3MMpvey+8skwf6szwI290caJWsMR3NuI4bd1r2i6RUEqWfq01m6bYINPrO6hzU00+V6sR+UgQsgW2GVeFsk9hzDB+meJ1KPdMQTAZ2rk4Wzooe/vai9HL2ltBCCUUss8XbUvmv3QRhg6aAGu9Fab+jtUHI+4BbSOJ+LKl3BK5c5yBIP0jm1B/9BrM+xq6FXtt+1+CSu6DbG/6kaRqa5hnEWuf5jh8IbZ9B/iQuqOhrcH4jXZ4ilxO15t91+3SBGxMmhHnuoSx984CBzyAMqxbjyAqwW87mQ7QARAQABtBJ0Zi10ZXJyYWZvcm0tc2V0dXCJAjgEEwECACIFAmL0Ix0CGy8GCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJEDZg9DULGRtOyJsP/jz18yjkWZ51NYdROCalQ2OhEEyDA1029LDSxKAni+L1t3EoQuaYJwE+G2VDaIDfpx5X2hZlwxJi+PLPl3NXGyiByr99N878TEK7R/BtP85MHWVtKiNWHaYT8udDW3te8oOqJl0nRLp2GmN4wzc27H8Sp38IpCzSf4FHXB9YcJIOI0sPcdGUYu9BWdhcFLJUY9ucqNJWhxt3bhSmqU60k52C6HwzC+5QHvCS2LyIG0GXpcLHh8/PMIpY1al/qZ63shKaFFRs/qr4AkFBazCLyK9gDG56G1ouFqZ/UFOV08oAWz4bG+1emNieXJ/t6kkerHkWID7IC6d6yU7IJIWkortcsXhmE5as6hswuzS+q+wutQDnCQXe6V5RAgct8vah+Iq3AM2rtCGeDAyliECb+JaeemuT0x0v9MPnl2OX89DdKb6Pzc2h7vP3aoO1nGBVM/LSrVvh1Zg4NlzjdQTM5rZLW68HBwkq8vtbWhzmPMt+gozbhB9IaQS1epeKYtYpqCbuIfvveT+o1Rx5jyb26Ri7+V2mbweQcngNTtT1zq4wcEh1axSUK47KydzXdjh1dCoVheZhafDFfWavavxn34kQKN3W+MacQsyDuNsmFBIaGte26Jgd+t0HcpEZkzZA5xEIpOcJfN9s0ziRLGOakzRRTAXViB1ueFVMNC2GvGO7uQINBGL0Ix0BEADYmiXbIcFBfwxZGBJALgDWCMo7Mmy/1uz3PVu3K2QS8e0Wir8k/dCm36MtD5JjYqh0YJg/gLJeCk28wXCUYGFGQPQ8e37hRYvHt9w8aaqh+TbEbL4izizT1VYOelwjjeD+5m9oEz5GD1yat0LH12eo7tUbDjVlkik1NcmdD9xfpMYJLbswU1eBSRz6dI601E39KIc2/RFXEO3sE0Swp1FmEeSGAzUxmrvu0KO3f05eZKtcD7j6uY4z5g5Ok2DQVNh2mmFyc1h2upp30eHrYskAuM6LBcU65TOoGoinjq5bjbzwZDNsqy7l0QKBe/2Sk2NhJg5oR8EZry0BM8+G9B+sa4u2WtPRQquXMMnuPjj7rp6tSwutFk8FRbfSXu+FEJmWvgC8bkmj+qpmDOZcfMwR6uyQH3rYDB+UCXgaYuZvnkEjkVcC5gIAlcoUgHlFYYDaT2Ds1QQ4+/OmAjv05uc7swC6dKsjW9dkHAhpLTIfx+C7TLPOCZRA6FG/h7++Z3jh8B9z9DqyRwbG7B6+OZTI23xY5d9zT30Vea5wAnUoEnhsV7aTMxJ28YiJ0gnLHUd8vH/FOT+lpWc4KHblOSF2il/JtXKtKRhnZQFwiPUZVvdx92tUtYY59WKOy19rQvB60FnlE5ivGw8Hh8M04oZflBZI6jDjCuTTxORJagDtqQARAQABiQQ+BBgBAgAJBQJi9CMdAhsuAikJEDZg9DULGRtOwV0gBBkBAgAGBQJi9CMdAAoJEPeS5hVP5cb0zXAP/173RbZSSUGgS0si3uVijsim33nka0gM2DfKYQvswZSOx/6Hp2OlpAtoBqNdqcWdluhT53ha4BFa/ycq5pHM24BZ4GXLHr3B+w3SDqpa1MiGLV9a02F+lsD3nh2AKg/CFrI+M+I9k7FrD4N8vnMqDyliY9FHLdZmq6XK0ERXiFqSUo5zIvONCfe4SJEBMcU23qcL5dIafCZIFSdsYlpzjP8jSMfJ3iRiG1WRH5MxJzAlrmd6KtEREUdtSLHNV/fIa5jEtDXdbEQBdBpqFmE/M9Z4o6BAIC8zZsu334wJdsDqx1g/aeCpWZ2ky8J6rqsBs72uQzFWzMWMoxyGC5j1I9pXAaPXHu1DRQb1w5nhb1GoXgl8pKsDFws0iR9khN9qzo0MApWwiasr7kUmVXE/WkgIZaw/bEtzsJv9WN8qDtwLJmZC05MOqE7s7sAgobHyUmvIJfr7yIbaXZx8CMZ7/OOYS9xGuhb6P2UWxKZj7NUc1A919OZKf3qBd+L7sjrMEu31oi/G3UznBtOd3rhtHss/hndcdkaMc1d1B2HmxgJ1/VZN1tZnLVIP5BekzWwq0cpdGoBkcdIXmS8ZOCWiVE9co+dOs7CSfdZ4hOXwMryl+OhetIyoN/uWqta2+87tdEDvfD88F0P/FoMNp88I/NkwYnYLvuA8B+UEFCPgZis+F9sQAL9bvk7xQrhpOgTC1VfHJt/3Ws69q+jS13bvUxELetef2hFT3fo1eocn/ilxHFhFzZUwtYR8xDsFQDsGdYPHFbf7ulkONwjq2LZiWRGt9GuHtRyrvvAnkI6LgYdcJrrffAVRFWKaHVkW/z8ngbu9FkAbkvZ0sPXNdRdYR49ZJxB7Mkr1lzVwfSPWtTvBp1ojR7mUZDzNq5ZbzCA/ggJpBr5IZ8ESRmYzJ2q5LYduk5w0unvnXZ08+9BCy0h+oZW4K4WTw2IHmU14hVRnSsIV7Q7kL7hoCX55Bzcse4A2nMsU4GoWsSzLU6btTonts9qtcWQXoXQLBMRnbZNpwGcUCUQlbCECkmhTLwbwsDc3eFCeYOXX1mKRBsegJaEL2O1/xQ0c0OZZaV+X7dunwtjBJwTuILnNGgHiSxJ/DBh+KEa04yJbF+jTBwSSxKzy+TrtvG01u9WhgYC/7x6edt7JV8b04pVXWXhAiNv7WAY1pWLxDUkRhfUG0eFddKqq5MB0nqdNZ04JUsVJjwEiZ9LNQHcBC0TAb6JW1RQ2S0gics522QEHVWFXqkoWxxoXb4phH/2ca5iPU8L+qGu64Z8skMAviPYW8XxMQh4j2OI7y4mk2u6b1Lnmhba87JpUog5sBANRRZ+nCfxcr/3w5PBCpgVzPijuSPXNnms38v5OiJ6U \ No newline at end of file +mQINBGL0Ix0BEADxKaN5o8kBgAQ2WUtsg9L/tq13XwVjXuSBBrIZ0HTaryUaqCZjCABFpky/WkG9R0PkHDLXFWmc6g/VvVi8AwU+GLztPrPbOsVfjjllxxz28JHxRMSpeetplhlplCg7ztLhpKkFKdcAzXeQkhaQkYLUWaR0w6WH7ARU3VZCn6UHIO8mmrTFzgZbalr3I+baylq8Y4pi+gie45lJXR0KCVb6ViUBwCjSKUL9nNloxZPhDkGpI+MYpp7UxBCdYaFo+Q8lYgfcQg19HTCPCICW6tGVNKUIglybx0f0V1R5r21piZhYVVVuvOQoIhz6OL2dl24m/pc+8no+3hkRlNkwdhmdQ/w8/p61uCqWRokyQ/Id2cswfM5eJDU4rgHZ17FGuzAZx2qsuc9sx3MMpvey+8skwf6szwI290caJWsMR3NuI4bd1r2i6RUEqWfq01m6bYINPrO6hzU00+V6sR+UgQsgW2GVeFsk9hzDB+meJ1KPdMQTAZ2rk4Wzooe/vai9HL2ltBCCUUss8XbUvmv3QRhg6aAGu9Fab+jtUHI+4BbSOJ+LKl3BK5c5yBIP0jm1B/9BrM+xq6FXtt+1+CSu6DbG/6kaRqa5hnEWuf5jh8IbZ9B/iQuqOhrcH4jXZ4ilxO15t91+3SBGxMmhHnuoSx984CBzyAMqxbjyAqwW87mQ7QARAQABtBJ0Zi10ZXJyYWZvcm0tc2V0dXCJAjgEEwECACIFAmL0Ix0CGy8GCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJEDZg9DULGRtOyJsP/jz18yjkWZ51NYdROCalQ2OhEEyDA1029LDSxKAni+L1t3EoQuaYJwE+G2VDaIDfpx5X2hZlwxJi+PLPl3NXGyiByr99N878TEK7R/BtP85MHWVtKiNWHaYT8udDW3te8oOqJl0nRLp2GmN4wzc27H8Sp38IpCzSf4FHXB9YcJIOI0sPcdGUYu9BWdhcFLJUY9ucqNJWhxt3bhSmqU60k52C6HwzC+5QHvCS2LyIG0GXpcLHh8/PMIpY1al/qZ63shKaFFRs/qr4AkFBazCLyK9gDG56G1ouFqZ/UFOV08oAWz4bG+1emNieXJ/t6kkerHkWID7IC6d6yU7IJIWkortcsXhmE5as6hswuzS+q+wutQDnCQXe6V5RAgct8vah+Iq3AM2rtCGeDAyliECb+JaeemuT0x0v9MPnl2OX89DdKb6Pzc2h7vP3aoO1nGBVM/LSrVvh1Zg4NlzjdQTM5rZLW68HBwkq8vtbWhzmPMt+gozbhB9IaQS1epeKYtYpqCbuIfvveT+o1Rx5jyb26Ri7+V2mbweQcngNTtT1zq4wcEh1axSUK47KydzXdjh1dCoVheZhafDFfWavavxn34kQKN3W+MacQsyDuNsmFBIaGte26Jgd+t0HcpEZkzZA5xEIpOcJfN9s0ziRLGOakzRRTAXViB1ueFVMNC2GvGO7uQINBGL0Ix0BEADYmiXbIcFBfwxZGBJALgDWCMo7Mmy/1uz3PVu3K2QS8e0Wir8k/dCm36MtD5JjYqh0YJg/gLJeCk28wXCUYGFGQPQ8e37hRYvHt9w8aaqh+TbEbL4izizT1VYOelwjjeD+5m9oEz5GD1yat0LH12eo7tUbDjVlkik1NcmdD9xfpMYJLbswU1eBSRz6dI601E39KIc2/RFXEO3sE0Swp1FmEeSGAzUxmrvu0KO3f05eZKtcD7j6uY4z5g5Ok2DQVNh2mmFyc1h2upp30eHrYskAuM6LBcU65TOoGoinjq5bjbzwZDNsqy7l0QKBe/2Sk2NhJg5oR8EZry0BM8+G9B+sa4u2WtPRQquXMMnuPjj7rp6tSwutFk8FRbfSXu+FEJmWvgC8bkmj+qpmDOZcfMwR6uyQH3rYDB+UCXgaYuZvnkEjkVcC5gIAlcoUgHlFYYDaT2Ds1QQ4+/OmAjv05uc7swC6dKsjW9dkHAhpLTIfx+C7TLPOCZRA6FG/h7++Z3jh8B9z9DqyRwbG7B6+OZTI23xY5d9zT30Vea5wAnUoEnhsV7aTMxJ28YiJ0gnLHUd8vH/FOT+lpWc4KHblOSF2il/JtXKtKRhnZQFwiPUZVvdx92tUtYY59WKOy19rQvB60FnlE5ivGw8Hh8M04oZflBZI6jDjCuTTxORJagDtqQARAQABiQQ+BBgBAgAJBQJi9CMdAhsuAikJEDZg9DULGRtOwV0gBBkBAgAGBQJi9CMdAAoJEPeS5hVP5cb0zXAP/173RbZSSUGgS0si3uVijsim33nka0gM2DfKYQvswZSOx/6Hp2OlpAtoBqNdqcWdluhT53ha4BFa/ycq5pHM24BZ4GXLHr3B+w3SDqpa1MiGLV9a02F+lsD3nh2AKg/CFrI+M+I9k7FrD4N8vnMqDyliY9FHLdZmq6XK0ERXiFqSUo5zIvONCfe4SJEBMcU23qcL5dIafCZIFSdsYlpzjP8jSMfJ3iRiG1WRH5MxJzAlrmd6KtEREUdtSLHNV/fIa5jEtDXdbEQBdBpqFmE/M9Z4o6BAIC8zZsu334wJdsDqx1g/aeCpWZ2ky8J6rqsBs72uQzFWzMWMoxyGC5j1I9pXAaPXHu1DRQb1w5nhb1GoXgl8pKsDFws0iR9khN9qzo0MApWwiasr7kUmVXE/WkgIZaw/bEtzsJv9WN8qDtwLJmZC05MOqE7s7sAgobHyUmvIJfr7yIbaXZx8CMZ7/OOYS9xGuhb6P2UWxKZj7NUc1A919OZKf3qBd+L7sjrMEu31oi/G3UznBtOd3rhtHss/hndcdkaMc1d1B2HmxgJ1/VZN1tZnLVIP5BekzWwq0cpdGoBkcdIXmS8ZOCWiVE9co+dOs7CSfdZ4hOXwMryl+OhetIyoN/uWqta2+87tdEDvfD88F0P/FoMNp88I/NkwYnYLvuA8B+UEFCPgZis+F9sQAL9bvk7xQrhpOgTC1VfHJt/3Ws69q+jS13bvUxELetef2hFT3fo1eocn/ilxHFhFzZUwtYR8xDsFQDsGdYPHFbf7ulkONwjq2LZiWRGt9GuHtRyrvvAnkI6LgYdcJrrffAVRFWKaHVkW/z8ngbu9FkAbkvZ0sPXNdRdYR49ZJxB7Mkr1lzVwfSPWtTvBp1ojR7mUZDzNq5ZbzCA/ggJpBr5IZ8ESRmYzJ2q5LYduk5w0unvnXZ08+9BCy0h+oZW4K4WTw2IHmU14hVRnSsIV7Q7kL7hoCX55Bzcse4A2nMsU4GoWsSzLU6btTonts9qtcWQXoXQLBMRnbZNpwGcUCUQlbCECkmhTLwbwsDc3eFCeYOXX1mKRBsegJaEL2O1/xQ0c0OZZaV+X7dunwtjBJwTuILnNGgHiSxJ/DBh+KEa04yJbF+jTBwSSxKzy+TrtvG01u9WhgYC/7x6edt7JV8b04pVXWXhAiNv7WAY1pWLxDUkRhfUG0eFddKqq5MB0nqdNZ04JUsVJjwEiZ9LNQHcBC0TAb6JW1RQ2S0gics522QEHVWFXqkoWxxoXb4phH/2ca5iPU8L+qGu64Z8skMAviPYW8XxMQh4j2OI7y4mk2u6b1Lnmhba87JpUog5sBANRRZ+nCfxcr/3w5PBCpgVzPijuSPXNnms38v5OiJ6U diff --git a/keys/gpg-public-keys/ATTIC/garne349.gpg.asc b/keys/gpg-public-keys/ATTIC/garne349.gpg.asc index c8ac4a7d..72258a9c 100644 --- a/keys/gpg-public-keys/ATTIC/garne349.gpg.asc +++ b/keys/gpg-public-keys/ATTIC/garne349.gpg.asc @@ -60,4 +60,4 @@ Co4AUVeQp2v8973PHgXY1Ft6+fFnLBpNgrEahaKBKMP9L+0avlTdsWQpdy0mSXfp Qne8rjp/te5RxchWXqidMLCvVNTzboelJbezfZw3ujLTENoqF/k9hmB+5rJQweqh f5hqL7s4fLc+Q58jFr4TNb3AMHJ0yPUmEo+bl3Y= =97aK ------END PGP PUBLIC KEY BLOCK----- \ No newline at end of file +-----END PGP PUBLIC KEY BLOCK----- diff --git a/keys/gpg-public-keys/ATTIC/gogel001.gpg.asc b/keys/gpg-public-keys/ATTIC/gogel001.gpg.asc index b782722c..3ffda639 100644 --- a/keys/gpg-public-keys/ATTIC/gogel001.gpg.asc +++ b/keys/gpg-public-keys/ATTIC/gogel001.gpg.asc @@ -60,4 +60,4 @@ kwi245ePpp/bweAvYLzPN4YZFhXD9130b37DNldNZoc/7ryHSZVS9HE1D88pjwO6 soFYAg0uLTqwImIt2UN9mC1MD2jEdKWghmZsnrRLwcBnj58i6nuoQUTnOQHYU3lN BiGNrLwREe51CZkYMzuQ31D9ufyJP4ZcsXGCkFaiCYg= =/qFG ------END PGP PUBLIC KEY BLOCK----- \ No newline at end of file +-----END PGP PUBLIC KEY BLOCK----- diff --git a/keys/gpg-public-keys/ATTIC/lopez539.gpg.asc b/keys/gpg-public-keys/ATTIC/lopez539.gpg.asc index 1f3d2d9a..d55471f5 100644 --- a/keys/gpg-public-keys/ATTIC/lopez539.gpg.asc +++ b/keys/gpg-public-keys/ATTIC/lopez539.gpg.asc @@ -60,4 +60,4 @@ HAwOx5U1qXKlf9EPj4b+AvxhyPUzDxBvqhauJs04wWAYQEnKMxHDxp5LWb1x0NN5 9/Iiv8vGCvUz7Ncp7II+Cz+wslkcbd+Hvt1iNW+ZK/0zK9C+AG05QPuHEv7MY45p mfsM11zZeXrn64pcMfxijlVOpUrvLvvJOxlXZUiWc1dnMGPobSMsuYOZYjaSdA== =vNEg ------END PGP PUBLIC KEY BLOCK----- \ No newline at end of file +-----END PGP PUBLIC KEY BLOCK----- diff --git a/keys/gpg-public-keys/ATTIC/lucas348.gpg.asc b/keys/gpg-public-keys/ATTIC/lucas348.gpg.asc index be6d2c69..8955e272 100644 --- a/keys/gpg-public-keys/ATTIC/lucas348.gpg.asc +++ b/keys/gpg-public-keys/ATTIC/lucas348.gpg.asc @@ -61,4 +61,4 @@ FjUXrXOPOeSVtrFCrjv376mtquOI7W7w+JEFK5wlVOAAqWi3Y+3mj3SKsJazs8rr JIzHhQRhw7UBBD/ne1Z8fHxHl7ufJBKp+4FlZHDBhX4ryjy4VNnlSfdq2W1UmNmN 7tE= =nR75 ------END PGP PUBLIC KEY BLOCK----- \ No newline at end of file +-----END PGP PUBLIC KEY BLOCK----- diff --git a/keys/gpg-public-keys/ATTIC/owusu013.gpg.asc b/keys/gpg-public-keys/ATTIC/owusu013.gpg.asc index 6524f517..c37b61b4 100644 --- a/keys/gpg-public-keys/ATTIC/owusu013.gpg.asc +++ b/keys/gpg-public-keys/ATTIC/owusu013.gpg.asc @@ -60,4 +60,4 @@ RjRPSoho8a1lX03DyFGuC+EJUHM3iBFVkVoAxi4iflR+xLHw0Gyt6PS65VZbDzLo rR8oA+vH3sApkJ2NIueUnReMGxFzxpL9mgKvZSeSKEaC80d9VtboR5xd+GZCM/zp rIdDTEeYSGfnMwI4rnYmumcNqdPWu4YSoaaCq3Ha9cFi3fve =F4DO ------END PGP PUBLIC KEY BLOCK----- \ No newline at end of file +-----END PGP PUBLIC KEY BLOCK----- diff --git a/keys/gpg-public-keys/ATTIC/tanyi001.gpg.asc b/keys/gpg-public-keys/ATTIC/tanyi001.gpg.asc index 930868ff..7f29222a 100644 --- a/keys/gpg-public-keys/ATTIC/tanyi001.gpg.asc +++ b/keys/gpg-public-keys/ATTIC/tanyi001.gpg.asc @@ -60,4 +60,4 @@ PvUTIn2PTg9GPnKG/joC6s4r7kC7aWhvjRZtjGk1cRZ2rMBVXiMdCO+R8ye919Ep CP0dWvztPNGYy2jOmwjYmelLz67zoRVeFVgl+57GpmH0CuuXJu80ZW1NXyS6gUVK uSCvEZobwgLQ6UQ467DpNjF8jMGoumJCA5bqJrIu5k4XoFU= =xBjq ------END PGP PUBLIC KEY BLOCK----- \ No newline at end of file +-----END PGP PUBLIC KEY BLOCK----- diff --git a/keys/gpg-public-keys/alsto316.gpg.asc b/keys/gpg-public-keys/alsto316.gpg.asc index 3b055b81..623ba162 100644 --- a/keys/gpg-public-keys/alsto316.gpg.asc +++ b/keys/gpg-public-keys/alsto316.gpg.asc @@ -60,4 +60,4 @@ MtuwNeOXSQdJqHdjEdZcJHVtwh7sJerGc1vNkt3Pqhhf6AbMiByAtwktJtubrChj pCgls5n1WPmpBzVPeAAYBr5DEkfQob+Jkg4dvgMhf2RMe6t1U6mOh2oePuTZSSZj Y62nOgaumEi4cZlKffWtWoLDzkuzqEruNv5joo64lb6lsuGM9Ou/2qOwYVnTWw== =NzKx ------END PGP PUBLIC KEY BLOCK----- \ No newline at end of file +-----END PGP PUBLIC KEY BLOCK----- diff --git a/keys/gpg-public-keys/ander693.gpg.asc b/keys/gpg-public-keys/ander693.gpg.asc index 4d23743b..d161153d 100644 --- a/keys/gpg-public-keys/ander693.gpg.asc +++ b/keys/gpg-public-keys/ander693.gpg.asc @@ -62,4 +62,3 @@ WUo5K0bHYnAszHSJuR4tIUHELCxFqRLUU7Va5ZfXdc7037sLXMYChNge1wRIkFNs Hk4bkF8jlQ== =FMyS -----END PGP PUBLIC KEY BLOCK----- - diff --git a/keys/gpg-public-keys/avadu001.gpg.asc b/keys/gpg-public-keys/avadu001.gpg.asc index 7790131f..f62bf6cb 100644 --- a/keys/gpg-public-keys/avadu001.gpg.asc +++ b/keys/gpg-public-keys/avadu001.gpg.asc @@ -60,4 +60,4 @@ HYhd+KAC5cZC2nck8gpkcVE17vAODU5Ubn4lJFB6b62GraMkiBMmrtcmFETv17FI PfiRt15pyEndtjUgbzVDQu81/aowc5P4NEJJcv5Ut2Y36QqDV0FLxDjyGhTjHKVi 3bxmgHFsQPI99leaW4y0FjVGI3TJ6Gh1lBE+QOReMoIocOWZnucC8f3ynjjD3DpT =/e0j ------END PGP PUBLIC KEY BLOCK----- \ No newline at end of file +-----END PGP PUBLIC KEY BLOCK----- diff --git a/keys/gpg-public-keys/dhond002.gpg.asc b/keys/gpg-public-keys/dhond002.gpg.asc index 536d740d..74a8debc 100644 --- a/keys/gpg-public-keys/dhond002.gpg.asc +++ b/keys/gpg-public-keys/dhond002.gpg.asc @@ -60,4 +60,4 @@ KNn0TiOCxAskNvmqFnjLCiFlsMwB9q/HOLpjJvRDweWtSlisdEYwrTLLSXJePm1G mn7jvRfdk5r1F1oHTwct2pA9rtbOzg8Vjm9v9zNB/1GN88vV39lSVjuNmU2G6HUy iWQjCoMg1pc31r/6caVvGuS5MWlfB1E7pi/aDl2kvXn4/XCkP5Tuvg== =QPr2 ------END PGP PUBLIC KEY BLOCK----- \ No newline at end of file +-----END PGP PUBLIC KEY BLOCK----- diff --git a/keys/gpg-public-keys/engli318.gpg.asc b/keys/gpg-public-keys/engli318.gpg.asc index 09c0ac4f..91ee8f17 100644 --- a/keys/gpg-public-keys/engli318.gpg.asc +++ b/keys/gpg-public-keys/engli318.gpg.asc @@ -60,4 +60,4 @@ oFfmfF2RwYH+bFkBnewQ/37/h0Bj4ny55hQMZtSSDlB252JLLw4OnZviM9bDs6LP uUOrruYnzjbNPmzbcI5lM2tU3vhq1NC46SJnhYtRm84grVac1ANBtPjCIWL1bF3X OYrcxSkwqVhmRJlPHQo0a2DW2Wgj0jc66avp =+W59 ------END PGP PUBLIC KEY BLOCK----- \ No newline at end of file +-----END PGP PUBLIC KEY BLOCK----- diff --git a/keys/gpg-public-keys/ibekw001.gpg.asc b/keys/gpg-public-keys/ibekw001.gpg.asc index c887a828..0d49d60e 100644 --- a/keys/gpg-public-keys/ibekw001.gpg.asc +++ b/keys/gpg-public-keys/ibekw001.gpg.asc @@ -61,4 +61,4 @@ anoiRG05vtDb7teLOgWU+ypP+LcXzm/4S/abEK6hZcDeWQ9n74WvnZbmf4g3P8fG eFIqEMST6B3qRI83mknM9WupAXj4SbPsprV55+wYeSyy9IlnwuCraxCp77LtPXRI kdEtwC0= =Qvj9 ------END PGP PUBLIC KEY BLOCK----- \ No newline at end of file +-----END PGP PUBLIC KEY BLOCK----- diff --git a/keys/gpg-public-keys/jain0009.gpg.asc b/keys/gpg-public-keys/jain0009.gpg.asc index 945a85da..59e7c3dd 100644 --- a/keys/gpg-public-keys/jain0009.gpg.asc +++ b/keys/gpg-public-keys/jain0009.gpg.asc @@ -60,4 +60,4 @@ wxZ58mpaRkIEFs0NfPzqlITsNksS/cTFyR+gHL7jbfliPxVsCw2OEO6Y6RWpQ4ne kBwDTWjtIM6CXHPmqEdbYQcDc9NellirSk085KXR0clw6mt3Ngbo3KvjzqVWiYsc rNRLJvFJs6Nykcld+vw+Osn3UA== =wxrC ------END PGP PUBLIC KEY BLOCK----- \ No newline at end of file +-----END PGP PUBLIC KEY BLOCK----- diff --git a/keys/gpg-public-keys/karim005.gpg.asc b/keys/gpg-public-keys/karim005.gpg.asc index 03f619d2..e061e5cb 100644 --- a/keys/gpg-public-keys/karim005.gpg.asc +++ b/keys/gpg-public-keys/karim005.gpg.asc @@ -62,4 +62,3 @@ VLh1BSWt77PblpNoMmNMQw6fEdsusCVki06CTd74W7K9CMUWQmqAApURYkidVAPw 7+4= =2zNf -----END PGP PUBLIC KEY BLOCK----- - diff --git a/keys/gpg-public-keys/marti926.gpg.asc b/keys/gpg-public-keys/marti926.gpg.asc new file mode 100644 index 00000000..a462c450 --- /dev/null +++ b/keys/gpg-public-keys/marti926.gpg.asc @@ -0,0 +1,89 @@ +-----BEGIN PGP PUBLIC KEY BLOCK----- + +mQENBGT57qoBCACsPgoYTHIRhpcxAH54h9Cp0nhgvuyE0i+B+AHcVB2ppUs/21nq +crmN98/pkaV09aAkohu35sd715AgMy5yAfzMlFHp663VvrLBCxikNkSyTUj3Q+Xb +pdRK0aThuHBjKqH7wqyKIPG8WT9YGxTWPGkeJA+Hxn4RNxCnhqZL2khDJqDb2fja +xgDWYfcF7WzGPD8SB+ggizDl/QPh8EvINwZRtXpFwKHrfiVy+yfdwjO6fXwVK+Ja +6DjI53E9MDpL24U24Qt9tEhvny1oZzxOMbrxOQgvb4Nvp+f0EBA/Nq35ytdhndC4 +ts0iUKtS7Em9ZSYj0E3DxudRhkphYy6Hv2QHABEBAAG0JWt5bGVtYXJ0aW4gPGt5 +bGUubS5tYXJ0aW5AY2Vuc3VzLmdvdj6JAVQEEwEIAD4WIQQkOQZz0nY21MSQUTJe +CeZSfwx2sAUCZPnuqgIbAwUJAHanAAULCQgHAgYVCgkICwIEFgIDAQIeAQIXgAAK +CRBeCeZSfwx2sCwPB/0UGIoZwA4NMlbTTgL6Z9rEHOzX2xK8RYQ7cocDItlsq4cL +ldrf5dQEjUMEXmyvRo5cy+jQIFTOz8p2assDyDTSLm57AwAICWFFirmTMRvKumBQ +RiTwMjZksaoU0eQ/Vt+jVtR0QhSWtuxqwDcWXdOg9/23jl0RnQyJ0rN8aIOLciZG +lJ4sjIxPeyaJnhjHS0Jakta7jdqjr+2JX4RSjR6VP330JmJFnzUNQw9PzeLlTw7l +KktU5hx7mZQnSDqgK5WGORQnIdzH1+I1paHbyT1gA3YBVg/b7Xs3/N6vnDgz/ySi +uJLFN+BmfrRK6aaC9wojbYrJc55Gd+srs968VtlAuQENBGT57qoBCAD2MU/eN/Qe +Oxle3FFkkJT/GiUdNP5yhoK6rDSj56+bfKgpnlwjKriyX8FQg+2ETCTaxaEY/NXD +SRPXEBgSo42NywZ7FB4/LZ5KXJ3bh8JLOrs1yWVT44Le2GOvqp72O24FYpNOJqSl +lYnuKSOS+whH6h3xgq0bwIPlA4hl5m1fylZ7lnEwiOlg+hvacwrBtcGZFjRBuzSc +fjhTPbq8BAA53DoVki1+IvRneEMRyKOIHD4xqHVuak1mcSyOm9QZcDCgVAMarwW1 +OKhq7HHqKYlChqmqa9BXLajUiENmcsAwMdidrCpfOKKyvXgQh0s1wITrtvFg1p5O +GjecJmPVQbJvABEBAAGJATwEGAEIACYWIQQkOQZz0nY21MSQUTJeCeZSfwx2sAUC +ZPnuqgIbDAUJAHanAAAKCRBeCeZSfwx2sMFMB/9Nk6YSp65OEf90VDvgYKXuWLdK +rnLd4/JIU0nXD8zqC57Wanx7UNTimwJ3ZpgOS1v3uWfPVZLjFiycE1e1+54aiTTl +NYX047FQwsbvQe2cJf6qPhYl0EDQmAnHrTWwWUfM4rA1ZtZRNGk61wX57zsGCxj/ +TU3GFmun9UTBKjRT0m2l8M7RwbBG2L5C56EmJA93/wcPNJlgnekm4532ETCxG/X+ +PWQwxQ+sJENh6b8sdEROv1xOnCpLVylDnE3xXsv0mGwZ6NBj+TZhQ/wjjPd5HnqF +/hW0aSxqVD8vsSqnn/39h4kRuXFuf+XJNZ/bB/v9Vt6ubNHDOMfIkM8shfAHmQIN +BGT/F84BEADMWg+cxfNTOA8XIxzQr8HMyxxqZ3oOpSOg7VjBJch4HLAJ6+TJnhsm +oY1ieAjjQMwJbxggG+zk+fbjV2wSQWCEVEUKGuU8kOiYTeCNuvnZqAVzGH0iN/RR +XSAAbRwS6K2werpS5MZw6J0b2OCXX2imPRsWPwhmlP3e5fShEjheRW3lEKvI2fve +Bmi6VMji/ha2r9Dr9ZLmO+HhpbfSk+V/5fK/lOedw5KJfRx5VXu5R9T3IbW0RtN9 +OwMjX/bHJ0NjeuhbZ0a1p+T/Mqku/Pmqd1PZ283Q81AAt1x0SLK3IJ3lg4X6grxS +v1unexNlmdaM/XzFWq++YX/SDPbFi+ynPSgRt8oPeT91p3sgLUQ7GyEXorTeW0R7 +P+SYxUqp3ruJapi1wBFKyMLeSB+ygEg9kK3qsAE6vN3JSqhd9hC6laqkMgVtsgBU +q+ExHGItAPz12PxxQwe6LC9TBpSVfDC4sBM9HjLYJwS7GaFnFXEl5Jy/kLde2d8Y +/GxNgxxhvTLunTXZg4CDU6rgnSvzbb9kMs+GStLZT23jM2RYTx1irRlFtRbX52/0 +Sv3XsMYn17r1Xx4UwVahBwbVaNW4EGZ8Z/4ZC+WBfC7ujP3yZHjcKG2DG3xld3ff +13WNVhlMjuwYtQf+VlwhbOxnQTVtTxesX6+YNl7ZfUdhv1yoHwGEwQARAQABtC1L +eWxlIE1hcmNlbCBNYXJ0aW4gPGt5bGUubS5tYXJ0aW5AY2Vuc3VzLmdvdj6JAk4E +EwEIADgWIQSIqzSLPGig7hD8yqPNaVvmSypHWQUCZP8XzgIbLwULCQgHAgYVCgkI +CwIEFgIDAQIeAQIXgAAKCRDNaVvmSypHWZupD/9FVVFu5MXqR+V9EssOTKK6Kfyp +YcXXUGhY0jeq7CHzr+s+oq8Y7h7EU5tR0XsXSofi49X5hXrF7NmngXuWk9QozNUF +dWL1wON/oDEHZpQZZcf0LpRTWjv+mC6fypG1LFxPyHjFYjN38wsRMl/WsMHgFsRK +aMoZguq++c7orNhGeaFMObHcYK38tjUI+L9uq4UYeROPEfS49PgjnvQx8ecwjcRA +9ok5gyBom8NFeRsTVTiJLA9LyiLk3WTKkDmdJSOatF9VwU47pBaJcQKTXyJ2n1H7 +ovEoSjyOpXsm66seRsstiKDk6FTfyI7uZ3hkjpckNOJ5ZV6I2rdvFDgKwVL4mrOg +CaJN2npXGDT6To5/+wSFPs/lsT6hHRKCa+vUHVP0jnaJU7G4gACuJ2viM1/iYawV +3tfkynjW1azZmXX5tk47sVhL9oo3phYpUaNK5L1LGiD5NDmpnnr9vFi7eA8N/jx3 +Z0fkJ4+3pFgJHo6OIVE6egzxlvwSeVhccHC/pPvyODqRTzZxVezfYnV5xNcbXs4d +POF0l6dJfQgptckOFf/px9P3QPmgOjBrYkzRsx0tBZUosLKLyFJNn9fkBGHk8Mui +vxu8QkuQfEHvhWehUUfgRCowptLJLKQkd3GZ0jwaV5xHoIynDCBqiLv++O6kx/Qm +NZTJYplQmu4A0G0TNrkCDQRk/xfOARAA60oGME9ikbt/H/JfbWyGyu/MPBH9+21L +v+z6hSXBFEjWSauUeQi13wl73DKCawrN7kn1Z9xxm7lCaeiU3/CfwHewU13aZiWt +E6sf2PDoV4M8ZnxsC/qeH9XfSdbx/hhKGjqRlrXpWPGUUucdSiXopNmn0msDBxTe +KkDLWmaVSaWZUhiIxpD3xVs8v9G52gKM0x6CniW1YeKFIunOfX/FrtXgICLqZ7Bs +YaoCzj7PNtr0vgVhFeak8QBc0lhJNtrQon+IHjGBwipemLmTU/UPRfgTBZ4ncyfB +/Ll2mWVn+pNpzWwTiaCDGfKRHZzDaUtkHXGkXEhruIQArfSsZQ969ppX75/A3gE+ +3rk9+NL+yBcuSZl1ggTf/l3SLbBiHaHN/kwy0EneqZVwj5LH4PDi9aKafGZYQ0uW +WhQXNog4NpJYyKELlLC77KuCmIZsts4ON85TTyor/Y2EPZr/5hA5/yHR7K8C2klb +4Ed4h5Oks4uDN0rwlNRjHYLHgFP6OO0uYaVktPjwZ97/maoqSVvsOvq7BUg4sQQn +X/SlqKwpuAG5C0y1lnx2oysp30DiBKa3OLs8tnqF9nsfq4TKFLyvTm62P1Zwl6hf +Ux2umowE8/tGftecdx+fByK8X1ZDLvA6YDhKk4YlCUZ5vPr6l1o5A4r6kW4RbsEa +0FnWpcF76kcAEQEAAYkEbAQYAQgAIBYhBIirNIs8aKDuEPzKo81pW+ZLKkdZBQJk +/xfOAhsuAkAJEM1pW+ZLKkdZwXQgBBkBCAAdFiEEC8Cp3IvPggNPBKOqCN/2uAwp +jrYFAmT/F84ACgkQCN/2uAwpjrZCFA/+Plq0IqX5dmjiyQLhqJ7Zf0qsasRrIW46 +vxVu6/yro8L3wqyIqJwMO8QNY3fjA1LPoSvJj9//CsELY+f91il9STYtnp84mnp+ +ePmJi9kR6XKW96hXh5zTRmfELl8CHOfRXQCp6gX6f1Ya80sDgIBLkL7NtWt4cuwJ +v5sQUXeymlkhXEpwmbWDEzKakqy9uWG6TZfXnNP1VaqXJYf8FYdl62MdsK/v4yyC +qXl/hD90I8QrNzQqOgXPMvagtB5Vavw3YXR1TTjf8bkVx39thpY56Js7ttg4IsD0 +JCZU8hw2sFDiwnrcpI1Rvcks0mdzWKWAaZDn4qayOtIrvHbyO9SPbnFeB3FiBpFs +pDxJYZBnuq62MUZKa/123yZBGMnUM7/8BNBfFCbu0FnWhaTCev8L1g7P9XXF42mk +bw9WlACmO+ZQcXuxD0UqwKFSMCnEu3lQzGkPleI9sDzzTZH4geVgRGtOKHIxNW7Q +p55g3CzlntyFBdsfKUbw9Zkx1Mw1ixZt63MuJf5Zpx0+k2wyleoyq9xjCfJvjAHi +zKY23VbdVv3DsLQkX5c3/VZ9GX/V7dc0o59XKgVXrmuN7rBaoelAa7l5aLJmlig4 +9on5aWVzFVq+uY5Fvm82lwiXyAAMqS4aYoUqhKZ4uhUop9dGLPfWQ/t/w4bWqmoQ +LvFuNLmzP7eMzxAAkSZ1aRGkvy07dzy001gfrj8pnj9W7tJdzSImx9BLFUnFm7dp +7yhZlapQE4IfeJ+IpQeGA3XUsK3i/PsOful1n7h4nDOLMdqnsNtpnN4S3NDtOcSv +4UGRltY0zBtzGXhhKBE5wVhGHJ8tFmBDgK7BYDAm6zYAA4Vd3Hb+vpJpJiJHyInn +Bt1+hs08mgsnoCrkAQ89/nU96ZR5aRUvs3/JLJQ5CF5xxHwQkQmUeUzDNcNLIbWB +Ez1QEq7mRK+Keq9s138bKZmm4ZvzogKNhFp3NJtTjN7NhjaOPJLVhWGvo5ob4ORX +qK+om0Nr/2r0YudEN1XtFtMMgq4eLAfhgTR1RxVCdyqxlQkAldLNtL/+uJIy5iG5 +AcUJk/zVR7K/3RZPhFhQByWysDXpwqwRhaa7/oI8xbUMzODwUkNlsGnh+FH8eF6q +0pvv/jKsl+qKzEIIO7TqgmFFTqpslO4ZxtDkrIiQ2FDXSC/qa/F7AuzMQEftx4zb +DMR9p0pZrELQXRbbXjnobTbaeqBhMaAo9+cEa34rJwRlO1VCU/IA864m5GMGpfxc +aC+3u4Zz863H16dpk4Hy8xPukApYz9AlKXjhXjw+gZN1CuWsRV3LBBTstpUKXW2s +uuEXimwrHDyS2NQr0G/rSFZiQ/3ra8Lh6qJNuZOTMF92JIvGvoCL0iTFBBQ= +=sXS6 +-----END PGP PUBLIC KEY BLOCK----- diff --git a/keys/gpg-public-keys/owens397.gpg.asc b/keys/gpg-public-keys/owens397.gpg.asc index e266f28e..ef6f63fe 100644 --- a/keys/gpg-public-keys/owens397.gpg.asc +++ b/keys/gpg-public-keys/owens397.gpg.asc @@ -61,4 +61,4 @@ IdvbRfYWvPkaTuksg3UcdUhQFz5iqoe6qoLQbKHaSSu+18Tl9YHOHEBrsnHnXWc7 JW+ldy0grWM+yZP8VqNV7U7lqWHtguOhxu49paAsGCL97KGd0xcWAoflRHWGxZvL hw== =upIN ------END PGP PUBLIC KEY BLOCK----- \ No newline at end of file +-----END PGP PUBLIC KEY BLOCK----- diff --git a/keys/gpg-public-keys/patel385.gpg.asc b/keys/gpg-public-keys/patel385.gpg.asc index 38794e57..2d73e560 100644 --- a/keys/gpg-public-keys/patel385.gpg.asc +++ b/keys/gpg-public-keys/patel385.gpg.asc @@ -61,4 +61,3 @@ DersDyQVsFh6RUS/fsAbG5nS2er806R7JZDR2CBYVNhMDTlUYZglIFvU3eC8Kboo sLrNOOKiU64jXyQJgSXYN+PoS3803Mp5WhIEIugjr7tZPT+mNtvuug0W2sE= =aZ1p -----END PGP PUBLIC KEY BLOCK----- - diff --git a/keys/gpg-public-keys/singa002.gpg.asc b/keys/gpg-public-keys/singa002.gpg.asc index 6c6f34b9..50a113e0 100644 --- a/keys/gpg-public-keys/singa002.gpg.asc +++ b/keys/gpg-public-keys/singa002.gpg.asc @@ -60,4 +60,4 @@ ukw+dk5PdZje2U/M0Kt1/rClFd7Gtmsb7dGBpwRs0fUd5amed0K4AxSGSsN3FjEY wWfAbfUmRm+FyTAdbOjifoPEwcbHfNOiv4E8u8XJQyxdnZkVNaE0Ts4uJOGLzFaN BEueSF93xBLgkNS7ymUX5v/5godN =S+G3 ------END PGP PUBLIC KEY BLOCK----- \ No newline at end of file +-----END PGP PUBLIC KEY BLOCK----- diff --git a/keys/gpg-public-keys/zulfi001.gpg.asc b/keys/gpg-public-keys/zulfi001.gpg.asc index 55fec60b..79f8f8b5 100644 --- a/keys/gpg-public-keys/zulfi001.gpg.asc +++ b/keys/gpg-public-keys/zulfi001.gpg.asc @@ -60,4 +60,4 @@ los06qXQu111OXwcyfaS3F5DMj2ev4xcXk92bCqBy9Wlb1xwLDPD8xixtutz+A8u nXZ7NvMtEhdhgmsoMc9q+cMppOvxvjV4AWV4b7p1/LVTnbmBQE0NW4vW5iRkHtlU q2m1mNKFkKUt5IjDfspQbwU6k7rmIP1ANgBSqZkkisxIq6Cyu9URtAHmKqI= =/tJU ------END PGP PUBLIC KEY BLOCK----- \ No newline at end of file +-----END PGP PUBLIC KEY BLOCK----- diff --git a/local-app/README.md b/local-app/README.md index 2795fb65..ea05675c 100644 --- a/local-app/README.md +++ b/local-app/README.md @@ -84,7 +84,7 @@ mkdir cto-webserver cd cto-webserver /apps/terraform/bin/setup-new-directory.sh /apps/terraform/bin/create-remote-state.sh > remote_state.yml -/apps/terraform/bin/setup-generate-rs-backend.py  +/apps/terraform/bin/setup-generate-rs-backend.py  ``` The last command outputs two `ln` commands. Execute the first one. You need a link @@ -105,4 +105,3 @@ git push origin mybranch ``` * Go to the GUI and make a pull request - diff --git a/local-app/ansible/inventory/inventory.yml b/local-app/ansible/inventory/inventory.yml index 8c00ea87..ecb633e9 100644 --- a/local-app/ansible/inventory/inventory.yml +++ b/local-app/ansible/inventory/inventory.yml @@ -18,4 +18,3 @@ all: # nfluorine.tco.census.gov: vars: ansible_connection: local - diff --git a/local-app/ansible/roles/aws-cli-v2/vars/main.yml b/local-app/ansible/roles/aws-cli-v2/vars/main.yml index 5776c792..5d0ca629 100644 --- a/local-app/ansible/roles/aws-cli-v2/vars/main.yml +++ b/local-app/ansible/roles/aws-cli-v2/vars/main.yml @@ -1,4 +1,4 @@ -url: "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" +url: "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" target_dir: "/apps/aws" bin_dir: "/apps/bin" unpack_dir: "/tmp/awscliv2" diff --git a/local-app/ansible/roles/aws-session-manager/tasks/main.yml b/local-app/ansible/roles/aws-session-manager/tasks/main.yml index e146b477..9d816d3b 100644 --- a/local-app/ansible/roles/aws-session-manager/tasks/main.yml +++ b/local-app/ansible/roles/aws-session-manager/tasks/main.yml @@ -38,5 +38,5 @@ https://s3.amazonaws.com/session-manager-downloads/plugin/latest/linux_64bit/ses changed_when: false -aws_cli_url: "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" +aws_cli_url: "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" aws_session_manager_url: https://s3.amazonaws.com/session-manager-downloads/plugin/latest/linux_64bit/session-manager-plugin.rpm diff --git a/local-app/ansible/roles/aws-session-manager/vars/main.yml b/local-app/ansible/roles/aws-session-manager/vars/main.yml index 30565cb0..341ba315 100644 --- a/local-app/ansible/roles/aws-session-manager/vars/main.yml +++ b/local-app/ansible/roles/aws-session-manager/vars/main.yml @@ -1,2 +1,2 @@ -aws_cli_url: "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" +aws_cli_url: "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" aws_session_manager_url: https://s3.amazonaws.com/session-manager-downloads/plugin/latest/linux_64bit/session-manager-plugin.rpm diff --git a/local-app/ansible/roles/aws-utilities/tasks/main.yml b/local-app/ansible/roles/aws-utilities/tasks/main.yml index e146b477..9d816d3b 100644 --- a/local-app/ansible/roles/aws-utilities/tasks/main.yml +++ b/local-app/ansible/roles/aws-utilities/tasks/main.yml @@ -38,5 +38,5 @@ https://s3.amazonaws.com/session-manager-downloads/plugin/latest/linux_64bit/ses changed_when: false -aws_cli_url: "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" +aws_cli_url: "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" aws_session_manager_url: https://s3.amazonaws.com/session-manager-downloads/plugin/latest/linux_64bit/session-manager-plugin.rpm diff --git a/local-app/ansible/roles/aws-utilities/vars/main.yml b/local-app/ansible/roles/aws-utilities/vars/main.yml index 30565cb0..341ba315 100644 --- a/local-app/ansible/roles/aws-utilities/vars/main.yml +++ b/local-app/ansible/roles/aws-utilities/vars/main.yml @@ -1,2 +1,2 @@ -aws_cli_url: "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" +aws_cli_url: "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" aws_session_manager_url: https://s3.amazonaws.com/session-manager-downloads/plugin/latest/linux_64bit/session-manager-plugin.rpm diff --git a/local-app/ansible/roles/setup-gpg/tasks/main.yml b/local-app/ansible/roles/setup-gpg/tasks/main.yml index 10b6e897..53815d2e 100644 --- a/local-app/ansible/roles/setup-gpg/tasks/main.yml +++ b/local-app/ansible/roles/setup-gpg/tasks/main.yml @@ -8,4 +8,3 @@ dest: "{{ app_dir_bin }}/{{ item }}" with_items: - setup-gpg.sh - diff --git a/local-app/ansible/roles/terraform-python/files/ansible/ansible.conda-info.txt b/local-app/ansible/roles/terraform-python/files/ansible/ansible.conda-info.txt index e8fec139..c30d6ae9 100644 --- a/local-app/ansible/roles/terraform-python/files/ansible/ansible.conda-info.txt +++ b/local-app/ansible/roles/terraform-python/files/ansible/ansible.conda-info.txt @@ -27,4 +27,3 @@ UID:GID : 1043:1408 netrc file : None offline mode : False - diff --git a/local-app/ansible/roles/terraform-python/files/ansible/ansible.revisions.txt b/local-app/ansible/roles/terraform-python/files/ansible/ansible.revisions.txt index 30ae2955..6e75f345 100644 --- a/local-app/ansible/roles/terraform-python/files/ansible/ansible.revisions.txt +++ b/local-app/ansible/roles/terraform-python/files/ansible/ansible.revisions.txt @@ -52,4 +52,3 @@ +xz-5.2.5 (defaults/linux-64) +yaml-0.2.5 (defaults/linux-64) +zlib-1.2.11 (defaults/linux-64) - diff --git a/local-app/ansible/roles/terraform-python/files/base/base.conda-info.txt b/local-app/ansible/roles/terraform-python/files/base/base.conda-info.txt index e8fec139..c30d6ae9 100644 --- a/local-app/ansible/roles/terraform-python/files/base/base.conda-info.txt +++ b/local-app/ansible/roles/terraform-python/files/base/base.conda-info.txt @@ -27,4 +27,3 @@ UID:GID : 1043:1408 netrc file : None offline mode : False - diff --git a/local-app/ansible/roles/terraform-python/files/base/base.revisions.txt b/local-app/ansible/roles/terraform-python/files/base/base.revisions.txt index 4716f8dd..4de9543f 100644 --- a/local-app/ansible/roles/terraform-python/files/base/base.revisions.txt +++ b/local-app/ansible/roles/terraform-python/files/base/base.revisions.txt @@ -107,4 +107,3 @@ +ruamel.yaml.clib-0.2.6 (https://nexus.it.census.gov:8443/repository/anaconda-proxy/main/linux-64) +termcolor-2.1.0 (https://nexus.it.census.gov:8443/repository/anaconda-proxy/main/linux-64) +toolz-0.12.0 (https://nexus.it.census.gov:8443/repository/anaconda-proxy/main/linux-64) - diff --git a/local-app/ansible/roles/terraform-python/files/conda-pre-install.sh b/local-app/ansible/roles/terraform-python/files/conda-pre-install.sh index fd8a3f41..9275de25 100755 --- a/local-app/ansible/roles/terraform-python/files/conda-pre-install.sh +++ b/local-app/ansible/roles/terraform-python/files/conda-pre-install.sh @@ -27,27 +27,27 @@ then echo "* conda list -n $PYENV > $PYENV.txt" conda list -n $PYENV > $PYENV.txt - + echo "* conda list -n $PYENV --revisions > $PYENV.revisions.txt" conda list -n $PYENV --revisions > $PYENV.revisions.txt - + echo "* conda list -n $PYENV --explicit > $PYENV.explicit.txt" conda list -n $PYENV --explicit > $PYENV.explicit.txt - + echo "* conda env export -n $PYENV > $PYENV.yml" conda env export -n $PYENV > $PYENV.yml - + echo "* pip list" > $PYENV.pip.txt pip list > $PYENV.pip.txt echo "* ldd /apps/anaconda/envs/$PYENV > $PYENV.ldd.txt" find /apps/anaconda/envs/$PYENV -name "*.so*" -exec ldd {} \; -print > $PYENV.ldd.txt 2> /dev/null fi - + if [ ! -z $PY_LIST_FILES ] then echo "* list files in enviromment (long)" find /apps/anaconda/envs/$PYENV -print -exec stat --format 'change="%z" modify="%y" %n' {} \; > $PYENV.find.txt && gzip $PYENV.find.txt -fi +fi # script conda.$PYENV.$SDATESTAMP.log diff --git a/local-app/ansible/roles/terraform-python/tasks/main.yml b/local-app/ansible/roles/terraform-python/tasks/main.yml index cac69e68..f413eb62 100644 --- a/local-app/ansible/roles/terraform-python/tasks/main.yml +++ b/local-app/ansible/roles/terraform-python/tasks/main.yml @@ -39,19 +39,19 @@ mode: "0644" src: "pip.conf" dest: "{{ anaconda_install_dir }}/pip.conf" - + - name: provision .condarc copy: mode: "0644" src: "condarc" dest: "{{ anaconda_install_dir }}/condarc" - + ## - name: setup miniconda.extra.yml ## template: ## mode: '0644' ## src: "miniconda.extra.yml.j2" ## dest: "{{ anaconda_install_dir }}/etc/miniconda.extra.yml" -## +## ## - name: install extras ## shell: | ## umask 022; @@ -60,14 +60,14 @@ ## http_proxy: "http://proxy.tco.census.gov:3128" ## https_proxy: "http://proxy.tco.census.gov:3128" ## no_proxy: "mirror1.csvd.census.gov,*.census.gov,169.254.169.254" -## +## - name: install certificate-update script copy: mode: "0755" src: "add-root-certificate.py" dest: "{{ app_dir_bin }}/add-root-certificate.py" - -# figure a way to run this when certifi changes + +# figure a way to run this when certifi changes # - name: update root certificates # command: "{{ anaconda_install_dir }}/bin/python {{ app_dir_bin }}/add-root-certificate.py" diff --git a/local-app/ansible/roles/terraform-python/vars/main.yml b/local-app/ansible/roles/terraform-python/vars/main.yml index 219c7c3d..163608e8 100644 --- a/local-app/ansible/roles/terraform-python/vars/main.yml +++ b/local-app/ansible/roles/terraform-python/vars/main.yml @@ -2,7 +2,7 @@ ## anaconda_dep_pkgs: ## - grep -## +## ## anaconda_checksum: '{{ anaconda_checksums[anaconda_installer_sh] }}' ## anaconda_install_dir: '{{ anaconda_parent_dir }}/{{ anaconda_name }}' ## anaconda_link_dir: '{{ anaconda_parent_dir }}/{{ anaconda_link_subdir }}' diff --git a/local-app/aws-account-setup/ansible/README.md b/local-app/aws-account-setup/ansible/README.md index 5cd12ff3..5b94d283 100644 --- a/local-app/aws-account-setup/ansible/README.md +++ b/local-app/aws-account-setup/ansible/README.md @@ -69,52 +69,52 @@ and then destroy it. The following roles exist with lots of various setup in each one. -* set-facts +* set-facts Sets up some variables to be used by the other roles. Required for each role. -* setup-directories +* setup-directories This setups up the directories: infrastructure, common, vpc, and region specific directories within each of those locations. -* setup-credentials +* setup-credentials This creates the regions specific credentials files in `TOP/credentials.d` to which links are made from throughout the structure. -* setup-variables +* setup-variables This creates the regions specific commong variables files in `TOP/variables.d` to which links are made from throughout the structure. -* setup-remote-state +* setup-remote-state This setups up the the initial `remote_state.yml` from a template for the directories in `setup-directories` above. -* setup-provider-configs +* setup-provider-configs This setups up provider specific configurations in `TOP/providers.d`, with .tf and .tfvars files accordingly. The .tfvars files will likely be encrypted with `git-secret` and usable only by those with the appropriate access. -* setup-git-repo +* setup-git-repo This creates a file in `TOP/init/git-setup` to be used one time for initializing the github repository. Once done, all files can be brought into git control easily. If modifying issue templates in this role's `files/ISSUE_TEMPLATE/` directory, you can use `--tags distribute-github-templates` to just copy them over. -* setup-gpg +* setup-gpg This creates the accounts-specific GPG key in `TOP/init/gpg-setup` which is used for encrypting passwords and access keys within various resource blocks and modules. To decrypt, one will need to import the private key and public key. -* setup-git-secret +* setup-git-secret This sets up the initial git secret configuration and provides directions on importing keys for use of git secret. -* [inf-common](roles/inf-common/files/INF.SETUP.md) +* [inf-common](roles/inf-common/files/INF.SETUP.md) Setup of SAML, IAM roles, IAM groups, IAM users, policies, KMS keys, etc. -* [inf-infrastructure](roles/inf-infrastructure/files/INF.SETUP.md) +* [inf-infrastructure](roles/inf-infrastructure/files/INF.SETUP.md) Setup of TF state bucket and dynamodb table, access log bucket, flow logs bucket, object logging bucket, cloudtrail, config, splunk things, etc. -* [inf-vpc](roles/inf-vpc/files/INF.SETUP.md) +* [inf-vpc](roles/inf-vpc/files/INF.SETUP.md) Setup of VPC structure for every region, including unused ones which will be in a directory `vpc/unused/{region}`. Within the unused regions, you'll need to visit each one to bring in the defaults and then remove them. We are required by ATO to remove resources (VPC, subnets, etc) from unused regions, and for us, that includes everything that does not diff --git a/local-app/aws-account-setup/ansible/README.md.pdf b/local-app/aws-account-setup/ansible/README.md.pdf index 4b51bb1d..4e45fe4e 100644 Binary files a/local-app/aws-account-setup/ansible/README.md.pdf and b/local-app/aws-account-setup/ansible/README.md.pdf differ diff --git a/local-app/aws-account-setup/ansible/TODO.md b/local-app/aws-account-setup/ansible/TODO.md index 2d9577b3..7e0321db 100644 --- a/local-app/aws-account-setup/ansible/TODO.md +++ b/local-app/aws-account-setup/ansible/TODO.md @@ -1,4 +1,3 @@ - [x] add ses-domain docs - [x] add config - [ ] add automation (scripts) - diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/README.md b/local-app/aws-account-setup/ansible/inventory/group_vars/README.md index 10b64adf..0d3dfcf2 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/README.md +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/README.md @@ -22,7 +22,7 @@ The ansible configuration is in the [terraform/support](https://github.e.it.cens `local-app/aws-account-setup/ansible`. We will use `ma99` as the example account alias for this description. ```script -# git clone git@github.e.it.census.gov:terraform/support.git +# git clone git@github.e.it.census.gov:terraform/support.git cd support cd local-app/aws-account-setup/ansible/inventory/group_vars mkdir ma99-ew ma99-gov @@ -148,7 +148,10 @@ Once merged, you'll be able to go on to the next step of deploying. - initial * 1.0.1 -- 2024-06-10 - add configuration for EDL accounts +<<<<<<< HEAD * 1.0.2 -- 2025-05-23 - change github to gitlab * 1.0.3 -- 2025-05-28 - change back to github from gitlab +======= +>>>>>>> 60c40e7 (wip) diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/adsd-dvs-prod-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/adsd-dvs-prod-gov/main.yml index ab24cdfa..69d08203 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/adsd-dvs-prod-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/adsd-dvs-prod-gov/main.yml @@ -5,4 +5,3 @@ tf_region_type: "gov" vpc_dns_servers: "{{ dns_servers['internal'] }}" vpc_ntp_servers: "{{ ntp_servers['internal'] }}" vpc_domain_name: "" - diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/all.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/all.yml index 1d96f8de..37f9d9f4 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/all.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/all.yml @@ -18,7 +18,7 @@ tf_provider_config_dir: provider_configs.d tf_includes_dir: includes.d aws_organization: "" -aws_organization_allowed: +aws_organization_allowed: - ent-ew - ent-gov - lab-gov @@ -40,7 +40,7 @@ tf_main_directories: tf_main_config_directories: - apps - + tf_main_other_directories: - infrastructure/global diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/cedsci-dev-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/cedsci-dev-ew/main.yml index cfc62105..94bf30bf 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/cedsci-dev-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/cedsci-dev-ew/main.yml @@ -10,4 +10,3 @@ host_admin_iam_accounts: hunte359: username: hunte359 generate_password: false - diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/cedsci-dev-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/cedsci-dev-gov/main.yml index 78599b9e..6c6e344c 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/cedsci-dev-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/cedsci-dev-gov/main.yml @@ -10,4 +10,3 @@ host_admin_iam_accounts: hunte359: username: hunte359 generate_password: false - diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/censusaws/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/censusaws/main.yml index 4bc51a80..3e8809bb 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/censusaws/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/censusaws/main.yml @@ -13,9 +13,9 @@ host_admin_iam_accounts: # ticket_number: "" ashle001: username: ashle001 - generate_password: true + generate_password: true ticket_number: "TBD" badra001: username: badra001 - generate_password: true + generate_password: true ticket_number: "TBD" diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/csd-vdi-dev-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/csd-vdi-dev-ew/main.yml index 18bc4305..213a7f9c 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/csd-vdi-dev-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/csd-vdi-dev-ew/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TBD" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/csd-vdi-dev-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/csd-vdi-dev-gov/main.yml index 4425cc66..a8ada305 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/csd-vdi-dev-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/csd-vdi-dev-gov/main.yml @@ -11,4 +11,3 @@ host_admin_iam_accounts: username: cymer001 generate_password: false ticket_number: "TBD" - diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/csvd-test-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/csvd-test-ew/main.yml index 07c984bf..04728e00 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/csvd-test-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/csvd-test-ew/main.yml @@ -11,5 +11,5 @@ vpc_domain_name: "" ### username: USERNAME ### generate_password: false ### ticket_number: "TICKETID" -### +### # diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/csvd-test-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/csvd-test-gov/main.yml index c4e818d7..142a408b 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/csvd-test-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/csvd-test-gov/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ### username: USERNAME ### generate_password: false ### ticket_number: "TICKETID" -### +### diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-ams-amt-ite-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-ams-amt-ite-ew/main.yml index f3fc318f..11c6f244 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-ams-amt-ite-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-ams-amt-ite-ew/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-ams-amt-ite-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-ams-amt-ite-gov/main.yml index f8777807..b74e3c18 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-ams-amt-ite-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-ams-amt-ite-gov/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-ams-amt-stage-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-ams-amt-stage-ew/main.yml index f8ad467f..cdef965c 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-ams-amt-stage-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-ams-amt-stage-ew/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-ams-amt-stage-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-ams-amt-stage-gov/main.yml index 97111582..097c05dc 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-ams-amt-stage-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-ams-amt-stage-gov/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-amt-dmz-prod-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-amt-dmz-prod-gov/main.yml index 61ab38bf..fad82707 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-amt-dmz-prod-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-amt-dmz-prod-gov/main.yml @@ -4,4 +4,4 @@ tf_account_use: "DITD AMT DMZ PROD GovCloud" tf_region_type: "gov" vpc_dns_servers: "{{ dns_servers['external'] }}" vpc_ntp_servers: "{{ ntp_servers['external'] }}" -vpc_domain_name: "" +vpc_domain_name: "" diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-amt-dmz-stage-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-amt-dmz-stage-gov/main.yml index 1496765e..e4a052aa 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-amt-dmz-stage-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-amt-dmz-stage-gov/main.yml @@ -4,4 +4,4 @@ tf_account_use: "DITD AMT DMZ STAGE GovCloud" tf_region_type: "gov" vpc_dns_servers: "{{ dns_servers['external'] }}" vpc_ntp_servers: "{{ ntp_servers['external'] }}" -vpc_domain_name: "" +vpc_domain_name: "" diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gppsys-prod-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gppsys-prod-ew/main.yml index 73f91ae8..8ca5b8f4 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gppsys-prod-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gppsys-prod-ew/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gppsys-prod-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gppsys-prod-gov/main.yml index 17026cac..633c6554 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gppsys-prod-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gppsys-prod-gov/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gppsys-stage-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gppsys-stage-ew/main.yml index 9397f029..db60428d 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gppsys-stage-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gppsys-stage-ew/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gppsys-stage-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gppsys-stage-gov/main.yml index bbe0f7f6..8e2a0bd0 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gppsys-stage-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gppsys-stage-gov/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gups-dmz-ite-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gups-dmz-ite-ew/main.yml index a6297f2d..b713b565 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gups-dmz-ite-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gups-dmz-ite-ew/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gups-dmz-ite-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gups-dmz-ite-gov/main.yml index cbf76a5e..848d4b6d 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gups-dmz-ite-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gups-dmz-ite-gov/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gups-dmz-prod-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gups-dmz-prod-ew/main.yml index 2e1b0e17..61086031 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gups-dmz-prod-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gups-dmz-prod-ew/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gups-dmz-prod-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gups-dmz-prod-gov/main.yml index 05bc0da2..d6152080 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gups-dmz-prod-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gups-dmz-prod-gov/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gups-dmz-stage-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gups-dmz-stage-gov/main.yml index e924ba57..4077a174 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gups-dmz-stage-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-gups-dmz-stage-gov/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-nonprod-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-nonprod-gov/main.yml index dc540503..cedad1af 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-nonprod-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-nonprod-gov/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-partnerportal-dmz-ite-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-partnerportal-dmz-ite-ew/main.yml index bc3cf17a..3d9de5c9 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-partnerportal-dmz-ite-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-partnerportal-dmz-ite-ew/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-partnerportal-dmz-ite-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-partnerportal-dmz-ite-gov/main.yml index 3390a949..75107e0b 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-partnerportal-dmz-ite-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-partnerportal-dmz-ite-gov/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-partnerportal-dmz-prod-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-partnerportal-dmz-prod-ew/main.yml index 3dec0f86..0e3520dd 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-partnerportal-dmz-prod-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-partnerportal-dmz-prod-ew/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-partnerportal-dmz-prod-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-partnerportal-dmz-prod-gov/main.yml index 06a5cc4b..3cdc71ee 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-partnerportal-dmz-prod-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-partnerportal-dmz-prod-gov/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-partnerportal-dmz-stage-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-partnerportal-dmz-stage-ew/main.yml index 522c928f..54a0a1f9 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-partnerportal-dmz-stage-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-partnerportal-dmz-stage-ew/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-partnerportal-dmz-stage-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-partnerportal-dmz-stage-gov/main.yml index 7c43ae3a..d27c9d79 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-partnerportal-dmz-stage-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-partnerportal-dmz-stage-gov/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-partnerportal-prod-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-partnerportal-prod-ew/main.yml index 61686caf..bcb61046 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-partnerportal-prod-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-partnerportal-prod-ew/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-partnerportal-prod-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-partnerportal-prod-gov/main.yml index ecc8546a..b9ed9717 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-partnerportal-prod-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-partnerportal-prod-gov/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-sdpcs-prod-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-sdpcs-prod-ew/main.yml index 98ac6b08..262f340c 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-sdpcs-prod-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-sdpcs-prod-ew/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-sdpcs-prod-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-sdpcs-prod-gov/main.yml index 22c6c7cb..f53cd559 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-sdpcs-prod-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-sdpcs-prod-gov/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-sdpcs-stage-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-sdpcs-stage-ew/main.yml index cf4ebbc8..3cff8d20 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-sdpcs-stage-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-sdpcs-stage-ew/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-sdpcs-stage-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-sdpcs-stage-gov/main.yml index 5463e24a..0fd8069e 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-sdpcs-stage-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ditd-sdpcs-stage-gov/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/econ-ead-prod-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/econ-ead-prod-gov/main.yml index d2a2051f..c7910ce4 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/econ-ead-prod-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/econ-ead-prod-gov/main.yml @@ -4,4 +4,4 @@ tf_account_use: "ECON EAD PROD GovCloud" tf_region_type: "gov" vpc_dns_servers: [ ] vpc_ntp_servers: [ ] -vpc_domain_name: "" +vpc_domain_name: "" diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adfo-common-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adfo-common-ew/main.yml index fd5f4859..0b84150e 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adfo-common-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adfo-common-ew/main.yml @@ -12,4 +12,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adfo-common-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adfo-common-gov/main.yml index 4064ccc3..77c3d3b3 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adfo-common-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adfo-common-gov/main.yml @@ -12,4 +12,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adfo-dev-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adfo-dev-ew/main.yml index 40eed3e7..c1b72bfc 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adfo-dev-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adfo-dev-ew/main.yml @@ -12,4 +12,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adfo-dev-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adfo-dev-gov/main.yml index dcfa7777..1f00c304 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adfo-dev-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adfo-dev-gov/main.yml @@ -12,4 +12,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adfo-nonprod-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adfo-nonprod-ew/main.yml index 7f80e379..cc1ef2ca 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adfo-nonprod-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adfo-nonprod-ew/main.yml @@ -12,4 +12,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adfo-nonprod-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adfo-nonprod-gov/main.yml index 7d30a2f0..8490f630 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adfo-nonprod-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adfo-nonprod-gov/main.yml @@ -12,4 +12,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adfo-prod-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adfo-prod-ew/main.yml index 3bd8efbd..e752ed12 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adfo-prod-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adfo-prod-ew/main.yml @@ -12,4 +12,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adfo-prod-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adfo-prod-gov/main.yml index d4111b89..c8e49e71 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adfo-prod-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adfo-prod-gov/main.yml @@ -12,4 +12,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adrm-common-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adrm-common-ew/main.yml index 6e00ff51..be52f050 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adrm-common-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adrm-common-ew/main.yml @@ -12,4 +12,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adrm-common-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adrm-common-gov/main.yml index d0604ac3..d23b6b79 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adrm-common-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adrm-common-gov/main.yml @@ -12,4 +12,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adrm-dev-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adrm-dev-ew/main.yml index f74ea9c8..cecf098d 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adrm-dev-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adrm-dev-ew/main.yml @@ -12,4 +12,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adrm-dev-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adrm-dev-gov/main.yml index 63895145..62c8a58d 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adrm-dev-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adrm-dev-gov/main.yml @@ -12,4 +12,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adrm-nonprod-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adrm-nonprod-ew/main.yml index 294a20f2..cc9c65bc 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adrm-nonprod-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adrm-nonprod-ew/main.yml @@ -12,4 +12,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adrm-nonprod-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adrm-nonprod-gov/main.yml index c02ae782..1bf5e251 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adrm-nonprod-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adrm-nonprod-gov/main.yml @@ -12,4 +12,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adrm-prod-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adrm-prod-ew/main.yml index b3198313..283f87c5 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adrm-prod-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adrm-prod-ew/main.yml @@ -12,4 +12,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adrm-prod-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adrm-prod-gov/main.yml index fa8f7d31..7f459acb 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adrm-prod-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-adrm-prod-gov/main.yml @@ -12,4 +12,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-cedsci-prod-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-cedsci-prod-gov/main.yml index 16ea60a0..e7202a6c 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-cedsci-prod-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-cedsci-prod-gov/main.yml @@ -5,4 +5,3 @@ tf_region_type: "gov" vpc_dns_servers: [ ] vpc_ntp_servers: [ ] vpc_domain_name: "" - diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-ocio-common-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-ocio-common-ew/main.yml index 73707a90..29273d6a 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-ocio-common-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-ocio-common-ew/main.yml @@ -12,4 +12,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-ocio-common-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-ocio-common-gov/main.yml index d47da747..6fb76baa 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-ocio-common-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-ocio-common-gov/main.yml @@ -12,4 +12,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-ocio-dev-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-ocio-dev-ew/main.yml index 132cbd8b..2bac9b19 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-ocio-dev-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-ocio-dev-ew/main.yml @@ -12,4 +12,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-ocio-dev-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-ocio-dev-gov/main.yml index f352c544..83ccd3ac 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-ocio-dev-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-ocio-dev-gov/main.yml @@ -12,4 +12,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-ocio-nonprod-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-ocio-nonprod-ew/main.yml index 36655fcd..5f703cc6 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-ocio-nonprod-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-ocio-nonprod-ew/main.yml @@ -12,4 +12,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-ocio-nonprod-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-ocio-nonprod-gov/main.yml index c57e23fc..b689e3c0 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-ocio-nonprod-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-ocio-nonprod-gov/main.yml @@ -12,4 +12,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-ocio-prod-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-ocio-prod-ew/main.yml index 3924d351..7216c3ca 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-ocio-prod-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-ocio-prod-ew/main.yml @@ -12,4 +12,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-ocio-prod-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-ocio-prod-gov/main.yml index 9a6d1671..8683ed0b 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-ocio-prod-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-ocio-prod-gov/main.yml @@ -12,4 +12,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-shared-dev-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-shared-dev-gov/main.yml index e40f84c5..1f80ca83 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-shared-dev-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-shared-dev-gov/main.yml @@ -4,4 +4,4 @@ tf_account_use: "EDL Shared GovCloud Dev" tf_region_type: "gov" vpc_dns_servers: [ ] vpc_ntp_servers: [ ] -vpc_domain_name: "" +vpc_domain_name: "" diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-shared-nonprod-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-shared-nonprod-gov/main.yml index e44dbbca..ccf31052 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/edl-shared-nonprod-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/edl-shared-nonprod-gov/main.yml @@ -4,4 +4,4 @@ tf_account_use: "EDL Shared GovCloud NonProd" tf_region_type: "gov" vpc_dns_servers: [ ] vpc_ntp_servers: [ ] -vpc_domain_name: "" +vpc_domain_name: "" diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ent-ew-network-nonprod/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ent-ew-network-nonprod/main.yml index bbfd6384..5b21e41f 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ent-ew-network-nonprod/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ent-ew-network-nonprod/main.yml @@ -5,4 +5,3 @@ tf_region_type: "ew" vpc_dns_servers: [ ] vpc_ntp_servers: [ ] vpc_domain_name: "" - diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ent-gov-network-nonprod/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ent-gov-network-nonprod/main.yml index 816fd43a..2851301a 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ent-gov-network-nonprod/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ent-gov-network-nonprod/main.yml @@ -5,4 +5,3 @@ tf_region_type: "gov" vpc_dns_servers: [ ] vpc_ntp_servers: [ ] vpc_domain_name: "" - diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/erd-dcdl-dev-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/erd-dcdl-dev-ew/main.yml index 0c48e89a..24fd56b7 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/erd-dcdl-dev-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/erd-dcdl-dev-ew/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/erd-dcdl-dev-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/erd-dcdl-dev-gov/main.yml index 7d3fdbda..6b3ab4eb 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/erd-dcdl-dev-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/erd-dcdl-dev-gov/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/erd-dcdl-ite-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/erd-dcdl-ite-ew/main.yml index 738b36dd..4cfc46ff 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/erd-dcdl-ite-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/erd-dcdl-ite-ew/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ### username: USERNAME ### generate_password: false ### ticket_number: "TICKETID" -### +### diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/erd-dcdl-ite-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/erd-dcdl-ite-gov/main.yml index 6c392ab1..0be26934 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/erd-dcdl-ite-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/erd-dcdl-ite-gov/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ### username: USERNAME ### generate_password: false ### ticket_number: "TICKETID" -### +### diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/erd-dcdl-prod-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/erd-dcdl-prod-ew/main.yml index a642abc9..bf2c1864 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/erd-dcdl-prod-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/erd-dcdl-prod-ew/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" #### username: USERNAME #### generate_password: false #### ticket_number: "TICKETID" -#### +#### diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/erd-dcdl-prod-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/erd-dcdl-prod-gov/main.yml index 53582590..309b35ef 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/erd-dcdl-prod-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/erd-dcdl-prod-gov/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" #### username: USERNAME #### generate_password: false #### ticket_number: "TICKETID" -#### +#### diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/lab-dev-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/lab-dev-ew/main.yml index a544cdbf..b89bf648 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/lab-dev-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/lab-dev-ew/main.yml @@ -12,4 +12,4 @@ is_lab_account: true ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/lab-dev-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/lab-dev-gov/main.yml index c543d280..23326bf1 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/lab-dev-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/lab-dev-gov/main.yml @@ -12,4 +12,4 @@ is_lab_account: true ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/lab-ew-management-nonprod/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/lab-ew-management-nonprod/main.yml index 112632ff..f95e2adf 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/lab-ew-management-nonprod/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/lab-ew-management-nonprod/main.yml @@ -12,4 +12,4 @@ is_lab_account: true ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/lab-ew-network-nonprod/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/lab-ew-network-nonprod/main.yml index 270d4a9e..7b6456fa 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/lab-ew-network-nonprod/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/lab-ew-network-nonprod/main.yml @@ -6,4 +6,3 @@ vpc_dns_servers: [ ] vpc_ntp_servers: [ ] vpc_domain_name: "" is_lab_account: true - diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/lab-gov-dmz-network-nonprod/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/lab-gov-dmz-network-nonprod/main.yml index 74102092..00f6dbb9 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/lab-gov-dmz-network-nonprod/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/lab-gov-dmz-network-nonprod/main.yml @@ -4,5 +4,5 @@ tf_account_use: "LAB DMZ Network GovCloud NonProd" tf_region_type: "gov" vpc_dns_servers: [ ] vpc_ntp_servers: [ ] -vpc_domain_name: "" +vpc_domain_name: "" is_lab_account: true diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/lab-gov-management-nonprod/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/lab-gov-management-nonprod/main.yml index 9527c4c6..720149e0 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/lab-gov-management-nonprod/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/lab-gov-management-nonprod/main.yml @@ -12,4 +12,4 @@ is_lab_account: true ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/lab-gov-network-sa/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/lab-gov-network-sa/main.yml index 3da88627..6c5c9136 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/lab-gov-network-sa/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/lab-gov-network-sa/main.yml @@ -6,4 +6,3 @@ vpc_dns_servers: [ ] vpc_ntp_servers: [ ] vpc_domain_name: "" is_lab_account: true - diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ma10-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ma10-gov/main.yml index cc47e729..6ce5480a 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ma10-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ma10-gov/main.yml @@ -15,11 +15,11 @@ host_admin_iam_accounts: # ticket_number: "" ashle001: username: ashle001 - generate_password: true + generate_password: true ticket_number: "TBD" badra001: username: badra001 - generate_password: true + generate_password: true ticket_number: "REQ000003062360" dwara001: username: dwara001 diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ma15-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ma15-ew/main.yml index 9a251a54..4fd97904 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ma15-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ma15-ew/main.yml @@ -4,4 +4,4 @@ tf_account_use: "DO NOT USE" tf_region_type: "ew" vpc_dns_servers: [ ] vpc_ntp_servers: [ ] -vpc_domain_name: "" +vpc_domain_name: "" diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ma16-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ma16-ew/main.yml index fc4eec1a..bdcc9d7f 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ma16-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ma16-ew/main.yml @@ -4,4 +4,4 @@ tf_account_use: "DO NOT USE" tf_region_type: "ew" vpc_dns_servers: [ ] vpc_ntp_servers: [ ] -vpc_domain_name: "" +vpc_domain_name: "" diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ma32-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ma32-ew/main.yml index f234afb2..7e34a93c 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ma32-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ma32-ew/main.yml @@ -4,4 +4,4 @@ tf_account_use: "dcom" tf_region_type: "ew" vpc_dns_servers: [ ] vpc_ntp_servers: [ ] -vpc_domain_name: "" +vpc_domain_name: "" diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ma42-ew/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ma42-ew/main.yml index 01085118..a0458c2b 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ma42-ew/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ma42-ew/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/ma42-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/ma42-gov/main.yml index 7904f0cd..8a09e03e 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/ma42-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/ma42-gov/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/npc-prod-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/npc-prod-gov/main.yml index 2e6969b8..34157b26 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/npc-prod-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/npc-prod-gov/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ### username: USERNAME ### generate_password: false ### ticket_number: "TICKETID" -### +### diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/tco-nonprod-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/tco-nonprod-gov/main.yml index d82137d1..75541860 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/tco-nonprod-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/tco-nonprod-gov/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ### username: USERNAME ### generate_password: false ### ticket_number: "TICKETID" -### +### diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/tco-prod-gov/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/tco-prod-gov/main.yml index 8c797915..e138f2bc 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/tco-prod-gov/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/tco-prod-gov/main.yml @@ -11,4 +11,4 @@ vpc_domain_name: "" ### username: USERNAME ### generate_password: false ### ticket_number: "TICKETID" -### +### diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/template/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/template/main.yml index 03d1aa50..a7f06f74 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/template/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/template/main.yml @@ -13,4 +13,4 @@ is_lab_account: false ## username: USERNAME ## generate_password: false ## ticket_number: "TICKETID" -## +## diff --git a/local-app/aws-account-setup/ansible/inventory/group_vars/uscensus-organization/main.yml b/local-app/aws-account-setup/ansible/inventory/group_vars/uscensus-organization/main.yml index 74c89289..164236ad 100644 --- a/local-app/aws-account-setup/ansible/inventory/group_vars/uscensus-organization/main.yml +++ b/local-app/aws-account-setup/ansible/inventory/group_vars/uscensus-organization/main.yml @@ -13,9 +13,9 @@ host_admin_iam_accounts: # ticket_number: "" ashle001: username: ashle001 - generate_password: true + generate_password: true ticket_number: "TBD" badra001: username: badra001 - generate_password: true + generate_password: true ticket_number: "TBD" diff --git a/local-app/aws-account-setup/ansible/inventory/inventory.yml b/local-app/aws-account-setup/ansible/inventory/inventory.yml index 847f3f43..4c7c4dbe 100644 --- a/local-app/aws-account-setup/ansible/inventory/inventory.yml +++ b/local-app/aws-account-setup/ansible/inventory/inventory.yml @@ -97,13 +97,13 @@ all: ma15-ew.cloud ansible_host=localhost ma15-gov: hosts: - ma15-gov.cloud ansible_host=localhost + ma15-gov.cloud ansible_host=localhost ma16-ew: hosts: ma16-ew.cloud ansible_host=localhost ma16-gov: hosts: - ma16-gov.cloud ansible_host=localhost + ma16-gov.cloud ansible_host=localhost ma17-ew: hosts: ma17-ew.cloud ansible_host=localhost @@ -145,7 +145,7 @@ all: ma32-ew.cloud ansible_host=localhost ma32-gov: hosts: - ma32-gov.cloud ansible_host=localhost + ma32-gov.cloud ansible_host=localhost ma33-ew: hosts: ma33-ew.cloud ansible_host=localhost @@ -397,67 +397,67 @@ all: edl-addcp-dev-gov.cloud ansible_host=localhost edl-addcp-common-ew: hosts: - edl-addcp-common-ew.cloud ansible_host=localhost + edl-addcp-common-ew.cloud ansible_host=localhost edl-addcp-common-gov: hosts: - edl-addcp-common-gov.cloud ansible_host=localhost + edl-addcp-common-gov.cloud ansible_host=localhost edl-addp-dev-ew: hosts: edl-addp-dev-ew.cloud ansible_host=localhost edl-addp-dev-gov: hosts: - edl-addp-dev-gov.cloud ansible_host=localhost + edl-addp-dev-gov.cloud ansible_host=localhost edl-addp-nonprod-ew: hosts: - edl-addp-nonprod-ew.cloud ansible_host=localhost + edl-addp-nonprod-ew.cloud ansible_host=localhost edl-addp-nonprod-gov: hosts: - edl-addp-nonprod-gov.cloud ansible_host=localhost + edl-addp-nonprod-gov.cloud ansible_host=localhost edl-addp-prod-ew: hosts: - edl-addp-prod-ew.cloud ansible_host=localhost + edl-addp-prod-ew.cloud ansible_host=localhost edl-addp-prod-gov: hosts: - edl-addp-prod-gov.cloud ansible_host=localhost + edl-addp-prod-gov.cloud ansible_host=localhost edl-adep-dev-ew: hosts: - edl-adep-dev-ew.cloud ansible_host=localhost + edl-adep-dev-ew.cloud ansible_host=localhost edl-adep-dev-gov: hosts: edl-adep-dev-gov.cloud ansible_host=localhost edl-adep-nonprod-ew: hosts: - edl-adep-nonprod-ew.cloud ansible_host=localhost + edl-adep-nonprod-ew.cloud ansible_host=localhost edl-adep-nonprod-gov: hosts: edl-adep-nonprod-gov.cloud ansible_host=localhost edl-adep-prod-ew: hosts: - edl-adep-prod-ew.cloud ansible_host=localhost + edl-adep-prod-ew.cloud ansible_host=localhost edl-adep-prod-gov: hosts: - edl-adep-prod-gov.cloud ansible_host=localhost + edl-adep-prod-gov.cloud ansible_host=localhost edl-cedsci-dev-ew: hosts: - edl-cedsci-dev-ew.cloud ansible_host=localhost + edl-cedsci-dev-ew.cloud ansible_host=localhost edl-cedsci-dev-gov: hosts: edl-cedsci-dev-gov.cloud ansible_host=localhost edl-cedsci-nonprod-ew: hosts: - edl-cedsci-nonprod-ew.cloud ansible_host=localhost + edl-cedsci-nonprod-ew.cloud ansible_host=localhost edl-cedsci-nonprod-gov: hosts: edl-cedsci-nonprod-gov.cloud ansible_host=localhost edl-cedsci-prod-ew: hosts: - edl-cedsci-prod-ew.cloud ansible_host=localhost + edl-cedsci-prod-ew.cloud ansible_host=localhost edl-cedsci-prod-gov: hosts: edl-cedsci-prod-gov.cloud ansible_host=localhost edl-management-ew: hosts: - edl-management-ew.cloud ansible_host=localhost + edl-management-ew.cloud ansible_host=localhost edl-management-gov: hosts: edl-management-gov.cloud ansible_host=localhost @@ -484,13 +484,13 @@ all: edl-addp-common-ew.cloud ansible_host=localhost edl-addp-common-gov: hosts: - edl-addp-common-gov.cloud ansible_host=localhost + edl-addp-common-gov.cloud ansible_host=localhost edl-adep-common-ew: hosts: edl-adep-common-ew.cloud ansible_host=localhost edl-adep-common-gov: hosts: - edl-adep-common-gov.cloud ansible_host=localhost + edl-adep-common-gov.cloud ansible_host=localhost edl-cedsci-common-ew: hosts: edl-cedsci-common-ew.cloud ansible_host=localhost @@ -542,7 +542,7 @@ all: edl-adfo-prod-ew: hosts: edl-adfo-prod-ew.cloud ansible_host=localhost - edl-adfo-prod-gov: + edl-adfo-prod-gov: hosts: edl-adfo-prod-gov.cloud ansible_host=localhost edl-ocio-common-ew: @@ -562,13 +562,13 @@ all: edl-ocio-nonprod-ew.cloud ansible_host=localhost edl-ocio-nonprod-gov: hosts: - edl-ocio-nonprod-gov.cloud ansible_host=localhost + edl-ocio-nonprod-gov.cloud ansible_host=localhost edl-ocio-prod-ew: hosts: edl-ocio-prod-ew.cloud ansible_host=localhost edl-ocio-prod-gov: hosts: - edl-ocio-prod-gov.cloud ansible_host=localhost + edl-ocio-prod-gov.cloud ansible_host=localhost ditd-partnerportal-dmz-ite-ew: hosts: ditd-partnerportal-dmz-ite-ew.cloud ansible_host=localhost @@ -610,19 +610,19 @@ all: ditd-gups-dmz-stage-ew.cloud ansible_host=localhost ditd-gups-dmz-stage-gov: hosts: - ditd-gups-dmz-stage-gov.cloud ansible_host=localhost + ditd-gups-dmz-stage-gov.cloud ansible_host=localhost ditd-gups-dmz-prod-ew: hosts: ditd-gups-dmz-prod-ew.cloud ansible_host=localhost ditd-gups-dmz-prod-gov: hosts: - ditd-gups-dmz-prod-gov.cloud ansible_host=localhost + ditd-gups-dmz-prod-gov.cloud ansible_host=localhost ditd-nonprod-ew: hosts: ditd-nonprod-ew.cloud ansible_host=localhost ditd-nonprod-gov: hosts: - ditd-nonprod-gov.cloud ansible_host=localhost + ditd-nonprod-gov.cloud ansible_host=localhost npc-nonprod-ew: hosts: npc-nonprod-ew.cloud ansible_host=localhost @@ -646,7 +646,7 @@ all: tco-prod-ew.cloud ansible_host=localhost tco-prod-gov: hosts: - tco-prod-gov.cloud ansible_host=localhost + tco-prod-gov.cloud ansible_host=localhost adsd-dvs-nonprod-ew: hosts: adsd-dvs-nonprod-ew.cloud ansible_host=localhost @@ -694,7 +694,7 @@ all: ditd-amt-dmz-ite-ew.cloud ansible_host=localhost ditd-amt-dmz-ite-gov: hosts: - ditd-amt-dmz-ite-gov.cloud ansible_host=localhost + ditd-amt-dmz-ite-gov.cloud ansible_host=localhost ditd-amt-dmz-stage-ew: hosts: ditd-amt-dmz-stage-ew.cloud ansible_host=localhost diff --git a/local-app/aws-account-setup/ansible/roles/inf-common/files/INF.SETUP.md b/local-app/aws-account-setup/ansible/roles/inf-common/files/INF.SETUP.md index ca904924..3fb64ad0 100644 --- a/local-app/aws-account-setup/ansible/roles/inf-common/files/INF.SETUP.md +++ b/local-app/aws-account-setup/ansible/roles/inf-common/files/INF.SETUP.md @@ -101,7 +101,7 @@ tf-apply -target=module.role_cloud-admin -target=module.group_cloud-admin ## inf-ip-restriction This creates the ipre-restriction group and associated permissions -(requires `general`). +(requires `general`). ```shell cd common/ @@ -238,4 +238,3 @@ unset TARGET % git commit -m'complete setup of common' -a % git push origin initial-setup ``` - diff --git a/local-app/aws-account-setup/ansible/roles/inf-common/files/INF.bootstrap.tf b/local-app/aws-account-setup/ansible/roles/inf-common/files/INF.bootstrap.tf index 60b25b59..6a85d68d 100644 --- a/local-app/aws-account-setup/ansible/roles/inf-common/files/INF.bootstrap.tf +++ b/local-app/aws-account-setup/ansible/roles/inf-common/files/INF.bootstrap.tf @@ -4,7 +4,7 @@ # import policy attachment # B_POLICY=$(aws --profile $(get-profile) iam list-attached-user-policies --user-name bootstrap --query 'AttachedPolicies[].PolicyArn' --output text) -# +# # import and remove account (count set to 0) # tf-import aws_iam_user.admin_bootstrap bootstrap # tf-import aws_iam_user_policy_attachment.admin_bootstrap bootstrap/$B_POLICY diff --git a/local-app/aws-account-setup/ansible/roles/inf-common/files/INF.group.inf-cloud-admin.tf b/local-app/aws-account-setup/ansible/roles/inf-common/files/INF.group.inf-cloud-admin.tf index 5e556510..e3b3e247 100644 --- a/local-app/aws-account-setup/ansible/roles/inf-common/files/INF.group.inf-cloud-admin.tf +++ b/local-app/aws-account-setup/ansible/roles/inf-common/files/INF.group.inf-cloud-admin.tf @@ -14,4 +14,3 @@ output "group_cloud-admin_name" { description = "Group Name for inf-cloud-admin" value = module.group_cloud-admin.group_name } - diff --git a/local-app/aws-account-setup/ansible/roles/inf-common/files/INF.group.inf-ip-restriction.tf b/local-app/aws-account-setup/ansible/roles/inf-common/files/INF.group.inf-ip-restriction.tf index c4dc494a..08f9d0c2 100644 --- a/local-app/aws-account-setup/ansible/roles/inf-common/files/INF.group.inf-ip-restriction.tf +++ b/local-app/aws-account-setup/ansible/roles/inf-common/files/INF.group.inf-ip-restriction.tf @@ -14,4 +14,3 @@ output "group_ip-restriction_name" { description = "Group Name for inf-ip-restriction" value = module.group_ip-restriction.group_name } - diff --git a/local-app/aws-account-setup/ansible/roles/inf-common/files/INF.role.billing.tf b/local-app/aws-account-setup/ansible/roles/inf-common/files/INF.role.billing.tf index 3d1eae64..498c9c81 100644 --- a/local-app/aws-account-setup/ansible/roles/inf-common/files/INF.role.billing.tf +++ b/local-app/aws-account-setup/ansible/roles/inf-common/files/INF.role.billing.tf @@ -33,4 +33,3 @@ module "role_limited_billing" { account_alias = var.account_alias inline_policies = [{ name = "limited-billing", policy = module.general.custom_policy_documents["limited_billing"].policy }] } - diff --git a/local-app/aws-account-setup/ansible/roles/inf-common/files/inf-cloud-admin.users.tf b/local-app/aws-account-setup/ansible/roles/inf-common/files/inf-cloud-admin.users.tf index 63f49fa3..1aae988d 100644 --- a/local-app/aws-account-setup/ansible/roles/inf-common/files/inf-cloud-admin.users.tf +++ b/local-app/aws-account-setup/ansible/roles/inf-common/files/inf-cloud-admin.users.tf @@ -20,5 +20,3 @@ data "ldap_object" "users" { search_values = { cn = each.key } select_attributes = ["cn", "dn"] } - - diff --git a/local-app/aws-account-setup/ansible/roles/inf-common/tasks/main.yml b/local-app/aws-account-setup/ansible/roles/inf-common/tasks/main.yml index 42be0234..d3a7b598 100644 --- a/local-app/aws-account-setup/ansible/roles/inf-common/tasks/main.yml +++ b/local-app/aws-account-setup/ansible/roles/inf-common/tasks/main.yml @@ -72,7 +72,7 @@ # var: region_map.values() # tags: # - debug -# +# # - name: debug # debug: # var: region_files @@ -94,7 +94,7 @@ # templated files - name: deploy common setup configs from templates - template: + template: mode: '0644' src: "{{ item }}.j2" dest: "{{ tf_top }}/common/{{ item }}" diff --git a/local-app/aws-account-setup/ansible/roles/inf-infrastructure/files/INF.preload-kms.tf b/local-app/aws-account-setup/ansible/roles/inf-infrastructure/files/INF.preload-kms.tf index 1f89a7c9..b129e56f 100644 --- a/local-app/aws-account-setup/ansible/roles/inf-infrastructure/files/INF.preload-kms.tf +++ b/local-app/aws-account-setup/ansible/roles/inf-infrastructure/files/INF.preload-kms.tf @@ -1,4 +1,3 @@ data "aws_kms_alias" "ebs" { name = "alias/aws/ebs" } - diff --git a/local-app/aws-account-setup/ansible/roles/inf-infrastructure/files/import_cloudforms.tf.source b/local-app/aws-account-setup/ansible/roles/inf-infrastructure/files/import_cloudforms.tf.source index 467404de..812e47b7 100644 --- a/local-app/aws-account-setup/ansible/roles/inf-infrastructure/files/import_cloudforms.tf.source +++ b/local-app/aws-account-setup/ansible/roles/inf-infrastructure/files/import_cloudforms.tf.source @@ -52,4 +52,3 @@ variable "csvd_cloudforms_config" { type = map(object({sqs_name=string})) default = null } - diff --git a/local-app/aws-account-setup/ansible/roles/inf-infrastructure/tasks/main.yml b/local-app/aws-account-setup/ansible/roles/inf-infrastructure/tasks/main.yml index 47dd1e92..134de20d 100644 --- a/local-app/aws-account-setup/ansible/roles/inf-infrastructure/tasks/main.yml +++ b/local-app/aws-account-setup/ansible/roles/inf-infrastructure/tasks/main.yml @@ -44,11 +44,11 @@ # - name: debug # debug: # var: region_map.values() -# +# # - name: debug # debug: # var: region_files -# +# - name: deploy per-region infrastructure setup configs copy: mode: '0644' @@ -70,7 +70,7 @@ tags: - always -#--- +#--- # templates #--- - name: Generate region infrastructure template files @@ -93,4 +93,3 @@ - "{{ template_main_files }}" tags: - first-time - diff --git a/local-app/aws-account-setup/ansible/roles/inf-infrastructure/templates/INF.ses-domain.tf.j2 b/local-app/aws-account-setup/ansible/roles/inf-infrastructure/templates/INF.ses-domain.tf.j2 index df869e73..15760e04 100644 --- a/local-app/aws-account-setup/ansible/roles/inf-infrastructure/templates/INF.ses-domain.tf.j2 +++ b/local-app/aws-account-setup/ansible/roles/inf-infrastructure/templates/INF.ses-domain.tf.j2 @@ -1,4 +1,4 @@ -{% set region = item.1 %} +{% set region = item.1 %} {% if ( 'us-east-1' in region and tf_region_type == "ew" ) or ( 'us-gov-west-1' in region and tf_region_type == "gov" ) %} module "ses" { source = "git@github.e.it.census.gov:terraform-modules/aws-inf-setup.git//ses-domain?ref=tf-upgrade" diff --git a/local-app/aws-account-setup/ansible/roles/inf-infrastructure/templates/tags.yml.j2 b/local-app/aws-account-setup/ansible/roles/inf-infrastructure/templates/tags.yml.j2 index c4ba4e3c..29004675 100644 --- a/local-app/aws-account-setup/ansible/roles/inf-infrastructure/templates/tags.yml.j2 +++ b/local-app/aws-account-setup/ansible/roles/inf-infrastructure/templates/tags.yml.j2 @@ -6,7 +6,7 @@ finops: - cloudtrail - config - flowlog - - objectlog + - objectlog - quota - route53 - ses diff --git a/local-app/aws-account-setup/ansible/roles/inf-vpc/files/shared-setup/provider.awscc.tf b/local-app/aws-account-setup/ansible/roles/inf-vpc/files/shared-setup/provider.awscc.tf index 845dac5d..510f8c90 100644 --- a/local-app/aws-account-setup/ansible/roles/inf-vpc/files/shared-setup/provider.awscc.tf +++ b/local-app/aws-account-setup/ansible/roles/inf-vpc/files/shared-setup/provider.awscc.tf @@ -18,4 +18,3 @@ provider "awscc" { region = var.region_map["west"] profile = var.profile } - diff --git a/local-app/aws-account-setup/ansible/roles/inf-vpc/files/shared-setup/region.tf b/local-app/aws-account-setup/ansible/roles/inf-vpc/files/shared-setup/region.tf index b7b1696e..f6175061 100644 --- a/local-app/aws-account-setup/ansible/roles/inf-vpc/files/shared-setup/region.tf +++ b/local-app/aws-account-setup/ansible/roles/inf-vpc/files/shared-setup/region.tf @@ -1,4 +1,3 @@ locals { region = var.region } - diff --git a/local-app/aws-account-setup/ansible/roles/inf-vpc/files/shared-setup/tf-run.data b/local-app/aws-account-setup/ansible/roles/inf-vpc/files/shared-setup/tf-run.data index 4b5d6788..9d3ca423 100644 --- a/local-app/aws-account-setup/ansible/roles/inf-vpc/files/shared-setup/tf-run.data +++ b/local-app/aws-account-setup/ansible/roles/inf-vpc/files/shared-setup/tf-run.data @@ -17,7 +17,7 @@ LINKTOP includes.d/variables.application_tags.auto.tfvars COMMAND rm -f provider.ldap.* provider.ldap_new.* TAG init -COMMAND tf-init +COMMAND tf-init TAG start ALL diff --git a/local-app/aws-account-setup/ansible/roles/inf-vpc/files/unused/tf-run.data b/local-app/aws-account-setup/ansible/roles/inf-vpc/files/unused/tf-run.data index 7a2fca2f..2501d25c 100644 --- a/local-app/aws-account-setup/ansible/roles/inf-vpc/files/unused/tf-run.data +++ b/local-app/aws-account-setup/ansible/roles/inf-vpc/files/unused/tf-run.data @@ -8,5 +8,3 @@ COMMAND tf-directory-setup.py -l s3 COMMENT Now go into each of the region directories and do: tf-run apply COMMENT You may use this loop to handle it: COMMENT for r in \$(ls \?\?-\*-[0-9] -d); do pushd \$r; TFARGS=-auto-approve tf-run apply; popd; done - - diff --git a/local-app/aws-account-setup/ansible/roles/inf-vpc/files/unused/tf-run.region.data b/local-app/aws-account-setup/ansible/roles/inf-vpc/files/unused/tf-run.region.data index 0db653c3..114bf3a5 100644 --- a/local-app/aws-account-setup/ansible/roles/inf-vpc/files/unused/tf-run.region.data +++ b/local-app/aws-account-setup/ansible/roles/inf-vpc/files/unused/tf-run.region.data @@ -6,7 +6,7 @@ COMMAND tf-directory-setup.py -l none -f COMMAND tf-init -upgrade module.vpc_defaults COMMAND mv INF.defaults.tf INF.defaults.tf.completed -COMMAND touch INF.defaults.tf +COMMAND touch INF.defaults.tf # tf-destroy -target=module.vpc_defaults ALL COMMAND setup/delete-defaults.sh true diff --git a/local-app/aws-account-setup/ansible/roles/inf-vpc/files/vpc0/README.md b/local-app/aws-account-setup/ansible/roles/inf-vpc/files/vpc0/README.md index 528652e7..90dd49ed 100644 --- a/local-app/aws-account-setup/ansible/roles/inf-vpc/files/vpc0/README.md +++ b/local-app/aws-account-setup/ansible/roles/inf-vpc/files/vpc0/README.md @@ -3,14 +3,14 @@ This VPC is created when a Route53 Private Hosted Zone PHZ is needed in an account which uses shared subnets from a central account. A [support case](https://us-gov-west-1.console.amazonaws-us-gov.com/support/home?region=us-gov-west-1#/case/?displayId=13210918551&language=en) was opened with AWS -in account `258852445129-ma50-gov`. +in account `258852445129-ma50-gov`. Initial problem: > We have deployed a network account (057405694017-ent-gov-network-prod) with shared VPCs. We shared a VPC to another account (258852445129-ma50-gov). We are trying to create a Route53 private hosted zone in this account (ma50-gov). We use Terraform. Here is what it wants to do. -> +> > Terraform will perform the following actions: -> +> > # aws_route53_zone.cluster_domain will be created > + resource "aws_route53_zone" "cluster_domain" { > + arn = (known after apply) @@ -37,69 +37,69 @@ Initial problem: > + "eks-cluster-name" = "eks-eis-dev" > } > + zone_id = (known after apply) -> +> > + vpc { > + vpc_id = "vpc-0c0f6344679b1164e" > + vpc_region = "us-gov-east-1" > } > } -> +> > Plan: 1 to add, 0 to change, 0 to destroy.y. -> +> > Note this is in account ma50-gov. -> +> > The error we get is: -> +> > Error: creating Route53 Hosted Zone: InvalidVPCId: The VPC: vpc-0c0f6344679b1164e in region us-gov-east-1 that you provided is not authorized to make the association. > status code: 400, request id: 734b3129-1ca5-4e06-ab41-e6746c6e1cf7 -> +> > with aws_route53_zone.cluster_domain, > on dns-zone.tf line 6, in resource "aws_route53_zone" "cluster_domain": > 6: resource "aws_route53_zone" "cluster_domain" { -> +> > I can't see to find a way to create a PHZ in this account when using a shared VPC from another account. I can't use a route53 vpc zone authorization as that require the zone to be created. I can't leave off the VPC, because it's required for a PHZ and I can't create a public zone in GovCloud. -> +> > I'm stuck. AWS Response: > Greetings, -> +> > Dustin here from AWS Networking Support. I'll be working with you regarding your hosted zone association today. -> +> > From what I'm understanding, you're wanting to associate multiple shared VPCs to a single private hosted zone that resides in this account in a GovCloud environment. The intended target VPC for creating this hosted zone is owned by a different account ID. When attempting to create a hosted zone, an InvalidVPCId error is given. If I've missed anything, please let me know. -> +> > First, an explanation of the error message InvalidVPCId: this error message most likely happened while attempting to associate a hosted zone with this VPC ID from an account that is not the account owner. [1] Since this VPC ID is owned by a different account ID than this account, triggering this error message. -> +> > If the target private hosted zone is to reside in this account, VPC association authorizations must first be set to allow for cross-account VPCs to be associated. Creating associations in this manner for shared, non-owned VPCs will require at least one VPC to be created within this account and be associated with the private hosted zone to be shared. In this case, a "dummy" VPC can be created to accomplish this solution. We have a rePost article that provides a step-by-step walkthrough of how to accomplish this [2]. This solution must be performed either with the AWS CLI or AWS CloudShell. -> +> > I tested this solution using CloudShell with accounts in my private environment with successful results by using the following commands: -> +> > Create command from the hosted zone account in step 5: > aws route53 create-vpc-association-authorization --hosted-zone-id --vpc VPCRegion=,VPCId= --region us-east-1 -> +> > Associate command from the different account in step 7: > aws route53 associate-vpc-with-hosted-zone --hosted-zone-id --vpc VPCRegion=,VPCId= --region us-east-1 -> +> > It is recommended to follow step 8 in the rePost article to prevent from recreating the same association at a later date. It is also at this step where the "dummy" VPC that was created in the hosted zone account can be disassociated from the hosted zone and deleted entirely. -> +> > A few considerations to take with this solution: -> +> > - Each VPC to be associated with this hosted zone that isn't owned by this account will need its own authorization request [3]. > - The private hosted zone must already exist before VPC association can be authorized. > - If using CloudShell, launching an instance in either VPC is not necessary for following the rePost article. > - For the 'create-vpc-association-authorization' and 'associate-vpc-with-hosted-zone' commands, the trailing "--region us-east-1" remains the same as this references where the hosted zone information is stored by default. > - If a dummy VPC was used in the hosted zone account and later deleted, a new VPC will need to be created if any new changes are to be made to hosted zone VPC associations. -> +> > I hope this information has helped you build your hosted zone solution. If you have any further questions or feedback, please feel free to reach back out to me and I'll be more than happy to continue working with you. Have a great day! -> -> -> [1] https://docs.aws.amazon.com/Route53/latest/APIReference/API_CreateVPCAssociationAuthorization.html -> [2] https://repost.aws/knowledge-center/route53-private-hosted-zone -> [3] https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zone-private-associate-vpcs-different-accounts.html -> +> +> +> [1] https://docs.aws.amazon.com/Route53/latest/APIReference/API_CreateVPCAssociationAuthorization.html +> [2] https://repost.aws/knowledge-center/route53-private-hosted-zone +> [3] https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zone-private-associate-vpcs-different-accounts.html +> > We value your feedback. Please share your experience by rating this and other correspondences in the AWS Support Center. You can rate a correspondence by selecting the stars in the top right corner of the correspondence. -> +> > Best regards, > Dustin M. > Amazon Web Services @@ -152,4 +152,4 @@ git add . git commit -m'add dummy vpc0' git push # add run.apply*log to PR, create PR -``` +``` diff --git a/local-app/aws-account-setup/ansible/roles/inf-vpc/files/vpc0/tf-run.data b/local-app/aws-account-setup/ansible/roles/inf-vpc/files/vpc0/tf-run.data index 0e47e9ae..f8b72342 100644 --- a/local-app/aws-account-setup/ansible/roles/inf-vpc/files/vpc0/tf-run.data +++ b/local-app/aws-account-setup/ansible/roles/inf-vpc/files/vpc0/tf-run.data @@ -14,6 +14,6 @@ LINKTOP includes.d/variables.application_tags.auto.tfvars COMMAND rm provider.ldap.* COMMAND tf-init -upgrade -#POLICY +#POLICY ALL COMMAND tf-directory-setup.py -l s3 diff --git a/local-app/aws-account-setup/ansible/roles/inf-vpc/tasks/main.yml b/local-app/aws-account-setup/ansible/roles/inf-vpc/tasks/main.yml index 79caba21..da3b34ab 100644 --- a/local-app/aws-account-setup/ansible/roles/inf-vpc/tasks/main.yml +++ b/local-app/aws-account-setup/ansible/roles/inf-vpc/tasks/main.yml @@ -18,7 +18,7 @@ mode: '644' src: "{{ item }}" dest: "{{ tf_top }}/vpc/{{ item }}" - with_items: + with_items: - tf-run.data - tf-run.region.data - tf-run.vpc.data @@ -140,4 +140,3 @@ - "{{ unused_regions | list }}" tags: - first-time - diff --git a/local-app/aws-account-setup/ansible/roles/inf-vpc/vars/main.yml b/local-app/aws-account-setup/ansible/roles/inf-vpc/vars/main.yml index 2efba1e5..b5ab788a 100644 --- a/local-app/aws-account-setup/ansible/roles/inf-vpc/vars/main.yml +++ b/local-app/aws-account-setup/ansible/roles/inf-vpc/vars/main.yml @@ -9,4 +9,3 @@ region_files: main_files_templates: [] region_files_templates: [] - diff --git a/local-app/aws-account-setup/ansible/roles/set-facts/tasks/debug.yml b/local-app/aws-account-setup/ansible/roles/set-facts/tasks/debug.yml index 79f66d2a..f9c4545b 100644 --- a/local-app/aws-account-setup/ansible/roles/set-facts/tasks/debug.yml +++ b/local-app/aws-account-setup/ansible/roles/set-facts/tasks/debug.yml @@ -11,7 +11,7 @@ tags: - always -- name: debug +- name: debug debug: var: tf_account_id tags: @@ -23,7 +23,7 @@ tags: - always -- name: debug +- name: debug debug: var: tf_top tags: @@ -35,7 +35,7 @@ tags: - always -- name: debug +- name: debug debug: var: these_regions tags: @@ -55,7 +55,7 @@ tags: - always -- name: debug +- name: debug debug: var: these_aliases tags: @@ -67,7 +67,7 @@ tags: - always -- name: debug +- name: debug debug: var: region_map tags: @@ -79,9 +79,8 @@ tags: - always -- name: debug +- name: debug debug: var: alias_map tags: - debug - diff --git a/local-app/aws-account-setup/ansible/roles/set-facts/tasks/main.yml b/local-app/aws-account-setup/ansible/roles/set-facts/tasks/main.yml index 28e6e897..ac81848a 100644 --- a/local-app/aws-account-setup/ansible/roles/set-facts/tasks/main.yml +++ b/local-app/aws-account-setup/ansible/roles/set-facts/tasks/main.yml @@ -83,4 +83,3 @@ when: tf_region_type == "ew" tags: - always - diff --git a/local-app/aws-account-setup/ansible/roles/setup-directories/files/init/tf-run.data b/local-app/aws-account-setup/ansible/roles/setup-directories/files/init/tf-run.data index 6c85b2a3..ef0fd6f9 100644 --- a/local-app/aws-account-setup/ansible/roles/setup-directories/files/init/tf-run.data +++ b/local-app/aws-account-setup/ansible/roles/setup-directories/files/init/tf-run.data @@ -19,7 +19,7 @@ COMMENT tf-run apply TAG finalize-init COMMENT git commit -m'finialize init/' -a -COMMENT git push +COMMENT git push STOP Once the full baseline is complete, we revisit the git-setup and add it to remote state. Requires tfstate to have been setup (post-baseline) diff --git a/local-app/aws-account-setup/ansible/roles/setup-directories/files/region.tf b/local-app/aws-account-setup/ansible/roles/setup-directories/files/region.tf index b7b1696e..f6175061 100644 --- a/local-app/aws-account-setup/ansible/roles/setup-directories/files/region.tf +++ b/local-app/aws-account-setup/ansible/roles/setup-directories/files/region.tf @@ -1,4 +1,3 @@ locals { region = var.region } - diff --git a/local-app/aws-account-setup/ansible/roles/setup-directories/tasks/submodule.yml b/local-app/aws-account-setup/ansible/roles/setup-directories/tasks/submodule.yml index 00a1edbe..f3ec97c3 100644 --- a/local-app/aws-account-setup/ansible/roles/setup-directories/tasks/submodule.yml +++ b/local-app/aws-account-setup/ansible/roles/setup-directories/tasks/submodule.yml @@ -55,7 +55,7 @@ mode: '755' path: "{{ tf_top_sub_structure }}/{{ item }}" state: directory - with_items: + with_items: - "{{ tf_credentials_dir }}" - "{{ tf_variables_dir }}" - "{{ tf_provider_config_dir }}" @@ -116,4 +116,3 @@ - "{{ region_map.values() | list }}" tags: - submodule - diff --git a/local-app/aws-account-setup/ansible/roles/setup-git-repo/files/.tf-control.tfrc b/local-app/aws-account-setup/ansible/roles/setup-git-repo/files/.tf-control.tfrc index 74254883..33c0dbe7 100644 --- a/local-app/aws-account-setup/ansible/roles/setup-git-repo/files/.tf-control.tfrc +++ b/local-app/aws-account-setup/ansible/roles/setup-git-repo/files/.tf-control.tfrc @@ -21,4 +21,3 @@ provider_installation { include = [ "*/*/*" ] } } - diff --git a/local-app/aws-account-setup/ansible/roles/setup-git-repo/files/ISSUE_TEMPLATE/bug_report.md b/local-app/aws-account-setup/ansible/roles/setup-git-repo/files/ISSUE_TEMPLATE/bug_report.md index e9b2d692..7926ac9f 100644 --- a/local-app/aws-account-setup/ansible/roles/setup-git-repo/files/ISSUE_TEMPLATE/bug_report.md +++ b/local-app/aws-account-setup/ansible/roles/setup-git-repo/files/ISSUE_TEMPLATE/bug_report.md @@ -12,9 +12,9 @@ A clear and concise description of what the bug is. **Identify Environment** - Terraform version [`terraform -version`]: - - Repository [`git remote -v show`]: + - Repository [`git remote -v show`]: - Git branch [`git branch`]: - - Current working directory [`pwd`]: + - Current working directory [`pwd`]: - Commands issued - Errors generated diff --git a/local-app/aws-account-setup/ansible/roles/setup-git-repo/files/PULL_REQUEST_TEMPLATE.md b/local-app/aws-account-setup/ansible/roles/setup-git-repo/files/PULL_REQUEST_TEMPLATE.md index c852621c..36b16f39 100644 --- a/local-app/aws-account-setup/ansible/roles/setup-git-repo/files/PULL_REQUEST_TEMPLATE.md +++ b/local-app/aws-account-setup/ansible/roles/setup-git-repo/files/PULL_REQUEST_TEMPLATE.md @@ -22,7 +22,7 @@ ## Requirements - Testing 1. Be sure you have a successful ouput from `tf-plan`, and if necessary for a multi-step configuration, `tf-plan -target=...`. - 1. Post (append) the `tf-plan` log + 1. Post (append) the `tf-plan` log 1. Commit/Push/PR before applying - Environment Checks diff --git a/local-app/aws-account-setup/ansible/roles/setup-git-repo/files/git-setup.sh.tpl b/local-app/aws-account-setup/ansible/roles/setup-git-repo/files/git-setup.sh.tpl index 69050cc0..8cee92fe 100644 --- a/local-app/aws-account-setup/ansible/roles/setup-git-repo/files/git-setup.sh.tpl +++ b/local-app/aws-account-setup/ansible/roles/setup-git-repo/files/git-setup.sh.tpl @@ -3,7 +3,7 @@ VERSION="1.0.1" export GIT_REMOTE="${git_ssh}" pushd ../.. -if [ ! -d ".git" ] +if [ ! -d ".git" ] then echo "* git setup for $GIT_REMOTE" git init . diff --git a/local-app/aws-account-setup/ansible/roles/setup-git-repo/files/submodule-TEMPLATE/.terraform-docs.yml b/local-app/aws-account-setup/ansible/roles/setup-git-repo/files/submodule-TEMPLATE/.terraform-docs.yml index 8391b9d3..5738e398 100644 --- a/local-app/aws-account-setup/ansible/roles/setup-git-repo/files/submodule-TEMPLATE/.terraform-docs.yml +++ b/local-app/aws-account-setup/ansible/roles/setup-git-repo/files/submodule-TEMPLATE/.terraform-docs.yml @@ -5,7 +5,7 @@ footer-from: "" sections: ## hide: [] - show: + show: - data-sources - header - footer @@ -15,7 +15,7 @@ sections: - providers - requirements - resources - + output: file: README.md mode: inject @@ -27,11 +27,11 @@ output: ## output-values: ## enabled: false ## from: "" -## +## ## sort: ## enabled: true ## by: name -## +## ## settings: ## anchor: true ## color: true diff --git a/local-app/aws-account-setup/ansible/roles/setup-git-repo/files/submodule-TEMPLATE/README.md b/local-app/aws-account-setup/ansible/roles/setup-git-repo/files/submodule-TEMPLATE/README.md index d3064bb6..1a557a5d 100644 --- a/local-app/aws-account-setup/ansible/roles/setup-git-repo/files/submodule-TEMPLATE/README.md +++ b/local-app/aws-account-setup/ansible/roles/setup-git-repo/files/submodule-TEMPLATE/README.md @@ -186,4 +186,3 @@ No modules. | [git\_url\_https](#output\_git\_url\_https) | Github repository URL for HTTPS | | [git\_url\_ssh](#output\_git\_url\_ssh) | Github repository URL for SSH | - diff --git a/local-app/aws-account-setup/ansible/roles/setup-git-repo/files/submodule-TEMPLATE/submodule.tf b/local-app/aws-account-setup/ansible/roles/setup-git-repo/files/submodule-TEMPLATE/submodule.tf index 78497936..c80c3001 100644 --- a/local-app/aws-account-setup/ansible/roles/setup-git-repo/files/submodule-TEMPLATE/submodule.tf +++ b/local-app/aws-account-setup/ansible/roles/setup-git-repo/files/submodule-TEMPLATE/submodule.tf @@ -87,15 +87,15 @@ resource "github_team" "app" { # for_each = toset(local.app_write_team) # name = each.key # description = format("Application: %v",each.key) -# privacy = "closed" +# privacy = "closed" # create_default_maintainer = false # } -# +# # resource "github_team" "app_read" { # for_each = toset(local.app_read_team) # name = each.key # description = format("Application: %v",each.key) -# privacy = "closed" +# privacy = "closed" # create_default_maintainer = false # } @@ -168,9 +168,8 @@ output "git_url_ssh" { ## https://github.com/integrations/terraform-provider-github/issues/716 ## ## github_repository.main: Creating... -## +## ## Error: DELETE https://github.e.it.census.gov/api/v3/repos/terraform/252903981224/vulnerability-alerts: 404 Not Found [] -## +## ## on INF.repo-setup.tf line 20, in resource "github_repository" "main": ## 20: resource "github_repository" "main" { - diff --git a/local-app/aws-account-setup/ansible/roles/setup-git-repo/files/submodules.settings.auto.tfvars b/local-app/aws-account-setup/ansible/roles/setup-git-repo/files/submodules.settings.auto.tfvars index b208c697..6ad9742a 100644 --- a/local-app/aws-account-setup/ansible/roles/setup-git-repo/files/submodules.settings.auto.tfvars +++ b/local-app/aws-account-setup/ansible/roles/setup-git-repo/files/submodules.settings.auto.tfvars @@ -4,4 +4,3 @@ app_team_members = { "read" : [], "write" : [], } - diff --git a/local-app/aws-account-setup/ansible/roles/setup-git-repo/tasks/main.yml b/local-app/aws-account-setup/ansible/roles/setup-git-repo/tasks/main.yml index 480fc67e..c17cc882 100644 --- a/local-app/aws-account-setup/ansible/roles/setup-git-repo/tasks/main.yml +++ b/local-app/aws-account-setup/ansible/roles/setup-git-repo/tasks/main.yml @@ -169,7 +169,7 @@ #--- # credentials #--- -- name: link first region credentials +- name: link first region credentials file: state: link src: "../../{{ tf_credentials_dir }}/{{ these_regions[0] }}.{{ tf_credentials }}" @@ -180,7 +180,7 @@ #--- # variables #--- -- name: link first region variables.common.auto +- name: link first region variables.common.auto file: state: link src: "../../{{ tf_variables_dir }}/{{ these_regions[0] }}.{{ tf_variables_common_auto }}" @@ -188,7 +188,7 @@ tags: - first-time -- name: link TF State variables.tfstate.tf +- name: link TF State variables.tfstate.tf file: state: link src: "../../{{ tf_variables_dir }}/{{ tf_variables_common_tfstate }}" @@ -211,4 +211,3 @@ include_tasks: "{{ role_path }}/tasks/submodule.yml" tags: - submodule - diff --git a/local-app/aws-account-setup/ansible/roles/setup-git-repo/tasks/submodule.yml b/local-app/aws-account-setup/ansible/roles/setup-git-repo/tasks/submodule.yml index aac73835..5da58f17 100644 --- a/local-app/aws-account-setup/ansible/roles/setup-git-repo/tasks/submodule.yml +++ b/local-app/aws-account-setup/ansible/roles/setup-git-repo/tasks/submodule.yml @@ -101,11 +101,11 @@ ## with_items: "{{ tf_provider_files }}" ## tags: ## - submodule -## +## ## #--- ## # credentials ## #--- -## - name: submodule link first region credentials +## - name: submodule link first region credentials ## file: ## state: link ## src: "../../../{{ tf_credentials_dir }}/{{ these_regions[0] }}.{{ tf_credentials }}" @@ -113,7 +113,7 @@ ## tags: ## - submodule -- name: submodule link first region credentials +- name: submodule link first region credentials file: state: link src: "../{{ these_regions[0] }}.{{ tf_credentials }}" @@ -124,16 +124,16 @@ ## #--- ## # variables ## #--- -## - name: submodule link first region variables.common.auto +## - name: submodule link first region variables.common.auto ## file: ## state: link ## src: "../../../{{ tf_variables_dir }}/{{ these_regions[0] }}.{{ tf_variables_common_auto }}" ## dest: "{{ tf_top }}/{{ tf_git_setup_directory }}/{{ tf_git_setup_submodule }}/{{ these_regions[0] }}.{{ tf_variables_common_auto }}" ## tags: ## - submodule -## -## -- name: submodule link TF State variables.tfstate.tf +## +## +- name: submodule link TF State variables.tfstate.tf file: state: link src: "../{{ tf_variables_common_tfstate }}" diff --git a/local-app/aws-account-setup/ansible/roles/setup-git-repo/templates/INF.repo-setup.tf.j2 b/local-app/aws-account-setup/ansible/roles/setup-git-repo/templates/INF.repo-setup.tf.j2 index 6cf7a179..5802746a 100644 --- a/local-app/aws-account-setup/ansible/roles/setup-git-repo/templates/INF.repo-setup.tf.j2 +++ b/local-app/aws-account-setup/ansible/roles/setup-git-repo/templates/INF.repo-setup.tf.j2 @@ -188,9 +188,9 @@ output "git_url_ssh" { ## https://github.com/integrations/terraform-provider-github/issues/716 ## ## github_repository.main: Creating... -## +## ## Error: DELETE https://github.e.it.census.gov/api/v3/repos/terraform/252903981224/vulnerability-alerts: 404 Not Found [] -## +## ## on INF.repo-setup.tf line 20, in resource "github_repository" "main": ## 20: resource "github_repository" "main" { @@ -215,4 +215,3 @@ resource "local_file" "git_setup" { filename = "${path.root}/setup/git-setup.sh" file_permission = "0755" } - diff --git a/local-app/aws-account-setup/ansible/roles/setup-git-secret/files/INF.git-secret.md b/local-app/aws-account-setup/ansible/roles/setup-git-secret/files/INF.git-secret.md index 9b609b24..b554ceab 100644 --- a/local-app/aws-account-setup/ansible/roles/setup-git-secret/files/INF.git-secret.md +++ b/local-app/aws-account-setup/ansible/roles/setup-git-secret/files/INF.git-secret.md @@ -12,7 +12,7 @@ TOP=$(git rev-parse --show-toplevel) cd $TOP git-secret init ``` - + ## Get GPG key Get keys from [terraform/support/keys/gpg-public-keys](https://github.e.it.census.gov/terraform/support/tree/master/keys/gpg-public-keys) @@ -163,4 +163,3 @@ is added, they may not be in your own GPG keyring. Be sure to follow the branch/commit/push/PR/merge with `git-secret` changes. It is super important to keep the TOP/.gitsecret directory accurate, as pushing old stuff can cause access issues for others - diff --git a/local-app/aws-account-setup/ansible/roles/setup-git-secret/tasks/main.yml b/local-app/aws-account-setup/ansible/roles/setup-git-secret/tasks/main.yml index 60437f5c..9c4283ce 100644 --- a/local-app/aws-account-setup/ansible/roles/setup-git-secret/tasks/main.yml +++ b/local-app/aws-account-setup/ansible/roles/setup-git-secret/tasks/main.yml @@ -72,7 +72,7 @@ tags: - always -- name: copy git-secret GPG public keys +- name: copy git-secret GPG public keys copy: mode: '644' src: "gpg-public-keys/{{ item }}.gpg.asc" diff --git a/local-app/aws-account-setup/ansible/roles/setup-gpg/files/INF.gpg-setup.md b/local-app/aws-account-setup/ansible/roles/setup-gpg/files/INF.gpg-setup.md index 693bffd6..f76936e0 100644 --- a/local-app/aws-account-setup/ansible/roles/setup-gpg/files/INF.gpg-setup.md +++ b/local-app/aws-account-setup/ansible/roles/setup-gpg/files/INF.gpg-setup.md @@ -18,7 +18,7 @@ in the support repo. ```shell cd init/gpg-setup tf-init -tf-plan +tf-plan tf-apply ``` diff --git a/local-app/aws-account-setup/ansible/roles/setup-gpg/tasks/main.yml b/local-app/aws-account-setup/ansible/roles/setup-gpg/tasks/main.yml index 6022942c..85dcb55e 100644 --- a/local-app/aws-account-setup/ansible/roles/setup-gpg/tasks/main.yml +++ b/local-app/aws-account-setup/ansible/roles/setup-gpg/tasks/main.yml @@ -68,4 +68,3 @@ with_items: "{{ tf_gpg_setup_template_files }}" tags: - first-time - diff --git a/local-app/aws-account-setup/ansible/roles/setup-includes/files/variables.account_tags.tf b/local-app/aws-account-setup/ansible/roles/setup-includes/files/variables.account_tags.tf index 54ec8db8..546a9c79 100644 --- a/local-app/aws-account-setup/ansible/roles/setup-includes/files/variables.account_tags.tf +++ b/local-app/aws-account-setup/ansible/roles/setup-includes/files/variables.account_tags.tf @@ -3,4 +3,3 @@ variable "account_tags" { type = map(string) default = {} } - diff --git a/local-app/aws-account-setup/ansible/roles/setup-includes/files/variables.application_tags.tf b/local-app/aws-account-setup/ansible/roles/setup-includes/files/variables.application_tags.tf index 0cfe6ae0..57d62d32 100644 --- a/local-app/aws-account-setup/ansible/roles/setup-includes/files/variables.application_tags.tf +++ b/local-app/aws-account-setup/ansible/roles/setup-includes/files/variables.application_tags.tf @@ -3,4 +3,3 @@ variable "application_tags" { type = map(string) default = {} } - diff --git a/local-app/aws-account-setup/ansible/roles/setup-includes/files/variables.infrastructure_tags.tf b/local-app/aws-account-setup/ansible/roles/setup-includes/files/variables.infrastructure_tags.tf index 8ab57985..e9b44875 100644 --- a/local-app/aws-account-setup/ansible/roles/setup-includes/files/variables.infrastructure_tags.tf +++ b/local-app/aws-account-setup/ansible/roles/setup-includes/files/variables.infrastructure_tags.tf @@ -3,4 +3,3 @@ variable "infrastructure_tags" { type = map(string) default = {} } - diff --git a/local-app/aws-account-setup/ansible/roles/setup-includes/tasks/main.yml b/local-app/aws-account-setup/ansible/roles/setup-includes/tasks/main.yml index cac76b7a..20d7f2d3 100644 --- a/local-app/aws-account-setup/ansible/roles/setup-includes/tasks/main.yml +++ b/local-app/aws-account-setup/ansible/roles/setup-includes/tasks/main.yml @@ -21,4 +21,3 @@ loop: "{{ tf_includes_tfvars_files }}" tags: - first-time - diff --git a/local-app/aws-account-setup/ansible/roles/setup-provider-configs/tasks/main.yml b/local-app/aws-account-setup/ansible/roles/setup-provider-configs/tasks/main.yml index b8f58a3f..4f5ca9a9 100644 --- a/local-app/aws-account-setup/ansible/roles/setup-provider-configs/tasks/main.yml +++ b/local-app/aws-account-setup/ansible/roles/setup-provider-configs/tasks/main.yml @@ -29,4 +29,3 @@ with_items: "{{ tf_provider_config_var_files }}" tags: - first-time - diff --git a/local-app/aws-account-setup/ansible/roles/setup-remote-state/tasks/main.yml b/local-app/aws-account-setup/ansible/roles/setup-remote-state/tasks/main.yml index 1c0704e0..534a2890 100644 --- a/local-app/aws-account-setup/ansible/roles/setup-remote-state/tasks/main.yml +++ b/local-app/aws-account-setup/ansible/roles/setup-remote-state/tasks/main.yml @@ -11,7 +11,7 @@ with_items: "{{ tf_main_directories }}" tags: - first-time - + - name: setup tf main config directories remote_state.yml template: mode: '644' diff --git a/local-app/aws-account-setup/ansible/roles/setup-remote-state/tasks/submodule.yml b/local-app/aws-account-setup/ansible/roles/setup-remote-state/tasks/submodule.yml index 49f24d27..cfae3e32 100644 --- a/local-app/aws-account-setup/ansible/roles/setup-remote-state/tasks/submodule.yml +++ b/local-app/aws-account-setup/ansible/roles/setup-remote-state/tasks/submodule.yml @@ -6,7 +6,7 @@ with_items: "{{ tf_main_directories }}" tags: - submodule - + - name: submodule setup tf main config per-region directories remote_state.yml template: mode: '644' diff --git a/local-app/aws-account-setup/ansible/test.yml b/local-app/aws-account-setup/ansible/test.yml index c758ce32..47e02442 100644 --- a/local-app/aws-account-setup/ansible/test.yml +++ b/local-app/aws-account-setup/ansible/test.yml @@ -14,7 +14,7 @@ - setup-git-repo # - setup-gpg # - setup-git-secret - + #- name: Deploy base configuration for account # hosts: all ## tags: diff --git a/local-app/aws-account-setup/ansible/variables.lab-dev-gov.txt b/local-app/aws-account-setup/ansible/variables.lab-dev-gov.txt index dc945dfc..4a3182e5 100644 --- a/local-app/aws-account-setup/ansible/variables.lab-dev-gov.txt +++ b/local-app/aws-account-setup/ansible/variables.lab-dev-gov.txt @@ -1,6 +1,6 @@ -[DEPRECATION WARNING]: [defaults]callback_whitelist option, normalizing names -to new standard, use callbacks_enabled instead. This feature will be removed -from ansible-core in version 2.15. Deprecation warnings can be disabled by +[DEPRECATION WARNING]: [defaults]callback_whitelist option, normalizing names +to new standard, use callbacks_enabled instead. This feature will be removed +from ansible-core in version 2.15. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. [WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details @@ -8,13 +8,13 @@ setting deprecation_warnings=False in ansible.cfg. PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* -Friday 10 January 2025 09:39:26 -0500 (0:00:00.062) 0:00:00.062 ******** -Friday 10 January 2025 09:39:26 -0500 (0:00:00.061) 0:00:00.061 ******** +Friday 10 January 2025 09:39:26 -0500 (0:00:00.062) 0:00:00.062 ******** +Friday 10 January 2025 09:39:26 -0500 (0:00:00.061) 0:00:00.061 ******** ok: [lab-dev-gov.cloud ansible_host=localhost] TASK [debug] ******************************************************************* -Friday 10 January 2025 09:39:28 -0500 (0:00:01.331) 0:00:01.394 ******** -Friday 10 January 2025 09:39:28 -0500 (0:00:01.331) 0:00:01.393 ******** +Friday 10 January 2025 09:39:28 -0500 (0:00:01.331) 0:00:01.394 ******** +Friday 10 January 2025 09:39:28 -0500 (0:00:01.331) 0:00:01.393 ******** [WARNING]: While constructing a mapping from /data/files/git- repos/terraform/support/local-app/aws-account- setup/ansible/inventory/group_vars/ent-ew-sectools-sa/main.yml, line 1, column @@ -472970,8 +472970,8 @@ ok: [lab-dev-gov.cloud ansible_host=localhost] => { } TASK [debug] ******************************************************************* -Friday 10 January 2025 09:39:51 -0500 (0:00:23.596) 0:00:24.990 ******** -Friday 10 January 2025 09:39:51 -0500 (0:00:23.596) 0:00:24.989 ******** +Friday 10 January 2025 09:39:51 -0500 (0:00:23.596) 0:00:24.990 ******** +Friday 10 January 2025 09:39:51 -0500 (0:00:23.596) 0:00:24.989 ******** ok: [lab-dev-gov.cloud ansible_host=localhost] => { "hostvars[inventory_hostname]": { "ansible_all_ipv4_addresses": [ @@ -477217,17 +477217,17 @@ ok: [lab-dev-gov.cloud ansible_host=localhost] => { } PLAY RECAP ********************************************************************* -lab-dev-gov.cloud ansible_host=localhost : ok=3 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 +lab-dev-gov.cloud ansible_host=localhost : ok=3 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 -Friday 10 January 2025 09:39:51 -0500 (0:00:00.331) 0:00:25.322 ******** -=============================================================================== +Friday 10 January 2025 09:39:51 -0500 (0:00:00.331) 0:00:25.322 ******** +=============================================================================== debug ------------------------------------------------------------------ 23.60s Gathering Facts --------------------------------------------------------- 1.33s debug ------------------------------------------------------------------- 0.33s -Friday 10 January 2025 09:39:51 -0500 (0:00:00.331) 0:00:25.321 ******** -=============================================================================== +Friday 10 January 2025 09:39:51 -0500 (0:00:00.331) 0:00:25.321 ******** +=============================================================================== debug ------------------------------------------------------------------ 23.93s gather_facts ------------------------------------------------------------ 1.33s -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ total ------------------------------------------------------------------ 25.26s Playbook run took 0 days, 0 hours, 0 minutes, 25 seconds diff --git a/local-app/aws-account-setup/ansible/vars.yml b/local-app/aws-account-setup/ansible/vars.yml index f8bfc3c0..2092d40a 100644 --- a/local-app/aws-account-setup/ansible/vars.yml +++ b/local-app/aws-account-setup/ansible/vars.yml @@ -16,7 +16,7 @@ ## - fail: ## msg: "You must specify a value for `hosts` variable - e.g.: ansible-playbook vars.yml -e 'hosts=localhost'" ## when: hosts is not defined -## +## ##- hosts: "{{ hosts }}" - hosts: all tasks: diff --git a/local-app/aws-sso-tools/aws-sso-login.conf b/local-app/aws-sso-tools/aws-sso-login.conf index 5922a191..50c2d17b 100644 --- a/local-app/aws-sso-tools/aws-sso-login.conf +++ b/local-app/aws-sso-tools/aws-sso-login.conf @@ -15,4 +15,3 @@ export HTTP_PROXY=http://proxy.tco.census.gov:3128 export HTTPS_PROXY=http://proxy.tco.census.gov:3128 export NO_PROXY=".census.gov,169.254.169.254,148.129.0.0/16,10.0.0.0/8,172.16.0/12" - diff --git a/local-app/aws-sso-tools/get-all-support-enterprise.ew.sh b/local-app/aws-sso-tools/get-all-support-enterprise.ew.sh index 97522b4c..83c0969e 100755 --- a/local-app/aws-sso-tools/get-all-support-enterprise.ew.sh +++ b/local-app/aws-sso-tools/get-all-support-enterprise.ew.sh @@ -5,5 +5,5 @@ environment="ew" for f in $(aws configure list-profiles | grep ^EW |grep administratoraccess) do - echo -n "$f "; aws --profile $f $WHAT describe-services --query 'services[0].code' + echo -n "$f "; aws --profile $f $WHAT describe-services --query 'services[0].code' done |& tee all-$WHAT.$environment.dat diff --git a/local-app/aws-sso-tools/get-all-vpc-endpoints.sh b/local-app/aws-sso-tools/get-all-vpc-endpoints.sh index 3f7513fe..bb727141 100755 --- a/local-app/aws-sso-tools/get-all-vpc-endpoints.sh +++ b/local-app/aws-sso-tools/get-all-vpc-endpoints.sh @@ -12,4 +12,3 @@ do # aws --profile $f --region $REGION ec2 describe-vpc-endpoints --query 'VpcEndpoints[*].{id:VpcEndpointId,type:VpcEndpointType,service:ServiceName,vpc:VpcId,zinterfaces:NetworkInterfaceIds[]}' --output text | sed -e "s/^/$f $REGION /" aws --profile $f --region $REGION ec2 describe-vpc-endpoints --query 'VpcEndpoints[*].{id:VpcEndpointId,type:VpcEndpointType,service:ServiceName,vpc:VpcId}' --output text | sed -e "s/^/$f $REGION /" done |& tee all-vpc-endpoints.txt - diff --git a/local-app/aws-sso-tools/get-all-vpc-flowlogs.sh b/local-app/aws-sso-tools/get-all-vpc-flowlogs.sh index b7de252b..447b7e16 100755 --- a/local-app/aws-sso-tools/get-all-vpc-flowlogs.sh +++ b/local-app/aws-sso-tools/get-all-vpc-flowlogs.sh @@ -9,4 +9,3 @@ do sed -e "s/^/$f $REGION /" done done |& tee all-vpc-flowlogs.dat - diff --git a/local-app/aws-sso-tools/profile-formatter.py b/local-app/aws-sso-tools/profile-formatter.py index 33c04a70..97ba6adc 100755 --- a/local-app/aws-sso-tools/profile-formatter.py +++ b/local-app/aws-sso-tools/profile-formatter.py @@ -1,11 +1,11 @@ #!/bin/env python -import sys -import re import os +import re +import sys -version='1.1.0' -DEBUG=os.getenv('AWSSSOUTIL_DEBUG',None)!=None +version = "1.1.0" +DEBUG = os.getenv("AWSSSOUTIL_DEBUG", None) != None sep = "." ( @@ -14,41 +14,44 @@ role_name, region_name, short_region_name, - region_index_str + region_index_str, ) = sys.argv[1:7] region_index = int(region_index_str) region_str = "" if region_index == 0 else sep + short_region_name -#print(account_name + sep + role_name + region_str) +# print(account_name + sep + role_name + region_str) -name=account_name.lower() -n_name=name +name = account_name.lower() +n_name = name if DEBUG: - print(f'IN account_name={account_name},account_id={account_id},role_name={role_name},region_name={region_name},short_region_name={short_region_name},region_index_str={region_index_str}',file=sys.stderr) + print( + f"IN account_name={account_name},account_id={account_id},role_name={role_name},region_name={region_name},short_region_name={short_region_name},region_index_str={region_index_str}", + file=sys.stderr, + ) -if re.search(r'us-gov',region_name): - if name=='census-esf': - n_name='do2-govcloud' - elif re.search(r'-ew$',name): - n_name=name.replace('-ew','-gov') - elif re.search(r'^(lab|ent)-ew-',name): - n_name=name.replace('-ew-','-gov-') - elif re.search(r'^multiaccount',name): - n_name=name.replace('multiaccount','do3-ma')+'-gov' +if re.search(r"us-gov", region_name): + if name == "census-esf": + n_name = "do2-govcloud" + elif re.search(r"-ew$", name): + n_name = name.replace("-ew", "-gov") + elif re.search(r"^(lab|ent)-ew-", name): + n_name = name.replace("-ew-", "-gov-") + elif re.search(r"^multiaccount", name): + n_name = name.replace("multiaccount", "do3-ma") + "-gov" else: - if name=='census-esf': - n_name='do1-ew' - elif re.search(r'^multiaccount',name): - n_name=name.replace('multiaccount','do3-ma')+'-ew' - elif re.search(r'^census.*esf2',name): - n_name='do2-cat' - elif re.search(r'^census.*esf3',name): - n_name='do2-prod' - elif name=='us-census-master': - n_name='censusaws' + if name == "census-esf": + n_name = "do1-ew" + elif re.search(r"^multiaccount", name): + n_name = name.replace("multiaccount", "do3-ma") + "-ew" + elif re.search(r"^census.*esf2", name): + n_name = "do2-cat" + elif re.search(r"^census.*esf3", name): + n_name = "do2-prod" + elif name == "us-census-master": + n_name = "censusaws" -role_name=role_name.lower() +role_name = role_name.lower() # ignore permissionset with terraform in it, now use create-terraform-profile.sh post run ## if re.search(r'terraform',role_name): ## is_terraform=True @@ -56,19 +59,22 @@ ## else: ## is_terraform=False ## new_name=f'{account_id}-{n_name}{sep}{role_name}' -new_name=f'{account_id}-{n_name}{sep}{role_name}' +new_name = f"{account_id}-{n_name}{sep}{role_name}" print(new_name) if DEBUG: - print(f'OUT n_name={n_name} new_name={new_name} is_terraform={is_terraform}', file=sys.stderr) - print('', file=sys.stderr) + print( + f"OUT n_name={n_name} new_name={new_name} is_terraform={is_terraform}", + file=sys.stderr, + ) + print("", file=sys.stderr) sys.exit(0) - + # profile name process # Finally, if you want total control over the generated profile names, you can provide a shell command with --profile-name-process and it will be executed with the following positional arguments: -# +# # Account name # Account id # Role name @@ -77,9 +83,9 @@ # Region index (zero-based index of what position the region is in the provided list of regions) # Number of regions # This must output a profile name to stdout and return an exit code of 0. If the output is the string SKIP, no profile will be created for that configuration. -# +# # The default formatting is roughly equivalent to the following code: -# +# # import sys # sep = "." # ( diff --git a/local-app/aws-sso-tools/refresh-profile.sh b/local-app/aws-sso-tools/refresh-profile.sh index 5295c80c..427457d6 100755 --- a/local-app/aws-sso-tools/refresh-profile.sh +++ b/local-app/aws-sso-tools/refresh-profile.sh @@ -43,9 +43,9 @@ then # --sso-region us-gov-east-1 --region us-gov-east-1 --account-name-case lower --role-name-case lower \ # --region-style long --components "{account_id}-{account_name}.{role_name}" --config-default output=json \ # |& tee refresh.gov.$(date +%s).log -# $SSOUTIL configure populate --sso-start-url https://start.us-gov-home.awsapps.com/directory/d-c2673e7ee9 +# $SSOUTIL configure populate --sso-start-url https://start.us-gov-home.awsapps.com/directory/d-c2673e7ee9 $SSOUTIL configure populate --sso-start-url ${sso_urls[$WHICH]} \ - --sso-region us-gov-east-1 --region us-gov-east-1 --profile-name-process $FORMATTER + --sso-region us-gov-east-1 --region us-gov-east-1 --profile-name-process $FORMATTER # |& tee refresh.gov.$(date +%s).log echo "# gov.admin.profiles.txt" aws configure list-profiles | grep administratoraccess | grep -v lab | grep -vE "(-ew)|(ew-)|us-census-master|do2-cat|do2-prod" @@ -58,20 +58,20 @@ then # --sso-region us-east-1 --region us-east-1 --account-name-case lower --role-name-case lower \ # --region-style long --components "EW-{account_id}-{account_name}.{role_name}" --config-default output=json \ # |& tee refresh.ew.$(date +%s).log -# $SSOUTIL configure populate --sso-start-url https://d-9067a863a6.awsapps.com/start +# $SSOUTIL configure populate --sso-start-url https://d-9067a863a6.awsapps.com/start $SSOUTIL configure populate --sso-start-url ${sso_urls[$WHICH]} \ - --sso-region us-east-1 --region us-east-1 --profile-name-process $FORMATTER + --sso-region us-east-1 --region us-east-1 --profile-name-process $FORMATTER # |& tee refresh.ew.$(date +%s).log echo "# ew.admin.profiles.txt" - aws configure list-profiles | grep administratoraccess | grep -E "(-ew)|(ew-)|(do2-prod)|(do2-cat)|(us-census-master)" + aws configure list-profiles | grep administratoraccess | grep -E "(-ew)|(ew-)|(do2-prod)|(do2-cat)|(us-census-master)" fi if [ $WHICH == "lab-gov" ] then echo "* refreshing profiles for Lab GovCloud" -# $SSOUTIL configure populate --sso-start-url https://start.us-gov-home.awsapps.com/directory/d-c2672d0b4e +# $SSOUTIL configure populate --sso-start-url https://start.us-gov-home.awsapps.com/directory/d-c2672d0b4e $SSOUTIL configure populate --sso-start-url ${sso_urls[$WHICH]} \ - --sso-region us-gov-east-1 --region us-gov-east-1 --profile-name-process $FORMATTER + --sso-region us-gov-east-1 --region us-gov-east-1 --profile-name-process $FORMATTER # |& tee refresh.gov.$(date +%s).log echo "# lab-gov.admin.profiles.txt" aws configure list-profiles | grep administratoraccess | grep lab | grep -vE "(-ew)|(ew-)" diff --git a/local-app/aws-sso-tools/show-network-interfaces.sh b/local-app/aws-sso-tools/show-network-interfaces.sh index c4206109..32b0bc5e 100755 --- a/local-app/aws-sso-tools/show-network-interfaces.sh +++ b/local-app/aws-sso-tools/show-network-interfaces.sh @@ -11,4 +11,3 @@ then fi aws --profile $(get_profile) --region $(get_region) ec2 describe-network-interfaces --filter Name=vpc-id,Values=$VPC --output yaml|grep "^ PrivateIpAddress:"|awk '{print $2}'|sort - diff --git a/local-app/aws-sso-tools/sso-get-roles.sh b/local-app/aws-sso-tools/sso-get-roles.sh index 0666e58a..33d85b6f 100755 --- a/local-app/aws-sso-tools/sso-get-roles.sh +++ b/local-app/aws-sso-tools/sso-get-roles.sh @@ -40,4 +40,3 @@ fi echo "* getting roles from '$WHICH', SSO start URL '$SSO_START_URL' from redirector '$R'" aws-sso-util roles --sso-start-url $SSO_START_URL - diff --git a/local-app/aws-sso-tools/submit-cases-enterprise-support.sh b/local-app/aws-sso-tools/submit-cases-enterprise-support.sh index 562c9fe0..43067be4 100755 --- a/local-app/aws-sso-tools/submit-cases-enterprise-support.sh +++ b/local-app/aws-sso-tools/submit-cases-enterprise-support.sh @@ -33,8 +33,8 @@ done ## language="en", ## issueType="customer-service", ## ) -## -## +## +## ## create-case ## --subject ## [--service-code ] @@ -60,4 +60,4 @@ done ## [--ca-bundle ] ## [--cli-read-timeout ] ## [--cli-connect-timeout ] -## +## diff --git a/local-app/bin/CHANGELOG.md b/local-app/bin/CHANGELOG.md index 8a8d6f29..ae244889 100644 --- a/local-app/bin/CHANGELOG.md +++ b/local-app/bin/CHANGELOG.md @@ -54,4 +54,3 @@ ## tf-aws ## vpc-migrate.sh - diff --git a/local-app/bin/check-tls-files.sh b/local-app/bin/check-tls-files.sh index 1e0084f6..e95f9ffc 100755 --- a/local-app/bin/check-tls-files.sh +++ b/local-app/bin/check-tls-files.sh @@ -10,7 +10,7 @@ if [ -z "$DNSNAME" ] then echo "* missing DNSNAME" exit 1 -fi +fi DNSNAME=$(echo "$DNSNAME" | tr A-Z a-z) if [[ "$DNSNAME" =~ \.census\.gov$ ]] diff --git a/local-app/bin/create-terraform-profile.sh b/local-app/bin/create-terraform-profile.sh index fa86fc54..ec83cdb0 100755 --- a/local-app/bin/create-terraform-profile.sh +++ b/local-app/bin/create-terraform-profile.sh @@ -35,7 +35,7 @@ then exit 0 fi -# ROLE is optional. Not specifying this role will create a COPY +# ROLE is optional. Not specifying this role will create a COPY # of the source profile into the proper terraform format {id}-{alias} ROLE=$2 @@ -95,7 +95,7 @@ do # then # tfprofile="${tfprofile}-$PROFILE_SUFFIX" # fi - + exists=${allprofiles["$tfprofile"]} # echo "$SOURCE $c/${#profiles[@]} $tfprofile $exists" @@ -108,7 +108,7 @@ do else echo "" fi - elif [[ ! -z "$OVERWRITE_PROFILE" ]] + elif [[ ! -z "$OVERWRITE_PROFILE" ]] then echo -n "* overwriting terraform profile $tfprofile with source $SOURCE $c/${#profiles[@]}" if [ ! -z "$ROLE" ] @@ -123,20 +123,20 @@ do continue fi - account_id=$(aws configure --profile $SOURCE get sso_account_id) + account_id=$(aws configure --profile $SOURCE get sso_account_id) if [ -z $account_id ] then skipped=$(( skipped + 1 )) echo "* profile $SOURCE not found or not an SSO profile, skipping" continue fi - + declare -A config=() - for s in sso_start_url sso_region sso_account_name sso_account_id sso_role_name region credential_process sso_auto_populated + for s in sso_start_url sso_region sso_account_name sso_account_id sso_role_name region credential_process sso_auto_populated do config["$s"]=$(aws configure --profile $SOURCE get $s) done - + account_name=${config["account_name"]} sso_role_name=${config["sso_role_name"]} region=${config["region"]} @@ -146,7 +146,7 @@ do else partition="aws" fi - + if [ ! -z "$ROLE" ] then role_arn="arn:${partition}:iam::${account_id}:role/$ROLE" @@ -161,7 +161,7 @@ do fi echo "" >> $AWS_CONFIG_FILE - + status=0 if [ ! -z "$ROLE" ] then @@ -170,11 +170,11 @@ do echo " region=$region" echo " role_arn=$role_arn" echo " role_session_name=$role_session_name" - + aws configure set profile.${tfprofile}.source_profile $SOURCE && \ aws configure set profile.${tfprofile}.region $region && \ aws configure set profile.${tfprofile}.role_arn $role_arn && \ - aws configure set profile.${tfprofile}.role_session_name $role_session_name + aws configure set profile.${tfprofile}.role_session_name $role_session_name status=$? else echo "* creating$overwrite terraform profile $tfprofile from $SOURCE via copy" @@ -182,14 +182,14 @@ do do echo " $s=${config[$s]}" done - + for s in "${!config[@]}" do aws configure set profile.${tfprofile}.$s "${config[$s]}" status=$status && [ $? == 0 ] done fi - + if [ $status == 0 ] then created=$(( created + 1 )) diff --git a/local-app/bin/create-terraform-workspace.sh b/local-app/bin/create-terraform-workspace.sh index 4675e51a..f1f6ad61 100755 --- a/local-app/bin/create-terraform-workspace.sh +++ b/local-app/bin/create-terraform-workspace.sh @@ -35,7 +35,7 @@ then echo "* unable to detect user $USER homedirectory, exiting" exit 1 fi - + USER_GID=$(id -g $USERNAME) status=$? if [ $status != 0 ] @@ -43,7 +43,7 @@ then echo "* unable to get USER_GID for $USERNAME status=$status" exit $status fi - + if [ -d "$USERTFWORKSPACE" ] then echo "* user workspace $USERTFWORKSPACE exists, exiting" diff --git a/local-app/bin/ldapsearch b/local-app/bin/ldapsearch index 26375e8f..0615dc2c 100755 --- a/local-app/bin/ldapsearch +++ b/local-app/bin/ldapsearch @@ -68,7 +68,7 @@ while getopts ":H:D:w:xLo:b:" opt; do fi LDAP_B=$OPTARG ;; - x) + x) if [ ! -z $LDAP_DEBUG ] then echo "#> passed -x, ignoreing" diff --git a/local-app/bin/manage-remote-state.sh b/local-app/bin/manage-remote-state.sh index a08d922a..9591d2fc 100755 --- a/local-app/bin/manage-remote-state.sh +++ b/local-app/bin/manage-remote-state.sh @@ -68,7 +68,7 @@ if [[ $ACTION == "list" ]] || [[ $ACTION == "delete" ]] then echo "* listing bucket s3://$bucket/$bucket_key" aws --profile $(get_profile) --region $region s3 ls s3://$bucket/$bucket_key - + echo "* listing ddb table entry $ddbentry" aws --profile $(get_profile) --region $region dynamodb get-item --table $ddbtable --key "{\"LockID\":{\"S\":\"$ddbentry\"}}" echo "" @@ -81,7 +81,7 @@ then OFILE=$(basename $bucket_key) echo "* getting bucket s3://$bucket/$bucket_key to logs/$STAMP/$OFILE" aws --profile $(get_profile) --region $region s3 cp s3://$bucket/$bucket_key logs/$STAMP/$OFILE - + OFILE="lock-entry.json" echo "* getting ddb table entry $ddbentry to logs/$STAMP/$OFILE" aws --profile $(get_profile) --region $region dynamodb get-item --table $ddbtable --key "{\"LockID\":{\"S\":\"$ddbentry\"}}" > logs/$STAMP/$OFILE @@ -93,9 +93,9 @@ then OFILE=$(basename $bucket_key) echo "* deleting bucket s3://$bucket/$bucket_key" aws --profile $(get_profile) --region $region s3 rm s3://$bucket/$bucket_key - + echo "* deleting ddb table entry $ddbentry" - aws --profile $(get_profile) --region $region dynamodb delete-item --table $ddbtable --key "{\"LockID\":{\"S\":\"$ddbentry\"}}" + aws --profile $(get_profile) --region $region dynamodb delete-item --table $ddbtable --key "{\"LockID\":{\"S\":\"$ddbentry\"}}" echo "" fi @@ -126,7 +126,7 @@ then aws --profile $(get_profile) --region $region dynamodb get-item --table $ddbtable --key "{\"LockID\":{\"S\":\"$ddbentry\"}}" > logs/$STAMP/$OFILE echo "* updating ddb table entry $ddbentry with value $VALUE" - aws --profile $(get_profile) --region $region dynamodb put-item --table $ddbtable --item "{\"LockID\":{\"S\":\"$ddbentry\"},\"Digest\":{\"S\":\"$VALUE\"}}" + aws --profile $(get_profile) --region $region dynamodb put-item --table $ddbtable --item "{\"LockID\":{\"S\":\"$ddbentry\"},\"Digest\":{\"S\":\"$VALUE\"}}" echo "" fi diff --git a/local-app/bin/reverse-ip.py b/local-app/bin/reverse-ip.py index 6fa86e15..4f0ada85 100755 --- a/local-app/bin/reverse-ip.py +++ b/local-app/bin/reverse-ip.py @@ -1,31 +1,32 @@ #!/apps/terraform/python/bin/python # /bin/env python +import ipaddress import json import sys -import ipaddress -#from pprint import pprint + +# from pprint import pprint # assumes mask is /24 for zone for ipv4, /64 for ipv6 -r=0 -outdata={'ipv4_mask_bits':'24','ipv6_mask_bits':'64'} +r = 0 +outdata = {"ipv4_mask_bits": "24", "ipv6_mask_bits": "64"} try: - indata=json.load(sys.stdin) - ipa=indata['ip_address'] - ip=ipaddress.ip_address(ipa) - if ip.version==4: - outdata['ipv4_ptr']=ip.reverse_pointer - outdata['ip_ptr']=outdata['ipv4_ptr'] - info=outdata['ip_ptr'].split('.',1) - outdata['name']=info[0] - outdata['zone']=info[1] - elif ip.version==6: - outdata['ipv6_ptr']=ip.reverse_pointer - outdata['ip_ptr']=outdata['ipv6_ptr'] - print(json.dumps(outdata)) + indata = json.load(sys.stdin) + ipa = indata["ip_address"] + ip = ipaddress.ip_address(ipa) + if ip.version == 4: + outdata["ipv4_ptr"] = ip.reverse_pointer + outdata["ip_ptr"] = outdata["ipv4_ptr"] + info = outdata["ip_ptr"].split(".", 1) + outdata["name"] = info[0] + outdata["zone"] = info[1] + elif ip.version == 6: + outdata["ipv6_ptr"] = ip.reverse_pointer + outdata["ip_ptr"] = outdata["ipv6_ptr"] + print(json.dumps(outdata)) # pprint(outdata) except: - sys.stderr.write("unable to parse input address\n") - r=1 + sys.stderr.write("unable to parse input address\n") + r = 1 sys.exit(r) diff --git a/local-app/bin/setup-generate-rs-backend.py b/local-app/bin/setup-generate-rs-backend.py index bf2627a1..bffb0945 100755 --- a/local-app/bin/setup-generate-rs-backend.py +++ b/local-app/bin/setup-generate-rs-backend.py @@ -1,46 +1,51 @@ #!/apps/terraform/python/bin/python # /bin/env python -from jinja2 import Environment,FileSystemLoader +import hashlib import os -#import csv -#import re + +# import csv +# import re import sys +from datetime import date, datetime, time from pprint import pprint -from datetime import datetime,date,time + +import yaml from dateutil import tz from dateutil.parser import parse as date_parse -import yaml -import hashlib +from jinja2 import Environment, FileSystemLoader + def touch_file(file): - if os.path.exists(file): - os.utime(file,None) - else: - open(file,'a').close() + if os.path.exists(file): + os.utime(file, None) + else: + open(file, "a").close() + def read_yaml(file): - data={} - with open(file, 'r') as stream: - try: - data=yaml.full_load(stream) - except yaml.YAMLError as e: - print(e) - return None - return data - -if len(sys.argv)>1: - file=sys.argv[1] + data = {} + with open(file, "r") as stream: + try: + data = yaml.full_load(stream) + except yaml.YAMLError as e: + print(e) + return None + return data + + +if len(sys.argv) > 1: + file = sys.argv[1] else: - file="remote_state.yml" -data=read_yaml(file) + file = "remote_state.yml" +data = read_yaml(file) pprint(data) print("") # subnets={} # indexes={} # units={} -# +# # for s in data['subnets']: # # pprint(s) # # for c in range(0,s['count']): @@ -56,53 +61,47 @@ def read_yaml(file): # # print(n2) # subnets[name]=n2 # indexes[name]=int(s['cidr']['index_start']) -# +# # for sn in n2: # for r in s['cidr']['reserved_start']: # n3=sn[r] # # print('reserved=%s ip=%s' % (r,n3)) -# +# -#--- +# --- # main -#--- -#file_loader=FileSystemLoader('./init/template') -file_loader=FileSystemLoader('/apps/terraform/template') -env=Environment( - loader=file_loader, - trim_blocks=True, - lstrip_blocks=True -) - -if data['directory'] == "": - print("* error, 'directory' cannot be empty") - sys.exit(1) - -tf_backend=env.get_template('remote_state.backend.tf.j2') -tf_backend_data=env.get_template('remote_state.data.tf.j2') - -tf_output=tf_backend.render(data=data) -tf_filename='remote_state.backend.tf' +# --- +# file_loader=FileSystemLoader('./init/template') +file_loader = FileSystemLoader("/apps/terraform/template") +env = Environment(loader=file_loader, trim_blocks=True, lstrip_blocks=True) + +if data["directory"] == "": + print("* error, 'directory' cannot be empty") + sys.exit(1) + +tf_backend = env.get_template("remote_state.backend.tf.j2") +tf_backend_data = env.get_template("remote_state.data.tf.j2") + +tf_output = tf_backend.render(data=data) +tf_filename = "remote_state.backend.tf" print("* creating file %s" % (tf_filename)) -with open(tf_filename, 'w') as tf_file: - tf_file.write(tf_output) +with open(tf_filename, "w") as tf_file: + tf_file.write(tf_output) -d=data['directory'].replace('/','_') -data['directory_replaced']=d -tf_output=tf_backend_data.render(data=data) -tf_filename='remote_state.%s.tf.s3' % d +d = data["directory"].replace("/", "_") +data["directory_replaced"] = d +tf_output = tf_backend_data.render(data=data) +tf_filename = "remote_state.%s.tf.s3" % d print("* creating file %s" % (tf_filename)) -with open(tf_filename, 'w') as tf_file: - tf_file.write(tf_output) +with open(tf_filename, "w") as tf_file: + tf_file.write(tf_output) -tf_filename='remote_state.%s.tf.none' % d +tf_filename = "remote_state.%s.tf.none" % d print("* touching file %s" % (tf_filename)) touch_file(tf_filename) -tf_filename='remote_state.%s.tf' % d +tf_filename = "remote_state.%s.tf" % d print("* sample ln commands to run\n") -print("# ln -sf %s.none %s" % (tf_filename,tf_filename)) -print("# ln -sf %s.s3 %s" % (tf_filename,tf_filename)) - - +print("# ln -sf %s.none %s" % (tf_filename, tf_filename)) +print("# ln -sf %s.s3 %s" % (tf_filename, tf_filename)) diff --git a/local-app/bin/setup-git-secret.sh b/local-app/bin/setup-git-secret.sh index bb6def65..50f81ae6 100755 --- a/local-app/bin/setup-git-secret.sh +++ b/local-app/bin/setup-git-secret.sh @@ -21,7 +21,7 @@ then else echo "* error finding TOP of git repository (check if git clone or git init done)" exit 1 -fi +fi HOMEDIR="$TOP/.gitsecret/keys" GPGARGS="--homedir $HOMEDIR" diff --git a/local-app/bin/setup-gpg.sh b/local-app/bin/setup-gpg.sh index 5e267ce6..75115911 100755 --- a/local-app/bin/setup-gpg.sh +++ b/local-app/bin/setup-gpg.sh @@ -5,7 +5,7 @@ THIS=$(basename $0 .sh) if [[ ! -z "$1" ]] && [[ "$1" == "help" ]] then - echo "* $THIS v$VERSION" + echo "* $THIS v$VERSION" echo " default get USER from environment, EMAIL from ldap" echo " environment variables:" echo " GPG_USERNAME=name use different name for key" @@ -128,8 +128,8 @@ if [ -z "$KEYNAME" ] then KEYNAME=$GPG_USERNAME fi -KEYID=$(gpg --list-key $KEYNAME 2> /dev/null |grep ^pub|awk '{print $2}' | awk -F/ '{print $2}') -if [[ ! -z "$KEYID" ]] || [[ $? == 0 ]] +KEYID=$(gpg --list-key $KEYNAME 2> /dev/null |grep ^pub|awk '{print $2}' | awk -F/ '{print $2}') +if [[ ! -z "$KEYID" ]] || [[ $? == 0 ]] then echo "* keyid $KEYID exist for $KEYNAME, exiting" exit 1 @@ -165,18 +165,18 @@ else fi echo "# listing key $KEYNAME: gpg --list-keys $KEYNAME" -gpg --list-keys $KEYNAME +gpg --list-keys $KEYNAME -# get keyid +# get keyid echo "# get keyid" -KEYID=$(gpg --list-key $KEYNAME |grep ^pub|awk '{print $2}' | awk -F/ '{print $2}') +KEYID=$(gpg --list-key $KEYNAME |grep ^pub|awk '{print $2}' | awk -F/ '{print $2}') -# public key +# public key echo "# export public key $KEYNAME as $GPG_USERNAME.gpg.asc and $GPG_USERNAME.gpg.b64" gpg --armor --export $KEYNAME > $GPG_USERNAME.gpg.asc gpg --export $KEYNAME | base64 -w 0 > $GPG_USERNAME.gpg.b64 - -# private key + +# private key echo "# export private key $KEYNAME as $GPG_USERNAME.gpg.secret-key" gpg --export-secret-keys $KEYID > $GPG_USERNAME.gpg.secret-key diff --git a/local-app/bin/show-instances.sh b/local-app/bin/show-instances.sh index 67a9f290..82e8e2f0 100755 --- a/local-app/bin/show-instances.sh +++ b/local-app/bin/show-instances.sh @@ -11,5 +11,4 @@ else FILTER="" fi -aws --profile $(get_profile) --region $(get_region) ec2 describe-instances $FILTER --output text --query 'Reservations[*].Instances[*].{a1:InstanceId,a2:NetworkInterfaces[0].PrivateIpAddress,b1:VpcId,c1:State.Name,t1:Tags[?Key==`Name`].Value|[0]}' - +aws --profile $(get_profile) --region $(get_region) ec2 describe-instances $FILTER --output text --query 'Reservations[*].Instances[*].{a1:InstanceId,a2:NetworkInterfaces[0].PrivateIpAddress,b1:VpcId,c1:State.Name,t1:Tags[?Key==`Name`].Value|[0]}' diff --git a/local-app/bin/show-network-interfaces.sh b/local-app/bin/show-network-interfaces.sh index c4206109..32b0bc5e 100755 --- a/local-app/bin/show-network-interfaces.sh +++ b/local-app/bin/show-network-interfaces.sh @@ -11,4 +11,3 @@ then fi aws --profile $(get_profile) --region $(get_region) ec2 describe-network-interfaces --filter Name=vpc-id,Values=$VPC --output yaml|grep "^ PrivateIpAddress:"|awk '{print $2}'|sort - diff --git a/local-app/bin/show-routes.sh b/local-app/bin/show-routes.sh index ca648448..cb7263b6 100755 --- a/local-app/bin/show-routes.sh +++ b/local-app/bin/show-routes.sh @@ -40,7 +40,7 @@ declare -A private_route_tables=() declare -A public_route_tables=() declare -A default_route_tables=() c=0 -while read name route_table_id +while read name route_table_id do c=$(( c + 1 )) is_default=$(echo $name |grep -c default) @@ -141,4 +141,3 @@ do done echo "peer_ids: ${!peers[@]}" - diff --git a/local-app/bin/show-tgw-tunnel-status.sh b/local-app/bin/show-tgw-tunnel-status.sh index c09682ce..d5af6e85 100755 --- a/local-app/bin/show-tgw-tunnel-status.sh +++ b/local-app/bin/show-tgw-tunnel-status.sh @@ -53,12 +53,12 @@ then aws --profile $(get_profile) --region $(get_region) ec2 describe-transit-gateway-route-tables $FILTER \ --query 'TransitGatewayRouteTables[*].{rtid:TransitGatewayRouteTableId,tgwid:TransitGatewayId,name:Tags[?Key==`Name`].Value|[0],vrf:Tags[?Key==`boc:network_vrf`].Value|[0],vpn_vrf:Tags[?Key==`boc:vpn_network_vrf`].Value|[0]}' --output text > /tmp/XXX - + #ent-gov-prod_inter-region-us-gov-east-1 tgw-rtb-00a727a17fce0712a tgw-019211a4ce95625bb inter-region #ent-gov-prod-vpn_stage-us-gov-east-1 tgw-rtb-0129086845ac51998 tgw-019211a4ce95625bb None #ent-gov-prod_services-us-gov-east-1 tgw-rtb-017d952c5e278d15f tgw-019211a4ce95625bb services #ent-gov-prod_stage-us-gov-east-1 tgw-rtb-01a60c1e4fbc7497a tgw-019211a4ce95625bb stage - + for f in $(cat /tmp/XXX | awk '{print $2}') do echo "# $(grep $f /tmp/XXX)" @@ -67,7 +67,7 @@ then --query 'Routes[*].{cidr:DestinationCidrBlock,type:Type,tgwrid:TransitGatewayAttachments[0].ResourceId,tgwrtype:TransitGatewayAttachments[0].ResourceType}' --output text echo "" done - + #148.129.0.0/16 vpn-0072853f39b22f6e9(18.252.20.87) vpn propagated #172.16.0.0/12 vpn-0072853f39b22f6e9(18.252.20.87) vpn propagated #192.168.0.0/16 vpn-0072853f39b22f6e9(18.252.20.87) vpn propagated diff --git a/local-app/bin/tgw-route-status.sh b/local-app/bin/tgw-route-status.sh index 53356147..aa3ff7b3 100755 --- a/local-app/bin/tgw-route-status.sh +++ b/local-app/bin/tgw-route-status.sh @@ -48,4 +48,3 @@ do echo "$v $rcount" | tee -a $OFILE done done - diff --git a/local-app/bin/vpc-migrate.sh b/local-app/bin/vpc-migrate.sh index adab5509..5453e113 100755 --- a/local-app/bin/vpc-migrate.sh +++ b/local-app/bin/vpc-migrate.sh @@ -128,7 +128,7 @@ declare -A private_route_tables=() declare -A public_route_tables=() declare -A default_route_tables=() c=0 -while read name route_table_id +while read name route_table_id do echo "# route-table: id=$route_table_id name=$name" c=$(( c + 1 )) @@ -263,7 +263,7 @@ while read name service endpoint_id endpoint_type do c=$(( c + 1 )) short_name=$(echo $service| sed -e "s/^.*${AWS_REGION}.//") - if [[ $endpoint_type == "Gateway" ]] + if [[ $endpoint_type == "Gateway" ]] then echo "tf-import 'module.routing.aws_vpc_endpoint.$short_name[0]' $endpoint_id" for rt_az in "${!private_route_tables[@]}" @@ -283,7 +283,7 @@ echo "" ## pl-e0b05589 vpce-0085d3bcfaf3f9fc8 ## None vgw-093a0a43e9ad0babf ## None vgw-093a0a43e9ad0babf -## +## echo "# get security groups" aws ec2 describe-security-groups --filter Name=vpc-id,Values=$vpc_id --output json --query 'SecurityGroups[*].{GroupName:GroupName,GroupId:GroupId,Name:Tags[?Key==`Name`].Value|[0]}' --output text > $TMPDIR/security-groups.txt @@ -333,14 +333,14 @@ declare -A nacls=() declare -A nacl_ids=() echo "# get network acls" aws ec2 describe-network-acls --filter Name=vpc-id,Values=$vpc_id --query 'NetworkAcls[*].{IsDefault:IsDefault,NetworkAclId:NetworkAclId,Name:Tags[?Key==`Name`].Value|[0]}' --output text > $TMPDIR/network-acls.txt -## +## ## True default-nacl-vpc1-services acl-055cf412d8884f358 ## False nacl-vpc1-services-private acl-09e76bc304d66a685 ## False nacl-vpc1-services-public acl-0e2c0f168ed146429 -## +## c=0 -while read is_default name nacl_id +while read is_default name nacl_id do c=$(( c + 1 )) is_public=$(echo $name |grep -c public) @@ -399,7 +399,7 @@ then echo "# found $c network-acl rules for private $naclid" echo "" fi - + # NETWORK_ACL_ID:RULE_NUMBER:PROTOCOL:EGRESS ## terraform import aws_network_acl_rule.my_rule acl-7aaabd18:100:6:false diff --git a/local-app/etc/profile.d/terraform.sh b/local-app/etc/profile.d/terraform.sh index 3e9700fb..98053943 100644 --- a/local-app/etc/profile.d/terraform.sh +++ b/local-app/etc/profile.d/terraform.sh @@ -19,7 +19,7 @@ pathappend() { } pathprepend() { - for ((i=$#; i>0; i--)); + for ((i=$#; i>0; i--)); do ARG=${!i} if [ -d "$ARG" ] && [[ ":$PATH:" != *":$ARG:"* ]]; then diff --git a/local-app/git-xargs/add-gpg-keys.edl.sh b/local-app/git-xargs/add-gpg-keys.edl.sh index 95eb7213..b9db499b 100755 --- a/local-app/git-xargs/add-gpg-keys.edl.sh +++ b/local-app/git-xargs/add-gpg-keys.edl.sh @@ -37,7 +37,7 @@ then echo "* adding users and running hide" (cd init/git-secret; bash setup-git-secret.sh) status=$? -# git-secret hide +# git-secret hide echo "* whoknows" git-secret whoknows fi diff --git a/local-app/git-xargs/add-gpg-keys.sh b/local-app/git-xargs/add-gpg-keys.sh index c64b58c8..eea6a6cf 100755 --- a/local-app/git-xargs/add-gpg-keys.sh +++ b/local-app/git-xargs/add-gpg-keys.sh @@ -6,8 +6,8 @@ if [[ -d "init/git-secret" ]] && [[ -d ".gitsecret" ]] then git-secret reveal -f # cp $HOME/terraform/support/keys/gpg-public-keys/ibekw001.gpg.asc init/git-secret/ -# init/git-secret/setup-git-secret.sh - git-secret hide +# init/git-secret/setup-git-secret.sh + git-secret hide fi exit $? diff --git a/local-app/git-xargs/get-tf-version.sh b/local-app/git-xargs/get-tf-version.sh index 696dea51..d07c3d02 100755 --- a/local-app/git-xargs/get-tf-version.sh +++ b/local-app/git-xargs/get-tf-version.sh @@ -96,7 +96,7 @@ do echo "* git-skip-archive $TFDIR (global=$GLOBAL)" >> $LOGFILE continue fi - + FILE="$TFDIR/.tf-control.override" if [ -r "$FILE" ] then @@ -173,7 +173,7 @@ done if [ $NEEDSUPGRADE -eq 0 ] then - gh label delete --repo $REPO --yes tf-upgrade + gh label delete --repo $REPO --yes tf-upgrade echo "* git-repo-label-delete: tf-upgrade status $?" >> $LOGFILE fi diff --git a/local-app/git-xargs/submit.update-git-secret.edl.sh b/local-app/git-xargs/submit.update-git-secret.edl.sh index 6c647501..4db69147 100755 --- a/local-app/git-xargs/submit.update-git-secret.edl.sh +++ b/local-app/git-xargs/submit.update-git-secret.edl.sh @@ -6,4 +6,3 @@ GITHUB_HOSTNAME=${GITSYSTEM}.e.it.census.gov GITHUB_OAUTH_TOKEN=ae2f950f5d628d66 --no-skip-ci \ --repos tf-repo.edl.txt \ "$(pwd)/update-git-secret.edl.sh" |& tee update-git-secret.edl.sh.$(date +%s).log - diff --git a/local-app/infoblox/history.1716915936 b/local-app/infoblox/history.1716915936 index 4003bcbb..f9d19980 100644 --- a/local-app/infoblox/history.1716915936 +++ b/local-app/infoblox/history.1716915936 @@ -1,8 +1,8 @@ 1 2024-05-15 15:50:19 get-profile 2 2024-05-15 15:50:29 cd .. 3 2024-05-15 15:50:33 git pull - 4 2024-05-15 15:50:39 vi organization.accounts.yml - 5 2024-05-15 15:51:44 grep ditd.*prod organization.accounts.yml + 4 2024-05-15 15:50:39 vi organization.accounts.yml + 5 2024-05-15 15:51:44 grep ditd.*prod organization.accounts.yml 6 2024-05-15 15:51:59 grep ditd.*prod organization.accounts.yml |grep name 7 2024-05-15 15:52:35 cd ../west/ 8 2024-05-15 15:52:41 cd vpc1/ @@ -49,14 +49,14 @@ 49 2024-05-15 09:23:34 s tf2 50 2024-05-15 08:01:23 s main 51 2024-05-16 09:35:19 r - 52 2024-05-16 09:35:39 vi ~/.main-screenrc - 53 2024-05-16 09:35:52 grep -v \# ~/.main-screenrc - 54 2024-05-16 09:35:58 vi ~/.main-screenrc - 55 2024-05-16 09:36:18 grep -v \# ~/.main-screenrc - 56 2024-05-16 09:36:21 vi ~/.main-screenrc - 57 2024-05-16 09:36:28 grep -v \# ~/.main-screenrc - 58 2024-05-16 09:36:35 vi ~/.main-screenrc - 59 2024-05-16 09:36:39 grep -v \# ~/.main-screenrc + 52 2024-05-16 09:35:39 vi ~/.main-screenrc + 53 2024-05-16 09:35:52 grep -v \# ~/.main-screenrc + 54 2024-05-16 09:35:58 vi ~/.main-screenrc + 55 2024-05-16 09:36:18 grep -v \# ~/.main-screenrc + 56 2024-05-16 09:36:21 vi ~/.main-screenrc + 57 2024-05-16 09:36:28 grep -v \# ~/.main-screenrc + 58 2024-05-16 09:36:35 vi ~/.main-screenrc + 59 2024-05-16 09:36:39 grep -v \# ~/.main-screenrc 60 2024-05-16 09:36:47 ls ~/terraform 61 2024-05-16 09:36:51 ls ~/git-repos/ 62 2024-05-16 09:36:52 ls ~/git-repos/badra001/ @@ -71,7 +71,7 @@ 71 2024-05-16 09:37:29 git commit -m'update' . 72 2024-05-16 09:37:30 git push 73 2024-05-16 09:37:31 popd - 74 2024-05-15 15:53:37 vi organization.accounts.yml + 74 2024-05-15 15:53:37 vi organization.accounts.yml 75 2024-05-16 10:57:45 /apps//terraform//bin/ldapsearch -LLL cn=lab-gov 76 2024-05-16 10:57:47 /apps//terraform//bin/ldapsearch -LLL cn=lab-gov dn 77 2024-05-16 11:06:13 git co master @@ -135,7 +135,7 @@ 135 2024-05-16 11:12:45 ls 136 2024-05-16 11:13:02 find . -name "pki*.key" 137 2024-05-16 11:13:14 git-secret list|grep pki.*key - 138 2024-05-16 11:13:30 git grep aws-tls-certificate + 138 2024-05-16 11:13:30 git grep aws-tls-certificate 139 2024-05-16 11:13:53 git grep acmpca- 140 2024-05-16 11:14:00 git grep acmpca- > x 141 2024-05-16 11:14:01 vi x @@ -145,7 +145,7 @@ 145 2024-05-16 11:16:27 cd .. 146 2024-05-16 11:16:36 for f in $(cat vpc/xx); do git-secret remove $f; done 147 2024-05-16 11:16:56 cat vpc/xx - 148 2024-05-16 11:17:22 vi + 148 2024-05-16 11:17:22 vi 149 2024-05-16 11:17:30 cd vpc 150 2024-05-16 11:17:32 cp xx xxx 151 2024-05-16 11:17:33 vi xxx @@ -166,7 +166,7 @@ 166 2024-05-16 11:19:56 ls 167 2024-05-16 11:19:57 cd terraform 168 2024-05-16 11:19:58 ls - 169 2024-05-16 11:20:02 ./get-terraform.sh + 169 2024-05-16 11:20:02 ./get-terraform.sh 170 2024-05-16 11:20:11 cat VERSION 171 2024-05-16 11:20:14 git status . 172 2024-05-16 11:20:26 git commit -m'update to 1.8.3' . @@ -218,7 +218,7 @@ 218 2024-05-16 11:46:53 cd ../acmpca-iam-rolesanywhere/ 219 2024-05-16 11:46:53 ls 220 2024-05-16 11:46:55 cd .. - 221 2024-05-16 11:46:56 vi CHANGELOG.md + 221 2024-05-16 11:46:56 vi CHANGELOG.md 222 2024-05-16 11:47:20 aws-sso-login.sh all 223 2024-05-16 11:48:40 ls 224 2024-05-16 11:48:44 vi common//version @@ -227,7 +227,7 @@ 227 2024-05-16 11:49:03 cd acm*roles* 228 2024-05-16 11:49:48 ls 229 2024-05-16 11:49:50 cat cert - 230 2024-05-16 11:49:52 cat certificate.tf + 230 2024-05-16 11:49:52 cat certificate.tf 231 2024-05-16 11:50:11 \ 232 2024-05-16 11:50:12 ls 233 2024-05-16 11:50:19 rm -rf logs/ @@ -235,7 +235,7 @@ 235 2024-05-16 11:50:59 vi main.tf output.tf variables.tf 236 2024-05-16 11:59:18 hgrept ldaps 237 2024-05-16 11:59:22 /apps//terraform//bin/ldapsearch -LLL cn=lab-gov dn - 238 2024-05-16 11:59:28 /apps//terraform//bin/ldapsearch -LLL cn=lab-gov + 238 2024-05-16 11:59:28 /apps//terraform//bin/ldapsearch -LLL cn=lab-gov 239 2024-05-16 11:59:32 /apps//terraform//bin/ldapsearch -LLL cn=lab-gov dn 240 2024-05-16 11:59:38 hgrept ldap.*-b 241 2024-05-16 11:59:40 hgrept ldap.*b @@ -333,7 +333,7 @@ 333 2024-05-17 07:38:26 tf-aws s3 ls s3://inf-services-logstash-066921446319-uge1/test-put/ 334 2024-05-17 07:38:51 tf-aws s3 presign get-object --bucket "inf-services-logstash-066921446319-uge1" --key "test-put/" 335 2024-05-17 07:39:04 aws s3 help - 336 2024-05-17 07:39:19 aws s3 presign + 336 2024-05-17 07:39:19 aws s3 presign 337 2024-05-17 07:39:23 aws s3 presign help 338 2024-05-17 07:40:11 tf-aws s3 presign get-object "s3://inf-services-logstash-066921446319-uge1/test-put/" --expires-in 3600 --region us-gov-east-1 339 2024-05-17 07:40:23 tf-aws s3 presign "s3://inf-services-logstash-066921446319-uge1/test-put/" --expires-in 3600 --region us-gov-east-1 @@ -346,7 +346,7 @@ 346 2024-05-17 07:45:40 vi presign.py 347 2024-05-17 07:45:57 hgrept aws 348 2024-05-17 07:46:08 source /apps/anaconda/bin/activate py3 - 349 2024-05-17 07:46:14 python presign.py + 349 2024-05-17 07:46:14 python presign.py 350 2024-05-17 07:46:22 vi presign.py 351 2024-05-17 07:46:47 hgrept aws 352 2024-05-17 07:46:58 python presign.py "inf-services-logstash-066921446319-uge1" "test-put/" @@ -357,29 +357,29 @@ 357 2024-05-17 07:48:13 python presign.py "inf-services-logstash-066921446319-uge1" "test-put/test.txt" 358 2024-05-17 07:48:39 curl -v -k --request --put --upload-file test.txt 'https://inf-services-logstash-066921446319-uge1.s3.us-gov-east-1.amazonaws.com/test-put/test.txt?AWSAccessKeyId=ASIAQ7FGUHOXZZEQ4B2B&Signature=lXjsuLX91ETG5BrWKfkZYkv3OLo%3D&x-amz-security-token=IQoJb3JpZ2luX2VjEEUaDXVzLWdvdi1lYXN0LTEiRzBFAiEApX%2Bz3ZnzyaFz%2FpeUMjspsOv4jhAqqkYFJYlWr%2BVhjkkCIEV3B7UiD7cu8%2F5U4yPssUbjPsVmE8iKtPUwPqOX9bFeKp4CCNL%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEQABoMMDY2OTIxNDQ2MzE5IgzuovKQkBQwDlt2zngq8gGliyqThVnyVlaowYE9n%2BESJiez2ShUFlcKAHXKux2ScuZVt27qljon%2B7WkGSAKDPkmlHGppwyG%2BwST%2BmuXHaHH%2BHUw11ul6YozhBY6v3s3wsbJpOi2WhANp3Gkn1WvRsEEQxgamRp7WT%2BeGCCCxCY%2FL9%2B2cu5TEr7%2FA4AxLR4P0o%2BWAjAcEKcokvFdx6XLjvzGysUgotv4GkXKAhS8MPsG4JVwoztNcMLUCJgNVfbGiCsHiAjYtKtOAXcb7Vun%2BMQHdKbiEJXchUwu4Ud%2BNOdO7GC66RJKEkBzYh%2FTOFc2bHFrGQkYvwYuf9g6LxsA5%2BBI9jD%2Bh52yBjqdAaAndWzRgM7fSskB8%2Bkp1mmEcOLy1oisF0zDu%2BSwdt95u00FiCPD5R%2BxQI9kj8ZkpJcVIyMgPpMHj%2Fin1LgwNhEJ8aRqAlpU81%2BdPzH6jYF8BdsXFYiMB7pUIkZBy9gNvF380QXhEuzgeAIgkIdYVaSfMVRFLBO0eXSH8anAbLYcuLGozL1v3Hzu0uaCB2sdN36Ttyx0My%2FkL1Jjgwc%3D&Expires=1715950094' 359 2024-05-17 07:49:09 vi pr - 360 2024-05-17 07:49:12 vi presign.py + 360 2024-05-17 07:49:12 vi presign.py 361 2024-05-17 07:51:50 python presign.py "inf-services-logstash-066921446319-uge1" "test-put/" - 362 2024-05-17 07:51:53 vi presign.py + 362 2024-05-17 07:51:53 vi presign.py 363 2024-05-17 07:52:00 python presign.py "inf-services-logstash-066921446319-uge1" "test-put/" - 364 2024-05-17 07:52:05 vi presign.py + 364 2024-05-17 07:52:05 vi presign.py 365 2024-05-17 07:52:10 python presign.py "inf-services-logstash-066921446319-uge1" "test-put/" - 366 2024-05-17 07:52:12 vi presign.py + 366 2024-05-17 07:52:12 vi presign.py 367 2024-05-17 07:52:25 python presign.py "inf-services-logstash-066921446319-uge1" "test-put/" - 368 2024-05-17 07:53:13 vi presign.py + 368 2024-05-17 07:53:13 vi presign.py 369 2024-05-17 07:54:37 python presign.py "inf-services-logstash-066921446319-uge1" "test-put/" - 370 2024-05-17 07:54:44 vi presign.py + 370 2024-05-17 07:54:44 vi presign.py 371 2024-05-17 07:55:06 python presign.py "inf-services-logstash-066921446319-uge1" "test-put/" - 372 2024-05-17 07:55:15 vi presign.py + 372 2024-05-17 07:55:15 vi presign.py 373 2024-05-17 07:57:41 python presign.py "inf-services-logstash-066921446319-uge1" "test-put/" - 374 2024-05-17 07:57:42 vi presign.py + 374 2024-05-17 07:57:42 vi presign.py 375 2024-05-17 07:58:24 python presign.py "inf-services-logstash-066921446319-uge1" "test-put/" - 376 2024-05-17 07:58:27 vi presign.py + 376 2024-05-17 07:58:27 vi presign.py 377 2024-05-17 07:58:34 python presign.py "inf-services-logstash-066921446319-uge1" "test-put/" - 378 2024-05-17 07:58:41 vi presign.py + 378 2024-05-17 07:58:41 vi presign.py 379 2024-05-17 07:58:53 python presign.py "inf-services-logstash-066921446319-uge1" "test-put/" - 380 2024-05-17 07:59:16 vi presign.py + 380 2024-05-17 07:59:16 vi presign.py 381 2024-05-17 07:59:27 python presign.py "inf-services-logstash-066921446319-uge1" "test-put/" - 382 2024-05-17 07:59:32 vi presign.py + 382 2024-05-17 07:59:32 vi presign.py 383 2024-05-17 07:59:36 python presign.py "inf-services-logstash-066921446319-uge1" "test-put/" 384 2024-05-17 07:59:42 hgrept curl 385 2024-05-17 08:00:03 curl -v -k --request --put --upload-file test.txt 'https://inf-services-logstash-066921446319-uge1.s3.us-gov-east-1.amazonaws.com/?key=test-put/&AWSAccessKeyId=ASIAQ7FGUHOXUXZMCHWJ&x-amz-security-token=IQoJb3JpZ2luX2VjEEUaDXVzLWdvdi1lYXN0LTEiRzBFAiBMnydSo1+0MlMphgLFZ5iiIoFVjzMI2ACUnSQc/ei/QwIhALgITguaXos0LazI8Ny2h1TEY6yuLljkgjXGjbKt6X14Kp4CCNL//////////wEQABoMMDY2OTIxNDQ2MzE5IgwB9t7tHq+ZIV3nC0Mq8gGoLSwkMSdnXjrSY7o3ZuL/Ozdr17IOB+55LO1D/3ygWWcNEsbrnEx2hIwhgTUKkoAxDoROIlvJ0qzY8BMJpIAGsvNEiVDzzZp0hRdtsE6lE2IIWylFGkcQf8qL8J+kQ/R797/oL7R7+MomTZD8UKWaJmBx8zRHiMoSUx7UiScig5u/1rh12uhcJpdrtIxAFKyQpalofERrrywaDR0PfbRlMNxCKU/0eEFbN1oA8GaqtZw0s2GLGNuUYHkRRS4aIvp8xK5B0j/SRAaWXNm8q1A1BI5v77jAPHRQENpglG66GpW+ZJR/oPAHlrESKNJsJFvjbzCpjZ2yBjqdAXJJ4WDnWc778qBCRuT6HRXtxAGXutU5RY48HM3BjZv08hfqy8WffkeUFjiXSkMoj8/G8vzV7VbmiVGelxlGm+nZoCd00cqt/pYLH7OS7ewv0K+3YR+a6FliMfw3hLuPVF75gQpNKOr4Cpr9O4xePX6m50CsrDsYi4bcwXexXl0IwKTBAeys9WPzzIZDM0EFOAqE0oPxRJw6mnLmpe4=&policy=eyJleHBpcmF0aW9uIjogIjIwMjQtMDUtMTdUMTI6NTk6MzdaIiwgImNvbmRpdGlvbnMiOiBbeyJidWNrZXQiOiAiaW5mLXNlcnZpY2VzLWxvZ3N0YXNoLTA2NjkyMTQ0NjMxOS11Z2UxIn0sIHsia2V5IjogInRlc3QtcHV0LyJ9LCB7IngtYW16LXNlY3VyaXR5LXRva2VuIjogIklRb0piM0pwWjJsdVgyVmpFRVVhRFhWekxXZHZkaTFsWVhOMExURWlSekJGQWlCTW55ZFNvMSswTWxNcGhnTEZaNWlpSW9GVmp6TUkyQUNVblNRYy9laS9Rd0loQUxnSVRndWFYb3MwTGF6SThOeTJoMVRFWTZ5dUxsamtnalhHamJLdDZYMTRLcDRDQ05MLy8vLy8vLy8vL3dFUUFCb01NRFkyT1RJeE5EUTJNekU1SWd3Qjl0N3RIcStaSVYzbkMwTXE4Z0dvTFN3a01TZG5YanJTWTdvM1p1TC9PemRyMTdJT0IrNTVMTzFELzN5Z1dXY05Fc2JybkV4MmhJd2hnVFVLa29BeERvUk9JbHZKMHF6WThCTUpwSUFHc3ZORWlWRHp6WnAwaFJkdHNFNmxFMklJV3lsRkdrY1FmOHFMOEora1EvUjc5Ny9vTDdSNytNb21UWkQ4VUtXYUptQng4elJIaU1vU1V4N1VpU2NpZzV1LzFyaDEydWhjSnBkcnRJeEFGS3lRcGFsb2ZFUnJyeXdhRFIwUGZiUmxNTnhDS1UvMGVFRmJOMW9BOEdhcXRadzBzMkdMR051VVlIa1JSUzRhSXZwOHhLNUIwai9TUkFhV1hObThxMUExQkk1djc3akFQSFJRRU5wZ2xHNjZHcFcrWkpSL29QQUhsckVTS05Kc0pGdmpiekNwaloyeUJqcWRBWEpKNFdEbldjNzc4cUJDUnVUNkhSWHR4QUdYdXRVNVJZNDhITTNCalp2MDhoZnF5OFdmZmtlVUZqaVhTa01vajgvRzh2elY3VmJtaVZHZWx4bEdtK25ab0NkMDBjcXQvcFlMSDdPUzdld3YwSyszWVIrYTZGbGlNZnczaEx1UFZGNzVnUXBOS09yNENwcjlPNHhlUFg2bTUwQ3NyRHNZaTRiY3dYZXhYbDBJd0tUQkFleXM5V1B6eklaRE0wRUZPQXFFMG9QeFJKdzZtbkxtcGU0PSJ9XX0=&signature=WaIcXGxpxFU/i2DfuTXGLBeracE= @@ -395,78 +395,78 @@ 395 2024-05-17 08:01:36 ls 396 2024-05-17 08:01:39 vi variables.tf 397 2024-05-17 08:05:44 ls - 398 2024-05-17 08:05:52 vi main.tf + 398 2024-05-17 08:05:52 vi main.tf 399 2024-05-17 08:05:59 cd ../acmpca 400 2024-05-17 08:05:59 ls - 401 2024-05-17 08:06:01 vi main.tf + 401 2024-05-17 08:06:01 vi main.tf 402 2024-05-17 08:07:21 ls 403 2024-05-17 08:07:24 cat s 404 2024-05-17 08:07:40 tf-state show module.certificate.tls_cert_request.certificate 405 2024-05-17 08:08:57 tf-state show 'module.certificate.local_sensitive_file.certificate_cert[0]' 406 2024-05-17 08:10:01 grep X5p09 * 407 2024-05-17 08:10:06 grep -i x509 * - 408 2024-05-17 08:06:03 vi certificate.tf + 408 2024-05-17 08:06:03 vi certificate.tf 409 2024-05-17 08:10:39 ls 410 2024-05-17 08:10:44 cd ../acmpca-iam-rolesanywhere/ 411 2024-05-17 08:10:44 ls - 412 2024-05-17 08:10:55 vi main.tf + 412 2024-05-17 08:10:55 vi main.tf 413 2024-05-17 08:11:29 cd ../acmpca 414 2024-05-17 08:11:29 ls - 415 2024-05-17 08:11:31 vi certificate.tf + 415 2024-05-17 08:11:31 vi certificate.tf 416 2024-05-17 08:11:37 grep ^out *tf - 417 2024-05-17 08:11:42 vi output.tf + 417 2024-05-17 08:11:42 vi output.tf 418 2024-05-17 08:16:04 cat s 419 2024-05-17 08:16:11 tf-state show 'module.certificate.local_sensitive_file.certificate_cert_chain[0]' 420 2024-05-17 08:16:20 ls 421 2024-05-17 08:16:22 ls certs 422 2024-05-17 08:16:25 less certs/*chain* 423 2024-05-17 08:16:48 ls - 424 2024-05-17 08:17:30 vi certificate.tf + 424 2024-05-17 08:17:30 vi certificate.tf 425 2024-05-17 08:19:22 cat s - 426 2024-05-17 08:19:27 vi certificate.tf + 426 2024-05-17 08:19:27 vi certificate.tf 427 2024-05-17 08:19:53 tf-plan 428 2024-05-17 08:20:14 ls - 429 2024-05-17 08:20:16 vi role.tf - 430 2024-05-17 08:20:22 vi certificate.tf + 429 2024-05-17 08:20:16 vi role.tf + 430 2024-05-17 08:20:22 vi certificate.tf 431 2024-05-17 08:20:35 ls .terraform/modules/ 432 2024-05-17 08:20:37 ls .terraform/modules//certificate/ - 433 2024-05-17 08:21:07 ls .terraform/modules//certificate/acmpca/output.tf - 434 2024-05-17 08:21:08 cat .terraform/modules//certificate/acmpca/output.tf - 435 2024-05-17 08:21:15 vi certificate.tf + 433 2024-05-17 08:21:07 ls .terraform/modules//certificate/acmpca/output.tf + 434 2024-05-17 08:21:08 cat .terraform/modules//certificate/acmpca/output.tf + 435 2024-05-17 08:21:15 vi certificate.tf 436 2024-05-17 08:21:49 tf-plan - 437 2024-05-17 08:23:11 vi .terraform/modules/certificate//acmpca/output.tf + 437 2024-05-17 08:23:11 vi .terraform/modules/certificate//acmpca/output.tf 438 2024-05-17 08:25:06 ls - 439 2024-05-17 08:25:10 vi certificate.tf + 439 2024-05-17 08:25:10 vi certificate.tf 440 2024-05-17 08:25:24 tf-0plan 441 2024-05-17 08:25:26 tf-plan - 442 2024-05-17 08:25:55 cat .terraform/modules/certificate/acmpca/output.tf - 443 2024-05-17 08:26:07 vi output.tf - 444 2024-05-17 08:26:40 tf-output + 442 2024-05-17 08:25:55 cat .terraform/modules/certificate/acmpca/output.tf + 443 2024-05-17 08:26:07 vi output.tf + 444 2024-05-17 08:26:40 tf-output 445 2024-05-17 08:26:51 tf-apply -refresh-only=true 446 2024-05-17 08:27:12 tf-state list > s - 447 2024-05-17 08:27:29 vi output.tf - 448 2024-05-17 08:27:43 vi .terraform/modules//certificate/acmpca/output.tf + 447 2024-05-17 08:27:29 vi output.tf + 448 2024-05-17 08:27:43 vi .terraform/modules//certificate/acmpca/output.tf 449 2024-05-17 08:28:02 tf-apply -refresh-only=true 450 2024-05-17 08:28:25 tf-state list > s 451 2024-05-17 08:28:29 cat s 452 2024-05-17 08:28:38 tf-state show 'data.tls_certificate.example_content' - 453 2024-05-17 08:29:59 vi certificate.tf + 453 2024-05-17 08:29:59 vi certificate.tf 454 2024-05-17 08:30:29 tf-state less - 455 2024-05-17 08:30:36 vi certificate.tf + 455 2024-05-17 08:30:36 vi certificate.tf 456 2024-05-17 08:33:13 hgrept app 457 2024-05-17 08:33:15 tf-apply -refresh-only=true - 458 2024-05-17 08:34:17 vi certificate.tf + 458 2024-05-17 08:34:17 vi certificate.tf 459 2024-05-17 08:34:24 tf-state less 460 2024-05-17 08:34:35 ! - 461 2024-05-17 08:34:38 vi certificate.tf + 461 2024-05-17 08:34:38 vi certificate.tf 462 2024-05-17 08:35:08 hgrept app 463 2024-05-17 08:35:11 tf-apply -refresh-only=true - 464 2024-05-17 09:13:12 vi certificate.tf + 464 2024-05-17 09:13:12 vi certificate.tf 465 2024-05-17 09:16:58 tf-output less - 466 2024-05-17 09:17:02 tf-output - 467 2024-05-17 09:17:03 vi certificate.tf - 468 2024-05-17 09:17:12 tf-output - 469 2024-05-17 09:17:25 vi certificate.tf + 466 2024-05-17 09:17:02 tf-output + 467 2024-05-17 09:17:03 vi certificate.tf + 468 2024-05-17 09:17:12 tf-output + 469 2024-05-17 09:17:25 vi certificate.tf 470 2024-05-17 09:19:12 tf-fmt 471 2024-05-17 09:19:13 tf-plan 472 2024-05-17 09:39:59 host pool.ntp.census.gov @@ -537,8 +537,8 @@ 537 2024-05-17 10:34:46 kubectl --kubeconfig setup/kube.config edit ns -n monitoring 538 2024-05-17 10:34:56 kubectl --kubeconfig setup/kube.config edit ns monitoring 539 2024-05-17 10:35:53 kubectl --kubeconfig setup/kube.config get ns -n monitoring -o yaml - 540 2024-05-17 10:35:59 kubectl --kubeconfig setup/kube.config get ns -n monitoring - 541 2024-05-17 10:36:07 kubectl --kubeconfig setup/kube.config get ns monitoring + 540 2024-05-17 10:35:59 kubectl --kubeconfig setup/kube.config get ns -n monitoring + 541 2024-05-17 10:36:07 kubectl --kubeconfig setup/kube.config get ns monitoring 542 2024-05-17 10:36:11 kubectl --kubeconfig setup/kube.config get ns monitoring -o yaml 543 2024-05-17 10:36:26 kubectl --kubeconfig setup/kube.config edit ns monitoring 544 2024-05-17 10:39:54 kubectl --kubeconfig setup/kube.config get ns monitoring -o yaml @@ -548,20 +548,20 @@ 548 2024-05-17 10:42:02 kubectl --kubeconfig setup/kube.config api-resources --verbs=list --namespaced | less 549 2024-05-17 10:42:13 kubectl --kubeconfig setup/kube.config api-resources --verbs=list --namespaced | grep -i monit 550 2024-05-17 10:42:59 kubectl --kubeconfig setup/kube.config get ns monitoring -o yaml - 551 2024-05-17 10:43:02 kubectl --kubeconfig setup/kube.config get ns monitoring -o yaml + 551 2024-05-17 10:43:02 kubectl --kubeconfig setup/kube.config get ns monitoring -o yaml 552 2024-05-17 10:43:09 kubectl --kubeconfig setup/kube.config get ns monitoring -o json |jq '.spec = {"finalizers":[]}' >temp.json 553 2024-05-17 10:43:12 cat temp.json 554 2024-05-17 10:44:24 kubectl --kubeconfig setup/kube.config get ns monitoring -o json | tr -d "\n" | sed "s/\"finalizers\": \[[^]]\+\]/\"finalizers\": []/" - 555 2024-05-17 10:44:35 ca temp.json - 556 2024-05-17 10:44:38 cat temp.json - 557 2024-05-17 10:45:19 kubectl --kubeconfig setup/kube.config replace --raw /api/v1/namespaces/monitoring/finalize -f temp.json + 555 2024-05-17 10:44:35 ca temp.json + 556 2024-05-17 10:44:38 cat temp.json + 557 2024-05-17 10:45:19 kubectl --kubeconfig setup/kube.config replace --raw /api/v1/namespaces/monitoring/finalize -f temp.json 558 2024-05-17 10:45:28 hgrept ns 559 2024-05-17 10:45:32 kubectl --kubeconfig setup/kube.config get ns monitoring -o yaml 560 2024-05-17 10:46:19 ls 561 2024-05-17 10:46:21 tf-state list 562 2024-05-17 10:46:35 tf-apply -refresh-only=true 563 2024-05-17 10:47:39 cat tf-run.d - 564 2024-05-17 10:47:41 tf-plan + 564 2024-05-17 10:47:41 tf-plan 565 2024-05-17 10:48:02 cat tf-run.d 566 2024-05-17 10:48:03 ls 567 2024-05-17 10:48:06 git-secret reveal -f @@ -573,7 +573,7 @@ 573 2024-05-17 10:50:03 grep var.name * 574 2024-05-17 10:50:19 cd terraform-modules 575 2024-05-17 10:50:20 cd aws-eks/ - 576 2024-05-17 10:50:28 cd + 576 2024-05-17 10:50:28 cd 577 2024-05-17 10:50:34 cd terraform-modules/aws-eks/ 578 2024-05-17 10:50:35 cd examples/ 579 2024-05-17 10:50:35 ls @@ -587,8 +587,8 @@ 587 2024-05-17 10:51:03 ls 588 2024-05-17 10:51:04 cat *auto* 589 2024-05-17 10:51:16 grep var.name * - 590 2024-05-17 10:51:42 vi variables.datadog.auto.tfvars - 591 2024-05-17 10:51:52 vi variables.datadog.auto.tfvars + 590 2024-05-17 10:51:42 vi variables.datadog.auto.tfvars + 591 2024-05-17 10:51:52 vi variables.datadog.auto.tfvars 592 2024-05-17 10:51:57 tf-plan summary 593 2024-05-17 10:52:02 tf-apply -auto-approve 594 2024-05-17 10:58:56 git pull @@ -596,10 +596,10 @@ 596 2024-05-17 10:59:03 cd infrastructure/global/ 597 2024-05-17 10:59:04 cd direct-connect/ 598 2024-05-17 10:59:04 ls - 599 2024-05-17 10:59:06 vi dx-hq-verizon.tf + 599 2024-05-17 10:59:06 vi dx-hq-verizon.tf 600 2024-05-17 11:08:21 tf-plan 601 2024-05-17 11:09:01 git status . - 602 2024-05-17 11:09:08 git add dx-hq-verizon.tf.private + 602 2024-05-17 11:09:08 git add dx-hq-verizon.tf.private 603 2024-05-17 11:09:17 vi dx-hq-verizon.tf 604 2024-05-17 11:10:20 git status . 605 2024-05-17 11:10:25 git commit -m'prep changes' . @@ -608,38 +608,38 @@ 608 2024-05-17 13:17:30 git push 609 2024-05-17 13:18:12 ls 610 2024-05-17 13:20:06 pwd - 611 2024-05-17 13:20:22 ls -al .terraform/modules//certificate//acmpca/output.tf + 611 2024-05-17 13:20:22 ls -al .terraform/modules//certificate//acmpca/output.tf 612 2024-05-17 13:20:30 ls -al .terraform/modules//certificate//acmpca 613 2024-05-17 13:20:35 ls -al .terraform/modules//certificate//acmpca|grep "May" - 614 2024-05-17 13:20:43 cp /tmp/ .terraform/modules//certificate//acmpca/output.tf + 614 2024-05-17 13:20:43 cp /tmp/ .terraform/modules//certificate//acmpca/output.tf 615 2024-05-17 13:20:51 cp .terraform/modules//certificate//acmpca/output.tf /tmp/ 616 2024-05-17 13:20:54 vi .terraform/modules//certificate//acmpca/output.tf /tmp/ - 617 2024-05-17 13:21:08 vi certificate.tf + 617 2024-05-17 13:21:08 vi certificate.tf 618 2024-05-17 13:21:29 vi .terraform/modules//certificate//acmpca/output.tf /tmp/ - 619 2024-05-17 13:21:57 vi .terraform/modules//certificate//acmpca/certificate.tf + 619 2024-05-17 13:21:57 vi .terraform/modules//certificate//acmpca/certificate.tf 620 2024-05-17 13:24:52 cd ../.. 621 2024-05-17 13:24:52 ls 622 2024-05-17 13:24:57 cd roles-anywhere/edl-cods/ 623 2024-05-17 13:24:58 ls 624 2024-05-17 13:25:02 vi cert - 625 2024-05-17 13:25:06 vi certificate.tf - 626 2024-05-17 13:25:27 vi .terraform/modules//certificate//acmpca/certificate.tf - 627 2024-05-17 13:25:55 vi certificate.tf - 628 2024-05-17 13:26:05 vi .terraform/modules//certificate//acmpca/certificate.tf - 629 2024-05-17 13:26:10 vi certificate.tf - 630 2024-05-17 13:26:29 vi .terraform/modules//certificate//acmpca/certificate.tf - 631 2024-05-17 13:26:32 vi certificate.tf + 625 2024-05-17 13:25:06 vi certificate.tf + 626 2024-05-17 13:25:27 vi .terraform/modules//certificate//acmpca/certificate.tf + 627 2024-05-17 13:25:55 vi certificate.tf + 628 2024-05-17 13:26:05 vi .terraform/modules//certificate//acmpca/certificate.tf + 629 2024-05-17 13:26:10 vi certificate.tf + 630 2024-05-17 13:26:29 vi .terraform/modules//certificate//acmpca/certificate.tf + 631 2024-05-17 13:26:32 vi certificate.tf 632 2024-05-17 13:27:12 tf-plan - 633 2024-05-17 13:27:42 vi certificate.tf + 633 2024-05-17 13:27:42 vi certificate.tf 634 2024-05-17 13:27:52 tf-apply -refresh-only=true - 635 2024-05-17 13:29:02 vi .terraform/modules//certificate//acmpca/certificate.tf + 635 2024-05-17 13:29:02 vi .terraform/modules//certificate//acmpca/certificate.tf 636 2024-05-17 13:29:30 ls - 637 2024-05-17 13:29:34 vi certificate.tf + 637 2024-05-17 13:29:34 vi certificate.tf 638 2024-05-17 13:38:19 tf-init 639 2024-05-17 13:44:29 tf-plan - 640 2024-05-17 13:44:54 vi certificate.tf + 640 2024-05-17 13:44:54 vi certificate.tf 641 2024-05-17 13:45:16 tf-plan - 642 2024-05-17 13:45:31 vi certificate.tf + 642 2024-05-17 13:45:31 vi certificate.tf 643 2024-05-17 13:46:07 tf-plan 644 2024-05-17 13:46:45 ls .terraform/modules/ 645 2024-05-17 13:46:46 cd .terraform/modules/ @@ -652,10 +652,10 @@ 652 2024-05-17 13:47:06 ls 653 2024-05-17 13:47:08 tf-plan 654 2024-05-17 13:47:37 tf-plan summary - 655 2024-05-17 13:47:40 tf-apply + 655 2024-05-17 13:47:40 tf-apply 656 2024-05-17 13:48:48 ls -al .terraform/modules//certificate/acmpca - 657 2024-05-17 13:48:54 ls -al .terraform/modules//certificate/acmpca/output.tf - 658 2024-05-17 13:48:55 ls .terraform/modules//certificate/acmpca/output.tf + 657 2024-05-17 13:48:54 ls -al .terraform/modules//certificate/acmpca/output.tf + 658 2024-05-17 13:48:55 ls .terraform/modules//certificate/acmpca/output.tf 659 2024-05-17 13:49:01 cp .terraform/modules//certificate/acmpca/output.tf /tmp/ 660 2024-05-17 13:49:08 cp .terraform/modules//certificate/acmpca/certificate.tf /tmp/ 661 2024-05-17 13:49:12 ls @@ -664,21 +664,21 @@ 664 2024-05-17 13:49:30 git diff output.tf 665 2024-05-17 13:49:38 cp /tmp/output.tf /tmp/certificate.tf . 666 2024-05-17 13:49:40 git diff . - 667 2024-05-17 13:49:42 vi output.tf - 668 2024-05-17 13:49:51 vi certificate.tf - 669 2024-05-17 13:52:14 vi output.tf + 667 2024-05-17 13:49:42 vi output.tf + 668 2024-05-17 13:49:51 vi certificate.tf + 669 2024-05-17 13:52:14 vi output.tf 670 2024-05-17 13:54:05 ls 671 2024-05-17 13:54:07 cd .. 672 2024-05-17 13:54:07 ls - 673 2024-05-17 13:54:11 vi CHANGELOG.md + 673 2024-05-17 13:54:11 vi CHANGELOG.md 674 2024-05-17 13:54:53 ls - 675 2024-05-17 13:54:57 vi common//versions.tf - 676 2024-05-17 13:55:02 vi common//version.tf + 675 2024-05-17 13:54:57 vi common//versions.tf + 676 2024-05-17 13:55:02 vi common//version.tf 677 2024-05-17 13:55:05 ls 678 2024-05-17 13:55:07 git status . 679 2024-05-17 13:55:11 rm X - 680 2024-05-17 13:55:15 vi common//version.tf - 681 2024-05-17 13:55:19 vi CHANGELOG.md + 680 2024-05-17 13:55:15 vi common//version.tf + 681 2024-05-17 13:55:19 vi CHANGELOG.md 682 2024-05-17 13:55:38 cat XX 683 2024-05-17 13:55:39 ls 684 2024-05-17 13:55:40 git statsu . @@ -692,23 +692,23 @@ 692 2024-05-17 13:57:54 tf-plan 693 2024-05-17 13:58:52 cd acmpca 694 2024-05-17 13:58:52 ls - 695 2024-05-17 13:58:55 vi certificate.tf + 695 2024-05-17 13:58:55 vi certificate.tf 696 2024-05-17 13:59:14 git commit -mfix -a 697 2024-05-17 13:59:18 git push 698 2024-05-17 13:59:24 tf-init -upgrade 699 2024-05-17 13:59:43 tf-plan - 700 2024-05-17 14:05:18 vi certificate.tf + 700 2024-05-17 14:05:18 vi certificate.tf 701 2024-05-17 14:05:37 git commit -mfix -a 702 2024-05-17 14:05:40 git push - 703 2024-05-17 14:05:54 vi certificate.tf + 703 2024-05-17 14:05:54 vi certificate.tf 704 2024-05-17 14:06:17 git commit -mfix -a 705 2024-05-17 14:06:20 git push 706 2024-05-17 14:06:25 tf-init -upgrade 707 2024-05-17 14:06:42 tf-plan - 708 2024-05-17 14:07:45 vi certificate.tf + 708 2024-05-17 14:07:45 vi certificate.tf 709 2024-05-17 14:08:08 tf-apply -refresh-only=true 710 2024-05-17 14:08:30 tf-plan - 711 2024-05-17 14:09:08 vi certificate.tf + 711 2024-05-17 14:09:08 vi certificate.tf 712 2024-05-17 14:09:22 rm -rf .terraform/modules/ 713 2024-05-17 14:09:23 tf-init 714 2024-05-17 14:09:30 tf-plan @@ -721,33 +721,33 @@ 721 2024-05-17 14:18:40 find . -exec grep -i hspd {} \; -print 722 2024-05-17 14:18:52 find . ! -type d -exec grep -i hspd {} \; -print 723 2024-05-17 14:25:37 ls - 724 2024-05-17 14:25:42 cat output.tf + 724 2024-05-17 14:25:42 cat output.tf 725 2024-05-17 14:25:58 cd ../acmpca-iam-rolesanywhere/ 726 2024-05-17 14:25:58 ls - 727 2024-05-17 14:25:59 vi output.tf + 727 2024-05-17 14:25:59 vi output.tf 728 2024-05-17 14:26:16 ls - 729 2024-05-17 14:26:18 vi main.tf - 730 2024-05-17 14:31:11 cat output.tf - 731 2024-05-17 14:31:28 cat ../acmpca/output.tf + 729 2024-05-17 14:26:18 vi main.tf + 730 2024-05-17 14:31:11 cat output.tf + 731 2024-05-17 14:31:28 cat ../acmpca/output.tf 732 2024-05-17 14:31:34 cat ../acmpca/output.tf |grep ^out - 733 2024-05-17 14:31:51 vi output.tf + 733 2024-05-17 14:31:51 vi output.tf 734 2024-05-17 14:31:54 git diff output.tf - 735 2024-05-17 14:31:56 vi output.tf - 736 2024-05-17 14:32:30 less ../acmpca//output.tf - 737 2024-05-17 14:32:35 vi output.tf + 735 2024-05-17 14:31:56 vi output.tf + 736 2024-05-17 14:32:30 less ../acmpca//output.tf + 737 2024-05-17 14:32:35 vi output.tf 738 2024-05-17 14:32:50 cat ../acmpca/output.tf |grep ^out -A2 739 2024-05-17 14:32:55 cat ../acmpca/output.tf |grep ^out -A2 > X - 740 2024-05-17 14:32:56 vi output.tf + 740 2024-05-17 14:32:56 vi output.tf 741 2024-05-17 14:34:50 ls 742 2024-05-17 14:34:53 vi variables.tf - 743 2024-05-17 14:35:00 grpe override + 743 2024-05-17 14:35:00 grpe override 744 2024-05-17 14:35:03 grep override * 745 2024-05-17 14:35:12 grep override ../acmpca/* 746 2024-05-17 14:35:18 vi variables.tf 747 2024-05-17 14:35:30 ls - 748 2024-05-17 14:36:02 vi main.tf + 748 2024-05-17 14:36:02 vi main.tf 749 2024-05-17 14:37:26 grep account_id * - 750 2024-05-17 14:37:31 vi main.tf + 750 2024-05-17 14:37:31 vi main.tf 751 2024-05-17 14:38:27 ls 752 2024-05-17 14:38:30 vi variables.tf 753 2024-05-17 14:38:34 grep ^var * @@ -759,72 +759,72 @@ 759 2024-05-17 14:39:17 less ../acmpca/variables.tf > v 760 2024-05-17 14:39:18 ls 761 2024-05-17 14:39:21 vi variables.tf - 762 2024-05-17 14:44:06 vi main.tf + 762 2024-05-17 14:44:06 vi main.tf 763 2024-05-17 14:51:00 grep arn * 764 2024-05-17 14:51:04 ls 765 2024-05-17 14:51:08 cd ../acmpca 766 2024-05-17 14:51:09 ls 767 2024-05-17 14:51:10 grep arn * - 768 2024-05-17 14:51:15 vi output.tf + 768 2024-05-17 14:51:15 vi output.tf 769 2024-05-17 14:51:19 ls - 770 2024-05-17 14:51:28 vi certificate.tf - 771 2024-05-17 14:51:43 vi output.tf - 772 2024-05-17 14:52:48 vi data.acmpca-parameters.tf - 773 2024-05-17 14:52:58 vi certificate.tf + 770 2024-05-17 14:51:28 vi certificate.tf + 771 2024-05-17 14:51:43 vi output.tf + 772 2024-05-17 14:52:48 vi data.acmpca-parameters.tf + 773 2024-05-17 14:52:58 vi certificate.tf 774 2024-05-17 14:53:09 ls - 775 2024-05-17 14:53:15 grep ^out output.tf + 775 2024-05-17 14:53:15 grep ^out output.tf 776 2024-05-17 14:53:23 cd ../acmpca-iam-rolesanywhere/ 777 2024-05-17 14:53:23 ls - 778 2024-05-17 14:53:25 vi output.tf - 779 2024-05-17 14:54:03 less ../acmpca/output.tf - 780 2024-05-17 14:54:11 vi output.tf + 778 2024-05-17 14:53:25 vi output.tf + 779 2024-05-17 14:54:03 less ../acmpca/output.tf + 780 2024-05-17 14:54:11 vi output.tf 781 2024-05-17 14:54:19 ls 782 2024-05-17 14:54:23 rm v 783 2024-05-17 14:54:28 cat X 784 2024-05-17 14:54:29 rm X 785 2024-05-17 14:54:31 git add . 786 2024-05-17 14:54:32 ls - 787 2024-05-17 14:54:48 vi certificate.tf + 787 2024-05-17 14:54:48 vi certificate.tf 788 2024-05-17 14:55:02 ls 789 2024-05-17 14:55:11 rep ca_name * 790 2024-05-17 14:55:13 grep ca_name * 791 2024-05-17 14:55:30 grpe ytrust_ca 792 2024-05-17 14:55:36 grpe trust_ca * 793 2024-05-17 14:55:38 grep trust_ca * - 794 2024-05-17 14:55:44 vi data.ssm.tf + 794 2024-05-17 14:55:44 vi data.ssm.tf 795 2024-05-17 14:56:14 tf-console - 796 2024-05-17 14:58:16 vi certificate.tf + 796 2024-05-17 14:58:16 vi certificate.tf 797 2024-05-17 14:58:53 cd ../acmpca 798 2024-05-17 14:58:53 ls - 799 2024-05-17 14:59:03 vi data.acmpca-parameters.tf + 799 2024-05-17 14:59:03 vi data.acmpca-parameters.tf 800 2024-05-17 14:59:05 ls - 801 2024-05-17 14:59:11 vi certificate.tf + 801 2024-05-17 14:59:11 vi certificate.tf 802 2024-05-17 15:00:52 grep mode * 803 2024-05-17 15:00:56 grep ca_mode * 804 2024-05-17 15:01:14 ls - 805 2024-05-17 15:01:16 vi certificate.tf + 805 2024-05-17 15:01:16 vi certificate.tf 806 2024-05-17 15:01:18 ls - 807 2024-05-17 15:01:20 vi data.acmpca-parameters.tf + 807 2024-05-17 15:01:20 vi data.acmpca-parameters.tf 808 2024-05-17 15:01:22 ls 809 2024-05-17 15:01:26 vi data.tf 810 2024-05-17 15:01:28 ls - 811 2024-05-17 15:01:29 vi main.tf + 811 2024-05-17 15:01:29 vi main.tf 812 2024-05-17 15:01:31 ls - 813 2024-05-17 15:01:37 vi certificate.tf + 813 2024-05-17 15:01:37 vi certificate.tf 814 2024-05-17 15:01:58 grep ca_settings * - 815 2024-05-17 15:02:13 vi output.tf + 815 2024-05-17 15:02:13 vi output.tf 816 2024-05-17 15:02:45 ls 817 2024-05-17 15:02:46 cd ../ - 818 2024-05-17 15:02:58 less acmpca/output.tf + 818 2024-05-17 15:02:58 less acmpca/output.tf 819 2024-05-17 15:03:07 cd acmpca-pk 820 2024-05-17 15:03:07 ls 821 2024-05-17 15:03:12 cd acmpca-eks-cert-manager/ 822 2024-05-17 15:03:13 ls - 823 2024-05-17 15:03:14 vi output.tf + 823 2024-05-17 15:03:14 vi output.tf 824 2024-05-17 15:03:24 cd ../acmpca-iam-rolesanywhere/ - 825 2024-05-17 15:03:26 vi output.tf + 825 2024-05-17 15:03:26 vi output.tf 826 2024-05-17 15:03:52 cd ../acmpca-eks-cert-manager/ - 827 2024-05-17 15:03:54 vi output.tf + 827 2024-05-17 15:03:54 vi output.tf 828 2024-05-17 15:04:21 cd .. 829 2024-05-17 15:04:22 git status . 830 2024-05-17 15:04:25 rm XX @@ -836,20 +836,20 @@ 836 2024-05-17 15:05:51 rm -rf .terraform/modules//certificate2 837 2024-05-17 15:05:55 tf-init -upgrade 838 2024-05-17 15:06:09 vi CHANGELOG.md common//version.tf X - 839 2024-05-17 15:06:16 vi certificate.tf + 839 2024-05-17 15:06:16 vi certificate.tf 840 2024-05-17 15:06:53 tf-init 841 2024-05-17 15:07:10 tf-plan - 842 2024-05-17 15:07:18 vi certificate.tf + 842 2024-05-17 15:07:18 vi certificate.tf 843 2024-05-17 15:07:30 tf-fmt 844 2024-05-17 15:07:33 tf-init 845 2024-05-17 15:10:10 tf-plan - 846 2024-05-17 15:10:27 vi certificate.tf + 846 2024-05-17 15:10:27 vi certificate.tf 847 2024-05-17 15:10:48 tf-fmt 848 2024-05-17 15:10:50 tf-plan 849 2024-05-17 15:11:22 cd acmpca-iam-rolesanywhere/ - 850 2024-05-17 15:11:27 vi main.tf - 851 2024-05-17 15:11:47 grep ^out ../acmpca/output.tf - 852 2024-05-17 15:11:52 vi output.tf + 850 2024-05-17 15:11:27 vi main.tf + 851 2024-05-17 15:11:47 grep ^out ../acmpca/output.tf + 852 2024-05-17 15:11:52 vi output.tf 853 2024-05-17 15:12:04 git commit -mfix -a 854 2024-05-17 15:12:08 git push 855 2024-05-17 15:12:12 tf-init -ugprade @@ -859,15 +859,15 @@ 859 2024-05-17 15:12:40 ls 860 2024-05-17 15:12:45 pwd 861 2024-05-17 15:12:45 ls - 862 2024-05-17 15:12:49 vi main.tf + 862 2024-05-17 15:12:49 vi main.tf 863 2024-05-17 15:13:02 rm -rf .terraform/modules/ 864 2024-05-17 15:13:03 tf-init 865 2024-05-17 15:13:10 tf-plan 866 2024-05-17 15:14:14 tf-plan summary - 867 2024-05-17 15:14:33 tf-apply - 868 2024-05-17 15:28:37 vi certificate.tf + 867 2024-05-17 15:14:33 tf-apply + 868 2024-05-17 15:28:37 vi certificate.tf 869 2024-05-17 15:29:54 grep validity * - 870 2024-05-17 15:29:57 vi certificate.tf + 870 2024-05-17 15:29:57 vi certificate.tf 871 2024-05-17 15:31:17 tf-fjmt 872 2024-05-17 15:31:19 tf-fm 873 2024-05-17 15:31:21 tf-fmt @@ -886,9 +886,9 @@ 886 2024-05-17 15:42:00 dig in a bob.ditd-gppsys-stage.stage.geo.csp1.census.gov +short 887 2024-05-17 15:42:36 tf-destroy -target=module.certificate -target=module.certificate2 888 2024-05-17 15:42:58 ls - 889 2024-05-17 15:43:00 vi role.tf + 889 2024-05-17 15:43:00 vi role.tf 890 2024-05-17 15:43:32 ls - 891 2024-05-17 15:43:37 vi variables.auto.tfvars + 891 2024-05-17 15:43:37 vi variables.auto.tfvars 892 2024-05-17 15:43:41 ls 893 2024-05-17 15:43:45 grep certificate_cond * 894 2024-05-17 15:43:48 vi variables.tf @@ -901,10 +901,10 @@ 901 2024-05-17 15:45:15 ls 902 2024-05-17 15:45:51 pwd 903 2024-05-17 15:45:52 ls - 904 2024-05-17 15:45:55 vi role.tf + 904 2024-05-17 15:45:55 vi role.tf 905 2024-05-17 15:45:59 ls 906 2024-05-17 15:46:01 vi outputs. - 907 2024-05-17 15:46:04 vi outputs.tf + 907 2024-05-17 15:46:04 vi outputs.tf 908 2024-05-17 15:46:20 ls 909 2024-05-17 15:46:33 vi aws_config.tmpl 910 2024-05-17 15:48:47 get-profile @@ -920,49 +920,49 @@ 920 2024-05-17 15:49:56 ls 921 2024-05-17 15:50:01 cd app-csvd-morpheus/ 922 2024-05-17 15:50:01 ls - 923 2024-05-17 15:50:05 vi stackset.tf + 923 2024-05-17 15:50:05 vi stackset.tf 924 2024-05-17 15:50:16 ls 925 2024-05-17 15:50:24 ls main 926 2024-05-17 15:52:46 grep local_ * 927 2024-05-17 15:52:50 cd ../acmpca 928 2024-05-17 15:52:50 grep local_ * - 929 2024-05-17 15:50:54 vi role.tf + 929 2024-05-17 15:50:54 vi role.tf 930 2024-05-17 15:55:36 grep profile_arn * 931 2024-05-17 15:55:41 grep profile_arn * -A3 - 932 2024-05-17 15:55:48 vi role.tf + 932 2024-05-17 15:55:48 vi role.tf 933 2024-05-17 15:55:56 grpe anchro * 934 2024-05-17 15:55:59 grpe anchor * 935 2024-05-17 15:56:02 grep anchor * 936 2024-05-17 15:56:08 grep anchor * -A3 - 937 2024-05-17 15:56:18 vi role.tf + 937 2024-05-17 15:56:18 vi role.tf 938 2024-05-17 15:56:54 tf-fmt 939 2024-05-17 15:56:57 tf-plan - 940 2024-05-17 15:52:54 vi certificate.tf - 941 2024-05-17 15:58:04 vi aws_config.tpl + 940 2024-05-17 15:52:54 vi certificate.tf + 941 2024-05-17 15:58:04 vi aws_config.tpl 942 2024-05-17 15:58:27 tf-plan - 943 2024-05-17 15:58:44 vi aws_config.tpl + 943 2024-05-17 15:58:44 vi aws_config.tpl 944 2024-05-17 15:58:56 tf-plan - 945 2024-05-17 15:59:10 vi aws_config.tpl - 946 2024-05-17 15:59:13 vi role.tf + 945 2024-05-17 15:59:10 vi aws_config.tpl + 946 2024-05-17 15:59:13 vi role.tf 947 2024-05-17 15:59:27 tf-fmt 948 2024-05-17 15:59:28 tf-plan 949 2024-05-17 15:59:54 vi outputs.tf 950 2024-05-17 16:00:12 tf-plan - 951 2024-05-17 16:00:57 tf-apply + 951 2024-05-17 16:00:57 tf-apply 952 2024-05-17 16:01:29 vi outputs.tf - 953 2024-05-17 16:01:34 vi certificate.tf + 953 2024-05-17 16:01:34 vi certificate.tf 954 2024-05-17 16:01:42 tf-fmt - 955 2024-05-17 16:01:44 tf-output - 956 2024-05-17 16:01:52 tf-apply + 955 2024-05-17 16:01:44 tf-output + 956 2024-05-17 16:01:52 tf-apply 957 2024-05-17 16:02:05 cat certs/aw 958 2024-05-17 16:02:12 tf-apply -auto-approve 959 2024-05-17 16:02:31 ls certs 960 2024-05-17 16:02:39 ls .terraform/modules//certificate - 961 2024-05-17 16:02:48 ls .terraform/modules//certificate/acmpca-iam-rolesanywhere/output.tf - 962 2024-05-17 16:02:50 vi .terraform/modules//certificate/acmpca-iam-rolesanywhere/output.tf + 961 2024-05-17 16:02:48 ls .terraform/modules//certificate/acmpca-iam-rolesanywhere/output.tf + 962 2024-05-17 16:02:50 vi .terraform/modules//certificate/acmpca-iam-rolesanywhere/output.tf 963 2024-05-17 16:03:00 ls 964 2024-05-17 16:03:03 ls certs - 965 2024-05-17 16:03:09 vi certs/r-edl-cods.aws_config + 965 2024-05-17 16:03:09 vi certs/r-edl-cods.aws_config 966 2024-05-17 16:03:30 pwd 967 2024-05-17 16:03:30 ls 968 2024-05-17 16:03:49 vi ~/.aws/config @@ -975,14 +975,14 @@ 975 2024-05-17 16:07:48 aws --profile edl-core-common-gov.r-edl-cods whoami 976 2024-05-17 16:08:11 ls 977 2024-05-17 16:08:13 git status . - 978 2024-05-17 16:08:20 git add aws_config.tpl + 978 2024-05-17 16:08:20 git add aws_config.tpl 979 2024-05-17 16:08:21 ls 980 2024-05-17 16:08:23 ls cert 981 2024-05-17 16:08:26 less cert 982 2024-05-17 16:08:30 rm cert 983 2024-05-17 16:08:32 ls certs - 984 2024-05-17 16:08:53 vi .gitignore - 985 2024-05-17 16:08:57 rm .gitignore + 984 2024-05-17 16:08:53 vi .gitignore + 985 2024-05-17 16:08:57 rm .gitignore 986 2024-05-17 16:09:02 vi ../.gitignore 987 2024-05-17 16:09:06 ls 988 2024-05-17 16:09:08 git status . @@ -996,8 +996,8 @@ 996 2024-05-17 16:09:37 rm acmpca/X 997 2024-05-17 16:09:40 git status 998 2024-05-17 16:09:43 cat X - 999 2024-05-17 16:09:55 #git tag -a -m 'acmpca-iam-rolesanywhere: new submodule' - 1000 2024-05-17 16:10:00 vi acmpca-iam-rolesanywhere/main.tf + 999 2024-05-17 16:09:55 #git tag -a -m 'acmpca-iam-rolesanywhere: new submodule' + 1000 2024-05-17 16:10:00 vi acmpca-iam-rolesanywhere/main.tf 1001 2024-05-17 16:14:19 git commit -m'update docs' -a 1002 2024-05-17 16:14:24 git push 1003 2024-05-17 16:14:29 cat X @@ -1011,7 +1011,7 @@ 1011 2024-05-17 16:15:06 git add certs/ 1012 2024-05-17 16:15:06 ls 1013 2024-05-17 16:15:08 git status . - 1014 2024-05-17 16:15:12 git-secret add certs/r-edl-cods.key + 1014 2024-05-17 16:15:12 git-secret add certs/r-edl-cods.key 1015 2024-05-17 16:15:15 git-secret hide -m 1016 2024-05-17 16:15:17 git pull 1017 2024-05-17 16:15:19 git status . @@ -1022,10 +1022,10 @@ 1022 2024-05-17 16:17:31 less logs//output.20240517* 1023 2024-05-17 16:17:55 cd acmpca-iam-rolesanywhere/ 1024 2024-05-17 16:17:55 ls - 1025 2024-05-17 16:17:58 vi main.tf + 1025 2024-05-17 16:17:58 vi main.tf 1026 2024-05-17 16:18:02 vi m.tf 1027 2024-05-17 16:18:27 mv m.tf m - 1028 2024-05-17 16:18:28 vi main.tf + 1028 2024-05-17 16:18:28 vi main.tf 1029 2024-05-17 16:20:40 git commit -m'update docs' -a 1030 2024-05-17 16:20:46 git push 1031 2024-05-17 14:25:08 find . ! -type d -exec grep -i hspd {} \; -print > find-hspd.txt 2> find-hspd.txt.err @@ -1059,7 +1059,7 @@ 1059 2024-05-20 07:56:23 cd apps/ 1060 2024-05-20 07:56:23 cd share-ami/ 1061 2024-05-20 07:56:24 ls - 1062 2024-05-20 07:56:27 vi variables.auto.tfvars + 1062 2024-05-20 07:56:27 vi variables.auto.tfvars 1063 2024-05-20 07:59:03 aws-sso-login.sh all 1064 2024-05-20 08:00:24 tf-plan 1065 2024-05-20 08:00:50 tf-plan summary @@ -1107,25 +1107,25 @@ 1107 2024-05-20 09:31:31 ls 1108 2024-05-20 09:31:36 cp variables.auto.tfvars /tmp/ 1109 2024-05-20 09:31:40 ls - 1110 2024-05-20 09:31:48 vi variables.auto.tfvars + 1110 2024-05-20 09:31:48 vi variables.auto.tfvars 1111 2024-05-20 09:32:14 git diff. 1112 2024-05-20 09:32:16 git diff . 1113 2024-05-20 09:32:17 git pull 1114 2024-05-20 09:32:20 tf-plan - 1115 2024-05-20 09:32:46 cat tf-run.data + 1115 2024-05-20 09:32:46 cat tf-run.data 1116 2024-05-20 09:33:02 tf-run plan tag:all 1117 2024-05-20 09:33:19 tf-plan summary - 1118 2024-05-20 09:52:23 cat certificate.tf + 1118 2024-05-20 09:52:23 cat certificate.tf 1119 2024-05-20 09:53:20 cat certs/r-edl-cods.ce 1120 2024-05-20 09:53:24 cat certs/r-edl-cods.conf 1121 2024-05-20 09:53:26 ls certs - 1122 2024-05-20 09:53:31 cat certs/r-edl-cods.aws_config + 1122 2024-05-20 09:53:31 cat certs/r-edl-cods.aws_config 1123 2024-05-20 09:55:53 ls 1124 2024-05-20 11:13:04 cat setup//in - 1125 2024-05-20 11:13:06 cat setup//ip-addresses-full.txt + 1125 2024-05-20 11:13:06 cat setup//ip-addresses-full.txt 1126 2024-05-20 11:13:10 cd ../.. 1127 2024-05-20 09:33:40 tf-run apply tag:all - 1128 2024-05-20 11:13:11 show-instances.sh + 1128 2024-05-20 11:13:11 show-instances.sh 1129 2024-05-20 11:13:35 tf-aws ec2 start-instances --instance-id i-00a1ea2bc2b1fb9ee 1130 2024-05-20 11:13:42 cd apps/test-instances/ 1131 2024-05-20 11:13:44 hgrept ssm @@ -1173,7 +1173,7 @@ 1173 2024-05-20 13:22:01 pwd 1174 2024-05-20 13:22:03 cd .. 1175 2024-05-20 13:22:03 ls - 1176 2024-05-20 13:22:04 vi README.md + 1176 2024-05-20 13:22:04 vi README.md 1177 2024-05-20 13:23:04 cd west/ 1178 2024-05-20 13:23:13 cd vpc4/ 1179 2024-05-20 13:23:13 ls @@ -1203,15 +1203,15 @@ 1203 2024-05-20 15:16:08 ls 1204 2024-05-20 15:16:11 cd common-services/datadog-agent/ 1205 2024-05-20 15:16:12 ls - 1206 2024-05-20 15:16:41 vi datadog.values.yml + 1206 2024-05-20 15:16:41 vi datadog.values.yml 1207 2024-05-20 15:17:47 ls 1208 2024-05-20 15:17:49 ls setup/ 1209 2024-05-20 15:18:01 kubectl --kubecondfig setup/kube.config get nodes -o wide 1210 2024-05-20 15:18:05 \ 1211 2024-05-20 15:18:14 kubectl --kubeconfig setup/kube.config get nodes -o wide 1212 2024-05-20 15:23:28 wpod - 1213 2024-05-20 11:30:28 tf-run apply tag:all - 1214 2024-05-20 15:23:42 TFARGS=-auto-approve tf-run apply tag:all + 1213 2024-05-20 11:30:28 tf-run apply tag:all + 1214 2024-05-20 15:23:42 TFARGS=-auto-approve tf-run apply tag:all 1215 2024-05-20 07:42:10 s hpc3 1216 2024-05-20 07:42:20 s tfinf 1217 2024-05-20 07:42:12 s hpc4 @@ -1277,18 +1277,18 @@ 1277 2024-05-21 08:01:37 cd /data/files/dock 1278 2024-05-21 08:01:41 mkdir datadog 1279 2024-05-21 08:01:43 vi README.md - 1280 2024-05-21 08:01:59 cat README.md + 1280 2024-05-21 08:01:59 cat README.md 1281 2024-05-21 08:02:03 curl -o https://s3.amazonaws.com/dd-agent/scripts/install_script_agent7.sh 1282 2024-05-21 08:02:06 curl -O https://s3.amazonaws.com/dd-agent/scripts/install_script_agent7.sh 1283 2024-05-21 08:02:08 ls 1284 2024-05-21 08:02:13 less install - 1285 2024-05-21 08:02:16 vi README.md + 1285 2024-05-21 08:02:16 vi README.md 1286 2024-05-21 08:02:21 mv README.md datadog/ 1287 2024-05-21 08:02:23 ls ins* 1288 2024-05-21 08:02:28 mv install_script_agent7.sh datadog/ 1289 2024-05-21 08:02:29 cd datadog/ - 1290 2024-05-21 08:02:30 less install_script_agent7.sh - 1291 2024-05-21 08:03:35 vi install_script_agent7.sh + 1290 2024-05-21 08:02:30 less install_script_agent7.sh + 1291 2024-05-21 08:03:35 vi install_script_agent7.sh 1292 2024-05-21 08:33:10 pwd 1293 2024-05-21 08:33:11 ls 1294 2024-05-21 08:33:12 git status . @@ -1305,14 +1305,14 @@ 1305 2024-05-21 10:01:57 ls 1306 2024-05-21 10:02:04 cd Documents/ 1307 2024-05-21 10:02:04 ls - 1308 2024-05-21 10:10:42 vi analyzer.tf + 1308 2024-05-21 10:10:42 vi analyzer.tf 1309 2024-05-21 10:11:17 ls - 1310 2024-05-21 10:11:19 vi analyzer.tf + 1310 2024-05-21 10:11:19 vi analyzer.tf 1311 2024-05-21 10:11:21 ls 1312 2024-05-21 10:11:23 pwd 1313 2024-05-21 10:11:38 cd .. 1314 2024-05-21 10:11:42 ls - 1315 2024-05-21 10:11:52 vi variables.delegation.auto.tfvars + 1315 2024-05-21 10:11:52 vi variables.delegation.auto.tfvars 1316 2024-05-21 10:12:06 tf-state list > s 1317 2024-05-21 10:12:08 cat s 1318 2024-05-21 10:12:10 tf-init @@ -1357,7 +1357,7 @@ 1357 2024-05-21 11:04:17 tf-aws help|grep id 1358 2024-05-21 11:04:30 tf-aws identitystore help 1359 2024-05-21 11:04:50 tf-output less - 1360 2024-05-21 11:04:52 tf-output + 1360 2024-05-21 11:04:52 tf-output 1361 2024-05-21 11:05:07 tf-aws identitystore list-users 1362 2024-05-21 11:05:14 tf-aws identitystore list-users --identity-store-id "d-c2672d0b4e" 1363 2024-05-21 11:05:49 vi uers.csv @@ -1368,7 +1368,7 @@ 1368 2024-05-21 11:07:30 git commit -m'add jones910, but not enabled due to scim provisioning' . 1369 2024-05-21 11:07:33 cd permissionsets/ALL 1370 2024-05-21 11:07:33 ls - 1371 2024-05-21 11:07:39 vi inf-idms-t1.yml + 1371 2024-05-21 11:07:39 vi inf-idms-t1.yml 1372 2024-05-21 11:08:58 vi inf-idms-t1.* 1373 2024-05-21 11:09:28 tf-plan 1374 2024-05-21 11:09:58 tf-init -upgrade @@ -1386,7 +1386,7 @@ 1386 2024-05-21 11:22:33 rm logs/plan.20240521.1716304856.log 1387 2024-05-21 11:22:36 tf-plan summary 1388 2024-05-21 11:22:41 ls - 1389 2024-05-21 11:22:43 vi analyzer.tf + 1389 2024-05-21 11:22:43 vi analyzer.tf 1390 2024-05-21 11:22:51 tf-state list > s 1391 2024-05-21 11:22:54 cat s 1392 2024-05-21 11:22:54 ls @@ -1406,7 +1406,7 @@ 1406 2024-05-21 11:23:21 ls 1407 2024-05-21 11:23:22 cd access-analyzer/ 1408 2024-05-21 11:23:22 ls - 1409 2024-05-21 11:23:24 cat analyzer.tf + 1409 2024-05-21 11:23:24 cat analyzer.tf 1410 2024-05-21 11:23:24 ls 1411 2024-05-21 11:23:28 trf-init 1412 2024-05-21 11:23:30 tf-init -upgrade @@ -1418,11 +1418,11 @@ 1418 2024-05-21 11:57:56 ls 1419 2024-05-21 11:58:02 grep alu ditd*/*yml 1420 2024-05-21 11:58:24 /apps//terraform//bin//ldapsearch cn=aluko003 fullname - 1421 2024-05-21 11:59:01 cat ditd-gups/ditd-gups-sc-developer.yml + 1421 2024-05-21 11:59:01 cat ditd-gups/ditd-gups-sc-developer.yml 1422 2024-05-21 11:59:11 /apps//terraform//bin//ldapsearch cn=aluko001 fullname - 1423 2024-05-21 11:59:55 /apps//terraform//bin//ldapsearch cn=aluko001 + 1423 2024-05-21 11:59:55 /apps//terraform//bin//ldapsearch cn=aluko001 1424 2024-05-21 12:00:03 cd .. - 1425 2024-05-21 12:00:09 vi users.csv + 1425 2024-05-21 12:00:09 vi users.csv 1426 2024-05-21 12:00:54 cd permissionsets 1427 2024-05-21 12:00:56 cd sc-developer/ 1428 2024-05-21 12:00:58 cd ditd-gups/ @@ -1434,23 +1434,23 @@ 1434 2024-05-21 12:02:59 ls 1435 2024-05-21 12:03:11 tf-state list > s 1436 2024-05-21 12:03:35 cat s - 1437 2024-05-21 12:03:45 /apps//terraform//bin//ldapsearch cn=aluko001 + 1437 2024-05-21 12:03:45 /apps//terraform//bin//ldapsearch cn=aluko001 1438 2024-05-21 12:03:56 /apps//terraform//bin//ldapsearch cn=akula001 1439 2024-05-21 12:04:01 /apps//terraform//bin//ldapsearch cn=akula001 fullname 1440 2024-05-21 12:05:38 git srtatus . 1441 2024-05-21 12:05:48 git commit -m'add jones910 to inf-idms-t1' . 1442 2024-05-21 12:05:50 git status 1443 2024-05-21 12:05:52 git push - 1444 2024-05-21 12:06:12 vi role.tf + 1444 2024-05-21 12:06:12 vi role.tf 1445 2024-05-21 12:07:42 hgrept policy 1446 2024-05-21 12:07:59 tf-aws iam list-role-policy 1447 2024-05-21 12:08:07 tf-aws iam get-role-policy 1448 2024-05-21 12:08:20 tf-aws iam get-role-policy --role-name r-edl-cods --policy-name cross-account-assume - 1449 2024-05-21 12:08:24 cat role.tf + 1449 2024-05-21 12:08:24 cat role.tf 1450 2024-05-21 12:08:31 tf-aws iam get-role-policy --role-name r-edl-cods --policy-name cross-account-role 1451 2024-05-21 12:10:18 cagt variables. 1452 2024-05-21 12:10:19 ls - 1453 2024-05-21 12:10:20 cat variables.auto.tfvars + 1453 2024-05-21 12:10:20 cat variables.auto.tfvars 1454 2024-05-21 12:11:34 pwd 1455 2024-05-21 12:11:34 ls 1456 2024-05-21 12:11:36 cd common/ @@ -1470,9 +1470,9 @@ 1470 2024-05-21 12:12:12 ls 1471 2024-05-21 12:12:14 vi ro 1472 2024-05-21 12:12:15 ls - 1473 2024-05-21 12:12:21 vi r-edl-cods.tf + 1473 2024-05-21 12:12:21 vi r-edl-cods.tf 1474 2024-05-21 12:12:48 hgrept iam - 1475 2024-05-21 12:13:00 tf-aws iam get-role + 1475 2024-05-21 12:13:00 tf-aws iam get-role 1476 2024-05-21 12:13:05 tf-aws iam get-role --role-name r-edl-cods 1477 2024-05-21 12:16:44 ls 1478 2024-05-21 12:16:46 pwd @@ -1494,27 +1494,27 @@ 1494 2024-05-21 12:17:39 tf-run superclean 1495 2024-05-21 12:17:41 ls 1496 2024-05-21 12:17:48 rm s - 1497 2024-05-21 12:17:52 cat tf-run.data + 1497 2024-05-21 12:17:52 cat tf-run.data 1498 2024-05-21 12:17:58 pwd 1499 2024-05-21 12:17:58 ls 1500 2024-05-21 12:18:27 mkdir edl-u-7533453 1501 2024-05-21 12:18:35 mv edl*{tf,yml} edl-u-7533453/ 1502 2024-05-21 12:18:35 ls - 1503 2024-05-21 12:18:37 less github.tf - 1504 2024-05-21 12:18:42 rm github.tf + 1503 2024-05-21 12:18:37 less github.tf + 1504 2024-05-21 12:18:42 rm github.tf 1505 2024-05-21 12:18:44 ls 1506 2024-05-21 12:18:46 vi * - 1507 2024-05-21 12:18:53 rm edl-management.txt + 1507 2024-05-21 12:18:53 rm edl-management.txt 1508 2024-05-21 12:18:53 ls 1509 2024-05-21 12:18:54 vi * 1510 2024-05-21 12:19:05 ls - 1511 2024-05-21 12:19:08 rm README.md + 1511 2024-05-21 12:19:08 rm README.md 1512 2024-05-21 12:19:08 ls - 1513 2024-05-21 12:19:11 cat tf-run.data - 1514 2024-05-21 12:19:18 vi tf-run.data + 1513 2024-05-21 12:19:11 cat tf-run.data + 1514 2024-05-21 12:19:18 vi tf-run.data 1515 2024-05-21 12:19:43 ls 1516 2024-05-21 12:19:44 ls ../ - 1517 2024-05-21 12:19:52 less ../sc-developer/tf-run.data + 1517 2024-05-21 12:19:52 less ../sc-developer/tf-run.data 1518 2024-05-21 12:19:56 ls 1519 2024-05-21 12:19:59 cd ../sc-developer/ 1520 2024-05-21 12:19:59 ls @@ -1533,7 +1533,7 @@ 1533 2024-05-21 12:20:56 ls 1534 2024-05-21 12:20:59 vi * 1535 2024-05-21 12:21:26 ls - 1536 2024-05-21 12:21:30 rm README.md + 1536 2024-05-21 12:21:30 rm README.md 1537 2024-05-21 12:21:30 ls 1538 2024-05-21 12:21:32 cd RTEM 1539 2024-05-21 12:21:32 ls @@ -1562,12 +1562,12 @@ 1562 2024-05-21 12:41:11 vi *yml 1563 2024-05-21 12:41:16 ls 1564 2024-05-21 13:14:54 vi edl-project.tf - 1565 2024-05-21 13:14:58 vi edl-project.permissionset.tf - 1566 2024-05-21 13:15:52 vi variables.environments.tf + 1565 2024-05-21 13:14:58 vi edl-project.permissionset.tf + 1566 2024-05-21 13:15:52 vi variables.environments.tf 1567 2024-05-21 13:17:19 ls - 1568 2024-05-21 13:17:21 vi variables.environments.tf + 1568 2024-05-21 13:17:21 vi variables.environments.tf 1569 2024-05-21 13:17:28 ls - 1570 2024-05-21 13:17:35 cvi edl-project.yml + 1570 2024-05-21 13:17:35 cvi edl-project.yml 1571 2024-05-21 13:17:38 vi edl* 1572 2024-05-21 13:18:00 ls 1573 2024-05-21 13:18:16 vi edl*tf @@ -1579,11 +1579,11 @@ 1579 2024-05-21 13:19:47 ls 1580 2024-05-21 13:19:50 ls .. 1581 2024-05-21 13:19:58 pwed - 1582 2024-05-21 13:20:00 cat ../base_arn.tf - 1583 2024-05-21 13:20:12 vi tf-run.data + 1582 2024-05-21 13:20:00 cat ../base_arn.tf + 1583 2024-05-21 13:20:12 vi tf-run.data 1584 2024-05-21 13:20:20 ls - 1585 2024-05-21 13:20:22 cat ../base_arn.tf - 1586 2024-05-21 13:20:26 vi tf-run.data + 1585 2024-05-21 13:20:22 cat ../base_arn.tf + 1586 2024-05-21 13:20:26 vi tf-run.data 1587 2024-05-21 13:20:28 ls 1588 2024-05-21 13:20:29 vi edl*tf 1589 2024-05-21 13:22:12 tf-run plan @@ -1593,22 +1593,22 @@ 1593 2024-05-21 13:24:46 tf-init 1594 2024-05-21 13:24:58 tf-run init 1595 2024-05-21 13:25:03 tf-init - 1596 2024-05-21 13:25:37 cat role.tf + 1596 2024-05-21 13:25:37 cat role.tf 1597 2024-05-21 13:26:20 ls 1598 2024-05-21 13:26:22 vi edl*tf 1599 2024-05-21 13:27:05 ls - 1600 2024-05-21 13:27:12 vi versions.tf + 1600 2024-05-21 13:27:12 vi versions.tf 1601 2024-05-21 13:27:20 ls 1602 2024-05-21 13:27:22 tf-init -upgrade 1603 2024-05-21 13:27:42 tf-plan - 1604 2024-05-21 13:28:09 vi edl-project.permissionset.tf + 1604 2024-05-21 13:28:09 vi edl-project.permissionset.tf 1605 2024-05-21 13:28:27 cat *yml - 1606 2024-05-21 13:28:29 vi edl-project.permissionset.tf + 1606 2024-05-21 13:28:29 vi edl-project.permissionset.tf 1607 2024-05-21 13:28:41 ls 1608 2024-05-21 13:28:44 tf-fmt 1609 2024-05-21 13:28:45 tf-plan 1610 2024-05-21 13:29:19 ls .terraform/modules/ - 1611 2024-05-21 13:29:28 vi .terraform/modules//group_edl-project/group-assignment/main.tf + 1611 2024-05-21 13:29:28 vi .terraform/modules//group_edl-project/group-assignment/main.tf 1612 2024-05-21 13:29:50 grep settings .terraform/modules//group_edl-project/group-assignment/*tf 1613 2024-05-21 13:30:04 vi edl*tf 1614 2024-05-21 13:30:19 tf-console @@ -1619,12 +1619,12 @@ 1619 2024-05-21 13:36:23 vi edl*tf 1620 2024-05-21 13:38:00 tf-plan 1621 2024-05-21 13:38:10 vi edl*tf - 1622 2024-05-21 13:38:12 vi versions.tf + 1622 2024-05-21 13:38:12 vi versions.tf 1623 2024-05-21 13:38:22 tf-fm,t 1624 2024-05-21 13:38:23 tf-fmt 1625 2024-05-21 13:38:25 tf-init 1626 2024-05-21 13:41:12 tf-plan - 1627 2024-05-21 13:41:45 vi versions.tf + 1627 2024-05-21 13:41:45 vi versions.tf 1628 2024-05-21 13:41:49 vi edl*tf 1629 2024-05-21 13:42:00 tf-plan 1630 2024-05-21 13:42:45 vi edl*tf @@ -1648,7 +1648,7 @@ 1648 2024-05-21 14:08:53 aws --profile 039006259752-edl-management-gov.edl-u-7533453 whoami 1649 2024-05-21 14:09:53 vi ~/.aws/config 1650 2024-05-21 14:10:22 which aws-sso-util - 1651 2024-05-21 14:10:27 source /apps//terraform//etc//set-proxy.sh + 1651 2024-05-21 14:10:27 source /apps//terraform//etc//set-proxy.sh 1652 2024-05-21 14:10:30 aws --profile 039006259752-edl-management-gov.edl-u-7533453 whoami 1653 2024-05-21 14:10:45 aws --profile 039006259752-edl-management-gov.edl-u-7533453 sts get-caller-identity 1654 2024-05-21 14:33:55 cd ../vpc @@ -1658,7 +1658,7 @@ 1658 2024-05-21 14:34:00 cd vpc7 1659 2024-05-21 14:34:00 ls 1660 2024-05-21 14:34:02 cd vpc-endpoints - 1661 2024-05-21 14:34:03 tf-output + 1661 2024-05-21 14:34:03 tf-output 1662 2024-05-21 14:34:23 tf-output less 1663 2024-05-21 14:35:55 cd $(pwd |sed -e 's/west/east/') 1664 2024-05-21 14:35:58 tf-output less @@ -1676,12 +1676,12 @@ 1676 2024-05-21 14:41:32 vi variables.common*auto* 1677 2024-05-21 14:41:41 ls 1678 2024-05-21 14:41:45 vi variables.images.* - 1679 2024-05-21 14:42:04 less ~/terraform-modules/aws-eks//examples//full-cluster-tf-upgrade/1.29/common-services//variables.images.auto.tfvars + 1679 2024-05-21 14:42:04 less ~/terraform-modules/aws-eks//examples//full-cluster-tf-upgrade/1.29/common-services//variables.images.auto.tfvars 1680 2024-05-21 14:42:25 less variables.images.* 1681 2024-05-21 14:42:37 cd .. 1682 2024-05-21 14:42:38 cd addons/ 1683 2024-05-21 14:42:38 ls - 1684 2024-05-21 14:42:42 less variables.addons.tf + 1684 2024-05-21 14:42:42 less variables.addons.tf 1685 2024-05-21 14:42:46 ls 1686 2024-05-21 14:42:47 cd .. 1687 2024-05-21 14:42:48 ls @@ -1707,7 +1707,7 @@ 1707 2024-05-21 15:22:31 git pull 1708 2024-05-21 15:22:34 cd datadog-agent/ 1709 2024-05-21 15:22:34 ls - 1710 2024-05-21 15:22:41 vi datadog.values.yml + 1710 2024-05-21 15:22:41 vi datadog.values.yml 1711 2024-05-21 15:23:16 tf-plan 1712 2024-05-21 15:23:23 aws-sso-login.sh all 1713 2024-05-21 15:24:36 tf-plan @@ -1752,7 +1752,7 @@ 1752 2024-05-22 09:22:23 cd account-deployment/ 1753 2024-05-22 09:22:24 ls 1754 2024-05-22 09:22:31 ls *hcl - 1755 2024-05-22 09:22:35 vi datadog.tags.hcl + 1755 2024-05-22 09:22:35 vi datadog.tags.hcl 1756 2024-05-22 09:31:19 ls 1757 2024-05-22 09:31:23 ma5 1758 2024-05-22 09:31:25 a5 @@ -1763,11 +1763,11 @@ 1763 2024-05-22 09:43:53 ls 1764 2024-05-22 09:43:54 vi r 1765 2024-05-22 09:43:55 ls - 1766 2024-05-22 09:43:57 vi r-edl-cods.tf + 1766 2024-05-22 09:43:57 vi r-edl-cods.tf 1767 2024-05-22 09:44:15 git status . - 1768 2024-05-22 09:44:17 vi r-edl-cods.tf + 1768 2024-05-22 09:44:17 vi r-edl-cods.tf 1769 2024-05-22 09:45:28 tf-plan - 1770 2024-05-22 09:45:52 vi variables.auto.tfvars + 1770 2024-05-22 09:45:52 vi variables.auto.tfvars 1771 2024-05-22 09:45:55 tf-plan 1772 2024-05-22 09:46:48 hgrept management 1773 2024-05-22 09:46:57 aws --profile edl-management-gov.r-edl-cods whoami @@ -1785,10 +1785,10 @@ 1785 2024-05-22 09:49:34 less *hier*yml 1786 2024-05-22 09:49:42 tf-aws organizations describe-organizational-unit 1787 2024-05-22 09:49:51 tf-aws organizations describe-organizational-unit --organizational-unit-id "ou-9go7-5fdk2tby" - 1788 2024-05-22 09:50:09 vi variables.auto.tfvars + 1788 2024-05-22 09:50:09 vi variables.auto.tfvars 1789 2024-05-22 09:50:39 aws --profile edl-core-common-gov.r-edl-cods whoami 1790 2024-05-22 09:55:42 less *hier*yml - 1791 2024-05-22 09:56:10 vi variables.auto.tfvars + 1791 2024-05-22 09:56:10 vi variables.auto.tfvars 1792 2024-05-22 09:56:42 tf-plan 1793 2024-05-22 09:56:55 aws --profile edl-core-common-gov.r-edl-cods whoami 1794 2024-05-22 09:57:08 tf-apply -auto-approve @@ -1820,12 +1820,12 @@ 1820 2024-05-23 07:51:23 cd terraform 1821 2024-05-23 07:51:23 ls 1822 2024-05-23 07:51:27 stty sane - 1823 2024-05-23 07:51:29 vi + 1823 2024-05-23 07:51:29 vi 1824 2024-05-23 07:51:35 pwd 1825 2024-05-23 07:51:35 ls 1826 2024-05-23 07:52:43 pwd 1827 2024-05-23 07:52:44 clear - 1828 2024-05-23 07:52:48 vi + 1828 2024-05-23 07:52:48 vi 1829 2024-05-23 07:52:52 stty 1830 2024-05-23 07:52:58 env|grep TERM 1831 2024-05-23 07:53:27 pwd @@ -1864,28 +1864,28 @@ 1864 2024-05-23 08:06:56 ls 1865 2024-05-23 08:07:06 git srtatus . 1866 2024-05-23 08:07:07 git status . - 1867 2024-05-23 08:07:13 vi datadog.values.yml + 1867 2024-05-23 08:07:13 vi datadog.values.yml 1868 2024-05-23 08:07:19 git diff . 1869 2024-05-23 08:07:23 git co -- datadog.values.yml - 1870 2024-05-23 08:07:36 less debug.log + 1870 2024-05-23 08:07:36 less debug.log 1871 2024-05-23 08:20:17 ls - 1872 2024-05-23 08:20:19 vi role.tf - 1873 2024-05-23 08:20:40 vi variables.auto.tfvars + 1872 2024-05-23 08:20:19 vi role.tf + 1873 2024-05-23 08:20:40 vi variables.auto.tfvars 1874 2024-05-23 08:21:01 ls - 1875 2024-05-23 08:21:03 vi r-edl-cods.tf + 1875 2024-05-23 08:21:03 vi r-edl-cods.tf 1876 2024-05-23 08:20:57 tf-plan - 1877 2024-05-23 08:21:07 vi variables.auto.tfvars + 1877 2024-05-23 08:21:07 vi variables.auto.tfvars 1878 2024-05-23 08:21:40 tf-plan - 1879 2024-05-23 08:21:22 tf-apply + 1879 2024-05-23 08:21:22 tf-apply 1880 2024-05-23 08:25:28 cd common-services/ 1881 2024-05-23 08:25:29 ls 1882 2024-05-23 08:25:29 cd .. - 1883 2024-05-23 08:25:30 vi role.tf + 1883 2024-05-23 08:25:30 vi role.tf 1884 2024-05-23 08:25:47 git pull 1885 2024-05-23 08:25:56 cd vpc/west//vpc8/apps//projects/ 1886 2024-05-23 08:25:57 cd 7533453 1887 2024-05-23 08:25:58 ls - 1888 2024-05-23 08:26:02 vi role.tf + 1888 2024-05-23 08:26:02 vi role.tf 1889 2024-05-23 08:26:21 pwd 1890 2024-05-23 08:26:21 ls 1891 2024-05-23 08:26:24 tf-init @@ -1893,27 +1893,27 @@ 1893 2024-05-23 08:26:49 ls 1894 2024-05-23 08:37:09 git pull 1895 2024-05-23 08:42:02 tf-plan - 1896 2024-05-23 08:26:53 vi README.md + 1896 2024-05-23 08:26:53 vi README.md 1897 2024-05-23 08:53:56 git status . 1898 2024-05-23 08:53:59 gi tadd . 1899 2024-05-23 08:54:01 git add . 1900 2024-05-23 08:54:09 git commit -minitial . 1901 2024-05-23 08:54:11 git push 1902 2024-05-23 08:55:18 ls - 1903 2024-05-23 08:55:19 vi role.tf + 1903 2024-05-23 08:55:19 vi role.tf 1904 2024-05-23 08:55:24 grei iam_ *tf 1905 2024-05-23 08:55:26 grep iam_ *tf 1906 2024-05-23 08:55:33 gre piam_arn * 1907 2024-05-23 08:55:37 grep iam_arn * - 1908 2024-05-23 08:54:33 vi role.tf + 1908 2024-05-23 08:54:33 vi role.tf 1909 2024-05-23 08:59:01 cp role.tf role.tf.new 1910 2024-05-23 08:59:02 tf-plan - 1911 2024-05-23 09:00:11 tf-apply + 1911 2024-05-23 09:00:11 tf-apply 1912 2024-05-23 09:02:24 hgrept profile 1913 2024-05-23 09:02:29 aws --profile edl-core-common-gov.r-edl-cods whoami 1914 2024-05-23 09:02:44 aws --profile 039006259752-edl-management-gov.edl-u-7533453 whoami 1915 2024-05-23 09:03:00 aws --profile 001522620024-edl-addcp-dev-gov.r-edl-dev-7533453 whoami - 1916 2024-05-23 09:13:48 vi role.tf + 1916 2024-05-23 09:13:48 vi role.tf 1917 2024-05-23 09:14:30 tf-plan 1918 2024-05-23 09:14:43 cd common-services/ 1919 2024-05-23 09:14:43 ls @@ -1929,10 +1929,10 @@ 1929 2024-05-23 09:15:31 ls 1930 2024-05-23 09:15:37 cd .. 1931 2024-05-23 09:15:37 ls - 1932 2024-05-23 09:15:39 vi role.tf + 1932 2024-05-23 09:15:39 vi role.tf 1933 2024-05-23 09:16:19 cd aws-auth/ 1934 2024-05-23 09:16:19 ls - 1935 2024-05-23 09:16:22 vi config_map.aws-auth.yaml.tpl + 1935 2024-05-23 09:16:22 vi config_map.aws-auth.yaml.tpl 1936 2024-05-23 09:16:26 vi *auto* 1937 2024-05-23 09:16:33 ls 1938 2024-05-23 09:16:47 kubectl --kubeconfig setup/kube.config get configmap aws-auth @@ -1960,9 +1960,9 @@ 1960 2024-05-23 10:02:29 git pull 1961 2024-05-23 10:02:31 cd managed-prefixes/ 1962 2024-05-23 10:02:31 ls - 1963 2024-05-23 10:02:34 vi transit-gateway-prefixes.yml + 1963 2024-05-23 10:02:34 vi transit-gateway-prefixes.yml 1964 2024-05-23 10:02:52 ls - 1965 2024-05-23 10:02:53 vi README.md + 1965 2024-05-23 10:02:53 vi README.md 1966 2024-05-23 10:02:56 ls 1967 2024-05-23 10:02:58 grep 10.132 * 1968 2024-05-23 10:03:02 grep 10.131 * @@ -2022,7 +2022,7 @@ 2022 2024-05-23 10:23:54 rm prov*info* 2023 2024-05-23 10:23:58 mv /tmp/provider.infoblox.tf . 2024 2024-05-23 10:24:01 git status . - 2025 2024-05-23 10:24:03 vi provider.infoblox.tf + 2025 2024-05-23 10:24:03 vi provider.infoblox.tf 2026 2024-05-23 10:24:09 ls 2027 2024-05-23 10:24:12 git status . 2028 2024-05-23 10:24:26 git commit -m'update to new parameter-based infoblox provider' -a @@ -2032,7 +2032,7 @@ 2032 2024-05-23 10:24:53 ls 2033 2024-05-23 10:25:01 cd vpc/east//vpc3/apps//fsx/ 2034 2024-05-23 10:25:01 ls - 2035 2024-05-23 10:25:10 vi tf-run.data + 2035 2024-05-23 10:25:10 vi tf-run.data 2036 2024-05-23 10:25:30 tf-run less 2037 2024-05-23 10:25:39 tf-run list 2038 2024-05-23 10:25:44 tf-run apply 10 10 @@ -2044,7 +2044,7 @@ 2044 2024-05-23 10:30:06 pwd 2045 2024-05-23 10:30:07 cd .. 2046 2024-05-23 10:30:08 ls - 2047 2024-05-23 10:30:10 less infoblox.tf + 2047 2024-05-23 10:30:10 less infoblox.tf 2048 2024-05-23 10:31:37 cd .. 2049 2024-05-23 10:31:40 cd stacksets/ 2050 2024-05-23 10:31:40 ls @@ -2056,16 +2056,16 @@ 2056 2024-05-23 10:31:58 ls 2057 2024-05-23 10:31:59 cd provider-infoblox/ 2058 2024-05-23 10:32:00 ls - 2059 2024-05-23 10:32:06 vi secret.tf + 2059 2024-05-23 10:32:06 vi secret.tf 2060 2024-05-23 10:32:09 ls 2061 2024-05-23 10:32:11 vi va*auto* 2062 2024-05-23 10:36:45 tf-apply -refresh-only=true - 2063 2024-05-23 10:37:06 vi versions.tf + 2063 2024-05-23 10:37:06 vi versions.tf 2064 2024-05-23 10:37:21 tf-init -upgrade 2065 2024-05-23 10:39:59 tf-apply -refresh-only=true 2066 2024-05-23 10:40:40 tf-state show data.aws_fsx_windows_file_system.fs - 2067 2024-05-23 10:41:44 vi versions.tf - 2068 2024-05-23 10:41:52 vi fsx-dns.tf + 2067 2024-05-23 10:41:44 vi versions.tf + 2068 2024-05-23 10:41:52 vi fsx-dns.tf 2069 2024-05-23 10:45:37 tf-plan 2070 2024-05-23 10:46:12 tf-plan summary 2071 2024-05-23 10:46:15 tf-plan less @@ -2078,7 +2078,7 @@ 2078 2024-05-23 10:47:30 git srtatus . 2079 2024-05-23 10:48:05 git commit -m'add dns entry for fsx' . 2080 2024-05-23 10:48:07 git push - 2081 2024-05-23 10:48:22 vi fsx-dns.tf + 2081 2024-05-23 10:48:22 vi fsx-dns.tf 2082 2024-05-23 10:48:59 git commit -m'add comment' . 2083 2024-05-23 10:49:01 git push 2084 2024-05-23 10:49:04 pwd @@ -2091,9 +2091,9 @@ 2091 2024-05-23 10:51:01 cd dns 2092 2024-05-23 10:51:04 tf-run init 2093 2024-05-23 10:51:06 tf-run apply - 2094 2024-05-23 10:51:23 vi versions.tf + 2094 2024-05-23 10:51:23 vi versions.tf 2095 2024-05-23 10:51:29 tf-fmt - 2096 2024-05-23 10:51:32 vi versions.tf + 2096 2024-05-23 10:51:32 vi versions.tf 2097 2024-05-23 10:51:35 tt-fmt 2098 2024-05-23 10:51:39 tf-fmt 2099 2024-05-23 10:51:41 tf-init -ugprade @@ -2123,9 +2123,9 @@ 2123 2024-05-23 11:40:36 terraform_1.8.2 init -upgrade 2124 2024-05-23 11:40:47 pwd 2125 2024-05-23 11:40:47 ls - 2126 2024-05-23 11:40:49 ./get-terraform.sh - 2127 2024-05-23 11:40:55 source /apps//terraform//etc//set-proxy.sh - 2128 2024-05-23 11:40:57 ./get-terraform.sh + 2126 2024-05-23 11:40:49 ./get-terraform.sh + 2127 2024-05-23 11:40:55 source /apps//terraform//etc//set-proxy.sh + 2128 2024-05-23 11:40:57 ./get-terraform.sh 2129 2024-05-23 11:41:01 ./get-terraform.sh current 2130 2024-05-23 11:41:14 ls -al *1.8* 2131 2024-05-23 11:41:25 ./get-terraform.sh current @@ -2152,25 +2152,25 @@ 2152 2024-05-23 12:11:58 tf-plan 2153 2024-05-23 12:11:46 /apps//terraform//bin/tgw-route-status.sh |& tee logs/tgw-status.$(date +%s).log 2154 2024-05-23 12:14:34 vi dx-hq-verizon.tf - 2155 2024-05-23 12:15:15 grep -iE "summary|^vpn" logs/tgw-status.1716480706.log + 2155 2024-05-23 12:15:15 grep -iE "summary|^vpn" logs/tgw-status.1716480706.log 2156 2024-05-23 12:15:19 grep -iE "summary|^vpn" logs/tgw-status.1716480706.log |grep _hq 2157 2024-05-23 12:15:39 grep -iE "summary|^vpn" logs/tgw-status.1716480706.log |grep _bcc 2158 2024-05-23 12:15:51 less logs/tgw-status.1716480706.log 2159 2024-05-23 12:17:15 cd ../west/ 2160 2024-05-23 12:17:18 hgrept tgw 2161 2024-05-23 12:17:23 /apps//terraform//bin/tgw-route-status.sh |& tee logs/tgw-status.$(date +%s).log - 2162 2024-05-23 12:21:22 less logs/tgw-status.1716481043.log + 2162 2024-05-23 12:21:22 less logs/tgw-status.1716481043.log 2163 2024-05-23 12:22:39 for f in services dev test stage prod cre; do show-tgw-tunnel-status.sh $f ; done |& tee logs/show-tgw-tunnel-status.$(date +%s).log 2164 2024-05-23 12:24:06 cd ../east/ 2165 2024-05-23 12:24:17 pwd 2166 2024-05-23 12:24:22 hgrept tgw- 2167 2024-05-23 12:24:08 for f in services dev test stage prod cre; do show-tgw-tunnel-status.sh $f ; done |& tee logs/show-tgw-tunnel-status.$(date +%s).log 2168 2024-05-23 12:24:29 /apps//terraform//bin/tgw-route-status.sh |& tee logs/tgw-status.$(date +%s).log - 2169 2024-05-23 12:25:03 grep ^vpn_hq logs/tgw-status.1716481469.log + 2169 2024-05-23 12:25:03 grep ^vpn_hq logs/tgw-status.1716481469.log 2170 2024-05-23 12:25:08 cd ../west/ 2171 2024-05-23 12:25:11 /apps//terraform//bin/tgw-route-status.sh |& tee logs/tgw-status.$(date +%s).log 2172 2024-05-23 12:25:48 grep ^vpn_hq logs/tgw-status.* - 2173 2024-05-23 12:30:27 tf-apply + 2173 2024-05-23 12:30:27 tf-apply 2174 2024-05-23 12:32:05 show-tgw-tunnel-status.sh services 2175 2024-05-23 12:32:21 show-tgw-tunnel-status.sh services|grep -A2 vpn_hq 2176 2024-05-23 12:55:23 date --date=@1716481827 @@ -2178,14 +2178,14 @@ 2178 2024-05-23 13:06:33 git push 2179 2024-05-23 13:08:21 skopeo inspect docker://docker.io/elastic/filebeat:8.13.4 2180 2024-05-23 13:08:31 env|grep -i proxy - 2181 2024-05-23 13:08:42 source /apps//terraform//etc//set-proxy.sh + 2181 2024-05-23 13:08:42 source /apps//terraform//etc//set-proxy.sh 2182 2024-05-23 13:08:45 skopeo inspect docker://docker.io/elastic/filebeat:8.13.4 2183 2024-05-23 13:44:01 pwd 2184 2024-05-23 13:44:03 cd .. 2185 2024-05-23 13:44:03 ls 2186 2024-05-23 13:44:04 cd acl/ 2187 2024-05-23 13:44:04 ls - 2188 2024-05-23 13:44:07 less test1.py + 2188 2024-05-23 13:44:07 less test1.py 2189 2024-05-23 13:44:26 ls -al 2190 2024-05-23 13:42:45 less find* 2191 2024-05-23 13:44:42 mkdir .svae @@ -2195,14 +2195,14 @@ 2195 2024-05-23 13:45:01 rmdir .save/.save/ 2196 2024-05-23 13:45:15 ls 2197 2024-05-23 13:45:19 getfacl * - 2198 2024-05-23 13:45:25 python test1.py + 2198 2024-05-23 13:45:25 python test1.py 2199 2024-05-23 13:45:36 source /apps//anaconda/bin//activate py3 - 2200 2024-05-23 13:45:37 python test1.py + 2200 2024-05-23 13:45:37 python test1.py 2201 2024-05-23 13:45:46 whcih pythoin 2202 2024-05-23 13:45:48 whcih pythyon 2203 2024-05-23 13:45:50 whcih python 2204 2024-05-23 13:45:53 which python - 2205 2024-05-23 13:46:27 cat test1.py + 2205 2024-05-23 13:46:27 cat test1.py 2206 2024-05-23 14:08:33 etwork 2207 2024-05-23 14:08:35 etwork 2208 2024-05-23 14:08:37 et @@ -2216,7 +2216,7 @@ 2216 2024-05-23 14:09:07 pwd 2217 2024-05-23 14:09:07 ls 2218 2024-05-23 14:09:11 ls *off - 2219 2024-05-23 14:09:15 mv sns.datadog.tf.off sns.datadog.tf.off + 2219 2024-05-23 14:09:15 mv sns.datadog.tf.off sns.datadog.tf.off 2220 2024-05-23 14:09:20 git mv sns.datadog.tf.off sns.datadog.tf 2221 2024-05-23 14:09:22 tf-plan 2222 2024-05-23 14:57:33 popd @@ -2228,18 +2228,18 @@ 2228 2024-05-23 14:57:54 show-tgw-tunnel-status.sh services|grep -A2 vpn_hq 2229 2024-05-23 13:47:03 sudo -s 2230 2024-05-23 15:08:32 ls - 2231 2024-05-23 15:08:35 python test1.py - 2232 2024-05-23 15:08:50 vi test1.py + 2231 2024-05-23 15:08:35 python test1.py + 2232 2024-05-23 15:08:50 vi test1.py 2233 2024-05-23 15:09:01 cp test1.py test2.py 2234 2024-05-23 15:09:02 vi teset2.py 2235 2024-05-23 15:09:05 ls - 2236 2024-05-23 15:09:08 vi test2.py - 2237 2024-05-23 15:09:32 python test2.py + 2236 2024-05-23 15:09:08 vi test2.py + 2237 2024-05-23 15:09:32 python test2.py 2238 2024-05-23 15:09:39 ls 2239 2024-05-23 15:09:47 mkdir file3.dir - 2240 2024-05-23 15:09:50 python test2.py + 2240 2024-05-23 15:09:50 python test2.py 2241 2024-05-23 15:09:54 mkdir file4.dir - 2242 2024-05-23 15:09:55 python test2.py + 2242 2024-05-23 15:09:55 python test2.py 2243 2024-05-23 15:40:29 cd .. 2244 2024-05-23 15:40:31 cd common-services/ 2245 2024-05-23 15:40:31 ls @@ -2250,10 +2250,10 @@ 2250 2024-05-23 15:40:50 pwd 2251 2024-05-23 15:40:51 ls 2252 2024-05-23 15:40:53 git status . - 2253 2024-05-23 15:40:58 tf-apply less + 2253 2024-05-23 15:40:58 tf-apply less 2254 2024-05-23 15:41:53 tf-apply -auto-approve 2255 2024-05-23 15:42:57 tf-aws whoami - 2256 2024-05-23 15:43:03 source /apps//terraform/etc/set-proxy.sh + 2256 2024-05-23 15:43:03 source /apps//terraform/etc/set-proxy.sh 2257 2024-05-23 15:43:09 tf-apply -auto-approve 2258 2024-05-23 15:50:11 cd /data/tmp/files/ent 2259 2024-05-23 15:50:13 cd /data/tmp/files/net @@ -2271,7 +2271,7 @@ 2271 2024-05-23 15:55:08 ls 2272 2024-05-23 16:40:10 hgrep brom 2273 2024-05-23 16:40:19 grpe bro - 2274 2024-05-23 16:40:22 grep bromine .vlab1-screenrc + 2274 2024-05-23 16:40:22 grep bromine .vlab1-screenrc 2275 2024-05-23 16:40:29 screen -t bromine ssh -4 -AXt jump.nfluorine.tco.census.gov "ssh -4 -AXt bromine.cto.census.gov" 2276 2024-05-23 17:41:04 cd infrastructure/ 2277 2024-05-23 17:41:05 git pull @@ -2286,7 +2286,7 @@ 2286 2024-05-23 17:46:58 ls 2287 2024-05-23 17:47:09 tf-aws whoami 2288 2024-05-23 17:47:15 aws-sso-login.sh ent-ew - 2289 2024-05-23 17:47:35 tf-apply + 2289 2024-05-23 17:47:35 tf-apply 2290 2024-05-23 18:04:01 pwd 2291 2024-05-23 18:04:01 ls 2292 2024-05-23 18:04:12 cat $(git root)/pro*d/pro*inf* @@ -2328,7 +2328,7 @@ 2328 2024-05-24 09:46:07 ls 2329 2024-05-24 09:46:11 cd aws-ecr-copy-images/ 2330 2024-05-24 09:46:11 ls - 2331 2024-05-24 09:46:13 vi main.tf + 2331 2024-05-24 09:46:13 vi main.tf 2332 2024-05-24 10:45:46 cd ../aws-eks/ 2333 2024-05-24 10:45:46 ls 2334 2024-05-24 10:45:48 cd examples/full-cluster-tf-upgrade/ @@ -2338,7 +2338,7 @@ 2338 2024-05-24 11:12:47 cd permissionsets 2339 2024-05-24 11:12:49 cd sc-tester/ 2340 2024-05-24 11:12:49 ls - 2341 2024-05-24 11:12:52 vi sc-tester.permissionset.tf + 2341 2024-05-24 11:12:52 vi sc-tester.permissionset.tf 2342 2024-05-24 11:16:17 git pull 2343 2024-05-24 11:16:19 cd .. 2344 2024-05-24 11:16:26 cd sc-finops/ @@ -2347,12 +2347,12 @@ 2347 2024-05-24 11:16:54 tf-aws iam list-roles|grep -i ReadOnly|less 2348 2024-05-24 11:17:08 tf-aws iam list-roles|grep -i Billing|less 2349 2024-05-24 11:17:22 tf-aws iam list-roles help - 2350 2024-05-24 11:31:05 vi sc-finops.permissionset.tf + 2350 2024-05-24 11:31:05 vi sc-finops.permissionset.tf 2351 2024-05-24 11:31:42 vi ../*/sc-*tf 2352 2024-05-24 11:31:55 vi ../sc*/sc-*tf 2353 2024-05-24 11:32:01 vi ../../sc*/sc-*tf 2354 2024-05-24 11:32:07 vi ../../*/sc*/sc-*tf - 2355 2024-05-24 11:32:14 vi sc-finops.permissionset.tf + 2355 2024-05-24 11:32:14 vi sc-finops.permissionset.tf 2356 2024-05-24 11:32:45 git pull 2357 2024-05-24 11:32:54 tf-fmt 2358 2024-05-24 11:32:57 tf-plan @@ -2369,9 +2369,9 @@ 2369 2024-05-24 11:35:41 tf-init 2370 2024-05-24 11:36:00 vi okta*6* 2371 2024-05-24 11:36:24 tf-plan - 2372 2024-05-24 11:37:58 vi .terraform/modules/group_okta-test6/group-assignment/main.tf + 2372 2024-05-24 11:37:58 vi .terraform/modules/group_okta-test6/group-assignment/main.tf 2373 2024-05-24 11:39:28 tf-plan - 2374 2024-05-24 11:41:23 vi .terraform/modules/group_okta-test6/group-assignment/main.tf + 2374 2024-05-24 11:41:23 vi .terraform/modules/group_okta-test6/group-assignment/main.tf 2375 2024-05-24 11:41:46 tf-plan;tf-plan summary 2376 2024-05-24 11:44:51 cd ../.. 2377 2024-05-24 11:44:55 show-instances.sh > i @@ -2386,15 +2386,15 @@ 2386 2024-05-24 11:47:35 ps -efww|grep tcp 2387 2024-05-24 11:47:37 sudo -s 2388 2024-05-24 11:47:50 hgrept ssm - 2389 2024-05-24 11:44:44 tf-apply - 2390 2024-05-24 11:52:11 vi .terraform/modules/group_okta-test6/group-assignment/main.tf + 2389 2024-05-24 11:44:44 tf-apply + 2390 2024-05-24 11:52:11 vi .terraform/modules/group_okta-test6/group-assignment/main.tf 2391 2024-05-24 11:52:22 vi okta-test6.tf 2392 2024-05-24 11:53:15 pwd 2393 2024-05-24 12:01:41 git status . 2394 2024-05-24 12:01:48 git commit -m'add ce:Describe*' . 2395 2024-05-24 12:01:50 git pull 2396 2024-05-24 12:01:52 git push - 2397 2024-05-24 11:52:47 tf-apply + 2397 2024-05-24 11:52:47 tf-apply 2398 2024-05-24 12:02:15 tf-apply -auto-approve 2399 2024-05-24 11:47:52 tf-aws ssm start-session --target i-080756d4c59237efd 2400 2024-05-24 12:53:50 exit @@ -2412,7 +2412,7 @@ 2412 2024-05-24 13:47:49 ls 2413 2024-05-24 13:47:50 cd v3 2414 2024-05-24 13:47:51 ls - 2415 2024-05-24 13:47:52 unzip Multi-Account\ Updated\ Roles\ \(5-23\).zip + 2415 2024-05-24 13:47:52 unzip Multi-Account\ Updated\ Roles\ \(5-23\).zip 2416 2024-05-24 13:47:54 ls 2417 2024-05-24 13:47:55 cd m 2418 2024-05-24 13:47:55 ls @@ -2432,7 +2432,7 @@ 2432 2024-05-24 13:54:58 yq -c 'users' r-*yaml 2433 2024-05-24 13:54:59 ls 2434 2024-05-24 13:55:29 cat r-adsd-dps-tier3support.yaml - 2435 2024-05-24 13:55:37 cat r-adsd-dps-tier3support.yaml | yq -c + 2435 2024-05-24 13:55:37 cat r-adsd-dps-tier3support.yaml | yq -c 2436 2024-05-24 13:55:44 cat r-adsd-dps-tier3support.yaml | yq -c 'users[]' 2437 2024-05-24 13:55:49 cat r-adsd-dps-tier3support.yaml | yq -c '{users[]}' 2438 2024-05-24 13:55:54 man jq @@ -2485,7 +2485,7 @@ 2485 2024-05-24 07:04:49 s f5 2486 2024-05-24 07:05:21 s tf2 2487 2024-05-24 07:04:44 s backup - 2488 2024-05-28 07:49:58 aws-sso-login.sh + 2488 2024-05-28 07:49:58 aws-sso-login.sh 2489 2024-05-28 07:50:03 aws-sso-login.sh all 2490 2024-05-28 07:57:27 curl -v https://cognito-idp.us-gov-west-1.amazonaws.com/us-gov-west-1_KgDaRItMB/.well-known/openid-configuration 2491 2024-05-28 07:57:37 hosw cognito-idp.us-gov-west-1.amazonaws.com @@ -2535,7 +2535,7 @@ 2535 2024-05-28 08:26:34 pwd 2536 2024-05-28 08:26:35 ls 2537 2024-05-28 08:26:41 git diff .|grep ^+ - 2538 2024-05-28 08:26:46 git diff .|grep ^+\ + 2538 2024-05-28 08:26:46 git diff .|grep ^+\ 2539 2024-05-28 08:26:49 git diff .|grep ^+\ |sort -u 2540 2024-05-28 08:26:53 git diff .|grep ^+\ |sort -u|awk '{print $3}' 2541 2024-05-28 08:27:00 git diff .|grep ^+\ |sort -u|awk '{print $3}' > u @@ -2543,12 +2543,12 @@ 2543 2024-05-28 08:27:11 rm u 2544 2024-05-28 08:27:12 cd ../.. 2545 2024-05-28 08:27:15 git pull - 2546 2024-05-28 08:27:23 ./sort-users.sh + 2546 2024-05-28 08:27:23 ./sort-users.sh 2547 2024-05-28 08:27:25 git diff . 2548 2024-05-28 08:27:30 git diff users.csv - 2549 2024-05-28 08:27:37 tf-apply + 2549 2024-05-28 08:27:37 tf-apply 2550 2024-05-28 08:32:24 tf-init -upgrade - 2551 2024-05-28 08:39:13 tf-apply + 2551 2024-05-28 08:39:13 tf-apply 2552 2024-05-28 08:42:04 git status . 2553 2024-05-28 08:42:11 git diff users.csv 2554 2024-05-28 08:42:21 cat permissionsets/edl-core/u @@ -2571,7 +2571,7 @@ 2571 2024-05-28 08:43:36 git diff .|grep ^+\ |sort -u|awk '{print $3}' > u 2572 2024-05-28 08:43:37 cat u 2573 2024-05-28 08:43:40 cd ../.. - 2574 2024-05-28 08:43:44 tf-output + 2574 2024-05-28 08:43:44 tf-output 2575 2024-05-28 08:43:53 tf-output > uu 2576 2024-05-28 08:43:56 vi uu 2577 2024-05-28 08:44:18 ls @@ -2585,8 +2585,8 @@ 2585 2024-05-28 08:45:42 ./check-users.sh u3 2586 2024-05-28 08:46:12 ./check-users.sh u3 > u4 2587 2024-05-28 08:46:36 for f in $(cat u3); do grep $f u4; done - 2588 2024-05-28 08:46:52 vi users.csv - 2589 2024-05-28 08:47:52 tf-apply + 2588 2024-05-28 08:46:52 vi users.csv + 2589 2024-05-28 08:47:52 tf-apply 2590 2024-05-28 08:50:33 ls u* 2591 2024-05-28 08:50:36 cat u3 2592 2024-05-28 08:50:41 cp u3 /tmp/ @@ -2609,10 +2609,10 @@ 2609 2024-05-28 09:11:30 pwd 2610 2024-05-28 09:11:35 pwd 2611 2024-05-28 09:11:35 ls - 2612 2024-05-28 09:11:42 vi vpn-tunnel-tracker.csv + 2612 2024-05-28 09:11:42 vi vpn-tunnel-tracker.csv 2613 2024-05-28 09:11:45 ls 2614 2024-05-28 09:11:47 grep 169.254 * - 2615 2024-05-28 09:11:52 vi vpn-endpoints.md + 2615 2024-05-28 09:11:52 vi vpn-endpoints.md 2616 2024-05-28 09:12:09 ls 2617 2024-05-28 09:12:22 cd .. 2618 2024-05-28 09:12:23 ls @@ -2620,24 +2620,24 @@ 2620 2024-05-28 09:12:26 ls 2621 2024-05-28 09:12:30 cd vpn-tunnel-cidr-allocation/ 2622 2024-05-28 09:12:30 ls - 2623 2024-05-28 09:12:36 vi tunnel-allocations.csv + 2623 2024-05-28 09:12:36 vi tunnel-allocations.csv 2624 2024-05-28 09:13:05 q! 2625 2024-05-28 09:13:17 sipcalc 169.254.240.0/20 2626 2024-05-28 09:13:48 ls - 2627 2024-05-28 09:13:52 vi gcp.tunnel-allocations.csv + 2627 2024-05-28 09:13:52 vi gcp.tunnel-allocations.csv 2628 2024-05-28 09:15:57 ls 2629 2024-05-28 09:07:52 tf-apply -auto-approve 2630 2024-05-28 09:16:07 vi cloudflare.tunnel-allocations.csv - 2631 2024-05-28 09:20:20 cat cloudflare.tunnel-allocations.csv - 2632 2024-05-28 09:21:23 vi gcp.tunnel-allocations.csv + 2631 2024-05-28 09:20:20 cat cloudflare.tunnel-allocations.csv + 2632 2024-05-28 09:21:23 vi gcp.tunnel-allocations.csv 2633 2024-05-28 09:22:08 :q! 2634 2024-05-28 09:22:09 ls - 2635 2024-05-28 09:22:12 vi tunnel-allocations.csv + 2635 2024-05-28 09:22:12 vi tunnel-allocations.csv 2636 2024-05-28 09:23:18 bc 2637 2024-05-28 09:33:27 git status 2638 2024-05-28 09:33:37 git commit -m'update edl-core users' -a 2639 2024-05-28 09:33:41 rm u u3 - 2640 2024-05-28 09:33:44 git status + 2640 2024-05-28 09:33:44 git status 2641 2024-05-28 09:33:45 git push 2642 2024-05-28 09:33:51 cd ../edl-projects/ 2643 2024-05-28 09:33:51 ls @@ -2659,9 +2659,9 @@ 2659 2024-05-28 09:39:22 git status 2660 2024-05-28 09:39:24 git push 2661 2024-05-28 09:46:03 aws-sso-util - 2662 2024-05-28 09:46:09 source /apps//terraform/etc//terraform.sh + 2662 2024-05-28 09:46:09 source /apps//terraform/etc//terraform.sh 2663 2024-05-28 09:46:11 aws-sso-util roles - 2664 2024-05-28 09:46:26 less /apps//terraform//bin/refresh-profile.sh + 2664 2024-05-28 09:46:26 less /apps//terraform//bin/refresh-profile.sh 2665 2024-05-28 09:47:11 aws-sso-util roles --sso-start-url "https://start.us-gov-home.awsapps.com/directory/d-c2673e7ee9" 2666 2024-05-28 09:47:42 curl -v http://r.census.gov/a/aws/sso/gov 2667 2024-05-28 09:47:51 curl -v http://r.census.gov/a/aws/sso/gov -D - @@ -2676,29 +2676,29 @@ 2676 2024-05-28 09:48:27 cd aws-sso-tools/ 2677 2024-05-28 09:48:27 ls 2678 2024-05-28 09:48:41 cp refresh-profile.sh sso-get-roles.sh - 2679 2024-05-28 09:48:43 vi sso-get-roles.sh - 2680 2024-05-28 09:53:46 chmod 755 sso-get-roles.sh - 2681 2024-05-28 09:53:51 ./sso-get-roles.sh + 2679 2024-05-28 09:48:43 vi sso-get-roles.sh + 2680 2024-05-28 09:53:46 chmod 755 sso-get-roles.sh + 2681 2024-05-28 09:53:51 ./sso-get-roles.sh 2682 2024-05-28 09:54:00 hgrep sso-start - 2683 2024-05-28 09:54:04 vi sso-get-roles.sh + 2683 2024-05-28 09:54:04 vi sso-get-roles.sh 2684 2024-05-28 09:54:12 hgrep sso-start - 2685 2024-05-28 09:54:15 ./sso-get-roles.sh - 2686 2024-05-28 09:54:45 vi sso-get-roles.sh - 2687 2024-05-28 09:55:22 ./sso-get-roles.sh - 2688 2024-05-28 09:55:27 vi sso-get-roles.sh - 2689 2024-05-28 09:56:19 ./sso-get-roles.sh + 2685 2024-05-28 09:54:15 ./sso-get-roles.sh + 2686 2024-05-28 09:54:45 vi sso-get-roles.sh + 2687 2024-05-28 09:55:22 ./sso-get-roles.sh + 2688 2024-05-28 09:55:27 vi sso-get-roles.sh + 2689 2024-05-28 09:56:19 ./sso-get-roles.sh 2690 2024-05-28 09:56:30 vi sso-gt - 2691 2024-05-28 09:56:33 vi sso-get-roles.sh - 2692 2024-05-28 09:56:56 bash -x ./sso-get-roles.sh - 2693 2024-05-28 09:57:05 vi sso-get-roles.sh - 2694 2024-05-28 09:57:13 bash -x ./sso-get-roles.sh - 2695 2024-05-28 09:57:36 vi sso-get-roles.sh - 2696 2024-05-28 09:57:53 bash -x ./sso-get-roles.sh - 2697 2024-05-28 09:58:00 vi sso-get-roles.sh + 2691 2024-05-28 09:56:33 vi sso-get-roles.sh + 2692 2024-05-28 09:56:56 bash -x ./sso-get-roles.sh + 2693 2024-05-28 09:57:05 vi sso-get-roles.sh + 2694 2024-05-28 09:57:13 bash -x ./sso-get-roles.sh + 2695 2024-05-28 09:57:36 vi sso-get-roles.sh + 2696 2024-05-28 09:57:53 bash -x ./sso-get-roles.sh + 2697 2024-05-28 09:58:00 vi sso-get-roles.sh 2698 2024-05-28 09:59:11 ls 2699 2024-05-28 09:59:12 cd .. 2700 2024-05-28 09:59:13 git status . - 2701 2024-05-28 09:58:19 bash -x ./sso-get-roles.sh + 2701 2024-05-28 09:58:19 bash -x ./sso-get-roles.sh 2702 2024-05-28 09:59:22 hgrept for.*users 2703 2024-05-28 09:59:34 history > history.$(date +%s) 2704 2024-05-28 09:59:37 cd .. @@ -2732,20 +2732,20 @@ 2732 2024-05-28 10:07:25 ls 2733 2024-05-28 10:07:27 cd ../.. 2734 2024-05-28 10:07:28 ls - 2735 2024-05-28 10:06:53 list-zones.sh + 2735 2024-05-28 10:06:53 list-zones.sh 2736 2024-05-28 10:07:31 cd vpc/east/ 2737 2024-05-28 10:07:32 cd vpc1 2738 2024-05-28 10:07:32 ls - 2739 2024-05-28 10:07:35 grep stage.dice list-zones.1713297291.txt + 2739 2024-05-28 10:07:35 grep stage.dice list-zones.1713297291.txt 2740 2024-05-28 10:07:50 ls ~/terraform/412295344020* 2741 2024-05-28 10:07:53 ls ~/terraform/412295344020* -d 2742 2024-05-28 10:09:18 pwd 2743 2024-05-28 10:09:18 ls 2744 2024-05-28 10:09:20 cat i 2745 2024-05-28 10:10:43 \ - 2746 2024-05-28 10:10:45 host SAS-7529993-WEST-1.DEV.EDL.CENSUS.GOV - 2747 2024-05-28 10:10:53 host - 2748 2024-05-28 10:11:11 host SAS-7529993-WEST-1.DEV.EDL.CENSUS.GOV + 2746 2024-05-28 10:10:45 host SAS-7529993-WEST-1.DEV.EDL.CENSUS.GOV + 2747 2024-05-28 10:10:53 host + 2748 2024-05-28 10:11:11 host SAS-7529993-WEST-1.DEV.EDL.CENSUS.GOV 2749 2024-05-28 10:11:15 host 10.196.102.82 2750 2024-05-28 10:11:27 host -t txt SAS-7529993-WEST-1.DEV.EDL.CENSUS.GOV 2751 2024-05-28 10:11:56 host -t a SAS-7529993-WEST-1.DEV.EDL.CENSUS.GOV @@ -2787,21 +2787,21 @@ 2787 2024-05-28 10:22:09 cd .. 2788 2024-05-28 10:22:14 cd local-app/infoblox/ 2789 2024-05-28 10:22:14 ls - 2790 2024-05-28 10:22:23 less infoblox-create-forwarding.py + 2790 2024-05-28 10:22:23 less infoblox-create-forwarding.py 2791 2024-05-28 10:23:00 source /apps/anaconda/bin/activate py3 2792 2024-05-28 10:23:31 #INFOBLOX_DMZ_VIEW=1 python infoblox-create-forwarding.py stage.dice.census.gov - 2793 2024-05-28 10:23:33 less infoblox-create-forwarding.py + 2793 2024-05-28 10:23:33 less infoblox-create-forwarding.py 2794 2024-05-28 10:24:55 git status info*py - 2795 2024-05-28 10:24:59 vi infoblox-create-forwarding.py + 2795 2024-05-28 10:24:59 vi infoblox-create-forwarding.py 2796 2024-05-28 10:28:13 git log infoblox-create-forwarding.py 2797 2024-05-28 10:28:19 git log infoblox-create-forwarding.py|grep -c ^comm - 2798 2024-05-28 10:28:21 vi infoblox-create-forwarding.py + 2798 2024-05-28 10:28:21 vi infoblox-create-forwarding.py 2799 2024-05-28 10:30:21 INFOBLOX_DMZ_VIEW=1 INFOBLOX_INTERNAL_VIEW=0 python infoblox-create-forwarding.py stage.dice.census.gov - 2800 2024-05-28 10:30:41 vi infoblox-create-forwarding.py - 2801 2024-05-28 10:30:55 INFOBLOX_DMZ_VIEW=1 INFOBLOX_INTERNAL_VIEW=0 python infoblox-restart.py - 2802 2024-05-28 10:34:45 vi infoblox-create-forwarding.py + 2800 2024-05-28 10:30:41 vi infoblox-create-forwarding.py + 2801 2024-05-28 10:30:55 INFOBLOX_DMZ_VIEW=1 INFOBLOX_INTERNAL_VIEW=0 python infoblox-restart.py + 2802 2024-05-28 10:34:45 vi infoblox-create-forwarding.py 2803 2024-05-28 10:35:17 INFOBLOX_DMZ=1 INFOBLOX_DMZ_VIEW=1 INFOBLOX_INTERNAL_VIEW=0 python infoblox-create-forwarding.py stage.dice.census.gov - 2804 2024-05-28 10:36:41 vi infoblox-create-forwarding.py + 2804 2024-05-28 10:36:41 vi infoblox-create-forwarding.py 2805 2024-05-28 10:37:51 hgrept rest 2806 2024-05-28 10:45:49 wpd 2807 2024-05-28 10:45:50 ls @@ -2854,90 +2854,90 @@ 2854 2024-05-28 11:35:01 tf-output less 2855 2024-05-28 11:54:54 pwd 2856 2024-05-28 11:54:55 git status . - 2857 2024-05-28 11:56:50 vi infoblox-create-forwarding.py + 2857 2024-05-28 11:56:50 vi infoblox-create-forwarding.py 2858 2024-05-28 11:56:57 pwd - 2859 2024-05-28 11:57:17 vi infoblox-create-forwarding.py + 2859 2024-05-28 11:57:17 vi infoblox-create-forwarding.py 2860 2024-05-28 12:00:04 grep argparse * 2861 2024-05-28 12:00:07 grep argparse ../* 2862 2024-05-28 12:00:09 grep argparse ../*/* 2863 2024-05-28 12:00:18 grep argparse ../*/*py 2864 2024-05-28 12:00:31 vi ../rotate-keys/rotate-keys.py - 2865 2024-05-28 12:00:44 vi ../rotate-keys/rotate-keys.py infoblox-create-forwarding.py + 2865 2024-05-28 12:00:44 vi ../rotate-keys/rotate-keys.py infoblox-create-forwarding.py 2866 2024-05-28 12:01:14 ls - 2867 2024-05-28 12:01:19 vi infoblox-create-forwarding.py + 2867 2024-05-28 12:01:19 vi infoblox-create-forwarding.py 2868 2024-05-28 12:01:30 git commit -m'update, but ready for redo' . - 2869 2024-05-28 12:01:32 vi infoblox-create-forwarding.py - 2870 2024-05-28 12:02:03 vi ../rotate-keys/rotate-keys.py infoblox-create-forwarding.py + 2869 2024-05-28 12:01:32 vi infoblox-create-forwarding.py + 2870 2024-05-28 12:02:03 vi ../rotate-keys/rotate-keys.py infoblox-create-forwarding.py 2871 2024-05-28 12:07:37 python infoblox-create-forwarding.py --help - 2872 2024-05-28 12:07:45 vi ../rotate-keys/rotate-keys.py infoblox-create-forwarding.py - 2873 2024-05-28 12:07:52 vi infoblox-create-forwarding.py - 2874 2024-05-28 12:11:43 vi ../rotate-keys/rotate-keys.py infoblox-create-forwarding.py + 2872 2024-05-28 12:07:45 vi ../rotate-keys/rotate-keys.py infoblox-create-forwarding.py + 2873 2024-05-28 12:07:52 vi infoblox-create-forwarding.py + 2874 2024-05-28 12:11:43 vi ../rotate-keys/rotate-keys.py infoblox-create-forwarding.py 2875 2024-05-28 12:11:46 python infoblox-create-forwarding.py --help - 2876 2024-05-28 12:11:51 vi ../rotate-keys/rotate-keys.py infoblox-create-forwarding.py - 2877 2024-05-28 12:12:01 vi infoblox-create-forwarding.py - 2878 2024-05-28 12:13:13 vi ../rotate-keys/rotate-keys.py infoblox-create-forwarding.py + 2876 2024-05-28 12:11:51 vi ../rotate-keys/rotate-keys.py infoblox-create-forwarding.py + 2877 2024-05-28 12:12:01 vi infoblox-create-forwarding.py + 2878 2024-05-28 12:13:13 vi ../rotate-keys/rotate-keys.py infoblox-create-forwarding.py 2879 2024-05-28 12:13:16 python infoblox-create-forwarding.py --help - 2880 2024-05-28 12:13:20 vi infoblox-create-forwarding.py + 2880 2024-05-28 12:13:20 vi infoblox-create-forwarding.py 2881 2024-05-28 12:13:35 python infoblox-create-forwarding.py --help - 2882 2024-05-28 12:13:47 vi infoblox-create-forwarding.py + 2882 2024-05-28 12:13:47 vi infoblox-create-forwarding.py 2883 2024-05-28 12:14:31 cat credentials.yml - 2884 2024-05-28 12:14:49 vi infoblox-create-forwarding.py + 2884 2024-05-28 12:14:49 vi infoblox-create-forwarding.py 2885 2024-05-28 12:15:51 python infoblox-create-forwarding.py --help - 2886 2024-05-28 12:15:53 vi infoblox-create-forwarding.py + 2886 2024-05-28 12:15:53 vi infoblox-create-forwarding.py 2887 2024-05-28 12:15:57 python infoblox-create-forwarding.py --help - 2888 2024-05-28 12:16:01 vi infoblox-create-forwarding.py + 2888 2024-05-28 12:16:01 vi infoblox-create-forwarding.py 2889 2024-05-28 12:25:04 python infoblox-create-forwarding.py --help - 2890 2024-05-28 12:25:10 python infoblox-create-forwarding.py - 2891 2024-05-28 12:25:20 vi infoblox-create-forwarding.py - 2892 2024-05-28 12:25:29 python infoblox-create-forwarding.py - 2893 2024-05-28 12:25:32 vi infoblox-create-forwarding.py - 2894 2024-05-28 12:25:44 python infoblox-create-forwarding.py - 2895 2024-05-28 12:25:53 vi infoblox-create-forwarding.py - 2896 2024-05-28 12:26:04 python infoblox-create-forwarding.py + 2890 2024-05-28 12:25:10 python infoblox-create-forwarding.py + 2891 2024-05-28 12:25:20 vi infoblox-create-forwarding.py + 2892 2024-05-28 12:25:29 python infoblox-create-forwarding.py + 2893 2024-05-28 12:25:32 vi infoblox-create-forwarding.py + 2894 2024-05-28 12:25:44 python infoblox-create-forwarding.py + 2895 2024-05-28 12:25:53 vi infoblox-create-forwarding.py + 2896 2024-05-28 12:26:04 python infoblox-create-forwarding.py 2897 2024-05-28 12:26:28 python infoblox-create-forwarding.py --help 2898 2024-05-28 12:27:03 python infoblox-create-forwarding.py --infoblox-nameserver-group aws-ent-gov-enterprise-dmz --infoblox-view Dmz - 2899 2024-05-28 12:27:10 vi infoblox-create-forwarding.py + 2899 2024-05-28 12:27:10 vi infoblox-create-forwarding.py 2900 2024-05-28 12:27:26 python infoblox-create-forwarding.py --infoblox-nameserver-group aws-ent-gov-enterprise-dmz --infoblox-view Dmz - 2901 2024-05-28 12:27:30 vi infoblox-create-forwarding.py + 2901 2024-05-28 12:27:30 vi infoblox-create-forwarding.py 2902 2024-05-28 12:27:46 python infoblox-create-forwarding.py --infoblox-nameserver-group aws-ent-gov-enterprise-dmz --infoblox-view Dmz 2903 2024-05-28 12:28:14 python infoblox-create-forwarding.py --infoblox-nameserver-group aws-ent-gov-enterprise --infoblox-view Dmz 2904 2024-05-28 12:28:25 python infoblox-create-forwarding.py --infoblox-nameserver-group aws-ent-gov-enterprise-dmz --infoblox-view Dmz - 2905 2024-05-28 12:28:36 vi infoblox-create-forwarding.py + 2905 2024-05-28 12:28:36 vi infoblox-create-forwarding.py 2906 2024-05-28 12:29:39 python infoblox-create-forwarding.py --infoblox-nameserver-group aws-ent-gov-enterprise-dmz --infoblox-view Dmz - 2907 2024-05-28 12:30:00 python infoblox-create-forwarding.py --infoblox-nameserver-group aws-ent-gov-enterprise-dmz --infoblox-forwader-group aws-ent-gov-enterprise --infoblox-view Dmz - 2908 2024-05-28 12:30:15 vi infoblox-create-forwarding.py - 2909 2024-05-28 12:31:17 python infoblox-create-forwarding.py --infoblox-nameserver-group aws-ent-gov-enterprise-dmz --infoblox-forwarder-group aws-ent-gov-enterprise --infoblox-view Dmz - 2910 2024-05-28 12:31:43 python infoblox-create-forwarding.py --infoblox-nameserver-group aws-ent-gov-enterprise --infoblox-forwarder-group aws-ent-gov-enterprise-dmz --infoblox-view Dmz - 2911 2024-05-28 12:34:49 vi infoblox-create-forwarding.py - 2912 2024-05-28 12:35:41 python infoblox-create-forwarding.py --infoblox-nameserver-group aws-ent-gov-enterprise --infoblox-forwarder-group aws-ent-gov-enterprise-dmz --infoblox-view Dmz - 2913 2024-05-28 12:35:48 vi infoblox-create-forwarding.py - 2914 2024-05-28 12:38:11 python infoblox-create-forwarding.py --infoblox-nameserver-group aws-ent-gov-enterprise --infoblox-forwarder-group aws-ent-gov-enterprise-dmz --infoblox-view Dmz + 2907 2024-05-28 12:30:00 python infoblox-create-forwarding.py --infoblox-nameserver-group aws-ent-gov-enterprise-dmz --infoblox-forwader-group aws-ent-gov-enterprise --infoblox-view Dmz + 2908 2024-05-28 12:30:15 vi infoblox-create-forwarding.py + 2909 2024-05-28 12:31:17 python infoblox-create-forwarding.py --infoblox-nameserver-group aws-ent-gov-enterprise-dmz --infoblox-forwarder-group aws-ent-gov-enterprise --infoblox-view Dmz + 2910 2024-05-28 12:31:43 python infoblox-create-forwarding.py --infoblox-nameserver-group aws-ent-gov-enterprise --infoblox-forwarder-group aws-ent-gov-enterprise-dmz --infoblox-view Dmz + 2911 2024-05-28 12:34:49 vi infoblox-create-forwarding.py + 2912 2024-05-28 12:35:41 python infoblox-create-forwarding.py --infoblox-nameserver-group aws-ent-gov-enterprise --infoblox-forwarder-group aws-ent-gov-enterprise-dmz --infoblox-view Dmz + 2913 2024-05-28 12:35:48 vi infoblox-create-forwarding.py + 2914 2024-05-28 12:38:11 python infoblox-create-forwarding.py --infoblox-nameserver-group aws-ent-gov-enterprise --infoblox-forwarder-group aws-ent-gov-enterprise-dmz --infoblox-view Dmz 2915 2024-05-28 12:38:18 python infoblox-create-forwarding.py --infoblox-nameserver-group aws-ent-gov-enterprise --infoblox-forwarder-group aws-ent-gov-enterprise-dmz --infoblox-view Dmz --zone stage.dice.census.gov - 2916 2024-05-28 12:38:29 vi infoblox-create-forwarding.py + 2916 2024-05-28 12:38:29 vi infoblox-create-forwarding.py 2917 2024-05-28 12:39:02 python infoblox-create-forwarding.py --infoblox-nameserver-group aws-ent-gov-enterprise --infoblox-forwarder-group aws-ent-gov-enterprise-dmz --infoblox-view Dmz --zone stage.dice.census.gov - 2918 2024-05-28 12:39:05 vi infoblox-create-forwarding.py + 2918 2024-05-28 12:39:05 vi infoblox-create-forwarding.py 2919 2024-05-28 12:39:15 python infoblox-create-forwarding.py --infoblox-nameserver-group aws-ent-gov-enterprise --infoblox-forwarder-group aws-ent-gov-enterprise-dmz --infoblox-view Dmz --zone stage.dice.census.gov - 2920 2024-05-28 12:39:19 vi infoblox-create-forwarding.py + 2920 2024-05-28 12:39:19 vi infoblox-create-forwarding.py 2921 2024-05-28 12:39:27 python infoblox-create-forwarding.py --infoblox-nameserver-group aws-ent-gov-enterprise --infoblox-forwarder-group aws-ent-gov-enterprise-dmz --infoblox-view Dmz --zone stage.dice.census.gov - 2922 2024-05-28 12:40:08 vi infoblox-create-forwarding.py + 2922 2024-05-28 12:40:08 vi infoblox-create-forwarding.py 2923 2024-05-28 12:44:56 python infoblox-create-forwarding.py --infoblox-nameserver-group aws-ent-gov-enterprise --infoblox-forwarder-group aws-ent-gov-enterprise-dmz --infoblox-view Dmz --cidr-block 10.255.0.0/19 - 2924 2024-05-28 12:45:13 vi infoblox-create-forwarding.py + 2924 2024-05-28 12:45:13 vi infoblox-create-forwarding.py 2925 2024-05-28 12:46:54 cp infoblox-create-forwarding.py infoblox-manage-forwarding.py 2926 2024-05-28 12:46:59 git co -- infoblox-create-forwarding.py - 2927 2024-05-28 12:47:04 vi infoblox-manage-forwarding.py + 2927 2024-05-28 12:47:04 vi infoblox-manage-forwarding.py 2928 2024-05-28 12:47:17 ls - 2929 2024-05-28 12:47:20 vi infoblox-delete-forwarding.py - 2930 2024-05-28 12:48:39 vi infoblox-manage-forwarding.py - 2931 2024-05-28 12:53:44 python infoblox-manage-forwarding.py --infoblox-nameserver-group aws-ent-gov-enterprise --infoblox-forwarder-group aws-ent-gov-enterprise-dmz --infoblox-view Dmz --zone stage.dice.census.gov + 2929 2024-05-28 12:47:20 vi infoblox-delete-forwarding.py + 2930 2024-05-28 12:48:39 vi infoblox-manage-forwarding.py + 2931 2024-05-28 12:53:44 python infoblox-manage-forwarding.py --infoblox-nameserver-group aws-ent-gov-enterprise --infoblox-forwarder-group aws-ent-gov-enterprise-dmz --infoblox-view Dmz --zone stage.dice.census.gov 2932 2024-05-28 12:53:52 python infoblox-manage-forwarding.py --infoblox-nameserver-group aws-ent-gov-enterprise --infoblox-forwarder-group aws-ent-gov-enterprise-dmz --infoblox-view Dmz --zone stage.dice.census.gov --action delete - 2933 2024-05-28 12:54:12 vi infoblox-manage-forwarding.py + 2933 2024-05-28 12:54:12 vi infoblox-manage-forwarding.py 2934 2024-05-28 12:54:34 python infoblox-manage-forwarding.py --infoblox-nameserver-group aws-ent-gov-enterprise --infoblox-forwarder-group aws-ent-gov-enterprise-dmz --infoblox-view Dmz --zone stage.dice.census.gov --action delete 2935 2024-05-28 12:54:54 python infoblox-manage-forwarding.py --infoblox-nameserver-group aws-ent-gov-enterprise --infoblox-forwarder-group aws-ent-gov-enterprise-dmz --infoblox-view Dmz --zone stage.dice.census.gov --action add - 2936 2024-05-28 12:55:27 vi infoblox-restart.py - 2937 2024-05-28 12:56:05 vi infoblox-manage-forwarding.py - 2938 2024-05-28 12:57:30 mv infoblox-manage-forwarding.py infoblox-manage-forwarding.py - 2939 2024-05-28 12:57:35 mv infoblox-manage-forwarding.py infoblox-manage.py - 2940 2024-05-28 12:57:41 vi infoblox-manage.py + 2936 2024-05-28 12:55:27 vi infoblox-restart.py + 2937 2024-05-28 12:56:05 vi infoblox-manage-forwarding.py + 2938 2024-05-28 12:57:30 mv infoblox-manage-forwarding.py infoblox-manage-forwarding.py + 2939 2024-05-28 12:57:35 mv infoblox-manage-forwarding.py infoblox-manage.py + 2940 2024-05-28 12:57:41 vi infoblox-manage.py 2941 2024-05-28 12:58:44 python infoblox-manage.py --help 2942 2024-05-28 12:59:00 python infoblox-manage.py --restart 2943 2024-05-28 13:01:40 pwd diff --git a/local-app/infoblox/infoblox-create-forwarding.py b/local-app/infoblox/infoblox-create-forwarding.py index 8d5369f3..b6f18f9d 100755 --- a/local-app/infoblox/infoblox-create-forwarding.py +++ b/local-app/infoblox/infoblox-create-forwarding.py @@ -1,214 +1,324 @@ #!/bin/env python -from pprint import pprint -from infoblox_client import connector -from infoblox_client import objects -import sys import os +import sys +from pprint import pprint + import urllib3 +from infoblox_client import connector, objects + urllib3.disable_warnings() -import boto3 +import argparse +import hashlib import ipaddress -#from credentials import credentials + +import boto3 + +# from credentials import credentials import yaml -import hashlib -import argparse + def read_yaml(file): - data={} - with open(file, 'r') as stream: - try: - data=yaml.full_load(stream) - except yaml.YAMLError as e: - print(e) - return None - return data - -version='1.1.0' - -ib_data=read_yaml('credentials.yml') -site=os.environ.get('INFOBLOX_SITE','network-prod').lower() -print(f'* using site {site}') -credentials=ib_data.get(site,None) + data = {} + with open(file, "r") as stream: + try: + data = yaml.full_load(stream) + except yaml.YAMLError as e: + print(e) + return None + return data + + +version = "1.1.0" + +ib_data = read_yaml("credentials.yml") +site = os.environ.get("INFOBLOX_SITE", "network-prod").lower() +print(f"* using site {site}") +credentials = ib_data.get(site, None) if credentials is None: - print(f'* no credentials found for site {site}') - sys.exit(1) + print(f"* no credentials found for site {site}") + sys.exit(1) -ib_host = credentials['host'] -ib_api_version = credentials['api_version'] -ib_username = credentials['username'] -ib_password = credentials['password'] +ib_host = credentials["host"] +ib_api_version = credentials["api_version"] +ib_username = credentials["username"] +ib_password = credentials["password"] -opts = {'host': ib_host, 'username': ib_username, 'password': ib_password} +opts = {"host": ib_host, "username": ib_username, "password": ib_password} conn = connector.Connector(opts) # get all network_views -network_views = conn.get_object('networkview') +network_views = conn.get_object("networkview") ## print('views',network_views) # search network by cidr in specific network view -network = conn.get_object('network', {'network': '10.189.0.0/16', 'network_view': 'default'}) +network = conn.get_object( + "network", {"network": "10.189.0.0/16", "network_view": "default"} +) ## print('network',network) -networkcontainer = conn.get_object('networkcontainer', {'network': '10.189.0.0/16', 'network_view': 'default'}) +networkcontainer = conn.get_object( + "networkcontainer", {"network": "10.189.0.0/16", "network_view": "default"} +) ## print('network container',networkcontainer) -#fzone = conn.get_object('zone_forward') -#print('foward zones') -#pprint(fzone) - -fzone_fields=[ - 'address', - 'comment', - 'disable', - 'disable_ns_generation', - 'display_domain', - 'dns_fqdn', - 'extattrs', - 'external_ns_group', - 'forward_to', - 'forwarders_only', - 'forwarding_servers', - 'fqdn', - 'locked', - 'locked_by', - 'mask_prefix', - 'ms_ad_integrated', - 'ms_ddns_mode', - 'ms_managed', - 'ms_read_only', - 'ms_sync_master_name', - 'ns_group', - 'parent', - 'prefix', - 'using_srg_associations', - 'view', - 'zone_format', +# fzone = conn.get_object('zone_forward') +# print('foward zones') +# pprint(fzone) + +fzone_fields = [ + "address", + "comment", + "disable", + "disable_ns_generation", + "display_domain", + "dns_fqdn", + "extattrs", + "external_ns_group", + "forward_to", + "forwarders_only", + "forwarding_servers", + "fqdn", + "locked", + "locked_by", + "mask_prefix", + "ms_ad_integrated", + "ms_ddns_mode", + "ms_managed", + "ms_read_only", + "ms_sync_master_name", + "ns_group", + "parent", + "prefix", + "using_srg_associations", + "view", + "zone_format", ] -forwarding_servers=[ - { 'forward_to':[], 'forwarders_only':False, 'name':"bcc-inf-idns01.tco.census.gov", 'use_override_forwarders':False }, - { 'forward_to':[], 'forwarders_only':False, 'name':"hq-inf-idns01.tco.census.gov", 'use_override_forwarders':False }, +forwarding_servers = [ + { + "forward_to": [], + "forwarders_only": False, + "name": "bcc-inf-idns01.tco.census.gov", + "use_override_forwarders": False, + }, + { + "forward_to": [], + "forwarders_only": False, + "name": "hq-inf-idns01.tco.census.gov", + "use_override_forwarders": False, + }, ] -forwarding_servers_dmz=[ - { 'forward_to':[], 'forwarders_only':False, 'name':"ns1e.census.gov", 'use_override_forwarders':False }, - { 'forward_to':[], 'forwarders_only':False, 'name':"ns2e.census.gov", 'use_override_forwarders':False }, +forwarding_servers_dmz = [ + { + "forward_to": [], + "forwarders_only": False, + "name": "ns1e.census.gov", + "use_override_forwarders": False, + }, + { + "forward_to": [], + "forwarders_only": False, + "name": "ns2e.census.gov", + "use_override_forwarders": False, + }, ] -forwarding_servers_lab=[ - { 'forward_to':[], 'forwarders_only':False, 'name':"vlab-hq-inf2.tco.census.gov", 'use_override_forwarders':False }, +forwarding_servers_lab = [ + { + "forward_to": [], + "forwarders_only": False, + "name": "vlab-hq-inf2.tco.census.gov", + "use_override_forwarders": False, + }, ] -fzone={ -# 'comment': 'AWS-EDL', - 'comment': os.environ.get('INFOBLOX_COMMENT','AWS'), - 'external_ns_group': 'aws-ent-gov-enterprise', - 'forward_to': [], - 'forwarders_only': False, - 'forwarding_servers': forwarding_servers, - 'zone_format': 'FORWARD', +fzone = { + # 'comment': 'AWS-EDL', + "comment": os.environ.get("INFOBLOX_COMMENT", "AWS"), + "external_ns_group": "aws-ent-gov-enterprise", + "forward_to": [], + "forwarders_only": False, + "forwarding_servers": forwarding_servers, + "zone_format": "FORWARD", } -fzone_dmz={ - 'comment': os.environ.get('INFOBLOX_COMMENT','AWS'), - 'external_ns_group': 'aws-ent-gov-enterprise-dmz', - 'forward_to': [], - 'forwarders_only': False, - 'forwarding_servers': forwarding_servers_dmz, - 'zone_format': 'FORWARD', +fzone_dmz = { + "comment": os.environ.get("INFOBLOX_COMMENT", "AWS"), + "external_ns_group": "aws-ent-gov-enterprise-dmz", + "forward_to": [], + "forwarders_only": False, + "forwarding_servers": forwarding_servers_dmz, + "zone_format": "FORWARD", } -fzone_internal_dmz={ - 'comment': os.environ.get('INFOBLOX_COMMENT','AWS'), - 'external_ns_group': 'aws-ent-gov-enterprise-dmz', - 'forward_to': [], - 'forwarders_only': False, - 'forwarding_servers': forwarding_servers, - 'zone_format': 'FORWARD', +fzone_internal_dmz = { + "comment": os.environ.get("INFOBLOX_COMMENT", "AWS"), + "external_ns_group": "aws-ent-gov-enterprise-dmz", + "forward_to": [], + "forwarders_only": False, + "forwarding_servers": forwarding_servers, + "zone_format": "FORWARD", } -fzone_lab={ - 'comment': os.environ.get('INFOBLOX_COMMENT','AWS'), - 'external_ns_group': 'aws-lab-gov-forwarder', - 'forward_to': [], - 'forwarders_only': False, - 'forwarding_servers': forwarding_servers_lab, - 'zone_format': 'FORWARD', +fzone_lab = { + "comment": os.environ.get("INFOBLOX_COMMENT", "AWS"), + "external_ns_group": "aws-lab-gov-forwarder", + "forward_to": [], + "forwarders_only": False, + "forwarding_servers": forwarding_servers_lab, + "zone_format": "FORWARD", } -if len(sys.argv)>1: - zone=sys.argv[1].replace('"','') +if len(sys.argv) > 1: + zone = sys.argv[1].replace('"', "") else: - print(f'* missing zone, skipping') - zone=None -if zone=='': - zone=None + print(f"* missing zone, skipping") + zone = None +if zone == "": + zone = None -if len(sys.argv)>2: - cidr=sys.argv[2] +if len(sys.argv) > 2: + cidr = sys.argv[2] else: - print(f'* missing cidr block, skipping') - cidr=None + print(f"* missing cidr block, skipping") + cidr = None -is_dmz=os.environ.get('INFOBLOX_DMZ','False').lower() in ('true','1','t','yes','y') -is_internal_view=os.environ.get('INFOBLOX_INTERNAL_VIEW','True').lower() in ('true','1','t','yes','y') -is_dmz_view=os.environ.get('INFOBLOX_DMZ_VIEW','False').lower() in ('true','1','t','yes','y') +is_dmz = os.environ.get("INFOBLOX_DMZ", "False").lower() in ( + "true", + "1", + "t", + "yes", + "y", +) +is_internal_view = os.environ.get("INFOBLOX_INTERNAL_VIEW", "True").lower() in ( + "true", + "1", + "t", + "yes", + "y", +) +is_dmz_view = os.environ.get("INFOBLOX_DMZ_VIEW", "False").lower() in ( + "true", + "1", + "t", + "yes", + "y", +) # to add to Dmz view (AWS internal, reachable by AWS DMZ through infoblox DMZ) # INFOBLOX_DMZ=1 INFOBLOX_DMZ_VIEW=1 INFOBLOX_INTERNAL_VIEW=0 python infoblox-create-forwarding.py ZONE -print(f'is_dmz={is_dmz} is_internal_view={is_internal_view} is_dmz_view={is_dmz_view}') +print(f"is_dmz={is_dmz} is_internal_view={is_internal_view} is_dmz_view={is_dmz_view}") # sys.exit(0) if zone is not None: - if site == 'lab-network-nonprod': - print(f'* creating forwarding zone {zone} in site {site}') - ozone = objects.DNSZoneForward.create(conn, view='Internal-LabView', fqdn=zone, check_if_exists=True, **fzone_lab) - print(ozone) - else: - if not is_dmz: - if is_internal_view: - print(f'* creating forwarding zone {zone} view internal-view in site {site}') - ozone = objects.DNSZoneForward.create(conn, view='internal-view', fqdn=zone, check_if_exists=True, **fzone) + if site == "lab-network-nonprod": + print(f"* creating forwarding zone {zone} in site {site}") + ozone = objects.DNSZoneForward.create( + conn, view="Internal-LabView", fqdn=zone, check_if_exists=True, **fzone_lab + ) print(ozone) - if is_dmz_view: - print(f'* creating forwarding zone {zone} view dmz in site {site}') - ozone = objects.DNSZoneForward.create(conn, view='Dmz', fqdn=zone, check_if_exists=True, **fzone) - print(ozone) - if is_dmz: - if is_internal_view: - print(f'* creating forwarding zone {zone} for dmz view internal-view in site {site}') - ozone = objects.DNSZoneForward.create(conn, view='internal-view', fqdn=zone, check_if_exists=True, **fzone_internal_dmz) - print(ozone) - if is_dmz_view: - print(f'* creating forwarding zone {zone} for dmz view dmz in site {site}') - ozone_dmz = objects.DNSZoneForward.create(conn, view='Dmz', fqdn=zone, check_if_exists=True, **fzone_dmz) - print(ozone_dmz) - -if cidr is not None: - fzone['zone_format']='IPV4' - fzone_dmz['zone_format']='IPV4' - fzone_internal_dmz['zone_format']='IPV4' - fzone_lab['zone_format']='IPV4' - network=ipaddress.ip_network(cidr) - for n in network.subnets(new_prefix=24): - ptr=(n[0].reverse_pointer.split('.',1))[1] - # print(f'network {n.network_address} ptr {ptr}') - zone = ptr - if site == 'lab-network-nonprod': - print(f'* creating forwarding zone {ptr} network {n} in site {site}') - ozone = objects.DNSZoneForward.create(conn, view='Internal-LabView', fqdn=n.with_prefixlen, check_if_exists=True, **fzone_lab) - print(ozone) else: - if not is_dmz: - if is_internal_view: - print(f'* creating forwarding zone {ptr} network {n} view internal-view in site {site}') - ozone = objects.DNSZoneForward.create(conn, view='internal-view', fqdn=n.with_prefixlen, check_if_exists=True, **fzone) - print(ozone) - if is_dmz_view: - print(f'* creating forwarding zone {ptr} network {n} view dmz in site {site}') - ozone = objects.DNSZoneForward.create(conn, view='Dmz', fqdn=n.with_prefixlen, check_if_exists=True, **fzone) - print(ozone) - - if is_dmz: - if is_internal_view: - print(f'* creating forwarding zone {ptr} network {n} for dmz internal-view in site {site}') - ozone = objects.DNSZoneForward.create(conn, view='internal-view', fqdn=n.with_prefixlen, check_if_exists=True, **fzone_internal_dmz) - print(ozone) - if is_dmz_view: - print(f'* creating forwarding zone {ptr} network {n} for dmz view in site {site}') - ozone_dmz = objects.DNSZoneForward.create(conn, view='Dmz', fqdn=n.with_prefixlen, check_if_exists=True, **fzone_dmz) - print(ozone_dmz) + if not is_dmz: + if is_internal_view: + print( + f"* creating forwarding zone {zone} view internal-view in site {site}" + ) + ozone = objects.DNSZoneForward.create( + conn, view="internal-view", fqdn=zone, check_if_exists=True, **fzone + ) + print(ozone) + if is_dmz_view: + print(f"* creating forwarding zone {zone} view dmz in site {site}") + ozone = objects.DNSZoneForward.create( + conn, view="Dmz", fqdn=zone, check_if_exists=True, **fzone + ) + print(ozone) + if is_dmz: + if is_internal_view: + print( + f"* creating forwarding zone {zone} for dmz view internal-view in site {site}" + ) + ozone = objects.DNSZoneForward.create( + conn, + view="internal-view", + fqdn=zone, + check_if_exists=True, + **fzone_internal_dmz, + ) + print(ozone) + if is_dmz_view: + print( + f"* creating forwarding zone {zone} for dmz view dmz in site {site}" + ) + ozone_dmz = objects.DNSZoneForward.create( + conn, view="Dmz", fqdn=zone, check_if_exists=True, **fzone_dmz + ) + print(ozone_dmz) + +if cidr is not None: + fzone["zone_format"] = "IPV4" + fzone_dmz["zone_format"] = "IPV4" + fzone_internal_dmz["zone_format"] = "IPV4" + fzone_lab["zone_format"] = "IPV4" + network = ipaddress.ip_network(cidr) + for n in network.subnets(new_prefix=24): + ptr = (n[0].reverse_pointer.split(".", 1))[1] + # print(f'network {n.network_address} ptr {ptr}') + zone = ptr + if site == "lab-network-nonprod": + print(f"* creating forwarding zone {ptr} network {n} in site {site}") + ozone = objects.DNSZoneForward.create( + conn, + view="Internal-LabView", + fqdn=n.with_prefixlen, + check_if_exists=True, + **fzone_lab, + ) + print(ozone) + else: + if not is_dmz: + if is_internal_view: + print( + f"* creating forwarding zone {ptr} network {n} view internal-view in site {site}" + ) + ozone = objects.DNSZoneForward.create( + conn, + view="internal-view", + fqdn=n.with_prefixlen, + check_if_exists=True, + **fzone, + ) + print(ozone) + if is_dmz_view: + print( + f"* creating forwarding zone {ptr} network {n} view dmz in site {site}" + ) + ozone = objects.DNSZoneForward.create( + conn, + view="Dmz", + fqdn=n.with_prefixlen, + check_if_exists=True, + **fzone, + ) + print(ozone) + + if is_dmz: + if is_internal_view: + print( + f"* creating forwarding zone {ptr} network {n} for dmz internal-view in site {site}" + ) + ozone = objects.DNSZoneForward.create( + conn, + view="internal-view", + fqdn=n.with_prefixlen, + check_if_exists=True, + **fzone_internal_dmz, + ) + print(ozone) + if is_dmz_view: + print( + f"* creating forwarding zone {ptr} network {n} for dmz view in site {site}" + ) + ozone_dmz = objects.DNSZoneForward.create( + conn, + view="Dmz", + fqdn=n.with_prefixlen, + check_if_exists=True, + **fzone_dmz, + ) + print(ozone_dmz) diff --git a/local-app/infoblox/infoblox-delete-forwarding.py b/local-app/infoblox/infoblox-delete-forwarding.py index 31a8aa32..ff3117f8 100755 --- a/local-app/infoblox/infoblox-delete-forwarding.py +++ b/local-app/infoblox/infoblox-delete-forwarding.py @@ -1,129 +1,163 @@ #!/bin/env python -from pprint import pprint -from infoblox_client import connector -from infoblox_client import objects -import sys import os +import sys +from pprint import pprint + import urllib3 +from infoblox_client import connector, objects + urllib3.disable_warnings() -import boto3 +import hashlib import ipaddress -#from credentials import credentials + +import boto3 + +# from credentials import credentials import yaml -import hashlib + def read_yaml(file): - data={} - with open(file, 'r') as stream: - try: - data=yaml.full_load(stream) - except yaml.YAMLError as e: - print(e) - return None - return data - -ib_data=read_yaml('credentials.yml') -site=os.environ.get('INFOBLOX_SITE','network-prod').lower() -print(f'* using site {site}') -credentials=ib_data.get(site,None) + data = {} + with open(file, "r") as stream: + try: + data = yaml.full_load(stream) + except yaml.YAMLError as e: + print(e) + return None + return data + + +ib_data = read_yaml("credentials.yml") +site = os.environ.get("INFOBLOX_SITE", "network-prod").lower() +print(f"* using site {site}") +credentials = ib_data.get(site, None) if credentials is None: - print(f'* no credentials found for site {site}') - sys.exit(1) + print(f"* no credentials found for site {site}") + sys.exit(1) -ib_host = credentials['host'] -ib_api_version = credentials['api_version'] -ib_username = credentials['username'] -ib_password = credentials['password'] +ib_host = credentials["host"] +ib_api_version = credentials["api_version"] +ib_username = credentials["username"] +ib_password = credentials["password"] -opts = {'host': ib_host, 'username': ib_username, 'password': ib_password} +opts = {"host": ib_host, "username": ib_username, "password": ib_password} conn = connector.Connector(opts) # get all network_views -network_views = conn.get_object('networkview') +network_views = conn.get_object("networkview") ## print('views',network_views) # search network by cidr in specific network view -network = conn.get_object('network', {'network': '10.189.0.0/16', 'network_view': 'default'}) +network = conn.get_object( + "network", {"network": "10.189.0.0/16", "network_view": "default"} +) ## print('network',network) -networkcontainer = conn.get_object('networkcontainer', {'network': '10.189.0.0/16', 'network_view': 'default'}) +networkcontainer = conn.get_object( + "networkcontainer", {"network": "10.189.0.0/16", "network_view": "default"} +) ## print('network container',networkcontainer) -#fzone = conn.get_object('zone_forward') -#print('foward zones') -#pprint(fzone) - -fzone_fields=[ - 'address', - 'comment', - 'disable', - 'disable_ns_generation', - 'display_domain', - 'dns_fqdn', - 'extattrs', - 'external_ns_group', - 'forward_to', - 'forwarders_only', - 'forwarding_servers', - 'fqdn', - 'locked', - 'locked_by', - 'mask_prefix', - 'ms_ad_integrated', - 'ms_ddns_mode', - 'ms_managed', - 'ms_read_only', - 'ms_sync_master_name', - 'ns_group', - 'parent', - 'prefix', - 'using_srg_associations', - 'view', - 'zone_format', +# fzone = conn.get_object('zone_forward') +# print('foward zones') +# pprint(fzone) + +fzone_fields = [ + "address", + "comment", + "disable", + "disable_ns_generation", + "display_domain", + "dns_fqdn", + "extattrs", + "external_ns_group", + "forward_to", + "forwarders_only", + "forwarding_servers", + "fqdn", + "locked", + "locked_by", + "mask_prefix", + "ms_ad_integrated", + "ms_ddns_mode", + "ms_managed", + "ms_read_only", + "ms_sync_master_name", + "ns_group", + "parent", + "prefix", + "using_srg_associations", + "view", + "zone_format", ] -forwarding_servers=[ - { 'forward_to':[], 'forwarders_only':False, 'name':"bcc-inf-idns01.tco.census.gov", 'use_override_forwarders':False }, - { 'forward_to':[], 'forwarders_only':False, 'name':"hq-inf-idns01.tco.census.gov", 'use_override_forwarders':False }, +forwarding_servers = [ + { + "forward_to": [], + "forwarders_only": False, + "name": "bcc-inf-idns01.tco.census.gov", + "use_override_forwarders": False, + }, + { + "forward_to": [], + "forwarders_only": False, + "name": "hq-inf-idns01.tco.census.gov", + "use_override_forwarders": False, + }, ] -forwarding_servers_dmz=[ - { 'forward_to':[], 'forwarders_only':False, 'name':"ns1e.census.gov", 'use_override_forwarders':False }, - { 'forward_to':[], 'forwarders_only':False, 'name':"ns2e.census.gov", 'use_override_forwarders':False }, +forwarding_servers_dmz = [ + { + "forward_to": [], + "forwarders_only": False, + "name": "ns1e.census.gov", + "use_override_forwarders": False, + }, + { + "forward_to": [], + "forwarders_only": False, + "name": "ns2e.census.gov", + "use_override_forwarders": False, + }, ] -forwarding_servers_lab=[ - { 'forward_to':[], 'forwarders_only':False, 'name':"vlab-hq-inf2.tco.census.gov", 'use_override_forwarders':False }, +forwarding_servers_lab = [ + { + "forward_to": [], + "forwarders_only": False, + "name": "vlab-hq-inf2.tco.census.gov", + "use_override_forwarders": False, + }, ] -fzone={ -# 'comment': 'AWS-EDL', - 'comment': os.environ.get('INFOBLOX_COMMENT','AWS'), - 'external_ns_group': 'aws-ent-gov-enterprise', - 'forward_to': [], - 'forwarders_only': False, - 'forwarding_servers': forwarding_servers, - 'zone_format': 'FORWARD', +fzone = { + # 'comment': 'AWS-EDL', + "comment": os.environ.get("INFOBLOX_COMMENT", "AWS"), + "external_ns_group": "aws-ent-gov-enterprise", + "forward_to": [], + "forwarders_only": False, + "forwarding_servers": forwarding_servers, + "zone_format": "FORWARD", } -fzone_dmz={ - 'comment': os.environ.get('INFOBLOX_COMMENT','AWS'), - 'external_ns_group': 'aws-ent-gov-enterprise-dmz', - 'forward_to': [], - 'forwarders_only': False, - 'forwarding_servers': forwarding_servers_dmz, - 'zone_format': 'FORWARD', +fzone_dmz = { + "comment": os.environ.get("INFOBLOX_COMMENT", "AWS"), + "external_ns_group": "aws-ent-gov-enterprise-dmz", + "forward_to": [], + "forwarders_only": False, + "forwarding_servers": forwarding_servers_dmz, + "zone_format": "FORWARD", } -fzone_internal_dmz={ - 'comment': os.environ.get('INFOBLOX_COMMENT','AWS'), - 'external_ns_group': 'aws-ent-gov-enterprise-dmz', - 'forward_to': [], - 'forwarders_only': False, - 'forwarding_servers': forwarding_servers, - 'zone_format': 'FORWARD', +fzone_internal_dmz = { + "comment": os.environ.get("INFOBLOX_COMMENT", "AWS"), + "external_ns_group": "aws-ent-gov-enterprise-dmz", + "forward_to": [], + "forwarders_only": False, + "forwarding_servers": forwarding_servers, + "zone_format": "FORWARD", } -fzone_lab={ - 'comment': os.environ.get('INFOBLOX_COMMENT','AWS'), - 'external_ns_group': 'aws-lab-gov-forwarder', - 'forward_to': [], - 'forwarders_only': False, - 'forwarding_servers': forwarding_servers_lab, - 'zone_format': 'FORWARD', +fzone_lab = { + "comment": os.environ.get("INFOBLOX_COMMENT", "AWS"), + "external_ns_group": "aws-lab-gov-forwarder", + "forward_to": [], + "forwarders_only": False, + "forwarding_servers": forwarding_servers_lab, + "zone_format": "FORWARD", } ## if True: @@ -132,95 +166,149 @@ def read_yaml(file): ## pprint(fwd_zone) ## print('vars(fwd_zone)') ## pprint(vars(fwd_zone)) -## +## ## if zone is not None and fwd_zone is not None: ## print(f'* deleting forwarding zone {zone}') ## r = fwd_zone.delete() ## print(f'zone deletion status {r}') -if len(sys.argv)>1: - zone=sys.argv[1].replace('"','') +if len(sys.argv) > 1: + zone = sys.argv[1].replace('"', "") else: - print(f'* missing zone, skipping') - zone=None -if zone=='': - zone=None + print(f"* missing zone, skipping") + zone = None +if zone == "": + zone = None -if len(sys.argv)>2: - cidr=sys.argv[2] +if len(sys.argv) > 2: + cidr = sys.argv[2] else: - print(f'* missing cidr block, skipping') - cidr=None + print(f"* missing cidr block, skipping") + cidr = None -is_dmz=os.environ.get('INFOBLOX_DMZ','False').lower() in ('true','1','t','yes','y') -is_internal_view=os.environ.get('INFOBLOX_INTERNAL_VIEW','True').lower() in ('true','1','t','yes','y') -is_dmz_view=os.environ.get('INFOBLOX_DMZ_VIEW','False').lower() in ('true','1','t','yes','y') +is_dmz = os.environ.get("INFOBLOX_DMZ", "False").lower() in ( + "true", + "1", + "t", + "yes", + "y", +) +is_internal_view = os.environ.get("INFOBLOX_INTERNAL_VIEW", "True").lower() in ( + "true", + "1", + "t", + "yes", + "y", +) +is_dmz_view = os.environ.get("INFOBLOX_DMZ_VIEW", "False").lower() in ( + "true", + "1", + "t", + "yes", + "y", +) -print(f'is_dmz={is_dmz} is_internal_view={is_internal_view} is_dmz_view={is_dmz_view}') +print(f"is_dmz={is_dmz} is_internal_view={is_internal_view} is_dmz_view={is_dmz_view}") # sys.exit(0) if zone is not None: - if site == 'lab-network-nonprod': - print(f'* deleting forwarding zone {zone} in site {site}') - ozone = objects.DNSZoneForward.search(conn, view='Internal-LabView', fqdn=zone, return_fields=fzone_fields) - print(ozone) - r = ozone.delete() - print(f'* zone {zone} site {site} deletion status {r}') - else: - if not is_dmz: - print(f'* deleting forwarding zone {zone} in site {site}') - ozone = objects.DNSZoneForward.search(conn, view='internal-view', fqdn=zone, return_fields=fzone_fields) - print(ozone) - r = ozone.delete() - print(f'* zone {zone} site {site} deletion status {r}') - if is_dmz: - if is_internal_view: - print(f'* deleting forwarding zone {zone} for dmz view internal-view in site {site}') - ozone = objects.DNSZoneForward.search(conn, view='internal-view', fqdn=zone, return_fields=fzone_fields) + if site == "lab-network-nonprod": + print(f"* deleting forwarding zone {zone} in site {site}") + ozone = objects.DNSZoneForward.search( + conn, view="Internal-LabView", fqdn=zone, return_fields=fzone_fields + ) print(ozone) r = ozone.delete() - print(f'* zone {zone} site {site} deletion status {r}') - if is_dmz_view: - print(f'* deleting forwarding zone {zone} for dmz view dmz in site {site}') - ozone_dmz = objects.DNSZoneForward.search(conn, view='Dmz', fqdn=zone, return_fields=fzone_fields) - print(ozone_dmz) - r = ozone_dmz.delete() - print(f'* zone {zone} site {site} deletion status {r}') - -if cidr is not None: - fzone['zone_format']='IPV4' - fzone_dmz['zone_format']='IPV4' - fzone_internal_dmz['zone_format']='IPV4' - fzone_lab['zone_format']='IPV4' - network=ipaddress.ip_network(cidr) - for n in network.subnets(new_prefix=24): - ptr=(n[0].reverse_pointer.split('.',1))[1] - # print(f'network {n.network_address} ptr {ptr}') - zone = ptr - if site == 'lab-network-nonprod': - print(f'* deleting forwarding zone {ptr} network {n} in site {site}') - ozone = objects.DNSZoneForward.search(conn, view='Internal-LabView', fqdn=n.with_prefixlen, return_fields=fzone_fields) - print(ozone) - r = ozone.delete() - print(f'* zone {ptr} network {n} site {site} deletion status {r}') + print(f"* zone {zone} site {site} deletion status {r}") else: - if not is_dmz: - print(f'* deleting forwarding zone {ptr} network {n} in site {site}') - ozone = objects.DNSZoneForward.search(conn, view='internal-view', fqdn=n.with_prefixlen, return_fields=fzone_fields) - print(ozone) - r = ozone.delete() - print(f'* zone {ptr} network {n} site {site} deletion status {r}') - - if is_dmz: - if is_internal_view: - print(f'* deleting forwarding zone {ptr} network {n} for dmz internal-view in site {site}') - ozone = objects.DNSZoneForward.search(conn, view='internal-view', fqdn=n.with_prefixlen, return_fields=fzone_fields) - print(ozone) - r = ozone.delete() - print(f'* zone {ptr} network {n} site {site} deletion status {r}') - if is_dmz_view: - print(f'* deleting forwarding zone {ptr} network {n} for dmz view in site {site}') - ozone_dmz = objects.DNSZoneForward.search(conn, view='Dmz', fqdn=n.with_prefixlen, return_fields=fzone_fields) - print(ozone_dmz) - r = ozone_dmz.delete() - print(f'* zone {ptr} network {n} site {site} deletion status {r}') + if not is_dmz: + print(f"* deleting forwarding zone {zone} in site {site}") + ozone = objects.DNSZoneForward.search( + conn, view="internal-view", fqdn=zone, return_fields=fzone_fields + ) + print(ozone) + r = ozone.delete() + print(f"* zone {zone} site {site} deletion status {r}") + if is_dmz: + if is_internal_view: + print( + f"* deleting forwarding zone {zone} for dmz view internal-view in site {site}" + ) + ozone = objects.DNSZoneForward.search( + conn, view="internal-view", fqdn=zone, return_fields=fzone_fields + ) + print(ozone) + r = ozone.delete() + print(f"* zone {zone} site {site} deletion status {r}") + if is_dmz_view: + print( + f"* deleting forwarding zone {zone} for dmz view dmz in site {site}" + ) + ozone_dmz = objects.DNSZoneForward.search( + conn, view="Dmz", fqdn=zone, return_fields=fzone_fields + ) + print(ozone_dmz) + r = ozone_dmz.delete() + print(f"* zone {zone} site {site} deletion status {r}") + +if cidr is not None: + fzone["zone_format"] = "IPV4" + fzone_dmz["zone_format"] = "IPV4" + fzone_internal_dmz["zone_format"] = "IPV4" + fzone_lab["zone_format"] = "IPV4" + network = ipaddress.ip_network(cidr) + for n in network.subnets(new_prefix=24): + ptr = (n[0].reverse_pointer.split(".", 1))[1] + # print(f'network {n.network_address} ptr {ptr}') + zone = ptr + if site == "lab-network-nonprod": + print(f"* deleting forwarding zone {ptr} network {n} in site {site}") + ozone = objects.DNSZoneForward.search( + conn, + view="Internal-LabView", + fqdn=n.with_prefixlen, + return_fields=fzone_fields, + ) + print(ozone) + r = ozone.delete() + print(f"* zone {ptr} network {n} site {site} deletion status {r}") + else: + if not is_dmz: + print(f"* deleting forwarding zone {ptr} network {n} in site {site}") + ozone = objects.DNSZoneForward.search( + conn, + view="internal-view", + fqdn=n.with_prefixlen, + return_fields=fzone_fields, + ) + print(ozone) + r = ozone.delete() + print(f"* zone {ptr} network {n} site {site} deletion status {r}") + + if is_dmz: + if is_internal_view: + print( + f"* deleting forwarding zone {ptr} network {n} for dmz internal-view in site {site}" + ) + ozone = objects.DNSZoneForward.search( + conn, + view="internal-view", + fqdn=n.with_prefixlen, + return_fields=fzone_fields, + ) + print(ozone) + r = ozone.delete() + print(f"* zone {ptr} network {n} site {site} deletion status {r}") + if is_dmz_view: + print( + f"* deleting forwarding zone {ptr} network {n} for dmz view in site {site}" + ) + ozone_dmz = objects.DNSZoneForward.search( + conn, + view="Dmz", + fqdn=n.with_prefixlen, + return_fields=fzone_fields, + ) + print(ozone_dmz) + r = ozone_dmz.delete() + print(f"* zone {ptr} network {n} site {site} deletion status {r}") diff --git a/local-app/infoblox/infoblox-manage.py b/local-app/infoblox/infoblox-manage.py index e1b454d2..62165ac1 100755 --- a/local-app/infoblox/infoblox-manage.py +++ b/local-app/infoblox/infoblox-manage.py @@ -1,229 +1,375 @@ #!/bin/env python -from pprint import pprint -from infoblox_client import connector -from infoblox_client import objects -from infoblox_client import utils -import sys import os +import sys +from pprint import pprint + import urllib3 +from infoblox_client import connector, objects, utils + urllib3.disable_warnings() -import boto3 +import argparse +import hashlib import ipaddress -#from credentials import credentials + +import boto3 + +# from credentials import credentials import yaml -import hashlib -import argparse + def read_yaml(file): - data={} - with open(file, 'r') as stream: - try: - data=yaml.full_load(stream) - except yaml.YAMLError as e: - print(e) - return None - return data + data = {} + with open(file, "r") as stream: + try: + data = yaml.full_load(stream) + except yaml.YAMLError as e: + print(e) + return None + return data + def parse_arguments(version): - parser = argparse.ArgumentParser(description="Manage Infoblox Forwarder Zones",add_help=True) - parser.add_argument('--version', action='version', version='%(prog)s v'+version) - parser.add_argument("-v","--verbose", action="store_true", dest="verbose", help="verbose output", default=False) - - parser.add_argument("-a","--action", action="store", dest="action", help="Infoblox Zone Action (d: add)", - choices=['add','delete','search','update'], - default='add'), - parser.add_argument("-r","--restart", action="store_true", dest="restart", help="Infoblox Restart DNS (d: False)",default=False) - - parser.add_argument("-z","--zone", action="store", dest="zone", help="DNS Zone Name") - parser.add_argument("-c","--cidr", action="store", dest="cidr_block", help="CIDR Block") - - parser.add_argument("-N","--infoblox-nameserver-group", action="store", dest="nameserver_group", help="Infoblox Nameserver Group (d: aws-ent-gov-enterprise)", - choices=['aws-ent-gov-enterprise','aws-ent-gov-enterprise-dmz','aws-lab-gov-forwarder'], - default="aws-ent-gov-enterprise") - parser.add_argument("-F","--infoblox-forwarder-group", action="store", dest="forwarder_group", help="Infoblox Forwarder Group (d: aws-ent-gov-enterprise)", - choices=['aws-ent-gov-enterprise','aws-ent-gov-enterprise-dmz','aws-lab-gov-forwarder'], - default="aws-ent-gov-enterprise") - parser.add_argument("-V","--infoblox-view", action="store", dest="view", help="Infoblox DNS View (d: internal-view)", - choices=['internal-view','Dmz','Public','Internal-LabView'], - default='internal-view') - parser.add_argument("-S","--infoblox-site", action="store", dest="site", help="Infoblox site for AWS Network Account (d: network-prod)", - choices=['network-prod', 'dmz-network-prod', 'lab-network-nonprod'], - default='network-prod') - parser.add_argument("-C","--infoblox-comment", action="store", dest="comment", help="Infoblox Comment (d: AWS)", default="AWS") - - args = parser.parse_args() - return args + parser = argparse.ArgumentParser( + description="Manage Infoblox Forwarder Zones", add_help=True + ) + parser.add_argument("--version", action="version", version="%(prog)s v" + version) + parser.add_argument( + "-v", + "--verbose", + action="store_true", + dest="verbose", + help="verbose output", + default=False, + ) + + parser.add_argument( + "-a", + "--action", + action="store", + dest="action", + help="Infoblox Zone Action (d: add)", + choices=["add", "delete", "search", "update"], + default="add", + ), + parser.add_argument( + "-r", + "--restart", + action="store_true", + dest="restart", + help="Infoblox Restart DNS (d: False)", + default=False, + ) + + parser.add_argument( + "-z", "--zone", action="store", dest="zone", help="DNS Zone Name" + ) + parser.add_argument( + "-c", "--cidr", action="store", dest="cidr_block", help="CIDR Block" + ) + + parser.add_argument( + "-N", + "--infoblox-nameserver-group", + action="store", + dest="nameserver_group", + help="Infoblox Nameserver Group (d: aws-ent-gov-enterprise)", + choices=[ + "aws-ent-gov-enterprise", + "aws-ent-gov-enterprise-dmz", + "aws-lab-gov-forwarder", + ], + default="aws-ent-gov-enterprise", + ) + parser.add_argument( + "-F", + "--infoblox-forwarder-group", + action="store", + dest="forwarder_group", + help="Infoblox Forwarder Group (d: aws-ent-gov-enterprise)", + choices=[ + "aws-ent-gov-enterprise", + "aws-ent-gov-enterprise-dmz", + "aws-lab-gov-forwarder", + ], + default="aws-ent-gov-enterprise", + ) + parser.add_argument( + "-V", + "--infoblox-view", + action="store", + dest="view", + help="Infoblox DNS View (d: internal-view)", + choices=["internal-view", "Dmz", "Public", "Internal-LabView"], + default="internal-view", + ) + parser.add_argument( + "-S", + "--infoblox-site", + action="store", + dest="site", + help="Infoblox site for AWS Network Account (d: network-prod)", + choices=["network-prod", "dmz-network-prod", "lab-network-nonprod"], + default="network-prod", + ) + parser.add_argument( + "-C", + "--infoblox-comment", + action="store", + dest="comment", + help="Infoblox Comment (d: AWS)", + default="AWS", + ) + + args = parser.parse_args() + return args + def main(): - version='2.1.0' - this=os.path.basename(sys.argv[0]) - print("# %s v%s" % (this,version)) - args = parse_arguments(version) - - ib_data=read_yaml('credentials.yml') - site=args.site - print(f'* using site {site}') - - credentials=ib_data.get(site,None) - if credentials is None: - print(f'* no credentials found for site {site}') - sys.exit(1) - - ib_host = credentials['host'] - ib_api_version = credentials['api_version'] - ib_username = credentials['username'] - ib_password = credentials['password'] - - opts = {'host': ib_host, 'username': ib_username, 'password': ib_password} - conn = connector.Connector(opts) - - fzone_fields=[ - 'address', - 'comment', - 'disable', - 'disable_ns_generation', - 'display_domain', - 'dns_fqdn', - 'extattrs', - 'external_ns_group', - 'forward_to', - 'forwarders_only', - 'forwarding_servers', - 'fqdn', - 'locked', - 'locked_by', - 'mask_prefix', - 'ms_ad_integrated', - 'ms_ddns_mode', - 'ms_managed', - 'ms_read_only', - 'ms_sync_master_name', - 'ns_group', - 'parent', - 'prefix', - 'using_srg_associations', - 'view', - 'zone_format', - ] - - ib_forwarding_servers={ - 'aws-ent-gov-enterprise': [ - { 'forward_to':[], 'forwarders_only':False, 'name':"bcc-inf-idns01.tco.census.gov", 'use_override_forwarders':False }, - { 'forward_to':[], 'forwarders_only':False, 'name':"hq-inf-idns01.tco.census.gov", 'use_override_forwarders':False }, - { 'forward_to':[], 'forwarders_only':False, 'name':"npc-inf-ns1.tco.census.gov", 'use_override_forwarders':False }, - ], - 'aws-ent-gov-enterprise-dmz': [ - { 'forward_to':[], 'forwarders_only':False, 'name':"ns1e.census.gov", 'use_override_forwarders':False }, - { 'forward_to':[], 'forwarders_only':False, 'name':"ns2e.census.gov", 'use_override_forwarders':False }, - ], - 'aws-lab-gov-forwarder': [ - { 'forward_to':[], 'forwarders_only':False, 'name':"vlab-hq-inf2.tco.census.gov", 'use_override_forwarders':False }, - ], - } - - ib_forwarding_config={ - 'comment': args.comment, - 'external_ns_group': args.nameserver_group, - 'forward_to': [], - 'forwarders_only': False, - 'forwarding_servers': ib_forwarding_servers[args.forwarder_group], - 'zone_format': 'FORWARD', - } - - zone=args.zone - cidr=args.cidr_block - -# pprint(args) -# pprint(ib_forwarding_config) - - if args.action == 'search': + version = "2.1.0" + this = os.path.basename(sys.argv[0]) + print("# %s v%s" % (this, version)) + args = parse_arguments(version) + + ib_data = read_yaml("credentials.yml") + site = args.site + print(f"* using site {site}") + + credentials = ib_data.get(site, None) + if credentials is None: + print(f"* no credentials found for site {site}") + sys.exit(1) + + ib_host = credentials["host"] + ib_api_version = credentials["api_version"] + ib_username = credentials["username"] + ib_password = credentials["password"] + + opts = {"host": ib_host, "username": ib_username, "password": ib_password} + conn = connector.Connector(opts) + + fzone_fields = [ + "address", + "comment", + "disable", + "disable_ns_generation", + "display_domain", + "dns_fqdn", + "extattrs", + "external_ns_group", + "forward_to", + "forwarders_only", + "forwarding_servers", + "fqdn", + "locked", + "locked_by", + "mask_prefix", + "ms_ad_integrated", + "ms_ddns_mode", + "ms_managed", + "ms_read_only", + "ms_sync_master_name", + "ns_group", + "parent", + "prefix", + "using_srg_associations", + "view", + "zone_format", + ] + + ib_forwarding_servers = { + "aws-ent-gov-enterprise": [ + { + "forward_to": [], + "forwarders_only": False, + "name": "bcc-inf-idns01.tco.census.gov", + "use_override_forwarders": False, + }, + { + "forward_to": [], + "forwarders_only": False, + "name": "hq-inf-idns01.tco.census.gov", + "use_override_forwarders": False, + }, + { + "forward_to": [], + "forwarders_only": False, + "name": "npc-inf-ns1.tco.census.gov", + "use_override_forwarders": False, + }, + ], + "aws-ent-gov-enterprise-dmz": [ + { + "forward_to": [], + "forwarders_only": False, + "name": "ns1e.census.gov", + "use_override_forwarders": False, + }, + { + "forward_to": [], + "forwarders_only": False, + "name": "ns2e.census.gov", + "use_override_forwarders": False, + }, + ], + "aws-lab-gov-forwarder": [ + { + "forward_to": [], + "forwarders_only": False, + "name": "vlab-hq-inf2.tco.census.gov", + "use_override_forwarders": False, + }, + ], + } + + ib_forwarding_config = { + "comment": args.comment, + "external_ns_group": args.nameserver_group, + "forward_to": [], + "forwarders_only": False, + "forwarding_servers": ib_forwarding_servers[args.forwarder_group], + "zone_format": "FORWARD", + } + + zone = args.zone + cidr = args.cidr_block + + # pprint(args) + # pprint(ib_forwarding_config) + + if args.action == "search": + if zone is not None: + print(f"* searching zone {zone} site {args.site} for view {args.view}") + fwd_zones = objects.DNSZoneForward.search( + conn, view=args.view, fqdn=zone, return_fields=fzone_fields + ) + pprint(fwd_zones) + else: + print(f"* searching all zones site {args.site} for view {args.view}") + fwd_zones = objects.DNSZoneForward.search_all( + conn, view=args.view, paging=True, return_fields=fzone_fields + ) + p = 0 + c = 0 + for page in utils.paging(fwd_zones, max_results=100): + p += 1 + cc = len(page) + c += cc + print(f"# page {p} items {cc} total {c}") + # pprint(page) + for i in page: + print( + f"zone {i.display_domain} domain {i.dns_fqdn} view {i.view} ns_group {i.external_ns_group} type {i.zone_format} disable {i.disable}" + ) + print(f"# total page {p} total items {c}") + sys.exit(0) + + # [DNSZoneForward: disable="False", disable_ns_generation="False", display_domain="prod.das.rm.census.gov", dns_fqdn="prod.das.rm.census.gov", external_ns_group="aws-ent-gov-enterprise", forward_to="[]", forwarders_only="False", forwarding_servers="[Forwardingmemberserver: forward_to="[]", forwarders_only="False", name="bcc-inf-idns01.tco.census.gov", use_override_forwarders="False", Forwardingmemberserver: forward_to="[]", forwarders_only="False", name="hq-inf-idns01.tco.census.gov", use_override_forwarders="False"]", fqdn="prod.das.rm.census.gov", locked="False", ms_ad_integrated="False", ms_ddns_mode="NONE", ms_managed="NONE", ms_read_only="False", parent="rm.census.gov", using_srg_associations="False", view="internal-view", zone_format="FORWARD", _ref="zone_forward/ZG5zLnpvbmUkLl9kZWZhdWx0Lmdvdi5jZW5zdXMucm0uZGFzLnByb2Q:prod.das.rm.census.gov/internal-view", + if zone is not None: - print(f'* searching zone {zone} site {args.site} for view {args.view}') - fwd_zones = objects.DNSZoneForward.search(conn, view=args.view, fqdn=zone, return_fields=fzone_fields) - pprint(fwd_zones) - else: - print(f'* searching all zones site {args.site} for view {args.view}') - fwd_zones = objects.DNSZoneForward.search_all(conn, view=args.view, paging=True, return_fields=fzone_fields) - p=0 - c=0 - for page in utils.paging(fwd_zones, max_results=100): - p+=1 - cc=len(page) - c+=cc - print(f'# page {p} items {cc} total {c}') -# pprint(page) - for i in page: - print(f'zone {i.display_domain} domain {i.dns_fqdn} view {i.view} ns_group {i.external_ns_group} type {i.zone_format} disable {i.disable}') - print(f'# total page {p} total items {c}') + f_zone = ib_forwarding_config.copy() + if args.action == "update": + del f_zone["zone_format"] + print( + f"* updating forwarding zone {zone} site {args.site} for nameserver_group {args.nameserver_group} view {args.view} forwarder_group {args.forwarder_group}" + ) + ozone = objects.DNSZoneForward.create( + conn, + view=args.view, + fqdn=zone, + check_if_exists=True, + update_if_exists=True, + **f_zone, + ) + print(ozone) + elif args.action == "add": + print( + f"* creating forwarding zone {zone} site {args.site} for nameserver_group {args.nameserver_group} view {args.view} forwarder_group {args.forwarder_group}" + ) + ozone = objects.DNSZoneForward.create( + conn, view=args.view, fqdn=zone, check_if_exists=True, **f_zone + ) + print(ozone) + elif args.action == "delete": + print( + f"* deleting forwarding zone {zone} site {args.site} for nameserver_group {args.nameserver_group} view {args.view} forwarder_group {args.forwarder_group}" + ) + ozone = objects.DNSZoneForward.search( + conn, view=args.view, fqdn=zone, return_fields=fzone_fields + ) + print(ozone) + r = ozone.delete() + print(f"* zone {zone} site {args.site} deletion status {r}") + + if cidr is not None: + f_zone = ib_forwarding_config.copy() + f_zone["zone_format"] = "IPV4" + if args.action == "update": + del f_zone["zone_format"] + network = ipaddress.ip_network(cidr) + for n in network.subnets(new_prefix=24): + ptr = (n[0].reverse_pointer.split(".", 1))[1] + ptr_zone = n.with_prefixlen + if args.action == "update": + print( + f"* updating forwarding PTR zone {ptr} site {args.site} for nameserver_group {args.nameserver_group} view {args.view} forwarder_group {args.forwarder_group}" + ) + ozone = objects.DNSZoneForward.create( + conn, + view=args.view, + fqdn=ptr_zone, + check_if_exists=True, + update_if_exists=True, + **f_zone, + ) + print(ozone) + elif args.action == "add": + print( + f"* creating forwarding PTR zone {ptr} site {args.site} for nameserver_group {args.nameserver_group} view {args.view} forwarder_group {args.forwarder_group}" + ) + ozone = objects.DNSZoneForward.create( + conn, view=args.view, fqdn=ptr_zone, check_if_exists=True, **f_zone + ) + print(ozone) + elif args.action == "delete": + print( + f"* deleting forwarding PTR zone {ptr} site {args.site} for nameserver_group {args.nameserver_group} view {args.view} forwarder_group {args.forwarder_group}" + ) + ozone = objects.DNSZoneForward.search( + conn, view=args.view, fqdn=ptr_zone, return_fields=fzone_fields + ) + print(ozone) + r = ozone.delete() + print(f"* zone {ptr_zone} site {args.site} deletion status {r}") + + if args.action: + print(f"* restarting Infoblox using site {site} grid host {ib_host}") + grid = objects.Grid.search(conn) + if grid is not None: + gstatus = grid.requestrestartservicestatus({"service_option": "ALL"}) + gridrs = objects.Restartservicestatus.search(conn) + print(f"dns_status={gridrs.dns_status}") + if gridrs.dns_status == "REQUESTING": + status = grid.restartservices( + { + "restart_option": "RESTART_IF_NEEDED", + "service_option": "ALL", + "member_order": "SEQUENTIALLY", + "sequential_delay": 1, + "user_name": ib_username, + } + ) + print("restart status") + pprint(status) + sys.exit(0) -# [DNSZoneForward: disable="False", disable_ns_generation="False", display_domain="prod.das.rm.census.gov", dns_fqdn="prod.das.rm.census.gov", external_ns_group="aws-ent-gov-enterprise", forward_to="[]", forwarders_only="False", forwarding_servers="[Forwardingmemberserver: forward_to="[]", forwarders_only="False", name="bcc-inf-idns01.tco.census.gov", use_override_forwarders="False", Forwardingmemberserver: forward_to="[]", forwarders_only="False", name="hq-inf-idns01.tco.census.gov", use_override_forwarders="False"]", fqdn="prod.das.rm.census.gov", locked="False", ms_ad_integrated="False", ms_ddns_mode="NONE", ms_managed="NONE", ms_read_only="False", parent="rm.census.gov", using_srg_associations="False", view="internal-view", zone_format="FORWARD", _ref="zone_forward/ZG5zLnpvbmUkLl9kZWZhdWx0Lmdvdi5jZW5zdXMucm0uZGFzLnByb2Q:prod.das.rm.census.gov/internal-view", - - - if zone is not None: - f_zone=ib_forwarding_config.copy() - if args.action == 'update': - del f_zone['zone_format'] - print(f'* updating forwarding zone {zone} site {args.site} for nameserver_group {args.nameserver_group} view {args.view} forwarder_group {args.forwarder_group}') - ozone = objects.DNSZoneForward.create(conn, view=args.view, fqdn=zone, check_if_exists=True, update_if_exists=True, **f_zone) - print(ozone) - elif args.action == 'add': - print(f'* creating forwarding zone {zone} site {args.site} for nameserver_group {args.nameserver_group} view {args.view} forwarder_group {args.forwarder_group}') - ozone = objects.DNSZoneForward.create(conn, view=args.view, fqdn=zone, check_if_exists=True, **f_zone) - print(ozone) - elif args.action == 'delete': - print(f'* deleting forwarding zone {zone} site {args.site} for nameserver_group {args.nameserver_group} view {args.view} forwarder_group {args.forwarder_group}') - ozone = objects.DNSZoneForward.search(conn, view=args.view, fqdn=zone, return_fields=fzone_fields) - print(ozone) - r = ozone.delete() - print(f'* zone {zone} site {args.site} deletion status {r}') - - - if cidr is not None: - f_zone=ib_forwarding_config.copy() - f_zone['zone_format']='IPV4' - if args.action == 'update': - del f_zone['zone_format'] - network=ipaddress.ip_network(cidr) - for n in network.subnets(new_prefix=24): - ptr=(n[0].reverse_pointer.split('.',1))[1] - ptr_zone=n.with_prefixlen - if args.action == 'update': - print(f'* updating forwarding PTR zone {ptr} site {args.site} for nameserver_group {args.nameserver_group} view {args.view} forwarder_group {args.forwarder_group}') - ozone = objects.DNSZoneForward.create(conn, view=args.view, fqdn=ptr_zone, check_if_exists=True, update_if_exists=True, **f_zone) - print(ozone) - elif args.action == 'add': - print(f'* creating forwarding PTR zone {ptr} site {args.site} for nameserver_group {args.nameserver_group} view {args.view} forwarder_group {args.forwarder_group}') - ozone = objects.DNSZoneForward.create(conn, view=args.view, fqdn=ptr_zone, check_if_exists=True, **f_zone) - print(ozone) - elif args.action == 'delete': - print(f'* deleting forwarding PTR zone {ptr} site {args.site} for nameserver_group {args.nameserver_group} view {args.view} forwarder_group {args.forwarder_group}') - ozone = objects.DNSZoneForward.search(conn, view=args.view, fqdn=ptr_zone, return_fields=fzone_fields) - print(ozone) - r = ozone.delete() - print(f'* zone {ptr_zone} site {args.site} deletion status {r}') - - if args.action: - print(f'* restarting Infoblox using site {site} grid host {ib_host}') - grid = objects.Grid.search(conn) - if grid is not None: - gstatus = grid.requestrestartservicestatus({'service_option':'ALL'}) - gridrs = objects.Restartservicestatus.search(conn) - print(f'dns_status={gridrs.dns_status}') - if gridrs.dns_status=='REQUESTING': - status = grid.restartservices({'restart_option': 'RESTART_IF_NEEDED', 'service_option': 'ALL', 'member_order': 'SEQUENTIALLY', 'sequential_delay': 1, 'user_name': ib_username}) - print('restart status') - pprint(status) - - sys.exit(0) - -#--- +# --- # main -#--- -if __name__ == '__main__': - main() +# --- +if __name__ == "__main__": + main() ## 2917 2024-05-28 12:39:02 python infoblox-create-forwarding.py --infoblox-nameserver-group aws-ent-gov-enterprise --infoblox-forwarder-group aws-ent-gov-enterprise-dmz --infoblox-view Dmz --zone stage.dice.census.gov ## 2919 2024-05-28 12:39:15 python infoblox-create-forwarding.py --infoblox-nameserver-group aws-ent-gov-enterprise --infoblox-forwarder-group aws-ent-gov-enterprise-dmz --infoblox-view Dmz --zone stage.dice.census.gov @@ -235,4 +381,4 @@ def main(): ## 2935 2024-05-28 12:54:54 python infoblox-manage-forwarding.py --infoblox-nameserver-group aws-ent-gov-enterprise --infoblox-forwarder-group aws-ent-gov-enterprise-dmz --infoblox-view Dmz --zone stage.dice.census.gov --action add ## 2941 2024-05-28 12:58:44 python infoblox-manage.py --help ## 2942 2024-05-28 12:59:00 python infoblox-manage.py --restart -## +## diff --git a/local-app/infoblox/infoblox-migrate-edl.py b/local-app/infoblox/infoblox-migrate-edl.py index 6dc291d1..f1a44420 100755 --- a/local-app/infoblox/infoblox-migrate-edl.py +++ b/local-app/infoblox/infoblox-migrate-edl.py @@ -1,177 +1,213 @@ #!/bin/env python -from pprint import pprint -from infoblox_client import connector -from infoblox_client import objects -import sys import os +import sys +from pprint import pprint + import urllib3 +from infoblox_client import connector, objects + urllib3.disable_warnings() -import boto3 import ipaddress + +import boto3 from credentials import credentials -ib_host = credentials['host'] -ib_api_version = credentials['api_version'] -ib_username = credentials['username'] -ib_password = credentials['password'] +ib_host = credentials["host"] +ib_api_version = credentials["api_version"] +ib_username = credentials["username"] +ib_password = credentials["password"] -opts = {'host': ib_host, 'username': ib_username, 'password': ib_password} +opts = {"host": ib_host, "username": ib_username, "password": ib_password} conn = connector.Connector(opts) # get all network_views -network_views = conn.get_object('networkview') +network_views = conn.get_object("networkview") ## print('views',network_views) # search network by cidr in specific network view -network = conn.get_object('network', {'network': '10.189.0.0/16', 'network_view': 'default'}) +network = conn.get_object( + "network", {"network": "10.189.0.0/16", "network_view": "default"} +) ## print('network',network) -networkcontainer = conn.get_object('networkcontainer', {'network': '10.189.0.0/16', 'network_view': 'default'}) +networkcontainer = conn.get_object( + "networkcontainer", {"network": "10.189.0.0/16", "network_view": "default"} +) ## print('network container',networkcontainer) -#fzone = conn.get_object('zone_forward') -#print('foward zones') -#pprint(fzone) +# fzone = conn.get_object('zone_forward') +# print('foward zones') +# pprint(fzone) -fzone_fields=[ - 'address', - 'comment', - 'disable', - 'disable_ns_generation', - 'display_domain', - 'dns_fqdn', - 'extattrs', - 'external_ns_group', - 'forward_to', - 'forwarders_only', - 'forwarding_servers', - 'fqdn', - 'locked', - 'locked_by', - 'mask_prefix', - 'ms_ad_integrated', - 'ms_ddns_mode', - 'ms_managed', - 'ms_read_only', - 'ms_sync_master_name', - 'ns_group', - 'parent', - 'prefix', - 'using_srg_associations', - 'view', - 'zone_format', +fzone_fields = [ + "address", + "comment", + "disable", + "disable_ns_generation", + "display_domain", + "dns_fqdn", + "extattrs", + "external_ns_group", + "forward_to", + "forwarders_only", + "forwarding_servers", + "fqdn", + "locked", + "locked_by", + "mask_prefix", + "ms_ad_integrated", + "ms_ddns_mode", + "ms_managed", + "ms_read_only", + "ms_sync_master_name", + "ns_group", + "parent", + "prefix", + "using_srg_associations", + "view", + "zone_format", ] if False: - zone = objects.DNSZoneForward.search(conn, view='internal-view', fqdn='stage.dice.census.gov', return_fields=fzone_fields) - pprint(zone) - pprint(vars(zone)) + zone = objects.DNSZoneForward.search( + conn, + view="internal-view", + fqdn="stage.dice.census.gov", + return_fields=fzone_fields, + ) + pprint(zone) + pprint(vars(zone)) if False: - rzone = objects.DNSZoneForward.search(conn, view='internal-view', fqdn='10.188.10.in-addr.arpa', return_fields=fzone_fields) - pprint(rzone) - pprint(vars(rzone)) + rzone = objects.DNSZoneForward.search( + conn, + view="internal-view", + fqdn="10.188.10.in-addr.arpa", + return_fields=fzone_fields, + ) + pprint(rzone) + pprint(vars(rzone)) if False: - name2='0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.7.3.0.8.1.0.2.0.2.0.0.0.1.6.2.ip6.arpa' - rzone2 = objects.DNSZoneForward.search(conn, view='internal-view', fqdn=name2, return_fields=fzone_fields) - pprint(rzone2) - pprint(vars(rzone2)) + name2 = "0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.7.3.0.8.1.0.2.0.2.0.0.0.1.6.2.ip6.arpa" + rzone2 = objects.DNSZoneForward.search( + conn, view="internal-view", fqdn=name2, return_fields=fzone_fields + ) + pprint(rzone2) + pprint(vars(rzone2)) -forwarding_servers=[ - { 'forward_to':[], 'forwarders_only':False, 'name':"bcc-inf-idns01.tco.census.gov", 'use_override_forwarders':False }, - { 'forward_to':[], 'forwarders_only':False, 'name':"hq-inf-idns01.tco.census.gov", 'use_override_forwarders':False }, +forwarding_servers = [ + { + "forward_to": [], + "forwarders_only": False, + "name": "bcc-inf-idns01.tco.census.gov", + "use_override_forwarders": False, + }, + { + "forward_to": [], + "forwarders_only": False, + "name": "hq-inf-idns01.tco.census.gov", + "use_override_forwarders": False, + }, ] -fzone={ - 'comment': 'AWS: DICE', - 'external_ns_group': 'aws-ent-gov-enterprise', - 'forward_to': [], - 'forwarders_only': False, - 'forwarding_services': forwarding_servers, -# 'parent': 'dice.census.gov', - 'zone_format': 'FORWARD', +fzone = { + "comment": "AWS: DICE", + "external_ns_group": "aws-ent-gov-enterprise", + "forward_to": [], + "forwarders_only": False, + "forwarding_services": forwarding_servers, + # 'parent': 'dice.census.gov', + "zone_format": "FORWARD", } if False: - zone = objects.DNSZoneForward.create(conn, view='internal-view', fqdn='ite.dice.census.gov', check_if_exists=True, **fzone) - pprint(zone) - pprint(vars(zone)) + zone = objects.DNSZoneForward.create( + conn, + view="internal-view", + fqdn="ite.dice.census.gov", + check_if_exists=True, + **fzone, + ) + pprint(zone) + pprint(vars(zone)) -if len(sys.argv)>1: - zone=sys.argv[1] +if len(sys.argv) > 1: + zone = sys.argv[1] else: - zone='dev.edl.census.gov' + zone = "dev.edl.census.gov" -z3 = conn.get_object('record:a', {'name~': f'.{zone}'}) +z3 = conn.get_object("record:a", {"name~": f".{zone}"}) ## pprint(z3) -#pprint(vars(z3)) -z4 = conn.get_object('record:txt', {'name~': f'.{zone}'}) +# pprint(vars(z3)) +z4 = conn.get_object("record:txt", {"name~": f".{zone}"}) ## pprint(z4) -entries={} +entries = {} for entry in z3: - name=entry['name'] - entries[name]={'ipv4':entry} - ip_address=ipaddress.ip_address(entry['ipv4addr']) - entries[name]['ipv4_ptr']=ip_address.reverse_pointer - entries[name]['ipv4_ptr_zone']=(ip_address.reverse_pointer.split('.',1))[1] + name = entry["name"] + entries[name] = {"ipv4": entry} + ip_address = ipaddress.ip_address(entry["ipv4addr"]) + entries[name]["ipv4_ptr"] = ip_address.reverse_pointer + entries[name]["ipv4_ptr_zone"] = (ip_address.reverse_pointer.split(".", 1))[1] for entry in z4: - name=entry['name'] - if entries.get(name): - entries[name]['txt']=entry - else: - print(f'entry {name} has TXT but no A record: TXT="{entry["text"]}"') + name = entry["name"] + if entries.get(name): + entries[name]["txt"] = entry + else: + print(f'entry {name} has TXT but no A record: TXT="{entry["text"]}"') -profile=os.environ.get('INFOBLOX_AWS_PROFILE',None) +profile = os.environ.get("INFOBLOX_AWS_PROFILE", None) -#pprint(entries) -print(f'# zone {zone}') +# pprint(entries) +print(f"# zone {zone}") if profile is None: - for k,v in entries.items(): - print(f'{k}. IN A {v["ipv4"]["ipv4addr"]}') - if v.get('txt'): - print(f'{k}. IN TXT {v["txt"]["text"]}') - sys.exit(0) + for k, v in entries.items(): + print(f'{k}. IN A {v["ipv4"]["ipv4addr"]}') + if v.get("txt"): + print(f'{k}. IN TXT {v["txt"]["text"]}') + sys.exit(0) -print(f'\n* checking AWS for profile {profile}') -east_session=boto3.Session(profile_name=profile,region_name='us-gov-east-1') -west_session=boto3.Session(profile_name=profile,region_name='us-gov-west-1') -east_client=east_session.client('ec2') -west_client=west_session.client('ec2') +print(f"\n* checking AWS for profile {profile}") +east_session = boto3.Session(profile_name=profile, region_name="us-gov-east-1") +west_session = boto3.Session(profile_name=profile, region_name="us-gov-west-1") +east_client = east_session.client("ec2") +west_client = west_session.client("ec2") -rows=['ftype,zone,rr_type,rr_name,rr_value,region'] -for k,v in entries.items(): -# check for ipv4 address - filters=[ { 'Name': 'addresses.private-ip-address', 'Values': [v['ipv4']['ipv4addr']] } ] - e_response = east_client.describe_network_interfaces(Filters=filters,DryRun=False) - w_response = west_client.describe_network_interfaces(Filters=filters,DryRun=False) - e_found=len(e_response['NetworkInterfaces']) -# print('east response') -# pprint(e_response) - w_found=len(w_response['NetworkInterfaces']) -# print('west response') -# pprint(w_response) - region='' - if e_found>0: - print(f'# found name {k} in east, keeping') - region='us-gov-east-1' - elif w_found>0: - print(f'# found name {k} in west, keeping') - region='us-gov-west-1' - else: - print(f'# not found name {k}, removing') - print(f'#REMOVE {k}. IN A {v["ipv4"]["ipv4addr"]}') - if v.get('txt'): - print(f'#REMOVE {k}. IN TXT {v["txt"]["text"]}') - if e_found>0 or w_found>0: - print(f'{k}. IN A {v["ipv4"]["ipv4addr"]}') - rows.append(f'CSV,{zone},A,{k},{v["ipv4"]["ipv4addr"]},{region}') - rows.append(f'CSV,{v["ipv4_ptr_zone"]},PTR,{v["ipv4_ptr"]},{k},{region}') - if v.get('txt'): - print(f'{k}. IN TXT {v["txt"]["text"]}') - rows.append(f'CSV,{zone},TXT,{k},{v["txt"]["text"]},{region}') +rows = ["ftype,zone,rr_type,rr_name,rr_value,region"] +for k, v in entries.items(): + # check for ipv4 address + filters = [ + {"Name": "addresses.private-ip-address", "Values": [v["ipv4"]["ipv4addr"]]} + ] + e_response = east_client.describe_network_interfaces(Filters=filters, DryRun=False) + w_response = west_client.describe_network_interfaces(Filters=filters, DryRun=False) + e_found = len(e_response["NetworkInterfaces"]) + # print('east response') + # pprint(e_response) + w_found = len(w_response["NetworkInterfaces"]) + # print('west response') + # pprint(w_response) + region = "" + if e_found > 0: + print(f"# found name {k} in east, keeping") + region = "us-gov-east-1" + elif w_found > 0: + print(f"# found name {k} in west, keeping") + region = "us-gov-west-1" + else: + print(f"# not found name {k}, removing") + print(f'#REMOVE {k}. IN A {v["ipv4"]["ipv4addr"]}') + if v.get("txt"): + print(f'#REMOVE {k}. IN TXT {v["txt"]["text"]}') + if e_found > 0 or w_found > 0: + print(f'{k}. IN A {v["ipv4"]["ipv4addr"]}') + rows.append(f'CSV,{zone},A,{k},{v["ipv4"]["ipv4addr"]},{region}') + rows.append(f'CSV,{v["ipv4_ptr_zone"]},PTR,{v["ipv4_ptr"]},{k},{region}') + if v.get("txt"): + print(f'{k}. IN TXT {v["txt"]["text"]}') + rows.append(f'CSV,{zone},TXT,{k},{v["txt"]["text"]},{region}') print() for r in rows: - print(r) + print(r) ## {'_ref': 'record:a/ZG5zLmJpbmRfYSQuX2RlZmF1bHQuZ292LmNlbnN1cy5lZGwuZGV2LHQtMjAyMDA1MjYtOS01LDEwLjE5Ni4xMi45NA:t-20200526-9-5.dev.edl.census.gov/internal-view', ## 'ipv4addr': '10.196.12.94', @@ -193,34 +229,34 @@ ## 'text': '"last_modified: 2020-01-31T15:32:43Z instance_id: ' ## 'i-00de76d33dbf78d4e instance_region: us-gov-east-1"', ## 'view': 'internal-view'}, -## +## ## search(cls, connector, return_fields=None, search_extattrs=None, force_proxy=False, **kwargs) Search single object on NIOS side, returns first object that match search criteria. Requires connector passed as the first argument. return_fields can be set to retrieve particular fields from NIOS, for example return_fields=['view', 'name']. If return_fields is [] default return_fields are returned by NIOS side for current wapi_version. search_extattrs is used to filter out results by extensible attributes. force_proxy forces search request to be processed on Grid Master (applies only in cloud environment) -## -## +## +## ## DNSZoneFoward -## +## ## from infoblox_client import objects -## +## ## opts = {'host': '192.168.1.10', 'username': 'admin', 'password': 'admin'} ## conn = connector.Connector(opts) -## +## ## Create a network view, and network: -## +## ## .. code:: python -## +## ## nview = objects.NetworkView.create(conn, name='my_view') ## network = objects.Network.create(conn, network_view='my_view', cidr='192.168.1.0/24') -## +## ## Create a DNS view and zone: -## +## ## .. code:: python -## +## ## view = objects.DNSView.create(conn, network_view='my_view', name='my_dns_view') ## zone = objects.DNSZone.create(conn, view='my_dns_view', fqdn='my_zone.com') -## -## -## +## +## +## ## members. ## Fields ## These fields are actual members of the object; thus, they @@ -236,7 +272,7 @@ ## Object Reference ## Restrictions ## Fields -## +## ## address ## comment ## disable @@ -263,7 +299,7 @@ ## using_srg_associations ## view ## zone_format -## +## ## DNSZoneForward: comment="AWS: DICE", disable="False", disable_ns_generation="False", display_domain="stage.dice.census.gov", dns_fqdn="stage.dice.census.gov", external_ns_group="aws-ent-gov-enterprise", forward_to="[]", forwarders_only="False", forwarding_servers="[Forwardingmemberserver: forward_to="[]", forwarders_only="False", name="bcc-inf-idns01.tco.census.gov", use_override_forwarders="False", Forwardingmemberserver: forward_to="[]", forwarders_only="False", name="hq-inf-idns01.tco.census.gov", use_override_forwarders="False"]", fqdn="stage.dice.census.gov", locked="False", ms_ad_integrated="False", ms_ddns_mode="NONE", ms_managed="NONE", ms_read_only="False", parent="dice.census.gov", using_srg_associations="False", view="internal-view", zone_format="FORWARD", _ref="zone_forward/ZG5zLnpvbmUkLl9kZWZhdWx0Lmdvdi5jZW5zdXMuZGljZS5zdGFnZQ:stage.dice.census.gov/internal-view" ## {'_ref': 'zone_forward/ZG5zLnpvbmUkLl9kZWZhdWx0Lmdvdi5jZW5zdXMuZGljZS5zdGFnZQ:stage.dice.census.gov/internal-view', diff --git a/local-app/infoblox/infoblox-restart.py b/local-app/infoblox/infoblox-restart.py index 046c4e20..c3681723 100755 --- a/local-app/infoblox/infoblox-restart.py +++ b/local-app/infoblox/infoblox-restart.py @@ -1,73 +1,86 @@ #!/bin/env python -from pprint import pprint -from infoblox_client import connector -from infoblox_client import objects -import sys import os +import sys +from pprint import pprint + import urllib3 +from infoblox_client import connector, objects + urllib3.disable_warnings() -import boto3 +import hashlib import ipaddress -#from credentials import credentials + +import boto3 + +# from credentials import credentials import yaml -import hashlib + def read_yaml(file): - data={} - with open(file, 'r') as stream: - try: - data=yaml.full_load(stream) - except yaml.YAMLError as e: - print(e) - return None - return data - -ib_data=read_yaml('credentials.yml') -site=os.environ.get('INFOBLOX_SITE','network-prod').lower() -print(f'* using site {site}') -credentials=ib_data.get(site,None) + data = {} + with open(file, "r") as stream: + try: + data = yaml.full_load(stream) + except yaml.YAMLError as e: + print(e) + return None + return data + + +ib_data = read_yaml("credentials.yml") +site = os.environ.get("INFOBLOX_SITE", "network-prod").lower() +print(f"* using site {site}") +credentials = ib_data.get(site, None) if credentials is None: - print(f'* no credentials found for site {site}') - sys.exit(1) + print(f"* no credentials found for site {site}") + sys.exit(1) -ib_host = credentials['host'] -ib_api_version = credentials['api_version'] -ib_username = credentials['username'] -ib_password = credentials['password'] +ib_host = credentials["host"] +ib_api_version = credentials["api_version"] +ib_username = credentials["username"] +ib_password = credentials["password"] -opts = {'host': ib_host, 'username': ib_username, 'password': ib_password} +opts = {"host": ib_host, "username": ib_username, "password": ib_password} conn = connector.Connector(opts) -#network = conn.get_object('network', {'network': '10.189.0.0/16', 'network_view': 'default'}) -#networkcontainer = conn.get_object('networkcontainer', {'network': '10.189.0.0/16', 'network_view': 'default'}) +# network = conn.get_object('network', {'network': '10.189.0.0/16', 'network_view': 'default'}) +# networkcontainer = conn.get_object('networkcontainer', {'network': '10.189.0.0/16', 'network_view': 'default'}) -print(f'* using site {site} grid host {ib_host}') +print(f"* using site {site} grid host {ib_host}") grid = objects.Grid.search(conn) -print('grid') +print("grid") pprint(grid) -print('vars(grid)') +print("vars(grid)") pprint(vars(grid)) if grid is not None: - gstatus = grid.requestrestartservicestatus({'service_option':'ALL'}) - print('gstatus') - pprint(gstatus) - - gridrs = objects.Restartservicestatus.search(conn) - print('gridrs') - pprint(gridrs) - print('vars(gridrs)') - pprint(vars(gridrs)) - - print(f'dns_status={gridrs.dns_status}') - if gridrs.dns_status=='REQUESTING': - status = grid.restartservices({'restart_option': 'RESTART_IF_NEEDED', 'service_option': 'ALL', 'member_order': 'SEQUENTIALLY', 'sequential_delay': 1, 'user_name': ib_username}) - print('restart status') - pprint(status) + gstatus = grid.requestrestartservicestatus({"service_option": "ALL"}) + print("gstatus") + pprint(gstatus) + + gridrs = objects.Restartservicestatus.search(conn) + print("gridrs") + pprint(gridrs) + print("vars(gridrs)") + pprint(vars(gridrs)) + + print(f"dns_status={gridrs.dns_status}") + if gridrs.dns_status == "REQUESTING": + status = grid.restartservices( + { + "restart_option": "RESTART_IF_NEEDED", + "service_option": "ALL", + "member_order": "SEQUENTIALLY", + "sequential_delay": 1, + "user_name": ib_username, + } + ) + print("restart status") + pprint(status) # https://blogs.infoblox.com/community/restart-services-using-the-rest-api/ ## curl -k -u admin:infoblox -X POST https://192.168.1.2/wapi/v1.1/grid/b25lLmNsdXN0ZXIkMA:Infoblox?_function=restartservices -H 'Content-Type:application/json' -d '{'restart_option': 'RESTART_IF_NEEDED', 'service_option': 'ALL', 'member_order': 'SEQUENTIALLY', 'sequential_delay': '1'}' -## +## ## grid diff --git a/local-app/infoblox/infoblox-search.py b/local-app/infoblox/infoblox-search.py index 999c5b08..d85f5795 100755 --- a/local-app/infoblox/infoblox-search.py +++ b/local-app/infoblox/infoblox-search.py @@ -1,73 +1,79 @@ #!/bin/env python -from pprint import pprint -from infoblox_client import connector -from infoblox_client import objects -import sys import os +import sys +from pprint import pprint + import urllib3 +from infoblox_client import connector, objects + urllib3.disable_warnings() -import boto3 import ipaddress + +import boto3 from credentials import credentials -ib_host = credentials['host'] -ib_api_version = credentials['api_version'] -ib_username = credentials['username'] -ib_password = credentials['password'] +ib_host = credentials["host"] +ib_api_version = credentials["api_version"] +ib_username = credentials["username"] +ib_password = credentials["password"] -opts = {'host': ib_host, 'username': ib_username, 'password': ib_password} +opts = {"host": ib_host, "username": ib_username, "password": ib_password} conn = connector.Connector(opts) # get all network_views -network_views = conn.get_object('networkview') +network_views = conn.get_object("networkview") ## print('views',network_views) # search network by cidr in specific network view -network = conn.get_object('network', {'network': '10.189.0.0/16', 'network_view': 'default'}) +network = conn.get_object( + "network", {"network": "10.189.0.0/16", "network_view": "default"} +) ## print('network',network) -networkcontainer = conn.get_object('networkcontainer', {'network': '10.189.0.0/16', 'network_view': 'default'}) +networkcontainer = conn.get_object( + "networkcontainer", {"network": "10.189.0.0/16", "network_view": "default"} +) ## print('network container',networkcontainer) -#fzone = conn.get_object('zone_forward') -#print('foward zones') -#pprint(fzone) +# fzone = conn.get_object('zone_forward') +# print('foward zones') +# pprint(fzone) -fzone_fields=[ - 'address', - 'comment', - 'disable', - 'disable_ns_generation', - 'display_domain', - 'dns_fqdn', - 'extattrs', - 'external_ns_group', - 'forward_to', - 'forwarders_only', - 'forwarding_servers', - 'fqdn', - 'locked', - 'locked_by', - 'mask_prefix', - 'ms_ad_integrated', - 'ms_ddns_mode', - 'ms_managed', - 'ms_read_only', - 'ms_sync_master_name', - 'ns_group', - 'parent', - 'prefix', - 'using_srg_associations', - 'view', - 'zone_format', +fzone_fields = [ + "address", + "comment", + "disable", + "disable_ns_generation", + "display_domain", + "dns_fqdn", + "extattrs", + "external_ns_group", + "forward_to", + "forwarders_only", + "forwarding_servers", + "fqdn", + "locked", + "locked_by", + "mask_prefix", + "ms_ad_integrated", + "ms_ddns_mode", + "ms_managed", + "ms_read_only", + "ms_sync_master_name", + "ns_group", + "parent", + "prefix", + "using_srg_associations", + "view", + "zone_format", ] grid = objects.Grid.search(conn) -print('grid') +print("grid") pprint(grid) -print('vars(grid)') +print("vars(grid)") pprint(vars(grid)) # https://blogs.infoblox.com/community/restart-services-using-the-rest-api/ ## curl -k -u admin:infoblox -X POST https://192.168.1.2/wapi/v1.1/grid/b25lLmNsdXN0ZXIkMA:Infoblox?_function=restartservices -H "Content-Type:application/json" -d '{"restart_option": "RESTART_IF_NEEDED", "service_option": "ALL", "member_order": "SEQUENTIALLY", "sequential_delay": '1'}' -## +## ## grid diff --git a/local-app/infoblox/infoblox.py b/local-app/infoblox/infoblox.py index e6a82881..e0b0389e 100755 --- a/local-app/infoblox/infoblox.py +++ b/local-app/infoblox/infoblox.py @@ -1,122 +1,152 @@ #!/bin/env python -from pprint import pprint -from infoblox_client import connector -from infoblox_client import objects -import sys import os +import sys +from pprint import pprint + import urllib3 +from infoblox_client import connector, objects + urllib3.disable_warnings() -import boto3 import ipaddress + +import boto3 from credentials import credentials -ib_host = credentials['host'] -ib_api_version = credentials['api_version'] -ib_username = credentials['username'] -ib_password = credentials['password'] +ib_host = credentials["host"] +ib_api_version = credentials["api_version"] +ib_username = credentials["username"] +ib_password = credentials["password"] -opts = {'host': ib_host, 'username': ib_username, 'password': ib_password} +opts = {"host": ib_host, "username": ib_username, "password": ib_password} conn = connector.Connector(opts) # get all network_views -network_views = conn.get_object('networkview') +network_views = conn.get_object("networkview") ## print('views',network_views) # search network by cidr in specific network view -network = conn.get_object('network', {'network': '10.189.0.0/16', 'network_view': 'default'}) +network = conn.get_object( + "network", {"network": "10.189.0.0/16", "network_view": "default"} +) ## print('network',network) -networkcontainer = conn.get_object('networkcontainer', {'network': '10.189.0.0/16', 'network_view': 'default'}) +networkcontainer = conn.get_object( + "networkcontainer", {"network": "10.189.0.0/16", "network_view": "default"} +) ## print('network container',networkcontainer) -#fzone = conn.get_object('zone_forward') -#print('foward zones') -#pprint(fzone) +# fzone = conn.get_object('zone_forward') +# print('foward zones') +# pprint(fzone) -fzone_fields=[ - 'address', - 'comment', - 'disable', - 'disable_ns_generation', - 'display_domain', - 'dns_fqdn', - 'extattrs', - 'external_ns_group', - 'forward_to', - 'forwarders_only', - 'forwarding_servers', - 'fqdn', - 'locked', - 'locked_by', - 'mask_prefix', - 'ms_ad_integrated', - 'ms_ddns_mode', - 'ms_managed', - 'ms_read_only', - 'ms_sync_master_name', - 'ns_group', - 'parent', - 'prefix', - 'using_srg_associations', - 'view', - 'zone_format', +fzone_fields = [ + "address", + "comment", + "disable", + "disable_ns_generation", + "display_domain", + "dns_fqdn", + "extattrs", + "external_ns_group", + "forward_to", + "forwarders_only", + "forwarding_servers", + "fqdn", + "locked", + "locked_by", + "mask_prefix", + "ms_ad_integrated", + "ms_ddns_mode", + "ms_managed", + "ms_read_only", + "ms_sync_master_name", + "ns_group", + "parent", + "prefix", + "using_srg_associations", + "view", + "zone_format", ] if False: - zone = objects.DNSZoneForward.search(conn, view='internal-view', fqdn='stage.dice.census.gov', return_fields=fzone_fields) - pprint(zone) - pprint(vars(zone)) + zone = objects.DNSZoneForward.search( + conn, + view="internal-view", + fqdn="stage.dice.census.gov", + return_fields=fzone_fields, + ) + pprint(zone) + pprint(vars(zone)) if False: - rzone = objects.DNSZoneForward.search(conn, view='internal-view', fqdn='10.188.10.in-addr.arpa', return_fields=fzone_fields) - pprint(rzone) - pprint(vars(rzone)) + rzone = objects.DNSZoneForward.search( + conn, + view="internal-view", + fqdn="10.188.10.in-addr.arpa", + return_fields=fzone_fields, + ) + pprint(rzone) + pprint(vars(rzone)) if False: - name2='0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.7.3.0.8.1.0.2.0.2.0.0.0.1.6.2.ip6.arpa' - rzone2 = objects.DNSZoneForward.search(conn, view='internal-view', fqdn=name2, return_fields=fzone_fields) - pprint(rzone2) - pprint(vars(rzone2)) + name2 = "0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.7.3.0.8.1.0.2.0.2.0.0.0.1.6.2.ip6.arpa" + rzone2 = objects.DNSZoneForward.search( + conn, view="internal-view", fqdn=name2, return_fields=fzone_fields + ) + pprint(rzone2) + pprint(vars(rzone2)) -forwarding_servers=[ - { 'forward_to':[], 'forwarders_only':False, 'name':"bcc-inf-idns01.tco.census.gov", 'use_override_forwarders':False }, - { 'forward_to':[], 'forwarders_only':False, 'name':"hq-inf-idns01.tco.census.gov", 'use_override_forwarders':False }, +forwarding_servers = [ + { + "forward_to": [], + "forwarders_only": False, + "name": "bcc-inf-idns01.tco.census.gov", + "use_override_forwarders": False, + }, + { + "forward_to": [], + "forwarders_only": False, + "name": "hq-inf-idns01.tco.census.gov", + "use_override_forwarders": False, + }, ] -fzone={ - 'comment': 'AWS-EDL', - 'external_ns_group': 'aws-ent-gov-enterprise', - 'forward_to': [], - 'forwarders_only': False, - 'forwarding_servers': forwarding_servers, -# 'parent': 'dice.census.gov', - 'zone_format': 'FORWARD', +fzone = { + "comment": "AWS-EDL", + "external_ns_group": "aws-ent-gov-enterprise", + "forward_to": [], + "forwarders_only": False, + "forwarding_servers": forwarding_servers, + # 'parent': 'dice.census.gov', + "zone_format": "FORWARD", } -if len(sys.argv)>1: - zone=sys.argv[1] +if len(sys.argv) > 1: + zone = sys.argv[1] else: - print(f'* missing zone') - sys.exit(1) + print(f"* missing zone") + sys.exit(1) -zone = objects.DNSZoneForward.search(conn, view='internal-view', fqdn=zone, return_fields=fzone_fields) -print('zone') +zone = objects.DNSZoneForward.search( + conn, view="internal-view", fqdn=zone, return_fields=fzone_fields +) +print("zone") pprint(zone) -print('vars(zone)') +print("vars(zone)") pprint(vars(zone)) -#print('dict(zone)') -#pprint(dict(zone)) +# print('dict(zone)') +# pprint(dict(zone)) -## +## ## if len(sys.argv)>1: ## zone=sys.argv[1] ## else: ## zone='dev.edl.census.gov' -## +## ## z3 = conn.get_object('record:a', {'name~': f'.{zone}'}) ## ## pprint(z3) ## #pprint(vars(z3)) ## z4 = conn.get_object('record:txt', {'name~': f'.{zone}'}) ## ## pprint(z4) -## +## ## entries={} ## for entry in z3: ## name=entry['name'] @@ -124,16 +154,16 @@ ## ip_address=ipaddress.ip_address(entry['ipv4addr']) ## entries[name]['ipv4_ptr']=ip_address.reverse_pointer ## entries[name]['ipv4_ptr_zone']=(ip_address.reverse_pointer.split('.',1))[1] -## +## ## for entry in z4: ## name=entry['name'] ## if entries.get(name): ## entries[name]['txt']=entry ## else: ## print(f'entry {name} has TXT but no A record: TXT="{entry["text"]}"') -## +## ## profile=os.environ.get('INFOBLOX_AWS_PROFILE',None) -## +## ## #pprint(entries) ## print(f'# zone {zone}') ## if profile is None: @@ -142,13 +172,13 @@ ## if v.get('txt'): ## print(f'{k}. IN TXT {v["txt"]["text"]}') ## sys.exit(0) -## +## ## print(f'\n* checking AWS for profile {profile}') ## east_session=boto3.Session(profile_name=profile,region_name='us-gov-east-1') ## west_session=boto3.Session(profile_name=profile,region_name='us-gov-west-1') ## east_client=east_session.client('ec2') ## west_client=west_session.client('ec2') -## +## ## rows=['ftype,zone,rr_type,rr_name,rr_value,region'] ## for k,v in entries.items(): ## # check for ipv4 address @@ -180,11 +210,11 @@ ## if v.get('txt'): ## print(f'{k}. IN TXT {v["txt"]["text"]}') ## rows.append(f'CSV,{zone},TXT,{k},{v["txt"]["text"]},{region}') -## +## ## print() ## for r in rows: ## print(r) -## +## ## ## {'_ref': 'record:a/ZG5zLmJpbmRfYSQuX2RlZmF1bHQuZ292LmNlbnN1cy5lZGwuZGV2LHQtMjAyMDA1MjYtOS01LDEwLjE5Ni4xMi45NA:t-20200526-9-5.dev.edl.census.gov/internal-view', ## ## 'ipv4addr': '10.196.12.94', ## ## 'name': 't-20200526-9-5.dev.edl.census.gov', @@ -205,34 +235,34 @@ ## ## 'text': '"last_modified: 2020-01-31T15:32:43Z instance_id: ' ## ## 'i-00de76d33dbf78d4e instance_region: us-gov-east-1"', ## ## 'view': 'internal-view'}, -## ## -## +## ## +## ## ## search(cls, connector, return_fields=None, search_extattrs=None, force_proxy=False, **kwargs) Search single object on NIOS side, returns first object that match search criteria. Requires connector passed as the first argument. return_fields can be set to retrieve particular fields from NIOS, for example return_fields=['view', 'name']. If return_fields is [] default return_fields are returned by NIOS side for current wapi_version. search_extattrs is used to filter out results by extensible attributes. force_proxy forces search request to be processed on Grid Master (applies only in cloud environment) -## ## -## ## +## ## +## ## ## ## DNSZoneFoward -## ## +## ## ## ## from infoblox_client import objects -## ## +## ## ## ## opts = {'host': '192.168.1.10', 'username': 'admin', 'password': 'admin'} ## ## conn = connector.Connector(opts) -## ## +## ## ## ## Create a network view, and network: -## ## +## ## ## ## .. code:: python -## ## +## ## ## ## nview = objects.NetworkView.create(conn, name='my_view') ## ## network = objects.Network.create(conn, network_view='my_view', cidr='192.168.1.0/24') -## ## +## ## ## ## Create a DNS view and zone: -## ## +## ## ## ## .. code:: python -## ## +## ## ## ## view = objects.DNSView.create(conn, network_view='my_view', name='my_dns_view') ## ## zone = objects.DNSZone.create(conn, view='my_dns_view', fqdn='my_zone.com') -## ## -## ## -## ## +## ## +## ## +## ## ## ## members. ## ## Fields ## ## These fields are actual members of the object; thus, they @@ -248,7 +278,7 @@ ## ## Object Reference ## ## Restrictions ## ## Fields -## ## +## ## ## ## address ## ## comment ## ## disable @@ -275,8 +305,8 @@ ## ## using_srg_associations ## ## view ## ## zone_format -## ## -## +## ## +## ## ## DNSZoneForward: comment="AWS: DICE", disable="False", disable_ns_generation="False", display_domain="stage.dice.census.gov", dns_fqdn="stage.dice.census.gov", external_ns_group="aws-ent-gov-enterprise", forward_to="[]", forwarders_only="False", forwarding_servers="[Forwardingmemberserver: forward_to="[]", forwarders_only="False", name="bcc-inf-idns01.tco.census.gov", use_override_forwarders="False", Forwardingmemberserver: forward_to="[]", forwarders_only="False", name="hq-inf-idns01.tco.census.gov", use_override_forwarders="False"]", fqdn="stage.dice.census.gov", locked="False", ms_ad_integrated="False", ms_ddns_mode="NONE", ms_managed="NONE", ms_read_only="False", parent="dice.census.gov", using_srg_associations="False", view="internal-view", zone_format="FORWARD", _ref="zone_forward/ZG5zLnpvbmUkLl9kZWZhdWx0Lmdvdi5jZW5zdXMuZGljZS5zdGFnZQ:stage.dice.census.gov/internal-view" ## ## {'_ref': 'zone_forward/ZG5zLnpvbmUkLl9kZWZhdWx0Lmdvdi5jZW5zdXMuZGljZS5zdGFnZQ:stage.dice.census.gov/internal-view', ## ## 'address': None, diff --git a/local-app/prowler/prowler-report.sh b/local-app/prowler/prowler-report.sh index b44115c9..53f6aafd 100755 --- a/local-app/prowler/prowler-report.sh +++ b/local-app/prowler/prowler-report.sh @@ -25,7 +25,7 @@ then echo "* running $PROWLER at $(date)" # test -d logs || mkdir logs PROWLEROUT="prowler.$region.$profile.$DATE" -# $PROWLER -p $profile -r $region -f $region -M csv,json,html > $REPORTDIR/$PROWLEROUT.txt +# $PROWLER -p $profile -r $region -f $region -M csv,json,html > $REPORTDIR/$PROWLEROUT.txt $PROWLER -p $profile -r $region -f $region -M text > $REPORTDIR/$PROWLEROUT.txt 2> $REPORTDIR/$PROWLEROUT.txt.err # echo "* making html" # cat logs/$PROWLEROUT.txt | ansi2html -la > logs/$PROWLEROUT.html diff --git a/local-app/python-tools/VolumeReport/volume_report.py b/local-app/python-tools/VolumeReport/volume_report.py index 405a58bd..2001960b 100644 --- a/local-app/python-tools/VolumeReport/volume_report.py +++ b/local-app/python-tools/VolumeReport/volume_report.py @@ -1,96 +1,83 @@ -import boto3 +import argparse +import configparser import csv import datetime import smtplib +from email import encoders +from email.mime.base import MIMEBase from email.mime.multipart import MIMEMultipart from email.mime.text import MIMEText -from email.mime.base import MIMEBase -from email import encoders -import argparse -import configparser + +import boto3 namespace = "AWS/EBS" -profile = 'prod' -vpc_id = 'vpc-f770a892' +profile = "prod" +vpc_id = "vpc-f770a892" now = datetime.datetime.now() start = now - datetime.timedelta(days=95) -period=3600 +period = 3600 print(start) print(now) instances = [] + def percentage(numerator, denominator): - return round(100 * float(numerator)/float(denominator), 1) + return round(100 * float(numerator) / float(denominator), 1) + def get_tag_map(tags): tag_map = {} if tags: for tag in tags: - tag_map[tag['Key']] = tag['Value'] + tag_map[tag["Key"]] = tag["Value"] return tag_map + def get_metrics(cw, vol_id): queries = [ { - "Id":"wiops", - "MetricStat":{ - "Metric":{ - "Namespace":namespace, + "Id": "wiops", + "MetricStat": { + "Metric": { + "Namespace": namespace, "MetricName": "VolumeWriteOps", - "Dimensions":[ - { - "Name":"VolumeId", - "Value":vol_id - } - ] + "Dimensions": [{"Name": "VolumeId", "Value": vol_id}], }, - "Period":period, - "Stat":"Sum" + "Period": period, + "Stat": "Sum", }, - "ReturnData":False + "ReturnData": False, }, { - "Id":"riops", - "MetricStat":{ - "Metric":{ - "Namespace":namespace, + "Id": "riops", + "MetricStat": { + "Metric": { + "Namespace": namespace, "MetricName": "VolumeReadOps", - "Dimensions":[ - { - "Name":"VolumeId", - "Value":vol_id - } - ] + "Dimensions": [{"Name": "VolumeId", "Value": vol_id}], }, - "Period":period, - "Stat":"Sum" + "Period": period, + "Stat": "Sum", }, - "ReturnData":False + "ReturnData": False, }, - { - "Id":"iops", - "ReturnData":True, - "Expression": "(wiops+riops)/PERIOD(wiops)" - } + {"Id": "iops", "ReturnData": True, "Expression": "(wiops+riops)/PERIOD(wiops)"}, ] - paginator = cw.get_paginator('get_metric_data') - pages = paginator.paginate(MetricDataQueries=queries, - StartTime=start, - EndTime=now - ) + paginator = cw.get_paginator("get_metric_data") + pages = paginator.paginate(MetricDataQueries=queries, StartTime=start, EndTime=now) max_iops = 0 for page in pages: - for result in page['MetricDataResults']: - if len(result['Values']) > 0: - if result['Id'] == 'iops': - iops = max(result['Values']) + for result in page["MetricDataResults"]: + if len(result["Values"]) > 0: + if result["Id"] == "iops": + iops = max(result["Values"]) if iops > max_iops: max_iops = iops return max_iops @@ -98,17 +85,15 @@ def get_metrics(cw, vol_id): def get_instances(ec2): - filters = [ - {} - ] - skip_filters = [ - {} - ] + filters = [{}] + skip_filters = [{}] ##instances = [ec2.Instance(id) for id in instance_ids] instances = list(ec2.instances.filter(Filters=filters)) - #instances = list(ec2.instances.all()) + # instances = list(ec2.instances.all()) print("Original Instance List has {} instances".format(len(instances))) - skipped_ids = [instance.id for instance in ec2.instances.filter(Filters=skip_filters)] + skipped_ids = [ + instance.id for instance in ec2.instances.filter(Filters=skip_filters) + ] for instance in instances: if instance.id in skipped_ids: @@ -116,129 +101,210 @@ def get_instances(ec2): print("Filtered instance list has {} instances".format(len(instances))) return instances + def send_email(): - smtp_server = 'mail.census.gov' + smtp_server = "mail.census.gov" smtp_port = 25 server = smtplib.SMTP() - recipients = ['sida.ju@census.gov'] + recipients = ["sida.ju@census.gov"] msg = MIMEMultipart() - msg['From'] = 'VolumeReport_do_not_reply@census.gov' - msg['To'] = ", ".join(recipients) - msg['Subject'] = 'IOPs threshold exceeded' - body = 'Refer to csv' + msg["From"] = "VolumeReport_do_not_reply@census.gov" + msg["To"] = ", ".join(recipients) + msg["Subject"] = "IOPs threshold exceeded" + body = "Refer to csv" - part = MIMEBase('application', 'octet-stream') - part.set_payload(open('volumes.csv', 'rb').read()) + part = MIMEBase("application", "octet-stream") + part.set_payload(open("volumes.csv", "rb").read()) encoders.encode_base64(part) - part.add_header('Content-Disposition', 'attachment; filename=volumes.csv') + part.add_header("Content-Disposition", "attachment; filename=volumes.csv") msg.attach(part) - msg.attach(MIMEText(body, 'plain')) + msg.attach(MIMEText(body, "plain")) server.connect(smtp_server, smtp_port) server.ehlo() txt = msg.as_string() - server.sendmail(msg['From'], recipients, txt) + server.sendmail(msg["From"], recipients, txt) server.quit() + def main(): start_time = datetime.datetime.now() - #interpret parameters - parser = argparse.ArgumentParser(description='Manage our Unused Resources by identifying empty and unreferenced resources.') - parser.add_argument('--config','-c', help='Config file') + # interpret parameters + parser = argparse.ArgumentParser( + description="Manage our Unused Resources by identifying empty and unreferenced resources." + ) + parser.add_argument("--config", "-c", help="Config file") - parser.add_argument('--profiles','-p', help='Use profiles defined in configuration file', - action='store_true') + parser.add_argument( + "--profiles", + "-p", + help="Use profiles defined in configuration file", + action="store_true", + ) args = parser.parse_args() use_profiles = args.profiles if not args.config: - print('Invalid parameters. config file is required.') + print("Invalid parameters. config file is required.") exit() config = args.config - print(f'Starting timestamp[{str(start_time)}]') + print(f"Starting timestamp[{str(start_time)}]") - #interpret config file + # interpret config file configParser = configparser.RawConfigParser(allow_no_value=True) with open(config) as file: configParser.read_file(file) - profiles = {item[0]:item[1] for item in configParser.items("profiles")} - accounts_regions = {item[0]:item[1].split(",") for item in configParser.items("accounts_regions")} + profiles = {item[0]: item[1] for item in configParser.items("profiles")} + accounts_regions = { + item[0]: item[1].split(",") for item in configParser.items("accounts_regions") + } accounts = accounts_regions.keys() print(profiles) print(accounts_regions) print(accounts) - with open('volumes.csv', 'w', newline='') as csvfile: + with open("volumes.csv", "w", newline="") as csvfile: writer = csv.writer(csvfile) - writer.writerow(["profile","id","type","az","size","iops","max_iops", "device", "instance_id", "name", "application", - "ProjectNumber", "Project Name", "CostAllocation", "Environment", "Organization", "Creator", "boc:created_by", "Owner", - "Project Role","DeploymentID","aws:cloudformation:stack-name","aws:cloudformation:stack-id","aws:cloudformation:logical-id", - "aws:elasticmapreduce:job-flow-id","eks:cluster-name","kubernetes.io/created-for/pv/name"]) + writer.writerow( + [ + "profile", + "id", + "type", + "az", + "size", + "iops", + "max_iops", + "device", + "instance_id", + "name", + "application", + "ProjectNumber", + "Project Name", + "CostAllocation", + "Environment", + "Organization", + "Creator", + "boc:created_by", + "Owner", + "Project Role", + "DeploymentID", + "aws:cloudformation:stack-name", + "aws:cloudformation:stack-id", + "aws:cloudformation:logical-id", + "aws:elasticmapreduce:job-flow-id", + "eks:cluster-name", + "kubernetes.io/created-for/pv/name", + ] + ) for profile in profiles: for region in accounts_regions[profile]: - #create a session with explicit profile - boto3_session = boto3.session.Session(profile_name = profiles[profile]) - #create ec2 session and resource - ec2_resource = boto3_session.resource('ec2', region_name = region) - #create CloudWatch client - cw_client = boto3_session.client('cloudwatch', region_name = region) + # create a session with explicit profile + boto3_session = boto3.session.Session(profile_name=profiles[profile]) + # create ec2 session and resource + ec2_resource = boto3_session.resource("ec2", region_name=region) + # create CloudWatch client + cw_client = boto3_session.client("cloudwatch", region_name=region) instances = get_instances(ec2_resource) for instance in instances: - + for volume in instance.volumes.all(): - print(f'Evaluating volume [{volume.volume_id}] of type volume[{volume.volume_type}]') - if(volume.volume_type in ['gp3','gp2','io1']): + print( + f"Evaluating volume [{volume.volume_id}] of type volume[{volume.volume_type}]" + ) + if volume.volume_type in ["gp3", "gp2", "io1"]: max_iops = get_metrics(cw_client, volume.id) - print(volume.id, volume.iops, max_iops, '{}%'.format(str(percentage(max_iops,volume.iops)))) + print( + volume.id, + volume.iops, + max_iops, + "{}%".format(str(percentage(max_iops, volume.iops))), + ) if percentage(max_iops, volume.iops) >= 0.0: send_mail = True tag_map = get_tag_map(volume.tags) - writer.writerow([ - profile, - volume.volume_id, - volume.volume_type, - volume.availability_zone, - volume.size, - volume.iops, - max_iops, - (volume.attachments[0]['Device'] if len(volume.attachments) > 0 else ""), - (volume.attachments[0]['InstanceId'] if len(volume.attachments) > 0 else ""), - tag_map.get("Name", ""), - tag_map.get("Application", ""), - tag_map.get("ProjectNumber", "").replace('\u200b',''), - tag_map.get("Project Name", "").replace('\u200b',''), - tag_map.get("CostAllocation", "").replace('\u200b',''), - tag_map.get("Environment", "").replace('\u200b',''), - tag_map.get("Organization", "").replace('\u200b',''), - tag_map.get("Creator", "").replace('\u200b',''), - tag_map.get("boc:created_by", "".replace('\u200b','')), - tag_map.get("Owner", "").replace('\u200b',''), - tag_map.get("Project Role", "").replace('\u200b',''), - tag_map.get("DeploymentID", "").replace('\u200b',''), - tag_map.get("aws:cloudformation:stack-name", "").replace('\u200b',''), - tag_map.get("aws:cloudformation:stack-id", "").replace('\u200b',''), - tag_map.get("aws:cloudformation:logical-id", "").replace('\u200b',''), - tag_map.get("aws:elasticmapreduce:job-flow-id", "").replace('\u200b',''), - tag_map.get("eks:cluster-name", "").replace('\u200b',''), - tag_map.get("kubernetes.io/created-for/pv/name", "").replace('\u200b',''), - - ]) - - + writer.writerow( + [ + profile, + volume.volume_id, + volume.volume_type, + volume.availability_zone, + volume.size, + volume.iops, + max_iops, + ( + volume.attachments[0]["Device"] + if len(volume.attachments) > 0 + else "" + ), + ( + volume.attachments[0]["InstanceId"] + if len(volume.attachments) > 0 + else "" + ), + tag_map.get("Name", ""), + tag_map.get("Application", ""), + tag_map.get("ProjectNumber", "").replace( + "\u200b", "" + ), + tag_map.get("Project Name", "").replace( + "\u200b", "" + ), + tag_map.get("CostAllocation", "").replace( + "\u200b", "" + ), + tag_map.get("Environment", "").replace( + "\u200b", "" + ), + tag_map.get("Organization", "").replace( + "\u200b", "" + ), + tag_map.get("Creator", "").replace( + "\u200b", "" + ), + tag_map.get( + "boc:created_by", "".replace("\u200b", "") + ), + tag_map.get("Owner", "").replace("\u200b", ""), + tag_map.get("Project Role", "").replace( + "\u200b", "" + ), + tag_map.get("DeploymentID", "").replace( + "\u200b", "" + ), + tag_map.get( + "aws:cloudformation:stack-name", "" + ).replace("\u200b", ""), + tag_map.get( + "aws:cloudformation:stack-id", "" + ).replace("\u200b", ""), + tag_map.get( + "aws:cloudformation:logical-id", "" + ).replace("\u200b", ""), + tag_map.get( + "aws:elasticmapreduce:job-flow-id", "" + ).replace("\u200b", ""), + tag_map.get("eks:cluster-name", "").replace( + "\u200b", "" + ), + tag_map.get( + "kubernetes.io/created-for/pv/name", "" + ).replace("\u200b", ""), + ] + ) if __name__ == "__main__": diff --git a/local-app/python-tools/connect_to_aws/connect_to_aws.py b/local-app/python-tools/connect_to_aws/connect_to_aws.py index 247656c2..514897b1 100644 --- a/local-app/python-tools/connect_to_aws/connect_to_aws.py +++ b/local-app/python-tools/connect_to_aws/connect_to_aws.py @@ -1,208 +1,295 @@ -import boto3 import argparse import configparser import io import json -import pandas as pd import os import threading +from datetime import date, datetime + +import boto3 +import pandas as pd -from datetime import datetime, date def json_serial(obj): - """JSON serializer for objects not serializable by default json code""" - if isinstance(obj, (datetime, date)): - return obj.isoformat() - raise TypeError ("Type %s not serializable" % type(obj)) + """JSON serializer for objects not serializable by default json code""" + if isinstance(obj, (datetime, date)): + return obj.isoformat() + raise TypeError("Type %s not serializable" % type(obj)) + def main(): - parser = argparse.ArgumentParser(description='Connect to AWS accounts') - parser.add_argument('--profile', help='AWS profile') - parser.add_argument('--config', help='Config file') - parser.add_argument('--me', help='Run using available credentials', - action='store_true') - parser.add_argument('--assume', help='Run using assumed roles', - action='store_true') - parser.add_argument('--region', help='A region to run against') - parser.add_argument('--read', help = 'Read API calls from cached results', - action='store_true') - parser.add_argument('--write', help = 'Write API calls to cached results', - action='store_true') - parser.add_argument('--run_token', help ='ID for cached results to read and/or write') - - args = parser.parse_args() - - run_token = None - - if args.read: - read_from_cache = True - run_token = args.run_token - else: - read_from_cache = False - - if args.write: - write_to_cache = True - run_token = args.run_token - else: - write_to_cache = False - - if(args.config): - configParser = configparser.RawConfigParser(allow_no_value=True) + parser = argparse.ArgumentParser(description="Connect to AWS accounts") + parser.add_argument("--profile", help="AWS profile") + parser.add_argument("--config", help="Config file") + parser.add_argument( + "--me", help="Run using available credentials", action="store_true" + ) + parser.add_argument("--assume", help="Run using assumed roles", action="store_true") + parser.add_argument("--region", help="A region to run against") + parser.add_argument( + "--read", help="Read API calls from cached results", action="store_true" + ) + parser.add_argument( + "--write", help="Write API calls to cached results", action="store_true" + ) + parser.add_argument( + "--run_token", help="ID for cached results to read and/or write" + ) + + args = parser.parse_args() + + run_token = None + + if args.read: + read_from_cache = True + run_token = args.run_token + else: + read_from_cache = False + + if args.write: + write_to_cache = True + run_token = args.run_token + else: + write_to_cache = False + if args.config: - with open(args.config) as file: - configParser.read_file(file) - - if args.profile: - profiles = [args.profile] - elif args.config: - profiles = [item[0] for item in configParser.items("roles")] - profile_roles = {item[0]:item[1] for item in configParser.items("roles")} - else: - profiles = ['dev'] - - if args.region: - regions = [args.region] - elif args.config: - regions = [item[1] for item in configParser.items("regions")] - else: - regions =['us-gov-west-1'] - - if args.me: - use_roles = False - elif args.assume: - use_roles = True - else: - print ('Inconsistent configuration. Exiting. Use one of "me" or "assume".') - exit() - - session = {} - - headers = ['account','region','instance_id', 'private_dns_name','instance_type', 'vpc_id','subnet_id','state'] - - master_list_of_lists = [] - - thread_list = [] - - for profile in profiles: - for region in regions: - profile_region = f'{profile}_{region}' - if not use_roles: - session[profile_region] = boto3.Session(profile_name = profile, region_name = region) - else: - role = profile_roles[profile] - stsClient = boto3.client('sts') - response = stsClient.assume_role(RoleArn=role, RoleSessionName='AWSConnect',ExternalId='ti-jbr-2010') - accessKeyId = response['Credentials']['AccessKeyId'] - secretAccessKey = response['Credentials']['SecretAccessKey'] - sessionToken = response['Credentials']['SessionToken'] - session[profile_region] = boto3.Session(aws_access_key_id = accessKeyId, - aws_secret_access_key = secretAccessKey, - aws_session_token = sessionToken) - for profile in profiles: - account = profile.split('-')[0] - print(f'account [{account}]') - for region in regions: - profile_region = f'{profile}_{region}' - boto_client_ec2 = session[profile_region].client('ec2') - method_token = 'describe_instances' - thread = threading.Thread(name=profile_region+'_'+method_token, target = tabulate_ec2s, args = (account, region, boto_client_ec2, profile_region, method_token, master_list_of_lists, read_from_cache, write_to_cache, run_token)) - thread.start() - thread_list.append(thread) - #tabulate_ec2s(account, region, boto_client_ec2, profile_region, method_token, master_list_of_lists, read_from_cache, write_to_cache, run_token) - - for t in thread_list: - t.join() - df = data_frame(headers,master_list_of_lists) - print (df) - excel_name = ''.join(['./excel/ec2_attributes_',datetime.now().strftime('%Y%m%d%H%M%S'),'.xlsx']) - writer_keys = pd.ExcelWriter(excel_name, engine='xlsxwriter') - df.to_excel(writer_keys,sheet_name='key_attributes') - writer_keys.save() - print(f'main: wrote excel file of key details to disk') - -def tabulate_ec2s(account, region, boto_client, profile_region, method_token, master_list_of_lists, read, write, user_token): - response = paginate_wrapper(read = read, write = write, client = boto_client, client_token = f'ec2_{profile_region}', method_token = 'describe_instances', run_token =user_token) - print ('profile_region[{0}]'.format(profile_region)) - list_of_lists = flatten(account, region, response) - master_list_of_lists.extend(list_of_lists) + configParser = configparser.RawConfigParser(allow_no_value=True) + if args.config: + with open(args.config) as file: + configParser.read_file(file) + + if args.profile: + profiles = [args.profile] + elif args.config: + profiles = [item[0] for item in configParser.items("roles")] + profile_roles = {item[0]: item[1] for item in configParser.items("roles")} + else: + profiles = ["dev"] + + if args.region: + regions = [args.region] + elif args.config: + regions = [item[1] for item in configParser.items("regions")] + else: + regions = ["us-gov-west-1"] + + if args.me: + use_roles = False + elif args.assume: + use_roles = True + else: + print('Inconsistent configuration. Exiting. Use one of "me" or "assume".') + exit() + + session = {} + + headers = [ + "account", + "region", + "instance_id", + "private_dns_name", + "instance_type", + "vpc_id", + "subnet_id", + "state", + ] + + master_list_of_lists = [] + + thread_list = [] + + for profile in profiles: + for region in regions: + profile_region = f"{profile}_{region}" + if not use_roles: + session[profile_region] = boto3.Session( + profile_name=profile, region_name=region + ) + else: + role = profile_roles[profile] + stsClient = boto3.client("sts") + response = stsClient.assume_role( + RoleArn=role, RoleSessionName="AWSConnect", ExternalId="ti-jbr-2010" + ) + accessKeyId = response["Credentials"]["AccessKeyId"] + secretAccessKey = response["Credentials"]["SecretAccessKey"] + sessionToken = response["Credentials"]["SessionToken"] + session[profile_region] = boto3.Session( + aws_access_key_id=accessKeyId, + aws_secret_access_key=secretAccessKey, + aws_session_token=sessionToken, + ) + for profile in profiles: + account = profile.split("-")[0] + print(f"account [{account}]") + for region in regions: + profile_region = f"{profile}_{region}" + boto_client_ec2 = session[profile_region].client("ec2") + method_token = "describe_instances" + thread = threading.Thread( + name=profile_region + "_" + method_token, + target=tabulate_ec2s, + args=( + account, + region, + boto_client_ec2, + profile_region, + method_token, + master_list_of_lists, + read_from_cache, + write_to_cache, + run_token, + ), + ) + thread.start() + thread_list.append(thread) + # tabulate_ec2s(account, region, boto_client_ec2, profile_region, method_token, master_list_of_lists, read_from_cache, write_to_cache, run_token) + + for t in thread_list: + t.join() + df = data_frame(headers, master_list_of_lists) + print(df) + excel_name = "".join( + ["./excel/ec2_attributes_", datetime.now().strftime("%Y%m%d%H%M%S"), ".xlsx"] + ) + writer_keys = pd.ExcelWriter(excel_name, engine="xlsxwriter") + df.to_excel(writer_keys, sheet_name="key_attributes") + writer_keys.save() + print(f"main: wrote excel file of key details to disk") + + +def tabulate_ec2s( + account, + region, + boto_client, + profile_region, + method_token, + master_list_of_lists, + read, + write, + user_token, +): + response = paginate_wrapper( + read=read, + write=write, + client=boto_client, + client_token=f"ec2_{profile_region}", + method_token="describe_instances", + run_token=user_token, + ) + print("profile_region[{0}]".format(profile_region)) + list_of_lists = flatten(account, region, response) + master_list_of_lists.extend(list_of_lists) + def flatten(account, region, response): - list_of_lists = [] - for page in response: - for reservation in page['Reservations']: - for instance in reservation['Instances']: - state = instance['State']['Name'] - if state == 'running': - list = [account, region, instance['InstanceId'], instance['PrivateDnsName'], instance['InstanceType'], instance['VpcId'], instance['SubnetId'], state ] - list_of_lists.append(list) - return list_of_lists + list_of_lists = [] + for page in response: + for reservation in page["Reservations"]: + for instance in reservation["Instances"]: + state = instance["State"]["Name"] + if state == "running": + list = [ + account, + region, + instance["InstanceId"], + instance["PrivateDnsName"], + instance["InstanceType"], + instance["VpcId"], + instance["SubnetId"], + state, + ] + list_of_lists.append(list) + return list_of_lists + def data_frame(headers, list_of_lists): - df = pd.DataFrame(columns = headers, data=list_of_lists) + df = pd.DataFrame(columns=headers, data=list_of_lists) return df -def paginate_wrapper(read=None, write=None, client=None, client_token=None, run_token=None, method_token='default', parameter_dict = {},parameter_token=''): - # print(f'paginate_wrapper: processing client_token[{client_token}] method_token [{method_token}] run_token[{run_token}]') - # if neither read nor write is specified, then look for a serialization. If it doesn't exist, then write - normalized_parameter_token = parameter_token.translate({ord(':'):'-'}) - if read and write: - if serialization_exists([client_token, method_token, run_token,normalized_parameter_token]): - read_action = True - write_action = False +def paginate_wrapper( + read=None, + write=None, + client=None, + client_token=None, + run_token=None, + method_token="default", + parameter_dict={}, + parameter_token="", +): + # print(f'paginate_wrapper: processing client_token[{client_token}] method_token [{method_token}] run_token[{run_token}]') + # if neither read nor write is specified, then look for a serialization. If it doesn't exist, then write + normalized_parameter_token = parameter_token.translate({ord(":"): "-"}) + if read and write: + if serialization_exists( + [client_token, method_token, run_token, normalized_parameter_token] + ): + read_action = True + write_action = False + else: + write_action = True + read_action = False else: - write_action = True - read_action = False - else: - read_action = read - write_action = write - if read_action: - result = deserialize([client_token, method_token,run_token,normalized_parameter_token]) - else: - if client.can_paginate(method_token): - paginator = client.get_paginator(method_token) - # print(f'paginate_wrapper: method_token[{method_token}] keys in parameter_dict [{parameter_dict.keys()}]') - page_iterator = paginator.paginate(**parameter_dict) - result = [page for page in page_iterator] + read_action = read + write_action = write + if read_action: + result = deserialize( + [client_token, method_token, run_token, normalized_parameter_token] + ) else: - func = getattr(client,method_token) - # print(f'paginate_wrapper: method_token [{method_token}] keys in parameter_dict [{parameter_dict.keys()}]') - # parameter_string = ', '.join([f'parameter[{parameter}] value [{parameter_dict[parameter]}]' for parameter in parameter_dict]) - # print(f'paginate_wrapper: parameter_string <{parameter_string}>') - result = [func(**parameter_dict)] - # print(f'paginate_wrapper: result [{result}]') + if client.can_paginate(method_token): + paginator = client.get_paginator(method_token) + # print(f'paginate_wrapper: method_token[{method_token}] keys in parameter_dict [{parameter_dict.keys()}]') + page_iterator = paginator.paginate(**parameter_dict) + result = [page for page in page_iterator] + else: + func = getattr(client, method_token) + # print(f'paginate_wrapper: method_token [{method_token}] keys in parameter_dict [{parameter_dict.keys()}]') + # parameter_string = ', '.join([f'parameter[{parameter}] value [{parameter_dict[parameter]}]' for parameter in parameter_dict]) + # print(f'paginate_wrapper: parameter_string <{parameter_string}>') + result = [func(**parameter_dict)] + # print(f'paginate_wrapper: result [{result}]') + + if write_action: + serialize( + result, + [client_token, method_token, run_token, normalized_parameter_token], + ) + return result - if write_action: - serialize(result, [client_token, method_token, run_token,normalized_parameter_token]) - return result def serialize(obj, token_list): - serialized = json.dumps(obj, default=json_serial) - filename = build_json_filename(token_list) - with open(filename, 'w') as file: - file.write(serialized) + serialized = json.dumps(obj, default=json_serial) + filename = build_json_filename(token_list) + with open(filename, "w") as file: + file.write(serialized) + def deserialize(token_list): - filename = build_json_filename(token_list) - with open(filename) as file: - serialized = file.read() - deserialized = json.loads(serialized) - return deserialized + filename = build_json_filename(token_list) + with open(filename) as file: + serialized = file.read() + deserialized = json.loads(serialized) + return deserialized + def build_json_filename(string_list): - joined_list = '-'.join(string_list) - return f'c:/cache/json/{joined_list}.json' + joined_list = "-".join(string_list) + return f"c:/cache/json/{joined_list}.json" + def serialization_exists(token_list): - filename = build_json_filename(token_list) - return os.path.isfile(filename) + filename = build_json_filename(token_list) + return os.path.isfile(filename) + def json_serial(obj): - """JSON serializer for objects not serializable by default json code""" - if isinstance(obj, (datetime, date)): - return obj.isoformat() - raise TypeError ("Type %s not serializable" % type(obj)) + """JSON serializer for objects not serializable by default json code""" + if isinstance(obj, (datetime, date)): + return obj.isoformat() + raise TypeError("Type %s not serializable" % type(obj)) if __name__ == "__main__": - main() + main() diff --git a/local-app/python-tools/gfl-resource-actions/aws_resource_management.py b/local-app/python-tools/gfl-resource-actions/aws_resource_management.py index a578e284..c7913ec9 100644 --- a/local-app/python-tools/gfl-resource-actions/aws_resource_management.py +++ b/local-app/python-tools/gfl-resource-actions/aws_resource_management.py @@ -4,28 +4,28 @@ Acts as a wrapper for the main.py script. """ -import sys -import os -import tempfile -import subprocess import concurrent.futures import logging +import os +import subprocess +import sys +import tempfile # Configure logging logging.basicConfig( - level=logging.INFO, - format='%(asctime)s [%(levelname)s] %(name)s - %(message)s' + level=logging.INFO, format="%(asctime)s [%(levelname)s] %(name)s - %(message)s" ) -logger = logging.getLogger('aws_resource_management') +logger = logging.getLogger("aws_resource_management") + def process_account(account_id, script_path, script_dir, args): """Process a single account with the main script.""" try: cmd = [sys.executable, script_path] + args env = os.environ.copy() - env['PYTHONPATH'] = script_dir + os.pathsep + env.get('PYTHONPATH', '') - env['AWS_ACCOUNT'] = account_id - + env["PYTHONPATH"] = script_dir + os.pathsep + env.get("PYTHONPATH", "") + env["AWS_ACCOUNT"] = account_id + result = subprocess.run(cmd, env=env, capture_output=True, text=True) if result.returncode != 0: logger.error(f"Error processing account {account_id}: {result.stderr}") @@ -34,51 +34,50 @@ def process_account(account_id, script_path, script_dir, args): logger.error(f"Error processing account {account_id}: {str(e)}") return 1 + if __name__ == "__main__": # Get the current script directory script_dir = os.path.dirname(os.path.abspath(__file__)) - + # Path to main.py main_path = os.path.join(script_dir, "main.py") - + # Check if the main script exists if not os.path.exists(main_path): print(f"Error: Main script not found at {main_path}") sys.exit(1) - + # Create a temporary version of main.py with absolute imports with open(main_path, "r") as f: content = f.read() - + # Replace relative imports with absolute imports content = content.replace("from . import config", "import config") content = content.replace("from .logging_utils", "from logging_utils") content = content.replace("from .aws_utils", "from aws_utils") content = content.replace("from .discovery", "from discovery") content = content.replace("from .managers", "from managers") - + # Write to temporary file temp_file = tempfile.NamedTemporaryFile(suffix=".py", delete=False) - temp_file.write(content.encode('utf-8')) + temp_file.write(content.encode("utf-8")) temp_file.close() - + try: # Get list of accounts to process - accounts = [os.environ.get('AWS_ACCOUNT_ID', '')] - + accounts = [os.environ.get("AWS_ACCOUNT_ID", "")] + # Create a thread pool for parallel execution with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor: # Submit account processing tasks futures = { executor.submit( - process_account, - account, - temp_file.name, - script_dir, - sys.argv[1:] - ): account for account in accounts if account + process_account, account, temp_file.name, script_dir, sys.argv[1:] + ): account + for account in accounts + if account } - + # Wait for all tasks to complete results = [] for future in concurrent.futures.as_completed(futures): @@ -86,9 +85,11 @@ def process_account(account_id, script_path, script_dir, args): try: results.append(future.result()) except Exception as e: - logger.error(f"Thread execution error for account {account}: {str(e)}") + logger.error( + f"Thread execution error for account {account}: {str(e)}" + ) results.append(1) - + # Exit with error if any account processing failed sys.exit(1 if any(result != 0 for result in results) else 0) finally: diff --git a/local-app/python-tools/gfl-resource-actions/aws_resource_management/__main__.py b/local-app/python-tools/gfl-resource-actions/aws_resource_management/__main__.py index 5ea659d0..42096049 100644 --- a/local-app/python-tools/gfl-resource-actions/aws_resource_management/__main__.py +++ b/local-app/python-tools/gfl-resource-actions/aws_resource_management/__main__.py @@ -3,6 +3,7 @@ """ import sys + from aws_resource_management.cli_optimized import main if __name__ == "__main__": diff --git a/local-app/python-tools/gfl-resource-actions/aws_resource_management/aws_utils.py b/local-app/python-tools/gfl-resource-actions/aws_resource_management/aws_utils.py index 40d46bb3..fc0c15e8 100644 --- a/local-app/python-tools/gfl-resource-actions/aws_resource_management/aws_utils.py +++ b/local-app/python-tools/gfl-resource-actions/aws_resource_management/aws_utils.py @@ -1,123 +1,131 @@ """ AWS utility functions for credential management and account discovery. """ -import os -import logging -import boto3 -import botocore + +import concurrent.futures import configparser -from typing import Dict, List, Optional, Any, Tuple -from concurrent.futures import ThreadPoolExecutor, as_completed +import logging +import os import threading -import concurrent.futures +from concurrent.futures import ThreadPoolExecutor, as_completed +from typing import Any, Dict, List, Optional, Tuple +import boto3 +import botocore from aws_resource_management.logging_setup import setup_logging logger = setup_logging() # Thread-local storage for session caching _thread_local = threading.local() + def get_account_list() -> List[Dict[str, str]]: """ Get list of accounts from AWS Organizations. - + Returns: List of dictionaries containing account_id and account_name """ try: # First try to use organizations API session = boto3.Session() - org_client = session.client('organizations') - + org_client = session.client("organizations") + accounts = [] - paginator = org_client.get_paginator('list_accounts') - + paginator = org_client.get_paginator("list_accounts") + for page in paginator.paginate(): - for account in page['Accounts']: - if account['Status'] == 'ACTIVE': - accounts.append({ - 'account_id': account['Id'], - 'account_name': account['Name'] - }) - + for account in page["Accounts"]: + if account["Status"] == "ACTIVE": + accounts.append( + {"account_id": account["Id"], "account_name": account["Name"]} + ) + return accounts - + except (botocore.exceptions.ClientError, botocore.exceptions.BotoCoreError) as e: logger.warning(f"Failed to get accounts from Organizations API: {str(e)}") logger.info("Falling back to configured AWS profiles") - + # Fall back to configured profiles if Organizations API access fails return get_accounts_from_profiles() + def get_accounts_from_profiles() -> List[Dict[str, str]]: """ Get list of accounts from configured AWS profiles in ~/.aws/config. - + Returns: List of dictionaries containing account_id and account_name """ accounts = [] # Dictionary to store the first profile seen for each account ID account_profiles = {} - + try: # Read AWS config file config_path = os.path.expanduser("~/.aws/config") if not os.path.exists(config_path): logger.warning(f"AWS config file not found at {config_path}") return [] - + config = configparser.ConfigParser() config.read(config_path) - + # Extract account information from profiles for section in config.sections(): # Skip sections that don't have account ID if not config.has_option(section, "sso_account_id"): continue - + account_id = config.get(section, "sso_account_id") section_name = section if section.startswith("profile "): section_name = section[8:] # Remove "profile " prefix - + # For each account ID, remember only the first profile we encounter # Unless it's a profile with a preferred role like AdministratorAccess or inf-admin-t2 if account_id not in account_profiles or ( - config.has_option(section, "sso_role_name") and - config.get(section, "sso_role_name") in ["AdministratorAccess", "inf-admin-t2"] + config.has_option(section, "sso_role_name") + and config.get(section, "sso_role_name") + in ["AdministratorAccess", "inf-admin-t2"] ): # Get account name if available, otherwise use account ID account_name = account_id if config.has_option(section, "sso_account_name"): account_name = config.get(section, "sso_account_name") - + account_profiles[account_id] = { - 'account_id': account_id, - 'account_name': account_name, - 'profile': section_name + "account_id": account_id, + "account_name": account_name, + "profile": section_name, } - - logger.debug(f"Using profile {section_name} for account {account_id} ({account_name})") - + + logger.debug( + f"Using profile {section_name} for account {account_id} ({account_name})" + ) + # Convert dictionary values to list accounts = list(account_profiles.values()) - + logger.info(f"Found {len(accounts)} accounts from AWS profiles") return accounts - + except Exception as e: logger.error(f"Error reading AWS profiles: {str(e)}") return [] -def get_credentials(account_id: str, profile_name: Optional[str] = None) -> Optional[Dict[str, Any]]: + +def get_credentials( + account_id: str, profile_name: Optional[str] = None +) -> Optional[Dict[str, Any]]: """ Get AWS credentials for the specified account. - + Args: account_id: AWS account ID profile_name: Optional AWS profile name to use - + Returns: Dict with AWS credentials or None if failed """ @@ -127,57 +135,70 @@ def get_credentials(account_id: str, profile_name: Optional[str] = None) -> Opti logger.debug(f"Using specified profile: {profile_name}") try: session = boto3.Session(profile_name=profile_name) - + # Verify we can get credentials from the session if session.get_credentials() is None: - logger.warning(f"No credentials available for profile {profile_name}") + logger.warning( + f"No credentials available for profile {profile_name}" + ) return None - + return { - 'aws_access_key_id': session.get_credentials().access_key, - 'aws_secret_access_key': session.get_credentials().secret_key, - 'aws_session_token': session.get_credentials().token + "aws_access_key_id": session.get_credentials().access_key, + "aws_secret_access_key": session.get_credentials().secret_key, + "aws_session_token": session.get_credentials().token, } - except (botocore.exceptions.ProfileNotFound, botocore.exceptions.ClientError) as e: + except ( + botocore.exceptions.ProfileNotFound, + botocore.exceptions.ClientError, + ) as e: logger.warning(f"Error using profile {profile_name}: {str(e)}") # Fall through to try finding another profile - + # Find profile for the specified account matching_profile = find_profile_by_account_id(account_id) if matching_profile: logger.debug(f"Using profile {matching_profile} for account {account_id}") try: session = boto3.Session(profile_name=matching_profile) - + # Verify we can get credentials from the session if session.get_credentials() is None: - logger.warning(f"No credentials available for profile {matching_profile}") + logger.warning( + f"No credentials available for profile {matching_profile}" + ) return None - + return { - 'aws_access_key_id': session.get_credentials().access_key, - 'aws_secret_access_key': session.get_credentials().secret_key, - 'aws_session_token': session.get_credentials().token + "aws_access_key_id": session.get_credentials().access_key, + "aws_secret_access_key": session.get_credentials().secret_key, + "aws_session_token": session.get_credentials().token, } - except (botocore.exceptions.ProfileNotFound, botocore.exceptions.ClientError) as e: + except ( + botocore.exceptions.ProfileNotFound, + botocore.exceptions.ClientError, + ) as e: logger.warning(f"Error using profile {matching_profile}: {str(e)}") # Fall through to role assumption - + # If no profile is found or profile didn't work, fall back to role assumption - logger.debug(f"No working profile found for account {account_id}, falling back to role assumption") + logger.debug( + f"No working profile found for account {account_id}, falling back to role assumption" + ) return assume_role_credentials(account_id) - + except Exception as e: logger.error(f"Error getting credentials for account {account_id}: {str(e)}") return None + def find_profile_by_account_id(account_id: str) -> Optional[str]: """ Find an AWS profile name that matches the specified account ID. - + Args: account_id: AWS account ID - + Returns: Profile name or None if not found """ @@ -185,154 +206,165 @@ def find_profile_by_account_id(account_id: str) -> Optional[str]: config_path = os.path.expanduser("~/.aws/config") if not os.path.exists(config_path): return None - + config = configparser.ConfigParser() config.read(config_path) - + # Try to find profiles in order of preference preferred_profiles = [ f"{account_id}.AdministratorAccess", f"{account_id}.inf-admin-t2", f"{account_id}-gov.administratoraccess", - f"{account_id}-gov.inf-admin-t2" + f"{account_id}-gov.inf-admin-t2", ] - + # First check for preferred profiles for profile_name in preferred_profiles: for section in config.sections(): section_name = section if section.startswith("profile "): section_name = section[8:] # Remove "profile " prefix - + if section_name.lower() == profile_name.lower(): - logger.debug(f"Found preferred profile {section_name} for account {account_id}") + logger.debug( + f"Found preferred profile {section_name} for account {account_id}" + ) return section_name - + # If no preferred profile found, look for any profile with this account ID for section in config.sections(): if not config.has_option(section, "sso_account_id"): continue - + profile_account_id = config.get(section, "sso_account_id") if profile_account_id == account_id: section_name = section if section.startswith("profile "): section_name = section[8:] # Remove "profile " prefix - + logger.debug(f"Found profile {section_name} for account {account_id}") return section_name - + logger.debug(f"No profile found for account {account_id}") return None - + except Exception as e: logger.error(f"Error finding profile for account {account_id}: {str(e)}") return None + def assume_role_credentials(account_id: str) -> Optional[Dict[str, Any]]: """ Get credentials by assuming a role in the target account. - + Args: account_id: AWS account ID - + Returns: Dict with AWS credentials or None if failed """ try: - sts_client = boto3.client('sts') - + sts_client = boto3.client("sts") + # Try common role names - role_names = ['OrganizationAccountAccessRole', 'AWSControlTowerExecution'] - + role_names = ["OrganizationAccountAccessRole", "AWSControlTowerExecution"] + for role_name in role_names: try: role_arn = f"arn:aws:iam::{account_id}:role/{role_name}" response = sts_client.assume_role( - RoleArn=role_arn, - RoleSessionName='ResourceManagementSession' + RoleArn=role_arn, RoleSessionName="ResourceManagementSession" ) - - credentials = response['Credentials'] + + credentials = response["Credentials"] return { - 'aws_access_key_id': credentials['AccessKeyId'], - 'aws_secret_access_key': credentials['SecretAccessKey'], - 'aws_session_token': credentials['SessionToken'] + "aws_access_key_id": credentials["AccessKeyId"], + "aws_secret_access_key": credentials["SecretAccessKey"], + "aws_session_token": credentials["SessionToken"], } except botocore.exceptions.ClientError: continue - + logger.warning(f"Could not assume any role in account {account_id}") return None - + except Exception as e: logger.error(f"Error assuming role for account {account_id}: {str(e)}") return None + def get_partition_for_region(region: str) -> str: """ Get the AWS partition for a region. - + Args: region: AWS region name - + Returns: AWS partition (aws, aws-us-gov, etc.) """ # Default to commercial AWS partition = "aws" - + # Check for GovCloud regions if region.startswith("us-gov"): partition = "aws-us-gov" elif region.startswith("cn-"): partition = "aws-cn" - + return partition + def get_all_regions(partition: Optional[str] = None) -> List[str]: """ Get all available AWS regions, filtered by partition if specified. - + Args: partition: Optional AWS partition to filter by - + Returns: List of region names """ try: - if partition == 'aws-us-gov': - return ['us-gov-east-1', 'us-gov-west-1'] - elif partition == 'aws-cn': - return ['cn-north-1', 'cn-northwest-1'] - + if partition == "aws-us-gov": + return ["us-gov-east-1", "us-gov-west-1"] + elif partition == "aws-cn": + return ["cn-north-1", "cn-northwest-1"] + # For standard AWS or no partition specified, try to get all regions - ec2 = boto3.client('ec2', region_name='us-east-1') - regions = [region['RegionName'] for region in ec2.describe_regions()['Regions']] - + ec2 = boto3.client("ec2", region_name="us-east-1") + regions = [region["RegionName"] for region in ec2.describe_regions()["Regions"]] + # If a partition was specified, filter the results - if partition == 'aws': - regions = [r for r in regions if not (r.startswith('us-gov-') or r.startswith('cn-'))] - + if partition == "aws": + regions = [ + r + for r in regions + if not (r.startswith("us-gov-") or r.startswith("cn-")) + ] + return regions except Exception as e: logger.warning(f"Error getting all regions: {e}") # Default to common regions based on partition - if partition == 'aws-us-gov': - return ['us-gov-east-1', 'us-gov-west-1'] - elif partition == 'aws-cn': - return ['cn-north-1', 'cn-northwest-1'] + if partition == "aws-us-gov": + return ["us-gov-east-1", "us-gov-west-1"] + elif partition == "aws-cn": + return ["cn-north-1", "cn-northwest-1"] else: - return ['us-east-1', 'us-east-2', 'us-west-1', 'us-west-2'] + return ["us-east-1", "us-east-2", "us-west-1", "us-west-2"] + -def get_enabled_regions(credentials: Dict[str, str], partition: Optional[str] = None) -> List[str]: +def get_enabled_regions( + credentials: Dict[str, str], partition: Optional[str] = None +) -> List[str]: """ Get list of AWS regions enabled for the account. - + Args: credentials: AWS credentials dictionary partition: AWS partition (aws, aws-us-gov, aws-cn) - + Returns: List of enabled region names """ @@ -340,60 +372,63 @@ def get_enabled_regions(credentials: Dict[str, str], partition: Optional[str] = # If no partition was specified, detect it from credentials if not partition: partition = detect_partition_from_credentials(credentials) - + # For GovCloud or China partitions, return their standard regions # as the describe_regions call might not work with these credentials - if partition == 'aws-us-gov': - return ['us-gov-east-1', 'us-gov-west-1'] - elif partition == 'aws-cn': - return ['cn-north-1', 'cn-northwest-1'] - + if partition == "aws-us-gov": + return ["us-gov-east-1", "us-gov-west-1"] + elif partition == "aws-cn": + return ["cn-north-1", "cn-northwest-1"] + # For standard partition, try to discover all enabled regions session = boto3.Session( - aws_access_key_id=credentials.get('aws_access_key_id'), - aws_secret_access_key=credentials.get('aws_secret_access_key'), - aws_session_token=credentials.get('aws_session_token') + aws_access_key_id=credentials.get("aws_access_key_id"), + aws_secret_access_key=credentials.get("aws_secret_access_key"), + aws_session_token=credentials.get("aws_session_token"), ) - - ec2_client = session.client('ec2', region_name='us-east-1') - regions = [region['RegionName'] for region in ec2_client.describe_regions()['Regions']] - + + ec2_client = session.client("ec2", region_name="us-east-1") + regions = [ + region["RegionName"] for region in ec2_client.describe_regions()["Regions"] + ] + # Filter to only enabled regions enabled_regions = [] for region in regions: if _is_region_enabled(region, credentials): enabled_regions.append(region) - + return enabled_regions except Exception as e: logger.warning(f"Error getting all regions: {str(e)}") # Fallback to default regions based on partition - if partition == 'aws-us-gov': - return ['us-gov-east-1', 'us-gov-west-1'] - elif partition == 'aws-cn': - return ['cn-north-1', 'cn-northwest-1'] - return ['us-east-1', 'us-east-2', 'us-west-1', 'us-west-2'] + if partition == "aws-us-gov": + return ["us-gov-east-1", "us-gov-west-1"] + elif partition == "aws-cn": + return ["cn-north-1", "cn-northwest-1"] + return ["us-east-1", "us-east-2", "us-west-1", "us-west-2"] + def _is_region_enabled(region: str, credentials: Dict[str, str]) -> bool: """ Check if a region is enabled for the account. - + Args: region: AWS region name credentials: AWS credentials dictionary - + Returns: True if the region is enabled, False otherwise """ try: session = boto3.Session( - aws_access_key_id=credentials.get('aws_access_key_id'), - aws_secret_access_key=credentials.get('aws_secret_access_key'), - aws_session_token=credentials.get('aws_session_token') + aws_access_key_id=credentials.get("aws_access_key_id"), + aws_secret_access_key=credentials.get("aws_secret_access_key"), + aws_session_token=credentials.get("aws_session_token"), ) - + # Try to create a client and make a lightweight API call - ec2_client = session.client('ec2', region_name=region) + ec2_client = session.client("ec2", region_name=region) try: # Try a lightweight call first ec2_client.describe_regions(RegionNames=[region]) @@ -410,151 +445,169 @@ def _is_region_enabled(region: str, credentials: Dict[str, str]) -> bool: # Could not create client or all calls failed return False + def is_valid_region(region: str) -> bool: """ Check if the given region is valid. - + Args: region: AWS region name - + Returns: True if valid, False otherwise """ try: - boto3.session.Session().client('ec2', region_name=region) + boto3.session.Session().client("ec2", region_name=region) return True except botocore.exceptions.ClientError: return False -def get_session_for_account(account_id: str, region: str = None, profile_name: Optional[str] = None) -> Optional[boto3.Session]: + +def get_session_for_account( + account_id: str, region: str = None, profile_name: Optional[str] = None +) -> Optional[boto3.Session]: """ Get or create a boto3 session for the specified account and region. Uses thread-local storage to cache sessions for better performance. - + Args: account_id: AWS account ID region: Region name (optional) profile_name: AWS profile name (optional) - + Returns: boto3.Session object or None if failed """ # Get thread-local storage for sessions - if not hasattr(_thread_local, 'sessions'): + if not hasattr(_thread_local, "sessions"): _thread_local.sessions = {} - + # Create a key from account ID, region, and profile name key = f"{account_id}:{region or 'default'}:{profile_name or 'default'}" - + # Return cached session if it exists if key in _thread_local.sessions: return _thread_local.sessions[key] - + try: # Get credentials for the account credentials = get_credentials(account_id, profile_name=profile_name) if not credentials: return None - + # Create a new session session = boto3.Session( - aws_access_key_id=credentials['aws_access_key_id'], - aws_secret_access_key=credentials['aws_secret_access_key'], - aws_session_token=credentials.get('aws_session_token'), - region_name=region + aws_access_key_id=credentials["aws_access_key_id"], + aws_secret_access_key=credentials["aws_secret_access_key"], + aws_session_token=credentials.get("aws_session_token"), + region_name=region, ) - + # Cache the session _thread_local.sessions[key] = session return session - + except Exception as e: logger.error(f"Error creating session for account {account_id}: {str(e)}") return None + def detect_partition_from_credentials(credentials: Dict[str, str]) -> str: """ Detect which AWS partition these credentials are valid for. Tests credentials against key regions from different partitions to determine where they're valid. - + Args: credentials: AWS credentials dictionary - + Returns: AWS partition name (aws, aws-us-gov, aws-cn) """ logger.info("Detecting AWS partition from credentials...") - + # Try GovCloud first since that's a common use case in this system try: client = boto3.client( - 'sts', - region_name='us-gov-west-1', - aws_access_key_id=credentials.get('aws_access_key_id'), - aws_secret_access_key=credentials.get('aws_secret_access_key'), - aws_session_token=credentials.get('aws_session_token') + "sts", + region_name="us-gov-west-1", + aws_access_key_id=credentials.get("aws_access_key_id"), + aws_secret_access_key=credentials.get("aws_secret_access_key"), + aws_session_token=credentials.get("aws_session_token"), ) client.get_caller_identity() logger.info("Credentials valid for GovCloud partition (aws-us-gov)") - return 'aws-us-gov' + return "aws-us-gov" except Exception: logger.debug("Credentials not valid for GovCloud partition") - + # Try commercial AWS try: client = boto3.client( - 'sts', - region_name='us-east-1', - aws_access_key_id=credentials.get('aws_access_key_id'), - aws_secret_access_key=credentials.get('aws_secret_access_key'), - aws_session_token=credentials.get('aws_session_token') + "sts", + region_name="us-east-1", + aws_access_key_id=credentials.get("aws_access_key_id"), + aws_secret_access_key=credentials.get("aws_secret_access_key"), + aws_session_token=credentials.get("aws_session_token"), ) client.get_caller_identity() logger.info("Credentials valid for standard AWS partition (aws)") - return 'aws' + return "aws" except Exception: logger.debug("Credentials not valid for standard AWS partition") - + # Try China partition try: client = boto3.client( - 'sts', - region_name='cn-north-1', - aws_access_key_id=credentials.get('aws_access_key_id'), - aws_secret_access_key=credentials.get('aws_secret_access_key'), - aws_session_token=credentials.get('aws_session_token') + "sts", + region_name="cn-north-1", + aws_access_key_id=credentials.get("aws_access_key_id"), + aws_secret_access_key=credentials.get("aws_secret_access_key"), + aws_session_token=credentials.get("aws_session_token"), ) client.get_caller_identity() logger.info("Credentials valid for China AWS partition (aws-cn)") - return 'aws-cn' + return "aws-cn" except Exception: logger.debug("Credentials not valid for China AWS partition") - + # Default to standard AWS if we couldn't determine - logger.warning("Could not determine valid partition for credentials - defaulting to aws") - return 'aws' + logger.warning( + "Could not determine valid partition for credentials - defaulting to aws" + ) + return "aws" + def get_regions_for_partition(partition: str) -> List[str]: """ Get list of regions available in the specified AWS partition. - + Args: partition: AWS partition (aws, aws-us-gov, aws-cn) - + Returns: List of region names for the partition """ - if partition == 'aws-us-gov': - return ['us-gov-east-1', 'us-gov-west-1'] - elif partition == 'aws-cn': - return ['cn-north-1', 'cn-northwest-1'] + if partition == "aws-us-gov": + return ["us-gov-east-1", "us-gov-west-1"] + elif partition == "aws-cn": + return ["cn-north-1", "cn-northwest-1"] else: # Default to commercial AWS regions try: - ec2 = boto3.client('ec2', region_name='us-east-1') - regions = [region['RegionName'] for region in ec2.describe_regions()['Regions']] + ec2 = boto3.client("ec2", region_name="us-east-1") + regions = [ + region["RegionName"] for region in ec2.describe_regions()["Regions"] + ] return regions except Exception as e: logger.warning(f"Could not fetch regions for partition {partition}: {e}") # Return common commercial regions as fallback - return ['us-east-1', 'us-east-2', 'us-west-1', 'us-west-2', - 'eu-west-1', 'eu-central-1', 'ap-southeast-1', 'ap-southeast-2'] + return [ + "us-east-1", + "us-east-2", + "us-west-1", + "us-west-2", + "eu-west-1", + "eu-central-1", + "ap-southeast-1", + "ap-southeast-2", + ] diff --git a/local-app/python-tools/gfl-resource-actions/aws_resource_management/cli.py b/local-app/python-tools/gfl-resource-actions/aws_resource_management/cli.py index df7787cb..7b964f6f 100644 --- a/local-app/python-tools/gfl-resource-actions/aws_resource_management/cli.py +++ b/local-app/python-tools/gfl-resource-actions/aws_resource_management/cli.py @@ -6,12 +6,13 @@ import sys import warnings + from aws_resource_management.cli_optimized import main warnings.warn( "The cli module is deprecated. Please use cli_optimized module instead.", DeprecationWarning, - stacklevel=2 + stacklevel=2, ) # Redirect to the optimized CLI diff --git a/local-app/python-tools/gfl-resource-actions/aws_resource_management/cli_optimized.py b/local-app/python-tools/gfl-resource-actions/aws_resource_management/cli_optimized.py index e4d8ca69..7fb7d46a 100644 --- a/local-app/python-tools/gfl-resource-actions/aws_resource_management/cli_optimized.py +++ b/local-app/python-tools/gfl-resource-actions/aws_resource_management/cli_optimized.py @@ -5,82 +5,116 @@ """ import argparse -import sys import logging +import sys import traceback -from typing import List, Optional, Dict, Any +from typing import Any, Dict, List, Optional -from aws_resource_management.logging_setup import setup_logging -from aws_resource_management.core import ResourceManager from aws_resource_management.config_manager import get_config +from aws_resource_management.core import ResourceManager +from aws_resource_management.logging_setup import setup_logging logger = setup_logging() config = get_config() + def parse_args(): """Parse command line arguments.""" parser = argparse.ArgumentParser(description="AWS Resource Management CLI") - + # Action arguments action_group = parser.add_mutually_exclusive_group(required=True) - action_group.add_argument('--stop', action='store_true', help='Stop resources') - action_group.add_argument('--start', action='store_true', help='Start resources') - + action_group.add_argument("--stop", action="store_true", help="Stop resources") + action_group.add_argument("--start", action="store_true", help="Start resources") + # Resource selection - parser.add_argument('--resource-type', choices=['ec2', 'rds', 'eks', 'emr', 'all'], - default='all', help='Resource type to manage (default: all)') - + parser.add_argument( + "--resource-type", + choices=["ec2", "rds", "eks", "emr", "all"], + default="all", + help="Resource type to manage (default: all)", + ) + # Account filtering - parser.add_argument('--account', help='Specific account ID to process') - parser.add_argument('--exclude-account', action='append', dest='exclude_accounts', - help='Account ID to exclude (can be used multiple times)') - + parser.add_argument("--account", help="Specific account ID to process") + parser.add_argument( + "--exclude-account", + action="append", + dest="exclude_accounts", + help="Account ID to exclude (can be used multiple times)", + ) + # Region options - parser.add_argument('--region', action='append', dest='regions', - help='Regions to scan (can be used multiple times)') - parser.add_argument('--exclude-region', action='append', dest='exclude_regions', - help='Regions to exclude (can be used multiple times)') - parser.add_argument('--discover-regions', action='store_true', - help='Automatically discover enabled regions') - parser.add_argument('--partition', choices=['aws', 'aws-us-gov', 'aws-cn'], - help='AWS partition to operate in') - + parser.add_argument( + "--region", + action="append", + dest="regions", + help="Regions to scan (can be used multiple times)", + ) + parser.add_argument( + "--exclude-region", + action="append", + dest="exclude_regions", + help="Regions to exclude (can be used multiple times)", + ) + parser.add_argument( + "--discover-regions", + action="store_true", + help="Automatically discover enabled regions", + ) + parser.add_argument( + "--partition", + choices=["aws", "aws-us-gov", "aws-cn"], + help="AWS partition to operate in", + ) + # Authentication - parser.add_argument('--profile', help='AWS profile to use') - parser.add_argument('--use-profiles', action='store_true', - help='Use AWS profiles for account discovery') - + parser.add_argument("--profile", help="AWS profile to use") + parser.add_argument( + "--use-profiles", + action="store_true", + help="Use AWS profiles for account discovery", + ) + # Dry run - parser.add_argument('--dry-run', action='store_true', - help='Show what would be done without making changes') - + parser.add_argument( + "--dry-run", + action="store_true", + help="Show what would be done without making changes", + ) + return parser.parse_args() + def main(): """Main CLI entry point.""" args = parse_args() - + try: # Determine action - action = 'stop' if args.stop else 'start' - + action = "stop" if args.stop else "start" + # Determine resource types to process exclude_resources = [] - if args.resource_type != 'all': + if args.resource_type != "all": # If specific resource type is selected, exclude all others - all_resource_types = ['ec2', 'rds', 'eks', 'emr'] - exclude_resources = [rt for rt in all_resource_types if rt != args.resource_type] - - logger.info(f"Processing {args.resource_type} resources in {args.regions or 'all enabled'} regions for {action} action") - + all_resource_types = ["ec2", "rds", "eks", "emr"] + exclude_resources = [ + rt for rt in all_resource_types if rt != args.resource_type + ] + + logger.info( + f"Processing {args.resource_type} resources in {args.regions or 'all enabled'} regions for {action} action" + ) + # Initialize resource manager resource_manager = ResourceManager( profile_name=args.profile, use_profiles=args.use_profiles, discover_regions=args.discover_regions, - partition=args.partition + partition=args.partition, ) - + # Create account inclusion/exclusion list exclude_accounts = args.exclude_accounts or [] if args.account: @@ -88,7 +122,7 @@ def main(): # But don't add the specified account to the exclusion list logger.info(f"Processing only account {args.account}") # We'll handle this by post-filtering the account list in resource_manager - + # Process accounts - single scan for all resource types stats = resource_manager.process_accounts( action=action, @@ -96,14 +130,14 @@ def main(): exclude_accounts=exclude_accounts, exclude_resources=exclude_resources, exclude_regions=args.exclude_regions or [], - dry_run=args.dry_run + dry_run=args.dry_run, ) - + logger.info(f"Completed {action} action for {args.resource_type} resources") - + # Exit with success code sys.exit(0) - + except KeyboardInterrupt: logger.warning("Operation interrupted by user. Exiting gracefully...") sys.exit(1) @@ -112,5 +146,6 @@ def main(): logger.debug(traceback.format_exc()) sys.exit(1) + if __name__ == "__main__": main() diff --git a/local-app/python-tools/gfl-resource-actions/aws_resource_management/config_manager.py b/local-app/python-tools/gfl-resource-actions/aws_resource_management/config_manager.py index c8b29713..e7a42885 100644 --- a/local-app/python-tools/gfl-resource-actions/aws_resource_management/config_manager.py +++ b/local-app/python-tools/gfl-resource-actions/aws_resource_management/config_manager.py @@ -2,10 +2,12 @@ Configuration management for the AWS Resource Management tool. Handles loading configuration from files and environment variables. """ + import os -import yaml -from typing import Dict, Any, Optional from pathlib import Path +from typing import Any, Dict, Optional + +import yaml # Default configuration values DEFAULT_CONFIG = { @@ -21,44 +23,46 @@ # Environment variable prefix for overriding configuration ENV_PREFIX = "AWS_RESOURCE_MGMT_" + class ConfigManager: """ Configuration manager for AWS Resource Management tool. Handles loading config from files and environment variables. """ + _instance = None _config = None - + def __new__(cls): """Singleton pattern to ensure only one config instance.""" if cls._instance is None: cls._instance = super(ConfigManager, cls).__new__(cls) cls._instance._config = DEFAULT_CONFIG.copy() return cls._instance - + def load_config(self, config_file: Optional[str] = None) -> Dict[str, Any]: """ Load configuration from file and environment variables. - + Args: config_file: Path to the config file (YAML) - + Returns: Configuration dictionary """ # Start with defaults self._config = DEFAULT_CONFIG.copy() - + # Load from config file if provided if config_file and Path(config_file).exists(): try: - with open(config_file, 'r') as f: + with open(config_file, "r") as f: file_config = yaml.safe_load(f) if file_config and isinstance(file_config, dict): self._config.update(file_config) except Exception as e: print(f"Error loading config file: {e}") - + # Override with environment variables for key in self._config.keys(): env_key = f"{ENV_PREFIX}{key.upper()}" @@ -66,50 +70,51 @@ def load_config(self, config_file: Optional[str] = None) -> Dict[str, Any]: # Convert environment variable to appropriate type env_value = os.environ[env_key] if isinstance(self._config[key], bool): - self._config[key] = env_value.lower() in ('true', 'yes', '1') + self._config[key] = env_value.lower() in ("true", "yes", "1") elif isinstance(self._config[key], int): self._config[key] = int(env_value) else: self._config[key] = env_value - + return self._config - + def get(self, key: str, default: Any = None) -> Any: """ Get a configuration value by key. - + Args: key: Configuration key default: Default value if key doesn't exist - + Returns: Configuration value """ return self._config.get(key, default) - + def set(self, key: str, value: Any) -> None: """ Set a configuration value. - + Args: key: Configuration key value: Configuration value """ self._config[key] = value - + def get_all(self) -> Dict[str, Any]: """ Get the entire configuration dictionary. - + Returns: Configuration dictionary """ return self._config.copy() + def get_config() -> ConfigManager: """ Get the configuration manager instance. - + Returns: ConfigManager instance """ diff --git a/local-app/python-tools/gfl-resource-actions/aws_resource_management/core.py b/local-app/python-tools/gfl-resource-actions/aws_resource_management/core.py index 4117500a..4cf64809 100644 --- a/local-app/python-tools/gfl-resource-actions/aws_resource_management/core.py +++ b/local-app/python-tools/gfl-resource-actions/aws_resource_management/core.py @@ -2,33 +2,45 @@ Core business logic for AWS Resource Management tool. """ -from typing import Dict, List, Any, Optional, Union import logging import sys +from typing import Any, Dict, List, Optional, Union -from aws_resource_management.logging_setup import setup_logging -from aws_resource_management.config_manager import get_config from aws_resource_management import utils +from aws_resource_management.aws_utils import ( + detect_partition_from_credentials, + get_account_list, + get_accounts_from_profiles, + get_credentials, + get_enabled_regions, +) +from aws_resource_management.config_manager import get_config from aws_resource_management.discovery import get_account_resources -from aws_resource_management.managers import EC2Manager, RDSManager, EKSManager, EMRManager -from aws_resource_management.aws_utils import get_credentials, get_account_list, get_accounts_from_profiles, get_enabled_regions, detect_partition_from_credentials +from aws_resource_management.logging_setup import setup_logging +from aws_resource_management.managers import ( + EC2Manager, + EKSManager, + EMRManager, + RDSManager, +) logger = setup_logging() config = get_config() + class ResourceManager: """Main resource manager that orchestrates operations across accounts and resources.""" - + def __init__( - self, - profile_name: Optional[str] = None, + self, + profile_name: Optional[str] = None, use_profiles: bool = False, discover_regions: bool = False, - partition: Optional[str] = None + partition: Optional[str] = None, ) -> None: """ Initialize the resource manager. - + Args: profile_name: Optional AWS profile name to use for all operations use_profiles: Whether to use AWS profiles from ~/.aws/config @@ -44,23 +56,23 @@ def __init__( self.partition = partition # Cache for discovered regions by account self.region_cache = {} - + # If no partition was specified, we'll auto-detect based on credentials # when we process the first account - + def process_accounts( - self, + self, action: str, regions: List[str], exclude_accounts: List[str] = None, exclude_resources: List[str] = None, exclude_regions: List[str] = None, dry_run: bool = False, - stats: Dict[str, Any] = None + stats: Dict[str, Any] = None, ) -> Dict[str, Any]: """ Process accounts and perform the specified action. - + Args: action: Action to perform (stop or start) regions: List of regions to scan @@ -69,36 +81,42 @@ def process_accounts( exclude_regions: List of regions to exclude dry_run: Whether to perform a dry run stats: Statistics dictionary to update - + Returns: Updated statistics dictionary """ try: if exclude_accounts is None: exclude_accounts = [] - + if exclude_resources is None: exclude_resources = [] - + if exclude_regions is None: exclude_regions = [] - + # Initialize stats if not provided if stats is None: stats = { - 'accounts_processed': 0, - 'accounts_skipped': 0, - 'resources_processed': 0, - 'resources_skipped': 0, - 'regions_processed': 0, - 'regions_by_account': {} + "accounts_processed": 0, + "accounts_skipped": 0, + "resources_processed": 0, + "resources_skipped": 0, + "regions_processed": 0, + "regions_by_account": {}, } - + # Initialize resource-specific counters - for resource_type in ['ec2', 'rds', 'eks', 'emr']: - for stat_type in ['found', 'skipped', 'stopped', 'started', 'errors']: + for resource_type in ["ec2", "rds", "eks", "emr"]: + for stat_type in [ + "found", + "skipped", + "stopped", + "started", + "errors", + ]: stats[f"{resource_type}_{stat_type}"] = 0 - + # Get account list - either from profiles or Organizations API if self.use_profiles: logger.info("Using AWS SSO profiles for account discovery") @@ -106,64 +124,88 @@ def process_accounts( else: logger.info("Using AWS Organizations API for account discovery") accounts = get_account_list() - + # Log summary of accounts logger.info(f"Found {len(accounts)} accounts to process") if exclude_accounts: logger.info(f"Excluding {len(exclude_accounts)} accounts") - + # Process each account for account in accounts: - account_id = account.get('account_id') - account_name = account.get('account_name', 'Unknown') - + account_id = account.get("account_id") + account_name = account.get("account_name", "Unknown") + # Skip excluded accounts if account_id in exclude_accounts: - logger.info(f"Skipping excluded account {account_id} ({account_name})") - stats['accounts_skipped'] += 1 + logger.info( + f"Skipping excluded account {account_id} ({account_name})" + ) + stats["accounts_skipped"] += 1 continue - + try: # Get credentials for the account credentials = get_credentials(account_id, self.profile_name) if not credentials: - logger.warning(f"Could not obtain credentials for account {account_id}") - stats['accounts_skipped'] += 1 + logger.warning( + f"Could not obtain credentials for account {account_id}" + ) + stats["accounts_skipped"] += 1 continue - + # Determine regions to scan for this account account_regions = regions - + # If auto-discovery of regions is enabled, get all enabled regions for this account if self.discover_regions: try: - logger.info(f"Discovering enabled regions for account {account_id}") - account_regions = get_enabled_regions(credentials, self.partition) + logger.info( + f"Discovering enabled regions for account {account_id}" + ) + account_regions = get_enabled_regions( + credentials, self.partition + ) if exclude_regions: - account_regions = [r for r in account_regions if r not in exclude_regions] - logger.info(f"Found {len(account_regions)} enabled regions in account {account_id}") - + account_regions = [ + r + for r in account_regions + if r not in exclude_regions + ] + logger.info( + f"Found {len(account_regions)} enabled regions in account {account_id}" + ) + # Cache discovered regions self.region_cache[account_id] = account_regions except Exception as e: - logger.error(f"Error discovering regions for account {account_id}: {str(e)}") + logger.error( + f"Error discovering regions for account {account_id}: {str(e)}" + ) if regions: - logger.info(f"Falling back to provided regions: {', '.join(regions)}") + logger.info( + f"Falling back to provided regions: {', '.join(regions)}" + ) account_regions = regions else: - logger.info("No regions provided and region discovery failed. Skipping account.") - stats['accounts_skipped'] += 1 + logger.info( + "No regions provided and region discovery failed. Skipping account." + ) + stats["accounts_skipped"] += 1 continue elif exclude_regions: - account_regions = [r for r in account_regions if r not in exclude_regions] - + account_regions = [ + r for r in account_regions if r not in exclude_regions + ] + # Track regions processed for this account - stats['regions_by_account'][account_id] = account_regions - stats['regions_processed'] += len(account_regions) - + stats["regions_by_account"][account_id] = account_regions + stats["regions_processed"] += len(account_regions) + # Process this account with its regions - logger.info(f"Processing account: {account_name} ({account_id}) in {len(account_regions)} regions") - + logger.info( + f"Processing account: {account_name} ({account_id}) in {len(account_regions)} regions" + ) + self._process_single_account( account=account, credentials=credentials, @@ -171,354 +213,453 @@ def process_accounts( action=action, dry_run=dry_run, exclude_resources=exclude_resources, - stats=stats + stats=stats, ) - - stats['accounts_processed'] += 1 - + + stats["accounts_processed"] += 1 + except Exception as e: - logger.error(f"Error processing account {account_id}: {str(e)}", exc_info=True) - stats['accounts_skipped'] += 1 - + logger.error( + f"Error processing account {account_id}: {str(e)}", + exc_info=True, + ) + stats["accounts_skipped"] += 1 + # Log summary of regions processed if self.discover_regions: - logger.info(f"Total accounts processed: {stats.get('accounts_processed', 0)}") - logger.info(f"Total regions processed: {stats.get('regions_processed', 0)}") - for account_id, account_regions in stats.get('regions_by_account', {}).items(): - logger.debug(f"Account {account_id} regions: {', '.join(account_regions)}") - + logger.info( + f"Total accounts processed: {stats.get('accounts_processed', 0)}" + ) + logger.info( + f"Total regions processed: {stats.get('regions_processed', 0)}" + ) + for account_id, account_regions in stats.get( + "regions_by_account", {} + ).items(): + logger.debug( + f"Account {account_id} regions: {', '.join(account_regions)}" + ) + # Print detailed resource summary self._print_resource_summary(stats, action) - + return stats - + except KeyboardInterrupt: # Log the interruption but re-raise to ensure application exit logger.warning("Account processing interrupted by user") raise - + def _print_resource_summary(self, stats: Dict[str, Any], action: str) -> None: """ Print a detailed summary of resources processed, broken down by resource type. - + Args: stats: Statistics dictionary action: Action that was performed (stop or start) """ logger.info(f"\n{'=' * 30} SUMMARY {'=' * 30}") - logger.info(f"ACTION: {action.upper()} {'(DRY RUN)' if stats.get('dry_run', False) else ''}") - + logger.info( + f"ACTION: {action.upper()} {'(DRY RUN)' if stats.get('dry_run', False) else ''}" + ) + # General stats logger.info(f"\nGENERAL STATISTICS:") logger.info(f"Accounts processed: {stats.get('accounts_processed', 0)}") logger.info(f"Accounts skipped: {stats.get('accounts_skipped', 0)}") logger.info(f"Regions processed: {stats.get('regions_processed', 0)}") - + # EC2 resources logger.info(f"\nEC2 INSTANCES:") logger.info(f"Found: {stats.get('ec2_found', 0)}") - if action == 'stop': + if action == "stop": logger.info(f"Stopped: {stats.get('ec2_stopped', 0)}") else: logger.info(f"Started: {stats.get('ec2_started', 0)}") logger.info(f"Skipped: {stats.get('ec2_skipped', 0)}") logger.info(f"Errors: {stats.get('ec2_errors', 0)}") - + # RDS resources logger.info(f"\nRDS INSTANCES:") logger.info(f"Found: {stats.get('rds_found', 0)}") - if action == 'stop': + if action == "stop": logger.info(f"Stopped: {stats.get('rds_stopped', 0)}") else: logger.info(f"Started: {stats.get('rds_started', 0)}") logger.info(f"Skipped: {stats.get('rds_skipped', 0)}") logger.info(f"Errors: {stats.get('rds_errors', 0)}") - + # RDS engine breakdown if available - rds_engines = stats.get('rds_engines', {}) + rds_engines = stats.get("rds_engines", {}) if rds_engines: - logger.info(f"RDS engines: {', '.join(f'{engine}({count})' for engine, count in rds_engines.items())}") - + logger.info( + f"RDS engines: {', '.join(f'{engine}({count})' for engine, count in rds_engines.items())}" + ) + # EKS resources logger.info(f"\nEKS CLUSTERS:") logger.info(f"Found: {stats.get('eks_found', 0)}") - if action == 'stop': + if action == "stop": logger.info(f"Stopped: {stats.get('eks_stopped', 0)}") else: logger.info(f"Started: {stats.get('eks_started', 0)}") logger.info(f"Skipped: {stats.get('eks_skipped', 0)}") logger.info(f"Errors: {stats.get('eks_errors', 0)}") - + # EMR resources logger.info(f"\nEMR CLUSTERS:") logger.info(f"Found: {stats.get('emr_found', 0)}") - if action == 'stop': + if action == "stop": logger.info(f"Stopped: {stats.get('emr_stopped', 0)}") else: logger.info(f"Started: {stats.get('emr_started', 0)}") logger.info(f"Skipped: {stats.get('emr_skipped', 0)}") logger.info(f"Errors: {stats.get('emr_errors', 0)}") - + # Error summary if there were any errors - if stats.get('errors', []): + if stats.get("errors", []): logger.info(f"\nERROR SUMMARY:") logger.info(f"Total errors: {len(stats.get('errors', []))}") - for i, error in enumerate(stats.get('errors', [])[:5], 1): # Show first 5 errors + for i, error in enumerate( + stats.get("errors", [])[:5], 1 + ): # Show first 5 errors logger.info(f" {i}. {error}") - if len(stats.get('errors', [])) > 5: - logger.info(f" ... and {len(stats.get('errors', [])) - 5} more errors (see log for details)") - + if len(stats.get("errors", [])) > 5: + logger.info( + f" ... and {len(stats.get('errors', [])) - 5} more errors (see log for details)" + ) + logger.info(f"{'=' * 68}") def _process_single_account( - self, + self, account: Dict[str, str], credentials: Dict[str, str], regions: List[str], action: str, dry_run: bool, exclude_resources: List[str], - stats: Dict[str, Any] + stats: Dict[str, Any], ) -> None: """Process a single account for the specified action.""" # Fix keys - use consistent naming with process_accounts - account_id = account.get('account_id') - account_name = account.get('account_name', account_id) - + account_id = account.get("account_id") + account_name = account.get("account_name", account_id) + try: # Auto-detect partition if not specified if not self.partition: self.partition = detect_partition_from_credentials(credentials) logger.info(f"Auto-detected AWS partition: {self.partition}") - + # Ensure we're using regions from the right partition if regions and regions[0] != "auto": # Filter regions to match the detected partition filtered_regions = [] for region in regions: - if self.partition == 'aws-us-gov' and not region.startswith('us-gov-'): - logger.debug(f"Skipping region {region} - not in {self.partition} partition") + if self.partition == "aws-us-gov" and not region.startswith( + "us-gov-" + ): + logger.debug( + f"Skipping region {region} - not in {self.partition} partition" + ) continue - elif self.partition == 'aws-cn' and not region.startswith('cn-'): - logger.debug(f"Skipping region {region} - not in {self.partition} partition") + elif self.partition == "aws-cn" and not region.startswith("cn-"): + logger.debug( + f"Skipping region {region} - not in {self.partition} partition" + ) continue - elif self.partition == 'aws' and (region.startswith('us-gov-') or region.startswith('cn-')): - logger.debug(f"Skipping region {region} - not in {self.partition} partition") + elif self.partition == "aws" and ( + region.startswith("us-gov-") or region.startswith("cn-") + ): + logger.debug( + f"Skipping region {region} - not in {self.partition} partition" + ) continue filtered_regions.append(region) - + # If filtering removed all regions, use defaults for the partition if not filtered_regions: - if self.partition == 'aws-us-gov': - filtered_regions = ['us-gov-east-1', 'us-gov-west-1'] - elif self.partition == 'aws-cn': - filtered_regions = ['cn-north-1', 'cn-northwest-1'] + if self.partition == "aws-us-gov": + filtered_regions = ["us-gov-east-1", "us-gov-west-1"] + elif self.partition == "aws-cn": + filtered_regions = ["cn-north-1", "cn-northwest-1"] else: - filtered_regions = ['us-east-1', 'us-east-2', 'us-west-1', 'us-west-2'] - logger.info(f"Using default regions for partition {self.partition}: {', '.join(filtered_regions)}") - + filtered_regions = [ + "us-east-1", + "us-east-2", + "us-west-1", + "us-west-2", + ] + logger.info( + f"Using default regions for partition {self.partition}: {', '.join(filtered_regions)}" + ) + regions = filtered_regions elif not regions: # If no regions provided, use defaults for the detected partition - if self.partition == 'aws-us-gov': - regions = ['us-gov-east-1', 'us-gov-west-1'] - elif self.partition == 'aws-cn': - regions = ['cn-north-1', 'cn-northwest-1'] + if self.partition == "aws-us-gov": + regions = ["us-gov-east-1", "us-gov-west-1"] + elif self.partition == "aws-cn": + regions = ["cn-north-1", "cn-northwest-1"] else: - regions = ['us-east-1', 'us-east-2', 'us-west-1', 'us-west-2'] - logger.info(f"No regions specified, using defaults for partition {self.partition}: {', '.join(regions)}") - + regions = ["us-east-1", "us-east-2", "us-west-1", "us-west-2"] + logger.info( + f"No regions specified, using defaults for partition {self.partition}: {', '.join(regions)}" + ) + # Only log this once - removing duplicate message - logger.info(f"Processing account: {account_name} ({account_id}) in {len(regions)} regions") - + logger.info( + f"Processing account: {account_name} ({account_id}) in {len(regions)} regions" + ) + # Define special handling for EMR clusters - remove 'STOPPED' from valid states as it's not accepted by the API emr_states = None if "emr" not in exclude_resources: # Only include valid EMR states for the ListClusters API call # Note: 'STOPPED' is not a valid state for ListClusters API - emr_states = ['STARTING', 'BOOTSTRAPPING', 'RUNNING', 'WAITING', 'TERMINATING'] - logger.debug(f"Using valid EMR states for discovery: {', '.join(emr_states)}") - + emr_states = [ + "STARTING", + "BOOTSTRAPPING", + "RUNNING", + "WAITING", + "TERMINATING", + ] + logger.debug( + f"Using valid EMR states for discovery: {', '.join(emr_states)}" + ) + # Get resources with special handling for EMR resources = get_account_resources( - credentials, - regions, - exclude_resources, - account_id, + credentials, + regions, + exclude_resources, + account_id, account_name, - emr_cluster_states=emr_states + emr_cluster_states=emr_states, ) if not resources: logger.error(f"Failed to get resources for account {account_id}") return - + # Initialize stat counters if they don't exist - for stat_type in ['ec2_found', 'ec2_skipped', 'ec2_stopped', 'ec2_started', 'ec2_errors', - 'rds_found', 'rds_skipped', 'rds_stopped', 'rds_started', 'rds_errors', - 'eks_found', 'eks_skipped', 'eks_stopped', 'eks_started', 'eks_errors', - 'emr_found', 'emr_skipped', 'emr_stopped', 'emr_started', 'emr_errors']: + for stat_type in [ + "ec2_found", + "ec2_skipped", + "ec2_stopped", + "ec2_started", + "ec2_errors", + "rds_found", + "rds_skipped", + "rds_stopped", + "rds_started", + "rds_errors", + "eks_found", + "eks_skipped", + "eks_stopped", + "eks_started", + "eks_errors", + "emr_found", + "emr_skipped", + "emr_stopped", + "emr_started", + "emr_errors", + ]: if stat_type not in stats: stats[stat_type] = 0 - + # Initialize RDS engines stats if it doesn't exist - if 'rds_engines' not in stats: - stats['rds_engines'] = {} - + if "rds_engines" not in stats: + stats["rds_engines"] = {} + # Initialize errors array if it doesn't exist - if 'errors' not in stats: - stats['errors'] = [] - + if "errors" not in stats: + stats["errors"] = [] + # Normalize state and status fields for all resource types # For EC2 instances - ec2_instances = resources.get('ec2_instances', []) + ec2_instances = resources.get("ec2_instances", []) for instance in ec2_instances: - if 'state' not in instance and 'status' in instance: - instance['state'] = instance['status'] - elif 'status' not in instance and 'state' in instance: - instance['status'] = instance['state'] - elif 'state' not in instance and 'status' not in instance: - instance['state'] = 'unknown' - instance['status'] = 'unknown' - + if "state" not in instance and "status" in instance: + instance["state"] = instance["status"] + elif "status" not in instance and "state" in instance: + instance["status"] = instance["state"] + elif "state" not in instance and "status" not in instance: + instance["state"] = "unknown" + instance["status"] = "unknown" + # For RDS instances - rds_instances = resources.get('rds_instances', []) + rds_instances = resources.get("rds_instances", []) for instance in rds_instances: - if 'state' not in instance and 'status' in instance: - instance['state'] = instance['status'] - elif 'status' not in instance and 'state' in instance: - instance['status'] = instance['state'] - elif 'state' not in instance and 'status' not in instance: - instance['state'] = 'unknown' - instance['status'] = 'unknown' - + if "state" not in instance and "status" in instance: + instance["state"] = instance["status"] + elif "status" not in instance and "state" in instance: + instance["status"] = instance["state"] + elif "state" not in instance and "status" not in instance: + instance["state"] = "unknown" + instance["status"] = "unknown" + # For EKS clusters - eks_clusters = resources.get('eks_clusters', []) + eks_clusters = resources.get("eks_clusters", []) for cluster in eks_clusters: - if 'state' not in cluster and 'status' in cluster: - cluster['state'] = cluster['status'] - elif 'status' not in cluster and 'state' in cluster: - cluster['status'] = cluster['state'] - elif 'state' not in cluster and 'status' not in cluster: - cluster['state'] = 'unknown' - cluster['status'] = 'unknown' - + if "state" not in cluster and "status" in cluster: + cluster["state"] = cluster["status"] + elif "status" not in cluster and "state" in cluster: + cluster["status"] = cluster["state"] + elif "state" not in cluster and "status" not in cluster: + cluster["state"] = "unknown" + cluster["status"] = "unknown" + # For EMR clusters - emr_clusters = resources.get('emr_clusters', []) + emr_clusters = resources.get("emr_clusters", []) for cluster in emr_clusters: - if 'state' not in cluster: - cluster['state'] = cluster.get('status', 'UNKNOWN') - if 'status' not in cluster: - cluster['status'] = cluster.get('state', 'UNKNOWN') - + if "state" not in cluster: + cluster["state"] = cluster.get("status", "UNKNOWN") + if "status" not in cluster: + cluster["status"] = cluster.get("state", "UNKNOWN") + # Count service-managed EC2 instances - eks_tag = self.config.get('eks_tag', 'kubernetes.io/cluster/') - emr_tag = self.config.get('emr_tag', 'aws:elasticmapreduce:job-flow-id') - exclusion_tag = self.config.get('exclusion_tag', 'DoNotStop') - + eks_tag = self.config.get("eks_tag", "kubernetes.io/cluster/") + emr_tag = self.config.get("emr_tag", "aws:elasticmapreduce:job-flow-id") + exclusion_tag = self.config.get("exclusion_tag", "DoNotStop") + # Continue with existing code - eks_managed = sum(1 for i in resources.get('ec2_instances', []) - if any(tag.startswith(eks_tag) for tag in i.get('tags', {}))) - emr_managed = sum(1 for i in resources.get('ec2_instances', []) - if emr_tag in i.get('tags', {})) - + eks_managed = sum( + 1 + for i in resources.get("ec2_instances", []) + if any(tag.startswith(eks_tag) for tag in i.get("tags", {})) + ) + emr_managed = sum( + 1 + for i in resources.get("ec2_instances", []) + if emr_tag in i.get("tags", {}) + ) + # Update resource counts - total_ec2 = len(resources.get('ec2_instances', [])) - excluded_ec2 = sum(1 for i in resources.get('ec2_instances', []) - if exclusion_tag in i.get('tags', {})) - stats['ec2_found'] += total_ec2 - stats['ec2_skipped'] += excluded_ec2 - - stats['rds_found'] += len(resources.get('rds_instances', [])) - stats['eks_found'] += len(resources.get('eks_clusters', [])) - stats['emr_found'] += len(resources.get('emr_clusters', [])) - + total_ec2 = len(resources.get("ec2_instances", [])) + excluded_ec2 = sum( + 1 + for i in resources.get("ec2_instances", []) + if exclusion_tag in i.get("tags", {}) + ) + stats["ec2_found"] += total_ec2 + stats["ec2_skipped"] += excluded_ec2 + + stats["rds_found"] += len(resources.get("rds_instances", [])) + stats["eks_found"] += len(resources.get("eks_clusters", [])) + stats["emr_found"] += len(resources.get("emr_clusters", [])) + # Track RDS engines - for instance in resources.get('rds_instances', []): - if instance and 'engine' in instance: - engine = instance['engine'] - if engine in stats['rds_engines']: - stats['rds_engines'][engine] += 1 + for instance in resources.get("rds_instances", []): + if instance and "engine" in instance: + engine = instance["engine"] + if engine in stats["rds_engines"]: + stats["rds_engines"][engine] += 1 else: - stats['rds_engines'][engine] = 1 - + stats["rds_engines"][engine] = 1 + # Log discovered resources with service-managed counts - logger.info(f"Account {account_id} - Found {total_ec2} EC2 instances ({excluded_ec2} explicitly excluded, {eks_managed} EKS-managed, {emr_managed} EMR-managed)") - logger.info(f"Account {account_id} - Found {len(resources.get('rds_instances', []))} RDS instances") - logger.info(f"Account {account_id} - Found {len(resources.get('eks_clusters', []))} EKS clusters") - logger.info(f"Account {account_id} - Found {len(resources.get('emr_clusters', []))} EMR clusters") - + logger.info( + f"Account {account_id} - Found {total_ec2} EC2 instances ({excluded_ec2} explicitly excluded, {eks_managed} EKS-managed, {emr_managed} EMR-managed)" + ) + logger.info( + f"Account {account_id} - Found {len(resources.get('rds_instances', []))} RDS instances" + ) + logger.info( + f"Account {account_id} - Found {len(resources.get('eks_clusters', []))} EKS clusters" + ) + logger.info( + f"Account {account_id} - Found {len(resources.get('emr_clusters', []))} EMR clusters" + ) + # Initialize resource managers with account name ec2_manager = EC2Manager(credentials, account_id, account_name) rds_manager = RDSManager(credentials, account_id, account_name) eks_manager = EKSManager(credentials, account_id, account_name) emr_manager = EMRManager(credentials, account_id, account_name) - + # Helper function to safely process manager results def safe_get_result(result: Optional[Dict[str, int]], key: str) -> int: if result is None: return 0 return result.get(key, 0) - + # Perform actions based on selected mode if action == "stop": if "ec2" not in exclude_resources: - result = ec2_manager.stop(resources.get('ec2_instances', []), dry_run) - stats['ec2_stopped'] += safe_get_result(result, 'success') - stats['ec2_errors'] += safe_get_result(result, 'errors') + result = ec2_manager.stop( + resources.get("ec2_instances", []), dry_run + ) + stats["ec2_stopped"] += safe_get_result(result, "success") + stats["ec2_errors"] += safe_get_result(result, "errors") else: - stats['ec2_skipped'] += total_ec2 - excluded_ec2 - + stats["ec2_skipped"] += total_ec2 - excluded_ec2 + if "rds" not in exclude_resources: - result = rds_manager.stop(resources.get('rds_instances', []), dry_run) - stats['rds_stopped'] += safe_get_result(result, 'success') - stats['rds_errors'] += safe_get_result(result, 'errors') + result = rds_manager.stop( + resources.get("rds_instances", []), dry_run + ) + stats["rds_stopped"] += safe_get_result(result, "success") + stats["rds_errors"] += safe_get_result(result, "errors") else: - stats['rds_skipped'] += len(resources.get('rds_instances', [])) - + stats["rds_skipped"] += len(resources.get("rds_instances", [])) + if "eks" not in exclude_resources: - result = eks_manager.stop(resources.get('eks_clusters', []), dry_run) - stats['eks_stopped'] += safe_get_result(result, 'success') - stats['eks_errors'] += safe_get_result(result, 'errors') + result = eks_manager.stop( + resources.get("eks_clusters", []), dry_run + ) + stats["eks_stopped"] += safe_get_result(result, "success") + stats["eks_errors"] += safe_get_result(result, "errors") else: - stats['eks_skipped'] += len(resources.get('eks_clusters', [])) - + stats["eks_skipped"] += len(resources.get("eks_clusters", [])) + if "emr" not in exclude_resources: - result = emr_manager.stop(resources.get('emr_clusters', []), dry_run) - stats['emr_stopped'] += safe_get_result(result, 'success') - stats['emr_errors'] += safe_get_result(result, 'errors') + result = emr_manager.stop( + resources.get("emr_clusters", []), dry_run + ) + stats["emr_stopped"] += safe_get_result(result, "success") + stats["emr_errors"] += safe_get_result(result, "errors") else: - stats['emr_skipped'] += len(resources.get('emr_clusters', [])) + stats["emr_skipped"] += len(resources.get("emr_clusters", [])) else: # action == "start" if "ec2" not in exclude_resources: - result = ec2_manager.start(resources.get('ec2_instances', []), dry_run) - stats['ec2_started'] += safe_get_result(result, 'success') - stats['ec2_errors'] += safe_get_result(result, 'errors') + result = ec2_manager.start( + resources.get("ec2_instances", []), dry_run + ) + stats["ec2_started"] += safe_get_result(result, "success") + stats["ec2_errors"] += safe_get_result(result, "errors") else: - stats['ec2_skipped'] += total_ec2 - excluded_ec2 - + stats["ec2_skipped"] += total_ec2 - excluded_ec2 + if "rds" not in exclude_resources: - result = rds_manager.start(resources.get('rds_instances', []), dry_run) - stats['rds_started'] += safe_get_result(result, 'success') - stats['rds_errors'] += safe_get_result(result, 'errors') + result = rds_manager.start( + resources.get("rds_instances", []), dry_run + ) + stats["rds_started"] += safe_get_result(result, "success") + stats["rds_errors"] += safe_get_result(result, "errors") else: - stats['rds_skipped'] += len(resources.get('rds_instances', [])) - + stats["rds_skipped"] += len(resources.get("rds_instances", [])) + if "eks" not in exclude_resources: - result = eks_manager.start(resources.get('eks_clusters', []), dry_run) - stats['eks_started'] += safe_get_result(result, 'success') - stats['eks_errors'] += safe_get_result(result, 'errors') + result = eks_manager.start( + resources.get("eks_clusters", []), dry_run + ) + stats["eks_started"] += safe_get_result(result, "success") + stats["eks_errors"] += safe_get_result(result, "errors") else: - stats['eks_skipped'] += len(resources.get('eks_clusters', [])) - + stats["eks_skipped"] += len(resources.get("eks_clusters", [])) + if "emr" not in exclude_resources: - result = emr_manager.start(resources.get('emr_clusters', []), dry_run) - stats['emr_started'] += safe_get_result(result, 'success') - stats['emr_errors'] += safe_get_result(result, 'errors') + result = emr_manager.start( + resources.get("emr_clusters", []), dry_run + ) + stats["emr_started"] += safe_get_result(result, "success") + stats["emr_errors"] += safe_get_result(result, "errors") else: - stats['emr_skipped'] += len(resources.get('emr_clusters', [])) + stats["emr_skipped"] += len(resources.get("emr_clusters", [])) except Exception as e: logger.error(f"Error processing account {account_id}: {str(e)}") - if 'errors' not in stats: - stats['errors'] = [] - stats['errors'].append(f"Error processing account {account_id}: {str(e)}") + if "errors" not in stats: + stats["errors"] = [] + stats["errors"].append(f"Error processing account {account_id}: {str(e)}") return diff --git a/local-app/python-tools/gfl-resource-actions/aws_resource_management/discovery.py b/local-app/python-tools/gfl-resource-actions/aws_resource_management/discovery.py index f81b1f42..d327060f 100644 --- a/local-app/python-tools/gfl-resource-actions/aws_resource_management/discovery.py +++ b/local-app/python-tools/gfl-resource-actions/aws_resource_management/discovery.py @@ -1,34 +1,41 @@ """ Resource discovery module for finding AWS resources. """ -from typing import Dict, List, Any, Optional, Union -import boto3 + import concurrent.futures from collections import defaultdict -from botocore.exceptions import ClientError, EndpointConnectionError from concurrent.futures import ThreadPoolExecutor, as_completed +from typing import Any, Dict, List, Optional, Union +import boto3 +from aws_resource_management.aws_utils import ( + detect_partition_from_credentials, + get_all_regions, + get_regions_for_partition, + is_valid_region, +) from aws_resource_management.config_manager import get_config from aws_resource_management.logging_setup import setup_logging -from aws_resource_management.aws_utils import get_all_regions, is_valid_region, detect_partition_from_credentials, get_regions_for_partition +from botocore.exceptions import ClientError, EndpointConnectionError logger = setup_logging() config = get_config() # Default regions for different partitions DEFAULT_REGIONS = { - 'aws': ['us-east-1', 'us-east-2', 'us-west-1', 'us-west-2'], - 'aws-us-gov': ['us-gov-east-1', 'us-gov-west-1'], - 'aws-cn': ['cn-north-1', 'cn-northwest-1'] + "aws": ["us-east-1", "us-east-2", "us-west-1", "us-west-2"], + "aws-us-gov": ["us-gov-east-1", "us-gov-west-1"], + "aws-cn": ["cn-north-1", "cn-northwest-1"], } + class ResourceDiscovery: """Resource discovery for AWS resources.""" - + def __init__(self, credentials: Dict[str, str], account_id: str): """ Initialize resource discovery. - + Args: credentials: AWS credentials dictionary account_id: AWS account ID @@ -36,22 +43,24 @@ def __init__(self, credentials: Dict[str, str], account_id: str): self.credentials = credentials self.account_id = account_id self.partition = detect_partition_from_credentials(credentials) - logger.info(f"Resource discovery initialized for account {account_id} in partition {self.partition}") - + logger.info( + f"Resource discovery initialized for account {account_id} in partition {self.partition}" + ) + def discover_resources( - self, - resource_type: str, + self, + resource_type: str, regions: Union[str, List[str]] = "all", - resource_ids: Optional[List[str]] = None + resource_ids: Optional[List[str]] = None, ) -> List[Dict[str, Any]]: """ Discover resources of the specified type. - + Args: resource_type: Resource type (ec2_instance, rds_instance, etc.) regions: Regions to discover resources in, "all" for all regions resource_ids: List of resource IDs to filter by - + Returns: List of resource dictionaries """ @@ -60,32 +69,42 @@ def discover_resources( regions = get_regions_for_partition(self.partition) elif isinstance(regions, str): regions = [regions] - + # Filter regions to only include those in the detected partition valid_regions = [] for region in regions: # Skip regions that don't belong to our partition - if self.partition == 'aws-us-gov' and not region.startswith('us-gov-'): - logger.debug(f"Skipping region {region} - not in {self.partition} partition") + if self.partition == "aws-us-gov" and not region.startswith("us-gov-"): + logger.debug( + f"Skipping region {region} - not in {self.partition} partition" + ) continue - elif self.partition == 'aws-cn' and not region.startswith('cn-'): - logger.debug(f"Skipping region {region} - not in {self.partition} partition") + elif self.partition == "aws-cn" and not region.startswith("cn-"): + logger.debug( + f"Skipping region {region} - not in {self.partition} partition" + ) continue - elif self.partition == 'aws' and (region.startswith('us-gov-') or region.startswith('cn-')): - logger.debug(f"Skipping region {region} - not in {self.partition} partition") + elif self.partition == "aws" and ( + region.startswith("us-gov-") or region.startswith("cn-") + ): + logger.debug( + f"Skipping region {region} - not in {self.partition} partition" + ) continue - + # Add the valid region to our list valid_regions.append(region) - + # If we filtered out all regions, use default regions for the partition if not valid_regions: - valid_regions = DEFAULT_REGIONS.get(self.partition, DEFAULT_REGIONS['aws']) - logger.info(f"No valid regions found for partition {self.partition}, using defaults: {', '.join(valid_regions)}") - + valid_regions = DEFAULT_REGIONS.get(self.partition, DEFAULT_REGIONS["aws"]) + logger.info( + f"No valid regions found for partition {self.partition}, using defaults: {', '.join(valid_regions)}" + ) + # Use the filtered valid regions regions = valid_regions - + # Determine which regions to search region_list = [] if regions == "all": @@ -95,332 +114,392 @@ def discover_resources( else: logger.error(f"Invalid regions parameter: {regions}") return [] - + if not region_list: logger.warning("No valid regions specified for resource discovery") return [] - + # Call the appropriate discovery method based on resource type discovery_method = getattr(self, f"_discover_{resource_type}", None) if not discovery_method: - logger.error(f"No discovery method available for resource type: {resource_type}") + logger.error( + f"No discovery method available for resource type: {resource_type}" + ) return [] - + # Use thread pool to speed up discovery across regions resources = [] - with concurrent.futures.ThreadPoolExecutor(max_workers=min(10, len(region_list))) as executor: + with concurrent.futures.ThreadPoolExecutor( + max_workers=min(10, len(region_list)) + ) as executor: # Create a future for each region future_to_region = { - executor.submit(discovery_method, region, resource_ids): region + executor.submit(discovery_method, region, resource_ids): region for region in region_list } - + # Process results as they complete for future in concurrent.futures.as_completed(future_to_region): region = future_to_region[future] try: region_resources = future.result() resources.extend(region_resources) - logger.debug(f"Discovered {len(region_resources)} {resource_type} resources in {region}") + logger.debug( + f"Discovered {len(region_resources)} {resource_type} resources in {region}" + ) except Exception as e: logger.error(f"Error discovering {resource_type} in {region}: {e}") - + return resources - - def _discover_ec2(self, region: str, resource_ids: Optional[List[str]] = None) -> List[Dict[str, Any]]: + + def _discover_ec2( + self, region: str, resource_ids: Optional[List[str]] = None + ) -> List[Dict[str, Any]]: """ Discover EC2 instances in a region. - + Args: region: AWS region resource_ids: Specific instance IDs to filter by - + Returns: List of EC2 instance dictionaries """ # Create EC2 client ec2_client = boto3.client( - 'ec2', + "ec2", region_name=region, - aws_access_key_id=self.credentials.get('aws_access_key_id'), - aws_secret_access_key=self.credentials.get('aws_secret_access_key'), - aws_session_token=self.credentials.get('aws_session_token') + aws_access_key_id=self.credentials.get("aws_access_key_id"), + aws_secret_access_key=self.credentials.get("aws_secret_access_key"), + aws_session_token=self.credentials.get("aws_session_token"), ) - + # Build filters filters = [] if resource_ids: - filters.append({ - 'Name': 'instance-id', - 'Values': resource_ids - }) - + filters.append({"Name": "instance-id", "Values": resource_ids}) + try: # Get instances response = ec2_client.describe_instances(Filters=filters) - + # Extract instance information instances = [] - for reservation in response.get('Reservations', []): - for instance in reservation.get('Instances', []): + for reservation in response.get("Reservations", []): + for instance in reservation.get("Instances", []): # Convert tags to dictionary tags = {} - for tag in instance.get('Tags', []): - tags[tag['Key']] = tag['Value'] - + for tag in instance.get("Tags", []): + tags[tag["Key"]] = tag["Value"] + # Create a simplified instance dictionary instance_dict = { - 'id': instance['InstanceId'], - 'name': tags.get('Name', 'Unnamed'), - 'type': instance['InstanceType'], - 'status': instance['State']['Name'], - 'region': region, - 'tags': tags, - 'launch_time': instance.get('LaunchTime', '').isoformat() if instance.get('LaunchTime') else None, - 'public_ip': instance.get('PublicIpAddress'), - 'private_ip': instance.get('PrivateIpAddress') + "id": instance["InstanceId"], + "name": tags.get("Name", "Unnamed"), + "type": instance["InstanceType"], + "status": instance["State"]["Name"], + "region": region, + "tags": tags, + "launch_time": ( + instance.get("LaunchTime", "").isoformat() + if instance.get("LaunchTime") + else None + ), + "public_ip": instance.get("PublicIpAddress"), + "private_ip": instance.get("PrivateIpAddress"), } instances.append(instance_dict) - + return instances - + except Exception as e: logger.error(f"Error discovering EC2 instances in {region}: {e}") return [] - - def _discover_rds(self, region: str, resource_ids: Optional[List[str]] = None) -> List[Dict[str, Any]]: + + def _discover_rds( + self, region: str, resource_ids: Optional[List[str]] = None + ) -> List[Dict[str, Any]]: """ Discover RDS instances in a region. - + Args: region: AWS region resource_ids: Specific DB instance IDs to filter by - + Returns: List of RDS instance dictionaries """ # Create RDS client rds_client = boto3.client( - 'rds', + "rds", region_name=region, - aws_access_key_id=self.credentials.get('aws_access_key_id'), - aws_secret_access_key=self.credentials.get('aws_secret_access_key'), - aws_session_token=self.credentials.get('aws_session_token') + aws_access_key_id=self.credentials.get("aws_access_key_id"), + aws_secret_access_key=self.credentials.get("aws_secret_access_key"), + aws_session_token=self.credentials.get("aws_session_token"), ) - + try: # Get instances db_instances = [] - + # Use filters if specific IDs are provided if resource_ids: for db_id in resource_ids: try: - response = rds_client.describe_db_instances(DBInstanceIdentifier=db_id) - db_instances.extend(response.get('DBInstances', [])) + response = rds_client.describe_db_instances( + DBInstanceIdentifier=db_id + ) + db_instances.extend(response.get("DBInstances", [])) except Exception: continue else: # Get all instances response = rds_client.describe_db_instances() - db_instances = response.get('DBInstances', []) - + db_instances = response.get("DBInstances", []) + # Handle pagination - while 'Marker' in response: - response = rds_client.describe_db_instances(Marker=response['Marker']) - db_instances.extend(response.get('DBInstances', [])) - + while "Marker" in response: + response = rds_client.describe_db_instances( + Marker=response["Marker"] + ) + db_instances.extend(response.get("DBInstances", [])) + # Process instances instances = [] for db in db_instances: # Get tags try: tags_response = rds_client.list_tags_for_resource( - ResourceName=db['DBInstanceArn'] + ResourceName=db["DBInstanceArn"] ) - tags = {item['Key']: item['Value'] for item in tags_response.get('TagList', [])} + tags = { + item["Key"]: item["Value"] + for item in tags_response.get("TagList", []) + } except Exception: tags = {} - + # Create a simplified instance dictionary instance_dict = { - 'id': db['DBInstanceIdentifier'], - 'name': db['DBInstanceIdentifier'], - 'type': db['DBInstanceClass'], - 'status': db['DBInstanceStatus'], - 'region': region, - 'tags': tags, - 'engine': db['Engine'], - 'endpoint': db.get('Endpoint', {}).get('Address'), - 'multi_az': db.get('MultiAZ', False) + "id": db["DBInstanceIdentifier"], + "name": db["DBInstanceIdentifier"], + "type": db["DBInstanceClass"], + "status": db["DBInstanceStatus"], + "region": region, + "tags": tags, + "engine": db["Engine"], + "endpoint": db.get("Endpoint", {}).get("Address"), + "multi_az": db.get("MultiAZ", False), } instances.append(instance_dict) - + return instances - + except Exception as e: logger.error(f"Error discovering RDS instances in {region}: {e}") return [] - - def _discover_emr(self, region: str, resource_ids: Optional[List[str]] = None) -> List[Dict[str, Any]]: + + def _discover_emr( + self, region: str, resource_ids: Optional[List[str]] = None + ) -> List[Dict[str, Any]]: """ Discover EMR clusters in a region. - + Args: region: AWS region resource_ids: Specific cluster IDs to filter by - + Returns: List of EMR cluster dictionaries """ # Create EMR client emr_client = boto3.client( - 'emr', + "emr", region_name=region, - aws_access_key_id=self.credentials.get('aws_access_key_id'), - aws_secret_access_key=self.credentials.get('aws_secret_access_key'), - aws_session_token=self.credentials.get('aws_session_token') + aws_access_key_id=self.credentials.get("aws_access_key_id"), + aws_secret_access_key=self.credentials.get("aws_secret_access_key"), + aws_session_token=self.credentials.get("aws_session_token"), ) - + try: # Build filter for cluster states # Include all states: STARTING, BOOTSTRAPPING, RUNNING, WAITING, TERMINATING, TERMINATED, TERMINATED_WITH_ERRORS - cluster_states = ['STARTING', 'BOOTSTRAPPING', 'RUNNING', 'WAITING', 'TERMINATING', 'TERMINATED', 'TERMINATED_WITH_ERRORS'] - + cluster_states = [ + "STARTING", + "BOOTSTRAPPING", + "RUNNING", + "WAITING", + "TERMINATING", + "TERMINATED", + "TERMINATED_WITH_ERRORS", + ] + # Get clusters response = emr_client.list_clusters(ClusterStates=cluster_states) - clusters = response.get('Clusters', []) - + clusters = response.get("Clusters", []) + # Handle pagination - while 'Marker' in response: - response = emr_client.list_clusters(Marker=response['Marker'], ClusterStates=cluster_states) - clusters.extend(response.get('Clusters', [])) - + while "Marker" in response: + response = emr_client.list_clusters( + Marker=response["Marker"], ClusterStates=cluster_states + ) + clusters.extend(response.get("Clusters", [])) + # Filter by IDs if specified if resource_ids: - clusters = [c for c in clusters if c['Id'] in resource_ids] - + clusters = [c for c in clusters if c["Id"] in resource_ids] + # Get detailed information for each cluster detailed_clusters = [] for cluster in clusters: try: # Get cluster details including tags - cluster_details = emr_client.describe_cluster(ClusterId=cluster['Id']) - cluster_info = cluster_details.get('Cluster', {}) - + cluster_details = emr_client.describe_cluster( + ClusterId=cluster["Id"] + ) + cluster_info = cluster_details.get("Cluster", {}) + # Convert tags to dictionary tags = {} - for tag in cluster_info.get('Tags', []): - tags[tag['Key']] = tag['Value'] - + for tag in cluster_info.get("Tags", []): + tags[tag["Key"]] = tag["Value"] + # Ensure we have a state value - state = 'UNKNOWN' - if 'Status' in cluster and isinstance(cluster['Status'], dict) and 'State' in cluster['Status']: - state = cluster['Status']['State'] - + state = "UNKNOWN" + if ( + "Status" in cluster + and isinstance(cluster["Status"], dict) + and "State" in cluster["Status"] + ): + state = cluster["Status"]["State"] + # Create a simplified cluster dictionary cluster_dict = { - 'id': cluster['Id'], - 'name': cluster.get('Name', 'Unnamed'), - 'status': state, - 'state': state, # Add both fields to ensure compatibility - 'region': region, - 'tags': tags, - 'creation_time': cluster.get('Status', {}).get('Timeline', {}).get('CreationDateTime', '').isoformat() - if cluster.get('Status', {}).get('Timeline', {}).get('CreationDateTime') else None, - 'instance_count': cluster_info.get('InstanceCollectionType', 'Unknown'), - 'applications': [app.get('Name') for app in cluster_info.get('Applications', [])] + "id": cluster["Id"], + "name": cluster.get("Name", "Unnamed"), + "status": state, + "state": state, # Add both fields to ensure compatibility + "region": region, + "tags": tags, + "creation_time": ( + cluster.get("Status", {}) + .get("Timeline", {}) + .get("CreationDateTime", "") + .isoformat() + if cluster.get("Status", {}) + .get("Timeline", {}) + .get("CreationDateTime") + else None + ), + "instance_count": cluster_info.get( + "InstanceCollectionType", "Unknown" + ), + "applications": [ + app.get("Name") + for app in cluster_info.get("Applications", []) + ], } detailed_clusters.append(cluster_dict) except Exception as e: - logger.warning(f"Error getting details for EMR cluster {cluster['Id']}: {e}") - + logger.warning( + f"Error getting details for EMR cluster {cluster['Id']}: {e}" + ) + return detailed_clusters - + except Exception as e: logger.error(f"Error discovering EMR clusters in {region}: {e}") return [] - - def _discover_eks(self, region: str, resource_ids: Optional[List[str]] = None) -> List[Dict[str, Any]]: + + def _discover_eks( + self, region: str, resource_ids: Optional[List[str]] = None + ) -> List[Dict[str, Any]]: """ Discover EKS clusters in a region. - + Args: region: AWS region resource_ids: Specific cluster names to filter by - + Returns: List of EKS cluster dictionaries """ # Create EKS client eks_client = boto3.client( - 'eks', + "eks", region_name=region, - aws_access_key_id=self.credentials.get('aws_access_key_id'), - aws_secret_access_key=self.credentials.get('aws_secret_access_key'), - aws_session_token=self.credentials.get('aws_session_token') + aws_access_key_id=self.credentials.get("aws_access_key_id"), + aws_secret_access_key=self.credentials.get("aws_secret_access_key"), + aws_session_token=self.credentials.get("aws_session_token"), ) - + try: # Get clusters response = eks_client.list_clusters() - cluster_names = response.get('clusters', []) - + cluster_names = response.get("clusters", []) + # Handle pagination - while 'nextToken' in response: - response = eks_client.list_clusters(nextToken=response['nextToken']) - cluster_names.extend(response.get('clusters', [])) - + while "nextToken" in response: + response = eks_client.list_clusters(nextToken=response["nextToken"]) + cluster_names.extend(response.get("clusters", [])) + # Filter by names if specified if resource_ids: cluster_names = [name for name in cluster_names if name in resource_ids] - + # Get detailed information for each cluster clusters = [] for name in cluster_names: try: # Get cluster details - cluster = eks_client.describe_cluster(name=name)['cluster'] - + cluster = eks_client.describe_cluster(name=name)["cluster"] + # Get tags - tags = cluster.get('tags', {}) - + tags = cluster.get("tags", {}) + # Create ARN manually if not present - arn = cluster.get('arn', f"arn:aws:eks:{region}:{self.account_id}:cluster/{name}") - + arn = cluster.get( + "arn", f"arn:aws:eks:{region}:{self.account_id}:cluster/{name}" + ) + # Create a simplified cluster dictionary cluster_dict = { - 'id': name, - 'name': name, - 'arn': arn, - 'status': cluster.get('status', 'UNKNOWN'), - 'region': region, - 'tags': tags, - 'version': cluster.get('version'), - 'endpoint': cluster.get('endpoint'), - 'created_at': cluster.get('createdAt', '').isoformat() if cluster.get('createdAt') else None, + "id": name, + "name": name, + "arn": arn, + "status": cluster.get("status", "UNKNOWN"), + "region": region, + "tags": tags, + "version": cluster.get("version"), + "endpoint": cluster.get("endpoint"), + "created_at": ( + cluster.get("createdAt", "").isoformat() + if cluster.get("createdAt") + else None + ), } clusters.append(cluster_dict) except Exception as e: logger.warning(f"Error getting details for EKS cluster {name}: {e}") - + return clusters - + except Exception as e: logger.error(f"Error discovering EKS clusters in {region}: {e}") return [] + def get_account_resources( credentials: Dict[str, str], regions: List[str], exclude_resources: List[str], account_id: str, account_name: str, - emr_cluster_states: Optional[List[str]] = None + emr_cluster_states: Optional[List[str]] = None, ) -> Dict[str, List[Dict[str, Any]]]: """ Discover resources in an AWS account across multiple regions. - + Args: credentials: AWS credentials for the account regions: List of regions to scan @@ -428,7 +507,7 @@ def get_account_resources( account_id: AWS account ID account_name: AWS account name emr_cluster_states: Optional list of EMR cluster states to filter by - + Returns: Dictionary of discovered resources by type """ @@ -438,304 +517,341 @@ def get_account_resources( partition = detect_partition_from_credentials(credentials) # Get regions for the detected partition regions = get_regions_for_partition(partition) - logger.info(f"Using default regions for partition {partition} in account {account_id}") - - logger.info(f"Discovering resources in account {account_id} ({account_name}) across {len(regions)} regions") - + logger.info( + f"Using default regions for partition {partition} in account {account_id}" + ) + + logger.info( + f"Discovering resources in account {account_id} ({account_name}) across {len(regions)} regions" + ) + resources = { - 'ec2_instances': [], - 'rds_instances': [], - 'eks_clusters': [], - 'emr_clusters': [] + "ec2_instances": [], + "rds_instances": [], + "eks_clusters": [], + "emr_clusters": [], } - + # Create a ResourceDiscovery instance for this account discovery = ResourceDiscovery(credentials, account_id) - + # Use parallel processing for regions to speed up discovery with ThreadPoolExecutor(max_workers=min(10, len(regions))) as executor: # Submit tasks for each region and resource type futures = {} - + for region in regions: logger.debug(f"Scanning region {region} in account {account_id}") - + # Get EC2 instances if not excluded if "ec2" not in exclude_resources: - future = executor.submit( - discovery._discover_ec2, - region - ) - futures[(region, 'ec2')] = future - + future = executor.submit(discovery._discover_ec2, region) + futures[(region, "ec2")] = future + # Get RDS instances if not excluded if "rds" not in exclude_resources: - future = executor.submit( - discovery._discover_rds, - region - ) - futures[(region, 'rds')] = future - + future = executor.submit(discovery._discover_rds, region) + futures[(region, "rds")] = future + # Get EKS clusters if not excluded if "eks" not in exclude_resources: - future = executor.submit( - discovery._discover_eks, - region - ) - futures[(region, 'eks')] = future - + future = executor.submit(discovery._discover_eks, region) + futures[(region, "eks")] = future + # Get EMR clusters if not excluded if "emr" not in exclude_resources: future = executor.submit( - discovery._discover_emr, - region, - emr_cluster_states + discovery._discover_emr, region, emr_cluster_states ) - futures[(region, 'emr')] = future - + futures[(region, "emr")] = future + # Collect results as they complete for (region, resource_type), future in futures.items(): try: result = future.result() - if resource_type == 'ec2': - resources['ec2_instances'].extend(result) - elif resource_type == 'rds': - resources['rds_instances'].extend(result) - elif resource_type == 'eks': - resources['eks_clusters'].extend(result) - elif resource_type == 'emr': - resources['emr_clusters'].extend(result) - logger.debug(f"Found {len(result)} {resource_type} resources in {region}") + if resource_type == "ec2": + resources["ec2_instances"].extend(result) + elif resource_type == "rds": + resources["rds_instances"].extend(result) + elif resource_type == "eks": + resources["eks_clusters"].extend(result) + elif resource_type == "emr": + resources["emr_clusters"].extend(result) + logger.debug( + f"Found {len(result)} {resource_type} resources in {region}" + ) except Exception as e: logger.error(f"Error discovering {resource_type} in {region}: {str(e)}") - + # Log summary - logger.info(f"Account {account_id} - Found {len(resources['ec2_instances'])} EC2 instances") - logger.info(f"Account {account_id} - Found {len(resources['rds_instances'])} RDS instances") - logger.info(f"Account {account_id} - Found {len(resources['eks_clusters'])} EKS clusters") - logger.info(f"Account {account_id} - Found {len(resources['emr_clusters'])} EMR clusters") - + logger.info( + f"Account {account_id} - Found {len(resources['ec2_instances'])} EC2 instances" + ) + logger.info( + f"Account {account_id} - Found {len(resources['rds_instances'])} RDS instances" + ) + logger.info( + f"Account {account_id} - Found {len(resources['eks_clusters'])} EKS clusters" + ) + logger.info( + f"Account {account_id} - Found {len(resources['emr_clusters'])} EMR clusters" + ) + return resources -def discover_ec2_instances(credentials: Dict[str, str], region: str, account_id: str) -> List[Dict[str, Any]]: + +def discover_ec2_instances( + credentials: Dict[str, str], region: str, account_id: str +) -> List[Dict[str, Any]]: """ Discover EC2 instances in a region. - + Args: credentials: AWS credentials region: AWS region account_id: AWS account ID - + Returns: List of EC2 instance dictionaries """ - ec2_client = boto3.client('ec2', region_name=region, **credentials) + ec2_client = boto3.client("ec2", region_name=region, **credentials) instances = [] - + try: - paginator = ec2_client.get_paginator('describe_instances') - + paginator = ec2_client.get_paginator("describe_instances") + for page in paginator.paginate(): - for reservation in page['Reservations']: - for instance in reservation['Instances']: + for reservation in page["Reservations"]: + for instance in reservation["Instances"]: # Skip terminated instances - if instance['State']['Name'] == 'terminated': + if instance["State"]["Name"] == "terminated": continue - + # Extract tags as a dictionary tags = {} - if 'Tags' in instance: - tags = {tag['Key']: tag['Value'] for tag in instance['Tags']} - + if "Tags" in instance: + tags = {tag["Key"]: tag["Value"] for tag in instance["Tags"]} + # Create a simplified instance object instance_obj = { - 'id': instance['InstanceId'], - 'type': instance['InstanceType'], - 'state': instance['State']['Name'], - 'region': region, - 'account_id': account_id, - 'tags': tags, - 'name': tags.get('Name', instance['InstanceId']) + "id": instance["InstanceId"], + "type": instance["InstanceType"], + "state": instance["State"]["Name"], + "region": region, + "account_id": account_id, + "tags": tags, + "name": tags.get("Name", instance["InstanceId"]), } - + instances.append(instance_obj) - + return instances except ClientError as e: logger.error(f"Error discovering EC2 instances in {region}: {str(e)}") return [] -def discover_rds_instances(credentials: Dict[str, str], region: str, account_id: str) -> List[Dict[str, Any]]: + +def discover_rds_instances( + credentials: Dict[str, str], region: str, account_id: str +) -> List[Dict[str, Any]]: """ Discover RDS instances in a region. - + Args: credentials: AWS credentials region: AWS region account_id: AWS account ID - + Returns: List of RDS instance dictionaries """ - rds_client = boto3.client('rds', region_name=region, **credentials) + rds_client = boto3.client("rds", region_name=region, **credentials) instances = [] - + try: - paginator = rds_client.get_paginator('describe_db_instances') - + paginator = rds_client.get_paginator("describe_db_instances") + for page in paginator.paginate(): - for instance in page['DBInstances']: + for instance in page["DBInstances"]: # Create a simplified instance object instance_obj = { - 'id': instance['DBInstanceIdentifier'], - 'engine': instance['Engine'], - 'state': instance['DBInstanceStatus'], - 'region': region, - 'account_id': account_id, - 'name': instance['DBInstanceIdentifier'] + "id": instance["DBInstanceIdentifier"], + "engine": instance["Engine"], + "state": instance["DBInstanceStatus"], + "region": region, + "account_id": account_id, + "name": instance["DBInstanceIdentifier"], } - + instances.append(instance_obj) - + return instances except ClientError as e: logger.error(f"Error discovering RDS instances in {region}: {str(e)}") return [] -def discover_eks_clusters(credentials: Dict[str, str], region: str, account_id: str) -> List[Dict[str, Any]]: + +def discover_eks_clusters( + credentials: Dict[str, str], region: str, account_id: str +) -> List[Dict[str, Any]]: """ Discover EKS clusters in a region. - + Args: credentials: AWS credentials region: AWS region account_id: AWS account ID - + Returns: List of EKS cluster dictionaries """ - eks_client = boto3.client('eks', region_name=region, **credentials) + eks_client = boto3.client("eks", region_name=region, **credentials) clusters = [] - + try: response = eks_client.list_clusters() - - for cluster_name in response['clusters']: - cluster_details = eks_client.describe_cluster(name=cluster_name)['cluster'] - + + for cluster_name in response["clusters"]: + cluster_details = eks_client.describe_cluster(name=cluster_name)["cluster"] + # Create a simplified cluster object cluster_obj = { - 'id': cluster_name, - 'name': cluster_name, - 'status': cluster_details['status'], - 'region': region, - 'account_id': account_id, - 'version': cluster_details['version'] + "id": cluster_name, + "name": cluster_name, + "status": cluster_details["status"], + "region": region, + "account_id": account_id, + "version": cluster_details["version"], } - + clusters.append(cluster_obj) - + # Handle pagination if there are more clusters - while 'nextToken' in response: - response = eks_client.list_clusters(nextToken=response['nextToken']) - for cluster_name in response['clusters']: - cluster_details = eks_client.describe_cluster(name=cluster_name)['cluster'] - + while "nextToken" in response: + response = eks_client.list_clusters(nextToken=response["nextToken"]) + for cluster_name in response["clusters"]: + cluster_details = eks_client.describe_cluster(name=cluster_name)[ + "cluster" + ] + # Create a simplified cluster object cluster_obj = { - 'id': cluster_name, - 'name': cluster_name, - 'status': cluster_details['status'], - 'region': region, - 'account_id': account_id, - 'version': cluster_details['version'] + "id": cluster_name, + "name": cluster_name, + "status": cluster_details["status"], + "region": region, + "account_id": account_id, + "version": cluster_details["version"], } - + clusters.append(cluster_obj) - + return clusters except ClientError as e: logger.error(f"Error discovering EKS clusters in {region}: {str(e)}") return [] -def discover_emr_clusters(credentials: Dict[str, str], region: str, account_id: str, - cluster_states: Optional[List[str]] = None) -> List[Dict[str, Any]]: + +def discover_emr_clusters( + credentials: Dict[str, str], + region: str, + account_id: str, + cluster_states: Optional[List[str]] = None, +) -> List[Dict[str, Any]]: """ Discover EMR clusters in a region. - + Args: credentials: AWS credentials region: AWS region account_id: AWS account ID cluster_states: Optional list of EMR cluster states to filter by - + Returns: List of EMR cluster dictionaries """ - emr_client = boto3.client('emr', region_name=region, **credentials) + emr_client = boto3.client("emr", region_name=region, **credentials) clusters = [] - + try: - paginator = emr_client.get_paginator('list_clusters') - + paginator = emr_client.get_paginator("list_clusters") + # Only list active clusters with valid states # Default cluster states, 'STOPPED' is not a valid state for ListClusters API # See: https://docs.aws.amazon.com/emr/latest/APIReference/API_ListClusters.html if cluster_states is None: - cluster_states = ['STARTING', 'BOOTSTRAPPING', 'RUNNING', 'WAITING', 'TERMINATING'] - + cluster_states = [ + "STARTING", + "BOOTSTRAPPING", + "RUNNING", + "WAITING", + "TERMINATING", + ] + try: for page in paginator.paginate(ClusterStates=cluster_states): - for cluster in page.get('Clusters', []): + for cluster in page.get("Clusters", []): try: # Safely extract cluster state with fallbacks - if 'Status' in cluster and isinstance(cluster['Status'], dict) and 'State' in cluster['Status']: - state = cluster['Status']['State'] + if ( + "Status" in cluster + and isinstance(cluster["Status"], dict) + and "State" in cluster["Status"] + ): + state = cluster["Status"]["State"] else: - state = 'UNKNOWN' - + state = "UNKNOWN" + # Create a simplified cluster object with both state and status fields cluster_obj = { - 'id': cluster.get('Id', 'unknown-id'), - 'name': cluster.get('Name', 'Unknown'), - 'state': state, - 'status': state, # Duplicate to ensure both fields exist - 'region': region, - 'account_id': account_id + "id": cluster.get("Id", "unknown-id"), + "name": cluster.get("Name", "Unknown"), + "state": state, + "status": state, # Duplicate to ensure both fields exist + "region": region, + "account_id": account_id, } - + clusters.append(cluster_obj) except Exception as e: - logger.warning(f"Error processing EMR cluster in {region}: {str(e)}") + logger.warning( + f"Error processing EMR cluster in {region}: {str(e)}" + ) # Continue with next cluster continue - + except Exception as e: logger.warning(f"Error during EMR pagination in {region}: {str(e)}") # Try using list_clusters without pagination as fallback try: response = emr_client.list_clusters(ClusterStates=cluster_states) - for cluster in response.get('Clusters', []): + for cluster in response.get("Clusters", []): # Safely extract cluster state with fallbacks - if 'Status' in cluster and isinstance(cluster['Status'], dict) and 'State' in cluster['Status']: - state = cluster['Status']['State'] + if ( + "Status" in cluster + and isinstance(cluster["Status"], dict) + and "State" in cluster["Status"] + ): + state = cluster["Status"]["State"] else: - state = 'UNKNOWN' - + state = "UNKNOWN" + # Create a simplified cluster object with both state and status fields cluster_obj = { - 'id': cluster.get('Id', 'unknown-id'), - 'name': cluster.get('Name', 'Unknown'), - 'state': state, - 'status': state, # Duplicate to ensure both fields exist - 'region': region, - 'account_id': account_id + "id": cluster.get("Id", "unknown-id"), + "name": cluster.get("Name", "Unknown"), + "state": state, + "status": state, # Duplicate to ensure both fields exist + "region": region, + "account_id": account_id, } - + clusters.append(cluster_obj) except Exception as nested_e: - logger.error(f"Fallback EMR cluster retrieval failed in {region}: {str(nested_e)}") - + logger.error( + f"Fallback EMR cluster retrieval failed in {region}: {str(nested_e)}" + ) + return clusters except ClientError as e: logger.error(f"Error discovering EMR clusters in {region}: {str(e)}") diff --git a/local-app/python-tools/gfl-resource-actions/aws_resource_management/logging_setup.py b/local-app/python-tools/gfl-resource-actions/aws_resource_management/logging_setup.py index 1add8ec1..2969d039 100644 --- a/local-app/python-tools/gfl-resource-actions/aws_resource_management/logging_setup.py +++ b/local-app/python-tools/gfl-resource-actions/aws_resource_management/logging_setup.py @@ -1,177 +1,196 @@ """ Logging setup for AWS Resource Management tool. """ + +import csv +import datetime +import json import logging import os -import json import sys -import csv -import datetime from pathlib import Path -from typing import Optional, Dict, Any +from typing import Any, Dict, Optional from aws_resource_management.config_manager import get_config + class JSONFormatter(logging.Formatter): """ JSON formatter for structured logging. """ + def format(self, record: logging.LogRecord) -> str: """ Format the log record as a JSON string. - + Args: record: Log record - + Returns: JSON formatted string """ log_data = { - 'timestamp': datetime.fromtimestamp(record.created).isoformat(), - 'level': record.levelname, - 'message': record.getMessage(), - 'module': record.module, - 'function': record.funcName, - 'line': record.lineno, + "timestamp": datetime.fromtimestamp(record.created).isoformat(), + "level": record.levelname, + "message": record.getMessage(), + "module": record.module, + "function": record.funcName, + "line": record.lineno, } - + # Include exception info if present if record.exc_info: - log_data['exception'] = { - 'type': record.exc_info[0].__name__, - 'message': str(record.exc_info[1]), + log_data["exception"] = { + "type": record.exc_info[0].__name__, + "message": str(record.exc_info[1]), } - + # Include any custom fields - for key, value in getattr(record, 'extra_fields', {}).items(): + for key, value in getattr(record, "extra_fields", {}).items(): log_data[key] = value - + return json.dumps(log_data) + def setup_logging( log_level: Optional[str] = None, log_file: Optional[str] = None, - use_json: bool = False + use_json: bool = False, ) -> logging.Logger: """ Set up logging configuration. - + Args: log_level: Log level (DEBUG, INFO, WARNING, ERROR, CRITICAL) log_file: Path to log file use_json: Whether to use JSON formatting - + Returns: Logger instance """ # Get root logger - logger = logging.getLogger('aws_resource_management') - + logger = logging.getLogger("aws_resource_management") + # Clear existing handlers to avoid duplicate logs if logger.handlers: logger.handlers = [] - + # Set log level if not log_level: - log_level = os.environ.get('AWS_RESOURCE_MGMT_LOG_LEVEL', 'INFO') - + log_level = os.environ.get("AWS_RESOURCE_MGMT_LOG_LEVEL", "INFO") + logger.setLevel(getattr(logging, log_level)) - + # Create console handler console_handler = logging.StreamHandler(sys.stdout) - + # Set formatter based on format preference if use_json: formatter = JSONFormatter() else: formatter = logging.Formatter( - '%(asctime)s [%(levelname)s] %(name)s - %(message)s' + "%(asctime)s [%(levelname)s] %(name)s - %(message)s" ) - + console_handler.setFormatter(formatter) logger.addHandler(console_handler) - + # Add file handler if log file is specified if log_file: file_handler = logging.FileHandler(log_file) file_handler.setFormatter(formatter) logger.addHandler(file_handler) - + return logger + class LoggerAdapter(logging.LoggerAdapter): """ Logger adapter for adding context to log messages. """ + def __init__(self, logger: logging.Logger, context: Dict[str, Any]): """ Initialize logger adapter with context. - + Args: logger: Logger instance context: Context dictionary """ super().__init__(logger, context) - + def process(self, msg: str, kwargs: Dict[str, Any]) -> tuple: """ Process the log message by adding context. - + Args: msg: Log message kwargs: Keyword arguments - + Returns: Tuple of (modified message, modified kwargs) """ # Add extra_fields for JSON formatter - if 'extra' not in kwargs: - kwargs['extra'] = {} - - if 'extra_fields' not in kwargs['extra']: - kwargs['extra']['extra_fields'] = {} - + if "extra" not in kwargs: + kwargs["extra"] = {} + + if "extra_fields" not in kwargs["extra"]: + kwargs["extra"]["extra_fields"] = {} + for key, value in self.extra.items(): - kwargs['extra']['extra_fields'][key] = value - + kwargs["extra"]["extra_fields"][key] = value + return msg, kwargs + def initialize_csv_log(csv_file: Optional[str] = None) -> str: """ Initialize CSV log file with headers if it doesn't exist. - + Args: csv_file: CSV file name (default: resource_actions.csv from config) - + Returns: Path to the CSV log file """ config = get_config() - + if csv_file is None: - csv_file = config.get('action_csv_file') - + csv_file = config.get("action_csv_file") + # Configure CSV logging directory - csv_log_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), '..', 'logs') + csv_log_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "logs") csv_path = os.path.join(csv_log_dir, csv_file) - + # Create directory if it doesn't exist Path(csv_log_dir).mkdir(parents=True, exist_ok=True) - + # Check if file exists, if not create with headers file_exists = os.path.isfile(csv_path) - + if not file_exists: - with open(csv_path, 'w', newline='') as f: + with open(csv_path, "w", newline="") as f: writer = csv.writer(f) - writer.writerow([ - 'timestamp', 'account_id', 'account_name', 'resource_type', - 'resource_id', 'resource_name', 'action', 'region', - 'status', 'details', 'dry_run', 'schedule' - ]) - + writer.writerow( + [ + "timestamp", + "account_id", + "account_name", + "resource_type", + "resource_id", + "resource_name", + "action", + "region", + "status", + "details", + "dry_run", + "schedule", + ] + ) + return csv_path + def log_action_to_csv( account_id: str, account_name: str, @@ -181,13 +200,13 @@ def log_action_to_csv( action: str, region: str, status: str, - details: str = "", + details: str = "", dry_run: bool = False, - existing_schedule: Optional[str] = None + existing_schedule: Optional[str] = None, ) -> None: """ Log resource action to CSV file for tracking. - + Args: account_id: AWS account ID account_name: AWS account name @@ -202,32 +221,45 @@ def log_action_to_csv( existing_schedule: Existing schedule for the resource """ timestamp = datetime.datetime.now().isoformat() - + # Initialize CSV log file csv_path = initialize_csv_log() - - with open(csv_path, 'a', newline='') as csvfile: + + with open(csv_path, "a", newline="") as csvfile: fieldnames = [ - 'timestamp', 'account_id', 'account_name', 'resource_type', 'resource_id', - 'resource_name', 'action', 'region', 'status', 'details', 'dry_run', 'schedule' + "timestamp", + "account_id", + "account_name", + "resource_type", + "resource_id", + "resource_name", + "action", + "region", + "status", + "details", + "dry_run", + "schedule", ] writer = csv.DictWriter(csvfile, fieldnames=fieldnames) - + # Write the log entry - writer.writerow({ - 'timestamp': timestamp, - 'account_id': account_id, - 'account_name': account_name, - 'resource_type': resource_type, - 'resource_id': resource_id, - 'resource_name': resource_name, - 'action': action, - 'region': region, - 'status': status, - 'details': details, - 'dry_run': 'Yes' if dry_run else 'No', - 'schedule': existing_schedule or '' - }) + writer.writerow( + { + "timestamp": timestamp, + "account_id": account_id, + "account_name": account_name, + "resource_type": resource_type, + "resource_id": resource_id, + "resource_name": resource_name, + "action": action, + "region": region, + "status": status, + "details": details, + "dry_run": "Yes" if dry_run else "No", + "schedule": existing_schedule or "", + } + ) + # Create a logger instance for direct import logger = setup_logging() diff --git a/local-app/python-tools/gfl-resource-actions/aws_resource_management/managers/__init__.py b/local-app/python-tools/gfl-resource-actions/aws_resource_management/managers/__init__.py index d64cb725..e2526461 100644 --- a/local-app/python-tools/gfl-resource-actions/aws_resource_management/managers/__init__.py +++ b/local-app/python-tools/gfl-resource-actions/aws_resource_management/managers/__init__.py @@ -1,8 +1,9 @@ """ Resource manager modules for different AWS services. """ + from aws_resource_management.managers.base import ResourceManager from aws_resource_management.managers.ec2 import EC2Manager -from aws_resource_management.managers.rds import RDSManager from aws_resource_management.managers.eks import EKSManager from aws_resource_management.managers.emr import EMRManager +from aws_resource_management.managers.rds import RDSManager diff --git a/local-app/python-tools/gfl-resource-actions/aws_resource_management/managers/base.py b/local-app/python-tools/gfl-resource-actions/aws_resource_management/managers/base.py index 71bcfac4..a5602032 100644 --- a/local-app/python-tools/gfl-resource-actions/aws_resource_management/managers/base.py +++ b/local-app/python-tools/gfl-resource-actions/aws_resource_management/managers/base.py @@ -1,29 +1,36 @@ """ Base resource manager class that all specific managers inherit from. """ -from typing import Dict, List, Any, Optional, Union -import boto3 + import datetime from abc import ABC, abstractmethod +from typing import Any, Dict, List, Optional, Union +import boto3 from aws_resource_management.config_manager import get_config from aws_resource_management.logging_setup import setup_logging logger = setup_logging() config = get_config() + class ResourceManager(ABC): """ Base class for all resource managers. - + This abstract class defines the common interface and functionality that all resource managers should implement. """ - - def __init__(self, credentials: Dict[str, str], account_id: str, account_name: Optional[str] = None): + + def __init__( + self, + credentials: Dict[str, str], + account_id: str, + account_name: Optional[str] = None, + ): """ Initialize the resource manager. - + Args: credentials: AWS credentials dictionary account_id: AWS account ID @@ -33,54 +40,54 @@ def __init__(self, credentials: Dict[str, str], account_id: str, account_name: O self.account_id = account_id self.account_name = account_name self.resource_type = "generic" # Override in subclass - + def create_client(self, service_name: str, region_name: str) -> Any: """ Create a boto3 client for the specified AWS service. - + Args: service_name: AWS service name region_name: AWS region name - + Returns: boto3 client """ return boto3.client( service_name, region_name=region_name, - aws_access_key_id=self.credentials.get('aws_access_key_id'), - aws_secret_access_key=self.credentials.get('aws_secret_access_key'), - aws_session_token=self.credentials.get('aws_session_token') + aws_access_key_id=self.credentials.get("aws_access_key_id"), + aws_secret_access_key=self.credentials.get("aws_secret_access_key"), + aws_session_token=self.credentials.get("aws_session_token"), ) - + def get_timestamp(self) -> str: """ Get current timestamp in ISO format. - + Returns: Timestamp string """ return datetime.datetime.now().isoformat() - + def should_exclude(self, resource: Dict[str, Any]) -> bool: """ Check if a resource should be excluded from operations based on tags. - + Args: resource: Resource dictionary with a 'tags' key - + Returns: True if resource should be excluded, False otherwise """ - tags = resource.get('tags', {}) - exclusion_tag = config.get('exclusion_tag') - + tags = resource.get("tags", {}) + exclusion_tag = config.get("exclusion_tag") + # Check if the exclusion tag exists if exclusion_tag in tags: return True - + return False - + def log_action( self, resource_id: str, @@ -89,11 +96,11 @@ def log_action( details: str = "", resource_name: Optional[str] = None, dry_run: bool = False, - existing_schedule: Optional[str] = None + existing_schedule: Optional[str] = None, ) -> None: """ Log an action performed on a resource. - + Args: resource_id: Resource ID region: AWS region @@ -103,19 +110,23 @@ def log_action( dry_run: Whether this was a dry run existing_schedule: Existing schedule if any (optional) """ - resource_info = f"'{resource_name}' ({resource_id})" if resource_name else resource_id + resource_info = ( + f"'{resource_name}' ({resource_id})" if resource_name else resource_id + ) dry_run_prefix = "[DRY RUN] " if dry_run else "" schedule_info = f", Schedule: {existing_schedule}" if existing_schedule else "" - - logger.info(f"{dry_run_prefix}{action.upper()} {self.resource_type} {resource_info} in {region}{schedule_info} {details}") - + + logger.info( + f"{dry_run_prefix}{action.upper()} {self.resource_type} {resource_info} in {region}{schedule_info} {details}" + ) + def get_partition_for_region(self, region: str) -> str: """ Get the AWS partition for a given region. - + Args: region: AWS region - + Returns: AWS partition (e.g., aws, aws-us-gov) """ @@ -125,30 +136,34 @@ def get_partition_for_region(self, region: str) -> str: return "aws-cn" else: return "aws" - + @abstractmethod - def stop(self, resources: List[Dict[str, Any]], dry_run: bool = False) -> Dict[str, int]: + def stop( + self, resources: List[Dict[str, Any]], dry_run: bool = False + ) -> Dict[str, int]: """ Stop resources. - + Args: resources: List of resource dictionaries dry_run: If True, only simulate the action - + Returns: Dictionary with success and error counts """ pass - + @abstractmethod - def start(self, resources: List[Dict[str, Any]], dry_run: bool = False) -> Dict[str, int]: + def start( + self, resources: List[Dict[str, Any]], dry_run: bool = False + ) -> Dict[str, int]: """ Start resources. - + Args: resources: List of resource dictionaries dry_run: If True, only simulate the action - + Returns: Dictionary with success and error counts """ diff --git a/local-app/python-tools/gfl-resource-actions/aws_resource_management/managers/ec2.py b/local-app/python-tools/gfl-resource-actions/aws_resource_management/managers/ec2.py index 25656e41..d795b19e 100644 --- a/local-app/python-tools/gfl-resource-actions/aws_resource_management/managers/ec2.py +++ b/local-app/python-tools/gfl-resource-actions/aws_resource_management/managers/ec2.py @@ -1,22 +1,29 @@ """ EC2 resource manager class. """ -from typing import Dict, List, Any, Optional -from aws_resource_management.managers.base import ResourceManager +from typing import Any, Dict, List, Optional + from aws_resource_management.config_manager import get_config from aws_resource_management.logging_setup import setup_logging +from aws_resource_management.managers.base import ResourceManager logger = setup_logging() config = get_config() + class EC2Manager(ResourceManager): """Manager for EC2 instances.""" - - def __init__(self, credentials: Dict[str, str], account_id: str, account_name: Optional[str] = None): + + def __init__( + self, + credentials: Dict[str, str], + account_id: str, + account_name: Optional[str] = None, + ): """ Initialize EC2 manager. - + Args: credentials: AWS credentials dictionary account_id: AWS account ID @@ -24,143 +31,150 @@ def __init__(self, credentials: Dict[str, str], account_id: str, account_name: O """ super().__init__(credentials, account_id, account_name) self.resource_type = "ec2_instance" - + def should_exclude(self, instance: Dict[str, Any]) -> bool: """ Check if EC2 instance should be excluded based on tags. - + Args: instance: Instance dictionary with tags - + Returns: True if the instance should be excluded, False otherwise """ tags = instance.get("tags", {}) - + # Check for explicit exclusion tag exclusion_tag = config.get("exclusion_tag") if exclusion_tag in tags: - logger.info(f"Skipping EC2 instance {instance['id']} with exclusion tag {exclusion_tag}") + logger.info( + f"Skipping EC2 instance {instance['id']} with exclusion tag {exclusion_tag}" + ) return True - + # Skip instances that are part of EKS clusters eks_tag = config.get("eks_tag") if eks_tag in tags: - logger.info(f"Skipping EC2 instance {instance['id']} with tag {eks_tag}={tags[eks_tag]} (EKS managed node)") + logger.info( + f"Skipping EC2 instance {instance['id']} with tag {eks_tag}={tags[eks_tag]} (EKS managed node)" + ) return True - + # Skip instances that are part of EMR clusters emr_tag = config.get("emr_tag") if emr_tag in tags: - logger.info(f"Skipping EC2 instance {instance['id']} with tag {emr_tag}={tags[emr_tag]} (EMR managed node)") + logger.info( + f"Skipping EC2 instance {instance['id']} with tag {emr_tag}={tags[emr_tag]} (EMR managed node)" + ) return True - + return False - - def stop(self, instances: List[Dict[str, Any]], dry_run: bool = False) -> Dict[str, int]: + + def stop( + self, instances: List[Dict[str, Any]], dry_run: bool = False + ) -> Dict[str, int]: """ Stop running EC2 instances. - + Args: instances: List of EC2 instance dictionaries dry_run: If True, only simulate the action - + Returns: Dictionary with success and error counts """ success_count = 0 error_count = 0 - + for instance in instances: if self.should_exclude(instance): continue - + if instance["state"] == "running": try: region = instance["region"] - ec2_client = self.create_client('ec2', region) - + ec2_client = self.create_client("ec2", region) + # Create ISO 8601 timestamp timestamp = self.get_timestamp() - + # Tag the instance before stopping - logger.info(f"{'[DRY RUN] Would tag' if dry_run else 'Tagging'} EC2 instance {instance['id']} with {config.get('stop_tag')}={timestamp}") + logger.info( + f"{'[DRY RUN] Would tag' if dry_run else 'Tagging'} EC2 instance {instance['id']} with {config.get('stop_tag')}={timestamp}" + ) if not dry_run: ec2_client.create_tags( - Resources=[instance['id']], - Tags=[ - { - 'Key': config.get('stop_tag'), - 'Value': timestamp - } - ] + Resources=[instance["id"]], + Tags=[{"Key": config.get("stop_tag"), "Value": timestamp}], ) - + # Stop the instance - logger.info(f"{'[DRY RUN] Would stop' if dry_run else 'Stopping'} EC2 instance {instance['id']} in region {region}") + logger.info( + f"{'[DRY RUN] Would stop' if dry_run else 'Stopping'} EC2 instance {instance['id']} in region {region}" + ) if not dry_run: - ec2_client.stop_instances(InstanceIds=[instance['id']]) - + ec2_client.stop_instances(InstanceIds=[instance["id"]]) + # Log with detailed information - self.log_action(instance['id'], region, "stop", dry_run=dry_run) + self.log_action(instance["id"], region, "stop", dry_run=dry_run) success_count += 1 - + except Exception as e: - logger.error(f"Failed to {'simulate stopping' if dry_run else 'stop'} EC2 instance {instance['id']}: {e}") + logger.error( + f"Failed to {'simulate stopping' if dry_run else 'stop'} EC2 instance {instance['id']}: {e}" + ) error_count += 1 - return { - "success": success_count, - "errors": error_count - } + return {"success": success_count, "errors": error_count} - def start(self, instances: List[Dict[str, Any]], dry_run: bool = False) -> Dict[str, int]: + def start( + self, instances: List[Dict[str, Any]], dry_run: bool = False + ) -> Dict[str, int]: """ Start stopped EC2 instances. - + Args: instances: List of EC2 instance dictionaries dry_run: If True, only simulate the action - + Returns: Dictionary with success and error counts """ success_count = 0 error_count = 0 - + for instance in instances: if self.should_exclude(instance): continue - + if instance["state"] == "stopped": try: region = instance["region"] - ec2_client = self.create_client('ec2', region) - - logger.info(f"{'[DRY RUN] Would start' if dry_run else 'Starting'} EC2 instance {instance['id']} in region {region}") + ec2_client = self.create_client("ec2", region) + + logger.info( + f"{'[DRY RUN] Would start' if dry_run else 'Starting'} EC2 instance {instance['id']} in region {region}" + ) if not dry_run: - ec2_client.start_instances(InstanceIds=[instance['id']]) - + ec2_client.start_instances(InstanceIds=[instance["id"]]) + # Remove the stop tag after successful start - logger.info(f"Removing {config.get('stop_tag')} tag from EC2 instance {instance['id']}") + logger.info( + f"Removing {config.get('stop_tag')} tag from EC2 instance {instance['id']}" + ) ec2_client.delete_tags( - Resources=[instance['id']], - Tags=[ - { - 'Key': config.get('stop_tag') - } - ] + Resources=[instance["id"]], + Tags=[{"Key": config.get("stop_tag")}], ) - + # Log with detailed information - self.log_action(instance['id'], region, "start", dry_run=dry_run) + self.log_action(instance["id"], region, "start", dry_run=dry_run) success_count += 1 - + except Exception as e: - logger.error(f"Failed to {'simulate starting' if dry_run else 'start'} EC2 instance {instance['id']}: {e}") + logger.error( + f"Failed to {'simulate starting' if dry_run else 'start'} EC2 instance {instance['id']}: {e}" + ) error_count += 1 - - return { - "success": success_count, - "errors": error_count - } + + return {"success": success_count, "errors": error_count} diff --git a/local-app/python-tools/gfl-resource-actions/aws_resource_management/managers/eks.py b/local-app/python-tools/gfl-resource-actions/aws_resource_management/managers/eks.py index 56745897..dcc12aca 100644 --- a/local-app/python-tools/gfl-resource-actions/aws_resource_management/managers/eks.py +++ b/local-app/python-tools/gfl-resource-actions/aws_resource_management/managers/eks.py @@ -1,32 +1,39 @@ """ EKS resource manager class. """ -from typing import Dict, List, Any, Optional, Union -from aws_resource_management.managers.base import ResourceManager +from typing import Any, Dict, List, Optional, Union + from aws_resource_management.config_manager import get_config from aws_resource_management.logging_setup import setup_logging +from aws_resource_management.managers.base import ResourceManager logger = setup_logging() config = get_config() # Tag names for EKS scaling -EKS_MIN_NODES_TAG = 'eks-min-nodes' -EKS_MAX_NODES_TAG = 'eks-max-nodes' -EKS_DESIRED_NODES_TAG = 'eks-desired-nodes' -EKS_ORIGINAL_MIN_NODES_TAG = 'eks-original-min-nodes' -EKS_ORIGINAL_MAX_NODES_TAG = 'eks-original-max-nodes' -EKS_ORIGINAL_DESIRED_NODES_TAG = 'eks-original-desired-nodes' -CLUSTER_SIZE_TAG = 'cluster:size' -EKS_ORIGINAL_CLUSTER_SIZE_TAG = 'eks-original-cluster-size' +EKS_MIN_NODES_TAG = "eks-min-nodes" +EKS_MAX_NODES_TAG = "eks-max-nodes" +EKS_DESIRED_NODES_TAG = "eks-desired-nodes" +EKS_ORIGINAL_MIN_NODES_TAG = "eks-original-min-nodes" +EKS_ORIGINAL_MAX_NODES_TAG = "eks-original-max-nodes" +EKS_ORIGINAL_DESIRED_NODES_TAG = "eks-original-desired-nodes" +CLUSTER_SIZE_TAG = "cluster:size" +EKS_ORIGINAL_CLUSTER_SIZE_TAG = "eks-original-cluster-size" + class EKSManager(ResourceManager): """Manager for EKS clusters.""" - - def __init__(self, credentials: Dict[str, str], account_id: str, account_name: Optional[str] = None): + + def __init__( + self, + credentials: Dict[str, str], + account_id: str, + account_name: Optional[str] = None, + ): """ Initialize EKS manager. - + Args: credentials: AWS credentials dictionary account_id: AWS account ID @@ -34,51 +41,53 @@ def __init__(self, credentials: Dict[str, str], account_id: str, account_name: O """ super().__init__(credentials, account_id, account_name) self.resource_type = "eks_cluster" - - def stop(self, clusters: List[Dict[str, Any]], dry_run: bool = False) -> Dict[str, int]: + + def stop( + self, clusters: List[Dict[str, Any]], dry_run: bool = False + ) -> Dict[str, int]: """ Stop EKS clusters by scaling down their nodegroups. - + Args: clusters: List of EKS cluster dictionaries dry_run: If True, only simulate the action - + Returns: Dictionary with success and error counts """ - logger.info(f"Delegating EKS stop operation to scale_down for {len(clusters)} clusters") + logger.info( + f"Delegating EKS stop operation to scale_down for {len(clusters)} clusters" + ) self.scale_down(clusters, dry_run) - + # Return a result dictionary that matches the expected interface - return { - "success": len(clusters), - "errors": 0 - } - - def start(self, clusters: List[Dict[str, Any]], dry_run: bool = False) -> Dict[str, int]: + return {"success": len(clusters), "errors": 0} + + def start( + self, clusters: List[Dict[str, Any]], dry_run: bool = False + ) -> Dict[str, int]: """ Start EKS clusters by scaling up their nodegroups. - + Args: clusters: List of EKS cluster dictionaries dry_run: If True, only simulate the action - + Returns: Dictionary with success and error counts """ - logger.info(f"Delegating EKS start operation to scale_up for {len(clusters)} clusters") + logger.info( + f"Delegating EKS start operation to scale_up for {len(clusters)} clusters" + ) self.scale_up(clusters, dry_run) - + # Return a result dictionary that matches the expected interface - return { - "success": len(clusters), - "errors": 0 - } - + return {"success": len(clusters), "errors": 0} + def scale_down(self, clusters: List[Dict[str, Any]], dry_run: bool = False) -> None: """ Scale down EKS clusters by setting node counts to 0. - + Args: clusters: List of EKS cluster dictionaries dry_run: If True, only simulate the action @@ -86,95 +95,126 @@ def scale_down(self, clusters: List[Dict[str, Any]], dry_run: bool = False) -> N logger.info(f"Evaluating {len(clusters)} EKS clusters for scale down action") scaled_count = 0 skipped_count = 0 - + for cluster in clusters: # Skip clusters with the exclusion tag if self.should_exclude(cluster): - logger.info(f"Skipping EKS cluster {cluster['name']} with exclusion tag {config.get('exclusion_tag')}") + logger.info( + f"Skipping EKS cluster {cluster['name']} with exclusion tag {config.get('exclusion_tag')}" + ) skipped_count += 1 continue - + try: region = cluster["region"] - eks_client = self.create_client('eks', region) - + eks_client = self.create_client("eks", region) + # Get current scaling parameters from tags or nodegroups - min_nodes = cluster.get('min_nodes', '') - max_nodes = cluster.get('max_nodes', '') - desired_nodes = cluster.get('desired_nodes', '') - schedule = cluster.get('schedule', '') - + min_nodes = cluster.get("min_nodes", "") + max_nodes = cluster.get("max_nodes", "") + desired_nodes = cluster.get("desired_nodes", "") + schedule = cluster.get("schedule", "") + # Check if we have a combined cluster:size tag - has_combined_size_tag = CLUSTER_SIZE_TAG in cluster.get('tags', {}) - + has_combined_size_tag = CLUSTER_SIZE_TAG in cluster.get("tags", {}) + # Only proceed if we have some scaling values to work with if min_nodes or max_nodes or desired_nodes or has_combined_size_tag: - logger.info(f"{'[DRY RUN] Would scale down' if dry_run else 'Scaling down'} EKS cluster {cluster['name']} in region {region}") - + logger.info( + f"{'[DRY RUN] Would scale down' if dry_run else 'Scaling down'} EKS cluster {cluster['name']} in region {region}" + ) + if not dry_run: # First, backup the current values in special tags tags_to_update = {} - + # Handle combined size tag if it exists if has_combined_size_tag: - original_size_tag = cluster['tags'][CLUSTER_SIZE_TAG] - tags_to_update[EKS_ORIGINAL_CLUSTER_SIZE_TAG] = original_size_tag + original_size_tag = cluster["tags"][CLUSTER_SIZE_TAG] + tags_to_update[EKS_ORIGINAL_CLUSTER_SIZE_TAG] = ( + original_size_tag + ) tags_to_update[CLUSTER_SIZE_TAG] = "min:0-max:0-desired:0" else: # Store original values in backup tags (individual tags) if min_nodes: tags_to_update[EKS_ORIGINAL_MIN_NODES_TAG] = min_nodes - tags_to_update[EKS_MIN_NODES_TAG] = '0' + tags_to_update[EKS_MIN_NODES_TAG] = "0" if max_nodes: tags_to_update[EKS_ORIGINAL_MAX_NODES_TAG] = max_nodes - tags_to_update[EKS_MAX_NODES_TAG] = '0' + tags_to_update[EKS_MAX_NODES_TAG] = "0" if desired_nodes: - tags_to_update[EKS_ORIGINAL_DESIRED_NODES_TAG] = desired_nodes - tags_to_update[EKS_DESIRED_NODES_TAG] = '0' - + tags_to_update[EKS_ORIGINAL_DESIRED_NODES_TAG] = ( + desired_nodes + ) + tags_to_update[EKS_DESIRED_NODES_TAG] = "0" + # Apply the tag updates if tags_to_update: - logger.info(f"Updating scaling tags for EKS cluster {cluster['name']}: {tags_to_update}") + logger.info( + f"Updating scaling tags for EKS cluster {cluster['name']}: {tags_to_update}" + ) eks_client.tag_resource( - resourceArn=cluster.get('arn', f"arn:{self.get_partition_for_region(region)}:eks:{region}:{self.account_id}:cluster/{cluster['name']}"), - tags=tags_to_update + resourceArn=cluster.get( + "arn", + f"arn:{self.get_partition_for_region(region)}:eks:{region}:{self.account_id}:cluster/{cluster['name']}", + ), + tags=tags_to_update, ) - + # Now actually scale down the nodegroups - self._scale_nodegroups(eks_client, cluster['name'], 0, region, dry_run=False) + self._scale_nodegroups( + eks_client, cluster["name"], 0, region, dry_run=False + ) else: # Dry run reporting if has_combined_size_tag: - original_size = cluster['tags'][CLUSTER_SIZE_TAG] - logger.info(f"[DRY RUN] Would backup combined size tag: {original_size} and set to min:0-max:0-desired:0") + original_size = cluster["tags"][CLUSTER_SIZE_TAG] + logger.info( + f"[DRY RUN] Would backup combined size tag: {original_size} and set to min:0-max:0-desired:0" + ) else: - logger.info(f"[DRY RUN] Would backup scaling values: Min={min_nodes}, Max={max_nodes}, Desired={desired_nodes}") - - logger.info(f"[DRY RUN] Would set scaling values to 0 for EKS cluster {cluster['name']}") - self._scale_nodegroups(eks_client, cluster['name'], 0, region, dry_run=True) - + logger.info( + f"[DRY RUN] Would backup scaling values: Min={min_nodes}, Max={max_nodes}, Desired={desired_nodes}" + ) + + logger.info( + f"[DRY RUN] Would set scaling values to 0 for EKS cluster {cluster['name']}" + ) + self._scale_nodegroups( + eks_client, cluster["name"], 0, region, dry_run=True + ) + # Log the action with schedule information schedule_info = f", Schedule: {schedule}" if schedule else "" self.log_action( - cluster['name'], region, "scale_down", - details=f"Min={min_nodes}->{0}, Max={max_nodes}->{0}, Desired={desired_nodes}->{0}{schedule_info}", + cluster["name"], + region, + "scale_down", + details=f"Min={min_nodes}->{0}, Max={max_nodes}->{0}, Desired={desired_nodes}->{0}{schedule_info}", dry_run=dry_run, - existing_schedule=schedule + existing_schedule=schedule, ) scaled_count += 1 else: - logger.info(f"Skipping EKS cluster {cluster['name']}, no scaling tags found") + logger.info( + f"Skipping EKS cluster {cluster['name']}, no scaling tags found" + ) skipped_count += 1 - + except Exception as e: - logger.error(f"Failed to {'simulate scaling down' if dry_run else 'scale down'} EKS cluster {cluster['name']}: {e}") - - logger.info(f"EKS scale down summary: {scaled_count} clusters processed, {skipped_count} clusters skipped") - + logger.error( + f"Failed to {'simulate scaling down' if dry_run else 'scale down'} EKS cluster {cluster['name']}: {e}" + ) + + logger.info( + f"EKS scale down summary: {scaled_count} clusters processed, {skipped_count} clusters skipped" + ) + def scale_up(self, clusters: List[Dict[str, Any]], dry_run: bool = False) -> None: """ Scale up EKS clusters using original node counts from tags. - + Args: clusters: List of EKS cluster dictionaries dry_run: If True, only simulate the action @@ -182,51 +222,68 @@ def scale_up(self, clusters: List[Dict[str, Any]], dry_run: bool = False) -> Non logger.info(f"Evaluating {len(clusters)} EKS clusters for scale up action") scaled_count = 0 skipped_count = 0 - + for cluster in clusters: # Skip clusters with the exclusion tag if self.should_exclude(cluster): - logger.info(f"Skipping EKS cluster {cluster['name']} with exclusion tag {config.get('exclusion_tag')}") + logger.info( + f"Skipping EKS cluster {cluster['name']} with exclusion tag {config.get('exclusion_tag')}" + ) skipped_count += 1 continue - + try: region = cluster["region"] - eks_client = self.create_client('eks', region) - + eks_client = self.create_client("eks", region) + # Check for original values in backup tags - original_min = cluster.get('original_min', '') - original_max = cluster.get('original_max', '') - original_desired = cluster.get('original_desired', '') - schedule = cluster.get('schedule', '') - + original_min = cluster.get("original_min", "") + original_max = cluster.get("original_max", "") + original_desired = cluster.get("original_desired", "") + schedule = cluster.get("schedule", "") + # Check for original combined size tag - tags = cluster.get('tags', {}) + tags = cluster.get("tags", {}) has_original_combined_tag = EKS_ORIGINAL_CLUSTER_SIZE_TAG in tags - + # If we have backup values or original combined tag, restore them - if original_min or original_max or original_desired or has_original_combined_tag: - logger.info(f"{'[DRY RUN] Would scale up' if dry_run else 'Scaling up'} EKS cluster {cluster['name']} in region {region}") - + if ( + original_min + or original_max + or original_desired + or has_original_combined_tag + ): + logger.info( + f"{'[DRY RUN] Would scale up' if dry_run else 'Scaling up'} EKS cluster {cluster['name']} in region {region}" + ) + if not dry_run: # Restore the original values from backup tags tags_to_update = {} tags_to_remove = [] - + # Handle original combined size tag if it exists if has_original_combined_tag: original_size_tag = tags[EKS_ORIGINAL_CLUSTER_SIZE_TAG] tags_to_update[CLUSTER_SIZE_TAG] = original_size_tag tags_to_remove.append(EKS_ORIGINAL_CLUSTER_SIZE_TAG) - + # Parse the original desired value from the size tag desired = None - if 'desired:' in original_size_tag: + if "desired:" in original_size_tag: try: - desired_str = original_size_tag.split('desired:')[1].split('-')[0] - desired = int(desired_str) if desired_str.isdigit() else None + desired_str = original_size_tag.split("desired:")[ + 1 + ].split("-")[0] + desired = ( + int(desired_str) + if desired_str.isdigit() + else None + ) except: - logger.warning(f"Could not parse desired value from combined size tag: {original_size_tag}") + logger.warning( + f"Could not parse desired value from combined size tag: {original_size_tag}" + ) else: # Restore original values for individual tags if original_min: @@ -238,77 +295,130 @@ def scale_up(self, clusters: List[Dict[str, Any]], dry_run: bool = False) -> Non if original_desired: tags_to_update[EKS_DESIRED_NODES_TAG] = original_desired tags_to_remove.append(EKS_ORIGINAL_DESIRED_NODES_TAG) - + # Parse desired value - desired = int(original_desired) if original_desired and original_desired.isdigit() else None - + desired = ( + int(original_desired) + if original_desired and original_desired.isdigit() + else None + ) + # Apply the tag updates if tags_to_update: - logger.info(f"Restoring original scaling tags for EKS cluster {cluster['name']}: {tags_to_update}") + logger.info( + f"Restoring original scaling tags for EKS cluster {cluster['name']}: {tags_to_update}" + ) eks_client.tag_resource( - resourceArn=cluster.get('arn', f"arn:{self.get_partition_for_region(region)}:eks:{region}:{self.account_id}:cluster/{cluster['name']}"), - tags=tags_to_update + resourceArn=cluster.get( + "arn", + f"arn:{self.get_partition_for_region(region)}:eks:{region}:{self.account_id}:cluster/{cluster['name']}", + ), + tags=tags_to_update, ) - + # Now actually scale up the nodegroups to desired capacity - self._scale_nodegroups(eks_client, cluster['name'], desired, region, dry_run=False) - + self._scale_nodegroups( + eks_client, + cluster["name"], + desired, + region, + dry_run=False, + ) + # Remove the backup tags if tags_to_remove: - logger.info(f"Removing backup scaling tags: {tags_to_remove}") + logger.info( + f"Removing backup scaling tags: {tags_to_remove}" + ) eks_client.untag_resource( - resourceArn=cluster.get('arn', f"arn:{self.get_partition_for_region(region)}:eks:{region}:{self.account_id}:cluster/{cluster['name']}"), - tagKeys=tags_to_remove + resourceArn=cluster.get( + "arn", + f"arn:{self.get_partition_for_region(region)}:eks:{region}:{self.account_id}:cluster/{cluster['name']}", + ), + tagKeys=tags_to_remove, ) else: # Dry run reporting if has_original_combined_tag: original_size = tags[EKS_ORIGINAL_CLUSTER_SIZE_TAG] - logger.info(f"[DRY RUN] Would restore combined size tag: {original_size}") - + logger.info( + f"[DRY RUN] Would restore combined size tag: {original_size}" + ) + # Try to parse desired value for nodegroup scaling desired = None - if 'desired:' in original_size: + if "desired:" in original_size: try: - desired_str = original_size.split('desired:')[1].split('-')[0] - desired = int(desired_str) if desired_str.isdigit() else None + desired_str = original_size.split("desired:")[ + 1 + ].split("-")[0] + desired = ( + int(desired_str) + if desired_str.isdigit() + else None + ) except: pass else: - logger.info(f"[DRY RUN] Would restore scaling values: Min={original_min}, Max={original_max}, Desired={original_desired}") - desired = int(original_desired) if original_desired and original_desired.isdigit() else None - - self._scale_nodegroups(eks_client, cluster['name'], desired, region, dry_run=True) - + logger.info( + f"[DRY RUN] Would restore scaling values: Min={original_min}, Max={original_max}, Desired={original_desired}" + ) + desired = ( + int(original_desired) + if original_desired and original_desired.isdigit() + else None + ) + + self._scale_nodegroups( + eks_client, cluster["name"], desired, region, dry_run=True + ) + # Log the action with schedule information scale_details = "" if has_original_combined_tag: - original_size = tags.get(EKS_ORIGINAL_CLUSTER_SIZE_TAG, "Unknown") + original_size = tags.get( + EKS_ORIGINAL_CLUSTER_SIZE_TAG, "Unknown" + ) scale_details = f"Size tag: 0->'{original_size}'" else: scale_details = f"Min=0->{original_min}, Max=0->{original_max}, Desired=0->{original_desired}" - + schedule_info = f", Schedule: {schedule}" if schedule else "" self.log_action( - cluster['name'], region, "scale_up", - details=f"{scale_details}{schedule_info}", + cluster["name"], + region, + "scale_up", + details=f"{scale_details}{schedule_info}", dry_run=dry_run, - existing_schedule=schedule + existing_schedule=schedule, ) scaled_count += 1 else: - logger.info(f"Skipping EKS cluster {cluster['name']}, no original scaling values found in tags") + logger.info( + f"Skipping EKS cluster {cluster['name']}, no original scaling values found in tags" + ) skipped_count += 1 - + except Exception as e: - logger.error(f"Failed to {'simulate scaling up' if dry_run else 'scale up'} EKS cluster {cluster['name']}: {e}") - - logger.info(f"EKS scale up summary: {scaled_count} clusters processed, {skipped_count} clusters skipped") - - def _scale_nodegroups(self, eks_client: Any, cluster_name: str, desired_capacity: Optional[int], region: str, dry_run: bool = False) -> None: + logger.error( + f"Failed to {'simulate scaling up' if dry_run else 'scale up'} EKS cluster {cluster['name']}: {e}" + ) + + logger.info( + f"EKS scale up summary: {scaled_count} clusters processed, {skipped_count} clusters skipped" + ) + + def _scale_nodegroups( + self, + eks_client: Any, + cluster_name: str, + desired_capacity: Optional[int], + region: str, + dry_run: bool = False, + ) -> None: """ Helper method to scale nodegroups to the specified capacity. - + Args: eks_client: EKS boto3 client cluster_name: Name of the EKS cluster @@ -319,67 +429,77 @@ def _scale_nodegroups(self, eks_client: Any, cluster_name: str, desired_capacity try: # List all nodegroups for this cluster response = eks_client.list_nodegroups(clusterName=cluster_name) - nodegroup_names = response.get('nodegroups', []) - + nodegroup_names = response.get("nodegroups", []) + # Handle pagination - while 'nextToken' in response: + while "nextToken" in response: response = eks_client.list_nodegroups( - clusterName=cluster_name, - nextToken=response['nextToken'] + clusterName=cluster_name, nextToken=response["nextToken"] ) - nodegroup_names.extend(response.get('nodegroups', [])) - + nodegroup_names.extend(response.get("nodegroups", [])) + if not nodegroup_names: logger.info(f"No nodegroups found for EKS cluster {cluster_name}") return - + for nodegroup_name in nodegroup_names: try: # Get nodegroup details nodegroup = eks_client.describe_nodegroup( - clusterName=cluster_name, - nodegroupName=nodegroup_name - ).get('nodegroup', {}) - + clusterName=cluster_name, nodegroupName=nodegroup_name + ).get("nodegroup", {}) + # Get current scaling configuration - current_min = nodegroup.get('scalingConfig', {}).get('minSize') - current_max = nodegroup.get('scalingConfig', {}).get('maxSize') - current_desired = nodegroup.get('scalingConfig', {}).get('desiredSize') - + current_min = nodegroup.get("scalingConfig", {}).get("minSize") + current_max = nodegroup.get("scalingConfig", {}).get("maxSize") + current_desired = nodegroup.get("scalingConfig", {}).get( + "desiredSize" + ) + # Use current min and max when scaling up if desired capacity is provided if desired_capacity is not None: # When scaling up, we need a valid min and max - new_min = current_min if desired_capacity == 0 else min(current_min, desired_capacity) + new_min = ( + current_min + if desired_capacity == 0 + else min(current_min, desired_capacity) + ) new_max = max(current_max, desired_capacity) else: # When scaling down to zero new_min = 0 new_max = current_max desired_capacity = 0 - + if dry_run: - logger.info(f"[DRY RUN] Would update nodegroup {nodegroup_name} scaling: " + - f"Min: {current_min}->{new_min}, " + - f"Max: {current_max}->{new_max}, " + - f"Desired: {current_desired}->{desired_capacity}") + logger.info( + f"[DRY RUN] Would update nodegroup {nodegroup_name} scaling: " + + f"Min: {current_min}->{new_min}, " + + f"Max: {current_max}->{new_max}, " + + f"Desired: {current_desired}->{desired_capacity}" + ) else: - logger.info(f"Updating nodegroup {nodegroup_name} scaling: " + - f"Min: {current_min}->{new_min}, " + - f"Max: {current_max}->{new_max}, " + - f"Desired: {current_desired}->{desired_capacity}") - + logger.info( + f"Updating nodegroup {nodegroup_name} scaling: " + + f"Min: {current_min}->{new_min}, " + + f"Max: {current_max}->{new_max}, " + + f"Desired: {current_desired}->{desired_capacity}" + ) + # Update the nodegroup scaling configuration eks_client.update_nodegroup_config( clusterName=cluster_name, nodegroupName=nodegroup_name, scalingConfig={ - 'minSize': new_min, - 'maxSize': new_max, - 'desiredSize': desired_capacity - } + "minSize": new_min, + "maxSize": new_max, + "desiredSize": desired_capacity, + }, ) except Exception as e: logger.error(f"Error updating nodegroup {nodegroup_name}: {e}") - + except Exception as e: - logger.error(f"Error listing nodegroups for cluster {cluster_name} in region {region}: {e}") + logger.error( + f"Error listing nodegroups for cluster {cluster_name} in region {region}: {e}" + ) diff --git a/local-app/python-tools/gfl-resource-actions/aws_resource_management/managers/emr.py b/local-app/python-tools/gfl-resource-actions/aws_resource_management/managers/emr.py index bb214a0b..dcb6cad7 100644 --- a/local-app/python-tools/gfl-resource-actions/aws_resource_management/managers/emr.py +++ b/local-app/python-tools/gfl-resource-actions/aws_resource_management/managers/emr.py @@ -1,23 +1,30 @@ """ EMR resource manager class. """ -from typing import Dict, List, Any, Optional -from botocore.exceptions import ClientError -from aws_resource_management.managers.base import ResourceManager +from typing import Any, Dict, List, Optional + from aws_resource_management.config_manager import get_config from aws_resource_management.logging_setup import setup_logging +from aws_resource_management.managers.base import ResourceManager +from botocore.exceptions import ClientError logger = setup_logging() config = get_config() + class EMRManager(ResourceManager): """Manager for EMR clusters.""" - - def __init__(self, credentials: Dict[str, str], account_id: str, account_name: Optional[str] = None): + + def __init__( + self, + credentials: Dict[str, str], + account_id: str, + account_name: Optional[str] = None, + ): """ Initialize EMR manager. - + Args: credentials: AWS credentials dictionary account_id: AWS account ID @@ -25,77 +32,102 @@ def __init__(self, credentials: Dict[str, str], account_id: str, account_name: O """ super().__init__(credentials, account_id, account_name) self.resource_type = "emr_cluster" - - def discover_clusters(self, region: str, cluster_states: List[str] = None) -> List[Dict[str, Any]]: + + def discover_clusters( + self, region: str, cluster_states: List[str] = None + ) -> List[Dict[str, Any]]: """ Discover EMR clusters in the given region with the specified states. - + Args: region: AWS region cluster_states: List of EMR cluster states to filter by - + Returns: List of EMR cluster dictionaries """ try: # Default states if not provided - EMR API only accepts specific states if not cluster_states: - cluster_states = ['STARTING', 'BOOTSTRAPPING', 'RUNNING', 'WAITING', 'TERMINATING'] - - logger.debug(f"Discovering EMR clusters in {region} with states: {', '.join(cluster_states)}") - - emr_client = self.create_client('emr', region) + cluster_states = [ + "STARTING", + "BOOTSTRAPPING", + "RUNNING", + "WAITING", + "TERMINATING", + ] + + logger.debug( + f"Discovering EMR clusters in {region} with states: {', '.join(cluster_states)}" + ) + + emr_client = self.create_client("emr", region) clusters = [] - + # ListClusters API only returns a subset of data, with pagination - paginator = emr_client.get_paginator('list_clusters') + paginator = emr_client.get_paginator("list_clusters") page_iterator = paginator.paginate(ClusterStates=cluster_states) - + for page in page_iterator: - if 'Clusters' in page: - for cluster_summary in page['Clusters']: + if "Clusters" in page: + for cluster_summary in page["Clusters"]: # Get detailed info for each cluster try: - cluster_id = cluster_summary.get('Id') + cluster_id = cluster_summary.get("Id") if not cluster_id: logger.warning("Skipping cluster with missing ID") continue - + # Format basic info cluster = { - 'id': cluster_id, - 'name': cluster_summary.get('Name', 'Unknown'), - 'status': cluster_summary.get('Status', {}).get('State', 'UNKNOWN'), - 'region': region, - 'account_id': self.account_id, - 'account_name': self.account_name + "id": cluster_id, + "name": cluster_summary.get("Name", "Unknown"), + "status": cluster_summary.get("Status", {}).get( + "State", "UNKNOWN" + ), + "region": region, + "account_id": self.account_id, + "account_name": self.account_name, } - + # Get tags try: - response = emr_client.describe_cluster(ClusterId=cluster_id) - if 'Cluster' in response and 'Tags' in response['Cluster']: + response = emr_client.describe_cluster( + ClusterId=cluster_id + ) + if ( + "Cluster" in response + and "Tags" in response["Cluster"] + ): tags = {} - for tag in response['Cluster']['Tags']: - tags[tag.get('Key')] = tag.get('Value') - cluster['tags'] = tags + for tag in response["Cluster"]["Tags"]: + tags[tag.get("Key")] = tag.get("Value") + cluster["tags"] = tags except Exception as e: - logger.warning(f"Could not get tags for cluster {cluster_id}: {str(e)}") - cluster['tags'] = {} - + logger.warning( + f"Could not get tags for cluster {cluster_id}: {str(e)}" + ) + cluster["tags"] = {} + clusters.append(cluster) - + except Exception as e: logger.warning(f"Error processing EMR cluster: {str(e)}") - + logger.info(f"Discovered {len(clusters)} EMR clusters in {region}") return clusters - + except ClientError as e: - error_code = e.response.get('Error', {}).get('Code') - if error_code == 'AccessDeniedException' or error_code == 'UnauthorizedOperation': + error_code = e.response.get("Error", {}).get("Code") + if ( + error_code == "AccessDeniedException" + or error_code == "UnauthorizedOperation" + ): logger.warning(f"Access denied to EMR in region {region}: {str(e)}") - elif error_code == 'EndpointConnectionError' or error_code == 'UnknownEndpoint': + elif ( + error_code == "EndpointConnectionError" + or error_code == "UnknownEndpoint" + ): logger.debug(f"EMR not supported in region {region}") else: logger.error(f"Error discovering EMR clusters in {region}: {str(e)}") @@ -107,93 +139,118 @@ def discover_clusters(self, region: str, cluster_states: List[str] = None) -> Li def discover_all_clusters(self, regions: List[str]) -> List[Dict[str, Any]]: """ Discover all EMR clusters across multiple regions. - + Args: regions: List of AWS regions - + Returns: List of EMR cluster dictionaries """ all_clusters = [] - + # First try RUNNING and WAITING clusters (typical use case) - active_states = ['STARTING', 'BOOTSTRAPPING', 'RUNNING', 'WAITING'] + active_states = ["STARTING", "BOOTSTRAPPING", "RUNNING", "WAITING"] for region in regions: clusters = self.discover_clusters(region, active_states) all_clusters.extend(clusters) - + # Then try TERMINATED and TERMINATED_WITH_ERRORS clusters (historical) # Get STOPPED clusters (requires separate call due to EMR API limitations) for region in regions: try: - emr_client = self.create_client('emr', region) + emr_client = self.create_client("emr", region) logger.debug(f"Looking for STOPPED clusters in {region}") - - paginator = emr_client.get_paginator('list_clusters') - page_iterator = paginator.paginate(ClusterStates=['TERMINATED']) - + + paginator = emr_client.get_paginator("list_clusters") + page_iterator = paginator.paginate(ClusterStates=["TERMINATED"]) + for page in page_iterator: - if 'Clusters' in page: - for cluster_summary in page['Clusters']: + if "Clusters" in page: + for cluster_summary in page["Clusters"]: # Check if this was a STOPPED cluster (vs actually terminated) - cluster_id = cluster_summary.get('Id') + cluster_id = cluster_summary.get("Id") if cluster_id: try: - response = emr_client.describe_cluster(ClusterId=cluster_id) - if 'Cluster' in response and 'Status' in response['Cluster']: - status_reason = response['Cluster']['Status'].get('StateChangeReason', {}) + response = emr_client.describe_cluster( + ClusterId=cluster_id + ) + if ( + "Cluster" in response + and "Status" in response["Cluster"] + ): + status_reason = response["Cluster"][ + "Status" + ].get("StateChangeReason", {}) # EMR uses the TERMINATED state for stopped clusters with a specific code - if status_reason.get('Code') == 'USER_REQUEST' and 'stop clusters' in status_reason.get('Message', '').lower(): + if ( + status_reason.get("Code") == "USER_REQUEST" + and "stop clusters" + in status_reason.get("Message", "").lower() + ): cluster = { - 'id': cluster_id, - 'name': cluster_summary.get('Name', 'Unknown'), - 'status': 'STOPPED', # Override with our interpretation - 'region': region, - 'account_id': self.account_id, - 'account_name': self.account_name + "id": cluster_id, + "name": cluster_summary.get( + "Name", "Unknown" + ), + "status": "STOPPED", # Override with our interpretation + "region": region, + "account_id": self.account_id, + "account_name": self.account_name, } - + # Get tags - if 'Tags' in response['Cluster']: + if "Tags" in response["Cluster"]: tags = {} - for tag in response['Cluster']['Tags']: - tags[tag.get('Key')] = tag.get('Value') - cluster['tags'] = tags + for tag in response["Cluster"]["Tags"]: + tags[tag.get("Key")] = tag.get( + "Value" + ) + cluster["tags"] = tags else: - cluster['tags'] = {} - + cluster["tags"] = {} + all_clusters.append(cluster) - + except Exception as e: - logger.warning(f"Error checking if cluster {cluster_id} is stopped: {str(e)}") - + logger.warning( + f"Error checking if cluster {cluster_id} is stopped: {str(e)}" + ) + except Exception as e: - logger.error(f"Error discovering STOPPED EMR clusters in {region}: {str(e)}") - - logger.info(f"Discovered a total of {len(all_clusters)} EMR clusters across all regions") + logger.error( + f"Error discovering STOPPED EMR clusters in {region}: {str(e)}" + ) + + logger.info( + f"Discovered a total of {len(all_clusters)} EMR clusters across all regions" + ) return all_clusters - + def validate_credentials(self, region: str = None) -> bool: """ Validate if the credentials work for EMR service. - + Args: region: AWS region to test credentials in - + Returns: True if credentials are valid, False otherwise """ if not region: # Use a default region based on the detected partition - if any(cred for cred, val in self.credentials.items() if val and 'gov' in val.lower()): - region = 'us-gov-west-1' + if any( + cred + for cred, val in self.credentials.items() + if val and "gov" in val.lower() + ): + region = "us-gov-west-1" else: - region = 'us-east-1' - + region = "us-east-1" + try: logger.info(f"Validating EMR credentials in region {region}") - emr_client = self.create_client('emr', region) - + emr_client = self.create_client("emr", region) + # Just list a small number of clusters to verify permissions response = emr_client.list_clusters(MaxResults=1) logger.info(f"EMR credentials valid in region {region}") @@ -201,191 +258,226 @@ def validate_credentials(self, region: str = None) -> bool: except Exception as e: logger.warning(f"EMR credentials validation failed in {region}: {str(e)}") return False - - def stop(self, clusters: List[Dict[str, Any]], dry_run: bool = False) -> Dict[str, int]: + + def stop( + self, clusters: List[Dict[str, Any]], dry_run: bool = False + ) -> Dict[str, int]: """ Stop EMR clusters gracefully. This preserves the cluster configuration and data. - + Args: clusters: List of EMR cluster dictionaries dry_run: If True, only simulate the action - + Returns: Dictionary with success and error counts """ success_count = 0 error_count = 0 - + for cluster in clusters: if not cluster: logger.warning("Skipping empty or invalid EMR cluster record") continue - + # Get cluster ID and status, using multiple field names for compatibility - cluster_id = cluster.get('id') + cluster_id = cluster.get("id") if not cluster_id: logger.warning("Skipping EMR cluster with missing ID") continue - + # Handle either 'status' or 'state' field for cluster status - cluster_status = cluster.get('status', cluster.get('state')) + cluster_status = cluster.get("status", cluster.get("state")) if not cluster_status: - logger.warning(f"Cannot determine status for EMR cluster {cluster_id}, assuming UNKNOWN") - cluster_status = 'UNKNOWN' - - cluster_name = cluster.get('name', 'Unknown') - region = cluster.get('region') + logger.warning( + f"Cannot determine status for EMR cluster {cluster_id}, assuming UNKNOWN" + ) + cluster_status = "UNKNOWN" + + cluster_name = cluster.get("name", "Unknown") + region = cluster.get("region") if not region: logger.warning(f"Missing region for EMR cluster {cluster_id}, skipping") error_count += 1 continue - + # Skip clusters with the exclusion tag if self.should_exclude(cluster): - logger.info(f"Skipping EMR cluster {cluster_id} with exclusion tag {config.get('exclusion_tag')}") + logger.info( + f"Skipping EMR cluster {cluster_id} with exclusion tag {config.get('exclusion_tag')}" + ) continue - + # Only RUNNING and WAITING clusters can be stopped if cluster_status in ["RUNNING", "WAITING"]: try: - emr_client = self.create_client('emr', region) - + emr_client = self.create_client("emr", region) + timestamp = self.get_timestamp() - + # Tag the cluster before stopping - logger.info(f"{'[DRY RUN] Would tag' if dry_run else 'Tagging'} EMR cluster {cluster_name} ({cluster_id}) with {config.get('stop_tag')}={timestamp}") + logger.info( + f"{'[DRY RUN] Would tag' if dry_run else 'Tagging'} EMR cluster {cluster_name} ({cluster_id}) with {config.get('stop_tag')}={timestamp}" + ) if not dry_run: emr_client.add_tags( ResourceId=cluster_id, - Tags=[ - { - 'Key': config.get('stop_tag'), - 'Value': timestamp - } - ] + Tags=[{"Key": config.get("stop_tag"), "Value": timestamp}], ) - - logger.info(f"{'[DRY RUN] Would stop' if dry_run else 'Stopping'} EMR cluster {cluster_name} ({cluster_id}) in region {region}") + + logger.info( + f"{'[DRY RUN] Would stop' if dry_run else 'Stopping'} EMR cluster {cluster_name} ({cluster_id}) in region {region}" + ) if not dry_run: emr_client.stop_cluster(ClusterId=cluster_id) - + # Log with detailed information - self.log_action(cluster_id, region, "stop", resource_name=cluster_name, dry_run=dry_run) + self.log_action( + cluster_id, + region, + "stop", + resource_name=cluster_name, + dry_run=dry_run, + ) success_count += 1 - + except Exception as e: - logger.error(f"Failed to {'simulate stopping' if dry_run else 'stop'} EMR cluster {cluster_id}: {e}") + logger.error( + f"Failed to {'simulate stopping' if dry_run else 'stop'} EMR cluster {cluster_id}: {e}" + ) error_count += 1 - + # For STARTING and BOOTSTRAPPING clusters, we need to terminate them as they can't be stopped elif cluster_status in ["STARTING", "BOOTSTRAPPING"]: try: - emr_client = self.create_client('emr', region) - + emr_client = self.create_client("emr", region) + timestamp = self.get_timestamp() - + # Tag the cluster before terminating - logger.info(f"{'[DRY RUN] Would tag' if dry_run else 'Tagging'} EMR cluster {cluster_name} ({cluster_id}) with {config.get('stop_tag')}={timestamp}") + logger.info( + f"{'[DRY RUN] Would tag' if dry_run else 'Tagging'} EMR cluster {cluster_name} ({cluster_id}) with {config.get('stop_tag')}={timestamp}" + ) if not dry_run: emr_client.add_tags( ResourceId=cluster_id, - Tags=[ - { - 'Key': config.get('stop_tag'), - 'Value': timestamp - } - ] + Tags=[{"Key": config.get("stop_tag"), "Value": timestamp}], ) - - logger.info(f"{'[DRY RUN] Would terminate' if dry_run else 'Terminating'} EMR cluster {cluster_name} ({cluster_id}) in region {region} (cannot stop a cluster in {cluster_status} state)") + + logger.info( + f"{'[DRY RUN] Would terminate' if dry_run else 'Terminating'} EMR cluster {cluster_name} ({cluster_id}) in region {region} (cannot stop a cluster in {cluster_status} state)" + ) if not dry_run: emr_client.terminate_job_flows(JobFlowIds=[cluster_id]) - + # Log with detailed information - self.log_action(cluster_id, region, "terminate", resource_name=cluster_name, dry_run=dry_run) + self.log_action( + cluster_id, + region, + "terminate", + resource_name=cluster_name, + dry_run=dry_run, + ) success_count += 1 - + except Exception as e: - logger.error(f"Failed to {'simulate terminating' if dry_run else 'terminate'} EMR cluster {cluster_id}: {e}") + logger.error( + f"Failed to {'simulate terminating' if dry_run else 'terminate'} EMR cluster {cluster_id}: {e}" + ) error_count += 1 else: - logger.info(f"Skipping EMR cluster {cluster_id} in state {cluster_status} (not eligible for stop)") - - return { - "success": success_count, - "errors": error_count - } - - def start(self, clusters: List[Dict[str, Any]], dry_run: bool = False) -> Dict[str, int]: + logger.info( + f"Skipping EMR cluster {cluster_id} in state {cluster_status} (not eligible for stop)" + ) + + return {"success": success_count, "errors": error_count} + + def start( + self, clusters: List[Dict[str, Any]], dry_run: bool = False + ) -> Dict[str, int]: """ Start stopped EMR clusters. - + Args: clusters: List of EMR cluster dictionaries dry_run: If True, only simulate the action - + Returns: Dictionary with success and error counts """ success_count = 0 error_count = 0 - + for cluster in clusters: if not cluster: logger.warning("Skipping empty or invalid EMR cluster record") continue - + # Get cluster ID and status, using multiple field names for compatibility - cluster_id = cluster.get('id') + cluster_id = cluster.get("id") if not cluster_id: logger.warning("Skipping EMR cluster with missing ID") continue - + # Handle either 'status' or 'state' field for cluster status - cluster_status = cluster.get('status', cluster.get('state')) + cluster_status = cluster.get("status", cluster.get("state")) if not cluster_status: - logger.warning(f"Cannot determine status for EMR cluster {cluster_id}, assuming UNKNOWN") - cluster_status = 'UNKNOWN' - - cluster_name = cluster.get('name', 'Unknown') - region = cluster.get('region') + logger.warning( + f"Cannot determine status for EMR cluster {cluster_id}, assuming UNKNOWN" + ) + cluster_status = "UNKNOWN" + + cluster_name = cluster.get("name", "Unknown") + region = cluster.get("region") if not region: logger.warning(f"Missing region for EMR cluster {cluster_id}, skipping") error_count += 1 continue - + # Skip clusters with the exclusion tag if self.should_exclude(cluster): - logger.info(f"Skipping EMR cluster {cluster_id} with exclusion tag {config.get('exclusion_tag')}") + logger.info( + f"Skipping EMR cluster {cluster_id} with exclusion tag {config.get('exclusion_tag')}" + ) continue - + if cluster_status == "STOPPED": try: - emr_client = self.create_client('emr', region) - - logger.info(f"{'[DRY RUN] Would start' if dry_run else 'Starting'} EMR cluster {cluster_name} ({cluster_id}) in region {region}") + emr_client = self.create_client("emr", region) + + logger.info( + f"{'[DRY RUN] Would start' if dry_run else 'Starting'} EMR cluster {cluster_name} ({cluster_id}) in region {region}" + ) if not dry_run: emr_client.start_cluster(ClusterId=cluster_id) - + # Remove the stop tag after successful start - logger.info(f"Removing {config.get('stop_tag')} tag from EMR cluster {cluster_id}") + logger.info( + f"Removing {config.get('stop_tag')} tag from EMR cluster {cluster_id}" + ) emr_client.remove_tags( - ResourceId=cluster_id, - TagKeys=[config.get('stop_tag')] + ResourceId=cluster_id, TagKeys=[config.get("stop_tag")] ) - + # Log with detailed information - self.log_action(cluster_id, region, "start", resource_name=cluster_name, dry_run=dry_run) + self.log_action( + cluster_id, + region, + "start", + resource_name=cluster_name, + dry_run=dry_run, + ) success_count += 1 - + except Exception as e: - logger.error(f"Failed to {'simulate starting' if dry_run else 'start'} EMR cluster {cluster_id}: {e}") + logger.error( + f"Failed to {'simulate starting' if dry_run else 'start'} EMR cluster {cluster_id}: {e}" + ) error_count += 1 else: - logger.info(f"Skipping EMR cluster {cluster_id} in state {cluster_status} (not in STOPPED state)") - - return { - "success": success_count, - "errors": error_count - } + logger.info( + f"Skipping EMR cluster {cluster_id} in state {cluster_status} (not in STOPPED state)" + ) + + return {"success": success_count, "errors": error_count} diff --git a/local-app/python-tools/gfl-resource-actions/aws_resource_management/managers/rds.py b/local-app/python-tools/gfl-resource-actions/aws_resource_management/managers/rds.py index 0cbc9186..89885c0c 100644 --- a/local-app/python-tools/gfl-resource-actions/aws_resource_management/managers/rds.py +++ b/local-app/python-tools/gfl-resource-actions/aws_resource_management/managers/rds.py @@ -1,22 +1,29 @@ """ RDS resource manager class. """ -from typing import Dict, List, Any, Optional -from aws_resource_management.managers.base import ResourceManager +from typing import Any, Dict, List, Optional + from aws_resource_management.config_manager import get_config from aws_resource_management.logging_setup import setup_logging +from aws_resource_management.managers.base import ResourceManager logger = setup_logging() config = get_config() + class RDSManager(ResourceManager): """Manager for RDS instances.""" - - def __init__(self, credentials: Dict[str, str], account_id: str, account_name: Optional[str] = None): + + def __init__( + self, + credentials: Dict[str, str], + account_id: str, + account_name: Optional[str] = None, + ): """ Initialize RDS manager. - + Args: credentials: AWS credentials dictionary account_id: AWS account ID @@ -24,15 +31,17 @@ def __init__(self, credentials: Dict[str, str], account_id: str, account_name: O """ super().__init__(credentials, account_id, account_name) self.resource_type = "rds_instance" - - def stop(self, instances: List[Dict[str, Any]], dry_run: bool = False) -> Dict[str, int]: + + def stop( + self, instances: List[Dict[str, Any]], dry_run: bool = False + ) -> Dict[str, int]: """ Stop available RDS instances. - + Args: instances: List of RDS instance dictionaries dry_run: If True, only simulate the action - + Returns: Dictionary with success and error counts """ @@ -40,106 +49,119 @@ def stop(self, instances: List[Dict[str, Any]], dry_run: bool = False) -> Dict[s stopped_count = 0 skipped_count = 0 error_count = 0 - + for instance in instances: # Skip instances with the exclusion tag if self.should_exclude(instance): - logger.info(f"Skipping RDS instance {instance['id']} with exclusion tag {config.get('exclusion_tag')}") + logger.info( + f"Skipping RDS instance {instance['id']} with exclusion tag {config.get('exclusion_tag')}" + ) skipped_count += 1 continue - + # Log status for debugging logger.debug(f"RDS instance {instance['id']} status: {instance['status']}") - + if instance["status"] == "available": try: region = instance["region"] - rds_client = self.create_client('rds', region) - + rds_client = self.create_client("rds", region) + # Create ISO 8601 timestamp timestamp = self.get_timestamp() - + # Tag the instance before stopping - logger.info(f"{'[DRY RUN] Would tag' if dry_run else 'Tagging'} RDS instance {instance['id']} with {config.get('stop_tag')}={timestamp}") + logger.info( + f"{'[DRY RUN] Would tag' if dry_run else 'Tagging'} RDS instance {instance['id']} with {config.get('stop_tag')}={timestamp}" + ) if not dry_run: rds_client.add_tags_to_resource( ResourceName=f"arn:{self.get_partition_for_region(region)}:rds:{region}:{self.account_id}:db:{instance['id']}", - Tags=[ - { - 'Key': config.get('stop_tag'), - 'Value': timestamp - } - ] + Tags=[{"Key": config.get("stop_tag"), "Value": timestamp}], ) - - logger.info(f"{'[DRY RUN] Would stop' if dry_run else 'Stopping'} RDS instance {instance['id']} in region {region}") + + logger.info( + f"{'[DRY RUN] Would stop' if dry_run else 'Stopping'} RDS instance {instance['id']} in region {region}" + ) if not dry_run: - rds_client.stop_db_instance(DBInstanceIdentifier=instance['id']) - + rds_client.stop_db_instance(DBInstanceIdentifier=instance["id"]) + # Log with detailed information - self.log_action(instance['id'], region, "stop", dry_run=dry_run) - + self.log_action(instance["id"], region, "stop", dry_run=dry_run) + stopped_count += 1 - + except Exception as e: - logger.error(f"Failed to {'simulate stopping' if dry_run else 'stop'} RDS instance {instance['id']}: {e}") + logger.error( + f"Failed to {'simulate stopping' if dry_run else 'stop'} RDS instance {instance['id']}: {e}" + ) error_count += 1 else: - logger.debug(f"Skipping RDS instance {instance['id']} with non-available status: {instance['status']}") + logger.debug( + f"Skipping RDS instance {instance['id']} with non-available status: {instance['status']}" + ) skipped_count += 1 - - logger.info(f"RDS stop summary: {stopped_count} instances processed, {skipped_count} instances skipped") - - return { - "success": stopped_count, - "errors": error_count - } - - def start(self, instances: List[Dict[str, Any]], dry_run: bool = False) -> Dict[str, int]: + + logger.info( + f"RDS stop summary: {stopped_count} instances processed, {skipped_count} instances skipped" + ) + + return {"success": stopped_count, "errors": error_count} + + def start( + self, instances: List[Dict[str, Any]], dry_run: bool = False + ) -> Dict[str, int]: """ Start stopped RDS instances. - + Args: instances: List of RDS instance dictionaries dry_run: If True, only simulate the action - + Returns: Dictionary with success and error counts """ started_count = 0 error_count = 0 - + for instance in instances: # Skip instances with the exclusion tag if self.should_exclude(instance): - logger.info(f"Skipping RDS instance {instance['id']} with exclusion tag {config.get('exclusion_tag')}") + logger.info( + f"Skipping RDS instance {instance['id']} with exclusion tag {config.get('exclusion_tag')}" + ) continue - + if instance["status"] == "stopped": try: region = instance["region"] - rds_client = self.create_client('rds', region) - - logger.info(f"{'[DRY RUN] Would start' if dry_run else 'Starting'} RDS instance {instance['id']} in region {region}") + rds_client = self.create_client("rds", region) + + logger.info( + f"{'[DRY RUN] Would start' if dry_run else 'Starting'} RDS instance {instance['id']} in region {region}" + ) if not dry_run: - rds_client.start_db_instance(DBInstanceIdentifier=instance['id']) - + rds_client.start_db_instance( + DBInstanceIdentifier=instance["id"] + ) + # Remove the stop tag after successful start - logger.info(f"Removing {config.get('stop_tag')} tag from RDS instance {instance['id']}") + logger.info( + f"Removing {config.get('stop_tag')} tag from RDS instance {instance['id']}" + ) rds_client.remove_tags_from_resource( ResourceName=f"arn:{self.get_partition_for_region(region)}:rds:{region}:{self.account_id}:db:{instance['id']}", - TagKeys=[config.get('stop_tag')] + TagKeys=[config.get("stop_tag")], ) - + # Log with detailed information - self.log_action(instance['id'], region, "start", dry_run=dry_run) + self.log_action(instance["id"], region, "start", dry_run=dry_run) started_count += 1 - + except Exception as e: - logger.error(f"Failed to {'simulate starting' if dry_run else 'start'} RDS instance {instance['id']}: {e}") + logger.error( + f"Failed to {'simulate starting' if dry_run else 'start'} RDS instance {instance['id']}: {e}" + ) error_count += 1 - - return { - "success": started_count, - "errors": error_count - } + + return {"success": started_count, "errors": error_count} diff --git a/local-app/python-tools/gfl-resource-actions/aws_resource_management/resource_controller.py b/local-app/python-tools/gfl-resource-actions/aws_resource_management/resource_controller.py index db2d6fc1..82f6a459 100644 --- a/local-app/python-tools/gfl-resource-actions/aws_resource_management/resource_controller.py +++ b/local-app/python-tools/gfl-resource-actions/aws_resource_management/resource_controller.py @@ -5,41 +5,47 @@ """ import warnings + warnings.warn( "The resource_controller module is deprecated. Please use the core module instead.", DeprecationWarning, - stacklevel=2 + stacklevel=2, ) -from typing import Dict, List, Any, Optional, Union import logging import sys +from typing import Any, Dict, List, Optional, Union -from aws_resource_management.managers.ec2 import EC2Manager -from aws_resource_management.managers.rds import RDSManager -from aws_resource_management.managers.emr import EMRManager -from aws_resource_management.managers.eks import EKSManager +from aws_resource_management.aws_utils import ( + get_account_list, + get_credentials, + get_enabled_regions, +) from aws_resource_management.config_manager import get_config -from aws_resource_management.logging_setup import setup_logging -from aws_resource_management.aws_utils import get_credentials, get_account_list, get_enabled_regions from aws_resource_management.core import ResourceManager +from aws_resource_management.logging_setup import setup_logging +from aws_resource_management.managers.ec2 import EC2Manager +from aws_resource_management.managers.eks import EKSManager +from aws_resource_management.managers.emr import EMRManager +from aws_resource_management.managers.rds import RDSManager logger = setup_logging() config = get_config() + class ResourceController: """Controller class that orchestrates resource management across accounts.""" - + def __init__( - self, - profile_name: Optional[str] = None, + self, + profile_name: Optional[str] = None, use_profiles: bool = False, discover_regions: bool = False, - partition: Optional[str] = None + partition: Optional[str] = None, ): """ Initialize the resource controller. - + Args: profile_name: Optional AWS profile name to use for all operations use_profiles: Whether to use AWS profiles from ~/.aws/config @@ -48,14 +54,14 @@ def __init__( """ self.config = get_config() self.resource_manager = ResourceManager( - profile_name=profile_name, + profile_name=profile_name, use_profiles=use_profiles, discover_regions=discover_regions, - partition=partition + partition=partition, ) self.discover_regions = discover_regions self.partition = partition - + def process_resources( self, action: str, @@ -64,11 +70,11 @@ def process_resources( exclude_accounts: List[str] = None, exclude_resources: List[str] = None, exclude_regions: List[str] = None, - dry_run: bool = False + dry_run: bool = False, ) -> Dict[str, Any]: """ Process resources of the specified type across accounts. - + Args: action: Action to perform (stop or start) resource_type: Type of resources to manage @@ -77,24 +83,28 @@ def process_resources( exclude_resources: List of resource types to exclude exclude_regions: List of regions to exclude dry_run: Whether to perform a dry run - + Returns: Statistics dictionary with results """ try: # Validate action - if action not in ['start', 'stop']: + if action not in ["start", "stop"]: logger.error(f"Invalid action: {action}. Must be 'start' or 'stop'") - return {'error': f"Invalid action: {action}"} - + return {"error": f"Invalid action: {action}"} + # Validate resource type - if resource_type not in ['ec2', 'rds', 'eks', 'emr']: + if resource_type not in ["ec2", "rds", "eks", "emr"]: logger.error(f"Invalid resource type: {resource_type}") - return {'error': f"Invalid resource type: {resource_type}"} - - region_msg = "all enabled regions" if self.discover_regions else ", ".join(regions) - logger.info(f"Processing {resource_type} resources in {region_msg} for {action} action") - + return {"error": f"Invalid resource type: {resource_type}"} + + region_msg = ( + "all enabled regions" if self.discover_regions else ", ".join(regions) + ) + logger.info( + f"Processing {resource_type} resources in {region_msg} for {action} action" + ) + # Process accounts through the resource manager stats = self.resource_manager.process_accounts( action=action, @@ -102,9 +112,9 @@ def process_resources( exclude_accounts=exclude_accounts, exclude_resources=exclude_resources, exclude_regions=exclude_regions, - dry_run=dry_run + dry_run=dry_run, ) - + # Log summary logger.info(f"Completed {action} action for {resource_type} resources") logger.info(f"Accounts processed: {stats.get('accounts_processed', 0)}") @@ -112,120 +122,122 @@ def process_resources( logger.info(f"Resources processed: {stats.get('resources_processed', 0)}") logger.info(f"Resources skipped: {stats.get('resources_skipped', 0)}") logger.info(f"Regions processed: {stats.get('regions_processed', 0)}") - - if stats.get('errors'): + + if stats.get("errors"): logger.warning(f"Encountered {len(stats.get('errors', []))} errors") - + return stats - + except KeyboardInterrupt: # Propagate the keyboard interrupt to let the CLI handle it logger.warning("Operation interrupted by user in resource controller") raise - + def list_resources( self, resource_type: str, regions: List[str], exclude_accounts: List[str] = None, - exclude_regions: List[str] = None + exclude_regions: List[str] = None, ) -> Dict[str, Any]: """ List resources of the specified type across accounts. - + Args: resource_type: Type of resources to list regions: List of regions to scan exclude_accounts: List of account IDs to exclude exclude_regions: List of regions to exclude - + Returns: Dictionary containing the discovered resources """ logger.info(f"Listing {resource_type} resources in {', '.join(regions)}") - + # Get account list accounts = get_account_list() if not accounts: logger.warning("No accounts found or error retrieving accounts") - return {'error': 'No accounts found or error retrieving accounts'} - - results = { - 'resources': [], - 'accounts_processed': 0, - 'accounts_skipped': 0 - } - + return {"error": "No accounts found or error retrieving accounts"} + + results = {"resources": [], "accounts_processed": 0, "accounts_skipped": 0} + # Process each account for account in accounts: - account_id = account.get('account_id') - account_name = account.get('account_name') - + account_id = account.get("account_id") + account_name = account.get("account_name") + # Skip excluded accounts if exclude_accounts and account_id in exclude_accounts: logger.info(f"Skipping excluded account {account_id} ({account_name})") - results['accounts_skipped'] += 1 + results["accounts_skipped"] += 1 continue - + try: # Get credentials for the account credentials = get_credentials(account_id) if not credentials: - logger.warning(f"Could not obtain credentials for account {account_id}") - results['accounts_skipped'] += 1 + logger.warning( + f"Could not obtain credentials for account {account_id}" + ) + results["accounts_skipped"] += 1 continue - + # Create the appropriate manager for the resource type manager = self._create_resource_manager( resource_type=resource_type, credentials=credentials, account_id=account_id, - account_name=account_name + account_name=account_name, ) - + if not manager: - logger.error(f"Failed to create resource manager for type {resource_type}") + logger.error( + f"Failed to create resource manager for type {resource_type}" + ) continue - + # Get resources for each region for region in regions: resources = manager.list_resources(region) - results['resources'].extend(resources) - - results['accounts_processed'] += 1 - + results["resources"].extend(resources) + + results["accounts_processed"] += 1 + except Exception as e: - logger.error(f"Error listing resources for account {account_id}: {str(e)}") - + logger.error( + f"Error listing resources for account {account_id}: {str(e)}" + ) + logger.info(f"Found {len(results['resources'])} {resource_type} resources") return results - + def _create_resource_manager( self, resource_type: str, credentials: Dict[str, str], account_id: str, - account_name: Optional[str] = None + account_name: Optional[str] = None, ) -> Optional[Any]: """ Create the appropriate resource manager based on the resource type. - + Args: resource_type: Type of resources to manage credentials: AWS credentials account_id: AWS account ID account_name: AWS account name - + Returns: Resource manager instance or None if invalid type """ - if resource_type == 'ec2': + if resource_type == "ec2": return EC2Manager(credentials, account_id, account_name) - elif resource_type == 'rds': + elif resource_type == "rds": return RDSManager(credentials, account_id, account_name) - elif resource_type == 'eks': + elif resource_type == "eks": return EKSManager(credentials, account_id, account_name) - elif resource_type == 'emr': + elif resource_type == "emr": return EMRManager(credentials, account_id, account_name) else: return None @@ -233,10 +245,10 @@ def _create_resource_manager( def get_regions_for_account(self, account_id: str) -> List[str]: """ Get enabled regions for a specific account. - + Args: account_id: AWS account ID - + Returns: List of enabled region names """ @@ -246,11 +258,11 @@ def get_regions_for_account(self, account_id: str) -> List[str]: if not credentials: logger.warning(f"Could not obtain credentials for account {account_id}") return [] - + # Get enabled regions regions = get_enabled_regions(credentials, partition=self.partition) return regions - + except Exception as e: logger.error(f"Error getting regions for account {account_id}: {str(e)}") return [] diff --git a/local-app/python-tools/gfl-resource-actions/aws_resource_management/utils.py b/local-app/python-tools/gfl-resource-actions/aws_resource_management/utils.py index e2aacf76..4c0fb04b 100644 --- a/local-app/python-tools/gfl-resource-actions/aws_resource_management/utils.py +++ b/local-app/python-tools/gfl-resource-actions/aws_resource_management/utils.py @@ -1,137 +1,159 @@ """ AWS utility functions for resource management. """ -import boto3 + import os import re -from typing import Dict, List, Optional, Any, Union -from botocore.exceptions import ClientError +from typing import Any, Dict, List, Optional, Union +import boto3 from aws_resource_management.config_manager import get_config from aws_resource_management.logging_setup import setup_logging +from botocore.exceptions import ClientError logger = setup_logging() config = get_config() -def create_boto3_client(service: str, credentials: Dict[str, str], region: str) -> boto3.client: + +def create_boto3_client( + service: str, credentials: Dict[str, str], region: str +) -> boto3.client: """ Create a boto3 client for a specific service using the provided credentials. - + Args: service: AWS service name (e.g., 'ec2', 'rds') credentials: AWS credentials dict with access key, secret key, and token region: AWS region name - + Returns: Configured boto3 client """ return boto3.client( service, - aws_access_key_id=credentials['AccessKeyId'], - aws_secret_access_key=credentials['SecretAccessKey'], - aws_session_token=credentials['SessionToken'], - region_name=region + aws_access_key_id=credentials["AccessKeyId"], + aws_secret_access_key=credentials["SecretAccessKey"], + aws_session_token=credentials["SessionToken"], + region_name=region, ) + def get_available_regions() -> List[str]: """ Get a list of all available AWS regions. - + Returns: List of region names """ try: - ec2_client = boto3.client('ec2') + ec2_client = boto3.client("ec2") response = ec2_client.describe_regions() - return [region['RegionName'] for region in response['Regions']] + return [region["RegionName"] for region in response["Regions"]] except Exception as e: logger.error(f"Error getting available regions: {e}") # Return a default list of common regions as fallback - default_region = config.get('default_region') + default_region = config.get("default_region") if default_region: return [default_region] - return ['us-gov-east-1', 'us-gov-west-1'] + return ["us-gov-east-1", "us-gov-west-1"] + def get_partition_for_region(region: str) -> str: """ Determine the AWS partition for a given region. - + Args: region: AWS region name - + Returns: str: AWS partition name ('aws', 'aws-us-gov', 'aws-cn') """ - if region.startswith('us-gov-'): - return 'aws-us-gov' - elif region.startswith('cn-'): - return 'aws-cn' + if region.startswith("us-gov-"): + return "aws-us-gov" + elif region.startswith("cn-"): + return "aws-cn" else: - return 'aws' + return "aws" + -def find_profile_for_account(account_id: str, role_name: Optional[str] = None) -> Optional[str]: +def find_profile_for_account( + account_id: str, role_name: Optional[str] = None +) -> Optional[str]: """ Find an appropriate SSO profile for the given account ID and role name. - + Args: account_id: AWS account ID role_name: Preferred role name (e.g., 'AdministratorAccess', 'inf-admin-t2') - + Returns: Profile name if found, None otherwise """ try: # Try to read profiles from AWS config import configparser + config_file = os.path.expanduser("~/.aws/config") if not os.path.exists(config_file): return None - + aws_config = configparser.ConfigParser() aws_config.read(config_file) - + # Look for profiles matching the account ID matching_profiles = [] for section in aws_config.sections(): # Extract the actual profile name from section (removing 'profile ' prefix if present) profile_name = section - if section.startswith('profile '): + if section.startswith("profile "): profile_name = section[8:] - + # Check if this profile is for the target account ID - if 'sso_account_id' in aws_config[section] and aws_config[section]['sso_account_id'] == account_id: - matching_profiles.append((profile_name, aws_config[section].get('sso_role_name', ''))) - + if ( + "sso_account_id" in aws_config[section] + and aws_config[section]["sso_account_id"] == account_id + ): + matching_profiles.append( + (profile_name, aws_config[section].get("sso_role_name", "")) + ) + # If role_name is specified, try to find a profile with that role if role_name and matching_profiles: for profile, profile_role in matching_profiles: if profile_role.lower() == role_name.lower(): - logger.info(f"Found matching profile {profile} for account {account_id} with role {role_name}") + logger.info( + f"Found matching profile {profile} for account {account_id} with role {role_name}" + ) return profile - + # If we have any matching profiles, return the first one if matching_profiles: # Prefer profiles with admin roles if available - admin_profiles = [p for p, r in matching_profiles if 'admin' in r.lower()] + admin_profiles = [p for p, r in matching_profiles if "admin" in r.lower()] if admin_profiles: - logger.info(f"Using profile {admin_profiles[0]} for account {account_id}") + logger.info( + f"Using profile {admin_profiles[0]} for account {account_id}" + ) return admin_profiles[0] - - logger.info(f"Using profile {matching_profiles[0][0]} for account {account_id}") + + logger.info( + f"Using profile {matching_profiles[0][0]} for account {account_id}" + ) return matching_profiles[0][0] - + return None except Exception as e: logger.warning(f"Error finding profile for account {account_id}: {e}") return None + def create_boto3_session_from_profile(profile_name: str) -> Optional[boto3.Session]: """ Create a boto3 session using the specified profile. - + Args: profile_name: AWS profile name - + Returns: Configured boto3 session or None if failed """ @@ -139,22 +161,25 @@ def create_boto3_session_from_profile(profile_name: str) -> Optional[boto3.Sessi logger.info(f"Creating session using profile: {profile_name}") session = boto3.Session(profile_name=profile_name) # Test if the session works by getting the caller identity - sts = session.client('sts') + sts = session.client("sts") sts.get_caller_identity() return session except Exception as e: logger.warning(f"Failed to create session with profile {profile_name}: {e}") return None -def assume_role(account_id: str, region: Optional[str] = None, role_name: Optional[str] = None) -> Optional[Dict[str, str]]: + +def assume_role( + account_id: str, region: Optional[str] = None, role_name: Optional[str] = None +) -> Optional[Dict[str, str]]: """ Get credentials for the specified account, preferring SSO profiles when available. - + Args: account_id: AWS account ID to access region: AWS region, used to determine partition role_name: Role name to assume (defaults to config.assume_role_name) - + Returns: Credentials dictionary or None if failed """ @@ -168,51 +193,51 @@ def assume_role(account_id: str, region: Optional[str] = None, role_name: Option credentials = session.get_credentials() frozen_credentials = credentials.get_frozen_credentials() return { - 'AccessKeyId': frozen_credentials.access_key, - 'SecretAccessKey': frozen_credentials.secret_key, - 'SessionToken': frozen_credentials.token + "AccessKeyId": frozen_credentials.access_key, + "SecretAccessKey": frozen_credentials.secret_key, + "SessionToken": frozen_credentials.token, } - + # Fall back to assuming role directly if no profile found or profile didn't work # Use default region from config if not provided if region is None: - region = config.get('default_region') + region = config.get("default_region") # Determine the correct partition for the ARN partition = get_partition_for_region(region) - + # Use the specified role name or fall back to the default - actual_role_name = role_name if role_name else config.get('assume_role_name') - + actual_role_name = role_name if role_name else config.get("assume_role_name") + role_arn = f"arn:{partition}:iam::{account_id}:role/{actual_role_name}" - sts_client = boto3.client('sts', region_name=region) - + sts_client = boto3.client("sts", region_name=region) + logger.info(f"Assuming role {role_arn}") response = sts_client.assume_role( - RoleArn=role_arn, - RoleSessionName="ResourceManagementSession" + RoleArn=role_arn, RoleSessionName="ResourceManagementSession" ) - - return response['Credentials'] + + return response["Credentials"] except Exception as e: logger.error(f"Error accessing account {account_id}: {e}") return None + def create_boto3_client_for_account( - account_id: str, - service: str, - region: Optional[str] = None, - role_name: Optional[str] = None + account_id: str, + service: str, + region: Optional[str] = None, + role_name: Optional[str] = None, ) -> Optional[boto3.client]: """ Create a boto3 client for a specific service in the specified account. - + Args: account_id: AWS account ID service: AWS service name (e.g., 'ec2', 'rds') region: AWS region name role_name: Role name to use - + Returns: Configured boto3 client or None if failed """ @@ -225,42 +250,45 @@ def create_boto3_client_for_account( return session.client(service, region_name=region) else: return session.client(service) - + # Fall back to assume_role method credentials = assume_role(account_id, region, role_name) if not credentials: return None - + return create_boto3_client(service, credentials, region) -def get_organization_accounts(exclude_accounts: Optional[List[str]] = None) -> List[Dict[str, str]]: + +def get_organization_accounts( + exclude_accounts: Optional[List[str]] = None, +) -> List[Dict[str, str]]: """ Get a list of accounts in the AWS organization. - + Args: exclude_accounts: List of account IDs to exclude - + Returns: List of account dicts with id and name """ if exclude_accounts is None: exclude_accounts = [] - + try: - org_client = boto3.client('organizations') + org_client = boto3.client("organizations") accounts = [] - + # Get all accounts in the organization - paginator = org_client.get_paginator('list_accounts') + paginator = org_client.get_paginator("list_accounts") for page in paginator.paginate(): - for account in page['Accounts']: + for account in page["Accounts"]: # Skip suspended accounts and accounts in exclude list - if account['Status'] == 'ACTIVE' and account['Id'] not in exclude_accounts: - accounts.append({ - 'id': account['Id'], - 'name': account['Name'] - }) - + if ( + account["Status"] == "ACTIVE" + and account["Id"] not in exclude_accounts + ): + accounts.append({"id": account["Id"], "name": account["Name"]}) + return accounts except Exception as e: logger.error(f"Error getting organization accounts: {e}") diff --git a/local-app/python-tools/gfl-resource-actions/aws_utils.py b/local-app/python-tools/gfl-resource-actions/aws_utils.py index f1b6d899..31bf0db5 100644 --- a/local-app/python-tools/gfl-resource-actions/aws_utils.py +++ b/local-app/python-tools/gfl-resource-actions/aws_utils.py @@ -1,100 +1,104 @@ """ AWS utility functions for resource management. """ + import boto3 import config from logging_utils import setup_logging logger = setup_logging() + def create_boto3_client(service, credentials, region): """ Create a boto3 client for a specific service using the provided credentials. - + Args: service (str): AWS service name (e.g., 'ec2', 'rds') credentials (dict): AWS credentials dict with access key, secret key, and token region (str): AWS region name - + Returns: boto3.client: Configured boto3 client """ return boto3.client( service, - aws_access_key_id=credentials['AccessKeyId'], - aws_secret_access_key=credentials['SecretAccessKey'], - aws_session_token=credentials['SessionToken'], - region_name=region + aws_access_key_id=credentials["AccessKeyId"], + aws_secret_access_key=credentials["SecretAccessKey"], + aws_session_token=credentials["SessionToken"], + region_name=region, ) + def get_available_regions(): """ Get a list of all available AWS regions. - + Returns: list: List of region names """ try: - ec2_client = boto3.client('ec2') + ec2_client = boto3.client("ec2") response = ec2_client.describe_regions() - return [region['RegionName'] for region in response['Regions']] + return [region["RegionName"] for region in response["Regions"]] except Exception as e: logger.error(f"Error getting available regions: {e}") # Return a default list of common regions as fallback - return ['us-east-1', 'us-east-2', 'us-west-1', 'us-west-2'] + return ["us-east-1", "us-east-2", "us-west-1", "us-west-2"] + def assume_role(account_id): """ Assume the OrganizationAccountAccessRole in the specified account. - + Args: account_id (str): AWS account ID to assume role in - + Returns: dict: Credentials dictionary or None if failed """ try: role_arn = f"arn:aws:iam::{account_id}:role/{config.ASSUME_ROLE_NAME}" - sts_client = boto3.client('sts') - + sts_client = boto3.client("sts") + response = sts_client.assume_role( - RoleArn=role_arn, - RoleSessionName="ResourceManagementSession" + RoleArn=role_arn, RoleSessionName="ResourceManagementSession" ) - - return response['Credentials'] + + return response["Credentials"] except Exception as e: logger.error(f"Error assuming role in account {account_id}: {e}") return None + def get_organization_accounts(exclude_accounts=None): """ Get a list of accounts in the AWS organization. - + Args: exclude_accounts (list): List of account IDs to exclude - + Returns: list: List of account dicts with id and name """ if exclude_accounts is None: exclude_accounts = [] - + try: - org_client = boto3.client('organizations') + org_client = boto3.client("organizations") accounts = [] - + # Get all accounts in the organization - paginator = org_client.get_paginator('list_accounts') + paginator = org_client.get_paginator("list_accounts") for page in paginator.paginate(): - for account in page['Accounts']: + for account in page["Accounts"]: # Skip suspended accounts and accounts in exclude list - if account['Status'] == 'ACTIVE' and account['Id'] not in exclude_accounts: - accounts.append({ - 'id': account['Id'], - 'name': account['Name'] - }) - + if ( + account["Status"] == "ACTIVE" + and account["Id"] not in exclude_accounts + ): + accounts.append({"id": account["Id"], "name": account["Name"]}) + return accounts except Exception as e: logger.error(f"Error getting organization accounts: {e}") diff --git a/local-app/python-tools/gfl-resource-actions/config.py b/local-app/python-tools/gfl-resource-actions/config.py index c6d9defe..277c6138 100644 --- a/local-app/python-tools/gfl-resource-actions/config.py +++ b/local-app/python-tools/gfl-resource-actions/config.py @@ -25,4 +25,4 @@ # Log file name LOG_FILE = "aws_resource_management.log" -ACTION_CSV_FILE = 'resource_actions.csv' +ACTION_CSV_FILE = "resource_actions.csv" diff --git a/local-app/python-tools/gfl-resource-actions/discovery.py b/local-app/python-tools/gfl-resource-actions/discovery.py index de336b82..6e01aeb5 100644 --- a/local-app/python-tools/gfl-resource-actions/discovery.py +++ b/local-app/python-tools/gfl-resource-actions/discovery.py @@ -1,95 +1,99 @@ """ Resource discovery functions for AWS resources. """ + import config from aws_utils import create_boto3_client -from logging_utils import setup_logging, log_action_to_csv +from logging_utils import log_action_to_csv, setup_logging logger = setup_logging() + def get_ec2_instances(credentials, region, account_id=None, account_name=None): """Get EC2 instances in a region.""" try: - ec2_client = create_boto3_client('ec2', credentials, region) + ec2_client = create_boto3_client("ec2", credentials, region) instances = [] - + response = ec2_client.describe_instances() - for reservation in response.get('Reservations', []): - for instance in reservation.get('Instances', []): + for reservation in response.get("Reservations", []): + for instance in reservation.get("Instances", []): # Capture tags for exclusion check - tags = {tag['Key']: tag['Value'] for tag in instance.get('Tags', [])} - + tags = {tag["Key"]: tag["Value"] for tag in instance.get("Tags", [])} + # Extract name from tags - name = tags.get('Name', '') - + name = tags.get("Name", "") + instance_data = { - "id": instance['InstanceId'], + "id": instance["InstanceId"], "name": name, - "state": instance['State']['Name'], + "state": instance["State"]["Name"], "region": region, - "private_ip": instance.get('PrivateIpAddress', ''), - "public_ip": instance.get('PublicIpAddress', ''), - "tags": tags + "private_ip": instance.get("PrivateIpAddress", ""), + "public_ip": instance.get("PublicIpAddress", ""), + "tags": tags, } - + instances.append(instance_data) - + # Log the discovery action to CSV if account_id: log_action_to_csv( account_id=account_id, account_name=account_name or "Unknown", resource_type="ec2", - resource_id=instance['InstanceId'], + resource_id=instance["InstanceId"], resource_name=name, action="discover", region=region, status="success", - details=f"State: {instance['State']['Name']}, Private IP: {instance.get('PrivateIpAddress', 'N/A')}" + details=f"State: {instance['State']['Name']}, Private IP: {instance.get('PrivateIpAddress', 'N/A')}", ) - + # Handle pagination - while 'NextToken' in response: - response = ec2_client.describe_instances(NextToken=response['NextToken']) - for reservation in response.get('Reservations', []): - for instance in reservation.get('Instances', []): + while "NextToken" in response: + response = ec2_client.describe_instances(NextToken=response["NextToken"]) + for reservation in response.get("Reservations", []): + for instance in reservation.get("Instances", []): # Capture tags for exclusion check - tags = {tag['Key']: tag['Value'] for tag in instance.get('Tags', [])} - + tags = { + tag["Key"]: tag["Value"] for tag in instance.get("Tags", []) + } + # Extract name from tags - name = tags.get('Name', '') - + name = tags.get("Name", "") + instance_data = { - "id": instance['InstanceId'], + "id": instance["InstanceId"], "name": name, - "state": instance['State']['Name'], + "state": instance["State"]["Name"], "region": region, - "private_ip": instance.get('PrivateIpAddress', ''), - "public_ip": instance.get('PublicIpAddress', ''), - "tags": tags + "private_ip": instance.get("PrivateIpAddress", ""), + "public_ip": instance.get("PublicIpAddress", ""), + "tags": tags, } - + instances.append(instance_data) - + # Log the discovery action to CSV if account_id: log_action_to_csv( account_id=account_id, account_name=account_name or "Unknown", resource_type="ec2", - resource_id=instance['InstanceId'], + resource_id=instance["InstanceId"], resource_name=name, action="discover", region=region, status="success", - details=f"State: {instance['State']['Name']}, Private IP: {instance.get('PrivateIpAddress', 'N/A')}" + details=f"State: {instance['State']['Name']}, Private IP: {instance.get('PrivateIpAddress', 'N/A')}", ) - + return instances except Exception as e: error_msg = f"Error getting EC2 instances in region {region}: {e}" logger.error(error_msg) - + # Log the error to CSV if account_id: log_action_to_csv( @@ -101,142 +105,153 @@ def get_ec2_instances(credentials, region, account_id=None, account_name=None): action="discover", region=region, status="failed", - details=error_msg + details=error_msg, ) - + return [] + def get_rds_instances(credentials, region, account_id=None, account_name=None): """Get RDS instances in a region.""" try: - rds_client = create_boto3_client('rds', credentials, region) + rds_client = create_boto3_client("rds", credentials, region) instances = [] - + logger.info(f"Discovering RDS instances in region {region}...") response = rds_client.describe_db_instances() - + # Log the raw count of instances returned from API - raw_count = len(response.get('DBInstances', [])) + raw_count = len(response.get("DBInstances", [])) logger.info(f"Raw RDS API returned {raw_count} instances in region {region}") - - for instance in response.get('DBInstances', []): + + for instance in response.get("DBInstances", []): # Log the raw status as received from AWS - raw_status = instance['DBInstanceStatus'] - logger.debug(f"RDS instance {instance['DBInstanceIdentifier']} raw status: {raw_status}") - + raw_status = instance["DBInstanceStatus"] + logger.debug( + f"RDS instance {instance['DBInstanceIdentifier']} raw status: {raw_status}" + ) + # Capture tags for exclusion check tags = {} try: tag_list = rds_client.list_tags_for_resource( - ResourceName=instance['DBInstanceArn'] - ).get('TagList', []) - tags = {tag['Key']: tag['Value'] for tag in tag_list} + ResourceName=instance["DBInstanceArn"] + ).get("TagList", []) + tags = {tag["Key"]: tag["Value"] for tag in tag_list} except Exception as tag_e: - logger.warning(f"Error getting tags for RDS instance {instance['DBInstanceIdentifier']}: {tag_e}") - + logger.warning( + f"Error getting tags for RDS instance {instance['DBInstanceIdentifier']}: {tag_e}" + ) + # Get endpoint address endpoint_address = "" - if 'Endpoint' in instance and 'Address' in instance['Endpoint']: - endpoint_address = instance['Endpoint']['Address'] - + if "Endpoint" in instance and "Address" in instance["Endpoint"]: + endpoint_address = instance["Endpoint"]["Address"] + # Standardize status to lowercase for consistent comparison standardized_status = raw_status.lower() - + instance_data = { - "id": instance['DBInstanceIdentifier'], - "name": instance['DBInstanceIdentifier'], # Using ID as name + "id": instance["DBInstanceIdentifier"], + "name": instance["DBInstanceIdentifier"], # Using ID as name "status": standardized_status, # Using standardized status - "raw_status": raw_status, # Keep original status for debugging + "raw_status": raw_status, # Keep original status for debugging "region": region, "endpoint": endpoint_address, - "tags": tags + "tags": tags, } - + instances.append(instance_data) - + # Log the discovery action to CSV if account_id: log_action_to_csv( account_id=account_id, account_name=account_name or "Unknown", resource_type="rds", - resource_id=instance['DBInstanceIdentifier'], - resource_name=instance['DBInstanceIdentifier'], + resource_id=instance["DBInstanceIdentifier"], + resource_name=instance["DBInstanceIdentifier"], action="discover", region=region, status="success", - details=f"Status: {raw_status}, Endpoint: {endpoint_address}" + details=f"Status: {raw_status}, Endpoint: {endpoint_address}", ) - + # Handle pagination - while 'Marker' in response: - response = rds_client.describe_db_instances(Marker=response['Marker']) - for instance in response.get('DBInstances', []): + while "Marker" in response: + response = rds_client.describe_db_instances(Marker=response["Marker"]) + for instance in response.get("DBInstances", []): # Log the raw status as received from AWS - raw_status = instance['DBInstanceStatus'] - logger.debug(f"RDS instance {instance['DBInstanceIdentifier']} raw status: {raw_status}") - + raw_status = instance["DBInstanceStatus"] + logger.debug( + f"RDS instance {instance['DBInstanceIdentifier']} raw status: {raw_status}" + ) + # Capture tags for exclusion check tags = {} try: tag_list = rds_client.list_tags_for_resource( - ResourceName=instance['DBInstanceArn'] - ).get('TagList', []) - tags = {tag['Key']: tag['Value'] for tag in tag_list} + ResourceName=instance["DBInstanceArn"] + ).get("TagList", []) + tags = {tag["Key"]: tag["Value"] for tag in tag_list} except Exception as tag_e: - logger.warning(f"Error getting tags for RDS instance {instance['DBInstanceIdentifier']}: {tag_e}") - + logger.warning( + f"Error getting tags for RDS instance {instance['DBInstanceIdentifier']}: {tag_e}" + ) + # Get endpoint address endpoint_address = "" - if 'Endpoint' in instance and 'Address' in instance['Endpoint']: - endpoint_address = instance['Endpoint']['Address'] - + if "Endpoint" in instance and "Address" in instance["Endpoint"]: + endpoint_address = instance["Endpoint"]["Address"] + # Standardize status to lowercase for consistent comparison standardized_status = raw_status.lower() - + instance_data = { - "id": instance['DBInstanceIdentifier'], - "name": instance['DBInstanceIdentifier'], # Using ID as name + "id": instance["DBInstanceIdentifier"], + "name": instance["DBInstanceIdentifier"], # Using ID as name "status": standardized_status, # Using standardized status - "raw_status": raw_status, # Keep original status for debugging + "raw_status": raw_status, # Keep original status for debugging "region": region, "endpoint": endpoint_address, - "tags": tags + "tags": tags, } - + instances.append(instance_data) - + # Log the discovery action to CSV if account_id: log_action_to_csv( account_id=account_id, account_name=account_name or "Unknown", resource_type="rds", - resource_id=instance['DBInstanceIdentifier'], - resource_name=instance['DBInstanceIdentifier'], + resource_id=instance["DBInstanceIdentifier"], + resource_name=instance["DBInstanceIdentifier"], action="discover", region=region, status="success", - details=f"Status: {raw_status}, Endpoint: {endpoint_address}" + details=f"Status: {raw_status}, Endpoint: {endpoint_address}", ) - + # Log final count of instances found logger.info(f"Found {len(instances)} RDS instances in region {region}") - + # Log status distribution for debugging status_counts = {} for inst in instances: - status = inst['status'] + status = inst["status"] status_counts[status] = status_counts.get(status, 0) + 1 - + if status_counts: - logger.info(f"RDS instance status distribution in {region}: {status_counts}") - + logger.info( + f"RDS instance status distribution in {region}: {status_counts}" + ) + return instances except Exception as e: error_msg = f"Error getting RDS instances in region {region}: {e}" logger.error(error_msg) - + # Log the error to CSV if account_id: log_action_to_csv( @@ -248,75 +263,106 @@ def get_rds_instances(credentials, region, account_id=None, account_name=None): action="discover", region=region, status="failed", - details=error_msg + details=error_msg, ) - + return [] + def get_eks_clusters(credentials, region, account_id=None, account_name=None): """Get EKS clusters in a region.""" try: - eks_client = create_boto3_client('eks', credentials, region) + eks_client = create_boto3_client("eks", credentials, region) clusters = [] - + # List all clusters response = eks_client.list_clusters() - cluster_names = response.get('clusters', []) - + cluster_names = response.get("clusters", []) + logger.info(f"Found {len(cluster_names)} EKS clusters in region {region}") - + # Get detailed info for each cluster for cluster_name in cluster_names: try: - cluster_info = eks_client.describe_cluster(name=cluster_name)['cluster'] - + cluster_info = eks_client.describe_cluster(name=cluster_name)["cluster"] + # Get tags - tags = cluster_info.get('tags', {}) - + tags = cluster_info.get("tags", {}) + # Look for both individual scaling tags and combined format - min_nodes = tags.get('eks-min-nodes', '') - max_nodes = tags.get('eks-max-nodes', '') - desired_nodes = tags.get('eks-desired-nodes', '') - + min_nodes = tags.get("eks-min-nodes", "") + max_nodes = tags.get("eks-max-nodes", "") + desired_nodes = tags.get("eks-desired-nodes", "") + # Parse the combined cluster:size tag if it exists - if 'cluster:size' in tags: - size_tag = tags['cluster:size'] - logger.debug(f"Found cluster:size tag: {size_tag} for cluster {cluster_name}") + if "cluster:size" in tags: + size_tag = tags["cluster:size"] + logger.debug( + f"Found cluster:size tag: {size_tag} for cluster {cluster_name}" + ) try: # Parse min:X-max:Y-desired:Z format - if 'min:' in size_tag and 'max:' in size_tag and 'desired:' in size_tag: + if ( + "min:" in size_tag + and "max:" in size_tag + and "desired:" in size_tag + ): # Extract values using regex or string parsing - min_match = size_tag.split('min:')[1].split('-')[0] if 'min:' in size_tag else None - max_match = size_tag.split('max:')[1].split('-')[0] if 'max:' in size_tag else None - desired_match = size_tag.split('desired:')[1].split('-')[0] if 'desired:' in size_tag else None - - min_nodes = min_match if min_match and not min_nodes else min_nodes - max_nodes = max_match if max_match and not max_nodes else max_nodes - desired_nodes = desired_match if desired_match and not desired_nodes else desired_nodes - - logger.debug(f"Parsed cluster:size: min={min_nodes}, max={max_nodes}, desired={desired_nodes}") + min_match = ( + size_tag.split("min:")[1].split("-")[0] + if "min:" in size_tag + else None + ) + max_match = ( + size_tag.split("max:")[1].split("-")[0] + if "max:" in size_tag + else None + ) + desired_match = ( + size_tag.split("desired:")[1].split("-")[0] + if "desired:" in size_tag + else None + ) + + min_nodes = ( + min_match if min_match and not min_nodes else min_nodes + ) + max_nodes = ( + max_match if max_match and not max_nodes else max_nodes + ) + desired_nodes = ( + desired_match + if desired_match and not desired_nodes + else desired_nodes + ) + + logger.debug( + f"Parsed cluster:size: min={min_nodes}, max={max_nodes}, desired={desired_nodes}" + ) except Exception as parse_error: - logger.warning(f"Error parsing cluster:size tag for cluster {cluster_name}: {parse_error}") - + logger.warning( + f"Error parsing cluster:size tag for cluster {cluster_name}: {parse_error}" + ) + # Check for backup tags that store original values - original_min = tags.get('eks-original-min-nodes', '') - original_max = tags.get('eks-original-max-nodes', '') - original_desired = tags.get('eks-original-desired-nodes', '') - + original_min = tags.get("eks-original-min-nodes", "") + original_max = tags.get("eks-original-max-nodes", "") + original_desired = tags.get("eks-original-desired-nodes", "") + # Check for schedule tag schedule = None for tag_key, tag_value in tags.items(): - if 'schedule' in tag_key.lower(): + if "schedule" in tag_key.lower(): schedule = tag_value break - + cluster_data = { "name": cluster_name, "id": cluster_name, # Using name as ID for consistency - "status": cluster_info.get('status', 'UNKNOWN'), + "status": cluster_info.get("status", "UNKNOWN"), "region": region, - "endpoint": cluster_info.get('endpoint', ''), - "version": cluster_info.get('version', ''), + "endpoint": cluster_info.get("endpoint", ""), + "version": cluster_info.get("version", ""), "min_nodes": min_nodes, "max_nodes": max_nodes, "desired_nodes": desired_nodes, @@ -324,15 +370,15 @@ def get_eks_clusters(credentials, region, account_id=None, account_name=None): "original_max": original_max, "original_desired": original_desired, "schedule": schedule, - "tags": tags + "tags": tags, } - + clusters.append(cluster_data) - + # Log the discovery with scaling info and schedule scale_details = f"Min: {min_nodes or 'N/A'}, Max: {max_nodes or 'N/A'}, Desired: {desired_nodes or 'N/A'}" schedule_info = f", Schedule: {schedule}" if schedule else "" - + if account_id: log_action_to_csv( account_id=account_id, @@ -344,55 +390,59 @@ def get_eks_clusters(credentials, region, account_id=None, account_name=None): region=region, status="success", details=f"Status: {cluster_data['status']}, Version: {cluster_data['version']}, Scaling: {scale_details}{schedule_info}", - existing_schedule=schedule + existing_schedule=schedule, ) - + except Exception as e: - logger.warning(f"Error getting details for EKS cluster {cluster_name} in region {region}: {e}") - + logger.warning( + f"Error getting details for EKS cluster {cluster_name} in region {region}: {e}" + ) + # Handle pagination - while 'nextToken' in response: - response = eks_client.list_clusters(nextToken=response['nextToken']) - cluster_names = response.get('clusters', []) - + while "nextToken" in response: + response = eks_client.list_clusters(nextToken=response["nextToken"]) + cluster_names = response.get("clusters", []) + for cluster_name in cluster_names: try: - cluster_info = eks_client.describe_cluster(name=cluster_name)['cluster'] - + cluster_info = eks_client.describe_cluster(name=cluster_name)[ + "cluster" + ] + # Get tags - tags = cluster_info.get('tags', {}) - + tags = cluster_info.get("tags", {}) + # Extract scaling parameters from tags if they exist - min_nodes = tags.get('eks-min-nodes', '') - max_nodes = tags.get('eks-max-nodes', '') - desired_nodes = tags.get('eks-desired-nodes', '') - + min_nodes = tags.get("eks-min-nodes", "") + max_nodes = tags.get("eks-max-nodes", "") + desired_nodes = tags.get("eks-desired-nodes", "") + # Check for backup tags that store original values - original_min = tags.get('eks-original-min-nodes', '') - original_max = tags.get('eks-original-max-nodes', '') - original_desired = tags.get('eks-original-desired-nodes', '') - + original_min = tags.get("eks-original-min-nodes", "") + original_max = tags.get("eks-original-max-nodes", "") + original_desired = tags.get("eks-original-desired-nodes", "") + cluster_data = { "name": cluster_name, "id": cluster_name, # Using name as ID for consistency - "status": cluster_info.get('status', 'UNKNOWN'), + "status": cluster_info.get("status", "UNKNOWN"), "region": region, - "endpoint": cluster_info.get('endpoint', ''), - "version": cluster_info.get('version', ''), + "endpoint": cluster_info.get("endpoint", ""), + "version": cluster_info.get("version", ""), "min_nodes": min_nodes, "max_nodes": max_nodes, "desired_nodes": desired_nodes, "original_min": original_min, "original_max": original_max, "original_desired": original_desired, - "tags": tags + "tags": tags, } - + clusters.append(cluster_data) - + # Log the discovery with scaling info scale_details = f"Min: {min_nodes or 'N/A'}, Max: {max_nodes or 'N/A'}, Desired: {desired_nodes or 'N/A'}" - + if account_id: log_action_to_csv( account_id=account_id, @@ -403,30 +453,44 @@ def get_eks_clusters(credentials, region, account_id=None, account_name=None): action="discover", region=region, status="success", - details=f"Status: {cluster_data['status']}, Version: {cluster_data['version']}, Scaling: {scale_details}" + details=f"Status: {cluster_data['status']}, Version: {cluster_data['version']}, Scaling: {scale_details}", ) - + except Exception as e: - logger.warning(f"Error getting details for EKS cluster {cluster_name} in region {region}: {e}") - + logger.warning( + f"Error getting details for EKS cluster {cluster_name} in region {region}: {e}" + ) + # Log the summary scaling_stats = {} for cluster in clusters: - has_scaling = any([cluster.get('min_nodes'), cluster.get('max_nodes'), cluster.get('desired_nodes')]) - scaling_stats["with_scaling_tags"] = scaling_stats.get("with_scaling_tags", 0) + (1 if has_scaling else 0) - scaling_stats["without_scaling_tags"] = scaling_stats.get("without_scaling_tags", 0) + (0 if has_scaling else 1) - + has_scaling = any( + [ + cluster.get("min_nodes"), + cluster.get("max_nodes"), + cluster.get("desired_nodes"), + ] + ) + scaling_stats["with_scaling_tags"] = scaling_stats.get( + "with_scaling_tags", 0 + ) + (1 if has_scaling else 0) + scaling_stats["without_scaling_tags"] = scaling_stats.get( + "without_scaling_tags", 0 + ) + (0 if has_scaling else 1) + if scaling_stats: - logger.info(f"EKS clusters in {region}: {len(clusters)} total, " - f"{scaling_stats.get('with_scaling_tags', 0)} with scaling tags, " - f"{scaling_stats.get('without_scaling_tags', 0)} without scaling tags") - + logger.info( + f"EKS clusters in {region}: {len(clusters)} total, " + f"{scaling_stats.get('with_scaling_tags', 0)} with scaling tags, " + f"{scaling_stats.get('without_scaling_tags', 0)} without scaling tags" + ) + return clusters - + except Exception as e: error_msg = f"Error getting EKS clusters in region {region}: {e}" logger.error(error_msg) - + # Log the error to CSV if account_id: log_action_to_csv( @@ -438,119 +502,146 @@ def get_eks_clusters(credentials, region, account_id=None, account_name=None): action="discover", region=region, status="failed", - details=error_msg + details=error_msg, ) - + return [] + def get_emr_clusters(credentials, region, account_id=None, account_name=None): """Get EMR clusters in a region.""" try: - emr_client = create_boto3_client('emr', credentials, region) + emr_client = create_boto3_client("emr", credentials, region) clusters = [] - + # List active clusters (RUNNING, WAITING, STARTING, BOOTSTRAPPING, TERMINATING, TERMINATED_WITH_ERRORS) response = emr_client.list_clusters( - ClusterStates=['RUNNING', 'WAITING', 'STARTING', 'BOOTSTRAPPING', 'TERMINATING', 'TERMINATED_WITH_ERRORS'] + ClusterStates=[ + "RUNNING", + "WAITING", + "STARTING", + "BOOTSTRAPPING", + "TERMINATING", + "TERMINATED_WITH_ERRORS", + ] ) - + # Process the clusters - for cluster in response.get('Clusters', []): + for cluster in response.get("Clusters", []): try: # Get detailed cluster info - detail = emr_client.describe_cluster(ClusterId=cluster['Id']) - cluster_info = detail.get('Cluster', {}) - + detail = emr_client.describe_cluster(ClusterId=cluster["Id"]) + cluster_info = detail.get("Cluster", {}) + # Extract tags tags = {} - if 'Tags' in cluster_info: - for tag in cluster_info['Tags']: - tags[tag.get('Key', '')] = tag.get('Value', '') - + if "Tags" in cluster_info: + for tag in cluster_info["Tags"]: + tags[tag.get("Key", "")] = tag.get("Value", "") + # Build the cluster data structure cluster_data = { - "id": cluster['Id'], - "name": cluster.get('Name', 'Unnamed cluster'), - "status": cluster.get('Status', {}).get('State', 'UNKNOWN'), + "id": cluster["Id"], + "name": cluster.get("Name", "Unnamed cluster"), + "status": cluster.get("Status", {}).get("State", "UNKNOWN"), "region": region, - "type": cluster_info.get('InstanceCollectionType', 'UNKNOWN'), - "creation_time": str(cluster.get('Status', {}).get('Timeline', {}).get('CreationDateTime', '')), - "tags": tags + "type": cluster_info.get("InstanceCollectionType", "UNKNOWN"), + "creation_time": str( + cluster.get("Status", {}) + .get("Timeline", {}) + .get("CreationDateTime", "") + ), + "tags": tags, } - + clusters.append(cluster_data) - + # Log the discovery action to CSV if account_id: log_action_to_csv( account_id=account_id, account_name=account_name or "Unknown", resource_type="emr", - resource_id=cluster['Id'], - resource_name=cluster.get('Name', 'Unnamed cluster'), + resource_id=cluster["Id"], + resource_name=cluster.get("Name", "Unnamed cluster"), action="discover", region=region, status="success", - details=f"Status: {cluster_data['status']}, Type: {cluster_data['type']}" + details=f"Status: {cluster_data['status']}, Type: {cluster_data['type']}", ) except Exception as e: - logger.warning(f"Error getting details for EMR cluster {cluster['Id']} in region {region}: {e}") - + logger.warning( + f"Error getting details for EMR cluster {cluster['Id']} in region {region}: {e}" + ) + # Handle pagination - while 'Marker' in response: + while "Marker" in response: response = emr_client.list_clusters( - ClusterStates=['RUNNING', 'WAITING', 'STARTING', 'BOOTSTRAPPING', 'TERMINATING', 'TERMINATED_WITH_ERRORS'], - Marker=response['Marker'] + ClusterStates=[ + "RUNNING", + "WAITING", + "STARTING", + "BOOTSTRAPPING", + "TERMINATING", + "TERMINATED_WITH_ERRORS", + ], + Marker=response["Marker"], ) - - for cluster in response.get('Clusters', []): + + for cluster in response.get("Clusters", []): try: # Get detailed cluster info - detail = emr_client.describe_cluster(ClusterId=cluster['Id']) - cluster_info = detail.get('Cluster', {}) - + detail = emr_client.describe_cluster(ClusterId=cluster["Id"]) + cluster_info = detail.get("Cluster", {}) + # Extract tags tags = {} - if 'Tags' in cluster_info: - for tag in cluster_info['Tags']: - tags[tag.get('Key', '')] = tag.get('Value', '') - + if "Tags" in cluster_info: + for tag in cluster_info["Tags"]: + tags[tag.get("Key", "")] = tag.get("Value", "") + # Build the cluster data structure cluster_data = { - "id": cluster['Id'], - "name": cluster.get('Name', 'Unnamed cluster'), - "status": cluster.get('Status', {}).get('State', 'UNKNOWN'), + "id": cluster["Id"], + "name": cluster.get("Name", "Unnamed cluster"), + "status": cluster.get("Status", {}).get("State", "UNKNOWN"), "region": region, - "type": cluster_info.get('InstanceCollectionType', 'UNKNOWN'), - "creation_time": str(cluster.get('Status', {}).get('Timeline', {}).get('CreationDateTime', '')), - "tags": tags + "type": cluster_info.get("InstanceCollectionType", "UNKNOWN"), + "creation_time": str( + cluster.get("Status", {}) + .get("Timeline", {}) + .get("CreationDateTime", "") + ), + "tags": tags, } - + clusters.append(cluster_data) - + # Log the discovery action to CSV if account_id: log_action_to_csv( account_id=account_id, account_name=account_name or "Unknown", resource_type="emr", - resource_id=cluster['Id'], - resource_name=cluster.get('Name', 'Unnamed cluster'), + resource_id=cluster["Id"], + resource_name=cluster.get("Name", "Unnamed cluster"), action="discover", region=region, status="success", - details=f"Status: {cluster_data['status']}, Type: {cluster_data['type']}" + details=f"Status: {cluster_data['status']}, Type: {cluster_data['type']}", ) except Exception as e: - logger.warning(f"Error getting details for EMR cluster {cluster['Id']} in region {region}: {e}") - + logger.warning( + f"Error getting details for EMR cluster {cluster['Id']} in region {region}: {e}" + ) + logger.info(f"Found {len(clusters)} EMR clusters in region {region}") return clusters - + except Exception as e: error_msg = f"Error getting EMR clusters in region {region}: {e}" logger.error(error_msg) - + # Log the error to CSV if account_id: log_action_to_csv( @@ -562,38 +653,49 @@ def get_emr_clusters(credentials, region, account_id=None, account_name=None): action="discover", region=region, status="failed", - details=error_msg + details=error_msg, ) - + return [] -def get_account_resources(credentials, regions, exclude_resources=None, account_id=None, account_name=None): + +def get_account_resources( + credentials, regions, exclude_resources=None, account_id=None, account_name=None +): """Get all resources from an account across specified regions.""" if exclude_resources is None: exclude_resources = [] - + resources = { "ec2_instances": [], "rds_instances": [], "eks_clusters": [], - "emr_clusters": [] + "emr_clusters": [], } - + for region in regions: # Get EC2 instances if "ec2" not in exclude_resources: - resources["ec2_instances"].extend(get_ec2_instances(credentials, region, account_id, account_name)) - + resources["ec2_instances"].extend( + get_ec2_instances(credentials, region, account_id, account_name) + ) + # Get RDS instances if "rds" not in exclude_resources: - resources["rds_instances"].extend(get_rds_instances(credentials, region, account_id, account_name)) - + resources["rds_instances"].extend( + get_rds_instances(credentials, region, account_id, account_name) + ) + # Get EKS clusters if "eks" not in exclude_resources: - resources["eks_clusters"].extend(get_eks_clusters(credentials, region, account_id, account_name)) - + resources["eks_clusters"].extend( + get_eks_clusters(credentials, region, account_id, account_name) + ) + # Get EMR clusters if "emr" not in exclude_resources: - resources["emr_clusters"].extend(get_emr_clusters(credentials, region, account_id, account_name)) - + resources["emr_clusters"].extend( + get_emr_clusters(credentials, region, account_id, account_name) + ) + return resources diff --git a/local-app/python-tools/gfl-resource-actions/logging_setup.py b/local-app/python-tools/gfl-resource-actions/logging_setup.py index ec39d52b..f51e7f5f 100644 --- a/local-app/python-tools/gfl-resource-actions/logging_setup.py +++ b/local-app/python-tools/gfl-resource-actions/logging_setup.py @@ -12,44 +12,56 @@ # Store sensitive field patterns to mask in logs SENSITIVE_FIELDS = { - "password", "secret", "token", "key", "passphrase", "credential", - "auth", "jwt", "private_key", "access_key", "secret_key" + "password", + "secret", + "token", + "key", + "passphrase", + "credential", + "auth", + "jwt", + "private_key", + "access_key", + "secret_key", } + class LoggingContext: """Thread-local storage for logging context information.""" + _context = threading.local() - + @classmethod def set(cls, **kwargs) -> None: """Set context values.""" if not hasattr(cls._context, "data"): cls._context.data = {} cls._context.data.update(kwargs) - + @classmethod def get(cls) -> Dict[str, Any]: """Get all context values.""" return getattr(cls._context, "data", {}).copy() - + @classmethod def get_value(cls, key: str, default: Any = None) -> Any: """Get a specific context value.""" return getattr(cls._context, "data", {}).get(key, default) - + @classmethod def clear(cls) -> None: """Clear all context values.""" if hasattr(cls._context, "data"): cls._context.data = {} + class JSONFormatter(logging.Formatter): """Format log records as JSON with additional context.""" - + def __init__(self, sanitize_fields: Optional[Set[str]] = None): super().__init__() self.sanitize_fields = sanitize_fields or SENSITIVE_FIELDS - + def format(self, record: logging.LogRecord) -> str: """Format the record as a JSON string with context.""" # Start with basic log information @@ -63,51 +75,51 @@ def format(self, record: logging.LogRecord) -> str: "line": record.lineno, "process": record.process, "thread": record.thread, - "hostname": os.uname().nodename + "hostname": os.uname().nodename, } - + # Add correlation ID if not already present if not hasattr(record, "correlation_id"): log_data["correlation_id"] = str(uuid.uuid4()) - + # Add exception info if available if record.exc_info: log_data["exception"] = { "type": record.exc_info[0].__name__, "message": str(record.exc_info[1]), - "traceback": self.formatException(record.exc_info) + "traceback": self.formatException(record.exc_info), } - + # Add extra fields from the record if hasattr(record, "extra") and record.extra: for key, value in record.extra.items(): log_data[key] = value - + # Add context from LoggingContext context_data = LoggingContext.get() if context_data: for key, value in context_data.items(): if key not in log_data: # Don't overwrite existing fields log_data[key] = value - + # Sanitize sensitive fields self._sanitize_data(log_data) - + # Convert to JSON return json.dumps(log_data) - + def _sanitize_data(self, data: Dict[str, Any], path: str = "") -> None: """Recursively sanitize sensitive data.""" if isinstance(data, dict): for key, value in list(data.items()): current_path = f"{path}.{key}" if path else key - + # Check if this is a sensitive field is_sensitive = any( - sensitive in current_path.lower() + sensitive in current_path.lower() for sensitive in self.sanitize_fields ) - + if is_sensitive and isinstance(value, (str, int, float, bool)): data[key] = "********" elif isinstance(value, (dict, list)): @@ -118,57 +130,56 @@ def _sanitize_data(self, data: Dict[str, Any], path: str = "") -> None: if isinstance(item, (dict, list)): self._sanitize_data(item, current_path) + class ContextFilter(logging.Filter): """Filter for automatically adding AWS context to log records.""" - + def __init__( - self, + self, account_id: Optional[str] = None, region: Optional[str] = None, - service_name: Optional[str] = None + service_name: Optional[str] = None, ): super().__init__() self.account_id = account_id self.region = region self.service_name = service_name - + def filter(self, record: logging.LogRecord) -> bool: """Add AWS context to the log record.""" # Add basic context if not already present if not hasattr(record, "aws"): record.aws = {} - + # Add AWS account ID if self.account_id and not record.aws.get("account_id"): record.aws["account_id"] = self.account_id - + # Add AWS region if self.region and not record.aws.get("region"): record.aws["region"] = self.region - + # Add service name if self.service_name and not record.aws.get("service_name"): record.aws["service_name"] = self.service_name - + # Add context from LoggingContext if relevant context = LoggingContext.get() for field in ["resource_type", "resource_id", "operation_name"]: if field in context and not record.aws.get(field): record.aws[field] = context[field] - + # Always return True as we're just enriching, not filtering out return True + @contextmanager def log_operation( - logger: logging.Logger, - name: str, - level: int = logging.INFO, - **context + logger: logging.Logger, name: str, level: int = logging.INFO, **context ) -> None: """ Context manager for logging operations with timing and status. - + Args: logger: The logger to use name: Name of the operation @@ -177,50 +188,54 @@ def log_operation( """ # Generate correlation ID for this operation if not provided correlation_id = context.get("correlation_id", str(uuid.uuid4())) - + # Set initial context LoggingContext.set( operation_name=name, operation_start_time=time.time(), correlation_id=correlation_id, - **context + **context, + ) + + logger.log( + level, f"Starting operation: {name}", extra={"correlation_id": correlation_id} ) - - logger.log(level, f"Starting operation: {name}", extra={"correlation_id": correlation_id}) - + try: yield # Calculate duration and update context on successful completion - duration = (time.time() - LoggingContext.get_value("operation_start_time", 0)) * 1000 + duration = ( + time.time() - LoggingContext.get_value("operation_start_time", 0) + ) * 1000 LoggingContext.set(operation_status="success", duration_ms=duration) logger.log( - level, + level, f"Operation {name} completed successfully in {duration:.2f}ms", - extra={"correlation_id": correlation_id, "duration_ms": duration} + extra={"correlation_id": correlation_id, "duration_ms": duration}, ) except Exception as e: # Calculate duration and update context on failure - duration = (time.time() - LoggingContext.get_value("operation_start_time", 0)) * 1000 - LoggingContext.set(operation_status="failed", duration_ms=duration, error=str(e)) + duration = ( + time.time() - LoggingContext.get_value("operation_start_time", 0) + ) * 1000 + LoggingContext.set( + operation_status="failed", duration_ms=duration, error=str(e) + ) logger.error( f"Operation {name} failed after {duration:.2f}ms: {str(e)}", exc_info=True, - extra={"correlation_id": correlation_id, "duration_ms": duration} + extra={"correlation_id": correlation_id, "duration_ms": duration}, ) raise finally: # Clear context to avoid leaking between operations LoggingContext.clear() -def log_with_context( - logger: logging.Logger, - level: int, - msg: str, - **context -) -> None: + +def log_with_context(logger: logging.Logger, level: int, msg: str, **context) -> None: """ Log a message with the current context plus any additional context provided. - + Args: logger: Logger to use level: Log level @@ -230,10 +245,11 @@ def log_with_context( # Merge with existing context merged_context = LoggingContext.get() merged_context.update(context) - + # Log with the merged context logger.log(level, msg, extra={"extra": merged_context}) + def configure_logging( app_name: str, log_level: Union[int, str] = logging.INFO, @@ -242,11 +258,11 @@ def configure_logging( max_log_size: int = 10 * 1024 * 1024, # 10 MB backup_count: int = 5, console_output: bool = True, - aws_context: Optional[Dict[str, str]] = None + aws_context: Optional[Dict[str, str]] = None, ) -> logging.Logger: """ Configure application logging with advanced features. - + Args: app_name: Application name used for the logger log_level: Logging level (name or constant) @@ -256,19 +272,19 @@ def configure_logging( backup_count: Number of backup log files to keep console_output: Whether to output logs to console aws_context: Dict with AWS context (account_id, region, service_name) - + Returns: Configured logger instance """ # Convert string log level to int if needed if isinstance(log_level, str): log_level = getattr(logging, log_level.upper(), logging.INFO) - + # Create logger logger = logging.getLogger(app_name) logger.setLevel(log_level) logger.handlers = [] # Remove existing handlers - + # Create formatters if json_format: json_formatter = JSONFormatter() @@ -276,46 +292,45 @@ def configure_logging( else: # Human-readable format for console console_formatter = logging.Formatter( - '%(asctime)s [%(levelname)s] [%(name)s] [%(correlation_id)s] %(message)s' + "%(asctime)s [%(levelname)s] [%(name)s] [%(correlation_id)s] %(message)s" ) # JSON for file even if console is human-readable json_formatter = JSONFormatter() - + # Add AWS context filter if provided if aws_context: context_filter = ContextFilter( account_id=aws_context.get("account_id"), region=aws_context.get("region"), - service_name=aws_context.get("service_name") + service_name=aws_context.get("service_name"), ) logger.addFilter(context_filter) - + # Add console handler if requested if console_output: console_handler = logging.StreamHandler(sys.stdout) console_handler.setFormatter(console_formatter) logger.addHandler(console_handler) - + # Add file handler if requested if log_file: os.makedirs(os.path.dirname(os.path.abspath(log_file)), exist_ok=True) file_handler = RotatingFileHandler( - log_file, - maxBytes=max_log_size, - backupCount=backup_count + log_file, maxBytes=max_log_size, backupCount=backup_count ) file_handler.setFormatter(json_formatter) logger.addHandler(file_handler) - + return logger + def get_logger(name: str) -> logging.Logger: """ Get a logger with the given name, inheriting configuration from the root logger. - + Args: name: Logger name - + Returns: Logger instance """ diff --git a/local-app/python-tools/gfl-resource-actions/logging_utils.py b/local-app/python-tools/gfl-resource-actions/logging_utils.py index b0ac0083..a6c16659 100644 --- a/local-app/python-tools/gfl-resource-actions/logging_utils.py +++ b/local-app/python-tools/gfl-resource-actions/logging_utils.py @@ -1,109 +1,153 @@ """ Logging utilities for resource management. """ + +import csv +import datetime import logging import os import sys -import csv -import datetime -from config import ACTION_CSV_FILE - from pathlib import Path +from config import ACTION_CSV_FILE + # Configure log formats -DEFAULT_LOG_FORMAT = '%(asctime)s - %(name)s - %(levelname)s - %(message)s' +DEFAULT_LOG_FORMAT = "%(asctime)s - %(name)s - %(levelname)s - %(message)s" DEFAULT_LOG_LEVEL = logging.INFO # Configure CSV logging -CSV_LOG_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'logs') -CSV_FIELDS = ['timestamp', 'account_id', 'account_name', 'resource_type', 'resource_id', 'resource_name', 'action', 'region', 'status', 'details'] +CSV_LOG_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "logs") +CSV_FIELDS = [ + "timestamp", + "account_id", + "account_name", + "resource_type", + "resource_id", + "resource_name", + "action", + "region", + "status", + "details", +] + -def setup_logging(name='resource_management', level=logging.INFO): +def setup_logging(name="resource_management", level=logging.INFO): """Set up and configure logging.""" logger = logging.getLogger(name) - + # Only configure the logger if it hasn't been configured already if not logger.handlers: logger.setLevel(level) - + # Create console handler console_handler = logging.StreamHandler() console_handler.setLevel(level) - + # Create formatter - formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s') + formatter = logging.Formatter( + "%(asctime)s - %(name)s - %(levelname)s - %(message)s" + ) console_handler.setFormatter(formatter) - + # Add handler to logger logger.addHandler(console_handler) - + # Prevent propagation to the root logger to avoid duplicate messages logger.propagate = False - + return logger + def initialize_csv_log(csv_file=None): """ Initialize CSV log file with headers if it doesn't exist. - + Args: csv_file (str, optional): CSV file name (default: resource_actions.csv) - + Returns: str: Path to the CSV log file """ if csv_file is None: csv_file = ACTION_CSV_FILE - + csv_path = os.path.join(CSV_LOG_DIR, csv_file) - + # Create directory if it doesn't exist Path(CSV_LOG_DIR).mkdir(parents=True, exist_ok=True) - + # Check if file exists, if not create with headers file_exists = os.path.isfile(csv_path) - + if not file_exists: - with open(csv_path, 'w', newline='') as f: + with open(csv_path, "w", newline="") as f: writer = csv.writer(f) writer.writerow(CSV_FIELDS) - + return csv_path -def log_action_to_csv(account_id, account_name, resource_type, resource_id, resource_name, action, region, status, details="", dry_run=False, existing_schedule=None): + +def log_action_to_csv( + account_id, + account_name, + resource_type, + resource_id, + resource_name, + action, + region, + status, + details="", + dry_run=False, + existing_schedule=None, +): """Log resource action to CSV file for tracking.""" timestamp = datetime.datetime.now().isoformat() - + # Create directory if it doesn't exist os.makedirs(os.path.dirname(CSV_LOG_DIR), exist_ok=True) - + # Check if file exists to determine if we need to write headers file_exists = os.path.isfile(CSV_LOG_DIR) - - with open(CSV_LOG_DIR, 'a', newline='') as csvfile: - fieldnames = ['timestamp', 'account_id', 'account_name', 'resource_type', 'resource_id', - 'resource_name', 'action', 'region', 'status', 'details', 'dry_run', 'schedule'] + + with open(CSV_LOG_DIR, "a", newline="") as csvfile: + fieldnames = [ + "timestamp", + "account_id", + "account_name", + "resource_type", + "resource_id", + "resource_name", + "action", + "region", + "status", + "details", + "dry_run", + "schedule", + ] writer = csv.DictWriter(csvfile, fieldnames=fieldnames) - + # Write header if file is new if not file_exists: writer.writeheader() - + # Write the log entry - writer.writerow({ - 'timestamp': timestamp, - 'account_id': account_id, - 'account_name': account_name, - 'resource_type': resource_type, - 'resource_id': resource_id, - 'resource_name': resource_name, - 'action': action, - 'region': region, - 'status': status, - 'details': details, - 'dry_run': 'Yes' if dry_run else 'No', - 'schedule': existing_schedule or '' - }) + writer.writerow( + { + "timestamp": timestamp, + "account_id": account_id, + "account_name": account_name, + "resource_type": resource_type, + "resource_id": resource_id, + "resource_name": resource_name, + "action": action, + "region": region, + "status": status, + "details": details, + "dry_run": "Yes" if dry_run else "No", + "schedule": existing_schedule or "", + } + ) + # Create a logger instance for direct import logger = setup_logging() @@ -113,7 +157,8 @@ def log_action_to_csv(account_id, account_name, resource_type, resource_id, reso from logging_setup import LoggingContext, log_operation, log_with_context -logger = logging.getLogger('aws_resource_management') +logger = logging.getLogger("aws_resource_management") + def log_resource_operation( level: int, @@ -122,11 +167,11 @@ def log_resource_operation( resource_id: str, region: str, details: Optional[Dict[str, Any]] = None, - exception: Optional[Exception] = None + exception: Optional[Exception] = None, ) -> None: """ Log resource operations with consistent structure. - + Args: level: Log level operation: Operation being performed (e.g., 'start', 'stop') @@ -141,31 +186,32 @@ def log_resource_operation( "resource_type": resource_type, "resource_id": resource_id, "operation": operation, - "region": region + "region": region, } - + if details: context["details"] = details msg += f": {details}" - + if exception: context["exception"] = str(exception) context["exception_type"] = exception.__class__.__name__ msg += f" (failed: {exception})" - + log_with_context(logger, level, msg, **context) + def initialize_aws_logging( app_name: str, log_level: str = "INFO", account_id: Optional[str] = None, region: Optional[str] = None, log_to_file: bool = False, - log_file_path: Optional[str] = None + log_file_path: Optional[str] = None, ) -> logging.Logger: """ Initialize AWS-aware logging for the application. - + Args: app_name: Application name log_level: Log level name @@ -173,26 +219,29 @@ def initialize_aws_logging( region: AWS default region log_to_file: Whether to log to a file log_file_path: Path to log file (if log_to_file is True) - + Returns: Configured logger instance """ from logging_setup import configure_logging - + # Set up default log file path if needed if log_to_file and not log_file_path: import os - log_dir = os.path.join(os.path.expanduser("~"), ".aws-resource-management", "logs") + + log_dir = os.path.join( + os.path.expanduser("~"), ".aws-resource-management", "logs" + ) os.makedirs(log_dir, exist_ok=True) log_file_path = os.path.join(log_dir, f"{app_name}.log") - + # Configure AWS context aws_context = {} if account_id: aws_context["account_id"] = account_id if region: aws_context["region"] = region - + # Initialize logging return configure_logging( app_name=app_name, @@ -200,9 +249,10 @@ def initialize_aws_logging( json_format=True, log_file=log_file_path if log_to_file else None, console_output=True, - aws_context=aws_context + aws_context=aws_context, ) + # Example usage function to demonstrate the structured logging features def example_usage(): # Initialize logging @@ -211,18 +261,18 @@ def example_usage(): log_level="INFO", account_id="123456789012", region="us-west-2", - log_to_file=True + log_to_file=True, ) - + # Simple logging with context log_with_context( - logger, - logging.INFO, + logger, + logging.INFO, "Starting application", app_version="1.0.0", - environment="development" + environment="development", ) - + # Log resource operation log_resource_operation( logging.INFO, @@ -230,25 +280,30 @@ def example_usage(): "ec2_instance", "i-1234567890abcdef0", "us-west-2", - details={"instance_type": "t2.micro", "target_state": "running"} + details={"instance_type": "t2.micro", "target_state": "running"}, ) - + # Use operation context manager try: - with log_operation(logger, "resize_rds_instance", logging.INFO, - resource_type="rds_instance", - resource_id="mydb-instance-1", - region="us-west-2"): + with log_operation( + logger, + "resize_rds_instance", + logging.INFO, + resource_type="rds_instance", + resource_id="mydb-instance-1", + region="us-west-2", + ): # Simulated operation that takes time import time + time.sleep(1.5) - + # Set additional context during operation LoggingContext.set(instance_class="db.t3.large") - + # Log with the updated context log_with_context(logger, logging.INFO, "Changed instance class") - + # Simulate successful completion pass except Exception as e: diff --git a/local-app/python-tools/gfl-resource-actions/main.py b/local-app/python-tools/gfl-resource-actions/main.py index 37624fd0..3ed280d7 100644 --- a/local-app/python-tools/gfl-resource-actions/main.py +++ b/local-app/python-tools/gfl-resource-actions/main.py @@ -5,14 +5,16 @@ import argparse from concurrent.futures import ThreadPoolExecutor + import config -from logging_utils import setup_logging -from aws_utils import get_available_regions, assume_role, get_organization_accounts +from aws_utils import assume_role, get_available_regions, get_organization_accounts from discovery import get_account_resources -from managers import EC2Manager, RDSManager, EKSManager, EMRManager +from logging_utils import setup_logging +from managers import EC2Manager, EKSManager, EMRManager, RDSManager logger = setup_logging() + def process_region(credentials, region, exclude_resources): """Process resources in a single region.""" try: @@ -23,19 +25,24 @@ def process_region(credentials, region, exclude_resources): "ec2_instances": [], "rds_instances": [], "eks_clusters": [], - "emr_clusters": [] + "emr_clusters": [], } + def main(): # ...existing code until after args parsing... # Process each account for account in accounts: - logger.info(f"Processing account: {account['name']} ({account['id']}) in {len(regions)} regions") - - credentials = assume_role(account['id']) + logger.info( + f"Processing account: {account['name']} ({account['id']}) in {len(regions)} regions" + ) + + credentials = assume_role(account["id"]) if not credentials: - logger.warning(f"Skipping account {account['id']} due to role assumption failure") + logger.warning( + f"Skipping account {account['id']} due to role assumption failure" + ) continue # Process regions in parallel @@ -43,25 +50,30 @@ def main(): "ec2_instances": [], "rds_instances": [], "eks_clusters": [], - "emr_clusters": [] + "emr_clusters": [], } - + with ThreadPoolExecutor(max_workers=4) as executor: future_to_region = { - executor.submit(process_region, credentials, region, exclude_resources): region + executor.submit( + process_region, credentials, region, exclude_resources + ): region for region in regions } - + for future in concurrent.futures.as_completed(future_to_region): region = future_to_region[future] try: region_resources = future.result() for resource_type in all_resources: - all_resources[resource_type].extend(region_resources[resource_type]) + all_resources[resource_type].extend( + region_resources[resource_type] + ) except Exception as e: logger.error(f"Error processing region {region}: {e}") # ...rest of existing code... + if __name__ == "__main__": sys.exit(main()) diff --git a/local-app/python-tools/gfl-resource-actions/managers/__init__.py b/local-app/python-tools/gfl-resource-actions/managers/__init__.py index 72dde61f..535bbb30 100644 --- a/local-app/python-tools/gfl-resource-actions/managers/__init__.py +++ b/local-app/python-tools/gfl-resource-actions/managers/__init__.py @@ -3,6 +3,6 @@ """ from managers.ec2 import EC2Manager -from managers.rds import RDSManager from managers.eks import EKSManager from managers.emr import EMRManager +from managers.rds import RDSManager diff --git a/local-app/python-tools/gfl-resource-actions/managers/base.py b/local-app/python-tools/gfl-resource-actions/managers/base.py index f518ff30..961d1b10 100644 --- a/local-app/python-tools/gfl-resource-actions/managers/base.py +++ b/local-app/python-tools/gfl-resource-actions/managers/base.py @@ -4,27 +4,34 @@ This module contains the ResourceManager base class which handles common operations such as client creation, action logging, and resource filtering. """ + import datetime -from typing import Dict, Optional, Any, Union +from typing import Any, Dict, Optional, Union + import boto3 -from botocore.exceptions import ClientError import config from aws_utils import get_partition_for_region -from logging_utils import setup_logging, log_action_to_csv +from botocore.exceptions import ClientError +from logging_utils import log_action_to_csv, setup_logging logger = setup_logging() class ResourceManager: """Base class for all resource managers providing common AWS resource functionality.""" - - def __init__(self, credentials: Dict[str, str], account_id: str, account_name: Optional[str] = None): + + def __init__( + self, + credentials: Dict[str, str], + account_id: str, + account_name: Optional[str] = None, + ): """ Initialize resource manager with AWS credentials. - + Args: credentials: AWS credentials dictionary containing AccessKeyId, SecretAccessKey, and SessionToken - account_id: AWS account ID + account_id: AWS account ID account_name: AWS account name (optional, defaults to account_id if not provided) """ self.credentials = credentials @@ -36,86 +43,90 @@ def __init__(self, credentials: Dict[str, str], account_id: str, account_name: O def create_client(self, service: str, region: str) -> Optional[boto3.client]: """ Create a boto3 client for the specified service and region. - + Args: service: AWS service name (e.g., 'ec2', 's3') region: AWS region name (e.g., 'us-east-1') - + Returns: Configured boto3 client or None if credentials are not available - + Raises: ClientError: If there's an error creating the client """ if not self.credentials: - logger.warning(f"No credentials available to create {service} client in {region}") + logger.warning( + f"No credentials available to create {service} client in {region}" + ) return None - + # Use client caching to avoid creating redundant clients cache_key = f"{service}:{region}" if cache_key in self._clients: return self._clients[cache_key] - + try: client = boto3.client( service, region_name=region, - aws_access_key_id=self.credentials['AccessKeyId'], - aws_secret_access_key=self.credentials['SecretAccessKey'], - aws_session_token=self.credentials['SessionToken'] + aws_access_key_id=self.credentials["AccessKeyId"], + aws_secret_access_key=self.credentials["SecretAccessKey"], + aws_session_token=self.credentials["SessionToken"], ) self._clients[cache_key] = client return client except ClientError as e: logger.error(f"Failed to create {service} client in {region}: {str(e)}") raise - + def get_partition_for_region(self, region: str) -> str: """ Get AWS partition for the specified region. - + Args: region: AWS region name - + Returns: AWS partition (e.g., 'aws', 'aws-cn', 'aws-us-gov') """ return get_partition_for_region(region) - + def get_timestamp(self) -> str: """ Get ISO 8601 timestamp for tagging and logging purposes. - + Returns: Current UTC timestamp in ISO 8601 format """ return datetime.datetime.utcnow().isoformat() - + def should_exclude(self, resource: Dict[str, Any]) -> bool: """ Check if resource should be excluded based on tags. - + Args: resource: Resource dictionary containing tags - + Returns: True if the resource should be excluded, False otherwise """ tags = resource.get("tags", {}) return config.EXCLUSION_TAG in tags - - def log_action(self, - resource_id: str, - region: str, - action: str, - resource_name: Optional[str] = None, - details: Optional[str] = None, - dry_run: bool = False, - existing_schedule: Optional[str] = None, - status: str = "success") -> None: + + def log_action( + self, + resource_id: str, + region: str, + action: str, + resource_name: Optional[str] = None, + details: Optional[str] = None, + dry_run: bool = False, + existing_schedule: Optional[str] = None, + status: str = "success", + ) -> None: """ Log an action performed on a resource. - + Args: resource_id: Resource identifier region: AWS region @@ -127,24 +138,24 @@ def log_action(self, status: Action status (default: 'success') """ dry_run_prefix = "[DRY RUN] " if dry_run else "" - + # Build a detailed log message resource_desc = f"{resource_id}" if resource_name: resource_desc = f"{resource_name} ({resource_id})" - + account_desc = f"account {self.account_id}" if self.account_name and self.account_name != self.account_id: account_desc = f"{self.account_name} ({self.account_id})" - + # Convert action to past tense for logging - action_desc = f"{action}ed" if not action.endswith('e') else f"{action}d" - + action_desc = f"{action}ed" if not action.endswith("e") else f"{action}d" + message = f"{dry_run_prefix}{self.resource_type.upper()} {resource_desc} {action_desc} in {account_desc} region {region}" - + if details: message += f" - {details}" - + # Add schedule information if available if existing_schedule: message += f" (Schedule: {existing_schedule})" @@ -156,16 +167,17 @@ def log_action(self, account_name=self.account_name, resource_type=self.resource_type, resource_id=resource_id, - resource_name=resource_name or resource_id, # Use ID as name if not provided + resource_name=resource_name + or resource_id, # Use ID as name if not provided action=action, region=region, status=status_value, details=details, dry_run=dry_run, - existing_schedule=existing_schedule + existing_schedule=existing_schedule, ) logger.info(message) - + def cleanup(self) -> None: """ Perform cleanup operations when the manager is no longer needed. diff --git a/local-app/python-tools/gfl-resource-actions/managers/ec2.py b/local-app/python-tools/gfl-resource-actions/managers/ec2.py index f6272e10..a2471e04 100644 --- a/local-app/python-tools/gfl-resource-actions/managers/ec2.py +++ b/local-app/python-tools/gfl-resource-actions/managers/ec2.py @@ -1,107 +1,117 @@ """ EC2 resource manager class. """ -from managers import base + import config from logging_utils import setup_logging +from managers import base logger = setup_logging() + class EC2Manager(base.ResourceManager): """Manager for EC2 instances.""" - + def __init__(self, credentials, account_id, account_name=None): """Initialize EC2 manager.""" super().__init__(credentials, account_id, account_name) self.resource_type = "ec2_instance" - + def should_exclude(self, instance): """Check if EC2 instance should be excluded based on tags.""" tags = instance.get("tags", {}) - + # Check for explicit exclusion tag if config.EXCLUSION_TAG in tags: - logger.info(f"Skipping EC2 instance {instance['id']} with exclusion tag {config.EXCLUSION_TAG}") + logger.info( + f"Skipping EC2 instance {instance['id']} with exclusion tag {config.EXCLUSION_TAG}" + ) return True - + # Skip instances that are part of EKS clusters if config.EKS_TAG in tags: - logger.info(f"Skipping EC2 instance {instance['id']} with tag {config.EKS_TAG}={tags[config.EKS_TAG]} (EKS managed node)") + logger.info( + f"Skipping EC2 instance {instance['id']} with tag {config.EKS_TAG}={tags[config.EKS_TAG]} (EKS managed node)" + ) return True - + # Skip instances that are part of EMR clusters if config.EMR_TAG in tags: - logger.info(f"Skipping EC2 instance {instance['id']} with tag {config.EMR_TAG}={tags[config.EMR_TAG]} (EMR managed node)") + logger.info( + f"Skipping EC2 instance {instance['id']} with tag {config.EMR_TAG}={tags[config.EMR_TAG]} (EMR managed node)" + ) return True - + return False - + def stop(self, instances, dry_run=False): """Stop running EC2 instances.""" for instance in instances: if self.should_exclude(instance): continue - + if instance["state"] == "running": try: region = instance["region"] - ec2_client = self.create_client('ec2', region) - + ec2_client = self.create_client("ec2", region) + # Create ISO 8601 timestamp timestamp = self.get_timestamp() - + # Tag the instance before stopping - logger.info(f"{'[DRY RUN] Would tag' if dry_run else 'Tagging'} EC2 instance {instance['id']} with {config.STOP_TAG}={timestamp}") + logger.info( + f"{'[DRY RUN] Would tag' if dry_run else 'Tagging'} EC2 instance {instance['id']} with {config.STOP_TAG}={timestamp}" + ) if not dry_run: ec2_client.create_tags( - Resources=[instance['id']], - Tags=[ - { - 'Key': config.STOP_TAG, - 'Value': timestamp - } - ] + Resources=[instance["id"]], + Tags=[{"Key": config.STOP_TAG, "Value": timestamp}], ) - + # Stop the instance - logger.info(f"{'[DRY RUN] Would stop' if dry_run else 'Stopping'} EC2 instance {instance['id']} in region {region}") + logger.info( + f"{'[DRY RUN] Would stop' if dry_run else 'Stopping'} EC2 instance {instance['id']} in region {region}" + ) if not dry_run: - ec2_client.stop_instances(InstanceIds=[instance['id']]) - + ec2_client.stop_instances(InstanceIds=[instance["id"]]) + # Log with detailed information - self.log_action(instance['id'], region, "stop", dry_run=dry_run) - + self.log_action(instance["id"], region, "stop", dry_run=dry_run) + except Exception as e: - logger.error(f"Failed to {'simulate stopping' if dry_run else 'stop'} EC2 instance {instance['id']}: {e}") + logger.error( + f"Failed to {'simulate stopping' if dry_run else 'stop'} EC2 instance {instance['id']}: {e}" + ) def start(self, instances, dry_run=False): """Start stopped EC2 instances.""" for instance in instances: if self.should_exclude(instance): continue - + if instance["state"] == "stopped": try: region = instance["region"] - ec2_client = self.create_client('ec2', region) - - logger.info(f"{'[DRY RUN] Would start' if dry_run else 'Starting'} EC2 instance {instance['id']} in region {region}") + ec2_client = self.create_client("ec2", region) + + logger.info( + f"{'[DRY RUN] Would start' if dry_run else 'Starting'} EC2 instance {instance['id']} in region {region}" + ) if not dry_run: - ec2_client.start_instances(InstanceIds=[instance['id']]) - + ec2_client.start_instances(InstanceIds=[instance["id"]]) + # Remove the stop tag after successful start - logger.info(f"Removing {config.STOP_TAG} tag from EC2 instance {instance['id']}") + logger.info( + f"Removing {config.STOP_TAG} tag from EC2 instance {instance['id']}" + ) ec2_client.delete_tags( - Resources=[instance['id']], - Tags=[ - { - 'Key': config.STOP_TAG - } - ] + Resources=[instance["id"]], Tags=[{"Key": config.STOP_TAG}] ) - + # Log with detailed information - self.log_action(instance['id'], region, "start", dry_run=dry_run) - + self.log_action(instance["id"], region, "start", dry_run=dry_run) + except Exception as e: - logger.error(f"Failed to {'simulate starting' if dry_run else 'start'} EC2 instance {instance['id']}: {e}") + logger.error( + f"Failed to {'simulate starting' if dry_run else 'start'} EC2 instance {instance['id']}: {e}" + ) diff --git a/local-app/python-tools/gfl-resource-actions/managers/eks.py b/local-app/python-tools/gfl-resource-actions/managers/eks.py index 7467a81c..25e3843d 100644 --- a/local-app/python-tools/gfl-resource-actions/managers/eks.py +++ b/local-app/python-tools/gfl-resource-actions/managers/eks.py @@ -1,212 +1,260 @@ """ EKS resource manager class. """ -from managers import base + import config from logging_utils import setup_logging +from managers import base logger = setup_logging() # Tag names for EKS scaling -EKS_MIN_NODES_TAG = 'eks-min-nodes' -EKS_MAX_NODES_TAG = 'eks-max-nodes' -EKS_DESIRED_NODES_TAG = 'eks-desired-nodes' -EKS_ORIGINAL_MIN_NODES_TAG = 'eks-original-min-nodes' -EKS_ORIGINAL_MAX_NODES_TAG = 'eks-original-max-nodes' -EKS_ORIGINAL_DESIRED_NODES_TAG = 'eks-original-desired-nodes' -CLUSTER_SIZE_TAG = 'cluster:size' -EKS_ORIGINAL_CLUSTER_SIZE_TAG = 'eks-original-cluster-size' +EKS_MIN_NODES_TAG = "eks-min-nodes" +EKS_MAX_NODES_TAG = "eks-max-nodes" +EKS_DESIRED_NODES_TAG = "eks-desired-nodes" +EKS_ORIGINAL_MIN_NODES_TAG = "eks-original-min-nodes" +EKS_ORIGINAL_MAX_NODES_TAG = "eks-original-max-nodes" +EKS_ORIGINAL_DESIRED_NODES_TAG = "eks-original-desired-nodes" +CLUSTER_SIZE_TAG = "cluster:size" +EKS_ORIGINAL_CLUSTER_SIZE_TAG = "eks-original-cluster-size" + class EKSManager(base.ResourceManager): """Manager for EKS clusters.""" - + def __init__(self, credentials, account_id, account_name=None): """Initialize EKS manager.""" super().__init__(credentials, account_id, account_name) self.resource_type = "eks_cluster" - + # Add stop method that delegates to scale_down def stop(self, clusters, dry_run=False): """ Stop EKS clusters by scaling down their nodegroups. - + Args: clusters: List of EKS cluster dictionaries dry_run: If True, only simulate the action - + Returns: dict: Summary of actions taken """ - logger.info(f"Delegating EKS stop operation to scale_down for {len(clusters)} clusters") + logger.info( + f"Delegating EKS stop operation to scale_down for {len(clusters)} clusters" + ) self.scale_down(clusters, dry_run) - + # Return a result dictionary that matches the expected interface - return { - "success": len(clusters), - "errors": 0 - } - + return {"success": len(clusters), "errors": 0} + # Add start method that delegates to scale_up def start(self, clusters, dry_run=False): """ Start EKS clusters by scaling up their nodegroups. - + Args: clusters: List of EKS cluster dictionaries dry_run: If True, only simulate the action - + Returns: dict: Summary of actions taken """ - logger.info(f"Delegating EKS start operation to scale_up for {len(clusters)} clusters") + logger.info( + f"Delegating EKS start operation to scale_up for {len(clusters)} clusters" + ) self.scale_up(clusters, dry_run) - + # Return a result dictionary that matches the expected interface - return { - "success": len(clusters), - "errors": 0 - } - + return {"success": len(clusters), "errors": 0} + def scale_down(self, clusters, dry_run=False): """Scale down EKS clusters by setting node counts to 0.""" logger.info(f"Evaluating {len(clusters)} EKS clusters for scale down action") scaled_count = 0 skipped_count = 0 - + for cluster in clusters: # Skip clusters with the exclusion tag if self.should_exclude(cluster): - logger.info(f"Skipping EKS cluster {cluster['name']} with exclusion tag {config.EXCLUSION_TAG}") + logger.info( + f"Skipping EKS cluster {cluster['name']} with exclusion tag {config.EXCLUSION_TAG}" + ) skipped_count += 1 continue - + try: region = cluster["region"] - eks_client = self.create_client('eks', region) - + eks_client = self.create_client("eks", region) + # Get current scaling parameters from tags or nodegroups - min_nodes = cluster.get('min_nodes', '') - max_nodes = cluster.get('max_nodes', '') - desired_nodes = cluster.get('desired_nodes', '') - schedule = cluster.get('schedule', '') - + min_nodes = cluster.get("min_nodes", "") + max_nodes = cluster.get("max_nodes", "") + desired_nodes = cluster.get("desired_nodes", "") + schedule = cluster.get("schedule", "") + # Check if we have a combined cluster:size tag - has_combined_size_tag = CLUSTER_SIZE_TAG in cluster.get('tags', {}) - + has_combined_size_tag = CLUSTER_SIZE_TAG in cluster.get("tags", {}) + # Only proceed if we have some scaling values to work with if min_nodes or max_nodes or desired_nodes or has_combined_size_tag: - logger.info(f"{'[DRY RUN] Would scale down' if dry_run else 'Scaling down'} EKS cluster {cluster['name']} in region {region}") - + logger.info( + f"{'[DRY RUN] Would scale down' if dry_run else 'Scaling down'} EKS cluster {cluster['name']} in region {region}" + ) + if not dry_run: # First, backup the current values in special tags tags_to_update = {} - + # Handle combined size tag if it exists if has_combined_size_tag: - original_size_tag = cluster['tags'][CLUSTER_SIZE_TAG] - tags_to_update[EKS_ORIGINAL_CLUSTER_SIZE_TAG] = original_size_tag + original_size_tag = cluster["tags"][CLUSTER_SIZE_TAG] + tags_to_update[EKS_ORIGINAL_CLUSTER_SIZE_TAG] = ( + original_size_tag + ) tags_to_update[CLUSTER_SIZE_TAG] = "min:0-max:0-desired:0" else: # Store original values in backup tags (individual tags) if min_nodes: tags_to_update[EKS_ORIGINAL_MIN_NODES_TAG] = min_nodes - tags_to_update[EKS_MIN_NODES_TAG] = '0' + tags_to_update[EKS_MIN_NODES_TAG] = "0" if max_nodes: tags_to_update[EKS_ORIGINAL_MAX_NODES_TAG] = max_nodes - tags_to_update[EKS_MAX_NODES_TAG] = '0' + tags_to_update[EKS_MAX_NODES_TAG] = "0" if desired_nodes: - tags_to_update[EKS_ORIGINAL_DESIRED_NODES_TAG] = desired_nodes - tags_to_update[EKS_DESIRED_NODES_TAG] = '0' - + tags_to_update[EKS_ORIGINAL_DESIRED_NODES_TAG] = ( + desired_nodes + ) + tags_to_update[EKS_DESIRED_NODES_TAG] = "0" + # Apply the tag updates if tags_to_update: - logger.info(f"Updating scaling tags for EKS cluster {cluster['name']}: {tags_to_update}") + logger.info( + f"Updating scaling tags for EKS cluster {cluster['name']}: {tags_to_update}" + ) eks_client.tag_resource( - resourceArn=cluster.get('arn', f"arn:{self.get_partition_for_region(region)}:eks:{region}:{self.account_id}:cluster/{cluster['name']}"), - tags=tags_to_update + resourceArn=cluster.get( + "arn", + f"arn:{self.get_partition_for_region(region)}:eks:{region}:{self.account_id}:cluster/{cluster['name']}", + ), + tags=tags_to_update, ) - + # Now actually scale down the nodegroups - self._scale_nodegroups(eks_client, cluster['name'], 0, region, dry_run=False) + self._scale_nodegroups( + eks_client, cluster["name"], 0, region, dry_run=False + ) else: # Dry run reporting if has_combined_size_tag: - original_size = cluster['tags'][CLUSTER_SIZE_TAG] - logger.info(f"[DRY RUN] Would backup combined size tag: {original_size} and set to min:0-max:0-desired:0") + original_size = cluster["tags"][CLUSTER_SIZE_TAG] + logger.info( + f"[DRY RUN] Would backup combined size tag: {original_size} and set to min:0-max:0-desired:0" + ) else: - logger.info(f"[DRY RUN] Would backup scaling values: Min={min_nodes}, Max={max_nodes}, Desired={desired_nodes}") - - logger.info(f"[DRY RUN] Would set scaling values to 0 for EKS cluster {cluster['name']}") - self._scale_nodegroups(eks_client, cluster['name'], 0, region, dry_run=True) - + logger.info( + f"[DRY RUN] Would backup scaling values: Min={min_nodes}, Max={max_nodes}, Desired={desired_nodes}" + ) + + logger.info( + f"[DRY RUN] Would set scaling values to 0 for EKS cluster {cluster['name']}" + ) + self._scale_nodegroups( + eks_client, cluster["name"], 0, region, dry_run=True + ) + # Log the action with schedule information schedule_info = f", Schedule: {schedule}" if schedule else "" self.log_action( - cluster['name'], region, "scale_down", - details=f"Min={min_nodes}->{0}, Max={max_nodes}->{0}, Desired={desired_nodes}->{0}{schedule_info}", + cluster["name"], + region, + "scale_down", + details=f"Min={min_nodes}->{0}, Max={max_nodes}->{0}, Desired={desired_nodes}->{0}{schedule_info}", dry_run=dry_run, - existing_schedule=schedule + existing_schedule=schedule, ) scaled_count += 1 else: - logger.info(f"Skipping EKS cluster {cluster['name']}, no scaling tags found") + logger.info( + f"Skipping EKS cluster {cluster['name']}, no scaling tags found" + ) skipped_count += 1 - + except Exception as e: - logger.error(f"Failed to {'simulate scaling down' if dry_run else 'scale down'} EKS cluster {cluster['name']}: {e}") - - logger.info(f"EKS scale down summary: {scaled_count} clusters processed, {skipped_count} clusters skipped") - + logger.error( + f"Failed to {'simulate scaling down' if dry_run else 'scale down'} EKS cluster {cluster['name']}: {e}" + ) + + logger.info( + f"EKS scale down summary: {scaled_count} clusters processed, {skipped_count} clusters skipped" + ) + def scale_up(self, clusters, dry_run=False): """Scale up EKS clusters using original node counts from tags.""" logger.info(f"Evaluating {len(clusters)} EKS clusters for scale up action") scaled_count = 0 skipped_count = 0 - + for cluster in clusters: # Skip clusters with the exclusion tag if self.should_exclude(cluster): - logger.info(f"Skipping EKS cluster {cluster['name']} with exclusion tag {config.EXCLUSION_TAG}") + logger.info( + f"Skipping EKS cluster {cluster['name']} with exclusion tag {config.EXCLUSION_TAG}" + ) skipped_count += 1 continue - + try: region = cluster["region"] - eks_client = self.create_client('eks', region) - + eks_client = self.create_client("eks", region) + # Check for original values in backup tags - original_min = cluster.get('original_min', '') - original_max = cluster.get('original_max', '') - original_desired = cluster.get('original_desired', '') - schedule = cluster.get('schedule', '') - + original_min = cluster.get("original_min", "") + original_max = cluster.get("original_max", "") + original_desired = cluster.get("original_desired", "") + schedule = cluster.get("schedule", "") + # Check for original combined size tag - tags = cluster.get('tags', {}) + tags = cluster.get("tags", {}) has_original_combined_tag = EKS_ORIGINAL_CLUSTER_SIZE_TAG in tags - + # If we have backup values or original combined tag, restore them - if original_min or original_max or original_desired or has_original_combined_tag: - logger.info(f"{'[DRY RUN] Would scale up' if dry_run else 'Scaling up'} EKS cluster {cluster['name']} in region {region}") - + if ( + original_min + or original_max + or original_desired + or has_original_combined_tag + ): + logger.info( + f"{'[DRY RUN] Would scale up' if dry_run else 'Scaling up'} EKS cluster {cluster['name']} in region {region}" + ) + if not dry_run: # Restore the original values from backup tags tags_to_update = {} tags_to_remove = [] - + # Handle original combined size tag if it exists if has_original_combined_tag: original_size_tag = tags[EKS_ORIGINAL_CLUSTER_SIZE_TAG] tags_to_update[CLUSTER_SIZE_TAG] = original_size_tag tags_to_remove.append(EKS_ORIGINAL_CLUSTER_SIZE_TAG) - + # Parse the original desired value from the size tag desired = None - if 'desired:' in original_size_tag: + if "desired:" in original_size_tag: try: - desired_str = original_size_tag.split('desired:')[1].split('-')[0] - desired = int(desired_str) if desired_str.isdigit() else None + desired_str = original_size_tag.split("desired:")[ + 1 + ].split("-")[0] + desired = ( + int(desired_str) + if desired_str.isdigit() + else None + ) except: - logger.warning(f"Could not parse desired value from combined size tag: {original_size_tag}") + logger.warning( + f"Could not parse desired value from combined size tag: {original_size_tag}" + ) else: # Restore original values for individual tags if original_min: @@ -218,139 +266,197 @@ def scale_up(self, clusters, dry_run=False): if original_desired: tags_to_update[EKS_DESIRED_NODES_TAG] = original_desired tags_to_remove.append(EKS_ORIGINAL_DESIRED_NODES_TAG) - + # Parse desired value - desired = int(original_desired) if original_desired and original_desired.isdigit() else None - + desired = ( + int(original_desired) + if original_desired and original_desired.isdigit() + else None + ) + # Apply the tag updates if tags_to_update: - logger.info(f"Restoring original scaling tags for EKS cluster {cluster['name']}: {tags_to_update}") + logger.info( + f"Restoring original scaling tags for EKS cluster {cluster['name']}: {tags_to_update}" + ) eks_client.tag_resource( - resourceArn=cluster.get('arn', f"arn:{self.get_partition_for_region(region)}:eks:{region}:{self.account_id}:cluster/{cluster['name']}"), - tags=tags_to_update + resourceArn=cluster.get( + "arn", + f"arn:{self.get_partition_for_region(region)}:eks:{region}:{self.account_id}:cluster/{cluster['name']}", + ), + tags=tags_to_update, ) - + # Now actually scale up the nodegroups to desired capacity - self._scale_nodegroups(eks_client, cluster['name'], desired, region, dry_run=False) - + self._scale_nodegroups( + eks_client, + cluster["name"], + desired, + region, + dry_run=False, + ) + # Remove the backup tags if tags_to_remove: - logger.info(f"Removing backup scaling tags: {tags_to_remove}") + logger.info( + f"Removing backup scaling tags: {tags_to_remove}" + ) eks_client.untag_resource( - resourceArn=cluster.get('arn', f"arn:{self.get_partition_for_region(region)}:eks:{region}:{self.account_id}:cluster/{cluster['name']}"), - tagKeys=tags_to_remove + resourceArn=cluster.get( + "arn", + f"arn:{self.get_partition_for_region(region)}:eks:{region}:{self.account_id}:cluster/{cluster['name']}", + ), + tagKeys=tags_to_remove, ) else: # Dry run reporting if has_original_combined_tag: original_size = tags[EKS_ORIGINAL_CLUSTER_SIZE_TAG] - logger.info(f"[DRY RUN] Would restore combined size tag: {original_size}") - + logger.info( + f"[DRY RUN] Would restore combined size tag: {original_size}" + ) + # Try to parse desired value for nodegroup scaling desired = None - if 'desired:' in original_size: + if "desired:" in original_size: try: - desired_str = original_size.split('desired:')[1].split('-')[0] - desired = int(desired_str) if desired_str.isdigit() else None + desired_str = original_size.split("desired:")[ + 1 + ].split("-")[0] + desired = ( + int(desired_str) + if desired_str.isdigit() + else None + ) except: pass else: - logger.info(f"[DRY RUN] Would restore scaling values: Min={original_min}, Max={original_max}, Desired={original_desired}") - desired = int(original_desired) if original_desired and original_desired.isdigit() else None - - self._scale_nodegroups(eks_client, cluster['name'], desired, region, dry_run=True) - + logger.info( + f"[DRY RUN] Would restore scaling values: Min={original_min}, Max={original_max}, Desired={original_desired}" + ) + desired = ( + int(original_desired) + if original_desired and original_desired.isdigit() + else None + ) + + self._scale_nodegroups( + eks_client, cluster["name"], desired, region, dry_run=True + ) + # Log the action with schedule information scale_details = "" if has_original_combined_tag: - original_size = tags.get(EKS_ORIGINAL_CLUSTER_SIZE_TAG, "Unknown") + original_size = tags.get( + EKS_ORIGINAL_CLUSTER_SIZE_TAG, "Unknown" + ) scale_details = f"Size tag: 0->'{original_size}'" else: scale_details = f"Min=0->{original_min}, Max=0->{original_max}, Desired=0->{original_desired}" - + schedule_info = f", Schedule: {schedule}" if schedule else "" self.log_action( - cluster['name'], region, "scale_up", - details=f"{scale_details}{schedule_info}", + cluster["name"], + region, + "scale_up", + details=f"{scale_details}{schedule_info}", dry_run=dry_run, - existing_schedule=schedule + existing_schedule=schedule, ) scaled_count += 1 else: - logger.info(f"Skipping EKS cluster {cluster['name']}, no original scaling values found in tags") + logger.info( + f"Skipping EKS cluster {cluster['name']}, no original scaling values found in tags" + ) skipped_count += 1 - + except Exception as e: - logger.error(f"Failed to {'simulate scaling up' if dry_run else 'scale up'} EKS cluster {cluster['name']}: {e}") - - logger.info(f"EKS scale up summary: {scaled_count} clusters processed, {skipped_count} clusters skipped") - - def _scale_nodegroups(self, eks_client, cluster_name, desired_capacity, region, dry_run=False): + logger.error( + f"Failed to {'simulate scaling up' if dry_run else 'scale up'} EKS cluster {cluster['name']}: {e}" + ) + + logger.info( + f"EKS scale up summary: {scaled_count} clusters processed, {skipped_count} clusters skipped" + ) + + def _scale_nodegroups( + self, eks_client, cluster_name, desired_capacity, region, dry_run=False + ): """Helper method to scale nodegroups to the specified capacity.""" try: # List all nodegroups for this cluster response = eks_client.list_nodegroups(clusterName=cluster_name) - nodegroup_names = response.get('nodegroups', []) - + nodegroup_names = response.get("nodegroups", []) + # Handle pagination - while 'nextToken' in response: + while "nextToken" in response: response = eks_client.list_nodegroups( - clusterName=cluster_name, - nextToken=response['nextToken'] + clusterName=cluster_name, nextToken=response["nextToken"] ) - nodegroup_names.extend(response.get('nodegroups', [])) - + nodegroup_names.extend(response.get("nodegroups", [])) + if not nodegroup_names: logger.info(f"No nodegroups found for EKS cluster {cluster_name}") return - + for nodegroup_name in nodegroup_names: try: # Get nodegroup details nodegroup = eks_client.describe_nodegroup( - clusterName=cluster_name, - nodegroupName=nodegroup_name - ).get('nodegroup', {}) - + clusterName=cluster_name, nodegroupName=nodegroup_name + ).get("nodegroup", {}) + # Get current scaling configuration - current_min = nodegroup.get('scalingConfig', {}).get('minSize') - current_max = nodegroup.get('scalingConfig', {}).get('maxSize') - current_desired = nodegroup.get('scalingConfig', {}).get('desiredSize') - + current_min = nodegroup.get("scalingConfig", {}).get("minSize") + current_max = nodegroup.get("scalingConfig", {}).get("maxSize") + current_desired = nodegroup.get("scalingConfig", {}).get( + "desiredSize" + ) + # Use current min and max when scaling up if desired capacity is provided if desired_capacity is not None: # When scaling up, we need a valid min and max - new_min = current_min if desired_capacity == 0 else min(current_min, desired_capacity) + new_min = ( + current_min + if desired_capacity == 0 + else min(current_min, desired_capacity) + ) new_max = max(current_max, desired_capacity) else: # When scaling down to zero new_min = 0 new_max = current_max desired_capacity = 0 - + if dry_run: - logger.info(f"[DRY RUN] Would update nodegroup {nodegroup_name} scaling: " + - f"Min: {current_min}->{new_min}, " + - f"Max: {current_max}->{new_max}, " + - f"Desired: {current_desired}->{desired_capacity}") + logger.info( + f"[DRY RUN] Would update nodegroup {nodegroup_name} scaling: " + + f"Min: {current_min}->{new_min}, " + + f"Max: {current_max}->{new_max}, " + + f"Desired: {current_desired}->{desired_capacity}" + ) else: - logger.info(f"Updating nodegroup {nodegroup_name} scaling: " + - f"Min: {current_min}->{new_min}, " + - f"Max: {current_max}->{new_max}, " + - f"Desired: {current_desired}->{desired_capacity}") - + logger.info( + f"Updating nodegroup {nodegroup_name} scaling: " + + f"Min: {current_min}->{new_min}, " + + f"Max: {current_max}->{new_max}, " + + f"Desired: {current_desired}->{desired_capacity}" + ) + # Update the nodegroup scaling configuration eks_client.update_nodegroup_config( clusterName=cluster_name, nodegroupName=nodegroup_name, scalingConfig={ - 'minSize': new_min, - 'maxSize': new_max, - 'desiredSize': desired_capacity - } + "minSize": new_min, + "maxSize": new_max, + "desiredSize": desired_capacity, + }, ) except Exception as e: logger.error(f"Error updating nodegroup {nodegroup_name}: {e}") - + except Exception as e: - logger.error(f"Error listing nodegroups for cluster {cluster_name} in region {region}: {e}") + logger.error( + f"Error listing nodegroups for cluster {cluster_name} in region {region}: {e}" + ) diff --git a/local-app/python-tools/gfl-resource-actions/managers/emr.py b/local-app/python-tools/gfl-resource-actions/managers/emr.py index 9e58df24..90bd565e 100644 --- a/local-app/python-tools/gfl-resource-actions/managers/emr.py +++ b/local-app/python-tools/gfl-resource-actions/managers/emr.py @@ -1,20 +1,22 @@ """ EMR resource manager class. """ -from managers import base + import config from logging_utils import setup_logging +from managers import base logger = setup_logging() + class EMRManager(base.ResourceManager): """Manager for EMR clusters.""" - + def __init__(self, credentials, account_id, account_name=None): """Initialize EMR manager.""" super().__init__(credentials, account_id, account_name) self.resource_type = "emr_cluster" - + def stop(self, clusters, dry_run=False): """ Stop EMR clusters gracefully. @@ -23,98 +25,127 @@ def stop(self, clusters, dry_run=False): for cluster in clusters: # Skip clusters with the exclusion tag if self.should_exclude(cluster): - logger.info(f"Skipping EMR cluster {cluster['id']} with exclusion tag {config.EXCLUSION_TAG}") + logger.info( + f"Skipping EMR cluster {cluster['id']} with exclusion tag {config.EXCLUSION_TAG}" + ) continue - + # Only RUNNING and WAITING clusters can be stopped if cluster["status"] in ["RUNNING", "WAITING"]: try: region = cluster["region"] - emr_client = self.create_client('emr', region) - + emr_client = self.create_client("emr", region) + timestamp = self.get_timestamp() - + # Tag the cluster before stopping - logger.info(f"{'[DRY RUN] Would tag' if dry_run else 'Tagging'} EMR cluster {cluster['name']} ({cluster['id']}) with {config.STOP_TAG}={timestamp}") + logger.info( + f"{'[DRY RUN] Would tag' if dry_run else 'Tagging'} EMR cluster {cluster['name']} ({cluster['id']}) with {config.STOP_TAG}={timestamp}" + ) if not dry_run: emr_client.add_tags( - ResourceId=cluster['id'], - Tags=[ - { - 'Key': config.STOP_TAG, - 'Value': timestamp - } - ] + ResourceId=cluster["id"], + Tags=[{"Key": config.STOP_TAG, "Value": timestamp}], ) - - logger.info(f"{'[DRY RUN] Would stop' if dry_run else 'Stopping'} EMR cluster {cluster['name']} ({cluster['id']}) in region {region}") + + logger.info( + f"{'[DRY RUN] Would stop' if dry_run else 'Stopping'} EMR cluster {cluster['name']} ({cluster['id']}) in region {region}" + ) if not dry_run: - emr_client.stop_cluster(ClusterId=cluster['id']) - + emr_client.stop_cluster(ClusterId=cluster["id"]) + # Log with detailed information - self.log_action(cluster['id'], region, "stop", resource_name=cluster['name'], dry_run=dry_run) - + self.log_action( + cluster["id"], + region, + "stop", + resource_name=cluster["name"], + dry_run=dry_run, + ) + except Exception as e: - logger.error(f"Failed to {'simulate stopping' if dry_run else 'stop'} EMR cluster {cluster['id']}: {e}") + logger.error( + f"Failed to {'simulate stopping' if dry_run else 'stop'} EMR cluster {cluster['id']}: {e}" + ) # For STARTING and BOOTSTRAPPING clusters, we need to terminate them as they can't be stopped elif cluster["status"] in ["STARTING", "BOOTSTRAPPING"]: try: region = cluster["region"] - emr_client = self.create_client('emr', region) - + emr_client = self.create_client("emr", region) + timestamp = self.get_timestamp() - + # Tag the cluster before terminating - logger.info(f"{'[DRY RUN] Would tag' if dry_run else 'Tagging'} EMR cluster {cluster['name']} ({cluster['id']}) with {config.STOP_TAG}={timestamp}") + logger.info( + f"{'[DRY RUN] Would tag' if dry_run else 'Tagging'} EMR cluster {cluster['name']} ({cluster['id']}) with {config.STOP_TAG}={timestamp}" + ) if not dry_run: emr_client.add_tags( - ResourceId=cluster['id'], - Tags=[ - { - 'Key': config.STOP_TAG, - 'Value': timestamp - } - ] + ResourceId=cluster["id"], + Tags=[{"Key": config.STOP_TAG, "Value": timestamp}], ) - - logger.info(f"{'[DRY RUN] Would terminate' if dry_run else 'Terminating'} EMR cluster {cluster['name']} ({cluster['id']}) in region {region} (cannot stop a cluster in {cluster['status']} state)") + + logger.info( + f"{'[DRY RUN] Would terminate' if dry_run else 'Terminating'} EMR cluster {cluster['name']} ({cluster['id']}) in region {region} (cannot stop a cluster in {cluster['status']} state)" + ) if not dry_run: - emr_client.terminate_job_flows(JobFlowIds=[cluster['id']]) - + emr_client.terminate_job_flows(JobFlowIds=[cluster["id"]]) + # Log with detailed information - self.log_action(cluster['id'], region, "terminate", resource_name=cluster['name'], dry_run=dry_run) - + self.log_action( + cluster["id"], + region, + "terminate", + resource_name=cluster["name"], + dry_run=dry_run, + ) + except Exception as e: - logger.error(f"Failed to {'simulate terminating' if dry_run else 'terminate'} EMR cluster {cluster['id']}: {e}") + logger.error( + f"Failed to {'simulate terminating' if dry_run else 'terminate'} EMR cluster {cluster['id']}: {e}" + ) def start(self, clusters, dry_run=False): """Start stopped EMR clusters.""" for cluster in clusters: # Skip clusters with the exclusion tag if self.should_exclude(cluster): - logger.info(f"Skipping EMR cluster {cluster['id']} with exclusion tag {config.EXCLUSION_TAG}") + logger.info( + f"Skipping EMR cluster {cluster['id']} with exclusion tag {config.EXCLUSION_TAG}" + ) continue - + if cluster["status"] == "STOPPED": try: region = cluster["region"] - emr_client = self.create_client('emr', region) - + emr_client = self.create_client("emr", region) + timestamp = self.get_timestamp() - - logger.info(f"{'[DRY RUN] Would start' if dry_run else 'Starting'} EMR cluster {cluster['name']} ({cluster['id']}) in region {region}") + + logger.info( + f"{'[DRY RUN] Would start' if dry_run else 'Starting'} EMR cluster {cluster['name']} ({cluster['id']}) in region {region}" + ) if not dry_run: - emr_client.start_cluster(ClusterId=cluster['id']) - + emr_client.start_cluster(ClusterId=cluster["id"]) + # Remove the stop tag after successful start - logger.info(f"Removing {config.STOP_TAG} tag from EMR cluster {cluster['id']}") + logger.info( + f"Removing {config.STOP_TAG} tag from EMR cluster {cluster['id']}" + ) emr_client.remove_tags( - ResourceId=cluster['id'], - TagKeys=[config.STOP_TAG] + ResourceId=cluster["id"], TagKeys=[config.STOP_TAG] ) - + # Log with detailed information - self.log_action(cluster['id'], region, "start", resource_name=cluster['name'], dry_run=dry_run) - + self.log_action( + cluster["id"], + region, + "start", + resource_name=cluster["name"], + dry_run=dry_run, + ) + except Exception as e: - logger.error(f"Failed to {'simulate starting' if dry_run else 'start'} EMR cluster {cluster['id']}: {e}") + logger.error( + f"Failed to {'simulate starting' if dry_run else 'start'} EMR cluster {cluster['id']}: {e}" + ) diff --git a/local-app/python-tools/gfl-resource-actions/managers/rds.py b/local-app/python-tools/gfl-resource-actions/managers/rds.py index 18631761..37142699 100644 --- a/local-app/python-tools/gfl-resource-actions/managers/rds.py +++ b/local-app/python-tools/gfl-resource-actions/managers/rds.py @@ -1,100 +1,119 @@ """ RDS resource manager class. """ -from managers import base + import config from logging_utils import setup_logging +from managers import base logger = setup_logging() + class RDSManager(base.ResourceManager): """Manager for RDS instances.""" - + def __init__(self, credentials, account_id, account_name=None): """Initialize RDS manager.""" super().__init__(credentials, account_id, account_name) self.resource_type = "rds_instance" - + def stop(self, instances, dry_run=False): """Stop available RDS instances.""" logger.info(f"Evaluating {len(instances)} RDS instances for stop action") stopped_count = 0 skipped_count = 0 - + for instance in instances: # Skip instances with the exclusion tag if self.should_exclude(instance): - logger.info(f"Skipping RDS instance {instance['id']} with exclusion tag {config.EXCLUSION_TAG}") + logger.info( + f"Skipping RDS instance {instance['id']} with exclusion tag {config.EXCLUSION_TAG}" + ) skipped_count += 1 continue - + # Log status for debugging logger.debug(f"RDS instance {instance['id']} status: {instance['status']}") - + if instance["status"] == "available": try: region = instance["region"] - rds_client = self.create_client('rds', region) - + rds_client = self.create_client("rds", region) + # Create ISO 8601 timestamp timestamp = self.get_timestamp() - + # Tag the instance before stopping - logger.info(f"{'[DRY RUN] Would tag' if dry_run else 'Tagging'} RDS instance {instance['id']} with {config.STOP_TAG}={timestamp}") + logger.info( + f"{'[DRY RUN] Would tag' if dry_run else 'Tagging'} RDS instance {instance['id']} with {config.STOP_TAG}={timestamp}" + ) if not dry_run: rds_client.add_tags_to_resource( ResourceName=f"arn:{self.get_partition_for_region(region)}:rds:{region}:{self.account_id}:db:{instance['id']}", - Tags=[ - { - 'Key': config.STOP_TAG, - 'Value': timestamp - } - ] + Tags=[{"Key": config.STOP_TAG, "Value": timestamp}], ) - - logger.info(f"{'[DRY RUN] Would stop' if dry_run else 'Stopping'} RDS instance {instance['id']} in region {region}") + + logger.info( + f"{'[DRY RUN] Would stop' if dry_run else 'Stopping'} RDS instance {instance['id']} in region {region}" + ) if not dry_run: - rds_client.stop_db_instance(DBInstanceIdentifier=instance['id']) - + rds_client.stop_db_instance(DBInstanceIdentifier=instance["id"]) + # Log with detailed information - self.log_action(instance['id'], region, "stop", dry_run=dry_run) - + self.log_action(instance["id"], region, "stop", dry_run=dry_run) + stopped_count += 1 - + except Exception as e: - logger.error(f"Failed to {'simulate stopping' if dry_run else 'stop'} RDS instance {instance['id']}: {e}") + logger.error( + f"Failed to {'simulate stopping' if dry_run else 'stop'} RDS instance {instance['id']}: {e}" + ) else: - logger.debug(f"Skipping RDS instance {instance['id']} with non-available status: {instance['status']}") + logger.debug( + f"Skipping RDS instance {instance['id']} with non-available status: {instance['status']}" + ) skipped_count += 1 - - logger.info(f"RDS stop summary: {stopped_count} instances processed, {skipped_count} instances skipped") + + logger.info( + f"RDS stop summary: {stopped_count} instances processed, {skipped_count} instances skipped" + ) def start(self, instances, dry_run=False): """Start stopped RDS instances.""" for instance in instances: # Skip instances with the exclusion tag if self.should_exclude(instance): - logger.info(f"Skipping RDS instance {instance['id']} with exclusion tag {config.EXCLUSION_TAG}") + logger.info( + f"Skipping RDS instance {instance['id']} with exclusion tag {config.EXCLUSION_TAG}" + ) continue - + if instance["status"] == "stopped": try: region = instance["region"] - rds_client = self.create_client('rds', region) - - logger.info(f"{'[DRY RUN] Would start' if dry_run else 'Starting'} RDS instance {instance['id']} in region {region}") + rds_client = self.create_client("rds", region) + + logger.info( + f"{'[DRY RUN] Would start' if dry_run else 'Starting'} RDS instance {instance['id']} in region {region}" + ) if not dry_run: - rds_client.start_db_instance(DBInstanceIdentifier=instance['id']) - + rds_client.start_db_instance( + DBInstanceIdentifier=instance["id"] + ) + # Remove the stop tag after successful start - logger.info(f"Removing {config.STOP_TAG} tag from RDS instance {instance['id']}") + logger.info( + f"Removing {config.STOP_TAG} tag from RDS instance {instance['id']}" + ) rds_client.remove_tags_from_resource( ResourceName=f"arn:{self.get_partition_for_region(region)}:rds:{region}:{self.account_id}:db:{instance['id']}", - TagKeys=[config.STOP_TAG] + TagKeys=[config.STOP_TAG], ) - + # Log with detailed information - self.log_action(instance['id'], region, "start", dry_run=dry_run) - + self.log_action(instance["id"], region, "start", dry_run=dry_run) + except Exception as e: - logger.error(f"Failed to {'simulate starting' if dry_run else 'start'} RDS instance {instance['id']}: {e}") + logger.error( + f"Failed to {'simulate starting' if dry_run else 'start'} RDS instance {instance['id']}: {e}" + ) diff --git a/local-app/python-tools/gfl-resource-actions/plan.md b/local-app/python-tools/gfl-resource-actions/plan.md index 5209f1f6..7874143a 100644 --- a/local-app/python-tools/gfl-resource-actions/plan.md +++ b/local-app/python-tools/gfl-resource-actions/plan.md @@ -72,23 +72,23 @@ The logging implementation could be enhanced: 2.2 Granular Log Levels: - Implementation details for each log level: - - DEBUG: + - DEBUG: - AWS API request/response bodies and headers - Parameter values for all functions - Resource discovery detailed progress - Authentication token acquisition - Configuration loading and resolution steps - Connection attempts and retries - - - INFO: + + - INFO: - Resource operations start/completion (e.g., "Starting EC2 instance i-123456") - Resource state changes (e.g., "RDS instance changed to 'stopping'") - Important configuration values loaded - Number of resources discovered/affected - Operation timing information - Profile and region information in use - - - WARNING: + + - WARNING: - Resources excluded due to tags - Missing optional configuration - Slow API responses @@ -96,16 +96,16 @@ The logging implementation could be enhanced: - Deprecated API usage - Retrying operations after recoverable errors - Missing tags or metadata - - - ERROR: + + - ERROR: - Failed operations on specific resources - Authentication or permission issues - Invalid parameters or configurations - API rate limiting or throttling - Network connectivity issues to specific regions - Resource not found - - - CRITICAL: + + - CRITICAL: - Complete failure to authenticate - Fatal configuration errors - Inability to access any AWS services @@ -114,7 +114,7 @@ The logging implementation could be enhanced: - Implementation approach: - Create logging helper functions for consistent usage: ```python - def log_resource_operation(logger, level, operation, resource_type, resource_id, + def log_resource_operation(logger, level, operation, resource_type, resource_id, region, details=None, exception=None): """Log resource operations with consistent structure.""" msg = f"{operation} {resource_type} {resource_id} in {region}" @@ -132,36 +132,36 @@ The logging implementation could be enhanced: msg += f" (failed: {exception})" logger.log(level, msg, extra=extra) ``` - + - Add log level configuration: - Command line option to override log level - Environment variable for log level - Configuration file setting - Different log levels for console vs file output - + - Create module-level logger constants with appropriate levels: ```python # Base module logger logger = logging.getLogger('aws_resource_management') - + # Component-specific loggers with potential different levels api_logger = logger.getChild('api') # For AWS API calls discovery_logger = logger.getChild('discovery') # For resource discovery operation_logger = logger.getChild('operation') # For resource operations ``` - + - Add utility function to adjust verbosity: ```python def set_verbosity(level_name: str) -> None: """ Set the verbosity level of the application. - + Arguments: level_name: One of 'debug', 'info', 'warning', 'error', 'critical' """ level = getattr(logging, level_name.upper()) logger.setLevel(level) - + # Adjust component loggers as needed if level <= logging.DEBUG: api_logger.setLevel(logging.DEBUG) # Always show API details in debug mode @@ -175,32 +175,32 @@ The logging implementation could be enhanced: - Common attributes for all errors: error_code, message, details - Methods for serializing errors to JSON - Support for contextual information (account_id, region, etc.) - + - ConfigurationError (config issues): - Missing required configuration - Invalid configuration values - Configuration file parsing errors - Environment variable conflicts - + - AuthenticationError (credential issues): - Invalid credentials - Expired credentials - Missing credentials - Insufficient permissions - + - ResourceOperationError (resource actions): - Failed state transitions - Resource in incompatible state - Resource already in target state - Dependencies preventing operation - Resource-specific error subclasses (EC2Error, RDSError, etc.) - + - NetworkError (connectivity issues): - Connection timeouts - Rate limiting/throttling - DNS resolution failures - Endpoint unavailability - + - ValidationError (input validation): - Invalid parameter types - Value range violations @@ -214,40 +214,40 @@ The logging implementation could be enhanced: # General errors (1-99) UNKNOWN_ERROR = 1 INTERNAL_ERROR = 2 - + # Configuration errors (100-199) CONFIG_NOT_FOUND = 100 CONFIG_PARSE_ERROR = 101 CONFIG_VALIDATION_ERROR = 102 - + # Authentication errors (200-299) AUTH_INVALID_CREDENTIALS = 200 AUTH_EXPIRED_CREDENTIALS = 201 AUTH_INSUFFICIENT_PERMISSIONS = 202 - + # Resource operation errors (300-399) RESOURCE_NOT_FOUND = 300 RESOURCE_STATE_CONFLICT = 301 RESOURCE_DEPENDENCY_ERROR = 302 - + # Service-specific error ranges # EC2 errors (1000-1099) EC2_INVALID_STATE = 1000 EC2_INSTANCE_NOT_FOUND = 1001 - + # RDS errors (1100-1199) RDS_INVALID_STATE = 1100 RDS_INSTANCE_NOT_FOUND = 1101 ``` - + - Add contextual information to exceptions: ```python class BaseAWSResourceError(Exception): """Base exception for all AWS Resource Management errors.""" - + def __init__( - self, - message: str, + self, + message: str, error_code: ErrorCode = ErrorCode.UNKNOWN_ERROR, account_id: Optional[str] = None, region: Optional[str] = None, @@ -264,16 +264,16 @@ The logging implementation could be enhanced: self.resource_id = resource_id self.details = details or {} self.original_exception = original_exception - + # Build detailed error message detailed_msg = f"[{error_code.name}] {message}" if resource_type and resource_id: detailed_msg += f" (Resource: {resource_type}/{resource_id})" if region: detailed_msg += f" in region {region}" - + super().__init__(detailed_msg) - + def to_dict(self) -> Dict[str, Any]: """Convert exception to a dictionary for serialization.""" result = { @@ -281,19 +281,19 @@ The logging implementation could be enhanced: "error_name": self.error_code.name, "message": self.message, } - + # Add contextual information if available for field in ["account_id", "region", "resource_type", "resource_id"]: if getattr(self, field): result[field] = getattr(self, field) - + # Add any additional details if self.details: result["details"] = self.details - + return result ``` - + - Add utility functions for error handling: ```python def handle_boto3_error( @@ -306,7 +306,7 @@ The logging implementation could be enhanced: """Convert boto3 errors to our custom exceptions with proper context.""" error_code = boto3_error.response.get("Error", {}).get("Code", "Unknown") error_message = boto3_error.response.get("Error", {}).get("Message", str(boto3_error)) - + # Map boto3 error codes to our error codes if error_code == "AuthFailure": return AuthenticationError( @@ -318,7 +318,7 @@ The logging implementation could be enhanced: details={"operation": operation}, original_exception=boto3_error ) - + # Handle throttling/rate limiting if error_code == "Throttling" or error_code == "ThrottlingException": return NetworkError( @@ -330,7 +330,7 @@ The logging implementation could be enhanced: details={"operation": operation}, original_exception=boto3_error ) - + # Default to ResourceOperationError for other cases return ResourceOperationError( f"AWS operation failed: {error_message}", @@ -366,9 +366,9 @@ The logging implementation could be enhanced: - Implementation approach for stack trace logging: ```python def log_exception( - logger, - exc: Exception, - msg: str = "An exception occurred", + logger, + exc: Exception, + msg: str = "An exception occurred", level: int = logging.ERROR, include_traceback: bool = True, include_cause_chain: bool = True, @@ -376,7 +376,7 @@ The logging implementation could be enhanced: ) -> None: """ Log an exception with stack trace at the specified level. - + Args: logger: Logger to use exc: The exception to log @@ -387,12 +387,12 @@ The logging implementation could be enhanced: context: Additional context to include in the log """ exc_info = sys.exc_info() if include_traceback else None - + # Extract info from exception extra = context or {} extra['exception_type'] = exc.__class__.__name__ extra['exception_msg'] = str(exc) - + # Format cause chain if requested and available if include_cause_chain and exc.__cause__: causes = [] @@ -400,9 +400,9 @@ The logging implementation could be enhanced: while current: causes.append(f"{current.__class__.__name__}: {str(current)}") current = current.__cause__ - + extra['exception_causes'] = causes - + # Log the exception logger.log(level, msg, exc_info=exc_info, extra=extra) ``` @@ -419,21 +419,21 @@ The logging implementation could be enhanced: def add_request_logging_to_session(session: boto3.Session, log_level: int = logging.DEBUG) -> None: """ Add request/response logging hooks to a boto3 session. - + Args: session: boto3 Session to instrument log_level: Log level to use for API requests/responses """ api_logger = logging.getLogger('aws_resource_management.api') - + def log_request(request, **kwargs): # Create a unique ID for this request for correlation request_id = str(uuid.uuid4()) request.context['request_log_id'] = request_id - + # Get sanitized parameters (remove sensitive info) params = _sanitize_params(request.context.get('params', {})) - + # Log the request details api_logger.log( log_level, @@ -447,17 +447,17 @@ The logging implementation could be enhanced: 'endpoint_url': request.context.get('client_meta').endpoint_url, } ) - + def log_response(request, response, **kwargs): # Get the request ID we set earlier request_id = getattr(request.context, 'request_log_id', 'unknown') - + # Get AWS request ID from response aws_request_id = response.get('ResponseMetadata', {}).get('RequestId', 'unknown') - + # Calculate latency latency_ms = response.get('ResponseMetadata', {}).get('HTTPHeaders', {}).get('x-amz-request-latency', -1) - + # Log the response api_logger.log( log_level, @@ -471,7 +471,7 @@ The logging implementation could be enhanced: 'operation': request.context.get('operation_name'), } ) - + # Register the event handlers with boto3 session.events.register('before-send.*.*.', log_request) session.events.register('after-call.*.*.', log_response) @@ -487,65 +487,65 @@ The logging implementation could be enhanced: - Implementation approach for error reporting: ```python def format_error_for_display( - error: Exception, + error: Exception, verbose: bool = False, include_help: bool = True ) -> str: """ Format an error for display to users. - + Args: error: The exception to format verbose: Whether to include detailed information include_help: Whether to include help text for known errors - + Returns: Formatted error message """ # Handle our custom exceptions if isinstance(error, BaseAWSResourceError): message = f"Error ({error.error_code.name}): {error.message}" - + # Add context for resource errors if available if error.resource_type and error.resource_id: message += f"\nResource: {error.resource_type}/{error.resource_id}" if error.region: message += f"\nRegion: {error.region}" - + # Add details for verbose mode if verbose and error.details: message += "\nDetails:" for k, v in error.details.items(): message += f"\n {k}: {v}" - + # Add help text for known errors if include_help: help_text = ERROR_HELP_TEXT.get(error.error_code) if help_text: message += f"\n\nTroubleshooting:\n{help_text}" - + return message - + # Handle boto3 ClientError if isinstance(error, botocore.exceptions.ClientError): error_code = error.response.get("Error", {}).get("Code", "Unknown") error_message = error.response.get("Error", {}).get("Message", str(error)) - + message = f"AWS Error ({error_code}): {error_message}" - + # Add request ID if available for troubleshooting request_id = error.response.get("ResponseMetadata", {}).get("RequestId") if request_id and verbose: message += f"\nAWS Request ID: {request_id}" - + # Add help for common boto3 errors if include_help: help_text = BOTO3_ERROR_HELP.get(error_code) if help_text: message += f"\n\nTroubleshooting:\n{help_text}" - + return message - + # Default case for other exceptions return f"Error: {str(error)}" ``` @@ -572,7 +572,7 @@ The logging implementation could be enhanced: ): """ Decorator for retrying functions with exponential backoff. - + Args: max_retries: Maximum number of retries initial_backoff: Initial backoff time in seconds @@ -580,7 +580,7 @@ The logging implementation could be enhanced: max_backoff: Maximum backoff time in seconds retryable_exceptions: Tuple of exceptions that should trigger a retry should_retry_cb: Optional callback to determine if an exception is retryable - + Returns: Decorated function """ @@ -588,10 +588,10 @@ The logging implementation could be enhanced: @functools.wraps(func) def wrapper(*args, **kwargs): retry_logger = logging.getLogger('aws_resource_management.retry') - + retry_count = 0 backoff = initial_backoff - + while True: try: return func(*args, **kwargs) @@ -600,24 +600,24 @@ The logging implementation could be enhanced: should_retry = True if should_retry_cb: should_retry = should_retry_cb(e) - + # If we've reached max retries or shouldn't retry, re-raise if retry_count >= max_retries or not should_retry: raise - + # Calculate backoff with jitter for this retry jittered_backoff = backoff * (0.9 + 0.2 * random.random()) actual_backoff = min(jittered_backoff, max_backoff) - + # Log the retry attempt retry_logger.warning( f"Retrying {func.__name__} after error: {str(e)}. " f"Retry {retry_count + 1}/{max_retries} in {actual_backoff:.2f}s" ) - + # Wait before retrying time.sleep(actual_backoff) - + # Update for next iteration retry_count += 1 backoff = backoff * backoff_factor diff --git a/local-app/python-tools/gfl-resource-actions/setup.py b/local-app/python-tools/gfl-resource-actions/setup.py index 3f8d4140..f8a7ec77 100644 --- a/local-app/python-tools/gfl-resource-actions/setup.py +++ b/local-app/python-tools/gfl-resource-actions/setup.py @@ -1,7 +1,8 @@ """ Setup script for AWS Resource Management tool. """ -from setuptools import setup, find_packages + +from setuptools import find_packages, setup setup( name="aws_resource_management", @@ -16,8 +17,8 @@ "python-dateutil>=2.8.2", ], entry_points={ - 'console_scripts': [ - 'aws-resource-mgmt=aws_resource_management.cli:main', + "console_scripts": [ + "aws-resource-mgmt=aws_resource_management.cli:main", ], }, classifiers=[ diff --git a/local-app/python-tools/glacier_to_s3_restore/glacier_to_s3_restore.cfg b/local-app/python-tools/glacier_to_s3_restore/glacier_to_s3_restore.cfg index de5be127..b929db30 100644 --- a/local-app/python-tools/glacier_to_s3_restore/glacier_to_s3_restore.cfg +++ b/local-app/python-tools/glacier_to_s3_restore/glacier_to_s3_restore.cfg @@ -3,4 +3,4 @@ do1=None [regions] west=us-gov-east-1 [object-lists] -test-object-list=single-object-list.txt,do1,us-gov-east-1,v-s3-erd-dcdl,14 \ No newline at end of file +test-object-list=single-object-list.txt,do1,us-gov-east-1,v-s3-erd-dcdl,14 diff --git a/local-app/python-tools/glacier_to_s3_restore/glacier_to_s3_restore.py b/local-app/python-tools/glacier_to_s3_restore/glacier_to_s3_restore.py index 7b382216..7f46c118 100644 --- a/local-app/python-tools/glacier_to_s3_restore/glacier_to_s3_restore.py +++ b/local-app/python-tools/glacier_to_s3_restore/glacier_to_s3_restore.py @@ -1,204 +1,237 @@ -import boto3 import argparse import configparser import io import json -import pandas as pd import os import threading +from datetime import date, datetime + +import boto3 +import pandas as pd -from datetime import datetime, date def json_serial(obj): - """JSON serializer for objects not serializable by default json code""" - if isinstance(obj, (datetime, date)): - return obj.isoformat() - raise TypeError ("Type %s not serializable" % type(obj)) + """JSON serializer for objects not serializable by default json code""" + if isinstance(obj, (datetime, date)): + return obj.isoformat() + raise TypeError("Type %s not serializable" % type(obj)) + def main(): - parser = argparse.ArgumentParser(description='Connect to AWS accounts') - parser.add_argument('--profile', help='AWS profile') - parser.add_argument('--config', help='Config file') - parser.add_argument('--me', help='Run using available credentials', - action='store_true') - parser.add_argument('--assume', help='Run using assumed roles', - action='store_true') - parser.add_argument('--region', help='A region to run against') - parser.add_argument('--read', help = 'Read API calls from cached results', - action='store_true') - parser.add_argument('--write', help = 'Write API calls to cached results', - action='store_true') - parser.add_argument('--run_token', help ='ID for cached results to read and/or write') - parser.add_argument('--object_list', help='object list in format filename,profile,region') - parser.add_argument('--list', help="Use this option to list the objects and make API call to get object attributes, no effect", - action='store_true') - - - args = parser.parse_args() - - run_token = None - - list_control = False - if args.list: - print("List option is set") - list_control = True - - if args.read: - read_from_cache = True - run_token = args.run_token - else: - read_from_cache = False - - if args.write: - write_to_cache = True - run_token = args.run_token - else: - write_to_cache = False - - if(args.config): - configParser = configparser.RawConfigParser(allow_no_value=True) + parser = argparse.ArgumentParser(description="Connect to AWS accounts") + parser.add_argument("--profile", help="AWS profile") + parser.add_argument("--config", help="Config file") + parser.add_argument( + "--me", help="Run using available credentials", action="store_true" + ) + parser.add_argument("--assume", help="Run using assumed roles", action="store_true") + parser.add_argument("--region", help="A region to run against") + parser.add_argument( + "--read", help="Read API calls from cached results", action="store_true" + ) + parser.add_argument( + "--write", help="Write API calls to cached results", action="store_true" + ) + parser.add_argument( + "--run_token", help="ID for cached results to read and/or write" + ) + parser.add_argument( + "--object_list", help="object list in format filename,profile,region" + ) + parser.add_argument( + "--list", + help="Use this option to list the objects and make API call to get object attributes, no effect", + action="store_true", + ) + + args = parser.parse_args() + + run_token = None + + list_control = False + if args.list: + print("List option is set") + list_control = True + + if args.read: + read_from_cache = True + run_token = args.run_token + else: + read_from_cache = False + + if args.write: + write_to_cache = True + run_token = args.run_token + else: + write_to_cache = False + if args.config: - with open(args.config) as file: - configParser.read_file(file) - - if args.profile: - profiles = [args.profile] - elif args.config: - profiles = [item[0] for item in configParser.items("roles")] - profile_roles = {item[0]:item[1] for item in configParser.items("roles")} - else: - profiles = ['do1'] - - if args.region: - regions = [args.region] - elif args.config: - regions = [item[1] for item in configParser.items("regions")] - else: - regions =['us-gov-west-1'] - - if args.object_list: - object_list_spec = {'object_list':args.object_list} - elif args.config: - object_list_spec = {item[0]:item[1] for item in configParser.items("object-lists")} - else: - object_list_spec = {'object_list':'object_list.txt,do1,us-gov-east-1,v-s3-erd-dcdl'} - - object_lists = {} - for key in object_list_spec: - ol_filename, ol_profile, ol_region, ol_bucket, days = object_list_spec[key].split(',') - object_lists[key] = {'filename':ol_filename, - 'profile':ol_profile, - 'region':ol_region, - 'bucket':ol_bucket, - 'days':int(days)} - - for key in object_lists: - print(f'key [{key}] file [{object_lists[key]["filename"]}] profile [{object_lists[key]["profile"]}] region [{object_lists[key]["region"]}] bucket[{object_lists[key]["bucket"]}]') - - if args.me: - use_roles = False - elif args.assume: - use_roles = True - else: - print ('Inconsistent configuration. Exiting. Use one of "me" or "assume".') - exit() - - session = {} - - headers = ['account','region','instance_id', 'private_dns_name','instance_type', 'vpc_id','subnet_id','state'] - - master_list_of_lists = [] - - thread_list = [] - - for profile in profiles: - for region in regions: - profile_region = f'{profile}_{region}' - if not use_roles: - session[profile_region] = boto3.Session(profile_name = profile, region_name = region) - else: - role = profile_roles[profile] - stsClient = boto3.client('sts') - response = stsClient.assume_role(RoleArn=role, RoleSessionName='AWSConnect',ExternalId='ti-jbr-2010') - accessKeyId = response['Credentials']['AccessKeyId'] - secretAccessKey = response['Credentials']['SecretAccessKey'] - sessionToken = response['Credentials']['SessionToken'] - session[profile_region] = boto3.Session(aws_access_key_id = accessKeyId, - aws_secret_access_key = secretAccessKey, - aws_session_token = sessionToken) - boto_clients_s3 = {} - result_set = [] - for profile in profiles: - account = profile.split('-')[0] - print(f'account [{account}]') - for region in regions: - profile_region = f'{profile}_{region}' - boto_clients_s3[profile_region] = session[profile_region].client('s3') - # method_token = 'describe_instances' - # thread = threading.Thread(name=profile_region+'_'+method_token, target = tabulate_ec2s, args = (account, region, boto_client_ec2, profile_region, method_token, master_list_of_lists, read_from_cache, write_to_cache, run_token)) - # thread.start() - # thread_list.append(thread) - # #tabulate_ec2s(account, region, boto_client_ec2, profile_region, method_token, master_list_of_lists, read_from_cache, write_to_cache, run_token) - for key in object_lists: - profile = object_lists[key]['profile'] - region = object_lists[key]['region'] - bucket = object_lists[key]['bucket'] - file = object_lists[key]['filename'] - days = object_lists[key]['days'] - - print(f'key[{key}] profile[{profile}] region [{region}] bucket[{bucket}] file[{file}]') - - profile_region = f'{profile}_{region}' - - client_s3 = boto_clients_s3[profile_region] - - with open(file, "r") as f: - lines = f.readlines() - lines = [line.strip() for line in lines] - - for line in lines: - print(line) - line_without_s3prefix = line[5:] - first_slash_index = line_without_s3prefix.find('/') - object_key = line_without_s3prefix[first_slash_index+1:] - print(object_key) - - print(list_control) - if list_control: - print("List control execution") - response = client_s3.get_object_attributes( - Bucket = bucket, - Key = object_key, - ObjectAttributes = ["StorageClass"] + configParser = configparser.RawConfigParser(allow_no_value=True) + if args.config: + with open(args.config) as file: + configParser.read_file(file) + + if args.profile: + profiles = [args.profile] + elif args.config: + profiles = [item[0] for item in configParser.items("roles")] + profile_roles = {item[0]: item[1] for item in configParser.items("roles")} + else: + profiles = ["do1"] + + if args.region: + regions = [args.region] + elif args.config: + regions = [item[1] for item in configParser.items("regions")] + else: + regions = ["us-gov-west-1"] + + if args.object_list: + object_list_spec = {"object_list": args.object_list} + elif args.config: + object_list_spec = { + item[0]: item[1] for item in configParser.items("object-lists") + } + else: + object_list_spec = { + "object_list": "object_list.txt,do1,us-gov-east-1,v-s3-erd-dcdl" + } + + object_lists = {} + for key in object_list_spec: + ol_filename, ol_profile, ol_region, ol_bucket, days = object_list_spec[ + key + ].split(",") + object_lists[key] = { + "filename": ol_filename, + "profile": ol_profile, + "region": ol_region, + "bucket": ol_bucket, + "days": int(days), + } + + for key in object_lists: + print( + f'key [{key}] file [{object_lists[key]["filename"]}] profile [{object_lists[key]["profile"]}] region [{object_lists[key]["region"]}] bucket[{object_lists[key]["bucket"]}]' ) - print(response) - storage_class = response['StorageClass'] - result_list = [profile, region, bucket, object_key, storage_class] - print(result_list) - result_set.append(result_list) - else: - print("action implementation") - response = client_s3.restore_object( - Bucket = bucket, - Key = object_key, - RestoreRequest = { - 'Days': days, - 'GlacierJobParameters':{ - 'Tier': 'Standard' - } - } + + if args.me: + use_roles = False + elif args.assume: + use_roles = True + else: + print('Inconsistent configuration. Exiting. Use one of "me" or "assume".') + exit() + + session = {} + + headers = [ + "account", + "region", + "instance_id", + "private_dns_name", + "instance_type", + "vpc_id", + "subnet_id", + "state", + ] + + master_list_of_lists = [] + + thread_list = [] + + for profile in profiles: + for region in regions: + profile_region = f"{profile}_{region}" + if not use_roles: + session[profile_region] = boto3.Session( + profile_name=profile, region_name=region + ) + else: + role = profile_roles[profile] + stsClient = boto3.client("sts") + response = stsClient.assume_role( + RoleArn=role, RoleSessionName="AWSConnect", ExternalId="ti-jbr-2010" + ) + accessKeyId = response["Credentials"]["AccessKeyId"] + secretAccessKey = response["Credentials"]["SecretAccessKey"] + sessionToken = response["Credentials"]["SessionToken"] + session[profile_region] = boto3.Session( + aws_access_key_id=accessKeyId, + aws_secret_access_key=secretAccessKey, + aws_session_token=sessionToken, + ) + boto_clients_s3 = {} + result_set = [] + for profile in profiles: + account = profile.split("-")[0] + print(f"account [{account}]") + for region in regions: + profile_region = f"{profile}_{region}" + boto_clients_s3[profile_region] = session[profile_region].client("s3") + # method_token = 'describe_instances' + # thread = threading.Thread(name=profile_region+'_'+method_token, target = tabulate_ec2s, args = (account, region, boto_client_ec2, profile_region, method_token, master_list_of_lists, read_from_cache, write_to_cache, run_token)) + # thread.start() + # thread_list.append(thread) + # #tabulate_ec2s(account, region, boto_client_ec2, profile_region, method_token, master_list_of_lists, read_from_cache, write_to_cache, run_token) + for key in object_lists: + profile = object_lists[key]["profile"] + region = object_lists[key]["region"] + bucket = object_lists[key]["bucket"] + file = object_lists[key]["filename"] + days = object_lists[key]["days"] + + print( + f"key[{key}] profile[{profile}] region [{region}] bucket[{bucket}] file[{file}]" ) - - # for t in thread_list: - # t.join() - # df = data_frame(headers,master_list_of_lists) - # print (df) - # excel_name = ''.join(['./excel/ec2_attributes_',datetime.now().strftime('%Y%m%d%H%M%S'),'.xlsx']) - # writer_keys = pd.ExcelWriter(excel_name, engine='xlsxwriter') - # df.to_excel(writer_keys,sheet_name='key_attributes') - # writer_keys.save() - # print(f'main: wrote excel file of key details to disk') + profile_region = f"{profile}_{region}" + + client_s3 = boto_clients_s3[profile_region] + + with open(file, "r") as f: + lines = f.readlines() + lines = [line.strip() for line in lines] + + for line in lines: + print(line) + line_without_s3prefix = line[5:] + first_slash_index = line_without_s3prefix.find("/") + object_key = line_without_s3prefix[first_slash_index + 1 :] + print(object_key) + + print(list_control) + if list_control: + print("List control execution") + response = client_s3.get_object_attributes( + Bucket=bucket, Key=object_key, ObjectAttributes=["StorageClass"] + ) + print(response) + storage_class = response["StorageClass"] + result_list = [profile, region, bucket, object_key, storage_class] + print(result_list) + result_set.append(result_list) + else: + print("action implementation") + response = client_s3.restore_object( + Bucket=bucket, + Key=object_key, + RestoreRequest={ + "Days": days, + "GlacierJobParameters": {"Tier": "Standard"}, + }, + ) + + # for t in thread_list: + # t.join() + # df = data_frame(headers,master_list_of_lists) + # print (df) + # excel_name = ''.join(['./excel/ec2_attributes_',datetime.now().strftime('%Y%m%d%H%M%S'),'.xlsx']) + # writer_keys = pd.ExcelWriter(excel_name, engine='xlsxwriter') + # df.to_excel(writer_keys,sheet_name='key_attributes') + # writer_keys.save() + # print(f'main: wrote excel file of key details to disk') + # def tabulate_ec2s(account, region, boto_client, profile_region, method_token, master_list_of_lists, read, write, user_token): # response = paginate_wrapper(read = read, write = write, client = boto_client, client_token = f'ec2_{profile_region}', method_token = 'describe_instances', run_token =user_token) @@ -222,67 +255,88 @@ def main(): # return df -def paginate_wrapper(read=None, write=None, client=None, client_token=None, run_token=None, method_token='default', parameter_dict = {},parameter_token=''): - # print(f'paginate_wrapper: processing client_token[{client_token}] method_token [{method_token}] run_token[{run_token}]') - # if neither read nor write is specified, then look for a serialization. If it doesn't exist, then write - normalized_parameter_token = parameter_token.translate({ord(':'):'-'}) - if read and write: - if serialization_exists([client_token, method_token, run_token,normalized_parameter_token]): - read_action = True - write_action = False +def paginate_wrapper( + read=None, + write=None, + client=None, + client_token=None, + run_token=None, + method_token="default", + parameter_dict={}, + parameter_token="", +): + # print(f'paginate_wrapper: processing client_token[{client_token}] method_token [{method_token}] run_token[{run_token}]') + # if neither read nor write is specified, then look for a serialization. If it doesn't exist, then write + normalized_parameter_token = parameter_token.translate({ord(":"): "-"}) + if read and write: + if serialization_exists( + [client_token, method_token, run_token, normalized_parameter_token] + ): + read_action = True + write_action = False + else: + write_action = True + read_action = False else: - write_action = True - read_action = False - else: - read_action = read - write_action = write - if read_action: - result = deserialize([client_token, method_token,run_token,normalized_parameter_token]) - else: - if client.can_paginate(method_token): - paginator = client.get_paginator(method_token) - # print(f'paginate_wrapper: method_token[{method_token}] keys in parameter_dict [{parameter_dict.keys()}]') - page_iterator = paginator.paginate(**parameter_dict) - result = [page for page in page_iterator] + read_action = read + write_action = write + if read_action: + result = deserialize( + [client_token, method_token, run_token, normalized_parameter_token] + ) else: - func = getattr(client,method_token) - # print(f'paginate_wrapper: method_token [{method_token}] keys in parameter_dict [{parameter_dict.keys()}]') - # parameter_string = ', '.join([f'parameter[{parameter}] value [{parameter_dict[parameter]}]' for parameter in parameter_dict]) - # print(f'paginate_wrapper: parameter_string <{parameter_string}>') - result = [func(**parameter_dict)] - # print(f'paginate_wrapper: result [{result}]') + if client.can_paginate(method_token): + paginator = client.get_paginator(method_token) + # print(f'paginate_wrapper: method_token[{method_token}] keys in parameter_dict [{parameter_dict.keys()}]') + page_iterator = paginator.paginate(**parameter_dict) + result = [page for page in page_iterator] + else: + func = getattr(client, method_token) + # print(f'paginate_wrapper: method_token [{method_token}] keys in parameter_dict [{parameter_dict.keys()}]') + # parameter_string = ', '.join([f'parameter[{parameter}] value [{parameter_dict[parameter]}]' for parameter in parameter_dict]) + # print(f'paginate_wrapper: parameter_string <{parameter_string}>') + result = [func(**parameter_dict)] + # print(f'paginate_wrapper: result [{result}]') + + if write_action: + serialize( + result, + [client_token, method_token, run_token, normalized_parameter_token], + ) + return result - if write_action: - serialize(result, [client_token, method_token, run_token,normalized_parameter_token]) - return result def serialize(obj, token_list): - serialized = json.dumps(obj, default=json_serial) - filename = build_json_filename(token_list) - with open(filename, 'w') as file: - file.write(serialized) + serialized = json.dumps(obj, default=json_serial) + filename = build_json_filename(token_list) + with open(filename, "w") as file: + file.write(serialized) + def deserialize(token_list): - filename = build_json_filename(token_list) - with open(filename) as file: - serialized = file.read() - deserialized = json.loads(serialized) - return deserialized + filename = build_json_filename(token_list) + with open(filename) as file: + serialized = file.read() + deserialized = json.loads(serialized) + return deserialized + def build_json_filename(string_list): - joined_list = '-'.join(string_list) - return f'c:/cache/json/{joined_list}.json' + joined_list = "-".join(string_list) + return f"c:/cache/json/{joined_list}.json" + def serialization_exists(token_list): - filename = build_json_filename(token_list) - return os.path.isfile(filename) + filename = build_json_filename(token_list) + return os.path.isfile(filename) + def json_serial(obj): - """JSON serializer for objects not serializable by default json code""" - if isinstance(obj, (datetime, date)): - return obj.isoformat() - raise TypeError ("Type %s not serializable" % type(obj)) + """JSON serializer for objects not serializable by default json code""" + if isinstance(obj, (datetime, date)): + return obj.isoformat() + raise TypeError("Type %s not serializable" % type(obj)) if __name__ == "__main__": - main() + main() diff --git a/local-app/python-tools/list_kms_keys/list_kms_keys.py b/local-app/python-tools/list_kms_keys/list_kms_keys.py index be8a917a..2213f7b8 100644 --- a/local-app/python-tools/list_kms_keys/list_kms_keys.py +++ b/local-app/python-tools/list_kms_keys/list_kms_keys.py @@ -1,194 +1,270 @@ -import boto3 import argparse import configparser import io import json -import pandas as pd import os import threading +from datetime import date, datetime + +import boto3 +import pandas as pd -from datetime import datetime, date def json_serial(obj): - """JSON serializer for objects not serializable by default json code""" - if isinstance(obj, (datetime, date)): - return obj.isoformat() - raise TypeError ("Type %s not serializable" % type(obj)) + """JSON serializer for objects not serializable by default json code""" + if isinstance(obj, (datetime, date)): + return obj.isoformat() + raise TypeError("Type %s not serializable" % type(obj)) + def main(): - parser = argparse.ArgumentParser(description='Connect to AWS accounts') - parser.add_argument('--profile', help='AWS profile') - parser.add_argument('--config', help='Config file') - parser.add_argument('--me', help='Run using available credentials', - action='store_true') - parser.add_argument('--assume', help='Run using assumed roles', - action='store_true') - parser.add_argument('--region', help='A region to run against') - parser.add_argument('--read', help = 'Read API calls from cached results', - action='store_true') - parser.add_argument('--write', help = 'Write API calls to cached results', - action='store_true') - parser.add_argument('--run_token', help ='ID for cached results to read and/or write') - - args = parser.parse_args() - - run_token = None - - if args.read: - read_from_cache = True - run_token = args.run_token - else: - read_from_cache = False - - if args.write: - write_to_cache = True - run_token = args.run_token - else: - write_to_cache = False - - if(args.config): - configParser = configparser.RawConfigParser(allow_no_value=True) + parser = argparse.ArgumentParser(description="Connect to AWS accounts") + parser.add_argument("--profile", help="AWS profile") + parser.add_argument("--config", help="Config file") + parser.add_argument( + "--me", help="Run using available credentials", action="store_true" + ) + parser.add_argument("--assume", help="Run using assumed roles", action="store_true") + parser.add_argument("--region", help="A region to run against") + parser.add_argument( + "--read", help="Read API calls from cached results", action="store_true" + ) + parser.add_argument( + "--write", help="Write API calls to cached results", action="store_true" + ) + parser.add_argument( + "--run_token", help="ID for cached results to read and/or write" + ) + + args = parser.parse_args() + + run_token = None + + if args.read: + read_from_cache = True + run_token = args.run_token + else: + read_from_cache = False + + if args.write: + write_to_cache = True + run_token = args.run_token + else: + write_to_cache = False + if args.config: - with open(args.config) as file: - configParser.read_file(file) - - if args.profile: - profiles = [args.profile] - elif args.config: - profiles = [item[0] for item in configParser.items("roles")] - profile_roles = {item[0]:item[1] for item in configParser.items("roles")} - else: - profiles = ['dev'] - - if args.region: - regions = [args.region] - elif args.config: - regions = [item[1] for item in configParser.items("regions")] - else: - regions =['us-gov-west-1'] - - if args.me: - use_roles = False - elif args.assume: - use_roles = True - else: - print ('Inconsistent configuration. Exiting. Use one of "me" or "assume".') - exit() - - session = {} - - headers = ['account','region','key_id','key_manager','enabled','description','key_state','application_tag','name_tag'] - - master_list_of_lists = [] - - thread_list = [] - - for profile in profiles: - for region in regions: - profile_region = f'{profile}_{region}' - if not use_roles: - session[profile_region] = boto3.Session(profile_name = profile, region_name = region) - else: - role = profile_roles[profile] - stsClient = boto3.client('sts') - response = stsClient.assume_role(RoleArn=role, RoleSessionName='AWSConnect',ExternalId='ti-jbr-2010') - accessKeyId = response['Credentials']['AccessKeyId'] - secretAccessKey = response['Credentials']['SecretAccessKey'] - sessionToken = response['Credentials']['SessionToken'] - session[profile_region] = boto3.Session(aws_access_key_id = accessKeyId, - aws_secret_access_key = secretAccessKey, - aws_session_token = sessionToken) - for profile in profiles: - account = profile.split('-')[0] - for region in regions: - profile_region = f'{profile}_{region}' - boto_client_kms = session[profile_region].client('kms') - method_token = 'describe_instances' - thread = threading.Thread(name=profile_region+'_'+method_token, target = tabulate_keys, args = (account, region, boto_client_kms, profile_region, method_token, master_list_of_lists, read_from_cache, write_to_cache, run_token)) - thread.start() - thread_list.append(thread) - #tabulate_keys(account, region, boto_client_kms, profile_region, method_token, master_list_of_lists, read_from_cache, write_to_cache, run_token) - - for t in thread_list: - t.join() - df = data_frame(headers,master_list_of_lists) - print (df) - excel_name = ''.join(['./excel/key_attributes_',datetime.now().strftime('%Y%m%d%H%M%S'),'.xlsx']) - writer_keys = pd.ExcelWriter(excel_name, engine='xlsxwriter') - df.to_excel(writer_keys,sheet_name='key_attributes') - writer_keys.save() - print(f'main: wrote excel file of key details to disk') - -def tabulate_keys(account, region, boto_client, profile_region, method_token, master_list_of_lists, read, write, user_token): - response = paginate_wrapper(boto_client, profile_region, 'list_keys', read = read, write = write, user_token =user_token) - description_result_dict = {} - tag_result_dict = {} - for page in response: - for key in page['Keys']: - key_id = key['KeyId'] - description_result = boto_client.describe_key(KeyId=key_id) - try: - tag_result = boto_client.list_resource_tags(KeyId=key_id)['Tags'] - print(tag_result) - except: - tag_result = [] - tag_dict = {tag['TagKey']:tag['TagValue'] for tag in tag_result} - description_result_dict[key_id] = description_result['KeyMetadata'] - tag_result_dict[key_id] = tag_dict - print ('profile_region[{0}]'.format(profile_region)) - list_of_lists = flatten(account, region, response, description_result_dict, tag_result_dict) - master_list_of_lists.extend(list_of_lists) - -def flatten(account, region, response,description_result_dict, tag_result_dict): - list_of_lists = [] - for page in response: - for key in page['Keys']: - key_id = key['KeyId'] - enabled = description_result_dict[key_id]['Enabled'] - description = description_result_dict[key_id]['Description'] - key_state = description_result_dict[key_id]['KeyState'] - application_tag = tag_result_dict[key_id].get("Application","") - name_tag = tag_result_dict[key_id].get("Name","") - key_manager = description_result_dict[key_id]['KeyManager'] - list = [account, region, key_id,key_manager,enabled,description,key_state,application_tag,name_tag] - list_of_lists.append(list) - return list_of_lists + configParser = configparser.RawConfigParser(allow_no_value=True) + if args.config: + with open(args.config) as file: + configParser.read_file(file) + + if args.profile: + profiles = [args.profile] + elif args.config: + profiles = [item[0] for item in configParser.items("roles")] + profile_roles = {item[0]: item[1] for item in configParser.items("roles")} + else: + profiles = ["dev"] + + if args.region: + regions = [args.region] + elif args.config: + regions = [item[1] for item in configParser.items("regions")] + else: + regions = ["us-gov-west-1"] + + if args.me: + use_roles = False + elif args.assume: + use_roles = True + else: + print('Inconsistent configuration. Exiting. Use one of "me" or "assume".') + exit() + + session = {} + + headers = [ + "account", + "region", + "key_id", + "key_manager", + "enabled", + "description", + "key_state", + "application_tag", + "name_tag", + ] + + master_list_of_lists = [] + + thread_list = [] + + for profile in profiles: + for region in regions: + profile_region = f"{profile}_{region}" + if not use_roles: + session[profile_region] = boto3.Session( + profile_name=profile, region_name=region + ) + else: + role = profile_roles[profile] + stsClient = boto3.client("sts") + response = stsClient.assume_role( + RoleArn=role, RoleSessionName="AWSConnect", ExternalId="ti-jbr-2010" + ) + accessKeyId = response["Credentials"]["AccessKeyId"] + secretAccessKey = response["Credentials"]["SecretAccessKey"] + sessionToken = response["Credentials"]["SessionToken"] + session[profile_region] = boto3.Session( + aws_access_key_id=accessKeyId, + aws_secret_access_key=secretAccessKey, + aws_session_token=sessionToken, + ) + for profile in profiles: + account = profile.split("-")[0] + for region in regions: + profile_region = f"{profile}_{region}" + boto_client_kms = session[profile_region].client("kms") + method_token = "describe_instances" + thread = threading.Thread( + name=profile_region + "_" + method_token, + target=tabulate_keys, + args=( + account, + region, + boto_client_kms, + profile_region, + method_token, + master_list_of_lists, + read_from_cache, + write_to_cache, + run_token, + ), + ) + thread.start() + thread_list.append(thread) + # tabulate_keys(account, region, boto_client_kms, profile_region, method_token, master_list_of_lists, read_from_cache, write_to_cache, run_token) + + for t in thread_list: + t.join() + df = data_frame(headers, master_list_of_lists) + print(df) + excel_name = "".join( + ["./excel/key_attributes_", datetime.now().strftime("%Y%m%d%H%M%S"), ".xlsx"] + ) + writer_keys = pd.ExcelWriter(excel_name, engine="xlsxwriter") + df.to_excel(writer_keys, sheet_name="key_attributes") + writer_keys.save() + print(f"main: wrote excel file of key details to disk") + + +def tabulate_keys( + account, + region, + boto_client, + profile_region, + method_token, + master_list_of_lists, + read, + write, + user_token, +): + response = paginate_wrapper( + boto_client, + profile_region, + "list_keys", + read=read, + write=write, + user_token=user_token, + ) + description_result_dict = {} + tag_result_dict = {} + for page in response: + for key in page["Keys"]: + key_id = key["KeyId"] + description_result = boto_client.describe_key(KeyId=key_id) + try: + tag_result = boto_client.list_resource_tags(KeyId=key_id)["Tags"] + print(tag_result) + except: + tag_result = [] + tag_dict = {tag["TagKey"]: tag["TagValue"] for tag in tag_result} + description_result_dict[key_id] = description_result["KeyMetadata"] + tag_result_dict[key_id] = tag_dict + print("profile_region[{0}]".format(profile_region)) + list_of_lists = flatten( + account, region, response, description_result_dict, tag_result_dict + ) + master_list_of_lists.extend(list_of_lists) + + +def flatten(account, region, response, description_result_dict, tag_result_dict): + list_of_lists = [] + for page in response: + for key in page["Keys"]: + key_id = key["KeyId"] + enabled = description_result_dict[key_id]["Enabled"] + description = description_result_dict[key_id]["Description"] + key_state = description_result_dict[key_id]["KeyState"] + application_tag = tag_result_dict[key_id].get("Application", "") + name_tag = tag_result_dict[key_id].get("Name", "") + key_manager = description_result_dict[key_id]["KeyManager"] + list = [ + account, + region, + key_id, + key_manager, + enabled, + description, + key_state, + application_tag, + name_tag, + ] + list_of_lists.append(list) + return list_of_lists + def data_frame(headers, list_of_lists): - df = pd.DataFrame(columns = headers, data=list_of_lists) + df = pd.DataFrame(columns=headers, data=list_of_lists) return df -def paginate_wrapper(client, client_token, method_token, read = False, write = False, user_token = None): - failed_read = False - if read: - try: - result = deserialize(client_token, method_token, user_token) - except: - failed_read = True - if not read or (failed_read and write): - paginator = client.get_paginator(method_token) - page_iterator = paginator.paginate() - result = [page for page in page_iterator] - if write: - serialize(result,client_token, method_token, user_token) - return result +def paginate_wrapper( + client, client_token, method_token, read=False, write=False, user_token=None +): + failed_read = False + if read: + try: + result = deserialize(client_token, method_token, user_token) + except: + failed_read = True + if not read or (failed_read and write): + paginator = client.get_paginator(method_token) + page_iterator = paginator.paginate() + result = [page for page in page_iterator] + if write: + serialize(result, client_token, method_token, user_token) + return result + def serialize(obj, token_1, token_2, token_3): - serialized = json.dumps(obj, default=json_serial) - filename = build_json_filename(token_1, token_2, token_3) - with open(filename, 'w') as file: - file.write(serialized) + serialized = json.dumps(obj, default=json_serial) + filename = build_json_filename(token_1, token_2, token_3) + with open(filename, "w") as file: + file.write(serialized) + def deserialize(token_1, token_2, token_3): - filename = build_json_filename(token_1, token_2, token_3) - with open(filename) as file: - serialized = file.read() - deserialized = json.loads(serialized) - return deserialized + filename = build_json_filename(token_1, token_2, token_3) + with open(filename) as file: + serialized = file.read() + deserialized = json.loads(serialized) + return deserialized + def build_json_filename(token_1, token_2, token_3): - return os.path.join('json',f'{token_1}-{token_2}-{token_3}.json') + return os.path.join("json", f"{token_1}-{token_2}-{token_3}.json") + if __name__ == "__main__": - main() + main() diff --git a/local-app/python-tools/survey_bucket_encryption/survey_bucket_encryption.cfg b/local-app/python-tools/survey_bucket_encryption/survey_bucket_encryption.cfg index f394a91a..d65c3a67 100644 --- a/local-app/python-tools/survey_bucket_encryption/survey_bucket_encryption.cfg +++ b/local-app/python-tools/survey_bucket_encryption/survey_bucket_encryption.cfg @@ -4,4 +4,3 @@ ma10=None do1=None [regions] west=us-gov-west-1 - diff --git a/local-app/python-tools/survey_bucket_encryption/survey_bucket_encryption.py b/local-app/python-tools/survey_bucket_encryption/survey_bucket_encryption.py index 1567389e..77413c67 100644 --- a/local-app/python-tools/survey_bucket_encryption/survey_bucket_encryption.py +++ b/local-app/python-tools/survey_bucket_encryption/survey_bucket_encryption.py @@ -1,236 +1,353 @@ -import boto3 import argparse import configparser import io import json -import pandas as pd import os import threading +from datetime import date, datetime + +import boto3 +import pandas as pd -from datetime import datetime, date def json_serial(obj): - """JSON serializer for objects not serializable by default json code""" - if isinstance(obj, (datetime, date)): - return obj.isoformat() - raise TypeError ("Type %s not serializable" % type(obj)) + """JSON serializer for objects not serializable by default json code""" + if isinstance(obj, (datetime, date)): + return obj.isoformat() + raise TypeError("Type %s not serializable" % type(obj)) + def main(): - parser = argparse.ArgumentParser(description='Connect to AWS accounts') - parser.add_argument('--profile', help='AWS profile') - parser.add_argument('--config', help='Config file') - parser.add_argument('--me', help='Run using available credentials', - action='store_true') - parser.add_argument('--assume', help='Run using assumed roles', - action='store_true') - parser.add_argument('--region', help='A region to run against') - parser.add_argument('--read', help = 'Read API calls from cached results', - action='store_true') - parser.add_argument('--write', help = 'Write API calls to cached results', - action='store_true') - parser.add_argument('--run_token', help ='ID for cached results to read and/or write') - - args = parser.parse_args() - - run_token = None - - if args.read: - read_from_cache = True - run_token = args.run_token - else: - read_from_cache = False - - if args.write: - write_to_cache = True - run_token = args.run_token - else: - write_to_cache = False - - if(args.config): - configParser = configparser.RawConfigParser(allow_no_value=True) + parser = argparse.ArgumentParser(description="Connect to AWS accounts") + parser.add_argument("--profile", help="AWS profile") + parser.add_argument("--config", help="Config file") + parser.add_argument( + "--me", help="Run using available credentials", action="store_true" + ) + parser.add_argument("--assume", help="Run using assumed roles", action="store_true") + parser.add_argument("--region", help="A region to run against") + parser.add_argument( + "--read", help="Read API calls from cached results", action="store_true" + ) + parser.add_argument( + "--write", help="Write API calls to cached results", action="store_true" + ) + parser.add_argument( + "--run_token", help="ID for cached results to read and/or write" + ) + + args = parser.parse_args() + + run_token = None + + if args.read: + read_from_cache = True + run_token = args.run_token + else: + read_from_cache = False + + if args.write: + write_to_cache = True + run_token = args.run_token + else: + write_to_cache = False + if args.config: - with open(args.config) as file: - configParser.read_file(file) - - if args.profile: - profiles = [args.profile] - elif args.config: - profiles = [item[0] for item in configParser.items("roles")] - profile_roles = {item[0]:item[1] for item in configParser.items("roles")} - else: - profiles = ['dev'] - - if args.region: - regions = [args.region] - elif args.config: - regions = [item[1] for item in configParser.items("regions")] - else: - regions =['us-gov-west-1'] - - if args.me: - use_roles = False - elif args.assume: - use_roles = True - else: - print ('Inconsistent configuration. Exiting. Use one of "me" or "assume".') - exit() - - session = {} - - headers = ['account','region','bucket_name','encryption_type', 'key', 'bucket_key_enabled'] - - master_list_of_lists = [] - - thread_list = [] - - for profile in profiles: - for region in regions: - profile_region = f'{profile}_{region}' - if not use_roles: - session[profile_region] = boto3.Session(profile_name = profile, region_name = region) - else: - role = profile_roles[profile] - stsClient = boto3.client('sts') - response = stsClient.assume_role(RoleArn=role, RoleSessionName='AWSConnect',ExternalId='ti-jbr-2010') - accessKeyId = response['Credentials']['AccessKeyId'] - secretAccessKey = response['Credentials']['SecretAccessKey'] - sessionToken = response['Credentials']['SessionToken'] - session[profile_region] = boto3.Session(aws_access_key_id = accessKeyId, - aws_secret_access_key = secretAccessKey, - aws_session_token = sessionToken) - for profile in profiles: - account = profile.split('-')[0] - print(f'account [{account}]') - for region in regions: - profile_region = f'{profile}_{region}' - boto_client_ec2 = session[profile_region].client('ec2') - boto_client_s3 = session[profile_region].client('s3') - method_token = 'list_buckets' - # thread = threading.Thread(name=profile_region+'_'+method_token, target = tabulate_ec2s, args = (account, region, boto_client_ec2, profile_region, method_token, master_list_of_lists, read_from_cache, write_to_cache, run_token)) - thread = threading.Thread(name=profile_region+'_'+method_token, target = tabulate_bucket_encryption, args = (account, region, boto_client_s3, profile_region, method_token, master_list_of_lists, read_from_cache, write_to_cache, run_token)) - thread.start() - thread_list.append(thread) - #tabulate_ec2s(account, region, boto_client_ec2, profile_region, method_token, master_list_of_lists, read_from_cache, write_to_cache, run_token) - - for t in thread_list: - t.join() - df = data_frame(headers,master_list_of_lists) - print (df) - excel_name = ''.join(['./excel/key_attributes_',datetime.now().strftime('%Y%m%d%H%M%S'),'.xlsx']) - writer_keys = pd.ExcelWriter(excel_name, engine='xlsxwriter') - df.to_excel(writer_keys,sheet_name='key_attributes') - writer_keys.save() - print(f'main: wrote excel file of key details to disk') - -def tabulate_ec2s(account, region, boto_client, profile_region, method_token, master_list_of_lists, read, write, user_token): - response = paginate_wrapper(read = read, write = write, client = boto_client, client_token = f'ec2_{profile_region}', method_token = 'describe_instances', run_token =user_token) - print ('profile_region[{0}]'.format(profile_region)) - list_of_lists = flatten(account, region, response) - master_list_of_lists.extend(list_of_lists) - -def tabulate_bucket_encryption(account, region, boto_client_s3, profile_region, method_token, master_list_of_lists, read, write, user_token): - response = paginate_wrapper(read = read, write = write, client = boto_client_s3, client_token = f's3_{profile_region}', method_token = 'list_buckets', run_token =user_token) - print ('profile_region[{0}]'.format(profile_region)) - list_of_lists = flatten_buckets(account, region, response) - for bucket_entry in list_of_lists: - bucket_name = bucket_entry[2] - print(f'profile_region [{profile_region}] bucket_name[{bucket_name}]') - parameters = {'Bucket':bucket_name} - try: - bucket_response = paginate_wrapper(read = read, write=write, client = boto_client_s3, client_token = f's3-{profile_region}', method_token = 'get_bucket_encryption', run_token = user_token, parameter_dict= parameters, parameter_token = bucket_name) - except: - bucket_response = [{'ServerSideEncryptionConfiguration':{'Rules':[{'BucketKeyEnabled':'API Error','ApplyServerSideEncryptionByDefault':{'SSEAlgorithm':"API Error",'KMSMasterKeyId':'API Error'}}]}}] - encryption_configuration = bucket_response[0].get('ServerSideEncryptionConfiguration',{}) - - rules = encryption_configuration.get('Rules',[]) - if len(rules) > 0: - sse_default = rules[0].get('ApplyServerSideEncryptionByDefault',{}) - encryption_type = sse_default.get('SSEAlgorithm',"None") - key = sse_default.get('KMSMasterKeyID',"None") - bucket_key_enabled = rules[0].get('BucketKeyEnabled', "NoData") + configParser = configparser.RawConfigParser(allow_no_value=True) + if args.config: + with open(args.config) as file: + configParser.read_file(file) + + if args.profile: + profiles = [args.profile] + elif args.config: + profiles = [item[0] for item in configParser.items("roles")] + profile_roles = {item[0]: item[1] for item in configParser.items("roles")} else: - encryption_type = "No Rules" - key = "No Rules" - bucket_key_enabled = "No Rules" - bucket_entry.extend([encryption_type, key, bucket_key_enabled]) - master_list_of_lists.extend(list_of_lists) + profiles = ["dev"] + + if args.region: + regions = [args.region] + elif args.config: + regions = [item[1] for item in configParser.items("regions")] + else: + regions = ["us-gov-west-1"] + + if args.me: + use_roles = False + elif args.assume: + use_roles = True + else: + print('Inconsistent configuration. Exiting. Use one of "me" or "assume".') + exit() + + session = {} + + headers = [ + "account", + "region", + "bucket_name", + "encryption_type", + "key", + "bucket_key_enabled", + ] + + master_list_of_lists = [] + + thread_list = [] + + for profile in profiles: + for region in regions: + profile_region = f"{profile}_{region}" + if not use_roles: + session[profile_region] = boto3.Session( + profile_name=profile, region_name=region + ) + else: + role = profile_roles[profile] + stsClient = boto3.client("sts") + response = stsClient.assume_role( + RoleArn=role, RoleSessionName="AWSConnect", ExternalId="ti-jbr-2010" + ) + accessKeyId = response["Credentials"]["AccessKeyId"] + secretAccessKey = response["Credentials"]["SecretAccessKey"] + sessionToken = response["Credentials"]["SessionToken"] + session[profile_region] = boto3.Session( + aws_access_key_id=accessKeyId, + aws_secret_access_key=secretAccessKey, + aws_session_token=sessionToken, + ) + for profile in profiles: + account = profile.split("-")[0] + print(f"account [{account}]") + for region in regions: + profile_region = f"{profile}_{region}" + boto_client_ec2 = session[profile_region].client("ec2") + boto_client_s3 = session[profile_region].client("s3") + method_token = "list_buckets" + # thread = threading.Thread(name=profile_region+'_'+method_token, target = tabulate_ec2s, args = (account, region, boto_client_ec2, profile_region, method_token, master_list_of_lists, read_from_cache, write_to_cache, run_token)) + thread = threading.Thread( + name=profile_region + "_" + method_token, + target=tabulate_bucket_encryption, + args=( + account, + region, + boto_client_s3, + profile_region, + method_token, + master_list_of_lists, + read_from_cache, + write_to_cache, + run_token, + ), + ) + thread.start() + thread_list.append(thread) + # tabulate_ec2s(account, region, boto_client_ec2, profile_region, method_token, master_list_of_lists, read_from_cache, write_to_cache, run_token) + + for t in thread_list: + t.join() + df = data_frame(headers, master_list_of_lists) + print(df) + excel_name = "".join( + ["./excel/key_attributes_", datetime.now().strftime("%Y%m%d%H%M%S"), ".xlsx"] + ) + writer_keys = pd.ExcelWriter(excel_name, engine="xlsxwriter") + df.to_excel(writer_keys, sheet_name="key_attributes") + writer_keys.save() + print(f"main: wrote excel file of key details to disk") + + +def tabulate_ec2s( + account, + region, + boto_client, + profile_region, + method_token, + master_list_of_lists, + read, + write, + user_token, +): + response = paginate_wrapper( + read=read, + write=write, + client=boto_client, + client_token=f"ec2_{profile_region}", + method_token="describe_instances", + run_token=user_token, + ) + print("profile_region[{0}]".format(profile_region)) + list_of_lists = flatten(account, region, response) + master_list_of_lists.extend(list_of_lists) + + +def tabulate_bucket_encryption( + account, + region, + boto_client_s3, + profile_region, + method_token, + master_list_of_lists, + read, + write, + user_token, +): + response = paginate_wrapper( + read=read, + write=write, + client=boto_client_s3, + client_token=f"s3_{profile_region}", + method_token="list_buckets", + run_token=user_token, + ) + print("profile_region[{0}]".format(profile_region)) + list_of_lists = flatten_buckets(account, region, response) + for bucket_entry in list_of_lists: + bucket_name = bucket_entry[2] + print(f"profile_region [{profile_region}] bucket_name[{bucket_name}]") + parameters = {"Bucket": bucket_name} + try: + bucket_response = paginate_wrapper( + read=read, + write=write, + client=boto_client_s3, + client_token=f"s3-{profile_region}", + method_token="get_bucket_encryption", + run_token=user_token, + parameter_dict=parameters, + parameter_token=bucket_name, + ) + except: + bucket_response = [ + { + "ServerSideEncryptionConfiguration": { + "Rules": [ + { + "BucketKeyEnabled": "API Error", + "ApplyServerSideEncryptionByDefault": { + "SSEAlgorithm": "API Error", + "KMSMasterKeyId": "API Error", + }, + } + ] + } + } + ] + encryption_configuration = bucket_response[0].get( + "ServerSideEncryptionConfiguration", {} + ) + + rules = encryption_configuration.get("Rules", []) + if len(rules) > 0: + sse_default = rules[0].get("ApplyServerSideEncryptionByDefault", {}) + encryption_type = sse_default.get("SSEAlgorithm", "None") + key = sse_default.get("KMSMasterKeyID", "None") + bucket_key_enabled = rules[0].get("BucketKeyEnabled", "NoData") + else: + encryption_type = "No Rules" + key = "No Rules" + bucket_key_enabled = "No Rules" + bucket_entry.extend([encryption_type, key, bucket_key_enabled]) + master_list_of_lists.extend(list_of_lists) -def flatten_buckets(account, region, response): - list_of_lists = [] - for page in response: - for bucket in page['Buckets']: - list = [account,region, bucket['Name']] - list_of_lists.append(list) - return list_of_lists +def flatten_buckets(account, region, response): + list_of_lists = [] + for page in response: + for bucket in page["Buckets"]: + list = [account, region, bucket["Name"]] + list_of_lists.append(list) + return list_of_lists def data_frame(headers, list_of_lists): - df = pd.DataFrame(columns = headers, data=list_of_lists) + df = pd.DataFrame(columns=headers, data=list_of_lists) return df -def paginate_wrapper(read=None, write=None, client=None, client_token=None, run_token=None, method_token='default', parameter_dict = {},parameter_token=''): - # print(f'paginate_wrapper: processing client_token[{client_token}] method_token [{method_token}] run_token[{run_token}]') - # if neither read nor write is specified, then look for a serialization. If it doesn't exist, then write - normalized_parameter_token = parameter_token.translate({ord(':'):'-'}) - if read and write: - if serialization_exists([client_token, method_token, run_token,normalized_parameter_token]): - read_action = True - write_action = False +def paginate_wrapper( + read=None, + write=None, + client=None, + client_token=None, + run_token=None, + method_token="default", + parameter_dict={}, + parameter_token="", +): + # print(f'paginate_wrapper: processing client_token[{client_token}] method_token [{method_token}] run_token[{run_token}]') + # if neither read nor write is specified, then look for a serialization. If it doesn't exist, then write + normalized_parameter_token = parameter_token.translate({ord(":"): "-"}) + if read and write: + if serialization_exists( + [client_token, method_token, run_token, normalized_parameter_token] + ): + read_action = True + write_action = False + else: + write_action = True + read_action = False else: - write_action = True - read_action = False - else: - read_action = read - write_action = write - if read_action: - result = deserialize([client_token, method_token,run_token,normalized_parameter_token]) - else: - if client.can_paginate(method_token): - paginator = client.get_paginator(method_token) - # print(f'paginate_wrapper: method_token[{method_token}] keys in parameter_dict [{parameter_dict.keys()}]') - page_iterator = paginator.paginate(**parameter_dict) - result = [page for page in page_iterator] + read_action = read + write_action = write + if read_action: + result = deserialize( + [client_token, method_token, run_token, normalized_parameter_token] + ) else: - func = getattr(client,method_token) - # print(f'paginate_wrapper: method_token [{method_token}] keys in parameter_dict [{parameter_dict.keys()}]') - # parameter_string = ', '.join([f'parameter[{parameter}] value [{parameter_dict[parameter]}]' for parameter in parameter_dict]) - # print(f'paginate_wrapper: parameter_string <{parameter_string}>') - result = [func(**parameter_dict)] - # print(f'paginate_wrapper: result [{result}]') + if client.can_paginate(method_token): + paginator = client.get_paginator(method_token) + # print(f'paginate_wrapper: method_token[{method_token}] keys in parameter_dict [{parameter_dict.keys()}]') + page_iterator = paginator.paginate(**parameter_dict) + result = [page for page in page_iterator] + else: + func = getattr(client, method_token) + # print(f'paginate_wrapper: method_token [{method_token}] keys in parameter_dict [{parameter_dict.keys()}]') + # parameter_string = ', '.join([f'parameter[{parameter}] value [{parameter_dict[parameter]}]' for parameter in parameter_dict]) + # print(f'paginate_wrapper: parameter_string <{parameter_string}>') + result = [func(**parameter_dict)] + # print(f'paginate_wrapper: result [{result}]') + + if write_action: + serialize( + result, + [client_token, method_token, run_token, normalized_parameter_token], + ) + return result - if write_action: - serialize(result, [client_token, method_token, run_token,normalized_parameter_token]) - return result def serialize(obj, token_list): - serialized = json.dumps(obj, default=json_serial) - filename = build_json_filename(token_list) - with open(filename, 'w') as file: - file.write(serialized) + serialized = json.dumps(obj, default=json_serial) + filename = build_json_filename(token_list) + with open(filename, "w") as file: + file.write(serialized) + def deserialize(token_list): - filename = build_json_filename(token_list) - with open(filename) as file: - serialized = file.read() - deserialized = json.loads(serialized) - return deserialized + filename = build_json_filename(token_list) + with open(filename) as file: + serialized = file.read() + deserialized = json.loads(serialized) + return deserialized + def build_json_filename(string_list): - joined_list = '-'.join(string_list) - return f'c:/cache/json/{joined_list}.json' + joined_list = "-".join(string_list) + return f"c:/cache/json/{joined_list}.json" + def serialization_exists(token_list): - filename = build_json_filename(token_list) - return os.path.isfile(filename) + filename = build_json_filename(token_list) + return os.path.isfile(filename) + def json_serial(obj): - """JSON serializer for objects not serializable by default json code""" - if isinstance(obj, (datetime, date)): - return obj.isoformat() - raise TypeError ("Type %s not serializable" % type(obj)) + """JSON serializer for objects not serializable by default json code""" + if isinstance(obj, (datetime, date)): + return obj.isoformat() + raise TypeError("Type %s not serializable" % type(obj)) if __name__ == "__main__": - main() + main() diff --git a/local-app/python-tools/utility/594.py b/local-app/python-tools/utility/594.py index 09fa19fc..dba06241 100644 --- a/local-app/python-tools/utility/594.py +++ b/local-app/python-tools/utility/594.py @@ -1,7 +1,8 @@ import sys -import boto3 from datetime import datetime +import boto3 + AWS_REGION = "us-gov-east-1" ec2 = boto3.client("ec2", region_name=AWS_REGION) @@ -15,7 +16,7 @@ "csvd-sple-mc001.compute.csp1.census.gov", "csvd-sple-cm001.compute.csp1.census.gov", "csvd-sple-ds001.compute.csp1.census.gov", - "csvd-sple-dp001.compute.csp1.census.gov" + "csvd-sple-dp001.compute.csp1.census.gov", ] instance_string = str(instance_array) @@ -26,34 +27,36 @@ for volume in volume_array: volume_id = volume["VolumeId"] size = volume["Size"] - volumeMap[volume_id]=volume + volumeMap[volume_id] = volume # get all instance reservations reservation_array = ec2.describe_instances()["Reservations"] -volume_list=[] +volume_list = [] for reservation in reservation_array: - # loop over each instance in the reservation - for instance in reservation["Instances"]: - - instanceNAME='' - - # get the tags and find the start/stop/weekend tags present on the server - tags = instance["Tags"] - for tag in tags: - if tag["Key"] == "Name": - instanceNAME = tag["Value"] - - instance_in_list = instanceNAME in instance_string - if instanceNAME != '' and instance_in_list: - print(instanceNAME) - - block_devices = instance["BlockDeviceMappings"] - for block in block_devices: - device_name = block["DeviceName"] - volume_id = block["Ebs"]["VolumeId"] - - size = volumeMap[volume_id]["Size"] - type = volumeMap[volume_id]["VolumeType"] - print('\t'+ device_name +' '+ volume_id +" "+ str(size) +" " + type) + # loop over each instance in the reservation + for instance in reservation["Instances"]: + + instanceNAME = "" + + # get the tags and find the start/stop/weekend tags present on the server + tags = instance["Tags"] + for tag in tags: + if tag["Key"] == "Name": + instanceNAME = tag["Value"] + + instance_in_list = instanceNAME in instance_string + if instanceNAME != "" and instance_in_list: + print(instanceNAME) + + block_devices = instance["BlockDeviceMappings"] + for block in block_devices: + device_name = block["DeviceName"] + volume_id = block["Ebs"]["VolumeId"] + + size = volumeMap[volume_id]["Size"] + type = volumeMap[volume_id]["VolumeType"] + print( + "\t" + device_name + " " + volume_id + " " + str(size) + " " + type + ) diff --git a/local-app/python-tools/utility/ami-report.py b/local-app/python-tools/utility/ami-report.py index 0c9d900c..995de85c 100644 --- a/local-app/python-tools/utility/ami-report.py +++ b/local-app/python-tools/utility/ami-report.py @@ -1,10 +1,11 @@ -import boto3 -from datetime import datetime import argparse import subprocess +from datetime import datetime + +import boto3 argParser = argparse.ArgumentParser() -#argParser.add_argument("-c", "--cluster", help="specific cluster name to research - default is all clusters") +# argParser.add_argument("-c", "--cluster", help="specific cluster name to research - default is all clusters") args = argParser.parse_args() @@ -13,130 +14,130 @@ # load account list accountList = [] -with open('account-list.txt', 'r') as dat: - accountList = dat.readlines() +with open("account-list.txt", "r") as dat: + accountList = dat.readlines() regionList = ["us-gov-east-1", "us-gov-west-1"] # gather info on AMI -account="107742151971-do2-govcloud" +account = "107742151971-do2-govcloud" session = boto3.Session(profile_name=account) -permitted_map={} -kms_permissions_map={} +permitted_map = {} +kms_permissions_map = {} for region in regionList: - ec2 = session.client("ec2", region_name=region) - kms = session.client("kms", region_name=region) + ec2 = session.client("ec2", region_name=region) + kms = session.client("kms", region_name=region) - images = ec2.describe_images( - Owners=['self'], + images = ec2.describe_images( + Owners=["self"], Filters=[ - { - 'Name': 'name', - 'Values': [ - 'RHEL 8 Base OS', - ] - }, - ] - )["Images"] - - image=images[0] - ami_id = image["ImageId"] - snapshot_id = image["BlockDeviceMappings"][0]["Ebs"]["SnapshotId"] - - snapshots = ec2.describe_snapshots( SnapshotIds=[ snapshot_id ] )["Snapshots"] - - kms_key_id=snapshots[0]["KmsKeyId"] - - response = ec2.describe_image_attribute( - Attribute='launchPermission', - ImageId=ami_id, - ) - launch_permissions=response["LaunchPermissions"] - permitted_list = [] - for lp in launch_permissions: - permitted_list.append(lp["UserId"]) - - # pack the map by region with the list of accounts that can use this ami - permitted_map[region]=permitted_list - -#"Allow attachment of persistent resources" -#"Allow use of the key" - response = kms.get_key_policy( - KeyId=kms_key_id, - PolicyName='default', - ) - policy = response["Policy"] - policy_list = [y for y in (x.strip() for x in policy.splitlines()) if y] - block1AWS=False - block2AWS=False - block1_policy_accts_list = [] - block2_policy_accts_list = [] - block_permissions = {} - - for line in policy_list: - if line.find("Allow use of the key") > 0: - block1AWS = True - - if line.find("Allow attachment of persistent resources") > 0: - block2AWS = True - - if line.find("[") > 0 and block1AWS == True: - start=line.find("[") + 1 - end = line.find("]") - policy_accts = line[start:end] - block1_policy_accts_list = policy_accts.split() - fixed_list=[] - for entry in block1_policy_accts_list: - last_colon = entry.rindex(":") - new_entry = entry[:last_colon] - last_colon = new_entry.rindex(":") - new_entry = new_entry[last_colon+1:] - fixed_list.append(new_entry) - - block1AWS = False - block_permissions["b1"] = fixed_list - - if line.find("[") > 0 and block2AWS == True: - start=line.find("[") + 1 - end = line.find("]") - policy_accts = line[start:end] - block2_policy_accts_list = policy_accts.split() - for entry in block2_policy_accts_list: - last_colon = entry.rindex(":") - new_entry = entry[:last_colon] - last_colon = new_entry.rindex(":") - new_entry = new_entry[last_colon+1:] - fixed_list.append(new_entry) - - block2AWS = False - block_permissions["b2"] = fixed_list - - kms_permissions_map[region] = block_permissions - -#print(kms_permissions_map) -#print(permitted_map) + { + "Name": "name", + "Values": [ + "RHEL 8 Base OS", + ], + }, + ], + )["Images"] + + image = images[0] + ami_id = image["ImageId"] + snapshot_id = image["BlockDeviceMappings"][0]["Ebs"]["SnapshotId"] + + snapshots = ec2.describe_snapshots(SnapshotIds=[snapshot_id])["Snapshots"] + + kms_key_id = snapshots[0]["KmsKeyId"] + + response = ec2.describe_image_attribute( + Attribute="launchPermission", + ImageId=ami_id, + ) + launch_permissions = response["LaunchPermissions"] + permitted_list = [] + for lp in launch_permissions: + permitted_list.append(lp["UserId"]) + + # pack the map by region with the list of accounts that can use this ami + permitted_map[region] = permitted_list + + # "Allow attachment of persistent resources" + # "Allow use of the key" + response = kms.get_key_policy( + KeyId=kms_key_id, + PolicyName="default", + ) + policy = response["Policy"] + policy_list = [y for y in (x.strip() for x in policy.splitlines()) if y] + block1AWS = False + block2AWS = False + block1_policy_accts_list = [] + block2_policy_accts_list = [] + block_permissions = {} + + for line in policy_list: + if line.find("Allow use of the key") > 0: + block1AWS = True + + if line.find("Allow attachment of persistent resources") > 0: + block2AWS = True + + if line.find("[") > 0 and block1AWS == True: + start = line.find("[") + 1 + end = line.find("]") + policy_accts = line[start:end] + block1_policy_accts_list = policy_accts.split() + fixed_list = [] + for entry in block1_policy_accts_list: + last_colon = entry.rindex(":") + new_entry = entry[:last_colon] + last_colon = new_entry.rindex(":") + new_entry = new_entry[last_colon + 1 :] + fixed_list.append(new_entry) + + block1AWS = False + block_permissions["b1"] = fixed_list + + if line.find("[") > 0 and block2AWS == True: + start = line.find("[") + 1 + end = line.find("]") + policy_accts = line[start:end] + block2_policy_accts_list = policy_accts.split() + for entry in block2_policy_accts_list: + last_colon = entry.rindex(":") + new_entry = entry[:last_colon] + last_colon = new_entry.rindex(":") + new_entry = new_entry[last_colon + 1 :] + fixed_list.append(new_entry) + + block2AWS = False + block_permissions["b2"] = fixed_list + + kms_permissions_map[region] = block_permissions + +# print(kms_permissions_map) +# print(permitted_map) regionList = ["us-gov-east-1", "us-gov-west-1"] account_data = {} for acct in accountList: - account = acct.strip() + account = acct.strip() - account_data["account"] = account - account_number = account[0:account.index("-") ] + account_data["account"] = account + account_number = account[0 : account.index("-")] - for region in regionList: - in_permitted_map = permitted_map[region].count(account_number) - account_data["ami-"+region] = in_permitted_map + for region in regionList: + in_permitted_map = permitted_map[region].count(account_number) + account_data["ami-" + region] = in_permitted_map - kms_perms = kms_permissions_map[region] - b1=kms_perms["b1"] - in_permitted_map = b1.count(account_number) - account_data["kms-"+region+"-b1"]=in_permitted_map + kms_perms = kms_permissions_map[region] + b1 = kms_perms["b1"] + in_permitted_map = b1.count(account_number) + account_data["kms-" + region + "-b1"] = in_permitted_map - b2=kms_perms["b2"] - in_permitted_map = b2.count(account_number) - account_data["kms-"+region+"-b2"]=in_permitted_map + b2 = kms_perms["b2"] + in_permitted_map = b2.count(account_number) + account_data["kms-" + region + "-b2"] = in_permitted_map - print(account_data) + print(account_data) diff --git a/local-app/python-tools/utility/arginfo.py b/local-app/python-tools/utility/arginfo.py index d5210c40..91f86cf3 100644 --- a/local-app/python-tools/utility/arginfo.py +++ b/local-app/python-tools/utility/arginfo.py @@ -1,20 +1,20 @@ import sys - + # total arguments n = len(sys.argv) print("Total arguments passed:", n) - + # Arguments passed print("\nName of Python script:", sys.argv[0]) - -print("\nArguments passed:", end = " ") + +print("\nArguments passed:", end=" ") for i in range(1, n): - print(sys.argv[i], end = " ") - + print(sys.argv[i], end=" ") + # Addition of numbers Sum = 0 # Using argparse module for i in range(1, n): Sum += int(sys.argv[i]) - + print("\n\nResult:", Sum) diff --git a/local-app/python-tools/utility/cluster-report.py b/local-app/python-tools/utility/cluster-report.py index 7d911fc6..ca1ee97e 100644 --- a/local-app/python-tools/utility/cluster-report.py +++ b/local-app/python-tools/utility/cluster-report.py @@ -1,151 +1,208 @@ -import boto3 -from datetime import datetime import argparse +from datetime import datetime + +import boto3 argParser = argparse.ArgumentParser() argParser.add_argument("-p", "--profile", help="sso-profile to use") -argParser.add_argument("-r", "--region", help="[us-gov-west-1|us-gov-east-1] DEFAULT=us-gov-east-1") -argParser.add_argument("-c", "--cluster", help="specific cluster name to research - default is all clusters") +argParser.add_argument( + "-r", "--region", help="[us-gov-west-1|us-gov-east-1] DEFAULT=us-gov-east-1" +) +argParser.add_argument( + "-c", + "--cluster", + help="specific cluster name to research - default is all clusters", +) args = argParser.parse_args() # set region from command line args AWS_REGION = "us-gov-east-1" if args.region: - AWS_REGION = args.region + AWS_REGION = args.region if args.profile: - boto3.setup_default_session(profile_name=args.profile) + boto3.setup_default_session(profile_name=args.profile) ## GET THE EKS Clusters -autoscaling = boto3.client('autoscaling', region_name=AWS_REGION) +autoscaling = boto3.client("autoscaling", region_name=AWS_REGION) eks = boto3.client("eks", region_name=AWS_REGION) ec2 = boto3.client("ec2", region_name=AWS_REGION) -cluster_list = eks.list_clusters()["clusters"]; +cluster_list = eks.list_clusters()["clusters"] print("Resource,Identifier,Project Name,Cost Allocation,Project Number,Misc,Misc,Misc") for cluster in cluster_list: - clusterName='' - projectName='NOT TAGGED' - costAllocation='NOT TAGGED' - projectNumber='NOT TAGGED' - environment='NOT TAGGED' + clusterName = "" + projectName = "NOT TAGGED" + costAllocation = "NOT TAGGED" + projectNumber = "NOT TAGGED" + environment = "NOT TAGGED" response = eks.describe_cluster(name=cluster) version = response["cluster"]["version"] - tags=response["cluster"]["tags"] - cluster_name=tags["eks-cluster-name"] + tags = response["cluster"]["tags"] + cluster_name = tags["eks-cluster-name"] if args.cluster: if cluster_name != args.cluster: - continue; + continue for tag in tags: - if tag == "eks-cluster-name": - clusterName = tags[tag] - if tag == "Project Name": - projectName = tags[tag] - if tag == "CostAllocation": - costAllocation = tags[tag] - if tag == "ProjectNumber": - projectNumber = tags[tag] - if tag == "Environment": - environment = tags[tag] - - print("Cluster,"+ cluster_name +"," + projectName+"," + costAllocation +","+ projectNumber+","+ environment +","+ version) - - ## Get the nodegroups - response = eks.list_nodegroups( clusterName=clusterName) - nodeGroups = response["nodegroups"] - - for nodeGroup in nodeGroups: - response = eks.describe_nodegroup(clusterName=clusterName, nodegroupName=nodeGroup) - nodeGroup = response["nodegroup"] - - projectName='NOT TAGGED' - costAllocation='NOT TAGGED' - projectNumber='NOT TAGGED' - - nodeGroupName = nodeGroup["nodegroupName"] - scalingConfig = nodeGroup["scalingConfig"] - - tags = nodeGroup["tags"] - for tag in tags: + if tag == "eks-cluster-name": + clusterName = tags[tag] if tag == "Project Name": - projectName = tags[tag] + projectName = tags[tag] if tag == "CostAllocation": - costAllocation = tags[tag] + costAllocation = tags[tag] if tag == "ProjectNumber": - projectNumber = tags[tag] - - print("NodeGroup," + nodeGroupName +","+ projectName +"," + costAllocation +","+ projectNumber) + projectNumber = tags[tag] + if tag == "Environment": + environment = tags[tag] + + print( + "Cluster," + + cluster_name + + "," + + projectName + + "," + + costAllocation + + "," + + projectNumber + + "," + + environment + + "," + + version + ) - autoscalingGroups = nodeGroup["resources"]["autoScalingGroups"] - -## GET the Security Groups - -## Get the Route53 Domains + ## Get the nodegroups + response = eks.list_nodegroups(clusterName=clusterName) + nodeGroups = response["nodegroups"] - for autoscalingGroup in autoscalingGroups: - response = autoscaling.describe_auto_scaling_groups( - AutoScalingGroupNames=[ - autoscalingGroup["name"], - ] + for nodeGroup in nodeGroups: + response = eks.describe_nodegroup( + clusterName=clusterName, nodegroupName=nodeGroup + ) + nodeGroup = response["nodegroup"] + + projectName = "NOT TAGGED" + costAllocation = "NOT TAGGED" + projectNumber = "NOT TAGGED" + + nodeGroupName = nodeGroup["nodegroupName"] + scalingConfig = nodeGroup["scalingConfig"] + + tags = nodeGroup["tags"] + for tag in tags: + if tag == "Project Name": + projectName = tags[tag] + if tag == "CostAllocation": + costAllocation = tags[tag] + if tag == "ProjectNumber": + projectNumber = tags[tag] + + print( + "NodeGroup," + + nodeGroupName + + "," + + projectName + + "," + + costAllocation + + "," + + projectNumber ) - all_instances = [] - for group in response["AutoScalingGroups"]: - for instance in group["Instances"]: - all_instances.append(instance["InstanceId"]) - - for instance in all_instances: - - response = ec2.describe_instances( Filters=[ { 'Name': 'instance-id', 'Values': [ instance ] } ]) - - for reservation in response["Reservations"]: - for instance in reservation["Instances"]: - - instanceId = instance["InstanceId"] - - projectName='NOT TAGGED' - costAllocation='NOT TAGGED' - projectNumber='NOT TAGGED' - - tags = instance["Tags"] - for tag in tags: - if tag["Key"] == "Project Name": - projectName = tag["Value"] - if tag["Key"] == "CostAllocation": - costAllocation = tag["Value"] - if tag["Key"] == "ProjectNumber": - projectNumber = tag["Value"] - - print("Instance," + instanceId +","+ projectName +"," + costAllocation +","+ projectNumber +","+ cluster_name) - - volumes = instance["BlockDeviceMappings"] - for volume in volumes: - volumeId = volume["Ebs"]["VolumeId"] - response = ec2.describe_volumes( Filters=[ { 'Name': 'volume-id', 'Values': [ volumeId ] } ]) - - for volume in response["Volumes"]: - - projectName='NOT TAGGED' - costAllocation='NOT TAGGED' - projectNumber='NOT TAGGED' - - tags = volume["Tags"] - for tag in tags: - if tag["Key"] == "Project Name": - projectName = tag["Value"] - if tag["Key"] == "CostAllocation": - costAllocation = tag["Value"] - if tag["Key"] == "ProjectNumber": - projectNumber = tag["Value"] - - print("Volume," + volumeId +","+ projectName +"," + costAllocation +","+ projectNumber +","+ cluster_name) + autoscalingGroups = nodeGroup["resources"]["autoScalingGroups"] + + ## GET the Security Groups + + ## Get the Route53 Domains + + for autoscalingGroup in autoscalingGroups: + response = autoscaling.describe_auto_scaling_groups( + AutoScalingGroupNames=[ + autoscalingGroup["name"], + ] + ) + + all_instances = [] + for group in response["AutoScalingGroups"]: + for instance in group["Instances"]: + all_instances.append(instance["InstanceId"]) + + for instance in all_instances: + + response = ec2.describe_instances( + Filters=[{"Name": "instance-id", "Values": [instance]}] + ) + + for reservation in response["Reservations"]: + for instance in reservation["Instances"]: + + instanceId = instance["InstanceId"] + + projectName = "NOT TAGGED" + costAllocation = "NOT TAGGED" + projectNumber = "NOT TAGGED" + + tags = instance["Tags"] + for tag in tags: + if tag["Key"] == "Project Name": + projectName = tag["Value"] + if tag["Key"] == "CostAllocation": + costAllocation = tag["Value"] + if tag["Key"] == "ProjectNumber": + projectNumber = tag["Value"] + + print( + "Instance," + + instanceId + + "," + + projectName + + "," + + costAllocation + + "," + + projectNumber + + "," + + cluster_name + ) + + volumes = instance["BlockDeviceMappings"] + for volume in volumes: + volumeId = volume["Ebs"]["VolumeId"] + response = ec2.describe_volumes( + Filters=[{"Name": "volume-id", "Values": [volumeId]}] + ) + + for volume in response["Volumes"]: + + projectName = "NOT TAGGED" + costAllocation = "NOT TAGGED" + projectNumber = "NOT TAGGED" + + tags = volume["Tags"] + for tag in tags: + if tag["Key"] == "Project Name": + projectName = tag["Value"] + if tag["Key"] == "CostAllocation": + costAllocation = tag["Value"] + if tag["Key"] == "ProjectNumber": + projectNumber = tag["Value"] + + print( + "Volume," + + volumeId + + "," + + projectName + + "," + + costAllocation + + "," + + projectNumber + + "," + + cluster_name + ) exit(0) @@ -164,12 +221,12 @@ for volume in volume_array: if len(volume["Attachments"]) > 0: - key = volume["Attachments"][0]["InstanceId"] - volume_id = volume["VolumeId"] + key = volume["Attachments"][0]["InstanceId"] + volume_id = volume["VolumeId"] - vol_list = instanceIdToVolumeIdList.get(key) - if not vol_list: - vol_list=[] + vol_list = instanceIdToVolumeIdList.get(key) + if not vol_list: + vol_list = [] - vol_list.append(volume_id) - instanceIdToVolumeIdList[key] = vol_list + vol_list.append(volume_id) + instanceIdToVolumeIdList[key] = vol_list diff --git a/local-app/python-tools/utility/ead.py b/local-app/python-tools/utility/ead.py index 2b089fb4..12aa4bf4 100644 --- a/local-app/python-tools/utility/ead.py +++ b/local-app/python-tools/utility/ead.py @@ -1,6 +1,7 @@ -import boto3 from datetime import datetime +import boto3 + AWS_REGION = "us-gov-west-1" ec2 = boto3.client("ec2", region_name=AWS_REGION) @@ -11,28 +12,28 @@ for volume in volume_array: volume_id = volume["VolumeId"] - volumeMap[volume_id]=volume + volumeMap[volume_id] = volume # get all instance reservations reservation_array = ec2.describe_instances()["Reservations"] -volume_list=[] +volume_list = [] for reservation in reservation_array: - # loop over each instance in the reservation - for instance in reservation["Instances"]: + # loop over each instance in the reservation + for instance in reservation["Instances"]: - instance_id=instance["InstanceId"] + instance_id = instance["InstanceId"] - # get the tags and find the start/stop/weekend tags present on the server - tags = instance["Tags"] - for tag in tags: - if tag["Key"] == "Organization" and tag["Value"] == "census:adep:ead": + # get the tags and find the start/stop/weekend tags present on the server + tags = instance["Tags"] + for tag in tags: + if tag["Key"] == "Organization" and tag["Value"] == "census:adep:ead": - block_devices = instance["BlockDeviceMappings"] - for block in block_devices: - volume_id = block["Ebs"]["VolumeId"] - key = volumeMap[volume_id]["KmsKeyId"] + block_devices = instance["BlockDeviceMappings"] + for block in block_devices: + volume_id = block["Ebs"]["VolumeId"] + key = volumeMap[volume_id]["KmsKeyId"] - if key != "": - print(key) + if key != "": + print(key) diff --git a/local-app/python-tools/utility/find-instances.py b/local-app/python-tools/utility/find-instances.py index dfbd4ce4..42f77955 100644 --- a/local-app/python-tools/utility/find-instances.py +++ b/local-app/python-tools/utility/find-instances.py @@ -1,43 +1,43 @@ -import boto3 -from datetime import datetime import argparse import subprocess +from datetime import datetime + +import boto3 argParser = argparse.ArgumentParser() -#argParser.add_argument("-c", "--cluster", help="specific cluster name to research - default is all clusters") +# argParser.add_argument("-c", "--cluster", help="specific cluster name to research - default is all clusters") args = argParser.parse_args() accountList = [] -with open('account-list.txt', 'r') as dat: - accountList = dat.readlines() +with open("account-list.txt", "r") as dat: + accountList = dat.readlines() regionList = ["us-gov-east-1", "us-gov-west-1"] print("Instance,Name,Account") for acct in accountList: - account = acct.strip() - session = boto3.Session(profile_name=account) - - for region in regionList: - # get all instance reservations - ec2_client = session.client("ec2", region_name=region) + account = acct.strip() + session = boto3.Session(profile_name=account) - reservation_array = ec2_client.describe_instances()["Reservations"] + for region in regionList: + # get all instance reservations + ec2_client = session.client("ec2", region_name=region) - for reservation in reservation_array: + reservation_array = ec2_client.describe_instances()["Reservations"] - # loop over each instance in the reservation - for instance in reservation["Instances"]: - # build a map of instance data needed for this script - instance_id=instance["InstanceId"] - instance_name="" + for reservation in reservation_array: - tags = instance.get("Tags") - for tag in tags: - if tag["Key"] == "Name": - instance_name = tag["Value"] + # loop over each instance in the reservation + for instance in reservation["Instances"]: + # build a map of instance data needed for this script + instance_id = instance["InstanceId"] + instance_name = "" - print(instance_id +","+ instance_name +","+ account ) + tags = instance.get("Tags") + for tag in tags: + if tag["Key"] == "Name": + instance_name = tag["Value"] + print(instance_id + "," + instance_name + "," + account) diff --git a/local-app/python-tools/utility/hostnames.py b/local-app/python-tools/utility/hostnames.py index 17025aa8..76c6ad16 100644 --- a/local-app/python-tools/utility/hostnames.py +++ b/local-app/python-tools/utility/hostnames.py @@ -1,6 +1,7 @@ -import boto3 from datetime import datetime +import boto3 + AWS_REGION = "us-gov-east-1" ec2E = boto3.client("ec2", region_name=AWS_REGION) AWS_REGION = "us-gov-west-1" @@ -14,27 +15,27 @@ reservation_array = reservation_arrayE + reservation_arrayW for reservation in reservation_array: - # loop over each instance in the reservation - for instance in reservation["Instances"]: - - # initialize to blank - instance_id=instance["InstanceId"] - instance_name='(not running)' - ip_address='(not running)' - title_data='no_title' - - # only have dns or ip if running - instance_state = instance["State"]["Name"] - if instance_state=="running": - instance_name=instance["PrivateDnsName"] - ip_address = instance["PrivateIpAddress"] - - # get the tags and find the start/stop/weekend tags present on the server - tags = instance["Tags"] - for tag in tags: - if tag["Key"] == "Name": - instance_name = tag["Value"] - if tag["Key"] == "Title Data": - title_data = tag["Value"] - - print( instance_id +","+ instance_name +","+ ip_address +","+ title_data) + # loop over each instance in the reservation + for instance in reservation["Instances"]: + + # initialize to blank + instance_id = instance["InstanceId"] + instance_name = "(not running)" + ip_address = "(not running)" + title_data = "no_title" + + # only have dns or ip if running + instance_state = instance["State"]["Name"] + if instance_state == "running": + instance_name = instance["PrivateDnsName"] + ip_address = instance["PrivateIpAddress"] + + # get the tags and find the start/stop/weekend tags present on the server + tags = instance["Tags"] + for tag in tags: + if tag["Key"] == "Name": + instance_name = tag["Value"] + if tag["Key"] == "Title Data": + title_data = tag["Value"] + + print(instance_id + "," + instance_name + "," + ip_address + "," + title_data) diff --git a/local-app/python-tools/utility/input.py b/local-app/python-tools/utility/input.py index 935e5d38..b3b689c3 100644 --- a/local-app/python-tools/utility/input.py +++ b/local-app/python-tools/utility/input.py @@ -1,15 +1,14 @@ all_selected = False while True: if all_selected == False: - txt = input("Enter to continue. q to exit: ") + txt = input("Enter to continue. q to exit: ") - if txt == 'q': - break + if txt == "q": + break - if txt == 'a': - all_selected = True + if txt == "a": + all_selected = True - print(f"continuing...") + print(f"continuing...") if all_selected == True: - print(f"continuing...") - + print(f"continuing...") diff --git a/local-app/python-tools/utility/instance-scheduled.py b/local-app/python-tools/utility/instance-scheduled.py index 54530de3..0f71b0f4 100644 --- a/local-app/python-tools/utility/instance-scheduled.py +++ b/local-app/python-tools/utility/instance-scheduled.py @@ -1,6 +1,7 @@ -import boto3 from datetime import datetime +import boto3 + AWS_REGION = "us-gov-west-1" ec2 = boto3.client("ec2", region_name=AWS_REGION) @@ -16,23 +17,23 @@ for reservation in reservation_array: - # loop over each instance in the reservation - for instance in reservation["Instances"]: - - instance_id=instance["InstanceId"] - instance_name = '' - start_hour='' - stop_hour='' - - # get the tags and find the start/stop/weekend tags present on the server - tags = instance["Tags"] - for tag in tags: - if tag["Key"] == "Name": - instance_name = tag["Value"] - if tag["Key"] == "schedule_starthour": - start_hour = tag["Value"] - if tag["Key"] == "schedule_stophour": - stop_hour = tag["Value"] - - if instance_name.startswith("erd-app"): - print(instance_name +" "+ start_hour +" "+ stop_hour) + # loop over each instance in the reservation + for instance in reservation["Instances"]: + + instance_id = instance["InstanceId"] + instance_name = "" + start_hour = "" + stop_hour = "" + + # get the tags and find the start/stop/weekend tags present on the server + tags = instance["Tags"] + for tag in tags: + if tag["Key"] == "Name": + instance_name = tag["Value"] + if tag["Key"] == "schedule_starthour": + start_hour = tag["Value"] + if tag["Key"] == "schedule_stophour": + stop_hour = tag["Value"] + + if instance_name.startswith("erd-app"): + print(instance_name + " " + start_hour + " " + stop_hour) diff --git a/local-app/python-tools/utility/lambda-scheduler.py b/local-app/python-tools/utility/lambda-scheduler.py index 2ed415ca..4c870163 100644 --- a/local-app/python-tools/utility/lambda-scheduler.py +++ b/local-app/python-tools/utility/lambda-scheduler.py @@ -1,119 +1,144 @@ import json from datetime import datetime -from dateutil import tz + import boto3 +from dateutil import tz -ec2 = boto3.client('ec2') +ec2 = boto3.client("ec2") -platform_key="PlatformDetails" -patching_tag="schedule_patching" -skip_tag="schedule_skip" +platform_key = "PlatformDetails" +patching_tag = "schedule_patching" +skip_tag = "schedule_skip" -def lambda_handler(event, context): - # Hardcode zones: - from_zone = tz.gettz('UTC') - to_zone = tz.gettz('America/New_York') - - utc = datetime.now() - utc = utc.replace(tzinfo=from_zone) - - # Convert time zone - now = utc.astimezone(to_zone) - current_time_hour = int(now.strftime("%H")) - - # if at midnight, set to 24 to simplify logic - if current_time_hour == 0: - current_time_hour = 24 - - # weekend in python is days 5 and 6 (zero indexed starting monday) - is_weekend = ( now.weekday() >= 5) - - print( str(now) +" -- "+ str(current_time_hour) ) - print( "is weekend: "+ str(is_weekend)) - - # get all instance reservations - instance_array = [] - reservation_array = ec2.describe_instances()["Reservations"] - - stop_instances = [] - start_instances = [] - - for reservation in reservation_array: - # loop over each instance in the reservation - for instance in reservation["Instances"]: - - # build a map of instance data needed for this script - instanceId=instance["InstanceId"] - instance_state = instance["State"]["Name"] - platform = instance[platform_key] - - environment=stop_hour=start_hour=name=weekend='' - patching = skip = False - - ip_addr = '' - if instance_state=="running": - ip_addr = instance["PrivateIpAddress"] - - instanceId=instance["InstanceId"] - launch_time=instance["LaunchTime"] - - # get the tags and find the start/stop/weekend tags present on the server - tags = instance.get("Tags") - if tags is not None: - for tag in tags: - if tag["Key"] == 'schedule_stophour': - stop_hour = tag["Value"] - if tag["Key"] == 'schedule_starthour': - start_hour = tag["Value"] - if tag["Key"] == 'schedule_weekend': - weekend = tag["Value"] - if tag["Key"] == patching_tag: - patching = "True" - if tag["Key"] == skip_tag: - skip = "True" - if tag["Key"] == "Environment": - environment = tag["Value"] - if tag["Key"] == "Name": - name = tag["Value"] - - # hard-coded for now (just remove this line when windows maint tag is agreed upon) - if platform.upper() == 'WINDOWS' or platform=='': - patching = True - - if start_hour != '' or stop_hour != '': - - if instance_state == 'stopped': - launch_time = '' - - print (instanceId +","+ name +","+ ip_addr +","+ instance_state +","+start_hour +","+ stop_hour +","+ platform +","+ str(patching) +',' + environment +","+ str(launch_time)) - - if platform.upper() == 'WINDOWS' or patching == True or skip == True: - #Not scheduling (windows or) when patching in place - continue - - # if it is the weekend and the instance has weekend false, do nothing, otherwise do logic - if is_weekend and weekend == "false": - continue - - # if current time is == stop hour, and the instance is running, stop it - if current_time_hour == int(stop_hour) and instance_state == "running": - stop_instances.append(instanceId) - - # if current time == start hour and the instance is stopped, start it - elif current_time_hour == int(start_hour) and instance_state == "stopped": - start_instances.append(instanceId) - - for instance in start_instances: - response = ec2.start_instances(InstanceIds=[instance]) - print( str(response)) - - for instance in stop_instances: - response = ec2.stop_instances(InstanceIds=[instance]) - print( str(response)) - - return { - 'statusCode': 200, - 'body': json.dumps('current hour/weekend '+ str(current_time_hour) +'-'+ str(is_weekend)) - } +def lambda_handler(event, context): + # Hardcode zones: + from_zone = tz.gettz("UTC") + to_zone = tz.gettz("America/New_York") + + utc = datetime.now() + utc = utc.replace(tzinfo=from_zone) + + # Convert time zone + now = utc.astimezone(to_zone) + current_time_hour = int(now.strftime("%H")) + + # if at midnight, set to 24 to simplify logic + if current_time_hour == 0: + current_time_hour = 24 + + # weekend in python is days 5 and 6 (zero indexed starting monday) + is_weekend = now.weekday() >= 5 + + print(str(now) + " -- " + str(current_time_hour)) + print("is weekend: " + str(is_weekend)) + + # get all instance reservations + instance_array = [] + reservation_array = ec2.describe_instances()["Reservations"] + + stop_instances = [] + start_instances = [] + + for reservation in reservation_array: + # loop over each instance in the reservation + for instance in reservation["Instances"]: + + # build a map of instance data needed for this script + instanceId = instance["InstanceId"] + instance_state = instance["State"]["Name"] + platform = instance[platform_key] + + environment = stop_hour = start_hour = name = weekend = "" + patching = skip = False + + ip_addr = "" + if instance_state == "running": + ip_addr = instance["PrivateIpAddress"] + + instanceId = instance["InstanceId"] + launch_time = instance["LaunchTime"] + + # get the tags and find the start/stop/weekend tags present on the server + tags = instance.get("Tags") + if tags is not None: + for tag in tags: + if tag["Key"] == "schedule_stophour": + stop_hour = tag["Value"] + if tag["Key"] == "schedule_starthour": + start_hour = tag["Value"] + if tag["Key"] == "schedule_weekend": + weekend = tag["Value"] + if tag["Key"] == patching_tag: + patching = "True" + if tag["Key"] == skip_tag: + skip = "True" + if tag["Key"] == "Environment": + environment = tag["Value"] + if tag["Key"] == "Name": + name = tag["Value"] + + # hard-coded for now (just remove this line when windows maint tag is agreed upon) + if platform.upper() == "WINDOWS" or platform == "": + patching = True + + if start_hour != "" or stop_hour != "": + + if instance_state == "stopped": + launch_time = "" + + print( + instanceId + + "," + + name + + "," + + ip_addr + + "," + + instance_state + + "," + + start_hour + + "," + + stop_hour + + "," + + platform + + "," + + str(patching) + + "," + + environment + + "," + + str(launch_time) + ) + + if platform.upper() == "WINDOWS" or patching == True or skip == True: + # Not scheduling (windows or) when patching in place + continue + + # if it is the weekend and the instance has weekend false, do nothing, otherwise do logic + if is_weekend and weekend == "false": + continue + + # if current time is == stop hour, and the instance is running, stop it + if current_time_hour == int(stop_hour) and instance_state == "running": + stop_instances.append(instanceId) + + # if current time == start hour and the instance is stopped, start it + elif ( + current_time_hour == int(start_hour) and instance_state == "stopped" + ): + start_instances.append(instanceId) + + for instance in start_instances: + response = ec2.start_instances(InstanceIds=[instance]) + print(str(response)) + + for instance in stop_instances: + response = ec2.stop_instances(InstanceIds=[instance]) + print(str(response)) + + return { + "statusCode": 200, + "body": json.dumps( + "current hour/weekend " + str(current_time_hour) + "-" + str(is_weekend) + ), + } diff --git a/local-app/python-tools/utility/metrics.py b/local-app/python-tools/utility/metrics.py index 19a04ffc..dced7cd5 100644 --- a/local-app/python-tools/utility/metrics.py +++ b/local-app/python-tools/utility/metrics.py @@ -1,42 +1,48 @@ -import boto3 from datetime import datetime, timedelta +import boto3 + + def cw_metrics(regionapi, cwapi): - # get all instance reservations - reservation_array = ec2.describe_instances()["Reservations"] - - for reservation in reservation_array: - # loop over each instance in the reservation - for instance in reservation["Instances"]: - - # initialize to blank - instance_id=instance["InstanceId"] - instance_type = instance["InstanceType"] - - response = cw.get_metric_statistics( - Namespace = 'AWS/EC2', - Period = 300, - MetricName="CPUUtilization", - StartTime = datetime.utcnow() - timedelta(seconds = 4*24*3600), - EndTime = datetime.utcnow(), - Statistics=[ - 'Average' - ], - Dimensions = [ - {'Name': 'InstanceId', 'Value': instance_id} - ]) - - dataPoints = response["Datapoints"] - if (dataPoints): - print("{:10.4f}".format(dataPoints[0]['Average']) +","+ instance_id +","+ instance_type) + # get all instance reservations + reservation_array = ec2.describe_instances()["Reservations"] + + for reservation in reservation_array: + # loop over each instance in the reservation + for instance in reservation["Instances"]: + + # initialize to blank + instance_id = instance["InstanceId"] + instance_type = instance["InstanceType"] + + response = cw.get_metric_statistics( + Namespace="AWS/EC2", + Period=300, + MetricName="CPUUtilization", + StartTime=datetime.utcnow() - timedelta(seconds=4 * 24 * 3600), + EndTime=datetime.utcnow(), + Statistics=["Average"], + Dimensions=[{"Name": "InstanceId", "Value": instance_id}], + ) + + dataPoints = response["Datapoints"] + if dataPoints: + print( + "{:10.4f}".format(dataPoints[0]["Average"]) + + "," + + instance_id + + "," + + instance_type + ) + AWS_REGION = "us-gov-east-1" ec2 = boto3.client("ec2", region_name=AWS_REGION) cw = boto3.client("cloudwatch", region_name=AWS_REGION) -cw_metrics(ec2,cw); +cw_metrics(ec2, cw) AWS_REGION = "us-gov-west-1" ec2 = boto3.client("ec2", region_name=AWS_REGION) cw = boto3.client("cloudwatch", region_name=AWS_REGION) -cw_metrics(ec2,cw); +cw_metrics(ec2, cw) diff --git a/local-app/python-tools/utility/multi-cluster-report.py b/local-app/python-tools/utility/multi-cluster-report.py index 952d2947..61470fd0 100644 --- a/local-app/python-tools/utility/multi-cluster-report.py +++ b/local-app/python-tools/utility/multi-cluster-report.py @@ -1,10 +1,11 @@ -import boto3 -from datetime import datetime import argparse import subprocess +from datetime import datetime + +import boto3 argParser = argparse.ArgumentParser() -#argParser.add_argument("-c", "--cluster", help="specific cluster name to research - default is all clusters") +# argParser.add_argument("-c", "--cluster", help="specific cluster name to research - default is all clusters") args = argParser.parse_args() @@ -12,140 +13,212 @@ AWS_REGION = "us-gov-east-1" accountList = [] -with open('account-list.txt', 'r') as dat: - accountList = dat.readlines() +with open("account-list.txt", "r") as dat: + accountList = dat.readlines() regionList = ["us-gov-east-1", "us-gov-west-1"] -print("Resource,Identifier,Account,Region,Project Name,Cost Allocation,Project Number,Misc,Misc,Misc") +print( + "Resource,Identifier,Account,Region,Project Name,Cost Allocation,Project Number,Misc,Misc,Misc" +) for acct in accountList: - account = acct.strip() - session = boto3.Session(profile_name=account) - - for region in regionList: - ## GET THE EKS Clusters - autoscaling = session.client('autoscaling', region_name=region) - eks = session.client("eks", region_name=region) - ec2 = session.client("ec2", region_name=region) - - cluster_list = eks.list_clusters()["clusters"]; - - for cluster in cluster_list: - clusterName=cluster - projectName='NOT TAGGED' - costAllocation='NOT TAGGED' - projectNumber='NOT TAGGED' - environment='NOT TAGGED' - - response = eks.describe_cluster(name=cluster) - version = response["cluster"]["version"] - - tags=response["cluster"]["tags"] - - for tag in tags: - if tag == "Project Name": - projectName = tags[tag] - if tag == "CostAllocation": - costAllocation = tags[tag] - if tag == "ProjectNumber": - projectNumber = tags[tag] - if tag == "Environment": - environment = tags[tag] - - print("Cluster,"+ clusterName +"," + account +"," + region +"," + projectName+"," + costAllocation +","+ projectNumber+","+ environment +","+ version) - - ## Get the nodegroups - if len(clusterName) > 0: - response = eks.list_nodegroups( clusterName=clusterName) - nodeGroups = response["nodegroups"] - - for nodeGroup in nodeGroups: - response = eks.describe_nodegroup(clusterName=clusterName, nodegroupName=nodeGroup) - nodeGroup = response["nodegroup"] - - projectName='NOT TAGGED' - costAllocation='NOT TAGGED' - projectNumber='NOT TAGGED' - - nodeGroupName = nodeGroup["nodegroupName"] - scalingConfig = nodeGroup["scalingConfig"] - - tags = nodeGroup.get("tags") - if tags is not None: - for tag in tags: - if tag == "Project Name": - projectName = tags[tag] - if tag == "CostAllocation": - costAllocation = tags[tag] - if tag == "ProjectNumber": - projectNumber = tags[tag] + account = acct.strip() + session = boto3.Session(profile_name=account) - print("NodeGroup," + nodeGroupName +","+ account +"," + region +"," + projectName +"," + costAllocation +","+ projectNumber) - - autoscalingGroups = nodeGroup["resources"]["autoScalingGroups"] - - ## GET the Security Groups - - ## Get the Route53 Domains - - for autoscalingGroup in autoscalingGroups: - response = autoscaling.describe_auto_scaling_groups( - AutoScalingGroupNames=[ - autoscalingGroup["name"], - ] - ) + for region in regionList: + ## GET THE EKS Clusters + autoscaling = session.client("autoscaling", region_name=region) + eks = session.client("eks", region_name=region) + ec2 = session.client("ec2", region_name=region) - all_instances = [] - for group in response["AutoScalingGroups"]: - for instance in group["Instances"]: - all_instances.append(instance["InstanceId"]) + cluster_list = eks.list_clusters()["clusters"] - for instance in all_instances: + for cluster in cluster_list: + clusterName = cluster + projectName = "NOT TAGGED" + costAllocation = "NOT TAGGED" + projectNumber = "NOT TAGGED" + environment = "NOT TAGGED" - response = ec2.describe_instances( Filters=[ { 'Name': 'instance-id', 'Values': [ instance ] } ]) + response = eks.describe_cluster(name=cluster) + version = response["cluster"]["version"] - for reservation in response["Reservations"]: - for instance in reservation["Instances"]: + tags = response["cluster"]["tags"] - instanceId = instance["InstanceId"] - - projectName='NOT TAGGED' - costAllocation='NOT TAGGED' - projectNumber='NOT TAGGED' - - tags = instance.get("Tags") - if tags is not None: - for tag in tags: - if tag["Key"] == "Project Name": - projectName = tag["Value"] - if tag["Key"] == "CostAllocation": - costAllocation = tag["Value"] - if tag["Key"] == "ProjectNumber": - projectNumber = tag["Value"] - - print("Instance," + instanceId +","+ account +"," + region +"," + projectName +"," + costAllocation +","+ projectNumber +","+ clusterName) - - volumes = instance["BlockDeviceMappings"] - for volume in volumes: - volumeId = volume["Ebs"]["VolumeId"] - response = ec2.describe_volumes( Filters=[ { 'Name': 'volume-id', 'Values': [ volumeId ] } ]) - - for volume in response["Volumes"]: - - projectName='NOT TAGGED' - costAllocation='NOT TAGGED' - projectNumber='NOT TAGGED' - - tags = volume.get("Tags") - if tags is not None: - for tag in tags: - if tag["Key"] == "Project Name": - projectName = tag["Value"] - if tag["Key"] == "CostAllocation": - costAllocation = tag["Value"] - if tag["Key"] == "ProjectNumber": - projectNumber = tag["Value"] - - print("Volume," + volumeId +","+ account +"," + region +"," + projectName +"," + costAllocation +","+ projectNumber +","+ clusterName) + for tag in tags: + if tag == "Project Name": + projectName = tags[tag] + if tag == "CostAllocation": + costAllocation = tags[tag] + if tag == "ProjectNumber": + projectNumber = tags[tag] + if tag == "Environment": + environment = tags[tag] + + print( + "Cluster," + + clusterName + + "," + + account + + "," + + region + + "," + + projectName + + "," + + costAllocation + + "," + + projectNumber + + "," + + environment + + "," + + version + ) + ## Get the nodegroups + if len(clusterName) > 0: + response = eks.list_nodegroups(clusterName=clusterName) + nodeGroups = response["nodegroups"] + + for nodeGroup in nodeGroups: + response = eks.describe_nodegroup( + clusterName=clusterName, nodegroupName=nodeGroup + ) + nodeGroup = response["nodegroup"] + + projectName = "NOT TAGGED" + costAllocation = "NOT TAGGED" + projectNumber = "NOT TAGGED" + + nodeGroupName = nodeGroup["nodegroupName"] + scalingConfig = nodeGroup["scalingConfig"] + + tags = nodeGroup.get("tags") + if tags is not None: + for tag in tags: + if tag == "Project Name": + projectName = tags[tag] + if tag == "CostAllocation": + costAllocation = tags[tag] + if tag == "ProjectNumber": + projectNumber = tags[tag] + + print( + "NodeGroup," + + nodeGroupName + + "," + + account + + "," + + region + + "," + + projectName + + "," + + costAllocation + + "," + + projectNumber + ) + + autoscalingGroups = nodeGroup["resources"]["autoScalingGroups"] + + ## GET the Security Groups + + ## Get the Route53 Domains + + for autoscalingGroup in autoscalingGroups: + response = autoscaling.describe_auto_scaling_groups( + AutoScalingGroupNames=[ + autoscalingGroup["name"], + ] + ) + + all_instances = [] + for group in response["AutoScalingGroups"]: + for instance in group["Instances"]: + all_instances.append(instance["InstanceId"]) + + for instance in all_instances: + + response = ec2.describe_instances( + Filters=[{"Name": "instance-id", "Values": [instance]}] + ) + + for reservation in response["Reservations"]: + for instance in reservation["Instances"]: + + instanceId = instance["InstanceId"] + + projectName = "NOT TAGGED" + costAllocation = "NOT TAGGED" + projectNumber = "NOT TAGGED" + + tags = instance.get("Tags") + if tags is not None: + for tag in tags: + if tag["Key"] == "Project Name": + projectName = tag["Value"] + if tag["Key"] == "CostAllocation": + costAllocation = tag["Value"] + if tag["Key"] == "ProjectNumber": + projectNumber = tag["Value"] + + print( + "Instance," + + instanceId + + "," + + account + + "," + + region + + "," + + projectName + + "," + + costAllocation + + "," + + projectNumber + + "," + + clusterName + ) + + volumes = instance["BlockDeviceMappings"] + for volume in volumes: + volumeId = volume["Ebs"]["VolumeId"] + response = ec2.describe_volumes( + Filters=[ + { + "Name": "volume-id", + "Values": [volumeId], + } + ] + ) + + for volume in response["Volumes"]: + + projectName = "NOT TAGGED" + costAllocation = "NOT TAGGED" + projectNumber = "NOT TAGGED" + + tags = volume.get("Tags") + if tags is not None: + for tag in tags: + if tag["Key"] == "Project Name": + projectName = tag["Value"] + if tag["Key"] == "CostAllocation": + costAllocation = tag["Value"] + if tag["Key"] == "ProjectNumber": + projectNumber = tag["Value"] + + print( + "Volume," + + volumeId + + "," + + account + + "," + + region + + "," + + projectName + + "," + + costAllocation + + "," + + projectNumber + + "," + + clusterName + ) diff --git a/local-app/python-tools/utility/new-tagging.py b/local-app/python-tools/utility/new-tagging.py index f26d8938..50de3158 100644 --- a/local-app/python-tools/utility/new-tagging.py +++ b/local-app/python-tools/utility/new-tagging.py @@ -1,12 +1,13 @@ -import boto3 from datetime import datetime + +import boto3 import pandas as pd connection = {} ### BUILD A MAP of ALL COMBINATIONS OF IDENTIFIED REGION.SERVICE Clients -SERVICE='EC2' +SERVICE = "EC2" AWS_REGION = "us-gov-east-1" client = boto3.client("ec2", region_name=AWS_REGION) connection[AWS_REGION + SERVICE] = client @@ -15,7 +16,7 @@ client = boto3.client("ec2", region_name=AWS_REGION) connection[AWS_REGION + SERVICE] = client -SERVICE='EFS' +SERVICE = "EFS" AWS_REGION = "us-gov-east-1" client = boto3.client("efs", region_name=AWS_REGION) connection[AWS_REGION + SERVICE] = client @@ -24,7 +25,7 @@ client = boto3.client("efs", region_name=AWS_REGION) connection[AWS_REGION + SERVICE] = client -SERVICE='S3' +SERVICE = "S3" AWS_REGION = "us-gov-east-1" client = boto3.client("s3", region_name=AWS_REGION) connection[AWS_REGION + SERVICE] = client @@ -34,69 +35,80 @@ connection[AWS_REGION + SERVICE] = client # READ THE DATAFILE INTO A DataFrame -df = pd.read_csv('new-data-file.csv') +df = pd.read_csv("new-data-file.csv") # GET RID OF FIRST ROW AND COLUMN df = df[1:] df2 = df[df.columns[1:]] # use filter to select rows where column H (field_name) is not tagged -field_name="Current Tag: Project Name" -df3 = df2[df2[field_name] == "(not tagged)" ] +field_name = "Current Tag: Project Name" +df3 = df2[df2[field_name] == "(not tagged)"] ## GET THE RELEVANT FIELDS -df = df3[["Region.1", "Service", "Identifier", "Fix Tag: Project Name", "Fix Tag: Project Role", "Fix Tag: ProjectNumber"]] -df=df.astype(str) +df = df3[ + [ + "Region.1", + "Service", + "Identifier", + "Fix Tag: Project Name", + "Fix Tag: Project Role", + "Fix Tag: ProjectNumber", + ] +] +df = df.astype(str) # loop over the rows for ind in df.index: - region=df['Region.1'][ind] - service=df['Service'][ind] - identifier = df['Identifier'][ind] - project_name= df['Fix Tag: Project Name'][ind] - project_role=df['Fix Tag: Project Role'][ind] - project_number=df['Fix Tag: ProjectNumber'][ind] + region = df["Region.1"][ind] + service = df["Service"][ind] + identifier = df["Identifier"][ind] + project_name = df["Fix Tag: Project Name"][ind] + project_role = df["Fix Tag: Project Role"][ind] + project_number = df["Fix Tag: ProjectNumber"][ind] # in case the project name is undefined, skip this section if len(project_name) > 4: - # RETRIEVE THE CLIENT FROM THE MAP - client = connection[region + service] - - ## BUILD A MAP of tags - cluster_tags = {} - - # only add the tag if we have a value - if len(project_name) > 0: - cluster_tags["Project Name"] = project_name - if len(project_role) > 0: - cluster_tags["Project Role"] = project_role - if len(project_number) > 0: - cluster_tags["ProjectNumber"] = project_number - - # CONVERT THE MAP into KEY/VALUE List - tag_list = [] - for key in cluster_tags.keys(): - tag = {} - tag["Key"] = key - tag["Value"] = cluster_tags[key] - tag_list.append(tag) - - ## if there are tags to be added - if len(tag_list) > 0: - - try: - - # MAKE THE CORRECT CALL DEPENDING ON SERVICE TYPE - if service=='EC2': - client.create_tags( Resources=[identifier], Tags=tag_list) - if service=='EFS': - client.create_tags( FileSystemId=identifier, Tags=tag_list) - if service=='S3': - bucket_tagging = client.get_bucket_tagging(Bucket=identifier) - tags = bucket_tagging["TagSet"] - tags.extend(tag_list) - client.put_bucket_tagging( Bucket=identifier, Tagging={'TagSet':tags}) - - except: - print("unable to update: "+ identifier) + # RETRIEVE THE CLIENT FROM THE MAP + client = connection[region + service] + + ## BUILD A MAP of tags + cluster_tags = {} + + # only add the tag if we have a value + if len(project_name) > 0: + cluster_tags["Project Name"] = project_name + if len(project_role) > 0: + cluster_tags["Project Role"] = project_role + if len(project_number) > 0: + cluster_tags["ProjectNumber"] = project_number + + # CONVERT THE MAP into KEY/VALUE List + tag_list = [] + for key in cluster_tags.keys(): + tag = {} + tag["Key"] = key + tag["Value"] = cluster_tags[key] + tag_list.append(tag) + + ## if there are tags to be added + if len(tag_list) > 0: + + try: + + # MAKE THE CORRECT CALL DEPENDING ON SERVICE TYPE + if service == "EC2": + client.create_tags(Resources=[identifier], Tags=tag_list) + if service == "EFS": + client.create_tags(FileSystemId=identifier, Tags=tag_list) + if service == "S3": + bucket_tagging = client.get_bucket_tagging(Bucket=identifier) + tags = bucket_tagging["TagSet"] + tags.extend(tag_list) + client.put_bucket_tagging( + Bucket=identifier, Tagging={"TagSet": tags} + ) + + except: + print("unable to update: " + identifier) diff --git a/local-app/python-tools/utility/query-tag.py b/local-app/python-tools/utility/query-tag.py index ac398296..d5db2ec9 100644 --- a/local-app/python-tools/utility/query-tag.py +++ b/local-app/python-tools/utility/query-tag.py @@ -1,28 +1,31 @@ -import boto3 from datetime import datetime, timedelta +import boto3 + + def get_tags(regionapi, cwapi): - # get all instance reservations - reservation_array = ec2.describe_instances()["Reservations"] + # get all instance reservations + reservation_array = ec2.describe_instances()["Reservations"] + + for reservation in reservation_array: + # loop over each instance in the reservation + for instance in reservation["Instances"]: - for reservation in reservation_array: - # loop over each instance in the reservation - for instance in reservation["Instances"]: + # initialize to blank + instance_id = instance["InstanceId"] + instance_type = instance["InstanceType"] + tags = instance["Tags"] + if tags: + print(tags) - # initialize to blank - instance_id=instance["InstanceId"] - instance_type = instance["InstanceType"] - tags = instance["Tags"] - if (tags): - print(tags) AWS_REGION = "us-gov-east-1" ec2 = boto3.client("ec2", region_name=AWS_REGION) cw = boto3.client("cloudwatch", region_name=AWS_REGION) -get_tags(ec2,cw); +get_tags(ec2, cw) AWS_REGION = "us-gov-west-1" ec2 = boto3.client("ec2", region_name=AWS_REGION) cw = boto3.client("cloudwatch", region_name=AWS_REGION) -get_tags(ec2,cw); +get_tags(ec2, cw) diff --git a/local-app/python-tools/utility/reggie.py b/local-app/python-tools/utility/reggie.py index 2f66bb80..6360b28f 100644 --- a/local-app/python-tools/utility/reggie.py +++ b/local-app/python-tools/utility/reggie.py @@ -1,9 +1,10 @@ -import boto3 from datetime import datetime +import boto3 + AWS_REGION = "us-gov-west-1" -ec21 = boto3.client("ec2", region_name='us-gov-west-1') -ec22 = boto3.client("ec2", region_name='us-gov-east-1') +ec21 = boto3.client("ec2", region_name="us-gov-west-1") +ec22 = boto3.client("ec2", region_name="us-gov-east-1") instance_array = [] @@ -16,7 +17,7 @@ for volume in volume_array: volume_id = volume["VolumeId"] - volumeMap[volume_id]=volume + volumeMap[volume_id] = volume # get all instance reservations reservation_array1 = ec21.describe_instances()["Reservations"] @@ -24,21 +25,21 @@ reservation_array = reservation_array1 + reservation_array2 -volume_list=[] +volume_list = [] for reservation in reservation_array: - # loop over each instance in the reservation - for instance in reservation["Instances"]: + # loop over each instance in the reservation + for instance in reservation["Instances"]: - instance_id=instance["InstanceId"] + instance_id = instance["InstanceId"] - platform = instance["PlatformDetails"] + platform = instance["PlatformDetails"] - if platform == "Windows": - block_devices = instance["BlockDeviceMappings"] - for block in block_devices: - volume_id = block["Ebs"]["VolumeId"] - key = volumeMap[volume_id]["KmsKeyId"] + if platform == "Windows": + block_devices = instance["BlockDeviceMappings"] + for block in block_devices: + volume_id = block["Ebs"]["VolumeId"] + key = volumeMap[volume_id]["KmsKeyId"] - if key != "": - print(key) + if key != "": + print(key) diff --git a/local-app/python-tools/utility/table.py b/local-app/python-tools/utility/table.py index 1e5c5a5a..62188ef2 100644 --- a/local-app/python-tools/utility/table.py +++ b/local-app/python-tools/utility/table.py @@ -1,68 +1,72 @@ -import boto3 from datetime import datetime -ec2 = boto3.client('ec2') +import boto3 + +ec2 = boto3.client("ec2") + +platform_key = "PlatformDetails" +windows_maint_tag = "Patching" -platform_key="PlatformDetails" -windows_maint_tag="Patching" def find_instances(): - instance_array = [] - - # get all instance reservations - reservation_array = ec2.describe_instances()["Reservations"] - - for reservation in reservation_array: - # loop over each instance in the reservation - for instance in reservation["Instances"]: - - # build a map of instance data needed for this script - instance_name=instance["InstanceId"] - instance_state = instance["State"]["Name"] - platform = instance[platform_key] - - ip_addr = '' - if instance_state=="running": - ip_addr = instance["PrivateIpAddress"] - - stop_hour=start_hour=weekend='' - windows_maint = False - - # get the tags and find the start/stop/weekend tags present on the server - tags = instance["Tags"] - for tag in tags: - if tag["Key"] == 'schedule_stophour': - stop_hour = tag["Value"] - if tag["Key"] == 'schedule_starthour': - start_hour = tag["Value"] - if tag["Key"] == 'schedule_weekend': - weekend = tag["Value"] - if tag["Key"] == windows_maint_tag: - windows_maint = "True" - if tag["Key"] == "Name": - instanceNAME = tag["Value"] - if tag["Key"] == "Environment": - environment = tag["Value"] - - # hard-coded for now (just remove this line when windows maint tag is agreed upon) - windows_maint = True - - instance_item={ "InstanceId":instance_name, - "State":instance_state, - "StopHour":stop_hour, - "StartHour":start_hour, - "Weekend":weekend, - platform_key: platform, - windows_maint_tag: windows_maint, - "name": instanceNAME, - "env": environment, - "ip_addr": ip_addr} - - if start_hour != '' or stop_hour != '': - instance_array.append(instance_item) - - return instance_array + instance_array = [] + + # get all instance reservations + reservation_array = ec2.describe_instances()["Reservations"] + + for reservation in reservation_array: + # loop over each instance in the reservation + for instance in reservation["Instances"]: + + # build a map of instance data needed for this script + instance_name = instance["InstanceId"] + instance_state = instance["State"]["Name"] + platform = instance[platform_key] + + ip_addr = "" + if instance_state == "running": + ip_addr = instance["PrivateIpAddress"] + + stop_hour = start_hour = weekend = "" + windows_maint = False + + # get the tags and find the start/stop/weekend tags present on the server + tags = instance["Tags"] + for tag in tags: + if tag["Key"] == "schedule_stophour": + stop_hour = tag["Value"] + if tag["Key"] == "schedule_starthour": + start_hour = tag["Value"] + if tag["Key"] == "schedule_weekend": + weekend = tag["Value"] + if tag["Key"] == windows_maint_tag: + windows_maint = "True" + if tag["Key"] == "Name": + instanceNAME = tag["Value"] + if tag["Key"] == "Environment": + environment = tag["Value"] + + # hard-coded for now (just remove this line when windows maint tag is agreed upon) + windows_maint = True + + instance_item = { + "InstanceId": instance_name, + "State": instance_state, + "StopHour": stop_hour, + "StartHour": start_hour, + "Weekend": weekend, + platform_key: platform, + windows_maint_tag: windows_maint, + "name": instanceNAME, + "env": environment, + "ip_addr": ip_addr, + } + + if start_hour != "" or stop_hour != "": + instance_array.append(instance_item) + + return instance_array def toggle_instances(current_time_hour, is_weekend): @@ -84,39 +88,65 @@ def toggle_instances(current_time_hour, is_weekend): env = instance["env"] ip_addr = instance["ip_addr"] - print (instanceId +","+ name +","+ ip_addr +","+ state +","+start_hour +","+ stop_hour +","+ platform +',' + env) - - if platform.upper() == 'WINDOWS' and windows_maint == True: - #Not scheduling windows when patching in place - continue + print( + instanceId + + "," + + name + + "," + + ip_addr + + "," + + state + + "," + + start_hour + + "," + + stop_hour + + "," + + platform + + "," + + env + ) + + if platform.upper() == "WINDOWS" and windows_maint == True: + # Not scheduling windows when patching in place + continue # if it is the weekend and the instance has weekend false, do nothing, otherwise do logic if is_weekend and weekend == "false": - continue + continue # if stop hour is defined and current time is >= stop hour, and the instance is running, stop it - if stop_hour != '' and current_time_hour >= int(stop_hour) and state == "running": - stop_instances.append(instanceId) + if ( + stop_hour != "" + and current_time_hour >= int(stop_hour) + and state == "running" + ): + stop_instances.append(instanceId) # if start hour is defined and current time is >= start hour, and the instance is stopped, start it - if start_hour != '' and current_time_hour >= int(start_hour) and state == "stopped": - start_instances.append(instanceId) + if ( + start_hour != "" + and current_time_hour >= int(start_hour) + and state == "stopped" + ): + start_instances.append(instanceId) + + # print(start_instances)#response = ec2.start_instances(InstanceIds=[instanceId], DryRun=True) + # print(stop_instances)#response = ec2.stop_instances(InstanceIds=[instanceId], DryRun=True) - #print(start_instances)#response = ec2.start_instances(InstanceIds=[instanceId], DryRun=True) - #print(stop_instances)#response = ec2.stop_instances(InstanceIds=[instanceId], DryRun=True) def main(): now = datetime.now() - + # get the current hour current_time_hour = int(now.strftime("%H")) # weekend in python is days 5 and 6 (zero indexed starting monday) - is_weekend = ( now.weekday() >= 5) + is_weekend = now.weekday() >= 5 # toggle instances based upon current state and time/day toggle_instances(current_time_hour, is_weekend) + if __name__ == "__main__": main() diff --git a/local-app/python-tools/utility/tag-report.py b/local-app/python-tools/utility/tag-report.py index be1cef43..a971313b 100644 --- a/local-app/python-tools/utility/tag-report.py +++ b/local-app/python-tools/utility/tag-report.py @@ -1,16 +1,17 @@ -import boto3 -from datetime import datetime import argparse import subprocess +from datetime import datetime + +import boto3 argParser = argparse.ArgumentParser() -#argParser.add_argument("-c", "--cluster", help="specific cluster name to research - default is all clusters") +# argParser.add_argument("-c", "--cluster", help="specific cluster name to research - default is all clusters") args = argParser.parse_args() accountList = [] -with open('account-list2.txt', 'r') as dat: - accountList = dat.readlines() +with open("account-list2.txt", "r") as dat: + accountList = dat.readlines() regionList = ["us-gov-east-1", "us-gov-west-1"] @@ -18,67 +19,66 @@ all_tags_list = [] for acct in accountList: - account = acct.strip() - session = boto3.Session(profile_name=account) + account = acct.strip() + session = boto3.Session(profile_name=account) + + for region in regionList: + ec2 = session.client("ec2", region_name=region) - for region in regionList: - ec2 = session.client("ec2", region_name=region) + reservations = ec2.describe_instances()["Reservations"] - reservations = ec2.describe_instances()["Reservations"] + for reservation in reservations: + # loop over each instance in the reservation + for instance in reservation["Instances"]: - for reservation in reservations: - # loop over each instance in the reservation - for instance in reservation["Instances"]: + instance_id = instance["InstanceId"] + tags = instance.get("Tags") - instance_id=instance["InstanceId"] - tags = instance.get("Tags") + instance_tag_map = {} + instance_tag_map["Region"] = region - instance_tag_map = {} - instance_tag_map["Region"] = region + # loop over tags to get the names for the full array + for tag in tags: + key = tag["Key"] - # loop over tags to get the names for the full array - for tag in tags: - key = tag["Key"] - - # add the key to the full list - if not key in all_tags_list: - all_tags_list.append(key) + # add the key to the full list + if not key in all_tags_list: + all_tags_list.append(key) - value = tag["Value"] - instance_tag_map[key] = value + value = tag["Value"] + instance_tag_map[key] = value - # store the full map to the tags - tag_map[instance_id] = instance_tag_map + # store the full map to the tags + tag_map[instance_id] = instance_tag_map # sort the tags all_tags_list.sort(key=str.lower) all_tags_list.remove("Name") -all_tags_list.insert(0,"Name") -all_tags_list.insert(1,"Region") +all_tags_list.insert(0, "Name") +all_tags_list.insert(1, "Region") -tag_list_str = ','.join(all_tags_list) +tag_list_str = ",".join(all_tags_list) print("Instance ID,", tag_list_str) # loop over the keys in the map key_list = list(tag_map.keys()) for instance in key_list: - # get the tags for this instance from the map - instance_tags = tag_map[instance] - - output_array = [] + # get the tags for this instance from the map + instance_tags = tag_map[instance] - # add the instance id - output_array.append(instance) + output_array = [] - # loop over the full list of tags - for key in all_tags_list: - value = instance_tags.get(key) + # add the instance id + output_array.append(instance) - if value is None or len(value) == 0: - value='' + # loop over the full list of tags + for key in all_tags_list: + value = instance_tags.get(key) - output_array.append( str(value) ) + if value is None or len(value) == 0: + value = "" - print( ','.join(output_array) ) + output_array.append(str(value)) + print(",".join(output_array)) diff --git a/local-app/python-tools/utility/tagging.py b/local-app/python-tools/utility/tagging.py index 6619aaf8..3925287c 100644 --- a/local-app/python-tools/utility/tagging.py +++ b/local-app/python-tools/utility/tagging.py @@ -1,6 +1,7 @@ -import boto3 from datetime import datetime +import boto3 + AWS_REGION = "us-gov-west-1" ec2 = boto3.client("ec2", region_name=AWS_REGION) @@ -15,51 +16,53 @@ for volume in volume_array: if len(volume["Attachments"]) > 0: - key = volume["Attachments"][0]["InstanceId"] - volume_id = volume["VolumeId"] + key = volume["Attachments"][0]["InstanceId"] + volume_id = volume["VolumeId"] - vol_list = instanceIdToVolumeIdList.get(key) - if not vol_list: - vol_list=[] + vol_list = instanceIdToVolumeIdList.get(key) + if not vol_list: + vol_list = [] - vol_list.append(volume_id) - instanceIdToVolumeIdList[key] = vol_list + vol_list.append(volume_id) + instanceIdToVolumeIdList[key] = vol_list -#keys = instanceIdToVolumeIdList.keys() -#for key in keys: +# keys = instanceIdToVolumeIdList.keys() +# for key in keys: # print(key +" "+ ' '.join(map(str,instanceIdToVolumeIdList.get(key))) ) ### ### read data file containing the CLUSTER NAME, PROJECT ROLE, PROJECT NAME and PROJECT NUMBER from finance ### into a map of cluster name to map of the other three ### -datafile = open('data-file.txt', 'r') +datafile = open("data-file.txt", "r") lines = datafile.readlines() cluster_data = {} # build map with values for line in lines: - if len(line) > 1: - fieldlist = line.strip().split(',') - clustername=fieldlist[0] - project_role=fieldlist[1] - project_name=fieldlist[2] - project_number=fieldlist[3] - environment = fieldlist[4] - - # EXPECTED TAGS - #"Project Name" = "geo_partnerportal" - #"Project Role" = "geo_partnerportal_eks" - #"ProjectNumber" = "fs0000000066" - #"Environment" = "dev" - - cluster_tags = {} - cluster_tags["Project Name"] = project_name - cluster_tags["Project Role"] = project_role - cluster_tags["ProjectNumber"] = project_number ## NO SPACE BEFORE NUMBER IS CORRECT - cluster_tags["Environment"] = environment.lower() ## lower case expected - - cluster_data[clustername] = cluster_tags + if len(line) > 1: + fieldlist = line.strip().split(",") + clustername = fieldlist[0] + project_role = fieldlist[1] + project_name = fieldlist[2] + project_number = fieldlist[3] + environment = fieldlist[4] + + # EXPECTED TAGS + # "Project Name" = "geo_partnerportal" + # "Project Role" = "geo_partnerportal_eks" + # "ProjectNumber" = "fs0000000066" + # "Environment" = "dev" + + cluster_tags = {} + cluster_tags["Project Name"] = project_name + cluster_tags["Project Role"] = project_role + cluster_tags["ProjectNumber"] = ( + project_number ## NO SPACE BEFORE NUMBER IS CORRECT + ) + cluster_tags["Environment"] = environment.lower() ## lower case expected + + cluster_data[clustername] = cluster_tags ### ### FIND ALL EC2 INSTANCES working as EKS Worker Nodes @@ -68,70 +71,70 @@ match_string = "nodegroup-instance-name" reservation_array = ec2.describe_instances()["Reservations"] -account="" +account = "" clusters_not_tagged = [] for reservation in reservation_array: - if(len(account)) == 0: - account = reservation["OwnerId"] - print(account) - - # loop over each instance in the reservation - for instance in reservation["Instances"]: - - instance_id=instance["InstanceId"] - instance_name = '' - - # get the tags and find the start/stop/weekend tags present on the server - tags = instance["Tags"] - for tag in tags: - if tag["Key"] == "Name": - instance_name = tag["Value"] - - match_index = instance_name.find(match_string) - if match_index > 0: - cluster_name = instance_name[:(match_index-1)] - - inCluster = cluster_name in cluster_data.keys() - if inCluster == True: - cluster_tags = cluster_data[cluster_name] - - volumes = [] - thisVolume = instanceIdToVolumeIdList.get(instance_id) - if thisVolume and len(thisVolume) > 0: - volumes = thisVolume - - cluster_tags = cluster_data.get(cluster_name) - tag_list = [] - for key in cluster_tags.keys(): - tag = {} - tag["Key"] = key - tag["Value"] = cluster_tags[key] - tag_list.append(tag) - - print("\t"+ cluster_name ) - - print("\tTAG INSTANCE\t"+ instance_id) - - #OBVIOUSLY, uncomment this - ec2.create_tags( Resources=[instance_id], Tags=tag_list) - - print("\tTAG VOLUMES\t") - for volume in volumes: - - #OBVIOUSLY, uncomment this - print("\t\tTAG VOLUME:\t"+ volume) - - #OBVIOUSLY, uncomment this - ec2.create_tags( Resources=[volume], Tags=tag_list) - - print("\n\n") - else: - if not cluster_name in clusters_not_tagged: - clusters_not_tagged.append(cluster_name) - -print("For account: "+ account) + if (len(account)) == 0: + account = reservation["OwnerId"] + print(account) + + # loop over each instance in the reservation + for instance in reservation["Instances"]: + + instance_id = instance["InstanceId"] + instance_name = "" + + # get the tags and find the start/stop/weekend tags present on the server + tags = instance["Tags"] + for tag in tags: + if tag["Key"] == "Name": + instance_name = tag["Value"] + + match_index = instance_name.find(match_string) + if match_index > 0: + cluster_name = instance_name[: (match_index - 1)] + + inCluster = cluster_name in cluster_data.keys() + if inCluster == True: + cluster_tags = cluster_data[cluster_name] + + volumes = [] + thisVolume = instanceIdToVolumeIdList.get(instance_id) + if thisVolume and len(thisVolume) > 0: + volumes = thisVolume + + cluster_tags = cluster_data.get(cluster_name) + tag_list = [] + for key in cluster_tags.keys(): + tag = {} + tag["Key"] = key + tag["Value"] = cluster_tags[key] + tag_list.append(tag) + + print("\t" + cluster_name) + + print("\tTAG INSTANCE\t" + instance_id) + + # OBVIOUSLY, uncomment this + ec2.create_tags(Resources=[instance_id], Tags=tag_list) + + print("\tTAG VOLUMES\t") + for volume in volumes: + + # OBVIOUSLY, uncomment this + print("\t\tTAG VOLUME:\t" + volume) + + # OBVIOUSLY, uncomment this + ec2.create_tags(Resources=[volume], Tags=tag_list) + + print("\n\n") + else: + if not cluster_name in clusters_not_tagged: + clusters_not_tagged.append(cluster_name) + +print("For account: " + account) print("For region: " + AWS_REGION) print("These clusters not tagged: ") for cluster in clusters_not_tagged: - print("\t"+ cluster) + print("\t" + cluster) diff --git a/local-app/python-tools/utility/trial.py b/local-app/python-tools/utility/trial.py index 56b6bf34..c23cbace 100644 --- a/local-app/python-tools/utility/trial.py +++ b/local-app/python-tools/utility/trial.py @@ -1,10 +1,10 @@ -import boto3 from datetime import datetime -volume_id="vol-0b5fa10be8d27c932" +import boto3 + +volume_id = "vol-0b5fa10be8d27c932" AWS_REGION = "us-gov-west-1" ec2 = boto3.client("ec2", region_name=AWS_REGION) -response = ec2.modify_volume(VolumeId=volume_id, Size= 250, VolumeType='gp3') +response = ec2.modify_volume(VolumeId=volume_id, Size=250, VolumeType="gp3") print(response) - diff --git a/local-app/python-tools/utility/type-report.py b/local-app/python-tools/utility/type-report.py index df2fb157..f7a65937 100644 --- a/local-app/python-tools/utility/type-report.py +++ b/local-app/python-tools/utility/type-report.py @@ -1,20 +1,21 @@ -import boto3 from datetime import datetime -typePrice={} -price = '' -instance_type='' +import boto3 + +typePrice = {} +price = "" +instance_type = "" -with open("pricing.txt", 'r') as infile: +with open("pricing.txt", "r") as infile: lines = infile.readlines() - counter=0 + counter = 0 for line in lines: if counter == 9: counter = 0 - if counter==0: + if counter == 0: instance_type = line.strip() - if counter==1: + if counter == 1: price = line.strip() counter = counter + 1 @@ -34,17 +35,17 @@ # build the map of instances by type for reservation in reservation_array: - # loop over each instance in the reservation - for instance in reservation["Instances"]: + # loop over each instance in the reservation + for instance in reservation["Instances"]: + + instance_type = instance["InstanceType"] - instance_type=instance["InstanceType"] - - instance_id=instance["InstanceId"] - instance_type=instance["InstanceType"] - availability_zone=instance["Placement"]["AvailabilityZone"] + instance_id = instance["InstanceId"] + instance_type = instance["InstanceType"] + availability_zone = instance["Placement"]["AvailabilityZone"] - price = typePrice.get(instance_type) - if not price: - price = str(0) + price = typePrice.get(instance_type) + if not price: + price = str(0) - print(instance_type +","+ price +","+ instance_id +","+ availability_zone) + print(instance_type + "," + price + "," + instance_id + "," + availability_zone) diff --git a/local-app/python-tools/utility/upgrade.py b/local-app/python-tools/utility/upgrade.py index d52a2465..22609815 100644 --- a/local-app/python-tools/utility/upgrade.py +++ b/local-app/python-tools/utility/upgrade.py @@ -1,12 +1,12 @@ f = open("servers.txt", "r") lines = f.readlines() -change_to='' +change_to = "" for line in lines: if "C6" in line: - array = line.split(" ") - change_to=array[3] + array = line.split(" ") + change_to = array[3] if "ditdapp" in line: - array = line.split(" ") - print(array[0] +" "+ change_to.strip()) + array = line.split(" ") + print(array[0] + " " + change_to.strip()) diff --git a/local-app/python-tools/utility/vol-list.py b/local-app/python-tools/utility/vol-list.py index 0ded785c..ce8cdc09 100644 --- a/local-app/python-tools/utility/vol-list.py +++ b/local-app/python-tools/utility/vol-list.py @@ -1,6 +1,7 @@ -import boto3 from datetime import datetime +import boto3 + AWS_REGION = "us-gov-east-1" ec2 = boto3.client("ec2", region_name=AWS_REGION) @@ -13,14 +14,14 @@ volume_id = volume["VolumeId"] attachments = volume["Attachments"] size = volume["Size"] - volumeName = '' + volumeName = "" if len(attachments) == 0: tags = volume.get("Tags") if tags: - for tag in tags: - if tag["Key"] == "Name": - volumeName = tag["Value"] + for tag in tags: + if tag["Key"] == "Name": + volumeName = tag["Value"] if volumeName.startswith("kubernetes-dynamic-pvc"): - print(volumeName +","+ volume_id +","+ str(size)) + print(volumeName + "," + volume_id + "," + str(size)) diff --git a/local-app/python-tools/utility/volumes.py b/local-app/python-tools/utility/volumes.py index 3e675517..5bf9011f 100644 --- a/local-app/python-tools/utility/volumes.py +++ b/local-app/python-tools/utility/volumes.py @@ -1,6 +1,7 @@ -import boto3 from datetime import datetime +import boto3 + AWS_REGION = "us-gov-east-1" ec2 = boto3.client("ec2", region_name=AWS_REGION) @@ -12,34 +13,35 @@ for volume in volume_array: volume_id = volume["VolumeId"] size = volume["Size"] - volumeMap[volume_id]=volume + volumeMap[volume_id] = volume # get all instance reservations reservation_array = ec2.describe_instances()["Reservations"] -volume_list=[] +volume_list = [] for reservation in reservation_array: - # loop over each instance in the reservation - for instance in reservation["Instances"]: - - instanceNAME='' - - # get the tags and find the start/stop/weekend tags present on the server - tags = instance["Tags"] - for tag in tags: - if tag["Key"] == "Name": - instanceNAME = tag["Value"] - - if instanceNAME.startswith('csvd-sple-idx'): - print(instanceNAME) - - block_devices = instance["BlockDeviceMappings"] - for block in block_devices: - device_name = block["DeviceName"] - volume_id = block["Ebs"]["VolumeId"] - - size = volumeMap[volume_id]["Size"] - type = volumeMap[volume_id]["VolumeType"] - print('\t'+ device_name +' '+ volume_id +" "+ str(size) +" " + type) - + # loop over each instance in the reservation + for instance in reservation["Instances"]: + + instanceNAME = "" + + # get the tags and find the start/stop/weekend tags present on the server + tags = instance["Tags"] + for tag in tags: + if tag["Key"] == "Name": + instanceNAME = tag["Value"] + + if instanceNAME.startswith("csvd-sple-idx"): + print(instanceNAME) + + block_devices = instance["BlockDeviceMappings"] + for block in block_devices: + device_name = block["DeviceName"] + volume_id = block["Ebs"]["VolumeId"] + + size = volumeMap[volume_id]["Size"] + type = volumeMap[volume_id]["VolumeType"] + print( + "\t" + device_name + " " + volume_id + " " + str(size) + " " + type + ) diff --git a/local-app/rotate-keys/README.md b/local-app/rotate-keys/README.md index f48d7779..ef9834ac 100644 --- a/local-app/rotate-keys/README.md +++ b/local-app/rotate-keys/README.md @@ -18,5 +18,3 @@ Drives the rotation, extracton, listing, and disabling of old keys. * [setup-rotate-keys.sh](setup-rotate-keys.md) A script which will create a directory `access_keys.{{ IAM_USERNAME }}`, initialize the setup, create key 0, extract, and create a zip file (using a password you specify). - - diff --git a/local-app/rotate-keys/main.keys.tf.j2 b/local-app/rotate-keys/main.keys.tf.j2 index 6af2c099..cf295c62 100644 --- a/local-app/rotate-keys/main.keys.tf.j2 +++ b/local-app/rotate-keys/main.keys.tf.j2 @@ -3,7 +3,7 @@ resource "aws_iam_access_key" "iam_access_key_v{{ v }}" { {% if (data.version.id-v) <= 1 %} count = 1 {% else %} - count = 0 + count = 0 {% endif %} user = data.aws_iam_user.iam_user.user_name pgp_key = file("setup/terraform.gpg.b64") @@ -13,7 +13,7 @@ resource "null_resource" "wait_iam_access_key_v{{ v }}" { {% if (data.version.id-v) <= 1 %} count = 1 {% else %} - count = 0 + count = 0 {% endif %} provisioner "local-exec" { when = create diff --git a/local-app/rotate-keys/rotate-keys.md b/local-app/rotate-keys/rotate-keys.md index 46ba2a2b..41eb4bd1 100644 --- a/local-app/rotate-keys/rotate-keys.md +++ b/local-app/rotate-keys/rotate-keys.md @@ -50,7 +50,7 @@ Even though NOT a module, pull from terraform-module/ +-- gpg-key.gpg +-- gpg-key alias ``` - + ## Running ```shell @@ -98,7 +98,7 @@ rotate-keys.py # apply changes tf-apply -# get new key details +# get new key details rotate-keys.py -a # (update ```aws config``` with new keys) @@ -120,4 +120,3 @@ add a key (date) into the YAML file * --initialize-gpg Copy all the required files from some git repo, setup/copy a GPG key - diff --git a/local-app/rotate-keys/rotate-keys.py b/local-app/rotate-keys/rotate-keys.py index 8e0a83e7..c0caa5c7 100755 --- a/local-app/rotate-keys/rotate-keys.py +++ b/local-app/rotate-keys/rotate-keys.py @@ -1,33 +1,36 @@ #!/apps/terraform/python/bin/python # /bin/env python -from jinja2 import Environment,DictLoader -import os +import argparse +import base64 import csv +import hashlib +import json +import os import re +import subprocess import sys +from collections import defaultdict +from datetime import date, datetime, time from pprint import pprint -from datetime import datetime,date,time + +import boto3 +import gnupg +import yaml from dateutil import tz from dateutil.parser import parse as date_parse -import yaml -import hashlib -import argparse -import subprocess -import json -import base64 -import gnupg -import boto3 -from collections import defaultdict +from jinja2 import DictLoader, Environment + def is_gpg_agent_needed(): - value=True - gpg_version=gnupg.GPG().version - if gpg_version[0]>=2 and gpg_version[1]>=2: - value=False - elif os.environ.get('SKIP_GPG_AGENT') is not None: - value=False - return value + value = True + gpg_version = gnupg.GPG().version + if gpg_version[0] >= 2 and gpg_version[1] >= 2: + value = False + elif os.environ.get("SKIP_GPG_AGENT") is not None: + value = False + return value + # this is from # https://gist.github.com/fralau/061a4f6c13251367ef1d9a9a99fb3e8d @@ -41,11 +44,11 @@ def parse_var(s): or foo="hello world" """ - items = s.split('=') - key = items[0].strip() # we remove blanks around keys, as is logical + items = s.split("=") + key = items[0].strip() # we remove blanks around keys, as is logical if len(items) > 1: # rejoin the rest: - value = '='.join(items[1:]) + value = "=".join(items[1:]) return (key, value) @@ -54,8 +57,8 @@ def parse_vars(items): Parse a series of key-value pairs and return a dictionary """ d = defaultdict(str) - for x in ['username','profile','region','service_profile','gpg_key_file']: - d[x]="" + for x in ["username", "profile", "region", "service_profile", "gpg_key_file"]: + d[x] = "" if items: for item in items: @@ -63,24 +66,25 @@ def parse_vars(items): d[key] = value return d -#--- + +# --- # get files -#--- +# --- def init_content(vars): - if not vars.get('gpg_key_file',None): - vars['gpg_key_file']=find_gpg_key() - for k,v in vars.items(): - if v is None: - vars[k]="" - - content={ - 'main.keys.tf.j2':""" + if not vars.get("gpg_key_file", None): + vars["gpg_key_file"] = find_gpg_key() + for k, v in vars.items(): + if v is None: + vars[k] = "" + + content = { + "main.keys.tf.j2": """ {% for v in range(1,data.version.id+1) %} resource "aws_iam_access_key" "iam_access_key_v{{ v }}" { {% if (data.version.id-v) <= 1 %} count = 1 {% else %} - count = 0 + count = 0 {% endif %} user = data.aws_iam_user.iam_user.user_name pgp_key = file(var.gpg_key_file) @@ -90,7 +94,7 @@ def init_content(vars): {% if (data.version.id-v) <= 1 %} count = 1 {% else %} - count = 0 + count = 0 {% endif %} provisioner "local-exec" { when = create @@ -100,7 +104,7 @@ def init_content(vars): {% endfor %} """, - "output.keys.tf.j2":""" + "output.keys.tf.j2": """ {% for v in range(1,data.version.id+1) %} {% if data.version.id == 1 %} output "aws_access_key_id_prev" { @@ -139,19 +143,19 @@ def init_content(vars): {% endif %} {% endfor %} """, - 'credentials.tf':""" + "credentials.tf": """ provider "aws" { region = var.region profile = var.profile } """, - 'main.tf':""" + "main.tf": """ data "aws_iam_user" "iam_user" { user_name = var.username } """, - 'outputs.tf':""" + "outputs.tf": """ output "username" { description = "AWS IAM user to rotate" value = var.username @@ -173,14 +177,16 @@ def init_content(vars): } """, - 'variables.auto.tfvars':""" + "variables.auto.tfvars": """ username = "{username}" region = "{region}" profile = "{profile}" service_profile = "{service_profile}" gpg_key_file = "{gpg_key_file}" -""".format(**vars), - 'variables.tf':""" +""".format( + **vars + ), + "variables.tf": """ variable "region" { description = "AWS region" type = string @@ -209,347 +215,558 @@ def init_content(vars): } """, - } - return content + } + return content - -#--- +# --- # find gpg key file -#--- +# --- def find_gpg_key(): - """ - ind gpg key file (base64) from current path, cehcking in ./, ./setup and init (current, one and two levels up) - look for terraformg.gpg.b64 or tf-gpg-key.b64 - returns none if not found - """ - - p_files=['terraform.gpg.b64','tf-gpg-key.b64'] - p_dirs=[x for x in ['./','./setup','./init','../init','../../init'] if os.path.isdir(x)] -# print('p_files {}'.format(p_files)) -# print('p_dirs {}'.format(p_dirs)) - for p_dir in p_dirs: - for p_file in p_files: - file=os.path.join(p_dir,p_file) -# print('* checking file {}'.format(file)) - if os.path.exists(file): - return file - else: - file=None - return file - -#--- + """ + ind gpg key file (base64) from current path, cehcking in ./, ./setup and init (current, one and two levels up) + look for terraformg.gpg.b64 or tf-gpg-key.b64 + returns none if not found + """ + + p_files = ["terraform.gpg.b64", "tf-gpg-key.b64"] + p_dirs = [ + x + for x in ["./", "./setup", "./init", "../init", "../../init"] + if os.path.isdir(x) + ] + # print('p_files {}'.format(p_files)) + # print('p_dirs {}'.format(p_dirs)) + for p_dir in p_dirs: + for p_file in p_files: + file = os.path.join(p_dir, p_file) + # print('* checking file {}'.format(file)) + if os.path.exists(file): + return file + else: + file = None + return file + + +# --- # initialize files for running the rotation -#--- +# --- def initialize_files(content): - status=True - try: - for k in content.keys(): - if not k.endswith('.j2'): - if os.path.exists(k): - print('* file %s already exists, skipping' % k) - else: - with open(k, 'w') as kfile: - kfile.write(content[k]) - print('* file %s created' % k) - except: - status=False - return 0 if status else 1 + status = True + try: + for k in content.keys(): + if not k.endswith(".j2"): + if os.path.exists(k): + print("* file %s already exists, skipping" % k) + else: + with open(k, "w") as kfile: + kfile.write(content[k]) + print("* file %s created" % k) + except: + status = False + return 0 if status else 1 + def get_terraform_output(): - rdata={} -# look for terraform command in TFCOMMAND - tf_command=os.environ.get('TFCOMMAND','terraform') - cmd = f'{tf_command} output -json' - try: - proc = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True) - json_string,_ = proc.communicate() - data = json.loads(json_string) - rdata={x:data[x]['value'] for x in data.keys()} - if rdata['service_profile']!='': - profile=rdata['service_profile'] - else: - profile=rdata['profile'] - rdata['effective_profile']=profile - except: - print("Error getting output from '%s', check outputs.tf or current directory for terraform" % cmd) - return rdata + rdata = {} + # look for terraform command in TFCOMMAND + tf_command = os.environ.get("TFCOMMAND", "terraform") + cmd = f"{tf_command} output -json" + try: + proc = subprocess.Popen( + cmd, + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + shell=True, + ) + json_string, _ = proc.communicate() + data = json.loads(json_string) + rdata = {x: data[x]["value"] for x in data.keys()} + if rdata["service_profile"] != "": + profile = rdata["service_profile"] + else: + profile = rdata["profile"] + rdata["effective_profile"] = profile + except: + print( + "Error getting output from '%s', check outputs.tf or current directory for terraform" + % cmd + ) + return rdata + def decrypt_access_key(rdata): - value=None - if is_gpg_agent_needed() and not os.environ.get('GPG_AGENT_INFO'): - print("Error, please setup GnuPG Agent: eval $(gpg-agent --daemon)") + value = None + if is_gpg_agent_needed() and not os.environ.get("GPG_AGENT_INFO"): + print("Error, please setup GnuPG Agent: eval $(gpg-agent --daemon)") + return value + + try: + gpg = gnupg.GPG(use_agent=True) + aws_secret_e = base64.standard_b64decode(rdata["aws_secret_access_key"].strip()) + value = gpg.decrypt(aws_secret_e) + except: + print("Error decrypting or decoding secret access key: %s" % sys.exc_info()[0]) return value - try: - gpg = gnupg.GPG(use_agent=True) - aws_secret_e=base64.standard_b64decode(rdata['aws_secret_access_key'].strip()) - value=gpg.decrypt(aws_secret_e) - except: - print("Error decrypting or decoding secret access key: %s" % sys.exc_info()[0]) - return value - -def disable_access_key(rdata,key): - status=0 - client=setup_iam_client(rdata,rdata['profile']) - user=rdata['username'] - try: - client.update_access_key(UserName=user,AccessKeyId=key,Status='Inactive') - except: - print("Error disabling access key %s for user %s: %s" % (key,user,sys.exc_info()[0])) - status=1 - return status - -def get_access_key_text(rdata,kdata): - access_key_string='' - aws_secret=decrypt_access_key(rdata) - if aws_secret: - access_key_string+='# profile=%s username=%s\n# date=%s version=%s\n' % (rdata['effective_profile'],rdata['username'],kdata['date'].strftime('%Y%m%d'),kdata['id']) - access_key_string+='aws_access_key_id = %s\n' % rdata['aws_access_key_id'] - access_key_string+='aws_secret_access_key = %s\n' % aws_secret - return access_key_string - -def setup_iam_client(rdata,profile=None): - if profile is None: - profile=rdata['effective_profile'] - session=boto3.Session(profile_name=profile) - client=session.client('iam') - return client - -def list_access_keys(rdata,do_print=True): - user=rdata['username'] - client=setup_iam_client(rdata) - response=client.list_access_keys(UserName=user) - if response.get('AccessKeyMetadata'): - if do_print: - for ak in response['AccessKeyMetadata']: - print('user %(UserName)s access_key_id %(AccessKeyId)s status %(Status)-8s create_date %(CreateDate)s' % ak) - return response['AccessKeyMetadata'] - else: - return [] -def read_yaml(file): - data={} - if not os.path.exists(file): - return None - with open(file, 'r') as stream: +def disable_access_key(rdata, key): + status = 0 + client = setup_iam_client(rdata, rdata["profile"]) + user = rdata["username"] try: - data=yaml.full_load(stream) - except yaml.YAMLError as e: - return None - return data + client.update_access_key(UserName=user, AccessKeyId=key, Status="Inactive") + except: + print( + "Error disabling access key %s for user %s: %s" + % (key, user, sys.exc_info()[0]) + ) + status = 1 + return status + + +def get_access_key_text(rdata, kdata): + access_key_string = "" + aws_secret = decrypt_access_key(rdata) + if aws_secret: + access_key_string += "# profile=%s username=%s\n# date=%s version=%s\n" % ( + rdata["effective_profile"], + rdata["username"], + kdata["date"].strftime("%Y%m%d"), + kdata["id"], + ) + access_key_string += "aws_access_key_id = %s\n" % rdata["aws_access_key_id"] + access_key_string += "aws_secret_access_key = %s\n" % aws_secret + return access_key_string + + +def setup_iam_client(rdata, profile=None): + if profile is None: + profile = rdata["effective_profile"] + session = boto3.Session(profile_name=profile) + client = session.client("iam") + return client + + +def list_access_keys(rdata, do_print=True): + user = rdata["username"] + client = setup_iam_client(rdata) + response = client.list_access_keys(UserName=user) + if response.get("AccessKeyMetadata"): + if do_print: + for ak in response["AccessKeyMetadata"]: + print( + "user %(UserName)s access_key_id %(AccessKeyId)s status %(Status)-8s create_date %(CreateDate)s" + % ak + ) + return response["AccessKeyMetadata"] + else: + return [] + + +def read_yaml(file): + data = {} + if not os.path.exists(file): + return None + with open(file, "r") as stream: + try: + data = yaml.full_load(stream) + except yaml.YAMLError as e: + return None + return data + + +def write_yaml(data, file): + with open(file, "w") as yfile: + yfile.write(yaml.dump(data, default_flow_style=False)) -def write_yaml(data,file): - with open(file, 'w') as yfile: - yfile.write(yaml.dump(data,default_flow_style=False)) def get_account_details(rdata): - profile=rdata['effective_profile'] - account_id=None - try: - session=boto3.Session(profile_name=profile) - sts = session.client('sts') - account_id = sts.get_caller_identity().get('Account') - except: - print("warning: cannot call STS to get account number") - rdata['account_id']=account_id - return account_id + profile = rdata["effective_profile"] + account_id = None + try: + session = boto3.Session(profile_name=profile) + sts = session.client("sts") + account_id = sts.get_caller_identity().get("Account") + except: + print("warning: cannot call STS to get account number") + rdata["account_id"] = account_id + return account_id + def parse_arguments(version): - parser = argparse.ArgumentParser(description="Rotate AWS Keys from YML file",add_help=True) - parser.add_argument('filename', action='store', help="filename to read", default='rotate-keys.yml', nargs='?') - parser.add_argument('--version', action='version', version='%(prog)s v'+version) - parser.add_argument("-i","--info", action="store_true", dest="info", help="Get curent version info only",default=False) - parser.add_argument("-H","--history", action="store_true", dest="history", help="Get history",default=False) - parser.add_argument("-n","--dry-run", action="store_true", dest="dry_run", help="dry run, do not rotate keys and write files", default=False) - parser.add_argument("-d","--debug", action="store_true", dest="debug", help="debugging", default=False) - parser.add_argument("-v","--verbose", action="store_true", dest="verbose", help="verbose output", default=False) -# for initialization - parser.add_argument("-I","--init","--initialize", action="store_true", dest="initialize", help="Initialize environment with TF files",default=False) -# "(do not put spaces before or after the = sign). If a value contains spaces, you should define it with double quotes. values are always treated as strings. - parser.add_argument("--variables", metavar="KEY=VALUE", nargs='+', help="Set key/value pairs for username,profile,region,service_profile,gpg_key_file") - - parser.add_argument("-R","--no-rotate", action="store_true", dest="no_rotate", help="Do not rotate keys, even if expired", default=False) - parser.add_argument("-f","--force-rotate", action="store_true", dest="do_rotate", help="Force rotation of keys, even if not expired", default=False) - parser.add_argument("-D","--rotation-days", action="store", dest="rotation_days", type=int, help="Number of days before rotation required (90)", default=90) - -# for extracting - parser.add_argument("-l","--list", action="store_true", dest="do_list", help="List existing access keys",default=False) - parser.add_argument("-a","--access-key", action="store_true", dest="do_ak", help="output current access/secret key information", default=False) - parser.add_argument("-A","--access-key-file", action="store_true", dest="do_ak_file", help="output current access/secret key information to file aws.USER.ACCOUNT.VERSION.txt", default=False) - parser.add_argument("-x","--disable", action="store", dest="ak_disable", help="Disable access key provided") - parser.add_argument("-X","--disable-oldest", action="store_true", dest="do_ak_disable_oldest", help="Disable oldest access key", default=False) - - args = parser.parse_args() - return args + parser = argparse.ArgumentParser( + description="Rotate AWS Keys from YML file", add_help=True + ) + parser.add_argument( + "filename", + action="store", + help="filename to read", + default="rotate-keys.yml", + nargs="?", + ) + parser.add_argument("--version", action="version", version="%(prog)s v" + version) + parser.add_argument( + "-i", + "--info", + action="store_true", + dest="info", + help="Get curent version info only", + default=False, + ) + parser.add_argument( + "-H", + "--history", + action="store_true", + dest="history", + help="Get history", + default=False, + ) + parser.add_argument( + "-n", + "--dry-run", + action="store_true", + dest="dry_run", + help="dry run, do not rotate keys and write files", + default=False, + ) + parser.add_argument( + "-d", + "--debug", + action="store_true", + dest="debug", + help="debugging", + default=False, + ) + parser.add_argument( + "-v", + "--verbose", + action="store_true", + dest="verbose", + help="verbose output", + default=False, + ) + # for initialization + parser.add_argument( + "-I", + "--init", + "--initialize", + action="store_true", + dest="initialize", + help="Initialize environment with TF files", + default=False, + ) + # "(do not put spaces before or after the = sign). If a value contains spaces, you should define it with double quotes. values are always treated as strings. + parser.add_argument( + "--variables", + metavar="KEY=VALUE", + nargs="+", + help="Set key/value pairs for username,profile,region,service_profile,gpg_key_file", + ) + + parser.add_argument( + "-R", + "--no-rotate", + action="store_true", + dest="no_rotate", + help="Do not rotate keys, even if expired", + default=False, + ) + parser.add_argument( + "-f", + "--force-rotate", + action="store_true", + dest="do_rotate", + help="Force rotation of keys, even if not expired", + default=False, + ) + parser.add_argument( + "-D", + "--rotation-days", + action="store", + dest="rotation_days", + type=int, + help="Number of days before rotation required (90)", + default=90, + ) + + # for extracting + parser.add_argument( + "-l", + "--list", + action="store_true", + dest="do_list", + help="List existing access keys", + default=False, + ) + parser.add_argument( + "-a", + "--access-key", + action="store_true", + dest="do_ak", + help="output current access/secret key information", + default=False, + ) + parser.add_argument( + "-A", + "--access-key-file", + action="store_true", + dest="do_ak_file", + help="output current access/secret key information to file aws.USER.ACCOUNT.VERSION.txt", + default=False, + ) + parser.add_argument( + "-x", + "--disable", + action="store", + dest="ak_disable", + help="Disable access key provided", + ) + parser.add_argument( + "-X", + "--disable-oldest", + action="store_true", + dest="do_ak_disable_oldest", + help="Disable oldest access key", + default=False, + ) + + args = parser.parse_args() + return args + def main(): - version="1.14.0" - this=os.path.basename(sys.argv[0]) - print("# %s v%s" % (this,version)) - args = parse_arguments(version) - init_variables = parse_vars(args.variables) - - do_string='*' if not args.dry_run else '#(dry-run)' - - if args.do_list or args.do_ak or args.do_ak_file or args.ak_disable!=None or args.do_ak_disable_oldest: - do_keystuff=True - else: - do_keystuff=False - - file=args.filename - rotation_days=args.rotation_days - now=datetime.now() - if args.verbose: - print("* reading yaml file %s" % file) - data=read_yaml(file) - if data is None: - print("* initializing: using defaults for yaml file") - data={'version':{'id':0,'date':now},'history':[]} - if data.get('version',None) is None: - data={'version':{'id':0,'date':now}} - print("setting new version") - old_version=data['version']['id'] - old_date=data['version']['date'] - old_age=now-old_date - data['version']['age']=str(old_age) - data['version']['age_seconds']=old_age.total_seconds() - status=0 - - if do_keystuff: - rdata=get_terraform_output() - if len(rdata)==0: - sys.exit(1) - if not rdata.get('account_id'): - get_account_details(rdata) - - user=rdata['username'] - profile=rdata['effective_profile'] - - if args.do_ak: - print('%s new access keys for user %s profile %s\n' % (do_string,user,profile)) - if not args.dry_run: - string=get_access_key_text(rdata,data['version']) - print(string) - if args.do_ak_file: - ak_file='aws.%s.%s.v%s.txt' % (user,rdata['account_id'],data['version']['id']) - print('%s writing new access keys for user %s profile %s to file %s' % (do_string,user,profile,ak_file)) - if not args.dry_run: - string=get_access_key_text(rdata,data['version']) - with open(ak_file, 'w') as keyfile: - keyfile.write(string) - if args.do_list: - print('* existing access keys for user %s profile %s' % (user,profile)) - list_access_keys(rdata) - if args.ak_disable: - print('%s disabling access key %s for user %s profile %s' % (do_string,args.ak_disable,user,profile)) - if not args.dry_run: - disable_access_key(rdata,args.ak_disable) - if args.do_ak_disable_oldest: - keys=list_access_keys(rdata,do_print=False) - if len(keys)>1: - if keys[0]['CreateDate']>keys[1]['CreateDate']: - which_key=keys[1] - else: - which_key=keys[0] - if which_key['Status']=='Inactive': - print('* not disabling oldest access key %s [%s] for user %s profile %s, already disabled' % (which_key['AccessKeyId'],which_key['CreateDate'],user,profile)) - else: - print('%s disabling oldest access key %s [%s] for user %s profile %s' % (do_string,which_key['AccessKeyId'],which_key['CreateDate'],user,profile)) - if not args.dry_run: - disable_access_key(rdata,which_key['AccessKeyId']) - else: - print('* not disabling oldest access key (count=%s) for user %s profile %s ' % (len(keys),user,profile)) - - sys.exit(0) - - if args.verbose: - print("* version = %(id)s, last_rotate_date = %(date)s" % data['version']) - print("* old_date = %s, current_date = %s, age(days) = %s, age(seconds) = %s" % - (old_date,now,old_age,data['version']['age_seconds'])) - - if old_age.days rotation_days %s, rotating" % (do_string,old_age.days,rotation_days)) - - if not data.get('history'): - data['history']=[] - if args.history: - for h in data['history']: - h['age']=h.get('age',-1) - try: - print(" v%(id)s rotate_date=%(date)s by_user=%(by_user)s age=%(age)s" % (h)) - except: - pass - if args.info or args.history: - try: - print("* v%(id)s rotate_date=%(date)s by_user=%(by_user)s age=%(age)s" % (data['version'])) - except: - pass - - if args.info or args.history: - sys.exit(0) - elif status>0: - sys.exit(status) - - content=init_content(init_variables) - if args.initialize: - status=initialize_files(content) - sys.exit(status) - - if do_keystuff: - print("do_keystuff") - else: - data['history'].append(data['version'].copy()) - data['version']['id']+=1 - data['version']['date']=now - data['version']['by_user']=os.environ.get('USER','') - data['version']['script_version']=version - - print("%s rotating from v%s (%s) to v%s (%s)" % - (do_string,old_version,old_date,data['version']['id'],data['version']['date'])) -# -# file_loader=FileSystemLoader('/apps/terraform/template') -# env=Environment( -# loader=file_loader, -# trim_blocks=True, -# lstrip_blocks=True -# ) -# tf_main_keys=env.get_template('main.keys.tf.j2') -# tf_output_keys=env.get_template('output.keys.tf.j2') -# file_loader=BaseLoader() - dict_loader=DictLoader(content) - env=Environment(loader=dict_loader, trim_blocks=True, lstrip_blocks=True) -# tf_main_keys=env.from_string(templates['main.keys.tf']) -# tf_output_keys=env.from_string(templates['output.keys.tf']) - tf_main_keys=env.get_template('main.keys.tf.j2') - tf_output_keys=env.get_template('output.keys.tf.j2') - - tf_output=tf_main_keys.render(data=data) - tf_filename='main.keys.tf' - print("%s creating file %s" % (do_string,tf_filename)) - if not args.dry_run: - with open(tf_filename, 'w') as tf_file: - tf_file.write(tf_output) - - tf_output=tf_output_keys.render(data=data) - tf_filename='output.keys.tf' - print("%s creating file %s" % (do_string,tf_filename)) - if not args.dry_run: - with open(tf_filename, 'w') as tf_file: - tf_file.write(tf_output) - - print("%s creating/updating yaml file %s" % (do_string,file)) - if not args.dry_run: - write_yaml(data,file) - -#--- + version = "1.14.0" + this = os.path.basename(sys.argv[0]) + print("# %s v%s" % (this, version)) + args = parse_arguments(version) + init_variables = parse_vars(args.variables) + + do_string = "*" if not args.dry_run else "#(dry-run)" + + if ( + args.do_list + or args.do_ak + or args.do_ak_file + or args.ak_disable != None + or args.do_ak_disable_oldest + ): + do_keystuff = True + else: + do_keystuff = False + + file = args.filename + rotation_days = args.rotation_days + now = datetime.now() + if args.verbose: + print("* reading yaml file %s" % file) + data = read_yaml(file) + if data is None: + print("* initializing: using defaults for yaml file") + data = {"version": {"id": 0, "date": now}, "history": []} + if data.get("version", None) is None: + data = {"version": {"id": 0, "date": now}} + print("setting new version") + old_version = data["version"]["id"] + old_date = data["version"]["date"] + old_age = now - old_date + data["version"]["age"] = str(old_age) + data["version"]["age_seconds"] = old_age.total_seconds() + status = 0 + + if do_keystuff: + rdata = get_terraform_output() + if len(rdata) == 0: + sys.exit(1) + if not rdata.get("account_id"): + get_account_details(rdata) + + user = rdata["username"] + profile = rdata["effective_profile"] + + if args.do_ak: + print( + "%s new access keys for user %s profile %s\n" + % (do_string, user, profile) + ) + if not args.dry_run: + string = get_access_key_text(rdata, data["version"]) + print(string) + if args.do_ak_file: + ak_file = "aws.%s.%s.v%s.txt" % ( + user, + rdata["account_id"], + data["version"]["id"], + ) + print( + "%s writing new access keys for user %s profile %s to file %s" + % (do_string, user, profile, ak_file) + ) + if not args.dry_run: + string = get_access_key_text(rdata, data["version"]) + with open(ak_file, "w") as keyfile: + keyfile.write(string) + if args.do_list: + print("* existing access keys for user %s profile %s" % (user, profile)) + list_access_keys(rdata) + if args.ak_disable: + print( + "%s disabling access key %s for user %s profile %s" + % (do_string, args.ak_disable, user, profile) + ) + if not args.dry_run: + disable_access_key(rdata, args.ak_disable) + if args.do_ak_disable_oldest: + keys = list_access_keys(rdata, do_print=False) + if len(keys) > 1: + if keys[0]["CreateDate"] > keys[1]["CreateDate"]: + which_key = keys[1] + else: + which_key = keys[0] + if which_key["Status"] == "Inactive": + print( + "* not disabling oldest access key %s [%s] for user %s profile %s, already disabled" + % ( + which_key["AccessKeyId"], + which_key["CreateDate"], + user, + profile, + ) + ) + else: + print( + "%s disabling oldest access key %s [%s] for user %s profile %s" + % ( + do_string, + which_key["AccessKeyId"], + which_key["CreateDate"], + user, + profile, + ) + ) + if not args.dry_run: + disable_access_key(rdata, which_key["AccessKeyId"]) + else: + print( + "* not disabling oldest access key (count=%s) for user %s profile %s " + % (len(keys), user, profile) + ) + + sys.exit(0) + + if args.verbose: + print("* version = %(id)s, last_rotate_date = %(date)s" % data["version"]) + print( + "* old_date = %s, current_date = %s, age(days) = %s, age(seconds) = %s" + % (old_date, now, old_age, data["version"]["age_seconds"]) + ) + + if old_age.days < rotation_days and not args.do_rotate: + print( + "%s age %s < rotation_days %s, not rotating" + % (do_string, old_age.days, rotation_days) + ) + status = int(old_age.days) + elif old_age.days < rotation_days and args.do_rotate: + print( + "%s age %s < rotation_days %s, force rotating" + % (do_string, old_age.days, rotation_days) + ) + else: + print( + "%s age %s > rotation_days %s, rotating" + % (do_string, old_age.days, rotation_days) + ) + + if not data.get("history"): + data["history"] = [] + if args.history: + for h in data["history"]: + h["age"] = h.get("age", -1) + try: + print( + " v%(id)s rotate_date=%(date)s by_user=%(by_user)s age=%(age)s" + % (h) + ) + except: + pass + if args.info or args.history: + try: + print( + "* v%(id)s rotate_date=%(date)s by_user=%(by_user)s age=%(age)s" + % (data["version"]) + ) + except: + pass + + if args.info or args.history: + sys.exit(0) + elif status > 0: + sys.exit(status) + + content = init_content(init_variables) + if args.initialize: + status = initialize_files(content) + sys.exit(status) + + if do_keystuff: + print("do_keystuff") + else: + data["history"].append(data["version"].copy()) + data["version"]["id"] += 1 + data["version"]["date"] = now + data["version"]["by_user"] = os.environ.get("USER", "") + data["version"]["script_version"] = version + + print( + "%s rotating from v%s (%s) to v%s (%s)" + % ( + do_string, + old_version, + old_date, + data["version"]["id"], + data["version"]["date"], + ) + ) + # + # file_loader=FileSystemLoader('/apps/terraform/template') + # env=Environment( + # loader=file_loader, + # trim_blocks=True, + # lstrip_blocks=True + # ) + # tf_main_keys=env.get_template('main.keys.tf.j2') + # tf_output_keys=env.get_template('output.keys.tf.j2') + # file_loader=BaseLoader() + dict_loader = DictLoader(content) + env = Environment(loader=dict_loader, trim_blocks=True, lstrip_blocks=True) + # tf_main_keys=env.from_string(templates['main.keys.tf']) + # tf_output_keys=env.from_string(templates['output.keys.tf']) + tf_main_keys = env.get_template("main.keys.tf.j2") + tf_output_keys = env.get_template("output.keys.tf.j2") + + tf_output = tf_main_keys.render(data=data) + tf_filename = "main.keys.tf" + print("%s creating file %s" % (do_string, tf_filename)) + if not args.dry_run: + with open(tf_filename, "w") as tf_file: + tf_file.write(tf_output) + + tf_output = tf_output_keys.render(data=data) + tf_filename = "output.keys.tf" + print("%s creating file %s" % (do_string, tf_filename)) + if not args.dry_run: + with open(tf_filename, "w") as tf_file: + tf_file.write(tf_output) + + print("%s creating/updating yaml file %s" % (do_string, file)) + if not args.dry_run: + write_yaml(data, file) + + +# --- # main -#--- -if __name__ == '__main__': - main() - +# --- +if __name__ == "__main__": + main() diff --git a/local-app/rotate-keys/setup-rotate-keys.md b/local-app/rotate-keys/setup-rotate-keys.md index 43f407af..5696f367 100644 --- a/local-app/rotate-keys/setup-rotate-keys.md +++ b/local-app/rotate-keys/setup-rotate-keys.md @@ -20,7 +20,7 @@ Initial setup is required (details forthcoming). This is located at `/apps/terraform/bin/setup-rotate-keys.sh`. -* Ansible +* Ansible This is installed with Ansible @@ -141,4 +141,3 @@ gpg: unchanged: 1 gpg: secret keys read: 1 gpg: secret keys imported: 1 ``` - diff --git a/local-app/rotate-keys/setup-rotate-keys.sh b/local-app/rotate-keys/setup-rotate-keys.sh index 1af8bb1d..2e8a8e14 100755 --- a/local-app/rotate-keys/setup-rotate-keys.sh +++ b/local-app/rotate-keys/setup-rotate-keys.sh @@ -19,7 +19,7 @@ then exit 1 fi -# path or name of terraform binary +# path or name of terraform binary # get from top of git repo or $HOME/.tf-control CURRENTDIR=$(pwd) get_git_root @@ -71,7 +71,7 @@ then agent_count=0 else agent_count=$(ps -efww | grep -E "\bgpg-agent\b" | awk '{if ($8 == "gpg-agent") { print $0 }}' | wc -l) -fi +fi # check GPG version 2.2+ doesn't need an agent explicitly running GPG_VERSION=( $(gpg --version |head -n 1|awk '{print $3}' | awk -F. '{print $1,$2,$3}') ) @@ -230,7 +230,7 @@ fi # echo "% git push origin $ADIR" # echo "" # echo "* create PR and merge (not excuting)" -# +# # need to add remote state setup echo "* create zip file with password (specify when prompted)" diff --git a/local-app/terraform-python/base/base.conda-info.txt b/local-app/terraform-python/base/base.conda-info.txt index 6d100a71..6e167f34 100644 --- a/local-app/terraform-python/base/base.conda-info.txt +++ b/local-app/terraform-python/base/base.conda-info.txt @@ -28,5 +28,3 @@ UID:GID : 1043:1408 netrc file : None offline mode : False - - diff --git a/local-app/terraform-python/base/base.revisions.txt b/local-app/terraform-python/base/base.revisions.txt index d7849e9d..1fb1e3da 100644 --- a/local-app/terraform-python/base/base.revisions.txt +++ b/local-app/terraform-python/base/base.revisions.txt @@ -242,4 +242,3 @@ +office365-rest-python-client-2.4.3 (https://nexus.it.census.gov:8443/repository/conda-proxy/conda-forge/noarch) +pyjwt-2.4.0 (https://nexus.it.census.gov:8443/repository/anaconda-proxy/main/linux-64) +pytz-2023.3.post1 (https://nexus.it.census.gov:8443/repository/anaconda-proxy/main/linux-64) - diff --git a/local-app/terraform-python/conda-pre-install.sh b/local-app/terraform-python/conda-pre-install.sh index fd8a3f41..9275de25 100755 --- a/local-app/terraform-python/conda-pre-install.sh +++ b/local-app/terraform-python/conda-pre-install.sh @@ -27,27 +27,27 @@ then echo "* conda list -n $PYENV > $PYENV.txt" conda list -n $PYENV > $PYENV.txt - + echo "* conda list -n $PYENV --revisions > $PYENV.revisions.txt" conda list -n $PYENV --revisions > $PYENV.revisions.txt - + echo "* conda list -n $PYENV --explicit > $PYENV.explicit.txt" conda list -n $PYENV --explicit > $PYENV.explicit.txt - + echo "* conda env export -n $PYENV > $PYENV.yml" conda env export -n $PYENV > $PYENV.yml - + echo "* pip list" > $PYENV.pip.txt pip list > $PYENV.pip.txt echo "* ldd /apps/anaconda/envs/$PYENV > $PYENV.ldd.txt" find /apps/anaconda/envs/$PYENV -name "*.so*" -exec ldd {} \; -print > $PYENV.ldd.txt 2> /dev/null fi - + if [ ! -z $PY_LIST_FILES ] then echo "* list files in enviromment (long)" find /apps/anaconda/envs/$PYENV -print -exec stat --format 'change="%z" modify="%y" %n' {} \; > $PYENV.find.txt && gzip $PYENV.find.txt -fi +fi # script conda.$PYENV.$SDATESTAMP.log diff --git a/local-app/terraform-python/tf-add-root-certificate.py b/local-app/terraform-python/tf-add-root-certificate.py index 817bd18c..472d187b 100644 --- a/local-app/terraform-python/tf-add-root-certificate.py +++ b/local-app/terraform-python/tf-add-root-certificate.py @@ -1,38 +1,39 @@ #!/bin/env python -import shutil import os +import shutil import sys -import certifi from datetime import datetime +import certifi + cafile = certifi.where() print("* certifi ca file %s" % cafile) -file='' -files=[ - '/apps/terraform/etc/census-pki.bundle.crt', - '/etc/pki/tls/certs/census-root-ca.crt', - '/etc/pki/tls/certs/census-root-cert.crt', - '/etc/pki/tls/certs/cacert.crt', - '/etc/openldap/cacerts/cacert.pem' +file = "" +files = [ + "/apps/terraform/etc/census-pki.bundle.crt", + "/etc/pki/tls/certs/census-root-ca.crt", + "/etc/pki/tls/certs/census-root-cert.crt", + "/etc/pki/tls/certs/cacert.crt", + "/etc/openldap/cacerts/cacert.pem", ] for f in files: - if os.path.exists(f): - file=f - break -if file=='': - print("* no certificate file found, exiting") - sys.exit(1) + if os.path.exists(f): + file = f + break +if file == "": + print("* no certificate file found, exiting") + sys.exit(1) else: - print("* using certificate file %s" % file) + print("* using certificate file %s" % file) -backup_timestamp=datetime.now().isoformat() -cafile_backup=f'{cafile}.{backup_timestamp}' -with open(file, 'rb') as infile: - customca = infile.read() -with open(cafile, 'ab') as outfile: - print(f"* backup from {cafile} to {cafile_backup}") - shutil.copyfile(cafile,cafile_backup) - print("* writing new ca file") - outfile.write(customca) +backup_timestamp = datetime.now().isoformat() +cafile_backup = f"{cafile}.{backup_timestamp}" +with open(file, "rb") as infile: + customca = infile.read() +with open(cafile, "ab") as outfile: + print(f"* backup from {cafile} to {cafile_backup}") + shutil.copyfile(cafile, cafile_backup) + print("* writing new ca file") + outfile.write(customca) diff --git a/local-app/terraform-python/tf-install-miniconda.sh b/local-app/terraform-python/tf-install-miniconda.sh index 86956ce2..2504979e 100755 --- a/local-app/terraform-python/tf-install-miniconda.sh +++ b/local-app/terraform-python/tf-install-miniconda.sh @@ -46,4 +46,3 @@ $APPDIR/bin/python tf-add-root-certificate.py # echo "* link python to $BASEDIR" # ln -sf $APPDIR/bin/python $BASEDIR/bin/ echo "* activate with 'source /apps/terraform/python/bin/activate'" - diff --git a/local-app/tf-control/README.md b/local-app/tf-control/README.md index eb350b28..ca93bf18 100644 --- a/local-app/tf-control/README.md +++ b/local-app/tf-control/README.md @@ -21,10 +21,10 @@ which map back to the `tf-control.sh` script: Each of these runs `terraform FUNCTION |& tee logs/FUNCTION.DATESTAMP.log`. You can look at the output of the last run of each function with `tf-FUNCTION less`, and it will use the Linux less command on the last file for that function. -A common use case for these +A common use case for these ```bash -tf-init +tf-init tf-plan tf-plan less tf-apply @@ -158,55 +158,55 @@ Use this where you would normally run `terraform` bare commands. For example, t a locked state. This is important because of the dynamic determination of the TF version used in the repo or current directory. ```console -% tf-cli version -# starting v1.7.0 action cli file logs/cli.20221230.1672411481.log stamp 20221230.1672411481 time 1672411481 -# current_directory=/home/b/badra001/terraform/817869416306-do3-ma4-gov/vpc/east/vpc2 -# git_repository=git@github.e.it.census.gov:terraform/817869416306-do3-ma4-gov.git -# git_current_branch=master -# terraform_version=Terraform v1.3.6 -# TFCONTROL=/home/b/badra001/terraform/817869416306-do3-ma4-gov/vpc/east/vpc2/.tf-control -# TF_CLI_CONFIG_FILE=/home/b/badra001/terraform/817869416306-do3-ma4-gov/vpc/east/vpc2/.tf-control.tfrc -# TFARGS="" TFNOCLOR= TFNOLOG= - -Terraform v1.3.6 -on linux_amd64 -+ provider registry.terraform.io/hashicorp/aws v4.48.0 -+ provider registry.terraform.io/hashicorp/dns v3.2.3 -+ provider registry.terraform.io/hashicorp/external v2.2.3 -+ provider registry.terraform.io/hashicorp/local v2.2.3 -+ provider registry.terraform.io/hashicorp/null v3.2.1 -+ provider registry.terraform.io/hashicorp/random v3.4.3 -+ provider registry.terraform.io/hashicorp/template v2.2.0 -+ provider registry.terraform.io/trevex/ldap v0.5.4 -# ending v1.7.0 action cli file logs/cli.20221230.1672411481.log stamp 20221230.1672411481 start 1672411481 end 1672411481 elapsed 0 - - -# results in file logs/cli.20221230.1672411481.log stamp 20221230.1672411481 status=0 +% tf-cli version +# starting v1.7.0 action cli file logs/cli.20221230.1672411481.log stamp 20221230.1672411481 time 1672411481 +# current_directory=/home/b/badra001/terraform/817869416306-do3-ma4-gov/vpc/east/vpc2 +# git_repository=git@github.e.it.census.gov:terraform/817869416306-do3-ma4-gov.git +# git_current_branch=master +# terraform_version=Terraform v1.3.6 +# TFCONTROL=/home/b/badra001/terraform/817869416306-do3-ma4-gov/vpc/east/vpc2/.tf-control +# TF_CLI_CONFIG_FILE=/home/b/badra001/terraform/817869416306-do3-ma4-gov/vpc/east/vpc2/.tf-control.tfrc +# TFARGS="" TFNOCLOR= TFNOLOG= + +Terraform v1.3.6 +on linux_amd64 ++ provider registry.terraform.io/hashicorp/aws v4.48.0 ++ provider registry.terraform.io/hashicorp/dns v3.2.3 ++ provider registry.terraform.io/hashicorp/external v2.2.3 ++ provider registry.terraform.io/hashicorp/local v2.2.3 ++ provider registry.terraform.io/hashicorp/null v3.2.1 ++ provider registry.terraform.io/hashicorp/random v3.4.3 ++ provider registry.terraform.io/hashicorp/template v2.2.0 ++ provider registry.terraform.io/trevex/ldap v0.5.4 +# ending v1.7.0 action cli file logs/cli.20221230.1672411481.log stamp 20221230.1672411481 start 1672411481 end 1672411481 elapsed 0 + + +# results in file logs/cli.20221230.1672411481.log stamp 20221230.1672411481 status=0 ``` ```console -% tf-cli providers -# starting v1.7.0 action cli file logs/cli.20221230.1672411483.log stamp 20221230.1672411483 time 1672411483 -# current_directory=/home/b/badra001/terraform/817869416306-do3-ma4-gov/vpc/east/vpc2 -# git_repository=git@github.e.it.census.gov:terraform/817869416306-do3-ma4-gov.git -# git_current_branch=master -# terraform_version=Terraform v1.3.6 -# TFCONTROL=/home/b/badra001/terraform/817869416306-do3-ma4-gov/vpc/east/vpc2/.tf-control -# TF_CLI_CONFIG_FILE=/home/b/badra001/terraform/817869416306-do3-ma4-gov/vpc/east/vpc2/.tf-control.tfrc -# TFARGS="" TFNOCLOR= TFNOLOG= - - -Providers required by configuration: -. -├── provider[registry.terraform.io/hashicorp/aws] >= 3.0.0 -├── provider[terraform.io/builtin/terraform] -├── module.nacls_enterprise -│   ├── provider[registry.terraform.io/hashicorp/aws] >= 3.66.0 -│   ├── provider[registry.terraform.io/hashicorp/null] >= 3.0.0 -│   ├── provider[registry.terraform.io/hashicorp/random] >= 3.0.0 -│   ├── provider[registry.terraform.io/hashicorp/template] >= 2.0.0 -│   ├── provider[registry.terraform.io/trevex/ldap] >= 0.5.4 -│   └── provider[registry.terraform.io/hashicorp/local] >= 1.0.0 +% tf-cli providers +# starting v1.7.0 action cli file logs/cli.20221230.1672411483.log stamp 20221230.1672411483 time 1672411483 +# current_directory=/home/b/badra001/terraform/817869416306-do3-ma4-gov/vpc/east/vpc2 +# git_repository=git@github.e.it.census.gov:terraform/817869416306-do3-ma4-gov.git +# git_current_branch=master +# terraform_version=Terraform v1.3.6 +# TFCONTROL=/home/b/badra001/terraform/817869416306-do3-ma4-gov/vpc/east/vpc2/.tf-control +# TF_CLI_CONFIG_FILE=/home/b/badra001/terraform/817869416306-do3-ma4-gov/vpc/east/vpc2/.tf-control.tfrc +# TFARGS="" TFNOCLOR= TFNOLOG= + + +Providers required by configuration: +. +├── provider[registry.terraform.io/hashicorp/aws] >= 3.0.0 +├── provider[terraform.io/builtin/terraform] +├── module.nacls_enterprise +│   ├── provider[registry.terraform.io/hashicorp/aws] >= 3.66.0 +│   ├── provider[registry.terraform.io/hashicorp/null] >= 3.0.0 +│   ├── provider[registry.terraform.io/hashicorp/random] >= 3.0.0 +│   ├── provider[registry.terraform.io/hashicorp/template] >= 2.0.0 +│   ├── provider[registry.terraform.io/trevex/ldap] >= 0.5.4 +│   └── provider[registry.terraform.io/hashicorp/local] >= 1.0.0 . . diff --git a/local-app/tf-control/tf-control.sh b/local-app/tf-control/tf-control.sh index ef1b36f7..b936deec 100755 --- a/local-app/tf-control/tf-control.sh +++ b/local-app/tf-control/tf-control.sh @@ -9,7 +9,7 @@ get_git_root() fi } -do_help() +do_help() { local ACTIONS=$@ echo "* help: $THIS $VERSION" @@ -31,7 +31,7 @@ do_help() for a in $ACTIONS do echo " tf-${a}" - done + done echo "" echo "* Special Actions:" echo " tf-plan summary: produces a list of items to create, destroy, replace, or update. Requires having run 'tf-plan' first" @@ -40,7 +40,7 @@ do_help() return 0 } -# pass things like -target= +# pass things like -target= # make aliases # ln -s $BINDIR/tf-control.sh $BINDIR/tf-init # ln -s $BINDIR/tf-control.sh $BINDIR/tf-plan @@ -64,7 +64,7 @@ LOGDIR="logs" umask 002 -# path or name of terraform binary +# path or name of terraform binary # get from top of git repo or $HOME/.tf-control CURRENTDIR=$(pwd) get_git_root @@ -325,7 +325,7 @@ fi if [ $ACTION == "output" ] then -# $TFCOMMAND output $TFARGS $@ +# $TFCOMMAND output $TFARGS $@ # $TFCOMMAND output $TFARGS $TFCOLOR $@ |& tee -a $LOGFILE $TFCOMMAND output $TFCOLOR $@ |& tee -a $LOGFILE r=$? @@ -345,7 +345,7 @@ then # $TFCOMMAND state $TFARGS $@ |& tee -a $LOGFILE if [ ! -z "$TFNOLOG" ] then - $TFCOMMAND state $@ + $TFCOMMAND state $@ else $TFCOMMAND state $@ |& tee -a $LOGFILE fi diff --git a/local-app/tf-directory-setup/tf-directory-setup.py b/local-app/tf-directory-setup/tf-directory-setup.py index 7c7b5e15..158d3543 100755 --- a/local-app/tf-directory-setup/tf-directory-setup.py +++ b/local-app/tf-directory-setup/tf-directory-setup.py @@ -1,176 +1,229 @@ #!/apps/terraform/python/bin/python # /bin/env python -from jinja2 import Environment,FileSystemLoader +import argparse +import hashlib import os -#import csv -#import re + +# import csv +# import re import sys +from datetime import date, datetime, time +from pathlib import Path from pprint import pprint -from datetime import datetime,date,time + +import yaml from dateutil import tz from dateutil.parser import parse as date_parse -import yaml -import hashlib -import argparse -from pathlib import Path +from jinja2 import Environment, FileSystemLoader + def parse_arguments(version): - parser = argparse.ArgumentParser(description="Setup directory for Terraform (remote state, links)",add_help=True) - parser.add_argument('filename', action='store', help="Configuration filename to read (remote_state.yml)", default='remote_state.yml', nargs='?') - parser.add_argument('--version', action='version', version='%(prog)s '+version) - parser.add_argument("-n","--dry-run", action="store_true", dest="dry_run", help="Dry run, do not create links or remote state configuration", default=False) - parser.add_argument("-d","--debug", action="store_true", dest="debug", help="debugging", default=False) - parser.add_argument("-v","--verbose", action="store_true", dest="verbose", help="verbose output", default=False) - parser.add_argument("-f","--force", action="store_true", dest="force", help="Force", default=False) - parser.add_argument("-l","--link", action="store", dest="link", help="Make link to .tf", choices=['none', 'local', 's3']) - args = parser.parse_args() - return args + parser = argparse.ArgumentParser( + description="Setup directory for Terraform (remote state, links)", add_help=True + ) + parser.add_argument( + "filename", + action="store", + help="Configuration filename to read (remote_state.yml)", + default="remote_state.yml", + nargs="?", + ) + parser.add_argument("--version", action="version", version="%(prog)s " + version) + parser.add_argument( + "-n", + "--dry-run", + action="store_true", + dest="dry_run", + help="Dry run, do not create links or remote state configuration", + default=False, + ) + parser.add_argument( + "-d", + "--debug", + action="store_true", + dest="debug", + help="debugging", + default=False, + ) + parser.add_argument( + "-v", + "--verbose", + action="store_true", + dest="verbose", + help="verbose output", + default=False, + ) + parser.add_argument( + "-f", "--force", action="store_true", dest="force", help="Force", default=False + ) + parser.add_argument( + "-l", + "--link", + action="store", + dest="link", + help="Make link to .tf", + choices=["none", "local", "s3"], + ) + args = parser.parse_args() + return args + def touch_file(file): - if os.path.exists(file): - os.utime(file,None) - else: - open(file,'a').close() + if os.path.exists(file): + os.utime(file, None) + else: + open(file, "a").close() + def read_yaml(file): - data={} - with open(file, 'r') as stream: - try: - data=yaml.full_load(stream) - except yaml.YAMLError as e: - print(e) - return None - return data - -def create_backend(args,version): - data=read_yaml(args.filename) -# initialize missing fields - data['make_links']=data.get('make_links',True) - - if args.debug: - print('* data =') - pprint(data) - print('* args =',args) - print("") - - dry_s="[dry-run] " if args.dry_run else "" - this_dir=os.getcwd() - -# print('args',args) -# sys.exit(0) - - file_loader=FileSystemLoader('/apps/terraform/template') - env=Environment( - loader=file_loader, - trim_blocks=True, - lstrip_blocks=True - ) - - data['directory']=data.get('directory','') - if data['directory'] == "": - print("* error, 'directory' cannot be empty") - sys.exit(1) - - tf_backend=env.get_template('remote_state.backend.tf.j2') - tf_backend_data_local=env.get_template('remote_state.data.tf.local.j2') - tf_backend_data_s3=env.get_template('remote_state.data.tf.s3.j2') - - tf_output=tf_backend.render(data=data) - tf_filename='remote_state.backend.tf' - if os.path.exists(tf_filename): - do_create=args.force - else: - do_create=True - if do_create: - print("* {}creating file {}".format(dry_s,tf_filename)) - if not args.dry_run: - with open(tf_filename, 'w') as tf_file: - tf_file.write(tf_output) - else: - if args.debug or args.verbose: - print("* {}not creating file {}".format(dry_s,tf_filename)) - - d=data['directory'].replace('/','_').replace('.','_') - data['directory_replaced']=d - - base_dir=this_dir.replace(data['directory'],'') - dir_paths=data['directory'].split(os.path.sep) - rp=['..'] * len(dir_paths) - rel_path=os.path.join(*rp) - if args.debug: - print("* this_dir={}\n base_dir={}\n directory={}".format(this_dir,base_dir,data['directory'])) - print(" path_length={}\n relative_path_to_top={}/".format(len(dir_paths),rel_path)) - - - tf_output=tf_backend_data_s3.render(data=data) - tf_filename='remote_state.%s.tf.s3' % d - if do_create: - print("* {}creating file {}".format(dry_s,tf_filename)) - if not args.dry_run: - with open(tf_filename, 'w') as tf_file: - tf_file.write(tf_output) - else: - if args.debug or args.verbose: - print("* {}not creating file {}".format(dry_s,tf_filename)) - - tf_output=tf_backend_data_local.render(data=data) - tf_filename='remote_state.%s.tf.local' % d - if do_create: - print("* {}creating file {}".format(dry_s,tf_filename)) - if not args.dry_run: - with open(tf_filename, 'w') as tf_file: - tf_file.write(tf_output) - else: - if args.debug or args.verbose: - print("* {}not creating file {}".format(dry_s,tf_filename)) - - tf_filename='remote_state.%s.tf.none' % d - if do_create: - print("* {}touching file {}".format(dry_s,tf_filename)) - if not args.dry_run: - touch_file(tf_filename) - else: - if args.debug or args.verbose: - print("* {}not touching file {}".format(dry_s,tf_filename)) - - tf_filename='remote_state.%s.tf' % d - if args.link is not None: - source_file='{}.{}'.format(tf_filename,args.link) + data = {} + with open(file, "r") as stream: + try: + data = yaml.full_load(stream) + except yaml.YAMLError as e: + print(e) + return None + return data + + +def create_backend(args, version): + data = read_yaml(args.filename) + # initialize missing fields + data["make_links"] = data.get("make_links", True) + + if args.debug: + print("* data =") + pprint(data) + print("* args =", args) + print("") + + dry_s = "[dry-run] " if args.dry_run else "" + this_dir = os.getcwd() + + # print('args',args) + # sys.exit(0) + + file_loader = FileSystemLoader("/apps/terraform/template") + env = Environment(loader=file_loader, trim_blocks=True, lstrip_blocks=True) + + data["directory"] = data.get("directory", "") + if data["directory"] == "": + print("* error, 'directory' cannot be empty") + sys.exit(1) + + tf_backend = env.get_template("remote_state.backend.tf.j2") + tf_backend_data_local = env.get_template("remote_state.data.tf.local.j2") + tf_backend_data_s3 = env.get_template("remote_state.data.tf.s3.j2") + + tf_output = tf_backend.render(data=data) + tf_filename = "remote_state.backend.tf" if os.path.exists(tf_filename): - if not os.path.islink(tf_filename): - print("* {}target file {} is not a link, fixing".format(dry_s,tf_filename)) - if not args.dry_run: - os.remove(tf_filename) - if args.verbose: - print("* {}removing file {}".format(dry_s,tf_filename)) - if not args.dry_run: - os.symlink(source_file, tf_filename) - print("* {}link {} to {}".format(dry_s,source_file, tf_filename)) - else: + do_create = args.force + else: + do_create = True if do_create: - print("* sample ln commands to run\n") - print("# ln -sf {}.none {}".format(tf_filename,tf_filename)) - print("# ln -sf {}.local {}".format(tf_filename,tf_filename)) - print("# ln -sf {}.s3 {}".format(tf_filename,tf_filename)) + print("* {}creating file {}".format(dry_s, tf_filename)) + if not args.dry_run: + with open(tf_filename, "w") as tf_file: + tf_file.write(tf_output) + else: + if args.debug or args.verbose: + print("* {}not creating file {}".format(dry_s, tf_filename)) - return + d = data["directory"].replace("/", "_").replace(".", "_") + data["directory_replaced"] = d -#--- + base_dir = this_dir.replace(data["directory"], "") + dir_paths = data["directory"].split(os.path.sep) + rp = [".."] * len(dir_paths) + rel_path = os.path.join(*rp) + if args.debug: + print( + "* this_dir={}\n base_dir={}\n directory={}".format( + this_dir, base_dir, data["directory"] + ) + ) + print( + " path_length={}\n relative_path_to_top={}/".format( + len(dir_paths), rel_path + ) + ) + + tf_output = tf_backend_data_s3.render(data=data) + tf_filename = "remote_state.%s.tf.s3" % d + if do_create: + print("* {}creating file {}".format(dry_s, tf_filename)) + if not args.dry_run: + with open(tf_filename, "w") as tf_file: + tf_file.write(tf_output) + else: + if args.debug or args.verbose: + print("* {}not creating file {}".format(dry_s, tf_filename)) + + tf_output = tf_backend_data_local.render(data=data) + tf_filename = "remote_state.%s.tf.local" % d + if do_create: + print("* {}creating file {}".format(dry_s, tf_filename)) + if not args.dry_run: + with open(tf_filename, "w") as tf_file: + tf_file.write(tf_output) + else: + if args.debug or args.verbose: + print("* {}not creating file {}".format(dry_s, tf_filename)) + + tf_filename = "remote_state.%s.tf.none" % d + if do_create: + print("* {}touching file {}".format(dry_s, tf_filename)) + if not args.dry_run: + touch_file(tf_filename) + else: + if args.debug or args.verbose: + print("* {}not touching file {}".format(dry_s, tf_filename)) + + tf_filename = "remote_state.%s.tf" % d + if args.link is not None: + source_file = "{}.{}".format(tf_filename, args.link) + if os.path.exists(tf_filename): + if not os.path.islink(tf_filename): + print( + "* {}target file {} is not a link, fixing".format( + dry_s, tf_filename + ) + ) + if not args.dry_run: + os.remove(tf_filename) + if args.verbose: + print("* {}removing file {}".format(dry_s, tf_filename)) + if not args.dry_run: + os.symlink(source_file, tf_filename) + print("* {}link {} to {}".format(dry_s, source_file, tf_filename)) + else: + if do_create: + print("* sample ln commands to run\n") + print("# ln -sf {}.none {}".format(tf_filename, tf_filename)) + print("# ln -sf {}.local {}".format(tf_filename, tf_filename)) + print("# ln -sf {}.s3 {}".format(tf_filename, tf_filename)) + + return + + +# --- # main -#--- +# --- def main(): - version='2.2.2' - args=parse_arguments(version) + version = "2.2.2" + args = parse_arguments(version) - create_backend(args,version) - return + create_backend(args, version) + return -#--- + +# --- # main -#--- -if __name__ == '__main__': - main() +# --- +if __name__ == "__main__": + main() # if make_links, then read all links in the parent directory and make a link to them # if link_files [] exists, then make only links to the parent directory for those files @@ -207,13 +260,12 @@ def main(): # - common/apps/myapp1 - ## cwd=Path.cwd() ## top=None ## for p in cwd.parents: ## if (p / "TOP").exists() or ( (p / "init").exists() and (p / "init").is_dir() ): ## top=p -## +## ## if top: ## rel=cwd.relative_to(top) ## rel_top=['..'] * len(rel.parts) @@ -221,5 +273,5 @@ def main(): ## else: ## rel=None ## rel_top_s='' -## +## ## print('cwd={}\ntop={}\nrel={}\nrel_to_top={}'.format(cwd,top,rel,rel_top_s)) diff --git a/local-app/tf-directory-setup/tf-find-top.py b/local-app/tf-directory-setup/tf-find-top.py index eff114ab..915e0efe 100755 --- a/local-app/tf-directory-setup/tf-find-top.py +++ b/local-app/tf-directory-setup/tf-find-top.py @@ -1,70 +1,111 @@ #!/bin/env python -from pathlib import Path +import argparse import os import sys -import argparse +from pathlib import Path + def parse_arguments(version): - parser = argparse.ArgumentParser(description="Find the TOP level of the Terraform configuration directory. Finds only the closest occurence of the desired file.",add_help=True) - parser.add_argument('file', action='store', help="Path file to find (default: TOP or init/)", default='', nargs='?') - parser.add_argument('--version', action='version', version='%(prog)s '+version) - parser.add_argument("-s","--source", action="store_true", dest="source", help="Generate output suitable to be sourced into variables", default=False) - parser.add_argument("-v","--verbose", action="store_true", dest="verbose", help="verbose output", default=False) - parser.add_argument("-d","--direction", action="store", dest="direction", help="Select output path relationship", default='absolute', choices=['absolute','relative']) - args = parser.parse_args() - return args + parser = argparse.ArgumentParser( + description="Find the TOP level of the Terraform configuration directory. Finds only the closest occurence of the desired file.", + add_help=True, + ) + parser.add_argument( + "file", + action="store", + help="Path file to find (default: TOP or init/)", + default="", + nargs="?", + ) + parser.add_argument("--version", action="version", version="%(prog)s " + version) + parser.add_argument( + "-s", + "--source", + action="store_true", + dest="source", + help="Generate output suitable to be sourced into variables", + default=False, + ) + parser.add_argument( + "-v", + "--verbose", + action="store_true", + dest="verbose", + help="verbose output", + default=False, + ) + parser.add_argument( + "-d", + "--direction", + action="store", + dest="direction", + help="Select output path relationship", + default="absolute", + choices=["absolute", "relative"], + ) + args = parser.parse_args() + return args + def find_top(find_path=None): - cwd=Path.cwd() - top=None - for p in cwd.parents: - if find_path is None or find_path=='': - if (p / "TOP").exists() or ( (p / "init").exists() and (p / "init").is_dir() ): - top=p + cwd = Path.cwd() + top = None + for p in cwd.parents: + if find_path is None or find_path == "": + if (p / "TOP").exists() or ( + (p / "init").exists() and (p / "init").is_dir() + ): + top = p + else: + if (p / find_path).exists(): + top = p + + if top: + rel = cwd.relative_to(top) + rel_top = [".."] * len(rel.parts) + rel_top_s = os.path.join(*rel_top) else: - if (p / find_path).exists(): - top=p + rel = None + rel_top_s = "" - if top: - rel=cwd.relative_to(top) - rel_top=['..'] * len(rel.parts) - rel_top_s=os.path.join(*rel_top) - else: - rel=None - rel_top_s='' + return { + "status": top is not None, + "path_current": str(cwd), + "path_top": str(top), + "path_from_top": str(rel), + "path_to_top": rel_top_s, + } - return { - "status": top is not None, - "path_current": str(cwd), - "path_top": str(top), - "path_from_top": str(rel), - "path_to_top": rel_top_s - } -#--- +# --- # main -#--- +# --- def main(): - version='v1.0.0' - args=parse_arguments(version) - o_prefix='TFTOP' + version = "v1.0.0" + args = parse_arguments(version) + o_prefix = "TFTOP" + + # print('cwd={}\ntop={}\nrel={}\nrel_to_top={}'.format(cwd,top,rel,rel_top_s)) + results = find_top(args.file) + if results["status"]: + for k in sorted(results.keys()): + if "path" in k and ( + args.verbose + or (k == "path_top" and args.direction == "absolute") + or (k == "path_to_top" and args.direction == "relative") + ): + if args.source or args.verbose: + print('{}_{}="{}"'.format(o_prefix, k.upper(), results[k])) + else: + print("{}".format(results[k])) + return 0 + else: + return 1 -# print('cwd={}\ntop={}\nrel={}\nrel_to_top={}'.format(cwd,top,rel,rel_top_s)) - results=find_top(args.file) - if results['status']: - for k in sorted(results.keys()): - if 'path' in k and (args.verbose or (k=='path_top' and args.direction=='absolute') or (k=='path_to_top' and args.direction=='relative')): - if args.source or args.verbose: - print('{}_{}="{}"'.format(o_prefix,k.upper(),results[k])) - else: - print('{}'.format(results[k])) - return 0 - else: - return 1 -#--- +# --- # main -#--- -if __name__ == '__main__': - sys.exit(main()) +# --- +if __name__ == "__main__": + sys.exit(main()) diff --git a/local-app/tf-run/README.md b/local-app/tf-run/README.md index 12a116db..28bc661f 100644 --- a/local-app/tf-run/README.md +++ b/local-app/tf-run/README.md @@ -72,10 +72,10 @@ most current TF 1.x. They may be modified to use a specific version (say, as p ### clean -When copying files, or moving files from one directory to another, you often need to start with a fresh set of files. You may not +When copying files, or moving files from one directory to another, you often need to start with a fresh set of files. You may not need to remove the `logs` directory, or an established `.terraform` directory (in the case of a _move_). `clean` will remove all soft links in the current directory, and it will remove all the `remote_state.*` files. As `tf-run` handles the creation -of the remote state files from a parent file (data file directive `REMOTE-STATE`), there is no need for manually maintaining +of the remote state files from a parent file (data file directive `REMOTE-STATE`), there is no need for manually maintaining remote state files. The commands within the `tf-run.data` file are responsible for re-creating the needed links (through calling `setup-new-directory.sh`, which will be rolled into the `tf-directory-setup.py` script in the future). diff --git a/local-app/tf-run/applications/base/tf-run.data b/local-app/tf-run/applications/base/tf-run.data index b83d6236..2121255a 100644 --- a/local-app/tf-run/applications/base/tf-run.data +++ b/local-app/tf-run/applications/base/tf-run.data @@ -17,10 +17,10 @@ LINKTOP includes.d/variables.application_tags.auto.tfvars COMMAND rm -f provider.ldap.* TAG init -COMMAND tf-init +COMMAND tf-init TAG start -#POLICY +#POLICY ALL TAG state-link diff --git a/local-app/tf-run/applications/git-setup/submodule/tf-run.data b/local-app/tf-run/applications/git-setup/submodule/tf-run.data index b659b029..283a32bf 100644 --- a/local-app/tf-run/applications/git-setup/submodule/tf-run.data +++ b/local-app/tf-run/applications/git-setup/submodule/tf-run.data @@ -14,7 +14,7 @@ github_repository.app github_team_repository.apps github_repository_webhook.apps COMMENT extract git url: GITURL=\$(terraform output git_url_ssh) -STOP +STOP TAG clone-new-repo COMMENT clone the repo, add a file, commit and push, then come back and continue at TAG step2 @@ -40,4 +40,3 @@ COMMENT git submodule update --init TAG add-to-git COMMENT add all the newly created submodule stuff into tig - diff --git a/local-app/tf-run/applications/infrastructure/tf-run.data b/local-app/tf-run/applications/infrastructure/tf-run.data index 3798788d..53b80a82 100644 --- a/local-app/tf-run/applications/infrastructure/tf-run.data +++ b/local-app/tf-run/applications/infrastructure/tf-run.data @@ -11,7 +11,7 @@ LINKTOP includes.d/variables.application_tags.tf LINKTOP includes.d/variables.application_tags.auto.tfvars TAG init -COMMAND tf-init +COMMAND tf-init TAG start module.tfstate @@ -23,7 +23,7 @@ TAG to-common STOP go to TOP/common and tf-run.sh at TAG from-infrastructure TAG from-common -LINKTOP common/remote_state.common.tf +LINKTOP common/remote_state.common.tf STOP go to each region at TAG from-infrastructure-main and execute tf-run.sh COMMENT Update git: cd infrastructure, branch add commit push diff --git a/local-app/tf-run/applications/load-balancer/tf-run.data b/local-app/tf-run/applications/load-balancer/tf-run.data index 107e31a1..dfa27b43 100644 --- a/local-app/tf-run/applications/load-balancer/tf-run.data +++ b/local-app/tf-run/applications/load-balancer/tf-run.data @@ -6,7 +6,7 @@ COMMAND tf-init -upgrade module.cert COMMENT submit certs/*.csr file for signature from enterprise PKI COMMENT if provided a link, change app_cert_download to true and continue -COMMENT if provided a .cer or .crt file, drop it into certs/ and continue +COMMENT if provided a .cer or .crt file, drop it into certs/ and continue STOP continue with %%NEXT%% only after the certificate signing is complete module.cert module.cert diff --git a/local-app/tf-run/applications/vpc/tf-run.region.data b/local-app/tf-run/applications/vpc/tf-run.region.data index 8b0c5a1a..1c4594cc 100644 --- a/local-app/tf-run/applications/vpc/tf-run.region.data +++ b/local-app/tf-run/applications/vpc/tf-run.region.data @@ -2,13 +2,13 @@ VERSION 1.1.0 COMMAND tf-directory-setup.py -l none -f # do not execute setup-new-directory.sh # COMMAND setup-new-directory.sh -LINKTOP common/remote_state.common.tf -COMMAND tf-init +LINKTOP common/remote_state.common.tf +COMMAND tf-init TAG remove-defaults module.vpc_defaults COMMAND mv INF.defaults.tf INF.defaults.tf.completed -COMMAND touch INF.defaults.tf +COMMAND touch INF.defaults.tf COMMAND tf-directory-setup.py -l s3 ALL diff --git a/local-app/tf-run/read-run.sh b/local-app/tf-run/read-run.sh index 94c911c6..9f26ac0a 100755 --- a/local-app/tf-run/read-run.sh +++ b/local-app/tf-run/read-run.sh @@ -18,7 +18,7 @@ read_run_data() { echo "$c '$line'" fi c=$(( $c + 1 )) - done + done else status=1 fi diff --git a/local-app/tf-run/run.sh b/local-app/tf-run/run.sh index 2fabad47..6396b89d 100755 --- a/local-app/tf-run/run.sh +++ b/local-app/tf-run/run.sh @@ -59,7 +59,7 @@ then exit 1 fi -if [[ $ACTION == "plan" ]] || [[ $ACTION == "apply" ]] +if [[ $ACTION == "plan" ]] || [[ $ACTION == "apply" ]] then echo "* running action=$ACTION" else @@ -120,7 +120,7 @@ do words=( $t ) w=${words[0]} rest="${words[@]:1}" - case $w in + case $w in COMMAND) echo "* $c $w> $rest" if [ $LIST == 0 ] @@ -146,7 +146,7 @@ do continue ;; STOP) - echo "* $c $w" + echo "* $c $w" if [ $LIST == 0 ] then break @@ -168,7 +168,7 @@ do *) ;; esac - + tfargs="" if [ "$t" != "ALL" ] then @@ -189,7 +189,7 @@ do tf-$ACTION $tfargs else echo " (dry-run)" - fi + fi STATUS=$? if [ $STATUS != 0 ] then diff --git a/local-app/tf-run/tf-run.sh b/local-app/tf-run/tf-run.sh index 4cf21e5e..3c0bc01b 100755 --- a/local-app/tf-run/tf-run.sh +++ b/local-app/tf-run/tf-run.sh @@ -166,7 +166,7 @@ do_clean() then WHAT="clean" fi - + echo "* executing $WHAT, removing remote_state.*" echo -n "> " for f in $(ls remote_state.* -d) @@ -198,7 +198,7 @@ do_superclean() return 0 } -do_help() +do_help() { local ACTIONS=$@ echo "* help: $THIS $VERSION" @@ -284,7 +284,7 @@ fi if [[ ! -z "$ACTION" ]] && [[ "$ACTION" == "help" ]] then - do_help + do_help exit 0 fi @@ -433,10 +433,10 @@ then else echo "NOT-OK" echo "* found $c source statements, please fix to git@ format." - grep -E '^[^#]*source.*git::https' *.tf + grep -E '^[^#]*source.*git::https' *.tf fi echo "" - + echo -n "* [check] for .tf for eligible modules not using ?ref=tf-upgrade: " c=$(cat *.tf | grep -E "^[^#]*source.*git.*($_TFSTRING)" | grep -c -v ref=tf-upgrade) if [ $c == 0 ] @@ -447,7 +447,7 @@ then echo "* found $c source statements not referencing ref=tf-upgrade, please verify with the list at" echo " https://${GITSYSTEM}.e.it.census.gov/terraform/support/blob/master/docs/how-to/terraform-upgrade/upgrade-code.md#modules" echo " and change accordingly if the module here is on the list." - grep -E "^[^#]*source.*git.*($_TFSTRING)" *.tf | grep -v ref=tf-upgrade + grep -E "^[^#]*source.*git.*($_TFSTRING)" *.tf | grep -v ref=tf-upgrade fi echo "" exit 0 @@ -468,7 +468,7 @@ then else echo "* action $ACTION declined" fi - exit 0 + exit 0 fi TFRUNFILE_VERSION="" @@ -484,7 +484,7 @@ then else RUNFILE="tf-run.data" fi - + # read file tf-run.data declare -a targets=() declare -a targets_status=() @@ -659,7 +659,7 @@ do # echo "not break c=$c end=$END" fi - case $w in + case $w in REMOTE-STATE) echo "> [$c] $w> generate-remote-state" if [ $LIST == 0 ] @@ -811,7 +811,7 @@ do continue ;; STOP) - echo "> [$c] $w> $rest" + echo "> [$c] $w> $rest" status=$? if [ $LIST == 0 ] then @@ -823,7 +823,7 @@ do fi ;; CLEAN) - ;; + ;; POLICY) PFILES="${words[@]:1}" if [ -z $PFILES ] @@ -854,7 +854,7 @@ do echo "* error encountered, status=$status; exiting" exit $status fi - + tfargs="" if [ "$t" != "ALL" ] then @@ -875,7 +875,7 @@ do tf-$ACTION $TFOPTIONS $tfargs else echo " (dry-run)" - fi + fi status=$? targets_status[$c]=$status diff --git a/plan.md b/plan.md index fcd64839..54898914 100644 --- a/plan.md +++ b/plan.md @@ -398,7 +398,7 @@ Example README section: ``` make verify-tools ``` - + Make sure you see "All critical tools are available" before proceeding. 2. **Scan your Terraform directories**: @@ -410,7 +410,7 @@ Example README section: ``` make dry-run DIR=/path/to/terraform ``` - + Review the proposed changes carefully. 4. **Perform the upgrade**: @@ -640,7 +640,7 @@ The command-line interface has been extended with commands for assessment: - `tf-upgrade assess-risk` - Perform comprehensive risk assessment - `tf-upgrade dry-run` - Simulate upgrades without making changes -### Reporter System +### Reporter System ✅ **Implemented** in `tf_upgrade/reporters/` @@ -867,7 +867,7 @@ This pragmatic approach balances quality with practicality for a tool with limit While this tool is designed for a one-time operation, some components may be repurposed for future needs: 1. **HCL Transformation Engine**: Could be adapted for future syntax changes -2. **Dependency Graph Generator**: Useful for infrastructure visualization +2. **Dependency Graph Generator**: Useful for infrastructure visualization 3. **Configuration Scanner**: Helpful for infrastructure auditing These components have been designed with clean interfaces that could be extracted and reused if needed. @@ -917,7 +917,7 @@ Before deploying the tool for widespread use, a structured verification process - Check that no files are modified during dry-run 3. **Step-by-Step Mode** - - Run `tf-upgrade upgrade --interactive` + - Run `tf-upgrade upgrade --interactive` - Test aborting the upgrade mid-process - Verify that completed steps remain applied while uncompleted steps are skipped - Test completing the upgrade after an abort @@ -1021,9 +1021,9 @@ for dir in "${TEST_DIRS[@]}"; do echo -e "${YELLOW}!${NC} Test directory not found: $dir" continue fi - + echo "Testing configuration in $dir..." - + # Backup test echo " Testing backup creation..." $TOOL_CMD upgrade --dry-run $dir @@ -1032,7 +1032,7 @@ for dir in "${TEST_DIRS[@]}"; do else echo -e " ${RED}✗${NC} No backup created!" fi - + # Dry-run test echo " Testing dry-run output..." $TOOL_CMD dry-run $dir > /tmp/dryrun-output.txt @@ -1041,7 +1041,7 @@ for dir in "${TEST_DIRS[@]}"; do else echo -e " ${YELLOW}!${NC} No changes detected in dry-run" fi - + echo done diff --git a/providers/terraform-provider-infoblox/README.md b/providers/terraform-provider-infoblox/README.md index 6ac1bc0d..ad10f611 100644 --- a/providers/terraform-provider-infoblox/README.md +++ b/providers/terraform-provider-infoblox/README.md @@ -30,4 +30,3 @@ Resulting binary: ``` # Use - diff --git a/structure/init/README.md b/structure/init/README.md index ed83ef76..9731e1b2 100644 --- a/structure/init/README.md +++ b/structure/init/README.md @@ -1,7 +1,7 @@ # Creation of GPG key for use by Terraform #gpg --gen-key -((find /data | xargs file) 2> /dev/null &); gpg --gen-key --batch --passphrase-file tf-gpg-key-password.txt tf-gpg-gen-key-options.txt +((find /data | xargs file) 2> /dev/null &); gpg --gen-key --batch --passphrase-file tf-gpg-key-password.txt tf-gpg-gen-key-options.txt ### # tf-gpg-gen-key-otpions.txt diff --git a/structure/init/setup-generate-rs-backend.py b/structure/init/setup-generate-rs-backend.py index 3f69107d..a6e2f61b 100755 --- a/structure/init/setup-generate-rs-backend.py +++ b/structure/init/setup-generate-rs-backend.py @@ -1,39 +1,43 @@ #!/bin/env python -from jinja2 import Environment,FileSystemLoader +import hashlib import os -#import csv -#import re + +# import csv +# import re import sys +from datetime import date, datetime, time from pprint import pprint -from datetime import datetime,date,time + +import yaml from dateutil import tz from dateutil.parser import parse as date_parse -import yaml -import hashlib +from jinja2 import Environment, FileSystemLoader + def read_yaml(file): - data={} - with open(file, 'r') as stream: - try: - data=yaml.full_load(stream) - except yaml.YAMLError as e: - print(e) - return None - return data + data = {} + with open(file, "r") as stream: + try: + data = yaml.full_load(stream) + except yaml.YAMLError as e: + print(e) + return None + return data + -if len(sys.argv)>1: - file=sys.argv[1] +if len(sys.argv) > 1: + file = sys.argv[1] else: - file="remote_state.yml" -data=read_yaml(file) + file = "remote_state.yml" +data = read_yaml(file) pprint(data) print("") # subnets={} # indexes={} # units={} -# +# # for s in data['subnets']: # # pprint(s) # # for c in range(0,s['count']): @@ -49,40 +53,36 @@ def read_yaml(file): # # print(n2) # subnets[name]=n2 # indexes[name]=int(s['cidr']['index_start']) -# +# # for sn in n2: # for r in s['cidr']['reserved_start']: # n3=sn[r] # # print('reserved=%s ip=%s' % (r,n3)) -# +# -#--- +# --- # main -#--- -file_loader=FileSystemLoader('./init/template') -env=Environment( - loader=file_loader, - trim_blocks=True, - lstrip_blocks=True -) +# --- +file_loader = FileSystemLoader("./init/template") +env = Environment(loader=file_loader, trim_blocks=True, lstrip_blocks=True) -if data['directory'] == "": - print("* error, 'directory' cannot be empty") - sys.exit(1) +if data["directory"] == "": + print("* error, 'directory' cannot be empty") + sys.exit(1) -tf_backend=env.get_template('remote_state.backend.tf.j2') -tf_backend_data=env.get_template('remote_state.data.tf.j2') +tf_backend = env.get_template("remote_state.backend.tf.j2") +tf_backend_data = env.get_template("remote_state.data.tf.j2") -tf_output=tf_backend.render(data=data) -tf_filename='remote_state.backend.tf.new' +tf_output = tf_backend.render(data=data) +tf_filename = "remote_state.backend.tf.new" print("* creating file %s" % (tf_filename)) -with open(tf_filename, 'w') as tf_file: - tf_file.write(tf_output) +with open(tf_filename, "w") as tf_file: + tf_file.write(tf_output) -d=data['directory'].replace('/','_') -data['directory_replaced']=d -tf_output=tf_backend_data.render(data=data) -tf_filename='remote_state.%s.tf.s3' % d +d = data["directory"].replace("/", "_") +data["directory_replaced"] = d +tf_output = tf_backend_data.render(data=data) +tf_filename = "remote_state.%s.tf.s3" % d print("* creating file %s" % (tf_filename)) -with open(tf_filename, 'w') as tf_file: - tf_file.write(tf_output) +with open(tf_filename, "w") as tf_file: + tf_file.write(tf_output) diff --git a/structure/init/tf-control.sh b/structure/init/tf-control.sh index 712853c5..4fc58bf0 100755 --- a/structure/init/tf-control.sh +++ b/structure/init/tf-control.sh @@ -1,6 +1,6 @@ #!/bin/bash -# pass things like -target= +# pass things like -target= # make aliases # ln -s $BINDIR/tf-control.sh $BINDIR/tf-init # ln -s $BINDIR/tf-control.sh $BINDIR/tf-plan @@ -11,7 +11,7 @@ THIS=$(basename $0) ACTION=$(basename $THIS .sh | sed -e 's/^tf-//') LOGDIR="logs" -# path or name of terraform binary +# path or name of terraform binary # get from $HOME/.tf-control if [ -r $HOME/.tf-control ] then @@ -106,7 +106,7 @@ fi if [ $ACTION == "output" ] then # $TFCOMMAND output $@ |& tee -a $LOGFILE - $TFCOMMAND output $@ + $TFCOMMAND output $@ r=$? exit $r fi diff --git a/terraform-docs-releases/VERSION-0.14 b/terraform-docs-releases/VERSION-0.14 index c39e9c5f..930e3000 100644 --- a/terraform-docs-releases/VERSION-0.14 +++ b/terraform-docs-releases/VERSION-0.14 @@ -1 +1 @@ -0.14.1 \ No newline at end of file +0.14.1 diff --git a/terraform/LICENSE.txt b/terraform/LICENSE.txt index 8142708d..bc88130c 100644 --- a/terraform/LICENSE.txt +++ b/terraform/LICENSE.txt @@ -8,37 +8,37 @@ Licensed Work: Terraform Version 1.6.0 or later. The Licensed Work is (c) HashiCorp, Inc. Additional Use Grant: You may make production use of the Licensed Work, provided Your use does not include offering the Licensed Work to third - parties on a hosted or embedded basis in order to compete with - HashiCorp's paid version(s) of the Licensed Work. For purposes + parties on a hosted or embedded basis in order to compete with + HashiCorp's paid version(s) of the Licensed Work. For purposes of this license: A "competitive offering" is a Product that is offered to third - parties on a paid basis, including through paid support - arrangements, that significantly overlaps with the capabilities - of HashiCorp's paid version(s) of the Licensed Work. If Your - Product is not a competitive offering when You first make it + parties on a paid basis, including through paid support + arrangements, that significantly overlaps with the capabilities + of HashiCorp's paid version(s) of the Licensed Work. If Your + Product is not a competitive offering when You first make it generally available, it will not become a competitive offering - later due to HashiCorp releasing a new version of the Licensed - Work with additional capabilities. In addition, Products that + later due to HashiCorp releasing a new version of the Licensed + Work with additional capabilities. In addition, Products that are not provided on a paid basis are not competitive. - "Product" means software that is offered to end users to manage - in their own environments or offered as a service on a hosted + "Product" means software that is offered to end users to manage + in their own environments or offered as a service on a hosted basis. - "Embedded" means including the source code or executable code - from the Licensed Work in a competitive offering. "Embedded" - also means packaging the competitive offering in such a way - that the Licensed Work must be accessed or downloaded for the + "Embedded" means including the source code or executable code + from the Licensed Work in a competitive offering. "Embedded" + also means packaging the competitive offering in such a way + that the Licensed Work must be accessed or downloaded for the competitive offering to operate. - Hosting or using the Licensed Work(s) for internal purposes - within an organization is not considered a competitive - offering. HashiCorp considers your organization to include all + Hosting or using the Licensed Work(s) for internal purposes + within an organization is not considered a competitive + offering. HashiCorp considers your organization to include all of your affiliates under common control. - For binding interpretive guidance on using HashiCorp products - under the Business Source License, please visit our FAQ. + For binding interpretive guidance on using HashiCorp products + under the Business Source License, please visit our FAQ. (https://www.hashicorp.com/license-faq) Change Date: Four years from the date the Licensed Work is published. Change License: MPL 2.0 diff --git a/testplan.md b/testplan.md index 42c7524f..a0fd1700 100644 --- a/testplan.md +++ b/testplan.md @@ -621,7 +621,7 @@ The Terraform Upgrade Tool is ready for release when: **Objective:** Test upgrade of DynamoDB table definitions **Steps:** -1. Create test fixtures with various DynamoDB table configurations +1. Create test fixtures with various DynamoDB table configurations 2. Test attribute definitions with different types (S, N, B) 3. Verify handling of global secondary indexes 4. Test multiline item declarations with heredoc syntax diff --git a/tests/fixtures/0.12/simple/upgrade-logs/upgrade-progress.json b/tests/fixtures/0.12/simple/upgrade-logs/upgrade-progress.json index fa985564..79c787f6 100644 --- a/tests/fixtures/0.12/simple/upgrade-logs/upgrade-progress.json +++ b/tests/fixtures/0.12/simple/upgrade-logs/upgrade-progress.json @@ -4,4 +4,4 @@ "completed": 0, "total": 0, "status": "in_progress" -} \ No newline at end of file +} diff --git a/tflint-releases/get-tflint.sh b/tflint-releases/get-tflint.sh index 04f330fe..abb01e4c 100755 --- a/tflint-releases/get-tflint.sh +++ b/tflint-releases/get-tflint.sh @@ -1,7 +1,7 @@ #!/bin/bash get_latest_release() { - curl --silent "https://api.github.com/repos/terraform-linters/tflint/releases/latest" | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/' + curl --silent "https://api.github.com/repos/terraform-linters/tflint/releases/latest" | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/' } ARG=$1 @@ -96,4 +96,3 @@ then umask 022 cp tflint_${version} $BINDIR/ && ln -sf tflint_${version} $BINDIR/tflint fi -