-
Notifications
You must be signed in to change notification settings - Fork 8
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
4 changed files
with
313 additions
and
73 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| # Byte-compiled / optimized / DLL files | ||
| __pycache__/ | ||
| *.py[cod] | ||
| *$py.class | ||
|
|
||
| # Distribution / packaging | ||
| dist/ | ||
| build/ | ||
| *.egg-info/ | ||
| *.egg | ||
|
|
||
| # Virtual environments | ||
| venv/ | ||
| env/ | ||
| ENV/ | ||
| .env/ | ||
| .venv/ | ||
|
|
||
| # Unit test / coverage reports | ||
| htmlcov/ | ||
| .tox/ | ||
| .coverage | ||
| .coverage.* | ||
| coverage.xml | ||
| *.cover | ||
| .hypothesis/ | ||
| .pytest_cache/ | ||
| nosetests.xml | ||
|
|
||
| # IDE specific files | ||
| .idea/ | ||
| .vscode/ | ||
| *.swp | ||
| *.swo | ||
| .DS_Store | ||
|
|
||
| # Project specific | ||
| config.local.py | ||
| credentials.json | ||
| *.log | ||
| logs/ | ||
| temp/ | ||
|
|
||
| # AWS | ||
| .aws-sam/ | ||
| samconfig.toml | ||
| .chalice/ | ||
| .aws_credentials | ||
|
|
||
| # Terraform | ||
| .terraform/ | ||
| *.tfstate | ||
| *.tfstate.backup | ||
| *.tfplan | ||
| .terraform.lock.hcl |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,100 +1,156 @@ | ||
| # AWS Organization Resource Management Tool - Gliffy | ||
| # AWS Resource Management Utilities | ||
|
|
||
| This script discovers and manages (start/stop) AWS resources across multiple accounts in an AWS organization. | ||
| A collection of Python utilities for managing and interacting with AWS resources across multiple accounts, with special support for GovCloud environments and AWS SSO. | ||
|
|
||
| ## Prerequisites | ||
| ## Features | ||
|
|
||
| - Cross-account resource management | ||
| - Dynamic AWS partition detection (supports commercial AWS, GovCloud, AWS China) | ||
| - AWS SSO profile integration | ||
| - Fallback mechanisms for authentication | ||
| - Support for AWS Organizations | ||
|
|
||
| ## Requirements | ||
|
|
||
| - Python 3.6+ | ||
| - AWS SDK for Python (boto3) | ||
| - Proper IAM permissions: | ||
| - Access to AWS Organizations API (in the management account) | ||
| - Ability to assume `OrganizationAccountAccessRole` in member accounts | ||
| - Permissions to manage EC2, RDS, EKS, and EMR resources | ||
| - boto3 | ||
| - AWS credentials configured through AWS SSO or with standard credentials | ||
|
|
||
| ## Installation | ||
|
|
||
| ```bash | ||
| # Install dependencies | ||
| 1. Clone this repository or copy the files to your desired location | ||
| 2. Install required dependencies: | ||
| ``` | ||
| pip install boto3 | ||
| ``` | ||
|
|
||
| ## IAM Permissions | ||
| ## Usage | ||
|
|
||
| Ensure your IAM user/role has the following permissions: | ||
| - `organizations:ListAccounts` | ||
| - `sts:AssumeRole` | ||
| - Permission to assume a role in each target account | ||
| ### Basic Usage | ||
|
|
||
| The role being assumed in each account needs: | ||
| - `ec2:DescribeInstances`, `ec2:StopInstances`, `ec2:StartInstances` | ||
| - `rds:DescribeDBInstances`, `rds:StopDBInstance`, `rds:StartDBInstance` | ||
| - `eks:ListClusters`, `eks:DescribeCluster`, `eks:ListNodegroups`, `eks:DescribeNodegroup` | ||
| - `autoscaling:DescribeAutoScalingGroups`, `autoscaling:UpdateAutoScalingGroup` | ||
| - `emr:ListClusters`, `emr:DescribeCluster`, `emr:StopCluster`, `emr:StartCluster`, `emr:TerminateJobFlows` | ||
| ```python | ||
| import aws_utils | ||
|
|
||
| ## Usage | ||
| # Create a client using SSO profiles or role assumption | ||
| ec2_client = aws_utils.create_boto3_client_for_account( | ||
| '123456789012', # AWS account ID | ||
| 'ec2', # AWS service | ||
| 'us-gov-west-1' # AWS region | ||
| ) | ||
|
|
||
| # Use the client | ||
| instances = ec2_client.describe_instances() | ||
| ``` | ||
|
|
||
| ### Using AWS SSO Profiles | ||
|
|
||
| The library will automatically find and use appropriate AWS SSO profiles from your AWS config file (`~/.aws/config`). It will: | ||
|
|
||
| 1. Look for profiles matching the requested account ID | ||
| 2. If a role_name is specified, it will find a profile with that role | ||
| 3. Otherwise, it will prefer profiles with admin permissions | ||
| 4. Fall back to role assumption if no profile is found or profile authentication fails | ||
|
|
||
| ### Key Functions | ||
|
|
||
| #### `create_boto3_client_for_account(account_id, service, region=None, role_name=None)` | ||
|
|
||
| Creates a boto3 client for the specified AWS service in the target account. | ||
|
|
||
| - `account_id`: Target AWS account ID | ||
| - `service`: AWS service name (e.g., 'ec2', 'rds') | ||
| - `region`: AWS region (defaults to config.DEFAULT_REGION) | ||
| - `role_name`: Specific role to assume or look for in SSO profiles | ||
|
|
||
| #### `get_partition_for_region(region)` | ||
|
|
||
| Determines the correct AWS partition for a given region. | ||
|
|
||
| ```bash | ||
| # Stop resources (default action) | ||
| python aws_org_resource_shutdown.py | ||
| - `region`: AWS region name | ||
| - Returns: 'aws', 'aws-us-gov', or 'aws-cn' | ||
|
|
||
| # Start resources | ||
| python aws_org_resource_shutdown.py --action start | ||
| #### `find_profile_for_account(account_id, role_name=None)` | ||
|
|
||
| # List all resources without modifying them (dry run) | ||
| python aws_org_resource_shutdown.py --dry-run | ||
| Finds an SSO profile in the AWS config file for the specified account. | ||
|
|
||
| # Specify regions to scan | ||
| python aws_org_resource_shutdown.py --regions us-east-1,us-west-2 | ||
| - `account_id`: AWS account ID to find profile for | ||
| - `role_name`: Optional preferred role name | ||
| - Returns: Profile name if found, None otherwise | ||
|
|
||
| # Exclude specific accounts | ||
| python aws_org_resource_shutdown.py --exclude-accounts 123456789012,098765432109 | ||
| #### `assume_role(account_id, region=None, role_name=None)` | ||
|
|
||
| # Exclude specific resource types | ||
| python aws_org_resource_shutdown.py --exclude-resources ec2 eks emr | ||
| Gets credentials for the specified account, trying SSO profiles first. | ||
|
|
||
| - `account_id`: AWS account ID | ||
| - `region`: AWS region for determining partition | ||
| - `role_name`: Role name to assume | ||
| - Returns: Credentials dictionary | ||
|
|
||
| #### `get_organization_accounts(exclude_accounts=None)` | ||
|
|
||
| Gets a list of accounts in the AWS organization. | ||
|
|
||
| - `exclude_accounts`: List of account IDs to exclude | ||
| - Returns: List of dictionaries with account ID and name | ||
|
|
||
| ## Configuration | ||
|
|
||
| Create a `config.py` file with the following variables: | ||
|
|
||
| ```python | ||
| # Default AWS region to use | ||
| DEFAULT_REGION = 'us-gov-east-1' | ||
|
|
||
| # Default role name to assume if one is not provided | ||
| ASSUME_ROLE_NAME = 'OrganizationAccountAccessRole' | ||
| ``` | ||
|
|
||
| ## Makefile Usage | ||
|
|
||
| The project includes a Makefile to simplify common operations: | ||
|
|
||
| # Combine options | ||
| python aws_org_resource_shutdown.py --action start --regions us-east-1 --exclude-resources rds --dry-run | ||
| ``` | ||
| # Install dependencies | ||
| make install | ||
| ## How It Works | ||
| # Run tests | ||
| make test | ||
| 1. Connects to AWS Organizations API to list all member accounts | ||
| 2. For each account: | ||
| - Assumes the `OrganizationAccountAccessRole` role | ||
| - Queries for EC2 instances, RDS instances, EKS clusters, and EMR clusters | ||
| - Performs shutdown or startup operations based on the selected action (if not in dry-run mode) | ||
| # Run linting | ||
| make lint | ||
| ### Tag-Based Exclusion | ||
| # Format code | ||
| make format | ||
| Resources with the tag `gfl_exclude` will be automatically excluded from stop/start actions. The value of the tag is not checked, only its presence. This allows you to mark critical resources that should not be automatically managed. | ||
| # Clean up temporary files | ||
| make clean | ||
| Additionally, EC2 instances with the following tags are automatically excluded from direct EC2 management: | ||
| - `eks:cluster-name` - These instances are managed by EKS and should only be scaled through the EKS API | ||
| - `aws:elasticmapreduce:job-flow-id` - These instances are managed by EMR and should only be controlled through the EMR API | ||
| # Build distribution package | ||
| make dist | ||
| ### Stop Action (Default) | ||
| - EC2: Stops running instances (unless they have exclusion tags or are service-managed) | ||
| - RDS: Stops available database instances (unless they have the exclusion tag) | ||
| - EKS: Scales down node groups to 0 (unless the cluster has the exclusion tag) | ||
| - EMR: Stops clusters if in RUNNING or WAITING state, terminates if in STARTING or BOOTSTRAPPING state | ||
| # Deploy to development environment | ||
| make deploy-dev | ||
| ### Start Action | ||
| - EC2: Starts stopped instances (unless they have exclusion tags or are service-managed) | ||
| - RDS: Starts stopped database instances (unless they have the exclusion tag) | ||
| - EKS: Scales up node groups using original sizes from tags (unless the cluster has the exclusion tag) | ||
| - EMR: Starts stopped clusters | ||
| # Deploy to production environment | ||
| make deploy-prod | ||
| ``` | ||
|
|
||
| The Makefile helps standardize development workflows and ensures consistent environment setup across different machines. To view all available commands: | ||
|
|
||
| ## Logging | ||
| ``` | ||
| make help | ||
| ``` | ||
|
|
||
| The script logs all actions to both the console and a file named `aws_resource_management.log`. | ||
| ## Tips for GovCloud Usage | ||
|
|
||
| ## Safety Features | ||
| When working with GovCloud: | ||
|
|
||
| - Dry-run mode to list resources without modifying them | ||
| - Ability to exclude specific accounts or resource types | ||
| - Detailed logging of all actions | ||
| 1. Ensure your AWS SSO is properly configured | ||
| 2. AWS SSO profiles should include the GovCloud regions in their configuration | ||
| 3. The library will automatically detect GovCloud regions and use the correct partition ('aws-us-gov') | ||
|
|
||
| ## Note | ||
| ## Troubleshooting | ||
|
|
||
| This script requires proper IAM permissions and should be tested in a non-production environment first. | ||
| - **Authentication failures**: Check that your SSO session is active (`aws sso login --profile your-profile`) | ||
| - **Profile not found**: Verify your `~/.aws/config` file contains the correct account IDs | ||
| - **Permission issues**: Ensure the roles in your SSO profiles have the necessary permissions |
Oops, something went wrong.