From 26ed7d9dc1c883df99bded0f856e6e690bec78e4 Mon Sep 17 00:00:00 2001 From: badra001 Date: Fri, 17 Jul 2026 16:02:55 -0400 Subject: [PATCH] initial --- .../README.check_s3_buckets.md | 209 ++++++++++++++++++ 1 file changed, 209 insertions(+) create mode 100644 local-app/python-tools/cross-organization/README.check_s3_buckets.md diff --git a/local-app/python-tools/cross-organization/README.check_s3_buckets.md b/local-app/python-tools/cross-organization/README.check_s3_buckets.md new file mode 100644 index 00000000..ac7f6ce8 --- /dev/null +++ b/local-app/python-tools/cross-organization/README.check_s3_buckets.md @@ -0,0 +1,209 @@ +# AWS Organization S3 Management & Purgation Suite + +An enterprise-grade, multi-threaded compliance, telemetry, and automated remediation suite for large-scale AWS Organizations. This suite allows cloud infrastructure teams to safely discover, analyze, and purge S3 storage footprints across hundreds of AWS accounts in parallel. + +--- + +## 🛠️ Suite Component Matrix + +1. **`check_s3_buckets.py`** *(Worker Plugin)*: The scanning module executed concurrently across member accounts by the `org_runner3` orchestrator. It queries bucket configuration logic, resource policies, and CloudWatch metrics. +2. **`assess_check_s3_buckets.py`** *(Analytical Assessor)*: A data processing engine that reads raw multi-account JSON data dumps, applies regular expression filters, normalizes metrics into human-readable storage allocations (B, MB, GB, TB, PB), and exports optimization dashboards. +3. **`purge_s3_buckets.py`** *(Remediation Engine)*: A highly precise utility designed to ingest structural telemetry mappings, target specific workloads using regex, and safely execute massive, thread-safe object version deletions. + +--- + +## 📋 Tool Change Log Traceability + +### 🖥️ `assess_check_s3_buckets.py` + +#### v1.4.1 + +* Fixed nested layout variable interpolation error causing positional tuple substitution failures during markdown header drawing. +* Expanded operational reporting metadata tracking boundaries to explicitly list active module configurations alongside script versions. + +#### v1.4.0 + +* Added dual-layer profile analytics in Section 1 to report overall unfiltered global stats alongside filtered target metrics. +* Refined cascading data filter context ensuring all downstream sections only report on filtered matching objects. + +#### v1.3.1 + +* Re-ordered analytical report sections to elevate high-priority leaderboards above master tables. + +#### v1.3.0 + +* Introduced top-20 storage-volume leaderboards sorted descending with positional index ranks. +* Introduced top-20 object-density leaderboards tracking highly populated buckets. + +#### v1.2.1 + +* Fixed float token type crash caused by literal syntax formatting typos within nested account mapping code. + +#### v1.2.0 + +* Added `--bucket-name` optional parameter handling regular expression target namespace filtering. +* Realigned the execution default scale value for `--size-units` to standard Gigabytes (`GB`). + +--- + +### 🧨 `purge_s3_buckets.py` + +#### v2.5.0 + +* Fixed data truncation bug by storing and treating size arrays as floating-point decimals instead of integers, resolving a bug where small sizes (e.g., `0.05 GB`) evaluated to `0`. +* Dynamically extracts and preserves exact column string header identifiers from source datasets (`Size (GB)`, `Size (Bytes)`). + +#### v2.4.0 + +* Implemented white-space cleaning logic over incoming dictionary strings to secure key lookups. +* Fixed numerical text evaluation failures by stripping inline comma thousands-separators before converting strings to numbers. + +#### v2.3.0 + +* Revamped the worker row progress panel layout to drop account string flags, dedicating the visualization slot entirely to unclipped bucket namespaces. +* Lifted strict column length truncation boundaries to prevent clipping on long corporate names. + +#### v2.2.0 + +* Expanded parallel `BarColumn` width boundaries from 25 to 45 to protect long names against terminal clipping. +* Integrated the account alias value into progress swim-lane labels. + +#### v2.1.0 + +* Added structured logging via `--output`, generating matching `.json` and `.csv` telemetry exports. +* Captured millisecond-accurate timestamps (`start_time`, `end_time`, `elapsed_time_seconds`) and unhandled `error_count` tracking properties per bucket resource target. +* Appended comprehensive orchestration platform tracking records (command arguments, global execution metrics) inside the `.json` report metadata tracking block. + +#### v2.0.0 + +* Re-architected code to leverage thread-safe `queue.Queue` communication lines and parallel `rich.live.Live` rendering slots to mirror `org_runner3` visual mechanics. +* Realigned parameters to match standard core orchestration design principles (`--workers`, `--profile`, `--region`, `--role-name`, `--dry-run`). +* Added advanced regex options: `--filter-bucket` and `--filter-account` (targets account IDs or name descriptors). + +--- + +## 🚀 Technical Execution Examples + +### Phase 1: Distributed Fleet Discovery + +Run the orchestrated scan across your AWS Organization via `org_runner3` with parallel worker account lanes: + +```bash +python3 -u org_runner3.py \ + --module check_s3_buckets \ + --role-name OrganizationAccountAccessRole \ + --max-workers 12 \ + --progress-account + +``` + +### Phase 2: Fleet Footprint Assessment & Modeling + +#### Standard Execution + +Generate optimization reports using the standard default `GB` metrics alignment strategy over the latest discovered json file payload: + +```bash +python3 assess_check_s3_buckets.py + +``` + +#### Targeted Analytical Extraction + +Limit calculations to buckets containing the token `prod` or `logs`, and normalize all dashboard visualization blocks directly into Terabytes (`TB`): + +```bash +python3 assess_check_s3_buckets.py \ + --bucket-name "^.*(prod|logs).*$" \ + --size-units TB + +``` + +### Phase 3: Targeted Remediation Purgation + +#### Safe Telemetry Evaluation Dry-Run + +Run a safe dry-run scan across test caches matching dev environments without processing changes, outputting telemetry tracking records to a secure local folder: + +```bash +python3 purge_s3_buckets.py \ + --csv s3_global_inventory_execution.csv \ + --filter-bucket "^dev-.*-temporary-cache$" \ + --filter-account "development-sandbox" \ + --role-name SecOpsRemediationRole \ + --workers 8 \ + --output ./purge_telemetry_logs + +``` + +#### Armed Destructive Execution + +Arm the engine to execute thread-safe object deletions, forcing real-time parallel progress bars to paint the terminal display window grid: + +```bash +python3 purge_s3_buckets.py \ + --csv s3_global_inventory_execution.csv \ + --filter-bucket "^tmp-replication-pipeline$" \ + --role-name SecOpsRemediationRole \ + --workers 16 \ + --output ./purge_telemetry_logs \ + --execute + +``` + +--- + +## 📊 Sample Analytical Report Layout Format + +```text +======================================================================================================================= +AWS ORGANIZATIONAL S3 METRICS STORAGE OPTIMIZATION REPORT +Module: check_s3_buckets | Assessor Version: v1.4.1 | Source: audit_results.check_s3_buckets.20260717T153000.json +======================================================================================================================= + +[SECTION 1: FLEET METRICS PROFILE OVERVIEW] + ▶ Global Unfiltered Footprint: + • Total S3 Buckets Managed : 1450 Buckets + • Cumulative Raw Storage Volume : 4,512.45 GB + • Aggregate Object Count : 12,450,112 Objects + + ▶ Filtered Subset Footprint (Match Pattern: '^.*-temp-.*$'): + • Total Matching S3 Buckets : 42 Buckets + • Cumulative Subset Storage Vol : 12.04 GB + • Subset Target Object Count : 450,000 Objects + +[SECTION 2: ACCOUNT BILLING SECTOR SUMMARY BREAKDOWN] + • Target Account: core-prod-environment (111122223333) | Buckets: 5 | Size (GB): 8.150 | Objects: 250,000 + • Target Account: dev-sandbox-labs (444455556666) | Buckets: 37 | Size (GB): 3.890 | Objects: 200,000 + +[SECTION 3: SPACE DISTRIBUTION BY STORAGE TYPE TIER] + • Storage Tier: StandardStorage | Footprint (GB): 10.040 | Share %: 83.39 + • Storage Tier: IntelligentTieringStorage | Footprint (GB): 2.000 | Share %: 16.61 + +[SECTION 4: TOP 20 LARGEST BUCKETS BY INSTALLED VOLUME FOOTPRINT] + Idx | Account ID | Account Alias | Bucket Name | Region | Size (GB) | Objects + --------------------------------------------------------------------------------------------------------------------- + #1 | 111122223333 | core-prod-environment | core-prod-temp-cache-01 | us-east-1 | 5.12 | 100,000 + #2 | 444455556666 | dev-sandbox-labs | lab-test-temp-analytics | us-west-2 | 2.00 | 50,000 + +[SECTION 5: TOP 20 MOST POPULATED BUCKETS BY ALLOCATED OBJECT DENSITY] + Idx | Account ID | Account Alias | Bucket Name | Region | Size (GB) | Objects + --------------------------------------------------------------------------------------------------------------------- + #1 | 444455556666 | dev-sandbox-labs | lab-test-temp-analytics | us-west-2 | 2.00 | 150,000 + #2 | 111122223333 | core-prod-environment | core-prod-temp-cache-01 | us-east-1 | 5.12 | 100,000 + +[SECTION 6: GRANULAR BUCKET METADATA INVENTORY MATRIX] + Account ID | Account Alias | Bucket Name | Region | Size (GB) | Objects | Version | Lifecycle + -------------------------------------------------------------------------------------------------------------------------------------------------- + 111122223333 | core-prod-environment | core-prod-temp-cache-01 | us-east-1 | 5.12 | 100,000 | Enabled | True + 444455556666 | dev-sandbox-labs | lab-test-temp-analytics | us-west-2 | 2.00 | 50,000 | Suspended | False + +``` + +--- + +## 🔒 Security Best Practices + +1. **Least Privilege Principle**: Ensure that the IAM cross-account role assumed by the engine limits permissions to `s3:ListBucket`, `s3:GetBucketLocation`, `s3:GetBucketVersioning`, `s3:GetBucketLifecycleConfiguration`, and `s3:GetBucketTagging` for scanning. For purging operations, add `s3:DeleteObject` and `s3:DeleteObjectVersion`. +2. **Audit Logging Enabled**: Do not run `org_runner3` with the `--no-logfile` flag in production environments. Retain the auto-generated persistent logs for post-incident security tracking. +3. **Immutable Telemetry Tracking Assets**: Keep file logs output by the `--output` flags inside write-once (WORM) storage locations to provide an untamperable audit record of destructive operations.