initial

terraform · Jul 17, 2026 · 26ed7d9 · 26ed7d9
1 parent a13b657
commit 26ed7d9
Showing 1 changed file with 209 additions and 0 deletions.
diff --git a/local-app/python-tools/cross-organization/README.check_s3_buckets.md b/local-app/python-tools/cross-organization/README.check_s3_buckets.md
@@ -0,0 +1,209 @@
+# AWS Organization S3 Management & Purgation Suite
+
+An enterprise-grade, multi-threaded compliance, telemetry, and automated remediation suite for large-scale AWS Organizations. This suite allows cloud infrastructure teams to safely discover, analyze, and purge S3 storage footprints across hundreds of AWS accounts in parallel.
+
+---
+
+## 🛠️ Suite Component Matrix
+
+1. **`check_s3_buckets.py`** *(Worker Plugin)*: The scanning module executed concurrently across member accounts by the `org_runner3` orchestrator. It queries bucket configuration logic, resource policies, and CloudWatch metrics.
+2. **`assess_check_s3_buckets.py`** *(Analytical Assessor)*: A data processing engine that reads raw multi-account JSON data dumps, applies regular expression filters, normalizes metrics into human-readable storage allocations (B, MB, GB, TB, PB), and exports optimization dashboards.
+3. **`purge_s3_buckets.py`** *(Remediation Engine)*: A highly precise utility designed to ingest structural telemetry mappings, target specific workloads using regex, and safely execute massive, thread-safe object version deletions.
+
+---
+
+## 📋 Tool Change Log Traceability
+
+### 🖥️ `assess_check_s3_buckets.py`
+
+#### v1.4.1
+
+* Fixed nested layout variable interpolation error causing positional tuple substitution failures during markdown header drawing.
+* Expanded operational reporting metadata tracking boundaries to explicitly list active module configurations alongside script versions.
+
+#### v1.4.0
+
+* Added dual-layer profile analytics in Section 1 to report overall unfiltered global stats alongside filtered target metrics.
+* Refined cascading data filter context ensuring all downstream sections only report on filtered matching objects.
+
+#### v1.3.1
+
+* Re-ordered analytical report sections to elevate high-priority leaderboards above master tables.
+
+#### v1.3.0
+
+* Introduced top-20 storage-volume leaderboards sorted descending with positional index ranks.
+* Introduced top-20 object-density leaderboards tracking highly populated buckets.
+
+#### v1.2.1
+
+* Fixed float token type crash caused by literal syntax formatting typos within nested account mapping code.
+
+#### v1.2.0
+
+* Added `--bucket-name` optional parameter handling regular expression target namespace filtering.
+* Realigned the execution default scale value for `--size-units` to standard Gigabytes (`GB`).
+
+---
+
+### 🧨 `purge_s3_buckets.py`
+
+#### v2.5.0
+
+* Fixed data truncation bug by storing and treating size arrays as floating-point decimals instead of integers, resolving a bug where small sizes (e.g., `0.05 GB`) evaluated to `0`.
+* Dynamically extracts and preserves exact column string header identifiers from source datasets (`Size (GB)`, `Size (Bytes)`).
+
+#### v2.4.0
+
+* Implemented white-space cleaning logic over incoming dictionary strings to secure key lookups.
+* Fixed numerical text evaluation failures by stripping inline comma thousands-separators before converting strings to numbers.
+
+#### v2.3.0
+
+* Revamped the worker row progress panel layout to drop account string flags, dedicating the visualization slot entirely to unclipped bucket namespaces.
+* Lifted strict column length truncation boundaries to prevent clipping on long corporate names.
+
+#### v2.2.0
+
+* Expanded parallel `BarColumn` width boundaries from 25 to 45 to protect long names against terminal clipping.
+* Integrated the account alias value into progress swim-lane labels.
+
+#### v2.1.0
+
+* Added structured logging via `--output`, generating matching `.json` and `.csv` telemetry exports.
+* Captured millisecond-accurate timestamps (`start_time`, `end_time`, `elapsed_time_seconds`) and unhandled `error_count` tracking properties per bucket resource target.
+* Appended comprehensive orchestration platform tracking records (command arguments, global execution metrics) inside the `.json` report metadata tracking block.
+
+#### v2.0.0
+
+* Re-architected code to leverage thread-safe `queue.Queue` communication lines and parallel `rich.live.Live` rendering slots to mirror `org_runner3` visual mechanics.
+* Realigned parameters to match standard core orchestration design principles (`--workers`, `--profile`, `--region`, `--role-name`, `--dry-run`).
+* Added advanced regex options: `--filter-bucket` and `--filter-account` (targets account IDs or name descriptors).
+
+---
+
+## 🚀 Technical Execution Examples
+
+### Phase 1: Distributed Fleet Discovery
+
+Run the orchestrated scan across your AWS Organization via `org_runner3` with parallel worker account lanes:
+
+```bash
+python3 -u org_runner3.py \
+  --module check_s3_buckets \
+  --role-name OrganizationAccountAccessRole \
+  --max-workers 12 \
+  --progress-account
+
+```
+
+### Phase 2: Fleet Footprint Assessment & Modeling
+
+#### Standard Execution
+
+Generate optimization reports using the standard default `GB` metrics alignment strategy over the latest discovered json file payload:
+
+```bash
+python3 assess_check_s3_buckets.py
+
+```
+
+#### Targeted Analytical Extraction
+
+Limit calculations to buckets containing the token `prod` or `logs`, and normalize all dashboard visualization blocks directly into Terabytes (`TB`):
+
+```bash
+python3 assess_check_s3_buckets.py \
+  --bucket-name "^.*(prod|logs).*$" \
+  --size-units TB
+
+```
+
+### Phase 3: Targeted Remediation Purgation
+
+#### Safe Telemetry Evaluation Dry-Run
+
+Run a safe dry-run scan across test caches matching dev environments without processing changes, outputting telemetry tracking records to a secure local folder:
+
+```bash
+python3 purge_s3_buckets.py \
+  --csv s3_global_inventory_execution.csv \
+  --filter-bucket "^dev-.*-temporary-cache$" \
+  --filter-account "development-sandbox" \
+  --role-name SecOpsRemediationRole \
+  --workers 8 \
+  --output ./purge_telemetry_logs
+
+```
+
+#### Armed Destructive Execution
+
+Arm the engine to execute thread-safe object deletions, forcing real-time parallel progress bars to paint the terminal display window grid:
+
+```bash
+python3 purge_s3_buckets.py \
+  --csv s3_global_inventory_execution.csv \
+  --filter-bucket "^tmp-replication-pipeline$" \
+  --role-name SecOpsRemediationRole \
+  --workers 16 \
+  --output ./purge_telemetry_logs \
+  --execute
+
+```
+
+---
+
+## 📊 Sample Analytical Report Layout Format
+
+```text
+=======================================================================================================================
+AWS ORGANIZATIONAL S3 METRICS STORAGE OPTIMIZATION REPORT
+Module: check_s3_buckets | Assessor Version: v1.4.1 | Source: audit_results.check_s3_buckets.20260717T153000.json
+=======================================================================================================================
+
+[SECTION 1: FLEET METRICS PROFILE OVERVIEW]
+  ▶ Global Unfiltered Footprint:
+     • Total S3 Buckets Managed       : 1450 Buckets
+     • Cumulative Raw Storage Volume  : 4,512.45 GB
+     • Aggregate Object Count         : 12,450,112 Objects
+
+  ▶ Filtered Subset Footprint (Match Pattern: '^.*-temp-.*$'):
+     • Total Matching S3 Buckets      : 42 Buckets
+     • Cumulative Subset Storage Vol  : 12.04 GB
+     • Subset Target Object Count     : 450,000 Objects
+
+[SECTION 2: ACCOUNT BILLING SECTOR SUMMARY BREAKDOWN]
+     • Target Account: core-prod-environment (111122223333)       | Buckets:    5 | Size (GB):        8.150 | Objects:        250,000
+     • Target Account: dev-sandbox-labs (444455556666)          | Buckets:   37 | Size (GB):        3.890 | Objects:        200,000
+
+[SECTION 3: SPACE DISTRIBUTION BY STORAGE TYPE TIER]
+     • Storage Tier: StandardStorage                | Footprint (GB):            10.040 | Share %:    83.39
+     • Storage Tier: IntelligentTieringStorage      | Footprint (GB):             2.000 | Share %:    16.61
+
+[SECTION 4: TOP 20 LARGEST BUCKETS BY INSTALLED VOLUME FOOTPRINT]
+  Idx  | Account ID   | Account Alias         | Bucket Name                  | Region       | Size (GB)          | Objects     
+  ---------------------------------------------------------------------------------------------------------------------
+  #1   | 111122223333 | core-prod-environment | core-prod-temp-cache-01      | us-east-1    |               5.12 |      100,000
+  #2   | 444455556666 | dev-sandbox-labs      | lab-test-temp-analytics      | us-west-2    |               2.00 |       50,000
+
+[SECTION 5: TOP 20 MOST POPULATED BUCKETS BY ALLOCATED OBJECT DENSITY]
+  Idx  | Account ID   | Account Alias         | Bucket Name                  | Region       | Size (GB)          | Objects     
+  ---------------------------------------------------------------------------------------------------------------------
+  #1   | 444455556666 | dev-sandbox-labs      | lab-test-temp-analytics      | us-west-2    |               2.00 |      150,000
+  #2   | 111122223333 | core-prod-environment | core-prod-temp-cache-01      | us-east-1    |               5.12 |      100,000
+
+[SECTION 6: GRANULAR BUCKET METADATA INVENTORY MATRIX]
+  Account ID   | Account Alias         | Bucket Name                  | Region       | Size (GB)          | Objects      | Version    | Lifecycle    
+  --------------------------------------------------------------------------------------------------------------------------------------------------
+  111122223333 | core-prod-environment | core-prod-temp-cache-01      | us-east-1    |               5.12 |      100,000 | Enabled    | True         
+  444455556666 | dev-sandbox-labs      | lab-test-temp-analytics      | us-west-2    |               2.00 |       50,000 | Suspended  | False        
+
+```
+
+---
+
+## 🔒 Security Best Practices
+
+1. **Least Privilege Principle**: Ensure that the IAM cross-account role assumed by the engine limits permissions to `s3:ListBucket`, `s3:GetBucketLocation`, `s3:GetBucketVersioning`, `s3:GetBucketLifecycleConfiguration`, and `s3:GetBucketTagging` for scanning. For purging operations, add `s3:DeleteObject` and `s3:DeleteObjectVersion`.
+2. **Audit Logging Enabled**: Do not run `org_runner3` with the `--no-logfile` flag in production environments. Retain the auto-generated persistent logs for post-incident security tracking.
+3. **Immutable Telemetry Tracking Assets**: Keep file logs output by the `--output` flags inside write-once (WORM) storage locations to provide an untamperable audit record of destructive operations.