fix: use internal visibility for GHE repo creation; add REPO_VISIBILI…

…TY env var

GHE enterprise policy blocks private repo creation for org members.
Switch default visibility to 'internal' (visible to org members, not public)
which complies with both the enterprise policy and security posture.

Changes:
- repository_provider.py: _default_visibility() reads REPO_VISIBILITY env var,
  defaults to 'internal' instead of hardcoded 'private'
- github_provider.py: use effective_visibility so 'private' boolean field is
  False for 'internal' repos (previously always True, causing GHE 403)
- app.py: explicitly pass RepositorySettings(visibility=REPO_VISIBILITY) into
  get_repository() so env var is honoured by the running container
- packer.pkr.hcl: bake Census Bureau Root CA cert into container trust store
  so VERIFY_SSL=true can be re-enabled after next rebuild
- deploy/terraform.tfvars: GITHUB_API without /api/v3 (code appends it)

New scripts:
- scripts/test_workflow.py: end-to-end 6-step workflow tester with Rich UI
- scripts/check_github_permissions.py: 11-check GitHub PAT permissions verifier

Verified: Lambda successfully creates internal repos in SCT-Engineering org

deploy/terraform.tfvars

@@ -2,9 +2,11 @@
 
 aws_region = "us-gov-west-1"
 
-github_api_url           = "https://github.e.it.census.gov/api/v3" # Your GitHub Enterprise URL
+github_api_url           = "https://github.e.it.census.gov" # Your GitHub Enterprise URL (code adds /api/v3 automatically)
 github_org_name          = "SCT-Engineering"                       # Your GitHub organization name
 template_repo_name       = "template-eks-cluster"
+# GHE enterprise policy blocks 'private' repo creation for org members; use 'internal'
+repo_visibility          = "internal"
 create_service_catalog   = true
 github_token_secret_name = "/eks-cluster-deployment/github_token"
 

docs/SERVICE_CATALOG_RESOLUTION.md

@@ -0,0 +1,153 @@
+# Service Catalog S3 Access Issue - RESOLVED ✅
+
+**Date:** February 11, 2026  
+**Resolved By:** David Arnold & DeLong  
+**Issue Duration:** ~2 days
+
+## Problem Summary
+
+Service Catalog product deployment was failing with S3 Access Denied errors when attempting to launch the GitHub Repository Creator product. The CloudFormation template was uploaded to the enterprise artifacts bucket but Service Catalog couldn't read it during provisioning.
+
+## Root Cause
+
+The issue was **tag-based access control** enforced by Service Control Policies (SCPs) in the GovCloud environment. The launch role's IAM policy was missing a required condition that restricts S3 access to only objects tagged with `servicecatalog:provisioning = true`.
+
+### Key Insight
+All working Service Catalog products in the enterprise bucket had the `servicecatalog:provisioning = true` tag on their templates, while our newly uploaded template did not.
+
+## Solution Implemented
+
+### 1. Updated Launch Role IAM Policy
+
+Changed the `S3ReadTemplate` statement in `deploy/service_catalog.tf`:
+
+**Before:**
+```hcl
+{
+  Sid    = "S3ReadTemplate"
+  Effect = "Allow"
+  Action = [
+    "s3:GetObject"
+  ]
+  Resource = "arn:${data.aws_partition.current.partition}:s3:::${local.artifacts_bucket}/*"
+}
+```
+
+**After:**
+```hcl
+{
+  Sid    = "S3ReadTemplate"
+  Effect = "Allow"
+  Action = [
+    "s3:GetObject"
+  ]
+  Resource = "*"
+  Condition = {
+    "StringEquals" = {
+      "s3:ExistingObjectTag/servicecatalog:provisioning" = ["true"]
+    }
+  }
+}
+```
+
+### 2. Updated S3 Object Configuration
+
+Modified the `aws_s3_object.product_template` resource to automatically add the required tag:
+
+```hcl
+resource "aws_s3_object" "product_template" {
+  count  = local.create_sc ? 1 : 0
+  bucket = local.artifacts_bucket
+  key    = local.product_s3_key
+  source = local.product_template
+  etag   = filemd5(local.product_template)
+
+  tags = merge(var.tags, {
+    "servicecatalog:provisioning" = "true"
+  })
+}
+```
+
+## Verification
+
+After applying the changes:
+
+1. ✅ S3 object has the required tag:
+   ```json
+   {
+     "servicecatalog:provisioning": "true",
+     "ManagedBy": "Terraform",
+     "Environment": "production",
+     "Purpose": "ServiceCatalogRepoGenerator"
+   }
+   ```
+
+2. ✅ Service Catalog can read the template:
+   ```bash
+   aws servicecatalog describe-provisioning-parameters \
+     --product-id prod-w3uvfaxmeblxe \
+     --provisioning-artifact-name "v1.0" \
+     --path-id "lpv3-uchiqj7m3d57k" \
+     --region us-gov-west-1
+   ```
+   Returns all CloudFormation parameters successfully.
+
+3. ✅ Launch constraints properly configured:
+   - LAUNCH constraint: Uses `github-automation-sc-launch-role`
+   - TEMPLATE constraint: Locks Lambda ARN to deployed function
+
+## Product Details
+
+- **Product ID:** `prod-w3uvfaxmeblxe`
+- **Portfolio ID:** `port-uchiqj7m3d57k`
+- **Template Location:** `s3://servicecatalog-product-artifacts-20250904021619588100000003/github-repo-creator/v1.0/product-template.yaml`
+- **Template URL:** `https://servicecatalog-product-artifacts-20250904021619588100000003.s3.us-gov-west-1.amazonaws.com/github-repo-creator/v1.0/product-template.yaml`
+- **Console URL:** https://console.amazonaws-us-gov.com/servicecatalog/home?region=us-gov-west-1#/products/prod-w3uvfaxmeblxe
+
+## Parameters Available
+
+The product accepts the following parameters:
+
+| Parameter | Required | Default | Description |
+|-----------|----------|---------|-------------|
+| `ProjectName` | Yes | - | Repository name (lowercase, hyphens) |
+| `OwningTeam` | Yes | `tf-module-admins` | GitHub team with admin access |
+| `Environment` | Yes | `development` | Environment (development/staging/production/sandbox) |
+| `AwsRegion` | No | `us-gov-west-1` | Primary AWS region |
+| `OrganizationPath` | No | - | Organization path (e.g., dept:team:subteam) |
+| `FinOpsProjectNumber` | No | - | FinOps project number |
+| `FinOpsProjectName` | No | - | FinOps project name |
+| `AdditionalTags` | No | `{}` | Additional tags as JSON |
+| `LambdaFunctionArn` | No | (locked) | ARN of the Lambda function (enforced by template constraint) |
+
+## Lessons Learned
+
+1. **Tag-Based Access Control:** Enterprise S3 buckets in GovCloud environments often use tag-based SCPs to control access
+2. **Consistency Matters:** Always examine working examples to understand hidden requirements (tags, URL formats, etc.)
+3. **IAM Conditions:** IAM policy conditions can reference S3 object tags for fine-grained access control
+4. **Service Catalog Permissions:** Service Catalog service principal needs proper S3 access, which is often controlled differently than user access
+
+## Related Files
+
+- **Terraform Configuration:** `deploy/service_catalog.tf`
+- **Product Template:** `service-catalog/product-template.yaml`
+- **Variables:** `deploy/variables.tf`
+- **Config:** `deploy/terraform.tfvars`
+- **Request Document:** `docs/S3_ACCESS_REQUEST.md`
+
+## Next Steps
+
+1. ✅ Service Catalog product is ready to use
+2. ⏳ Test launching a repository through the product
+3. ⏳ Rebuild Lambda container image to fix stale code issue (separate task)
+4. ⏳ Document end-user instructions for using the Service Catalog product
+5. ⏳ Set up CI/CD pipeline to automatically update product template on changes
+
+## Acknowledgments
+
+Special thanks to **DeLong** for identifying the tag-based access control requirement that was the key to resolving this issue!
+
+---
+
+**Status:** ✅ RESOLVED  
+**Last Updated:** February 11, 2026

packer.pkr.hcl

@@ -99,6 +99,10 @@ build {
   provisioner "shell" {
     inline = [
       "cd /var/task",
+      "echo 'Installing Census Bureau Root CA certificate...'",
+      "cp /etc/pip-cert.pem /etc/pki/ca-trust/source/anchors/census-root-ca.pem || true",
+      "update-ca-trust extract || true",
+      "cat /etc/pip-cert.pem >> /var/lang/lib/python3.11/site-packages/certifi/cacert.pem || true",
       "echo 'Installing dependencies for Service Catalog Lambda function...'",
       "pip install -r requirements.txt",
       "echo 'Lambda function built for AWS Service Catalog integration'"