From ebf13ae312c075eae15ca0ec59c96cc92553a8c0 Mon Sep 17 00:00:00 2001 From: badra001 Date: Tue, 21 Jul 2026 08:31:38 -0400 Subject: [PATCH] update --- aws/projects/adsd-chris/README.md | 1 + aws/projects/adsd-chris/fsx-issue.md | 280 +++++++++++++++++++++++++++ 2 files changed, 281 insertions(+) create mode 100644 aws/projects/adsd-chris/fsx-issue.md diff --git a/aws/projects/adsd-chris/README.md b/aws/projects/adsd-chris/README.md index 2176d79a..1be9e860 100644 --- a/aws/projects/adsd-chris/README.md +++ b/aws/projects/adsd-chris/README.md @@ -28,6 +28,7 @@ to any future MSSQL setups. ## FSx Windows * Integration with boc.ad.census.gov +* [Major Issue](fsx-issue.md) with FSx for Windows ## Active Directory Integrations diff --git a/aws/projects/adsd-chris/fsx-issue.md b/aws/projects/adsd-chris/fsx-issue.md new file mode 100644 index 00000000..830269b8 --- /dev/null +++ b/aws/projects/adsd-chris/fsx-issue.md @@ -0,0 +1,280 @@ +# FSx for Windows Issue with boc.ad.census.gov. + +We are unable to create an FSx for Windows file system in AWS using the boc.ad.census.gov +self-managed AD. This describes the problem. + +## Summary + +* Cannot create FSx for Windows file system with boc.ad.census.gov +* FSx expects A records returned in a SRV query in the additional section +* A (IPs) records are not returned because we have 17 domain controllers (17 SRV records). + * The size of a standard DNS packet is 512 bytes + * Larger than that, it moved to EDNS0 and TCP + * Infoblox is set to 1220 as the max (grid-wide), even though the specification allows 4096 + * While it can be increased, Infoblox does not recommend to do so because of possible packet fragmentation + * The response is 1191 bytes, so not large enough for the 17 A and AAAA records +* When 13 or more SRV records are present, even with enough space in the packet, Infoblox will not send additional records due to a patch to address + * [CVE-2024-11187](https://nvd.nist.gov/vuln/detail/CVE-2024-11187) + * [NIOS 9.0.6 changes](https://docs.infoblox.com/space/nios90/1323565220/NIOS+9.0.6+-+Changes+to+Default+Behavior) +* The AWS PowerShell test script fails when enumerating the SRV records and checks the parent ad.census.gov (which + isn't so useful) but it only finds 5 of the 6 DCs. +* This worked in the lab because we have only fewer DCs, and it worked in csp1.ead.census.gov because we have only 2 DCs. + +## Next Steps + +* A support case for account adsd-chris-nonprod-gov ID 178397328406861 is opened for this and has been active for a couple weeks now +* It is now referred to the FSx service team to address. +* The fix should be to enumerate the A records for every SRV record found in the DNS query. That will fix + not only this problem, but also allow the VPC+2 resolver to work (it too provides minimal answers and never actually + returns any additional A records, even for the small domains) +* I tried creating an unbound resolver on a Linux machine, serving only the needed DC data for AD locally, and forwarding + all other boc.ad.census.gov to Infoblox. While this works and provides the expected responses, it'll ultimately result + in a failure of the file system at a later time when FSx needs to renew Kerberos tokens and rediscover the domain + +## Information + +* dig for csp1.ead.census.gov + +You see this returns the two A records for the DCs and the packet site is 452, under the 512 for UDP. + +```console +% dig @148.129.127.22 in srv _ldap._tcp.dc._msdcs.csp1.ead.census.gov +edns +stats +tcp + +; <<>> DiG 9.11.36-RedHat-9.11.36-16.el8_10.8 <<>> @148.129.127.22 in srv _ldap._tcp.dc._msdcs.csp1.ead.census.gov +edns +stats +tcp +; (1 server found) +;; global options: +cmd +;; Got answer: +;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25030 +;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 4, ADDITIONAL: 9 + +;; OPT PSEUDOSECTION: +; EDNS: version: 0, flags:; udp: 1220 +; COOKIE: b137b3e93df4002f010000006a5f655c35dc2bb9871ea770 (good) +;; QUESTION SECTION: +;_ldap._tcp.dc._msdcs.csp1.ead.census.gov. IN SRV + +;; ANSWER SECTION: +_ldap._tcp.dc._msdcs.csp1.ead.census.gov. 600 IN SRV 0 100 389 csp1-csp1-dc4.csp1.ead.census.gov. +_ldap._tcp.dc._msdcs.csp1.ead.census.gov. 600 IN SRV 0 100 389 csp1-csp1-dc3.csp1.ead.census.gov. + +;; AUTHORITY SECTION: +csp1.ead.census.gov. 86400 IN NS ns2.csp1.census.gov. +csp1.ead.census.gov. 86400 IN NS ns1.csp1.census.gov. +csp1.ead.census.gov. 86400 IN NS ns1.census.gov. +csp1.ead.census.gov. 86400 IN NS ns2.census.gov. + +;; ADDITIONAL SECTION: +csp1-csp1-dc3.csp1.ead.census.gov. 1200 IN A 10.193.0.143 +csp1-csp1-dc4.csp1.ead.census.gov. 1200 IN A 10.193.3.223 +ns1.csp1.census.gov. 28800 IN A 10.193.0.22 +ns1.census.gov. 900 IN A 148.129.191.22 +ns2.csp1.census.gov. 28800 IN A 10.193.2.22 +ns2.census.gov. 900 IN A 148.129.127.22 +ns1.census.gov. 900 IN AAAA 2610:20:2008:2000::16:0 +ns2.census.gov. 900 IN AAAA 2610:20:2018:2000::16:0 + +;; Query time: 0 msec +;; SERVER: 148.129.127.22#53(148.129.127.22) +;; WHEN: Tue Jul 21 08:26:04 EDT 2026 +;; MSG SIZE rcvd: 452 +``` + +* dig for boc.ad.census.gov + +You see the 17 SRV records, and zero A records for DCs. The A (and AAAA) records you see here are glue records +for the DNS servers. + +```console +% dig @148.129.127.22 in srv _ldap._tcp.dc._msdcs.boc.ad.census.gov +edns +stats +tcp + +; <<>> DiG 9.11.36-RedHat-9.11.36-16.el8_10.8 <<>> @148.129.127.22 in srv _ldap._tcp.dc._msdcs.boc.ad.census.gov +edns +stats +tcp +; (1 server found) +;; global options: +cmd +;; Got answer: +;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 47031 +;; flags: qr aa rd ra; QUERY: 1, ANSWER: 17, AUTHORITY: 3, ADDITIONAL: 7 + +;; OPT PSEUDOSECTION: +; EDNS: version: 0, flags:; udp: 1220 +; COOKIE: 06b540e10a603be1010000006a5f65a4ac07bd33913f965e (good) +;; QUESTION SECTION: +;_ldap._tcp.dc._msdcs.boc.ad.census.gov. IN SRV + +;; ANSWER SECTION: +_ldap._tcp.dc._msdcs.boc.ad.census.gov. 600 IN SRV 0 100 389 it285bccbocdc.boc.ad.census.gov. +_ldap._tcp.dc._msdcs.boc.ad.census.gov. 600 IN SRV 0 100 389 atro013bocdc.boc.ad.census.gov. +_ldap._tcp.dc._msdcs.boc.ad.census.gov. 600 IN SRV 0 100 389 nyro013bocdc.boc.ad.census.gov. +_ldap._tcp.dc._msdcs.boc.ad.census.gov. 600 IN SRV 0 100 389 cgro013bocdc.boc.ad.census.gov. +_ldap._tcp.dc._msdcs.boc.ad.census.gov. 600 IN SRV 0 100 389 phro013bocdc.boc.ad.census.gov. +_ldap._tcp.dc._msdcs.boc.ad.census.gov. 600 IN SRV 0 100 389 it062hqbocdc.boc.ad.census.gov. +_ldap._tcp.dc._msdcs.boc.ad.census.gov. 600 IN SRV 0 100 389 laro013bocdc.boc.ad.census.gov. +_ldap._tcp.dc._msdcs.boc.ad.census.gov. 600 IN SRV 0 100 389 dnro013bocdc.boc.ad.census.gov. +_ldap._tcp.dc._msdcs.boc.ad.census.gov. 600 IN SRV 0 100 389 it056npcbocdc.boc.ad.census.gov. +_ldap._tcp.dc._msdcs.boc.ad.census.gov. 600 IN SRV 0 100 389 it279bccbocdc.boc.ad.census.gov. +_ldap._tcp.dc._msdcs.boc.ad.census.gov. 600 IN SRV 0 100 389 csp1-boc-dc1.boc.ad.census.gov. +_ldap._tcp.dc._msdcs.boc.ad.census.gov. 600 IN SRV 0 100 389 csp1-boc-dc2.boc.ad.census.gov. +_ldap._tcp.dc._msdcs.boc.ad.census.gov. 600 IN SRV 0 100 389 it297sccbocdc.boc.ad.census.gov. +_ldap._tcp.dc._msdcs.boc.ad.census.gov. 600 IN SRV 0 100 389 it324npcbocdc.boc.ad.census.gov. +_ldap._tcp.dc._msdcs.boc.ad.census.gov. 600 IN SRV 0 100 389 it296npcbocdc.boc.ad.census.gov. +_ldap._tcp.dc._msdcs.boc.ad.census.gov. 600 IN SRV 0 100 389 csp2-bocad-dc1.boc.ad.census.gov. +_ldap._tcp.dc._msdcs.boc.ad.census.gov. 600 IN SRV 0 100 389 csp2-bocad-dc2.boc.ad.census.gov. + +;; AUTHORITY SECTION: +_msdcs.boc.ad.census.gov. 28800 IN NS npc-inf-ns1.tco.census.gov. +_msdcs.boc.ad.census.gov. 28800 IN NS bcc-inf-idns01.tco.census.gov. +_msdcs.boc.ad.census.gov. 28800 IN NS hq-inf-idns01.tco.census.gov. + +;; ADDITIONAL SECTION: +npc-inf-ns1.tco.census.gov. 28800 IN A 148.129.15.22 +hq-inf-idns01.tco.census.gov. 28800 IN A 148.129.191.22 +bcc-inf-idns01.tco.census.gov. 28800 IN A 148.129.127.22 +npc-inf-ns1.tco.census.gov. 28800 IN AAAA 2610:20:2028:2000::16:0 +hq-inf-idns01.tco.census.gov. 28800 IN AAAA 2610:20:2008:2000::16:0 +bcc-inf-idns01.tco.census.gov. 28800 IN AAAA 2610:20:2018:2000::16:0 + +;; Query time: 0 msec +;; SERVER: 148.129.127.22#53(148.129.127.22) +;; WHEN: Tue Jul 21 08:27:16 EDT 2026 +;; MSG SIZE rcvd: 1191 +``` + +* dig for ad.census.gov + +Here you see 6 DC A records (as well as 4 AAAA). The AWS Test script returned only 5 here, not sure why. Size is under the 1220 at 808. +Not that it matters, as we are not putting this in the ad.census.gov domain. + +```console +% dig @148.129.127.22 in srv _ldap._tcp.dc._msdcs.ad.census.gov +edns +stats +tcp + +; <<>> DiG 9.11.36-RedHat-9.11.36-16.el8_10.8 <<>> @148.129.127.22 in srv _ldap._tcp.dc._msdcs.ad.census.gov +edns +stats +tcp +; (1 server found) +;; global options: +cmd +;; Got answer: +;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16970 +;; flags: qr aa rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 3, ADDITIONAL: 17 + +;; OPT PSEUDOSECTION: +; EDNS: version: 0, flags:; udp: 1220 +; COOKIE: 8b1b34b8d671fc8e010000006a5f65e4a488bdcb5a743d55 (good) +;; QUESTION SECTION: +;_ldap._tcp.dc._msdcs.ad.census.gov. IN SRV + +;; ANSWER SECTION: +_ldap._tcp.dc._msdcs.ad.census.gov. 600 IN SRV 0 100 389 it274npcaddc.ad.census.gov. +_ldap._tcp.dc._msdcs.ad.census.gov. 600 IN SRV 0 100 389 csp2-ad-dc1.ad.census.gov. +_ldap._tcp.dc._msdcs.ad.census.gov. 600 IN SRV 0 100 389 csp2-ad-dc2.ad.census.gov. +_ldap._tcp.dc._msdcs.ad.census.gov. 600 IN SRV 0 100 389 it272hqaddc.ad.census.gov. +_ldap._tcp.dc._msdcs.ad.census.gov. 600 IN SRV 0 100 389 it273sccaddc.ad.census.gov. +_ldap._tcp.dc._msdcs.ad.census.gov. 600 IN SRV 0 100 389 it271bccaddc.ad.census.gov. + +;; AUTHORITY SECTION: +_msdcs.ad.census.gov. 28800 IN NS npc-inf-ns1.tco.census.gov. +_msdcs.ad.census.gov. 28800 IN NS bcc-inf-idns01.tco.census.gov. +_msdcs.ad.census.gov. 28800 IN NS hq-inf-idns01.tco.census.gov. + +;; ADDITIONAL SECTION: +it272hqaddc.ad.census.gov. 28800 IN A 172.24.107.121 +it273sccaddc.ad.census.gov. 1200 IN A 172.24.107.151 +it271bccaddc.ad.census.gov. 1200 IN A 172.24.107.119 +it274npcaddc.ad.census.gov. 28800 IN A 172.25.18.71 +csp2-ad-dc1.ad.census.gov. 1200 IN A 10.198.31.136 +csp2-ad-dc2.ad.census.gov. 1200 IN A 10.198.31.137 +npc-inf-ns1.tco.census.gov. 28800 IN A 148.129.15.22 +hq-inf-idns01.tco.census.gov. 28800 IN A 148.129.191.22 +bcc-inf-idns01.tco.census.gov. 28800 IN A 148.129.127.22 +it272hqaddc.ad.census.gov. 28800 IN AAAA 2610:20:2018:374:172:24:107:121 +it273sccaddc.ad.census.gov. 28800 IN AAAA 2610:20:2018:374:172:24:107:151 +it271bccaddc.ad.census.gov. 28800 IN AAAA 2610:20:2018:374:172:24:107:119 +it274npcaddc.ad.census.gov. 28800 IN AAAA 2610:20:2026:646:172:25:18:71 +npc-inf-ns1.tco.census.gov. 28800 IN AAAA 2610:20:2028:2000::16:0 +hq-inf-idns01.tco.census.gov. 28800 IN AAAA 2610:20:2008:2000::16:0 +bcc-inf-idns01.tco.census.gov. 28800 IN AAAA 2610:20:2018:2000::16:0 + +;; Query time: 0 msec +;; SERVER: 148.129.127.22#53(148.129.127.22) +;; WHEN: Tue Jul 21 08:28:20 EDT 2026 +;; MSG SIZE rcvd: 808 +``` + +## Test Script Output + +* setup credentials +* setup validaton args + +```script +PS> $Result = Test-FSxADConfiguration @FSxADValidationArgs +Testing service account permissions is not currently enabled. If enabled, script will create test Active Directory computer objects in the organizational unit. This will be cleaned up by the script unless delete permissions are not properly configured on the provided service account in which case manual cleanup may be necessary. Do you want to enable testing? [y/n]: y +Running Active Directory validation with following input parameters: + + +DomainDNSRoot : BOC.AD.CENSUS.GOV +DnsIpAddresses : {148.129.127.22, 148.129.191.22} +SubnetIds : {subnet-06ade3a77c96d9df8} +OrganizationalUnit : +AdminGroup : +ComputerObjects : +TestServiceAccountPermissions : False + + + + +Test 1 - Validate EC2 Subnets ... +Found region from EC2 instance metadata + +SubnetId VpcId AvailabilityZoneId CidrBlock +-------- ----- ------------------ --------- +subnet-06ade3a77c96d9df8 vpc-061d3af8f9c2a02b3 usge1-az1 10.180.234.0/23 + + + +Test 2 - Validate connectivity with DNS Servers ... +WARNING: No domain controllers found for _ldap._tcp.dc._msdcs.BOC.AD.CENSUS.GOV! +WARNING: Failed to resolve DNS record for domain BOC.AD.CENSUS.GOV +Trying variation _ldap._tcp.dc._msdcs.AD.CENSUS.GOV + +Found the following 5 Domain Controllers by querying _ldap._tcp.dc._msdcs.AD.CENSUS.GOV: + +Name IPAddress Priority Weight +---- --------- -------- ------ +csp2-ad-dc1.ad.census.gov 10.198.31.136 0 100 +it271bccaddc.ad.census.gov 172.24.107.119 0 100 +it273sccaddc.ad.census.gov 172.24.107.151 0 100 +it272hqaddc.ad.census.gov 172.24.107.121 0 100 +it274npcaddc.ad.census.gov 172.25.18.71 0 100 + + + +WARNING: Did you mean to pass in AD.CENSUS.GOV instead of BOC.AD.CENSUS.GOV for DomainDNSRoot? +Skipping Validate FSx service user credentials ... +Skipping Validate all Domain Controllers are reachable ... +Skipping Validate domain properties ... +Skipping Validate organizational unit ... +Skipping Validate Admin Group ... +Skipping Validate that provided EC2 Subnets belong to a single AD Site ... +Skipping Validate connectivity with DNS Servers ... +Skipping Validate FSx service user credentials ... +Skipping Validate 'Create Computer Objects' permission ... +Skipping Validate 'Validated write to DNS host name' permission ... +Skipping Validate 'Validated write to service principal name' permission ... +Skipping Validate 'Reset Password' permission ... +Skipping Validate 'This Organization' list children permission ... +Skipping Validate 'Read and write Account Restrictions' permission ... +Skipping Validate 'Modify computer ACL' permission ... +Skipping Validate 'Delete Computer Objects' permission ... +Skipping Validate existing computer objects ... +Forest name was not found +Skipping Validate global catalog ... +18 of 20 tests skipped. +FAILURE - Tests failed. Please see error details below: + +Name Value +---- ----- +InferredDomainName AD.CENSUS.GOV +InvalidDomain _ldap._tcp.dc._msdcs.BOC.AD.CENSUS.GOV + + + +Please address all errors and warnings above prior to re-running validation to confirm fix. +Please refer to the included Troubleshooting Guide (TROUBLESHOOTING.md) for each warning and error. +The Client Powershell version running this script is- 5.1.20348.4294 +```