Skip to content
New issue
Jump to bottom

1 Instance failed to create DNS (out of 50) #20

Closed
cho00013 opened this issue Mar 25, 2022 · 3 comments · Fixed by #23
Closed

1 Instance failed to create DNS (out of 50) #20

cho00013 opened this issue Mar 25, 2022 · 3 comments · Fixed by #23
Labels
bug Something isn't working

Comments

@cho00013
Copy link
Contributor

cho00013 commented Mar 25, 2022 •
edited

Of the 50, one failed - per Don

for f in $(cat testn-ips.txt ); do host $f; done
31.18.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-1-apps-us-gov-west-1a.ite.das.rm.census.gov.
106.24.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-10-emr-us-gov-west-1a.ite.das.rm.census.gov.
7.26.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-11-emr-us-gov-west-1b.ite.das.rm.census.gov.
181.29.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-12-emr-us-gov-west-1c.ite.das.rm.census.gov.
59.18.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-13-apps-us-gov-west-1a.ite.das.rm.census.gov.
135.18.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-14-apps-us-gov-west-1b.ite.das.rm.census.gov.
35.19.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-15-apps-us-gov-west-1c.ite.das.rm.census.gov.
48.25.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-16-emr-us-gov-west-1a.ite.das.rm.census.gov.
224.26.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-17-emr-us-gov-west-1b.ite.das.rm.census.gov.
199.29.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-18-emr-us-gov-west-1c.ite.das.rm.census.gov.
92.18.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-19-apps-us-gov-west-1a.ite.das.rm.census.gov.
189.18.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-2-apps-us-gov-west-1b.ite.das.rm.census.gov.
175.18.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-20-apps-us-gov-west-1b.ite.das.rm.census.gov.
112.19.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-21-apps-us-gov-west-1c.ite.das.rm.census.gov.
42.25.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-22-emr-us-gov-west-1a.ite.das.rm.census.gov.
137.26.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-23-emr-us-gov-west-1b.ite.das.rm.census.gov.
246.29.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-24-emr-us-gov-west-1c.ite.das.rm.census.gov.
55.18.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-25-apps-us-gov-west-1a.ite.das.rm.census.gov.
165.18.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-26-apps-us-gov-west-1b.ite.das.rm.census.gov.
114.19.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-27-apps-us-gov-west-1c.ite.das.rm.census.gov.
196.25.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-28-emr-us-gov-west-1a.ite.das.rm.census.gov.
25.26.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-29-emr-us-gov-west-1b.ite.das.rm.census.gov.
Host 20.19.191.10.in-addr.arpa. not found: 3(NXDOMAIN)
74.29.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-30-emr-us-gov-west-1c.ite.das.rm.census.gov.
35.18.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-31-apps-us-gov-west-1a.ite.das.rm.census.gov.
207.18.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-32-apps-us-gov-west-1b.ite.das.rm.census.gov.
115.19.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-33-apps-us-gov-west-1c.ite.das.rm.census.gov.
167.24.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-34-emr-us-gov-west-1a.ite.das.rm.census.gov.
155.26.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-35-emr-us-gov-west-1b.ite.das.rm.census.gov.
201.29.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-36-emr-us-gov-west-1c.ite.das.rm.census.gov.
96.18.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-37-apps-us-gov-west-1a.ite.das.rm.census.gov.
166.18.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-38-apps-us-gov-west-1b.ite.das.rm.census.gov.
102.19.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-39-apps-us-gov-west-1c.ite.das.rm.census.gov.
186.24.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-4-emr-us-gov-west-1a.ite.das.rm.census.gov.
119.24.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-40-emr-us-gov-west-1a.ite.das.rm.census.gov.
156.27.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-41-emr-us-gov-west-1b.ite.das.rm.census.gov.
218.29.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-42-emr-us-gov-west-1c.ite.das.rm.census.gov.
73.18.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-43-apps-us-gov-west-1a.ite.das.rm.census.gov.
198.18.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-44-apps-us-gov-west-1b.ite.das.rm.census.gov.
15.19.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-45-apps-us-gov-west-1c.ite.das.rm.census.gov.
216.24.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-46-emr-us-gov-west-1a.ite.das.rm.census.gov.
195.26.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-47-emr-us-gov-west-1b.ite.das.rm.census.gov.
128.29.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-48-emr-us-gov-west-1c.ite.das.rm.census.gov.
95.18.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-49-apps-us-gov-west-1a.ite.das.rm.census.gov.
168.26.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-5-emr-us-gov-west-1b.ite.das.rm.census.gov.
215.18.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-50-apps-us-gov-west-1b.ite.das.rm.census.gov.
29.29.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-6-emr-us-gov-west-1c.ite.das.rm.census.gov.
58.18.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-7-apps-us-gov-west-1a.ite.das.rm.census.gov.
216.18.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-8-apps-us-gov-west-1b.ite.das.rm.census.gov.
118.19.191.10.in-addr.arpa domain name pointer othername-vpc3-testn-9-apps-us-gov-west-1c.ite.das.rm.census.gov.

This one did not get done

  "vpc3-testn-3" = {
    "id" = "i-0476aed93505b2ba9"
    "ip_address" = "10.191.19.20"
    "name" = "vpc3-testn-3"
    "subnet_id" = "subnet-0236ef947802d4406"
    "tags" = {
      "CostAllocation" = "csvd:infrastructure"
      "Environment" = "infrastructure"
      "Name" = "vpc3-testn-3-apps-us-gov-west-1c.ite.das.rm.census.gov"
      "boc:created_by" = "terraform"
      "boc:dns:name" = "othername-vpc3-testn-3-apps-us-gov-west-1c.ite.das.rm.census.gov"
      "test_details" = "4. valid boc:dns:name othername-NAME, normal NAME is valid too, should not show up"
    }
  }

sns message

{
  "Type" : "Notification",
  "MessageId" : "1c737c8c-7fdf-5382-8ec2-263ad03ccef3",
  "TopicArn" : "arn:aws-us-gov:sns:us-gov-west-1:252999262699:inf-dynamic-route53-us-gov-west-1",
  "Message" : "{\"instance_id\": \"i-0476aed93505b2ba9\", \"account_id\": \"252999262699\", \"region\": \"us-gov-west-1\", \"client\": \"ec2\", \"boto3_method\": \"describe_vpc_attribute\", \"message\": \"You have to enable DNS hostnames to use Route 53 private hosted zones. DNS hostnames disabled for vpc-0f111ee54c6971d9f\"}",
  "Timestamp" : "2022-03-24T17:42:59.979Z",
  "SignatureVersion" : "1",
  "Signature" : "QOpwo2n/nEnMC1pTHn39XHoytNEjTKlnvz9xFvI31ALMov5BLP/K2kTArLCiIfEvnzczSA7jytVnLwIXA29n+gH1yeA4r8ul38cUU1KcotK+K4Dp+DEhIbzw6gm6fSRkPs6jckKhdUsx4jhQqKVyA0Ski0bofn7c7LTch+igjJVFp+B6vIuIwsZUsyE2EX+so+VJ5xmhThSOdbMCzsft6Kc8bxZ0hgtZGw2weB7Fdf3StYlxyCfidSvgjEZMbLjRqkPR+qaGg0EHihPAkztfU/u4XX9KBrcvF/10P/+Ox56bv1eNEziTog8eNosEy9FTU62cF8qKn2hWXPb4e1LLww==",
  "SigningCertURL" : "https://sns.us-gov-west-1.amazonaws.com/SimpleNotificationService-9cff44d7b285d0190eeed8082728a947.pem",
  "UnsubscribeURL" : "https://sns.us-gov-west-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws-us-gov:sns:us-gov-west-1:252999262699:inf-dynamic-route53-us-gov-west-1:de7c18d3-81b1-4eea-8584-465b07ca48da"
}

DNS not disabled. All of these instances are in the same VPC.

Let's see if I can find the relevant log.
@cho00013 cho00013 added the bug Something isn't working label Mar 25, 2022
@cho00013
Copy link
Contributor Author

cho00013 commented Mar 25, 2022 •
edited

CloudWatch
Log groups
/aws/lambda/inf-dynamic-route53-us-gov-west-1
2022/03/24/[$LATEST]649b7938e98f4b528a6a5f35287f9ceb

key log entry

[INFO]	2022-03-24T17:42:59.798Z	6b8c5753-5fce-4d32-8c00-41e20649ca3e	unexpected error. <class 'botocore.exceptions.ClientError'> - line number: 2241

[ERROR]	2022-03-24T17:42:59.810Z	6b8c5753-5fce-4d32-8c00-41e20649ca3e	DNS hostnames disabled for vpc-0f111ee54c6971d9f. You have to enable DNS hostnames to use Route 53 private hosted zones.  - line number: 366

The unexpected error refers to the command below.

def is_dns_hostnames_enabled(client, vpc_id):
    """
    Whether dns hostnames is enabled
    :param client:
    :param vpc_id:
    :return:
    """
    try:
        response = client.describe_vpc_attribute(
            Attribute='enableDnsHostnames',
            VpcId=vpc_id
        )

        LOGGER.debug("%s", str(response) + lineno())
        return response['EnableDnsHostnames']['Value']
    except:
        LOGGER.info("unexpected error. %s\n", str(sys.exc_info()[0]) + lineno())

@cho00013
Copy link
Contributor Author

Instance Info

Instance info

% aws --profile $(get-profile) --region $(get-region) ec2 describe-instances --instance-ids "i-0476aed93505b2ba9" --output text
RESERVATIONS    252999262699    r-0585b8067f03f6889
INSTANCES       0       x86_64  960CD387-89FD-4D42-A997-B3E7F47BAF6F    False   True    xen     ami-0f282f1684662b6e8   i-0476aed93505b2ba9     t3.medium       vpc3-test-ec2-keypair   2022-03-24T17:42:41+00:00       Linux/UNIX      ip-10-191-19-20.us-gov-west-1.compute.internal  10.191.19.20            /dev/xvda       ebs     True            subnet-0236ef947802d4406        RunInstances    2022-03-24T17:42:41+00:00       hvm     vpc-0f111ee54c6971d9f
BLOCKDEVICEMAPPINGS     /dev/xvda
EBS     2022-03-24T17:42:41+00:00       True    attached        vol-09861b5ae984f9b70
CAPACITYRESERVATIONSPECIFICATION        open
CPUOPTIONS      1       2
ENCLAVEOPTIONS  False
HIBERNATIONOPTIONS      False
IAMINSTANCEPROFILE      arn:aws-us-gov:iam::252999262699:instance-profile/r-vpc-test-ec2-role   AIPATVZ7OMHVRNVEFRSNV
METADATAOPTIONS enabled disabled        1       optional        disabled        applied
MONITORING      disabled
NETWORKINTERFACES               interface       0a:2f:06:84:79:dc       eni-058bde7f54e0d1353   252999262699    ip-10-191-19-20.us-gov-west-1.compute.internal  10.191.19.20    True    in-use  subnet-0236ef947802d4406        vpc-0f111ee54c6971d9f
ATTACHMENT      2022-03-24T17:42:41+00:00       eni-attach-0424994a792a3d0ff    True    0       0       attached
GROUPS  sg-04c40c32406357361    it-linux-base
PRIVATEIPADDRESSES      True    ip-10-191-19-20.us-gov-west-1.compute.internal  10.191.19.20
PLACEMENT       us-gov-west-1c          default
PRIVATEDNSNAMEOPTIONS   False   False   ip-name
SECURITYGROUPS  sg-04c40c32406357361    it-linux-base
STATE   16      running
TAGS    test_details    4. valid boc:dns:name othername-NAME, normal NAME is valid too, should not show up
TAGS    Environment     infrastructure
TAGS    boc:created_by  terraform
TAGS    CostAllocation  csvd:infrastructure
TAGS    Name    vpc3-testn-3-apps-us-gov-west-1c.ite.das.rm.census.gov
TAGS    boc:dns:name    othername-vpc3-testn-3-apps-us-gov-west-1c.ite.das.rm.census.gov

@cho00013
Copy link
Contributor Author

Added MAX_API_RETRY logic to handle Ec2 API limit

Fixed in PR #23

@badra001 badra001 mentioned this issue Mar 28, 2022
Merged
badra001 added a commit that referenced this issue Mar 28, 2022
@badra001
- code 0.2.2 fixes for json.loads and API limits (#20, #22)
9b8f0aa
@cho00013 cho00013 linked a pull request Mar 28, 2022 that will close this issue
Fixed issue 20 and 22 #23
Merged
Sign in to join this conversation on GitHub.
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fixed issue 20 and 22 terraform-modules/aws-dynamic-route53
2 participants
@badra001 @cho00013