**CloudWatch Logs Insights** region: us-gov-west-1 log-group-names: /aws/lambda/inf-dynamic-route53-us-gov-west-1 start-time: 2022-03-28T16:53:00.000Z end-time: 2022-03-28T16:59:59.000Z query-string: ``` filter @message LIKE 'waiting before retry' | fields @timestamp, @message | sort @timestamp desc ``` --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | @timestamp | @message | |-------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | 2022-03-28 16:55:05.720 | [INFO] 2022-03-28T16:55:05.719Z e5aa97d3-4a14-48d2-8623-5dc8784d23b7 instance: i-0e004fa40a5c0e272, change_resource_record_sets UPSERT returned error, waiting before retry. 1 - line number: 1736 | | 2022-03-28 16:55:03.630 | [INFO] 2022-03-28T16:55:03.629Z 78f15925-f1c5-4066-abc8-1269754265c0 instance: i-0601736d1978c4a42, change_resource_record_sets UPSERT returned error, waiting before retry. 1 - line number: 1736 | | 2022-03-28 16:55:03.410 | [INFO] 2022-03-28T16:55:03.410Z 06c3270d-4951-4138-9fd6-270671263694 instance: i-09ee7613e6c5ead49, change_resource_record_sets UPSERT returned error, waiting before retry. 1 - line number: 1736 | | 2022-03-28 16:55:01.806 | [INFO] 2022-03-28T16:55:01.806Z 82e60507-d373-445c-b11b-985ae5c8de66 instance: i-0ec902d7230632812, change_resource_record_sets UPSERT returned error, waiting before retry. 1 - line number: 1736 | | 2022-03-28 16:55:01.156 | [INFO] 2022-03-28T16:55:01.156Z 1bf96f90-6e09-498b-9cfe-f0cd447a0ca1 instance: i-09833fecca9ec17e8, get_hosted_zone returned error, waiting before retry. 2 - line number: 2506 | | 2022-03-28 16:55:00.138 | [INFO] 2022-03-28T16:55:00.138Z 8343d8fd-d0ba-44b5-8f02-9629a01b021b instance: i-0f1c92856988ca50f, change_resource_record_sets UPSERT returned error, waiting before retry. 3 - line number: 1736 | | 2022-03-28 16:54:58.318 | [INFO] 2022-03-28T16:54:58.317Z ab651fb9-8c81-4ca9-bf65-cd45e370c41c instance: i-091aa810717417cb6, get_hosted_zone returned error, waiting before retry. 1 - line number: 2506 | | 2022-03-28 16:54:57.636 | [INFO] 2022-03-28T16:54:57.635Z 0d376b76-77b5-4ccd-9c96-81ee00c312ed instance: i-0ccb20d64282a3e69, change_resource_record_sets UPSERT returned error, waiting before retry. 1 - line number: 1736 | | 2022-03-28 16:54:57.408 | [INFO] 2022-03-28T16:54:57.408Z 2adf27a8-41ee-4f31-a9a5-5a79ac5f3ce3 instance: i-0cbee59e07b9de1d0, change_resource_record_sets UPSERT returned error, waiting before retry. 1 - line number: 1736 | | 2022-03-28 16:54:56.481 | [INFO] 2022-03-28T16:54:56.481Z c7ae43f5-1620-4f15-a902-932459bd626a instance: i-0116aa2d5b3957168, get_hosted_zone returned error, waiting before retry. 1 - line number: 2506 | | 2022-03-28 16:54:55.515 | [INFO] 2022-03-28T16:54:55.515Z 5808d4ef-851f-4d34-9d74-939bdec2ab0b instance: i-000d5d33213178fe3, change_resource_record_sets UPSERT returned error, waiting before retry. 1 - line number: 1736 | | 2022-03-28 16:54:55.220 | [INFO] 2022-03-28T16:54:55.220Z f00a5807-2cc7-4390-b38c-64dc5d584f8a instance: i-0ee094c6bdcc21e9b, list_hosted_zones returned error, waiting before retry. 2 - line number: 1401 | | 2022-03-28 16:54:54.618 | [INFO] 2022-03-28T16:54:54.618Z e5aa97d3-4a14-48d2-8623-5dc8784d23b7 instance: i-0e004fa40a5c0e272, get_hosted_zone returned error, waiting before retry. 1 - line number: 2506 | | 2022-03-28 16:54:54.508 | [INFO] 2022-03-28T16:54:54.507Z 28a907a0-ad13-48e9-a150-039af47b5531 instance: i-0c0e588f92b3eed6d, change_resource_record_sets UPSERT returned error, waiting before retry. 1 - line number: 1736 | | 2022-03-28 16:54:54.306 | [INFO] 2022-03-28T16:54:54.305Z d8fa8755-8cd1-4cef-9fb5-bbee6954175d instance: i-0b501c884c0ad4295, change_resource_record_sets UPSERT returned error, waiting before retry. 1 - line number: 1736 | | 2022-03-28 16:54:53.723 | [INFO] 2022-03-28T16:54:53.722Z 0b8506b9-4815-489e-8737-b2826f20f8ba instance: i-0d1f660c6f8cad29a, change_resource_record_sets UPSERT returned error, waiting before retry. 2 - line number: 1736 | | 2022-03-28 16:54:53.717 | [INFO] 2022-03-28T16:54:53.717Z 5d853428-806e-46bf-94f5-05ee97b71078 instance: i-04f1c336ad505d251, change_resource_record_sets UPSERT returned error, waiting before retry. 1 - line number: 1736 | | 2022-03-28 16:54:53.289 | [INFO] 2022-03-28T16:54:53.289Z b7275e5b-c73d-4bca-8ad9-c3c690090f61 instance: i-0eb8097d89d4b7737, get_hosted_zone returned error, waiting before retry. 1 - line number: 2506 | | 2022-03-28 16:54:53.274 | [INFO] 2022-03-28T16:54:53.273Z 78f15925-f1c5-4066-abc8-1269754265c0 instance: i-0601736d1978c4a42, change_resource_record_sets UPSERT returned error, waiting before retry. 2 - line number: 1736 | | 2022-03-28 16:54:51.015 | [INFO] 2022-03-28T16:54:51.15Z 2deae644-7feb-4bb3-94e8-8df6dc4ce3e0 instance: i-06e4c35c8e1207956, change_resource_record_sets UPSERT returned error, waiting before retry. 2 - line number: 1736 | | 2022-03-28 16:54:50.834 | [INFO] 2022-03-28T16:54:50.833Z 1bf96f90-6e09-498b-9cfe-f0cd447a0ca1 instance: i-09833fecca9ec17e8, get_hosted_zone returned error, waiting before retry. 1 - line number: 2506 | | 2022-03-28 16:54:50.294 | [INFO] 2022-03-28T16:54:50.294Z caa4e848-d876-431b-9cae-4edb5152cd37 instance: i-0826e53332959569d, get_hosted_zone returned error, waiting before retry. 1 - line number: 2506 | | 2022-03-28 16:54:48.775 | [INFO] 2022-03-28T16:54:48.775Z 97ea1263-413b-4bb0-8ccf-a1c0ba886593 instance: i-020a0fb5d585aa0fc, change_resource_record_sets UPSERT returned error, waiting before retry. 1 - line number: 1736 | | 2022-03-28 16:54:47.535 | [INFO] 2022-03-28T16:54:47.535Z 7d8c22ab-0cf8-4d32-8efd-4712f8332eb1 instance: i-00499f59fe43f64f8, get_hosted_zone returned error, waiting before retry. 1 - line number: 2506 | | 2022-03-28 16:54:47.510 | [INFO] 2022-03-28T16:54:47.509Z 06c3270d-4951-4138-9fd6-270671263694 instance: i-09ee7613e6c5ead49, get_hosted_zone returned error, waiting before retry. 1 - line number: 2506 | | 2022-03-28 16:54:46.821 | [INFO] 2022-03-28T16:54:46.821Z 0b8506b9-4815-489e-8737-b2826f20f8ba instance: i-0d1f660c6f8cad29a, change_resource_record_sets UPSERT returned error, waiting before retry. 1 - line number: 1736 | | 2022-03-28 16:54:46.381 | [INFO] 2022-03-28T16:54:46.380Z f00a5807-2cc7-4390-b38c-64dc5d584f8a instance: i-0ee094c6bdcc21e9b, list_hosted_zones returned error, waiting before retry. 1 - line number: 1401 | | 2022-03-28 16:54:45.838 | [INFO] 2022-03-28T16:54:45.838Z 8343d8fd-d0ba-44b5-8f02-9629a01b021b instance: i-0f1c92856988ca50f, change_resource_record_sets UPSERT returned error, waiting before retry. 2 - line number: 1736 | | 2022-03-28 16:54:45.170 | [INFO] 2022-03-28T16:54:45.170Z 78f15925-f1c5-4066-abc8-1269754265c0 instance: i-0601736d1978c4a42, change_resource_record_sets UPSERT returned error, waiting before retry. 1 - line number: 1736 | | 2022-03-28 16:54:44.963 | [INFO] 2022-03-28T16:54:44.963Z aa5b8c0a-451b-4150-9e9a-33408d25693b instance: i-0d01e9ff1888fe1f4, change_resource_record_sets UPSERT returned error, waiting before retry. 2 - line number: 1736 | | 2022-03-28 16:54:44.726 | [INFO] 2022-03-28T16:54:44.725Z 82e60507-d373-445c-b11b-985ae5c8de66 instance: i-0ec902d7230632812, change_resource_record_sets UPSERT returned error, waiting before retry. 3 - line number: 1736 | | 2022-03-28 16:54:44.449 | [INFO] 2022-03-28T16:54:44.449Z e1e221c8-2bb8-4cf5-8585-812aec4d310e instance: i-0a11c7b5161cd7657, change_resource_record_sets UPSERT returned error, waiting before retry. 1 - line number: 1736 | | 2022-03-28 16:54:44.053 | [INFO] 2022-03-28T16:54:44.53Z 1bf96f90-6e09-498b-9cfe-f0cd447a0ca1 instance: i-09833fecca9ec17e8, list_hosted_zones returned error, waiting before retry. 1 - line number: 1401 | | 2022-03-28 16:54:43.415 | [INFO] 2022-03-28T16:54:43.415Z e5aa97d3-4a14-48d2-8623-5dc8784d23b7 instance: i-0e004fa40a5c0e272, list_hosted_zones returned error, waiting before retry. 2 - line number: 1401 | | 2022-03-28 16:54:42.713 | [INFO] 2022-03-28T16:54:42.713Z 2adf27a8-41ee-4f31-a9a5-5a79ac5f3ce3 instance: i-0cbee59e07b9de1d0, get_hosted_zone returned error, waiting before retry. 1 - line number: 2506 | | 2022-03-28 16:54:42.152 | [INFO] 2022-03-28T16:54:42.152Z 5808d4ef-851f-4d34-9d74-939bdec2ab0b instance: i-000d5d33213178fe3, change_resource_record_sets UPSERT returned error, waiting before retry. 1 - line number: 1736 | | 2022-03-28 16:54:40.532 | [INFO] 2022-03-28T16:54:40.532Z d840b31e-1191-4928-b380-65111f3a80d9 instance: i-0a1c2e5cc35e69ba5, change_resource_record_sets UPSERT returned error, waiting before retry. 1 - line number: 1736 | | 2022-03-28 16:54:40.286 | [INFO] 2022-03-28T16:54:40.286Z 06c3270d-4951-4138-9fd6-270671263694 instance: i-09ee7613e6c5ead49, list_hosted_zones returned error, waiting before retry. 1 - line number: 1401 | | 2022-03-28 16:54:40.266 | [INFO] 2022-03-28T16:54:40.250Z b8297c6b-581e-4d95-be7d-12c6d4bf7dc7 instance: i-0411b98da12a58e6a, get_hosted_zone returned error, waiting before retry. 2 - line number: 2506 | | 2022-03-28 16:54:39.426 | [INFO] 2022-03-28T16:54:39.426Z d8fa8755-8cd1-4cef-9fb5-bbee6954175d instance: i-0b501c884c0ad4295, change_resource_record_sets UPSERT returned error, waiting before retry. 1 - line number: 1736 | | 2022-03-28 16:54:38.938 | [INFO] 2022-03-28T16:54:38.938Z 8343d8fd-d0ba-44b5-8f02-9629a01b021b instance: i-0f1c92856988ca50f, change_resource_record_sets UPSERT returned error, waiting before retry. 1 - line number: 1736 | | 2022-03-28 16:54:38.656 | [INFO] 2022-03-28T16:54:38.655Z 2deae644-7feb-4bb3-94e8-8df6dc4ce3e0 instance: i-06e4c35c8e1207956, change_resource_record_sets UPSERT returned error, waiting before retry. 1 - line number: 1736 | | 2022-03-28 16:54:38.641 | [INFO] 2022-03-28T16:54:38.641Z c7ae43f5-1620-4f15-a902-932459bd626a instance: i-0116aa2d5b3957168, list_hosted_zones returned error, waiting before retry. 1 - line number: 1401 | | 2022-03-28 16:54:37.471 | [INFO] 2022-03-28T16:54:37.471Z d8f14549-9c99-45aa-ac19-2a63130a3705 instance: i-0d82992d6100e0dc5, get_hosted_zone returned error, waiting before retry. 2 - line number: 2506 | | 2022-03-28 16:54:37.089 | [INFO] 2022-03-28T16:54:37.89Z b7275e5b-c73d-4bca-8ad9-c3c690090f61 instance: i-0eb8097d89d4b7737, get_hosted_zone returned error, waiting before retry. 1 - line number: 2506 | | 2022-03-28 16:54:36.748 | [INFO] 2022-03-28T16:54:36.748Z c81f9a7a-15a2-479e-bd20-929f65940968 instance: i-09a0027ccd09fd49f, change_resource_record_sets UPSERT returned error, waiting before retry. 1 - line number: 1736 | | 2022-03-28 16:54:36.403 | [INFO] 2022-03-28T16:54:36.402Z aa5b8c0a-451b-4150-9e9a-33408d25693b instance: i-0d01e9ff1888fe1f4, change_resource_record_sets UPSERT returned error, waiting before retry. 1 - line number: 1736 | | 2022-03-28 16:54:35.595 | [INFO] 2022-03-28T16:54:35.595Z e5aa97d3-4a14-48d2-8623-5dc8784d23b7 instance: i-0e004fa40a5c0e272, list_hosted_zones returned error, waiting before retry. 1 - line number: 1401 | | 2022-03-28 16:54:35.462 | [INFO] 2022-03-28T16:54:35.462Z 82e60507-d373-445c-b11b-985ae5c8de66 instance: i-0ec902d7230632812, change_resource_record_sets UPSERT returned error, waiting before retry. 2 - line number: 1736 | | 2022-03-28 16:54:34.816 | [INFO] 2022-03-28T16:54:34.815Z 7d8c22ab-0cf8-4d32-8efd-4712f8332eb1 instance: i-00499f59fe43f64f8, list_hosted_zones returned error, waiting before retry. 1 - line number: 1401 | | 2022-03-28 16:54:32.619 | [INFO] 2022-03-28T16:54:32.618Z 20c83bfe-e6eb-45bf-83e8-6fb04118801c instance: i-0905c5f02a4c0dbac, change_resource_record_sets UPSERT returned error, waiting before retry. 1 - line number: 1736 | | 2022-03-28 16:54:32.143 | [INFO] 2022-03-28T16:54:32.143Z 8343d8fd-d0ba-44b5-8f02-9629a01b021b instance: i-0f1c92856988ca50f, get_hosted_zone returned error, waiting before retry. 1 - line number: 2506 | | 2022-03-28 16:54:30.592 | [INFO] 2022-03-28T16:54:30.591Z d3958c34-2525-46aa-83ac-4d99cc9d5312 instance: i-032f7fb80c62077cf, change_resource_record_sets UPSERT returned error, waiting before retry. 1 - line number: 1736 | | 2022-03-28 16:54:30.481 | [INFO] 2022-03-28T16:54:30.481Z 0b8506b9-4815-489e-8737-b2826f20f8ba instance: i-0d1f660c6f8cad29a, get_hosted_zone returned error, waiting before retry. 2 - line number: 2506 | | 2022-03-28 16:54:30.337 | [INFO] 2022-03-28T16:54:30.337Z b8297c6b-581e-4d95-be7d-12c6d4bf7dc7 instance: i-0411b98da12a58e6a, get_hosted_zone returned error, waiting before retry. 1 - line number: 2506 | | 2022-03-28 16:54:28.020 | [INFO] 2022-03-28T16:54:28.20Z 97ea1263-413b-4bb0-8ccf-a1c0ba886593 instance: i-020a0fb5d585aa0fc, get_hosted_zone returned error, waiting before retry. 1 - line number: 2506 | | 2022-03-28 16:54:27.247 | [INFO] 2022-03-28T16:54:27.246Z 75a18a45-bb95-483c-b31c-29261d283650 instance: i-05fd0af5be063de20, change_resource_record_sets UPSERT returned error, waiting before retry. 1 - line number: 1736 | | 2022-03-28 16:54:27.006 | [INFO] 2022-03-28T16:54:27.6Z 82e60507-d373-445c-b11b-985ae5c8de66 instance: i-0ec902d7230632812, change_resource_record_sets UPSERT returned error, waiting before retry. 1 - line number: 1736 | | 2022-03-28 16:54:26.851 | [INFO] 2022-03-28T16:54:26.851Z d8f14549-9c99-45aa-ac19-2a63130a3705 instance: i-0d82992d6100e0dc5, get_hosted_zone returned error, waiting before retry. 1 - line number: 2506 | | 2022-03-28 16:54:26.395 | [INFO] 2022-03-28T16:54:26.395Z 0d376b76-77b5-4ccd-9c96-81ee00c312ed instance: i-0ccb20d64282a3e69, get_hosted_zone returned error, waiting before retry. 1 - line number: 2506 | | 2022-03-28 16:54:25.817 | [INFO] 2022-03-28T16:54:25.817Z 0b8506b9-4815-489e-8737-b2826f20f8ba instance: i-0d1f660c6f8cad29a, get_hosted_zone returned error, waiting before retry. 1 - line number: 2506 | | 2022-03-28 16:54:24.757 | [INFO] 2022-03-28T16:54:24.757Z c8dc8671-85c1-4063-9707-2942cc73b5a7 instance: i-022b90665946d3ba0, change_resource_record_sets UPSERT returned error, waiting before retry. 1 - line number: 1736 | | 2022-03-28 16:54:23.983 | [INFO] 2022-03-28T16:54:23.983Z aa5b8c0a-451b-4150-9e9a-33408d25693b instance: i-0d01e9ff1888fe1f4, get_hosted_zone returned error, waiting before retry. 1 - line number: 2506 | | 2022-03-28 16:54:23.395 | [INFO] 2022-03-28T16:54:23.394Z 3b700ef0-c77b-41ac-b0ca-6983e3a15c04 instance: i-0b70d26cf173fa6d2, list_hosted_zones returned error, waiting before retry. 1 - line number: 1401 | | 2022-03-28 16:54:21.163 | [INFO] 2022-03-28T16:54:21.162Z 78865d1a-5692-48fa-b6ef-2d72d7ddc343 instance: i-09c8eec23e6c469ce, change_resource_record_sets UPSERT returned error, waiting before retry. 1 - line number: 1736 | | 2022-03-28 16:54:19.399 | [INFO] 2022-03-28T16:54:19.399Z f710481a-233a-4455-a2e7-99fb24a6b14f instance: i-0260807d4782ab6b8, describe_instances returned RequestLimitExceeded, waiting before retry. 1 - line number: 1359 | | 2022-03-28 16:54:16.597 | [INFO] 2022-03-28T16:54:16.597Z c81f9a7a-15a2-479e-bd20-929f65940968 instance: i-09a0027ccd09fd49f, describe_instances returned RequestLimitExceeded, waiting before retry. 1 - line number: 1359 | | 2022-03-28 16:54:16.117 | [INFO] 2022-03-28T16:54:16.106Z 8343d8fd-d0ba-44b5-8f02-9629a01b021b instance: i-0f1c92856988ca50f, describe_instances returned RequestLimitExceeded, waiting before retry. 1 - line number: 1359 | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------