EventBridge order not preserved

[badra001]

Describe the bug

When stopping and starting an instance, the events can be returned out of order and then are processed as ADD and DELETE, thus removing the recently added Route53 record.

Identify Environment

• Lambda/Module version: 2.0.4

Expected behavior

Processing in the order of the events received.

Screenshots

• ec2 start event (ADD)

[INFO]	2023-08-15T12:07:42.146Z	f769fbec-7f1b-4221-84b5-f6715657c090	event: {'version': '0', 'id': '44b004e6-f1e3-f267-e994-dd48721265f8', 'detail-type': 'EC2 Instance State-change Notification', 'source': 'aws.ec2', 'account': '412295344020', 'time': '2023-08-15T12:07:41Z', 'region': 'us-gov-east-1', 'resources': ['arn:aws-us-gov:ec2:us-gov-east-1:412295344020:instance/i-08c8967c555e58aa1'], 'detail': {'instance-id': 'i-08c8967c555e58aa1', 'state': 'running'}} - line number: 304

[INFO] 2023-08-15T12:07:42.146Z f769fbec-7f1b-4221-84b5-f6715657c090 event_action ** ADD ** instance_id i-08c8967c555e58aa1 instance_state running

[INFO]	2023-08-15T12:07:52.584Z	f769fbec-7f1b-4221-84b5-f6715657c090	dns_data records written:[DnsData(zone_id='Z025821015U3JJT1NNEGH', rr_name='adsd-app001-stage', zone_name='stage.dice.census.gov.', rr_type='A', rr_value='10.188.104.151'), DnsData(zone_id='Z025821015U3JJT1NNEGH', rr_name='adsd-app001-stage', zone_name='stage.dice.census.gov.', rr_type='TXT', rr_value='"heritage=dynr53,dynr53/version=2.0.6,dynr53/account_id=412295344020,dynr53/region=us-gov-east-1,dynr53/instance_id=i-08c8967c555e58aa1,dynr53/create_time=1692101258"'), DnsData(zone_id='Z018384334G0S5NZOHKUE', rr_name='151', zone_name='104.188.10.in-addr.arpa.', rr_type='PTR', rr_value='adsd-app001-stage.stage.dice.census.gov.'), DnsData(zone_id='Z018384334G0S5NZOHKUE', rr_name='151', zone_name='104.188.10.in-addr.arpa.', rr_type='TXT', rr_value='"heritage=dynr53,dynr53/version=2.0.6,dynr53/account_id=412295344020,dynr53/region=us-gov-east-1,dynr53/instance_id=i-08c8967c555e58aa1,dynr53/create_time=1692101258"')] | [INFO] 2023-08-15T12:07:52.584Z f769fbec-7f1b-4221-84b5-f6715657c090 dns_data records written: [DnsData(zone_id='Z025821015U3JJT1NNEGH', rr_name='adsd-app001-stage', zone_name='stage.dice.census.gov.', rr_type='A', rr_value='10.188.104.151'), DnsData(zone_id='Z025821015U3JJT1NNEGH', rr_name='adsd-app001-stage', zone_name='stage.dice.census.gov.', rr_type='TXT', rr_value='"heritage=dynr53,dynr53/version=2.0.6,dynr53/account_id=412295344020,dynr53/region=us-gov-east-1,dynr53/instance_id=i-08c8967c555e58aa1,dynr53/create_time=1692101258"'), DnsData(zone_id='Z018384334G0S5NZOHKUE', rr_name='151', zone_name='104.188.10.in-addr.arpa.', rr_type='PTR', rr_value='adsd-app001-stage.stage.dice.census.gov.'), DnsData(zone_id='Z018384334G0S5NZOHKUE', rr_name='151', zone_name='104.188.10.in-addr.arpa.', rr_type='TXT', rr_value='"heritage=dynr53,dynr53/version=2.0.6,dynr53/account_id=412295344020,dynr53/region=us-gov-east-1,dynr53/instance_id=i-08c8967c555e58aa1,dynr53/create_time=1692101258"')]

[INFO]	2023-08-15T12:07:52.585Z	f769fbec-7f1b-4221-84b5-f6715657c090	inf-dynamic-route53 stats: source=aws.ec2 state=running assumed_role.cached=4 elapsed_ms=10438.964 end=2023-08-15 12:07:52.585050 sleep.count=2 sleep.time=5.988897884228269 start=2023-08-15 12:07:42.146086 success=4 zones=94 zones.pagination=

• ec2 stop event (DELETE)

[INFO]	2023-08-15T12:08:22.043Z	4e91067c-ef43-4c0a-84ff-34a4d7511714	event: {'version': '0', 'id': '4d780661-21b5-e6fc-f413-9a71fc9db2f5', 'detail-type': 'EC2 Instance State-change Notification', 'source': 'aws.ec2', 'account': '412295344020', 'time': '2023-08-15T12:07:13Z', 'region': 'us-gov-east-1', 'resources': ['arn:aws-us-gov:ec2:us-gov-east-1:412295344020:instance/i-0d087e68380f9dded'], 'detail': {'instance-id': 'i-0d087e68380f9dded', 'state': 'stopped'}} - line number: 304

[INFO]	2023-08-15T12:08:22.044Z	4e91067c-ef43-4c0a-84ff-34a4d7511714	event_action ** DELETE ** instance_id i-0d087e68380f9dded instance_state stopped

[INFO] 2023-08-15T12:08:27.825Z 4e91067c-ef43-4c0a-84ff-34a4d7511714 dns_data records written: []

[INFO]	2023-08-15T12:08:27.825Z	4e91067c-ef43-4c0a-84ff-34a4d7511714	inf-dynamic-route53 stats: source=aws.ec2 state=stopped assumed_role.cached=10 delete_success.True=2 elapsed_ms=5782.316 end=2023-08-15 12:08:27.825934 sleep.count=3 sleep.time=7.077367106872938 start=2023-08-15 12:08:22.043618 success=4 zones=94 zones.pagination=0

Note the timestamps of the events:

• start: 'time': '2023-08-15T12:07:41Z'
• stop: 'time': '2023-08-15T12:07:13Z'

but they were processed in the order of start, stop

Additional context

First observed in ma12-gov.