I'm in for #2. And no to #1, but I'm not sure we'd notice if we did.
Here's a theory that you might be able to chase down with AWS EC2 support folks:
Many, many EC2 instances are either scheduled to be created on the hour (e.g. by cloudformation/knife ec2/whatever) or are running cron jobs that run hourly;
EC2 provisioning tasks and cron jobs usually require connections to outside servers - package installs, apt-get updates, sending logs to s3, etc. - and that means looking up hostnames;
Lots of hostnames are being looked up on the hour as a result, and <resource X> is being exhausted hourly when a flood of lookups go to the DNS server.
Important caveat: resource X may not be the AWS internal DNS server itself! It can be the port it's connected to being saturated, or a particular uplink on a two port portchannel being flaky (and the flakiness is only evident when it's under high load>, or the elastic interface that is attached to the DNS server, or any one of another dozen things.
Are you seeing this behavior across multiple AZs and regions, or just one?
(This is just a theory, mind you, but I've seen this same behavior when managing other large DNS clusters, and it sounds like a good fit.)
Comments
I'm in for #2. And no to #1, but I'm not sure we'd notice if we did.
Here's a theory that you might be able to chase down with AWS EC2 support folks:
Many, many EC2 instances are either scheduled to be created on the hour (e.g. by cloudformation/knife ec2/whatever) or are running cron jobs that run hourly;
EC2 provisioning tasks and cron jobs usually require connections to outside servers - package installs, apt-get updates, sending logs to s3, etc. - and that means looking up hostnames;
Lots of hostnames are being looked up on the hour as a result, and <resource X> is being exhausted hourly when a flood of lookups go to the DNS server.
Important caveat: resource X may not be the AWS internal DNS server itself! It can be the port it's connected to being saturated, or a particular uplink on a two port portchannel being flaky (and the flakiness is only evident when it's under high load>, or the elastic interface that is attached to the DNS server, or any one of another dozen things.
Are you seeing this behavior across multiple AZs and regions, or just one?
(This is just a theory, mind you, but I've seen this same behavior when managing other large DNS clusters, and it sounds like a good fit.)