We are in the process of mitigating a large scale DDoS attack against our global DNS platform. We expect service to return to normal very shortly. Stay tuned and let me know if you have any questions. ted@namecheap.com
Just for future reference, it's usually considered a good idea to put your status page on completely independent infrastructure so that it stays up even when the rest of your stuff goes down. A status page that doesn't work during an outage isn't particularly useful.
Good point. The status page is on another cloud but since this is a DNS issue, the subdomain is down. In the future, we'll investigate running this page on a secondary DNS.
According to whois for namecheap.com, the DNS for that domain is hosted on dynect.net, and "host status.namecheap.com" resolves just fine (to 204.232.212.56), so it does not appear to be a DNS issue that is preventing your status page from working.
Perhaps also investigate allowing customers to slave their own secondary DNS, too? (That is, allow AXFRs.)
This has been a feature requested for, as far as I can tell from the support forums, four years now. It should be possible to make this allowed/disallowed per-zone, and as far as I'm concerned, I don't care who is able to download my tiny zone file. It would allow me to add more diversified DNS servers in the face of things like a DoS against Namecheap.
Posting up-to-the-minute updates on Twitter is also a good idea. Lots of tweets come back from a search for "Namecheap" & you want to be sure you're a part of that conversation!
Cloudflare has also been nailed the last 3 weeks in a row causing outages. Just search Twitter for cloudflare DDOS or NTP or etc. etc. etc. At the end of the day there are currently so many slave machines out there we are all vulnerable. It's just the nature of things. At least outages are only temporary. It's much better than the early days were we'd be down for 12 hours at a time. https://twitter.com/search?q=cloudflare%20attack&src=typd
I suspect the hackers are targeting one of namecheap's customers, not namecheap directly. Because that's usually the case, a good approach is to not give all customers the exact same nameservers.
I agree. We left Network Solutions b/c they were hit 3 times in less than a year. Maybe the website targets moved to Namecheap, so they were targeted also?
Yes please, thank you :) All my domains are down now but I understand how shitty (the panic!) it feels to have things going down. I'll be checking this.
I tried this and now for some reason my apache server is returning the default page. also I think it wiped out my mailserver settings -- my mail is hosted on namecheap. i guess i gotta wait till everything is fixed.
I had a couple of domains to try this on. One domain switched over within a minute. The other has been almost an hour. I'm assuming the TLD makes a difference?
ETA how long it will take for the transfer to be completed? I know that editing host records usually is instantaneous (unlike other providers), but we're talking about changing the DNS servers here.
Was going to switch to route 53 while you guys were down and switch back, but the page says it'll take a day...Might as well just wait at that point. I know it's panic mode over there, but some kind of failover record would be awesome for when this happens in the future(or an option for one).
Everything related to DNS says it'll take a day but that's "worst case"... I've successfully transferred DNS for 5 Namecheap-registered domains to Route 53 since this all started a couple hours ago, and they're now up & running smoothly.
Their "update DNS server" page was acting a bit wonky, kept saying some of my nameservers were invalid when they weren't, but I eventually got them all switched.
This isn't a dig against Namecheap, it sounds like this attack is pretty bad, but for important domains Route53 just seems like a much better setup (geographically dispersed, different nameservers for each hosted zone, etc).
Hi, I work on Route 53, can you clarify; are you seeing a problem from your registrar when you try to move to Route 53? We're also always keen to hear about features customers would like.
It's definitely on our list, but in the meantime there is https://code.google.com/p/route53d/ which will let you translate an AXFR/IXFR from a bind server directly into Route 53 authenticated calls.
Does it take the same amount of time to complete that switching to another provider's DNS server takes?
Many of us already switched to other DNS servers during the panic, and we'd like to know whether you'd suggest we reverse that change in order to jump to v1 instead for a quicker result. Thank you.
You need to provide some gift to your customers for this downtime. I am using NC for 10 years, every time on bad issues I continue to use. But this outage very bad, I lost money...
While it's not a customer friendly viewpoint, it's not exactly what happened here. You have to consider what is being provided and how much was paid into it for said service. Had they been paying a lot of money for solid DNS I could agree with you, but they weren't.
It's a completely free service. With an extremely low price for domains.
Even though it's become almost a standard for domain registrars to provide DNS, it's still free. I've been using NC for years for my business (also a free service) and the downtime for what it provides has been mostly minimal.
Now if I was paying namecheap specifically for enterprise DNS something like $10/month or $40/year and it included DDoS protection then yes I should be compensated. Namecheap gave people what they paid for - a domain, the DNS is just a really great perk.
If namecheap chose to stop assigning all customers the same list of DNS servers, it would benefit namecheap as well. There's 200+ people in the queue for live chat right now :)
It's not DDOS mitigation or H/A, or some other high end feature.
The only way this would be successful is if the "customer IP's" were spread accross separate networks, and the announcement for the attacked network was sent somewhere else.
Assigning customers across lots of IP's in the same /24 isn't going to do anything. A volumetric attack is still going to succeed there.
In this landscape of ever-dwindling portable IPv4 subnets, it's harder and harder to get a /24. You won't get an assignment if your justification is "to fight a DDoS."
Are you idiot or kid? We are doing professional business and we were using namecheap DNS with registered domains (not free dns). The main point is protection, they provide this service FREE (with or without domain) yeah good but where is protection where is responsibility?
I just suggested something, if money important for me, I didn't move all of my domains from namecheap to another provider :)
That protection and responsibility is defined in your contract or registration agreement with NameCheap. You can consider DNS or DDoS protection NameCheap's responsibility, and they'll decide -- like any business -- which one they can afford.
While offering some gift or compensation for ANY negative incident -- especially one outside anyone's control -- is nice, that's not something everyone can afford under ANY circumstance.
If they messed up your cheeseburger, I could see this. How does this help this situation? It doesn't. Namecheap should take that money and invest in their infrastructure.
Comments
We are in the process of mitigating a large scale DDoS attack against our global DNS platform. We expect service to return to normal very shortly. Stay tuned and let me know if you have any questions. ted@namecheap.com
Just for future reference, it's usually considered a good idea to put your status page on completely independent infrastructure so that it stays up even when the rest of your stuff goes down. A status page that doesn't work during an outage isn't particularly useful.
Good point. The status page is on another cloud but since this is a DNS issue, the subdomain is down. In the future, we'll investigate running this page on a secondary DNS.
According to whois for namecheap.com, the DNS for that domain is hosted on dynect.net, and "host status.namecheap.com" resolves just fine (to 204.232.212.56), so it does not appear to be a DNS issue that is preventing your status page from working.
I stand corrected. It's on Rackspace, which apparently is also experiencing issues.
Rackspace helped us get our status page back online: http://status.namecheap.com/
Perhaps also investigate allowing customers to slave their own secondary DNS, too? (That is, allow AXFRs.)
This has been a feature requested for, as far as I can tell from the support forums, four years now. It should be possible to make this allowed/disallowed per-zone, and as far as I'm concerned, I don't care who is able to download my tiny zone file. It would allow me to add more diversified DNS servers in the face of things like a DoS against Namecheap.
Posting up-to-the-minute updates on Twitter is also a good idea. Lots of tweets come back from a search for "Namecheap" & you want to be sure you're a part of that conversation!
They've been posting updates 30 minutes before your post buddy, and still are now. https://twitter.com/Namecheap
Yes, we are actively posting updates on Twitter.
If the hackers really want to take you down, adding another server in isn't very hard...
If you're as big as NameCheap, you can afford to pay CloudFlare or somebody like it to protect your status page.
Cloudflare has also been nailed the last 3 weeks in a row causing outages. Just search Twitter for cloudflare DDOS or NTP or etc. etc. etc. At the end of the day there are currently so many slave machines out there we are all vulnerable. It's just the nature of things. At least outages are only temporary. It's much better than the early days were we'd be down for 12 hours at a time. https://twitter.com/search?q=cloudflare%20attack&src=typd
I suspect the hackers are targeting one of namecheap's customers, not namecheap directly. Because that's usually the case, a good approach is to not give all customers the exact same nameservers.
I agree. We left Network Solutions b/c they were hit 3 times in less than a year. Maybe the website targets moved to Namecheap, so they were targeted also?
Like!
Not having separate status hosting is liken not salting passwords. As a NC user, I may leave just for this reason
WOOH the Akamai realtime attack visualization is completely red at the moment: http://www.akamai.com/html/technology/dataviz1.html
Someone is apparently pissed at the state of Indiana, 105 attacks in the last 24hours
370,000 attacks/hour, but that is still 56% below average.. hmmm interesting!
Most DDOS attacks against a company like yours are actually attacks against a specific customer. If:
a) you had a pool of DNS server names, say 20, all with unrelated hostnames
b) you assigned 2 to each customer, randomly, when they configured a domain to use your servers.
Then, a DDOS attack would impact 10% of your customers instead of 100%. (Assuming other practices, like null routing the target until resolved)
Yes, this. Hopefully they take heed after this painful experience.
Yes please, thank you :) All my domains are down now but I understand how shitty (the panic!) it feels to have things going down. I'll be checking this.
I'd also like to add that we have redunancy on our DNS V1. I advise switching over to this, in the meantime. Please find the tutorial here: https://www.namecheap.com/support/knowledgebase/article.aspx...
I tried this and now for some reason my apache server is returning the default page. also I think it wiped out my mailserver settings -- my mail is hosted on namecheap. i guess i gotta wait till everything is fixed.
I had a couple of domains to try this on. One domain switched over within a minute. The other has been almost an hour. I'm assuming the TLD makes a difference?
.com was < 1 minute, .cm is > 55 minutes.
ETA how long it will take for the transfer to be completed? I know that editing host records usually is instantaneous (unlike other providers), but we're talking about changing the DNS servers here.
Looks to be up around five to ten minutes after executing the change. Not too terrible.
Thanks! I just did the switch and the website was up in few minutes.
Was going to switch to route 53 while you guys were down and switch back, but the page says it'll take a day...Might as well just wait at that point. I know it's panic mode over there, but some kind of failover record would be awesome for when this happens in the future(or an option for one).
Everything related to DNS says it'll take a day but that's "worst case"... I've successfully transferred DNS for 5 Namecheap-registered domains to Route 53 since this all started a couple hours ago, and they're now up & running smoothly.
Their "update DNS server" page was acting a bit wonky, kept saying some of my nameservers were invalid when they weren't, but I eventually got them all switched.
This isn't a dig against Namecheap, it sounds like this attack is pretty bad, but for important domains Route53 just seems like a much better setup (geographically dispersed, different nameservers for each hosted zone, etc).
Yeah, Route53 isn't as feature rich though. Also I have the same problem, keeps giving me errors about bad nameservers. I guess I'll keep trying.
Hi, I work on Route 53, can you clarify; are you seeing a problem from your registrar when you try to move to Route 53? We're also always keen to hear about features customers would like.
Secondary DNS from a normal bind nameserver.
I second this. I'll add that Route 53 would need to respond to notifies for this to be useful.
It's definitely on our list, but in the meantime there is https://code.google.com/p/route53d/ which will let you translate an AXFR/IXFR from a bind server directly into Route 53 authenticated calls.
You guys should really put a notice on your homepage or something.
Good luck.
The homepage wasn't accessible until just a bit ago. We're still working on it.
tamar (also at Namecheap)
I took your advice and transferred a domain to DNSv1. That domain is still not back online, but all the domains I left as DNSv2 have come back!
You guys should add an option for us to put in an optional backup DNS record in the settings.
DNS V1 is available and functioning.
I dont have this option in my account
The quick tut on switching to DNSv1 is nice however the option doesnt exist in my account
Thanks for posting here!
Kudos for using a HN page to stay in touch
Ho quickly will the switch to v1 execute?
It generally takes less than an hour.
Does it take the same amount of time to complete that switching to another provider's DNS server takes?
Many of us already switched to other DNS servers during the panic, and we'd like to know whether you'd suggest we reverse that change in order to jump to v1 instead for a quicker result. Thank you.
hi already have OK's, but the DNS are all using the same IP :( bad sign?
When did this attack start ?
a little more than 2 hours ago
Thanks for the update, guys.
Good luck guys!
Down again My domain down again
do you have ETA?
We're working hard to resolve this and we will keep you posted with updates here. Unfortunately, I can't give you an exact ETA.
You need to provide some gift to your customers for this downtime. I am using NC for 10 years, every time on bad issues I continue to use. But this outage very bad, I lost money...
DDOS attacks are not up to Namecheap. You want money, go talk to the people doing it. These are very hard to prevent.
[deleted]
While it's not a customer friendly viewpoint, it's not exactly what happened here. You have to consider what is being provided and how much was paid into it for said service. Had they been paying a lot of money for solid DNS I could agree with you, but they weren't.
It's a completely free service. With an extremely low price for domains.
Even though it's become almost a standard for domain registrars to provide DNS, it's still free. I've been using NC for years for my business (also a free service) and the downtime for what it provides has been mostly minimal.
Now if I was paying namecheap specifically for enterprise DNS something like $10/month or $40/year and it included DDoS protection then yes I should be compensated. Namecheap gave people what they paid for - a domain, the DNS is just a really great perk.
If namecheap chose to stop assigning all customers the same list of DNS servers, it would benefit namecheap as well. There's 200+ people in the queue for live chat right now :)
It's not DDOS mitigation or H/A, or some other high end feature.
The only way this would be successful is if the "customer IP's" were spread accross separate networks, and the announcement for the attacked network was sent somewhere else.
Assigning customers across lots of IP's in the same /24 isn't going to do anything. A volumetric attack is still going to succeed there.
In this landscape of ever-dwindling portable IPv4 subnets, it's harder and harder to get a /24. You won't get an assignment if your justification is "to fight a DDoS."
So namecheap would need a few different, separate IPv4 subnets.
Like the ones they already have?
http://bgp.he.net/AS22612#_prefixes
Good point. They should be spreading things around more.
That's what insurance is for.
They could better protect against it by not assigning the same list of DNS servers to every customer.
Are you idiot or kid? We are doing professional business and we were using namecheap DNS with registered domains (not free dns). The main point is protection, they provide this service FREE (with or without domain) yeah good but where is protection where is responsibility?
I just suggested something, if money important for me, I didn't move all of my domains from namecheap to another provider :)
That protection and responsibility is defined in your contract or registration agreement with NameCheap. You can consider DNS or DDoS protection NameCheap's responsibility, and they'll decide -- like any business -- which one they can afford.
While offering some gift or compensation for ANY negative incident -- especially one outside anyone's control -- is nice, that's not something everyone can afford under ANY circumstance.
If they messed up your cheeseburger, I could see this. How does this help this situation? It doesn't. Namecheap should take that money and invest in their infrastructure.
I'm sure they will. Just a matter of balancing stuff, of course.