I effectively cut over to a spare domain, but wow... 14h of incident, and about 1h3m of outage, but this was a SaaS platform that had run successfully for more than a decade prior... so that domain name was hard-coded everywhere, and I needed to change everything.
it's an insane DR recovery, I'm not sure losing your primary domain as a SaaS service is on anyone's failure mode, I'm pleased I could respond and get this done... but wow.
root cause? Gandi never said, but a few threads elsewhere on the internet from a few individuals impacted who were Italian suggested that Gandi had failed to pay their annual registration with nic.sm and so nic.sm suspended all .sm domains that Gandi managed... totally unbelievable and also very believable.
looks like it... it looks like San Marino has a very solid response for lack of payment, and that is damaging to the real end users / customers of registrar services
Comments
my worst domain story is of Gandi hosting microco.sm , the .sm is San Marino, and the domain costs €300 per year as a special business domain.
but... halfway through last year (in May) the domain was suspended, and as DNS caches expired the domain became increasingly unavailable.
I used microco.sm to host an OSS forum platform, and this instance runs over 300 domains.
what then happened was a scramble... Gandi support were effectively non-existent and I was against the clock for those caches expiring.
full incident thread written in public view: https://www.lfgss.com/conversations/386644/
I effectively cut over to a spare domain, but wow... 14h of incident, and about 1h3m of outage, but this was a SaaS platform that had run successfully for more than a decade prior... so that domain name was hard-coded everywhere, and I needed to change everything.
it's an insane DR recovery, I'm not sure losing your primary domain as a SaaS service is on anyone's failure mode, I'm pleased I could respond and get this done... but wow.
root cause? Gandi never said, but a few threads elsewhere on the internet from a few individuals impacted who were Italian suggested that Gandi had failed to pay their annual registration with nic.sm and so nic.sm suspended all .sm domains that Gandi managed... totally unbelievable and also very believable.
I tried to find info on this and ended up in a 2018 HN thread which links to a similar incident but with 101domain instead of Gandi: https://puri.sm/posts/the-great-purism-dns-outage-of-2018/
Makes me wonder if San Marino was at fault in both of these incidents.
looks like it... it looks like San Marino has a very solid response for lack of payment, and that is damaging to the real end users / customers of registrar services