The reliance of Ukraine’s substations on GPS for time synchronization, a standard in industrial control systems for its accuracy and affordability, becomes a vulnerability when faced with such jamming.
Cisco’s response involved shipping a large order of modified equipment, specifically designed to maintain accurate time even under radio jamming conditions. This solution employs the Cisco Industrial Ethernet switch with an internal crystal oscillator, enabling new clock recovery algorithms for accurate timekeeping when GPS is unavailable.
That's really neat. Didn't know GPS was also used for time synchronization.
The electric grid is already extremely fragile. Its uptime is dependent on constant maintenance and repairs and many people on 24/7 standby to make more repairs in outages.
Its resilience is in the support network, not in the physical structures itself.
That support network building custom hardware in a GPS outage is that system functioning as expected. A GPS outage without a bunch of concerned humans doing everything they can to fix it isn’t going to happen.
Also on making sure that it is balanced all the time. The effort it takes to make sure that there's always input exactly as much energy as is taken out of the system is just staggering. And if only a few of the things needed to adjust it fail .. here comes the brown/blackout.
Most of the effort has been done in design. Rotating mass of generators is a nice self-correcting system, also the frequency is a near-zero latency signaling mechanism across the entire grid. Each participant can measure it and make independent decisions whether the grid needs more power or more consumption.
There is a lot of work that goes into predicting and commanding large participants and it's a complex system but some of the design is just beautiful in it's simplicity.
That's really interesting! I assumed that everybody would aim for a resilient system to reduce support costs, but you're saying the electric grid is built cheap & fragile and the cost of support is just an accepted part of it?
How come the support costs are cheaper for maintaining the electric grid than paying a bit more for hardware that requires less support?
You can't even outsource your support costs to a cheaper COL country.
Is it because the infra is subject to nature & thus always being degraded in all sorts of ways in all different environments?
Yes, once you realize even the ground you walk on is shifting cm's per year like a slow moving ocean, you see that nothing fits nicely in the world indefinitely with then precision needed for electricity infra.
There's a lot of slop in electricity infrastructure especially in those big towers with cable drapes of hundreds of feet.
Australia, the continent, has shifted by 4.9 feet since the last local mapping grid adjustment was made to GPS coordinates back in 1994.
That's about 2.7 inches a year .. that took entire regional electrical service areas along with it as a single unit with no stretching of cables.
In your scenario you need to find electrical works spanning a fault line, rigidly fixed to both sides, and built by Engineers (civil, mech, not software) clueless to drift .. like a steel bridge constructed without expansion joints to account for tempreture changes.
My whole point is that it is indeed incredibly reliable, but only because it is undergoing constant maintenance 24/7 by trained professionals.
Take the humans out of the loop and it falls over by itself very soon.
There is no situation where, say, one of its dependencies goes away and lots of smart people don’t engineer a fix ASAP. This was in response to someone expressing concern that the grid is dependent on GPS. It’s dependent on a lot of things, one of which is probably Home Depot. If any of those dependencies go away, we will re-engineer the dependency tree, because it’s not operating in a vacuum. It’s resilient and reliable because it is staffed and continuously engineered.
There's only two ways you can lose GPS from a fixed location: local jamming, or all of the satellites getting disabled somehow.
A jammer will typically be detected and stopped before any damage is done. Even owning a GPS jammer is illegal in most places.
All of the satellites going down is probably a cataclysmic scenario, like global nuclear war or solar flares. Not worth worrying about because you'll have much bigger problems.
GPS is one of the most reliable systems there is. There are redundant constellations of satellites run by different companies in different countries. It'd be near impossible to disrupt service from all of them at once.
GPS is one of the most reliable systems there is. There are redundant constellations of satellites run by different companies in different countries.
1. GPS refers specifically to the system run by the united states government. The generic term is GNSS
2. Even though there are multiple GNSS systems out there, and modern phones support multiple, it's unknown whether the same applies to other systems. Supporting multiple systems costs more money, so I suspect hardware vendors might skimp on that, especially when the benefits are so marginal.
Basically, GPS satellites need ridiculous precision in their clocks to make navigation work. They all have atomic reference clocks to accomplish this.
Since GPS satellites are already broadcasting a ridiculously precise signal to the entire planet for free, it makes a lot of sense to use that signal when you need very precise time.
Though in cases like this where critical infrastructure is involved, I'm confused why they don't have an adequate clock source locally. You don't usually rely totally on GPS, you use it as a reference to check your local clock against.
More to the point, all GPS satellites broadcast the time - the receiver takes the multiple (different) times from multiple satellites and the satellite's known location to determine the receiver's location.
It’s related to power generation. Finding line faults, keeping equipment at 60hz or 50hz whatever they use there.
They need precision location information for multiple power plants. They can do without it but you end up with dirty power which most equipment doesn’t like very much and some will shut off and not work at all or break.
You can make a reference oscillator almost as precise as an atomic clock with a temperature compensated crystal oscillator. They cost tens of dollars.
We've also been synchronizing grid generation since long before spaceflight. It seems that the core problems here have been solved a long time ago.
Then again, I don't know much about the grid at this scale. I very much doubt that a few PPM phase difference would actually matter, but I don't really know.
Either way, it still seems insane to rely on GPS for your sole source of truth, even with its remarkable reliability.
Radio clocks [1] often don’t use GPS. Many countries have a radio signal they can pick up. In the continental US, it comes from Fort Collins, Colorado. [2]
Technically you can broadcast a new time to your neighborhood and have most people late to work (the ones having that kind of clock alarm), it's far from secure and accurate.
So the pragmatic approach that everyone is utilizing (including AWS’ latest announcement for microsecond accuracy) relies on hardware GPS. Why am I being attacked here. This is how it’s done.
I'm not the one attacking you, but I think I can explain.
Your original statement was quite absolutist "It’s really the only way to do accurate time." A statement which would have had more agreement would be "It’s really the only practical way to do accurate time."
Precision Time Protocol [1] is an alternative. CERN has been using it as part of a project to achieve sub-nanosecond accuracy over a wired network [2].
Comments
That's really neat. Didn't know GPS was also used for time synchronization.
Bit horrifying if you ask me, because I bet that happens everywhere. So if we lose GPS, we lose electricity? Awesome. Not at all fragile.
The electric grid is already extremely fragile. Its uptime is dependent on constant maintenance and repairs and many people on 24/7 standby to make more repairs in outages.
Its resilience is in the support network, not in the physical structures itself.
That support network building custom hardware in a GPS outage is that system functioning as expected. A GPS outage without a bunch of concerned humans doing everything they can to fix it isn’t going to happen.
Also on making sure that it is balanced all the time. The effort it takes to make sure that there's always input exactly as much energy as is taken out of the system is just staggering. And if only a few of the things needed to adjust it fail .. here comes the brown/blackout.
Most of the effort has been done in design. Rotating mass of generators is a nice self-correcting system, also the frequency is a near-zero latency signaling mechanism across the entire grid. Each participant can measure it and make independent decisions whether the grid needs more power or more consumption.
There is a lot of work that goes into predicting and commanding large participants and it's a complex system but some of the design is just beautiful in it's simplicity.
That's really interesting! I assumed that everybody would aim for a resilient system to reduce support costs, but you're saying the electric grid is built cheap & fragile and the cost of support is just an accepted part of it?
How come the support costs are cheaper for maintaining the electric grid than paying a bit more for hardware that requires less support?
You can't even outsource your support costs to a cheaper COL country.
Is it because the infra is subject to nature & thus always being degraded in all sorts of ways in all different environments?
Yes, once you realize even the ground you walk on is shifting cm's per year like a slow moving ocean, you see that nothing fits nicely in the world indefinitely with then precision needed for electricity infra.
There's a lot of slop in electricity infrastructure especially in those big towers with cable drapes of hundreds of feet.
Australia, the continent, has shifted by 4.9 feet since the last local mapping grid adjustment was made to GPS coordinates back in 1994.
That's about 2.7 inches a year .. that took entire regional electrical service areas along with it as a single unit with no stretching of cables.
In your scenario you need to find electrical works spanning a fault line, rigidly fixed to both sides, and built by Engineers (civil, mech, not software) clueless to drift .. like a steel bridge constructed without expansion joints to account for tempreture changes.
Is it? How often do you experience outages?
It needs maintenance but it's incredibly reliable and resilient if you zoom back a bit.
My whole point is that it is indeed incredibly reliable, but only because it is undergoing constant maintenance 24/7 by trained professionals.
Take the humans out of the loop and it falls over by itself very soon.
There is no situation where, say, one of its dependencies goes away and lots of smart people don’t engineer a fix ASAP. This was in response to someone expressing concern that the grid is dependent on GPS. It’s dependent on a lot of things, one of which is probably Home Depot. If any of those dependencies go away, we will re-engineer the dependency tree, because it’s not operating in a vacuum. It’s resilient and reliable because it is staffed and continuously engineered.
That is as valuable a comment as humans are incredibly fragile as we need to drink water every day.
But fails to mention that we have a self healing immune system along with the background processing that makes sure we drink water every fay.
There's only two ways you can lose GPS from a fixed location: local jamming, or all of the satellites getting disabled somehow.
A jammer will typically be detected and stopped before any damage is done. Even owning a GPS jammer is illegal in most places.
All of the satellites going down is probably a cataclysmic scenario, like global nuclear war or solar flares. Not worth worrying about because you'll have much bigger problems.
GPS is one of the most reliable systems there is. There are redundant constellations of satellites run by different companies in different countries. It'd be near impossible to disrupt service from all of them at once.
1. GPS refers specifically to the system run by the united states government. The generic term is GNSS
2. Even though there are multiple GNSS systems out there, and modern phones support multiple, it's unknown whether the same applies to other systems. Supporting multiple systems costs more money, so I suspect hardware vendors might skimp on that, especially when the benefits are so marginal.
"hindering the identification of issues like line breaks" doesn't quite sound like completely losing electricity.
Basically, GPS satellites need ridiculous precision in their clocks to make navigation work. They all have atomic reference clocks to accomplish this.
Since GPS satellites are already broadcasting a ridiculously precise signal to the entire planet for free, it makes a lot of sense to use that signal when you need very precise time.
Though in cases like this where critical infrastructure is involved, I'm confused why they don't have an adequate clock source locally. You don't usually rely totally on GPS, you use it as a reference to check your local clock against.
More to the point, all GPS satellites broadcast the time - the receiver takes the multiple (different) times from multiple satellites and the satellite's known location to determine the receiver's location.
It’s related to power generation. Finding line faults, keeping equipment at 60hz or 50hz whatever they use there.
They need precision location information for multiple power plants. They can do without it but you end up with dirty power which most equipment doesn’t like very much and some will shut off and not work at all or break.
I remember UK had an issue where they produced low frequency power and everyone’s kettles started turning on early due to the clock mismatch. https://medium.com/drax/what-is-electrical-frequency-and-why...
You can make a reference oscillator almost as precise as an atomic clock with a temperature compensated crystal oscillator. They cost tens of dollars.
We've also been synchronizing grid generation since long before spaceflight. It seems that the core problems here have been solved a long time ago.
Then again, I don't know much about the grid at this scale. I very much doubt that a few PPM phase difference would actually matter, but I don't really know.
Either way, it still seems insane to rely on GPS for your sole source of truth, even with its remarkable reliability.
It’s really the only way to do accurate time.
Radio clocks [1] often don’t use GPS. Many countries have a radio signal they can pick up. In the continental US, it comes from Fort Collins, Colorado. [2]
[1] https://en.m.wikipedia.org/wiki/Radio_clock
[2] https://en.m.wikipedia.org/wiki/WWVB
To be fair, this isn't really _that_ accurate, it's only good to about 10ms, GPS will get you to 0.001ms accuracy, at least with the right kit.
Technically you can broadcast a new time to your neighborhood and have most people late to work (the ones having that kind of clock alarm), it's far from secure and accurate.
That suffers from the same vulnerabilities.
I have trouble picking up WWVB indoors at home even without jamming.
Well no, it isn't - it's just that every other solution is several orders of magnitude more expensive.
So the pragmatic approach that everyone is utilizing (including AWS’ latest announcement for microsecond accuracy) relies on hardware GPS. Why am I being attacked here. This is how it’s done.
I'm not the one attacking you, but I think I can explain.
Your original statement was quite absolutist "It’s really the only way to do accurate time." A statement which would have had more agreement would be "It’s really the only practical way to do accurate time."
Fair.
Precision Time Protocol [1] is an alternative. CERN has been using it as part of a project to achieve sub-nanosecond accuracy over a wired network [2].
[1] https://en.wikipedia.org/wiki/Precision_Time_Protocol
[2] https://en.wikipedia.org/wiki/White_Rabbit_Project
But PTP needs a grandmaster clock, and that is usually a GPS receiver.
Or atomic clock.