You don’t even need a tcxo. If you don’t care about holdover, connecting the pps pin from a $5 GPS module to an $15 SBC will give you a time source that is significantly better than anything you can serve over NTP to your clients. NTP is always gonna be the weak link for a service like this.
With that underneath you need just one gps clock reciever for the whole LAN, and PTP capable Ethernet adapter and switches throughout, of course. Which most are, these days.
Do you have experience with this? Playing around with PTP has been on my TODO list for quite a while, but it seems to be quite a hassle.
At home, I already have a GPS module with its PPS pin connected to an Pi running an NTP server. In my data center, I use clock.sjc.he.net and that's about as good as NTP gets. Certainly good enough for one-way latency measurements.
Are there any SBCs with hardware timestamping and a PPS input? I've heard mixed reviews of the BeagleBone Black as a PTP server. Then again, even if I get that working I have no way to run PTP in the data center since there's no GPS reception in the cabinets.
It _is_ a hassle, and you should have a need for that level of synchronicity. We did it in our team for an in-vehicle network, with the vehicle GPS reciever as the network PTP time master and we figured how to access the ethernet PTP register both from userspace and from the kernel. Kernel for the system clock, user space to avoid latencies from kernel Userland interaction in some high precision real time task. It was a major effort and it required low level hackery way above my personal head.
"Amazon provides the Amazon Time Sync Service, which is accessible from all EC2 instances, and is also used by other AWS services. This service uses a fleet of satellite-connected and atomic reference clocks in each AWS Region to deliver accurate current time readings of the Coordinated Universal Time (UTC) global standard through Network Time Protocol (NTP). The Amazon Time Sync Service automatically smooths any leap seconds that are added to UTC.
"The Amazon Time Sync Service is available through NTP"
As I understand it, “amazon time sync service” is just a bunch of NTP servers run by Amazon. Their ClockBound library calculates error bounds the same way you would if you were using a different NTP server: |Local Offset| + Root Dispersion + (Root Delay / 2)
how exactly would GPS work at scale for 10,000s of servers
I’m not sure what you meant by “clocks on the system board” but if you’re talking about the system clock on the NTP server or the NTP client, it really doesn’t matter. Any old crystal on the NTP server is accurate enough to stay locked to a PPS input. The clients are limited by NTP so their clocks don’t really matter either.
Comments
Amazon time sync likely also requires extremely accurate clocks on the system board as well to prevent there from being too much drift.
You don’t even need a tcxo. If you don’t care about holdover, connecting the pps pin from a $5 GPS module to an $15 SBC will give you a time source that is significantly better than anything you can serve over NTP to your clients. NTP is always gonna be the weak link for a service like this.
That's when you move from NTP to PTP, which in a LAN provides sub microsecond accuracy.
https://en.m.wikipedia.org/wiki/Precision_Time_Protocol
With that underneath you need just one gps clock reciever for the whole LAN, and PTP capable Ethernet adapter and switches throughout, of course. Which most are, these days.
Do you have experience with this? Playing around with PTP has been on my TODO list for quite a while, but it seems to be quite a hassle.
At home, I already have a GPS module with its PPS pin connected to an Pi running an NTP server. In my data center, I use clock.sjc.he.net and that's about as good as NTP gets. Certainly good enough for one-way latency measurements.
Are there any SBCs with hardware timestamping and a PPS input? I've heard mixed reviews of the BeagleBone Black as a PTP server. Then again, even if I get that working I have no way to run PTP in the data center since there's no GPS reception in the cabinets.
It _is_ a hassle, and you should have a need for that level of synchronicity. We did it in our team for an in-vehicle network, with the vehicle GPS reciever as the network PTP time master and we figured how to access the ethernet PTP register both from userspace and from the kernel. Kernel for the system clock, user space to avoid latencies from kernel Userland interaction in some high precision real time task. It was a major effort and it required low level hackery way above my personal head.
I was referring to the amazon time sync service which I imagine has pretty tight timing requirements (similar to spanner), not NTP.
Also, how exactly would GPS work at scale for 10,000s of servers within 1000s of racks within a DC?
"Amazon provides the Amazon Time Sync Service, which is accessible from all EC2 instances, and is also used by other AWS services. This service uses a fleet of satellite-connected and atomic reference clocks in each AWS Region to deliver accurate current time readings of the Coordinated Universal Time (UTC) global standard through Network Time Protocol (NTP). The Amazon Time Sync Service automatically smooths any leap seconds that are added to UTC.
"The Amazon Time Sync Service is available through NTP"
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time...
As I understand it, “amazon time sync service” is just a bunch of NTP servers run by Amazon. Their ClockBound library calculates error bounds the same way you would if you were using a different NTP server: |Local Offset| + Root Dispersion + (Root Delay / 2)
GPS module (+ GPSDO + Rb clock optional) -> SBC running NTP server -> machines running NTP client
I’m not sure what you meant by “clocks on the system board” but if you’re talking about the system clock on the NTP server or the NTP client, it really doesn’t matter. Any old crystal on the NTP server is accurate enough to stay locked to a PPS input. The clients are limited by NTP so their clocks don’t really matter either.
I see. This different than what I thought it was. If it is indeed a simple NTP server, this makes sense.
By using PTP in the data center LAN to distribute the GPS time locally.
https://en.m.wikipedia.org/wiki/Precision_Time_Protocol