You can code a failsafe, or what they call in other industries, an interlock. If an order exceeds some sane limit, or a loss is incurred that exceeds a limit, or the limit-checker fails to proactively ping the interlock, then an independent mechanism shuts it down.
This is how, say, a modern x-ray machine prevents itself from failing in a way that radiation gets stuck on.
Shit happens. I wrote the firmware for a multi-link HDLC card which earned vast amounts of money for the company that commissioned it. I was quite aware of my responsibility and so was the hardware team. We stress tested the crap out of it before releasing to production and fortunately it never locked up when it was in actual use but we had some pretty wild and very rare (so hard to trigger and reproduce) bugs which delayed deployment considerably. But none of this gave me the kind of feeling that working on fuel estimation software for aircraft gave. That's when there are in the most literal sense lives on the line, and that's a completely different kind of pressure. You simply can not fuck up. It also really rammed home the value of code review and having a good specification so that you can ensure that within the envelope of input parameters your software does what it is supposed to do.
Multi-link HDLC. That brings me back to 1985 and the Intel 8274, Zilog Z8530 and the Western Digital WD2511 (that implemented the layer 2 protocol in silicon).
Yes, that was around that time. There were a couple of problems, the first batch of chips we got was very early in the development stage of the chip, pre-production issues and there were some bugs in the chips themselves which could cause lock-up under some circumstances. We found ways to work around those and then of course there were all of the niceties around dealing with a device that generates an extremely high rate of interrupts. So the code would either have to attempt to service all channels on any interrupt or be re-entrant. I don't remember which solution we picked but in the end the thing was, once we had the bugs worked out fairly bullet proof.
One funny bit about the development process was that initially I was going to be nice and implement every layer as its own stand-alone bit of software communicating via a defined set of primitives with the layers above and below. But that was slow as molasses so in the end all of that elegance got discarded for an absolutely unholy sandwich that did all of the layers in a single chunk of code. But with the experience from the 'slow' version that was actually doable, I would have never been able to write that as the first implementation. Classic illustration of 'first make it work, then make it fast'.
Cool. We had a small company in San Diego called Metacomp (long gone now) design a Z8530 add on module for their 80186 based Multibus board. It had two Z8530 chips on it and you could have two modules on the base board. So 8 channels total per slot.
Yes, 8 was just about the maximum, simply because that would approach the limits of how many interrupts you could process before you'd inevitably start losing them. Also, space and power delivery limitations, and connector space on the backside made doing more than that very impractical. Or you had to take more than one interrupt per board but that was 'not done'. I've always loved working on that division line between hardware and software, as close to the metal as possible. Funny, I'd all but forgotten about this project, now I'm wondering if I still have the software somewhere, just searched for a bit and can't locate it, so there is a good chance I wrote it on company hardware. I did find another project that I'd forgotten about, a CAD program for sails for sailboats that I wrote in the 80's for TD Sails in the Netherlands. That was a great project to work on and I met some awesome people doing it.
This is becoming less and less true every day. 24/7 markets are coming whether you want them or not, and if you're not trading outside of US Equity hours you're leaving significant money on the table.
This is becoming less and less true every day. 24/7 markets are coming whether you want them or not
This has still to exist for real. Every couple of years there is a resurgence of "we should have 24/7 equity markets", followed by milestone announcements of Nasdaq.
The reality is more... nuanced.
First, there has to be a wide enough window for corporate actions and news dissemination to happen. Not only is that a regulatory requirement, but anyway no sane investor would like to trade in such a window even if possible.
Second, there are already early sessions, pre opening sessions, and late sessions on most American markets. The liquidity there is almost inexistant, so I wouldn't say there is a huge demand for extended trading hours by non-retail participants.
And last, half the liquidity of US markets is not on lit books, and half the remaining lit liquidity is close to the auction. I don't think it would even be a net profit for lit exchanges to extend their hours in these conditions. As for dark pool, I think only 2 or 3 of the smaller ones are 24/7, like the one from IB.
if you're not trading outside of US Equity hours you're leaving significant money on the table
HFTs need a lot of liquidity and tight spreads, that's pretty much only doable at scale in the US and Japan. Europe and EM are either not liquid enough, or only really able to sustain low capital ad-hoc strategies.
Genuinely curious - would 24x7 low-latency trading matter then? Wouldn't after-hours trading (for your local timezone) be happening on an exchange in another location?
Or do you mean dark pools/private exchanges that may run 24x7?
Comments
I feel like it's also got to be super stressful
Imagine getting a call you system is down and is costing them millions per hour
You can code a failsafe, or what they call in other industries, an interlock. If an order exceeds some sane limit, or a loss is incurred that exceeds a limit, or the limit-checker fails to proactively ping the interlock, then an independent mechanism shuts it down.
This is how, say, a modern x-ray machine prevents itself from failing in a way that radiation gets stuck on.
The way to think is the opposite: your system is making millions per hour.
No the way to think is ‘it’s down less than our closest competitors system’
What about when something goes wrong because you overlooked some detail?
Shit happens. I wrote the firmware for a multi-link HDLC card which earned vast amounts of money for the company that commissioned it. I was quite aware of my responsibility and so was the hardware team. We stress tested the crap out of it before releasing to production and fortunately it never locked up when it was in actual use but we had some pretty wild and very rare (so hard to trigger and reproduce) bugs which delayed deployment considerably. But none of this gave me the kind of feeling that working on fuel estimation software for aircraft gave. That's when there are in the most literal sense lives on the line, and that's a completely different kind of pressure. You simply can not fuck up. It also really rammed home the value of code review and having a good specification so that you can ensure that within the envelope of input parameters your software does what it is supposed to do.
Multi-link HDLC. That brings me back to 1985 and the Intel 8274, Zilog Z8530 and the Western Digital WD2511 (that implemented the layer 2 protocol in silicon).
Yes, that was around that time. There were a couple of problems, the first batch of chips we got was very early in the development stage of the chip, pre-production issues and there were some bugs in the chips themselves which could cause lock-up under some circumstances. We found ways to work around those and then of course there were all of the niceties around dealing with a device that generates an extremely high rate of interrupts. So the code would either have to attempt to service all channels on any interrupt or be re-entrant. I don't remember which solution we picked but in the end the thing was, once we had the bugs worked out fairly bullet proof.
One funny bit about the development process was that initially I was going to be nice and implement every layer as its own stand-alone bit of software communicating via a defined set of primitives with the layers above and below. But that was slow as molasses so in the end all of that elegance got discarded for an absolutely unholy sandwich that did all of the layers in a single chunk of code. But with the experience from the 'slow' version that was actually doable, I would have never been able to write that as the first implementation. Classic illustration of 'first make it work, then make it fast'.
Cool. We had a small company in San Diego called Metacomp (long gone now) design a Z8530 add on module for their 80186 based Multibus board. It had two Z8530 chips on it and you could have two modules on the base board. So 8 channels total per slot.
Yes, 8 was just about the maximum, simply because that would approach the limits of how many interrupts you could process before you'd inevitably start losing them. Also, space and power delivery limitations, and connector space on the backside made doing more than that very impractical. Or you had to take more than one interrupt per board but that was 'not done'. I've always loved working on that division line between hardware and software, as close to the metal as possible. Funny, I'd all but forgotten about this project, now I'm wondering if I still have the software somewhere, just searched for a bit and can't locate it, so there is a good chance I wrote it on company hardware. I did find another project that I'd forgotten about, a CAD program for sails for sailboats that I wrote in the 80's for TD Sails in the Netherlands. That was a great project to work on and I met some awesome people doing it.
True. If you put in the whole system maybe.
On the other hand, no nights or weekends.
This is becoming less and less true every day. 24/7 markets are coming whether you want them or not, and if you're not trading outside of US Equity hours you're leaving significant money on the table.
This has still to exist for real. Every couple of years there is a resurgence of "we should have 24/7 equity markets", followed by milestone announcements of Nasdaq.
The reality is more... nuanced.
First, there has to be a wide enough window for corporate actions and news dissemination to happen. Not only is that a regulatory requirement, but anyway no sane investor would like to trade in such a window even if possible.
Second, there are already early sessions, pre opening sessions, and late sessions on most American markets. The liquidity there is almost inexistant, so I wouldn't say there is a huge demand for extended trading hours by non-retail participants.
And last, half the liquidity of US markets is not on lit books, and half the remaining lit liquidity is close to the auction. I don't think it would even be a net profit for lit exchanges to extend their hours in these conditions. As for dark pool, I think only 2 or 3 of the smaller ones are 24/7, like the one from IB.
HFTs need a lot of liquidity and tight spreads, that's pretty much only doable at scale in the US and Japan. Europe and EM are either not liquid enough, or only really able to sustain low capital ad-hoc strategies.
Genuinely curious - would 24x7 low-latency trading matter then? Wouldn't after-hours trading (for your local timezone) be happening on an exchange in another location?
Or do you mean dark pools/private exchanges that may run 24x7?
PoE is the way to use low voltage in electronics.
I'd want to know why they had such a system as a single point of failure
No business is designed ideally. Now do you want the job or we move to another one who will accept this?
Fortunately in important industries that attitude doesn't fly.
Imagine a single person making a single mistake causing a building to collapse? Or a nuclear plant to meltdown.
Failure modes can be a bug making a bad trade while doing so at 100% uptime.