What Kind of Technology is Used in HFT?
There's been some interest in the comments about the tech of High Frequency Trading, so I figured it might be good to do a quick overview of some of the technology used in HFT.
First, a bit of terminology...
A lot of times I hear the terms "algo" or "algorithmic" trading used synonymous to HFT. While just about every HFT shop is also algo, just because a firm describes themselves as algo doesn't necessarily imply they are HFT. Algo trading just means you use algorithms to do one or more of the following tasks 1) decide when and where to execute a trade and/or 2) generate the signal that initiated the trade.
There are plenty of shops that do either 1 or 2 but not both. These are usually mid to longer term strategies.
That might sound wierd, to hear me decompose the terms "trade" and "signal" so let me explain a bit. A trade is simply the act of buying or selling something. The signal is the reason for the trade. The signal is why we are doing the trade.
So anyways, in what follows I'm really talking about algo HFT shops here where both the signals and the trade execution are almost completely done by machines. There are a number of technologies which shops tend to cluster around, and I'll go into those first. Then I'll talk about alternative technologies that some firms use to accomplish the same tasks in a different way. Finally, I'll talk about the benefits of being slightly different living and trading in a world where decisions are off-loaded to computers. The speed and scale at which you can do something stupid in HFT is mind-boggling.
![](https://steemitimages.com/640x0/https://steemitimages.com/DQmRkBafpUR7KrZ2arZutYfMsZ72tHPg2h1hJCEEGoqRUqL/image.png)
There are a lot of well-trodden paths in HFT. Many, many shops host their trading software on machines running the Linux OS. The vast majority of strategies are written in some descendant of C. Many firms use FPGA technology to speed up their processing of network messages. There are good technical reasons to do this. Linux is free, open source, provides lots of customization, and is extremely extensible. C and its ilk are fast, while FPGA's offload tasks to hardware which used to be performed by software, minimizing reaction time considerably.
Likewise, quant's can follow the road more traveled and use R to analyze and manipulate their high frequency datasets. Those frustrated by R's limitations -- or those who want greater flexibility -- can plump for Python and its PyData stack. We can all use big data tech like Hadoop or Spark for big data processing.
On the other hand, some choose to be wildly different.
Some people prefer to remain on the fringe, using technologies that haven't been adopted by the broader market. Consider the prop trading firm Jane Street, which utilizes functional programming languages like oCaml to run their trading desk.
Quants who prefer to be different can use tools like Scala for data processing or HDF for big data storage (we are partial to HDF, might be partially because of the UIUC connection). Neither technology has experienced wide-spread adoption, despite active development from their backers.
In the context of High Frequency Trading, what are the benefits of walking the road less traveled?
While there are many arguments we could make, the biggest benefit comes from synchronicity, or the simultaneous occurrence of events that appear significantly related but have no discernible causal connection. This definition brings to mind events like the equity flash crash in May of 2010 or the bond flash rally in October 2014. But it's perhaps subtler than that: the technology we use shapes the way our minds think in the same way human language shapes individual thought.
Flash events like this are examples of extreme synchronicity in the markets. This sync can occur from many sources.
For example, suppose everyone individually implements a rule to shut off their strategies when the market has moved X% in Y seconds. This seemingly prudent rule on an individual scale can have big consequences on the global, market scale. Consider that if everyone implemented this kind of rule, then a move bigger than X% will synchronize everyone's strategies BY SHUTTING EVERYONE'S STRATEGIES OFF AT THE SAME TIME. Ever noticed the market just disappear while you're trading it during a big move?
Another example: do you trade the same market, using the same technology stack, with the same datafeed as your competitor? Then your strategies will share a latent synchronicity with your competitors.
A great example would be two HFT competitors using the same microwave or undersea line to connect two exchanges which have a wide geographical separation. There were days when it rained in Ohio and rendered a multi-million dollar microwave line useless. Firms which built their lines on a slightly different path could be unaffected, and thus enjoy outsized gains from increased market share.
Moreover, a quant's choice of technology has a direct impact on their strategy's implementation. This includes mathematical models. For example, a lot of statistical arbitrage traders are fading relative value relationships between different assets. All of these strategies accept a certain amount of risk, but if enough other traders are fading the same spread, the relationship can become a self-reinforcing blow up.
Consider the situation where many traders are short the 10-year and long the 30-year US treasury futures; this often occurs after large moves in a spread. Market makers become increasingly loaded up in one direction by definition: they provide liquidity in the spread at all times. As such they get selected by the market, so to speak, and at some point they have to cut risk to trade another day.
As they exit their positions it pushes the spread further out of equilibrium, further exacerbating the pain of trader's who have yet to liquidate their positions. This can snowball, creating a liquidity hole on the exit side of the trade.
In a world where automation becomes imperative to survival, technological choices have a profound influence on the correlation of overall strategies in the market. When we cluster around certain technologies we can experience a clustering of performance as well.
Interesting article! Although I struggled to understand certain parts, I learned a lot about HFT. It is a huge part of the financial industry now, and I think it is important I learn a little bit about it. I will be entering the finance industry in 4-5 years, and part of me fears that there will be no jobs for me after college because computers completely take over! @marketstack what do you think of the future job market? Will computers/AI take over most jobs in the next 4-5 years?
it's nothing to be afraid of... you can build your own trading AI and trade with your own money... make those hedge funds irrelevant to your career
agreed! thats the really cool thing about trading these days... you really dont need a firm to back you if you've got a little cash and an edge
nice! upvote this one. i suggest you start with Python, and install Anaconda for it. You can view my post if you don't know how. Then you learn scikit learn and dive into decision tree of it. Search decision tree on YouTube, and you'll find tons of tutorials. I learned it from a lot of people, and I like Sentdex's one. If you want a simplified version, I'm planning on writing one, but I'm still wrecking my brain about it
XD
only super rich people who own hundreds of multi-trillion dollar hedge funds can afford HFT. those people's worlds are far too hard for us to understand... LOL
For ordinary people like us, algo trading will suffice...
it would be interesting to price out what would be the bare minimum one would have to spend to set up a small HFT server in a colo for a year... another post perhaps... five figures for comp (w fpga networking modifications), few grand a month for data center... market data... it will def eat a six figure hole in your ass and thats before you've done any trading. and of course thats for just one computer in one market lmao. if you need a microwave line now we're getting costly.
a trillion might be a bit overkill LOL. however, for single market HFT, its definitely something that newly minted crypto millionaires could start to think about however, not that i'm advocating anybody to do anything with that statement, just to illustrate scale (disclaimer)
Wow... a very sophisticated article!!
I don't know how you write these articles.. amazing!
You must be a professional writer.
Thank you for voting on my postings!!
I really appreciated.
Upvoted & resteemed!
thanks! i certainly appreciate the sentiment, i have been paid to write in the past but the majority of my work has been for free on my old wordpress blog although it has been picked up on abnornmal returns, zerohedge, and such on occassion
Thank you for sharing. I read with fascination about HFT. I have always been fascinated but always wondered how I could get my hands on this HFT thingi for a reasonable fee but being a small time trader, I never had the access nor the capital to pursue this initiative.
They are more profitable traders because they employ powerful computers in transacting a large number of orders at a very fast speeds. They used complex algorithms to analyze multiple market and executivecirders based on market conditions.
Thanks @marketstack for sharing this wonderful topic
this post has been upvoted and resteemed, to my community members
thanks!
Wow, didn't know that
Thank you for sharing. It is very difficult to master new technologies.
Thank you for the interesting information. It is very difficult to master new technologies. Technologies are constantly changing. And you should always be aware of the changes.
The technology used also depends on the asset class that HFT algos are being used to transact.