Even if this product allowed you to have picosecond processing of market data and generation of trading signals, if you don't have exchange co-location, inherent network latency in your market data from providers and order routing to exchanges will make this thing pretty useless. For a hobbyist, throw leased lines out the door, now a Linode box, even a desktop running Windows XP with full-blown Nagle's algorithm, would have good-enough performance unless you can pour serious capital into a project.
They're also very vague on the software architecture or framework "kit". They don't say which feeds they support or if they provide any domain-specific data structures/standard trading algorithms like VWAP (slicing an order up over the day to give you "better" fills). I do see mention of an optional Feed Parser reference design, which I can't ascertain much value from. I am guessing it is mostly a generic kit for writing on their specialized hardware, and not specific to trading. There will be a lot of pain, maybe impossible to take vendor or open source libraries for market feed handling and compile it down using their embedded libraries. You'll be stuck doing tons of TCP/IP work, fun for hacking, but relative to the latency issues in your market data/order routing, it won't help you profit. Most people buy these appliances with included feed support.
With a big budget and co-location, this may be useful in bootstrapping some really low-level processing of market data, although there are a variety of hardware and software messaging solutions that would be a good alternative in driving latency lower if you have some money, Solace (HW), Tibco, 29West, etc to name a few.
I wish there was a hobbyist solution out there, but sorry, my initial impression says this isn't it. I would like more information, but maybe there are better hobbyist efforts out there. Interactive Brokers API isn't bad, and here's a full-blown real open-source trading kit that has a similar business model to MySQL. http://www.marketcetera.com/ Note: I feel that there are a few issues with this as well in some of the technologies they chose (JMS, Eclipse RCP).
Despite all of this, I find the most worrying thing about the recent HFT developments is the simple fact that hobbyists want to get involved. Surely turning amateurs loose in such a necessarily complex world will result in large sums of money being lost.
This is definitely not a solution intended for hobbyists. This is a solution for big HFT players that have optimized already all their trading components, from exchange proximity to fine grained application optimization.(and everything in between).
The competition is leveling out because most players have gotten the basic tricks; this is just the next logical step in evolution. It has been proved in practice that fpga based trading systems consistently edge out normal cpu based ones(atm anyway). Most likely, in the future, companies will be forced to move to such a solution or get left behind.
In all seriousness, I would rather see a market designed explicitly around the barring of high frequency trades. All trades taking a fixed and relatively long period (say, 30 minutes) to complete.
This sort of investment strategy doesn't seem to be creating long-term market value, it's exploiting short-term edge effects in a zero-sum strategy. The arms race to pursue ever more inhumanly fast automatic trades seems to have no net value at all yet is consuming resources to impoverish the algorithmically and connectively poorer at the expense of the richer.
Markets seem to be favoured because they allow prices to find their natural level from supply and demand. How is this facility helped by permitting trades to be completed and then reversed faster than any human can ever perceive? This is algorithmic gambling on edge effects far more than a true market.
I can see from your post that you're very misinformed about how markets operate and the role of high frequency traders (a.k.a. liquidity providers). Read my response to a previous comment: http://news.ycombinator.com/item?id=2164458
I have nothing against HFT but I don't buy your liquidity argument. Liquidity is needed most when markets are falling outside the norm. Any algorithm with a fail-safe or kill switch will immediately shut off when times get bad thereby ending their contribution to liquidity at a time when it is needed most.
Please help me understand this better if I'm incorrect.
"Liquidity is needed most when markets are falling outside the norm"
True but humans have fail-safes and kill switches too, thus this is not a valid argument against HFT in favor of human market makers. At any time, humans could step in and provide liquidity during the flash crash and during other crashes.
The main point that you're missing is that there is a huge demand for liquidity during a majority of market hours. On average, 7 billion shares of US equities are traded every day. That is a monumentally large number if you think about it. According to the TABB consulting group, roughly 50-70% of American stock trades are done by HFT [1]. Let's assume that 10-20% of trading is non-profit motivated (utilitarian, you could say). That means that roughly 1 billion shares are exchanged every day by utilitarian traders. That in itself should be clear evidence that there is a large demand for liquidity.
HFT market makers play an important role in those transactions. Specifically, they make it cheaper to buy and sell stocks by (1) tightening the bid-ask spread and (2) providing more quantity at each price, so that the average cost of executing an order is less. Not only that, when markets become tighter, they actually enable transactions to occur that would not have happened before. In other words, previously where buyers and sellers would NOT have traded because the transaction costs of crossing the bid-ask spread were too high, those two parties can now trade. Specifically, without HFT, there would be far fewer than 1 billion shares traded by utilitarian traders on a daily basis.
To address the other point of providing liquidity "when the markets need it most", let's take a step back. When you say that "market makers should step in to provide liquidity [for society's benefit]", you're implying that there's some externality to lack of liquidity in financial markets (if so, this is yet another reason that we need HFT on a daily basis). Suppose that this is the case: there is some negative externality to society when markets are illiquid, as is oft to happen when things go crazy in the world. During those times, volatility is insanely high because the risk of being in any position is also insanely high. Remember, market makers get compensated (on average) for holding risk that you don't want. If risk is higher, naturally, the compensation should be also. This is manifested in higher costs of execution: spreads widen and the available quantity at each level decreases.
If you want to force HFT market makers, which are private corporations, to step in to provide more liquidity, then you are forcing these companies to pay for that externality. In effect, they would take on huge risk for far diminished expected returns. That doesn't make any particular sense to me. However, if society as a whole has this view that some private corporations need to pay for public externalities, then that should be a matter of regulation. But if that's the case, why pick on HFT in particular? Why not force McDonalds and Whole Foods to give food to hungry people during famines? Surely, there is an externality to the food industry NOT stepping in during periods of extended hunger, "just when people need it the most."
I'm still in the "learning the syntax" stage of finance so this is great information. thank you. do you think that one reason why people say bad things about HFT is because they see them being unfairly compensated for taking on what they perceive is a small amount of risk? as in - "come on, how hard can it be to grab a block of stock on an upward trend and quickly sell it." Of course, the more I dig into this, the more I realize how hard it is to find those trends and how easy it is to lose your ass. But, people don't dig in themselves.
do you think that one reason why people say bad things about HFT is because they see them being unfairly compensated for taking on what they perceive is a small amount of risk?
I think that's definitely part of it. The issue is that people don't know how much high frequency traders actually make per contract. The amount is pretty small (on average). Typically, each trade takes on "bite-sized" risk and expects "bite-sized" returns. Do that thousands of times a day, and your bite-sized returns grow to something reasonable, but not ridiculous. Certainly, it's not on the order of how much other forms of proprietary trading make.
I think the other issue is that people don't understand that all traders, HFT or not, take on risk when they enter a trade. A lot of people use terms like "skimming off the top" or "making money for nothing". This indicates to me that they don't realize why market makers should even make money (on average). It's simply risk transfer, and that has a price (on average).
You're right though, identifying bona fide trends are pretty difficult. If it were easy, everyone would be doing it and arbing those trends away.
They're just piggybacking trend and making profit of it, that how there used. Maybe not why and how they were designed. Still, IMHO, it is not fair and goes against the very definition of stock exchange.
High frequency traders are simply are able to provide the most competitive price in the market. They usually do this with a strong hunch that they will be able to get out of that position in a short period of time at a better price. Are they providing a service? Absolutely. When HF traders are involved in a transaction, risk is transferred to them. No matter how short the interval is that they're trading on, they're still taking on risk. The party trading against them doesn't have to worry about what way the market is going to move after the trade is complete and they are willing to pay for that.
Consider a hypothetical stock currently trading at 34.05 bid, 34.06 ask. Trader A puts in a bid for 100 @ 34.06, matching Trader B's ask of 34.06. Normally, B would then sell to A, but thanks to HFT, institute C has the unique and intrinsically unfair position of being able to fulfill A's order. Assuming a moderate holding of stock, you are then able to manipulate the market up and down second-to-second by selectively fulfilling buy or sell orders and so causing supply or demand to pile up temporarily.
In a zero sum game where knowledge is power, you know exactly who is buying and who is selling - before John Q. Public knows. To compound this, you can fulfill orders using immediate or cancel (IOC) orders before they hit the markets - making it impossible for others to know how much or even whether you are buying/selling.[1]
In itself, there is really not much wrong with this - market makers have been doing this for decades, possibly even centuries. And as long as there is a level playing, where everyone sees orders at the same time, there really is no problem - an invaluable service in the form of liquidity is provided.
Having the advantage of being able to front run the markets is not limited to just market making, however. Limits are one of the ubiquitous features of stock markets - the ability to say, I will buy or sell X shares of this stick up until it hits price Y.
This can be exploited easily by HFT/front running - you have 30 ms [2] to probe out where the limit is with small IOC transactions. In this way, you are able to meet orders at prices that are different than the currently trading price - effectively bypassing all competition.
Just to respond to one of your statements from the linked post:
I am interested in any civil responses to this. I am not professing to be any sort of expert on HFT. [3]
Normally I would not bother responding to posts like this, preferring to avoid talking to anyone, but your slightly condescending tone was infuriating.
1. This may not be correct, in any case, the IOC would only be of the required size and so not provide any indication of holding size.
3. I have used HFT and front running as synonyms because they seem to be inescapably tied together. Why the need to have picosecond advantages if you aren't trying to beat another bottom feeder to the prey?
edit: bah, I really dislike how HN treats newlines - I can't find any way to simply start a new line without a space in between the two sentences. Comes out like a block of text at 1366x768, or ends up looking too spread out.
Consider a hypothetical stock currently trading at 34.05 bid, 34.06 ask. Trader A puts in a bid for 100 @ 34.06, matching Trader B's ask of 34.06. Normally, B would then sell to A, but thanks to HFT, institute C has the unique and intrinsically unfair position of being able to fulfill A's order.
Explain? Who is institution C? How are they able to violate time priority?
Assuming a moderate holding of stock, you are then able to manipulate the market up and down second-to-second by selectively fulfilling buy or sell orders and so causing supply or demand to pile up temporarily.
On some stock exchanges, for example NASDAQ, you can pay a fee to be allowed access to market orders earlier than anyone else. Paired with colo services (literally your servers are next door - nearly eliminating network latency) you have a massive time advantage.
Note that the SEC has not banned flash trading; it only proposed to do so - as of so far they have not done anything.
The second point - it is the core of how market makers work to provide liquidity, with the added advantage of being able to see orders first. Simple supply/demand dictates that selling stock will drive the market down, since existing sell orders are not filled and slowly pile up at lower and lower prices. Then beginning to buy shares will raise the stock price temporarily, at which point the cycle repeats. Being able to figure out the limits on the buy/sell orders is simply icing.
---
It appears that NASDAQ probably discontinued market front running voluntarily in late '09; I wasn't aware of that. I will have to rethink my positions - HFT as such isn't wrong in my opinion, although being able to make a profit 97% of the time (and having 3 straight quarters of only profitable days) is ridiculous, as is profiting purely from having a connection that is faster than what everyone else has.
You're point that market makers provide liquidity by flash orders is entirely false. You clearly have no understanding of how market makers actually go about their day to day business. Please don't pontificate as if you do, or if you plan to, acknowledge that you're only speculating based on marginal facts and poorly written articles (yes, even the NY Times has horribly inaccurate articles) you've gathered on the internet.
In fact, that entire paragraph is pure specious reasoning. The main point that really drove it home for me was:
Being able to figure out the limits on the buy/sell orders is simply icing.
I can gather you have not actually tried to find a statistical trend in market prices. Simply put, it's difficult to do in practice.
rgarcia already pointed out that your statement about institute C violating price-time priority is false. You then make a defamatory remark against the industry with zero evidence. Market manipulation is illegal. No respectable trading firm does it. Why? It is unethical and illegal.
In a zero sum game where knowledge is power, you know exactly who is buying and who is selling - before John Q. Public knows.
Since when was it the prerogative of every market participant to get quotes at the exact same time? There is a huge cost to colocation--it doesn't come free. The fact that trading has as high barrier to entry is a moot point. Starting a biotech company takes hundreds of millions of dollars in capital. Is it unfair to John Q. Public that Amgen and Novartis have competitive advantages in that field? Of course. Is it unethical? Absolutely not. When the steam engine was invented, was it unfair for guys who drove horse buggies? Of course. The new technology replaced him because it was able to do what he did, but more efficiently. This is no different. Day traders (I'm assuming that's what you're referring to when yous ay John Q. Public) are horse and buggy drivers. It isn't the fault of high frequency traders that they are that way. If John Q. Public wants to stay competitive, he has to change what he's doing.
If, instead, you're talking about your pension fund buying 1 million shares of Microsoft, then you have to realize that HFT is actually decreasing the cost of their doing that. Tighter spreads and more liquidity lower transaction costs for everybody.
I find it ridiculous that the article you cite makes no mention of HFT! In fact, I'm pretty sure that those profits are from their prop trading desks that don't do HFT. I can tell you that with certainty, because I know the ballpark of how much HFT firms make, and it's nowhere near that number. Please don't cite facts that are irrelevant to your point and then carry on as if they prove your case. I perused the article, did you even read it?
Having the advantage of being able to front run the markets is not limited to just market making, however. Limits are one of the ubiquitous features of stock markets - the ability to say, I will buy or sell X shares of this stick up until it hits price Y. This can be exploited easily by HFT/front running - you have 30 ms [2] to probe out where the limit is with small IOC transactions. In this way, you are able to meet orders at prices that are different than the currently trading price - effectively bypassing all competition.
Flash orders, if they're even used by anybody anymore, must account for some negligible fraction of market volume.
3. I have used HFT and front running as synonyms because they seem to be inescapably tied together. Why the need to have picosecond advantages if you aren't trying to beat another bottom feeder to the prey?
Market makers need high speed mostly to avoid getting picked off by other aggressive HFT firms. The prey isn't day traders, it's other HFT firms. Day traders don't operate on the same timescales as HFT. Just the fact that a day trader is using a mouse to enter and exit positions already precludes that possibility. If day traders are placing limit orders into the market and are upset that they can't cancel fast enough, they need to understand that their edge isn't in cancelling quickly, but purportedly something else (predicting longer term price movement?).
I'm all for a civil debate, but please checks your facts. Most of the sources you cite are irrelevant to this discussion :(
Virtually no hobbyist would be able to compete against big financial firms' hardware and algorithms. It might be a smart move for a large high frequency trading firm to put out these sorts of tools for free. An incredibly easy to use free api with good (though not superior) speed would be pretty popular. More subpar competition == more money for them. Though, admittedly, probably nowhere near what they are making now.
Most solutions of this type are an order of magnitude higher than that. It's very easy to spend $20k/mo.
To give an idea of scale / endpoint, a top-tier HFT infrastructure costs on the order of $100k/day. There are probably only a half dozen total of those in the world, though.
Even with this solution and well-designed code, a trader needs external infrastructure that plays on the same low-latency level. Most HFT strategies aren't as secret as you think; there are usually dozens of shops running similar trades. For that reason, something like this needs to be jacked directly into an exchange co-lo or connected through dedicated fiber. In short, the cost of this blade is just the start if you want to play with the gunslingers. Somebody else mentioned that there are prop shops that will give you the infrastructure you need for a cut of your profits. Some will even front the cash you need to trade. One such place is HTG Capital Partners of Chicago, which is where much of the HFT world is centered.
So, if we limit our consideration of HFT to strategies which rely primarily upon speed-based exploits (trade decisions based on the examination of current market data w/o correlation to external data sources like, say, Twitter feeds for market "sentiment"), can we conclude from the trading stack proposed for this box that most arbitrage opportunities have been exhausted by fierce competition? If the current state of the art is measured in microseconds, how much more room is there for improvement? I understand that there's a whole world of data streams from which one might attempt to predict markets, and I am explicitly limiting my question to the genre of trading which seems to be profitable via latency reduction alone.
I am waiting for one of these shops to open up the opportunity to hobbyists/devs. I would love to try and create my own decision/trade algorithms but dont have the kind of money it would take to mitigate the cost effect of the trade volume (ie, at $8 per trade and making 1000 trades per day with an average return of 2% per day you would need to be working with more than 400K to even cover your tx cost).
Strikes me that it would be a great research incubator for them to understand/learn new ideas and techniques and hunt for talent.
There are plenty of prop shops that would let you trade with a very modest (low 5-figure or even 4-figure) dollar amount. Some will leverage your money, some will flat-out stake you in addition to your input. These places have the benefit of offering you relatively low costs -- minuscule compared to the cost of a retail trade at your average discount broker -- but they do not necessarily have the sort of investment in supercharged hardware.
It is a good thing that your actual trading costs are not anywhere near those at the retail brokers you have in mind. 2%/day is an outrageously huge return. $1,000 at 2% compounded 250 times (approximate number of trading days in a earth year) is over $140k; you would return 14,000% annually.
The bottom line is that it is harder than it looks.
Set it up, trade 'virtually' (i.e., just simulate), prove your algos, get bankrolled by someone who is convinced of the algos worth. Quite straightforward. I'm highly skeptical of a hobbyist beating the competition though (maybe in micro niches, where one would add domain specific knowledge to get an edge - that's my pet idea on this topic :) )
It is straightforward, but impractical. Getting good historical market data is very, very expensive. Even recording current market data is not cheap and not without significant difficulty.
For a non-pro setup you're going to be accepting significant scope limitations in production - you're not going to be scanning the full real time feed across the whole market. Without colocation HFT isn't really an option either. So you can restrict your strategies to medium frequency against a small universe of assets which makes the technical side easier. Also the data side you won't need a full set of data, e.g. tickdata.com has data at $20/symbol/year (https://store.tickdata.com/prices?productMarketId=PRODUCT_MA...)
True, but it's not impossible to get data sets to work from - on all quant forums the question comes up at least daily and the answers are always the same.
Why would one even consider entering these markets without the ability to demonstrate alpha over historical results?
Yes, your alpha needs to include the ability to beat transaction costs... so if you can't pay for the [relatively exceptionally cheap] cost of several years of your chosen market's historical data, maybe you're not really serious.
It's fun to play at being a financier, but let's not pretend that access to market data is the significant barrier to entry which is preventing disruption in the world financial markets.
The cost of market data and a system equipped to process it are not inconsiderable.
Historical tick data (quotes and trades) for US equities will run you $20k/year, minimum. US equities runs about 30-40GB/day so you are looking at almost 10TB/year. You need to be able to access the data quickly so you probably don't want to compress it and probably do want to duplicate the data, slicing and dicing it in different ways. Call it 20TB, minimum, per year of data.
Futures and options data will run another $20k/year, each, and futures has about as much data as equities and options has an order of magnitude more. That is another 100TB/year, minimum, per year, probably closer to 200TB.
How are going to to study these data? Even if you were able to operate at disk speed (which seems unlikely unless you have lots of disks and have duplicated and distributed your data amongst them) it would take you five minutes to run against one day's worth of data, or close to a day to run a year's simulation. If you consider more than a few cores in parallel, reading from disk is going to be seriously hampered.
None of this even considers the cost of the software necessary to support this effort: contending with corporate actions, tracking halts, etc.
This all adds up to a significant financial hurdle.
Surely though, one could pre-calculate potential future positions or other useful data on stock in advance of next day's trade between close and re-open of the markets?
he cost of market data and a system equipped to process it are not inconsiderable.
Not even close, if you consider the alpha for generating value. I know it's awesome on HN to think that generating 10M over 2 years entitles one to a "50M exit."... but that's not the way things work in finance.
20K is nothing... if one thinks it is than one is playing the wrong game.
And to be perfectly clear. I will take all of your action, every day, in prediction of market movements.. You tell me what you want. I will set a price and we will play for as much money as you like [up to my assets of 20MM or so.] I have no fear that I will beat you [or any one else] before you get me broke.
It's not really that simple. You can't really ever simulate your trading with 100% confidence since you're never entirely sure what your participation in the market will do to the results you see when you /aren't/ participating.
There is also plenty of anecdotal evidence that as you try and increase the amounts of capital you're putting to work in the market that it becomes tougher and tougher to maintain an edge in the markets.
I wonder what exactly the FPGA does, and how they can make it practical and flexible. Of course they could always just test with C and then compile that to HDL.
A month or two ago I spoke to someone working in these shops. Apparently, they're using the FPGAs to run many of the high-bandwidth algorithms, because going from NIC to motherboard to CPU and back again added too much latency. So they program the algorithms into the FPGA (which has its own Ethernet port) so it can receive, process, and send the data without ever hitting circuitry not dedicated to the task.
This is true, but FPGAs can still only handle relatively simple algorithms. FPGAs make certain kinds of tasks nearly trivial (think truly parallel tasks such as decoding market data for different instruments), but there's just not enough gates (in most cases) to do interesting stuff in the application / logic layer.
FPGAs canonical use case is still as very fast codecs. There are exceptions, but they're really only the very simplest of arbitrage strategies. And in order to make those work you need top-of-the-line network gear and engineers as well.
This seems a bit surprising: what is the latency coming from the NIC to the motherboard ? I somehow doubt the latency due to distance within one piece of hardware matters (current CPU cannot do much more than a few 1000s cycles for the electricity to go around a few times in a motherboard, which is nothing to do any kind of advanced processing).
The problem is that a high-bandwidth link overruns the CPU with thousands of interrupts per second. It can paralyse the system, where the CPU is so busy handling I/O that it doesn't get around to actually scheduling processes.
Disclaimer: I sometimes consult for HFTs but we don't use FPGAs (yet).
So it is more an architecture issue with conventional hardware ? It makes much more sense to me with this explanation - the issue is the same as in most real-time systems then (like in embedded hardware for audio gears, which is more familiar to me than HFT).
Don't things like real time linux or soft real time patches for linux help, though ?
> Don't things like real time linux or soft real time patches for linux help, though ?
I don't see how it could.
The CPU essentially drops everything it's doing when a packet comes in. Have that happen often enough and there won't be time left to do anything else, unless you start dropping packets.
My guess, as is usually the case with these HFT technology vendors, is that this company is trying to sell the residual technology IP from a hedge fund that bellied up.
To those "hobbyists", what would you look for in a solution?
It looks like you got a few downvotes because people assumed you're being snarky (maybe you are?) -- but I think that most people (even hackers) who work outside of the financial sector are going to wonder why this is on the front page. In short though, it's a 1U blade server with a full-size FPGA in it; implementing your algorithms in hardware makes them extremely fast (thus the mention of the C-to-HDL compiler), which is obviously of critical importance in high-frequency trading.
I used to work in the FPGA industry, and we had quite a few people looking at trying to do hardware acceleration of stuff currently done on CPUs. In short, it is hard.
There are plenty of suppliers who offer stuff in the market, for example replacing one half of a dual CPU system with an FPGA and in general some variant of 'pick a fast interconnect and stick an FPGA at the end'
I can't see why this offers anything more to the discussion over a product announcement.
We've tried using OpenCL for pricing, the biggest bottle neck is getting data into and out of video card memory.
It's just too slow to move it from main to video memory and back.
Also OpenCL works best when working on large pieces of data at once, with High frequency trading often you want to work on very small pieces, 50 bytes, and process them as fast as possible. Batching up the data and working on it in a SIMD type fashion just takes too long:(
And that's exactly what the world needed. As if Wall Street was not enough to destroy the world. I also question the validity of posting this link to where entrepreneurs, a.k.a value creating people, frequent.
They're also very vague on the software architecture or framework "kit". They don't say which feeds they support or if they provide any domain-specific data structures/standard trading algorithms like VWAP (slicing an order up over the day to give you "better" fills). I do see mention of an optional Feed Parser reference design, which I can't ascertain much value from. I am guessing it is mostly a generic kit for writing on their specialized hardware, and not specific to trading. There will be a lot of pain, maybe impossible to take vendor or open source libraries for market feed handling and compile it down using their embedded libraries. You'll be stuck doing tons of TCP/IP work, fun for hacking, but relative to the latency issues in your market data/order routing, it won't help you profit. Most people buy these appliances with included feed support.
With a big budget and co-location, this may be useful in bootstrapping some really low-level processing of market data, although there are a variety of hardware and software messaging solutions that would be a good alternative in driving latency lower if you have some money, Solace (HW), Tibco, 29West, etc to name a few.
I wish there was a hobbyist solution out there, but sorry, my initial impression says this isn't it. I would like more information, but maybe there are better hobbyist efforts out there. Interactive Brokers API isn't bad, and here's a full-blown real open-source trading kit that has a similar business model to MySQL. http://www.marketcetera.com/ Note: I feel that there are a few issues with this as well in some of the technologies they chose (JMS, Eclipse RCP).