Why would one even consider entering these markets without the ability to demonstrate alpha over historical results?
Yes, your alpha needs to include the ability to beat transaction costs... so if you can't pay for the [relatively exceptionally cheap] cost of several years of your chosen market's historical data, maybe you're not really serious.
It's fun to play at being a financier, but let's not pretend that access to market data is the significant barrier to entry which is preventing disruption in the world financial markets.
The cost of market data and a system equipped to process it are not inconsiderable.
Historical tick data (quotes and trades) for US equities will run you $20k/year, minimum. US equities runs about 30-40GB/day so you are looking at almost 10TB/year. You need to be able to access the data quickly so you probably don't want to compress it and probably do want to duplicate the data, slicing and dicing it in different ways. Call it 20TB, minimum, per year of data.
Futures and options data will run another $20k/year, each, and futures has about as much data as equities and options has an order of magnitude more. That is another 100TB/year, minimum, per year, probably closer to 200TB.
How are going to to study these data? Even if you were able to operate at disk speed (which seems unlikely unless you have lots of disks and have duplicated and distributed your data amongst them) it would take you five minutes to run against one day's worth of data, or close to a day to run a year's simulation. If you consider more than a few cores in parallel, reading from disk is going to be seriously hampered.
None of this even considers the cost of the software necessary to support this effort: contending with corporate actions, tracking halts, etc.
This all adds up to a significant financial hurdle.
Surely though, one could pre-calculate potential future positions or other useful data on stock in advance of next day's trade between close and re-open of the markets?
he cost of market data and a system equipped to process it are not inconsiderable.
Not even close, if you consider the alpha for generating value. I know it's awesome on HN to think that generating 10M over 2 years entitles one to a "50M exit."... but that's not the way things work in finance.
20K is nothing... if one thinks it is than one is playing the wrong game.
And to be perfectly clear. I will take all of your action, every day, in prediction of market movements.. You tell me what you want. I will set a price and we will play for as much money as you like [up to my assets of 20MM or so.] I have no fear that I will beat you [or any one else] before you get me broke.
Yes, your alpha needs to include the ability to beat transaction costs... so if you can't pay for the [relatively exceptionally cheap] cost of several years of your chosen market's historical data, maybe you're not really serious.
It's fun to play at being a financier, but let's not pretend that access to market data is the significant barrier to entry which is preventing disruption in the world financial markets.