Deep Learning In Finance: Zero Sum Games and Arbitrage

Deep learning in finance has always been a bit like a game of cat and mouse. On one side, you have the traders and investors trying to outmaneuver each other. On the other side, you have the machine learning algorithms, constantly scouring the data for opportunities to exploit and eke out even the slightest edge.

It's a fascinating idea to think about the prospect of an algorithm that given enough data can predict price or volatility based on patterns that humans can't see.

This is what it looks like to try to build an AI powered trading bot.

1. Feature extraction: One of the first steps in using DNNs for financial data analysis is to extract relevant features from the data. This might involve using techniques such as principal component analysis (PCA) or independent component analysis (ICA) to reduce the dimensionality of the data and identify the most important features for predicting arbitrage opportunities.
2. Model training: Once the features have been extracted, the next step is to train a DNN model on the data. This might involve using techniques such as backpropagation and gradient descent to optimize the model's weights and biases, as well as techniques such as regularization and early stopping to prevent over-fitting.
3. Model evaluation: After the model has been trained, it's important to evaluate its performance to ensure that it is accurate and reliable. This might involve using techniques such as cross-validation and hyperparameter tuning to optimize the model's performance, as well as techniques such as testing and validation to ensure that the model generalizes well to new data.
4. Model deployment: Once the model has been trained and evaluated, it can be deployed in a live trading environment to search for arbitrage opportunities in real-time. This might involve integrating the model into a trading platform or other financial software, as well as setting up automatic trades or alerts based on the predictions made by the model.

So you've done all these steps and now you have a profitable AI bot?

Unfortunately no.

The downside to a zero-sum game like high-frequency trading is that, any edge that you gain is effectively an edge that you are taking away from someone else. In addition to that financial markets are a level two complex system, as opposed to a level one complex system like the weather.

Allow me to explain: The weather (although admittedly insanely complex) is theoretically predictable if we knew the position and velocity of every particle on earth. If we had perfect knowledge of how it worked, it wouldn't change our model for the weather. However, this is not the case in finance. Knowledge of how the system works is itself an input into the system. i.e. If knowledge is derived of the perfect trading strategy and people use this then the strategy will soon be no longer effective because execution of the knowledge itself changes the dynamics of the market.

This is fundamentally the zero-sum information arbitrage game that is being played out in trading. Which, is why historically anything that contributes to a small edge is kept as secret as possible - especially data, which is often the limiting reagent for wining formulas.

On a side note, I think that the work that Numerai is doing to democratize data and to align incentives to change the zero-sum dynamics is fascinating.

Of course, this doesn't mean that the game of financial cat and mouse has come to an end. Far from it. The markets are constantly evolving, and there will always be those looking to gain an advantage over their competitors. However, I find the hard problems in finance intriguing, and potentially one of the areas where AI's edge will always be cancelled out (or reduced in effectiveness) by other competing AIs.