Estimating Large Noisy Matrices: Random Matrix Theory Meets Cross-Validation
Large sample covariance matrices (SCM) are not the best estimates of true covariances. Using tools from random matrix theory (RMT) and free probability one can compute the eigenvalue spectrum of the SCM from independent or even auto-correlated samples. One can also compute the optimal estimator of true covariance from sample data under a natural hypothesis of absence of prior knowledge about eigenvectors. This estimator can be expressed in RMT language but is easier to understand in the optimization/validation (O/V) framework used in machine learning. A recent leave-one-out algorithm is actually the best numerical implementation of the RMT optimal estimator.
Large sample covariance matrices (SCM) are not the best estimates of true covariances. Using tools from random matrix theory (RMT) and free probability one can compute the eigenvalue spectrum of the SCM from independent or even auto-correlated samples. One can also compute the optimal estimator of true covariance from sample data under a natural hypothesis of absence of prior knowledge about eigenvectors. This estimator can be expressed in RMT language but is easier to understand in the optimization/validation (O/V) framework used in machine learning. A recent leave-one-out algorithm is actually the best numerical implementation of the RMT optimal estimator.
Functions of the true covariance matrix (such as its inverse) can also be estimated using the same schemes. I speculate that the link between the RMT and O/V frameworks can help us distinguish between signal and noise in very complex noisy data sets such as neural recording data. References: Bun, Bouchaud and Potters, Physics Reports 666 (2017), forthcoming book by Potters and Bouchaud, Cambridge (2020).
Speaker Bio: Marc Potters is Chief Investment Officer of CFM, an investment firm based in Paris. Together with Jean-Philippe Bouchaud, he supervises the research team with particular focus on developing concrete applications in financial forecasting, portfolio construction, risk control and execution. Marc maintains strong links with academia and as an expert in Random Matrix Theory has taught at UCLA and Sorbonne University. Marc obtained his PhD in physics from Princeton University and joined CFM in October 1995 as a researcher in quantitative finance. Marc continues to publish papers in statistical finance with his research team and co-authored ‘Theory of Financial Risk and Derivative Pricing’ with Jean-Philippe.