The modern financial industry has been required to deal with large and diverse portfolios in a variety of asset classes often with limited market data available. Financial Signal Processing and Machine Learning unifies a number of recent advances made in signal processing and machine learning for the design and management of investment portfolios and financial engineering. This book bridges the gap between these disciplines, offering the latest information on key topics including characterizing statistical dependence and correlation in high dimensions, constructing effective and robust risk measures, and their use in portfolio optimization and rebalancing. The book focuses on signal processing approaches to model return, momentum, and mean reversion, addressing theoretical and implementation aspects. It highlights the connections between portfolio theory, sparse learning and compressed sensing, sparse eigen-portfolios, robust optimization, non-Gaussian data-driven risk measures, graphical models, causal analysis through temporal-causal modeling, and large-scale copula-based approaches.
* Highlights signal processing and machine learning as key approaches to quantitative finance.
* Offers advanced mathematical tools for high-dimensional portfolio construction, monitoring, and post-trade analysis problems.
* Presents portfolio theory, sparse learning and compressed sensing, sparsity methods for investment portfolios. including eigen-portfolios, model return, momentum, mean reversion and non-Gaussian data-driven risk measures with real-world applications of these techniques.
* Includes contributions from leading researchers and practitioners in both the signal and information processing communities, and the quantitative finance community.
Ali N. Akansu, Electrical and Computer Engineering Department, New Jersey Institute of Technology (NJIT), USA
Dr. Akansu is a Professor of Electrical and Computer Engineering at NJIT, USA. Prof. Akansu was VP R&D at IDT Corporation and the founding President and CEO of PixWave, Inc. He has sat on the board of an investment fund and has been an academic visitor at David Sarnoff Research Center, IBM T.J. Watson Research Center, and GEC-Marconi Electronic Systems.Prof. Akansu was a Visiting Professor at Courant Institute of Mathematical Sciences of New York University performing research on Quantitative Finance. He is a Fellow of the IEEE and was the Lead Guest Editor of the recent special issue of IEEE Journal of Selected Topics in Signal Processing on Signal Processing Methods in Finance and Electronic Trading.
Sanjeev R. Kulkarni, Department of Electrical Engineering, Princeton University, USA
Dr. Kulkarni is currently Professor of Electrical Engineering at Princeton University, and Director of Princeton's Keller Center. He is an affiliated faculty member of the Department of Operations Research and Financial Engineering and the Department of Philosophy, and has taught a broad range of courses across a number of departments (Electrical Engineering, Computer Science, Philosophy, and Operations Research & Financial Engineering). He has received 7 E-Council Excellence in Teaching Awards. He spent 1998 with Susquehanna International Group and was a regular consultant there from 1997 to 2001, working on statistical arbitrage and analysis of firm-wide stock trading. Prof. Kulkarni is a Fellow of the IEEE.
Dmitry Malioutev, IBM Research, USA
Dr. Dmitry Malioutov is a research staff member in the machine learning group of the Cognitive Algorithms department at IBM Research. Dmitry received the Ph.D. and the S.M. degrees in Electrical Engineering and Computer Science from MIT where he was part of the Laboratory for Information and Decision Systems. Prior to joining IBM, Dmitry had spent several years as an applied researcher in high-frequency trading in DRW Trading, Chicago, and as a postdoctoral researcher in Microsoft Research, Cambridge, UK. His research interests include interpretable machine learning; sparse signal representation; inference and learning in graphical models, message passing algorithms; Statistical risk modeling, robust covariance estimation; portfolio optimization. Dr. Malioutov received the 2010 IEEE Signal Processing Society best 5-year paper award, and a 2006 IEEE ICASSP student paper award, and the MIT Presidential fellowship. Dr. Malioutov serves on the IEEE-SPS machine learning for signal processing technical committee, and is an associate editor of the IEEE Transactions on Signal Processing, and a guest editor of the IEEE Journal on Selected Topics in Signal Processing.
Financial Signal Processing and Machine Learning
Ali N. Akansu1, Sanjeev R. Kulkarni2 and Dmitry Malioutov3
1New Jersey Institute of Technology, USA
2Princeton University, USA
3IBM T.J. Watson Research Center, USA
In the last decade, we have seen dramatic growth in applications for signal-processing and machine-learning techniques in many enterprise and industrial settings. Advertising, real estate, healthcare, e-commerce, and many other industries have been radically transformed by new processes and practices relying on collecting and analyzing data about operations, customers, competitors, new opportunities, and other aspects of business. The financial industry has been one of the early adopters, with a long history of applying sophisticated methods and models to analyze relevant data and make intelligent decisions - ranging from the quadratic programming formulation in Markowitz portfolio selection (Markowitz, 1952), factor analysis for equity modeling (Fama and French, 1993), stochastic differential equations for option pricing (Black and Scholes, 1973), stochastic volatility models in risk management (Engle, 1982; Hull and White, 1987), reinforcement learning for optimal trade execution (Bertsimas and Lo, 1998), and many other examples. While there is a great deal of overlap among techniques in machine learning, signal processing and financial econometrics, historically, there has been rather limited awareness and slow permeation of new ideas among these areas of research. For example, the ideas of stochastic volatility and copula modeling, which are quite central in financial econometrics, are less known in the signal-processing literature, and the concepts of sparse modeling and optimization that have had a transformative impact on signal processing and statistics have only started to propagate slowly into financial applications. The aim of this book is to raise awareness of possible synergies and interactions among these disciplines, present some recent developments in signal processing and machine learning with applications in finance, and also facilitate interested experts in signal processing to learn more about applications and tools that have been developed and widely used by the financial community.
We start this chapter with a brief summary of basic concepts in finance and risk management that appear throughout the rest of the book. We present the underlying technical themes, including sparse learning, convex optimization, and non-Gaussian modeling, followed by brief overviews of the chapters in the book. Finally, we mention a number of highly relevant topics that have not been included in the volume due to lack of space.
1.2 A Bird's-Eye View of Finance
The financial ecosystem and markets have been transformed with the advent of new technologies where almost any financial product can be traded in the globally interconnected cyberspace of financial exchanges by anyone, anywhere, and anytime. This systemic change has placed real-time data acquisition and handling, low-latency communications technologies and services, and high-performance processing and automated decision making at the core of such complex systems. The industry has already coined the term big data finance, and it is interesting to see that technology is leading the financial industry as it has been in other sectors like e-commerce, internet multimedia, and wireless communications. In contrast, the knowledge base and exposure of the engineering community to the financial sector and its relevant activity have been quite limited. Recently, there have been an increasing number of publications by the engineering community in the finance literature, including A Primer for Financial Engineering (Akansu and Torun, 2015) and research contributions like Akansu et al., (2012) and Pollak et al., (2011). This volume facilitates that trend, and it is composed of chapter contributions on selected topics written by prominent researchers in quantitative finance and financial engineering.
We start by sketching a very broad-stroke view of the field of finance, its objectives, and its participants to put the chapters into context for readers with engineering expertise. Finance broadly deals with all aspects of money management, including borrowing and lending, transfer of money across continents, investment and price discovery, and asset and liability management by governments, corporations, and individuals. We focus specifically on trading where the main participants may be roughly classified into hedgers, investors, speculators, and market makers (and other intermediaries). Despite their different goals, all participants try to balance the two basic objectives in trading: to maximize future expected rewards (returns) and to minimize the risk of potential losses.
Naturally, one desires to buy a product cheap and sell it at a higher price in order to achieve the ultimate goal of profiting from this trading activity. Therefore, the expected return of an investment over any holding time (horizon) is one of the two fundamental performance metrics of a trade. The complementary metric is its variation, often measured as the standard deviation over a time window, and called investment risk or market risk.1 Return and risk are two typically conflicting but interwoven measures, and risk-normalized return (Sharpe ratio) finds its common use in many areas of finance. Portfolio optimization involves balancing risk and reward to achieve investment objectives by optimally combining multiple financial instruments into a portfolio. The critical ingredient in forming portfolios is to characterize the statistical dependence between prices of various financial instruments in the portfolio. The celebrated Markowitz portfolio formulation (Markowitz, 1952) was the first principled mathematical framework to balance risk and reward based on the covariance matrix (also known as the variance-covariance or VCV matrix in finance) of returns (or log-returns) of financial instruments as a measure of statistical dependence. Portfolio management is a rich and active field, and many other formulations have been proposed, including risk parity portfolios (Roncalli, 2013), Black-Litterman portfolios (Black and Litterman, 1992), log-optimal portfolios (Cover and Ordentlich, 1996), and conditional value at risk (cVaR) and coherent risk measures for portfolios (Rockafellar and Uryasev, 2000) that address various aspects ranging from the difficulty of estimating the risk and return for large portfolios to the non-Gaussian nature of financial time series, and to more complex utility functions of investors.
The recognition of a price inefficiency is one of the crucial pieces of information to trade that product. If the price is deemed to be low based on some analysis (e.g. fundamental or statistical), an investor would like to buy it with the expectation that the price will go up in time. Similarly, one would shortsell it (borrow the product from a lender with some fee and sell it at the current market price) when its price is forecast to be higher than what it should be. Then, the investor would later buy to cover it (buy from the market and return the borrowed product back to the lender) when the price goes down. This set of transactions is the building block of any sophisticated financial trading activity. The main challenge is to identify price inefficiencies, also called alpha of a product, and swiftly act upon it for the purpose of making a profit from the trade. The efficient market hypothesis (EMH) stipulates that the market instantaneously aggregates and reflects all of the relevant information to price various securities; hence, it is impossible to beat the market. However, violations of the EMH assumptions abound: unequal availability of information, access to high-speed infrastructure, and various frictions and regulations in the market have fostered a vast and thriving trading industry.
Fundamental investors find alpha (i.e., predict the expected return) based on their knowledge of enterprise strategy, competitive advantage, aptitude of its leadership, economic and political developments, and future outlook. Traders often find inefficiencies that arise due to the complexity of market operations. Inefficiencies come from various sources such as market regulations, complexity of exchange operations, varying latency, private sources of information, and complex statistical considerations. An arbitrage is a typically short-lived market anomaly where the same financial instrument can be bought at one venue (exchange) for a lower price than it can be simultaneously sold at another venue. Relative value strategies recognize that similar instruments can exhibit significant (unjustified) price differences. Statistical trading strategies, including statistical arbitrage, find patterns and correlations in historical trading data using machine-learning methods and tools like factor models, and attempt to exploit them hoping that these relations will persist in the future. Some market inefficiencies arise due to unequal access to information, or the speed of dissemination of this information. The various sources of market inefficiencies give rise to trading strategies at different frequencies, from high-frequency traders who hold their positions on the order of milliseconds, to midfrequency trading that ranges from intraday (holding no overnight position) to a span of a few days, and to long-term trading ranging from a few weeks to years. High-frequency trading requires state-of-the-art computing, network communications, and trading infrastructure: a...