Separating signal from noise with natural language processing techniques.
Investors know that even a small improvement in stock selection can have a material impact on portfolio performance, which is why we emphasize the importance of investing in equity research. Traditionally, quantitative equity researchers have used vast amounts of well-structured numerical data from company financials to look for key indicators that guide investment decisions. In addition to analyzing numerical data, fundamental analysts often attempt to improve predictions of company performance by analyzing the text contained in company filings. Indeed, interpreting management commentary to help predict future earnings is something fundamental analysts have done for decades. It is a task historically reserved for humans, who with experience learn to “read between the lines”. However, there are limits to this and fundamental analysts are often short of time during the reporting season to carefully read often lengthy reports littered with boilerplate language. This is difficult for even the most skilled analyst to do at scale…but not for machines.
Recent academic research suggests that predictions of company performance can be improved by using Natural Language Processing techniques on text contained in company filings. Li (Journal of Accounting Research, 2010) showed that companies expressing positive sentiment in their MD&A (management discussion & analysis) section tend to have continuing business momentum that will translate into rising share prices.1 Identifying positive sentiment within hundreds of stocks would be difficult and time consuming for a human analyst, but, as noted above, machines, using NLP technology, are able to analyze thousands of company filings to help improve forecasts.
But machines are not perfect, and academia is not reality
While the academic research suggests that this idea can be valuable, there can be a huge gap between what works in theory versus what works in practice. Oftentimes after research on return prediction factors is published, the signal gets arbitraged away and its efficacy declines. Rather than blindly follow academic research, we set out to test these for ourselves.
The first step in the process was to build a text database from public companies’ SEC 10-K filings from the past 10 years. All SEC 10-K filings contain two key sections: the MD&A and “Risk” sections; however, the filings often use inconsistent formats that make extraction of the right text extremely complex. The next step was to massage the text data into a form that can be used by the NLP algorithms, including removing special characters and truncating word variations into their root form (e.g., changing words like “investments,” “investable,” and “investing” to the root word “invest”). Finally, the right NLP algorithms were selected and applied to the text. For example, to gauge sentiment, we combined dictionary-based metrics with more advanced language models (e.g. FinBert) which were created using machine learning techniques.
The testing proved that the new NLP-based signals not only predicted subsequent company performance (Figure 1), but also provided new information that was not contained in the existing numeric-based quant signals. The text-based signal also could have helped navigate the current difficult market environment, as stocks with management who were bullish when speaking about last year’s operating performance outperformed those with management who were less positive. This corroborates how the COVID crisis, while unprecedented for around a century, did much to accelerate already existing trends favoring the most innovative companies.
Sources: Voya IM and Voya Enterprise Data Science Team analysis, Factset; hypothetical cumulative simulated performance of quintile portfolios of stocks ranked by sentiment of most recently available 10K filing (equal-weighted, monthly rebalanced), out of point-in-time Russell 3000 constituents. Benchmark is equal-weighted Russell 3000 constituents. Composite sentiment based on dictionary approach and BERT model. Past performance is not indicative of future returns. Investing inherently risks loss of capital. Performance reported gross of any fees or transaction costs, which would have reduced returns.
Improving our models is only the beginning...
Implementing a Natural Language Processing capability was a logical next step in our quest to use the latest data science techniques in support of our active investment strategies. It’s just a start in what is possible using Machine Intelligence to “read” and interpret textual sources with material company information, be it explicit or between the lines. Pending enhancements include red flags to spot deception, such as the use of overly complex language or the lack of hard numbers in financial disclosures, as well as language models specifically trained to predict developments such as improvement in a company’s ESG practices. It can easily move beyond company filings to spoken language, for instance during company earnings calls.
Text-based metrics can be used to help our human analysts in picking stocks, adding to the proverbial mosaic and possibly triggering a review of existing portfolio holdings when the filing indicates a change versus the original investment thesis. Moreover, we use Machine Intelligence to combine the resultant signals with factors computed from traditional and alternative structured data sources to run systematic investment strategies. As a result, a new set of intelligent insights, gleaned from unstructured text, is now built into Voya’s quantitative equity models to improve stock selection. However, the benefits of machine learning and NLP extend beyond helping our active fundamental analysts and improvements to our equity models. In our next blog, we will explore how the combined ‘team’ of human and machine, consisting of virtual and human analysts, can result in superior stock selection and why it is important to not rely entirely on one or the other.
1 Li, F. (2010) “The Information Content of Forward-Looking Statements in Corporate Filings—A Naïve Bayesian Machine Learning Approach,” Journal of Accounting Research, vol. 48(5), pp. 1049-1102.
This commentary has been prepared by Voya Investment Management for informational purposes. Nothing contained herein should be construed as (i) an offer to sell or solicitation of an offer to buy any security or (ii) a recommendation as to the advisability of investing in, purchasing or selling any security. Any opinions expressed herein reflect our judgment and are subject to change. Certain of the statements contained herein are statements of future expectations and other forward-looking statements that are based on management’s current views and assumptions and involve known and unknown risks and uncertainties that could cause actual results, performance or events to differ materially from those expressed or implied in such statements. Actual results, performance or events may differ materially from those in such statements due to, without limitation, (1) general economic conditions, (2) performance of financial markets, (3) changes in laws and regulations and (4) changes in the policies of governments and/or regulatory authorities. The opinions, views and information expressed in this commentary regarding holdings are subject to change without notice. The information provided regarding holdings is not a recommendation to buy or sell any security. Fund holdings are fluid and are subject to daily change based on market conditions and other factors.
Past performance is no guarantee of future results.