New Ingredients from AI: Turning Language into Food for ML Algorithms

November 1, 2024

To the uninitiated masses 🙂 the markets often seem to be a jumble of ticker symbols, numbers, and charts that must be entirely driven by purely numerical factors—things they couldn’t possibly hope to ever understand in a million years. 🙂 However, to those of us used to studying numbers and mathematical structures, price charts often look much more like biological movements than something with an underlying mathematical truth. And there is a wealth of academic literature describing financial markets through the features of biological organisms.

...to those of us used to studying numbers and mathematical structures, price charts often look much more like biological movements than something with an underlying mathematical truth.

Yes, there are underlying fundamentals. And yes, unemotional, high-frequency algorithmic trading does impact price movements. But there is a decidedly organic, human, emotional element to markets.

The human element is often overlooked in favor of the more tangible, numerical aspects of trading in various types of analysis. Historically, the interplay of human emotions, expectations, and reactions has been too complex a dance to truly model. While rudimentary sentiment analysis can provide valuable inputs, traditional analysis falls well short of being able to capture the full spectrum of market dynamics, especially during times of extreme volatility—and fear.

This is also why using only technical analysis and traditional indicators as inputs into machine learning models makes them—well—not entirely useless, but we view them as incomplete. Beyond basic sentiment analysis, the general lack of ‘awareness’ of the human perspective makes them rather limited in scope.

...using only technical analysis and traditional indicators as inputs into machine learning models makes them—well—not entirely useless, but we view them as incomplete.

The transformative role of advanced Natural Language Processing (NLP) and Large Language Models (LLMs) is pivotal in this context. These technologies are adept at processing and analyzing language data, turning qualitative information—words, sentiment, and unstructured data—into quantifiable features. This process effectively converts what was once purely qualitative into a format that can be easily integrated into machine learning algorithms.

By transforming words and unstructured data into numerical values, NLP and LLMs bridge the gap between traditional fundamentals and the nuanced world of human sentiment. For example, the sentiment expressed in news articles or social media posts can now be quantified and used to predict market trends or investor behavior. This quantification allows for a more comprehensive approach to financial analysis, combining the precision of numbers with the depth of qualitative insights.

However, it's crucial to recognize that while NLP and LLMs significantly enhance our ability to process and analyze qualitative data, they do not yet replace the need for human expertise. The interpretation of this transformed data still requires a nuanced understanding of both the markets and human psychology. In our view, removing human expertise from the process will be iterative. For now, we believe the most effective approach is one that combines advanced technology with human insight, leveraging the strengths of both to gain a more complete understanding of market dynamics.

Ayano is a virtual writer we are developing specifically to focus on publishing educational and introductory content covering AI, LLMs, financial analysis, and other related topics. Humans are currently responsible for ideation, prompt engineering, fact-checking, copy editing, and overall guidance and training—including finalizing translations, while LLMs cover initial research, analysis, copywriting, and drafting translations into multiple languages.