September 29, 2026
Sentiment in Big Data as an Indicator of Financial Market Behavior
Sentiment in Big Data as an Indicator of Financial Market Behavior
Amanda Panico Gasperini; Julio Cesar Ruivo Costa
DOI: 10.22167/2675-6528-202602629
Article derived from a Course Conclusion Work (TCC), with content based on the student’s original work and adapted to the editorial format of the E&S Magazine with the support of the ResumeAI tool, an artificial intelligence solution developed by Instituto Pecege for textual synthesis and organization.
Summary
The relationship between financial news and asset behavior was investigated in the Brazilian context during a period of high volatility. The study analyzed whether the sentiment present in news headlines, collected between 2020 and 2024, was associated with the daily fluctuations of IBOVESPA, S&P 500, and three B3 stocks. To this end, 675 headlines were collected through Google News RSS service, classified by emotional polarity using a customized financial lexicon for Portuguese, developed after generalist BERT models presented systematic errors. Sentiment scores were cross-referenced with market returns from Yahoo Finance, generating 394 valid observations. Pearson correlation revealed a positive and significant association between sentiment and the return of IBOVESPA (r = 0.13; p = 0.013) and PETR4 (r = 0.10; p = 0.041). The analysis with a one- to three-day lag did not indicate predictive power of sentiment on subsequent trading sessions, a result consistent with the Efficient Market Hypothesis. The findings indicated that headlines and asset prices move jointly in response to the same events, and that generalist language models are insufficient to capture the specific vocabulary of financial news in Portuguese. The main contribution of the work was the construction of the customized financial lexicon, highlighting the need for specific tools for the language and domain. Media sentiment functions as a thermometer of the present, not as a predictive tool.
Keywords: COVID-19; Behavioral finance; IBOVESPA; Natural language processing; Media sentiment.
1. Introduction
Classical economic theory, formalized by Fama’s Efficient Market Hypothesis (1970), postulates that financial asset prices reflect all publicly available information. In its semi-strong version, this hypothesis suggests that the analysis of released news does not confer informational advantage, as its content would already be incorporated into prices.
However, events such as the 14.78% drop in the IBOVESPA in March 2020, coinciding with the peak of journalistic coverage on the pandemic, challenge this premise. Models based exclusively on economic fundamentals did not predict such a movement. The question then arises about the existence of a measurable relationship between media tone and asset behavior, and under what conditions this relationship manifests.
This questioning aligns with behavioral finance, which documents how emotions and cognitive shortcuts can distort investment decisions (Kahneman, 2012; Shiller, 2014; Mosca, 2009). In parallel, Natural Language Processing (NLP) offers tools to automatically extract semantic information from large-scale journalistic texts (Bird, Klein, Loper, 2009). Lexicon-based sentiment analysis, a modality of unsupervised text mining, was the solution employed in this work to classify financial headlines.
The integration between behavioral finance and NLP has international precedents, such as Tetlock’s (2007) study, which associated pessimism in journalistic columns with drops in the American market. In Brazil, Improta (2023) tested the incorporation of sentiment into financial time series forecasting models, with inconclusive results that highlight the complexity of combining textual and financial data in the national market. Brazilian academic production that crosses NLP with the financial market, especially in crisis scenarios, is still limited.
The period from 2020 to 2024, marked by the COVID-19 pandemic, represented a context of exceptional news volume and historical volatility, offering a unique opportunity to investigate this relationship with real data. Given this scenario, this work aimed to verify whether the sentiment of headlines collected between 2020 and 2024 was associated with the daily fluctuations of IBOVESPA and S&P 500, specifically seeking to organize a database of headlines with identified dates, develop a classification method suitable for the financial vocabulary in Portuguese, test the correlation between sentiment and returns, verify whether the sentiment of one day anticipates the following trading session, and compare the behavior of this relationship in different moments of the pandemic cycle.
2. Material and Methods
The research was characterized as a quantitative and correlational study, with the objective of investigating the association between the sentiment expressed in financial news headlines and the daily fluctuations of market assets. The methodological approach involved the collection of textual and financial data, the development of a customized sentiment lexicon for Portuguese, and the application of statistical techniques to analyze the temporal and contemporaneous relationships between the variables.
The unit of analysis consisted of financial news headlines and asset price data. Headline collection was carried out between January 2020 and December 2024, using the Google News RSS service. Thirteen specific queries, with terms related to the Brazilian financial market, IBOVESPA, S&P 500, and the COVID-19 pandemic, were used to gather the titles. After removing duplicates and filtering by period, a database of 675 valid headlines was obtained, each with its title and publication date.
For market data, the public API of Yahoo Finance was used, through the yfinance library in Python. Adjusted closing prices were downloaded for IBOVESPA (^BVSP), S&P 500 (^GSPC), VALE3.SA, PETR4.SA, and B3SA3.SA, covering the same period as the headlines. The selection of these assets aimed to represent the broad Brazilian stock exchange index, distinct sectors of the national economy, and the main American benchmark index, allowing for the capture of domestic market dynamics and a global benchmark.
Before sentiment analysis, the headlines underwent a four-step text preprocessing process. First, the name of the news outlet was removed. Then, residual HTML characters and URLs were eliminated. Finally, multiple spaces were normalized. Portuguese accents and special characters were preserved so as not to compromise the identification of specific financial terms. Headlines with fewer than four words after cleaning were discarded.
Sentiment classification was a central technical step. Initially, pre-trained language models based on the BERT architecture (Devlin et al., 2019) were tested, specifically nlptown/bert-base-multilingual-uncased-sentiment and Ixyuan/distilbert-base-multilingual-cased-sentiments-student. However, both models presented systematic errors when classifying financial headlines in Portuguese, such as categorizing “Ibovespa rises 5% with optimism about vaccines” as negative or “Circuit breaker is triggered for the third time” as positive.
These errors stemmed from training the models on generalist domains, such as consumer reviews and social media, where financial vocabulary has distinct meanings. The BERTimbau model (Souza et al., 2020), although trained for Brazilian Portuguese, did not have fine-tuning on a specific financial corpus, which would maintain the problem. Therefore, we opted for the construction of a customized financial lexicon for the Brazilian context, following the dictionary-based sentiment analysis approach (Tetlock, 2007).
The custom lexicon was developed with manual validation of each term. The set of negative terms included words such as “fall”, “crisis”, “collapse”, “circuit breaker”, “recession”, “plummet”, “loss”, “lockdown”, “default”, “contraction”, and “volatility”. The positive set gathered terms such as “rise”, “up”, “soars”, “record”, “growth”, “recovery”, “optimism”, “appreciation”, “expansion”, and “profit”.
The sentiment score for each headline was calculated as the normalized difference between the counts of positive and negative terms present in the text, resulting in values between -1 and +1. Headlines without classifiable terms received a score of zero. The lexicon classified 54.2% of the 675 headlines with non-zero polarity. This metric captured the direction of the sentiment, without inferring its intensity, which would be unfeasible without an annotated financial corpus in Portuguese for reliable validation.
The sentiment and market databases were integrated by date. Sentiment scores were aggregated by date as the average of all headlines from that day. This series was cross-referenced with the market database through an inner join, keeping only the days when there were headlines and the stock market was open simultaneously. This process resulted in 394 valid observations with complete data for analysis.
The daily returns of the assets were calculated as the percentage change in the closing price relative to the previous day, multiplied by 100. This transformation aimed to eliminate the long-term trend of absolute price series, which could artificially inflate correlation coefficients. All analyses were implemented in Python, using the pandas and scipy libraries, in the Google Colaboratory environment.
Three categories of statistical analysis were applied. The first tested the Pearson correlation between the daily average sentiment and the percentage returns of each asset, calculating the coefficient (r) and the p-value for a two-tailed test with a significance level of 5%. The second category investigated lags of one, two, and three business days, to verify the predictive power of sentiment on subsequent trading days.
The third category of analysis divided the sample into four sub-periods: acute pandemic (2020), recovery (2021), normalization (2022-2023), and post-pandemic (2024). This division allowed for the comparison of the behavior of the relationship between sentiment and returns throughout the different phases of the pandemic cycle. Additionally, a simple linear regression model was estimated, considering asset returns as the dependent variable and daily average sentiment as the independent variable, to complement the correlation analysis.
3. Results and Discussion
The investigation of the relationship between the sentiment expressed in financial news headlines and market asset behavior, in the period from 2020 to 2024, revealed important findings about informational dynamics in the Brazilian context. The results indicated a positive and significant association between media sentiment and the returns of IBOVESPA and PETR4, but without any predictive power over future trading sessions. This study also highlighted the need for Natural Language Processing (NLP) tools specialized for financial Portuguese, culminating in the construction of a customized lexicon, a central methodological contribution to the field.
The analysis of the database, composed of 675 headlines collected via RSS from Google News, demonstrated a temporal distribution that directly reflects the historical events of the period. It was found that 26.4% of the headlines were from 2020 and 26.8% from 2021, concentrating the largest volume of journalistic coverage during the most intense phases of the COVID-19 pandemic. This concentration does not represent a sampling bias, but rather a mirror of the reality of the time, characterized by intense news production and high market volatility.
March 2020, in particular, stood out as the month with the highest density of negative headlines in the entire historical series. This peak of media pessimism coincided with critical events, such as the successive activations of *circuit breakers* on B3 and the official declaration of the pandemic by the World Health Organization. Market data corroborated this observation, with the largest daily drop in the IBOVESPA, of 14.78%, recorded on March 12, 2020, and in the S&P 500, of 11.98%, on March 16 of the same year, both aligned with the peak of negative headlines.
Regarding sentiment classification, the customized financial lexicon developed for this work managed to assign non-null polarity to 54.2% of the headlines. Of this total, 29.7% were classified as negative and 24.5% as positive. The distribution of sentiment over time also followed the pandemic’s chronology: the months from March to June 2020 presented the highest density of negative headlines, while the period from October 2020 to March 2021, marked by the first vaccine efficacy announcements, recorded the highest positive scores.
The construction of a customized financial lexicon was a crucial step, motivated by the inadequacy of general-purpose language models, such as those based on the BERT architecture, for the Portuguese financial vocabulary. Initial tests with models like *nlptown/bert-base-multilingual-uncased-sentiment* and *lxyuan/distilbert-base-multilingual-cased-sentiments-student* demonstrated systematic errors. For example, the headline “Ibovespa rises 5% with optimism about vaccines” was classified as negative by one model, and “Circuit breaker is triggered for the third time” as positive by another, highlighting the failure to capture the contextual meaning of specific financial market terms.
This limitation of generalist models stems from the fact that they are trained on consumer reviews or social media comments, domains where the semantics of terms like “circuit breaker” differ drastically from the financial context. The absence of an annotated financial *corpus* in Portuguese, as identified in the study, reinforces the relevance of the lexicon-based approach, which, although technically simpler, proved effective for the specific domain. The developed lexicon included negative terms such as “queda” (fall), “crise” (crisis), “colapso” (collapse), and “circuit breaker”, and positive terms such as “alta” (rise), “sobe” (rises), “recorde” (record), and “crescimento” (growth), with scores ranging from -1 to +1 to indicate the direction of sentiment.
The contemporary correlation analysis between the daily average headline sentiment and the percentage returns of the assets revealed significant results for IBOVESPA and PETR4. The Pearson coefficient for IBOVESPA was +0.1255, with a p-value of 0.013, indicating a statistically significant positive association. For PETR4, the coefficient was +0.1029, with a p-value of 0.041, also pointing to a statistically significant positive correlation. The other assets analyzed, S&P 500, VALE3, and B3SA3, presented positive coefficients of similar magnitude, but without statistical significance at the 5% level.
The positive sign of these correlations suggests that days with a more favorable tone in the headlines tended to coincide with positive returns in assets, while pessimistic coverage accompanied declines. It is crucial to interpret this relationship as a co-movement phenomenon, rather than direct causality, as highlighted by Tetlock (2007). Headlines and asset prices likely react jointly to the same events, with the market pricing in the information during the trading day and the press publishing the corresponding tone in near real-time.
The relatively weak magnitude of the correlation coefficients was expected, as asset prices are influenced by a multiplicity of factors, such as interest rates, exchange rates, external capital flow, and corporate results. Headline sentiment, in this scenario, acts as a signal in a noisy and complex informational environment. The absence of significance for the S&P 500, an index from a developed market, is consistent with the hypothesis that markets with greater analytical depth and diversity of institutional participants are less sensitive to the narrative climate of the general media.
The time-lagged analysis, which sought to verify if the sentiment of one day would anticipate market movements in the following days, did not produce significant results. Tests with lags of one, two, and three business days for IBOVESPA and S&P 500 did not reveal predictive power of sentiment. For example, for IBOVESPA, the correlation with a one-day lag was +0.0038 (p = 0.9397), and for S&P 500, it was -0.0555 (p = 0.2728), both without statistical significance.
This finding is consistent with Fama’s (1970) Semi-Strong Market Efficiency Hypothesis, which posits that asset prices already rapidly incorporate all publicly available information, including news content. If media sentiment had predictive power, it would be possible to develop simple and profitable investment strategies, which the data do not support. The window of opportunity to react to headline information closes on the same day, distinguishing media sentiment from variables with genuine predictive power, such as macroeconomic indicators with lagged effects.
The sub-period analysis, which divided the sample into pandemic phases (acute pandemic, recovery, normalization, and post-pandemic), did not find statistical significance in any of the groups. This lack of significance may be attributed to the reduction in the number of observations in each sub-period, ranging from 75 to 127 records, which decreases the statistical power to detect effects of weak magnitude. However, the pattern of coefficients over time offers relevant analytical *insights*.
In the period of acute pandemic (2020), the correlation coefficient for IBOVESPA was the highest positive (r = +0.1682), although not significant. This result suggests that in times of high uncertainty, when economic fundamentals are difficult to assess, the tone of the news may carry more weight in investors’ decisions. This observation aligns with Kahneman’s (2012) description of intuitive processing dominating in chaotic environments, making agents more reactive to the emotional climate of the news.
In 2021, during the recovery phase, the correlation coefficient for IBOVESPA was close to zero and slightly negative (r = -0.0090). This may indicate a divergence between market enthusiasm, driven by vaccines and economic reopening, and media coverage, which still paid attention to virus variants and uncertainties about the recovery. In the normalization (2022-2023) and post-pandemic (2024) sub-periods, the coefficients returned to positive values (r = +0.1495 and r = +0.1229, respectively), close to the significance threshold, reinforcing the idea that the relationship between media sentiment and the market is not constant, intensifying during periods of greater uncertainty.
From a practical point of view, the findings of this study indicate that media sentiment, although not a predictive tool, has value as a thermometer of the present and an auxiliary contextual variable for investment managers. In times of high uncertainty, such as those experienced during the pandemic, strongly negative sentiment scores can signal a deterioration in risk appetite, complementing the reading of other market stress indicators. This information can assist in decisions regarding exposure management, hedging, or the timing of portfolio rebalancing.
It is fundamental to emphasize that the use of media sentiment should be understood within its limitations. It does not confer an advantage for price movement anticipation strategies and should not be treated as an isolated buy or sell signal. Its character is diagnostic and contemporary, offering an additional input for the interpretation of the market environment, but not a decision-making model. The main contribution of this work, the construction of a customized financial lexicon for Portuguese, highlights the need for specific tools for the language and domain, overcoming the deficiencies of generalist models and paving the way for future more robust research.
4. Conclusion
This study aimed to verify the association between the sentiment expressed in financial news headlines and the daily oscillations of assets in the Brazilian market, covering the period from 2020 to 2024. A positive and statistically significant association was found between media sentiment and the returns of IBOVESPA and PETR4. However, the time-lagged analysis did not indicate predictive power of sentiment on subsequent trading days, a finding that aligns with the Semi-Strong Form Efficient Market Hypothesis. The results suggest that headlines and asset prices tend to move together, reacting to the same market events.
The main contribution of this work lies in the construction of a customized financial lexicon for Brazilian Portuguese. This tool was developed after observing the inadequacy of general-purpose language models, such as those based on the BERT architecture, to capture the specific vocabulary of financial news in Portuguese, which presented systematic errors in the classification of contextual terms. The absence of an annotated financial corpus for the national language was a relevant methodological finding, reinforcing the need for specialized solutions and the importance of manual lexicon curation for this domain. From a practical standpoint, the findings indicate that media sentiment, although not a predictive tool, offers value as a thermometer of the present and an auxiliary context variable for investment managers. In periods of high uncertainty, such as those experienced during the pandemic, strongly negative sentiment scores can signal a deterioration in risk appetite, complementing other market stress indicators and aiding in decisions regarding exposure management or portfolio rebalancing. However, it is crucial to acknowledge the study’s limitations, such as the weak magnitude of correlation coefficients, expected in an environment with multiple influencing factors, and the reduction in statistical power in sub-period analysis due to the smaller number of observations. For future studies, it is suggested to replicate the methodology for the American market, directly comparing the magnitude and significance of the S&P 500’s correlations with those of the IBOVESPA, which could deepen the understanding of how emerging and developed markets react to media coverage.
Bibliographic References
BIRD, S.; KLEIN, E.; LOPER, E. Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit. Sebastopol: O’Reilly Media, 2009.
DEVLIN, J.; CHANG, M.-W.; LEE, K.; TOUTANOVA, K. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2019. Disponível em: https://arxiv.org/abs/1810.04805. Acesso em: mar. 2025.
FAMA, E. F. Efficient capital markets: a review of theory and empirical work. The Journal of Finance, v. 25, n. 2, p. 383-417, 1970.
IMPROTA, A. D. S. S. Incorporando análise de sentimentos na predição em séries temporais financeiras por meio de modelos TabNet. 2023. Monografia (MBA em Inteligência Artificial e Big Data) Instituto de Ciências Matemáticas e de Computação, Universidade de São Paulo, São Carlos, 2023. Disponível em: https://bdta.abcd.usp.br/directbitstream/aa074227-cb33-4cff-a6de-099f775784a0/Alexandre%20Improta.pdf. Acesso em: mar. 2025.
KAHNEMAN, D. Rápido e Devagar: Duas Formas de Pensar. Rio de Janeiro: Objetiva, 2012.
MOSCA, A. Finanças Comportamentais. Rio de Janeiro: Elsevier, 2009.
SHILLER, R. J. Exuberância Irracional. 3. ed. São Paulo: Saraiva, 2014.
SOUZA, F.; NOGUEIRA, R.; LOTUFO, R. BERTimbau: Pretrained BERT models for Brazilian Portuguese. In: BRAZILIAN CONFERENCE ON INTELLIGENT SYSTEMS (BRACIS), 9., 2020, Rio Grande. Anais… Rio Grande: BRACIS, 2020. p. 403-417.
TETLOCK, P. C. Giving content to investor sentiment: the role of media in the stock market. The Journal of Finance, v. 62, n. 3, p. 1139-1168, jun. 2007.
Article originating from the Final Course Project of the Specialization in Data Science and Analytics of the MBA USP/Esalq
To learn more about the course, click here and access the MBX Academy platform