Hadoop-based Financial News Text-Driven Market Volatility Prediction: TF-IDF, Machine Learning, and SHAP Interpretability Analysis

Abstract

The variation of financial market is largely influenced by the emergence of new information like macroeconomic policies, corporate notifications and international political events. With the growing amount of financial textual data, it has become an important issue to predict market trends from news materials. In our research, we have collected an empirical data set including 1,985 news articles concerning market volatility. A binary target label has been established based on whether the daily volatility exceeds a 1% level, resulting in a positive class ratio approximately 0.27. We carry out the Hadoop framework (HDFS and distributed processing) in the first stage of text processing to preprocess and organize the raw data. For feature extraction, a TF-IDF model (with 1-2 grams and 5,000 maximum features) is used to obtain the text vector representation. Then, we train and test two kinds of algorithms: a calibrated probability linear Support Vector Machine (SVM) and an XGBoost classifier. We conduct validation tests based on a TimeSeriesSplit method. The performance measures consist of the area under the Receiver Operating Characteristic curve (ROC-AUC), Precision-Recall curve (PR-AUC) and the optimal F1 score. Additionally, we integrate other features such as VADER sentiment indices, token entropy and special shock words to do an ablation study. The SHAP parameters give explanations for the results both globally and locally. The experimental results indicate that there are strong predictive hints in the news texts and the interpretive step is beneficial to confirm the model logic.