An Action Selection Method Using Student’s T-Distributed Noise for Optimising Deep Deterministic Policy Gradient Algorithm in Recommender Systems.

Abstract

The exponential growth of digital content, driven by the proliferation of the internet, e-commerce, social media, and multimedia services, has intensified the problem of information overload. To address this, recommender systems have emerged as indispensable tools for filtering and prioritising relevant content. While traditional systems rely on collaborative filtering or content-based techniques, recent advancements increasingly frame recommendation as a sequential decision-making problem, leveraging the theoretical foundations of Reinforcement Learning (RL). Within this paradigm, Deep Reinforcement Learning (DRL) and specifically actor–critic methods such as the Deep Deterministic Policy Gradient (DDPG) algorithm have shown promise due to their ability to model complex user–item interactions in continuous action spaces. Despite their potential, DRL algorithms are susceptible to fundamental issues like overestimation bias and the compounding of approximation errors in Temporal Difference (TD) learning. In recommender systems, these shortcomings can cause the agent to overcommit to a narrow set of high-reward items, diminishing recommendation diversity and limiting long-term user engagement. A common remedy involves injecting exploration noise into the policy’s action space. By encouraging deviation from deterministic behaviour, this noise enhances the agent’s capacity to discover underexplored but potentially rewarding actions, thereby mitigating overestimation and promoting robust policy learning. This thesis develops a parameterised heavy-tailed stochastic exploration framework for actor–critic reinforcement learning, defined by three design parameters: the noise distribution family, the scale schedule, and the bound region. The framework is instantiated by integrating Student’s t-distributed action perturbation, linear scale annealing, and value clipping into DDPG, resulting in the proposed t-DDPG method. Unlike standard Gaussian or Ornstein–Uhlenbeck (OU) noise, both light-tailed distributions that produce predominantly local perturbations, the heavy-tailed Student’s t-distribution generates occasional large excursions while retaining mathematical tractability and bounded operating behaviour. The framework is evaluated using three benchmark recommendation datasets: MovieLens 100K, MovieLens 1M, and Jester D4. Its performance was compared against three baseline DDPG configurations: vanilla DDPG without exploration noise, DDPG with Gaussian noise, and DDPG with Ornstein–Uhlenbeck noise. The proposed t-DDPG variant achieved the highest HR@10 and nDCG@10 across all three datasets. On MovieLens 100K, evaluated on ten random seeds, t-DDPG attained HR@10 of 0.4543±0.0193 and nDCG@10 of 0.2375±0.0136, exceeding the Gaussian baseline by 6.5% and 8.0% relative respectively. A Bonferroni-corrected statistical analysis at family-wise ! = 0.05 confirmed significance for all six pairwise comparisons (three baselines → two metrics) under both the paired t-test and the non-parametric Wilcoxon signed-rank test. On MovieLens 1M and Jester D4, evaluated on three random seeds, t-DDPG again outperformed all baselines on both metrics. Notably, t-DDPG was the only configuration of the four to deliver a positive gain over the deterministic baseline on every dataset tested; the lighter-tailed alternatives underperformed the deterministic baseline on Jester D4. These findings establish heavy-tailed action-level perturbation as a theoretically grounded and empirically validated exploration mechanism for actor–critic recommendation, with the framework formulation supporting principled extension to alternative heavy-tailed distributions and to other actor–critic methods.