Feature-level Knowledge Distillation Framework for Horizontal Dimensionality Reduction Process in Multitask Learning Using UNITI Strategy

Abstract

Horizontal data dimensionality reduction is a technique that simplifies data by reducing complexity along the task and output dimensions instead of the input dimension. This approach decreases redundancy between tasks, which enhances knowledge sharing and lowers computational demands. Multi-label learning and other machine learning approaches faces challenges from high-dimensional data. However, existing dimensionality reduction frameworks make it complex to apply semi-supervised methods, enforce instance correlation constraints, or implement feature selection and extraction mechanisms. A major challenge in multi-task learning is the loss of important discriminative information. To overcome this issue, the proposed model integrates dimensionality reduction in a multi-task learning approach. Initially, high-dimensional data are collected from the relevant databases and then applied to Iterative Linear Discriminant Analysis (LDA)-based dimensional reduction, which results in a discriminative and stable low-dimensional representation that maintains the essential features. Then, multitask learning is performed using Feature-level Knowledge Distillation-based Unifying Neural networks with Inter-dataset for Task Integration (FKD-UNITI) framework, which performs continuous refinement of shared features, boosts the tasks integration across datasets and achieves scalable learning over heterogeneous data sources. The performance of the suggested dimensional reduction model is evaluated among existing techniques to ensure its effectiveness using different performance metrics.