A supervised weeding method to cluster high dimensional predictors with application to job market analysis.

Area of Science:

• Data mining
• Bioinformatics
• Text analysis

Background:

• Clustering high-dimensional predictors is crucial in fields like text mining and biological data analysis.
• Standard clustering methods identify patterns within predictors but cannot predict response variables.
• Existing methods often focus on either clustering or prediction, not both.

Purpose of the Study:

• To develop a supervised weeding algorithm for detecting sparse clusters and capturing prediction effects simultaneously.
• To address the dual requirement of clustering and prediction in high-dimensional data analysis.
• To introduce a novel approach for supervised predictor selection and clustering.

Main Methods:

• An iterative feature screening and coherence evaluation procedure.
• Backward elimination of unimportant predictors to form nested sets.
• Monte Carlo simulations for assessing finite-sample performance.
• Application to a job description dataset to analyze keyword impact on salaries.

Main Results:

• The proposed supervised weeding algorithm demonstrates comparable clustering and prediction performance to existing specialized methods.
• The method effectively identifies significant predictor groups that influence response variables.
• Simulation results confirm the algorithm's finite-sample performance.

Conclusions:

• The supervised weeding algorithm offers a robust solution for the dual problem of clustering and prediction in high-dimensional data.
• This approach enhances the utility of clustering by incorporating predictive capabilities.
• The method provides valuable insights into feature importance and relationships within complex datasets.