The first step is the hardest: pitfalls of representing and tokenizing temporal data for large language models

Area of Science:

• Artificial Intelligence
• Machine Learning
• Data Science

Background:

• Large language models (LLMs) show advanced reasoning, making them suitable as personal assistants.
• Integrating numerical and temporal data, such as from wearables or electronic health records, presents a significant challenge for LLMs.
• LLM tokenizers, designed for text, often fail to capture the nuances of numerical and temporal patterns, leading to a 'modality gap'.

Purpose of the Study:

• To investigate the challenges of representing and tokenizing temporal data for LLMs.
• To analyze the effectiveness of current methods for feeding time-series data into LLMs.
• To propose directions for improving LLM performance with temporal data.

Main Methods:

• A case study was conducted using the OpenAI tokenizer on a mobile sensing dataset.
• A review of recent literature on feeding time-series data into LLMs for human-centric tasks was performed.
• Experimental setups like zero-shot prompting and few-shot learning were examined.

Main Results:

• The case study demonstrated that LLMs tokenize timestamps and sensor values into meaningless units, failing to grasp temporal relationships.
• Existing research relies heavily on prompt engineering and data aggregation to bridge the modality gap.
• A trade-off exists between model expressiveness (domain-specific models) and parameter efficiency (zero-shot LLMs) for temporal data.

Conclusions:

• Current LLM tokenizers are not optimized for numerical data, and limited time-series examples in training data exacerbate this issue.
• Balancing model expressiveness and computational efficiency is key for effective temporal data integration.
• Prompt tuning, model grafting, and enhanced tokenizers are promising avenues for future research, particularly in biomedical signal processing.