GPTRC: A large-scale dataset for evaluating question-answering capabilities and limitations of large language models

A Generative Pre-Trained Transformer (GPT) Responses Corpus (GPTRC), a large-scale dataset designed to facilitate comprehensive analysis of ChatGPT's performance was presented in this paper. The dataset consists of 50,134 question-response pairs generated by GPT-3.5 Turbo model. It covers broad range of topics and question formats for the evaluation of the Large Language Models (LLMs) strength and limitations. It is also comparable to human-generated text. The records in the dataset includes metadata like thinking time, response time, response length, and extracted keywords. These metadata are helpful for benchmarking studies. The dataset was curated through a combination of web-scraped and manually generated questions, which were submitted to ChatGPT via OpenAI's API to capture raw, unfiltered responses. The question types considered are both factual and reasoning-based. It also covers multilingual record for studying LLMs capabilities in terms of coherence, relevance, adaptability, and processing efficiency. The GPTRC dataset is a transparent, reproducible, and openly accessible resource for analyzing LLM behavior across varied domains. It supports researchers aiming to benchmark LLMs on multiple fronts such as accuracy, efficiency, linguistic richness, and consistency. The comprehensive metadata allows researchers to draw nuanced insights into model performance under different conditions. By facilitating fine-grained analysis, this dataset serves as a foundational resource for improving LLM robustness and guiding the development of future AI systems.