BanglaMix: a dataset of token-level annotated Bengali-English code-mixed social media comments with Bengali...

This article describes BanglaMix, a dataset of 12,576 Bengali-English code-mixed social media comments collected from a publicly available video-sharing platform (YouTube) using its public Data API. Each record pairs the original (raw) comment text with a lightly cleaned version, a Bengali-script (Unicode) transliteration of that text, and a word-level, token-by-token language annotation using five labels (BAN, ENG, ROM_BAN, OTHER, UNK). Each comment is additionally characterised by three comment-level attributes: script type (latin, bengali, mixed, or other), code-mix type (an eight-category label describing which languages and scripts co-occur in the comment, e.g., monolingual_BAN, ENG+ROM_BAN, BAN+ENG), and a boolean flag indicating whether the comment is likely to be code-mixed. All annotations, including the Bengali transliterations and the token-level language labels, were produced by a single primary annotator (the author); an inter-annotator reliability study in which a second bilingual annotator independently re-labelled a random 500-comment sub-sample found substantial agreement at both the token level (Cohen's kappa = 0.74) and comment level (Cohen's kappa = 0.67). Comments were retained as anonymous text records identified only by an alphanumeric comment identifier; no usernames, channel names, or other personal identifiers were stored, and records that had retained a public @-mention of another commenter were removed before release. Across the dataset, 57.3% of comments are written primarily in Latin script, 28.4% in Bengali script, and 13.6% in mixed script; 36.8% of comments are flagged as likely code-mixed. At the token level, of 120,866 labelled tokens, 44.7% are tagged as Bengali (BAN), 23.5% as English (ENG), and 22.1% as Romanized Bengali (ROM_BAN). Comments are short on average (median 6 words; mean 9.61 words per comment) and exhibit no duplicate source texts. The dataset is intended for reuse in word-level language identification, Bengali back-transliteration, and other Bengali-English code-mixed natural language processing tasks, for which large, token-annotated, naturalistic social media resources remain scarce.