Simulated data for census-scale entity resolution research without privacy restrictions: a large-scale dataset...

Area of Science:

• Data Science
• Computational Social Science

Background:

• Entity Resolution (ER) is crucial for data integration but hindered by privacy concerns surrounding personally identifiable information (PII).
• Restrictions on accessing authentic PII slow the development and testing of new ER methods and software.
• The pseudopeople Python package addresses this by generating simulated datasets for ER research.

Purpose of the Study:

• To develop and release a tool for generating realistic, noisy simulated population data for entity resolution (ER).
• To enable researchers to develop and test ER algorithms without compromising sensitive personal information.

Main Methods:

• Utilized the Vivarium simulation platform to create a dynamic model of individuals, families, households, and employment.
• Generated simulated censuses, surveys, and administrative data reflecting real-world population dynamics.
• Developed the pseudopeople Python package to add configurable noise to simulated data for realistic ER challenges.

Main Results:

• Produced over 900 gigabytes of simulated population data, representing hundreds of millions of individuals.
• Made a sample population of thousands openly available via the pseudopeople package.
• Provided access to larger simulated populations upon request, structured for ER research.

Conclusions:

• The pseudopeople package and its associated simulated data overcome PII access barriers in ER research.
• Facilitates the development, testing, and adoption of novel ER algorithms and software for large-scale population data.