dedupe - :id: A python library for accurate and scaleable fuzzy matching, record deduplication and entity-resolution

•    Python

---

dedupe is a python library that uses machine learning to perform fuzzy matching, deduplication and entity resolution quickly on structured data. dedupe takes in human training data and comes up with the best rules for your dataset to quickly and automatically find similar records, even with very large databases.

dedupe record-linkage python-library entity-resolution

Zingg - Scalable fuzzy matching for data mastering, deduplication and entity resolution

•    Java

---

Zingg is a scalable fuzzy matching for data mastering, deduplication and entity resolution. Real world data contains multiple records belonging to the same customer. These records can be in single or multiple systems and they have variations across fields which makes it hard to combine them together, especially with growing data volumes. Zingg integrates different records of an entity like customer, patient, supplier, product etc in same or disparate data sources.

data-science identity-resolution spark etl dedupe entity-resolution data-transformation ml fuzzy-matching deduplication masterdata dataengineering fuzzymatch dataquality datapreparation analytics-engineering data-transformations

csvdedupe - :id: Command line tool for deduplicating CSV files

•    Python

---

Command line tools for using the dedupe python library for deduplicating CSV files. csvdedupe - takes a messy input file or STDIN pipe and identifies duplicates.

dedupe cli record-linkage entity-resolution csv-files

conciliator - OpenRefine reconciliation services for VIAF, ORCID, and Open Library + framework for creating more

•    Java

---

conciliator is a growing collection of OpenRefine reconciliation services, as well as a Java framework for creating them. A reconciliation service tries to match variant text (usually names of things) to standard IDs for the entity represented by that text. This project supercedes refine_viaf.

reconciliation-service viaf openrefine orcid solr entity-resolution openlibrary

whatis - WhatIs.this: simple entity resolution through Wikipedia

•    Ruby

---

WhatIs.this is a quick probe for the meaning and metadata of concepts through Wikipedia. gem install whatis or add gem "whatis" to your Gemfile.

wikipedia entity-resolution

dedupe-examples - :id: Examples for using the dedupe library

•    Python

---

Example scripts for the dedupe, a library that uses machine learning to perform de-duplication and entity resolution quickly on structured data. We recommend using virtualenv and virtualenvwrapper for working in a virtualized development environment. Read how to set up virtualenv.

dedupe record-linkage entity-resolution

rltk - Record Linkage ToolKit (Find and link entities)

•    Python

---

The Record Linkage ToolKit (RLTK) is a general-purpose open-source record linkage platform that allows users to build powerful Python programs that link records referring to the same underlying entity. Record linkage is an extremely important problem that shows up in domains extending from social networks to bibliographic data and biomedicine. Current open platforms for record linkage have problems scaling even to moderately sized datasets, or are just not easy to use (even by experts). RLTK attempts to address all of these issues. RLTK supports a full, scalable record linkage pipeline, including multi-core algorithms for blocking, profiling data, computing a wide variety of features, and training and applying machine learning classifiers based on Python’s sklearn library. An end-to-end RLTK pipeline can be jump-started with only a few lines of code. However, RLTK is also designed to be extensible and customizable, allowing users arbitrary degrees of control over many of the individual components. You can add new features to RLTK (e.g. a custom string similarity) very easily.

linkage similarity similarity-metric string-similarity record-linkage entity-resolution