go-edlib - Golang string comparison and edit distance algorithms library, featuring : Levenshtein, LCS, Hamming, Damerau levenshtein (OSA and Adjacent transpositions algorithms), Jaro-Winkler, Cosine, etc

•    Go

---

Golang string comparison and edit distance algorithms library featuring : Levenshtein, LCS, Hamming, Damerau levenshtein (OSA and Adjacent transpositions algorithms), Jaro-Winkler, Cosine, etc...

unicode algorithms edit-distance levenshtein jaro-winkler levenshtein-distance similarity-measures string-distance cosine string-matching damerau-levenshtein lcs lcs-distance hamming string-comparison golang-string-comparison edit-distance-algorithms

leven - Measure the difference between two strings with the fastest JS implementation of the Levenshtein distance algorithm

•    Javascript

---

Measure the difference between two strings using the fastest JS implementation of the Levenshtein distance algorithm

leven levenshtein distance algorithm algo string difference diff fast fuzzy similar similarity compare comparison edit text match matching

jieba-php - "結巴"中文分詞：做最好的 PHP 中文分詞、中文斷詞組件。 / "Jieba" (Chinese for "to stutter") Chinese text segmentation: built to be the best PHP Chinese word segmentation module

•    PHP

---

"Jieba" (Chinese for "to stutter") Chinese text segmentation: built to be the best PHP Chinese word segmentation module. Scroll down for English documentation.

nlp natural-language-processing chinese-text-segmentation machine-learning

java-string-similarity - Implementation of various string similarity and distance algorithms: Levenshtein, Jaro-winkler, n-Gram, Q-Gram, Jaccard index, Longest Common Subsequence edit distance, cosine similarity

•    Java

---

Implementation of various string similarity and distance algorithms: Levenshtein, Jaro-winkler, n-Gram, Q-Gram, Jaccard index, Longest Common Subsequence edit distance, cosine similarity...

levenshtein-distance cosine-similarity string-distance damerau-levenshtein distance distance-measure jaro-winkler similarity-measures shingles algorithm jvm

fuzzysearch - :pig: Tiny and fast fuzzy search in Go

•    Go

---

Inspired by bevacqua/fuzzysearch, a fuzzy matching library written in JavaScript. But contains some extras like ranking using Levenshtein distance (see RankMatch()) and finding matches in a list of words (see Find()). Fuzzy searching allows for flexibly matching a string with partial input, useful for filtering data very quickly based on lightweight user input.

fuzzy-search algorithm

fuzzysearch - :pig: Tiny and fast fuzzy search in Go

•    Go

---

Inspired by bevacqua/fuzzysearch, a fuzzy matching library written in JavaScript. But contains some extras like ranking using Levenshtein distance (see RankMatch()) and finding matches in a list of words (see Find()). Fuzzy searching allows for flexibly matching a string with partial input, useful for filtering data very quickly based on lightweight user input.

fuzzy-search algorithm

lac - Baidu's open-source lexical analysis tool for Chinese, including word segmentation, part-of-speech tagging & named entity recognition

•    C++

---

Baidu's open-source lexical analysis tool for Chinese, including word segmentation, part-of-speech tagging & named entity recognition.

lexical-analysis word-segmentation part-of-speech-tagger named-entity-recognition chinese-word-segmentation chinese-nlp

fuzzywuzzy - Fuzzy String Matching in Python

•    Python

---

Fuzzy string matching like a boss. It uses Levenshtein Distance to calculate the differences between sequences in a simple-to-use package.

gse - Go efficient text segmentation; support english, chinese, japanese and other. Go 语言高性能分词

•    Go

---

Go efficient text segmentation; support english, chinese, japanese and other. Dictionary with double array trie (Double-Array Trie) to achieve, Sender algorithm is the shortest path based on word frequency plus dynamic programming.

segment nlp gse chinese english japanese trie

textdistance - Compute distance between sequences

•    Python

---

TextDistance -- python library for comparing distance between two or more sequences by many algorithms. Work in progress. Now all algorithms compare two strings as array of bits.

distance algorithm textdistance hamming-distance levenshtein-distance damerau-levenshtein damerau-levenshtein-distance algorithms distance-calculation jellyfish

jcseg - Jcseg是基于mmseg算法的一个轻量级中文分词器，同时集成了关键字提取，关键短语提取，关键句子提取和文章自动摘要等功能，并且提供了一个基于Jetty的web服务器，方便各大语言直接http调用，同时提供了最新版本的lucene, solr, elasticsearch的分词接口！

•    Java

---

A lightweight open source chinese tokenizer with keywords, key sentences, summary extracts support, offered the latest lucene,solr,elasticsearch API.

jcseg mmseg chinese-word-segmentation natural-language-processing pos-tagging nlp nlp-keywords-extraction lucene-analyzer lucene-tokenizer solr-plugin elasticsearch-analyzer chinese-text-segmentation chinese-nlp keywords-extraction jcseg-analyzer

jiebaR - Chinese text segmentation with R. R语言中文分词 （文档已更新 🎉 ：https://qinwenfeng.com/jiebaR/ )

•    C++

---

This is a package for Chinese text segmentation, keyword extraction and speech tagging.

cppjieba jieba chinese-text-segmentation lexical-analysis chinese nlp

jieba - 结巴中文分词

•    Python

---

"Jieba" (Chinese for "to stutter") Chinese text segmentation: built to be the best Python Chinese word segmentation module.

natural-language-processing nlp

SentencePiece - Unsupervised text tokenizer for Neural Network-based text generation

•    C++

---

SentencePiece is an unsupervised text tokenizer and detokenizer mainly for Neural Network-based text generation systems where the vocabulary size is predetermined prior to the neural model training. SentencePiece implements subword units (e.g., byte-pair-encoding (BPE) and unigram language model with the extension of direct training from raw sentences. SentencePiece allows us to make a purely end-to-end system that does not depend on language-specific pre postprocessing.

neural-machine-translation natural-language-processing word-segmentation translation text tokenizer neural-network

Chinese-Word-Vectors - 100+ Chinese Word Vectors 上百种预训练中文词向量

•    Python

---

This project provides 100+ Chinese Word Vectors (embeddings) trained with different representations (dense and sparse), context features (word, ngram, character, and more), and corpora. One can easily obtain pre-trained vectors with different properties and use them for downstream tasks. Moreover, we provide a Chinese analogical reasoning dataset CA8 and an evaluation toolkit for users to evaluate the quality of their word vectors.

chinese chinese-word-segmentation embeddings word-embeddings vectors-trained embedding

jellyfish - 🎐 a python library for doing approximate and phonetic matching of strings.

•    Python

---

Jellyfish is a python library for doing approximate and phonetic matching of strings. Written by James Turk <james.p.turk@gmail.com> and Michael Stephens.

levenshtein soundex hamming metaphone jaro-winkler fuzzy-search

rmmseg-cpp - an re-implementation of rmmseg (Chinese word segmentation library for Ruby) in C++

•    Ruby

---

an re-implementation of rmmseg (Chinese word segmentation library for Ruby) in C++

Arab Techies

•    Javascript

---

A collection of open source libraries and tools that provide solutions for common problems in processing Arabic text, especially in web applications. text normalization, phrase segmentation, text indexing, stop word lists, common spelling mistakes.

talisman - A straightforward & modular NLP, machine learning & fuzzy matching library for JavaScript

•    Javascript

---

The library's full documentation can be found here. Be sure to lint & pass the unit tests before submitting your pull request.

natural-language-processing machine-learning fuzzy-matching clustering record-linkage bayes bloom-filter canberra caverphone chebyshev cologne cosine classifier daitch-mokotoff dice fingerprint fuzzy hamming k-means jaccard jaro lancaster levenshtein lig metaphone mra ngrams nlp nysiis perceptron phonetic porter punkt schinke sorensen soundex stats tfidf tokenizer tversky vectorizer winkler

HanLP - 自然语言处理 中文分词 词性标注 命名实体识别 依存句法分析 关键词提取 新词发现 短语提取 自动摘要 文本分类 拼音简繁

•    Java

---

汉语言处理包

nlp natural-language-processing hanlp crf hmm trie textrank doublearraytrie neural-network chinese-word-segmentation text-mining pos-tagging dependency-parser text-classification word2vec perceptron named-entity-recognition text-clustering