14 best open source chunking projects.

We have collection of more than 1 Million open source products ranging from Enterprise product to small libraries in all platforms. We aggregate information from all open source repositories. Search and find the best for your needs. Check out projects section.

Displaying 1 to 14 from 14 results

NCRFpp - NCRF++, an Open-source Neural Sequence Labeling Toolkit

Python

Sequence labeling models are quite popular in many NLP tasks, such as Named Entity Recognition (NER), part-of-speech (POS) tagging and word segmentation. State-of-the-art sequence labeling models mostly utilize the CRF structure with input word features. LSTM (or bidirectional LSTM) is a popular deep learning based feature extractor in sequence labeling task. And CNN can also be used due to faster computation. Besides, features within word are also useful to represent word, which can be captured by character LSTM or character CNN structure or human-defined neural features. NCRF++ is a PyTorch based framework with flexiable choices of input features and output structures. The design of neural sequence labeling models with NCRF++ is fully configurable through a configuration file, which does not require any code work. NCRF++ is a neural version of CRF++, which is a famous statistical CRF framework.

pytorch ner sequence-labeling crf lstm-crf char-rnn char-cnn named-entity-recognition part-of-speech-tagger chunking neural-networks nbest lstm cnn batch

fine-uploader - Multiple file upload plugin with image previews, drag and drop, progress bars

Javascript

FineUploader is also simple to use. In the simplest case, you only need to include one JavaScript file. There are absolutely no other required external dependencies. For more information, please see the documentation. If you'd like to help and keep this project strong and relevant, you have several options.

fineuploader web-development files vanilla-javascript vanilla-js standalone api file-upload javascript-library dropzone amazon aws azure chunk chunking cross-domain cross-site drag drop file file-input file-uploader input jquery-plugin multiple preview progress resume s3 selection upload widget

casync - Content-Addressable Data Synchronization Tool

Encoding: Let's take a large linear data stream, split it into variable-sized chunks (the size of each being a function of the chunk's contents), and store these chunks in individual, compressed files in some directory, each file named after a strong hash value of its contents, so that the hash value may be used to as key for retrieving the full chunk data. Let's call this directory a "chunk store". At the same time, generate a "chunk index" file that lists these chunk hash values plus their respective chunk sizes in a simple linear array. The chunking algorithm is supposed to create variable, but similarly sized chunks from the data stream, and do so in a way that the same data results in the same chunks even if placed at varying offsets. For more information see this blog story. Decoding: Let's take the chunk index file, and reassemble the large linear data stream by concatenating the uncompressed chunks retrieved from the chunk store, keyed by the listed chunk hash values.

archive tar file-system http chunking synchronization download upload delivery

flair - A very simple framework for state-of-the-art NLP

Python

A very simple framework for state-of-the-art NLP. Developed by Zalando Research. A powerful syntactic-semantic tagger / classifier. Flair allows you to apply our state-of-the-art models for named entity recognition (NER), part-of-speech tagging (PoS), frame sense disambiguation, chunking and classification to your text.

pytorch nlp named-entity-recognition sequence-labeling chunking semantic-role-labeling word-embeddings

nacl-stream-js - Streaming encryption based on TweetNaCl.js

Javascript

Inputs: 32-byte key, 16-byte nonce, a stream (or a file).Stream is split into chunks of the specified length.

encryption decryption streaming chunking nacl tweetnacl file

node-chunking-streams - A set of NodeJS streams aimed on chunking data

Javascript

Simple TransformStream which counts lines (\n is a separator) and emit data chunks contains exactly specified number of them. If some tail data is not fit fully into specified chunk size, it can be emitted at the end if flushTail flag is set.

stream chunking s3 gzip

desync - Alternative casync implementation

This project re-implements many features of upstream casync in Go. It seeks to maintain compatibility with casync's data structures, protocols and types, such as chunk stores (castr), index files (caibx/caidx) and archives (catar) in order to function as a drop-in replacement in many use cases. It also tries to maintain support for platforms other than Linux and simplify build/installation. It consists of a library that implements the features, available for integration into any 3rd-party product as well as a command-line tool. For support and discussion, see . Feature requests should be discussed there before filing, unless you're interested in doing the work to implement them yourself.

chunking archive synchronization casync

cryptor - Privacy, Anonimity, Freedom

Cryptor is a P2P network designed for sharing data without revealing one's true identity or the nature of the shared information. Data security is accomplished using asymetric encryption (AES256) and public-private key (RSA) for safe peer-to-peer communication and data integrity validation. All local data stored on your machine is encrypted using your master password. The chunk design of the file sharing protocol also further improves security.

p2p encryption cryptography network file-sharing chunking cryptor under-development

SAPO - SAPO

CSharp

This is an implementation of the paper "Towards Easier and Faster Sequence Labeling: A Search-based Probabilistic Online Learning Framework (SAPO)". The codes are also used for the paper "Towards Shockingly Easy Structured Classification: A Search-based Probabilistic Online Learning Framework".

natural-language-processing chunking named-entity-recognition postagging

rabin - node native addon for rabin fingerprinting data streams

Node native addon module (C/C++) for Rabin fingerprinting data streams. Uses the implementation of Rabin fingerprinting from LBFS.

rabin cdc chunking fingerprint rolling-hash dedupe deduplication rsync

split - Split large files into smaller ones using deterministic Content Defined Chunking

Split large files into smaller ones using the same Content Defined Chunking algorithm the restic backup program uses. If you're interested in the mathematical foundation for Content Defined Chunking with Rabin Fingerprints, head over to the restic blog which has an introductory article.

data split chunking cdc