Quickwit - Fast and cost-efficient distributed search engine for immutable data

  •        112

Quickwit is a distributed search engine built from the ground up to offer cost-efficiency and high reliability. By mere mortals for mere mortals, Quickwit's architecture is as simple as possible. Quickwit is written in Rust and built on top of the mighty tantivy library. It designed to index big datasets straight from object storage like AWS S3 in a stateless manner.

  • Scalable distributed search: Host an arbitrary number of indexes on Amazon S3 and answer search queries in less than a second with a small pool of stateless search instances.
  • Stream indexing: Ingest TB of data from your favorite distributed event streaming service. As of today, Quickwit supports Apache Kafka natively. The next releases will bring support for more platforms.
  • Fault-tolerant architecture that won't lose data: Quickwit achieves exactly-once processing for indexing and safely stores your data on highly reliable object storage services such as Amazon S3.
  • Cloud-native, easy to operate: Thanks to true decoupled compute and storage, search instances are stateless, add or remove search nodes within seconds.
  • Sub-second full-text search on cloud / distributed storage: Quickwit Search re-designed indexing and index data structure to open it in less than 60ms on Amazon S3.
  • Time-based sharding: Quickwit shards data by time when enabled. And you can use a second dimension to shard data thanks to our tags feature. Time-based queries only access splits (a data piece of the index) that match the time range of the query which leads to significant performance improvements.
  • Painless multi-tenant search: Create indexes for each tenant without hurting query performance. Or group tenants into one index and use tagging to prune irrelevant splits for your tenant query to improve significantly performance.

https://quickwit.io
https://github.com/quickwit-inc/quickwit

Tags
Implementation
License
Platform

   




Related Projects

Solr - Blazing-fast, open source enterprise search platform

  •    Java

Solr is the popular, blazing fast open source enterprise search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, dynamic clustering, database integration, and rich document (e.g., Word, PDF) handling. Solr is highly scalable, providing distributed search and index replication, and it powers the search and navigation features of many of the world's largest internet sites.

Manticore Search - High performance full-text search engine with SQL and JSON support

  •    C++

Manticore Search is an open source high performance full-text search oriented engine. It is a fork of Sphinx Search. Manticore Search is written in C++. It means speed and low resource consumption, it means you don’t have to worry about a garbage collector that suddenly makes a trouble.

Jumper - Collaborative search engine in PHP

  •    PHP

Jumper 2.0 is a collaborative community search platform that revolutionizes search by crowdsourcing knowledge management powered by a shared bookmarking engine. It is easily and quickly deployed into a community of practice that benefits users with complex and specialized search requirements. Jumper delivers universal search of any databases, flat files, fileshares, content systems, web pages, blogs and wikis, even people - through one simple search box.

Sphinix - Search server

  •    C++

Sphinix is free open-source SQL full-text search engine. How do you implement full-text search for that 10+ million row table, keep up with the load, and stay relevant? Sphinx is good at those kinds of riddles.

ElasticSearch - Distributed, RESTful search and analytics engine

  •    Java

Elasticsearch is a distributed, RESTful search and analytics engine capable of solving a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data so you can discover the expected and uncover the unexpected.


Tantivy - Full-text search engine library inspired by Lucene and written in Rust

  •    Rust

Tantivy is a full text search engine library written in rust. It is closer to Lucene than to Elastic Search and Solr in the sense it is not an off-the-shelf search engine server, but rather a crate that can be used to build such a search engine.

Elasticlunr.js - Lightweight full-text search engine in Javascript for browser search and offline search

  •    Javascript

Elasticlunr.js is a lightweight full-text search engine in Javascript for browser search and offline search. Elasticlunr.js is developed based on Lunr.js, but more flexible than lunr.js. Elasticlunr.js provides Query-Time boosting and field search. Elasticlunr.js is a bit like Solr, but much smaller and not as bright, but also provide flexible configuration and query-time boosting.

Open Search Server

  •    C++

Open Search Server is both a modern crawler and search engine and a suite of high-powered full text search algorithms. Built using the best open source technologies like lucene, zkoss, tomcat, poi, tagsoup. Open Search Server is a stable, high-performance piece of software.

SenseiDB - Distributed, Realtime, Semi-Structured Database from LinkedIn

  •    Java

Sensei is a distributed data system that was built to support many product initiatives at LinkedIn, including the real-time faceted search in LinkedIn Signal and the news feed and tabs on the Homepage. Sensei is both a search engine and a database. It is designed to query and navigate through documents that consist of unstructured text and well-formed and structured metadata. Sensei is both a search engine and a database.

LUNR.js - A bit like Solr, but much smaller and not as bright

  •    Javascript

Lunr.js is a small, full-text search library for use in the browser. It indexes JSON documents and provides a simple search interface for retrieving documents that best match text queries. A bit like Solr, but much smaller and not as bright. Lunr enables you to provide a great search experience without the need for external, server-side, search services. Lunr has no external dependencies and works in your browser or on the server with node.js.

OpenSearch - Open source distributed and RESTful search engine

  •    Java

OpenSearch is a community-driven, open source search and analytics suite derived from Apache 2.0 licensed Elasticsearch 7.10.2 & Kibana 7.10.2. It consists of a search engine daemon, OpenSearch, and a visualization and user interface, OpenSearch Dashboards. OpenSearch enables people to easily ingest, secure, search, aggregate, view, and analyze data. These capabilities are popular for use cases such as application search, log analytics, and more.

Lucene - A high-performance, full-featured text search engine library

  •    Java

Apache Lucene is a high-performance, full-featured text search engine library written entirely in Java. It is a technology suitable for nearly any application that requires full-text search, especially cross-platform.

Crate - The fast, scalable, easy to use SQL database with native full text search

  •    Java

Crate is an open source, highly scalable, shared-nothing distributed SQL database. Crate offers the scalability and performance of a modern No-SQL database with the power of Standard SQL. Crate’s distributed SQL query engine lets you use the same syntax that already exists in your applications or integrations, and have queries seamlessly executed across the crate cluster, including any aggregations, if needed.

mnoGoSearch

  •    C

mnoGoSearch for UNIX consists of a command line indexer and a search program which can be run under Apache Web Server, or any other HTTP server supporting CGI interface. mnoGoSearch for Unix is distributed in sources and can be compiled with a number of databases, depending on user's choice. It is known to work on a wide variety of the modern Unix operating systems including Linux, FreeBSD, Mac OSX, Solaris and others.

IndexTank - Search Engine powers Reddit

  •    Java

IndexTank search engine powers search in Reddit, Social bookmarking site. IndexTank is acquired by LinkedIn and released the project as open source. It includes features like Variables boosts, Facets, Faceted search, Snippeting, Custom scoring functions, Suggest, and Autocomplete.

Strus - Full text Search Engine in C++

  •    C++

The open source project strus provides a collection of C++ (C++98) libraries and command line tools for building a full-text search engine. The strus search engine can be build using any key value store database that provides an upper bound seek function for the stored key/value pairs. Currently there exists an implementation based on the LevelDB library.

riot - Go Open Source, Distributed, Simple and efficient Search Engine

  •    Go

Supporting riot, buy me a coffee.Riot is primarily distributed under the terms of the Apache License (Version 2.0), base on wukong.

MeiliSearch - Ultra relevant, instant and typo-tolerant full-text search API

  •    Rust

MeiliSearch is a powerful, fast, open-source, easy to use, and deploy search engine. The search and indexation are fully customizable and handles features like typo-tolerance, filters, and synonyms. It features include Search as-you-type experience, Full text search, Supports Synonym, Highly customizable, RESTfull API and lot more.

Nutch - Highly extensible, highly scalable Web crawler

  •    Java

Nutch is open source web-search software. It builds on Lucene Java, adding web-specifics, such as a crawler, a link-graph database, parsers for HTML and other document formats, etc.






We have large collection of open source products. Follow the tags from Tag Cloud >>


Open source products are scattered around the web. Please provide information about the open source projects you own / you use. Add Projects.