An Introduction to the UnQLite Embedded NoSQL Database Engine

  •        0
  

We aggregate and tag open source projects. We have collections of more than one million projects. Check out the projects section.

UnQLite is an embedded NoSQL database engine. It's a standard Key/Value store similar to the more popular Berkeley DB and a document-store database similar to MongoDB with a built-in scripting language called Jx9 that looks like Javascript. Unlike most other NoSQL databases, UnQLite does not have a separate server process. UnQLite reads and writes directly to ordinary disk files. A complete database with multiple collections is contained in a single disk file. The database file format is cross-platform, you can freely copy a database between 32-bit and 64-bit systems or between big-endian and little-endian architectures.

In this article, we shall introduce the concept behind UnQLite, it's architecture and a high level overview of its C/C++ API.

Web Site: https://unqlite.org

Stable Version: 1.1.8 (January 2018)

System requirements: Any OS with a modern C compiler

License: open source, http://unqlite.org/licensing.html

Downloads: https://unqlite.org/downloads.html

Support: http://unqlite.org/suport.html


The UnQLite Architecture

UnQLite Architecture

Like most modern database engines, UnQLite is built-up on layers, The upper-layers hence the document-store and the Key/Value store layers are presented to the host application via a set of exported interfaces (i.e. The UnQLite API).

The principal task of a database engine is to store and retrieve records. UnQLite support both structured and raw database record storage. The Document-store layer is used to store JSON docs (i.e. Objects, Arrays, Strings, etc.) in the database and is powered by the Jx9 programming language while the Key/Value store layer is used to store raw records in the database.

Key/Value store layer.

UnQLite is a standard key/value store similar to BerkeleyDB, Tokyo Cabinet, LevelDB, etc. but, with a rich feature set including support for transactions (ACID), concurrent reader, etc. Under the KV store, both keys and values are treated as simple arrays of bytes, so content can be anything from ASCII strings, binary blob and even disk files. The KV store layer is presented to host applications via a set of interfaces, these includes: unqlite_kv_store(), unqlite_kv_append(), unqlite_kv_fetch_callback(), etc.

Document store layer.

The document store that is used to store JSON docs (i.e. Objects, Arrays, Strings, etc.) in the database is powered by the Jx9 programming language.

Jx9 is an embeddable scripting language also called extension language designed to support general procedural programming with data description facilities. Jx9 is a Turing-Complete, dynamically typed programming language based on JSON and implemented as a library in the UnQLite core.

Jx9 is built with a ton of features and has a clean and familiar syntax similar to C and Javascript. Being an extension language, Jx9 has no notion of a main program, it only works embedded in a host application. The host program (UnQLite in our case) can write and read Jx9 variables and can register C/C++ functions to be called by Jx9 code.

Pluggable Run-time Interchangeable Storage Engines.

UnQLite works with run-time interchangeable storage engines (i.e. Hash, B+Tree, R+Tree, LSM, etc.). The storage engine works with key/value pairs where both the key and the value are byte arrays of arbitrary length and with no restrictions on content. UnQLite come with two built-in KV storage engine: A Virtual Linear Hash (VLH) storage engine is used for persistent on-disk databases with O(1) lookup time and an in-memory hash-table or Red-black tree storage engine is used for in-memory databases. Future versions of UnQLite might add other built-in storage engines (i.e. LSM).

Transaction Manger/Pager module.

The underlying storage engine requests information from the disk in fixed-size chunks. The default chunk size is 4096 bytes but can vary between 512 and 65536 bytes. The pager module is responsible for reading, writing, and caching these chunks. The pager module also provides the rollback and atomic commit abstraction and takes care of locking of the database file. The storage engine requests particular pages from the page cache and notifies the pager module when it wants to modify pages or commit or rollback changes. The pager module handles all the messy details of making sure the requests are handled quickly, safely, and efficiently.

Virtual File System.

UnQLite is designed to run on multitude of platforms. In order to provide portability between these platforms (i.e. Between POSIX and Win32/64 operating systems), UnQLite uses an abstraction layer to interface with the operating system. Each supported operating system has its own implementation.

An instance of the unqlite_vfs object defines the interface between the UnQLite core and the underlying operating system. The "vfs" in the name of the object stands for "Virtual File System".

Introduction to the UnQLite C/C++ API

Early versions of UnQLite were very easy to learn since they only supported 12 C/C++ interfaces. But as the UnQLite engine has grown in capability, new C/C++ interfaces have been added so that now there are over 95 distinct APIs. This can be overwhelming to a new programmer. Fortunately, most of the C/C++ interfaces in UnQLite are very specialized and never need to be used. Despite having so many entry points, the core API is still relatively simple and easy to code to. This article aims to provide all of the background information needed to easily understand how the UnQLite database engine works.

The principal task of a database engine is to store and retrieve records. UnQLite support both structured and raw database record storage.

In order to accomplish this purpose, the developer needs to know about two objects:

  • The Database Engine Handle: unqlite
  • The UnQLite (Via Jx9) Virtual Machine Object: unqlite_vm

The database engine handle and optionally, the virtual machine object are controlled by a small set of C/C++ interface routines. The dozens of the C/C++ interface routines and two objects form the core functionality of UnQLite. The developer who understands them will have a good foundation for using UnQLite.

Typical Usage of the Core C/C++ Interfaces

An application that wants to use UnQLite will typically use the following interfaces with their optional components such as the transaction manager and the cursors interfaces. Note that an application may switch between the Key/Value store and the Document-store interfaces without any problem.

   unqlite_open()

This routine opens a connection to an UnQLite database file and returns a database handle. This is often the first UnQLite API call that an application makes and is a prerequisite for most other UnQLite APIs. Many UnQLite interfaces require a pointer to the database handle as their first parameter and can be thought of as methods on the database handle. This routine is the constructor for the database handle.

   unqlite_kv_store()

   unqlite_kv_append()

   unqlite_kv_store_fmt()

   unqlite_kv_append_fmt()

   unqlite_kv_delete()

   unqlite_kv_fetch()

   unqlite_kv_fetch_callback()

Key/Value Store Interfaces

Under the Key/Value store, both keys and values are treated as simple arrays of bytes, so content can anything from ASCII strings, binary blob and even disk files. This set of interfaces allows clients to store and retrieve raw database records efficiently regardless of the underlying KV storage engine.

  unqlite_compile()

  unqlite_compile_file()

  unqlite_vm_config()

  unqlite_vm_exec()

  unqlite_vm_reset()

  unqlite_vm_extract_variable()

  unqlite_vm_release()

Document Store Interfaces

These set of interfaces works with the unqlite_vm object which is obtained after successful compilation of the target Jx9 script. Jx9 is the scripting language which power the document-store interface to UnQLite.

The Document-Store interface to UnQLite works as follows:

Obtain a new database handle via unqlite_open().

Compile your Jx9 script using one of the compile interfaces such as unqlite_compile() or unqlite_compile_file(). On successful compilation, the engine will automatically create an instance of this structure (unqlite_vm) and a pointer to this structure is made available to the caller.

When something goes wrong during compilation of the target Jx9 script due to a compile-time error, the caller must discard this pointer and fix its erroneous Jx9 code. Compile-time error logs can be extracted via a call to unqlite_config() with a configuration verb set to UNQLITE_CONFIG_JX9_ERR_LOG.

Optionally, configure the virtual machine using unqlite_vm_config().

Optionally, register one or more foreign functions or constants using the unqlite_create_function() or unqlite_create_constant() interfaces.

Execute the compiled Jx9 program by calling unqlite_vm_exec().

Optionally, extract the contents of one or more variables declared inside your Jx9 script using unqlite_vm_extract_variable().

Optionally, Reset the virtual machine using unqlite_vm_reset(), then go back to step 6. Do this zero or more times.

Destroy the virtual machine using unqlite_vm_release()

  unqlite_kv_cursor_init()

  unqlite_kv_cursor_first_entry()

  unqlite_kv_cursor_seek()

  unqlite_kv_cursor_last_entry()

  and many more

Database Cursors

Cursors provide a mechanism by which you can iterate over the records in a database. Using cursors, you can seek, fetch, move, and delete database records.

  unqlite_begin()

  unqlite_commit()

  unqlite_rollback()

Manual Transaction Manager

This set of interfaces allows the host application to manually start a write-transaction. Note that UnQLite is smart enough and will automatically start a write-transaction in the background when needed and so call to these routines is usually not necessary.

  unqlite_close()

This routine is the destructor for the unqlite handle obtained by a prior successful call to unqlite_open().

Each database connection must be closed in order to avoid memory leaks and malformed database image.

 

This article only mentions the foundational UnQLite interfaces. The UnQLite library includes many other APIs implementing useful features that are not described here. A complete list of functions that form the UnQLite application programming interface is found at the C/C++ Interface Specification. Refer to that document for complete and authoritative information about all UnQLite interfaces.

 


   

We publish blog post about open source products. If you are interested in sharing knowledge about open source products, please visit write for us




Related Articles

An introduction to MongoDB

  • mongodb database document-oriented-databse no-sql c++ data-mining

MongoDB is the most exciting SQL-free database currently available in the market. The new kid on the block, called MongoDB is a scalable, high-performance, open source, schema free and document oriented database that focuses on the ideas of NoSQL Approach. Written in C++, it has taken rapid strides since its emergence into the public sphere as a popular way to build your database applications.

Read More


Lucene / Solr as NoSQL database

  • lucene solr no-sql nosql document-store

Lucene and Solr are most popular and widely used search engine. It indexes the content and delivers the search result faster. It has all capabilities of NoSQL database. This article describes about its pros and cons.

Read More


Whats new in Lucene / Solr 4.0

  • lucene solr new-release

The release 4.0 is one of the important milestone for Lucene and Solr. It has lot of new features and performance important. Few important ones are highliggted in this article.

Read More


Univention Corporate Server - An open source identity management system

  • ucs identity-management-system

Univention Corporate Server is an open source identity management system, an IT infrastructure and device management solution and an extensible platform with a store-like App Center that includes tested third party applications and further UCS components: This is what Univention combines in their main product Univention Corporate Server, a Debian GNU/Linux based enterprise distribution. This article provides you the overview of Univention Corporate Server, its feature and installation.

Read More


Should web application store images in Database or File system?

  • database image-store filesystem

Web developers most frequent question, Should user images be stored in database or file system? Which is the best way. Both has some pros and cons.

Read More



Column database vs OLAP

  • business-intelligence olap column-database

OLAP (Online Analytical Processing), Reporting, Data mining related tasks are usually done by Business intelligence products. They do powerful Extraction, Transformation and Loading (ETL) the data and provides various reports. They use relational database as its back end. How could they generate better reports? Will column DB do a better job?

Read More


How hashmap works in Java. My style of learning.

  • java hashmap opensource-learning

This is the most frequently asked questions in the interview. Googling will throw many links related to this topic. How to learn the implementation of hash map? My style of learning using open source learning technique.

Read More


How to create SEO friendly url

  • seo url searchengine

SEO friendly URL is recommended for any website which wants to be indexed and wants its presence in search results. Searchengine mostly index the static URL. It will avoid the URL which has lot of query strings. Almost all websites generate content dynamically then how could the URL be static. That is the job of the programmer.

Read More


Best situation to use Column database

  • column database reporting

Column oriented database or datastore as the name sounds it stores the data by column rather than by row. It has some advantages and disadvantages over traditional RDBMS. Developer should know the typical situation to choose column oriented database.

Read More


Exonum Blockchain Framework by the Bitfury Group

  • blockchain bitcoin hyperledger blockchain-framework

Exonum is an extensible open source blockchain framework for building private blockchains which offers outstanding performance, data security, as well as fault tolerance. The framework does not include any business logic, instead, you can develop and add the services that meet your specific needs. Exonum can be used to build various solutions from a document registry to a DevOps facilitation system.

Read More


LogicalDOC - Open Source DMS

  • dms document-management-system

LogicalDOC is both a document management and a collaboration system. The software is loaded with many functions and allows organizing, indexing, retrieving, controlling and distributing important business documents securely and safely for any organization and individual.

Read More


Is Unix time end by 2038

  • unix-time year-2038

In 32 bit operating system, Dates are calculated using number of seconds differece between 1 January 1970 and current date. This difference will reset to 00:00:00 on January 2038. This is called year-2038 Bug.

Read More


LetsEncrypt certificate using ZeroSSL tools

  • ssl-certificate certificate security

Let’s Encrypt is a free, automated, and open Certificate Authority. It uses ACME protocol to validate your domain. If you have complete control over your domain, you can get a certificate for free. In order to provide secure access to your public network like HTTPS, LDAPS etc you need a certificate from a Certificate Authority. The cost of the certificate range from 10$ to 100$. If you want a wildcard certificate then it may cost more. The certificate is valid for one year and you need to pay and renew every year. Let's Encrypt comes for the rescue. You can create and renew certificate for few.

Read More


How Bitcoin works? A simple introduction.

Bitcoin is an open source digital currency which could be transferred in a P2P payment network. It is decentralized and it is not controlled by any central authority or banks. It is transferred from person to person and no authority will be aware of your transaction. Its quite different from PayPal or Banks.

Read More


Why Elastic Search is gaining more popularity than Solr?

  • solr elastic-search search-engine

Solr and Elastic Search both are built on top of Lucene library. Both are compratively equal. Any new feature / enhancement which get introduced in Lucene, will also get added to Solr. But still Elastic Search which uses Lucene as it core gained more popularity than Solr in recent years.

Read More


AbanteCart - Easy to use open source e-commerce platform, helps selling online

  • e-commerce ecommerce cart

AbanteCart is a free, open source shopping cart that was built by developers with a passion for free and accessible software. Founded in 2010 (launched in 2011), the platform is coded in PHP and supports MySQL. AbanteCart’s easy to use admin and basic layout management tool make this open source solution both easy to use and customizable, depending on the skills of the user. AbanteCart is very user-friendly, it is entirely possible for a user with little to no coding experience to set up and use this cart. If the user would be limited to the themes and features available in base AbanteCart, there is a marketplace where third-party extensions or plugins come to the rescue.

Read More


Is ZooKeeper mandatory for Cloud

  • zookeeper distributed cloud

Cloud is nothing but more than one system or application distributed across the network, across the globe. It may have couple of application servers, database server, shared data storage, backup server and lot more. The resources in the distributed environment must have information about each other so that they could co-ordinate and share without any issues. ZooKeeper helps to do that.

Read More


ONLYOFFICE Document Server, an online office app for Nextcloud and ownCloud

  • office office-suite word spreadsheet

ONLYOFFICE Document Server is a free collaborative online office suite including viewers and editors for texts, spreadsheets and presentations, fully compatible with Office Open XML formats (.docx, .xlsx, .pptx). This article provides you the overview of ONLYOFFICE Document Server, its features, installation and integration with Nextcloud and ownCloud.

Read More


10 Free services for your Website / Blog. Just plug it.

  • free website blog free-service free-resources

Each website / blog delivers useful content or service to its users. But website themselves requires some service to monitor and increase its presence. Here are few free services which could be used by Website / Blog. This will be very much helpful for small business owners.

Read More


An introduction to web cache proxy server - nuster

  • web-cache proxy-server load-balancer

Nuster is a simple yet powerful web caching proxy server based on HAProxy. It is 100% compatible with HAProxy, and takes full advantage of the ACL functionality of HAProxy to provide fine-grained caching policy based on the content of request, response or server status. This article gives an overview of nuster - web cache proxy server, its installation and few examples of how to use it.

Read More