An Embeddable NoSQL Database Engine

Star

Follow @symisc

Tweet

Follow @unqlite_db

Introduction

This document describes the architecture of the UnQLite database engine. The information here is useful to those who want to understand or modify the inner workings of UnQLite.

A block diagram showing the main components of UnQLite and how they interrelate is shown above. The text that follows will provide a quick overview of each of these components.

Like most modern database engines, UnQLite is built-up on layers, The upper-layers hence the document-store and the Key/Value store layers are presented to the host application via a set of exported interfaces (i.e. The UnQLite API).

The principal task of a database engine is to store and retrieve records. UnQLite support both structured and raw database record storage. The Document-store layer is used to store JSON docs (i.e. Objects, Arrays, Strings, etc.) in the database and is powered by the Jx9 programming language while the Key/Value store layer is used to store raw records in the database.

Key/Value store layer.

UnQLite is a standard key/value store similar to BerkeleyDB, Tokyo Cabinet, LevelDB, etc. but, with a rich feature set including support for transactions (ACID), concurrent reader, etc. Under the KV store, both keys and values are treated as simple arrays of bytes, so content can be anything from ASCII strings, binary blob and even disk files. The KV store layer is presented to host applications via a set of interfaces, these includes: unqlite_kv_store(), unqlite_kv_append(), unqlite_kv_fetch_callback(), unqlite_kv_append_fmt(), unqlite_kv_delete(), etc.

Document store layer.

The Document store to UnQLite which is used to store JSON docs (i.e. Objects, Arrays, Strings, etc.) in the database is powered by the Jx9 programming language.

Jx9 is an embeddable scripting language also called extension language designed to support general procedural programming with data description facilities. Jx9 is a Turing-Complete, dynamically typed programming language based on JSON and implemented as a library in the UnQLite core.

Jx9 is built with a tons of features and has a clean and familiar syntax similar to C and Javascript. Being an extension language, Jx9 has no notion of a main program, it only works embedded in a host application. The host program (UnQLite in our case) can write and read Jx9 variables and can register C/C++ functions to be called by Jx9 code.

Pluggable Run-time Interchangeable Storage Engines.

UnQLite works with run-time interchangeable storage engines (i.e. Hash, B+Tree, R+Tree, LSM, etc.). The storage engine works with key/value pairs where both the key and the value are byte arrays of arbitrary length and with no restrictions on content. UnQLite come with two built-in KV storage engine: A Virtual Linear Hash (VLH) storage engine is used for persistent on-disk databases with O(1) lookup time and an in-memory hash-table or Red-black tree storage engine is used for in-memory databases. Future versions of UnQLite might add other built-in storage engines (i.e. LSM).

Transaction Manger/Pager module.

The underlying storage engine requests information from the disk in fixed-size chunks. The default chunk size is 4096 bytes but can vary between 512 and 65536 bytes. The pager module is responsible for reading, writing, and caching these chunks. The pager module also provides the rollback and atomic commit abstraction and takes care of locking of the database file. The storage engine requests particular pages from the page cache and notifies the pager module when it wants to modify pages or commit or rollback changes. The pager module handles all the messy details of making sure the requests are handled quickly, safely, and efficiently.

Virtual File System.

UnQLite is designed to run on multitude of platforms. In order to provide portability between these platforms (i.e. Between POSIX and Win32/64 operating systems), UnQLite uses an abstraction layer to interface with the operating system. Each supported operating system has its own implementation.

An instance of the unqlite_vfs object defines the interface between the UnQLite core and the underlying operating system. The "vfs" in the name of the object stands for "Virtual File System".

useful links to start with.

UnQLite is very easy to learn, even for new programmer. Here is some useful links to start with:

UnQLite In 5 Minutes Or Less gives a high-level overview on the how to embed UnQLite in a host application.
An Introduction To The UnQLite C/C++ Interface gives an overview and roadmap to the dozens of C/C++ interface to UnQLite.

An Introduction To Jx9 - Jx9 is the scripting language which power the document store interface to UnQLite.

Download some working examples from the download page.

Interested in Machine Learning & Computer Vision? try out Pixlab