An Embeddable NoSQL Database Engine

Star

Follow @symisc

Tweet

Follow @unqlite_db

This article provides an overview and roadmap to the C/C++ interface to UnQLite.

Early versions of UnQLite were very easy to learn since they only supported 12 C/C++ interfaces. But as the UnQLite engine has grown in capability, new C/C++ interfaces have been added so that now there are over 95 distinct APIs. This can be overwhelming to a new programmer. Fortunately, most of the C/C++ interfaces in UnQLite are very specialized and never need to be used. Despite having so many entry points, the core API is still relatively simple and easy to code to. This article aims to provide all of the background information needed to easily understand how the UnQLite database engine works.

A separate document, The UnQLite C/C++ Interface, provides detailed specifications for all of the various C/C++ APIs for UnQLite. Once the reader understands the basic principles of operation for UnQLite, that document should be used as a reference guide. This article is intended as introduction only and is neither a complete nor authoritative reference for the UnQLite API.

1.0 Core Objects And Interfaces

The principal task of a database engine is to store and retrieve records. UnQLite support both structured and raw database record storage.

Structured data storage is presented to clients via the Document store interface to UnQLite. The document store itself is used to store JSON docs (i.e. Objects, Arrays, Strings, etc.) in the database and is powered by the Jx9 programming language. Jx9 is an embeddable scripting language also called extension language designed to support general procedural programming with data description facilities. Jx9 is a Turing-Complete, dynamically typed programming language based on JSON and implemented as a library in the UnQLite core. You can find more on Jx9 at the Introduction to Jx9 page.

Raw data storage is presented to clients via the Key/Value store interfaces. UnQLite is a standard key/value store similar to Berkeley DB, Tokyo Cabinet, LevelDB, etc. but with a rich feature set including support for transactions (ACID), concurrent reader, etc. Under the KV store, both keys and values are treated as simple arrays of bytes, so content can be anything from ASCII strings, binary blob and even disk files. The KV store layer is presented to clients via a set of interfaces, these includes: unqlite_kv_store(), unqlite_kv_append(), unqlite_kv_fetch_callback(), unqlite_kv_append_fmt(), etc.

In order to accomplish this purpose, the developer needs to know about two objects:

• The Database Engine Handle: unqlite

• The UnQLite (Via Jx9) Virtual Machine Object: unqlite_vm

The database engine handle and optionally, the virtual machine object are controlled by a small set of C/C++ interface routine listed below.

• unqlite_open()

  Key/Value Store Interface

• unqlite_kv_store()

• unqlite_kv_append()

• unqlite_kv_store_fmt()

• unqlite_kv_append_fmt()

• unqlite_kv_delete()

• unqlite_kv_fetch()

• unqlite_kv_fetch_callback()

  Document Store Interfaces

• unqlite_compile()

• unqlite_compile_file()

• unqlite_vm_config()

• unqlite_vm_exec()

• unqlite_vm_extract_variable()
  

• unqlite_vm_reset()

• unqlite_vm_release()
  Optionally, the Transaction Manager

• unqlite_begin()
• unqlite_commit()

• unqlite_rollback()
  

  Optionally, the Cursor Interfaces

• unqlite_kv_cursor_init()

• unqlite_kv_cursor_first_entry()

• unqlite_kv_cursor_seek()

• unqlite_kv_cursor_last_entry()

• and many more

  Closing the database

• unqlite_close()

The dozens of the C/C++ interface routines and two objects listed above form the core functionality of UnQLite. The developer who understands them will have a good foundation for using UnQLite.

2.0 Typical Usage of the Core C/C++ Interfaces

An application that wants to use UnQLite will typically use the following interfaces with their optional components such as the transaction manager and the cursors interfaces. Note that an application may switch between the Key/Value store and the Document-store interfaces without any problem.

unqlite_open()	This routine opens a connection to a UnQLite database file and returns a database handle. This is often the first UnQLite API call that an application makes and is a prerequisite for most other UnQLite APIs. Many UnQLite interfaces require a pointer to the database handle as their first parameter and can be thought of as methods on the database handle. This routine is the constructor for the database handle.
unqlite_kv_store() unqlite_kv_append() unqlite_kv_store_fmt() unqlite_kv_append_fmt() unqlite_kv_delete() unqlite_kv_fetch() unqlite_kv_fetch_callback()	Key/Value Store Interfaces Under the Key/Value store, both keys and values are treated as simple arrays of bytes, so content can anything from ASCII strings, binary blob and even disk files. These set of interfaces allow clients to store and retrieve raw database records efficiently regardless of the underlying KV storage engine.
unqlite_compile() unqlite_compile_file() unqlite_vm_config() unqlite_vm_exec() unqlite_vm_reset() unqlite_vm_extract_variable() unqlite_vm_release()	Document Store Interfaces These set of interfaces works with the unqlite_vm object which is obtained after successful compilation of the target Jx9 script. Jx9 is the scripting language which power the document-store interface to UnQLite. The Document-Store interface to UnQLite works as follows: Obtain a new database handle via unqlite_open(). Compile your Jx9 script using one of the compile iterfaces such as unqlite_compile() or unqlite_compile_file(). On successful compilation, the engine will automatically create an instance of this structure (unqlite_vm) and a pointer to this structure is made available to the caller. When something goes wrong during compilation of the target Jx9 script due to a compile-time error, the caller must discard this pointer and fix its erroneous Jx9 code. Compile-time error logs can be extracted via a call to unqlite_config() with a configuration verb set to UNQLITE_CONFIG_JX9_ERR_LOG. Optionally, configure the virtual machine using  unqlite_vm_config(). Optionally, register one or more foreign functions or constants using the unqlite_create_function() or unqlite_create_constant() interfaces. Execute the compiled Jx9 program by calling unqlite_vm_exec(). Optionally, extract the contents of one or more variables declared inside your Jx9 script using unqlite_vm_extract_variable(). Optionally, Reset the virtual machine using unqlite_vm_reset(), then go back to step 6. Do this zero or more times. Destroy the virtual machine using unqlite_vm_release().
unqlite_kv_cursor_init() unqlite_kv_cursor_first_entry() unqlite_kv_cursor_seek() unqlite_kv_cursor_last_entry() and many more	Database Cursors Cursors provide a mechanism by which you can iterate over the records in a database. Using cursors, you can seek, fetch, move, and delete database records.
unqlite_begin() unqlite_commit() unqlite_rollback()	Manual Transaction Manager These set of interfaces allow the host application to manually start a write-transaction. Note that UnQLite is smart enough and will automatically start a write-transaction in the background when needed and so call to these routines is usually not necessary.
unqlite_close()	This routine is the destructor for the unqlite handle obtained by a prior successful call to unqlite_open().  Each database connection must be closed in order to avoid memory leaks and malformed database image.

3.0 In-Process Extending

UnQLite includes interfaces that can be used to extend its functionality. Such routines includes unqlite_create_function() and unqlite_create_constant().

The unqlite_create_function() interface is used to install a foreign function (typically a C/C++ function) that can be invoked from your Jx9 code. The new function implementation typically makes use of the following additional interfaces.

All of the built-in Jx9 functions (over 312) of Jx9 such as db_create(), db_store(), db_fetch(), db_drop_record(), etc. are created using exactly this interface.

Refer to the following tutorial for a step by step guide on how to create, install and call foreign functions from your Jx9 script.

The unqlite_create_constant() interface is used to associate a constant name (An identifier in the Jx9 world such as JX9_EOL, JX9_OS, __TIME__, etc.) with a C callback which is responsible of expanding the constant name to the desired value such as 3,14 the value of PI.

Refer to the following tutorial for a step by step guide on how to create, install and expand constants from your Jx9 code.

4.0 Others Interfaces

This article only mentions the foundational UnQLite interfaces. The UnQLite library includes many other APIs implementing useful features that are not described here. A complete list of functions that form the UnQLite application programming interface is found at the C/C++ Interface Specification. Refer to that document for complete and authoritative information about all UnQLite interfaces.