Symisc UnQLite

An Embeddable NoSQL Database Engine
Star
Follow @symisc

unQLite in 5 minutes or Less.

Here is what you do to start experimenting with the UnQLite database engine without having to do a lot of tedious reading and configuration:

download The Code

Get a copy of the last public release of UnQLite (A single 1.8 MB C file). Visit the download page for additional information.

Write Programs That Use UnQLite

The principal task of a database engine is to store and retrieve records. UnQLite support both structured and raw database record storage.


Structured data storage is presented to clients via the Document store interface to UnQLite. The document store itself is used to store JSON docs (i.e. Objects, Arrays, Strings, etc.) in the database and is powered by the Jx9 programming language. Jx9 is an embeddable scripting language also called extension language designed to support general procedural programming with data description facilities. Jx9 is a Turing-Complete, dynamically typed programming language based on JSON and implemented as a library in the UnQLite core. You can find more on Jx9 at the Introduction to Jx9 page.


Raw data storage is presented to clients via the Key/Value store interfaces. UnQLite is a standard key/value store similar to Berkeley DB, Tokyo Cabinet, LevelDB, etc. but with a rich feature set including support for transactions (ACID), concurrent reader, etc. Under the KV store, both keys and values are treated as simple arrays of bytes, so content can be anything from ASCII strings, binary blob and even disk files.

Introduction to the Key/Value Store C/C++ Interfaces 

By far, the UnQLite Key/Value store layer is the simplest, yet the most flexible since it does not impose any structure or schema to the database records.

The Key/Value store layer is presented to clients via a set of interfaces, these includes: unqlite_kv_store(), unqlite_kv_append(), unqlite_kv_fetch_callback(), unqlite_kv_append_fmt(), and many more.


Below is a simple C program that demonstrates how to use the Key/Value C/C++ interface to UnQLite.

  1. #include <unqlite.h>
  2. int i,rc;
  3. unqlite *pDb;

  5.  // Open our database;
  6.  rc = unqlite_open(&pDb,"test.db",UNQLITE_OPEN_CREATE);
  7.  if( rc != UNQLITE_OK ){ return; }

  9.  // Store some records
  10.  rc = unqlite_kv_store(pDb,"test",-1,"Hello World",11); //test => 'Hello World'
  11.  if( rc != UNQLITE_OK ){
  12.    //Insertion fail, Hande error (See below)
  13.    return;
  14. }
  15.  // A small formatted string
  16.  rc = unqlite_kv_store_fmt(pDb,"date",-1,"Current date: %d:%d:%d",2013,06,07);
  17.  if( rc != UNQLITE_OK ){
  18.    //Insertion fail, Hande error (See below)
  19.    return;
  20.  }

  22.  //Switch to the append interface
  23.   rc = unqlite_kv_append(pDb,"msg",-1,"Hello, ",7); //msg => 'Hello, '
  24.   if( rc == UNQLITE_OK ){
  25.    //The second chunk
  26.    rc = unqlite_kv_append(pDb,"msg",-1,"Current time is: ",17); //msg => 'Hello, Current time is: '
  27.    if( rc == UNQLITE_OK ){
  28.      //The last formatted chunk
  29.      rc = unqlite_kv_append_fmt(pDb,"msg",-1,"%d:%d:%d",10,16,53); //msg => 'Hello, Current time is: 10:16:53'
  30.    }
  31.   }

  33. //Delete a record
  34. unqlite_kv_delete(pDb,"test",-1);

  36. //Store 20 random records.
  37. for(i = 0 ; i < 20 ; ++i ){
  38.   char zKey[12]; //Random generated key
  39.   char zData[34]; //Dummy data

  41.   // generate the random key
  42.   unqlite_util_random_string(pDb,zKey,sizeof(zKey));
  43.  
  44.   // Perform the insertion
  45.   rc = unqlite_kv_store(pDb,zKey,sizeof(zKey),zData,sizeof(zData));
  46.   if( rc != UNQLITE_OK ){
  47.    break;
  48.   }
  49. }

  51. if( rc != UNQLITE_OK ){
  52.  //Insertion fail, Handle error
  53.   const char *zBuf;
  54.   int iLen;
  55.   /* Something goes wrong, extract the database error log */
  56.   unqlite_config(pDb,UNQLITE_CONFIG_ERR_LOG,&zBuf,&iLen);
  57.   if( iLen > 0 ){
  58.     puts(zBuf);
  59.   }
  60.   if( rc != UNQLITE_BUSY && rc != UNQLITE_NOTIMPLEMENTED ){
  61.     /* Rollback */
      unqlite_rollback(pDb);
  62.   }
  63. }

  65. //Auto-commit the transaction and close our handle.
  66. unqlite_close(pDb);


Get a copy of the C File


Under the KV store, since keys and values are treated as a simple array of bytes, you can even insert external files perhaps some XML or any other files you need in your UnQLite database and turn it (i.e. The UnQLite database) into a Tar archive with O(1) lookup. Here is a snippet.


  1. #include <unqlite.h>
  2. // Usage example: ./unqlite_tar config.xml license.txt audio.wav splash.jpeg,...
  3. void *pMap;
  4. unqlite_int64 iSize;
  5. int rc,i;
  6. unqlite *pDb;

  8. // Open our database;
  9. rc = unqlite_open(&pDb,"test.db",UNQLITE_OPEN_CREATE);

  11. if( rc != UNQLITE_OK ){ /* Handle error */ return; }

  13. // Store the given files in our database
  14. for( i = 1 ; i < argc ; ++i ){
  15.    const char *zName = argv[i]; //Name of the target file
  16.  
  17. // Obtain a read-only memory view of the target file;
  18.   rc = unqlite_util_load_mmaped_file(zName,&pMap,&iSize);
  19.   if( rc != UNQLITE_OK ){ /* Handle error */ return; }
  20.  
  21. // Store the whole file in our database;
  22.    rc = unqlite_kv_store(pDb,zName,-1,pMap,iSize);
  23.   if( rc != UNQLITE_OK ){ /* Handle error */ return; }
  24.  
  25. // Discard the memory view;
  26.   unqlite_util_release_mmaped_file(pMap,iSize);
  27. }


  30. //Auto-commit the transaction and close our handle
  31. unqlite_close(pDb);


Get a copy of the C File


The function calls to pay attention to here are the call to unqlite_util_load_mmaped_file() on line 18 which is used to obtain a read-only memory view of the whole file. After that, a standard KV insertion is done using a call to unqlite_kv_store(). Don't forget to release the memory view of the target file and closing your database.


Fetching data under the KV store is straightforward and involve only a single call to: unqlite_kv_fetch_callback() or unqlite_kv_fetch(). The unqlite_kv_fetch_callback() interface is the recommended one where the caller supply a simple callback responsible of consuming the record data perhaps redirecting it (i.e. Record data) to its standard output (STDOUT) or to the connected peer (See interface documentation for a working example). The unqlite_kv_fetch() interface should be familiar to everyone where it expects a user supplied buffer (either statistically or dynamically allocated) where record data is copied in.

Using Database Cursors

Cursors provide a mechanism by which you can iterate over the records in a database. Using cursors, you can seek, fetch, move, and delete database records. 

For an introduction to the cursor interfaces, please refer to their API documentation available here.

Below is a simple C program that demonstrates how to use cursors. This program will iterate over the database from the last record to the first.


#include <unqlite.h>

 int rc;
unqlite *pDb;
unqlite_kv_cursor *pCursor;
unqlite_int64 iData;

// Open our database;
rc = unqlite_open(&pDb,"test.db",UNQLITE_OPEN_CREATE);
if( rc != UNQLITE_OK ){ return; }

 //Store some records unqlite_kv_store(), unqlite_kv_append()...

/* Allocate a new cursor instance */

rc = unqlite_kv_cursor_init(pDb,&pCursor);

if( rc != UNQLITE_OK ){ return; }


/* Point to the last record */
rc = unqlite_kv_cursor_last_entry(pCursor);
if( rc != UNQLITE_OK ){ return; }

/* Iterate over the records */

while( unqlite_kv_cursor_valid_entry(pCursor) ){

      /* Consume the key */

   printf("\nKey ==>\n\t");

   unqlite_kv_cursor_key_callback(pCursor,DataConsumerCallback,0);


   /* Extract data length */

   unqlite_kv_cursor_data(pCursor,NULL,&iData);

   printf("\nData length ==> %lld\n\t",iData);

      

   /* Consume the data */

   unqlite_kv_cursor_data_callback(pCursor,DataConsumerCallback,0);

 

   /* Point to the previous record */

   unqlite_kv_cursor_prev_entry(pCursor);

}


/* Finally, Release our cursor */

unqlite_kv_cursor_release(pDb,pCursor);


//Auto-commit the transaction and close our handle
unqlite_close(pDb);
Get a copy of the C File

Introduction to the Document-Store C/C++ Interfaces

Structured data storage is presented to clients via the Document store interface to UnQLite. The Document-store itself is used to store JSON docs (i.e. Objects, Arrays, Strings, etc.) in the database and is powered by the Jx9 programming language.
The following is the Jx9 script to compile by the C program defined below that demonstrates how to use the document store interfaces to UnQLite.


/* Create the collection 'users'  */

if( !db_exists('users') ){

    /* Try to create it */

   $rc = db_create('users');

   if ( !$rc ){

     //Handle error

      print db_errlog();

   return;

   }

}

//The following is the JSON objects to be stored shortly in our 'users' collection

$zRec = [

{

   name : 'james',

   age  : 27,

   mail : 'dude@example.com'

},

{

   name : 'robert',

   age  : 35,

   mail : 'rob@example.com'

},

{

   name : 'monji',

   age  : 47,

   mail : 'monji@example.com'

},
{
  name : 'barzini',

  age  : 52,

  mail : 'barz@mobster.com'

}

];

//Store our records

$rc = db_store('users',$zRec);
if( !$rc ){
 //Handle error

    print db_errlog();

  return;

}
//One more record
$rc = db_store('users',{ name : 'alex', age : 19, mail : 'alex@example.com'  });
if( !$rc ){
 //Handle error

    print db_errlog();

  return;

}
print "Total number of stored records: ",db_total_records('users'),JX9_EOL;

//Fetch data using db_fetch_all(), db_fetch_by_id() and db_fetch().

Now, the C program which compile and executes the Jx9 script defined above.

  1. #include <unqlite.h>

  3. int rc;
  4. unqlite *pDb;
  5. unqlite_vm *pDb;

  7. // Open our database;
  8. rc = unqlite_open(&pDb,"test.db",UNQLITE_OPEN_CREATE);
  9. if( rc != UNQLITE_OK ){ /* Handle error */ return;

  11. /* Compile the Jx9 script defined above */

  12. rc = unqlite_compile(pDb,JX9_PROG,sizeof(JX9_PROG)-1,&pVm);

  13. if( rc != UNQLITE_OK ){

  14.   if( rc == UNQLITE_COMPILE_ERR ){

  15.      const char *zBuf;

  16.      int iLen;

  17.      /* Compile-time error, extract the compiler error log */

  18.      unqlite_config(pDb,UNQLITE_CONFIG_JX9_ERR_LOG,&zBuf,&iLen);

  19.      if( iLen > 0 ){

  20.        puts(zBuf);

  21.      }

  22.   }

  23.  return;

  24. }


  26. /* Install a VM output consumer callback */

  27. rc = unqlite_vm_config(pVm,UNQLITE_VM_CONFIG_OUTPUT,OutputConsumer,0);

  28. if( rc != UNQLITE_OK ){

  29.  return;

  30. }


  32. /* Execute our script */

  33. rc = unqlite_vm_exec(pVm);

  34. if( rc != UNQLITE_OK ){

  35.  return;

  36. }


  38. /* Finally, release our VM */

  39. unqlite_vm_release(pVm);


  41. unqlite_close(pDb);

Get a copy of the C File


The Document-Store interface to UnQLite works as follows:

  1. Obtain a new database handle via unqlite_open() (line 8 in our example).

  2. Compile your Jx9 script (line 12 in our example) using one of the compile iterfaces such as unqlite_compile() or unqlite_compile_file(). On successful compilation, the engine will automatically create an instance of the structure unqlite_vm and a pointer to this structure is made available to the caller.

  3. When something goes wrong during compilation of the target Jx9 script due to a compile-time error, the caller must discard this pointer and fix its erroneous Jx9 code. Compile-time error logs can be extracted (line 18 in our example) via a call to unqlite_config() with a configuration verb set to UNQLITE_CONFIG_JX9_ERR_LOG.

  4. Optionally, configure the virtual machine using  unqlite_vm_config().

  5. Optionally, register one or more foreign functions or constants using the unqlite_create_function() or unqlite_create_constant() interfaces.

  6. Execute the compiled Jx9 program (line 33 in our example) by calling unqlite_vm_exec().

  7. Optionally, extract the contents of one or more variables declared inside your Jx9 script using unqlite_vm_extract_variable().

  8. Optionally, Reset the virtual machine using unqlite_vm_reset(), then go back to step 6. Do this zero or more times.

  9. Destroy the virtual machine (line 39 in our example) using unqlite_vm_release().

Other useful Links to start with

Check out the Introduction To The UnQLite C/C++ Interface for an introductory overview and roadmap to the dozens of UnQLite interface functions. A separate document, The UnQLite C/C++ Interface, provides detailed specifications for all of the various C/C++ APIs for UnQLite. Once the reader understands the basic principles of operation for UnQLite, that document should be used as a reference guide.

Any questions, visit the Support Page for additional information.


Interested in Machine Learning & Computer Vision? try out Pixlab

PixLab
Symisc Systems
Copyright © Symisc Systems