Merge pull request #256

7d8acd1 explicitly disable AES on ARM (Riccardo Spagni)
4222b37 add arm6 target and flags (Riccardo Spagni)
5675325 new makefile targets for static builds (Riccardo Spagni)
9519526 Only compile BerkeleyDB as an option in non-static (Thomas Winget)
7b14d4a Steps toward multiple dbs available -- working (Thomas Winget)
dbdcf11 blockchain_converter: Add support for resume from last block (warptangent)
4880803 blockchain_import: lengthen string for line clear (warptangent)
7476d2e blockchain_export: show progress during export (warptangent)
4bedd68 Update Blockchain::get_db() to return reference instead of pointer (warptangent)
0386e99 Add README for blockchain converter, importer, and exporter utilities (warptangent)
ead7fad BerkeleyDB implementation of BlockchainDB seems to be working! (Thomas Winget)
ffadb65 blockchain_export: Add compile-time support for BlockchainDB (warptangent)
f6cbfb6 Add blockchain_export utility (warptangent)
275cbd4 Add support for database open with flags (warptangent)
cb862cb Add mdb_flags variable to LMDB database open (warptangent)
260cc56 Add blockchain_import utility (warptangent)
ca75b47 Blockchain: add get_db() accessor, needed for blockchain_import (warptangent)
a3dd9d1 blockchain_converter: Add support for batch transactions (warptangent)
acb5d29 Update and relocate comment that applies class wide (warptangent)
bd9da6d Merges #33 (Thomas Winget)
6e9e8ab Move db_drivers/ to external/ (Thomas Winget)
56a1bf5 fixed BUILD_64 option (Riccardo Spagni)
c97a685 fixed arch_width option (Riccardo Spagni)
3a3c07c fixed msys2 / mingw folders based on architecture, added license to unbound CMakeList as that is not part of standard Unbound (Riccardo Spagni)
3b3da86 moved 32-bit/64-bit detection into main CMakeLists (Riccardo Spagni)
a0cbf73 removed findlmdb.cmake, no longer required (Riccardo Spagni)
0e8bbdb 32-bit LMDB, removed check for in-system LMDB as it is consensus-critical (Riccardo Spagni)
11e8157 Revert "Moved db_drivers/ into external/ for consistency" (Thomas Winget)
daf2a8f Revert "Build fixed, goofed up some CMake" (Thomas Winget)
e146027 BlockchainBDB passes unit tests (Thomas Winget)
43477b7 BerkeleyDB Blockchain building, not working yet (Thomas Winget)
cade0da CMake wiring, minor cleanup, minor test addition (Thomas Winget)
1bc8939 BerkeleyDB BlockchainDB impl copy/paste/modify (Thomas Winget)
5112dc3 Try to not pollute cryptonote namespace (Thomas Winget)
edef0bb Initial commit of BDB BlockchainDB implementation (Thomas Winget)
eee3ee7 BlockchainDB implementations have names now (Thomas Winget)
5d2a2b7 Fixed includes in BlockchainDB unit tests (Thomas Winget)
8b82f3c Build fixed, goofed up some CMake (Thomas Winget)
b213356 Moved db_drivers/ into external/ for consistency (Thomas Winget)
5eab480 Moved BlockchainDB into its own src/ subfolder (Thomas Winget)
0ad0784 Changed log level of debug message -- too spammy (Thomas Winget)
6485dac BlockchainLMDB: Add profiling to tx_exists() (warptangent)
83fb6d8 BlockchainLMDB: Add batch transaction support to tx_exists() (warptangent)
8529c0e BlockchainDB, BlockchainLMDB: Add profiling for DB commits (warptangent)
7a66b8b BlockchainDB: Add virtual function declarations for batch transactions (warptangent)
b7a2d84 BlockchainLMDB: Add check for open database to two functions (warptangent)
58ecc58 BlockchainLMDB: Add support for batch transactions (warptangent)
8909d7d Improve block and tx processing efficiency by less repeat hashing (warptangent)
3676ac5 Add profiling to block and tx processing (warptangent)
ce71abd Move LMDB storage to subfolder (warptangent)
42f8fe5 Fix formatting (warptangent)
aa82f78 Fix log statement (warptangent)
26873db Remove unused variable (warptangent)
4b90fd3 Add log statement (warptangent)
2531aa3 Add and extend log statements (warptangent)
59305d3 Blockchain: match original function declaration from blockchain_storage (warptangent)
cd972bd Update year and formatting in license (warptangent)
b88ab64 Fix Blockchain::get_tail_id() to set parameter to last block number instead of height (warptangent)
963bc09 Revert "Bounds error, should fix #27" (warptangent)
6f1c4b4 Bounds error, should fix #27 (Thomas Winget)
84fe5fb Add compile-time support for both db implementations: in-memory and LMDB (warptangent)
8bd1983 Blockchain: reflect log updates from blockchain_storage (warptangent)
7f9b070 Blockchain: reflect log and assert updates from blockchain_storage (warptangent)
70342ec Blockchain: reflect log level of blockchain_storage (warptangent)
c8d27fb Blockchain: reflect assert behavior of blockchain_storage for get_tx_outputs_gindexs() (warptangent)
d00ee78 Update recently added log statement to fix possible null dereference (warptangent)
acd4c36 Should fix std::min issues related to size_t (Thomas Winget)
3488359 Add in-source lmdb to build process (Thomas Winget)
4e45fae Initial import of lmdb source (Thomas Winget)
800d9b9 Remove code previously made unused and marked unused (warptangent)
0840c2f Fix height assertion in Blockchain::handle_alternative_block() (warptangent)
63051be Fix comparison between main and alternate chain's cumulative difficulty. (warptangent)
909ea81 Remove a have_block() check so alternate block can be processed (warptangent)
4d0a94b Complete implementation of transaction removal (warptangent)
1701c26 Use block index when obtaining block's difficulty for log statement (warptangent)
4eba21f Fix transfers to support mixins (warptangent)
d045dfa Fix transfers (without mixins) (warptangent)
429a740 throw inline functions need to keep exception type (Thomas Winget)
14555ee Fixes segfault in Blockchain::handle_alternative_block (Thomas Winget)
5086ca1 add BlockchainDB tests to new cmake (Thomas Winget)
c5c100c Obtain tx hash and tx output index from amount and output offset (warptangent)
ad8200a db_lmdb: fix global index calculation off by 1 (moneromooo-monero)
57b80c5 db_lmdb: remove redundant checks (moneromooo-monero)
1362846 blockchain_converter: add --testnet for converting testnet blockchain (moneromooo-monero)
c50cd95 Fixes a bug with getting output metadata from BlockchainDB (Thomas Winget)
c3fa07b update comments to reflect changed code (Thomas Winget)
59d2b0e db_lmdb: do not give the group database write permissions (moneromooo-monero)
4c2a452 db_lmdb: catch attempt to remove block from an empty blockchain (moneromooo-monero)
3a3459d db_lmdb: factor all the log+throw code paths (moneromooo-monero)
3fcb8da db_lmdb: factor the MDB_val setup code (moneromooo-monero)
609cf7f blockchain_converter: a bit more user friendly output (moneromooo-monero)
2b9f737 blockchain_converter: only call data path function once (moneromooo-monero)
1860658 blockchain: do not append "testnet" to the data directory (moneromooo-monero)
1c578ad db_lmdb: remove block timestamp too when removing a block (moneromooo-monero)
c93a186 db_lmdb: do not cast const away (moneromooo-monero)
198368b blockchain: fix wallet syncing from scratch (moneromooo-monero)
1d23db2 db_lmdb: do not keep a dangling pointer to stack objects (moneromooo-monero)
29b5876 db_lmdb: make cursor internal members private (moneromooo-monero)
a3157d7 blockchain_storage: refactor genesis block creation (moneromooo-monero)
8e41b1e blockchain_storage: add consts where appropriate (moneromooo-monero)
98bdadc blockchain_converter: delete blockchain on succesful exit (moneromooo-monero)
256162f checkpoints: add consts where appropriate (moneromooo-monero)
b7270ab blockchain: add consts where appropriate (moneromooo-monero)
23f3cb4 blockchain_db: add consts where appropriate (moneromooo-monero)
10fd6ca blockchain_db: factor some exception code (moneromooo-monero)
11129b9 blockchain_converter: use the actual blockchain location (moneromooo-monero)
0886183 build: add liblmdb to the cmake autodetection system (moneromooo-monero)
215e63b extraneous semicolon in Blockchain::complete_timestamps_vector (Thomas Winget)
8e1b7e2 raised maximum mapsize for lmdb to ~16GB (Thomas Winget)
6c8b8ac more blockchain height-related fixes, syncing other nodes code this time (Thomas Winget)
9455e0c ~ didn't work, need hard path.  debug print. (Thomas Winget)
4af0918 very, VERY primitive blockchain converter (Thomas Winget)
26a7db3 add new checkpointing behavior to Blockchain class (Thomas Winget)
767aac2 Remove unused dependency (Thomas Winget)
006e106 Store output pubkeys separately, bug fixes (Thomas Winget)
ab7951d Minor bugfixes, redundancy removal (Thomas Winget)
71b18d7 moar bug fixes, removed debug prints (Thomas Winget)
0915913 BlockchainLMDB seems to be working*! (Thomas Winget)
1a546e3 some bug fixes, but still needs work (Thomas Winget)
006afe2 Minor bug fixes and debug prints (Thomas Winget)
90f402e minor fixes to Blockchain.cpp (Thomas Winget)
74a1a89 Integrate BlockchainDB into cryptonote_core (Thomas Winget)
d8c570b All LMDB BlockchainDB implemented, not tested (Thomas Winget)
e47e343 LMDB blockchain: remove outputs and spent keys (Thomas Winget)
a0af217 Adding block data to LMDB BlockchainDB coded (Thomas Winget)
db00ce0 Parts of LMDB impl of BlockchainDB done and working (Thomas Winget)
1240cf8 BlockchainDB unit tests, lmdb linker flag (Thomas Winget)
b98b964 Initial commit of lmdb BlockchainDB impl (Thomas Winget)
bc44bc1 Initial commit of BlockchainDB tests, other misc (Thomas Winget)
90d6f8b Adding libglim as an external library (Thomas Winget)
07733f9 update new blockchain to build with new changes (Thomas Winget)
1ffbeb2 stupid past me, fixing typos and shit... (Thomas Winget)
67515b8 missing typedef (Thomas Winget)
aba548c import of BlockchainDB files (Thomas Winget)

CMakeLists.txt

@@ -79,6 +79,20 @@ enable_testing()
 
 option(BUILD_DOCUMENTATION "Build the Doxygen documentation." ON)
 
+# Check whether we're on a 32-bit or 64-bit system
+if(CMAKE_SIZEOF_VOID_P EQUAL "8")
+  set(DEFAULT_BUILD_64 ON)
+else()
+  set(DEFAULT_BUILD_64 OFF)
+endif()
+option(BUILD_64 "Build for 64-bit? 'OFF' builds for 32-bit." ${DEFAULT_BUILD_64})
+
+if(BUILD_64)
+  set(ARCH_WIDTH "64")
+else()
+  set(ARCH_WIDTH "32")
+endif()
+message(STATUS "Building for a ${ARCH_WIDTH}-bit system")
 
 # Check if we're on FreeBSD so we can exclude the local miniupnpc (it should be installed from ports instead)
 # CMAKE_SYSTEM_NAME checks are commonly known, but specifically taken from libsdl's CMakeLists
@@ -111,16 +125,16 @@ endif()
 option(STATIC "Link libraries statically" ${DEFAULT_STATIC})
 
 if(MINGW)
-  get_filename_component(msys2_install_path "[HKEY_CURRENT_USER\\Software\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\MSYS2 64bit;InstallLocation]" ABSOLUTE)
+  get_filename_component(msys2_install_path "[HKEY_CURRENT_USER\\Software\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\MSYS2 ${ARCH_WIDTH}bit;InstallLocation]" ABSOLUTE)
-  set(CMAKE_INCLUDE_PATH "${msys2_install_path}/mingw64/include")
+  set(CMAKE_INCLUDE_PATH "${msys2_install_path}/mingw${ARCH_WIDTH}/include")
   # This is necessary because otherwise CMake will make Boost libraries -lfoo
   # rather than a full path. Unfortunately, this makes the shared libraries get
   # linked due to a bug in CMake which misses putting -static flags around the
   # -lfoo arguments.
   list(REMOVE_ITEM CMAKE_C_IMPLICIT_LINK_DIRECTORIES
-    "${msys2_install_path}/mingw64/lib")
+    "${msys2_install_path}/mingw${ARCH_WIDTH}/lib")
   list(REMOVE_ITEM CMAKE_CXX_IMPLICIT_LINK_DIRECTORIES
-    "${msys2_install_path}/mingw64/lib")
+    "${msys2_install_path}/mingw${ARCH_WIDTH}/lib")
 endif()
 
 if(STATIC)
@@ -131,6 +145,27 @@ if(STATIC)
   endif()
 endif()
 
+# default database:
+# should be lmdb for testing, memory for production still
+# set(DATABASE memory)
+set(DATABASE lmdb)
+
+if (DEFINED ENV{DATABASE})
+  set(DATABASE $ENV{DATABASE})
+  message(STATUS "DATABASE set: ${DATABASE}")
+else()
+  message(STATUS "Could not find DATABASE in env (not required unless you want to change database type from default: ${DATABASE})")
+endif()
+if (DATABASE STREQUAL "lmdb")
+  set(BLOCKCHAIN_DB DB_LMDB)
+elseif (DATABASE STREQUAL "memory")
+  set(BLOCKCHAIN_DB DB_MEMORY)
+else()
+  die("Invalid database type: ${DATABASE}")
+endif()
+
+add_definitions("-DBLOCKCHAIN_DB=${BLOCKCHAIN_DB}")
+
 if (UNIX AND NOT APPLE)
   # Note that at the time of this writing the -Wstrict-prototypes flag added below will make this fail
   find_package(Threads)
@@ -153,6 +188,14 @@ link_directories(${UNBOUND_LIBRARY_DIRS})
 # Final setup for rapidjson
 include_directories(external/rapidjson)
 
+# Final setup for liblmdb
+include_directories(${LMDB_INCLUDE})
+
+# Final setup for Berkeley DB
+if (NOT STATIC)
+  include_directories(${BDB_INCLUDE})
+endif()
+
 if(MSVC)
   add_definitions("/bigobj /MP /W3 /GS- /D_CRT_SECURE_NO_WARNINGS /wd4996 /wd4345 /D_WIN32_WINNT=0x0600 /DWIN32_LEAN_AND_MEAN /DGTEST_HAS_TR1_TUPLE=0 /FIinline_c.h /D__SSE4_1__")
   # set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} /Dinline=__inline")
@@ -170,9 +213,9 @@ else()
   else()
     set(ARCH_FLAG "-march=${ARCH}")
   endif()
-  set(WARNINGS "-Wall -pedantic -Wextra -Wpointer-arith -Wundef -Wvla -Wwrite-strings -Wno-error=extra -Wno-error=deprecated-declarations -Wno-error=sign-compare -Wno-error=strict-aliasing -Wno-error=type-limits -Wno-unused-parameter -Wno-error=unused-variable -Wno-error=undef -Wno-error=uninitialized")
+  set(WARNINGS "-Wall -Wextra -Wpointer-arith -Wundef -Wvla -Wwrite-strings -Wno-error=extra -Wno-error=deprecated-declarations -Wno-error=sign-compare -Wno-error=strict-aliasing -Wno-error=type-limits -Wno-unused-parameter -Wno-error=unused-variable -Wno-error=undef -Wno-error=uninitialized")
   if(NOT MINGW)
- #   set(WARNINGS "${WARNINGS} -Werror") # to allow pedantic but not stop compilation
+    set(WARNINGS "${WARNINGS} -Werror") # to allow pedantic but not stop compilation
   endif()
   if(CMAKE_C_COMPILER_ID STREQUAL "Clang")
     set(WARNINGS "${WARNINGS} -Wno-error=mismatched-tags -Wno-error=null-conversion -Wno-overloaded-shift-op-parentheses -Wno-error=shift-count-overflow -Wno-error=tautological-constant-out-of-range-compare -Wno-error=unused-private-field -Wno-error=unneeded-internal-declaration")
@@ -207,7 +250,15 @@ else()
     set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -std=c11 -D_GNU_SOURCE ${MINGW_FLAG} ${STATIC_ASSERT_FLAG} ${WARNINGS} ${C_WARNINGS} ${ARCH_FLAG} -maes")
     set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11 -D_GNU_SOURCE ${MINGW_FLAG} ${WARNINGS} ${CXX_WARNINGS} ${ARCH_FLAG} -maes")
   endif()
-  
+
+  string(SUBSTRING ${ARCH} 0 3 ARM_TEST)
+  string(TOLOWER ${ARM_TEST} ARM_TEST)
+  if(${ARM_TEST} STREQUAL "arm")
+    message(STATUS "Setting ARM C and C++ flags")
+    set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -mfpu=vfp -mfloat-abi=hard")
+    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -mfpu=vfp -mfloat-abi=hard")
+  endif()
+
   if(APPLE)
     set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -DGTEST_HAS_TR1_TUPLE=0")
   endif()

Makefile

@@ -30,9 +30,27 @@ release-all:
 	mkdir -p build/release
 	cd build/release && cmake -D BUILD_TESTS=ON -D CMAKE_BUILD_TYPE=release ../.. && $(MAKE)
 
-release-static:
+release-static: release-static-64
+
+release-static-64:
 	mkdir -p build/release
-	cd build/release && cmake -D STATIC=ON -D ARCH="x86-64" -D CMAKE_BUILD_TYPE=release ../.. && $(MAKE)
+	cd build/release && cmake -D STATIC=ON -D ARCH="x86-64" -D BUILD_64=ON -D CMAKE_BUILD_TYPE=release ../.. && $(MAKE)
+
+release-static-32:
+	mkdir -p build/release
+	cd build/release && cmake -D STATIC=ON -D ARCH="i686" -D BUILD_64=OFF -D CMAKE_BUILD_TYPE=release ../.. && $(MAKE)
+
+release-static-arm6:
+	mkdir -p build/release
+	cd build/release && cmake -D STATIC=ON -D ARCH="armv6zk" -D BUILD_64=OFF -D NO_AES=ON -D CMAKE_BUILD_TYPE=release ../.. && $(MAKE)
+
+release-static-win64:
+	mkdir -p build/release
+	cmake -G "MSYS Makefiles" -D STATIC=ON -D ARCH="x86-64" -D BUILD_64=ON -D CMAKE_BUILD_TYPE=Release -D CMAKE_TOOLCHAIN_FILE=../cmake/64-bit-toolchain.cmake -D MSYS2_FOLDER=c:/msys64 ../.. && $(MAKE)
+
+release-static-win32:
+	mkdir -p build/release
+	cmake -G "MSYS Makefiles" -D STATIC=ON -D ARCH="i686" -D BUILD_64=OFF -D CMAKE_BUILD_TYPE=Release -D CMAKE_TOOLCHAIN_FILE=../cmake/32-bit-toolchain.cmake -D MSYS2_FOLDER=c:/msys32 ../.. && $(MAKE)
 
 clean:
 	@echo "WARNING: Back-up your wallet if it exists within ./build!" ; \

cmake/FindBerkeleyDB.cmake

@@ -0,0 +1,25 @@
+# - Try to find Berkeley DB
+# Once done this will define
+#
+#  BERKELEY_DB_FOUND - system has Berkeley DB
+#  BERKELEY_DB_INCLUDE_DIR - the Berkeley DB include directory
+#  BERKELEY_DB_LIBRARIES - Link these to use Berkeley DB
+#  BERKELEY_DB_DEFINITIONS - Compiler switches required for using Berkeley DB
+
+# Copyright (c) 2006, Alexander Dymo, <adymo@kdevelop.org>
+#
+# Redistribution and use is allowed according to the terms of the BSD license.
+# For details see the accompanying COPYING-CMAKE-SCRIPTS file.
+
+find_path(BERKELEY_DB_INCLUDE_DIR db_cxx.h
+  /usr/include/db4
+  /usr/local/include/db4
+)
+
+find_library(BERKELEY_DB_LIBRARIES NAMES db_cxx )
+
+include(FindPackageHandleStandardArgs)
+find_package_handle_standard_args(Berkeley "Could not find Berkeley DB >= 4.1" BERKELEY_DB_INCLUDE_DIR BERKELEY_DB_LIBRARIES)
+# show the BERKELEY_DB_INCLUDE_DIR and BERKELEY_DB_LIBRARIES variables only in the advanced view
+mark_as_advanced(BERKELEY_DB_INCLUDE_DIR BERKELEY_DB_LIBRARIES )
+

external/CMakeLists.txt

@@ -97,3 +97,5 @@ else()
     die("Found libunbound includes, but could not find libunbound library. Please make sure you have installed libunbound or libunbound-dev or the equivalent")
   endif()
 endif()
+
+add_subdirectory(db_drivers)

external/db_drivers/CMakeLists.txt

@@ -0,0 +1,59 @@
+# Copyright (c) 2014-2015, The Monero Project
+# 
+# All rights reserved.
+# 
+# Redistribution and use in source and binary forms, with or without modification, are
+# permitted provided that the following conditions are met:
+# 
+# 1. Redistributions of source code must retain the above copyright notice, this list of
+#    conditions and the following disclaimer.
+# 
+# 2. Redistributions in binary form must reproduce the above copyright notice, this list
+#    of conditions and the following disclaimer in the documentation and/or other
+#    materials provided with the distribution.
+# 
+# 3. Neither the name of the copyright holder nor the names of its contributors may be
+#    used to endorse or promote products derived from this software without specific
+#    prior written permission.
+# 
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+# THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+# THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+# We aren't even going to check the system for an installed LMDB driver, as it is too
+# critical a consensus component to rely on dynamically linked libraries
+message(STATUS "Using ${ARCH_WIDTH}-bit LMDB from source tree")
+add_subdirectory(liblmdb${ARCH_WIDTH})
+set(LMDB_INCLUDE "${CMAKE_CURRENT_SOURCE_DIR}/liblmdb${ARCH_WIDTH}" CACHE STRING "LMDB Include path")
+
+set(LMDB_LIBRARY "lmdb" CACHE STRING "LMDB Library name")
+
+if (NOT STATIC)
+  find_package(BerkeleyDB)
+
+  if(NOT BERKELEY_DB_LIBRARIES OR STATIC)
+    add_subdirectory(libdb)
+    message(STATUS "BerkeleyDB not found, building from src tree")
+
+    set(BDB_STATIC true CACHE BOOL "BDB Static flag")
+    set(BDB_INCLUDE "${CMAKE_CURRENT_SOURCE_DIR}/libdb" CACHE STRING "BDB include path")
+    set(BDB_LIBRARY "db" CACHE STRING "BDB library name")
+  else()
+    message(STATUS "Found BerkeleyDB include (db.h) in ${BERKELEY_DB_INCLUDE_DIR}")
+    if(BERKELEY_DB_LIBRARIES)
+      message(STATUS "Found BerkeleyDB shared library")
+      set(BDB_STATIC false CACHE BOOL "BDB Static flag")
+      set(BDB_INCLUDE ${BERKELEY_DB_INCLUDE_DIR} CACHE STRING "BDB include path")
+      set(BDB_LIBRARY ${BERKELEY_DB_LIBRARIES} CACHE STRING "BDB library name")
+      set(BDB_LIBRARY_DIRS "" CACHE STRING "BDB Library dirs")
+    else()
+      die("Found BerkeleyDB includes, but could not find BerkeleyDB library. Please make sure you have installed libdb and libdb-dev or the equivalent")
+    endif()
+  endif()
+endif()

external/db_drivers/liblmdb32/.gitignore

@@ -0,0 +1,20 @@
+mtest
+mtest[23456]
+testdb
+mdb_copy
+mdb_stat
+*.[ao]
+*.so
+*.exe
+*[~#]
+*.bak
+*.orig
+*.rej
+*.gcov
+*.gcda
+*.gcno
+core
+core.*
+valgrind.*
+man/
+html/

external/db_drivers/liblmdb32/CHANGES

@@ -0,0 +1,112 @@
+LMDB 0.9 Change Log
+
+LMDB 0.9.14 Release (2014/09/20)
+	Fix to support 64K page size (ITS#7713)
+	Fix to persist decreased as well as increased mapsizes (ITS#7789)
+	Fix cursor bug when deleting last node of a DUPSORT key
+	Fix mdb_env_info to return FIXEDMAP address
+	Fix ambiguous error code from writing to closed DBI (ITS#7825)
+	Fix mdb_copy copying past end of file (ITS#7886)
+	Fix cursor bugs from page_merge/rebalance
+	Fix to dirty fewer pages in deletes (mdb_page_loose())
+	Fix mdb_dbi_open creating subDBs (ITS#7917)
+	Fix mdb_cursor_get(_DUP) with single value (ITS#7913)
+	Fix Windows compat issues in mtests (ITS#7879)
+	Add compacting variant of mdb_copy
+	Add BigEndian integer key compare code
+	Add mdb_dump/mdb_load utilities
+
+LMDB 0.9.13 Release (2014/06/18)
+	Fix mdb_page_alloc unlimited overflow page search
+	Documentation
+		Re-fix MDB_CURRENT doc (ITS#7793)
+		Fix MDB_GET_MULTIPLE/MDB_NEXT_MULTIPLE doc
+
+LMDB 0.9.12 Release (2014/06/13)
+	Fix MDB_GET_BOTH regression (ITS#7875,#7681)
+	Fix MDB_MULTIPLE writing multiple keys (ITS#7834)
+	Fix mdb_rebalance (ITS#7829)
+	Fix mdb_page_split (ITS#7815)
+	Fix md_entries count (ITS#7861,#7828,#7793)
+	Fix MDB_CURRENT (ITS#7793)
+	Fix possible crash on Windows DLL detach
+	Misc code cleanup
+	Documentation
+		mdb_cursor_put: cursor moves on error (ITS#7771)
+
+
+LMDB 0.9.11 Release (2014/01/15)
+	Add mdb_env_set_assert() (ITS#7775)
+	Fix: invalidate txn on page allocation errors (ITS#7377)
+	Fix xcursor tracking in mdb_cursor_del0() (ITS#7771)
+	Fix corruption from deletes (ITS#7756)
+	Fix Windows/MSVC build issues
+	Raise safe limit of max MDB_MAXKEYSIZE
+	Misc code cleanup
+	Documentation
+		Remove spurious note about non-overlapping flags (ITS#7665)
+
+LMDB 0.9.10 Release (2013/11/12)
+	Add MDB_NOMEMINIT option
+	Fix mdb_page_split() again (ITS#7589)
+	Fix MDB_NORDAHEAD definition (ITS#7734)
+	Fix mdb_cursor_del() positioning (ITS#7733)
+	Partial fix for larger page sizes (ITS#7713)
+	Fix Windows64/MSVC build issues
+
+LMDB 0.9.9 Release (2013/10/24)
+	Add mdb_env_get_fd()
+	Add MDB_NORDAHEAD option
+	Add MDB_NOLOCK option
+	Avoid wasting space in mdb_page_split() (ITS#7589)
+	Fix mdb_page_merge() cursor fixup (ITS#7722)
+	Fix mdb_cursor_del() on last delete (ITS#7718)
+	Fix adding WRITEMAP on existing env (ITS#7715)
+	Fix nested txns (ITS#7515)
+	Fix mdb_env_copy() O_DIRECT bug (ITS#7682)
+	Fix mdb_cursor_set(SET_RANGE) return code (ITS#7681)
+	Fix mdb_rebalance() cursor fixup (ITS#7701)
+	Misc code cleanup
+	Documentation
+		Note that by default, readers need write access
+
+
+LMDB 0.9.8 Release (2013/09/09)
+	Allow mdb_env_set_mapsize() on an open environment
+	Fix mdb_dbi_flags() (ITS#7672)
+	Fix mdb_page_unspill() in nested txns
+	Fix mdb_cursor_get(CURRENT|NEXT) after a delete
+	Fix mdb_cursor_get(DUP) to always return key (ITS#7671)
+	Fix mdb_cursor_del() to always advance to next item (ITS#7670)
+	Fix mdb_cursor_set(SET_RANGE) for tree with single page (ITS#7681)
+	Fix mdb_env_copy() retry open if O_DIRECT fails (ITS#7682)
+	Tweak mdb_page_spill() to be less aggressive
+	Documentation
+		Update caveats since mdb_reader_check() added in 0.9.7
+
+LMDB 0.9.7 Release (2013/08/17)
+	Don't leave stale lockfile on failed RDONLY open (ITS#7664)
+	Fix mdb_page_split() ref beyond cursor depth
+	Fix read txn data race (ITS#7635)
+	Fix mdb_rebalance (ITS#7536, #7538)
+	Fix mdb_drop() (ITS#7561)
+	Misc DEBUG macro fixes
+	Add MDB_NOTLS envflag
+	Add mdb_env_copyfd()
+	Add mdb_txn_env() (ITS#7660)
+	Add mdb_dbi_flags() (ITS#7661)
+	Add mdb_env_get_maxkeysize()
+	Add mdb_env_reader_list()/mdb_env_reader_check()
+	Add mdb_page_spill/unspill, remove hard txn size limit
+	Use shorter names for semaphores (ITS#7615)
+	Build
+		Fix install target (ITS#7656)
+	Documentation
+		Misc updates for cursors, DB handles, data lifetime
+
+LMDB 0.9.6 Release (2013/02/25)
+	Many fixes/enhancements
+
+LMDB 0.9.5 Release (2012/11/30)
+	Renamed from libmdb to liblmdb
+	Many fixes/enhancements

external/db_drivers/liblmdb32/CMakeLists.txt

@@ -0,0 +1,39 @@
+# Copyright (c) 2014-2015, The Monero Project
+# 
+# All rights reserved.
+# 
+# Redistribution and use in source and binary forms, with or without modification, are
+# permitted provided that the following conditions are met:
+# 
+# 1. Redistributions of source code must retain the above copyright notice, this list of
+#    conditions and the following disclaimer.
+# 
+# 2. Redistributions in binary form must reproduce the above copyright notice, this list
+#    of conditions and the following disclaimer in the documentation and/or other
+#    materials provided with the distribution.
+# 
+# 3. Neither the name of the copyright holder nor the names of its contributors may be
+#    used to endorse or promote products derived from this software without specific
+#    prior written permission.
+# 
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+# THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+# THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+set (lmdb_sources
+mdb.c
+midl.c)
+
+include_directories("${CMAKE_CURRENT_SOURCE_DIR}")
+
+add_library(lmdb
+  ${lmdb_sources})
+target_link_libraries(lmdb
+  LINK_PRIVATE
+    ${CMAKE_THREAD_LIBS_INIT})

external/db_drivers/liblmdb32/COPYRIGHT

@@ -0,0 +1,20 @@
+Copyright 2011-2014 Howard Chu, Symas Corp.
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted only as authorized by the OpenLDAP
+Public License.
+
+A copy of this license is available in the file LICENSE in the
+top-level directory of the distribution or, alternatively, at
+<http://www.OpenLDAP.org/license.html>.
+
+OpenLDAP is a registered trademark of the OpenLDAP Foundation.
+
+Individual files and/or contributed packages may be copyright by
+other parties and/or subject to additional restrictions.
+
+This work also contains materials derived from public sources.
+
+Additional information about OpenLDAP can be obtained at
+<http://www.openldap.org/>.

external/db_drivers/liblmdb32/LICENSE

@@ -0,0 +1,47 @@
+The OpenLDAP Public License
+  Version 2.8, 17 August 2003
+
+Redistribution and use of this software and associated documentation
+("Software"), with or without modification, are permitted provided
+that the following conditions are met:
+
+1. Redistributions in source form must retain copyright statements
+   and notices,
+
+2. Redistributions in binary form must reproduce applicable copyright
+   statements and notices, this list of conditions, and the following
+   disclaimer in the documentation and/or other materials provided
+   with the distribution, and
+
+3. Redistributions must contain a verbatim copy of this document.
+
+The OpenLDAP Foundation may revise this license from time to time.
+Each revision is distinguished by a version number.  You may use
+this Software under terms of this license revision or under the
+terms of any subsequent revision of the license.
+
+THIS SOFTWARE IS PROVIDED BY THE OPENLDAP FOUNDATION AND ITS
+CONTRIBUTORS ``AS IS'' AND ANY EXPRESSED OR IMPLIED WARRANTIES,
+INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
+AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.  IN NO EVENT
+SHALL THE OPENLDAP FOUNDATION, ITS CONTRIBUTORS, OR THE AUTHOR(S)
+OR OWNER(S) OF THE SOFTWARE BE LIABLE FOR ANY DIRECT, INDIRECT,
+INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
+ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGE.
+
+The names of the authors and copyright holders must not be used in
+advertising or otherwise to promote the sale, use or other dealing
+in this Software without specific, written prior permission.  Title
+to copyright in this Software shall at all times remain with copyright
+holders.
+
+OpenLDAP is a registered trademark of the OpenLDAP Foundation.
+
+Copyright 1999-2003 The OpenLDAP Foundation, Redwood City,
+California, USA.  All Rights Reserved.  Permission to copy and
+distribute verbatim copies of this document is granted.

external/db_drivers/liblmdb32/Makefile

@@ -0,0 +1,98 @@
+# Makefile for liblmdb (Lightning memory-mapped database library).
+
+########################################################################
+# Configuration. The compiler options must enable threaded compilation.
+#
+# Preprocessor macros (for CPPFLAGS) of interest...
+# Note that the defaults should already be correct for most
+# platforms; you should not need to change any of these.
+# Read their descriptions in mdb.c if you do:
+#
+# - MDB_USE_POSIX_SEM
+# - MDB_DSYNC
+# - MDB_FDATASYNC
+# - MDB_USE_PWRITEV
+#
+# There may be other macros in mdb.c of interest. You should
+# read mdb.c before changing any of them.
+#
+CC	= gcc
+W	= -W -Wall -Wno-unused-parameter -Wbad-function-cast -Wuninitialized
+THREADS = -pthread
+OPT = -O2 -g
+CFLAGS	= $(THREADS) $(OPT) $(W) $(XCFLAGS)
+LDLIBS	=
+SOLIBS	=
+prefix	= /usr/local
+XCFLAGS = -DVL32
+
+########################################################################
+
+IHDRS	= lmdb.h
+ILIBS	= liblmdb.a liblmdb.so
+IPROGS	= mdb_stat mdb_copy mdb_dump mdb_load
+IDOCS	= mdb_stat.1 mdb_copy.1 mdb_dump.1 mdb_load.1
+PROGS	= $(IPROGS) mtest mtest2 mtest3 mtest4 mtest5
+all:	$(ILIBS) $(PROGS)
+
+install: $(ILIBS) $(IPROGS) $(IHDRS)
+	for f in $(IPROGS); do cp $$f $(DESTDIR)$(prefix)/bin; done
+	for f in $(ILIBS); do cp $$f $(DESTDIR)$(prefix)/lib; done
+	for f in $(IHDRS); do cp $$f $(DESTDIR)$(prefix)/include; done
+	for f in $(IDOCS); do cp $$f $(DESTDIR)$(prefix)/man/man1; done
+
+clean:
+	rm -rf $(PROGS) *.[ao] *.so *~ testdb
+
+test:	all
+	mkdir testdb
+	./mtest && ./mdb_stat testdb
+
+liblmdb.a:	mdb.o midl.o
+	ar rs $@ mdb.o midl.o
+
+liblmdb.so:	mdb.o midl.o
+#	$(CC) $(LDFLAGS) -pthread -shared -Wl,-Bsymbolic -o $@ mdb.o midl.o $(SOLIBS)
+	$(CC) $(LDFLAGS) -pthread -shared -o $@ mdb.o midl.o $(SOLIBS)
+
+mdb_stat: mdb_stat.o liblmdb.a
+mdb_copy: mdb_copy.o liblmdb.a
+mdb_dump: mdb_dump.o liblmdb.a
+mdb_load: mdb_load.o liblmdb.a
+mtest:    mtest.o    liblmdb.a
+mtest2:	mtest2.o liblmdb.a
+mtest3:	mtest3.o liblmdb.a
+mtest4:	mtest4.o liblmdb.a
+mtest5:	mtest5.o liblmdb.a
+mtest6:	mtest6.o liblmdb.a
+
+mdb.o: mdb.c lmdb.h midl.h
+	$(CC) $(CFLAGS) -fPIC $(CPPFLAGS) -c mdb.c
+
+midl.o: midl.c midl.h
+	$(CC) $(CFLAGS) -fPIC $(CPPFLAGS) -c midl.c
+
+%:	%.o
+	$(CC) $(CFLAGS) $(LDFLAGS) $^ $(LDLIBS) -o $@
+
+%.o:	%.c lmdb.h
+	$(CC) $(CFLAGS) $(CPPFLAGS) -c $<
+
+COV_FLAGS=-fprofile-arcs -ftest-coverage
+COV_OBJS=xmdb.o xmidl.o
+
+coverage: xmtest
+	for i in mtest*.c [0-9]*.c; do j=`basename \$$i .c`; $(MAKE) $$j.o; \
+		gcc -o x$$j $$j.o $(COV_OBJS) -pthread $(COV_FLAGS); \
+		rm -rf testdb; mkdir testdb; ./x$$j; done
+	gcov xmdb.c
+	gcov xmidl.c
+
+xmtest:	mtest.o xmdb.o xmidl.o
+	gcc -o xmtest mtest.o xmdb.o xmidl.o -pthread $(COV_FLAGS)
+
+xmdb.o: mdb.c lmdb.h midl.h
+	$(CC) $(CFLAGS) -fPIC $(CPPFLAGS) -O0 $(COV_FLAGS) -c mdb.c -o $@
+
+xmidl.o: midl.c midl.h
+	$(CC) $(CFLAGS) -fPIC $(CPPFLAGS) -O0 $(COV_FLAGS) -c midl.c -o $@

external/db_drivers/liblmdb32/mdb_copy.1

@@ -0,0 +1,54 @@
+.TH MDB_COPY 1 "2014/06/20" "LMDB 0.9.14"
+.\" Copyright 2012-2014 Howard Chu, Symas Corp. All Rights Reserved.
+.\" Copying restrictions apply.  See COPYRIGHT/LICENSE.
+.SH NAME
+mdb_copy \- LMDB environment copy tool
+.SH SYNOPSIS
+.B mdb_copy
+[\c
+.BR \-V ]
+[\c
+.BR \-c ]
+[\c
+.BR \-n ]
+.B srcpath
+[\c
+.BR dstpath ]
+.SH DESCRIPTION
+The
+.B mdb_copy
+utility copies an LMDB environment. The environment can
+be copied regardless of whether it is currently in use.
+No lockfile is created, since it gets recreated at need.
+
+If
+.I dstpath
+is specified it must be the path of an empty directory
+for storing the backup. Otherwise, the backup will be
+written to stdout.
+
+.SH OPTIONS
+.TP
+.BR \-V
+Write the library version number to the standard output, and exit.
+.TP
+.BR \-c
+Compact while copying. Only current data pages will be copied; freed
+or unused pages will be omitted from the copy. This option will
+slow down the backup process as it is more CPU-intensive.
+.TP
+.BR \-n
+Open LDMB environment(s) which do not use subdirectories.
+
+.SH DIAGNOSTICS
+Exit status is zero if no errors occur.
+Errors result in a non-zero exit status and
+a diagnostic message being written to standard error.
+.SH CAVEATS
+This utility can trigger significant file size growth if run
+in parallel with write transactions, because pages which they
+free during copying cannot be reused until the copy is done.
+.SH "SEE ALSO"
+.BR mdb_stat (1)
+.SH AUTHOR
+Howard Chu of Symas Corporation <http://www.symas.com>

external/db_drivers/liblmdb32/mdb_copy.c

@@ -0,0 +1,82 @@
+/* mdb_copy.c - memory-mapped database backup tool */
+/*
+ * Copyright 2012 Howard Chu, Symas Corp.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted only as authorized by the OpenLDAP
+ * Public License.
+ *
+ * A copy of this license is available in the file LICENSE in the
+ * top-level directory of the distribution or, alternatively, at
+ * <http://www.OpenLDAP.org/license.html>.
+ */
+#ifdef _WIN32
+#include <windows.h>
+#define	MDB_STDOUT	GetStdHandle(STD_OUTPUT_HANDLE)
+#else
+#define	MDB_STDOUT	1
+#endif
+#include <stdio.h>
+#include <stdlib.h>
+#include <signal.h>
+#include "lmdb.h"
+
+static void
+sighandle(int sig)
+{
+}
+
+int main(int argc,char * argv[])
+{
+	int rc;
+	MDB_env *env;
+	const char *progname = argv[0], *act;
+	unsigned flags = MDB_RDONLY;
+	unsigned cpflags = 0;
+
+	for (; argc > 1 && argv[1][0] == '-'; argc--, argv++) {
+		if (argv[1][1] == 'n' && argv[1][2] == '\0')
+			flags |= MDB_NOSUBDIR;
+		else if (argv[1][1] == 'c' && argv[1][2] == '\0')
+			cpflags |= MDB_CP_COMPACT;
+		else if (argv[1][1] == 'V' && argv[1][2] == '\0') {
+			printf("%s\n", MDB_VERSION_STRING);
+			exit(0);
+		} else
+			argc = 0;
+	}
+
+	if (argc<2 || argc>3) {
+		fprintf(stderr, "usage: %s [-V] [-c] [-n] srcpath [dstpath]\n", progname);
+		exit(EXIT_FAILURE);
+	}
+
+#ifdef SIGPIPE
+	signal(SIGPIPE, sighandle);
+#endif
+#ifdef SIGHUP
+	signal(SIGHUP, sighandle);
+#endif
+	signal(SIGINT, sighandle);
+	signal(SIGTERM, sighandle);
+
+	act = "opening environment";
+	rc = mdb_env_create(&env);
+	if (rc == MDB_SUCCESS) {
+		rc = mdb_env_open(env, argv[1], flags, 0664);
+	}
+	if (rc == MDB_SUCCESS) {
+		act = "copying";
+		if (argc == 2)
+			rc = mdb_env_copyfd2(env, MDB_STDOUT, cpflags);
+		else
+			rc = mdb_env_copy2(env, argv[2], cpflags);
+	}
+	if (rc)
+		fprintf(stderr, "%s: %s failed, error %d (%s)\n",
+			progname, act, rc, mdb_strerror(rc));
+	mdb_env_close(env);
+
+	return rc ? EXIT_FAILURE : EXIT_SUCCESS;
+}

external/db_drivers/liblmdb32/mdb_dump.1

@@ -0,0 +1,75 @@
+.TH MDB_DUMP 1 "2014/06/20" "LMDB 0.9.14"
+.\" Copyright 2014 Howard Chu, Symas Corp. All Rights Reserved.
+.\" Copying restrictions apply.  See COPYRIGHT/LICENSE.
+.SH NAME
+mdb_dump \- LMDB environment export tool
+.SH SYNOPSIS
+.B mdb_dump
+.BR \ envpath
+[\c
+.BR \-V ]
+[\c
+.BI \-f \ file\fR]
+[\c
+.BR \-l ]
+[\c
+.BR \-n ]
+[\c
+.BR \-p ]
+[\c
+.BR \-a \ |
+.BI \-s \ subdb\fR]
+.SH DESCRIPTION
+The
+.B mdb_dump
+utility reads a database and writes its contents to the
+standard output using a portable flat-text format
+understood by the
+.BR mdb_load (1)
+utility.
+.SH OPTIONS
+.TP
+.BR \-V
+Write the library version number to the standard output, and exit.
+.TP
+.BR \-f \ file
+Write to the specified file instead of to the standard output.
+.TP
+.BR \-l
+List the databases stored in the environment. Just the
+names will be listed, no data will be output.
+.TP
+.BR \-n
+Dump an LMDB database which does not use subdirectories.
+.TP
+.BR \-p
+If characters in either the key or data items are printing characters (as
+defined by isprint(3)), output them directly. This option permits users to
+use standard text editors and tools to modify the contents of databases.
+
+Note: different systems may have different notions about what characters
+are considered printing characters, and databases dumped in this manner may
+be less portable to external systems. 
+.TP
+.BR \-a
+Dump all of the subdatabases in the environment.
+.TP
+.BR \-s \ subdb
+Dump a specific subdatabase. If no database is specified, only the main database is dumped.
+.SH DIAGNOSTICS
+Exit status is zero if no errors occur.
+Errors result in a non-zero exit status and
+a diagnostic message being written to standard error.
+
+Dumping and reloading databases that use user-defined comparison functions
+will result in new databases that use the default comparison functions.
+\fBIn this case it is quite likely that the reloaded database will be
+damaged beyond repair permitting neither record storage nor retrieval.\fP
+
+The only available workaround is to modify the source for the
+.BR mdb_load (1)
+utility to load the database using the correct comparison functions.
+.SH "SEE ALSO"
+.BR mdb_load (1)
+.SH AUTHOR
+Howard Chu of Symas Corporation <http://www.symas.com>

external/db_drivers/liblmdb32/mdb_dump.c

@@ -0,0 +1,317 @@
+/* mdb_dump.c - memory-mapped database dump tool */
+/*
+ * Copyright 2011-2014 Howard Chu, Symas Corp.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted only as authorized by the OpenLDAP
+ * Public License.
+ *
+ * A copy of this license is available in the file LICENSE in the
+ * top-level directory of the distribution or, alternatively, at
+ * <http://www.OpenLDAP.org/license.html>.
+ */
+#include <stdio.h>
+#include <errno.h>
+#include <stdlib.h>
+#include <string.h>
+#include <ctype.h>
+#include <unistd.h>
+#include <signal.h>
+#include "lmdb.h"
+
+#ifdef _WIN32
+#define Z	"I"
+#else
+#define Z	"z"
+#endif
+
+#define PRINT	1
+static int mode;
+
+typedef struct flagbit {
+	int bit;
+	char *name;
+} flagbit;
+
+flagbit dbflags[] = {
+	{ MDB_REVERSEKEY, "reversekey" },
+	{ MDB_DUPSORT, "dupsort" },
+	{ MDB_INTEGERKEY, "integerkey" },
+	{ MDB_DUPFIXED, "dupfixed" },
+	{ MDB_INTEGERDUP, "integerdup" },
+	{ MDB_REVERSEDUP, "reversedup" },
+	{ 0, NULL }
+};
+
+static volatile sig_atomic_t gotsig;
+
+static void dumpsig( int sig )
+{
+	gotsig=1;
+}
+
+static const char hexc[] = "0123456789abcdef";
+
+static void hex(unsigned char c)
+{
+	putchar(hexc[c >> 4]);
+	putchar(hexc[c & 0xf]);
+}
+
+static void text(MDB_val *v)
+{
+	unsigned char *c, *end;
+
+	putchar(' ');
+	c = v->mv_data;
+	end = c + v->mv_size;
+	while (c < end) {
+		if (isprint(*c)) {
+			putchar(*c);
+		} else {
+			putchar('\\');
+			hex(*c);
+		}
+		c++;
+	}
+	putchar('\n');
+}
+
+static void byte(MDB_val *v)
+{
+	unsigned char *c, *end;
+
+	putchar(' ');
+	c = v->mv_data;
+	end = c + v->mv_size;
+	while (c < end) {
+		hex(*c++);
+	}
+	putchar('\n');
+}
+
+/* Dump in BDB-compatible format */
+static int dumpit(MDB_txn *txn, MDB_dbi dbi, char *name)
+{
+	MDB_cursor *mc;
+	MDB_stat ms;
+	MDB_val key, data;
+	MDB_envinfo info;
+	unsigned int flags;
+	int rc, i;
+
+	rc = mdb_dbi_flags(txn, dbi, &flags);
+	if (rc) return rc;
+
+	rc = mdb_stat(txn, dbi, &ms);
+	if (rc) return rc;
+
+	rc = mdb_env_info(mdb_txn_env(txn), &info);
+	if (rc) return rc;
+
+	printf("VERSION=3\n");
+	printf("format=%s\n", mode & PRINT ? "print" : "bytevalue");
+	if (name)
+		printf("database=%s\n", name);
+	printf("type=btree\n");
+	printf("mapsize=%" Z "u\n", info.me_mapsize);
+	if (info.me_mapaddr)
+		printf("mapaddr=%p\n", info.me_mapaddr);
+	printf("maxreaders=%u\n", info.me_maxreaders);
+
+	if (flags & MDB_DUPSORT)
+		printf("duplicates=1\n");
+
+	for (i=0; dbflags[i].bit; i++)
+		if (flags & dbflags[i].bit)
+			printf("%s=1\n", dbflags[i].name);
+
+	printf("db_pagesize=%d\n", ms.ms_psize);
+	printf("HEADER=END\n");
+
+	rc = mdb_cursor_open(txn, dbi, &mc);
+	if (rc) return rc;
+
+	while ((rc = mdb_cursor_get(mc, &key, &data, MDB_NEXT) == MDB_SUCCESS)) {
+		if (gotsig) {
+			rc = EINTR;
+			break;
+		}
+		if (mode & PRINT) {
+			text(&key);
+			text(&data);
+		} else {
+			byte(&key);
+			byte(&data);
+		}
+	}
+	printf("DATA=END\n");
+	if (rc == MDB_NOTFOUND)
+		rc = MDB_SUCCESS;
+
+	return rc;
+}
+
+static void usage(char *prog)
+{
+	fprintf(stderr, "usage: %s dbpath [-V] [-f output] [-l] [-n] [-p] [-a|-s subdb]\n", prog);
+	exit(EXIT_FAILURE);
+}
+
+int main(int argc, char *argv[])
+{
+	int i, rc;
+	MDB_env *env;
+	MDB_txn *txn;
+	MDB_dbi dbi;
+	char *prog = argv[0];
+	char *envname;
+	char *subname = NULL;
+	int alldbs = 0, envflags = 0, list = 0;
+
+	if (argc < 2) {
+		usage(prog);
+	}
+
+	/* -a: dump main DB and all subDBs
+	 * -s: dump only the named subDB
+	 * -n: use NOSUBDIR flag on env_open
+	 * -p: use printable characters
+	 * -f: write to file instead of stdout
+	 * -V: print version and exit
+	 * (default) dump only the main DB
+	 */
+	while ((i = getopt(argc, argv, "af:lnps:V")) != EOF) {
+		switch(i) {
+		case 'V':
+			printf("%s\n", MDB_VERSION_STRING);
+			exit(0);
+			break;
+		case 'l':
+			list = 1;
+			/*FALLTHROUGH*/;
+		case 'a':
+			if (subname)
+				usage(prog);
+			alldbs++;
+			break;
+		case 'f':
+			if (freopen(optarg, "w", stdout) == NULL) {
+				fprintf(stderr, "%s: %s: reopen: %s\n",
+					prog, optarg, strerror(errno));
+				exit(EXIT_FAILURE);
+			}
+			break;
+		case 'n':
+			envflags |= MDB_NOSUBDIR;
+			break;
+		case 'p':
+			mode |= PRINT;
+			break;
+		case 's':
+			if (alldbs)
+				usage(prog);
+			subname = optarg;
+			break;
+		default:
+			usage(prog);
+		}
+	}
+
+	if (optind != argc - 1)
+		usage(prog);
+
+#ifdef SIGPIPE
+	signal(SIGPIPE, dumpsig);
+#endif
+#ifdef SIGHUP
+	signal(SIGHUP, dumpsig);
+#endif
+	signal(SIGINT, dumpsig);
+	signal(SIGTERM, dumpsig);
+
+	envname = argv[optind];
+	rc = mdb_env_create(&env);
+	if (rc) {
+		fprintf(stderr, "mdb_env_create failed, error %d %s\n", rc, mdb_strerror(rc));
+		return EXIT_FAILURE;
+	}
+
+	if (alldbs || subname) {
+		mdb_env_set_maxdbs(env, 2);
+	}
+
+	rc = mdb_env_open(env, envname, envflags | MDB_RDONLY, 0664);
+	if (rc) {
+		fprintf(stderr, "mdb_env_open failed, error %d %s\n", rc, mdb_strerror(rc));
+		goto env_close;
+	}
+
+	rc = mdb_txn_begin(env, NULL, MDB_RDONLY, &txn);
+	if (rc) {
+		fprintf(stderr, "mdb_txn_begin failed, error %d %s\n", rc, mdb_strerror(rc));
+		goto env_close;
+	}
+
+	rc = mdb_open(txn, subname, 0, &dbi);
+	if (rc) {
+		fprintf(stderr, "mdb_open failed, error %d %s\n", rc, mdb_strerror(rc));
+		goto txn_abort;
+	}
+
+	if (alldbs) {
+		MDB_cursor *cursor;
+		MDB_val key;
+		int count = 0;
+
+		rc = mdb_cursor_open(txn, dbi, &cursor);
+		if (rc) {
+			fprintf(stderr, "mdb_cursor_open failed, error %d %s\n", rc, mdb_strerror(rc));
+			goto txn_abort;
+		}
+		while ((rc = mdb_cursor_get(cursor, &key, NULL, MDB_NEXT_NODUP)) == 0) {
+			char *str;
+			MDB_dbi db2;
+			if (memchr(key.mv_data, '\0', key.mv_size))
+				continue;
+			count++;
+			str = malloc(key.mv_size+1);
+			memcpy(str, key.mv_data, key.mv_size);
+			str[key.mv_size] = '\0';
+			rc = mdb_open(txn, str, 0, &db2);
+			if (rc == MDB_SUCCESS) {
+				if (list) {
+					printf("%s\n", str);
+					list++;
+				} else {
+					rc = dumpit(txn, db2, str);
+					if (rc)
+						break;
+				}
+				mdb_close(env, db2);
+			}
+			free(str);
+			if (rc) continue;
+		}
+		mdb_cursor_close(cursor);
+		if (!count) {
+			fprintf(stderr, "%s: %s does not contain multiple databases\n", prog, envname);
+			rc = MDB_NOTFOUND;
+		} else if (rc == MDB_NOTFOUND) {
+			rc = MDB_SUCCESS;
+		}
+	} else {
+		rc = dumpit(txn, dbi, subname);
+	}
+	if (rc && rc != MDB_NOTFOUND)
+		fprintf(stderr, "%s: %s: %s\n", prog, envname, mdb_strerror(rc));
+
+	mdb_close(env, dbi);
+txn_abort:
+	mdb_txn_abort(txn);
+env_close:
+	mdb_env_close(env);
+
+	return rc ? EXIT_FAILURE : EXIT_SUCCESS;
+}

external/db_drivers/liblmdb32/mdb_load.1

@@ -0,0 +1,77 @@
+.TH MDB_LOAD 1 "2014/06/20" "LMDB 0.9.14"
+.\" Copyright 2014 Howard Chu, Symas Corp. All Rights Reserved.
+.\" Copying restrictions apply.  See COPYRIGHT/LICENSE.
+.SH NAME
+mdb_load \- LMDB environment import tool
+.SH SYNOPSIS
+.B mdb_load
+.BR \ envpath
+[\c
+.BR \-V ]
+[\c
+.BI \-f \ file\fR]
+[\c
+.BR \-n ]
+[\c
+.BI \-s \ subdb\fR]
+[\c
+.BR \-N ]
+[\c
+.BR \-T ]
+.SH DESCRIPTION
+The
+.B mdb_load
+utility reads from the standard input and loads it into the
+LMDB environment
+.BR envpath .
+
+The input to
+.B mdb_load
+must be in the output format specified by the
+.BR mdb_dump (1)
+utility or as specified by the
+.B -T
+option below.
+.SH OPTIONS
+.TP
+.BR \-V
+Write the library version number to the standard output, and exit.
+.TP
+.BR \-f \ file
+Read from the specified file instead of from the standard input.
+.TP
+.BR \-n
+Load an LMDB database which does not use subdirectories.
+.TP
+.BR \-s \ subdb
+Load a specific subdatabase. If no database is specified, data is loaded into the main database.
+.TP
+.BR \-N
+Don't overwrite existing records when loading into an already existing database; just skip them.
+.TP
+.BR \-T
+Load data from simple text files. The input must be paired lines of text, where the first
+line of the pair is the key item, and the second line of the pair is its corresponding
+data item.
+
+A simple escape mechanism, where newline and backslash (\\) characters are special, is
+applied to the text input. Newline characters are interpreted as record separators.
+Backslash characters in the text will be interpreted in one of two ways: If the backslash
+character precedes another backslash character, the pair will be interpreted as a literal
+backslash. If the backslash character precedes any other character, the two characters
+following the backslash will be interpreted as a hexadecimal specification of a single
+character; for example, \\0a is a newline character in the ASCII character set.
+
+For this reason, any backslash or newline characters that naturally occur in the text
+input must be escaped to avoid misinterpretation by
+.BR mdb_load . 
+
+.SH DIAGNOSTICS
+Exit status is zero if no errors occur.
+Errors result in a non-zero exit status and
+a diagnostic message being written to standard error.
+
+.SH "SEE ALSO"
+.BR mdb_dump (1)
+.SH AUTHOR
+Howard Chu of Symas Corporation <http://www.symas.com>

external/db_drivers/liblmdb32/mdb_load.c

@@ -0,0 +1,452 @@
+/* mdb_load.c - memory-mapped database load tool */
+/*
+ * Copyright 2011-2014 Howard Chu, Symas Corp.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted only as authorized by the OpenLDAP
+ * Public License.
+ *
+ * A copy of this license is available in the file LICENSE in the
+ * top-level directory of the distribution or, alternatively, at
+ * <http://www.OpenLDAP.org/license.html>.
+ */
+#include <stdio.h>
+#include <stdlib.h>
+#include <errno.h>
+#include <string.h>
+#include <ctype.h>
+#include <unistd.h>
+#include "lmdb.h"
+
+#define PRINT	1
+#define NOHDR	2
+static int mode;
+
+static char *subname = NULL;
+
+static size_t lineno;
+static int version;
+
+static int flags;
+
+static char *prog;
+
+static int Eof;
+
+static MDB_envinfo info;
+
+static MDB_val kbuf, dbuf;
+
+#ifdef _WIN32
+#define Z	"I"
+#else
+#define Z	"z"
+#endif
+
+#define STRLENOF(s)	(sizeof(s)-1)
+
+typedef struct flagbit {
+	int bit;
+	char *name;
+	int len;
+} flagbit;
+
+#define S(s)	s, STRLENOF(s)
+
+flagbit dbflags[] = {
+	{ MDB_REVERSEKEY, S("reversekey") },
+	{ MDB_DUPSORT, S("dupsort") },
+	{ MDB_INTEGERKEY, S("integerkey") },
+	{ MDB_DUPFIXED, S("dupfixed") },
+	{ MDB_INTEGERDUP, S("integerdup") },
+	{ MDB_REVERSEDUP, S("reversedup") },
+	{ 0, NULL, 0 }
+};
+
+static void readhdr(void)
+{
+	char *ptr;
+
+	while (fgets(dbuf.mv_data, dbuf.mv_size, stdin) != NULL) {
+		lineno++;
+		if (!strncmp(dbuf.mv_data, "VERSION=", STRLENOF("VERSION="))) {
+			version=atoi((char *)dbuf.mv_data+STRLENOF("VERSION="));
+			if (version > 3) {
+				fprintf(stderr, "%s: line %" Z "d: unsupported VERSION %d\n",
+					prog, lineno, version);
+				exit(EXIT_FAILURE);
+			}
+		} else if (!strncmp(dbuf.mv_data, "HEADER=END", STRLENOF("HEADER=END"))) {
+			break;
+		} else if (!strncmp(dbuf.mv_data, "format=", STRLENOF("format="))) {
+			if (!strncmp((char *)dbuf.mv_data+STRLENOF("FORMAT="), "print", STRLENOF("print")))
+				mode |= PRINT;
+			else if (strncmp((char *)dbuf.mv_data+STRLENOF("FORMAT="), "bytevalue", STRLENOF("bytevalue"))) {
+				fprintf(stderr, "%s: line %" Z "d: unsupported FORMAT %s\n",
+					prog, lineno, (char *)dbuf.mv_data+STRLENOF("FORMAT="));
+				exit(EXIT_FAILURE);
+			}
+		} else if (!strncmp(dbuf.mv_data, "database=", STRLENOF("database="))) {
+			ptr = memchr(dbuf.mv_data, '\n', dbuf.mv_size);
+			if (ptr) *ptr = '\0';
+			if (subname) free(subname);
+			subname = strdup((char *)dbuf.mv_data+STRLENOF("database="));
+		} else if (!strncmp(dbuf.mv_data, "type=", STRLENOF("type="))) {
+			if (strncmp((char *)dbuf.mv_data+STRLENOF("type="), "btree", STRLENOF("btree")))  {
+				fprintf(stderr, "%s: line %" Z "d: unsupported type %s\n",
+					prog, lineno, (char *)dbuf.mv_data+STRLENOF("type="));
+				exit(EXIT_FAILURE);
+			}
+		} else if (!strncmp(dbuf.mv_data, "mapaddr=", STRLENOF("mapaddr="))) {
+			int i;
+			ptr = memchr(dbuf.mv_data, '\n', dbuf.mv_size);
+			if (ptr) *ptr = '\0';
+			i = sscanf((char *)dbuf.mv_data+STRLENOF("mapaddr="), "%p", &info.me_mapaddr);
+			if (i != 1) {
+				fprintf(stderr, "%s: line %" Z "d: invalid mapaddr %s\n",
+					prog, lineno, (char *)dbuf.mv_data+STRLENOF("mapaddr="));
+				exit(EXIT_FAILURE);
+			}
+		} else if (!strncmp(dbuf.mv_data, "mapsize=", STRLENOF("mapsize="))) {
+			int i;
+			ptr = memchr(dbuf.mv_data, '\n', dbuf.mv_size);
+			if (ptr) *ptr = '\0';
+			i = sscanf((char *)dbuf.mv_data+STRLENOF("mapsize="), "%" Z "u", &info.me_mapsize);
+			if (i != 1) {
+				fprintf(stderr, "%s: line %" Z "d: invalid mapsize %s\n",
+					prog, lineno, (char *)dbuf.mv_data+STRLENOF("mapsize="));
+				exit(EXIT_FAILURE);
+			}
+		} else if (!strncmp(dbuf.mv_data, "maxreaders=", STRLENOF("maxreaders="))) {
+			int i;
+			ptr = memchr(dbuf.mv_data, '\n', dbuf.mv_size);
+			if (ptr) *ptr = '\0';
+			i = sscanf((char *)dbuf.mv_data+STRLENOF("maxreaders="), "%u", &info.me_maxreaders);
+			if (i != 1) {
+				fprintf(stderr, "%s: line %" Z "d: invalid maxreaders %s\n",
+					prog, lineno, (char *)dbuf.mv_data+STRLENOF("maxreaders="));
+				exit(EXIT_FAILURE);
+			}
+		} else {
+			int i;
+			for (i=0; dbflags[i].bit; i++) {
+				if (!strncmp(dbuf.mv_data, dbflags[i].name, dbflags[i].len) &&
+					((char *)dbuf.mv_data)[dbflags[i].len] == '=') {
+					flags |= dbflags[i].bit;
+					break;
+				}
+			}
+			if (!dbflags[i].bit) {
+				ptr = memchr(dbuf.mv_data, '=', dbuf.mv_size);
+				if (!ptr) {
+					fprintf(stderr, "%s: line %" Z "d: unexpected format\n",
+						prog, lineno);
+					exit(EXIT_FAILURE);
+				} else {
+					*ptr = '\0';
+					fprintf(stderr, "%s: line %" Z "d: unrecognized keyword ignored: %s\n",
+						prog, lineno, (char *)dbuf.mv_data);
+				}
+			}
+		}
+	}
+}
+
+static void badend(void)
+{
+	fprintf(stderr, "%s: line %" Z "d: unexpected end of input\n",
+		prog, lineno);
+}
+
+static int unhex(unsigned char *c2)
+{
+	int x, c;
+	x = *c2++ & 0x4f;
+	if (x & 0x40)
+		x -= 55;
+	c = x << 4;
+	x = *c2 & 0x4f;
+	if (x & 0x40)
+		x -= 55;
+	c |= x;
+	return c;
+}
+
+static int readline(MDB_val *out, MDB_val *buf)
+{
+	unsigned char *c1, *c2, *end;
+	size_t len;
+	int c;
+
+	if (!(mode & NOHDR)) {
+		c = fgetc(stdin);
+		if (c == EOF) {
+			Eof = 1;
+			return EOF;
+		}
+		if (c != ' ') {
+			lineno++;
+			if (fgets(buf->mv_data, buf->mv_size, stdin) == NULL) {
+badend:
+				Eof = 1;
+				badend();
+				return EOF;
+			}
+			if (c == 'D' && !strncmp(buf->mv_data, "ATA=END", STRLENOF("ATA=END")))
+				return EOF;
+			goto badend;
+		}
+	}
+	if (fgets(buf->mv_data, buf->mv_size, stdin) == NULL) {
+		Eof = 1;
+		return EOF;
+	}
+	lineno++;
+
+	c1 = buf->mv_data;
+	len = strlen((char *)c1);
+
+	/* Is buffer too short? */
+	while (c1[len-1] != '\n') {
+		buf->mv_data = realloc(buf->mv_data, buf->mv_size*2);
+		if (!buf->mv_data) {
+			Eof = 1;
+			fprintf(stderr, "%s: line %" Z "d: out of memory, line too long\n",
+				prog, lineno);
+			return EOF;
+		}
+		c1 = buf->mv_data;
+		c1 += buf->mv_size;
+		if (fgets((char *)c1, buf->mv_size, stdin) == NULL) {
+			Eof = 1;
+			badend();
+			return EOF;
+		}
+		buf->mv_size *= 2;
+		len = strlen((char *)c1);
+	}
+	c1 = c2 = buf->mv_data;
+	len = strlen((char *)c1);
+	c1[--len] = '\0';
+	end = c1 + len;
+
+	if (mode & PRINT) {
+		while (c2 < end) {
+			if (*c2 == '\\') {
+				if (c2[1] == '\\') {
+					c1++; c2 += 2;
+				} else {
+					if (c2+3 > end || !isxdigit(c2[1]) || !isxdigit(c2[2])) {
+						Eof = 1;
+						badend();
+						return EOF;
+					}
+					*c1++ = unhex(++c2);
+					c2 += 2;
+				}
+			} else {
+				c1++; c2++;
+			}
+		}
+	} else {
+		/* odd length not allowed */
+		if (len & 1) {
+			Eof = 1;
+			badend();
+			return EOF;
+		}
+		while (c2 < end) {
+			if (!isxdigit(*c2) || !isxdigit(c2[1])) {
+				Eof = 1;
+				badend();
+				return EOF;
+			}
+			*c1++ = unhex(c2);
+			c2 += 2;
+		}
+	}
+	c2 = out->mv_data = buf->mv_data;
+	out->mv_size = c1 - c2;
+
+	return 0;
+}
+
+static void usage(void)
+{
+	fprintf(stderr, "usage: %s dbpath [-V] [-f input] [-n] [-s name] [-N] [-T]\n", prog);
+	exit(EXIT_FAILURE);
+}
+
+int main(int argc, char *argv[])
+{
+	int i, rc;
+	MDB_env *env;
+	MDB_txn *txn;
+	MDB_cursor *mc;
+	MDB_dbi dbi;
+	char *envname;
+	int envflags = 0, putflags = 0;
+	int dohdr = 0;
+
+	prog = argv[0];
+
+	if (argc < 2) {
+		usage();
+	}
+
+	/* -f: load file instead of stdin
+	 * -n: use NOSUBDIR flag on env_open
+	 * -s: load into named subDB
+	 * -N: use NOOVERWRITE on puts
+	 * -T: read plaintext
+	 * -V: print version and exit
+	 */
+	while ((i = getopt(argc, argv, "f:ns:NTV")) != EOF) {
+		switch(i) {
+		case 'V':
+			printf("%s\n", MDB_VERSION_STRING);
+			exit(0);
+			break;
+		case 'f':
+			if (freopen(optarg, "r", stdin) == NULL) {
+				fprintf(stderr, "%s: %s: reopen: %s\n",
+					prog, optarg, strerror(errno));
+				exit(EXIT_FAILURE);
+			}
+			break;
+		case 'n':
+			envflags |= MDB_NOSUBDIR;
+			break;
+		case 's':
+			subname = strdup(optarg);
+			break;
+		case 'N':
+			putflags = MDB_NOOVERWRITE|MDB_NODUPDATA;
+			break;
+		case 'T':
+			mode |= NOHDR;
+			break;
+		default:
+			usage();
+		}
+	}
+
+	if (optind != argc - 1)
+		usage();
+
+	dbuf.mv_size = 4096;
+	dbuf.mv_data = malloc(dbuf.mv_size);
+
+	if (!(mode & NOHDR))
+		readhdr();
+
+	envname = argv[optind];
+	rc = mdb_env_create(&env);
+	if (rc) {
+		fprintf(stderr, "mdb_env_create failed, error %d %s\n", rc, mdb_strerror(rc));
+		return EXIT_FAILURE;
+	}
+
+	mdb_env_set_maxdbs(env, 2);
+
+	if (info.me_maxreaders)
+		mdb_env_set_maxreaders(env, info.me_maxreaders);
+
+	if (info.me_mapsize)
+		mdb_env_set_mapsize(env, info.me_mapsize);
+
+	if (info.me_mapaddr)
+		envflags |= MDB_FIXEDMAP;
+
+	rc = mdb_env_open(env, envname, envflags, 0664);
+	if (rc) {
+		fprintf(stderr, "mdb_env_open failed, error %d %s\n", rc, mdb_strerror(rc));
+		goto env_close;
+	}
+
+	kbuf.mv_size = mdb_env_get_maxkeysize(env) * 2 + 2;
+	kbuf.mv_data = malloc(kbuf.mv_size);
+
+	while(!Eof) {
+		MDB_val key, data;
+		int batch = 0;
+		flags = 0;
+
+		if (!dohdr) {
+			dohdr = 1;
+		} else if (!(mode & NOHDR))
+			readhdr();
+		
+		rc = mdb_txn_begin(env, NULL, 0, &txn);
+		if (rc) {
+			fprintf(stderr, "mdb_txn_begin failed, error %d %s\n", rc, mdb_strerror(rc));
+			goto env_close;
+		}
+
+		rc = mdb_open(txn, subname, flags|MDB_CREATE, &dbi);
+		if (rc) {
+			fprintf(stderr, "mdb_open failed, error %d %s\n", rc, mdb_strerror(rc));
+			goto txn_abort;
+		}
+
+		rc = mdb_cursor_open(txn, dbi, &mc);
+		if (rc) {
+			fprintf(stderr, "mdb_cursor_open failed, error %d %s\n", rc, mdb_strerror(rc));
+			goto txn_abort;
+		}
+
+		while(1) {
+			rc = readline(&key, &kbuf);
+			if (rc == EOF)
+				break;
+			if (rc)
+				goto txn_abort;
+
+			rc = readline(&data, &dbuf);
+			if (rc)
+				goto txn_abort;
+			
+			rc = mdb_cursor_put(mc, &key, &data, putflags);
+			if (rc == MDB_KEYEXIST && putflags)
+				continue;
+			if (rc)
+				goto txn_abort;
+			batch++;
+			if (batch == 100) {
+				rc = mdb_txn_commit(txn);
+				if (rc) {
+					fprintf(stderr, "%s: line %" Z "d: txn_commit: %s\n",
+						prog, lineno, mdb_strerror(rc));
+					goto env_close;
+				}
+				rc = mdb_txn_begin(env, NULL, 0, &txn);
+				if (rc) {
+					fprintf(stderr, "mdb_txn_begin failed, error %d %s\n", rc, mdb_strerror(rc));
+					goto env_close;
+				}
+				rc = mdb_cursor_open(txn, dbi, &mc);
+				if (rc) {
+					fprintf(stderr, "mdb_cursor_open failed, error %d %s\n", rc, mdb_strerror(rc));
+					goto txn_abort;
+				}
+				batch = 0;
+			}
+		}
+		rc = mdb_txn_commit(txn);
+		txn = NULL;
+		if (rc) {
+			fprintf(stderr, "%s: line %" Z "d: txn_commit: %s\n",
+				prog, lineno, mdb_strerror(rc));
+			goto env_close;
+		}
+		mdb_dbi_close(env, dbi);
+	}
+
+txn_abort:
+	mdb_txn_abort(txn);
+env_close:
+	mdb_env_close(env);
+
+	return rc ? EXIT_FAILURE : EXIT_SUCCESS;
+}

external/db_drivers/liblmdb32/mdb_stat.1

@@ -0,0 +1,64 @@
+.TH MDB_STAT 1 "2014/06/20" "LMDB 0.9.14"
+.\" Copyright 2012-2014 Howard Chu, Symas Corp. All Rights Reserved.
+.\" Copying restrictions apply.  See COPYRIGHT/LICENSE.
+.SH NAME
+mdb_stat \- LMDB environment status tool
+.SH SYNOPSIS
+.B mdb_stat
+.BR \ envpath
+[\c
+.BR \-V ]
+[\c
+.BR \-e ]
+[\c
+.BR \-f [ f [ f ]]]
+[\c
+.BR \-n ]
+[\c
+.BR \-r [ r ]]
+[\c
+.BR \-a \ |
+.BI \-s \ subdb\fR]
+.SH DESCRIPTION
+The
+.B mdb_stat
+utility displays the status of an LMDB environment.
+.SH OPTIONS
+.TP
+.BR \-V
+Write the library version number to the standard output, and exit.
+.TP
+.BR \-e
+Display information about the database environment.
+.TP
+.BR \-f
+Display information about the environment freelist.
+If \fB\-ff\fP is given, summarize each freelist entry.
+If \fB\-fff\fP is given, display the full list of page IDs in the freelist.
+.TP
+.BR \-n
+Display the status of an LMDB database which does not use subdirectories.
+.TP
+.BR \-r
+Display information about the environment reader table.
+Shows the process ID, thread ID, and transaction ID for each active
+reader slot. The process ID and transaction ID are in decimal, the
+thread ID is in hexadecimal. The transaction ID is displayed as "-"
+if the reader does not currently have a read transaction open.
+If \fB\-rr\fP is given, check for stale entries in the reader
+table and clear them. The reader table will be printed again
+after the check is performed.
+.TP
+.BR \-a
+Display the status of all of the subdatabases in the environment.
+.TP
+.BR \-s \ subdb
+Display the status of a specific subdatabase.
+.SH DIAGNOSTICS
+Exit status is zero if no errors occur.
+Errors result in a non-zero exit status and
+a diagnostic message being written to standard error.
+.SH "SEE ALSO"
+.BR mdb_copy (1)
+.SH AUTHOR
+Howard Chu of Symas Corporation <http://www.symas.com>

external/db_drivers/liblmdb32/mdb_stat.c

@@ -0,0 +1,263 @@
+/* mdb_stat.c - memory-mapped database status tool */
+/*
+ * Copyright 2011-2013 Howard Chu, Symas Corp.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted only as authorized by the OpenLDAP
+ * Public License.
+ *
+ * A copy of this license is available in the file LICENSE in the
+ * top-level directory of the distribution or, alternatively, at
+ * <http://www.OpenLDAP.org/license.html>.
+ */
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include "lmdb.h"
+
+#ifdef	_WIN32
+#define	Z	"I"
+#else
+#define	Z	"z"
+#endif
+
+static void prstat(MDB_stat *ms)
+{
+#if 0
+	printf("  Page size: %u\n", ms->ms_psize);
+#endif
+	printf("  Tree depth: %u\n", ms->ms_depth);
+	printf("  Branch pages: %"Z"u\n", ms->ms_branch_pages);
+	printf("  Leaf pages: %"Z"u\n", ms->ms_leaf_pages);
+	printf("  Overflow pages: %"Z"u\n", ms->ms_overflow_pages);
+	printf("  Entries: %"Z"u\n", ms->ms_entries);
+}
+
+static void usage(char *prog)
+{
+	fprintf(stderr, "usage: %s dbpath [-V] [-n] [-e] [-r[r]] [-f[f[f]]] [-a|-s subdb]\n", prog);
+	exit(EXIT_FAILURE);
+}
+
+int main(int argc, char *argv[])
+{
+	int i, rc;
+	MDB_env *env;
+	MDB_txn *txn;
+	MDB_dbi dbi;
+	MDB_stat mst;
+	MDB_envinfo mei;
+	char *prog = argv[0];
+	char *envname;
+	char *subname = NULL;
+	int alldbs = 0, envinfo = 0, envflags = 0, freinfo = 0, rdrinfo = 0;
+
+	if (argc < 2) {
+		usage(prog);
+	}
+
+	/* -a: print stat of main DB and all subDBs
+	 * -s: print stat of only the named subDB
+	 * -e: print env info
+	 * -f: print freelist info
+	 * -r: print reader info
+	 * -n: use NOSUBDIR flag on env_open
+	 * -V: print version and exit
+	 * (default) print stat of only the main DB
+	 */
+	while ((i = getopt(argc, argv, "Vaefnrs:")) != EOF) {
+		switch(i) {
+		case 'V':
+			printf("%s\n", MDB_VERSION_STRING);
+			exit(0);
+			break;
+		case 'a':
+			if (subname)
+				usage(prog);
+			alldbs++;
+			break;
+		case 'e':
+			envinfo++;
+			break;
+		case 'f':
+			freinfo++;
+			break;
+		case 'n':
+			envflags |= MDB_NOSUBDIR;
+			break;
+		case 'r':
+			rdrinfo++;
+			break;
+		case 's':
+			if (alldbs)
+				usage(prog);
+			subname = optarg;
+			break;
+		default:
+			usage(prog);
+		}
+	}
+
+	if (optind != argc - 1)
+		usage(prog);
+
+	envname = argv[optind];
+	rc = mdb_env_create(&env);
+	if (rc) {
+		fprintf(stderr, "mdb_env_create failed, error %d %s\n", rc, mdb_strerror(rc));
+		return EXIT_FAILURE;
+	}
+
+	if (alldbs || subname) {
+		mdb_env_set_maxdbs(env, 4);
+	}
+
+	rc = mdb_env_open(env, envname, envflags | MDB_RDONLY, 0664);
+	if (rc) {
+		fprintf(stderr, "mdb_env_open failed, error %d %s\n", rc, mdb_strerror(rc));
+		goto env_close;
+	}
+
+	if (envinfo) {
+		(void)mdb_env_stat(env, &mst);
+		(void)mdb_env_info(env, &mei);
+		printf("Environment Info\n");
+		printf("  Map address: %p\n", mei.me_mapaddr);
+		printf("  Map size: %"Z"u\n", mei.me_mapsize);
+		printf("  Page size: %u\n", mst.ms_psize);
+		printf("  Max pages: %"Z"u\n", mei.me_mapsize / mst.ms_psize);
+		printf("  Number of pages used: %"Z"u\n", mei.me_last_pgno+1);
+		printf("  Last transaction ID: %"Z"u\n", mei.me_last_txnid);
+		printf("  Max readers: %u\n", mei.me_maxreaders);
+		printf("  Number of readers used: %u\n", mei.me_numreaders);
+	}
+
+	if (rdrinfo) {
+		printf("Reader Table Status\n");
+		rc = mdb_reader_list(env, (MDB_msg_func *)fputs, stdout);
+		if (rdrinfo > 1) {
+			int dead;
+			mdb_reader_check(env, &dead);
+			printf("  %d stale readers cleared.\n", dead);
+			rc = mdb_reader_list(env, (MDB_msg_func *)fputs, stdout);
+		}
+		if (!(subname || alldbs || freinfo))
+			goto env_close;
+	}
+
+	rc = mdb_txn_begin(env, NULL, MDB_RDONLY, &txn);
+	if (rc) {
+		fprintf(stderr, "mdb_txn_begin failed, error %d %s\n", rc, mdb_strerror(rc));
+		goto env_close;
+	}
+
+	if (freinfo) {
+		MDB_cursor *cursor;
+		MDB_val key, data;
+		size_t pages = 0, *iptr;
+
+		printf("Freelist Status\n");
+		dbi = 0;
+		rc = mdb_cursor_open(txn, dbi, &cursor);
+		if (rc) {
+			fprintf(stderr, "mdb_cursor_open failed, error %d %s\n", rc, mdb_strerror(rc));
+			goto txn_abort;
+		}
+		rc = mdb_stat(txn, dbi, &mst);
+		if (rc) {
+			fprintf(stderr, "mdb_stat failed, error %d %s\n", rc, mdb_strerror(rc));
+			goto txn_abort;
+		}
+		prstat(&mst);
+		while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
+			iptr = data.mv_data;
+			pages += *iptr;
+			if (freinfo > 1) {
+				char *bad = "";
+				size_t pg, prev;
+				ssize_t i, j, span = 0;
+				j = *iptr++;
+				for (i = j, prev = 1; --i >= 0; ) {
+					pg = iptr[i];
+					if (pg <= prev)
+						bad = " [bad sequence]";
+					prev = pg;
+					pg += span;
+					for (; i >= span && iptr[i-span] == pg; span++, pg++) ;
+				}
+				printf("    Transaction %"Z"u, %"Z"d pages, maxspan %"Z"d%s\n",
+					*(size_t *)key.mv_data, j, span, bad);
+				if (freinfo > 2) {
+					for (--j; j >= 0; ) {
+						pg = iptr[j];
+						for (span=1; --j >= 0 && iptr[j] == pg+span; span++) ;
+						printf(span>1 ? "     %9"Z"u[%"Z"d]\n" : "     %9"Z"u\n",
+							pg, span);
+					}
+				}
+			}
+		}
+		mdb_cursor_close(cursor);
+		printf("  Free pages: %"Z"u\n", pages);
+	}
+
+	rc = mdb_open(txn, subname, 0, &dbi);
+	if (rc) {
+		fprintf(stderr, "mdb_open failed, error %d %s\n", rc, mdb_strerror(rc));
+		goto txn_abort;
+	}
+
+	rc = mdb_stat(txn, dbi, &mst);
+	if (rc) {
+		fprintf(stderr, "mdb_stat failed, error %d %s\n", rc, mdb_strerror(rc));
+		goto txn_abort;
+	}
+	printf("Status of %s\n", subname ? subname : "Main DB");
+	prstat(&mst);
+
+	if (alldbs) {
+		MDB_cursor *cursor;
+		MDB_val key;
+
+		rc = mdb_cursor_open(txn, dbi, &cursor);
+		if (rc) {
+			fprintf(stderr, "mdb_cursor_open failed, error %d %s\n", rc, mdb_strerror(rc));
+			goto txn_abort;
+		}
+		while ((rc = mdb_cursor_get(cursor, &key, NULL, MDB_NEXT_NODUP)) == 0) {
+			char *str;
+			MDB_dbi db2;
+			if (memchr(key.mv_data, '\0', key.mv_size))
+				continue;
+			str = malloc(key.mv_size+1);
+			memcpy(str, key.mv_data, key.mv_size);
+			str[key.mv_size] = '\0';
+			rc = mdb_open(txn, str, 0, &db2);
+			if (rc == MDB_SUCCESS)
+				printf("Status of %s\n", str);
+			free(str);
+			if (rc) continue;
+			rc = mdb_stat(txn, db2, &mst);
+			if (rc) {
+				fprintf(stderr, "mdb_stat failed, error %d %s\n", rc, mdb_strerror(rc));
+				goto txn_abort;
+			}
+			prstat(&mst);
+			mdb_close(env, db2);
+		}
+		mdb_cursor_close(cursor);
+	}
+
+	if (rc == MDB_NOTFOUND)
+		rc = MDB_SUCCESS;
+
+	mdb_close(env, dbi);
+txn_abort:
+	mdb_txn_abort(txn);
+env_close:
+	mdb_env_close(env);
+
+	return rc ? EXIT_FAILURE : EXIT_SUCCESS;
+}

external/db_drivers/liblmdb32/midl.c

@@ -0,0 +1,360 @@
+/**	@file midl.c
+ *	@brief ldap bdb back-end ID List functions */
+/* $OpenLDAP$ */
+/* This work is part of OpenLDAP Software <http://www.openldap.org/>.
+ *
+ * Copyright 2000-2014 The OpenLDAP Foundation.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted only as authorized by the OpenLDAP
+ * Public License.
+ *
+ * A copy of this license is available in the file LICENSE in the
+ * top-level directory of the distribution or, alternatively, at
+ * <http://www.OpenLDAP.org/license.html>.
+ */
+
+#include <limits.h>
+#include <string.h>
+#include <stdlib.h>
+#include <errno.h>
+#include <sys/types.h>
+#include "midl.h"
+
+/** @defgroup internal	LMDB Internals
+ *	@{
+ */
+/** @defgroup idls	ID List Management
+ *	@{
+ */
+#define CMP(x,y)	 ( (x) < (y) ? -1 : (x) > (y) )
+
+unsigned mdb_midl_search( MDB_IDL ids, MDB_ID id )
+{
+	/*
+	 * binary search of id in ids
+	 * if found, returns position of id
+	 * if not found, returns first position greater than id
+	 */
+	unsigned base = 0;
+	unsigned cursor = 1;
+	int val = 0;
+	unsigned n = ids[0];
+
+	while( 0 < n ) {
+		unsigned pivot = n >> 1;
+		cursor = base + pivot + 1;
+		val = CMP( ids[cursor], id );
+
+		if( val < 0 ) {
+			n = pivot;
+
+		} else if ( val > 0 ) {
+			base = cursor;
+			n -= pivot + 1;
+
+		} else {
+			return cursor;
+		}
+	}
+
+	if( val > 0 ) {
+		++cursor;
+	}
+	return cursor;
+}
+
+#if 0	/* superseded by append/sort */
+int mdb_midl_insert( MDB_IDL ids, MDB_ID id )
+{
+	unsigned x, i;
+
+	x = mdb_midl_search( ids, id );
+	assert( x > 0 );
+
+	if( x < 1 ) {
+		/* internal error */
+		return -2;
+	}
+
+	if ( x <= ids[0] && ids[x] == id ) {
+		/* duplicate */
+		assert(0);
+		return -1;
+	}
+
+	if ( ++ids[0] >= MDB_IDL_DB_MAX ) {
+		/* no room */
+		--ids[0];
+		return -2;
+
+	} else {
+		/* insert id */
+		for (i=ids[0]; i>x; i--)
+			ids[i] = ids[i-1];
+		ids[x] = id;
+	}
+
+	return 0;
+}
+#endif
+
+MDB_IDL mdb_midl_alloc(int num)
+{
+	MDB_IDL ids = malloc((num+2) * sizeof(MDB_ID));
+	if (ids) {
+		*ids++ = num;
+		*ids = 0;
+	}
+	return ids;
+}
+
+void mdb_midl_free(MDB_IDL ids)
+{
+	if (ids)
+		free(ids-1);
+}
+
+int mdb_midl_shrink( MDB_IDL *idp )
+{
+	MDB_IDL ids = *idp;
+	if (*(--ids) > MDB_IDL_UM_MAX &&
+		(ids = realloc(ids, (MDB_IDL_UM_MAX+1) * sizeof(MDB_ID))))
+	{
+		*ids++ = MDB_IDL_UM_MAX;
+		*idp = ids;
+		return 1;
+	}
+	return 0;
+}
+
+static int mdb_midl_grow( MDB_IDL *idp, int num )
+{
+	MDB_IDL idn = *idp-1;
+	/* grow it */
+	idn = realloc(idn, (*idn + num + 2) * sizeof(MDB_ID));
+	if (!idn)
+		return ENOMEM;
+	*idn++ += num;
+	*idp = idn;
+	return 0;
+}
+
+int mdb_midl_need( MDB_IDL *idp, unsigned num )
+{
+	MDB_IDL ids = *idp;
+	num += ids[0];
+	if (num > ids[-1]) {
+		num = (num + num/4 + (256 + 2)) & -256;
+		if (!(ids = realloc(ids-1, num * sizeof(MDB_ID))))
+			return ENOMEM;
+		*ids++ = num - 2;
+		*idp = ids;
+	}
+	return 0;
+}
+
+int mdb_midl_append( MDB_IDL *idp, MDB_ID id )
+{
+	MDB_IDL ids = *idp;
+	/* Too big? */
+	if (ids[0] >= ids[-1]) {
+		if (mdb_midl_grow(idp, MDB_IDL_UM_MAX))
+			return ENOMEM;
+		ids = *idp;
+	}
+	ids[0]++;
+	ids[ids[0]] = id;
+	return 0;
+}
+
+int mdb_midl_append_list( MDB_IDL *idp, MDB_IDL app )
+{
+	MDB_IDL ids = *idp;
+	/* Too big? */
+	if (ids[0] + app[0] >= ids[-1]) {
+		if (mdb_midl_grow(idp, app[0]))
+			return ENOMEM;
+		ids = *idp;
+	}
+	memcpy(&ids[ids[0]+1], &app[1], app[0] * sizeof(MDB_ID));
+	ids[0] += app[0];
+	return 0;
+}
+
+int mdb_midl_append_range( MDB_IDL *idp, MDB_ID id, unsigned n )
+{
+	MDB_ID *ids = *idp, len = ids[0];
+	/* Too big? */
+	if (len + n > ids[-1]) {
+		if (mdb_midl_grow(idp, n | MDB_IDL_UM_MAX))
+			return ENOMEM;
+		ids = *idp;
+	}
+	ids[0] = len + n;
+	ids += len;
+	while (n)
+		ids[n--] = id++;
+	return 0;
+}
+
+void mdb_midl_xmerge( MDB_IDL idl, MDB_IDL merge )
+{
+	MDB_ID old_id, merge_id, i = merge[0], j = idl[0], k = i+j, total = k;
+	idl[0] = (MDB_ID)-1;		/* delimiter for idl scan below */
+	old_id = idl[j];
+	while (i) {
+		merge_id = merge[i--];
+		for (; old_id < merge_id; old_id = idl[--j])
+			idl[k--] = old_id;
+		idl[k--] = merge_id;
+	}
+	idl[0] = total;
+}
+
+/* Quicksort + Insertion sort for small arrays */
+
+#define SMALL	8
+#define	MIDL_SWAP(a,b)	{ itmp=(a); (a)=(b); (b)=itmp; }
+
+void
+mdb_midl_sort( MDB_IDL ids )
+{
+	/* Max possible depth of int-indexed tree * 2 items/level */
+	int istack[sizeof(int)*CHAR_BIT * 2];
+	int i,j,k,l,ir,jstack;
+	MDB_ID a, itmp;
+
+	ir = (int)ids[0];
+	l = 1;
+	jstack = 0;
+	for(;;) {
+		if (ir - l < SMALL) {	/* Insertion sort */
+			for (j=l+1;j<=ir;j++) {
+				a = ids[j];
+				for (i=j-1;i>=1;i--) {
+					if (ids[i] >= a) break;
+					ids[i+1] = ids[i];
+				}
+				ids[i+1] = a;
+			}
+			if (jstack == 0) break;
+			ir = istack[jstack--];
+			l = istack[jstack--];
+		} else {
+			k = (l + ir) >> 1;	/* Choose median of left, center, right */
+			MIDL_SWAP(ids[k], ids[l+1]);
+			if (ids[l] < ids[ir]) {
+				MIDL_SWAP(ids[l], ids[ir]);
+			}
+			if (ids[l+1] < ids[ir]) {
+				MIDL_SWAP(ids[l+1], ids[ir]);
+			}
+			if (ids[l] < ids[l+1]) {
+				MIDL_SWAP(ids[l], ids[l+1]);
+			}
+			i = l+1;
+			j = ir;
+			a = ids[l+1];
+			for(;;) {
+				do i++; while(ids[i] > a);
+				do j--; while(ids[j] < a);
+				if (j < i) break;
+				MIDL_SWAP(ids[i],ids[j]);
+			}
+			ids[l+1] = ids[j];
+			ids[j] = a;
+			jstack += 2;
+			if (ir-i+1 >= j-l) {
+				istack[jstack] = ir;
+				istack[jstack-1] = i;
+				ir = j-1;
+			} else {
+				istack[jstack] = j-1;
+				istack[jstack-1] = l;
+				l = i;
+			}
+		}
+	}
+}
+
+unsigned mdb_mid2l_search( MDB_ID2L ids, MDB_ID id )
+{
+	/*
+	 * binary search of id in ids
+	 * if found, returns position of id
+	 * if not found, returns first position greater than id
+	 */
+	unsigned base = 0;
+	unsigned cursor = 1;
+	int val = 0;
+	unsigned n = (unsigned)ids[0].mid;
+
+	while( 0 < n ) {
+		unsigned pivot = n >> 1;
+		cursor = base + pivot + 1;
+		val = CMP( id, ids[cursor].mid );
+
+		if( val < 0 ) {
+			n = pivot;
+
+		} else if ( val > 0 ) {
+			base = cursor;
+			n -= pivot + 1;
+
+		} else {
+			return cursor;
+		}
+	}
+
+	if( val > 0 ) {
+		++cursor;
+	}
+	return cursor;
+}
+
+int mdb_mid2l_insert( MDB_ID2L ids, MDB_ID2 *id )
+{
+	unsigned x, i;
+
+	x = mdb_mid2l_search( ids, id->mid );
+
+	if( x < 1 ) {
+		/* internal error */
+		return -2;
+	}
+
+	if ( x <= ids[0].mid && ids[x].mid == id->mid ) {
+		/* duplicate */
+		return -1;
+	}
+
+	if ( ids[0].mid >= MDB_IDL_UM_MAX ) {
+		/* too big */
+		return -2;
+
+	} else {
+		/* insert id */
+		ids[0].mid++;
+		for (i=(unsigned)ids[0].mid; i>x; i--)
+			ids[i] = ids[i-1];
+		ids[x] = *id;
+	}
+
+	return 0;
+}
+
+int mdb_mid2l_append( MDB_ID2L ids, MDB_ID2 *id )
+{
+	/* Too big? */
+	if (ids[0].mid >= MDB_IDL_UM_MAX) {
+		return -2;
+	}
+	ids[0].mid++;
+	ids[ids[0].mid] = *id;
+	return 0;
+}
+
+/** @} */
+/** @} */

external/db_drivers/liblmdb32/midl.h

@@ -0,0 +1,190 @@
+/**	@file midl.h
+ *	@brief LMDB ID List header file.
+ *
+ *	This file was originally part of back-bdb but has been
+ *	modified for use in libmdb. Most of the macros defined
+ *	in this file are unused, just left over from the original.
+ *
+ *	This file is only used internally in libmdb and its definitions
+ *	are not exposed publicly.
+ */
+/* $OpenLDAP$ */
+/* This work is part of OpenLDAP Software <http://www.openldap.org/>.
+ *
+ * Copyright 2000-2014 The OpenLDAP Foundation.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted only as authorized by the OpenLDAP
+ * Public License.
+ *
+ * A copy of this license is available in the file LICENSE in the
+ * top-level directory of the distribution or, alternatively, at
+ * <http://www.OpenLDAP.org/license.html>.
+ */
+
+#ifndef _MDB_MIDL_H_
+#define _MDB_MIDL_H_
+
+#include <stddef.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/** @defgroup internal	LMDB Internals
+ *	@{
+ */
+
+/** @defgroup idls	ID List Management
+ *	@{
+ */
+	/** A generic unsigned ID number. These were entryIDs in back-bdb.
+	 *	Preferably it should have the same size as a pointer.
+	 */
+typedef size_t MDB_ID;
+
+	/** An IDL is an ID List, a sorted array of IDs. The first
+	 * element of the array is a counter for how many actual
+	 * IDs are in the list. In the original back-bdb code, IDLs are
+	 * sorted in ascending order. For libmdb IDLs are sorted in
+	 * descending order.
+	 */
+typedef MDB_ID *MDB_IDL;
+
+/* IDL sizes - likely should be even bigger
+ *   limiting factors: sizeof(ID), thread stack size
+ */
+#ifdef VL32
+#define	MDB_IDL_LOGN	10	/* DB_SIZE is 2^10, UM_SIZE is 2^11 */
+#else
+#define	MDB_IDL_LOGN	16	/* DB_SIZE is 2^16, UM_SIZE is 2^17 */
+#endif
+#define MDB_IDL_DB_SIZE		(1<<MDB_IDL_LOGN)
+#define MDB_IDL_UM_SIZE		(1<<(MDB_IDL_LOGN+1))
+
+#define MDB_IDL_DB_MAX		(MDB_IDL_DB_SIZE-1)
+#define MDB_IDL_UM_MAX		(MDB_IDL_UM_SIZE-1)
+
+#define MDB_IDL_SIZEOF(ids)		(((ids)[0]+1) * sizeof(MDB_ID))
+#define MDB_IDL_IS_ZERO(ids) ( (ids)[0] == 0 )
+#define MDB_IDL_CPY( dst, src ) (memcpy( dst, src, MDB_IDL_SIZEOF( src ) ))
+#define MDB_IDL_FIRST( ids )	( (ids)[1] )
+#define MDB_IDL_LAST( ids )		( (ids)[(ids)[0]] )
+
+	/** Current max length of an #mdb_midl_alloc()ed IDL */
+#define MDB_IDL_ALLOCLEN( ids )	( (ids)[-1] )
+
+	/** Append ID to IDL. The IDL must be big enough. */
+#define mdb_midl_xappend(idl, id) do { \
+		MDB_ID *xidl = (idl), xlen = ++(xidl[0]); \
+		xidl[xlen] = (id); \
+	} while (0)
+
+	/** Search for an ID in an IDL.
+	 * @param[in] ids	The IDL to search.
+	 * @param[in] id	The ID to search for.
+	 * @return	The index of the first ID greater than or equal to \b id.
+	 */
+unsigned mdb_midl_search( MDB_IDL ids, MDB_ID id );
+
+	/** Allocate an IDL.
+	 * Allocates memory for an IDL of the given size.
+	 * @return	IDL on success, NULL on failure.
+	 */
+MDB_IDL mdb_midl_alloc(int num);
+
+	/** Free an IDL.
+	 * @param[in] ids	The IDL to free.
+	 */
+void mdb_midl_free(MDB_IDL ids);
+
+	/** Shrink an IDL.
+	 * Return the IDL to the default size if it has grown larger.
+	 * @param[in,out] idp	Address of the IDL to shrink.
+	 * @return	0 on no change, non-zero if shrunk.
+	 */
+int mdb_midl_shrink(MDB_IDL *idp);
+
+	/** Make room for num additional elements in an IDL.
+	 * @param[in,out] idp	Address of the IDL.
+	 * @param[in] num	Number of elements to make room for.
+	 * @return	0 on success, ENOMEM on failure.
+	 */
+int mdb_midl_need(MDB_IDL *idp, unsigned num);
+
+	/** Append an ID onto an IDL.
+	 * @param[in,out] idp	Address of the IDL to append to.
+	 * @param[in] id	The ID to append.
+	 * @return	0 on success, ENOMEM if the IDL is too large.
+	 */
+int mdb_midl_append( MDB_IDL *idp, MDB_ID id );
+
+	/** Append an IDL onto an IDL.
+	 * @param[in,out] idp	Address of the IDL to append to.
+	 * @param[in] app	The IDL to append.
+	 * @return	0 on success, ENOMEM if the IDL is too large.
+	 */
+int mdb_midl_append_list( MDB_IDL *idp, MDB_IDL app );
+
+	/** Append an ID range onto an IDL.
+	 * @param[in,out] idp	Address of the IDL to append to.
+	 * @param[in] id	The lowest ID to append.
+	 * @param[in] n		Number of IDs to append.
+	 * @return	0 on success, ENOMEM if the IDL is too large.
+	 */
+int mdb_midl_append_range( MDB_IDL *idp, MDB_ID id, unsigned n );
+
+	/** Merge an IDL onto an IDL. The destination IDL must be big enough.
+	 * @param[in] idl	The IDL to merge into.
+	 * @param[in] merge	The IDL to merge.
+	 */
+void mdb_midl_xmerge( MDB_IDL idl, MDB_IDL merge );
+
+	/** Sort an IDL.
+	 * @param[in,out] ids	The IDL to sort.
+	 */
+void mdb_midl_sort( MDB_IDL ids );
+
+	/** An ID2 is an ID/pointer pair.
+	 */
+typedef struct MDB_ID2 {
+	MDB_ID mid;		/**< The ID */
+	void *mptr;		/**< The pointer */
+} MDB_ID2;
+
+	/** An ID2L is an ID2 List, a sorted array of ID2s.
+	 * The first element's \b mid member is a count of how many actual
+	 * elements are in the array. The \b mptr member of the first element is unused.
+	 * The array is sorted in ascending order by \b mid.
+	 */
+typedef MDB_ID2 *MDB_ID2L;
+
+	/** Search for an ID in an ID2L.
+	 * @param[in] ids	The ID2L to search.
+	 * @param[in] id	The ID to search for.
+	 * @return	The index of the first ID2 whose \b mid member is greater than or equal to \b id.
+	 */
+unsigned mdb_mid2l_search( MDB_ID2L ids, MDB_ID id );
+
+
+	/** Insert an ID2 into a ID2L.
+	 * @param[in,out] ids	The ID2L to insert into.
+	 * @param[in] id	The ID2 to insert.
+	 * @return	0 on success, -1 if the ID was already present in the ID2L.
+	 */
+int mdb_mid2l_insert( MDB_ID2L ids, MDB_ID2 *id );
+
+	/** Append an ID2 into a ID2L.
+	 * @param[in,out] ids	The ID2L to append into.
+	 * @param[in] id	The ID2 to append.
+	 * @return	0 on success, -2 if the ID2L is too big.
+	 */
+int mdb_mid2l_append( MDB_ID2L ids, MDB_ID2 *id );
+
+/** @} */
+/** @} */
+#ifdef __cplusplus
+}
+#endif
+#endif	/* _MDB_MIDL_H_ */

external/db_drivers/liblmdb32/mtest.c

@@ -0,0 +1,173 @@
+/* mtest.c - memory-mapped database tester/toy */
+/*
+ * Copyright 2011-2014 Howard Chu, Symas Corp.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted only as authorized by the OpenLDAP
+ * Public License.
+ *
+ * A copy of this license is available in the file LICENSE in the
+ * top-level directory of the distribution or, alternatively, at
+ * <http://www.OpenLDAP.org/license.html>.
+ */
+#include <stdio.h>
+#include <stdlib.h>
+#include <time.h>
+#include "lmdb.h"
+
+#define E(expr) CHECK((rc = (expr)) == MDB_SUCCESS, #expr)
+#define RES(err, expr) ((rc = expr) == (err) || (CHECK(!rc, #expr), 0))
+#define CHECK(test, msg) ((test) ? (void)0 : ((void)fprintf(stderr, \
+	"%s:%d: %s: %s\n", __FILE__, __LINE__, msg, mdb_strerror(rc)), abort()))
+
+int main(int argc,char * argv[])
+{
+	int i = 0, j = 0, rc;
+	MDB_env *env;
+	MDB_dbi dbi;
+	MDB_val key, data;
+	MDB_txn *txn;
+	MDB_stat mst;
+	MDB_cursor *cursor, *cur2;
+	MDB_cursor_op op;
+	int count;
+	int *values;
+	char sval[32] = "";
+
+	srand(time(NULL));
+
+	    count = (rand()%384) + 64;
+	    values = (int *)malloc(count*sizeof(int));
+
+	    for(i = 0;i<count;i++) {
+			values[i] = rand()%1024;
+	    }
+    
+		E(mdb_env_create(&env));
+		E(mdb_env_set_mapsize(env, 10485760));
+		E(mdb_env_open(env, "./testdb", 0 /* MDB_FIXEDMAP |MDB_NOSYNC*/, 0664));
+		E(mdb_txn_begin(env, NULL, 0, &txn));
+		E(mdb_open(txn, NULL, 0, &dbi));
+   
+		key.mv_size = sizeof(int);
+		key.mv_data = sval;
+		data.mv_size = sizeof(sval);
+		data.mv_data = sval;
+
+		printf("Adding %d values\n", count);
+	    for (i=0;i<count;i++) {	
+			sprintf(sval, "%03x %d foo bar", values[i], values[i]);
+			if (RES(MDB_KEYEXIST, mdb_put(txn, dbi, &key, &data, MDB_NOOVERWRITE))) {
+				j++;
+				data.mv_size = sizeof(sval);
+				data.mv_data = sval;
+			}
+	    }
+		if (j) printf("%d duplicates skipped\n", j);
+		E(mdb_txn_commit(txn));
+		E(mdb_env_stat(env, &mst));
+
+		E(mdb_txn_begin(env, NULL, 1, &txn));
+		E(mdb_cursor_open(txn, dbi, &cursor));
+		while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
+			printf("key: %p %.*s, data: %p %.*s\n",
+				key.mv_data,  (int) key.mv_size,  (char *) key.mv_data,
+				data.mv_data, (int) data.mv_size, (char *) data.mv_data);
+		}
+		CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+		mdb_cursor_close(cursor);
+		mdb_txn_abort(txn);
+
+		j=0;
+		key.mv_data = sval;
+	    for (i= count - 1; i > -1; i-= (rand()%5)) {
+			j++;
+			txn=NULL;
+			E(mdb_txn_begin(env, NULL, 0, &txn));
+			sprintf(sval, "%03x ", values[i]);
+			if (RES(MDB_NOTFOUND, mdb_del(txn, dbi, &key, NULL))) {
+				j--;
+				mdb_txn_abort(txn);
+			} else {
+				E(mdb_txn_commit(txn));
+			}
+	    }
+	    free(values);
+		printf("Deleted %d values\n", j);
+
+		E(mdb_env_stat(env, &mst));
+		E(mdb_txn_begin(env, NULL, 1, &txn));
+		E(mdb_cursor_open(txn, dbi, &cursor));
+		printf("Cursor next\n");
+		while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
+			printf("key: %.*s, data: %.*s\n",
+				(int) key.mv_size,  (char *) key.mv_data,
+				(int) data.mv_size, (char *) data.mv_data);
+		}
+		CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+		printf("Cursor last\n");
+		E(mdb_cursor_get(cursor, &key, &data, MDB_LAST));
+		printf("key: %.*s, data: %.*s\n",
+			(int) key.mv_size,  (char *) key.mv_data,
+			(int) data.mv_size, (char *) data.mv_data);
+		printf("Cursor prev\n");
+		while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_PREV)) == 0) {
+			printf("key: %.*s, data: %.*s\n",
+				(int) key.mv_size,  (char *) key.mv_data,
+				(int) data.mv_size, (char *) data.mv_data);
+		}
+		CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+		printf("Cursor last/prev\n");
+		E(mdb_cursor_get(cursor, &key, &data, MDB_LAST));
+			printf("key: %.*s, data: %.*s\n",
+				(int) key.mv_size,  (char *) key.mv_data,
+				(int) data.mv_size, (char *) data.mv_data);
+		E(mdb_cursor_get(cursor, &key, &data, MDB_PREV));
+			printf("key: %.*s, data: %.*s\n",
+				(int) key.mv_size,  (char *) key.mv_data,
+				(int) data.mv_size, (char *) data.mv_data);
+
+		mdb_txn_abort(txn);
+
+		printf("Deleting with cursor\n");
+		E(mdb_txn_begin(env, NULL, 0, &txn));
+		E(mdb_cursor_open(txn, dbi, &cur2));
+		for (i=0; i<50; i++) {
+			if (RES(MDB_NOTFOUND, mdb_cursor_get(cur2, &key, &data, MDB_NEXT)))
+				break;
+			printf("key: %p %.*s, data: %p %.*s\n",
+				key.mv_data,  (int) key.mv_size,  (char *) key.mv_data,
+				data.mv_data, (int) data.mv_size, (char *) data.mv_data);
+			E(mdb_del(txn, dbi, &key, NULL));
+		}
+
+		printf("Restarting cursor in txn\n");
+		for (op=MDB_FIRST, i=0; i<=32; op=MDB_NEXT, i++) {
+			if (RES(MDB_NOTFOUND, mdb_cursor_get(cur2, &key, &data, op)))
+				break;
+			printf("key: %p %.*s, data: %p %.*s\n",
+				key.mv_data,  (int) key.mv_size,  (char *) key.mv_data,
+				data.mv_data, (int) data.mv_size, (char *) data.mv_data);
+		}
+		mdb_cursor_close(cur2);
+		E(mdb_txn_commit(txn));
+
+		printf("Restarting cursor outside txn\n");
+		E(mdb_txn_begin(env, NULL, 0, &txn));
+		E(mdb_cursor_open(txn, dbi, &cursor));
+		for (op=MDB_FIRST, i=0; i<=32; op=MDB_NEXT, i++) {
+			if (RES(MDB_NOTFOUND, mdb_cursor_get(cursor, &key, &data, op)))
+				break;
+			printf("key: %p %.*s, data: %p %.*s\n",
+				key.mv_data,  (int) key.mv_size,  (char *) key.mv_data,
+				data.mv_data, (int) data.mv_size, (char *) data.mv_data);
+		}
+		mdb_cursor_close(cursor);
+		mdb_close(env, dbi);
+
+		mdb_txn_abort(txn);
+		mdb_env_close(env);
+
+	return 0;
+}

external/db_drivers/liblmdb32/mtest2.c

@@ -0,0 +1,123 @@
+/* mtest2.c - memory-mapped database tester/toy */
+/*
+ * Copyright 2011-2014 Howard Chu, Symas Corp.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted only as authorized by the OpenLDAP
+ * Public License.
+ *
+ * A copy of this license is available in the file LICENSE in the
+ * top-level directory of the distribution or, alternatively, at
+ * <http://www.OpenLDAP.org/license.html>.
+ */
+
+/* Just like mtest.c, but using a subDB instead of the main DB */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <time.h>
+#include "lmdb.h"
+
+#define E(expr) CHECK((rc = (expr)) == MDB_SUCCESS, #expr)
+#define RES(err, expr) ((rc = expr) == (err) || (CHECK(!rc, #expr), 0))
+#define CHECK(test, msg) ((test) ? (void)0 : ((void)fprintf(stderr, \
+	"%s:%d: %s: %s\n", __FILE__, __LINE__, msg, mdb_strerror(rc)), abort()))
+
+int main(int argc,char * argv[])
+{
+	int i = 0, j = 0, rc;
+	MDB_env *env;
+	MDB_dbi dbi;
+	MDB_val key, data;
+	MDB_txn *txn;
+	MDB_stat mst;
+	MDB_cursor *cursor;
+	int count;
+	int *values;
+	char sval[32] = "";
+
+	srand(time(NULL));
+
+	count = (rand()%384) + 64;
+	values = (int *)malloc(count*sizeof(int));
+
+	for(i = 0;i<count;i++) {
+		values[i] = rand()%1024;
+	}
+
+	E(mdb_env_create(&env));
+	E(mdb_env_set_mapsize(env, 10485760));
+	E(mdb_env_set_maxdbs(env, 4));
+	E(mdb_env_open(env, "./testdb", MDB_FIXEDMAP|MDB_NOSYNC, 0664));
+	E(mdb_txn_begin(env, NULL, 0, &txn));
+	E(mdb_open(txn, "id1", MDB_CREATE, &dbi));
+   
+	key.mv_size = sizeof(int);
+	key.mv_data = sval;
+	data.mv_size = sizeof(sval);
+	data.mv_data = sval;
+
+	printf("Adding %d values\n", count);
+	for (i=0;i<count;i++) {	
+		sprintf(sval, "%03x %d foo bar", values[i], values[i]);
+		if (RES(MDB_KEYEXIST, mdb_put(txn, dbi, &key, &data, MDB_NOOVERWRITE)))
+			j++;
+	}
+	if (j) printf("%d duplicates skipped\n", j);
+	E(mdb_txn_commit(txn));
+	E(mdb_env_stat(env, &mst));
+
+	E(mdb_txn_begin(env, NULL, 1, &txn));
+	E(mdb_cursor_open(txn, dbi, &cursor));
+	while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
+		printf("key: %p %.*s, data: %p %.*s\n",
+			key.mv_data,  (int) key.mv_size,  (char *) key.mv_data,
+			data.mv_data, (int) data.mv_size, (char *) data.mv_data);
+	}
+	CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+	mdb_cursor_close(cursor);
+	mdb_txn_abort(txn);
+
+	j=0;
+	key.mv_data = sval;
+	for (i= count - 1; i > -1; i-= (rand()%5)) {
+		j++;
+		txn=NULL;
+		E(mdb_txn_begin(env, NULL, 0, &txn));
+		sprintf(sval, "%03x ", values[i]);
+		if (RES(MDB_NOTFOUND, mdb_del(txn, dbi, &key, NULL))) {
+			j--;
+			mdb_txn_abort(txn);
+		} else {
+			E(mdb_txn_commit(txn));
+		}
+	}
+	free(values);
+	printf("Deleted %d values\n", j);
+
+	E(mdb_env_stat(env, &mst));
+	E(mdb_txn_begin(env, NULL, 1, &txn));
+	E(mdb_cursor_open(txn, dbi, &cursor));
+	printf("Cursor next\n");
+	while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
+		printf("key: %.*s, data: %.*s\n",
+			(int) key.mv_size,  (char *) key.mv_data,
+			(int) data.mv_size, (char *) data.mv_data);
+	}
+	CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+	printf("Cursor prev\n");
+	while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_PREV)) == 0) {
+		printf("key: %.*s, data: %.*s\n",
+			(int) key.mv_size,  (char *) key.mv_data,
+			(int) data.mv_size, (char *) data.mv_data);
+	}
+	CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+	mdb_cursor_close(cursor);
+	mdb_close(env, dbi);
+
+	mdb_txn_abort(txn);
+	mdb_env_close(env);
+
+	return 0;
+}

external/db_drivers/liblmdb32/mtest3.c

@@ -0,0 +1,133 @@
+/* mtest3.c - memory-mapped database tester/toy */
+/*
+ * Copyright 2011-2014 Howard Chu, Symas Corp.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted only as authorized by the OpenLDAP
+ * Public License.
+ *
+ * A copy of this license is available in the file LICENSE in the
+ * top-level directory of the distribution or, alternatively, at
+ * <http://www.OpenLDAP.org/license.html>.
+ */
+
+/* Tests for sorted duplicate DBs */
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <time.h>
+#include "lmdb.h"
+
+#define E(expr) CHECK((rc = (expr)) == MDB_SUCCESS, #expr)
+#define RES(err, expr) ((rc = expr) == (err) || (CHECK(!rc, #expr), 0))
+#define CHECK(test, msg) ((test) ? (void)0 : ((void)fprintf(stderr, \
+	"%s:%d: %s: %s\n", __FILE__, __LINE__, msg, mdb_strerror(rc)), abort()))
+
+int main(int argc,char * argv[])
+{
+	int i = 0, j = 0, rc;
+	MDB_env *env;
+	MDB_dbi dbi;
+	MDB_val key, data;
+	MDB_txn *txn;
+	MDB_stat mst;
+	MDB_cursor *cursor;
+	int count;
+	int *values;
+	char sval[32];
+	char kval[sizeof(int)];
+
+	srand(time(NULL));
+
+	memset(sval, 0, sizeof(sval));
+
+	count = (rand()%384) + 64;
+	values = (int *)malloc(count*sizeof(int));
+
+	for(i = 0;i<count;i++) {
+		values[i] = rand()%1024;
+	}
+
+	E(mdb_env_create(&env));
+	E(mdb_env_set_mapsize(env, 10485760));
+	E(mdb_env_set_maxdbs(env, 4));
+	E(mdb_env_open(env, "./testdb", MDB_FIXEDMAP|MDB_NOSYNC, 0664));
+	E(mdb_txn_begin(env, NULL, 0, &txn));
+	E(mdb_open(txn, "id2", MDB_CREATE|MDB_DUPSORT, &dbi));
+
+	key.mv_size = sizeof(int);
+	key.mv_data = kval;
+	data.mv_size = sizeof(sval);
+	data.mv_data = sval;
+
+	printf("Adding %d values\n", count);
+	for (i=0;i<count;i++) {
+		if (!(i & 0x0f))
+			sprintf(kval, "%03x", values[i]);
+		sprintf(sval, "%03x %d foo bar", values[i], values[i]);
+		if (RES(MDB_KEYEXIST, mdb_put(txn, dbi, &key, &data, MDB_NODUPDATA)))
+			j++;
+	}
+	if (j) printf("%d duplicates skipped\n", j);
+	E(mdb_txn_commit(txn));
+	E(mdb_env_stat(env, &mst));
+
+	E(mdb_txn_begin(env, NULL, 1, &txn));
+	E(mdb_cursor_open(txn, dbi, &cursor));
+	while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
+		printf("key: %p %.*s, data: %p %.*s\n",
+			key.mv_data,  (int) key.mv_size,  (char *) key.mv_data,
+			data.mv_data, (int) data.mv_size, (char *) data.mv_data);
+	}
+	CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+	mdb_cursor_close(cursor);
+	mdb_txn_abort(txn);
+
+	j=0;
+
+	for (i= count - 1; i > -1; i-= (rand()%5)) {
+		j++;
+		txn=NULL;
+		E(mdb_txn_begin(env, NULL, 0, &txn));
+		sprintf(kval, "%03x", values[i & ~0x0f]);
+		sprintf(sval, "%03x %d foo bar", values[i], values[i]);
+		key.mv_size = sizeof(int);
+		key.mv_data = kval;
+		data.mv_size = sizeof(sval);
+		data.mv_data = sval;
+		if (RES(MDB_NOTFOUND, mdb_del(txn, dbi, &key, &data))) {
+			j--;
+			mdb_txn_abort(txn);
+		} else {
+			E(mdb_txn_commit(txn));
+		}
+	}
+	free(values);
+	printf("Deleted %d values\n", j);
+
+	E(mdb_env_stat(env, &mst));
+	E(mdb_txn_begin(env, NULL, 1, &txn));
+	E(mdb_cursor_open(txn, dbi, &cursor));
+	printf("Cursor next\n");
+	while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
+		printf("key: %.*s, data: %.*s\n",
+			(int) key.mv_size,  (char *) key.mv_data,
+			(int) data.mv_size, (char *) data.mv_data);
+	}
+	CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+	printf("Cursor prev\n");
+	while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_PREV)) == 0) {
+		printf("key: %.*s, data: %.*s\n",
+			(int) key.mv_size,  (char *) key.mv_data,
+			(int) data.mv_size, (char *) data.mv_data);
+	}
+	CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+	mdb_cursor_close(cursor);
+	mdb_close(env, dbi);
+
+	mdb_txn_abort(txn);
+	mdb_env_close(env);
+
+	return 0;
+}

external/db_drivers/liblmdb32/mtest4.c

@@ -0,0 +1,168 @@
+/* mtest4.c - memory-mapped database tester/toy */
+/*
+ * Copyright 2011-2014 Howard Chu, Symas Corp.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted only as authorized by the OpenLDAP
+ * Public License.
+ *
+ * A copy of this license is available in the file LICENSE in the
+ * top-level directory of the distribution or, alternatively, at
+ * <http://www.OpenLDAP.org/license.html>.
+ */
+
+/* Tests for sorted duplicate DBs with fixed-size keys */
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <time.h>
+#include "lmdb.h"
+
+#define E(expr) CHECK((rc = (expr)) == MDB_SUCCESS, #expr)
+#define RES(err, expr) ((rc = expr) == (err) || (CHECK(!rc, #expr), 0))
+#define CHECK(test, msg) ((test) ? (void)0 : ((void)fprintf(stderr, \
+	"%s:%d: %s: %s\n", __FILE__, __LINE__, msg, mdb_strerror(rc)), abort()))
+
+int main(int argc,char * argv[])
+{
+	int i = 0, j = 0, rc;
+	MDB_env *env;
+	MDB_dbi dbi;
+	MDB_val key, data;
+	MDB_txn *txn;
+	MDB_stat mst;
+	MDB_cursor *cursor;
+	int count;
+	int *values;
+	char sval[8];
+	char kval[sizeof(int)];
+
+	memset(sval, 0, sizeof(sval));
+
+	count = 510;
+	values = (int *)malloc(count*sizeof(int));
+
+	for(i = 0;i<count;i++) {
+		values[i] = i*5;
+	}
+
+	E(mdb_env_create(&env));
+	E(mdb_env_set_mapsize(env, 10485760));
+	E(mdb_env_set_maxdbs(env, 4));
+	E(mdb_env_open(env, "./testdb", MDB_FIXEDMAP|MDB_NOSYNC, 0664));
+	E(mdb_txn_begin(env, NULL, 0, &txn));
+	E(mdb_open(txn, "id4", MDB_CREATE|MDB_DUPSORT|MDB_DUPFIXED, &dbi));
+
+	key.mv_size = sizeof(int);
+	key.mv_data = kval;
+	data.mv_size = sizeof(sval);
+	data.mv_data = sval;
+
+	printf("Adding %d values\n", count);
+	strcpy(kval, "001");
+	for (i=0;i<count;i++) {
+		sprintf(sval, "%07x", values[i]);
+		if (RES(MDB_KEYEXIST, mdb_put(txn, dbi, &key, &data, MDB_NODUPDATA)))
+			j++;
+	}
+	if (j) printf("%d duplicates skipped\n", j);
+	E(mdb_txn_commit(txn));
+	E(mdb_env_stat(env, &mst));
+
+	/* there should be one full page of dups now.
+	 */
+	E(mdb_txn_begin(env, NULL, 1, &txn));
+	E(mdb_cursor_open(txn, dbi, &cursor));
+	while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
+		printf("key: %p %.*s, data: %p %.*s\n",
+			key.mv_data,  (int) key.mv_size,  (char *) key.mv_data,
+			data.mv_data, (int) data.mv_size, (char *) data.mv_data);
+	}
+	CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+	mdb_cursor_close(cursor);
+	mdb_txn_abort(txn);
+
+	/* test all 3 branches of split code:
+	 * 1: new key in lower half
+	 * 2: new key at split point
+	 * 3: new key in upper half
+	 */
+
+	key.mv_size = sizeof(int);
+	key.mv_data = kval;
+	data.mv_size = sizeof(sval);
+	data.mv_data = sval;
+
+	sprintf(sval, "%07x", values[3]+1);
+	E(mdb_txn_begin(env, NULL, 0, &txn));
+	(void)RES(MDB_KEYEXIST, mdb_put(txn, dbi, &key, &data, MDB_NODUPDATA));
+	mdb_txn_abort(txn);
+
+	sprintf(sval, "%07x", values[255]+1);
+	E(mdb_txn_begin(env, NULL, 0, &txn));
+	(void)RES(MDB_KEYEXIST, mdb_put(txn, dbi, &key, &data, MDB_NODUPDATA));
+	mdb_txn_abort(txn);
+
+	sprintf(sval, "%07x", values[500]+1);
+	E(mdb_txn_begin(env, NULL, 0, &txn));
+	(void)RES(MDB_KEYEXIST, mdb_put(txn, dbi, &key, &data, MDB_NODUPDATA));
+	E(mdb_txn_commit(txn));
+
+	/* Try MDB_NEXT_MULTIPLE */
+	E(mdb_txn_begin(env, NULL, 0, &txn));
+	E(mdb_cursor_open(txn, dbi, &cursor));
+	while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT_MULTIPLE)) == 0) {
+		printf("key: %.*s, data: %.*s\n",
+			(int) key.mv_size,  (char *) key.mv_data,
+			(int) data.mv_size, (char *) data.mv_data);
+	}
+	CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+	mdb_cursor_close(cursor);
+	mdb_txn_abort(txn);
+	j=0;
+
+	for (i= count - 1; i > -1; i-= (rand()%3)) {
+		j++;
+		txn=NULL;
+		E(mdb_txn_begin(env, NULL, 0, &txn));
+		sprintf(sval, "%07x", values[i]);
+		key.mv_size = sizeof(int);
+		key.mv_data = kval;
+		data.mv_size = sizeof(sval);
+		data.mv_data = sval;
+		if (RES(MDB_NOTFOUND, mdb_del(txn, dbi, &key, &data))) {
+			j--;
+			mdb_txn_abort(txn);
+		} else {
+			E(mdb_txn_commit(txn));
+		}
+	}
+	free(values);
+	printf("Deleted %d values\n", j);
+
+	E(mdb_env_stat(env, &mst));
+	E(mdb_txn_begin(env, NULL, 1, &txn));
+	E(mdb_cursor_open(txn, dbi, &cursor));
+	printf("Cursor next\n");
+	while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
+		printf("key: %.*s, data: %.*s\n",
+			(int) key.mv_size,  (char *) key.mv_data,
+			(int) data.mv_size, (char *) data.mv_data);
+	}
+	CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+	printf("Cursor prev\n");
+	while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_PREV)) == 0) {
+		printf("key: %.*s, data: %.*s\n",
+			(int) key.mv_size,  (char *) key.mv_data,
+			(int) data.mv_size, (char *) data.mv_data);
+	}
+	CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+	mdb_cursor_close(cursor);
+	mdb_close(env, dbi);
+
+	mdb_txn_abort(txn);
+	mdb_env_close(env);
+
+	return 0;
+}

external/db_drivers/liblmdb32/mtest5.c

@@ -0,0 +1,135 @@
+/* mtest5.c - memory-mapped database tester/toy */
+/*
+ * Copyright 2011-2014 Howard Chu, Symas Corp.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted only as authorized by the OpenLDAP
+ * Public License.
+ *
+ * A copy of this license is available in the file LICENSE in the
+ * top-level directory of the distribution or, alternatively, at
+ * <http://www.OpenLDAP.org/license.html>.
+ */
+
+/* Tests for sorted duplicate DBs using cursor_put */
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <time.h>
+#include "lmdb.h"
+
+#define E(expr) CHECK((rc = (expr)) == MDB_SUCCESS, #expr)
+#define RES(err, expr) ((rc = expr) == (err) || (CHECK(!rc, #expr), 0))
+#define CHECK(test, msg) ((test) ? (void)0 : ((void)fprintf(stderr, \
+	"%s:%d: %s: %s\n", __FILE__, __LINE__, msg, mdb_strerror(rc)), abort()))
+
+int main(int argc,char * argv[])
+{
+	int i = 0, j = 0, rc;
+	MDB_env *env;
+	MDB_dbi dbi;
+	MDB_val key, data;
+	MDB_txn *txn;
+	MDB_stat mst;
+	MDB_cursor *cursor;
+	int count;
+	int *values;
+	char sval[32];
+	char kval[sizeof(int)];
+
+	srand(time(NULL));
+
+	memset(sval, 0, sizeof(sval));
+
+	count = (rand()%384) + 64;
+	values = (int *)malloc(count*sizeof(int));
+
+	for(i = 0;i<count;i++) {
+		values[i] = rand()%1024;
+	}
+
+	E(mdb_env_create(&env));
+	E(mdb_env_set_mapsize(env, 10485760));
+	E(mdb_env_set_maxdbs(env, 4));
+	E(mdb_env_open(env, "./testdb", MDB_FIXEDMAP|MDB_NOSYNC, 0664));
+	E(mdb_txn_begin(env, NULL, 0, &txn));
+	E(mdb_open(txn, "id2", MDB_CREATE|MDB_DUPSORT, &dbi));
+	E(mdb_cursor_open(txn, dbi, &cursor));
+
+	key.mv_size = sizeof(int);
+	key.mv_data = kval;
+	data.mv_size = sizeof(sval);
+	data.mv_data = sval;
+
+	printf("Adding %d values\n", count);
+	for (i=0;i<count;i++) {
+		if (!(i & 0x0f))
+			sprintf(kval, "%03x", values[i]);
+		sprintf(sval, "%03x %d foo bar", values[i], values[i]);
+		if (RES(MDB_KEYEXIST, mdb_cursor_put(cursor, &key, &data, MDB_NODUPDATA)))
+			j++;
+	}
+	if (j) printf("%d duplicates skipped\n", j);
+	mdb_cursor_close(cursor);
+	E(mdb_txn_commit(txn));
+	E(mdb_env_stat(env, &mst));
+
+	E(mdb_txn_begin(env, NULL, 1, &txn));
+	E(mdb_cursor_open(txn, dbi, &cursor));
+	while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
+		printf("key: %p %.*s, data: %p %.*s\n",
+			key.mv_data,  (int) key.mv_size,  (char *) key.mv_data,
+			data.mv_data, (int) data.mv_size, (char *) data.mv_data);
+	}
+	CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+	mdb_cursor_close(cursor);
+	mdb_txn_abort(txn);
+
+	j=0;
+
+	for (i= count - 1; i > -1; i-= (rand()%5)) {
+		j++;
+		txn=NULL;
+		E(mdb_txn_begin(env, NULL, 0, &txn));
+		sprintf(kval, "%03x", values[i & ~0x0f]);
+		sprintf(sval, "%03x %d foo bar", values[i], values[i]);
+		key.mv_size = sizeof(int);
+		key.mv_data = kval;
+		data.mv_size = sizeof(sval);
+		data.mv_data = sval;
+		if (RES(MDB_NOTFOUND, mdb_del(txn, dbi, &key, &data))) {
+			j--;
+			mdb_txn_abort(txn);
+		} else {
+			E(mdb_txn_commit(txn));
+		}
+	}
+	free(values);
+	printf("Deleted %d values\n", j);
+
+	E(mdb_env_stat(env, &mst));
+	E(mdb_txn_begin(env, NULL, 1, &txn));
+	E(mdb_cursor_open(txn, dbi, &cursor));
+	printf("Cursor next\n");
+	while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
+		printf("key: %.*s, data: %.*s\n",
+			(int) key.mv_size,  (char *) key.mv_data,
+			(int) data.mv_size, (char *) data.mv_data);
+	}
+	CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+	printf("Cursor prev\n");
+	while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_PREV)) == 0) {
+		printf("key: %.*s, data: %.*s\n",
+			(int) key.mv_size,  (char *) key.mv_data,
+			(int) data.mv_size, (char *) data.mv_data);
+	}
+	CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+	mdb_cursor_close(cursor);
+	mdb_close(env, dbi);
+
+	mdb_txn_abort(txn);
+	mdb_env_close(env);
+
+	return 0;
+}

external/db_drivers/liblmdb32/mtest6.c

@@ -0,0 +1,137 @@
+/* mtest6.c - memory-mapped database tester/toy */
+/*
+ * Copyright 2011-2014 Howard Chu, Symas Corp.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted only as authorized by the OpenLDAP
+ * Public License.
+ *
+ * A copy of this license is available in the file LICENSE in the
+ * top-level directory of the distribution or, alternatively, at
+ * <http://www.OpenLDAP.org/license.html>.
+ */
+
+/* Tests for DB splits and merges */
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <time.h>
+#include "lmdb.h"
+
+#define E(expr) CHECK((rc = (expr)) == MDB_SUCCESS, #expr)
+#define RES(err, expr) ((rc = expr) == (err) || (CHECK(!rc, #expr), 0))
+#define CHECK(test, msg) ((test) ? (void)0 : ((void)fprintf(stderr, \
+	"%s:%d: %s: %s\n", __FILE__, __LINE__, msg, mdb_strerror(rc)), abort()))
+
+char dkbuf[1024];
+
+int main(int argc,char * argv[])
+{
+	int i = 0, j = 0, rc;
+	MDB_env *env;
+	MDB_dbi dbi;
+	MDB_val key, data;
+	MDB_txn *txn;
+	MDB_stat mst;
+	MDB_cursor *cursor;
+	int count;
+	int *values;
+	long kval;
+	char *sval;
+
+	srand(time(NULL));
+
+	E(mdb_env_create(&env));
+	E(mdb_env_set_mapsize(env, 10485760));
+	E(mdb_env_set_maxdbs(env, 4));
+	E(mdb_env_open(env, "./testdb", MDB_FIXEDMAP|MDB_NOSYNC, 0664));
+	E(mdb_txn_begin(env, NULL, 0, &txn));
+	E(mdb_open(txn, "id6", MDB_CREATE|MDB_INTEGERKEY, &dbi));
+	E(mdb_cursor_open(txn, dbi, &cursor));
+	E(mdb_stat(txn, dbi, &mst));
+
+	sval = calloc(1, mst.ms_psize / 4);
+	key.mv_size = sizeof(long);
+	key.mv_data = &kval;
+	data.mv_size = mst.ms_psize / 4 - 30;
+	data.mv_data = sval;
+
+	printf("Adding 12 values, should yield 3 splits\n");
+	for (i=0;i<12;i++) {
+		kval = i*5;
+		sprintf(sval, "%08x", kval);
+		(void)RES(MDB_KEYEXIST, mdb_cursor_put(cursor, &key, &data, MDB_NOOVERWRITE));
+	}
+	printf("Adding 12 more values, should yield 3 splits\n");
+	for (i=0;i<12;i++) {
+		kval = i*5+4;
+		sprintf(sval, "%08x", kval);
+		(void)RES(MDB_KEYEXIST, mdb_cursor_put(cursor, &key, &data, MDB_NOOVERWRITE));
+	}
+	printf("Adding 12 more values, should yield 3 splits\n");
+	for (i=0;i<12;i++) {
+		kval = i*5+1;
+		sprintf(sval, "%08x", kval);
+		(void)RES(MDB_KEYEXIST, mdb_cursor_put(cursor, &key, &data, MDB_NOOVERWRITE));
+	}
+	E(mdb_cursor_get(cursor, &key, &data, MDB_FIRST));
+
+	do {
+		printf("key: %p %s, data: %p %.*s\n",
+			key.mv_data,  mdb_dkey(&key, dkbuf),
+			data.mv_data, (int) data.mv_size, (char *) data.mv_data);
+	} while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0);
+	CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+	mdb_cursor_close(cursor);
+	mdb_txn_commit(txn);
+
+#if 0
+	j=0;
+
+	for (i= count - 1; i > -1; i-= (rand()%5)) {
+		j++;
+		txn=NULL;
+		E(mdb_txn_begin(env, NULL, 0, &txn));
+		sprintf(kval, "%03x", values[i & ~0x0f]);
+		sprintf(sval, "%03x %d foo bar", values[i], values[i]);
+		key.mv_size = sizeof(int);
+		key.mv_data = kval;
+		data.mv_size = sizeof(sval);
+		data.mv_data = sval;
+		if (RES(MDB_NOTFOUND, mdb_del(txn, dbi, &key, &data))) {
+			j--;
+			mdb_txn_abort(txn);
+		} else {
+			E(mdb_txn_commit(txn));
+		}
+	}
+	free(values);
+	printf("Deleted %d values\n", j);
+
+	E(mdb_env_stat(env, &mst));
+	E(mdb_txn_begin(env, NULL, 1, &txn));
+	E(mdb_cursor_open(txn, dbi, &cursor));
+	printf("Cursor next\n");
+	while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
+		printf("key: %.*s, data: %.*s\n",
+			(int) key.mv_size,  (char *) key.mv_data,
+			(int) data.mv_size, (char *) data.mv_data);
+	}
+	CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+	printf("Cursor prev\n");
+	while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_PREV)) == 0) {
+		printf("key: %.*s, data: %.*s\n",
+			(int) key.mv_size,  (char *) key.mv_data,
+			(int) data.mv_size, (char *) data.mv_data);
+	}
+	CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+	mdb_cursor_close(cursor);
+	mdb_close(env, dbi);
+
+	mdb_txn_abort(txn);
+#endif
+	mdb_env_close(env);
+
+	return 0;
+}

external/db_drivers/liblmdb32/sample-bdb.txt

@@ -0,0 +1,73 @@
+/* sample-bdb.txt - BerkeleyDB toy/sample
+ *
+ * Do a line-by-line comparison of this and sample-mdb.txt
+ */
+/*
+ * Copyright 2012 Howard Chu, Symas Corp.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted only as authorized by the OpenLDAP
+ * Public License.
+ *
+ * A copy of this license is available in the file LICENSE in the
+ * top-level directory of the distribution or, alternatively, at
+ * <http://www.OpenLDAP.org/license.html>.
+ */
+#include <stdio.h>
+#include <string.h>
+#include <db.h>
+
+int main(int argc,char * argv[])
+{
+	int rc;
+	DB_ENV *env;
+	DB *dbi;
+	DBT key, data;
+	DB_TXN *txn;
+	DBC *cursor;
+	char sval[32], kval[32];
+
+	/* Note: Most error checking omitted for simplicity */
+
+#define FLAGS (DB_INIT_LOCK|DB_INIT_LOG|DB_INIT_TXN|DB_INIT_MPOOL|DB_CREATE|DB_THREAD)
+	rc = db_env_create(&env, 0);
+	rc = env->open(env, "./testdb", FLAGS, 0664);
+	rc = db_create(&dbi, env, 0);
+	rc = env->txn_begin(env, NULL, &txn, 0);
+	rc = dbi->open(dbi, txn, "test.bdb", NULL, DB_BTREE, DB_CREATE, 0664);
+
+	memset(&key, 0, sizeof(DBT));
+	memset(&data, 0, sizeof(DBT));
+	key.size = sizeof(int);
+	key.data = sval;
+	data.size = sizeof(sval);
+	data.data = sval;
+
+	sprintf(sval, "%03x %d foo bar", 32, 3141592);
+	rc = dbi->put(dbi, txn, &key, &data, 0);
+	rc = txn->commit(txn, 0);
+	if (rc) {
+		fprintf(stderr, "txn->commit: (%d) %s\n", rc, db_strerror(rc));
+		goto leave;
+	}
+	rc = env->txn_begin(env, NULL, &txn, 0);
+	rc = dbi->cursor(dbi, txn, &cursor, 0);
+	key.flags = DB_DBT_USERMEM;
+	key.data = kval;
+	key.ulen = sizeof(kval);
+	data.flags = DB_DBT_USERMEM;
+	data.data = sval;
+	data.ulen = sizeof(sval);
+	while ((rc = cursor->c_get(cursor, &key, &data, DB_NEXT)) == 0) {
+		printf("key: %p %.*s, data: %p %.*s\n",
+			key.data,  (int) key.size,  (char *) key.data,
+			data.data, (int) data.size, (char *) data.data);
+	}
+	rc = cursor->c_close(cursor);
+	rc = txn->abort(txn);
+leave:
+	rc = dbi->close(dbi, 0);
+	rc = env->close(env, 0);
+	return rc;
+}

external/db_drivers/liblmdb32/sample-mdb.txt

@@ -0,0 +1,62 @@
+/* sample-mdb.txt - MDB toy/sample
+ *
+ * Do a line-by-line comparison of this and sample-bdb.txt
+ */
+/*
+ * Copyright 2012 Howard Chu, Symas Corp.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted only as authorized by the OpenLDAP
+ * Public License.
+ *
+ * A copy of this license is available in the file LICENSE in the
+ * top-level directory of the distribution or, alternatively, at
+ * <http://www.OpenLDAP.org/license.html>.
+ */
+#include <stdio.h>
+#include "lmdb.h"
+
+int main(int argc,char * argv[])
+{
+	int rc;
+	MDB_env *env;
+	MDB_dbi dbi;
+	MDB_val key, data;
+	MDB_txn *txn;
+	MDB_cursor *cursor;
+	char sval[32];
+
+	/* Note: Most error checking omitted for simplicity */
+
+	rc = mdb_env_create(&env);
+	rc = mdb_env_open(env, "./testdb", 0, 0664);
+	rc = mdb_txn_begin(env, NULL, 0, &txn);
+	rc = mdb_open(txn, NULL, 0, &dbi);
+
+	key.mv_size = sizeof(int);
+	key.mv_data = sval;
+	data.mv_size = sizeof(sval);
+	data.mv_data = sval;
+
+	sprintf(sval, "%03x %d foo bar", 32, 3141592);
+	rc = mdb_put(txn, dbi, &key, &data, 0);
+	rc = mdb_txn_commit(txn);
+	if (rc) {
+		fprintf(stderr, "mdb_txn_commit: (%d) %s\n", rc, mdb_strerror(rc));
+		goto leave;
+	}
+	rc = mdb_txn_begin(env, NULL, MDB_RDONLY, &txn);
+	rc = mdb_cursor_open(txn, dbi, &cursor);
+	while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
+		printf("key: %p %.*s, data: %p %.*s\n",
+			key.mv_data,  (int) key.mv_size,  (char *) key.mv_data,
+			data.mv_data, (int) data.mv_size, (char *) data.mv_data);
+	}
+	mdb_cursor_close(cursor);
+	mdb_txn_abort(txn);
+leave:
+	mdb_close(env, dbi);
+	mdb_env_close(env);
+	return 0;
+}

external/db_drivers/liblmdb32/tooltag

@@ -0,0 +1,22 @@
+<tagfile>
+  <compound kind="page">
+    <name>mdb_copy_1</name>
+	<title>mdb_copy - environment copy tool</title>
+	<filename>mdb_copy.1</filename>
+  </compound>
+  <compound kind="page">
+    <name>mdb_dump_1</name>
+	<title>mdb_dump - environment export tool</title>
+	<filename>mdb_dump.1</filename>
+  </compound>
+  <compound kind="page">
+    <name>mdb_load_1</name>
+	<title>mdb_load - environment import tool</title>
+	<filename>mdb_load.1</filename>
+  </compound>
+  <compound kind="page">
+    <name>mdb_stat_1</name>
+	<title>mdb_stat - environment status tool</title>
+	<filename>mdb_stat.1</filename>
+  </compound>
+</tagfile>

external/db_drivers/liblmdb64/.gitignore

@@ -0,0 +1,23 @@
+mtest
+mtest[23456]
+testdb
+mdb_copy
+mdb_stat
+mdb_dump
+mdb_load
+*.lo
+*.[ao]
+*.so
+*.exe
+*[~#]
+*.bak
+*.orig
+*.rej
+*.gcov
+*.gcda
+*.gcno
+core
+core.*
+valgrind.*
+man/
+html/

external/db_drivers/liblmdb64/CHANGES

@@ -0,0 +1,112 @@
+LMDB 0.9 Change Log
+
+LMDB 0.9.14 Release (2014/09/20)
+	Fix to support 64K page size (ITS#7713)
+	Fix to persist decreased as well as increased mapsizes (ITS#7789)
+	Fix cursor bug when deleting last node of a DUPSORT key
+	Fix mdb_env_info to return FIXEDMAP address
+	Fix ambiguous error code from writing to closed DBI (ITS#7825)
+	Fix mdb_copy copying past end of file (ITS#7886)
+	Fix cursor bugs from page_merge/rebalance
+	Fix to dirty fewer pages in deletes (mdb_page_loose())
+	Fix mdb_dbi_open creating subDBs (ITS#7917)
+	Fix mdb_cursor_get(_DUP) with single value (ITS#7913)
+	Fix Windows compat issues in mtests (ITS#7879)
+	Add compacting variant of mdb_copy
+	Add BigEndian integer key compare code
+	Add mdb_dump/mdb_load utilities
+
+LMDB 0.9.13 Release (2014/06/18)
+	Fix mdb_page_alloc unlimited overflow page search
+	Documentation
+		Re-fix MDB_CURRENT doc (ITS#7793)
+		Fix MDB_GET_MULTIPLE/MDB_NEXT_MULTIPLE doc
+
+LMDB 0.9.12 Release (2014/06/13)
+	Fix MDB_GET_BOTH regression (ITS#7875,#7681)
+	Fix MDB_MULTIPLE writing multiple keys (ITS#7834)
+	Fix mdb_rebalance (ITS#7829)
+	Fix mdb_page_split (ITS#7815)
+	Fix md_entries count (ITS#7861,#7828,#7793)
+	Fix MDB_CURRENT (ITS#7793)
+	Fix possible crash on Windows DLL detach
+	Misc code cleanup
+	Documentation
+		mdb_cursor_put: cursor moves on error (ITS#7771)
+
+
+LMDB 0.9.11 Release (2014/01/15)
+	Add mdb_env_set_assert() (ITS#7775)
+	Fix: invalidate txn on page allocation errors (ITS#7377)
+	Fix xcursor tracking in mdb_cursor_del0() (ITS#7771)
+	Fix corruption from deletes (ITS#7756)
+	Fix Windows/MSVC build issues
+	Raise safe limit of max MDB_MAXKEYSIZE
+	Misc code cleanup
+	Documentation
+		Remove spurious note about non-overlapping flags (ITS#7665)
+
+LMDB 0.9.10 Release (2013/11/12)
+	Add MDB_NOMEMINIT option
+	Fix mdb_page_split() again (ITS#7589)
+	Fix MDB_NORDAHEAD definition (ITS#7734)
+	Fix mdb_cursor_del() positioning (ITS#7733)
+	Partial fix for larger page sizes (ITS#7713)
+	Fix Windows64/MSVC build issues
+
+LMDB 0.9.9 Release (2013/10/24)
+	Add mdb_env_get_fd()
+	Add MDB_NORDAHEAD option
+	Add MDB_NOLOCK option
+	Avoid wasting space in mdb_page_split() (ITS#7589)
+	Fix mdb_page_merge() cursor fixup (ITS#7722)
+	Fix mdb_cursor_del() on last delete (ITS#7718)
+	Fix adding WRITEMAP on existing env (ITS#7715)
+	Fix nested txns (ITS#7515)
+	Fix mdb_env_copy() O_DIRECT bug (ITS#7682)
+	Fix mdb_cursor_set(SET_RANGE) return code (ITS#7681)
+	Fix mdb_rebalance() cursor fixup (ITS#7701)
+	Misc code cleanup
+	Documentation
+		Note that by default, readers need write access
+
+
+LMDB 0.9.8 Release (2013/09/09)
+	Allow mdb_env_set_mapsize() on an open environment
+	Fix mdb_dbi_flags() (ITS#7672)
+	Fix mdb_page_unspill() in nested txns
+	Fix mdb_cursor_get(CURRENT|NEXT) after a delete
+	Fix mdb_cursor_get(DUP) to always return key (ITS#7671)
+	Fix mdb_cursor_del() to always advance to next item (ITS#7670)
+	Fix mdb_cursor_set(SET_RANGE) for tree with single page (ITS#7681)
+	Fix mdb_env_copy() retry open if O_DIRECT fails (ITS#7682)
+	Tweak mdb_page_spill() to be less aggressive
+	Documentation
+		Update caveats since mdb_reader_check() added in 0.9.7
+
+LMDB 0.9.7 Release (2013/08/17)
+	Don't leave stale lockfile on failed RDONLY open (ITS#7664)
+	Fix mdb_page_split() ref beyond cursor depth
+	Fix read txn data race (ITS#7635)
+	Fix mdb_rebalance (ITS#7536, #7538)
+	Fix mdb_drop() (ITS#7561)
+	Misc DEBUG macro fixes
+	Add MDB_NOTLS envflag
+	Add mdb_env_copyfd()
+	Add mdb_txn_env() (ITS#7660)
+	Add mdb_dbi_flags() (ITS#7661)
+	Add mdb_env_get_maxkeysize()
+	Add mdb_env_reader_list()/mdb_env_reader_check()
+	Add mdb_page_spill/unspill, remove hard txn size limit
+	Use shorter names for semaphores (ITS#7615)
+	Build
+		Fix install target (ITS#7656)
+	Documentation
+		Misc updates for cursors, DB handles, data lifetime
+
+LMDB 0.9.6 Release (2013/02/25)
+	Many fixes/enhancements
+
+LMDB 0.9.5 Release (2012/11/30)
+	Renamed from libmdb to liblmdb
+	Many fixes/enhancements

external/db_drivers/liblmdb64/CMakeLists.txt

@@ -0,0 +1,39 @@
+# Copyright (c) 2014-2015, The Monero Project
+# 
+# All rights reserved.
+# 
+# Redistribution and use in source and binary forms, with or without modification, are
+# permitted provided that the following conditions are met:
+# 
+# 1. Redistributions of source code must retain the above copyright notice, this list of
+#    conditions and the following disclaimer.
+# 
+# 2. Redistributions in binary form must reproduce the above copyright notice, this list
+#    of conditions and the following disclaimer in the documentation and/or other
+#    materials provided with the distribution.
+# 
+# 3. Neither the name of the copyright holder nor the names of its contributors may be
+#    used to endorse or promote products derived from this software without specific
+#    prior written permission.
+# 
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+# THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+# THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+set (lmdb_sources
+mdb.c
+midl.c)
+
+include_directories("${CMAKE_CURRENT_SOURCE_DIR}")
+
+add_library(lmdb
+  ${lmdb_sources})
+target_link_libraries(lmdb
+  LINK_PRIVATE
+    ${CMAKE_THREAD_LIBS_INIT})

external/db_drivers/liblmdb64/COPYRIGHT

@@ -0,0 +1,20 @@
+Copyright 2011-2014 Howard Chu, Symas Corp.
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted only as authorized by the OpenLDAP
+Public License.
+
+A copy of this license is available in the file LICENSE in the
+top-level directory of the distribution or, alternatively, at
+<http://www.OpenLDAP.org/license.html>.
+
+OpenLDAP is a registered trademark of the OpenLDAP Foundation.
+
+Individual files and/or contributed packages may be copyright by
+other parties and/or subject to additional restrictions.
+
+This work also contains materials derived from public sources.
+
+Additional information about OpenLDAP can be obtained at
+<http://www.openldap.org/>.

external/db_drivers/liblmdb64/LICENSE

@@ -0,0 +1,47 @@
+The OpenLDAP Public License
+  Version 2.8, 17 August 2003
+
+Redistribution and use of this software and associated documentation
+("Software"), with or without modification, are permitted provided
+that the following conditions are met:
+
+1. Redistributions in source form must retain copyright statements
+   and notices,
+
+2. Redistributions in binary form must reproduce applicable copyright
+   statements and notices, this list of conditions, and the following
+   disclaimer in the documentation and/or other materials provided
+   with the distribution, and
+
+3. Redistributions must contain a verbatim copy of this document.
+
+The OpenLDAP Foundation may revise this license from time to time.
+Each revision is distinguished by a version number.  You may use
+this Software under terms of this license revision or under the
+terms of any subsequent revision of the license.
+
+THIS SOFTWARE IS PROVIDED BY THE OPENLDAP FOUNDATION AND ITS
+CONTRIBUTORS ``AS IS'' AND ANY EXPRESSED OR IMPLIED WARRANTIES,
+INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
+AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.  IN NO EVENT
+SHALL THE OPENLDAP FOUNDATION, ITS CONTRIBUTORS, OR THE AUTHOR(S)
+OR OWNER(S) OF THE SOFTWARE BE LIABLE FOR ANY DIRECT, INDIRECT,
+INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
+ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGE.
+
+The names of the authors and copyright holders must not be used in
+advertising or otherwise to promote the sale, use or other dealing
+in this Software without specific, written prior permission.  Title
+to copyright in this Software shall at all times remain with copyright
+holders.
+
+OpenLDAP is a registered trademark of the OpenLDAP Foundation.
+
+Copyright 1999-2003 The OpenLDAP Foundation, Redwood City,
+California, USA.  All Rights Reserved.  Permission to copy and
+distribute verbatim copies of this document is granted.

external/db_drivers/liblmdb64/Makefile

@@ -0,0 +1,104 @@
+# Makefile for liblmdb (Lightning memory-mapped database library).
+
+########################################################################
+# Configuration. The compiler options must enable threaded compilation.
+#
+# Preprocessor macros (for CPPFLAGS) of interest...
+# Note that the defaults should already be correct for most
+# platforms; you should not need to change any of these.
+# Read their descriptions in mdb.c if you do:
+#
+# - MDB_USE_POSIX_SEM
+# - MDB_DSYNC
+# - MDB_FDATASYNC
+# - MDB_FDATASYNC_WORKS
+# - MDB_USE_PWRITEV
+#
+# There may be other macros in mdb.c of interest. You should
+# read mdb.c before changing any of them.
+#
+CC	= gcc
+W	= -W -Wall -Wno-unused-parameter -Wbad-function-cast -Wuninitialized
+THREADS = -pthread
+OPT = -O2 -g
+CFLAGS	= $(THREADS) $(OPT) $(W) $(XCFLAGS)
+LDLIBS	=
+SOLIBS	=
+prefix	= /usr/local
+
+########################################################################
+
+IHDRS	= lmdb.h
+ILIBS	= liblmdb.a liblmdb.so
+IPROGS	= mdb_stat mdb_copy mdb_dump mdb_load
+IDOCS	= mdb_stat.1 mdb_copy.1 mdb_dump.1 mdb_load.1
+PROGS	= $(IPROGS) mtest mtest2 mtest3 mtest4 mtest5
+all:	$(ILIBS) $(PROGS)
+
+install: $(ILIBS) $(IPROGS) $(IHDRS)
+	for f in $(IPROGS); do cp $$f $(DESTDIR)$(prefix)/bin; done
+	for f in $(ILIBS); do cp $$f $(DESTDIR)$(prefix)/lib; done
+	for f in $(IHDRS); do cp $$f $(DESTDIR)$(prefix)/include; done
+	for f in $(IDOCS); do cp $$f $(DESTDIR)$(prefix)/man/man1; done
+
+clean:
+	rm -rf $(PROGS) *.[ao] *.[ls]o *~ testdb
+
+test:	all
+	rm -rf testdb && mkdir testdb
+	./mtest && ./mdb_stat testdb
+
+liblmdb.a:	mdb.o midl.o
+	ar rs $@ mdb.o midl.o
+
+liblmdb.so:	mdb.lo midl.lo
+#	$(CC) $(LDFLAGS) -pthread -shared -Wl,-Bsymbolic -o $@ mdb.o midl.o $(SOLIBS)
+	$(CC) $(LDFLAGS) -pthread -shared -o $@ mdb.lo midl.lo $(SOLIBS)
+
+mdb_stat: mdb_stat.o liblmdb.a
+mdb_copy: mdb_copy.o liblmdb.a
+mdb_dump: mdb_dump.o liblmdb.a
+mdb_load: mdb_load.o liblmdb.a
+mtest:    mtest.o    liblmdb.a
+mtest2:	mtest2.o liblmdb.a
+mtest3:	mtest3.o liblmdb.a
+mtest4:	mtest4.o liblmdb.a
+mtest5:	mtest5.o liblmdb.a
+mtest6:	mtest6.o liblmdb.a
+
+mdb.o: mdb.c lmdb.h midl.h
+	$(CC) $(CFLAGS) $(CPPFLAGS) -c mdb.c
+
+midl.o: midl.c midl.h
+	$(CC) $(CFLAGS) $(CPPFLAGS) -c midl.c
+
+mdb.lo: mdb.c lmdb.h midl.h
+	$(CC) $(CFLAGS) -fPIC $(CPPFLAGS) -c mdb.c -o $@
+
+midl.lo: midl.c midl.h
+	$(CC) $(CFLAGS) -fPIC $(CPPFLAGS) -c midl.c -o $@
+
+%:	%.o
+	$(CC) $(CFLAGS) $(LDFLAGS) $^ $(LDLIBS) -o $@
+
+%.o:	%.c lmdb.h
+	$(CC) $(CFLAGS) $(CPPFLAGS) -c $<
+
+COV_FLAGS=-fprofile-arcs -ftest-coverage
+COV_OBJS=xmdb.o xmidl.o
+
+coverage: xmtest
+	for i in mtest*.c [0-9]*.c; do j=`basename \$$i .c`; $(MAKE) $$j.o; \
+		gcc -o x$$j $$j.o $(COV_OBJS) -pthread $(COV_FLAGS); \
+		rm -rf testdb; mkdir testdb; ./x$$j; done
+	gcov xmdb.c
+	gcov xmidl.c
+
+xmtest:	mtest.o xmdb.o xmidl.o
+	gcc -o xmtest mtest.o xmdb.o xmidl.o -pthread $(COV_FLAGS)
+
+xmdb.o: mdb.c lmdb.h midl.h
+	$(CC) $(CFLAGS) -fPIC $(CPPFLAGS) -O0 $(COV_FLAGS) -c mdb.c -o $@
+
+xmidl.o: midl.c midl.h
+	$(CC) $(CFLAGS) -fPIC $(CPPFLAGS) -O0 $(COV_FLAGS) -c midl.c -o $@

external/db_drivers/liblmdb64/mdb_copy.1

@@ -0,0 +1,54 @@
+.TH MDB_COPY 1 "2014/06/20" "LMDB 0.9.14"
+.\" Copyright 2012-2014 Howard Chu, Symas Corp. All Rights Reserved.
+.\" Copying restrictions apply.  See COPYRIGHT/LICENSE.
+.SH NAME
+mdb_copy \- LMDB environment copy tool
+.SH SYNOPSIS
+.B mdb_copy
+[\c
+.BR \-V ]
+[\c
+.BR \-c ]
+[\c
+.BR \-n ]
+.B srcpath
+[\c
+.BR dstpath ]
+.SH DESCRIPTION
+The
+.B mdb_copy
+utility copies an LMDB environment. The environment can
+be copied regardless of whether it is currently in use.
+No lockfile is created, since it gets recreated at need.
+
+If
+.I dstpath
+is specified it must be the path of an empty directory
+for storing the backup. Otherwise, the backup will be
+written to stdout.
+
+.SH OPTIONS
+.TP
+.BR \-V
+Write the library version number to the standard output, and exit.
+.TP
+.BR \-c
+Compact while copying. Only current data pages will be copied; freed
+or unused pages will be omitted from the copy. This option will
+slow down the backup process as it is more CPU-intensive.
+.TP
+.BR \-n
+Open LDMB environment(s) which do not use subdirectories.
+
+.SH DIAGNOSTICS
+Exit status is zero if no errors occur.
+Errors result in a non-zero exit status and
+a diagnostic message being written to standard error.
+.SH CAVEATS
+This utility can trigger significant file size growth if run
+in parallel with write transactions, because pages which they
+free during copying cannot be reused until the copy is done.
+.SH "SEE ALSO"
+.BR mdb_stat (1)
+.SH AUTHOR
+Howard Chu of Symas Corporation <http://www.symas.com>

external/db_drivers/liblmdb64/mdb_copy.c

@@ -0,0 +1,82 @@
+/* mdb_copy.c - memory-mapped database backup tool */
+/*
+ * Copyright 2012 Howard Chu, Symas Corp.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted only as authorized by the OpenLDAP
+ * Public License.
+ *
+ * A copy of this license is available in the file LICENSE in the
+ * top-level directory of the distribution or, alternatively, at
+ * <http://www.OpenLDAP.org/license.html>.
+ */
+#ifdef _WIN32
+#include <windows.h>
+#define	MDB_STDOUT	GetStdHandle(STD_OUTPUT_HANDLE)
+#else
+#define	MDB_STDOUT	1
+#endif
+#include <stdio.h>
+#include <stdlib.h>
+#include <signal.h>
+#include "lmdb.h"
+
+static void
+sighandle(int sig)
+{
+}
+
+int main(int argc,char * argv[])
+{
+	int rc;
+	MDB_env *env;
+	const char *progname = argv[0], *act;
+	unsigned flags = MDB_RDONLY;
+	unsigned cpflags = 0;
+
+	for (; argc > 1 && argv[1][0] == '-'; argc--, argv++) {
+		if (argv[1][1] == 'n' && argv[1][2] == '\0')
+			flags |= MDB_NOSUBDIR;
+		else if (argv[1][1] == 'c' && argv[1][2] == '\0')
+			cpflags |= MDB_CP_COMPACT;
+		else if (argv[1][1] == 'V' && argv[1][2] == '\0') {
+			printf("%s\n", MDB_VERSION_STRING);
+			exit(0);
+		} else
+			argc = 0;
+	}
+
+	if (argc<2 || argc>3) {
+		fprintf(stderr, "usage: %s [-V] [-c] [-n] srcpath [dstpath]\n", progname);
+		exit(EXIT_FAILURE);
+	}
+
+#ifdef SIGPIPE
+	signal(SIGPIPE, sighandle);
+#endif
+#ifdef SIGHUP
+	signal(SIGHUP, sighandle);
+#endif
+	signal(SIGINT, sighandle);
+	signal(SIGTERM, sighandle);
+
+	act = "opening environment";
+	rc = mdb_env_create(&env);
+	if (rc == MDB_SUCCESS) {
+		rc = mdb_env_open(env, argv[1], flags, 0600);
+	}
+	if (rc == MDB_SUCCESS) {
+		act = "copying";
+		if (argc == 2)
+			rc = mdb_env_copyfd2(env, MDB_STDOUT, cpflags);
+		else
+			rc = mdb_env_copy2(env, argv[2], cpflags);
+	}
+	if (rc)
+		fprintf(stderr, "%s: %s failed, error %d (%s)\n",
+			progname, act, rc, mdb_strerror(rc));
+	mdb_env_close(env);
+
+	return rc ? EXIT_FAILURE : EXIT_SUCCESS;
+}

external/db_drivers/liblmdb64/mdb_dump.1

@@ -0,0 +1,75 @@
+.TH MDB_DUMP 1 "2014/06/20" "LMDB 0.9.14"
+.\" Copyright 2014 Howard Chu, Symas Corp. All Rights Reserved.
+.\" Copying restrictions apply.  See COPYRIGHT/LICENSE.
+.SH NAME
+mdb_dump \- LMDB environment export tool
+.SH SYNOPSIS
+.B mdb_dump
+.BR \ envpath
+[\c
+.BR \-V ]
+[\c
+.BI \-f \ file\fR]
+[\c
+.BR \-l ]
+[\c
+.BR \-n ]
+[\c
+.BR \-p ]
+[\c
+.BR \-a \ |
+.BI \-s \ subdb\fR]
+.SH DESCRIPTION
+The
+.B mdb_dump
+utility reads a database and writes its contents to the
+standard output using a portable flat-text format
+understood by the
+.BR mdb_load (1)
+utility.
+.SH OPTIONS
+.TP
+.BR \-V
+Write the library version number to the standard output, and exit.
+.TP
+.BR \-f \ file
+Write to the specified file instead of to the standard output.
+.TP
+.BR \-l
+List the databases stored in the environment. Just the
+names will be listed, no data will be output.
+.TP
+.BR \-n
+Dump an LMDB database which does not use subdirectories.
+.TP
+.BR \-p
+If characters in either the key or data items are printing characters (as
+defined by isprint(3)), output them directly. This option permits users to
+use standard text editors and tools to modify the contents of databases.
+
+Note: different systems may have different notions about what characters
+are considered printing characters, and databases dumped in this manner may
+be less portable to external systems. 
+.TP
+.BR \-a
+Dump all of the subdatabases in the environment.
+.TP
+.BR \-s \ subdb
+Dump a specific subdatabase. If no database is specified, only the main database is dumped.
+.SH DIAGNOSTICS
+Exit status is zero if no errors occur.
+Errors result in a non-zero exit status and
+a diagnostic message being written to standard error.
+
+Dumping and reloading databases that use user-defined comparison functions
+will result in new databases that use the default comparison functions.
+\fBIn this case it is quite likely that the reloaded database will be
+damaged beyond repair permitting neither record storage nor retrieval.\fP
+
+The only available workaround is to modify the source for the
+.BR mdb_load (1)
+utility to load the database using the correct comparison functions.
+.SH "SEE ALSO"
+.BR mdb_load (1)
+.SH AUTHOR
+Howard Chu of Symas Corporation <http://www.symas.com>

external/db_drivers/liblmdb64/mdb_dump.c

@@ -0,0 +1,317 @@
+/* mdb_dump.c - memory-mapped database dump tool */
+/*
+ * Copyright 2011-2014 Howard Chu, Symas Corp.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted only as authorized by the OpenLDAP
+ * Public License.
+ *
+ * A copy of this license is available in the file LICENSE in the
+ * top-level directory of the distribution or, alternatively, at
+ * <http://www.OpenLDAP.org/license.html>.
+ */
+#include <stdio.h>
+#include <errno.h>
+#include <stdlib.h>
+#include <string.h>
+#include <ctype.h>
+#include <unistd.h>
+#include <signal.h>
+#include "lmdb.h"
+
+#ifdef _WIN32
+#define Z	"I"
+#else
+#define Z	"z"
+#endif
+
+#define PRINT	1
+static int mode;
+
+typedef struct flagbit {
+	int bit;
+	char *name;
+} flagbit;
+
+flagbit dbflags[] = {
+	{ MDB_REVERSEKEY, "reversekey" },
+	{ MDB_DUPSORT, "dupsort" },
+	{ MDB_INTEGERKEY, "integerkey" },
+	{ MDB_DUPFIXED, "dupfixed" },
+	{ MDB_INTEGERDUP, "integerdup" },
+	{ MDB_REVERSEDUP, "reversedup" },
+	{ 0, NULL }
+};
+
+static volatile sig_atomic_t gotsig;
+
+static void dumpsig( int sig )
+{
+	gotsig=1;
+}
+
+static const char hexc[] = "0123456789abcdef";
+
+static void hex(unsigned char c)
+{
+	putchar(hexc[c >> 4]);
+	putchar(hexc[c & 0xf]);
+}
+
+static void text(MDB_val *v)
+{
+	unsigned char *c, *end;
+
+	putchar(' ');
+	c = v->mv_data;
+	end = c + v->mv_size;
+	while (c < end) {
+		if (isprint(*c)) {
+			putchar(*c);
+		} else {
+			putchar('\\');
+			hex(*c);
+		}
+		c++;
+	}
+	putchar('\n');
+}
+
+static void byte(MDB_val *v)
+{
+	unsigned char *c, *end;
+
+	putchar(' ');
+	c = v->mv_data;
+	end = c + v->mv_size;
+	while (c < end) {
+		hex(*c++);
+	}
+	putchar('\n');
+}
+
+/* Dump in BDB-compatible format */
+static int dumpit(MDB_txn *txn, MDB_dbi dbi, char *name)
+{
+	MDB_cursor *mc;
+	MDB_stat ms;
+	MDB_val key, data;
+	MDB_envinfo info;
+	unsigned int flags;
+	int rc, i;
+
+	rc = mdb_dbi_flags(txn, dbi, &flags);
+	if (rc) return rc;
+
+	rc = mdb_stat(txn, dbi, &ms);
+	if (rc) return rc;
+
+	rc = mdb_env_info(mdb_txn_env(txn), &info);
+	if (rc) return rc;
+
+	printf("VERSION=3\n");
+	printf("format=%s\n", mode & PRINT ? "print" : "bytevalue");
+	if (name)
+		printf("database=%s\n", name);
+	printf("type=btree\n");
+	printf("mapsize=%" Z "u\n", info.me_mapsize);
+	if (info.me_mapaddr)
+		printf("mapaddr=%p\n", info.me_mapaddr);
+	printf("maxreaders=%u\n", info.me_maxreaders);
+
+	if (flags & MDB_DUPSORT)
+		printf("duplicates=1\n");
+
+	for (i=0; dbflags[i].bit; i++)
+		if (flags & dbflags[i].bit)
+			printf("%s=1\n", dbflags[i].name);
+
+	printf("db_pagesize=%d\n", ms.ms_psize);
+	printf("HEADER=END\n");
+
+	rc = mdb_cursor_open(txn, dbi, &mc);
+	if (rc) return rc;
+
+	while ((rc = mdb_cursor_get(mc, &key, &data, MDB_NEXT) == MDB_SUCCESS)) {
+		if (gotsig) {
+			rc = EINTR;
+			break;
+		}
+		if (mode & PRINT) {
+			text(&key);
+			text(&data);
+		} else {
+			byte(&key);
+			byte(&data);
+		}
+	}
+	printf("DATA=END\n");
+	if (rc == MDB_NOTFOUND)
+		rc = MDB_SUCCESS;
+
+	return rc;
+}
+
+static void usage(char *prog)
+{
+	fprintf(stderr, "usage: %s dbpath [-V] [-f output] [-l] [-n] [-p] [-a|-s subdb]\n", prog);
+	exit(EXIT_FAILURE);
+}
+
+int main(int argc, char *argv[])
+{
+	int i, rc;
+	MDB_env *env;
+	MDB_txn *txn;
+	MDB_dbi dbi;
+	char *prog = argv[0];
+	char *envname;
+	char *subname = NULL;
+	int alldbs = 0, envflags = 0, list = 0;
+
+	if (argc < 2) {
+		usage(prog);
+	}
+
+	/* -a: dump main DB and all subDBs
+	 * -s: dump only the named subDB
+	 * -n: use NOSUBDIR flag on env_open
+	 * -p: use printable characters
+	 * -f: write to file instead of stdout
+	 * -V: print version and exit
+	 * (default) dump only the main DB
+	 */
+	while ((i = getopt(argc, argv, "af:lnps:V")) != EOF) {
+		switch(i) {
+		case 'V':
+			printf("%s\n", MDB_VERSION_STRING);
+			exit(0);
+			break;
+		case 'l':
+			list = 1;
+			/*FALLTHROUGH*/;
+		case 'a':
+			if (subname)
+				usage(prog);
+			alldbs++;
+			break;
+		case 'f':
+			if (freopen(optarg, "w", stdout) == NULL) {
+				fprintf(stderr, "%s: %s: reopen: %s\n",
+					prog, optarg, strerror(errno));
+				exit(EXIT_FAILURE);
+			}
+			break;
+		case 'n':
+			envflags |= MDB_NOSUBDIR;
+			break;
+		case 'p':
+			mode |= PRINT;
+			break;
+		case 's':
+			if (alldbs)
+				usage(prog);
+			subname = optarg;
+			break;
+		default:
+			usage(prog);
+		}
+	}
+
+	if (optind != argc - 1)
+		usage(prog);
+
+#ifdef SIGPIPE
+	signal(SIGPIPE, dumpsig);
+#endif
+#ifdef SIGHUP
+	signal(SIGHUP, dumpsig);
+#endif
+	signal(SIGINT, dumpsig);
+	signal(SIGTERM, dumpsig);
+
+	envname = argv[optind];
+	rc = mdb_env_create(&env);
+	if (rc) {
+		fprintf(stderr, "mdb_env_create failed, error %d %s\n", rc, mdb_strerror(rc));
+		return EXIT_FAILURE;
+	}
+
+	if (alldbs || subname) {
+		mdb_env_set_maxdbs(env, 2);
+	}
+
+	rc = mdb_env_open(env, envname, envflags | MDB_RDONLY, 0664);
+	if (rc) {
+		fprintf(stderr, "mdb_env_open failed, error %d %s\n", rc, mdb_strerror(rc));
+		goto env_close;
+	}
+
+	rc = mdb_txn_begin(env, NULL, MDB_RDONLY, &txn);
+	if (rc) {
+		fprintf(stderr, "mdb_txn_begin failed, error %d %s\n", rc, mdb_strerror(rc));
+		goto env_close;
+	}
+
+	rc = mdb_open(txn, subname, 0, &dbi);
+	if (rc) {
+		fprintf(stderr, "mdb_open failed, error %d %s\n", rc, mdb_strerror(rc));
+		goto txn_abort;
+	}
+
+	if (alldbs) {
+		MDB_cursor *cursor;
+		MDB_val key;
+		int count = 0;
+
+		rc = mdb_cursor_open(txn, dbi, &cursor);
+		if (rc) {
+			fprintf(stderr, "mdb_cursor_open failed, error %d %s\n", rc, mdb_strerror(rc));
+			goto txn_abort;
+		}
+		while ((rc = mdb_cursor_get(cursor, &key, NULL, MDB_NEXT_NODUP)) == 0) {
+			char *str;
+			MDB_dbi db2;
+			if (memchr(key.mv_data, '\0', key.mv_size))
+				continue;
+			count++;
+			str = malloc(key.mv_size+1);
+			memcpy(str, key.mv_data, key.mv_size);
+			str[key.mv_size] = '\0';
+			rc = mdb_open(txn, str, 0, &db2);
+			if (rc == MDB_SUCCESS) {
+				if (list) {
+					printf("%s\n", str);
+					list++;
+				} else {
+					rc = dumpit(txn, db2, str);
+					if (rc)
+						break;
+				}
+				mdb_close(env, db2);
+			}
+			free(str);
+			if (rc) continue;
+		}
+		mdb_cursor_close(cursor);
+		if (!count) {
+			fprintf(stderr, "%s: %s does not contain multiple databases\n", prog, envname);
+			rc = MDB_NOTFOUND;
+		} else if (rc == MDB_NOTFOUND) {
+			rc = MDB_SUCCESS;
+		}
+	} else {
+		rc = dumpit(txn, dbi, subname);
+	}
+	if (rc && rc != MDB_NOTFOUND)
+		fprintf(stderr, "%s: %s: %s\n", prog, envname, mdb_strerror(rc));
+
+	mdb_close(env, dbi);
+txn_abort:
+	mdb_txn_abort(txn);
+env_close:
+	mdb_env_close(env);
+
+	return rc ? EXIT_FAILURE : EXIT_SUCCESS;
+}

external/db_drivers/liblmdb64/mdb_load.1

@@ -0,0 +1,77 @@
+.TH MDB_LOAD 1 "2014/06/20" "LMDB 0.9.14"
+.\" Copyright 2014 Howard Chu, Symas Corp. All Rights Reserved.
+.\" Copying restrictions apply.  See COPYRIGHT/LICENSE.
+.SH NAME
+mdb_load \- LMDB environment import tool
+.SH SYNOPSIS
+.B mdb_load
+.BR \ envpath
+[\c
+.BR \-V ]
+[\c
+.BI \-f \ file\fR]
+[\c
+.BR \-n ]
+[\c
+.BI \-s \ subdb\fR]
+[\c
+.BR \-N ]
+[\c
+.BR \-T ]
+.SH DESCRIPTION
+The
+.B mdb_load
+utility reads from the standard input and loads it into the
+LMDB environment
+.BR envpath .
+
+The input to
+.B mdb_load
+must be in the output format specified by the
+.BR mdb_dump (1)
+utility or as specified by the
+.B -T
+option below.
+.SH OPTIONS
+.TP
+.BR \-V
+Write the library version number to the standard output, and exit.
+.TP
+.BR \-f \ file
+Read from the specified file instead of from the standard input.
+.TP
+.BR \-n
+Load an LMDB database which does not use subdirectories.
+.TP
+.BR \-s \ subdb
+Load a specific subdatabase. If no database is specified, data is loaded into the main database.
+.TP
+.BR \-N
+Don't overwrite existing records when loading into an already existing database; just skip them.
+.TP
+.BR \-T
+Load data from simple text files. The input must be paired lines of text, where the first
+line of the pair is the key item, and the second line of the pair is its corresponding
+data item.
+
+A simple escape mechanism, where newline and backslash (\\) characters are special, is
+applied to the text input. Newline characters are interpreted as record separators.
+Backslash characters in the text will be interpreted in one of two ways: If the backslash
+character precedes another backslash character, the pair will be interpreted as a literal
+backslash. If the backslash character precedes any other character, the two characters
+following the backslash will be interpreted as a hexadecimal specification of a single
+character; for example, \\0a is a newline character in the ASCII character set.
+
+For this reason, any backslash or newline characters that naturally occur in the text
+input must be escaped to avoid misinterpretation by
+.BR mdb_load . 
+
+.SH DIAGNOSTICS
+Exit status is zero if no errors occur.
+Errors result in a non-zero exit status and
+a diagnostic message being written to standard error.
+
+.SH "SEE ALSO"
+.BR mdb_dump (1)
+.SH AUTHOR
+Howard Chu of Symas Corporation <http://www.symas.com>

external/db_drivers/liblmdb64/mdb_load.c

@@ -0,0 +1,452 @@
+/* mdb_load.c - memory-mapped database load tool */
+/*
+ * Copyright 2011-2014 Howard Chu, Symas Corp.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted only as authorized by the OpenLDAP
+ * Public License.
+ *
+ * A copy of this license is available in the file LICENSE in the
+ * top-level directory of the distribution or, alternatively, at
+ * <http://www.OpenLDAP.org/license.html>.
+ */
+#include <stdio.h>
+#include <stdlib.h>
+#include <errno.h>
+#include <string.h>
+#include <ctype.h>
+#include <unistd.h>
+#include "lmdb.h"
+
+#define PRINT	1
+#define NOHDR	2
+static int mode;
+
+static char *subname = NULL;
+
+static size_t lineno;
+static int version;
+
+static int flags;
+
+static char *prog;
+
+static int Eof;
+
+static MDB_envinfo info;
+
+static MDB_val kbuf, dbuf;
+
+#ifdef _WIN32
+#define Z	"I"
+#else
+#define Z	"z"
+#endif
+
+#define STRLENOF(s)	(sizeof(s)-1)
+
+typedef struct flagbit {
+	int bit;
+	char *name;
+	int len;
+} flagbit;
+
+#define S(s)	s, STRLENOF(s)
+
+flagbit dbflags[] = {
+	{ MDB_REVERSEKEY, S("reversekey") },
+	{ MDB_DUPSORT, S("dupsort") },
+	{ MDB_INTEGERKEY, S("integerkey") },
+	{ MDB_DUPFIXED, S("dupfixed") },
+	{ MDB_INTEGERDUP, S("integerdup") },
+	{ MDB_REVERSEDUP, S("reversedup") },
+	{ 0, NULL, 0 }
+};
+
+static void readhdr(void)
+{
+	char *ptr;
+
+	while (fgets(dbuf.mv_data, dbuf.mv_size, stdin) != NULL) {
+		lineno++;
+		if (!strncmp(dbuf.mv_data, "VERSION=", STRLENOF("VERSION="))) {
+			version=atoi((char *)dbuf.mv_data+STRLENOF("VERSION="));
+			if (version > 3) {
+				fprintf(stderr, "%s: line %" Z "d: unsupported VERSION %d\n",
+					prog, lineno, version);
+				exit(EXIT_FAILURE);
+			}
+		} else if (!strncmp(dbuf.mv_data, "HEADER=END", STRLENOF("HEADER=END"))) {
+			break;
+		} else if (!strncmp(dbuf.mv_data, "format=", STRLENOF("format="))) {
+			if (!strncmp((char *)dbuf.mv_data+STRLENOF("FORMAT="), "print", STRLENOF("print")))
+				mode |= PRINT;
+			else if (strncmp((char *)dbuf.mv_data+STRLENOF("FORMAT="), "bytevalue", STRLENOF("bytevalue"))) {
+				fprintf(stderr, "%s: line %" Z "d: unsupported FORMAT %s\n",
+					prog, lineno, (char *)dbuf.mv_data+STRLENOF("FORMAT="));
+				exit(EXIT_FAILURE);
+			}
+		} else if (!strncmp(dbuf.mv_data, "database=", STRLENOF("database="))) {
+			ptr = memchr(dbuf.mv_data, '\n', dbuf.mv_size);
+			if (ptr) *ptr = '\0';
+			if (subname) free(subname);
+			subname = strdup((char *)dbuf.mv_data+STRLENOF("database="));
+		} else if (!strncmp(dbuf.mv_data, "type=", STRLENOF("type="))) {
+			if (strncmp((char *)dbuf.mv_data+STRLENOF("type="), "btree", STRLENOF("btree")))  {
+				fprintf(stderr, "%s: line %" Z "d: unsupported type %s\n",
+					prog, lineno, (char *)dbuf.mv_data+STRLENOF("type="));
+				exit(EXIT_FAILURE);
+			}
+		} else if (!strncmp(dbuf.mv_data, "mapaddr=", STRLENOF("mapaddr="))) {
+			int i;
+			ptr = memchr(dbuf.mv_data, '\n', dbuf.mv_size);
+			if (ptr) *ptr = '\0';
+			i = sscanf((char *)dbuf.mv_data+STRLENOF("mapaddr="), "%p", &info.me_mapaddr);
+			if (i != 1) {
+				fprintf(stderr, "%s: line %" Z "d: invalid mapaddr %s\n",
+					prog, lineno, (char *)dbuf.mv_data+STRLENOF("mapaddr="));
+				exit(EXIT_FAILURE);
+			}
+		} else if (!strncmp(dbuf.mv_data, "mapsize=", STRLENOF("mapsize="))) {
+			int i;
+			ptr = memchr(dbuf.mv_data, '\n', dbuf.mv_size);
+			if (ptr) *ptr = '\0';
+			i = sscanf((char *)dbuf.mv_data+STRLENOF("mapsize="), "%" Z "u", &info.me_mapsize);
+			if (i != 1) {
+				fprintf(stderr, "%s: line %" Z "d: invalid mapsize %s\n",
+					prog, lineno, (char *)dbuf.mv_data+STRLENOF("mapsize="));
+				exit(EXIT_FAILURE);
+			}
+		} else if (!strncmp(dbuf.mv_data, "maxreaders=", STRLENOF("maxreaders="))) {
+			int i;
+			ptr = memchr(dbuf.mv_data, '\n', dbuf.mv_size);
+			if (ptr) *ptr = '\0';
+			i = sscanf((char *)dbuf.mv_data+STRLENOF("maxreaders="), "%u", &info.me_maxreaders);
+			if (i != 1) {
+				fprintf(stderr, "%s: line %" Z "d: invalid maxreaders %s\n",
+					prog, lineno, (char *)dbuf.mv_data+STRLENOF("maxreaders="));
+				exit(EXIT_FAILURE);
+			}
+		} else {
+			int i;
+			for (i=0; dbflags[i].bit; i++) {
+				if (!strncmp(dbuf.mv_data, dbflags[i].name, dbflags[i].len) &&
+					((char *)dbuf.mv_data)[dbflags[i].len] == '=') {
+					flags |= dbflags[i].bit;
+					break;
+				}
+			}
+			if (!dbflags[i].bit) {
+				ptr = memchr(dbuf.mv_data, '=', dbuf.mv_size);
+				if (!ptr) {
+					fprintf(stderr, "%s: line %" Z "d: unexpected format\n",
+						prog, lineno);
+					exit(EXIT_FAILURE);
+				} else {
+					*ptr = '\0';
+					fprintf(stderr, "%s: line %" Z "d: unrecognized keyword ignored: %s\n",
+						prog, lineno, (char *)dbuf.mv_data);
+				}
+			}
+		}
+	}
+}
+
+static void badend(void)
+{
+	fprintf(stderr, "%s: line %" Z "d: unexpected end of input\n",
+		prog, lineno);
+}
+
+static int unhex(unsigned char *c2)
+{
+	int x, c;
+	x = *c2++ & 0x4f;
+	if (x & 0x40)
+		x -= 55;
+	c = x << 4;
+	x = *c2 & 0x4f;
+	if (x & 0x40)
+		x -= 55;
+	c |= x;
+	return c;
+}
+
+static int readline(MDB_val *out, MDB_val *buf)
+{
+	unsigned char *c1, *c2, *end;
+	size_t len;
+	int c;
+
+	if (!(mode & NOHDR)) {
+		c = fgetc(stdin);
+		if (c == EOF) {
+			Eof = 1;
+			return EOF;
+		}
+		if (c != ' ') {
+			lineno++;
+			if (fgets(buf->mv_data, buf->mv_size, stdin) == NULL) {
+badend:
+				Eof = 1;
+				badend();
+				return EOF;
+			}
+			if (c == 'D' && !strncmp(buf->mv_data, "ATA=END", STRLENOF("ATA=END")))
+				return EOF;
+			goto badend;
+		}
+	}
+	if (fgets(buf->mv_data, buf->mv_size, stdin) == NULL) {
+		Eof = 1;
+		return EOF;
+	}
+	lineno++;
+
+	c1 = buf->mv_data;
+	len = strlen((char *)c1);
+
+	/* Is buffer too short? */
+	while (c1[len-1] != '\n') {
+		buf->mv_data = realloc(buf->mv_data, buf->mv_size*2);
+		if (!buf->mv_data) {
+			Eof = 1;
+			fprintf(stderr, "%s: line %" Z "d: out of memory, line too long\n",
+				prog, lineno);
+			return EOF;
+		}
+		c1 = buf->mv_data;
+		c1 += buf->mv_size;
+		if (fgets((char *)c1, buf->mv_size, stdin) == NULL) {
+			Eof = 1;
+			badend();
+			return EOF;
+		}
+		buf->mv_size *= 2;
+		len = strlen((char *)c1);
+	}
+	c1 = c2 = buf->mv_data;
+	len = strlen((char *)c1);
+	c1[--len] = '\0';
+	end = c1 + len;
+
+	if (mode & PRINT) {
+		while (c2 < end) {
+			if (*c2 == '\\') {
+				if (c2[1] == '\\') {
+					c1++; c2 += 2;
+				} else {
+					if (c2+3 > end || !isxdigit(c2[1]) || !isxdigit(c2[2])) {
+						Eof = 1;
+						badend();
+						return EOF;
+					}
+					*c1++ = unhex(++c2);
+					c2 += 2;
+				}
+			} else {
+				c1++; c2++;
+			}
+		}
+	} else {
+		/* odd length not allowed */
+		if (len & 1) {
+			Eof = 1;
+			badend();
+			return EOF;
+		}
+		while (c2 < end) {
+			if (!isxdigit(*c2) || !isxdigit(c2[1])) {
+				Eof = 1;
+				badend();
+				return EOF;
+			}
+			*c1++ = unhex(c2);
+			c2 += 2;
+		}
+	}
+	c2 = out->mv_data = buf->mv_data;
+	out->mv_size = c1 - c2;
+
+	return 0;
+}
+
+static void usage(void)
+{
+	fprintf(stderr, "usage: %s dbpath [-V] [-f input] [-n] [-s name] [-N] [-T]\n", prog);
+	exit(EXIT_FAILURE);
+}
+
+int main(int argc, char *argv[])
+{
+	int i, rc;
+	MDB_env *env;
+	MDB_txn *txn;
+	MDB_cursor *mc;
+	MDB_dbi dbi;
+	char *envname;
+	int envflags = 0, putflags = 0;
+	int dohdr = 0;
+
+	prog = argv[0];
+
+	if (argc < 2) {
+		usage();
+	}
+
+	/* -f: load file instead of stdin
+	 * -n: use NOSUBDIR flag on env_open
+	 * -s: load into named subDB
+	 * -N: use NOOVERWRITE on puts
+	 * -T: read plaintext
+	 * -V: print version and exit
+	 */
+	while ((i = getopt(argc, argv, "f:ns:NTV")) != EOF) {
+		switch(i) {
+		case 'V':
+			printf("%s\n", MDB_VERSION_STRING);
+			exit(0);
+			break;
+		case 'f':
+			if (freopen(optarg, "r", stdin) == NULL) {
+				fprintf(stderr, "%s: %s: reopen: %s\n",
+					prog, optarg, strerror(errno));
+				exit(EXIT_FAILURE);
+			}
+			break;
+		case 'n':
+			envflags |= MDB_NOSUBDIR;
+			break;
+		case 's':
+			subname = strdup(optarg);
+			break;
+		case 'N':
+			putflags = MDB_NOOVERWRITE|MDB_NODUPDATA;
+			break;
+		case 'T':
+			mode |= NOHDR;
+			break;
+		default:
+			usage();
+		}
+	}
+
+	if (optind != argc - 1)
+		usage();
+
+	dbuf.mv_size = 4096;
+	dbuf.mv_data = malloc(dbuf.mv_size);
+
+	if (!(mode & NOHDR))
+		readhdr();
+
+	envname = argv[optind];
+	rc = mdb_env_create(&env);
+	if (rc) {
+		fprintf(stderr, "mdb_env_create failed, error %d %s\n", rc, mdb_strerror(rc));
+		return EXIT_FAILURE;
+	}
+
+	mdb_env_set_maxdbs(env, 2);
+
+	if (info.me_maxreaders)
+		mdb_env_set_maxreaders(env, info.me_maxreaders);
+
+	if (info.me_mapsize)
+		mdb_env_set_mapsize(env, info.me_mapsize);
+
+	if (info.me_mapaddr)
+		envflags |= MDB_FIXEDMAP;
+
+	rc = mdb_env_open(env, envname, envflags, 0664);
+	if (rc) {
+		fprintf(stderr, "mdb_env_open failed, error %d %s\n", rc, mdb_strerror(rc));
+		goto env_close;
+	}
+
+	kbuf.mv_size = mdb_env_get_maxkeysize(env) * 2 + 2;
+	kbuf.mv_data = malloc(kbuf.mv_size);
+
+	while(!Eof) {
+		MDB_val key, data;
+		int batch = 0;
+		flags = 0;
+
+		if (!dohdr) {
+			dohdr = 1;
+		} else if (!(mode & NOHDR))
+			readhdr();
+		
+		rc = mdb_txn_begin(env, NULL, 0, &txn);
+		if (rc) {
+			fprintf(stderr, "mdb_txn_begin failed, error %d %s\n", rc, mdb_strerror(rc));
+			goto env_close;
+		}
+
+		rc = mdb_open(txn, subname, flags|MDB_CREATE, &dbi);
+		if (rc) {
+			fprintf(stderr, "mdb_open failed, error %d %s\n", rc, mdb_strerror(rc));
+			goto txn_abort;
+		}
+
+		rc = mdb_cursor_open(txn, dbi, &mc);
+		if (rc) {
+			fprintf(stderr, "mdb_cursor_open failed, error %d %s\n", rc, mdb_strerror(rc));
+			goto txn_abort;
+		}
+
+		while(1) {
+			rc = readline(&key, &kbuf);
+			if (rc == EOF)
+				break;
+			if (rc)
+				goto txn_abort;
+
+			rc = readline(&data, &dbuf);
+			if (rc)
+				goto txn_abort;
+			
+			rc = mdb_cursor_put(mc, &key, &data, putflags);
+			if (rc == MDB_KEYEXIST && putflags)
+				continue;
+			if (rc)
+				goto txn_abort;
+			batch++;
+			if (batch == 100) {
+				rc = mdb_txn_commit(txn);
+				if (rc) {
+					fprintf(stderr, "%s: line %" Z "d: txn_commit: %s\n",
+						prog, lineno, mdb_strerror(rc));
+					goto env_close;
+				}
+				rc = mdb_txn_begin(env, NULL, 0, &txn);
+				if (rc) {
+					fprintf(stderr, "mdb_txn_begin failed, error %d %s\n", rc, mdb_strerror(rc));
+					goto env_close;
+				}
+				rc = mdb_cursor_open(txn, dbi, &mc);
+				if (rc) {
+					fprintf(stderr, "mdb_cursor_open failed, error %d %s\n", rc, mdb_strerror(rc));
+					goto txn_abort;
+				}
+				batch = 0;
+			}
+		}
+		rc = mdb_txn_commit(txn);
+		txn = NULL;
+		if (rc) {
+			fprintf(stderr, "%s: line %" Z "d: txn_commit: %s\n",
+				prog, lineno, mdb_strerror(rc));
+			goto env_close;
+		}
+		mdb_dbi_close(env, dbi);
+	}
+
+txn_abort:
+	mdb_txn_abort(txn);
+env_close:
+	mdb_env_close(env);
+
+	return rc ? EXIT_FAILURE : EXIT_SUCCESS;
+}

external/db_drivers/liblmdb64/mdb_stat.1

@@ -0,0 +1,64 @@
+.TH MDB_STAT 1 "2014/06/20" "LMDB 0.9.14"
+.\" Copyright 2012-2014 Howard Chu, Symas Corp. All Rights Reserved.
+.\" Copying restrictions apply.  See COPYRIGHT/LICENSE.
+.SH NAME
+mdb_stat \- LMDB environment status tool
+.SH SYNOPSIS
+.B mdb_stat
+.BR \ envpath
+[\c
+.BR \-V ]
+[\c
+.BR \-e ]
+[\c
+.BR \-f [ f [ f ]]]
+[\c
+.BR \-n ]
+[\c
+.BR \-r [ r ]]
+[\c
+.BR \-a \ |
+.BI \-s \ subdb\fR]
+.SH DESCRIPTION
+The
+.B mdb_stat
+utility displays the status of an LMDB environment.
+.SH OPTIONS
+.TP
+.BR \-V
+Write the library version number to the standard output, and exit.
+.TP
+.BR \-e
+Display information about the database environment.
+.TP
+.BR \-f
+Display information about the environment freelist.
+If \fB\-ff\fP is given, summarize each freelist entry.
+If \fB\-fff\fP is given, display the full list of page IDs in the freelist.
+.TP
+.BR \-n
+Display the status of an LMDB database which does not use subdirectories.
+.TP
+.BR \-r
+Display information about the environment reader table.
+Shows the process ID, thread ID, and transaction ID for each active
+reader slot. The process ID and transaction ID are in decimal, the
+thread ID is in hexadecimal. The transaction ID is displayed as "-"
+if the reader does not currently have a read transaction open.
+If \fB\-rr\fP is given, check for stale entries in the reader
+table and clear them. The reader table will be printed again
+after the check is performed.
+.TP
+.BR \-a
+Display the status of all of the subdatabases in the environment.
+.TP
+.BR \-s \ subdb
+Display the status of a specific subdatabase.
+.SH DIAGNOSTICS
+Exit status is zero if no errors occur.
+Errors result in a non-zero exit status and
+a diagnostic message being written to standard error.
+.SH "SEE ALSO"
+.BR mdb_copy (1)
+.SH AUTHOR
+Howard Chu of Symas Corporation <http://www.symas.com>

external/db_drivers/liblmdb64/mdb_stat.c

@@ -0,0 +1,263 @@
+/* mdb_stat.c - memory-mapped database status tool */
+/*
+ * Copyright 2011-2013 Howard Chu, Symas Corp.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted only as authorized by the OpenLDAP
+ * Public License.
+ *
+ * A copy of this license is available in the file LICENSE in the
+ * top-level directory of the distribution or, alternatively, at
+ * <http://www.OpenLDAP.org/license.html>.
+ */
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include "lmdb.h"
+
+#ifdef	_WIN32
+#define	Z	"I"
+#else
+#define	Z	"z"
+#endif
+
+static void prstat(MDB_stat *ms)
+{
+#if 0
+	printf("  Page size: %u\n", ms->ms_psize);
+#endif
+	printf("  Tree depth: %u\n", ms->ms_depth);
+	printf("  Branch pages: %"Z"u\n", ms->ms_branch_pages);
+	printf("  Leaf pages: %"Z"u\n", ms->ms_leaf_pages);
+	printf("  Overflow pages: %"Z"u\n", ms->ms_overflow_pages);
+	printf("  Entries: %"Z"u\n", ms->ms_entries);
+}
+
+static void usage(char *prog)
+{
+	fprintf(stderr, "usage: %s dbpath [-V] [-n] [-e] [-r[r]] [-f[f[f]]] [-a|-s subdb]\n", prog);
+	exit(EXIT_FAILURE);
+}
+
+int main(int argc, char *argv[])
+{
+	int i, rc;
+	MDB_env *env;
+	MDB_txn *txn;
+	MDB_dbi dbi;
+	MDB_stat mst;
+	MDB_envinfo mei;
+	char *prog = argv[0];
+	char *envname;
+	char *subname = NULL;
+	int alldbs = 0, envinfo = 0, envflags = 0, freinfo = 0, rdrinfo = 0;
+
+	if (argc < 2) {
+		usage(prog);
+	}
+
+	/* -a: print stat of main DB and all subDBs
+	 * -s: print stat of only the named subDB
+	 * -e: print env info
+	 * -f: print freelist info
+	 * -r: print reader info
+	 * -n: use NOSUBDIR flag on env_open
+	 * -V: print version and exit
+	 * (default) print stat of only the main DB
+	 */
+	while ((i = getopt(argc, argv, "Vaefnrs:")) != EOF) {
+		switch(i) {
+		case 'V':
+			printf("%s\n", MDB_VERSION_STRING);
+			exit(0);
+			break;
+		case 'a':
+			if (subname)
+				usage(prog);
+			alldbs++;
+			break;
+		case 'e':
+			envinfo++;
+			break;
+		case 'f':
+			freinfo++;
+			break;
+		case 'n':
+			envflags |= MDB_NOSUBDIR;
+			break;
+		case 'r':
+			rdrinfo++;
+			break;
+		case 's':
+			if (alldbs)
+				usage(prog);
+			subname = optarg;
+			break;
+		default:
+			usage(prog);
+		}
+	}
+
+	if (optind != argc - 1)
+		usage(prog);
+
+	envname = argv[optind];
+	rc = mdb_env_create(&env);
+	if (rc) {
+		fprintf(stderr, "mdb_env_create failed, error %d %s\n", rc, mdb_strerror(rc));
+		return EXIT_FAILURE;
+	}
+
+	if (alldbs || subname) {
+		mdb_env_set_maxdbs(env, 4);
+	}
+
+	rc = mdb_env_open(env, envname, envflags | MDB_RDONLY, 0664);
+	if (rc) {
+		fprintf(stderr, "mdb_env_open failed, error %d %s\n", rc, mdb_strerror(rc));
+		goto env_close;
+	}
+
+	if (envinfo) {
+		(void)mdb_env_stat(env, &mst);
+		(void)mdb_env_info(env, &mei);
+		printf("Environment Info\n");
+		printf("  Map address: %p\n", mei.me_mapaddr);
+		printf("  Map size: %"Z"u\n", mei.me_mapsize);
+		printf("  Page size: %u\n", mst.ms_psize);
+		printf("  Max pages: %"Z"u\n", mei.me_mapsize / mst.ms_psize);
+		printf("  Number of pages used: %"Z"u\n", mei.me_last_pgno+1);
+		printf("  Last transaction ID: %"Z"u\n", mei.me_last_txnid);
+		printf("  Max readers: %u\n", mei.me_maxreaders);
+		printf("  Number of readers used: %u\n", mei.me_numreaders);
+	}
+
+	if (rdrinfo) {
+		printf("Reader Table Status\n");
+		rc = mdb_reader_list(env, (MDB_msg_func *)fputs, stdout);
+		if (rdrinfo > 1) {
+			int dead;
+			mdb_reader_check(env, &dead);
+			printf("  %d stale readers cleared.\n", dead);
+			rc = mdb_reader_list(env, (MDB_msg_func *)fputs, stdout);
+		}
+		if (!(subname || alldbs || freinfo))
+			goto env_close;
+	}
+
+	rc = mdb_txn_begin(env, NULL, MDB_RDONLY, &txn);
+	if (rc) {
+		fprintf(stderr, "mdb_txn_begin failed, error %d %s\n", rc, mdb_strerror(rc));
+		goto env_close;
+	}
+
+	if (freinfo) {
+		MDB_cursor *cursor;
+		MDB_val key, data;
+		size_t pages = 0, *iptr;
+
+		printf("Freelist Status\n");
+		dbi = 0;
+		rc = mdb_cursor_open(txn, dbi, &cursor);
+		if (rc) {
+			fprintf(stderr, "mdb_cursor_open failed, error %d %s\n", rc, mdb_strerror(rc));
+			goto txn_abort;
+		}
+		rc = mdb_stat(txn, dbi, &mst);
+		if (rc) {
+			fprintf(stderr, "mdb_stat failed, error %d %s\n", rc, mdb_strerror(rc));
+			goto txn_abort;
+		}
+		prstat(&mst);
+		while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
+			iptr = data.mv_data;
+			pages += *iptr;
+			if (freinfo > 1) {
+				char *bad = "";
+				size_t pg, prev;
+				ssize_t i, j, span = 0;
+				j = *iptr++;
+				for (i = j, prev = 1; --i >= 0; ) {
+					pg = iptr[i];
+					if (pg <= prev)
+						bad = " [bad sequence]";
+					prev = pg;
+					pg += span;
+					for (; i >= span && iptr[i-span] == pg; span++, pg++) ;
+				}
+				printf("    Transaction %"Z"u, %"Z"d pages, maxspan %"Z"d%s\n",
+					*(size_t *)key.mv_data, j, span, bad);
+				if (freinfo > 2) {
+					for (--j; j >= 0; ) {
+						pg = iptr[j];
+						for (span=1; --j >= 0 && iptr[j] == pg+span; span++) ;
+						printf(span>1 ? "     %9"Z"u[%"Z"d]\n" : "     %9"Z"u\n",
+							pg, span);
+					}
+				}
+			}
+		}
+		mdb_cursor_close(cursor);
+		printf("  Free pages: %"Z"u\n", pages);
+	}
+
+	rc = mdb_open(txn, subname, 0, &dbi);
+	if (rc) {
+		fprintf(stderr, "mdb_open failed, error %d %s\n", rc, mdb_strerror(rc));
+		goto txn_abort;
+	}
+
+	rc = mdb_stat(txn, dbi, &mst);
+	if (rc) {
+		fprintf(stderr, "mdb_stat failed, error %d %s\n", rc, mdb_strerror(rc));
+		goto txn_abort;
+	}
+	printf("Status of %s\n", subname ? subname : "Main DB");
+	prstat(&mst);
+
+	if (alldbs) {
+		MDB_cursor *cursor;
+		MDB_val key;
+
+		rc = mdb_cursor_open(txn, dbi, &cursor);
+		if (rc) {
+			fprintf(stderr, "mdb_cursor_open failed, error %d %s\n", rc, mdb_strerror(rc));
+			goto txn_abort;
+		}
+		while ((rc = mdb_cursor_get(cursor, &key, NULL, MDB_NEXT_NODUP)) == 0) {
+			char *str;
+			MDB_dbi db2;
+			if (memchr(key.mv_data, '\0', key.mv_size))
+				continue;
+			str = malloc(key.mv_size+1);
+			memcpy(str, key.mv_data, key.mv_size);
+			str[key.mv_size] = '\0';
+			rc = mdb_open(txn, str, 0, &db2);
+			if (rc == MDB_SUCCESS)
+				printf("Status of %s\n", str);
+			free(str);
+			if (rc) continue;
+			rc = mdb_stat(txn, db2, &mst);
+			if (rc) {
+				fprintf(stderr, "mdb_stat failed, error %d %s\n", rc, mdb_strerror(rc));
+				goto txn_abort;
+			}
+			prstat(&mst);
+			mdb_close(env, db2);
+		}
+		mdb_cursor_close(cursor);
+	}
+
+	if (rc == MDB_NOTFOUND)
+		rc = MDB_SUCCESS;
+
+	mdb_close(env, dbi);
+txn_abort:
+	mdb_txn_abort(txn);
+env_close:
+	mdb_env_close(env);
+
+	return rc ? EXIT_FAILURE : EXIT_SUCCESS;
+}

external/db_drivers/liblmdb64/midl.c

@@ -0,0 +1,360 @@
+/**	@file midl.c
+ *	@brief ldap bdb back-end ID List functions */
+/* $OpenLDAP$ */
+/* This work is part of OpenLDAP Software <http://www.openldap.org/>.
+ *
+ * Copyright 2000-2014 The OpenLDAP Foundation.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted only as authorized by the OpenLDAP
+ * Public License.
+ *
+ * A copy of this license is available in the file LICENSE in the
+ * top-level directory of the distribution or, alternatively, at
+ * <http://www.OpenLDAP.org/license.html>.
+ */
+
+#include <limits.h>
+#include <string.h>
+#include <stdlib.h>
+#include <errno.h>
+#include <sys/types.h>
+#include "midl.h"
+
+/** @defgroup internal	LMDB Internals
+ *	@{
+ */
+/** @defgroup idls	ID List Management
+ *	@{
+ */
+#define CMP(x,y)	 ( (x) < (y) ? -1 : (x) > (y) )
+
+unsigned mdb_midl_search( MDB_IDL ids, MDB_ID id )
+{
+	/*
+	 * binary search of id in ids
+	 * if found, returns position of id
+	 * if not found, returns first position greater than id
+	 */
+	unsigned base = 0;
+	unsigned cursor = 1;
+	int val = 0;
+	unsigned n = ids[0];
+
+	while( 0 < n ) {
+		unsigned pivot = n >> 1;
+		cursor = base + pivot + 1;
+		val = CMP( ids[cursor], id );
+
+		if( val < 0 ) {
+			n = pivot;
+
+		} else if ( val > 0 ) {
+			base = cursor;
+			n -= pivot + 1;
+
+		} else {
+			return cursor;
+		}
+	}
+
+	if( val > 0 ) {
+		++cursor;
+	}
+	return cursor;
+}
+
+#if 0	/* superseded by append/sort */
+int mdb_midl_insert( MDB_IDL ids, MDB_ID id )
+{
+	unsigned x, i;
+
+	x = mdb_midl_search( ids, id );
+	assert( x > 0 );
+
+	if( x < 1 ) {
+		/* internal error */
+		return -2;
+	}
+
+	if ( x <= ids[0] && ids[x] == id ) {
+		/* duplicate */
+		assert(0);
+		return -1;
+	}
+
+	if ( ++ids[0] >= MDB_IDL_DB_MAX ) {
+		/* no room */
+		--ids[0];
+		return -2;
+
+	} else {
+		/* insert id */
+		for (i=ids[0]; i>x; i--)
+			ids[i] = ids[i-1];
+		ids[x] = id;
+	}
+
+	return 0;
+}
+#endif
+
+MDB_IDL mdb_midl_alloc(int num)
+{
+	MDB_IDL ids = malloc((num+2) * sizeof(MDB_ID));
+	if (ids) {
+		*ids++ = num;
+		*ids = 0;
+	}
+	return ids;
+}
+
+void mdb_midl_free(MDB_IDL ids)
+{
+	if (ids)
+		free(ids-1);
+}
+
+int mdb_midl_shrink( MDB_IDL *idp )
+{
+	MDB_IDL ids = *idp;
+	if (*(--ids) > MDB_IDL_UM_MAX &&
+		(ids = realloc(ids, (MDB_IDL_UM_MAX+1) * sizeof(MDB_ID))))
+	{
+		*ids++ = MDB_IDL_UM_MAX;
+		*idp = ids;
+		return 1;
+	}
+	return 0;
+}
+
+static int mdb_midl_grow( MDB_IDL *idp, int num )
+{
+	MDB_IDL idn = *idp-1;
+	/* grow it */
+	idn = realloc(idn, (*idn + num + 2) * sizeof(MDB_ID));
+	if (!idn)
+		return ENOMEM;
+	*idn++ += num;
+	*idp = idn;
+	return 0;
+}
+
+int mdb_midl_need( MDB_IDL *idp, unsigned num )
+{
+	MDB_IDL ids = *idp;
+	num += ids[0];
+	if (num > ids[-1]) {
+		num = (num + num/4 + (256 + 2)) & -256;
+		if (!(ids = realloc(ids-1, num * sizeof(MDB_ID))))
+			return ENOMEM;
+		*ids++ = num - 2;
+		*idp = ids;
+	}
+	return 0;
+}
+
+int mdb_midl_append( MDB_IDL *idp, MDB_ID id )
+{
+	MDB_IDL ids = *idp;
+	/* Too big? */
+	if (ids[0] >= ids[-1]) {
+		if (mdb_midl_grow(idp, MDB_IDL_UM_MAX))
+			return ENOMEM;
+		ids = *idp;
+	}
+	ids[0]++;
+	ids[ids[0]] = id;
+	return 0;
+}
+
+int mdb_midl_append_list( MDB_IDL *idp, MDB_IDL app )
+{
+	MDB_IDL ids = *idp;
+	/* Too big? */
+	if (ids[0] + app[0] >= ids[-1]) {
+		if (mdb_midl_grow(idp, app[0]))
+			return ENOMEM;
+		ids = *idp;
+	}
+	memcpy(&ids[ids[0]+1], &app[1], app[0] * sizeof(MDB_ID));
+	ids[0] += app[0];
+	return 0;
+}
+
+int mdb_midl_append_range( MDB_IDL *idp, MDB_ID id, unsigned n )
+{
+	MDB_ID *ids = *idp, len = ids[0];
+	/* Too big? */
+	if (len + n > ids[-1]) {
+		if (mdb_midl_grow(idp, n | MDB_IDL_UM_MAX))
+			return ENOMEM;
+		ids = *idp;
+	}
+	ids[0] = len + n;
+	ids += len;
+	while (n)
+		ids[n--] = id++;
+	return 0;
+}
+
+void mdb_midl_xmerge( MDB_IDL idl, MDB_IDL merge )
+{
+	MDB_ID old_id, merge_id, i = merge[0], j = idl[0], k = i+j, total = k;
+	idl[0] = (MDB_ID)-1;		/* delimiter for idl scan below */
+	old_id = idl[j];
+	while (i) {
+		merge_id = merge[i--];
+		for (; old_id < merge_id; old_id = idl[--j])
+			idl[k--] = old_id;
+		idl[k--] = merge_id;
+	}
+	idl[0] = total;
+}
+
+/* Quicksort + Insertion sort for small arrays */
+
+#define SMALL	8
+#define	MIDL_SWAP(a,b)	{ itmp=(a); (a)=(b); (b)=itmp; }
+
+void
+mdb_midl_sort( MDB_IDL ids )
+{
+	/* Max possible depth of int-indexed tree * 2 items/level */
+	int istack[sizeof(int)*CHAR_BIT * 2];
+	int i,j,k,l,ir,jstack;
+	MDB_ID a, itmp;
+
+	ir = (int)ids[0];
+	l = 1;
+	jstack = 0;
+	for(;;) {
+		if (ir - l < SMALL) {	/* Insertion sort */
+			for (j=l+1;j<=ir;j++) {
+				a = ids[j];
+				for (i=j-1;i>=1;i--) {
+					if (ids[i] >= a) break;
+					ids[i+1] = ids[i];
+				}
+				ids[i+1] = a;
+			}
+			if (jstack == 0) break;
+			ir = istack[jstack--];
+			l = istack[jstack--];
+		} else {
+			k = (l + ir) >> 1;	/* Choose median of left, center, right */
+			MIDL_SWAP(ids[k], ids[l+1]);
+			if (ids[l] < ids[ir]) {
+				MIDL_SWAP(ids[l], ids[ir]);
+			}
+			if (ids[l+1] < ids[ir]) {
+				MIDL_SWAP(ids[l+1], ids[ir]);
+			}
+			if (ids[l] < ids[l+1]) {
+				MIDL_SWAP(ids[l], ids[l+1]);
+			}
+			i = l+1;
+			j = ir;
+			a = ids[l+1];
+			for(;;) {
+				do i++; while(ids[i] > a);
+				do j--; while(ids[j] < a);
+				if (j < i) break;
+				MIDL_SWAP(ids[i],ids[j]);
+			}
+			ids[l+1] = ids[j];
+			ids[j] = a;
+			jstack += 2;
+			if (ir-i+1 >= j-l) {
+				istack[jstack] = ir;
+				istack[jstack-1] = i;
+				ir = j-1;
+			} else {
+				istack[jstack] = j-1;
+				istack[jstack-1] = l;
+				l = i;
+			}
+		}
+	}
+}
+
+unsigned mdb_mid2l_search( MDB_ID2L ids, MDB_ID id )
+{
+	/*
+	 * binary search of id in ids
+	 * if found, returns position of id
+	 * if not found, returns first position greater than id
+	 */
+	unsigned base = 0;
+	unsigned cursor = 1;
+	int val = 0;
+	unsigned n = (unsigned)ids[0].mid;
+
+	while( 0 < n ) {
+		unsigned pivot = n >> 1;
+		cursor = base + pivot + 1;
+		val = CMP( id, ids[cursor].mid );
+
+		if( val < 0 ) {
+			n = pivot;
+
+		} else if ( val > 0 ) {
+			base = cursor;
+			n -= pivot + 1;
+
+		} else {
+			return cursor;
+		}
+	}
+
+	if( val > 0 ) {
+		++cursor;
+	}
+	return cursor;
+}
+
+int mdb_mid2l_insert( MDB_ID2L ids, MDB_ID2 *id )
+{
+	unsigned x, i;
+
+	x = mdb_mid2l_search( ids, id->mid );
+
+	if( x < 1 ) {
+		/* internal error */
+		return -2;
+	}
+
+	if ( x <= ids[0].mid && ids[x].mid == id->mid ) {
+		/* duplicate */
+		return -1;
+	}
+
+	if ( ids[0].mid >= MDB_IDL_UM_MAX ) {
+		/* too big */
+		return -2;
+
+	} else {
+		/* insert id */
+		ids[0].mid++;
+		for (i=(unsigned)ids[0].mid; i>x; i--)
+			ids[i] = ids[i-1];
+		ids[x] = *id;
+	}
+
+	return 0;
+}
+
+int mdb_mid2l_append( MDB_ID2L ids, MDB_ID2 *id )
+{
+	/* Too big? */
+	if (ids[0].mid >= MDB_IDL_UM_MAX) {
+		return -2;
+	}
+	ids[0].mid++;
+	ids[ids[0].mid] = *id;
+	return 0;
+}
+
+/** @} */
+/** @} */

external/db_drivers/liblmdb64/midl.h

@@ -0,0 +1,186 @@
+/**	@file midl.h
+ *	@brief LMDB ID List header file.
+ *
+ *	This file was originally part of back-bdb but has been
+ *	modified for use in libmdb. Most of the macros defined
+ *	in this file are unused, just left over from the original.
+ *
+ *	This file is only used internally in libmdb and its definitions
+ *	are not exposed publicly.
+ */
+/* $OpenLDAP$ */
+/* This work is part of OpenLDAP Software <http://www.openldap.org/>.
+ *
+ * Copyright 2000-2014 The OpenLDAP Foundation.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted only as authorized by the OpenLDAP
+ * Public License.
+ *
+ * A copy of this license is available in the file LICENSE in the
+ * top-level directory of the distribution or, alternatively, at
+ * <http://www.OpenLDAP.org/license.html>.
+ */
+
+#ifndef _MDB_MIDL_H_
+#define _MDB_MIDL_H_
+
+#include <stddef.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/** @defgroup internal	LMDB Internals
+ *	@{
+ */
+
+/** @defgroup idls	ID List Management
+ *	@{
+ */
+	/** A generic unsigned ID number. These were entryIDs in back-bdb.
+	 *	Preferably it should have the same size as a pointer.
+	 */
+typedef size_t MDB_ID;
+
+	/** An IDL is an ID List, a sorted array of IDs. The first
+	 * element of the array is a counter for how many actual
+	 * IDs are in the list. In the original back-bdb code, IDLs are
+	 * sorted in ascending order. For libmdb IDLs are sorted in
+	 * descending order.
+	 */
+typedef MDB_ID *MDB_IDL;
+
+/* IDL sizes - likely should be even bigger
+ *   limiting factors: sizeof(ID), thread stack size
+ */
+#define	MDB_IDL_LOGN	16	/* DB_SIZE is 2^16, UM_SIZE is 2^17 */
+#define MDB_IDL_DB_SIZE		(1<<MDB_IDL_LOGN)
+#define MDB_IDL_UM_SIZE		(1<<(MDB_IDL_LOGN+1))
+
+#define MDB_IDL_DB_MAX		(MDB_IDL_DB_SIZE-1)
+#define MDB_IDL_UM_MAX		(MDB_IDL_UM_SIZE-1)
+
+#define MDB_IDL_SIZEOF(ids)		(((ids)[0]+1) * sizeof(MDB_ID))
+#define MDB_IDL_IS_ZERO(ids) ( (ids)[0] == 0 )
+#define MDB_IDL_CPY( dst, src ) (memcpy( dst, src, MDB_IDL_SIZEOF( src ) ))
+#define MDB_IDL_FIRST( ids )	( (ids)[1] )
+#define MDB_IDL_LAST( ids )		( (ids)[(ids)[0]] )
+
+	/** Current max length of an #mdb_midl_alloc()ed IDL */
+#define MDB_IDL_ALLOCLEN( ids )	( (ids)[-1] )
+
+	/** Append ID to IDL. The IDL must be big enough. */
+#define mdb_midl_xappend(idl, id) do { \
+		MDB_ID *xidl = (idl), xlen = ++(xidl[0]); \
+		xidl[xlen] = (id); \
+	} while (0)
+
+	/** Search for an ID in an IDL.
+	 * @param[in] ids	The IDL to search.
+	 * @param[in] id	The ID to search for.
+	 * @return	The index of the first ID greater than or equal to \b id.
+	 */
+unsigned mdb_midl_search( MDB_IDL ids, MDB_ID id );
+
+	/** Allocate an IDL.
+	 * Allocates memory for an IDL of the given size.
+	 * @return	IDL on success, NULL on failure.
+	 */
+MDB_IDL mdb_midl_alloc(int num);
+
+	/** Free an IDL.
+	 * @param[in] ids	The IDL to free.
+	 */
+void mdb_midl_free(MDB_IDL ids);
+
+	/** Shrink an IDL.
+	 * Return the IDL to the default size if it has grown larger.
+	 * @param[in,out] idp	Address of the IDL to shrink.
+	 * @return	0 on no change, non-zero if shrunk.
+	 */
+int mdb_midl_shrink(MDB_IDL *idp);
+
+	/** Make room for num additional elements in an IDL.
+	 * @param[in,out] idp	Address of the IDL.
+	 * @param[in] num	Number of elements to make room for.
+	 * @return	0 on success, ENOMEM on failure.
+	 */
+int mdb_midl_need(MDB_IDL *idp, unsigned num);
+
+	/** Append an ID onto an IDL.
+	 * @param[in,out] idp	Address of the IDL to append to.
+	 * @param[in] id	The ID to append.
+	 * @return	0 on success, ENOMEM if the IDL is too large.
+	 */
+int mdb_midl_append( MDB_IDL *idp, MDB_ID id );
+
+	/** Append an IDL onto an IDL.
+	 * @param[in,out] idp	Address of the IDL to append to.
+	 * @param[in] app	The IDL to append.
+	 * @return	0 on success, ENOMEM if the IDL is too large.
+	 */
+int mdb_midl_append_list( MDB_IDL *idp, MDB_IDL app );
+
+	/** Append an ID range onto an IDL.
+	 * @param[in,out] idp	Address of the IDL to append to.
+	 * @param[in] id	The lowest ID to append.
+	 * @param[in] n		Number of IDs to append.
+	 * @return	0 on success, ENOMEM if the IDL is too large.
+	 */
+int mdb_midl_append_range( MDB_IDL *idp, MDB_ID id, unsigned n );
+
+	/** Merge an IDL onto an IDL. The destination IDL must be big enough.
+	 * @param[in] idl	The IDL to merge into.
+	 * @param[in] merge	The IDL to merge.
+	 */
+void mdb_midl_xmerge( MDB_IDL idl, MDB_IDL merge );
+
+	/** Sort an IDL.
+	 * @param[in,out] ids	The IDL to sort.
+	 */
+void mdb_midl_sort( MDB_IDL ids );
+
+	/** An ID2 is an ID/pointer pair.
+	 */
+typedef struct MDB_ID2 {
+	MDB_ID mid;		/**< The ID */
+	void *mptr;		/**< The pointer */
+} MDB_ID2;
+
+	/** An ID2L is an ID2 List, a sorted array of ID2s.
+	 * The first element's \b mid member is a count of how many actual
+	 * elements are in the array. The \b mptr member of the first element is unused.
+	 * The array is sorted in ascending order by \b mid.
+	 */
+typedef MDB_ID2 *MDB_ID2L;
+
+	/** Search for an ID in an ID2L.
+	 * @param[in] ids	The ID2L to search.
+	 * @param[in] id	The ID to search for.
+	 * @return	The index of the first ID2 whose \b mid member is greater than or equal to \b id.
+	 */
+unsigned mdb_mid2l_search( MDB_ID2L ids, MDB_ID id );
+
+
+	/** Insert an ID2 into a ID2L.
+	 * @param[in,out] ids	The ID2L to insert into.
+	 * @param[in] id	The ID2 to insert.
+	 * @return	0 on success, -1 if the ID was already present in the ID2L.
+	 */
+int mdb_mid2l_insert( MDB_ID2L ids, MDB_ID2 *id );
+
+	/** Append an ID2 into a ID2L.
+	 * @param[in,out] ids	The ID2L to append into.
+	 * @param[in] id	The ID2 to append.
+	 * @return	0 on success, -2 if the ID2L is too big.
+	 */
+int mdb_mid2l_append( MDB_ID2L ids, MDB_ID2 *id );
+
+/** @} */
+/** @} */
+#ifdef __cplusplus
+}
+#endif
+#endif	/* _MDB_MIDL_H_ */

external/db_drivers/liblmdb64/mtest.c

@@ -0,0 +1,173 @@
+/* mtest.c - memory-mapped database tester/toy */
+/*
+ * Copyright 2011-2014 Howard Chu, Symas Corp.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted only as authorized by the OpenLDAP
+ * Public License.
+ *
+ * A copy of this license is available in the file LICENSE in the
+ * top-level directory of the distribution or, alternatively, at
+ * <http://www.OpenLDAP.org/license.html>.
+ */
+#include <stdio.h>
+#include <stdlib.h>
+#include <time.h>
+#include "lmdb.h"
+
+#define E(expr) CHECK((rc = (expr)) == MDB_SUCCESS, #expr)
+#define RES(err, expr) ((rc = expr) == (err) || (CHECK(!rc, #expr), 0))
+#define CHECK(test, msg) ((test) ? (void)0 : ((void)fprintf(stderr, \
+	"%s:%d: %s: %s\n", __FILE__, __LINE__, msg, mdb_strerror(rc)), abort()))
+
+int main(int argc,char * argv[])
+{
+	int i = 0, j = 0, rc;
+	MDB_env *env;
+	MDB_dbi dbi;
+	MDB_val key, data;
+	MDB_txn *txn;
+	MDB_stat mst;
+	MDB_cursor *cursor, *cur2;
+	MDB_cursor_op op;
+	int count;
+	int *values;
+	char sval[32] = "";
+
+	srand(time(NULL));
+
+	    count = (rand()%384) + 64;
+	    values = (int *)malloc(count*sizeof(int));
+
+	    for(i = 0;i<count;i++) {
+			values[i] = rand()%1024;
+	    }
+    
+		E(mdb_env_create(&env));
+		E(mdb_env_set_mapsize(env, 10485760));
+		E(mdb_env_open(env, "./testdb", MDB_FIXEDMAP /*|MDB_NOSYNC*/, 0664));
+		E(mdb_txn_begin(env, NULL, 0, &txn));
+		E(mdb_open(txn, NULL, 0, &dbi));
+   
+		key.mv_size = sizeof(int);
+		key.mv_data = sval;
+		data.mv_size = sizeof(sval);
+		data.mv_data = sval;
+
+		printf("Adding %d values\n", count);
+	    for (i=0;i<count;i++) {	
+			sprintf(sval, "%03x %d foo bar", values[i], values[i]);
+			if (RES(MDB_KEYEXIST, mdb_put(txn, dbi, &key, &data, MDB_NOOVERWRITE))) {
+				j++;
+				data.mv_size = sizeof(sval);
+				data.mv_data = sval;
+			}
+	    }
+		if (j) printf("%d duplicates skipped\n", j);
+		E(mdb_txn_commit(txn));
+		E(mdb_env_stat(env, &mst));
+
+		E(mdb_txn_begin(env, NULL, 1, &txn));
+		E(mdb_cursor_open(txn, dbi, &cursor));
+		while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
+			printf("key: %p %.*s, data: %p %.*s\n",
+				key.mv_data,  (int) key.mv_size,  (char *) key.mv_data,
+				data.mv_data, (int) data.mv_size, (char *) data.mv_data);
+		}
+		CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+		mdb_cursor_close(cursor);
+		mdb_txn_abort(txn);
+
+		j=0;
+		key.mv_data = sval;
+	    for (i= count - 1; i > -1; i-= (rand()%5)) {
+			j++;
+			txn=NULL;
+			E(mdb_txn_begin(env, NULL, 0, &txn));
+			sprintf(sval, "%03x ", values[i]);
+			if (RES(MDB_NOTFOUND, mdb_del(txn, dbi, &key, NULL))) {
+				j--;
+				mdb_txn_abort(txn);
+			} else {
+				E(mdb_txn_commit(txn));
+			}
+	    }
+	    free(values);
+		printf("Deleted %d values\n", j);
+
+		E(mdb_env_stat(env, &mst));
+		E(mdb_txn_begin(env, NULL, 1, &txn));
+		E(mdb_cursor_open(txn, dbi, &cursor));
+		printf("Cursor next\n");
+		while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
+			printf("key: %.*s, data: %.*s\n",
+				(int) key.mv_size,  (char *) key.mv_data,
+				(int) data.mv_size, (char *) data.mv_data);
+		}
+		CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+		printf("Cursor last\n");
+		E(mdb_cursor_get(cursor, &key, &data, MDB_LAST));
+		printf("key: %.*s, data: %.*s\n",
+			(int) key.mv_size,  (char *) key.mv_data,
+			(int) data.mv_size, (char *) data.mv_data);
+		printf("Cursor prev\n");
+		while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_PREV)) == 0) {
+			printf("key: %.*s, data: %.*s\n",
+				(int) key.mv_size,  (char *) key.mv_data,
+				(int) data.mv_size, (char *) data.mv_data);
+		}
+		CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+		printf("Cursor last/prev\n");
+		E(mdb_cursor_get(cursor, &key, &data, MDB_LAST));
+			printf("key: %.*s, data: %.*s\n",
+				(int) key.mv_size,  (char *) key.mv_data,
+				(int) data.mv_size, (char *) data.mv_data);
+		E(mdb_cursor_get(cursor, &key, &data, MDB_PREV));
+			printf("key: %.*s, data: %.*s\n",
+				(int) key.mv_size,  (char *) key.mv_data,
+				(int) data.mv_size, (char *) data.mv_data);
+
+		mdb_txn_abort(txn);
+
+		printf("Deleting with cursor\n");
+		E(mdb_txn_begin(env, NULL, 0, &txn));
+		E(mdb_cursor_open(txn, dbi, &cur2));
+		for (i=0; i<50; i++) {
+			if (RES(MDB_NOTFOUND, mdb_cursor_get(cur2, &key, &data, MDB_NEXT)))
+				break;
+			printf("key: %p %.*s, data: %p %.*s\n",
+				key.mv_data,  (int) key.mv_size,  (char *) key.mv_data,
+				data.mv_data, (int) data.mv_size, (char *) data.mv_data);
+			E(mdb_del(txn, dbi, &key, NULL));
+		}
+
+		printf("Restarting cursor in txn\n");
+		for (op=MDB_FIRST, i=0; i<=32; op=MDB_NEXT, i++) {
+			if (RES(MDB_NOTFOUND, mdb_cursor_get(cur2, &key, &data, op)))
+				break;
+			printf("key: %p %.*s, data: %p %.*s\n",
+				key.mv_data,  (int) key.mv_size,  (char *) key.mv_data,
+				data.mv_data, (int) data.mv_size, (char *) data.mv_data);
+		}
+		mdb_cursor_close(cur2);
+		E(mdb_txn_commit(txn));
+
+		printf("Restarting cursor outside txn\n");
+		E(mdb_txn_begin(env, NULL, 0, &txn));
+		E(mdb_cursor_open(txn, dbi, &cursor));
+		for (op=MDB_FIRST, i=0; i<=32; op=MDB_NEXT, i++) {
+			if (RES(MDB_NOTFOUND, mdb_cursor_get(cursor, &key, &data, op)))
+				break;
+			printf("key: %p %.*s, data: %p %.*s\n",
+				key.mv_data,  (int) key.mv_size,  (char *) key.mv_data,
+				data.mv_data, (int) data.mv_size, (char *) data.mv_data);
+		}
+		mdb_cursor_close(cursor);
+		mdb_close(env, dbi);
+
+		mdb_txn_abort(txn);
+		mdb_env_close(env);
+
+	return 0;
+}

external/db_drivers/liblmdb64/mtest2.c

@@ -0,0 +1,123 @@
+/* mtest2.c - memory-mapped database tester/toy */
+/*
+ * Copyright 2011-2014 Howard Chu, Symas Corp.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted only as authorized by the OpenLDAP
+ * Public License.
+ *
+ * A copy of this license is available in the file LICENSE in the
+ * top-level directory of the distribution or, alternatively, at
+ * <http://www.OpenLDAP.org/license.html>.
+ */
+
+/* Just like mtest.c, but using a subDB instead of the main DB */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <time.h>
+#include "lmdb.h"
+
+#define E(expr) CHECK((rc = (expr)) == MDB_SUCCESS, #expr)
+#define RES(err, expr) ((rc = expr) == (err) || (CHECK(!rc, #expr), 0))
+#define CHECK(test, msg) ((test) ? (void)0 : ((void)fprintf(stderr, \
+	"%s:%d: %s: %s\n", __FILE__, __LINE__, msg, mdb_strerror(rc)), abort()))
+
+int main(int argc,char * argv[])
+{
+	int i = 0, j = 0, rc;
+	MDB_env *env;
+	MDB_dbi dbi;
+	MDB_val key, data;
+	MDB_txn *txn;
+	MDB_stat mst;
+	MDB_cursor *cursor;
+	int count;
+	int *values;
+	char sval[32] = "";
+
+	srand(time(NULL));
+
+	count = (rand()%384) + 64;
+	values = (int *)malloc(count*sizeof(int));
+
+	for(i = 0;i<count;i++) {
+		values[i] = rand()%1024;
+	}
+
+	E(mdb_env_create(&env));
+	E(mdb_env_set_mapsize(env, 10485760));
+	E(mdb_env_set_maxdbs(env, 4));
+	E(mdb_env_open(env, "./testdb", MDB_FIXEDMAP|MDB_NOSYNC, 0664));
+	E(mdb_txn_begin(env, NULL, 0, &txn));
+	E(mdb_open(txn, "id1", MDB_CREATE, &dbi));
+   
+	key.mv_size = sizeof(int);
+	key.mv_data = sval;
+	data.mv_size = sizeof(sval);
+	data.mv_data = sval;
+
+	printf("Adding %d values\n", count);
+	for (i=0;i<count;i++) {	
+		sprintf(sval, "%03x %d foo bar", values[i], values[i]);
+		if (RES(MDB_KEYEXIST, mdb_put(txn, dbi, &key, &data, MDB_NOOVERWRITE)))
+			j++;
+	}
+	if (j) printf("%d duplicates skipped\n", j);
+	E(mdb_txn_commit(txn));
+	E(mdb_env_stat(env, &mst));
+
+	E(mdb_txn_begin(env, NULL, 1, &txn));
+	E(mdb_cursor_open(txn, dbi, &cursor));
+	while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
+		printf("key: %p %.*s, data: %p %.*s\n",
+			key.mv_data,  (int) key.mv_size,  (char *) key.mv_data,
+			data.mv_data, (int) data.mv_size, (char *) data.mv_data);
+	}
+	CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+	mdb_cursor_close(cursor);
+	mdb_txn_abort(txn);
+
+	j=0;
+	key.mv_data = sval;
+	for (i= count - 1; i > -1; i-= (rand()%5)) {
+		j++;
+		txn=NULL;
+		E(mdb_txn_begin(env, NULL, 0, &txn));
+		sprintf(sval, "%03x ", values[i]);
+		if (RES(MDB_NOTFOUND, mdb_del(txn, dbi, &key, NULL))) {
+			j--;
+			mdb_txn_abort(txn);
+		} else {
+			E(mdb_txn_commit(txn));
+		}
+	}
+	free(values);
+	printf("Deleted %d values\n", j);
+
+	E(mdb_env_stat(env, &mst));
+	E(mdb_txn_begin(env, NULL, 1, &txn));
+	E(mdb_cursor_open(txn, dbi, &cursor));
+	printf("Cursor next\n");
+	while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
+		printf("key: %.*s, data: %.*s\n",
+			(int) key.mv_size,  (char *) key.mv_data,
+			(int) data.mv_size, (char *) data.mv_data);
+	}
+	CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+	printf("Cursor prev\n");
+	while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_PREV)) == 0) {
+		printf("key: %.*s, data: %.*s\n",
+			(int) key.mv_size,  (char *) key.mv_data,
+			(int) data.mv_size, (char *) data.mv_data);
+	}
+	CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+	mdb_cursor_close(cursor);
+	mdb_close(env, dbi);
+
+	mdb_txn_abort(txn);
+	mdb_env_close(env);
+
+	return 0;
+}

external/db_drivers/liblmdb64/mtest3.c

@@ -0,0 +1,133 @@
+/* mtest3.c - memory-mapped database tester/toy */
+/*
+ * Copyright 2011-2014 Howard Chu, Symas Corp.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted only as authorized by the OpenLDAP
+ * Public License.
+ *
+ * A copy of this license is available in the file LICENSE in the
+ * top-level directory of the distribution or, alternatively, at
+ * <http://www.OpenLDAP.org/license.html>.
+ */
+
+/* Tests for sorted duplicate DBs */
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <time.h>
+#include "lmdb.h"
+
+#define E(expr) CHECK((rc = (expr)) == MDB_SUCCESS, #expr)
+#define RES(err, expr) ((rc = expr) == (err) || (CHECK(!rc, #expr), 0))
+#define CHECK(test, msg) ((test) ? (void)0 : ((void)fprintf(stderr, \
+	"%s:%d: %s: %s\n", __FILE__, __LINE__, msg, mdb_strerror(rc)), abort()))
+
+int main(int argc,char * argv[])
+{
+	int i = 0, j = 0, rc;
+	MDB_env *env;
+	MDB_dbi dbi;
+	MDB_val key, data;
+	MDB_txn *txn;
+	MDB_stat mst;
+	MDB_cursor *cursor;
+	int count;
+	int *values;
+	char sval[32];
+	char kval[sizeof(int)];
+
+	srand(time(NULL));
+
+	memset(sval, 0, sizeof(sval));
+
+	count = (rand()%384) + 64;
+	values = (int *)malloc(count*sizeof(int));
+
+	for(i = 0;i<count;i++) {
+		values[i] = rand()%1024;
+	}
+
+	E(mdb_env_create(&env));
+	E(mdb_env_set_mapsize(env, 10485760));
+	E(mdb_env_set_maxdbs(env, 4));
+	E(mdb_env_open(env, "./testdb", MDB_FIXEDMAP|MDB_NOSYNC, 0664));
+	E(mdb_txn_begin(env, NULL, 0, &txn));
+	E(mdb_open(txn, "id2", MDB_CREATE|MDB_DUPSORT, &dbi));
+
+	key.mv_size = sizeof(int);
+	key.mv_data = kval;
+	data.mv_size = sizeof(sval);
+	data.mv_data = sval;
+
+	printf("Adding %d values\n", count);
+	for (i=0;i<count;i++) {
+		if (!(i & 0x0f))
+			sprintf(kval, "%03x", values[i]);
+		sprintf(sval, "%03x %d foo bar", values[i], values[i]);
+		if (RES(MDB_KEYEXIST, mdb_put(txn, dbi, &key, &data, MDB_NODUPDATA)))
+			j++;
+	}
+	if (j) printf("%d duplicates skipped\n", j);
+	E(mdb_txn_commit(txn));
+	E(mdb_env_stat(env, &mst));
+
+	E(mdb_txn_begin(env, NULL, 1, &txn));
+	E(mdb_cursor_open(txn, dbi, &cursor));
+	while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
+		printf("key: %p %.*s, data: %p %.*s\n",
+			key.mv_data,  (int) key.mv_size,  (char *) key.mv_data,
+			data.mv_data, (int) data.mv_size, (char *) data.mv_data);
+	}
+	CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+	mdb_cursor_close(cursor);
+	mdb_txn_abort(txn);
+
+	j=0;
+
+	for (i= count - 1; i > -1; i-= (rand()%5)) {
+		j++;
+		txn=NULL;
+		E(mdb_txn_begin(env, NULL, 0, &txn));
+		sprintf(kval, "%03x", values[i & ~0x0f]);
+		sprintf(sval, "%03x %d foo bar", values[i], values[i]);
+		key.mv_size = sizeof(int);
+		key.mv_data = kval;
+		data.mv_size = sizeof(sval);
+		data.mv_data = sval;
+		if (RES(MDB_NOTFOUND, mdb_del(txn, dbi, &key, &data))) {
+			j--;
+			mdb_txn_abort(txn);
+		} else {
+			E(mdb_txn_commit(txn));
+		}
+	}
+	free(values);
+	printf("Deleted %d values\n", j);
+
+	E(mdb_env_stat(env, &mst));
+	E(mdb_txn_begin(env, NULL, 1, &txn));
+	E(mdb_cursor_open(txn, dbi, &cursor));
+	printf("Cursor next\n");
+	while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
+		printf("key: %.*s, data: %.*s\n",
+			(int) key.mv_size,  (char *) key.mv_data,
+			(int) data.mv_size, (char *) data.mv_data);
+	}
+	CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+	printf("Cursor prev\n");
+	while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_PREV)) == 0) {
+		printf("key: %.*s, data: %.*s\n",
+			(int) key.mv_size,  (char *) key.mv_data,
+			(int) data.mv_size, (char *) data.mv_data);
+	}
+	CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+	mdb_cursor_close(cursor);
+	mdb_close(env, dbi);
+
+	mdb_txn_abort(txn);
+	mdb_env_close(env);
+
+	return 0;
+}

external/db_drivers/liblmdb64/mtest4.c

@@ -0,0 +1,168 @@
+/* mtest4.c - memory-mapped database tester/toy */
+/*
+ * Copyright 2011-2014 Howard Chu, Symas Corp.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted only as authorized by the OpenLDAP
+ * Public License.
+ *
+ * A copy of this license is available in the file LICENSE in the
+ * top-level directory of the distribution or, alternatively, at
+ * <http://www.OpenLDAP.org/license.html>.
+ */
+
+/* Tests for sorted duplicate DBs with fixed-size keys */
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <time.h>
+#include "lmdb.h"
+
+#define E(expr) CHECK((rc = (expr)) == MDB_SUCCESS, #expr)
+#define RES(err, expr) ((rc = expr) == (err) || (CHECK(!rc, #expr), 0))
+#define CHECK(test, msg) ((test) ? (void)0 : ((void)fprintf(stderr, \
+	"%s:%d: %s: %s\n", __FILE__, __LINE__, msg, mdb_strerror(rc)), abort()))
+
+int main(int argc,char * argv[])
+{
+	int i = 0, j = 0, rc;
+	MDB_env *env;
+	MDB_dbi dbi;
+	MDB_val key, data;
+	MDB_txn *txn;
+	MDB_stat mst;
+	MDB_cursor *cursor;
+	int count;
+	int *values;
+	char sval[8];
+	char kval[sizeof(int)];
+
+	memset(sval, 0, sizeof(sval));
+
+	count = 510;
+	values = (int *)malloc(count*sizeof(int));
+
+	for(i = 0;i<count;i++) {
+		values[i] = i*5;
+	}
+
+	E(mdb_env_create(&env));
+	E(mdb_env_set_mapsize(env, 10485760));
+	E(mdb_env_set_maxdbs(env, 4));
+	E(mdb_env_open(env, "./testdb", MDB_FIXEDMAP|MDB_NOSYNC, 0664));
+	E(mdb_txn_begin(env, NULL, 0, &txn));
+	E(mdb_open(txn, "id4", MDB_CREATE|MDB_DUPSORT|MDB_DUPFIXED, &dbi));
+
+	key.mv_size = sizeof(int);
+	key.mv_data = kval;
+	data.mv_size = sizeof(sval);
+	data.mv_data = sval;
+
+	printf("Adding %d values\n", count);
+	strcpy(kval, "001");
+	for (i=0;i<count;i++) {
+		sprintf(sval, "%07x", values[i]);
+		if (RES(MDB_KEYEXIST, mdb_put(txn, dbi, &key, &data, MDB_NODUPDATA)))
+			j++;
+	}
+	if (j) printf("%d duplicates skipped\n", j);
+	E(mdb_txn_commit(txn));
+	E(mdb_env_stat(env, &mst));
+
+	/* there should be one full page of dups now.
+	 */
+	E(mdb_txn_begin(env, NULL, 1, &txn));
+	E(mdb_cursor_open(txn, dbi, &cursor));
+	while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
+		printf("key: %p %.*s, data: %p %.*s\n",
+			key.mv_data,  (int) key.mv_size,  (char *) key.mv_data,
+			data.mv_data, (int) data.mv_size, (char *) data.mv_data);
+	}
+	CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+	mdb_cursor_close(cursor);
+	mdb_txn_abort(txn);
+
+	/* test all 3 branches of split code:
+	 * 1: new key in lower half
+	 * 2: new key at split point
+	 * 3: new key in upper half
+	 */
+
+	key.mv_size = sizeof(int);
+	key.mv_data = kval;
+	data.mv_size = sizeof(sval);
+	data.mv_data = sval;
+
+	sprintf(sval, "%07x", values[3]+1);
+	E(mdb_txn_begin(env, NULL, 0, &txn));
+	(void)RES(MDB_KEYEXIST, mdb_put(txn, dbi, &key, &data, MDB_NODUPDATA));
+	mdb_txn_abort(txn);
+
+	sprintf(sval, "%07x", values[255]+1);
+	E(mdb_txn_begin(env, NULL, 0, &txn));
+	(void)RES(MDB_KEYEXIST, mdb_put(txn, dbi, &key, &data, MDB_NODUPDATA));
+	mdb_txn_abort(txn);
+
+	sprintf(sval, "%07x", values[500]+1);
+	E(mdb_txn_begin(env, NULL, 0, &txn));
+	(void)RES(MDB_KEYEXIST, mdb_put(txn, dbi, &key, &data, MDB_NODUPDATA));
+	E(mdb_txn_commit(txn));
+
+	/* Try MDB_NEXT_MULTIPLE */
+	E(mdb_txn_begin(env, NULL, 0, &txn));
+	E(mdb_cursor_open(txn, dbi, &cursor));
+	while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT_MULTIPLE)) == 0) {
+		printf("key: %.*s, data: %.*s\n",
+			(int) key.mv_size,  (char *) key.mv_data,
+			(int) data.mv_size, (char *) data.mv_data);
+	}
+	CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+	mdb_cursor_close(cursor);
+	mdb_txn_abort(txn);
+	j=0;
+
+	for (i= count - 1; i > -1; i-= (rand()%3)) {
+		j++;
+		txn=NULL;
+		E(mdb_txn_begin(env, NULL, 0, &txn));
+		sprintf(sval, "%07x", values[i]);
+		key.mv_size = sizeof(int);
+		key.mv_data = kval;
+		data.mv_size = sizeof(sval);
+		data.mv_data = sval;
+		if (RES(MDB_NOTFOUND, mdb_del(txn, dbi, &key, &data))) {
+			j--;
+			mdb_txn_abort(txn);
+		} else {
+			E(mdb_txn_commit(txn));
+		}
+	}
+	free(values);
+	printf("Deleted %d values\n", j);
+
+	E(mdb_env_stat(env, &mst));
+	E(mdb_txn_begin(env, NULL, 1, &txn));
+	E(mdb_cursor_open(txn, dbi, &cursor));
+	printf("Cursor next\n");
+	while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
+		printf("key: %.*s, data: %.*s\n",
+			(int) key.mv_size,  (char *) key.mv_data,
+			(int) data.mv_size, (char *) data.mv_data);
+	}
+	CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+	printf("Cursor prev\n");
+	while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_PREV)) == 0) {
+		printf("key: %.*s, data: %.*s\n",
+			(int) key.mv_size,  (char *) key.mv_data,
+			(int) data.mv_size, (char *) data.mv_data);
+	}
+	CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+	mdb_cursor_close(cursor);
+	mdb_close(env, dbi);
+
+	mdb_txn_abort(txn);
+	mdb_env_close(env);
+
+	return 0;
+}

external/db_drivers/liblmdb64/mtest5.c

@@ -0,0 +1,135 @@
+/* mtest5.c - memory-mapped database tester/toy */
+/*
+ * Copyright 2011-2014 Howard Chu, Symas Corp.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted only as authorized by the OpenLDAP
+ * Public License.
+ *
+ * A copy of this license is available in the file LICENSE in the
+ * top-level directory of the distribution or, alternatively, at
+ * <http://www.OpenLDAP.org/license.html>.
+ */
+
+/* Tests for sorted duplicate DBs using cursor_put */
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <time.h>
+#include "lmdb.h"
+
+#define E(expr) CHECK((rc = (expr)) == MDB_SUCCESS, #expr)
+#define RES(err, expr) ((rc = expr) == (err) || (CHECK(!rc, #expr), 0))
+#define CHECK(test, msg) ((test) ? (void)0 : ((void)fprintf(stderr, \
+	"%s:%d: %s: %s\n", __FILE__, __LINE__, msg, mdb_strerror(rc)), abort()))
+
+int main(int argc,char * argv[])
+{
+	int i = 0, j = 0, rc;
+	MDB_env *env;
+	MDB_dbi dbi;
+	MDB_val key, data;
+	MDB_txn *txn;
+	MDB_stat mst;
+	MDB_cursor *cursor;
+	int count;
+	int *values;
+	char sval[32];
+	char kval[sizeof(int)];
+
+	srand(time(NULL));
+
+	memset(sval, 0, sizeof(sval));
+
+	count = (rand()%384) + 64;
+	values = (int *)malloc(count*sizeof(int));
+
+	for(i = 0;i<count;i++) {
+		values[i] = rand()%1024;
+	}
+
+	E(mdb_env_create(&env));
+	E(mdb_env_set_mapsize(env, 10485760));
+	E(mdb_env_set_maxdbs(env, 4));
+	E(mdb_env_open(env, "./testdb", MDB_FIXEDMAP|MDB_NOSYNC, 0664));
+	E(mdb_txn_begin(env, NULL, 0, &txn));
+	E(mdb_open(txn, "id2", MDB_CREATE|MDB_DUPSORT, &dbi));
+	E(mdb_cursor_open(txn, dbi, &cursor));
+
+	key.mv_size = sizeof(int);
+	key.mv_data = kval;
+	data.mv_size = sizeof(sval);
+	data.mv_data = sval;
+
+	printf("Adding %d values\n", count);
+	for (i=0;i<count;i++) {
+		if (!(i & 0x0f))
+			sprintf(kval, "%03x", values[i]);
+		sprintf(sval, "%03x %d foo bar", values[i], values[i]);
+		if (RES(MDB_KEYEXIST, mdb_cursor_put(cursor, &key, &data, MDB_NODUPDATA)))
+			j++;
+	}
+	if (j) printf("%d duplicates skipped\n", j);
+	mdb_cursor_close(cursor);
+	E(mdb_txn_commit(txn));
+	E(mdb_env_stat(env, &mst));
+
+	E(mdb_txn_begin(env, NULL, 1, &txn));
+	E(mdb_cursor_open(txn, dbi, &cursor));
+	while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
+		printf("key: %p %.*s, data: %p %.*s\n",
+			key.mv_data,  (int) key.mv_size,  (char *) key.mv_data,
+			data.mv_data, (int) data.mv_size, (char *) data.mv_data);
+	}
+	CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+	mdb_cursor_close(cursor);
+	mdb_txn_abort(txn);
+
+	j=0;
+
+	for (i= count - 1; i > -1; i-= (rand()%5)) {
+		j++;
+		txn=NULL;
+		E(mdb_txn_begin(env, NULL, 0, &txn));
+		sprintf(kval, "%03x", values[i & ~0x0f]);
+		sprintf(sval, "%03x %d foo bar", values[i], values[i]);
+		key.mv_size = sizeof(int);
+		key.mv_data = kval;
+		data.mv_size = sizeof(sval);
+		data.mv_data = sval;
+		if (RES(MDB_NOTFOUND, mdb_del(txn, dbi, &key, &data))) {
+			j--;
+			mdb_txn_abort(txn);
+		} else {
+			E(mdb_txn_commit(txn));
+		}
+	}
+	free(values);
+	printf("Deleted %d values\n", j);
+
+	E(mdb_env_stat(env, &mst));
+	E(mdb_txn_begin(env, NULL, 1, &txn));
+	E(mdb_cursor_open(txn, dbi, &cursor));
+	printf("Cursor next\n");
+	while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
+		printf("key: %.*s, data: %.*s\n",
+			(int) key.mv_size,  (char *) key.mv_data,
+			(int) data.mv_size, (char *) data.mv_data);
+	}
+	CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+	printf("Cursor prev\n");
+	while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_PREV)) == 0) {
+		printf("key: %.*s, data: %.*s\n",
+			(int) key.mv_size,  (char *) key.mv_data,
+			(int) data.mv_size, (char *) data.mv_data);
+	}
+	CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+	mdb_cursor_close(cursor);
+	mdb_close(env, dbi);
+
+	mdb_txn_abort(txn);
+	mdb_env_close(env);
+
+	return 0;
+}

external/db_drivers/liblmdb64/mtest6.c

@@ -0,0 +1,137 @@
+/* mtest6.c - memory-mapped database tester/toy */
+/*
+ * Copyright 2011-2014 Howard Chu, Symas Corp.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted only as authorized by the OpenLDAP
+ * Public License.
+ *
+ * A copy of this license is available in the file LICENSE in the
+ * top-level directory of the distribution or, alternatively, at
+ * <http://www.OpenLDAP.org/license.html>.
+ */
+
+/* Tests for DB splits and merges */
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <time.h>
+#include "lmdb.h"
+
+#define E(expr) CHECK((rc = (expr)) == MDB_SUCCESS, #expr)
+#define RES(err, expr) ((rc = expr) == (err) || (CHECK(!rc, #expr), 0))
+#define CHECK(test, msg) ((test) ? (void)0 : ((void)fprintf(stderr, \
+	"%s:%d: %s: %s\n", __FILE__, __LINE__, msg, mdb_strerror(rc)), abort()))
+
+char dkbuf[1024];
+
+int main(int argc,char * argv[])
+{
+	int i = 0, j = 0, rc;
+	MDB_env *env;
+	MDB_dbi dbi;
+	MDB_val key, data;
+	MDB_txn *txn;
+	MDB_stat mst;
+	MDB_cursor *cursor;
+	int count;
+	int *values;
+	long kval;
+	char *sval;
+
+	srand(time(NULL));
+
+	E(mdb_env_create(&env));
+	E(mdb_env_set_mapsize(env, 10485760));
+	E(mdb_env_set_maxdbs(env, 4));
+	E(mdb_env_open(env, "./testdb", MDB_FIXEDMAP|MDB_NOSYNC, 0664));
+	E(mdb_txn_begin(env, NULL, 0, &txn));
+	E(mdb_open(txn, "id6", MDB_CREATE|MDB_INTEGERKEY, &dbi));
+	E(mdb_cursor_open(txn, dbi, &cursor));
+	E(mdb_stat(txn, dbi, &mst));
+
+	sval = calloc(1, mst.ms_psize / 4);
+	key.mv_size = sizeof(long);
+	key.mv_data = &kval;
+	data.mv_size = mst.ms_psize / 4 - 30;
+	data.mv_data = sval;
+
+	printf("Adding 12 values, should yield 3 splits\n");
+	for (i=0;i<12;i++) {
+		kval = i*5;
+		sprintf(sval, "%08x", kval);
+		(void)RES(MDB_KEYEXIST, mdb_cursor_put(cursor, &key, &data, MDB_NOOVERWRITE));
+	}
+	printf("Adding 12 more values, should yield 3 splits\n");
+	for (i=0;i<12;i++) {
+		kval = i*5+4;
+		sprintf(sval, "%08x", kval);
+		(void)RES(MDB_KEYEXIST, mdb_cursor_put(cursor, &key, &data, MDB_NOOVERWRITE));
+	}
+	printf("Adding 12 more values, should yield 3 splits\n");
+	for (i=0;i<12;i++) {
+		kval = i*5+1;
+		sprintf(sval, "%08x", kval);
+		(void)RES(MDB_KEYEXIST, mdb_cursor_put(cursor, &key, &data, MDB_NOOVERWRITE));
+	}
+	E(mdb_cursor_get(cursor, &key, &data, MDB_FIRST));
+
+	do {
+		printf("key: %p %s, data: %p %.*s\n",
+			key.mv_data,  mdb_dkey(&key, dkbuf),
+			data.mv_data, (int) data.mv_size, (char *) data.mv_data);
+	} while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0);
+	CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+	mdb_cursor_close(cursor);
+	mdb_txn_commit(txn);
+
+#if 0
+	j=0;
+
+	for (i= count - 1; i > -1; i-= (rand()%5)) {
+		j++;
+		txn=NULL;
+		E(mdb_txn_begin(env, NULL, 0, &txn));
+		sprintf(kval, "%03x", values[i & ~0x0f]);
+		sprintf(sval, "%03x %d foo bar", values[i], values[i]);
+		key.mv_size = sizeof(int);
+		key.mv_data = kval;
+		data.mv_size = sizeof(sval);
+		data.mv_data = sval;
+		if (RES(MDB_NOTFOUND, mdb_del(txn, dbi, &key, &data))) {
+			j--;
+			mdb_txn_abort(txn);
+		} else {
+			E(mdb_txn_commit(txn));
+		}
+	}
+	free(values);
+	printf("Deleted %d values\n", j);
+
+	E(mdb_env_stat(env, &mst));
+	E(mdb_txn_begin(env, NULL, 1, &txn));
+	E(mdb_cursor_open(txn, dbi, &cursor));
+	printf("Cursor next\n");
+	while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
+		printf("key: %.*s, data: %.*s\n",
+			(int) key.mv_size,  (char *) key.mv_data,
+			(int) data.mv_size, (char *) data.mv_data);
+	}
+	CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+	printf("Cursor prev\n");
+	while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_PREV)) == 0) {
+		printf("key: %.*s, data: %.*s\n",
+			(int) key.mv_size,  (char *) key.mv_data,
+			(int) data.mv_size, (char *) data.mv_data);
+	}
+	CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get");
+	mdb_cursor_close(cursor);
+	mdb_close(env, dbi);
+
+	mdb_txn_abort(txn);
+#endif
+	mdb_env_close(env);
+
+	return 0;
+}

external/db_drivers/liblmdb64/sample-bdb.txt

@@ -0,0 +1,73 @@
+/* sample-bdb.txt - BerkeleyDB toy/sample
+ *
+ * Do a line-by-line comparison of this and sample-mdb.txt
+ */
+/*
+ * Copyright 2012 Howard Chu, Symas Corp.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted only as authorized by the OpenLDAP
+ * Public License.
+ *
+ * A copy of this license is available in the file LICENSE in the
+ * top-level directory of the distribution or, alternatively, at
+ * <http://www.OpenLDAP.org/license.html>.
+ */
+#include <stdio.h>
+#include <string.h>
+#include <db.h>
+
+int main(int argc,char * argv[])
+{
+	int rc;
+	DB_ENV *env;
+	DB *dbi;
+	DBT key, data;
+	DB_TXN *txn;
+	DBC *cursor;
+	char sval[32], kval[32];
+
+	/* Note: Most error checking omitted for simplicity */
+
+#define FLAGS (DB_INIT_LOCK|DB_INIT_LOG|DB_INIT_TXN|DB_INIT_MPOOL|DB_CREATE|DB_THREAD)
+	rc = db_env_create(&env, 0);
+	rc = env->open(env, "./testdb", FLAGS, 0664);
+	rc = db_create(&dbi, env, 0);
+	rc = env->txn_begin(env, NULL, &txn, 0);
+	rc = dbi->open(dbi, txn, "test.bdb", NULL, DB_BTREE, DB_CREATE, 0664);
+
+	memset(&key, 0, sizeof(DBT));
+	memset(&data, 0, sizeof(DBT));
+	key.size = sizeof(int);
+	key.data = sval;
+	data.size = sizeof(sval);
+	data.data = sval;
+
+	sprintf(sval, "%03x %d foo bar", 32, 3141592);
+	rc = dbi->put(dbi, txn, &key, &data, 0);
+	rc = txn->commit(txn, 0);
+	if (rc) {
+		fprintf(stderr, "txn->commit: (%d) %s\n", rc, db_strerror(rc));
+		goto leave;
+	}
+	rc = env->txn_begin(env, NULL, &txn, 0);
+	rc = dbi->cursor(dbi, txn, &cursor, 0);
+	key.flags = DB_DBT_USERMEM;
+	key.data = kval;
+	key.ulen = sizeof(kval);
+	data.flags = DB_DBT_USERMEM;
+	data.data = sval;
+	data.ulen = sizeof(sval);
+	while ((rc = cursor->c_get(cursor, &key, &data, DB_NEXT)) == 0) {
+		printf("key: %p %.*s, data: %p %.*s\n",
+			key.data,  (int) key.size,  (char *) key.data,
+			data.data, (int) data.size, (char *) data.data);
+	}
+	rc = cursor->c_close(cursor);
+	rc = txn->abort(txn);
+leave:
+	rc = dbi->close(dbi, 0);
+	rc = env->close(env, 0);
+	return rc;
+}

external/db_drivers/liblmdb64/sample-mdb.txt

@@ -0,0 +1,62 @@
+/* sample-mdb.txt - MDB toy/sample
+ *
+ * Do a line-by-line comparison of this and sample-bdb.txt
+ */
+/*
+ * Copyright 2012 Howard Chu, Symas Corp.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted only as authorized by the OpenLDAP
+ * Public License.
+ *
+ * A copy of this license is available in the file LICENSE in the
+ * top-level directory of the distribution or, alternatively, at
+ * <http://www.OpenLDAP.org/license.html>.
+ */
+#include <stdio.h>
+#include "lmdb.h"
+
+int main(int argc,char * argv[])
+{
+	int rc;
+	MDB_env *env;
+	MDB_dbi dbi;
+	MDB_val key, data;
+	MDB_txn *txn;
+	MDB_cursor *cursor;
+	char sval[32];
+
+	/* Note: Most error checking omitted for simplicity */
+
+	rc = mdb_env_create(&env);
+	rc = mdb_env_open(env, "./testdb", 0, 0664);
+	rc = mdb_txn_begin(env, NULL, 0, &txn);
+	rc = mdb_open(txn, NULL, 0, &dbi);
+
+	key.mv_size = sizeof(int);
+	key.mv_data = sval;
+	data.mv_size = sizeof(sval);
+	data.mv_data = sval;
+
+	sprintf(sval, "%03x %d foo bar", 32, 3141592);
+	rc = mdb_put(txn, dbi, &key, &data, 0);
+	rc = mdb_txn_commit(txn);
+	if (rc) {
+		fprintf(stderr, "mdb_txn_commit: (%d) %s\n", rc, mdb_strerror(rc));
+		goto leave;
+	}
+	rc = mdb_txn_begin(env, NULL, MDB_RDONLY, &txn);
+	rc = mdb_cursor_open(txn, dbi, &cursor);
+	while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) {
+		printf("key: %p %.*s, data: %p %.*s\n",
+			key.mv_data,  (int) key.mv_size,  (char *) key.mv_data,
+			data.mv_data, (int) data.mv_size, (char *) data.mv_data);
+	}
+	mdb_cursor_close(cursor);
+	mdb_txn_abort(txn);
+leave:
+	mdb_close(env, dbi);
+	mdb_env_close(env);
+	return 0;
+}

external/db_drivers/liblmdb64/tooltag

@@ -0,0 +1,22 @@
+<tagfile>
+  <compound kind="page">
+    <name>mdb_copy_1</name>
+	<title>mdb_copy - environment copy tool</title>
+	<filename>mdb_copy.1</filename>
+  </compound>
+  <compound kind="page">
+    <name>mdb_dump_1</name>
+	<title>mdb_dump - environment export tool</title>
+	<filename>mdb_dump.1</filename>
+  </compound>
+  <compound kind="page">
+    <name>mdb_load_1</name>
+	<title>mdb_load - environment import tool</title>
+	<filename>mdb_load.1</filename>
+  </compound>
+  <compound kind="page">
+    <name>mdb_stat_1</name>
+	<title>mdb_stat - environment status tool</title>
+	<filename>mdb_stat.1</filename>
+  </compound>
+</tagfile>

external/unbound/CMakeLists.txt

@@ -1,3 +1,31 @@
+# Copyright (c) 2014-2015, The Monero Project
+# 
+# All rights reserved.
+# 
+# Redistribution and use in source and binary forms, with or without modification, are
+# permitted provided that the following conditions are met:
+# 
+# 1. Redistributions of source code must retain the above copyright notice, this list of
+#    conditions and the following disclaimer.
+# 
+# 2. Redistributions in binary form must reproduce the above copyright notice, this list
+#    of conditions and the following disclaimer in the documentation and/or other
+#    materials provided with the distribution.
+# 
+# 3. Neither the name of the copyright holder nor the names of its contributors may be
+#    used to endorse or promote products derived from this software without specific
+#    prior written permission.
+# 
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+# THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+# THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
 cmake_minimum_required(VERSION 2.8.7)
 
 project(unbound C)

src/CMakeLists.txt

@@ -89,6 +89,7 @@ endfunction ()
 add_subdirectory(common)
 add_subdirectory(crypto)
 add_subdirectory(cryptonote_core)
+add_subdirectory(blockchain_db)
 add_subdirectory(mnemonics)
 add_subdirectory(rpc)
 add_subdirectory(wallet)
@@ -100,3 +101,5 @@ add_subdirectory(miner)
 add_subdirectory(simplewallet)
 add_subdirectory(daemonizer)
 add_subdirectory(daemon)
+
+add_subdirectory(blockchain_converter)

src/blockchain_converter/CMakeLists.txt

@@ -0,0 +1,115 @@
+# Copyright (c) 2014-2015, The Monero Project
+#
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are
+# permitted provided that the following conditions are met:
+#
+# 1. Redistributions of source code must retain the above copyright notice, this list of
+#    conditions and the following disclaimer.
+#
+# 2. Redistributions in binary form must reproduce the above copyright notice, this list
+#    of conditions and the following disclaimer in the documentation and/or other
+#    materials provided with the distribution.
+#
+# 3. Neither the name of the copyright holder nor the names of its contributors may be
+#    used to endorse or promote products derived from this software without specific
+#    prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+# THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+# THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+set(blockchain_converter_sources
+  blockchain_converter.cpp
+  )
+
+set(blockchain_converter_private_headers)
+
+bitmonero_private_headers(blockchain_converter
+	  ${blockchain_converter_private_headers})
+
+set(blockchain_import_sources
+  blockchain_import.cpp
+  )
+
+set(blockchain_import_private_headers
+  import.h
+  fake_core.h
+  )
+
+bitmonero_private_headers(blockchain_import
+	  ${blockchain_import_private_headers})
+
+set(blockchain_export_sources
+  blockchain_export.cpp
+  )
+
+set(blockchain_export_private_headers
+  import.h
+  blockchain_export.h
+  )
+
+bitmonero_private_headers(blockchain_export
+	  ${blockchain_export_private_headers})
+
+
+
+if (BLOCKCHAIN_DB STREQUAL DB_LMDB)
+bitmonero_add_executable(blockchain_converter
+  ${blockchain_converter_sources}
+  ${blockchain_converter_private_headers})
+
+target_link_libraries(blockchain_converter
+  LINK_PRIVATE
+    cryptonote_core
+	p2p
+	blockchain_db
+    ${CMAKE_THREAD_LIBS_INIT})
+
+add_dependencies(blockchain_converter
+	version)
+set_property(TARGET blockchain_converter
+	PROPERTY
+	OUTPUT_NAME "blockchain_converter")
+endif ()
+
+bitmonero_add_executable(blockchain_import
+  ${blockchain_import_sources}
+  ${blockchain_import_private_headers})
+
+target_link_libraries(blockchain_import
+  LINK_PRIVATE
+    cryptonote_core
+	blockchain_db
+	p2p
+    ${CMAKE_THREAD_LIBS_INIT})
+
+add_dependencies(blockchain_import
+	version)
+set_property(TARGET blockchain_import
+	PROPERTY
+	OUTPUT_NAME "blockchain_import")
+
+bitmonero_add_executable(blockchain_export
+  ${blockchain_export_sources}
+  ${blockchain_export_private_headers})
+
+target_link_libraries(blockchain_export
+  LINK_PRIVATE
+    cryptonote_core
+	p2p
+	blockchain_db
+    ${CMAKE_THREAD_LIBS_INIT})
+
+add_dependencies(blockchain_export
+	version)
+set_property(TARGET blockchain_export
+	PROPERTY
+	OUTPUT_NAME "blockchain_export")

src/blockchain_converter/README.md

@@ -0,0 +1,54 @@
+
+For importing into the LMDB database, compile with `DATABASE=lmdb`
+
+e.g.
+
+`DATABASE=lmdb make release`
+
+This is also the default compile setting on the blockchain branch.
+
+By default, the exporter will use the original in-memory database (blockchain.bin) as its source.
+This default is to make migrating to an LMDB database easy, without having to recompile anything.
+To change the source, adjust `SOURCE_DB` in `src/blockchain_converter/blockchain_export.h` according to the comments.
+
+# Usage:
+
+See also each utility's "--help" option.
+
+## Export an existing in-memory database
+
+`$ blockchain_export`
+
+This loads the existing blockchain, for whichever database type it was compiled for, and exports it to `$MONERO_DATA_DIR/export/blockchain.raw`
+
+## Import the exported file
+
+`$ blockchain_import`
+
+This imports blocks from `$MONERO_DATA_DIR/export/blockchain.raw` into the current database.
+
+Defaults: `--batch on`, `--batch size 20000`, `--verify on`
+
+Batch size refers to number of blocks and can be adjusted for performance based on available RAM.
+
+Verification should only be turned off if importing from a trusted blockchain.
+
+```bash
+# use default settings to import blockchain.raw into database
+$ blockchain_import
+
+# fast import with large batch size, verification off
+$ blockchain_import --batch-size 100000 --verify off
+
+# LMDB flags can be set by appending them to the database type:
+# flags: nosync, nometasync, writemap, mapasync
+$ blockchain_import --database lmdb#nosync
+$ blockchain_import --database lmdb#nosync,nometasync
+```
+
+## Blockchain converter with batching
+`blockchain_converter` has also been updated and includes batching for faster writes. However, on lower RAM systems, this will be slower than using the exporter and importer utilities. The converter needs to keep the blockchain in memory for the duration of the conversion, like the original bitmonerod, thus leaving less memory available to the destination database to operate.
+
+```bash
+$ blockchain_converter --batch on --batch-size 20000
+```

src/blockchain_converter/blockchain_converter.cpp

@@ -0,0 +1,299 @@
+// Copyright (c) 2014-2015, The Monero Project
+//
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+//
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+//
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+//
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#include "include_base_utils.h"
+#include "common/util.h"
+#include "warnings.h"
+#include "crypto/crypto.h"
+#include "cryptonote_config.h"
+#include "cryptonote_core/cryptonote_format_utils.h"
+#include "misc_language.h"
+#include "cryptonote_core/blockchain_storage.h"
+#include "blockchain_db/blockchain_db.h"
+#include "cryptonote_core/blockchain.h"
+#include "blockchain_db/lmdb/db_lmdb.h"
+#include "cryptonote_core/tx_pool.h"
+#include "common/command_line.h"
+#include "serialization/json_utils.h"
+#include "include_base_utils.h"
+#include "version.h"
+#include <iostream>
+
+
+unsigned int epee::g_test_dbg_lock_sleep = 0;
+
+namespace
+{
+
+// CONFIG
+bool opt_batch   = true;
+bool opt_resume  = true;
+bool opt_testnet = false;
+
+// number of blocks per batch transaction
+// adjustable through command-line argument according to available RAM
+uint64_t db_batch_size = 20000;
+
+}
+
+namespace po = boost::program_options;
+
+using namespace cryptonote;
+using namespace epee;
+
+struct fake_core
+{
+  Blockchain dummy;
+  tx_memory_pool m_pool;
+
+  blockchain_storage m_storage;
+
+#if !defined(BLOCKCHAIN_DB)
+  // for multi_db_runtime:
+  fake_core(const boost::filesystem::path &path, const bool use_testnet) : dummy(m_pool), m_pool(&dummy), m_storage(m_pool)
+#else
+  // for multi_db_compile:
+  fake_core(const boost::filesystem::path &path, const bool use_testnet) : dummy(m_pool), m_pool(dummy), m_storage(&m_pool)
+#endif
+  {
+    m_pool.init(path.string());
+    m_storage.init(path.string(), use_testnet);
+  }
+};
+
+int main(int argc, char* argv[])
+{
+  uint64_t height = 0;
+  uint64_t start_block = 0;
+  uint64_t end_block = 0;
+  uint64_t num_blocks = 0;
+
+  boost::filesystem::path default_data_path {tools::get_default_data_dir()};
+  boost::filesystem::path default_testnet_data_path {default_data_path / "testnet"};
+
+
+  po::options_description desc_cmd_only("Command line options");
+  po::options_description desc_cmd_sett("Command line options and settings options");
+  const command_line::arg_descriptor<uint32_t> arg_log_level   =  {"log-level", "", LOG_LEVEL_0};
+  const command_line::arg_descriptor<uint64_t> arg_batch_size  =  {"batch-size", "", db_batch_size};
+  const command_line::arg_descriptor<bool>     arg_testnet_on  = {
+    "testnet"
+      , "Run on testnet."
+      , opt_testnet
+  };
+  const command_line::arg_descriptor<uint64_t> arg_block_number =
+  {"block-number", "Number of blocks (default: use entire source blockchain)",
+    0};
+
+  command_line::add_arg(desc_cmd_sett, command_line::arg_data_dir, default_data_path.string());
+  command_line::add_arg(desc_cmd_sett, command_line::arg_testnet_data_dir, default_testnet_data_path.string());
+  command_line::add_arg(desc_cmd_sett, arg_log_level);
+  command_line::add_arg(desc_cmd_sett, arg_batch_size);
+  command_line::add_arg(desc_cmd_sett, arg_testnet_on);
+  command_line::add_arg(desc_cmd_sett, arg_block_number);
+
+  command_line::add_arg(desc_cmd_only, command_line::arg_help);
+
+  const command_line::arg_descriptor<bool> arg_batch  =  {"batch",
+    "Batch transactions for faster import", true};
+  const command_line::arg_descriptor<bool> arg_resume =  {"resume",
+        "Resume from current height if output database already exists", true};
+
+  // call add_options() directly for these arguments since command_line helpers
+  // support only boolean switch, not boolean argument
+  desc_cmd_sett.add_options()
+    (arg_batch.name,  make_semantic(arg_batch),  arg_batch.description)
+    (arg_resume.name, make_semantic(arg_resume), arg_resume.description)
+    ;
+
+  po::options_description desc_options("Allowed options");
+  desc_options.add(desc_cmd_only).add(desc_cmd_sett);
+
+
+  po::variables_map vm;
+  bool r = command_line::handle_error_helper(desc_options, [&]()
+  {
+    po::store(po::parse_command_line(argc, argv, desc_options), vm);
+    po::notify(vm);
+
+    return true;
+  });
+  if (!r)
+    return 1;
+
+  int log_level = command_line::get_arg(vm, arg_log_level);
+  opt_batch     = command_line::get_arg(vm, arg_batch);
+  opt_resume    = command_line::get_arg(vm, arg_resume);
+  db_batch_size = command_line::get_arg(vm, arg_batch_size);
+
+  if (command_line::get_arg(vm, command_line::arg_help))
+  {
+    std::cout << CRYPTONOTE_NAME << " v" << MONERO_VERSION_FULL << ENDL << ENDL;
+    std::cout << desc_options << std::endl;
+    return 1;
+  }
+
+  if (! opt_batch && ! vm["batch-size"].defaulted())
+  {
+    std::cerr << "Error: batch-size set, but batch option not enabled" << ENDL;
+    return 1;
+  }
+  if (! db_batch_size)
+  {
+    std::cerr << "Error: batch-size must be > 0" << ENDL;
+    return 1;
+  }
+
+  log_space::get_set_log_detalisation_level(true, log_level);
+  log_space::log_singletone::add_logger(LOGGER_CONSOLE, NULL, NULL);
+  LOG_PRINT_L0("Starting...");
+
+  std::string src_folder;
+  opt_testnet = command_line::get_arg(vm, arg_testnet_on);
+  auto data_dir_arg = opt_testnet ? command_line::arg_testnet_data_dir : command_line::arg_data_dir;
+  src_folder = command_line::get_arg(vm, data_dir_arg);
+  boost::filesystem::path dest_folder(src_folder);
+
+  num_blocks = command_line::get_arg(vm, arg_block_number);
+
+  if (opt_batch)
+  {
+    LOG_PRINT_L0("batch:   " << std::boolalpha << opt_batch << std::noboolalpha
+        << "  batch size: " << db_batch_size);
+  }
+  else
+  {
+    LOG_PRINT_L0("batch:   " << std::boolalpha << opt_batch << std::noboolalpha);
+  }
+  LOG_PRINT_L0("resume:  " << std::boolalpha << opt_resume  << std::noboolalpha);
+  LOG_PRINT_L0("testnet: " << std::boolalpha << opt_testnet << std::noboolalpha);
+
+  fake_core c(src_folder, opt_testnet);
+
+  height = c.m_storage.get_current_blockchain_height();
+
+  BlockchainDB *blockchain;
+  blockchain = new BlockchainLMDB(opt_batch);
+  dest_folder /= blockchain->get_db_name();
+  LOG_PRINT_L0("Source blockchain: " << src_folder);
+  LOG_PRINT_L0("Dest blockchain:   " << dest_folder.string());
+  LOG_PRINT_L0("Opening dest blockchain (BlockchainDB " << blockchain->get_db_name() << ")");
+  blockchain->open(dest_folder.string());
+  LOG_PRINT_L0("Source blockchain height: " << height);
+  LOG_PRINT_L0("Dest blockchain height:   " << blockchain->height());
+
+  if (opt_resume)
+    // next block number to add is same as current height
+    start_block = blockchain->height();
+
+  if (! num_blocks || (start_block + num_blocks > height))
+    end_block = height - 1;
+  else
+    end_block = start_block + num_blocks - 1;
+
+  LOG_PRINT_L0("start height: " << start_block+1 << "  stop height: " <<
+            end_block+1);
+
+  if (start_block > end_block)
+  {
+    LOG_PRINT_L0("Finished: no blocks to add");
+    delete blockchain;
+    return 0;
+  }
+
+  if (opt_batch)
+    blockchain->batch_start();
+  uint64_t i = 0;
+  for (i = start_block; i < end_block + 1; ++i)
+  {
+    // block: i  height: i+1  end height: end_block + 1
+    if ((i+1) % 10 == 0)
+    {
+      std::cout << "\r                   \r" << "height " << i+1 << "/" <<
+        end_block+1 << " (" << (i+1)*100/(end_block+1)<< "%)" << std::flush;
+    }
+    // for debugging:
+    // std::cout << "height " << i+1 << "/" << end_block+1
+    //   << " ((" << i+1 << ")*100/(end_block+1))" << "%)" << ENDL;
+
+    block b = c.m_storage.get_block(i);
+    size_t bsize = c.m_storage.get_block_size(i);
+    difficulty_type bdiff = c.m_storage.get_block_cumulative_difficulty(i);
+    uint64_t bcoins = c.m_storage.get_block_coins_generated(i);
+    std::vector<transaction> txs;
+    std::vector<crypto::hash> missed;
+
+    c.m_storage.get_transactions(b.tx_hashes, txs, missed);
+    if (missed.size())
+    {
+      std::cout << ENDL;
+      std::cerr << "Missed transaction(s) for block at height " << i + 1 << ", exiting" << ENDL;
+      delete blockchain;
+      return 1;
+    }
+
+    try
+    {
+      blockchain->add_block(b, bsize, bdiff, bcoins, txs);
+
+      if (opt_batch)
+      {
+        if ((i < end_block) && ((i + 1) % db_batch_size == 0))
+        {
+          std::cout << "\r                   \r";
+          std::cout << "[- batch commit at height " << i + 1 << " -]" << ENDL;
+          blockchain->batch_stop();
+          blockchain->batch_start();
+          std::cout << ENDL;
+          blockchain->show_stats();
+        }
+      }
+    }
+    catch (const std::exception& e)
+    {
+      std::cout << ENDL;
+      std::cerr << "Error adding block " << i << " to new blockchain: " << e.what() << ENDL;
+      delete blockchain;
+      return 2;
+    }
+  }
+  if (opt_batch)
+  {
+    std::cout << "\r                   \r" << "height " << i << "/" <<
+      end_block+1 << " (" << (i)*100/(end_block+1)<< "%)" << std::flush;
+    std::cout << ENDL;
+    std::cout << "[- batch commit at height " << i << " -]" << ENDL;
+    blockchain->batch_stop();
+  }
+  std::cout << ENDL;
+  blockchain->show_stats();
+  std::cout << "Finished at height: " << i << "  block: " << i-1 << ENDL;
+
+  delete blockchain;
+  return 0;
+}

src/blockchain_converter/blockchain_export.cpp

@@ -0,0 +1,396 @@
+// Copyright (c) 2014-2015, The Monero Project
+//
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+//
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+//
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+//
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#include <algorithm>
+#include <cstdio>
+#include <fstream>
+#include <boost/iostreams/copy.hpp>
+#include <atomic>
+
+#include <boost/archive/binary_oarchive.hpp>
+#include <boost/archive/binary_iarchive.hpp>
+#include <boost/iostreams/stream_buffer.hpp>
+#include <boost/iostreams/stream.hpp>
+#include <boost/iostreams/device/back_inserter.hpp>
+#include <boost/iostreams/filtering_streambuf.hpp>
+#include <boost/iostreams/filter/bzip2.hpp>
+#include "common/command_line.h"
+#include "version.h"
+#include "blockchain_export.h"
+#include "cryptonote_core/cryptonote_boost_serialization.h"
+
+#include "import.h"
+
+unsigned int epee::g_test_dbg_lock_sleep = 0;
+
+static int max_chunk = 0;
+static size_t height;
+
+namespace po = boost::program_options;
+
+using namespace cryptonote;
+using namespace epee;
+
+bool BlockchainExport::open(const boost::filesystem::path& dir_path)
+{
+  if (boost::filesystem::exists(dir_path))
+  {
+    if (!boost::filesystem::is_directory(dir_path))
+    {
+      LOG_PRINT_RED_L0("export directory path is a file: " << dir_path);
+      return false;
+    }
+  }
+  else
+  {
+    if (!boost::filesystem::create_directory(dir_path))
+    {
+      LOG_PRINT_RED_L0("Failed to create directory " << dir_path);
+      return false;
+    }
+  }
+
+  std::string file_path = (dir_path / BLOCKCHAIN_RAW).string();
+  m_raw_data_file = new std::ofstream();
+  m_raw_data_file->open(file_path , std::ios_base::binary | std::ios_base::out| std::ios::trunc);
+  if (m_raw_data_file->fail())
+    return false;
+
+  m_output_stream = new boost::iostreams::stream<boost::iostreams::back_insert_device<buffer_type>>(m_buffer);
+  m_raw_archive = new boost::archive::binary_oarchive(*m_output_stream);
+  if (m_raw_archive == NULL)
+    return false;
+
+  return true;
+}
+
+void BlockchainExport::flush_chunk()
+{
+  m_output_stream->flush();
+  char buffer[STR_LENGTH_OF_INT + 1];
+  int chunk_size = (int) m_buffer.size();
+  if (chunk_size > BUFFER_SIZE)
+  {
+    LOG_PRINT_L0("WARNING: chunk_size " << chunk_size << " > BUFFER_SIZE " << BUFFER_SIZE);
+  }
+  sprintf(buffer, STR_FORMAT_OF_INT, chunk_size);
+  m_raw_data_file->write(buffer, STR_LENGTH_OF_INT);
+  if (max_chunk < chunk_size)
+  {
+    max_chunk = chunk_size;
+  }
+  long pos_before = m_raw_data_file->tellp();
+  std::copy(m_buffer.begin(), m_buffer.end(), std::ostreambuf_iterator<char>(*m_raw_data_file));
+  m_raw_data_file->flush();
+  long pos_after = m_raw_data_file->tellp();
+  long num_chars_written = pos_after - pos_before;
+  if ((int) num_chars_written != chunk_size)
+  {
+    LOG_PRINT_RED_L0("INTERNAL ERROR: num chars wrote NEQ buffer size. height = " << height);
+  }
+
+  m_buffer.clear();
+  delete m_raw_archive;
+  delete m_output_stream;
+  m_output_stream = new boost::iostreams::stream<boost::iostreams::back_insert_device<buffer_type>>(m_buffer);
+  m_raw_archive = new boost::archive::binary_oarchive(*m_output_stream);
+}
+
+void BlockchainExport::serialize_block_to_text_buffer(const block& block)
+{
+  *m_raw_archive << block;
+}
+
+void BlockchainExport::buffer_serialize_tx(const transaction& tx)
+{
+  *m_raw_archive << tx;
+}
+
+void BlockchainExport::buffer_write_num_txs(const std::list<transaction> txs)
+{
+  int n = txs.size();
+  *m_raw_archive << n;
+}
+
+void BlockchainExport::write_block(block& block)
+{
+  serialize_block_to_text_buffer(block);
+  std::list<transaction> txs;
+
+  uint64_t block_height = boost::get<txin_gen>(block.miner_tx.vin.front()).height;
+
+  // put coinbase transaction first
+  transaction coinbase_tx = block.miner_tx;
+  crypto::hash coinbase_tx_hash = get_transaction_hash(coinbase_tx);
+#if SOURCE_DB == DB_MEMORY
+  const transaction* cb_tx_full = m_blockchain_storage->get_tx(coinbase_tx_hash);
+#else
+  transaction cb_tx_full = m_blockchain_storage->get_db().get_tx(coinbase_tx_hash);
+#endif
+
+#if SOURCE_DB == DB_MEMORY
+  if (cb_tx_full != NULL)
+  {
+    txs.push_back(*cb_tx_full);
+  }
+#else
+  // TODO: should check and abort if cb_tx_full equals null_hash?
+  txs.push_back(cb_tx_full);
+#endif
+
+  // now add all regular transactions
+  BOOST_FOREACH(const auto& tx_id, block.tx_hashes)
+  {
+#if SOURCE_DB == DB_MEMORY
+    const transaction* tx = m_blockchain_storage->get_tx(tx_id);
+#else
+    transaction tx = m_blockchain_storage->get_db().get_tx(tx_id);
+#endif
+
+#if SOURCE_DB == DB_MEMORY
+    if(tx == NULL)
+    {
+      if (! m_tx_pool)
+        throw std::runtime_error("Aborting: tx == NULL, so memory pool required to get tx, but memory pool isn't enabled");
+      else
+      {
+        transaction tx;
+        if(m_tx_pool->get_transaction(tx_id, tx))
+          txs.push_back(tx);
+        else
+          throw std::runtime_error("Aborting: tx not found in pool");
+      }
+    }
+    else
+      txs.push_back(*tx);
+#else
+    txs.push_back(tx);
+#endif
+  }
+
+  // serialize all txs to the persistant storage
+  buffer_write_num_txs(txs);
+  BOOST_FOREACH(const auto& tx, txs)
+  {
+    buffer_serialize_tx(tx);
+  }
+
+  // These three attributes are currently necessary for a fast import that adds blocks without verification.
+  bool include_extra_block_data = true;
+  if (include_extra_block_data)
+  {
+#if SOURCE_DB == DB_MEMORY
+    size_t block_size = m_blockchain_storage->get_block_size(block_height);
+#else
+    size_t block_size = m_blockchain_storage->get_db().get_block_size(block_height);
+#endif
+#if SOURCE_DB == DB_MEMORY
+    difficulty_type cumulative_difficulty = m_blockchain_storage->get_block_cumulative_difficulty(block_height);
+#else
+    difficulty_type cumulative_difficulty = m_blockchain_storage->get_db().get_block_cumulative_difficulty(block_height);
+#endif
+#if SOURCE_DB == DB_MEMORY
+    uint64_t coins_generated = m_blockchain_storage->get_block_coins_generated(block_height);
+#else
+    // TODO TEST to verify that this is the equivalent. make sure no off-by-one error with block height vs block number
+    uint64_t coins_generated = m_blockchain_storage->get_db().get_block_already_generated_coins(block_height);
+#endif
+
+    *m_raw_archive << block_size;
+    *m_raw_archive << cumulative_difficulty;
+    *m_raw_archive << coins_generated;
+  }
+}
+
+bool BlockchainExport::BlockchainExport::close()
+{
+  if (m_raw_data_file->fail())
+    return false;
+
+  m_raw_data_file->flush();
+  delete m_raw_archive;
+  delete m_output_stream;
+  delete m_raw_data_file;
+  return true;
+}
+
+
+#if SOURCE_DB == DB_MEMORY
+bool BlockchainExport::store_blockchain_raw(blockchain_storage* _blockchain_storage, tx_memory_pool* _tx_pool, boost::filesystem::path& output_dir, uint64_t requested_block_height)
+#else
+bool BlockchainExport::store_blockchain_raw(Blockchain* _blockchain_storage, tx_memory_pool* _tx_pool, boost::filesystem::path& output_dir, uint64_t requested_block_height)
+#endif
+{
+  uint64_t block_height = 0;
+  m_blockchain_storage = _blockchain_storage;
+  m_tx_pool = _tx_pool;
+  uint64_t progress_interval = 100;
+  std::string refresh_string = "\r                                    \r";
+  LOG_PRINT_L0("Storing blocks raw data...");
+  if (!BlockchainExport::open(output_dir))
+  {
+    LOG_PRINT_RED_L0("failed to open raw file for write");
+    return false;
+  }
+  block b;
+  LOG_PRINT_L0("source blockchain height: " <<  m_blockchain_storage->get_current_blockchain_height());
+  LOG_PRINT_L0("requested block height: " << requested_block_height);
+  if ((requested_block_height > 0) && (requested_block_height < m_blockchain_storage->get_current_blockchain_height()))
+    block_height = requested_block_height;
+  else
+  {
+    block_height = m_blockchain_storage->get_current_blockchain_height();
+    LOG_PRINT_L0("Using block height of source blockchain: " << block_height);
+  }
+  for (height=0; height < block_height; ++height)
+  {
+    crypto::hash hash = m_blockchain_storage->get_block_id_by_height(height);
+    m_blockchain_storage->get_block_by_hash(hash, b);
+    write_block(b);
+    if (height % NUM_BLOCKS_PER_CHUNK == 0) {
+      flush_chunk();
+    }
+    if (height % progress_interval == 0) {
+      std::cout << refresh_string;
+      std::cout << "height " << height << "/" << block_height << std::flush;
+    }
+  }
+  if (height % NUM_BLOCKS_PER_CHUNK != 0)
+  {
+    flush_chunk();
+  }
+  std::cout << refresh_string;
+  std::cout << "height " << height << "/" << block_height << ENDL;
+
+  LOG_PRINT_L0("longest chunk was " << max_chunk << " bytes");
+  return BlockchainExport::close();
+}
+
+
+int main(int argc, char* argv[])
+{
+  uint32_t log_level = 0;
+  uint64_t block_height = 0;
+  std::string import_filename = BLOCKCHAIN_RAW;
+
+  boost::filesystem::path default_data_path {tools::get_default_data_dir()};
+  boost::filesystem::path default_testnet_data_path {default_data_path / "testnet"};
+
+  po::options_description desc_cmd_only("Command line options");
+  po::options_description desc_cmd_sett("Command line options and settings options");
+  const command_line::arg_descriptor<uint32_t> arg_log_level   =  {"log-level",  "", log_level};
+  const command_line::arg_descriptor<uint64_t> arg_block_height =  {"block-number", "", block_height};
+  const command_line::arg_descriptor<bool>     arg_testnet_on  = {
+    "testnet"
+      , "Run on testnet."
+      , false
+  };
+
+
+  command_line::add_arg(desc_cmd_sett, command_line::arg_data_dir, default_data_path.string());
+  command_line::add_arg(desc_cmd_sett, command_line::arg_testnet_data_dir, default_testnet_data_path.string());
+  command_line::add_arg(desc_cmd_sett, arg_log_level);
+  command_line::add_arg(desc_cmd_sett, arg_block_height);
+  command_line::add_arg(desc_cmd_sett, arg_testnet_on);
+
+  command_line::add_arg(desc_cmd_only, command_line::arg_help);
+
+  po::options_description desc_options("Allowed options");
+  desc_options.add(desc_cmd_only).add(desc_cmd_sett);
+
+  po::variables_map vm;
+  bool r = command_line::handle_error_helper(desc_options, [&]()
+  {
+    po::store(po::parse_command_line(argc, argv, desc_options), vm);
+    po::notify(vm);
+    return true;
+  });
+  if (! r)
+    return 1;
+
+  if (command_line::get_arg(vm, command_line::arg_help))
+  {
+    std::cout << CRYPTONOTE_NAME << " v" << MONERO_VERSION_FULL << ENDL << ENDL;
+    std::cout << desc_options << std::endl;
+    return 1;
+  }
+
+  log_level    = command_line::get_arg(vm, arg_log_level);
+  block_height = command_line::get_arg(vm, arg_block_height);
+
+  log_space::get_set_log_detalisation_level(true, log_level);
+  log_space::log_singletone::add_logger(LOGGER_CONSOLE, NULL, NULL);
+  LOG_PRINT_L0("Starting...");
+  LOG_PRINT_L0("Setting log level = " << log_level);
+
+  bool opt_testnet = command_line::get_arg(vm, arg_testnet_on);
+
+  std::string m_config_folder;
+
+  auto data_dir_arg = opt_testnet ? command_line::arg_testnet_data_dir : command_line::arg_data_dir;
+  m_config_folder = command_line::get_arg(vm, data_dir_arg);
+  boost::filesystem::path output_dir {m_config_folder};
+  output_dir /= "export";
+  LOG_PRINT_L0("Export directory: " << output_dir.string());
+
+  // If we wanted to use the memory pool, we would set up a fake_core.
+
+#if SOURCE_DB == DB_MEMORY
+  // blockchain_storage* core_storage = NULL;
+  // tx_memory_pool m_mempool(*core_storage); // is this fake anyway? just passing in NULL! so m_mempool can't be used anyway, right?
+  // core_storage = new blockchain_storage(&m_mempool);
+
+  blockchain_storage* core_storage = new blockchain_storage(NULL);
+  LOG_PRINT_L0("Initializing source blockchain (in-memory database)");
+  r = core_storage->init(m_config_folder, opt_testnet);
+#else
+  // Use Blockchain instead of lower-level BlockchainDB for two reasons:
+  // 1. Blockchain has the init() method for easy setup
+  // 2. exporter needs to use get_current_blockchain_height(), get_block_id_by_height(), get_block_by_hash()
+  //
+  // cannot match blockchain_storage setup above with just one line,
+  // e.g.
+  //   Blockchain* core_storage = new Blockchain(NULL);
+  // because unlike blockchain_storage constructor, which takes a pointer to
+  // tx_memory_pool, Blockchain's constructor takes tx_memory_pool object.
+  LOG_PRINT_L0("Initializing source blockchain (BlockchainDB)");
+  Blockchain* core_storage = NULL;
+  tx_memory_pool m_mempool(*core_storage);
+  core_storage = new Blockchain(m_mempool);
+  r = core_storage->init(m_config_folder, opt_testnet);
+#endif
+
+  CHECK_AND_ASSERT_MES(r, false, "Failed to initialize source blockchain storage");
+  LOG_PRINT_L0("Source blockchain storage initialized OK");
+  LOG_PRINT_L0("Exporting blockchain raw data...");
+
+  BlockchainExport be;
+  r = be.store_blockchain_raw(core_storage, NULL, output_dir, block_height);
+  CHECK_AND_ASSERT_MES(r, false, "Failed to export blockchain raw data");
+  LOG_PRINT_L0("Blockchain raw data exported OK");
+}

src/blockchain_converter/blockchain_export.h

@@ -0,0 +1,87 @@
+// Copyright (c) 2014-2015, The Monero Project
+//
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+//
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+//
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+//
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#pragma once
+
+#include <boost/archive/binary_oarchive.hpp>
+#include <boost/iostreams/stream_buffer.hpp>
+#include <boost/iostreams/stream.hpp>
+#include <boost/iostreams/device/back_inserter.hpp>
+#include "cryptonote_core/cryptonote_basic.h"
+#include "cryptonote_core/blockchain_storage.h"
+#include "cryptonote_core/blockchain.h"
+#include "blockchain_db/blockchain_db.h"
+#include "blockchain_db/lmdb/db_lmdb.h"
+
+// CONFIG: choose one of the three #define's
+//
+// DB_MEMORY is a sensible default for users migrating to LMDB, as it allows
+// the exporter to use the in-memory blockchain while the other binaries
+// work with LMDB, without recompiling anything.
+//
+#define SOURCE_DB DB_MEMORY
+// #define SOURCE_DB DB_LMDB
+// to use global compile-time setting (DB_MEMORY or DB_LMDB):
+// #define SOURCE_DB BLOCKCHAIN_DB
+
+using namespace cryptonote;
+
+class BlockchainExport
+{
+public:
+#if SOURCE_DB == DB_MEMORY
+  bool store_blockchain_raw(cryptonote::blockchain_storage* cs, cryptonote::tx_memory_pool* txp,
+      boost::filesystem::path& output_dir, uint64_t use_block_height=0);
+#else
+  bool store_blockchain_raw(cryptonote::Blockchain* cs, cryptonote::tx_memory_pool* txp,
+      boost::filesystem::path& output_dir, uint64_t use_block_height=0);
+#endif
+
+protected:
+#if SOURCE_DB == DB_MEMORY
+  blockchain_storage* m_blockchain_storage;
+#else
+  Blockchain* m_blockchain_storage;
+#endif
+
+  tx_memory_pool* m_tx_pool;
+  typedef std::vector<char> buffer_type;
+  std::ofstream * m_raw_data_file;
+  boost::archive::binary_oarchive * m_raw_archive;
+  buffer_type m_buffer;
+  boost::iostreams::stream<boost::iostreams::back_insert_device<buffer_type>>* m_output_stream;
+
+  // open export file for write
+  bool open(const boost::filesystem::path& dir_path);
+  bool close();
+  void write_block(block& block);
+  void serialize_block_to_text_buffer(const block& block);
+  void buffer_serialize_tx(const transaction& tx);
+  void buffer_write_num_txs(const std::list<transaction> txs);
+  void flush_chunk();
+};

src/blockchain_converter/blockchain_import.cpp

@@ -0,0 +1,756 @@
+// Copyright (c) 2014-2015, The Monero Project
+//
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+//
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+//
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+//
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#include <atomic>
+#include <cstdio>
+#include <algorithm>
+#include <fstream>
+
+#include <boost/filesystem.hpp>
+#include <boost/iostreams/stream.hpp>
+#include <boost/archive/binary_iarchive.hpp>
+#include "cryptonote_core/cryptonote_basic.h"
+#include "cryptonote_core/cryptonote_format_utils.h"
+#include "cryptonote_core/cryptonote_boost_serialization.h"
+#include "serialization/json_utils.h" // dump_json()
+#include "include_base_utils.h"
+#include "common/command_line.h"
+#include "version.h"
+
+#include <lmdb.h> // for db flag arguments
+
+#include "import.h"
+#include "fake_core.h"
+
+unsigned int epee::g_test_dbg_lock_sleep = 0;
+
+// CONFIG
+static bool opt_batch   = true;
+static bool opt_verify  = true; // use add_new_block, which does verification before calling add_block
+static bool opt_resume  = true;
+static bool opt_testnet = true;
+
+// number of blocks per batch transaction
+// adjustable through command-line argument according to available RAM
+static uint64_t db_batch_size = 20000;
+
+static std::string refresh_string = "\r                                    \r";
+
+
+namespace po = boost::program_options;
+
+using namespace cryptonote;
+using namespace epee;
+
+
+int parse_db_arguments(const std::string& db_arg_str, std::string& db_engine, int& mdb_flags)
+{
+  std::vector<std::string> db_args;
+  boost::split(db_args, db_arg_str, boost::is_any_of("#"));
+  db_engine = db_args.front();
+  boost::algorithm::trim(db_engine);
+
+  if (db_args.size() == 1)
+  {
+    return 0;
+  }
+  else if (db_args.size() > 2)
+  {
+    std::cerr << "unrecognized database argument format: " << db_arg_str << ENDL;
+    return 1;
+  }
+
+  std::string db_arg_str2 = db_args[1];
+  boost::split(db_args, db_arg_str2, boost::is_any_of(","));
+  for (auto& it : db_args)
+  {
+    boost::algorithm::trim(it);
+    if (it.empty())
+      continue;
+    LOG_PRINT_L1("LMDB flag: " << it);
+    if (it == "nosync")
+    {
+      mdb_flags |= MDB_NOSYNC;
+    }
+    else if (it == "nometasync")
+    {
+      mdb_flags |= MDB_NOMETASYNC;
+    }
+    else if (it == "writemap")
+    {
+      mdb_flags |= MDB_WRITEMAP;
+    }
+    else if (it == "mapasync")
+    {
+      mdb_flags |= MDB_MAPASYNC;
+    }
+    else
+    {
+      std::cerr << "unrecognized database flag: " << it << ENDL;
+      return 1;
+    }
+  }
+  return 0;
+}
+
+
+int count_blocks(std::string& import_file_path)
+{
+  boost::filesystem::path raw_file_path(import_file_path);
+  boost::system::error_code ec;
+  if (!boost::filesystem::exists(raw_file_path, ec))
+  {
+    LOG_PRINT_L0("import file not found: " << raw_file_path);
+    throw std::runtime_error("Aborting");
+  }
+  std::ifstream import_file;
+  import_file.open(import_file_path, std::ios_base::binary | std::ifstream::in);
+
+  uint64_t h = 0;
+  if (import_file.fail())
+  {
+    LOG_PRINT_L0("import_file.open() fail");
+    throw std::runtime_error("Aborting");
+  }
+  LOG_PRINT_L0("Scanning blockchain from import file...");
+  char buffer1[STR_LENGTH_OF_INT + 1];
+  block b;
+  transaction tx;
+  bool quit = false;
+  uint64_t bytes_read = 0;
+  int progress_interval = 10;
+
+  while (! quit)
+  {
+    int chunk_size;
+    import_file.read(buffer1, STR_LENGTH_OF_INT);
+    if (!import_file) {
+      std::cout << refresh_string;
+      LOG_PRINT_L1("End of import file reached");
+      quit = true;
+      break;
+    }
+    h += NUM_BLOCKS_PER_CHUNK;
+    if (h % progress_interval == 0)
+    {
+      std::cout << refresh_string << "block height: " << h <<
+        std::flush;
+    }
+    bytes_read += STR_LENGTH_OF_INT;
+    buffer1[STR_LENGTH_OF_INT] = '\0';
+    chunk_size = atoi(buffer1);
+    if (chunk_size > BUFFER_SIZE)
+    {
+      std::cout << refresh_string;
+      LOG_PRINT_L0("WARNING: chunk_size " << chunk_size << " > BUFFER_SIZE " << BUFFER_SIZE
+          << "  height: " << h);
+      throw std::runtime_error("Aborting: chunk size exceeds buffer size");
+    }
+    if (chunk_size > 100000)
+    {
+      std::cout << refresh_string;
+      LOG_PRINT_L0("WARNING: chunk_size " << chunk_size << " > 100000" << "  height: "
+          << h);
+    }
+    else if (chunk_size <= 0) {
+      std::cout << refresh_string;
+      LOG_PRINT_L0("ERROR: chunk_size " << chunk_size << " <= 0" << "  height: " << h);
+      throw std::runtime_error("Aborting");
+    }
+    // skip to next expected block size value
+    import_file.seekg(chunk_size, std::ios_base::cur);
+    if (! import_file) {
+      std::cout << refresh_string;
+      LOG_PRINT_L0("ERROR: unexpected end of import file: bytes read before error: "
+          << import_file.gcount() << " of chunk_size " << chunk_size);
+      throw std::runtime_error("Aborting");
+    }
+    bytes_read += chunk_size;
+    std::cout << refresh_string;
+
+    LOG_PRINT_L3("Total bytes scanned: " << bytes_read);
+  }
+
+  import_file.close();
+
+  std::cout << ENDL;
+  std::cout << "Done scanning import file" << ENDL;
+  std::cout << "Total bytes scanned: " << bytes_read << ENDL;
+  std::cout << "Height: " << h << ENDL;
+
+  return h;
+}
+
+template <typename FakeCore>
+int import_from_file(FakeCore& simple_core, std::string& import_file_path)
+{
+#if !defined(BLOCKCHAIN_DB)
+  static_assert(std::is_same<fake_core_memory, FakeCore>::value || std::is_same<fake_core_lmdb, FakeCore>::value,
+      "FakeCore constraint error");
+#endif
+#if !defined(BLOCKCHAIN_DB) || (BLOCKCHAIN_DB == DB_LMDB)
+  if (std::is_same<fake_core_lmdb, FakeCore>::value)
+  {
+    // Reset stats, in case we're using newly created db, accumulating stats
+    // from addition of genesis block.
+    // This aligns internal db counts with importer counts.
+    simple_core.m_storage.get_db().reset_stats();
+  }
+#endif
+  boost::filesystem::path raw_file_path(import_file_path);
+  boost::system::error_code ec;
+  if (!boost::filesystem::exists(raw_file_path, ec))
+  {
+    LOG_PRINT_L0("import file not found: " << raw_file_path);
+    return false;
+  }
+
+  uint64_t source_height = count_blocks(import_file_path);
+  LOG_PRINT_L0("import file blockchain height: " << source_height);
+
+  std::ifstream import_file;
+  import_file.open(import_file_path, std::ios_base::binary | std::ifstream::in);
+
+  uint64_t h = 0;
+  if (import_file.fail())
+  {
+    LOG_PRINT_L0("import_file.open() fail");
+    return false;
+  }
+  char buffer1[STR_LENGTH_OF_INT + 1];
+  char buffer_block[BUFFER_SIZE];
+  block b;
+  transaction tx;
+  int quit = 0;
+  uint64_t bytes_read = 0;
+
+  uint64_t start_height = 1;
+  if (opt_resume)
+    start_height = simple_core.m_storage.get_current_blockchain_height();
+
+  // Note that a new blockchain will start with a height of 1 (block number 0)
+  // due to genesis block being added at initialization.
+
+  // CONFIG
+  // TODO: can expand on this, e.g. with --block-number option
+  uint64_t stop_height = source_height;
+
+  // These are what we'll try to use, and they don't have to be a determination
+  // from source and destination blockchains, but those are the current
+  // defaults.
+  LOG_PRINT_L0("start height: " << start_height << "  stop height: " <<
+      stop_height);
+
+  bool use_batch = false;
+  if (opt_batch)
+  {
+    if (simple_core.support_batch)
+      use_batch = true;
+    else
+      LOG_PRINT_L0("WARNING: batch transactions enabled but unsupported or unnecessary for this database engine - ignoring");
+  }
+
+  if (use_batch)
+    simple_core.batch_start();
+
+  LOG_PRINT_L0("Reading blockchain from import file...");
+  std::cout << ENDL;
+
+  // Within the loop, we skip to start_height before we start adding.
+  // TODO: Not a bottleneck, but we can use what's done in count_blocks() and
+  // only do the chunk size reads, skipping the chunk content reads until we're
+  // at start_height.
+  while (! quit)
+  {
+    int chunk_size;
+    import_file.read(buffer1, STR_LENGTH_OF_INT);
+    if (! import_file) {
+      std::cout << refresh_string;
+      LOG_PRINT_L0("End of import file reached");
+      quit = 1;
+      break;
+    }
+    bytes_read += STR_LENGTH_OF_INT;
+    buffer1[STR_LENGTH_OF_INT] = '\0';
+    chunk_size = atoi(buffer1);
+    if (chunk_size > BUFFER_SIZE)
+    {
+      LOG_PRINT_L0("WARNING: chunk_size " << chunk_size << " > BUFFER_SIZE " << BUFFER_SIZE);
+      throw std::runtime_error("Aborting: chunk size exceeds buffer size");
+    }
+    if (chunk_size > 100000)
+    {
+      LOG_PRINT_L0("WARNING: chunk_size " << chunk_size << " > 100000");
+    }
+    else if (chunk_size < 0) {
+      LOG_PRINT_L0("ERROR: chunk_size " << chunk_size << " < 0");
+      return 2;
+    }
+    import_file.read(buffer_block, chunk_size);
+    if (! import_file) {
+      LOG_PRINT_L0("ERROR: unexpected end of import file: bytes read before error: "
+          << import_file.gcount() << " of chunk_size " << chunk_size);
+      return 2;
+    }
+    bytes_read += chunk_size;
+    LOG_PRINT_L3("Total bytes read: " << bytes_read);
+
+    if (h + NUM_BLOCKS_PER_CHUNK < start_height + 1)
+    {
+      h += NUM_BLOCKS_PER_CHUNK;
+      continue;
+    }
+    if (h > stop_height)
+    {
+      LOG_PRINT_L0("Specified height reached - stopping.  height: " << h << "  block: " << h-1);
+      quit = 1;
+      break;
+    }
+
+    try
+    {
+      boost::iostreams::basic_array_source<char> device(buffer_block, chunk_size);
+      boost::iostreams::stream<boost::iostreams::basic_array_source<char>> s(device);
+      boost::archive::binary_iarchive a(s);
+
+      int display_interval = 1000;
+      int progress_interval = 10;
+      for (int chunk_ind = 0; chunk_ind < NUM_BLOCKS_PER_CHUNK; chunk_ind++)
+      {
+        h++;
+        if (h % display_interval == 0)
+        {
+          std::cout << refresh_string;
+          LOG_PRINT_L0("loading block height " << h);
+        }
+        else
+        {
+          LOG_PRINT_L3("loading block height " << h);
+        }
+        try {
+          a >> b;
+        }
+        catch (const std::exception& e)
+        {
+          std::cout << refresh_string;
+          LOG_PRINT_RED_L0("exception while de-archiving block, height=" << h);
+          quit = 1;
+          break;
+        }
+        LOG_PRINT_L2("block prev_id: " << b.prev_id << ENDL);
+
+        if (h % progress_interval == 0)
+        {
+          std::cout << refresh_string << "block " << h-1
+            << std::flush;
+        }
+
+        std::vector<transaction> txs;
+
+        int num_txs;
+        try
+        {
+          a >> num_txs;
+        }
+        catch (const std::exception& e)
+        {
+          std::cout << refresh_string;
+          LOG_PRINT_RED_L0("exception while de-archiving tx-num, height=" << h);
+          quit = 1;
+          break;
+        }
+        for(int tx_num = 1; tx_num <= num_txs; tx_num++)
+        {
+          try {
+                a >> tx;
+          }
+          catch (const std::exception& e)
+          {
+            LOG_PRINT_RED_L0("exception while de-archiving tx, height=" << h <<", tx_num=" << tx_num);
+            quit = 1;
+            break;
+          }
+          // if (tx_num == 1) {
+          //   std::cout << "coinbase transaction" << ENDL;
+          // }
+          // crypto::hash hsh = null_hash;
+          // size_t blob_size = 0;
+          // NOTE: all tx hashes except for coinbase tx are available in the block data
+          // get_transaction_hash(tx, hsh, blob_size);
+          // LOG_PRINT_L0("tx " << tx_num << "  " << hsh << " : " << ENDL);
+          // LOG_PRINT_L0(obj_to_json_str(tx) << ENDL);
+
+          // add blocks with verification.
+          // for Blockchain and blockchain_storage add_new_block().
+          if (opt_verify)
+          {
+            if (tx_num == 1) {
+              continue; // coinbase transaction. no need to insert to tx_pool.
+            }
+            // crypto::hash hsh = null_hash;
+            // size_t blob_size = 0;
+            // get_transaction_hash(tx, hsh, blob_size);
+            tx_verification_context tvc = AUTO_VAL_INIT(tvc);
+            bool r = true;
+            r = simple_core.m_pool.add_tx(tx, tvc, true);
+            if (!r)
+            {
+              LOG_PRINT_RED_L0("failed to add transaction to transaction pool, height=" << h <<", tx_num=" << tx_num);
+              quit = 1;
+              break;
+            }
+          }
+          else
+          {
+            // for add_block() method, without (much) processing.
+            // don't add coinbase transaction to txs.
+            //
+            // because add_block() calls
+            // add_transaction(blk_hash, blk.miner_tx) first, and
+            // then a for loop for the transactions in txs.
+            if (tx_num > 1)
+            {
+              txs.push_back(tx);
+            }
+          }
+        }
+
+        if (opt_verify)
+        {
+          block_verification_context bvc = boost::value_initialized<block_verification_context>();
+          simple_core.m_storage.add_new_block(b, bvc);
+
+          if (bvc.m_verifivation_failed)
+          {
+            LOG_PRINT_L0("Failed to add block to blockchain, verification failed, height = " << h);
+            LOG_PRINT_L0("skipping rest of import file");
+            // ok to commit previously batched data because it failed only in
+            // verification of potential new block with nothing added to batch
+            // yet
+            quit = 1;
+            break;
+          }
+          if (! bvc.m_added_to_main_chain)
+          {
+            LOG_PRINT_L0("Failed to add block to blockchain, height = " << h);
+            LOG_PRINT_L0("skipping rest of import file");
+            // make sure we don't commit partial block data
+            quit = 2;
+            break;
+          }
+        }
+        else
+        {
+          size_t block_size;
+          difficulty_type cumulative_difficulty;
+          uint64_t coins_generated;
+
+          a >> block_size;
+          a >> cumulative_difficulty;
+          a >> coins_generated;
+
+          std::cout << refresh_string;
+          LOG_PRINT_L2("block_size: " << block_size);
+          LOG_PRINT_L2("cumulative_difficulty: " << cumulative_difficulty);
+          LOG_PRINT_L2("coins_generated: " << coins_generated);
+
+          try
+          {
+            simple_core.add_block(b, block_size, cumulative_difficulty, coins_generated, txs);
+          }
+          catch (const std::exception& e)
+          {
+            std::cout << refresh_string;
+            LOG_PRINT_RED_L0("Error adding block to blockchain: " << e.what());
+            quit = 2; // make sure we don't commit partial block data
+            break;
+          }
+        }
+
+        if (use_batch)
+        {
+          if (h % db_batch_size == 0)
+          {
+            std::cout << refresh_string;
+            std::cout << ENDL << "[- batch commit at height " << h << " -]" << ENDL;
+            simple_core.batch_stop();
+            simple_core.batch_start();
+            std::cout << ENDL;
+#if !defined(BLOCKCHAIN_DB) || (BLOCKCHAIN_DB == DB_LMDB)
+            simple_core.m_storage.get_db().show_stats();
+#endif
+          }
+        }
+      }
+    }
+    catch (const std::exception& e)
+    {
+      std::cout << refresh_string;
+      LOG_PRINT_RED_L0("exception while reading from import file, height=" << h);
+      return 2;
+    }
+  } // while
+
+  import_file.close();
+
+  if (use_batch)
+  {
+    if (quit > 1)
+    {
+      // There was an error, so don't commit pending data.
+      // Destructor will abort write txn.
+    }
+    else
+    {
+      simple_core.batch_stop();
+    }
+#if !defined(BLOCKCHAIN_DB) || (BLOCKCHAIN_DB == DB_LMDB)
+    simple_core.m_storage.get_db().show_stats();
+#endif
+    if (h > 0)
+      LOG_PRINT_L0("Finished at height: " << h << "  block: " << h-1);
+  }
+  std::cout << ENDL;
+  return 0;
+}
+
+int main(int argc, char* argv[])
+{
+  std::string import_filename = BLOCKCHAIN_RAW;
+#if defined(BLOCKCHAIN_DB) && (BLOCKCHAIN_DB == DB_MEMORY)
+  std::string default_db_engine = "memory";
+#else
+  std::string default_db_engine = "lmdb";
+#endif
+
+  uint32_t log_level = LOG_LEVEL_0;
+  std::string dirname;
+  std::string db_arg_str;
+
+  boost::filesystem::path default_data_path {tools::get_default_data_dir()};
+  boost::filesystem::path default_testnet_data_path {default_data_path / "testnet"};
+
+  po::options_description desc_cmd_only("Command line options");
+  po::options_description desc_cmd_sett("Command line options and settings options");
+  const command_line::arg_descriptor<uint32_t> arg_log_level   =  {"log-level",  "", log_level};
+  const command_line::arg_descriptor<uint64_t> arg_batch_size  =  {"batch-size", "", db_batch_size};
+  const command_line::arg_descriptor<bool>     arg_testnet_on  = {
+    "testnet"
+      , "Run on testnet."
+      , false
+  };
+  const command_line::arg_descriptor<bool>     arg_count_blocks = {
+    "count-blocks"
+      , "Count blocks in import file and exit"
+      , false
+  };
+  const command_line::arg_descriptor<std::string> arg_database = {
+    "database", "available: memory, lmdb"
+      , default_db_engine
+  };
+  const command_line::arg_descriptor<bool> arg_verify =  {"verify",
+    "Verify blocks and transactions during import", true};
+  const command_line::arg_descriptor<bool> arg_batch  =  {"batch",
+    "Batch transactions for faster import", true};
+  const command_line::arg_descriptor<bool> arg_resume =  {"resume",
+    "Resume from current height if output database already exists", true};
+
+  command_line::add_arg(desc_cmd_sett, command_line::arg_data_dir, default_data_path.string());
+  command_line::add_arg(desc_cmd_sett, command_line::arg_testnet_data_dir, default_testnet_data_path.string());
+  command_line::add_arg(desc_cmd_sett, arg_log_level);
+  command_line::add_arg(desc_cmd_sett, arg_batch_size);
+  command_line::add_arg(desc_cmd_sett, arg_testnet_on);
+  command_line::add_arg(desc_cmd_sett, arg_database);
+
+  command_line::add_arg(desc_cmd_only, arg_count_blocks);
+  command_line::add_arg(desc_cmd_only, command_line::arg_help);
+
+  // call add_options() directly for these arguments since
+  // command_line helpers support only boolean switch, not boolean argument
+  desc_cmd_sett.add_options()
+    (arg_verify.name, make_semantic(arg_verify), arg_verify.description)
+    (arg_batch.name,  make_semantic(arg_batch),  arg_batch.description)
+    (arg_resume.name, make_semantic(arg_resume), arg_resume.description)
+    ;
+
+  po::options_description desc_options("Allowed options");
+  desc_options.add(desc_cmd_only).add(desc_cmd_sett);
+
+  po::variables_map vm;
+  bool r = command_line::handle_error_helper(desc_options, [&]()
+  {
+    po::store(po::parse_command_line(argc, argv, desc_options), vm);
+    po::notify(vm);
+    return true;
+  });
+  if (! r)
+    return 1;
+
+  log_level     = command_line::get_arg(vm, arg_log_level);
+  opt_verify    = command_line::get_arg(vm, arg_verify);
+  opt_batch     = command_line::get_arg(vm, arg_batch);
+  opt_resume    = command_line::get_arg(vm, arg_resume);
+  db_batch_size = command_line::get_arg(vm, arg_batch_size);
+
+  if (command_line::get_arg(vm, command_line::arg_help))
+  {
+    std::cout << CRYPTONOTE_NAME << " v" << MONERO_VERSION_FULL << ENDL << ENDL;
+    std::cout << desc_options << std::endl;
+    return 1;
+  }
+
+  if (! opt_batch && ! vm["batch-size"].defaulted())
+  {
+    std::cerr << "Error: batch-size set, but batch option not enabled" << ENDL;
+    exit(1);
+  }
+  if (! db_batch_size)
+  {
+    std::cerr << "Error: batch-size must be > 0" << ENDL;
+    exit(1);
+  }
+
+  std::vector<std::string> db_engines {"memory", "lmdb"};
+
+  opt_testnet = command_line::get_arg(vm, arg_testnet_on);
+  auto data_dir_arg = opt_testnet ? command_line::arg_testnet_data_dir : command_line::arg_data_dir;
+  dirname = command_line::get_arg(vm, data_dir_arg);
+  db_arg_str = command_line::get_arg(vm, arg_database);
+
+  log_space::get_set_log_detalisation_level(true, log_level);
+  log_space::log_singletone::add_logger(LOGGER_CONSOLE, NULL, NULL);
+  LOG_PRINT_L0("Starting...");
+  LOG_PRINT_L0("Setting log level = " << log_level);
+
+  boost::filesystem::path file_path {dirname};
+  std::string import_file_path;
+
+  import_file_path = (file_path / "export" / import_filename).string();
+
+  if (command_line::has_arg(vm, arg_count_blocks))
+  {
+    count_blocks(import_file_path);
+    exit(0);
+  }
+
+
+  std::string db_engine;
+  int mdb_flags = 0;
+  int res = 0;
+  res = parse_db_arguments(db_arg_str, db_engine, mdb_flags);
+  if (res)
+  {
+    std::cerr << "Error parsing database argument(s)" << ENDL;
+    exit(1);
+  }
+
+  if (std::find(db_engines.begin(), db_engines.end(), db_engine) == db_engines.end())
+  {
+    std::cerr << "Invalid database engine: " << db_engine << std::endl;
+    exit(1);
+  }
+
+  LOG_PRINT_L0("database: " << db_engine);
+  LOG_PRINT_L0("verify:  " << std::boolalpha << opt_verify << std::noboolalpha);
+  if (opt_batch)
+  {
+    LOG_PRINT_L0("batch:   " << std::boolalpha << opt_batch << std::noboolalpha
+        << "  batch size: " << db_batch_size);
+  }
+  else
+  {
+    LOG_PRINT_L0("batch:   " << std::boolalpha << opt_batch << std::noboolalpha);
+  }
+  LOG_PRINT_L0("resume:  " << std::boolalpha << opt_resume  << std::noboolalpha);
+  LOG_PRINT_L0("testnet: " << std::boolalpha << opt_testnet << std::noboolalpha);
+
+  std::cout << "import file path: " << import_file_path << ENDL;
+  std::cout << "database path:    " << file_path.string() << ENDL;
+
+  try
+  {
+
+  // fake_core needed for verification to work when enabled.
+  //
+  // NOTE: don't need fake_core method of doing things when we're going to call
+  // BlockchainDB add_block() directly and have available the 3 block
+  // properties to do so. Both ways work, but fake core isn't necessary in that
+  // circumstance.
+
+  // for multi_db_runtime:
+#if !defined(BLOCKCHAIN_DB)
+  if (db_engine == "lmdb")
+  {
+    fake_core_lmdb simple_core(dirname, opt_testnet, opt_batch, mdb_flags);
+    import_from_file(simple_core, import_file_path);
+  }
+  else if (db_engine == "memory")
+  {
+    fake_core_memory simple_core(dirname, opt_testnet);
+    import_from_file(simple_core, import_file_path);
+  }
+  else
+  {
+    std::cerr << "database engine unrecognized" << ENDL;
+    exit(1);
+  }
+
+  // for multi_db_compile:
+#else
+  if (db_engine != default_db_engine)
+  {
+    std::cerr << "Invalid database engine for compiled version: " << db_engine << std::endl;
+    exit(1);
+  }
+#if BLOCKCHAIN_DB == DB_LMDB
+  fake_core_lmdb simple_core(dirname, opt_testnet, opt_batch, mdb_flags);
+#else
+  fake_core_memory simple_core(dirname, opt_testnet);
+#endif
+
+  import_from_file(simple_core, import_file_path);
+#endif
+
+  }
+  catch (const DB_ERROR& e)
+  {
+    std::cout << std::string("Error loading blockchain db: ") + e.what() + " -- shutting down now" << ENDL;
+    exit(1);
+  }
+
+  // destructors called at exit:
+  //
+  // ensure db closed
+  //   - transactions properly checked and handled
+  //   - disk sync if needed
+  //
+  // fake_core object's destructor is called when it goes out of scope. For an
+  // LMDB fake_core, it calls Blockchain::deinit() on its object, which in turn
+  // calls delete on its BlockchainDB derived class' object, which closes its
+  // files.
+}

src/blockchain_converter/fake_core.h

@@ -0,0 +1,165 @@
+// Copyright (c) 2014-2015, The Monero Project
+//
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+//
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+//
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+//
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#pragma once
+
+#include <boost/filesystem.hpp>
+#include "cryptonote_core/blockchain.h" // BlockchainDB
+#include "cryptonote_core/blockchain_storage.h" // in-memory DB
+#include "blockchain_db/blockchain_db.h"
+#include "blockchain_db/lmdb/db_lmdb.h"
+
+using namespace cryptonote;
+
+
+#if !defined(BLOCKCHAIN_DB) || BLOCKCHAIN_DB == DB_LMDB
+
+struct fake_core_lmdb
+{
+  Blockchain m_storage;
+  tx_memory_pool m_pool;
+  bool support_batch;
+  bool support_add_block;
+
+  // for multi_db_runtime:
+#if !defined(BLOCKCHAIN_DB)
+  fake_core_lmdb(const boost::filesystem::path &path, const bool use_testnet=false, const bool do_batch=true, const int mdb_flags=0) : m_pool(&m_storage), m_storage(m_pool)
+  // for multi_db_compile:
+#else
+  fake_core_lmdb(const boost::filesystem::path &path, const bool use_testnet=false, const bool do_batch=true, const int mdb_flags=0) : m_pool(m_storage), m_storage(m_pool)
+#endif
+  {
+    m_pool.init(path.string());
+
+    BlockchainDB* db = new BlockchainLMDB();
+
+    boost::filesystem::path folder(path);
+
+    folder /= db->get_db_name();
+
+    LOG_PRINT_L0("Loading blockchain from folder " << folder.c_str() << " ...");
+
+    const std::string filename = folder.string();
+    try
+    {
+      db->open(filename, mdb_flags);
+    }
+    catch (const std::exception& e)
+    {
+      LOG_PRINT_L0("Error opening database: " << e.what());
+      throw;
+    }
+
+    m_storage.init(db, use_testnet);
+    if (do_batch)
+      m_storage.get_db().set_batch_transactions(do_batch);
+    support_batch = true;
+    support_add_block = true;
+  }
+  ~fake_core_lmdb()
+  {
+    m_storage.deinit();
+  }
+
+  uint64_t add_block(const block& blk
+                            , const size_t& block_size
+                            , const difficulty_type& cumulative_difficulty
+                            , const uint64_t& coins_generated
+                            , const std::vector<transaction>& txs
+                            )
+  {
+    return m_storage.get_db().add_block(blk, block_size, cumulative_difficulty, coins_generated, txs);
+  }
+
+  void batch_start()
+  {
+    m_storage.get_db().batch_start();
+  }
+
+  void batch_stop()
+  {
+    m_storage.get_db().batch_stop();
+  }
+
+};
+#endif
+
+#if !defined(BLOCKCHAIN_DB) || BLOCKCHAIN_DB == DB_MEMORY
+
+struct fake_core_memory
+{
+  blockchain_storage m_storage;
+  tx_memory_pool m_pool;
+  bool support_batch;
+  bool support_add_block;
+
+  // for multi_db_runtime:
+#if !defined(BLOCKCHAIN_DB)
+  fake_core_memory(const boost::filesystem::path &path, const bool use_testnet=false) : m_pool(&m_storage), m_storage(m_pool)
+#else
+  // for multi_db_compile:
+  fake_core_memory(const boost::filesystem::path &path, const bool use_testnet=false) : m_pool(m_storage), m_storage(&m_pool)
+#endif
+  {
+    m_pool.init(path.string());
+    m_storage.init(path.string(), use_testnet);
+    support_batch = false;
+    support_add_block = false;
+  }
+  ~fake_core_memory()
+  {
+    LOG_PRINT_L3("fake_core_memory() destructor called - want to see it ripple down");
+    m_storage.deinit();
+  }
+
+  uint64_t add_block(const block& blk
+                            , const size_t& block_size
+                            , const difficulty_type& cumulative_difficulty
+                            , const uint64_t& coins_generated
+                            , const std::vector<transaction>& txs
+                            )
+  {
+    // TODO:
+    // would need to refactor handle_block_to_main_chain() to have a direct add_block() method like Blockchain class
+    throw std::runtime_error("direct add_block() method not implemented for in-memory db");
+    return 2;
+  }
+
+  void batch_start()
+  {
+    LOG_PRINT_L0("WARNING: [batch_start] opt_batch set, but this database doesn't support/need transactions - ignoring");
+  }
+
+  void batch_stop()
+  {
+    LOG_PRINT_L0("WARNING: [batch_stop] opt_batch set, but this database doesn't support/need transactions - ignoring");
+  }
+
+};
+
+#endif

src/blockchain_converter/import.h

@@ -0,0 +1,37 @@
+// Copyright (c) 2014-2015, The Monero Project
+//
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+//
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+//
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+//
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#pragma once
+
+// TODO: bounds checking is done before writing to buffer, but buffer size
+// should be a sensible maximum
+#define BUFFER_SIZE 1000000
+#define NUM_BLOCKS_PER_CHUNK 1
+#define STR_LENGTH_OF_INT 9
+#define STR_FORMAT_OF_INT "%09d"
+#define BLOCKCHAIN_RAW "blockchain.raw"

src/blockchain_db/CMakeLists.txt

@@ -0,0 +1,76 @@
+# Copyright (c) 2014-2015, The Monero Project
+#
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are
+# permitted provided that the following conditions are met:
+#
+# 1. Redistributions of source code must retain the above copyright notice, this list of
+#    conditions and the following disclaimer.
+#
+# 2. Redistributions in binary form must reproduce the above copyright notice, this list
+#    of conditions and the following disclaimer in the documentation and/or other
+#    materials provided with the distribution.
+#
+# 3. Neither the name of the copyright holder nor the names of its contributors may be
+#    used to endorse or promote products derived from this software without specific
+#    prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+# THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+# THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+set(blockchain_db_sources
+  blockchain_db.cpp
+  lmdb/db_lmdb.cpp
+  )
+
+if (NOT STATIC)
+  set(blockchain_db_sources
+  ${blockchain_db_sources}
+  berkeleydb/db_bdb.cpp
+  )
+endif()
+
+
+set(blockchain_db_headers)
+
+set(blockchain_db_private_headers
+  blockchain_db.h
+  lmdb/db_lmdb.h
+  )
+
+if (NOT STATIC)
+  set(blockchain_db_private_headers
+  ${blockchain_db_private_headers}
+  berkeleydb/db_bdb.h
+  )
+endif()
+
+bitmonero_private_headers(blockchain_db
+  ${crypto_private_headers})
+bitmonero_add_library(blockchain_db
+  ${blockchain_db_sources}
+  ${blockchain_db_headers}
+  ${blockchain_db_private_headers})
+target_link_libraries(blockchain_db
+  LINK_PUBLIC
+    common
+    crypto
+    cryptonote_core
+    ${Boost_DATE_TIME_LIBRARY}
+    ${Boost_PROGRAM_OPTIONS_LIBRARY}
+    ${Boost_SERIALIZATION_LIBRARY}
+    ${LMDB_LIBRARY}
+    ${BDB_LIBRARY}
+  LINK_PRIVATE
+    ${Boost_FILESYSTEM_LIBRARY}
+    ${Boost_SYSTEM_LIBRARY}
+    ${Boost_THREAD_LIBRARY}
+    ${EXTRA_LIBRARIES})

src/blockchain_db/berkeleydb/db_bdb.h

@@ -0,0 +1,288 @@
+// Copyright (c) 2014, The Monero Project
+// All rights reserved.
+// 
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+// 
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+// 
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+// 
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+// 
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#include <db_cxx.h>
+
+#include "blockchain_db/blockchain_db.h"
+#include "cryptonote_protocol/blobdatatype.h" // for type blobdata
+
+namespace cryptonote
+{
+
+struct bdb_txn_safe
+{
+  bdb_txn_safe() : m_txn(NULL) { }
+  ~bdb_txn_safe()
+  {
+    LOG_PRINT_L3("bdb_txn_safe: destructor");
+
+    if (m_txn != NULL)
+      abort();
+  }
+
+  void commit(std::string message = "")
+  {
+    if (message.size() == 0)
+    {
+      message = "Failed to commit a transaction to the db";
+    }
+
+    if (m_txn->commit(0))
+    {
+      m_txn = NULL;
+      LOG_PRINT_L0(message);
+      throw DB_ERROR(message.c_str());
+    }
+    m_txn = NULL;
+  }
+
+  void abort()
+  {
+    LOG_PRINT_L3("bdb_txn_safe: abort()");
+    if(m_txn != NULL)
+    {
+      m_txn->abort();
+      m_txn = NULL;
+    }
+    else
+    {
+      LOG_PRINT_L0("WARNING: bdb_txn_safe: abort() called, but m_txn is NULL");
+    }
+  }
+
+  operator DbTxn*()
+  {
+    return m_txn;
+  }
+
+  operator DbTxn**()
+  {
+    return &m_txn;
+  }
+
+  DbTxn* m_txn;
+};
+
+class BlockchainBDB : public BlockchainDB
+{
+public:
+  BlockchainBDB(bool batch_transactions=false);
+  ~BlockchainBDB();
+
+  virtual void open(const std::string& filename, const int db_flags);
+
+  virtual void close();
+
+  virtual void sync();
+
+  virtual void reset();
+
+  virtual std::vector<std::string> get_filenames() const;
+
+  virtual std::string get_db_name() const;
+
+  virtual bool lock();
+
+  virtual void unlock();
+
+  virtual bool block_exists(const crypto::hash& h) const;
+
+  virtual block get_block(const crypto::hash& h) const;
+
+  virtual uint64_t get_block_height(const crypto::hash& h) const;
+
+  virtual block_header get_block_header(const crypto::hash& h) const;
+
+  virtual block get_block_from_height(const uint64_t& height) const;
+
+  virtual uint64_t get_block_timestamp(const uint64_t& height) const;
+
+  virtual uint64_t get_top_block_timestamp() const;
+
+  virtual size_t get_block_size(const uint64_t& height) const;
+
+  virtual difficulty_type get_block_cumulative_difficulty(const uint64_t& height) const;
+
+  virtual difficulty_type get_block_difficulty(const uint64_t& height) const;
+
+  virtual uint64_t get_block_already_generated_coins(const uint64_t& height) const;
+
+  virtual crypto::hash get_block_hash_from_height(const uint64_t& height) const;
+
+  virtual std::vector<block> get_blocks_range(const uint64_t& h1, const uint64_t& h2) const;
+
+  virtual std::vector<crypto::hash> get_hashes_range(const uint64_t& h1, const uint64_t& h2) const;
+
+  virtual crypto::hash top_block_hash() const;
+
+  virtual block get_top_block() const;
+
+  virtual uint64_t height() const;
+
+  virtual bool tx_exists(const crypto::hash& h) const;
+
+  virtual uint64_t get_tx_unlock_time(const crypto::hash& h) const;
+
+  virtual transaction get_tx(const crypto::hash& h) const;
+
+  virtual uint64_t get_tx_count() const;
+
+  virtual std::vector<transaction> get_tx_list(const std::vector<crypto::hash>& hlist) const;
+
+  virtual uint64_t get_tx_block_height(const crypto::hash& h) const;
+
+  virtual uint64_t get_random_output(const uint64_t& amount) const;
+
+  virtual uint64_t get_num_outputs(const uint64_t& amount) const;
+
+  virtual crypto::public_key get_output_key(const uint64_t& amount, const uint64_t& index) const;
+
+  virtual tx_out get_output(const crypto::hash& h, const uint64_t& index) const;
+
+  /**
+   * @brief get an output from its global index
+   *
+   * @param index global index of the output desired
+   *
+   * @return the output associated with the index.
+   * Will throw OUTPUT_DNE if not output has that global index.
+   * Will throw DB_ERROR if there is a non-specific LMDB error in fetching
+   */
+  tx_out get_output(const uint64_t& index) const;
+
+  virtual tx_out_index get_output_tx_and_index_from_global(const uint64_t& index) const;
+
+  virtual tx_out_index get_output_tx_and_index(const uint64_t& amount, const uint64_t& index) const;
+
+  virtual std::vector<uint64_t> get_tx_output_indices(const crypto::hash& h) const;
+  virtual std::vector<uint64_t> get_tx_amount_output_indices(const crypto::hash& h) const;
+
+  virtual bool has_key_image(const crypto::key_image& img) const;
+
+  virtual uint64_t add_block( const block& blk
+                            , const size_t& block_size
+                            , const difficulty_type& cumulative_difficulty
+                            , const uint64_t& coins_generated
+                            , const std::vector<transaction>& txs
+                            );
+
+  virtual void set_batch_transactions(bool batch_transactions);
+  virtual void batch_start();
+  virtual void batch_commit();
+  virtual void batch_stop();
+  virtual void batch_abort();
+
+  virtual void pop_block(block& blk, std::vector<transaction>& txs);
+
+private:
+  virtual void add_block( const block& blk
+                , const size_t& block_size
+                , const difficulty_type& cumulative_difficulty
+                , const uint64_t& coins_generated
+                , const crypto::hash& block_hash
+                );
+
+  virtual void remove_block();
+
+  virtual void add_transaction_data(const crypto::hash& blk_hash, const transaction& tx, const crypto::hash& tx_hash);
+
+  virtual void remove_transaction_data(const crypto::hash& tx_hash, const transaction& tx);
+
+  virtual void add_output(const crypto::hash& tx_hash, const tx_out& tx_output, const uint64_t& local_index);
+
+  virtual void remove_output(const tx_out& tx_output);
+
+  void remove_tx_outputs(const crypto::hash& tx_hash, const transaction& tx);
+
+  void remove_output(const uint64_t& out_index, const uint64_t amount);
+  void remove_amount_output_index(const uint64_t amount, const uint64_t global_output_index);
+
+  virtual void add_spent_key(const crypto::key_image& k_image);
+
+  virtual void remove_spent_key(const crypto::key_image& k_image);
+
+  /**
+   * @brief convert a tx output to a blob for storage
+   *
+   * @param output the output to convert
+   *
+   * @return the resultant blob
+   */
+  blobdata output_to_blob(const tx_out& output);
+
+  /**
+   * @brief convert a tx output blob to a tx output
+   *
+   * @param blob the blob to convert
+   *
+   * @return the resultant tx output
+   */
+  tx_out output_from_blob(const blobdata& blob) const;
+
+  /**
+   * @brief get the global index of the index-th output of the given amount
+   *
+   * @param amount the output amount
+   * @param index the index into the set of outputs of that amount
+   *
+   * @return the global index of the desired output
+   */
+  uint64_t get_output_global_index(const uint64_t& amount, const uint64_t& index) const;
+
+  void check_open() const;
+
+  DbEnv* m_env;
+
+  Db* m_blocks;
+  Db* m_block_heights;
+  Db* m_block_hashes;
+  Db* m_block_timestamps;
+  Db* m_block_sizes;
+  Db* m_block_diffs;
+  Db* m_block_coins;
+
+  Db* m_txs;
+  Db* m_tx_unlocks;
+  Db* m_tx_heights;
+  Db* m_tx_outputs;
+
+  Db* m_output_txs;
+  Db* m_output_indices;
+  Db* m_output_amounts;
+  Db* m_output_keys;
+
+  Db* m_spent_keys;
+
+  uint64_t m_height;
+  uint64_t m_num_outputs;
+  std::string m_folder;
+  bdb_txn_safe *m_write_txn;
+
+  bool m_batch_transactions; // support for batch transactions
+};
+
+}  // namespace cryptonote

src/blockchain_db/blockchain_db.cpp

@@ -0,0 +1,185 @@
+// Copyright (c) 2014, The Monero Project
+// 
+// All rights reserved.
+// 
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+// 
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+// 
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+// 
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+// 
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#include "blockchain_db.h"
+#include "cryptonote_core/cryptonote_format_utils.h"
+#include "profile_tools.h"
+
+using epee::string_tools::pod_to_hex;
+
+namespace cryptonote
+{
+
+void BlockchainDB::pop_block()
+{
+  block blk;
+  std::vector<transaction> txs;
+  pop_block(blk, txs);
+}
+
+void BlockchainDB::add_transaction(const crypto::hash& blk_hash, const transaction& tx, const crypto::hash* tx_hash_ptr)
+{
+  crypto::hash tx_hash;
+  if (!tx_hash_ptr)
+  {
+    // should only need to compute hash for miner transactions
+    tx_hash = get_transaction_hash(tx);
+    LOG_PRINT_L3("null tx_hash_ptr - needed to compute: " << tx_hash);
+  }
+  else
+  {
+    tx_hash = *tx_hash_ptr;
+  }
+
+  add_transaction_data(blk_hash, tx, tx_hash);
+
+  // iterate tx.vout using indices instead of C++11 foreach syntax because
+  // we need the index
+  if (tx.vout.size() != 0)  // it may be technically possible for a tx to have no outputs
+  {
+    for (uint64_t i = 0; i < tx.vout.size(); ++i)
+    {
+      add_output(tx_hash, tx.vout[i], i);
+    }
+
+    for (const txin_v& tx_input : tx.vin)
+    {
+      if (tx_input.type() == typeid(txin_to_key))
+      {
+        add_spent_key(boost::get<txin_to_key>(tx_input).k_image);
+      }
+    }
+  }
+}
+
+uint64_t BlockchainDB::add_block( const block& blk
+                                , const size_t& block_size
+                                , const difficulty_type& cumulative_difficulty
+                                , const uint64_t& coins_generated
+                                , const std::vector<transaction>& txs
+                                )
+{
+  TIME_MEASURE_START(time1);
+  crypto::hash blk_hash = get_block_hash(blk);
+  TIME_MEASURE_FINISH(time1);
+  time_blk_hash += time1;
+
+  // call out to subclass implementation to add the block & metadata
+  time1 = epee::misc_utils::get_tick_count();
+  add_block(blk, block_size, cumulative_difficulty, coins_generated, blk_hash);
+  TIME_MEASURE_FINISH(time1);
+  time_add_block1 += time1;
+
+  // call out to add the transactions
+
+  time1 = epee::misc_utils::get_tick_count();
+  add_transaction(blk_hash, blk.miner_tx);
+  int tx_i = 0;
+  crypto::hash tx_hash = null_hash;
+  for (const transaction& tx : txs)
+  {
+    tx_hash = blk.tx_hashes[tx_i];
+    add_transaction(blk_hash, tx, &tx_hash);
+    ++tx_i;
+  }
+  TIME_MEASURE_FINISH(time1);
+  time_add_transaction += time1;
+
+  ++num_calls;
+
+  return height();
+}
+
+void BlockchainDB::pop_block(block& blk, std::vector<transaction>& txs)
+{
+  blk = get_top_block();
+
+  remove_block();
+  
+  remove_transaction(get_transaction_hash(blk.miner_tx));
+  for (const auto& h : blk.tx_hashes)
+  {
+    txs.push_back(get_tx(h));
+    remove_transaction(h);
+  }
+}
+
+bool BlockchainDB::is_open()
+{
+  return m_open;
+}
+
+void BlockchainDB::remove_transaction(const crypto::hash& tx_hash)
+{
+  transaction tx = get_tx(tx_hash);
+
+  for (const txin_v& tx_input : tx.vin)
+  {
+    if (tx_input.type() == typeid(txin_to_key))
+    {
+      remove_spent_key(boost::get<txin_to_key>(tx_input).k_image);
+    }
+  }
+
+  // need tx as tx.vout has the tx outputs, and the output amounts are needed
+  remove_transaction_data(tx_hash, tx);
+}
+
+void BlockchainDB::reset_stats()
+{
+  num_calls = 0;
+  time_blk_hash = 0;
+  time_tx_exists = 0;
+  time_add_block1 = 0;
+  time_add_transaction = 0;
+  time_commit1 = 0;
+}
+
+void BlockchainDB::show_stats()
+{
+  LOG_PRINT_L1(ENDL
+    << "*********************************"
+    << ENDL
+    << "num_calls: " << num_calls
+    << ENDL
+    << "time_blk_hash: " << time_blk_hash << "ms"
+    << ENDL
+    << "time_tx_exists: " << time_tx_exists << "ms"
+    << ENDL
+    << "time_add_block1: " << time_add_block1 << "ms"
+    << ENDL
+    << "time_add_transaction: " << time_add_transaction << "ms"
+    << ENDL
+    << "time_commit1: " << time_commit1 << "ms"
+    << ENDL
+    << "*********************************"
+    << ENDL
+  );
+}
+
+}  // namespace cryptonote

src/blockchain_db/blockchain_db.h

@@ -0,0 +1,492 @@
+// Copyright (c) 2014, The Monero Project
+// 
+// All rights reserved.
+// 
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+// 
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+// 
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+// 
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+// 
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#ifndef BLOCKCHAIN_DB_H
+#define BLOCKCHAIN_DB_H
+
+#include <list>
+#include <string>
+#include <exception>
+#include "crypto/hash.h"
+#include "cryptonote_core/cryptonote_basic.h"
+#include "cryptonote_core/difficulty.h"
+
+/* DB Driver Interface
+ *
+ * The DB interface is a store for the canonical block chain.
+ * It serves as a persistent storage for the blockchain.
+ *
+ * For the sake of efficiency, the reference implementation will also
+ * store some blockchain data outside of the blocks, such as spent
+ * transfer key images, unspent transaction outputs, etc.
+ *
+ * Transactions are duplicated so that we don't have to fetch a whole block
+ * in order to fetch a transaction from that block.  If this is deemed
+ * unnecessary later, this can change.
+ *
+ * Spent key images are duplicated outside of the blocks so it is quick
+ * to verify an output hasn't already been spent
+ *
+ * Unspent transaction outputs are duplicated to quickly gather random
+ * outputs to use for mixins
+ *
+ * IMPORTANT:
+ * A concrete implementation of this interface should populate these
+ * duplicated members!  It is possible to have a partial implementation
+ * of this interface call to private members of the interface to be added
+ * later that will then populate as needed.
+ *
+ * General:
+ *   open()
+ *   is_open()
+ *   close()
+ *   sync()
+ *   reset()
+ *
+ *   Lock and unlock provided for reorg externally, and for block
+ *   additions internally, this way threaded reads are completely fine
+ *   unless the blockchain is changing.
+ *   bool lock()
+ *   unlock()
+ *
+ * vector<str>   get_filenames()
+ *
+ * Blocks:
+ *   bool        block_exists(hash)
+ *   height      add_block(block, block_size, cumulative_difficulty, coins_generated, transactions)
+ *   block       get_block(hash)
+ *   height      get_block_height(hash)
+ *   header      get_block_header(hash)
+ *   block       get_block_from_height(height)
+ *   size_t      get_block_size(height)
+ *   difficulty  get_block_cumulative_difficulty(height)
+ *   uint64_t    get_block_already_generated_coins(height)
+ *   uint64_t    get_block_timestamp(height) 
+ *   uint64_t    get_top_block_timestamp()
+ *   hash        get_block_hash_from_height(height)
+ *   blocks      get_blocks_range(height1, height2)
+ *   hashes      get_hashes_range(height1, height2)
+ *   hash        top_block_hash()
+ *   block       get_top_block()
+ *   height      height()
+ *   void        pop_block(block&, tx_list&)
+ *
+ * Transactions:
+ *   bool        tx_exists(hash)
+ *   uint64_t    get_tx_unlock_time(hash)
+ *   tx          get_tx(hash)
+ *   uint64_t    get_tx_count()
+ *   tx_list     get_tx_list(hash_list)
+ *   height      get_tx_block_height(hash)
+ *
+ * Outputs:
+ *   index       get_random_output(amount)
+ *   uint64_t    get_num_outputs(amount)
+ *   pub_key     get_output_key(amount, index)
+ *   tx_out      get_output(tx_hash, index)
+ *   hash,index  get_output_tx_and_index_from_global(index)
+ *   hash,index  get_output_tx_and_index(amount, index)
+ *   vec<uint64> get_tx_output_indices(tx_hash)
+ *
+ *
+ * Spent Output Key Images:
+ *   bool        has_key_image(key_image)
+ *
+ * Exceptions:
+ *   DB_ERROR -- generic
+ *   DB_OPEN_FAILURE
+ *   DB_CREATE_FAILURE
+ *   DB_SYNC_FAILURE
+ *   BLOCK_DNE
+ *   BLOCK_PARENT_DNE
+ *   BLOCK_EXISTS
+ *   BLOCK_INVALID -- considering making this multiple errors
+ *   TX_DNE
+ *   TX_EXISTS
+ *   OUTPUT_DNE
+ *   OUTPUT_EXISTS
+ *   KEY_IMAGE_EXISTS
+ */
+
+namespace cryptonote
+{
+
+// typedef for convenience
+typedef std::pair<crypto::hash, uint64_t> tx_out_index;
+
+/***********************************
+ * Exception Definitions
+ ***********************************/
+class DB_EXCEPTION : public std::exception
+{
+  private:
+    std::string m;
+
+  protected:
+    DB_EXCEPTION(const char *s) : m(s) { }
+
+  public:
+    virtual ~DB_EXCEPTION() { }
+
+    const char* what() const throw()
+    {
+      return m.c_str();
+    }
+};
+
+class DB_ERROR : public DB_EXCEPTION
+{
+  public:
+    DB_ERROR() : DB_EXCEPTION("Generic DB Error") { }
+    DB_ERROR(const char* s) : DB_EXCEPTION(s) { }
+};
+
+class DB_OPEN_FAILURE : public DB_EXCEPTION
+{
+  public:
+    DB_OPEN_FAILURE() : DB_EXCEPTION("Failed to open the db") { }
+    DB_OPEN_FAILURE(const char* s) : DB_EXCEPTION(s) { }
+};
+
+class DB_CREATE_FAILURE : public DB_EXCEPTION
+{
+  public:
+    DB_CREATE_FAILURE() : DB_EXCEPTION("Failed to create the db") { }
+    DB_CREATE_FAILURE(const char* s) : DB_EXCEPTION(s) { }
+};
+
+class DB_SYNC_FAILURE : public DB_EXCEPTION
+{
+  public:
+    DB_SYNC_FAILURE() : DB_EXCEPTION("Failed to sync the db") { }
+    DB_SYNC_FAILURE(const char* s) : DB_EXCEPTION(s) { }
+};
+
+class BLOCK_DNE : public DB_EXCEPTION
+{
+  public:
+    BLOCK_DNE() : DB_EXCEPTION("The block requested does not exist") { }
+    BLOCK_DNE(const char* s) : DB_EXCEPTION(s) { }
+};
+
+class BLOCK_PARENT_DNE : public DB_EXCEPTION
+{
+  public:
+    BLOCK_PARENT_DNE() : DB_EXCEPTION("The parent of the block does not exist") { }
+    BLOCK_PARENT_DNE(const char* s) : DB_EXCEPTION(s) { }
+};
+
+class BLOCK_EXISTS : public DB_EXCEPTION
+{
+  public:
+    BLOCK_EXISTS() : DB_EXCEPTION("The block to be added already exists!") { }
+    BLOCK_EXISTS(const char* s) : DB_EXCEPTION(s) { }
+};
+
+class BLOCK_INVALID : public DB_EXCEPTION
+{
+  public:
+    BLOCK_INVALID() : DB_EXCEPTION("The block to be added did not pass validation!") { }
+    BLOCK_INVALID(const char* s) : DB_EXCEPTION(s) { }
+};
+
+class TX_DNE : public DB_EXCEPTION
+{
+  public:
+    TX_DNE() : DB_EXCEPTION("The transaction requested does not exist") { }
+    TX_DNE(const char* s) : DB_EXCEPTION(s) { }
+};
+
+class TX_EXISTS : public DB_EXCEPTION
+{
+  public:
+    TX_EXISTS() : DB_EXCEPTION("The transaction to be added already exists!") { }
+    TX_EXISTS(const char* s) : DB_EXCEPTION(s) { }
+};
+
+class OUTPUT_DNE : public DB_EXCEPTION
+{
+  public:
+    OUTPUT_DNE() : DB_EXCEPTION("The output requested does not exist!") { }
+    OUTPUT_DNE(const char* s) : DB_EXCEPTION(s) { }
+};
+
+class OUTPUT_EXISTS : public DB_EXCEPTION
+{
+  public:
+    OUTPUT_EXISTS() : DB_EXCEPTION("The output to be added already exists!") { }
+    OUTPUT_EXISTS(const char* s) : DB_EXCEPTION(s) { }
+};
+
+class KEY_IMAGE_EXISTS : public DB_EXCEPTION
+{
+  public:
+    KEY_IMAGE_EXISTS() : DB_EXCEPTION("The spent key image to be added already exists!") { }
+    KEY_IMAGE_EXISTS(const char* s) : DB_EXCEPTION(s) { }
+};
+
+/***********************************
+ * End of Exception Definitions
+ ***********************************/
+
+
+class BlockchainDB
+{
+private:
+  /*********************************************************************
+   * private virtual members
+   *********************************************************************/
+
+  // tells the subclass to add the block and metadata to storage
+  virtual void add_block( const block& blk
+                , const size_t& block_size
+                , const difficulty_type& cumulative_difficulty
+                , const uint64_t& coins_generated
+                , const crypto::hash& blk_hash
+                ) = 0;
+
+  // tells the subclass to remove data about the top block
+  virtual void remove_block() = 0;
+
+  // tells the subclass to store the transaction and its metadata
+  virtual void add_transaction_data(const crypto::hash& blk_hash, const transaction& tx, const crypto::hash& tx_hash) = 0;
+
+  // tells the subclass to remove data about a transaction
+  virtual void remove_transaction_data(const crypto::hash& tx_hash, const transaction& tx) = 0;
+
+  // tells the subclass to store an output
+  virtual void add_output(const crypto::hash& tx_hash, const tx_out& tx_output, const uint64_t& local_index) = 0;
+
+  // tells the subclass to remove an output
+  virtual void remove_output(const tx_out& tx_output) = 0;
+
+  // tells the subclass to store a spent key
+  virtual void add_spent_key(const crypto::key_image& k_image) = 0;
+
+  // tells the subclass to remove a spent key
+  virtual void remove_spent_key(const crypto::key_image& k_image) = 0;
+
+
+  /*********************************************************************
+   * private concrete members
+   *********************************************************************/ 
+  // private version of pop_block, for undoing if an add_block goes tits up
+  void pop_block();
+
+  // helper function for add_transactions, to add each individual tx
+  void add_transaction(const crypto::hash& blk_hash, const transaction& tx, const crypto::hash* tx_hash_ptr = NULL);
+
+  // helper function to remove transaction from blockchain
+  void remove_transaction(const crypto::hash& tx_hash);
+
+  uint64_t num_calls = 0;
+  uint64_t time_blk_hash = 0;
+  uint64_t time_add_block1 = 0;
+  uint64_t time_add_transaction = 0;
+
+
+protected:
+
+  mutable uint64_t time_tx_exists = 0;
+  uint64_t time_commit1 = 0;
+
+
+public:
+
+  // virtual dtor
+  virtual ~BlockchainDB() { };
+
+  // reset profiling stats
+  void reset_stats();
+
+  // show profiling stats
+  void show_stats();
+
+  // open the db at location <filename>, or create it if there isn't one.
+  virtual void open(const std::string& filename, const int db_flags = 0) = 0;
+
+  // returns true of the db is open/ready, else false
+  bool is_open();
+
+  // close and sync the db
+  virtual void close() = 0;
+
+  // sync the db
+  virtual void sync() = 0;
+
+  // reset the db -- USE WITH CARE
+  virtual void reset() = 0;
+
+  // get all files used by this db (if any)
+  virtual std::vector<std::string> get_filenames() const = 0;
+
+  // return the name of the folder the db's file(s) should reside in
+  virtual std::string get_db_name() const = 0;
+
+
+  // FIXME: these are just for functionality mocking, need to implement
+  // RAII-friendly and multi-read one-write friendly locking mechanism
+  //
+  // acquire db lock
+  virtual bool lock() = 0;
+
+  // release db lock
+  virtual void unlock() = 0;
+
+  virtual void batch_start() = 0;
+  virtual void batch_stop() = 0;
+  virtual void set_batch_transactions(bool) = 0;
+
+  // adds a block with the given metadata to the top of the blockchain, returns the new height
+  // NOTE: subclass implementations of this (or the functions it calls) need
+  // to handle undoing any partially-added blocks in the event of a failure.
+  virtual uint64_t add_block( const block& blk
+                            , const size_t& block_size
+                            , const difficulty_type& cumulative_difficulty
+                            , const uint64_t& coins_generated
+                            , const std::vector<transaction>& txs
+                            );
+
+  // return true if a block with hash <h> exists in the blockchain
+  virtual bool block_exists(const crypto::hash& h) const = 0;
+
+  // return block with hash <h>
+  virtual block get_block(const crypto::hash& h) const = 0;
+
+  // return the height of the block with hash <h> on the blockchain,
+  // throw if it doesn't exist
+  virtual uint64_t get_block_height(const crypto::hash& h) const = 0;
+
+  // return header for block with hash <h>
+  virtual block_header get_block_header(const crypto::hash& h) const = 0;
+
+  // return block at height <height>
+  virtual block get_block_from_height(const uint64_t& height) const = 0;
+
+  // return timestamp of block at height <height>
+  virtual uint64_t get_block_timestamp(const uint64_t& height) const = 0;
+
+  // return timestamp of most recent block
+  virtual uint64_t get_top_block_timestamp() const = 0;
+
+  // return block size of block at height <height>
+  virtual size_t get_block_size(const uint64_t& height) const = 0;
+
+  // return cumulative difficulty up to and including block at height <height>
+  virtual difficulty_type get_block_cumulative_difficulty(const uint64_t& height) const = 0;
+
+  // return difficulty of block at height <height>
+  virtual difficulty_type get_block_difficulty(const uint64_t& height) const = 0;
+
+  // return number of coins generated up to and including block at height <height>
+  virtual uint64_t get_block_already_generated_coins(const uint64_t& height) const = 0;
+
+  // return hash of block at height <height>
+  virtual crypto::hash get_block_hash_from_height(const uint64_t& height) const = 0;
+
+  // return vector of blocks in range <h1,h2> of height (inclusively)
+  virtual std::vector<block> get_blocks_range(const uint64_t& h1, const uint64_t& h2) const = 0;
+
+  // return vector of block hashes in range <h1, h2> of height (inclusively)
+  virtual std::vector<crypto::hash> get_hashes_range(const uint64_t& h1, const uint64_t& h2) const = 0;
+
+  // return the hash of the top block on the chain
+  virtual crypto::hash top_block_hash() const = 0;
+
+  // return the block at the top of the blockchain
+  virtual block get_top_block() const = 0;
+
+  // return the height of the chain
+  virtual uint64_t height() const = 0;
+
+  // pops the top block off the blockchain.
+  // Returns by reference the popped block and its associated transactions
+  //
+  // IMPORTANT:
+  // When a block is popped, the transactions associated with it need to be
+  // removed, as well as outputs and spent key images associated with
+  // those transactions.
+  virtual void pop_block(block& blk, std::vector<transaction>& txs);
+
+
+  // return true if a transaction with hash <h> exists
+  virtual bool tx_exists(const crypto::hash& h) const = 0;
+
+  // return unlock time of tx with hash <h>
+  virtual uint64_t get_tx_unlock_time(const crypto::hash& h) const = 0;
+
+  // return tx with hash <h>
+  // throw if no such tx exists
+  virtual transaction get_tx(const crypto::hash& h) const = 0;
+
+  // returns the total number of transactions in all blocks
+  virtual uint64_t get_tx_count() const = 0;
+
+  // return list of tx with hashes <hlist>.
+  // TODO: decide if a missing hash means return empty list
+  // or just skip that hash
+  virtual std::vector<transaction> get_tx_list(const std::vector<crypto::hash>& hlist) const = 0;
+
+  // returns height of block that contains transaction with hash <h>
+  virtual uint64_t get_tx_block_height(const crypto::hash& h) const = 0;
+
+  // return global output index of a random output of amount <amount>
+  virtual uint64_t get_random_output(const uint64_t& amount) const = 0;
+
+  // returns the total number of outputs of amount <amount>
+  virtual uint64_t get_num_outputs(const uint64_t& amount) const = 0;
+
+  // return public key for output with global output amount <amount> and index <index>
+  virtual crypto::public_key get_output_key(const uint64_t& amount, const uint64_t& index) const = 0;
+
+  // returns the output indexed by <index> in the transaction with hash <h>
+  virtual tx_out get_output(const crypto::hash& h, const uint64_t& index) const = 0;
+
+  // returns the tx hash associated with an output, referenced by global output index
+  virtual tx_out_index get_output_tx_and_index_from_global(const uint64_t& index) const = 0;
+
+  // returns the transaction-local reference for the output with <amount> at <index>
+  // return type is pair of tx hash and index
+  virtual tx_out_index get_output_tx_and_index(const uint64_t& amount, const uint64_t& index) const = 0;
+
+  // return a vector of indices corresponding to the global output index for
+  // each output in the transaction with hash <h>
+  virtual std::vector<uint64_t> get_tx_output_indices(const crypto::hash& h) const = 0;
+  // return a vector of indices corresponding to the amount output index for
+  // each output in the transaction with hash <h>
+  virtual std::vector<uint64_t> get_tx_amount_output_indices(const crypto::hash& h) const = 0;
+
+  // returns true if key image <img> is present in spent key images storage
+  virtual bool has_key_image(const crypto::key_image& img) const = 0;
+
+  bool m_open;
+};  // class BlockchainDB
+
+
+}  // namespace cryptonote
+
+#endif  // BLOCKCHAIN_DB_H

src/blockchain_db/db_types.h

@@ -0,0 +1,40 @@
+// Copyright (c) 2015, The Monero Project
+//
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+//
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+//
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+//
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+//
+// Parts of this file are originally copyright (c) 2012-2013 The Cryptonote developers
+#pragma once
+
+namespace cryptonote
+{
+
+  const std::unordered_set<std::string> blockchain_db_types = 
+  { "lmdb"
+  , "berkeley"
+  };
+
+} // namespace cryptonote

src/blockchain_db/lmdb/db_lmdb.h

@@ -0,0 +1,314 @@
+// Copyright (c) 2014, The Monero Project
+// All rights reserved.
+// 
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+// 
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+// 
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+// 
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+// 
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#pragma once
+
+#include "blockchain_db/blockchain_db.h"
+#include "cryptonote_protocol/blobdatatype.h" // for type blobdata
+
+#include <lmdb.h>
+
+namespace cryptonote
+{
+
+struct mdb_txn_safe
+{
+  mdb_txn_safe() : m_txn(NULL) { }
+  ~mdb_txn_safe()
+  {
+    LOG_PRINT_L3("mdb_txn_safe: destructor");
+    if (m_txn != NULL)
+    {
+      if (m_batch_txn) // this is a batch txn and should have been handled before this point for safety
+      {
+        LOG_PRINT_L0("WARNING: mdb_txn_safe: m_txn is a batch txn and it's not NULL in destructor - calling mdb_txn_abort()");
+      }
+      else
+      {
+        // Example of when this occurs: a lookup fails, so a read-only txn is
+        // aborted through this destructor. However, successful read-only txns
+        // ideally should have been committed when done and not end up here.
+        //
+        // NOTE: not sure if this is ever reached for a non-batch write
+        // transaction, but it's probably not ideal if it did.
+        LOG_PRINT_L3("mdb_txn_safe: m_txn not NULL in destructor - calling mdb_txn_abort()");
+      }
+      mdb_txn_abort(m_txn);
+    }
+  }
+
+  void commit(std::string message = "")
+  {
+    if (message.size() == 0)
+    {
+      message = "Failed to commit a transaction to the db";
+    }
+
+    if (mdb_txn_commit(m_txn))
+    {
+      m_txn = NULL;
+      LOG_PRINT_L0(message);
+      throw DB_ERROR(message.c_str());
+    }
+    m_txn = NULL;
+  }
+
+  // This should only be needed for batch transaction which must be ensured to
+  // be aborted before mdb_env_close, not after. So we can't rely on
+  // BlockchainLMDB destructor to call mdb_txn_safe destructor, as that's too late
+  // to properly abort, since mdb_env_close would have been called earlier.
+  void abort()
+  {
+    LOG_PRINT_L3("mdb_txn_safe: abort()");
+    if(m_txn != NULL)
+    {
+      mdb_txn_abort(m_txn);
+      m_txn = NULL;
+    }
+    else
+    {
+      LOG_PRINT_L0("WARNING: mdb_txn_safe: abort() called, but m_txn is NULL");
+    }
+  }
+
+  operator MDB_txn*()
+  {
+    return m_txn;
+  }
+
+  operator MDB_txn**()
+  {
+    return &m_txn;
+  }
+
+  MDB_txn* m_txn;
+  bool m_batch_txn = false;
+};
+
+
+class BlockchainLMDB : public BlockchainDB
+{
+public:
+  BlockchainLMDB(bool batch_transactions=false);
+  ~BlockchainLMDB();
+
+  virtual void open(const std::string& filename, const int mdb_flags=0);
+
+  virtual void close();
+
+  virtual void sync();
+
+  virtual void reset();
+
+  virtual std::vector<std::string> get_filenames() const;
+
+  virtual std::string get_db_name() const;
+
+  virtual bool lock();
+
+  virtual void unlock();
+
+  virtual bool block_exists(const crypto::hash& h) const;
+
+  virtual block get_block(const crypto::hash& h) const;
+
+  virtual uint64_t get_block_height(const crypto::hash& h) const;
+
+  virtual block_header get_block_header(const crypto::hash& h) const;
+
+  virtual block get_block_from_height(const uint64_t& height) const;
+
+  virtual uint64_t get_block_timestamp(const uint64_t& height) const;
+
+  virtual uint64_t get_top_block_timestamp() const;
+
+  virtual size_t get_block_size(const uint64_t& height) const;
+
+  virtual difficulty_type get_block_cumulative_difficulty(const uint64_t& height) const;
+
+  virtual difficulty_type get_block_difficulty(const uint64_t& height) const;
+
+  virtual uint64_t get_block_already_generated_coins(const uint64_t& height) const;
+
+  virtual crypto::hash get_block_hash_from_height(const uint64_t& height) const;
+
+  virtual std::vector<block> get_blocks_range(const uint64_t& h1, const uint64_t& h2) const;
+
+  virtual std::vector<crypto::hash> get_hashes_range(const uint64_t& h1, const uint64_t& h2) const;
+
+  virtual crypto::hash top_block_hash() const;
+
+  virtual block get_top_block() const;
+
+  virtual uint64_t height() const;
+
+  virtual bool tx_exists(const crypto::hash& h) const;
+
+  virtual uint64_t get_tx_unlock_time(const crypto::hash& h) const;
+
+  virtual transaction get_tx(const crypto::hash& h) const;
+
+  virtual uint64_t get_tx_count() const;
+
+  virtual std::vector<transaction> get_tx_list(const std::vector<crypto::hash>& hlist) const;
+
+  virtual uint64_t get_tx_block_height(const crypto::hash& h) const;
+
+  virtual uint64_t get_random_output(const uint64_t& amount) const;
+
+  virtual uint64_t get_num_outputs(const uint64_t& amount) const;
+
+  virtual crypto::public_key get_output_key(const uint64_t& amount, const uint64_t& index) const;
+
+  virtual tx_out get_output(const crypto::hash& h, const uint64_t& index) const;
+
+  /**
+   * @brief get an output from its global index
+   *
+   * @param index global index of the output desired
+   *
+   * @return the output associated with the index.
+   * Will throw OUTPUT_DNE if not output has that global index.
+   * Will throw DB_ERROR if there is a non-specific LMDB error in fetching
+   */
+  tx_out get_output(const uint64_t& index) const;
+
+  virtual tx_out_index get_output_tx_and_index_from_global(const uint64_t& index) const;
+
+  virtual tx_out_index get_output_tx_and_index(const uint64_t& amount, const uint64_t& index) const;
+
+  virtual std::vector<uint64_t> get_tx_output_indices(const crypto::hash& h) const;
+  virtual std::vector<uint64_t> get_tx_amount_output_indices(const crypto::hash& h) const;
+
+  virtual bool has_key_image(const crypto::key_image& img) const;
+
+  virtual uint64_t add_block( const block& blk
+                            , const size_t& block_size
+                            , const difficulty_type& cumulative_difficulty
+                            , const uint64_t& coins_generated
+                            , const std::vector<transaction>& txs
+                            );
+
+  virtual void set_batch_transactions(bool batch_transactions);
+  virtual void batch_start();
+  virtual void batch_commit();
+  virtual void batch_stop();
+  virtual void batch_abort();
+
+  virtual void pop_block(block& blk, std::vector<transaction>& txs);
+
+private:
+  virtual void add_block( const block& blk
+                , const size_t& block_size
+                , const difficulty_type& cumulative_difficulty
+                , const uint64_t& coins_generated
+                , const crypto::hash& block_hash
+                );
+
+  virtual void remove_block();
+
+  virtual void add_transaction_data(const crypto::hash& blk_hash, const transaction& tx, const crypto::hash& tx_hash);
+
+  virtual void remove_transaction_data(const crypto::hash& tx_hash, const transaction& tx);
+
+  virtual void add_output(const crypto::hash& tx_hash, const tx_out& tx_output, const uint64_t& local_index);
+
+  virtual void remove_output(const tx_out& tx_output);
+
+  void remove_tx_outputs(const crypto::hash& tx_hash, const transaction& tx);
+
+  void remove_output(const uint64_t& out_index, const uint64_t amount);
+  void remove_amount_output_index(const uint64_t amount, const uint64_t global_output_index);
+
+  virtual void add_spent_key(const crypto::key_image& k_image);
+
+  virtual void remove_spent_key(const crypto::key_image& k_image);
+
+  /**
+   * @brief convert a tx output to a blob for storage
+   *
+   * @param output the output to convert
+   *
+   * @return the resultant blob
+   */
+  blobdata output_to_blob(const tx_out& output);
+
+  /**
+   * @brief convert a tx output blob to a tx output
+   *
+   * @param blob the blob to convert
+   *
+   * @return the resultant tx output
+   */
+  tx_out output_from_blob(const blobdata& blob) const;
+
+  /**
+   * @brief get the global index of the index-th output of the given amount
+   *
+   * @param amount the output amount
+   * @param index the index into the set of outputs of that amount
+   *
+   * @return the global index of the desired output
+   */
+  uint64_t get_output_global_index(const uint64_t& amount, const uint64_t& index) const;
+
+  void check_open() const;
+
+  MDB_env* m_env;
+
+  MDB_dbi m_blocks;
+  MDB_dbi m_block_heights;
+  MDB_dbi m_block_hashes;
+  MDB_dbi m_block_timestamps;
+  MDB_dbi m_block_sizes;
+  MDB_dbi m_block_diffs;
+  MDB_dbi m_block_coins;
+
+  MDB_dbi m_txs;
+  MDB_dbi m_tx_unlocks;
+  MDB_dbi m_tx_heights;
+  MDB_dbi m_tx_outputs;
+
+  MDB_dbi m_output_txs;
+  MDB_dbi m_output_indices;
+  MDB_dbi m_output_gindices;
+  MDB_dbi m_output_amounts;
+  MDB_dbi m_output_keys;
+  MDB_dbi m_outputs;
+
+  MDB_dbi m_spent_keys;
+
+  uint64_t m_height;
+  uint64_t m_num_outputs;
+  std::string m_folder;
+  mdb_txn_safe* m_write_txn; // may point to either a short-lived txn or a batch txn
+  mdb_txn_safe m_write_batch_txn; // persist batch txn outside of BlockchainLMDB
+
+  bool m_batch_transactions; // support for batch transactions
+  bool m_batch_active; // whether batch transaction is in progress
+};
+
+}  // namespace cryptonote

src/cryptonote_core/CMakeLists.txt

@@ -29,6 +29,7 @@
 set(cryptonote_core_sources
   account.cpp
   blockchain_storage.cpp
+  blockchain.cpp
   checkpoints.cpp
   checkpoints_create.cpp
   cryptonote_basic_impl.cpp
@@ -45,6 +46,7 @@ set(cryptonote_core_private_headers
   account_boost_serialization.h
   blockchain_storage.h
   blockchain_storage_boost_serialization.h
+  blockchain.h
   checkpoints.h
   checkpoints_create.h
   connection_context.h
@@ -71,6 +73,7 @@ target_link_libraries(cryptonote_core
     common
     crypto
     otshell_utils
+    blockchain_db
     ${Boost_DATE_TIME_LIBRARY}
     ${Boost_PROGRAM_OPTIONS_LIBRARY}
     ${Boost_SERIALIZATION_LIBRARY}

src/cryptonote_core/blockchain.h

@@ -0,0 +1,229 @@
+// Copyright (c) 2014, The Monero Project
+// 
+// All rights reserved.
+// 
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+// 
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+// 
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+// 
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+// 
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+// 
+// Parts of this file are originally copyright (c) 2012-2013 The Cryptonote developers
+
+#pragma once
+#include <boost/serialization/serialization.hpp>
+#include <boost/serialization/version.hpp>
+#include <boost/serialization/list.hpp>
+#include <boost/multi_index_container.hpp>
+#include <boost/multi_index/global_fun.hpp>
+#include <boost/multi_index/hashed_index.hpp>
+#include <boost/multi_index/member.hpp>
+#include <boost/foreach.hpp>
+#include <atomic>
+
+#include "syncobj.h"
+#include "string_tools.h"
+#include "cryptonote_basic.h"
+#include "common/util.h"
+#include "cryptonote_protocol/cryptonote_protocol_defs.h"
+#include "rpc/core_rpc_server_commands_defs.h"
+#include "difficulty.h"
+#include "cryptonote_core/cryptonote_format_utils.h"
+#include "verification_context.h"
+#include "crypto/hash.h"
+#include "checkpoints.h"
+#include "blockchain_db/blockchain_db.h"
+
+namespace cryptonote
+{
+  class tx_memory_pool;
+
+  /************************************************************************/
+  /*                                                                      */
+  /************************************************************************/
+  class Blockchain
+  {
+  public:
+    struct transaction_chain_entry
+    {
+      transaction tx;
+      uint64_t m_keeper_block_height;
+      size_t m_blob_size;
+      std::vector<uint64_t> m_global_output_indexes;
+    };
+
+    struct block_extended_info
+    {
+      block   bl;
+      uint64_t height;
+      size_t block_cumulative_size;
+      difficulty_type cumulative_difficulty;
+      uint64_t already_generated_coins;
+    };
+
+    Blockchain(tx_memory_pool& tx_pool);
+
+    bool init(BlockchainDB* db, const bool testnet = false);
+    bool deinit();
+
+    void set_checkpoints(checkpoints&& chk_pts) { m_checkpoints = chk_pts; }
+
+    //bool push_new_block();
+    bool get_blocks(uint64_t start_offset, size_t count, std::list<block>& blocks, std::list<transaction>& txs) const;
+    bool get_blocks(uint64_t start_offset, size_t count, std::list<block>& blocks) const;
+    bool get_alternative_blocks(std::list<block>& blocks) const;
+    size_t get_alternative_blocks_count() const;
+    crypto::hash get_block_id_by_height(uint64_t height) const;
+    bool get_block_by_hash(const crypto::hash &h, block &blk) const;
+    void get_all_known_block_ids(std::list<crypto::hash> &main, std::list<crypto::hash> &alt, std::list<crypto::hash> &invalid) const;
+
+    template<class archive_t>
+    void serialize(archive_t & ar, const unsigned int version);
+
+    bool have_tx(const crypto::hash &id) const;
+    bool have_tx_keyimges_as_spent(const transaction &tx) const;
+    bool have_tx_keyimg_as_spent(const crypto::key_image &key_im) const;
+
+    template<class visitor_t>
+    bool scan_outputkeys_for_indexes(const txin_to_key& tx_in_to_key, visitor_t& vis, uint64_t* pmax_related_block_height = NULL) const;
+
+    uint64_t get_current_blockchain_height() const;
+    crypto::hash get_tail_id() const;
+    crypto::hash get_tail_id(uint64_t& height) const;
+    difficulty_type get_difficulty_for_next_block() const;
+    bool add_new_block(const block& bl_, block_verification_context& bvc);
+    bool reset_and_set_genesis_block(const block& b);
+    bool create_block_template(block& b, const account_public_address& miner_address, difficulty_type& di, uint64_t& height, const blobdata& ex_nonce) const;
+    bool have_block(const crypto::hash& id) const;
+    size_t get_total_transactions() const;
+    bool get_short_chain_history(std::list<crypto::hash>& ids) const;
+    bool find_blockchain_supplement(const std::list<crypto::hash>& qblock_ids, NOTIFY_RESPONSE_CHAIN_ENTRY::request& resp) const;
+    bool find_blockchain_supplement(const std::list<crypto::hash>& qblock_ids, uint64_t& starter_offset) const;
+    bool find_blockchain_supplement(const uint64_t req_start_block, const std::list<crypto::hash>& qblock_ids, std::list<std::pair<block, std::list<transaction> > >& blocks, uint64_t& total_height, uint64_t& start_height, size_t max_count) const;
+    bool handle_get_objects(NOTIFY_REQUEST_GET_OBJECTS::request& arg, NOTIFY_RESPONSE_GET_OBJECTS::request& rsp);
+    bool handle_get_objects(const COMMAND_RPC_GET_RANDOM_OUTPUTS_FOR_AMOUNTS::request& req, COMMAND_RPC_GET_RANDOM_OUTPUTS_FOR_AMOUNTS::response& res);
+    bool get_random_outs_for_amounts(const COMMAND_RPC_GET_RANDOM_OUTPUTS_FOR_AMOUNTS::request& req, COMMAND_RPC_GET_RANDOM_OUTPUTS_FOR_AMOUNTS::response& res) const;
+    bool get_tx_outputs_gindexs(const crypto::hash& tx_id, std::vector<uint64_t>& indexs) const;
+    bool store_blockchain();
+    bool check_tx_input(const txin_to_key& txin, const crypto::hash& tx_prefix_hash, const std::vector<crypto::signature>& sig, uint64_t* pmax_related_block_height = NULL) const;
+    bool check_tx_inputs(const transaction& tx, uint64_t* pmax_used_block_height = NULL) const;
+    bool check_tx_inputs(const transaction& tx, uint64_t& pmax_used_block_height, crypto::hash& max_used_block_id) const;
+    uint64_t get_current_cumulative_blocksize_limit() const;
+    bool is_storing_blockchain()const{return m_is_blockchain_storing;}
+    uint64_t block_difficulty(uint64_t i) const;
+
+    template<class t_ids_container, class t_blocks_container, class t_missed_container>
+    bool get_blocks(const t_ids_container& block_ids, t_blocks_container& blocks, t_missed_container& missed_bs) const;
+
+    template<class t_ids_container, class t_tx_container, class t_missed_container>
+    bool get_transactions(const t_ids_container& txs_ids, t_tx_container& txs, t_missed_container& missed_txs) const;
+
+    //debug functions
+    void print_blockchain(uint64_t start_index, uint64_t end_index);
+    void print_blockchain_index();
+    void print_blockchain_outs(const std::string& file);
+
+    void check_against_checkpoints(const checkpoints& points, bool enforce);
+    void set_enforce_dns_checkpoints(bool enforce);
+    bool update_checkpoints(const std::string& file_path, bool check_dns);
+
+    BlockchainDB& get_db()
+    {
+      return *m_db;
+    }
+
+  private:
+    typedef std::unordered_map<crypto::hash, size_t> blocks_by_id_index;
+    typedef std::unordered_map<crypto::hash, transaction_chain_entry> transactions_container;
+    typedef std::unordered_set<crypto::key_image> key_images_container;
+    typedef std::vector<block_extended_info> blocks_container;
+    typedef std::unordered_map<crypto::hash, block_extended_info> blocks_ext_by_hash;
+    typedef std::unordered_map<crypto::hash, block> blocks_by_hash;
+    typedef std::map<uint64_t, std::vector<std::pair<crypto::hash, size_t>>> outputs_container; //crypto::hash - tx hash, size_t - index of out in transaction
+
+    BlockchainDB* m_db;
+
+    tx_memory_pool& m_tx_pool;
+    mutable epee::critical_section m_blockchain_lock; // TODO: add here reader/writer lock
+
+    // main chain
+    blocks_container m_blocks;               // height  -> block_extended_info
+    blocks_by_id_index m_blocks_index;       // crypto::hash -> height
+    transactions_container m_transactions;
+    key_images_container m_spent_keys;
+    size_t m_current_block_cumul_sz_limit;
+
+
+    // all alternative chains
+    blocks_ext_by_hash m_alternative_chains; // crypto::hash -> block_extended_info
+
+    // some invalid blocks
+    blocks_ext_by_hash m_invalid_blocks;     // crypto::hash -> block_extended_info
+    outputs_container m_outputs;
+
+
+    checkpoints m_checkpoints;
+    std::atomic<bool> m_is_in_checkpoint_zone;
+    std::atomic<bool> m_is_blockchain_storing;
+    bool m_enforce_dns_checkpoints;
+
+    bool switch_to_alternative_blockchain(std::list<blocks_ext_by_hash::iterator>& alt_chain, bool discard_disconnected_chain);
+    block pop_block_from_blockchain();
+    bool purge_transaction_from_blockchain(const crypto::hash& tx_id);
+    bool purge_transaction_keyimages_from_blockchain(const transaction& tx, bool strict_check);
+
+    bool handle_block_to_main_chain(const block& bl, block_verification_context& bvc);
+    bool handle_block_to_main_chain(const block& bl, const crypto::hash& id, block_verification_context& bvc);
+    bool handle_alternative_block(const block& b, const crypto::hash& id, block_verification_context& bvc);
+    difficulty_type get_next_difficulty_for_alternative_chain(const std::list<blocks_ext_by_hash::iterator>& alt_chain, block_extended_info& bei) const;
+    bool prevalidate_miner_transaction(const block& b, uint64_t height);
+    bool validate_miner_transaction(const block& b, size_t cumulative_block_size, uint64_t fee, uint64_t& base_reward, uint64_t already_generated_coins);
+    bool validate_transaction(const block& b, uint64_t height, const transaction& tx);
+    bool rollback_blockchain_switching(std::list<block>& original_chain, uint64_t rollback_height);
+    bool add_transaction_from_block(const transaction& tx, const crypto::hash& tx_id, const crypto::hash& bl_id, uint64_t bl_height);
+    bool push_transaction_to_global_outs_index(const transaction& tx, const crypto::hash& tx_id, std::vector<uint64_t>& global_indexes);
+    bool pop_transaction_from_global_index(const transaction& tx, const crypto::hash& tx_id);
+    void get_last_n_blocks_sizes(std::vector<size_t>& sz, size_t count) const;
+    void add_out_to_get_random_outs(COMMAND_RPC_GET_RANDOM_OUTPUTS_FOR_AMOUNTS::outs_for_amount& result_outs, uint64_t amount, size_t i) const;
+    bool is_tx_spendtime_unlocked(uint64_t unlock_time) const;
+    bool add_block_as_invalid(const block& bl, const crypto::hash& h);
+    bool add_block_as_invalid(const block_extended_info& bei, const crypto::hash& h);
+    bool check_block_timestamp(const block& b) const;
+    bool check_block_timestamp(std::vector<uint64_t>& timestamps, const block& b) const;
+    uint64_t get_adjusted_time() const;
+    bool complete_timestamps_vector(uint64_t start_height, std::vector<uint64_t>& timestamps);
+    bool update_next_cumulative_size_limit();
+
+    bool check_for_double_spend(const transaction& tx, key_images_container& keys_this_block) const;
+  };
+
+
+  /************************************************************************/
+  /*                                                                      */
+  /************************************************************************/
+
+  #define CURRENT_BLOCKCHAIN_STORAGE_ARCHIVE_VER    12
+
+  //------------------------------------------------------------------
+
+}  // namespace cryptonote
+
+BOOST_CLASS_VERSION(cryptonote::Blockchain, CURRENT_BLOCKCHAIN_STORAGE_ARCHIVE_VER)

src/cryptonote_core/blockchain_storage.cpp

@@ -58,19 +58,19 @@ using namespace cryptonote;
 DISABLE_VS_WARNINGS(4267)
 
 //------------------------------------------------------------------
-bool blockchain_storage::have_tx(const crypto::hash &id)
+bool blockchain_storage::have_tx(const crypto::hash &id) const
 {
   CRITICAL_REGION_LOCAL(m_blockchain_lock);
   return m_transactions.find(id) != m_transactions.end();
 }
 //------------------------------------------------------------------
-bool blockchain_storage::have_tx_keyimg_as_spent(const crypto::key_image &key_im)
+bool blockchain_storage::have_tx_keyimg_as_spent(const crypto::key_image &key_im) const
 {
   CRITICAL_REGION_LOCAL(m_blockchain_lock);
   return  m_spent_keys.find(key_im) != m_spent_keys.end();
 }
 //------------------------------------------------------------------
-transaction *blockchain_storage::get_tx(const crypto::hash &id)
+const transaction *blockchain_storage::get_tx(const crypto::hash &id) const
 {
   CRITICAL_REGION_LOCAL(m_blockchain_lock);
   auto it = m_transactions.find(id);
@@ -80,7 +80,7 @@ transaction *blockchain_storage::get_tx(const crypto::hash &id)
   return &it->second.tx;
 }
 //------------------------------------------------------------------
-uint64_t blockchain_storage::get_current_blockchain_height()
+uint64_t blockchain_storage::get_current_blockchain_height() const
 {
   CRITICAL_REGION_LOCAL(m_blockchain_lock);
   return m_blocks.size();
@@ -117,24 +117,14 @@ bool blockchain_storage::init(const std::string& config_folder, bool testnet)
   else
   {
       LOG_PRINT_L0("Can't load blockchain storage from file, generating genesis block.");
-      block bl = boost::value_initialized<block>();
+      if (!store_genesis_block(testnet, true))
-      block_verification_context bvc = boost::value_initialized<block_verification_context>();
+        return false;
-      if (testnet)
-      {
-        generate_genesis_block(bl, config::testnet::GENESIS_TX, config::testnet::GENESIS_NONCE);
-      }
-      else
-      {
-        generate_genesis_block(bl, config::GENESIS_TX, config::GENESIS_NONCE);
-      }
-      add_new_block(bl, bvc);
-      CHECK_AND_ASSERT_MES(!bvc.m_verifivation_failed && bvc.m_added_to_main_chain, false, "Failed to add genesis block to blockchain");
   }
   if(!m_blocks.size())
   {
     LOG_PRINT_L0("Blockchain not loaded, generating genesis block.");
 
-    if (!store_genesis_block(testnet)) {
+    if (!store_genesis_block(testnet, false)) {
       return false;
     }
   } else {
@@ -163,7 +153,7 @@ bool blockchain_storage::init(const std::string& config_folder, bool testnet)
   return true;
 }
 //------------------------------------------------------------------
-bool blockchain_storage::store_genesis_block(bool testnet) {
+bool blockchain_storage::store_genesis_block(bool testnet, bool check_added) {
   block bl = ::boost::value_initialized<block>();
   block_verification_context bvc = boost::value_initialized<block_verification_context>();
 
@@ -177,7 +167,7 @@ bool blockchain_storage::store_genesis_block(bool testnet) {
   }
 
   add_new_block(bl, bvc);
-  CHECK_AND_ASSERT_MES(!bvc.m_verifivation_failed, false, "Failed to add genesis block to blockchain");
+  CHECK_AND_ASSERT_MES(!bvc.m_verifivation_failed && (bvc.m_added_to_main_chain || !check_added), false, "Failed to add genesis block to blockchain");
   return true;
 }
 //------------------------------------------------------------------
@@ -235,7 +225,7 @@ bool blockchain_storage::pop_block_from_blockchain()
   m_blocks_index.erase(bl_ind);
   //pop block from core
   m_blocks.pop_back();
-  m_tx_pool.on_blockchain_dec(m_blocks.size()-1, get_tail_id());
+  m_tx_pool->on_blockchain_dec(m_blocks.size()-1, get_tail_id());
   return true;
 }
 //------------------------------------------------------------------
@@ -311,7 +301,7 @@ bool blockchain_storage::purge_transaction_from_blockchain(const crypto::hash& t
   if(!is_coinbase(tx))
   {
     cryptonote::tx_verification_context tvc = AUTO_VAL_INIT(tvc);
-    bool r = m_tx_pool.add_tx(tx, tvc, true);
+    bool r = m_tx_pool->add_tx(tx, tvc, true);
     CHECK_AND_ASSERT_MES(r, false, "purge_block_data_from_blockchain: failed to add transaction to transaction pool");
   }
 
@@ -337,14 +327,14 @@ bool blockchain_storage::purge_block_data_from_blockchain(const block& bl, size_
   return res;
 }
 //------------------------------------------------------------------
-crypto::hash blockchain_storage::get_tail_id(uint64_t& height)
+crypto::hash blockchain_storage::get_tail_id(uint64_t& height) const
 {
   CRITICAL_REGION_LOCAL(m_blockchain_lock);
   height = get_current_blockchain_height()-1;
   return get_tail_id();
 }
 //------------------------------------------------------------------
-crypto::hash blockchain_storage::get_tail_id()
+crypto::hash blockchain_storage::get_tail_id() const
 {
   CRITICAL_REGION_LOCAL(m_blockchain_lock);
   crypto::hash id = null_hash;
@@ -355,7 +345,7 @@ crypto::hash blockchain_storage::get_tail_id()
   return id;
 }
 //------------------------------------------------------------------
-bool blockchain_storage::get_short_chain_history(std::list<crypto::hash>& ids)
+bool blockchain_storage::get_short_chain_history(std::list<crypto::hash>& ids) const
 {
   CRITICAL_REGION_LOCAL(m_blockchain_lock);
   size_t i = 0;
@@ -385,7 +375,7 @@ bool blockchain_storage::get_short_chain_history(std::list<crypto::hash>& ids)
   return true;
 }
 //------------------------------------------------------------------
-crypto::hash blockchain_storage::get_block_id_by_height(uint64_t height)
+crypto::hash blockchain_storage::get_block_id_by_height(uint64_t height) const
 {
   CRITICAL_REGION_LOCAL(m_blockchain_lock);
   if(height >= m_blocks.size())
@@ -394,7 +384,7 @@ crypto::hash blockchain_storage::get_block_id_by_height(uint64_t height)
   return get_block_hash(m_blocks[height].bl);
 }
 //------------------------------------------------------------------
-bool blockchain_storage::get_block_by_hash(const crypto::hash &h, block &blk) {
+bool blockchain_storage::get_block_by_hash(const crypto::hash &h, block &blk) const {
   CRITICAL_REGION_LOCAL(m_blockchain_lock);
 
   // try to find block in main chain
@@ -414,20 +404,20 @@ bool blockchain_storage::get_block_by_hash(const crypto::hash &h, block &blk) {
   return false;
 }
 //------------------------------------------------------------------
-void blockchain_storage::get_all_known_block_ids(std::list<crypto::hash> &main, std::list<crypto::hash> &alt, std::list<crypto::hash> &invalid) {
+void blockchain_storage::get_all_known_block_ids(std::list<crypto::hash> &main, std::list<crypto::hash> &alt, std::list<crypto::hash> &invalid) const {
   CRITICAL_REGION_LOCAL(m_blockchain_lock);
 
-  BOOST_FOREACH(blocks_by_id_index::value_type &v, m_blocks_index)
+  BOOST_FOREACH(const blocks_by_id_index::value_type &v, m_blocks_index)
     main.push_back(v.first);
 
-  BOOST_FOREACH(blocks_ext_by_hash::value_type &v, m_alternative_chains)
+  BOOST_FOREACH(const blocks_ext_by_hash::value_type &v, m_alternative_chains)
     alt.push_back(v.first);
 
-  BOOST_FOREACH(blocks_ext_by_hash::value_type &v, m_invalid_blocks)
+  BOOST_FOREACH(const blocks_ext_by_hash::value_type &v, m_invalid_blocks)
     invalid.push_back(v.first);
 }
 //------------------------------------------------------------------
-difficulty_type blockchain_storage::get_difficulty_for_next_block()
+difficulty_type blockchain_storage::get_difficulty_for_next_block() const
 {
   CRITICAL_REGION_LOCAL(m_blockchain_lock);
   std::vector<uint64_t> timestamps;
@@ -539,7 +529,7 @@ bool blockchain_storage::switch_to_alternative_blockchain(std::list<blocks_ext_b
   return true;
 }
 //------------------------------------------------------------------
-difficulty_type blockchain_storage::get_next_difficulty_for_alternative_chain(const std::list<blocks_ext_by_hash::iterator>& alt_chain, block_extended_info& bei)
+difficulty_type blockchain_storage::get_next_difficulty_for_alternative_chain(const std::list<blocks_ext_by_hash::iterator>& alt_chain, block_extended_info& bei) const
 {
   std::vector<uint64_t> timestamps;
   std::vector<difficulty_type> commulative_difficulties;
@@ -584,7 +574,7 @@ difficulty_type blockchain_storage::get_next_difficulty_for_alternative_chain(co
   return next_difficulty(timestamps, commulative_difficulties);
 }
 //------------------------------------------------------------------
-bool blockchain_storage::prevalidate_miner_transaction(const block& b, uint64_t height)
+bool blockchain_storage::prevalidate_miner_transaction(const block& b, uint64_t height) const
 {
   CHECK_AND_ASSERT_MES(b.miner_tx.vin.size() == 1, false, "coinbase transaction in the block has no inputs");
   CHECK_AND_ASSERT_MES(b.miner_tx.vin[0].type() == typeid(txin_gen), false, "coinbase transaction in the block has the wrong type");
@@ -607,7 +597,7 @@ bool blockchain_storage::prevalidate_miner_transaction(const block& b, uint64_t
   return true;
 }
 //------------------------------------------------------------------
-bool blockchain_storage::validate_miner_transaction(const block& b, size_t cumulative_block_size, uint64_t fee, uint64_t& base_reward, uint64_t already_generated_coins)
+bool blockchain_storage::validate_miner_transaction(const block& b, size_t cumulative_block_size, uint64_t fee, uint64_t& base_reward, uint64_t already_generated_coins) const
 {
   //validate reward
   uint64_t money_in_use = 0;
@@ -634,7 +624,7 @@ bool blockchain_storage::validate_miner_transaction(const block& b, size_t cumul
   return true;
 }
 //------------------------------------------------------------------
-bool blockchain_storage::get_backward_blocks_sizes(size_t from_height, std::vector<size_t>& sz, size_t count)
+bool blockchain_storage::get_backward_blocks_sizes(size_t from_height, std::vector<size_t>& sz, size_t count) const
 {
   CRITICAL_REGION_LOCAL(m_blockchain_lock);
   CHECK_AND_ASSERT_MES(from_height < m_blocks.size(), false, "Internal error: get_backward_blocks_sizes called with from_height=" << from_height << ", blockchain height = " << m_blocks.size());
@@ -646,7 +636,7 @@ bool blockchain_storage::get_backward_blocks_sizes(size_t from_height, std::vect
   return true;
 }
 //------------------------------------------------------------------
-bool blockchain_storage::get_last_n_blocks_sizes(std::vector<size_t>& sz, size_t count)
+bool blockchain_storage::get_last_n_blocks_sizes(std::vector<size_t>& sz, size_t count) const
 {
   CRITICAL_REGION_LOCAL(m_blockchain_lock);
   if(!m_blocks.size())
@@ -654,12 +644,12 @@ bool blockchain_storage::get_last_n_blocks_sizes(std::vector<size_t>& sz, size_t
   return get_backward_blocks_sizes(m_blocks.size() -1, sz, count);
 }
 //------------------------------------------------------------------
-uint64_t blockchain_storage::get_current_comulative_blocksize_limit()
+uint64_t blockchain_storage::get_current_cumulative_blocksize_limit() const
 {
   return m_current_block_cumul_sz_limit;
 }
 //------------------------------------------------------------------
-bool blockchain_storage::create_block_template(block& b, const account_public_address& miner_address, difficulty_type& diffic, uint64_t& height, const blobdata& ex_nonce)
+bool blockchain_storage::create_block_template(block& b, const account_public_address& miner_address, difficulty_type& diffic, uint64_t& height, const blobdata& ex_nonce) const
 {
   size_t median_size;
   uint64_t already_generated_coins;
@@ -680,16 +670,16 @@ bool blockchain_storage::create_block_template(block& b, const account_public_ad
 
   size_t txs_size;
   uint64_t fee;
-  if (!m_tx_pool.fill_block_template(b, median_size, already_generated_coins, txs_size, fee)) {
+  if (!m_tx_pool->fill_block_template(b, median_size, already_generated_coins, txs_size, fee)) {
     return false;
   }
 #if defined(DEBUG_CREATE_BLOCK_TEMPLATE)
   size_t real_txs_size = 0;
   uint64_t real_fee = 0;
-  CRITICAL_REGION_BEGIN(m_tx_pool.m_transactions_lock);
+  CRITICAL_REGION_BEGIN(m_tx_pool->m_transactions_lock);
   BOOST_FOREACH(crypto::hash &cur_hash, b.tx_hashes) {
-    auto cur_res = m_tx_pool.m_transactions.find(cur_hash);
+    auto cur_res = m_tx_pool->m_transactions.find(cur_hash);
-    if (cur_res == m_tx_pool.m_transactions.end()) {
+    if (cur_res == m_tx_pool->m_transactions.end()) {
       LOG_ERROR("Creating block template: error: transaction not found");
       continue;
     }
@@ -778,7 +768,7 @@ bool blockchain_storage::create_block_template(block& b, const account_public_ad
   return false;
 }
 //------------------------------------------------------------------
-bool blockchain_storage::complete_timestamps_vector(uint64_t start_top_height, std::vector<uint64_t>& timestamps)
+bool blockchain_storage::complete_timestamps_vector(uint64_t start_top_height, std::vector<uint64_t>& timestamps) const
 {
 
   if(timestamps.size() >= BLOCKCHAIN_TIMESTAMP_CHECK_WINDOW)
@@ -943,7 +933,7 @@ bool blockchain_storage::handle_alternative_block(const block& b, const crypto::
   return true;
 }
 //------------------------------------------------------------------
-bool blockchain_storage::get_blocks(uint64_t start_offset, size_t count, std::list<block>& blocks, std::list<transaction>& txs)
+bool blockchain_storage::get_blocks(uint64_t start_offset, size_t count, std::list<block>& blocks, std::list<transaction>& txs) const
 {
   CRITICAL_REGION_LOCAL(m_blockchain_lock);
   if(start_offset >= m_blocks.size())
@@ -959,7 +949,7 @@ bool blockchain_storage::get_blocks(uint64_t start_offset, size_t count, std::li
   return true;
 }
 //------------------------------------------------------------------
-bool blockchain_storage::get_blocks(uint64_t start_offset, size_t count, std::list<block>& blocks)
+bool blockchain_storage::get_blocks(uint64_t start_offset, size_t count, std::list<block>& blocks) const
 {
   CRITICAL_REGION_LOCAL(m_blockchain_lock);
   if(start_offset >= m_blocks.size())
@@ -1003,7 +993,7 @@ bool blockchain_storage::handle_get_objects(NOTIFY_REQUEST_GET_OBJECTS::request&
   return true;
 }
 //------------------------------------------------------------------
-bool blockchain_storage::get_alternative_blocks(std::list<block>& blocks)
+bool blockchain_storage::get_alternative_blocks(std::list<block>& blocks) const
 {
   CRITICAL_REGION_LOCAL(m_blockchain_lock);
 
@@ -1014,21 +1004,21 @@ bool blockchain_storage::get_alternative_blocks(std::list<block>& blocks)
   return true;
 }
 //------------------------------------------------------------------
-size_t blockchain_storage::get_alternative_blocks_count()
+size_t blockchain_storage::get_alternative_blocks_count() const
 {
   CRITICAL_REGION_LOCAL(m_blockchain_lock);
   return m_alternative_chains.size();
 }
 //------------------------------------------------------------------
-bool blockchain_storage::add_out_to_get_random_outs(std::vector<std::pair<crypto::hash, size_t> >& amount_outs, COMMAND_RPC_GET_RANDOM_OUTPUTS_FOR_AMOUNTS::outs_for_amount& result_outs, uint64_t amount, size_t i)
+bool blockchain_storage::add_out_to_get_random_outs(const std::vector<std::pair<crypto::hash, size_t> >& amount_outs, COMMAND_RPC_GET_RANDOM_OUTPUTS_FOR_AMOUNTS::outs_for_amount& result_outs, uint64_t amount, size_t i) const
 {
   CRITICAL_REGION_LOCAL(m_blockchain_lock);
-  transactions_container::iterator tx_it = m_transactions.find(amount_outs[i].first);
+  transactions_container::const_iterator tx_it = m_transactions.find(amount_outs[i].first);
   CHECK_AND_ASSERT_MES(tx_it != m_transactions.end(), false, "internal error: transaction with id " << amount_outs[i].first << ENDL <<
     ", used in mounts global index for amount=" << amount << ": i=" << i << "not found in transactions index");
   CHECK_AND_ASSERT_MES(tx_it->second.tx.vout.size() > amount_outs[i].second, false, "internal error: in global outs index, transaction out index="
     << amount_outs[i].second << " more than transaction outputs = " << tx_it->second.tx.vout.size() << ", for tx id = " << amount_outs[i].first);
-  transaction& tx = tx_it->second.tx;
+  const transaction& tx = tx_it->second.tx;
   CHECK_AND_ASSERT_MES(tx.vout[amount_outs[i].second].target.type() == typeid(txout_to_key), false, "unknown tx out type");
 
   //check if transaction is unlocked
@@ -1041,7 +1031,7 @@ bool blockchain_storage::add_out_to_get_random_outs(std::vector<std::pair<crypto
   return true;
 }
 //------------------------------------------------------------------
-size_t blockchain_storage::find_end_of_allowed_index(const std::vector<std::pair<crypto::hash, size_t> >& amount_outs)
+size_t blockchain_storage::find_end_of_allowed_index(const std::vector<std::pair<crypto::hash, size_t> >& amount_outs) const
 {
   CRITICAL_REGION_LOCAL(m_blockchain_lock);
   if(!amount_outs.size())
@@ -1050,7 +1040,7 @@ size_t blockchain_storage::find_end_of_allowed_index(const std::vector<std::pair
   do
   {
     --i;
-    transactions_container::iterator it = m_transactions.find(amount_outs[i].first);
+    transactions_container::const_iterator it = m_transactions.find(amount_outs[i].first);
     CHECK_AND_ASSERT_MES(it != m_transactions.end(), 0, "internal error: failed to find transaction from outputs index with tx_id=" << amount_outs[i].first);
     if(it->second.m_keeper_block_height + CRYPTONOTE_MINED_MONEY_UNLOCK_WINDOW <= get_current_blockchain_height() )
       return i+1;
@@ -1058,7 +1048,7 @@ size_t blockchain_storage::find_end_of_allowed_index(const std::vector<std::pair
   return 0;
 }
 //------------------------------------------------------------------
-bool blockchain_storage::get_random_outs_for_amounts(const COMMAND_RPC_GET_RANDOM_OUTPUTS_FOR_AMOUNTS::request& req, COMMAND_RPC_GET_RANDOM_OUTPUTS_FOR_AMOUNTS::response& res)
+bool blockchain_storage::get_random_outs_for_amounts(const COMMAND_RPC_GET_RANDOM_OUTPUTS_FOR_AMOUNTS::request& req, COMMAND_RPC_GET_RANDOM_OUTPUTS_FOR_AMOUNTS::response& res) const
 {
   CRITICAL_REGION_LOCAL(m_blockchain_lock);
   BOOST_FOREACH(uint64_t amount, req.amounts)
@@ -1071,7 +1061,7 @@ bool blockchain_storage::get_random_outs_for_amounts(const COMMAND_RPC_GET_RANDO
       LOG_PRINT_L1("COMMAND_RPC_GET_RANDOM_OUTPUTS_FOR_AMOUNTS: not outs for amount " << amount << ", wallet should use some real outs when it lookup for some mix, so, at least one out for this amount should exist");
       continue;//actually this is strange situation, wallet should use some real outs when it lookup for some mix, so, at least one out for this amount should exist
     }
-    std::vector<std::pair<crypto::hash, size_t> >& amount_outs  = it->second;
+    const std::vector<std::pair<crypto::hash, size_t> >& amount_outs  = it->second;
     //it is not good idea to use top fresh outs, because it increases possibility of transaction canceling on split
     //lets find upper bound of not fresh outs
     size_t up_index_limit = find_end_of_allowed_index(amount_outs);
@@ -1106,7 +1096,7 @@ bool blockchain_storage::get_random_outs_for_amounts(const COMMAND_RPC_GET_RANDO
   return true;
 }
 //------------------------------------------------------------------
-bool blockchain_storage::find_blockchain_supplement(const std::list<crypto::hash>& qblock_ids, uint64_t& starter_offset)
+bool blockchain_storage::find_blockchain_supplement(const std::list<crypto::hash>& qblock_ids, uint64_t& starter_offset) const
 {
   CRITICAL_REGION_LOCAL(m_blockchain_lock);
 
@@ -1153,7 +1143,7 @@ bool blockchain_storage::find_blockchain_supplement(const std::list<crypto::hash
   return true;
 }
 //------------------------------------------------------------------
-uint64_t blockchain_storage::block_difficulty(size_t i)
+uint64_t blockchain_storage::block_difficulty(size_t i) const
 {
   CRITICAL_REGION_LOCAL(m_blockchain_lock);
   CHECK_AND_ASSERT_MES(i < m_blocks.size(), false, "wrong block index i = " << i << " at blockchain_storage::block_difficulty()");
@@ -1163,7 +1153,7 @@ uint64_t blockchain_storage::block_difficulty(size_t i)
   return m_blocks[i].cumulative_difficulty - m_blocks[i-1].cumulative_difficulty;
 }
 //------------------------------------------------------------------
-double blockchain_storage::get_avg_block_size( size_t count)
+double blockchain_storage::get_avg_block_size( size_t count) const
 {
 		if (count > get_current_blockchain_height()) return 500;
 
@@ -1241,7 +1231,7 @@ void blockchain_storage::print_blockchain_outs(const std::string& file)
   }
 }
 //------------------------------------------------------------------
-bool blockchain_storage::find_blockchain_supplement(const std::list<crypto::hash>& qblock_ids, NOTIFY_RESPONSE_CHAIN_ENTRY::request& resp)
+bool blockchain_storage::find_blockchain_supplement(const std::list<crypto::hash>& qblock_ids, NOTIFY_RESPONSE_CHAIN_ENTRY::request& resp) const
 {
   CRITICAL_REGION_LOCAL(m_blockchain_lock);
   if(!find_blockchain_supplement(qblock_ids, resp.start_height))
@@ -1254,7 +1244,7 @@ bool blockchain_storage::find_blockchain_supplement(const std::list<crypto::hash
   return true;
 }
 //------------------------------------------------------------------
-bool blockchain_storage::find_blockchain_supplement(const uint64_t req_start_block, const std::list<crypto::hash>& qblock_ids, std::list<std::pair<block, std::list<transaction> > >& blocks, uint64_t& total_height, uint64_t& start_height, size_t max_count)
+bool blockchain_storage::find_blockchain_supplement(const uint64_t req_start_block, const std::list<crypto::hash>& qblock_ids, std::list<std::pair<block, std::list<transaction> > >& blocks, uint64_t& total_height, uint64_t& start_height, size_t max_count) const
 {
   CRITICAL_REGION_LOCAL(m_blockchain_lock);
   if(req_start_block > 0) {
@@ -1293,7 +1283,7 @@ bool blockchain_storage::add_block_as_invalid(const block_extended_info& bei, co
   return true;
 }
 //------------------------------------------------------------------
-bool blockchain_storage::have_block(const crypto::hash& id)
+bool blockchain_storage::have_block(const crypto::hash& id) const
 {
   CRITICAL_REGION_LOCAL(m_blockchain_lock);
   if(m_blocks_index.count(id))
@@ -1331,13 +1321,13 @@ bool blockchain_storage::push_transaction_to_global_outs_index(const transaction
   return true;
 }
 //------------------------------------------------------------------
-size_t blockchain_storage::get_total_transactions()
+size_t blockchain_storage::get_total_transactions() const
 {
   CRITICAL_REGION_LOCAL(m_blockchain_lock);
   return m_transactions.size();
 }
 //------------------------------------------------------------------
-bool blockchain_storage::get_outs(uint64_t amount, std::list<crypto::public_key>& pkeys)
+bool blockchain_storage::get_outs(uint64_t amount, std::list<crypto::public_key>& pkeys) const
 {
   CRITICAL_REGION_LOCAL(m_blockchain_lock);
   auto it = m_outputs.find(amount);
@@ -1428,7 +1418,7 @@ bool blockchain_storage::add_transaction_from_block(const transaction& tx, const
   return true;
 }
 //------------------------------------------------------------------
-bool blockchain_storage::get_tx_outputs_gindexs(const crypto::hash& tx_id, std::vector<uint64_t>& indexs)
+bool blockchain_storage::get_tx_outputs_gindexs(const crypto::hash& tx_id, std::vector<uint64_t>& indexs) const
 {
   CRITICAL_REGION_LOCAL(m_blockchain_lock);
   auto it = m_transactions.find(tx_id);
@@ -1443,7 +1433,7 @@ bool blockchain_storage::get_tx_outputs_gindexs(const crypto::hash& tx_id, std::
   return true;
 }
 //------------------------------------------------------------------
-bool blockchain_storage::check_tx_inputs(const transaction& tx, uint64_t& max_used_block_height, crypto::hash& max_used_block_id)
+bool blockchain_storage::check_tx_inputs(const transaction& tx, uint64_t& max_used_block_height, crypto::hash& max_used_block_id) const
 {
   CRITICAL_REGION_LOCAL(m_blockchain_lock);
   bool res = check_tx_inputs(tx, &max_used_block_height);
@@ -1453,7 +1443,7 @@ bool blockchain_storage::check_tx_inputs(const transaction& tx, uint64_t& max_us
   return true;
 }
 //------------------------------------------------------------------
-bool blockchain_storage::have_tx_keyimges_as_spent(const transaction &tx)
+bool blockchain_storage::have_tx_keyimges_as_spent(const transaction &tx) const
 {
   BOOST_FOREACH(const txin_v& in, tx.vin)
   {
@@ -1464,13 +1454,13 @@ bool blockchain_storage::have_tx_keyimges_as_spent(const transaction &tx)
   return false;
 }
 //------------------------------------------------------------------
-bool blockchain_storage::check_tx_inputs(const transaction& tx, uint64_t* pmax_used_block_height)
+bool blockchain_storage::check_tx_inputs(const transaction& tx, uint64_t* pmax_used_block_height) const
 {
   crypto::hash tx_prefix_hash = get_transaction_prefix_hash(tx);
   return check_tx_inputs(tx, tx_prefix_hash, pmax_used_block_height);
 }
 //------------------------------------------------------------------
-bool blockchain_storage::check_tx_inputs(const transaction& tx, const crypto::hash& tx_prefix_hash, uint64_t* pmax_used_block_height)
+bool blockchain_storage::check_tx_inputs(const transaction& tx, const crypto::hash& tx_prefix_hash, uint64_t* pmax_used_block_height) const
 {
   size_t sig_index = 0;
   if(pmax_used_block_height)
@@ -1502,7 +1492,7 @@ bool blockchain_storage::check_tx_inputs(const transaction& tx, const crypto::ha
   return true;
 }
 //------------------------------------------------------------------
-bool blockchain_storage::is_tx_spendtime_unlocked(uint64_t unlock_time)
+bool blockchain_storage::is_tx_spendtime_unlocked(uint64_t unlock_time) const
 {
   if(unlock_time < CRYPTONOTE_MAX_BLOCK_NUMBER)
   {
@@ -1523,15 +1513,15 @@ bool blockchain_storage::is_tx_spendtime_unlocked(uint64_t unlock_time)
   return false;
 }
 //------------------------------------------------------------------
-bool blockchain_storage::check_tx_input(const txin_to_key& txin, const crypto::hash& tx_prefix_hash, const std::vector<crypto::signature>& sig, uint64_t* pmax_related_block_height)
+bool blockchain_storage::check_tx_input(const txin_to_key& txin, const crypto::hash& tx_prefix_hash, const std::vector<crypto::signature>& sig, uint64_t* pmax_related_block_height) const
 {
   CRITICAL_REGION_LOCAL(m_blockchain_lock);
 
   struct outputs_visitor
   {
     std::vector<const crypto::public_key *>& m_results_collector;
-    blockchain_storage& m_bch;
+    const blockchain_storage& m_bch;
-    outputs_visitor(std::vector<const crypto::public_key *>& results_collector, blockchain_storage& bch):m_results_collector(results_collector), m_bch(bch)
+    outputs_visitor(std::vector<const crypto::public_key *>& results_collector, const blockchain_storage& bch):m_results_collector(results_collector), m_bch(bch)
     {}
     bool handle_output(const transaction& tx, const tx_out& out)
     {
@@ -1573,13 +1563,13 @@ bool blockchain_storage::check_tx_input(const txin_to_key& txin, const crypto::h
   return crypto::check_ring_signature(tx_prefix_hash, txin.k_image, output_keys, sig.data());
 }
 //------------------------------------------------------------------
-uint64_t blockchain_storage::get_adjusted_time()
+uint64_t blockchain_storage::get_adjusted_time() const
 {
   //TODO: add collecting median time
   return time(NULL);
 }
 //------------------------------------------------------------------
-bool blockchain_storage::check_block_timestamp_main(const block& b)
+bool blockchain_storage::check_block_timestamp_main(const block& b) const
 {
   if(b.timestamp > get_adjusted_time() + CRYPTONOTE_BLOCK_FUTURE_TIME_LIMIT)
   {
@@ -1595,7 +1585,7 @@ bool blockchain_storage::check_block_timestamp_main(const block& b)
   return check_block_timestamp(std::move(timestamps), b);
 }
 //------------------------------------------------------------------
-bool blockchain_storage::check_block_timestamp(std::vector<uint64_t> timestamps, const block& b)
+bool blockchain_storage::check_block_timestamp(std::vector<uint64_t> timestamps, const block& b) const
 {
   if(timestamps.size() < BLOCKCHAIN_TIMESTAMP_CHECK_WINDOW)
     return true;
@@ -1696,7 +1686,7 @@ bool blockchain_storage::handle_block_to_main_chain(const block& bl, const crypt
     transaction tx;
     size_t blob_size = 0;
     uint64_t fee = 0;
-    if(!m_tx_pool.take_tx(tx_id, tx, blob_size, fee))
+    if(!m_tx_pool->take_tx(tx_id, tx, blob_size, fee))
     {
       LOG_PRINT_L1("Block with id: " << id  << "has at least one unknown transaction with id: " << tx_id);
       purge_block_data_from_blockchain(bl, tx_processed_count);
@@ -1708,7 +1698,7 @@ bool blockchain_storage::handle_block_to_main_chain(const block& bl, const crypt
     {
       LOG_PRINT_L1("Block with id: " << id  << "has at least one transaction (id: " << tx_id << ") with wrong inputs.");
       cryptonote::tx_verification_context tvc = AUTO_VAL_INIT(tvc);
-      bool add_res = m_tx_pool.add_tx(tx, tvc, true);
+      bool add_res = m_tx_pool->add_tx(tx, tvc, true);
       CHECK_AND_ASSERT_MES2(add_res, "handle_block_to_main_chain: failed to add transaction back to transaction pool");
       purge_block_data_from_blockchain(bl, tx_processed_count);
       add_block_as_invalid(bl, id);
@@ -1721,7 +1711,7 @@ bool blockchain_storage::handle_block_to_main_chain(const block& bl, const crypt
     {
        LOG_PRINT_L1("Block with id: " << id << " failed to add transaction to blockchain storage");
        cryptonote::tx_verification_context tvc = AUTO_VAL_INIT(tvc);
-       bool add_res = m_tx_pool.add_tx(tx, tvc, true);
+       bool add_res = m_tx_pool->add_tx(tx, tvc, true);
        CHECK_AND_ASSERT_MES2(add_res, "handle_block_to_main_chain: failed to add transaction back to transaction pool");
        purge_block_data_from_blockchain(bl, tx_processed_count);
        bvc.m_verifivation_failed = true;
@@ -1785,7 +1775,7 @@ bool blockchain_storage::handle_block_to_main_chain(const block& bl, const crypt
   /*if(!m_orphanes_reorganize_in_work)
     review_orphaned_blocks_with_new_block_id(id, true);*/
 
-  m_tx_pool.on_blockchain_inc(bei.height, id);
+  m_tx_pool->on_blockchain_inc(bei.height, id);
   //LOG_PRINT_L0("BLOCK: " << ENDL << "" << dump_obj_as_json(bei.bl));
   return true;
 }
@@ -1808,7 +1798,7 @@ bool blockchain_storage::add_new_block(const block& bl_, block_verification_cont
   //copy block here to let modify block.target
   block bl = bl_;
   crypto::hash id = get_block_hash(bl);
-  CRITICAL_REGION_LOCAL(m_tx_pool);//to avoid deadlock lets lock tx_pool for whole add/reorganize process
+  CRITICAL_REGION_LOCAL(*m_tx_pool);//to avoid deadlock lets lock tx_pool for whole add/reorganize process
   CRITICAL_REGION_LOCAL1(m_blockchain_lock);
   if(have_block(id))
   {
@@ -1829,7 +1819,7 @@ bool blockchain_storage::add_new_block(const block& bl_, block_verification_cont
   return handle_block_to_main_chain(bl, id, bvc);
 }
 //------------------------------------------------------------------
-void blockchain_storage::check_against_checkpoints(checkpoints& points, bool enforce)
+void blockchain_storage::check_against_checkpoints(const checkpoints& points, bool enforce)
 {
   const auto& pts = points.get_points();
 

src/cryptonote_core/blockchain_storage.h

@@ -78,7 +78,7 @@ namespace cryptonote
       uint64_t already_generated_coins;
     };
 
-    blockchain_storage(tx_memory_pool& tx_pool):m_tx_pool(tx_pool), m_current_block_cumul_sz_limit(0), m_is_in_checkpoint_zone(false), m_is_blockchain_storing(false), m_enforce_dns_checkpoints(false)
+    blockchain_storage(tx_memory_pool* tx_pool):m_tx_pool(tx_pool), m_current_block_cumul_sz_limit(0), m_is_in_checkpoint_zone(false), m_is_blockchain_storing(false), m_enforce_dns_checkpoints(false)
     {};
 
     bool init() { return init(tools::get_default_data_dir(), true); }
@@ -88,56 +88,56 @@ namespace cryptonote
     void set_checkpoints(checkpoints&& chk_pts) { m_checkpoints = chk_pts; }
 
     //bool push_new_block();
-    bool get_blocks(uint64_t start_offset, size_t count, std::list<block>& blocks, std::list<transaction>& txs);
+    bool get_blocks(uint64_t start_offset, size_t count, std::list<block>& blocks, std::list<transaction>& txs) const;
-    bool get_blocks(uint64_t start_offset, size_t count, std::list<block>& blocks);
+    bool get_blocks(uint64_t start_offset, size_t count, std::list<block>& blocks) const;
-    bool get_alternative_blocks(std::list<block>& blocks);
+    bool get_alternative_blocks(std::list<block>& blocks) const;
-    size_t get_alternative_blocks_count();
+    size_t get_alternative_blocks_count() const;
-    crypto::hash get_block_id_by_height(uint64_t height);
+    crypto::hash get_block_id_by_height(uint64_t height) const;
-    bool get_block_by_hash(const crypto::hash &h, block &blk);
+    bool get_block_by_hash(const crypto::hash &h, block &blk) const;
-    void get_all_known_block_ids(std::list<crypto::hash> &main, std::list<crypto::hash> &alt, std::list<crypto::hash> &invalid);
+    void get_all_known_block_ids(std::list<crypto::hash> &main, std::list<crypto::hash> &alt, std::list<crypto::hash> &invalid) const;
 
     template<class archive_t>
     void serialize(archive_t & ar, const unsigned int version);
 
-    bool have_tx(const crypto::hash &id);
+    bool have_tx(const crypto::hash &id) const;
-    bool have_tx_keyimges_as_spent(const transaction &tx);
+    bool have_tx_keyimges_as_spent(const transaction &tx) const;
-    bool have_tx_keyimg_as_spent(const crypto::key_image &key_im);
+    bool have_tx_keyimg_as_spent(const crypto::key_image &key_im) const;
-    transaction *get_tx(const crypto::hash &id);
+    const transaction *get_tx(const crypto::hash &id) const;
 
     template<class visitor_t>
-    bool scan_outputkeys_for_indexes(const txin_to_key& tx_in_to_key, visitor_t& vis, uint64_t* pmax_related_block_height = NULL);
+    bool scan_outputkeys_for_indexes(const txin_to_key& tx_in_to_key, visitor_t& vis, uint64_t* pmax_related_block_height = NULL) const;
 
-    uint64_t get_current_blockchain_height();
+    uint64_t get_current_blockchain_height() const;
-    crypto::hash get_tail_id();
+    crypto::hash get_tail_id() const;
-    crypto::hash get_tail_id(uint64_t& height);
+    crypto::hash get_tail_id(uint64_t& height) const;
-    difficulty_type get_difficulty_for_next_block();
+    difficulty_type get_difficulty_for_next_block() const;
     bool add_new_block(const block& bl_, block_verification_context& bvc);
     bool reset_and_set_genesis_block(const block& b);
-    bool create_block_template(block& b, const account_public_address& miner_address, difficulty_type& di, uint64_t& height, const blobdata& ex_nonce);
+    bool create_block_template(block& b, const account_public_address& miner_address, difficulty_type& di, uint64_t& height, const blobdata& ex_nonce) const;
-    bool have_block(const crypto::hash& id);
+    bool have_block(const crypto::hash& id) const;
-    size_t get_total_transactions();
+    size_t get_total_transactions() const;
-    bool get_outs(uint64_t amount, std::list<crypto::public_key>& pkeys);
+    bool get_outs(uint64_t amount, std::list<crypto::public_key>& pkeys) const;
-    bool get_short_chain_history(std::list<crypto::hash>& ids);
+    bool get_short_chain_history(std::list<crypto::hash>& ids) const;
-    bool find_blockchain_supplement(const std::list<crypto::hash>& qblock_ids, NOTIFY_RESPONSE_CHAIN_ENTRY::request& resp);
+    bool find_blockchain_supplement(const std::list<crypto::hash>& qblock_ids, NOTIFY_RESPONSE_CHAIN_ENTRY::request& resp) const;
-    bool find_blockchain_supplement(const std::list<crypto::hash>& qblock_ids, uint64_t& starter_offset);
+    bool find_blockchain_supplement(const std::list<crypto::hash>& qblock_ids, uint64_t& starter_offset) const;
-    bool find_blockchain_supplement(const uint64_t req_start_block, const std::list<crypto::hash>& qblock_ids, std::list<std::pair<block, std::list<transaction> > >& blocks, uint64_t& total_height, uint64_t& start_height, size_t max_count);
+    bool find_blockchain_supplement(const uint64_t req_start_block, const std::list<crypto::hash>& qblock_ids, std::list<std::pair<block, std::list<transaction> > >& blocks, uint64_t& total_height, uint64_t& start_height, size_t max_count) const;
     bool handle_get_objects(NOTIFY_REQUEST_GET_OBJECTS::request& arg, NOTIFY_RESPONSE_GET_OBJECTS::request& rsp);
     bool handle_get_objects(const COMMAND_RPC_GET_RANDOM_OUTPUTS_FOR_AMOUNTS::request& req, COMMAND_RPC_GET_RANDOM_OUTPUTS_FOR_AMOUNTS::response& res);
-    bool get_random_outs_for_amounts(const COMMAND_RPC_GET_RANDOM_OUTPUTS_FOR_AMOUNTS::request& req, COMMAND_RPC_GET_RANDOM_OUTPUTS_FOR_AMOUNTS::response& res);
+    bool get_random_outs_for_amounts(const COMMAND_RPC_GET_RANDOM_OUTPUTS_FOR_AMOUNTS::request& req, COMMAND_RPC_GET_RANDOM_OUTPUTS_FOR_AMOUNTS::response& res) const;
-    bool get_backward_blocks_sizes(size_t from_height, std::vector<size_t>& sz, size_t count);
+    bool get_backward_blocks_sizes(size_t from_height, std::vector<size_t>& sz, size_t count) const;
-    bool get_tx_outputs_gindexs(const crypto::hash& tx_id, std::vector<uint64_t>& indexs);
+    bool get_tx_outputs_gindexs(const crypto::hash& tx_id, std::vector<uint64_t>& indexs) const;
     bool store_blockchain();
-    bool check_tx_input(const txin_to_key& txin, const crypto::hash& tx_prefix_hash, const std::vector<crypto::signature>& sig, uint64_t* pmax_related_block_height = NULL);
+    bool check_tx_input(const txin_to_key& txin, const crypto::hash& tx_prefix_hash, const std::vector<crypto::signature>& sig, uint64_t* pmax_related_block_height = NULL) const;
-    bool check_tx_inputs(const transaction& tx, const crypto::hash& tx_prefix_hash, uint64_t* pmax_used_block_height = NULL);
+    bool check_tx_inputs(const transaction& tx, const crypto::hash& tx_prefix_hash, uint64_t* pmax_used_block_height = NULL) const;
-    bool check_tx_inputs(const transaction& tx, uint64_t* pmax_used_block_height = NULL);
+    bool check_tx_inputs(const transaction& tx, uint64_t* pmax_used_block_height = NULL) const;
-    bool check_tx_inputs(const transaction& tx, uint64_t& pmax_used_block_height, crypto::hash& max_used_block_id);
+    bool check_tx_inputs(const transaction& tx, uint64_t& pmax_used_block_height, crypto::hash& max_used_block_id) const;
-    uint64_t get_current_comulative_blocksize_limit();
+    uint64_t get_current_cumulative_blocksize_limit() const;
-    bool is_storing_blockchain(){return m_is_blockchain_storing;}
+    bool is_storing_blockchain()const{return m_is_blockchain_storing;}
-    uint64_t block_difficulty(size_t i);
+    uint64_t block_difficulty(size_t i) const;
-    double get_avg_block_size( size_t count);
+    double get_avg_block_size( size_t count) const;
 
     template<class t_ids_container, class t_blocks_container, class t_missed_container>
-    bool get_blocks(const t_ids_container& block_ids, t_blocks_container& blocks, t_missed_container& missed_bs)
+    bool get_blocks(const t_ids_container& block_ids, t_blocks_container& blocks, t_missed_container& missed_bs) const
     {
       CRITICAL_REGION_LOCAL(m_blockchain_lock);
 
@@ -157,7 +157,7 @@ namespace cryptonote
     }
 
     template<class t_ids_container, class t_tx_container, class t_missed_container>
-    bool get_transactions(const t_ids_container& txs_ids, t_tx_container& txs, t_missed_container& missed_txs)
+    bool get_transactions(const t_ids_container& txs_ids, t_tx_container& txs, t_missed_container& missed_txs) const
     {
       CRITICAL_REGION_LOCAL(m_blockchain_lock);
 
@@ -167,7 +167,7 @@ namespace cryptonote
         if(it == m_transactions.end())
         {
           transaction tx;
-          if(!m_tx_pool.get_transaction(tx_id, tx))
+          if(!m_tx_pool->get_transaction(tx_id, tx))
             missed_txs.push_back(tx_id);
           else
             txs.push_back(tx);
@@ -181,10 +181,15 @@ namespace cryptonote
     void print_blockchain(uint64_t start_index, uint64_t end_index);
     void print_blockchain_index();
     void print_blockchain_outs(const std::string& file);
-    void check_against_checkpoints(checkpoints& points, bool enforce);
+    void check_against_checkpoints(const checkpoints& points, bool enforce);
     bool update_checkpoints(const std::string& file_path, bool check_dns);
     void set_enforce_dns_checkpoints(bool enforce_checkpoints);
 
+    block get_block(uint64_t height) const { return m_blocks[height].bl; }
+    size_t get_block_size(uint64_t height) const { return m_blocks[height].block_cumulative_size; }
+    difficulty_type get_block_cumulative_difficulty(uint64_t height) const { return m_blocks[height].cumulative_difficulty; }
+    uint64_t get_block_coins_generated(uint64_t height) const { return m_blocks[height].already_generated_coins; }
+
   private:
     typedef std::unordered_map<crypto::hash, size_t> blocks_by_id_index;
     typedef std::unordered_map<crypto::hash, transaction_chain_entry> transactions_container;
@@ -194,8 +199,8 @@ namespace cryptonote
     typedef std::unordered_map<crypto::hash, block> blocks_by_hash;
     typedef std::map<uint64_t, std::vector<std::pair<crypto::hash, size_t>>> outputs_container; //crypto::hash - tx hash, size_t - index of out in transaction
 
-    tx_memory_pool& m_tx_pool;
+    tx_memory_pool* m_tx_pool;
-    epee::critical_section m_blockchain_lock; // TODO: add here reader/writer lock
+    mutable epee::critical_section m_blockchain_lock; // TODO: add here reader/writer lock
 
     // main chain
     blocks_container m_blocks;               // height  -> block_extended_info
@@ -230,26 +235,26 @@ namespace cryptonote
     bool handle_block_to_main_chain(const block& bl, block_verification_context& bvc);
     bool handle_block_to_main_chain(const block& bl, const crypto::hash& id, block_verification_context& bvc);
     bool handle_alternative_block(const block& b, const crypto::hash& id, block_verification_context& bvc);
-    difficulty_type get_next_difficulty_for_alternative_chain(const std::list<blocks_ext_by_hash::iterator>& alt_chain, block_extended_info& bei);
+    difficulty_type get_next_difficulty_for_alternative_chain(const std::list<blocks_ext_by_hash::iterator>& alt_chain, block_extended_info& bei) const;
-    bool prevalidate_miner_transaction(const block& b, uint64_t height);
+    bool prevalidate_miner_transaction(const block& b, uint64_t height) const;
-    bool validate_miner_transaction(const block& b, size_t cumulative_block_size, uint64_t fee, uint64_t& base_reward, uint64_t already_generated_coins);
+    bool validate_miner_transaction(const block& b, size_t cumulative_block_size, uint64_t fee, uint64_t& base_reward, uint64_t already_generated_coins) const;
-    bool validate_transaction(const block& b, uint64_t height, const transaction& tx);
+    bool validate_transaction(const block& b, uint64_t height, const transaction& tx) const;
     bool rollback_blockchain_switching(std::list<block>& original_chain, size_t rollback_height);
     bool add_transaction_from_block(const transaction& tx, const crypto::hash& tx_id, const crypto::hash& bl_id, uint64_t bl_height, size_t blob_size);
     bool push_transaction_to_global_outs_index(const transaction& tx, const crypto::hash& tx_id, std::vector<uint64_t>& global_indexes);
     bool pop_transaction_from_global_index(const transaction& tx, const crypto::hash& tx_id);
-    bool get_last_n_blocks_sizes(std::vector<size_t>& sz, size_t count);
+    bool get_last_n_blocks_sizes(std::vector<size_t>& sz, size_t count) const;
-    bool add_out_to_get_random_outs(std::vector<std::pair<crypto::hash, size_t> >& amount_outs, COMMAND_RPC_GET_RANDOM_OUTPUTS_FOR_AMOUNTS::outs_for_amount& result_outs, uint64_t amount, size_t i);
+    bool add_out_to_get_random_outs(const std::vector<std::pair<crypto::hash, size_t> >& amount_outs, COMMAND_RPC_GET_RANDOM_OUTPUTS_FOR_AMOUNTS::outs_for_amount& result_outs, uint64_t amount, size_t i) const;
-    bool is_tx_spendtime_unlocked(uint64_t unlock_time);
+    bool is_tx_spendtime_unlocked(uint64_t unlock_time) const;
     bool add_block_as_invalid(const block& bl, const crypto::hash& h);
     bool add_block_as_invalid(const block_extended_info& bei, const crypto::hash& h);
-    size_t find_end_of_allowed_index(const std::vector<std::pair<crypto::hash, size_t> >& amount_outs);
+    size_t find_end_of_allowed_index(const std::vector<std::pair<crypto::hash, size_t> >& amount_outs) const;
-    bool check_block_timestamp_main(const block& b);
+    bool check_block_timestamp_main(const block& b) const;
-    bool check_block_timestamp(std::vector<uint64_t> timestamps, const block& b);
+    bool check_block_timestamp(std::vector<uint64_t> timestamps, const block& b) const;
-    uint64_t get_adjusted_time();
+    uint64_t get_adjusted_time() const;
-    bool complete_timestamps_vector(uint64_t start_height, std::vector<uint64_t>& timestamps);
+    bool complete_timestamps_vector(uint64_t start_height, std::vector<uint64_t>& timestamps) const;
     bool update_next_comulative_size_limit();
-    bool store_genesis_block(bool testnet);
+    bool store_genesis_block(bool testnet, bool check_added = false);
   };
 
 
@@ -317,7 +322,7 @@ namespace cryptonote
 
   //------------------------------------------------------------------
   template<class visitor_t>
-  bool blockchain_storage::scan_outputkeys_for_indexes(const txin_to_key& tx_in_to_key, visitor_t& vis, uint64_t* pmax_related_block_height)
+  bool blockchain_storage::scan_outputkeys_for_indexes(const txin_to_key& tx_in_to_key, visitor_t& vis, uint64_t* pmax_related_block_height) const
   {
     CRITICAL_REGION_LOCAL(m_blockchain_lock);
     auto it = m_outputs.find(tx_in_to_key.amount);
@@ -327,7 +332,7 @@ namespace cryptonote
     std::vector<uint64_t> absolute_offsets = relative_output_offsets_to_absolute(tx_in_to_key.key_offsets);
 
 
-    std::vector<std::pair<crypto::hash, size_t> >& amount_outs_vec = it->second;
+    const std::vector<std::pair<crypto::hash, size_t> >& amount_outs_vec = it->second;
     size_t count = 0;
     BOOST_FOREACH(uint64_t i, absolute_offsets)
     {
@@ -336,7 +341,7 @@ namespace cryptonote
         LOG_PRINT_L0("Wrong index in transaction inputs: " << i << ", expected maximum " << amount_outs_vec.size() - 1);
         return false;
       }
-      transactions_container::iterator tx_it = m_transactions.find(amount_outs_vec[i].first);
+      transactions_container::const_iterator tx_it = m_transactions.find(amount_outs_vec[i].first);
       CHECK_AND_ASSERT_MES(tx_it != m_transactions.end(), false, "Wrong transaction id in output indexes: " << epee::string_tools::pod_to_hex(amount_outs_vec[i].first));
       CHECK_AND_ASSERT_MES(amount_outs_vec[i].second < tx_it->second.tx.vout.size(), false,
         "Wrong index in transaction outputs: " << amount_outs_vec[i].second << ", expected less then " << tx_it->second.tx.vout.size());

src/cryptonote_core/checkpoints.cpp

@@ -84,6 +84,10 @@ namespace cryptonote
     return check_block(height, h, ignored);
   }
   //---------------------------------------------------------------------------
+  // this basically says if the blockchain is smaller than the first
+  // checkpoint then alternate blocks are allowed.  Alternatively, if the
+  // last checkpoint *before* the end of the current chain is also before
+  // the block to be added, then this is fine.
   bool checkpoints::is_alternative_block_allowed(uint64_t blockchain_height, uint64_t block_height) const
   {
     if (0 == block_height)
@@ -99,7 +103,7 @@ namespace cryptonote
     return checkpoint_height < block_height;
   }
   //---------------------------------------------------------------------------
-  uint64_t checkpoints::get_max_height()
+  uint64_t checkpoints::get_max_height() const
   {
     std::map< uint64_t, crypto::hash >::const_iterator highest = 
         std::max_element( m_points.begin(), m_points.end(),
@@ -108,18 +112,18 @@ namespace cryptonote
     return highest->first;
   }
   //---------------------------------------------------------------------------
-  const std::map<uint64_t, crypto::hash>& checkpoints::get_points()
+  const std::map<uint64_t, crypto::hash>& checkpoints::get_points() const
   {
     return m_points;
   }
 
-  bool checkpoints::check_for_conflicts(checkpoints& other)
+  bool checkpoints::check_for_conflicts(const checkpoints& other) const
   {
     for (auto& pt : other.get_points())
     {
       if (m_points.count(pt.first))
       {
-        CHECK_AND_ASSERT_MES(pt.second == m_points[pt.first], false, "Checkpoint at given height already exists, and hash for new checkpoint was different!");
+        CHECK_AND_ASSERT_MES(pt.second == m_points.at(pt.first), false, "Checkpoint at given height already exists, and hash for new checkpoint was different!");
       }
     }
     return true;

src/cryptonote_core/checkpoints.h

@@ -45,9 +45,9 @@ namespace cryptonote
     bool check_block(uint64_t height, const crypto::hash& h) const;
     bool check_block(uint64_t height, const crypto::hash& h, bool& is_a_checkpoint) const;
     bool is_alternative_block_allowed(uint64_t blockchain_height, uint64_t block_height) const;
-    uint64_t get_max_height();
+    uint64_t get_max_height() const;
-    const std::map<uint64_t, crypto::hash>& get_points();
+    const std::map<uint64_t, crypto::hash>& get_points() const;
-    bool check_for_conflicts(checkpoints& other);
+    bool check_for_conflicts(const checkpoints& other) const;
   private:
     std::map<uint64_t, crypto::hash> m_points;
   };

src/cryptonote_core/cryptonote_basic.h

@@ -50,6 +50,8 @@
 #include "misc_language.h"
 #include "tx_extra.h"
 
+#define DB_MEMORY 1
+#define DB_LMDB   2
 
 namespace cryptonote
 {

src/cryptonote_core/cryptonote_core.cpp

@@ -44,6 +44,11 @@ using namespace epee;
 #include <csignal>
 #include "daemon/command_line_args.h"
 #include "cryptonote_core/checkpoints_create.h"
+#include "blockchain_db/blockchain_db.h"
+#include "blockchain_db/lmdb/db_lmdb.h"
+#ifndef STATICLIB
+#include "blockchain_db/berkeleydb/db_bdb.h"
+#endif
 
 DISABLE_VS_WARNINGS(4355)
 
@@ -53,7 +58,11 @@ namespace cryptonote
   //-----------------------------------------------------------------------------------------------
   core::core(i_cryptonote_protocol* pprotocol):
               m_mempool(m_blockchain_storage),
+#if BLOCKCHAIN_DB == DB_LMDB
               m_blockchain_storage(m_mempool),
+#else
+              m_blockchain_storage(&m_mempool),
+#endif
               m_miner(this),
               m_miner_address(boost::value_initialized<account_public_address>()), 
               m_starter_message_showed(false),
@@ -195,7 +204,50 @@ namespace cryptonote
     r = m_mempool.init(m_config_folder);
     CHECK_AND_ASSERT_MES(r, false, "Failed to initialize memory pool");
 
+#if BLOCKCHAIN_DB == DB_LMDB
+    std::string db_type = command_line::get_arg(vm, daemon_args::arg_db_type);
+
+    BlockchainDB* db = nullptr;
+    if (db_type == "lmdb")
+    {
+      db = new BlockchainLMDB();
+    }
+    else if (db_type == "berkeley")
+    {
+#ifndef STATICLIB
+      db = new BlockchainBDB();
+#else
+      LOG_ERROR("BlockchainBDB not supported on STATIC builds");
+      return false;
+#endif
+    }
+    else
+    {
+      LOG_ERROR("Attempted to use non-existant database type");
+      return false;
+    }
+
+    boost::filesystem::path folder(m_config_folder);
+
+    folder /= db->get_db_name();
+
+    LOG_PRINT_L0("Loading blockchain from folder " << folder.c_str() << " ...");
+
+    const std::string filename = folder.string();
+    try
+    {
+      db->open(filename);
+    }
+    catch (const DB_ERROR& e)
+    {
+      LOG_PRINT_L0("Error opening database: " << e.what());
+      return false;
+    }
+
+    r = m_blockchain_storage.init(db, m_testnet);
+#else
     r = m_blockchain_storage.init(m_config_folder, m_testnet);
+#endif
     CHECK_AND_ASSERT_MES(r, false, "Failed to initialize blockchain storage");
 
     // load json & DNS checkpoints, and verify them
@@ -363,9 +415,9 @@ namespace cryptonote
       return false;
     }
 
-    if(!keeped_by_block && get_object_blobsize(tx) >= m_blockchain_storage.get_current_comulative_blocksize_limit() - CRYPTONOTE_COINBASE_BLOB_RESERVED_SIZE)
+    if(!keeped_by_block && get_object_blobsize(tx) >= m_blockchain_storage.get_current_cumulative_blocksize_limit() - CRYPTONOTE_COINBASE_BLOB_RESERVED_SIZE)
     {
-      LOG_PRINT_RED_L1("tx is too large " << get_object_blobsize(tx) << ", expected not bigger than " << m_blockchain_storage.get_current_comulative_blocksize_limit() - CRYPTONOTE_COINBASE_BLOB_RESERVED_SIZE);
+      LOG_PRINT_RED_L1("tx is too large " << get_object_blobsize(tx) << ", expected not bigger than " << m_blockchain_storage.get_current_cumulative_blocksize_limit() - CRYPTONOTE_COINBASE_BLOB_RESERVED_SIZE);
       return false;
     }
 
@@ -650,7 +702,11 @@ namespace cryptonote
       m_starter_message_showed = true;
     }
 
+#if BLOCKCHAIN_DB == DB_LMDB
+    m_store_blockchain_interval.do_call(boost::bind(&Blockchain::store_blockchain, &m_blockchain_storage));
+#else
     m_store_blockchain_interval.do_call(boost::bind(&blockchain_storage::store_blockchain, &m_blockchain_storage));
+#endif
     m_miner.on_idle();
     m_mempool.on_idle();
     return true;

src/cryptonote_core/cryptonote_core.h

@@ -39,7 +39,11 @@
 #include "cryptonote_protocol/cryptonote_protocol_handler_common.h"
 #include "storages/portable_storage_template_helper.h"
 #include "tx_pool.h"
+#if BLOCKCHAIN_DB == DB_LMDB
+#include "blockchain.h"
+#else
 #include "blockchain_storage.h"
+#endif
 #include "miner.h"
 #include "connection_context.h"
 #include "cryptonote_core/cryptonote_stat_info.h"
@@ -118,7 +122,11 @@ namespace cryptonote
      bool get_random_outs_for_amounts(const COMMAND_RPC_GET_RANDOM_OUTPUTS_FOR_AMOUNTS::request& req, COMMAND_RPC_GET_RANDOM_OUTPUTS_FOR_AMOUNTS::response& res);
      void pause_mine();
      void resume_mine();
+#if BLOCKCHAIN_DB == DB_LMDB
+     Blockchain& get_blockchain_storage(){return m_blockchain_storage;}
+#else
      blockchain_storage& get_blockchain_storage(){return m_blockchain_storage;}
+#endif
      //debug functions
      void print_blockchain(uint64_t start_index, uint64_t end_index);
      void print_blockchain_index();
@@ -159,7 +167,11 @@ namespace cryptonote
 	 uint64_t m_test_drop_download_height = 0;
 
      tx_memory_pool m_mempool;
+#if BLOCKCHAIN_DB == DB_LMDB
+     Blockchain m_blockchain_storage;
+#else
      blockchain_storage m_blockchain_storage;
+#endif
      i_cryptonote_protocol* m_pprotocol;
      epee::critical_section m_incoming_tx_lock;
      //m_miner and m_miner_addres are probably temporary here

src/cryptonote_core/tx_pool.cpp

@@ -37,7 +37,11 @@
 #include "cryptonote_format_utils.h"
 #include "cryptonote_boost_serialization.h"
 #include "cryptonote_config.h"
+#if BLOCKCHAIN_DB == DB_LMDB
+#include "blockchain.h"
+#else
 #include "blockchain_storage.h"
+#endif
 #include "common/boost_serialization_helper.h"
 #include "common/int-util.h"
 #include "misc_language.h"
@@ -52,12 +56,19 @@ namespace cryptonote
   {
     size_t const TRANSACTION_SIZE_LIMIT = (((CRYPTONOTE_BLOCK_GRANTED_FULL_REWARD_ZONE * 125) / 100) - CRYPTONOTE_COINBASE_BLOB_RESERVED_SIZE);
   }
-
   //---------------------------------------------------------------------------------
+#if BLOCKCHAIN_DB == DB_LMDB
+  //---------------------------------------------------------------------------------
+  tx_memory_pool::tx_memory_pool(Blockchain& bchs): m_blockchain(bchs)
+  {
+
+  }
+#else
   tx_memory_pool::tx_memory_pool(blockchain_storage& bchs): m_blockchain(bchs)
   {
 
   }
+#endif
   //---------------------------------------------------------------------------------
   bool tx_memory_pool::add_tx(const transaction &tx, /*const crypto::hash& tx_prefix_hash,*/ const crypto::hash &id, size_t blob_size, tx_verification_context& tvc, bool kept_by_block)
   {

src/cryptonote_core/tx_pool.h

@@ -44,10 +44,13 @@
 #include "verification_context.h"
 #include "crypto/hash.h"
 
-
 namespace cryptonote
 {
+#if BLOCKCHAIN_DB == DB_LMDB
+  class Blockchain;
+#else
   class blockchain_storage;
+#endif
   /************************************************************************/
   /*                                                                      */
   /************************************************************************/
@@ -55,7 +58,11 @@ namespace cryptonote
   class tx_memory_pool: boost::noncopyable
   {
   public:
+#if BLOCKCHAIN_DB == DB_LMDB
+    tx_memory_pool(Blockchain& bchs);
+#else
     tx_memory_pool(blockchain_storage& bchs);
+#endif
     bool add_tx(const transaction &tx, const crypto::hash &id, size_t blob_size, tx_verification_context& tvc, bool keeped_by_block);
     bool add_tx(const transaction &tx, tx_verification_context& tvc, bool keeped_by_block);
     //gets tx and remove it from pool
@@ -127,7 +134,11 @@ namespace cryptonote
     //transactions_container m_alternative_transactions;
 
     std::string m_config_folder;
+#if BLOCKCHAIN_DB == DB_LMDB
+    Blockchain& m_blockchain;
+#else
     blockchain_storage& m_blockchain;
+#endif
     /************************************************************************/
     /*                                                                      */
     /************************************************************************/
@@ -170,8 +181,11 @@ namespace cryptonote
       uint64_t operator()(const txin_to_scripthash& tx) const {return 0;}
     };
 
+#if BLOCKCHAIN_DB == DB_LMDB
+#else
 #if defined(DEBUG_CREATE_BLOCK_TEMPLATE)
     friend class blockchain_storage;
+#endif
 #endif
   };
 }

src/daemon/CMakeLists.txt

@@ -74,6 +74,7 @@ bitmonero_add_executable(daemon
 target_link_libraries(daemon
   LINK_PRIVATE
     rpc
+    blockchain_db
     cryptonote_core
     crypto
     common

src/daemon/command_line_args.h

@@ -70,6 +70,11 @@ namespace daemon_args
   , "checkpoints from DNS server will be enforced"
   , false
   };
+  const command_line::arg_descriptor<std::string> arg_db_type = {
+    "db-type"
+  , "Specify database type"
+  , "lmdb"
+  };
 
 }  // namespace daemon_args
 

src/daemon/main.cpp

@@ -42,6 +42,7 @@
 #include "rpc/core_rpc_server.h"
 #include <boost/program_options.hpp>
 #include "daemon/command_line_args.h"
+#include "blockchain_db/db_types.h"
 
 namespace po = boost::program_options;
 namespace bf = boost::filesystem;
@@ -84,6 +85,7 @@ int main(int argc, char const * argv[])
       command_line::add_arg(core_settings, daemon_args::arg_log_level);
       command_line::add_arg(core_settings, daemon_args::arg_testnet_on);
       command_line::add_arg(core_settings, daemon_args::arg_dns_checkpoints);
+      command_line::add_arg(core_settings, daemon_args::arg_db_type);
       daemonizer::init_options(hidden_options, visible_options);
       daemonize::t_executor::init_options(core_settings);
 
@@ -136,6 +138,19 @@ int main(int argc, char const * argv[])
     
     epee::g_test_dbg_lock_sleep = command_line::get_arg(vm, command_line::arg_test_dbg_lock_sleep);
 
+    std::string db_type = command_line::get_arg(vm, daemon_args::arg_db_type);
+
+    // verify that blockchaindb type is valid
+    if(cryptonote::blockchain_db_types.count(db_type) == 0)
+    {
+      std::cout << "Invalid database type (" << db_type << "), available types are:" << std::endl;
+      for (const auto& type : cryptonote::blockchain_db_types)
+      {
+        std::cout << "\t" << type << std::endl;
+      }
+      return 0;
+    }
+
     bool testnet_mode = command_line::get_arg(vm, daemon_args::arg_testnet_on);
 
     auto data_dir_arg = testnet_mode ? command_line::arg_testnet_data_dir : command_line::arg_data_dir;

tests/unit_tests/BlockchainDB.cpp

@@ -0,0 +1,325 @@
+// Copyright (c) 2014, The Monero Project
+// 
+// All rights reserved.
+// 
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+// 
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+// 
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+// 
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+// 
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#include <boost/filesystem.hpp>
+#include <boost/algorithm/string/predicate.hpp>
+#include <cstdio>
+#include <iostream>
+#include <chrono>
+#include <thread>
+
+#include "gtest/gtest.h"
+
+#include "blockchain_db/blockchain_db.h"
+#include "blockchain_db/lmdb/db_lmdb.h"
+#include "blockchain_db/berkeleydb/db_bdb.h"
+#include "cryptonote_core/cryptonote_format_utils.h"
+
+using namespace cryptonote;
+using epee::string_tools::pod_to_hex;
+
+#define ASSERT_HASH_EQ(a,b) ASSERT_EQ(pod_to_hex(a), pod_to_hex(b))
+
+namespace {  // anonymous namespace
+
+const std::vector<std::string> t_blocks =
+  {
+    "0100d5adc49a053b8818b2b6023cd2d532c6774e164a8fcacd603651cb3ea0cb7f9340b28ec016b4bc4ca301aa0101ff6e08acbb2702eab03067870349139bee7eab2ca2e030a6bb73d4f68ab6a3b6ca937214054cdac0843d028bbe23b57ea9bae53f12da93bb57bf8a2e40598d9fccd10c2921576e987d93cd80b4891302468738e391f07c4f2b356f7957160968e0bfef6e907c3cee2d8c23cbf04b089680c6868f01025a0f41f063e195a966051e3a29e17130a9ce97d48f55285b9bb04bdd55a09ae78088aca3cf0202d0f26169290450fe17e08974789c3458910b4db18361cdc564f8f2d0bdd2cf568090cad2c60e02d6f3483ec45505cc3be841046c7a12bf953ac973939bc7b727e54258e1881d4d80e08d84ddcb0102dae6dfb16d3e28aaaf43e00170b90606b36f35f38f8a3dceb5ee18199dd8f17c80c0caf384a30202385d7e57a4daba4cdd9e550a92dcc188838386e7581f13f09de796cbed4716a42101c052492a077abf41996b50c1b2e67fd7288bcd8c55cdc657b4e22d0804371f6901beb76a82ea17400cd6d7f595f70e1667d2018ed8f5a78d1ce07484222618c3cd"
+  , "0100f9adc49a057d3113f562eac36f14afa08c22ae20bbbf8cffa31a4466d24850732cb96f80e9762365ee01ab0101ff6f08cc953502be76deb845c431f2ed9a4862457654b914003693b8cd672abc935f0d97b16380c08db7010291819f2873e3efbae65ecd5a736f5e8a26318b591c21e39a03fb536520ac63ba80dac40902439a10fde02e39e48e0b31e57cc084a07eedbefb8cbea0143aedd0442b189caa80c6868f010227b84449de4cd7a48cbdce8974baf0b6646e03384e32055e705c243a86bef8a58088aca3cf0202fa7bd15e4e7e884307ab130bb9d50e33c5fcea6546042a26f948efd5952459ee8090cad2c60e028695583dbb8f8faab87e3ef3f88fa827db097bbf51761d91924f5c5b74c6631780e08d84ddcb010279d2f247b54690e3b491e488acff16014a825fd740c23988a25df7c4670c1f2580c0caf384a302022599dfa3f8788b66295051d85937816e1c320cdb347a0fba5219e3fe60c83b2421010576509c5672025d28fd5d3f38efce24e1f9aaf65dd3056b2504e6e2b7f19f7800"
+  };
+
+const std::vector<size_t> t_sizes =
+  {
+    1122
+  , 347
+  };
+
+const std::vector<difficulty_type> t_diffs =
+  {
+    4003674
+  , 4051757
+  };
+
+const std::vector<uint64_t> t_coins =
+  {
+    1952630229575370
+  , 1970220553446486
+  };
+
+const std::vector<std::vector<std::string>> t_transactions =
+  {
+    {
+      "0100010280e08d84ddcb0106010401110701f254220bb50d901a5523eaed438af5d43f8c6d0e54ba0632eb539884f6b7c02008c0a8a50402f9c7cf807ae74e56f4ec84db2bd93cfb02c2249b38e306f5b54b6e05d00d543b8095f52a02b6abb84e00f47f0a72e37b6b29392d906a38468404c57db3dbc5e8dd306a27a880d293ad0302cfc40a86723e7d459e90e45d47818dc0e81a1f451ace5137a4af8110a89a35ea80b4c4c321026b19c796338607d5a2c1ba240a167134142d72d1640ef07902da64fed0b10cfc8088aca3cf02021f6f655254fee84161118b32e7b6f8c31de5eb88aa00c29a8f57c0d1f95a24dd80d0b8e1981a023321af593163cea2ae37168ab926efd87f195756e3b723e886bdb7e618f751c480a094a58d1d0295ed2b08d1cf44482ae0060a5dcc4b7d810a85dea8c62e274f73862f3d59f8ed80a0e5b9c2910102dc50f2f28d7ceecd9a1147f7106c8d5b4e08b2ec77150f52dd7130ee4f5f50d42101d34f90ac861d0ee9fe3891656a234ea86a8a93bf51a237db65baa00d3f4aa196a9e1d89bc06b40e94ea9a26059efc7ba5b2de7ef7c139831ca62f3fe0bb252008f8c7ee810d3e1e06313edf2db362fc39431755779466b635f12f9f32e44470a3e85e08a28fcd90633efc94aa4ae39153dfaf661089d045521343a3d63e8da08d7916753c66aaebd4eefcfe8e58e5b3d266b752c9ca110749fa33fce7c44270386fcf2bed4f03dd5dadb2dc1fd4c505419f8217b9eaec07521f0d8963e104603c926745039cf38d31de6ed95ace8e8a451f5a36f818c151f517546d55ac0f500e54d07b30ea7452f2e93fa4f60bdb30d71a0a97f97eb121e662006780fbf69002228224a96bff37893d47ec3707b17383906c0cd7d9e7412b3e6c8ccf1419b093c06c26f96e3453b424713cdc5c9575f81cda4e157052df11f4c40809edf420f88a3dd1f7909bbf77c8b184a933389094a88e480e900bcdbf6d1824742ee520fc0032e7d892a2b099b8c6edfd1123ce58a34458ee20cad676a7f7cfd80a28f0cb0888af88838310db372986bdcf9bfcae2324480ca7360d22bff21fb569a530e"
+    }
+  , {
+    }
+  };
+
+// if the return type (blobdata for now) of block_to_blob ever changes
+// from std::string, this might break.
+bool compare_blocks(const block& a, const block& b)
+{
+  auto hash_a = pod_to_hex(get_block_hash(a));
+  auto hash_b = pod_to_hex(get_block_hash(b));
+
+  return hash_a == hash_b;
+}
+
+/*
+void print_block(const block& blk, const std::string& prefix = "")
+{
+  std::cerr << prefix << ": " << std::endl
+            << "\thash - " << pod_to_hex(get_block_hash(blk)) << std::endl
+            << "\tparent - " << pod_to_hex(blk.prev_id) << std::endl
+            << "\ttimestamp - " << blk.timestamp << std::endl
+  ;
+}
+
+// if the return type (blobdata for now) of tx_to_blob ever changes
+// from std::string, this might break.
+bool compare_txs(const transaction& a, const transaction& b)
+{
+  auto ab = tx_to_blob(a);
+  auto bb = tx_to_blob(b);
+
+  return ab == bb;
+}
+*/
+
+// convert hex string to string that has values based on that hex
+// thankfully should automatically ignore null-terminator.
+std::string h2b(const std::string& s)
+{
+  bool upper = true;
+  std::string result;
+  unsigned char val = 0;
+  for (char c : s)
+  {
+    if (upper)
+    {
+      val = 0;
+      if (c <= 'f' && c >= 'a')
+      {
+        val = ((c - 'a') + 10) << 4;
+      }
+      else
+      {
+        val = (c - '0') << 4;
+      }
+    }
+    else
+    {
+      if (c <= 'f' && c >= 'a')
+      {
+        val |= (c - 'a') + 10;
+      }
+      else
+      {
+        val |= c - '0';
+      }
+      result += (char)val;
+    }
+    upper = !upper;
+  }
+  return result;
+}
+
+template <typename T>
+class BlockchainDBTest : public testing::Test
+{
+protected:
+  BlockchainDBTest() : m_db(new T())
+  {
+    for (auto& i : t_blocks)
+    {
+      block bl;
+      blobdata bd = h2b(i);
+      parse_and_validate_block_from_blob(bd, bl);
+      m_blocks.push_back(bl);
+    }
+    for (auto& i : t_transactions)
+    {
+      std::vector<transaction> txs;
+      for (auto& j : i)
+      {
+        transaction tx;
+        blobdata bd = h2b(j);
+        parse_and_validate_tx_from_blob(bd, tx);
+        txs.push_back(tx);
+      }
+      m_txs.push_back(txs);
+    }
+  }
+
+  ~BlockchainDBTest() {
+    delete m_db;
+    remove_files();
+  }
+
+  BlockchainDB* m_db;
+  std::string m_prefix;
+  std::vector<block> m_blocks;
+  std::vector<std::vector<transaction> > m_txs;
+  std::vector<std::string> m_filenames;
+
+  void get_filenames()
+  {
+    m_filenames = m_db->get_filenames();
+    for (auto& f : m_filenames)
+    {
+      std::cerr << "File created by test: " << f << std::endl;
+    }
+  }
+
+  void remove_files()
+  {
+    // remove each file the db created, making sure it starts with fname.
+    for (auto& f : m_filenames)
+    {
+      if (boost::starts_with(f, m_prefix))
+      {
+        boost::filesystem::remove(f);
+      }
+      else
+      {
+        std::cerr << "File created by test not to be removed (for safety): " << f << std::endl;
+      }
+    }
+
+    // remove directory if it still exists
+    boost::filesystem::remove_all(m_prefix);
+  }
+
+  void set_prefix(const std::string& prefix)
+  {
+    m_prefix = prefix;
+  }
+};
+
+using testing::Types;
+
+typedef Types<BlockchainLMDB, BlockchainBDB> implementations;
+
+TYPED_TEST_CASE(BlockchainDBTest, implementations);
+
+TYPED_TEST(BlockchainDBTest, OpenAndClose)
+{
+  std::string fname(tmpnam(NULL));
+
+  this->set_prefix(fname);
+
+  // make sure open does not throw
+  ASSERT_NO_THROW(this->m_db->open(fname));
+  this->get_filenames();
+
+  // make sure open when already open DOES throw
+  ASSERT_THROW(this->m_db->open(fname), DB_OPEN_FAILURE);
+
+  ASSERT_NO_THROW(this->m_db->close());
+}
+
+TYPED_TEST(BlockchainDBTest, AddBlock)
+{
+  std::string fname(tmpnam(NULL));
+  this->set_prefix(fname);
+
+  // make sure open does not throw
+  ASSERT_NO_THROW(this->m_db->open(fname));
+  this->get_filenames();
+
+  // adding a block with no parent in the blockchain should throw.
+  // note: this shouldn't be possible, but is a good (and cheap) failsafe.
+  //
+  // TODO: need at least one more block to make this reasonable, as the
+  // BlockchainDB implementation should not check for parent if
+  // no blocks have been added yet (because genesis has no parent).
+  //ASSERT_THROW(this->m_db->add_block(this->m_blocks[1], t_sizes[1], t_diffs[1], t_coins[1], this->m_txs[1]), BLOCK_PARENT_DNE);
+
+  ASSERT_NO_THROW(this->m_db->add_block(this->m_blocks[0], t_sizes[0], t_diffs[0], t_coins[0], this->m_txs[0]));
+  ASSERT_NO_THROW(this->m_db->add_block(this->m_blocks[1], t_sizes[1], t_diffs[1], t_coins[1], this->m_txs[1]));
+
+  block b;
+  ASSERT_TRUE(this->m_db->block_exists(get_block_hash(this->m_blocks[0])));
+  ASSERT_NO_THROW(b = this->m_db->get_block(get_block_hash(this->m_blocks[0])));
+
+  ASSERT_TRUE(compare_blocks(this->m_blocks[0], b));
+
+  ASSERT_NO_THROW(b = this->m_db->get_block_from_height(0));
+
+  ASSERT_TRUE(compare_blocks(this->m_blocks[0], b));
+
+  // assert that we can't add the same block twice
+  ASSERT_THROW(this->m_db->add_block(this->m_blocks[0], t_sizes[0], t_diffs[0], t_coins[0], this->m_txs[0]), BLOCK_EXISTS);
+
+  for (auto& h : this->m_blocks[0].tx_hashes)
+  {
+    transaction tx;
+    ASSERT_TRUE(this->m_db->tx_exists(h));
+    ASSERT_NO_THROW(tx = this->m_db->get_tx(h));
+
+    ASSERT_HASH_EQ(h, get_transaction_hash(tx));
+  }
+}
+
+TYPED_TEST(BlockchainDBTest, RetrieveBlockData)
+{
+  std::string fname(tmpnam(NULL));
+  this->set_prefix(fname);
+
+  // make sure open does not throw
+  ASSERT_NO_THROW(this->m_db->open(fname));
+  this->get_filenames();
+
+  ASSERT_NO_THROW(this->m_db->add_block(this->m_blocks[0], t_sizes[0], t_diffs[0], t_coins[0], this->m_txs[0]));
+
+  ASSERT_EQ(t_sizes[0], this->m_db->get_block_size(0));
+  ASSERT_EQ(t_diffs[0], this->m_db->get_block_cumulative_difficulty(0));
+  ASSERT_EQ(t_diffs[0], this->m_db->get_block_difficulty(0));
+  ASSERT_EQ(t_coins[0], this->m_db->get_block_already_generated_coins(0));
+
+  ASSERT_NO_THROW(this->m_db->add_block(this->m_blocks[1], t_sizes[1], t_diffs[1], t_coins[1], this->m_txs[1]));
+  ASSERT_EQ(t_diffs[1] - t_diffs[0], this->m_db->get_block_difficulty(1));
+
+  ASSERT_HASH_EQ(get_block_hash(this->m_blocks[0]), this->m_db->get_block_hash_from_height(0));
+
+  std::vector<block> blks;
+  ASSERT_NO_THROW(blks = this->m_db->get_blocks_range(0, 1));
+  ASSERT_EQ(2, blks.size());
+  
+  ASSERT_HASH_EQ(get_block_hash(this->m_blocks[0]), get_block_hash(blks[0]));
+  ASSERT_HASH_EQ(get_block_hash(this->m_blocks[1]), get_block_hash(blks[1]));
+
+  std::vector<crypto::hash> hashes;
+  ASSERT_NO_THROW(hashes = this->m_db->get_hashes_range(0, 1));
+  ASSERT_EQ(2, hashes.size());
+
+  ASSERT_HASH_EQ(get_block_hash(this->m_blocks[0]), hashes[0]);
+  ASSERT_HASH_EQ(get_block_hash(this->m_blocks[1]), hashes[1]);
+}
+
+}  // anonymous namespace

tests/unit_tests/CMakeLists.txt

@@ -29,6 +29,7 @@
 set(unit_tests_sources
   address_from_url.cpp
   base58.cpp
+  BlockchainDB.cpp
   block_reward.cpp
   chacha8.cpp
   checkpoints.cpp
@@ -56,6 +57,7 @@ add_executable(unit_tests
 target_link_libraries(unit_tests
   LINK_PRIVATE
     cryptonote_core
+    blockchain_db
     rpc
     wallet
     p2p