1. leveldb 的部署和使用
1.1. 部署
部署比较简单，按照官网即可
1.1.1. C++
git clone --recurse-submodules https://github.com/google/leveldb.git
mkdir -p build && cd build
cmake -DCMAKE_BUILD_TYPE=Release .. && cmake --build .
make install
1.1.2. python
pip install leveldb
1.1.3. 小插曲
开始用 mac 部署，遇到挺多问题，不管是brew安装还是编译安装都有
leveldb/db.h file not found
实际上在 /usr/local/include/leveldb/db.h 是存在的
stat /usr/local/include/leveldb/db.h
16777221 47682495 -rw-r--r-- 1 root admin 0 6803 "Apr 5 14:55:15 2021" "Apr 5 14:52:17 2021" "Apr 5 14:55:15 2021" "Apr 5 14:52:17 2021" 4096 16 0 /usr/local/include/leveldb/db.h
起初一直怀疑是因为路径问题，查看路径是正确的
clang -x c -v -E /dev/null
# ---
Apple clang version 12.0.0 (clang-1200.0.32.29)
Target: x86_64-apple-darwin19.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
"/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang" -cc1 -triple x86_64-apple-macosx10.15.0 -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -Werror=implicit-function-declaration -E -disable-free -disable-llvm-verifier -discard-value-names -main-file-name null -mrelocation-model pic -pic-level 2 -mthread-model posix -mframe-pointer=all -fno-strict-return -masm-verbose -munwind-tables -target-sdk-version=10.15.6 -fcompatibility-qualified-id-block-type-checking -target-cpu penryn -dwarf-column-info -debugger-tuning=lldb -target-linker-version 609.8 -v -resource-dir /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -I/usr/local/include -internal-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/local/include -internal-isystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0/include -internal-externc-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include -internal-externc-isystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include -Wno-reorder-init-list -Wno-implicit-int-float-conversion -Wno-c99-designator -Wno-final-dtor-non-final-class -Wno-extra-semi-stmt -Wno-misleading-indentation -Wno-quoted-include-in-framework-header -Wno-implicit-fallthrough -Wno-enum-enum-conversion -Wno-enum-float-conversion -fdebug-compilation-dir /Users/wanggangfeng/work/tstack-code/practice/cpp-prectice/leveldbtest -ferror-limit 19 -fmessage-length 178 -stack-protector 1 -fstack-check -mdarwin-stkchk-strong-link -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fgnuc-version=4.2.1 -fobjc-runtime=macosx-10.15.0 -fmax-type-align=16 -fdiagnostics-show-option -fcolor-diagnostics -o - -x c /dev/null
clang -cc1 version 12.0.0 (clang-1200.0.32.29) default target x86_64-apple-darwin19.6.0
ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/local/include"
ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/Library/Frameworks"
#include "..." search starts here:
#include <...> search starts here:
/usr/local/include
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0/include
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks (framework directory)
End of search list.
# 1 "/dev/null"
# 1 "<built-in>" 1
# 1 "<built-in>" 3
# 366 "<built-in>" 3
# 1 "<command line>" 1
# 1 "<built-in>" 2
# 1 "/dev/null" 2
从新连接时链接报错
clang -Xlinker -v
# ----
@(#)PROGRAM:ld PROJECT:ld64-609.8
BUILD 15:07:46 Dec 18 2020
configured to support archs: armv6 armv7 armv7s arm64 arm64e arm64_32 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em
Library search paths:
/usr/local/lib
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/lib
Framework search paths:
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/
Undefined symbols for architecture x86_64:
"_main", referenced from:
implicit entry/start for main executable
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
花了很多时间最终没有解决这个问题，最终在centos上一次性成功，后面开发工作都搞centos会比mac少出很多问题
1.2. 主要功能
- open – 打开数据库
- put – 插入数据
- get – 获取数据
- Iterator – 遍历数据
其他功能后面再列
1.3. 使用
C++ 参照
- https://blog.csdn.net/joelcat/article/details/89240584 python 参照
- http://www.idcat.cn/leveldb-python%E5%9F%BA%E7%A1%80%E6%93%8D%E4%BD%9C%E6%96%B9%E6%B3%95.html
1.4. 宏观视角
使用 DB::Open 之后会在本地生成一个数据库目录, 目录路径为db_path 代码中指定的是 /var/tmp/ldb1.ldb
leveldb::DB::Open(options, db_path, &ldbptr)
tree /var/tmp/ldb1.ldb
/var/tmp/ldb1.ldb
├── 000003.log
├── CURRENT
├── LOCK
├── LOG
└── MANIFEST-000002
1.4.1. 首次打开文件解析
首先看看每个文件里面有什么东西
- 000003.log
看内容主要为刚刚写进去的键值，这个文件应该是数据库数据文件
hexdump -Cv 000003.log
00000000 71 69 19 21 20 00 01 01 00 00 00 00 00 00 00 01 |qi.! ...........|
00000010 00 00 00 01 04 4b 45 59 31 0d 48 45 4c 4c 4f 5f |.....KEY1.HELLO_|
00000020 4c 45 56 45 4c 44 42 4d 86 03 c7 38 00 01 02 00 |LEVELDBM...8....|
00000030 00 00 00 00 00 00 03 00 00 00 01 08 75 73 65 72 |............user|
00000040 6e 61 6d 65 04 4a 6f 65 6c 01 06 67 65 6e 64 65 |name.Joel..gende|
00000050 72 04 4d 61 6c 65 01 03 6a 6f 62 0a 50 72 6f 67 |r.Male..job.Prog|
00000060 72 61 6d 6d 65 72 |rammer|
00000066
- CURRENT
记录了MANIFEST-000002的文件名，推测和压缩有关， manifest应该是压缩标头，current指向manifest
hexdump -Cv CURRENT
00000000 4d 41 4e 49 46 45 53 54 2d 30 30 30 30 30 32 0a |MANIFEST-000002.|
00000010
- LOCK
LOCk顾名思义是个锁文件
ll LOCK
-rw-r--r-- 1 root root 0 4月 5 15:26 LOCK
可以先打开一个连接，然后在去打开连接，不出意外会报错
>>> import leveldb
>>> import os, sys
>>> db = leveldb.LevelDB("/var/tmp/ldb1.ldb")
>>> db = leveldb.LevelDB("/var/tmp/ldb1.ldb")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
leveldb.LevelDBError: IO error: lock /var/tmp/ldb1.ldb/LOCK: already held by process
python 没找到关闭db的接口 c关闭db接口就是删除db对象
delete db
- LOG
也是个空文件，猜测不到他是干啥的
- MANIFEST–000002
看dump的关键字是跟压缩有关
hexdump -Cv MANIFEST-000002
00000000 56 f9 b8 f8 1c 00 01 01 1a 6c 65 76 65 6c 64 62 |V........leveldb|
00000010 2e 42 79 74 65 77 69 73 65 43 6f 6d 70 61 72 61 |.BytewiseCompara|
00000020 74 6f 72 a4 9c 8b be 08 00 01 02 03 09 00 03 04 |tor.............|
00000030 04 00 |..|
00000032
1.5. 多次写入数据
-rw-r--r-- 1 root root 197 4月 5 16:46 000005.ldb
-rw-r--r-- 1 root root 102 4月 5 16:59 000008.log
-rw-r--r-- 1 root root 16 4月 5 16:59 CURRENT
-rw-r--r-- 1 root root 0 4月 5 15:26 LOCK
-rw-r--r-- 1 root root 181 4月 5 16:59 LOG
-rw-r--r-- 1 root root 0 4月 5 16:47 LOG.old
-rw-r--r-- 1 root root 85 4月 5 16:59 MANIFEST-000007
多了一个 000005.ldb 文件 log 和 MANIFEST的id也增加了
LOG文件大小也有了，看下变化
- 000008.log
大体上看变化不大
hexdump -Cv 000008.log
00000000 87 7e 52 9b 20 00 01 05 00 00 00 00 00 00 00 01 |.~R. ...........|
00000010 00 00 00 01 04 4b 45 59 31 0d 48 45 4c 4c 4f 5f |.....KEY1.HELLO_|
00000020 4c 45 56 45 4c 44 42 6a 69 07 b3 38 00 01 06 00 |LEVELDBji..8....|
00000030 00 00 00 00 00 00 03 00 00 00 01 08 75 73 65 72 |............user|
00000040 6e 61 6d 65 04 4a 6f 65 6c 01 06 67 65 6e 64 65 |name.Joel..gende|
00000050 72 04 4d 61 6c 65 01 03 6a 6f 62 0a 50 72 6f 67 |r.Male..job.Prog|
00000060 72 61 6d 6d 65 72 |rammer|
00000066
- 000005.ldb
和log内容类似，暂时也看不出什么
hexdump -Cv 000005.ldb
00000000 00 0c 0d 4b 45 59 31 01 01 00 00 00 00 00 00 48 |...KEY1........H|
00000010 45 4c 4c 4f 5f 4c 45 56 45 4c 44 42 00 0e 04 67 |ELLO_LEVELDB...g|
00000020 65 6e 64 65 72 01 03 00 00 00 00 00 00 4d 61 6c |ender........Mal|
00000030 65 00 0b 0a 6a 6f 62 01 04 00 00 00 00 00 00 50 |e...job........P|
00000040 72 6f 67 72 61 6d 6d 65 72 00 10 04 75 73 65 72 |rogrammer...user|
00000050 6e 61 6d 65 01 02 00 00 00 00 00 00 4a 6f 65 6c |name........Joel|
00000060 00 00 00 00 01 00 00 00 00 3b 06 83 55 00 00 00 |.........;..U...|
00000070 00 01 00 00 00 00 c0 f2 a1 b0 00 09 02 76 01 ff |.............v..|
00000080 ff ff ff ff ff ff 00 68 00 00 00 00 01 00 00 00 |.......h........|
00000090 00 eb 94 53 92 6d 08 7a 16 00 00 00 00 00 00 00 |...S.m.z........|
000000a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000b0 00 00 00 00 00 00 00 00 00 00 00 00 00 57 fb 80 |.............W..|
000000c0 8b 24 75 47 db |.$uG.|
000000c5
- LOG
写入几次之后有内容了
几个关键字 Recovery log Delete type=3 Delete type=0 剩下的就是时间戳了， 先记一下后面读代码的时候在回过来看看
hexdump -Cv LOG
00000000 32 30 32 31 2f 30 34 2f 30 35 2d 31 36 3a 35 39 |2021/04/05-16:59|
00000010 3a 35 32 2e 37 35 35 34 37 32 20 31 34 30 32 30 |:52.755472 14020|
00000020 39 36 36 39 35 39 36 39 39 32 20 52 65 63 6f 76 |9669596992 Recov|
00000030 65 72 69 6e 67 20 6c 6f 67 20 23 36 0a 32 30 32 |ering log #6.202|
00000040 31 2f 30 34 2f 30 35 2d 31 36 3a 35 39 3a 35 32 |1/04/05-16:59:52|
00000050 2e 37 37 32 38 34 32 20 31 34 30 32 30 39 36 36 |.772842 14020966|
00000060 39 35 39 36 39 39 32 20 44 65 6c 65 74 65 20 74 |9596992 Delete t|
00000070 79 70 65 3d 33 20 23 34 0a 32 30 32 31 2f 30 34 |ype=3 #4.2021/04|
00000080 2f 30 35 2d 31 36 3a 35 39 3a 35 32 2e 37 37 32 |/05-16:59:52.772|
00000090 38 35 32 20 31 34 30 32 30 39 36 36 39 35 39 36 |852 140209669596|
000000a0 39 39 32 20 44 65 6c 65 74 65 20 74 79 70 65 3d |992 Delete type=|
000000b0 30 20 23 36 0a |0 #6.|
000000b5
每次连接到数据库之后LOG内容都会变 LOG.old 为上次LOG的备份
再次连接上去这次变得比较复杂了,关键字多了Level table
hexdump -Cv LOG
00000000 32 30 32 31 2f 30 34 2f 30 35 2d 31 37 3a 31 35 |2021/04/05-17:15|
00000010 3a 31 32 2e 34 32 36 30 30 32 20 37 66 65 31 32 |:12.426002 7fe12|
00000020 37 64 32 36 37 34 30 20 52 65 63 6f 76 65 72 69 |7d26740 Recoveri|
00000030 6e 67 20 6c 6f 67 20 23 38 0a 32 30 32 31 2f 30 |ng log #8.2021/0|
00000040 34 2f 30 35 2d 31 37 3a 31 35 3a 31 32 2e 34 32 |4/05-17:15:12.42|
00000050 36 30 34 38 20 37 66 65 31 32 37 64 32 36 37 34 |6048 7fe127d2674|
00000060 30 20 4c 65 76 65 6c 2d 30 20 74 61 62 6c 65 20 |0 Level-0 table |
00000070 23 31 30 3a 20 73 74 61 72 74 65 64 0a 32 30 32 |#10: started.202|
00000080 31 2f 30 34 2f 30 35 2d 31 37 3a 31 35 3a 31 32 |1/04/05-17:15:12|
00000090 2e 34 32 39 36 39 36 20 37 66 65 31 32 37 64 32 |.429696 7fe127d2|
000000a0 36 37 34 30 20 4c 65 76 65 6c 2d 30 20 74 61 62 |6740 Level-0 tab|
000000b0 6c 65 20 23 31 30 3a 20 31 39 37 20 62 79 74 65 |le #10: 197 byte|
000000c0 73 20 4f 4b 0a 32 30 32 31 2f 30 34 2f 30 35 2d |s OK.2021/04/05-|
000000d0 31 37 3a 31 35 3a 31 32 2e 34 34 30 33 30 38 20 |17:15:12.440308 |
000000e0 37 66 65 31 32 37 64 32 36 37 34 30 20 44 65 6c |7fe127d26740 Del|
000000f0 65 74 65 20 74 79 70 65 3d 33 20 23 37 0a 32 30 |ete type=3 #7.20|
00000100 32 31 2f 30 34 2f 30 35 2d 31 37 3a 31 35 3a 31 |21/04/05-17:15:1|
00000110 32 2e 34 34 30 33 32 34 20 37 66 65 31 32 37 64 |2.440324 7fe127d|
00000120 32 36 37 34 30 20 44 65 6c 65 74 65 20 74 79 70 |26740 Delete typ|
00000130 65 3d 30 20 23 38 0a |e=0 #8.|
00000137
- MANIFEST
MANIFEST有两个 KEY1 和 两个 username 没有其他的键
hexdump -Cv MANIFEST-000009
00000000 be b7 4c 6d 3f 00 01 01 1a 6c 65 76 65 6c 64 62 |..Lm?....leveldb|
00000010 2e 42 79 74 65 77 69 73 65 43 6f 6d 70 61 72 61 |.BytewiseCompara|
00000020 74 6f 72 07 00 05 c5 01 0c 4b 45 59 31 01 01 00 |tor......KEY1...|
00000030 00 00 00 00 00 10 75 73 65 72 6e 61 6d 65 01 02 |......username..|
00000040 00 00 00 00 00 00 3a 8b 3a 2a 2b 00 01 02 0b 09 |......:.:*+.....|
00000050 00 03 0c 04 08 07 00 0a c5 01 0c 4b 45 59 31 01 |...........KEY1.|
00000060 05 00 00 00 00 00 00 10 75 73 65 72 6e 61 6d 65 |........username|
00000070 01 06 00 00 00 00 00 00 |........|
00000078
到目前位置，没法从宏观的使用上观察到其他信息，那就看看代码吧
