feat: floyd supports one key for one data structure #240

Mixficsol · 2024-03-27T08:11:59Z

背景

当前 Pika 的一个 Key 可以对应多种数据结构，和 Redis 不一致

解决方案

在 Floyd 现有的设计之上，将之前 String 类型所在的Column-Family 用于存放所有的 Key 的 Meta 信息，其对应的 value 为数据类型 type，并在内存中建立一个所有 Key 的 cuckoo bloom filter. 在增加一个 Key 时，先去读 cuckoo bloom filter 判断被增加的 Key 是否存在，如果不存在，则直接进行后续操作，如果存在，但可能是假阳性，所以需要一次读盘操作，去新增加的 Column-Family 中读取，验证被增加的 Key 是否真正存在，然后再进行后续的操作，单独设置 block cache ，为这个新增加的 Column-Family 单独设置一个，防止和别的数据共用，导致 Key 的 Colunm-Family 淘汰出内存

修改前：
enum ColumnFamilyIndex {
    kStringsCF = 0,
    kHashesMetaCF = 1,
    kHashesDataCF = 2,
    kSetsMetaCF = 3,
    kSetsDataCF = 4,
    kListsMetaCF = 5,
    kListsDataCF = 6,
    kZsetsMetaCF = 7,
    kZsetsDataCF = 8,
    kZsetsScoreCF = 9,
};

修改后：
enum ColumnFamilyIndex {
    kMetaCF = 0,
    kHashesDataCF = 1,
    kSetsDataCF = 2,
    kListsDataCF = 3,
    kZsetsDataCF = 4,
    kZsetsScoreCF = 5,
};

kMetaCF 字段设计

修改前

String

key格式
| reserve1 | key | reserve2 | 
|    8B    |     |   16B    |

value格式
| value | reserve | cdate | timestamp | 
|       |   16B   |   8B  |    8B     |

Hash

key格式
| reserve1 | key | reserve2 | 
|    8B    |     |    16B   |

value格式
| hash_size | version |  reserve  | cdate | timestamp | 
|    4B     |    8B   |    16B    |   8B  |     8B    |

List

key格式
| reserve1 | key | reserve2 |
|    8B    |     |   16B    |

value格式
|list_size| version | left index | right index | reserve |  cdate | timestamp | 
|   8B    |   8B    |     8B     |     8B      |   16B   |   8B   |    8B     |

set

key格式
| reserve1 | key | reserve2 |
|    8B    |     |    16B    |

value格式
| set_size | version | reserve | cdate | timestamp | 
|    4B    |    8B   |   16B   |   8B  |    8B     |

zset

key格式
| reserve1 | key | reserve2 |
|    8B    |     |    16B   |  

value格式
| zset_size | version | reserve  | cdate | timestamp | 
|    4B     |    8B   |    16B   |   8B  |    8B     |

修改后

我们对每一种数据类型的 Meta 的 value 前增加一个字段 Type 用于区别每个 Key 对应的数据结构

以 Set 类型举例

key格式
| reserve1 | key | reserve2 |
|    8B    |     |    16B   |

value格式
| type | set_size | version | reserve | cdate | timestamp | 
|  1B  |    4B    |    8B   |   16B   |   8B  |    8B     |

String 的 Meta 格式

key格式
| reserve1 | key | reserve2 | 
|    8B    |     |   16B    |

value格式
| Type | value | reserve | cdate | timestamp | 
|  1B  |       |   16B   |  8B   |    8B     |

在解析的时候，先解析头部的第一个字节，然后根据类型判断是否需要继续解析下去

issue #269

Mixficsol · 2024-03-27T08:13:59Z

第一次提交只修改了 Sadd 和 Hset 这两个命令在遇到多 Key 时的判断，确保和 Redis 的返回值保持一致

PikiwiDB

Redis

Mixficsol · 2024-03-27T09:12:57Z

目前遇到的一个问题是我们使用的是 RocksDB 的 Status 作为每一个命令的状态返回值，但是对于这种由于多 Key 问题导致的操作无法进行下去我们应该使用 RocksDB 中的哪种状态返回值呢？还是说修改 RocksDB 代码新加一种返回值

Issues-translate-bot · 2024-03-27T09:13:08Z

One problem currently encountered is that we are using Status of RocksDB as the status return value of each command, but for this kind of operation that cannot be carried out due to multiple Key problems, we should use RocksDB Which status return value does it have? Or should we modify the RocksDB code to add a new return value?

src/storage/src/base_meta_value_format.h

hotjump · 2024-03-27T11:51:14Z

如果要增加 cuckoo bloom filter ，建议增加性能对比的测试用例好判断是否有提升以及提升的幅度。

Issues-translate-bot · 2024-03-27T11:51:26Z

If you want to add cuckoo bloom filter, it is recommended to add test cases for performance comparison to judge whether there is improvement and the extent of improvement.

Mixficsol · 2024-03-28T03:44:07Z

第二次提交把除 string 以外的四种数据结构都加上了多 key 的判断，并修复测试用例通过

Issues-translate-bot · 2024-03-28T03:44:16Z

The second submission added multi-key judgment to the four data structures except string, and tried to repair the test cases.

Mixficsol · 2024-03-28T05:55:02Z

嗯嗯，这个算是第二期的任务，第一期先把CF弄好

Issues-translate-bot · 2024-03-28T05:55:11Z

Hmm, this is considered the task of the second period. In the first period, we should finish the CF first.

src/storage/src/lists_meta_value_format.h

src/storage/src/base_meta_value_format.h

src/storage/src/storage.cc

Mixficsol · 2024-03-29T05:54:19Z

第三次提交，新设计了 String 的 MetaCF, 对于之前 Floyd 中的 Del, Exists, Expire 等之前需要操作每一个数据类型的 CF，做了优化，只需要判断第一个字节 Type，然后调用相应的接口即可，当然还有后续类似的命令也需要进行优化

Issues-translate-bot · 2024-03-29T05:54:29Z

The third submission, newly designed MetaCF of String, and optimized the CF of Del, Exists, Expire in Floyd that previously needed to operate each data type. , you only need to determine the first byte Type, and then call the corresponding interface. Of course, subsequent similar commands also need to be optimized.

Mixficsol · 2024-03-29T08:34:44Z

第四次提交合并了一下代码的冲突，但是发现 List 的 GO 测试通过失败了，但是本地手动复现是没问题的，本地开启了 Debug 模式也是没问题的

Issues-translate-bot · 2024-03-29T08:34:55Z

The fourth submission merged some code conflicts.

Mixficsol · 2024-03-29T08:35:49Z

第五次提交移植了 redis 的 TCL 测试，目前还在调试

Issues-translate-bot · 2024-03-29T08:35:59Z

The fifth submission requires changing Persist, TTL, GetType, Keys, ScanDatabase, DoCompactRange, GetCurrentTaskType, IsExist, which were previously traversal data type interfaces, into pairs Specify the data type interface to operate

dingxiaoshuai123 · 2024-04-10T12:05:46Z

src/cmd_keys.cc

-    }
-    client->SetRes(CmdRes::kErrOther, cnt);
-  }
+  auto res = PSTORE.GetBackend(client->GetCurrentDB())->GetStorage()->Persist(client->Key());


feat: add type

…o floyd-improve

…oyd-improve

* add constexpr string

AlexStocks · 2024-04-11T14:42:03Z

from #144

…o floyd-improve

qqricky · 2024-04-21T15:17:47Z

🐮

Issues-translate-bot · 2024-04-21T15:17:59Z

🐮

wangshao1 · 2024-04-22T12:33:52Z

src/storage/src/storage.cc

@@ -1210,55 +1129,19 @@ int64_t Storage::DelByType(const std::vector<std::string>& keys, const DataType&

 int64_t Storage::Exists(const std::vector<std::string>& keys) {
  int64_t count = 0;
-  int32_t ret;
-  uint64_t llen;
-  std::string value;
  Status s;
  bool is_corruption = false;


is_corruption没有用到的变量

wangshao1 · 2024-04-22T12:35:39Z

src/storage/src/storage.cc

@@ -1515,225 +1398,48 @@ Status Storage::Scanx(const DataType& data_type, const std::string& start_key, c
 int32_t Storage::Expireat(const Slice& key, uint64_t timestamp) {
  Status s;
  int32_t count = 0;
-  bool is_corruption = false;
-
  auto& inst = GetDBInstance(key);
  s = inst->StringsExpireat(key, timestamp);


这里调用stringsexpireat是说expireat接口只支持string类型？

wangshao1 · 2024-04-22T12:36:22Z

src/storage/src/storage.cc

-
-  if (is_corruption) {
-    return -1;
+    return 1;


是不是少了corruption的情况？

wangshao1 · 2024-04-22T12:38:05Z

src/storage/src/redis_hashes.cc

  if (s.ok()) {
    ParsedHashesMetaValue parsed_hashes_meta_value(&meta_value);
+    auto type = static_cast<enum Type>(static_cast<uint8_t>(meta_value[0]));


我看类似的逻辑在挺多的地方都有调用，而且得到的type变量其实也没有其他地方在用，是不是封装一个函数出来？

github-actions bot added the ✏️ Feature New feature or request label Mar 27, 2024

Mixficsol force-pushed the floyd-improve branch 2 times, most recently from 1721c06 to 4ebdfd8 Compare March 27, 2024 08:24

Mixficsol force-pushed the floyd-improve branch from 4ebdfd8 to 79a34f8 Compare March 27, 2024 09:45

Complete the multi-key judgment of a pair of simple commands

4fc9b57

Mixficsol force-pushed the floyd-improve branch from 79a34f8 to 4fc9b57 Compare March 27, 2024 09:46

hotjump reviewed Mar 27, 2024

View reviewed changes

src/storage/src/base_meta_value_format.h Outdated Show resolved Hide resolved

Multiple key judgments are added to the remaining four data structures

21fddc0

Mixficsol force-pushed the floyd-improve branch from ce8a587 to 21fddc0 Compare March 28, 2024 07:16

lqxhub reviewed Mar 28, 2024

View reviewed changes

src/storage/src/lists_meta_value_format.h Outdated Show resolved Hide resolved

src/storage/src/base_meta_value_format.h Outdated Show resolved Hide resolved

src/storage/src/base_meta_value_format.h Outdated Show resolved Hide resolved

src/storage/src/storage.cc Show resolved Hide resolved

add string meta-CF

6e09141

Mixficsol force-pushed the floyd-improve branch from 68dfa7a to 2d2a013 Compare March 29, 2024 07:57

fix merge conflict

acb52b3

Mixficsol force-pushed the floyd-improve branch from 2d2a013 to acb52b3 Compare March 29, 2024 08:40

github-actions bot added the 📒 Documentation Improvements or additions to documentation label Mar 29, 2024

github-actions bot added the 📒 Documentation Improvements or additions to documentation label Apr 10, 2024

fix conflict

5295f09

Mixficsol force-pushed the floyd-improve branch from cc6261e to 5295f09 Compare April 10, 2024 10:24

dingxiaoshuai123 added 2 commits April 10, 2024 19:47

add enum as type & merge string&meta cf

d3a5973

Merge branch 'floyd-improve' of github.com:Mixficsol/pikiwidb into floyd

4bce5d9

dingxiaoshuai123 reviewed Apr 10, 2024

View reviewed changes

dingxiaoshuai123 and others added 8 commits April 10, 2024 21:18

change enum name

8fb77bf

Merge pull request #1 from Mixficsol/floyd

778e78e

feat: add type

Merge branch 'unstable' of github.com:OpenAtomFoundation/pikiwidb int…

1183a37

…o floyd-improve

Merge branch 'floyd-improve' of github.com:Mixficsol/pikiwidb into fl…

4ae1948

…oyd-improve

Changed the policy for RocksDB iterator scanning keys in Meta's CF

14246ac

change meta cf compaction filter (#3)

81759ff

Added redis latest tcl test

8be47ed

feat: add constexpr string (#4)

3d7cc93

* add constexpr string

Mixficsol force-pushed the floyd-improve branch from d471caa to 3d7cc93 Compare April 11, 2024 12:13

Mixficsol force-pushed the floyd-improve branch from dce08d0 to 534ad95 Compare April 12, 2024 07:58

Merge branch 'unstable' of github.com:OpenAtomFoundation/pikiwidb int…

5c2e2df

…o floyd-improve

Mixficsol force-pushed the floyd-improve branch from 534ad95 to 5c2e2df Compare April 12, 2024 08:14

Mixficsol removed the 📒 Documentation Improvements or additions to documentation label Apr 12, 2024

Mixficsol force-pushed the floyd-improve branch from f8892f2 to 009cb05 Compare April 15, 2024 12:33

remove tcl

2c3ceb9

Mixficsol force-pushed the floyd-improve branch from 009cb05 to 2c3ceb9 Compare April 15, 2024 12:35

wangshao1 reviewed Apr 22, 2024

View reviewed changes

Mixficsol closed this May 17, 2024

fix conflict

24bd514

Mixficsol reopened this May 23, 2024

Mixficsol closed this May 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: floyd supports one key for one data structure #240

feat: floyd supports one key for one data structure #240

Mixficsol commented Mar 27, 2024 •

edited

Loading

Mixficsol commented Mar 27, 2024 •

edited

Loading

Mixficsol commented Mar 27, 2024

Issues-translate-bot commented Mar 27, 2024

hotjump commented Mar 27, 2024

Issues-translate-bot commented Mar 27, 2024

Mixficsol commented Mar 28, 2024 •

edited

Loading

Issues-translate-bot commented Mar 28, 2024

Mixficsol commented Mar 28, 2024

Issues-translate-bot commented Mar 28, 2024

Mixficsol commented Mar 29, 2024

Issues-translate-bot commented Mar 29, 2024

Mixficsol commented Mar 29, 2024 •

edited

Loading

Issues-translate-bot commented Mar 29, 2024

Mixficsol commented Mar 29, 2024 •

edited

Loading

Issues-translate-bot commented Mar 29, 2024

dingxiaoshuai123 Apr 10, 2024

Mixficsol Apr 11, 2024

AlexStocks commented Apr 11, 2024

qqricky commented Apr 21, 2024

Issues-translate-bot commented Apr 21, 2024

wangshao1 Apr 22, 2024

wangshao1 Apr 22, 2024

wangshao1 Apr 22, 2024

wangshao1 Apr 22, 2024

feat: floyd supports one key for one data structure #240

feat: floyd supports one key for one data structure #240

Conversation

Mixficsol commented Mar 27, 2024 • edited Loading

背景

解决方案

kMetaCF 字段设计

Mixficsol commented Mar 27, 2024 • edited Loading

Mixficsol commented Mar 27, 2024

Issues-translate-bot commented Mar 27, 2024

hotjump commented Mar 27, 2024

Issues-translate-bot commented Mar 27, 2024

Mixficsol commented Mar 28, 2024 • edited Loading

Issues-translate-bot commented Mar 28, 2024

Mixficsol commented Mar 28, 2024

Issues-translate-bot commented Mar 28, 2024

Mixficsol commented Mar 29, 2024

Issues-translate-bot commented Mar 29, 2024

Mixficsol commented Mar 29, 2024 • edited Loading

Issues-translate-bot commented Mar 29, 2024

Mixficsol commented Mar 29, 2024 • edited Loading

Issues-translate-bot commented Mar 29, 2024

dingxiaoshuai123 Apr 10, 2024

Choose a reason for hiding this comment

Mixficsol Apr 11, 2024

Choose a reason for hiding this comment

AlexStocks commented Apr 11, 2024

qqricky commented Apr 21, 2024

Issues-translate-bot commented Apr 21, 2024

wangshao1 Apr 22, 2024

Choose a reason for hiding this comment

wangshao1 Apr 22, 2024

Choose a reason for hiding this comment

wangshao1 Apr 22, 2024

Choose a reason for hiding this comment

wangshao1 Apr 22, 2024

Choose a reason for hiding this comment

Mixficsol commented Mar 27, 2024 •

edited

Loading

Mixficsol commented Mar 27, 2024 •

edited

Loading

Mixficsol commented Mar 28, 2024 •

edited

Loading

Mixficsol commented Mar 29, 2024 •

edited

Loading

Mixficsol commented Mar 29, 2024 •

edited

Loading