You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
When a barrier reaches some actor. it will acquire some quota and write the data of memtable to shared buffer at first. If there are not enough memory quota, this write operation will be blocked and the actor will also be blocked.
Describe the solution you'd like
When the memory usage does not exceed 70% of shared buffer memory capacity, allow data of any barrier writing to hummock.
When the memory usage exceed 70% of shared buffer memory capacity, only allow the first barrier writing and let actor with larger barrier waiting for notify.
We use two variables to decide whether a request is the first barrier, max_seal_epoch and min_unseal_epoch. Uploader will update max_seal_epoch every time all data of one barrier has written to hummock storage. For example: Currently, max_seal_epoch is 10, while min_unseal_epoch is 11. Then, epoch-11 finish writing and max_seal_epoch update to 11. Then a request with epoch 14 try to allocate some quota, it will succeed because max_seal_epoch == min_unseal_epoch means that there is no first barrier. And this request will set min_unseal_epoch 14 and MemoryLimiter will refuse all requests with epoch larger than 14. But a few minutes later, some request with epoch-12 reaches, it will also get its quota and set min_unseal_epoch 12. Now all requests with epoch-14 will also fail to get quota.
For refused requests, we use a BTreeMap to group them by epoch. When the memory usage of storage is less than 70%, it will notify some request with smallest epoch.
Describe alternatives you've considered
No response
Additional context
No response
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
When a barrier reaches some actor. it will acquire some quota and write the data of memtable to shared buffer at first. If there are not enough memory quota, this write operation will be blocked and the actor will also be blocked.
Describe the solution you'd like
We use two variables to decide whether a request is the first barrier,
max_seal_epoch
andmin_unseal_epoch
. Uploader will updatemax_seal_epoch
every time all data of one barrier has written to hummock storage. For example: Currently,max_seal_epoch
is 10, whilemin_unseal_epoch
is 11. Then, epoch-11 finish writing andmax_seal_epoch
update to 11. Then a request with epoch 14 try to allocate some quota, it will succeed becausemax_seal_epoch == min_unseal_epoch
means that there is no first barrier. And this request will setmin_unseal_epoch
14 andMemoryLimiter
will refuse all requests with epoch larger than 14. But a few minutes later, some request with epoch-12 reaches, it will also get its quota and setmin_unseal_epoch
12. Now all requests with epoch-14 will also fail to get quota.For refused requests, we use a
BTreeMap
to group them by epoch. When the memory usage of storage is less than 70%, it will notify some request with smallest epoch.Describe alternatives you've considered
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: