[Feature] Support Read and Write Limitation on Hstore Backend

[shirleyStorage]

Feature Description (功能描述)

Each hstore server may be accessed by multiple graph servers, so we need to increase traffic control on the hstore side to protect the hstore server from being suspended due to too much traffic in high-concurrency scenarios

So, I want to implement rate limiting for read and write speeds in Hstore when CPU or memory reaches threshold levels.

[shirleyStorage]

Hi, @VGalaxies @imbajin I would love to hear your feedback on this feature. What do you think?

[dosubot]

This feature looks interesting, but I could really use your expertise on this, @imbajin.

To continue talking to Dosu, mention @dosu.

[imbajin]

Hi, @VGalaxies @imbajin I would love to hear your feedback on this feature. What do you think?

This (issue) range is quite large, need time to discuss ~

And there are some similar operations designed and implemented in higher versions of PD, such as multiple groups to separate hot and cold data

@dosu u can give your ideas and suggestions, as well as the current Store read and write overall generalization

[dosubot]

To address the feature request for implementing rate limiting in the Hstore backend, here are some ideas and suggestions:

Current Mechanisms for Traffic Control

The Hstore backend already includes mechanisms for traffic control using RateLimiter instances for both write and read operations. These are initialized based on configuration options:

private final RateLimiter writeRateLimiter;
private final RateLimiter readRateLimiter;

final int writeLimit = config.get(CoreOptions.RATE_LIMIT_WRITE);
this.writeRateLimiter = writeLimit > 0 ? RateLimiter.create(writeLimit) : null;
final int readLimit = config.get(CoreOptions.RATE_LIMIT_READ);
this.readRateLimiter = readLimit > 0 ? RateLimiter.create(readLimit) : null;

^[1]

Handling High-Concurrency Scenarios

The system uses a ReentrantReadWriteLock to manage concurrent access, allowing multiple threads to read concurrently while ensuring exclusive write operations:

private final ReadWriteLock storeLock = new ReentrantReadWriteLock();

When querying the store, a read lock is acquired to allow concurrent read operations:

Lock readLock = this.storeLock.readLock();
readLock.lock();
try {
    // Perform read operations
} finally {
    readLock.unlock();
}

^[2]

Leveraging Higher Versions of PD

Higher versions of PD include several modules that could be referenced or integrated for this feature. The hugegraph-pd project includes modules such as hg-pd-grpc, hg-pd-common, hg-pd-client, and others. These modules provide functionalities like partition management, heartbeat streaming, and store registration, which could be useful for implementing advanced traffic control features.

Suggestions for Implementation

1. Dynamic Rate Limiting: Implement dynamic rate limiting based on CPU and memory usage thresholds. This can be achieved by monitoring system metrics and adjusting the rate limits accordingly.
2. Integration with PD Modules: Utilize the existing modules in higher versions of PD for advanced traffic control features. For example, separating hot and cold data into different groups to manage load more effectively.
3. Enhanced Locking Mechanisms: Consider enhancing the current locking mechanisms to better handle high-concurrency scenarios, possibly by introducing more granular locks or optimizing the existing read-write locks.

By leveraging the existing mechanisms and integrating advanced features from higher versions of PD, we can effectively implement rate limiting and traffic control in the Hstore backend.

To continue talking to Dosu, mention @dosu.