Redis medium Key and hot Key solution

I. Definition

1.1. What is a Big key

• A large key is a key that contains a large amount of data. (Sum up in one word is big)

  Space occupied: A large key usually refers to a key that contains a large amount of data, so that the memory occupied by the corresponding value of the key is beyond the normal range. This size threshold is not fixed, but is relative to the available memory of the Redis instance. When the size of a key exceeds the available memory of the Redis instance, it is considered a large key.

  Operation time: If the operation time on a key is too long, the performance deteriorates or the processing speed of other requests is affected. This is usually due to the large amount of data contained under the key.

1.2. What is Hot key

• A hot key is a key that is frequently accessed. (To sum it up, hot, frequent visits.)

  Frequent access: A key that is accessed frequently within a certain period of time is a hot key.

  Business: For example, in the case of mall promotion, the cache of a certain product may become a hot key. In this case, the hot key reflects not only the high access frequency of the key, but also the user's enthusiasm for a certain service function.

  Performance: The CPU usage of Redis is too high due to frequent access of hot keys, resulting in prolonged response time or blocked requests, resulting in system crashes.

key big and not big, hot and not hot according to their own business, from the actual situation to evaluate.

Ii. Influence

2.1 Impact of a large key

• Memory consumption: The cache efficiency is reduced during caching, occupying a large amount of memory space, causing a sharp increase in the memory consumption of Redis, and may lead to insufficient memory resources of Redis instances, or even start memory elimination policies, thus affecting the normal operation of the system.
• Performance degradationProcessing large keys consumes more CPU time and bandwidth, which degrades Redis performance. Since Redis is still single-threaded, processingBig keyThis in turn blocks the processing of other requests, affecting system performance.
• The persistence efficiency decreases: When performing persistent operations,AOFwithRDBAll because of theBig keyIt takes more time, thus delaying the persistence time, and even causing cache inconsistency in distributed environments.
• Network transmission delay:Big keyDuring network transmission, the delay of network transmission will be increased, and data inconsistency may be caused during data synchronization in distributed environment.

2.2. Impact of hot key

• The CPU usage is highBecause it isHot keyTherefore, the CPU has been occupied, which in turn causes the CPU load of the Redis instance to increase.
• Request blocking: If key has access priority,Hot keyThe presence of the request queue may cause other requests to be blocked.
• Extended response time: BecauseHot keyOther requests are blocked, resulting in longer response times.
• Unbalanced performance: Traffic spikes and system performance is unbalanced.

2.3. Summary

Both large and hot keys can have a series of effects on Redis instances, such as high memory usage, increased CPU load, longer persistence time, and reduced performance.

Iii. Reason analysis

3.1 Causes of large keys

There are many reasons for the big key, and here we will take a look at these often encountered in the work.

3.1.1. Store large amounts of data

Storing a large amount of data is also the most common reason why we often encounter big keys.

For example, the String type directly holds a large text or binary data; The Hash structure stores a large number of key-value pairs.

• String

SET zuiyu_large_text_key "very large text content..."

• Hash

HMSET zuiyu_large_hash_key field1 value1 field2 value2 ... fieldN valueN

3.1.2 The cache time is set improperly

The reason for the large key is probably a very deep hidden old bug. In some business scenarios, Redis is used to cache data, and the business is to write data to the key regularly. As the key is not set the cache time, the key consumes more and more memory as time goes by. For this point, you only need to set a reasonable expiration time.

The premise is that multiple writes are not overwritten, but appended to the problem.

SETEX zuiyu_key_with_expiry value 3600 # Sets the expiry time to 3600 seconds

3.1.3 Improper use of data structure

When the List data structure is used to store data, repeated data is added to the key, causing the key to become larger and larger. In fact, services do not need duplicate data.

• List

LPUSH zuiyu_large_list_key value

3.1.4 Summary

The root cause of large keys is that too much data is stored under one key.

3.2 Causes of the Hot key

3.2.1 Hot Data

Hot key generation generally means that the system access is hot, but hot only one of the points or n points. Similar to the melon of a star in Weibo, when it hits the headlines, a large number of people visit it, causing the key corresponding to the star to become a hot key.

3.2.2. Frequent updates

In some service scenarios, if a key is updated frequently within a specified period of time, the key becomes a hot key.

3.2.3. Popular Search

Similar to the hot data in the first, hot data is generated, and the hot keyword corresponding to the data is also searched by a large number of users, resulting in frequent access to the keyword, and finally the key is also called the hot key.

3.2.4 Summary

The generation of hot keys is nothing more than hot data, hot keywords generated by hot data and frequent access to the same key in a certain period of time.

Fourth, solutions

4.1, big key solution

• Reasonable data structure
• Reasonable cache time
• Big keyTo split into multipleSmall key
• Periodic pairBig keyClean up

4.2, hot key solution

• Reasonable cache elimination policies
• Hotspot data fragmentation

Spread hot data across different Redis instances to improve system throughput.

• Cache preheating

The cache is preheated before the system starts or the active peak starts. The required data is loaded to the cache in advance, reducing the time for the first access to hot data.

• Random cache expiration time

Avoid cache avalanche and cache penetration when a large number of keys fail in batches at the same time.

• Cache penetration

Buffer requests are filtered using Bloom filters to prevent invalid requests from entering the cache layer.

V. Summary

For large keys, we want to avoid as much data as possible under the same key. For the hot key, we should set the expiration time reasonably, add Blum filter and other technologies to filter invalid requests, and preheat the incoming data cache and fragment the hot data.

The above is the details of the solution of Redis big Key and hot Key, more information about Redis big Key and hot Key, please pay attention to other related articles in the script home!

Latest comments