Write it at the front

Hello everyone , I'm Lao Tian , We've shared five serial gun articles before , Today we go on . Today we're going to talk about Redis.Redis It has become one of the necessary skills for our developers , At the same time, the interview is also a must . Here's the right Redis Make a summary , Then give it away 47 Serial guns .

Redis Of 47 Serial guns :

1、 Tell me what is Redis?

2、Redis What are the advantages and disadvantages ?

3、Redis What are the data types of ?

4、Redis Is it single threaded ?

5、Redis Why is it designed to be single threaded ?

6、Redis and Memcached What are the differences ?

7、 Please talk about Redis Thread model of ?

8、 Why? Redis Single thread model can also be so efficient ?

9、Redis It's single threaded , How to improve multicore CPU Utilization ratio ?

10、Redis We need to understand what the synchronization mechanism is ?

11、 What is? Redis Pipelining ?

12、Redis There are several ways of persistence ?

13、 say something RDB Of Advantages and disadvantages

14、 say something AOF Of Advantages and disadvantages

15、 How to choose the two persistence methods ?

16、 The interviewer asked what would happen if the machine suddenly turned off ?

17、 The interviewer asked bgsave What is the principle of ?

18、Redis There are several kinds of data “ Be overdue ” Strategy ?

19、Redis What kinds of data are there “ Eliminate ” Strategy ?

20、 What is the maximum storage capacity of a string type value ?

21、 be familiar with Redis Which clients of ?

22、 What is? Redis Business ?

23、Redis What are the points for attention in business ?

24、 Why? Redis Rollback is not supported for transactions ?

25、Redis What are the usage scenarios ?

26、 How to use Redis Implement distributed locks ?

27、 Implementation conditions of distributed lock ?

28、Redis and Zookeeper What's the difference between implementing distributed locks , Which is better ?

29、 How to use Redis Realize distributed current limiting ?

30、 How to use Redis Implement message queuing ?

31、Redis What are the high availability solutions ?

32、 What is? Redis Master slave synchronization ?

33、 How to use Redis Sentinel High availability ?

34、 If you use Redis Cluster High availability ?

35、 say something Redis The concept of hash slot ?

36、Redis Cluster What is the master-slave replication model of ?

37、Redis What's the function of your sentry ?

38、Redis What's the difference between a sentry and a group ?

39、 What does cache hit ratio mean ?

40、 If Redis There are 1 One hundred million key, Among them is 10w individual key It starts with a fixed known prefix , If you find them all ?

41、 Please talk about... In your production environment Redis How it was deployed ?

42、 You know what Redis Partition implementation scheme ?

43、 How to improve Redis shooting ?

44、 How to optimize the Redis Memory footprint ?

45、 What is cache penetration ? How to solve ?

46、 What is a cache avalanche ? How to solve ?

47、 Yes Redis Performance optimization , Some suggestions ?

.....( Continuously improve ing)

Redis Summary of core knowledge

I got you a Redis Summary of the core knowledge of mind map :

 picture

You need mind maps , Please add wechat tj20120622

1、 Tell me what is Redis?

Redis It's open source (BSD The license ), Memory storage data structure server , It can be used as a database , Caching and message queuing agents . It supports strings 、 Hashtable 、 list 、 aggregate 、 Ordered set , Bitmap ,HyperLogLogs And so on . Built in replication 、Lua Script 、LRU Take back 、 Business , And different levels of disk persistence , At the same time through Redis Sentinel Provide high availability , adopt Redis Cluster Provide automatic zoning . According to the monthly ranking website DB-Engines The data of ,Redis Is the most popular key value pair for storing databases .

The above is the official introduction , We can also make a simple answer :

Redis Its full name is :Remote Dictionary Server( Remote data services ), It's a high-performance memory based system that supports persistence key-value database . It has the following three basic characteristics :

  • Multiple data types
  • Persistence mechanism
  • Master slave synchronization

2、Redis What are the advantages and disadvantages ?

Redis advantage

  • Excellent reading and writing performance , Redis The speed at which you can read is 110000 Time /s, The speed of writing is 81000 Time /s.
  • Support data persistence , Support AOF and RDB Two ways to persist .
  • Support transactions ,Redis All operations of are atomic , meanwhile Redis It also supports atomic execution after several operations are combined .
  • Rich data structure , In addition to supporting string Type of value Support in addition hash、set、zset、list And so on .
  • Support master-slave replication , The host will automatically synchronize the data to the slave , Can be read-write separation .

Redis shortcoming

  • Database capacity is limited by physical memory , Cannot be used as high-performance read-write of massive data , therefore Redis Suitable scenarios are mainly limited to high-performance operations and operations with small amount of data .
  • Redis No automatic fault tolerance and recovery functions , The downtime of the host and slave will lead to the failure of the front-end read and write requests , You need to wait for the machine to restart or manually switch the front end IP To recover .
  • Host down , Before the outage, some data could not be synchronized to the slave in time , Switch IP After that, the problem of data inconsistency will be introduced , Reduced system availability .
  • Redis Difficult to support online capacity expansion , When the cluster capacity reaches the upper limit, online capacity expansion will become very complex . To avoid this problem , The operation and maintenance personnel must ensure enough space when the system is online , This is a great waste of resources .

3、Redis What are the data types of ?

Redis There are mainly the following data types :

  • String: This is the simplest type , It's just plain set and get, Do simple things. KV cache .
  • Hashe: This is similar map A structure of , In general, structured data can be used , Like an object ( The premise is that this object does not nest other objects ) To cache in Redis in , And every time I read and write the cache , It can be operated hash A field in .
  • List:List There is a sequence table , This can play a lot of tricks . For example, through list Store some list data structures , Similar to fan list 、 Comment list of articles and so on .
  • Sets: Is an unordered set , Automatic weight removal . Based on the direct set Throw in the data that needs to be de duplicated in the system , It's automatic , If you need fast global de duplication of some data , Of course, you can also jvm In memory HashSet Deduplication , But what if your system is deployed on multiple machines ? It has to be based on Redis Make a global set duplicate removal .
  • Sorted Set: Sort of set, De duplication but sorting , Give a score when you write it in , Auto sort by score . It can be used for ranking related functions .

Most articles are based on Redis Only 5 Type of data , also Bitmaps、HyperLogLogs、Streams etc. .

The explanation on the Chinese official website :

 picture


4、Redis Is it single threaded ?

Single thread here means Redis The network request module uses a thread ( So you don't need to consider concurrent security ), That is, a thread handles all network requests , Other modules still use multiple threads .

5、Redis Why is it designed to be single threaded ?

  1. Most requests are purely memory operations ( Very fast )
  2. Using single thread , Unnecessary context switches and race conditions are avoided
  3. Non blocking IO, Internal use epoll,epoll Reading in Chinese 、 Write 、 close 、 Connections are converted to events , And then use it epoll Multiplexing characteristics of , avoid IO cost .

6、Redis and Memcached What are the differences ?

From the following 8 In two ways :

  • Redis and Memcache It's all about storing data in memory , It's all memory databases . however Memcache It can also be used to cache other things , For example, pictures 、 Video and so on .
  • Memcache Support only key-value The data type of the structure ,Redis It's not just about supporting simple key-value Data of type , It also provides list,set,hash Such as data structure storage .
  • Virtual memory – Redis When physical memory runs out , You can use something that hasn't been used for a long time value Switch to disk distributed – Set up Memcache colony , utilize magent Be one master and many followers ; Redis You can be a master and a slave . You can have one master and one slave
  • Storage data security – Memcache After hanging up , The data is gone ;Redis Can be saved to disk on a regular basis ( Persistence )
  • Memcache The individual value Maximum 1m , Redis The individual value Maximum 512m .
  • disaster recovery – Memcache After hanging up , Data is not recoverable ;  Redis Data can be lost through aof recovery
  • Redis It supports cluster mode , Redis3.0 In the version , The government can support Cluster The model , Memcached No native cluster mode , Need to rely on the client to achieve , And then write data to the cluster in pieces .
  • Memcached The Internet IO The model is multithreaded , Non blocking IO Multiplexing network model , The prototype is close to nignx . and Redis Use single threaded IO Reuse model , I encapsulated a simple AeEvent Event processing framework , Primary implementation class epoll,kqueue and select , Closer to Apache Early models .

7、 Please talk about Redis Thread model of ?

This is because the answer to the question mentioned Redis It's based on non blocking IO Reuse model . If this question can't be answered , The answer is to dig a hole for yourself , Because you can't answer , The interviewer's impression of you may have to be discounted .

Redis Internal use of file event handler  file event handler , This file event handler is single threaded , therefore Redis It's called a single threaded model . It uses IO Multiplexing mechanism listens to multiple channels at the same time socket , according to socket Select the corresponding event handler to handle the event on .

The structure of the file event handler contains 4 Parts of :

  1. Multiple socket .
  2. IO Multiplexing program .
  3. File event dispatcher .
  4. Event handler ( Connect to answer processor 、 Command request processor 、 Command reply processor ).

Multiple socket Different operations may occur concurrently , Each operation corresponds to a different file event , however IO Multiplexers listen to multiple socket, Will socket The generated events are queued , The event dispatcher takes an event from the queue one at a time , Give the event to the corresponding event handler for processing .

Look at the client and Redis A communication process of :

 picture


Now let's talk about this picture :

  1. client Socket01 towards  Redis Of Server Socket Request to establish a connection , here Server Socket Will produce a AE_READABLE event ,IO Multiplexer listens to server socket After the event , Push the event into the queue in . The file event dispatcher gets the event from the queue , To the connection answering processor . Connecting to the answering processor creates a Can communicate with the client  Socket01, And put the Socket01 Of AE_READABLE The event is related to the command request processor United .
  2. Suppose the client sends a set key value request , here  Redis Medium  Socket01 Will produce AE_READABLE event ,IO Multiplexing programs push events into the queue , At this point, the event dispatcher gets the event from the queue Pieces of , Because of the front  Socket01 Of  AE_READABLE Event is already associated with the command request processor , So the event dispatcher Give the event to the command request processor for processing . Command request processor read  Socket01 Of set key value And in myself Done in memory set key value Set up . After the operation is completed , It will be  Socket01 Of AE_WRITABLE Events and orders Reply processor Association .
  3. If the client is ready to receive the return result at this time , that Redis Medium  Socket01 Will produce a AE_WRITABLE event , Also pushed into the queue , The event dispatcher found the associated command reply handler , By command reply processor  Socket01 Enter a result of this operation , such as  ok , And then release  Socket01 Of AE_WRITABLE Event and command reply processor Association .

This completes a communication . Don't be afraid of this passage , Combined with the picture , One time, not twice , I really can't. I can check some information online Combined with , Make sure that , Otherwise, it will be in vain .

8、 Why? Redis Single thread model can also be so efficient ?

It can be from below 5 Answer in three ways :

  1. C Language implementation , Efficient
  2. Pure memory operation
  3. Based on non blocking IO Reuse model mechanism
  4. Single thread can avoid the frequent context switching problem of multithreading
  5. Rich data structure ( The full name is hash structure , Very fast read speed , Some optimizations for data storage , Like ya Cable list , Jump watch, etc )

9、Redis It's single threaded , How to improve multicore CPU Utilization ratio ?

CPU It's not likely to be Redis Bottleneck , General memory and network can be . For example, using Redis The pipe (pipelining) stay liunx Running on the system can achieve 500K Of RPS(requests per second) , therefore , If your application primarily uses O(N) perhaps O(log(N)) Of command , They hardly need to use anything CPU.

However , In order to maximize the use of CPU, You can deploy multiple on the same server Redis Example , And use them as different servers , At some point , In any case, a server is not enough , therefore , If you want to use more than one CPU, You can think about slicing (shard) .

stay Redis In the client class library of , such as RB(Ruby The client of ) and Predis( Most commonly used PHP One of the clients ), Be able to use consistent hashing (consistent hashing) To deal with multiple Redis example .

10、Redis We need to understand what the synchronization mechanism is ?

Redis You can use master-slave synchronization , From slave synchronization . The first synchronization , The master node does it once bgsave, At the same time, the subsequent modification operations are recorded in memory buffer, To be completed rdb Full file synchronization to replication node , When the replication node acceptance is complete, it will rdb The image is loaded into memory .

After loading , Then inform the main node to synchronize the modified operation records to the replication node for replay, and the synchronization process is completed .

11、 What is? Redis Pipelining ?

Redis They have more complex data structures and provide atomic operations on them , This is a different evolutionary path from other databases .Redis All data types are based on basic data structure and transparent to programmers , There is no need for additional abstraction .

Redis Run in memory but persist to disk , So we need to balance memory when we read and write different data sets at high speed , Because the amount of data cannot be larger than the hardware memory . Another advantage of in memory databases is , Compared to the same complex data structure on disk , It's very easy to operate in memory , such Redis Can do a lot of internal complexity of things .

meanwhile , In terms of disk format, they are generated in a compact and additional way , Because they don't need random interviews .

12、Redis There are several ways of persistence ?

During the interview , If you can't answer it completely , There won't be a big problem . a key , It's about being organized , Yes RDB and AOF Have understanding .

Redis It provides two ways , Realize data persistence to hard disk .

1、【 Total quantity 】RDB Persistence , It means that in the specified time interval, the Dataset snapshot Write to disk . The actual operation process is ,fork A subprocess , Write the dataset to the temporary file first , After writing successfully , Replace the previous file , Compress storage with binary .

2、【 The incremental 】AOF Persistence , Log everything that the server handles Write 、 Delete operation , The query operation will not record , Record as text , You can open the file to see the detailed operation record .

13、 say something RDB Of Advantages and disadvantages

advantage

  • Flexible setting of backup frequency and cycle . You may want to file every hour recently 24 Hours of data , At the same time, it needs to be archived once a day. Recently 30 Days of data . Through such a backup strategy , In the event of a catastrophic system failure , We can recover very easily .
  • Perfect for cold backup , For disaster recovery ,RDB It's a very good choice . Because we can easily compress a single file and then transfer it to other storage media . recommend , This complete data file can be sent to some remote secure storage , for instance Amazon Of S3 Cloud services , It can be Alibaba cloud's in China OSS Distributed storage .
  • Maximize performance . about Redis In terms of service process , At the beginning of persistence , The only thing it needs to do is fork Out of child process , After that, the subprocess completes these persistent tasks , This can greatly avoid the execution of service process IO Operation . in other words ,RDB Yes Redis External reading and writing services , Very little impact , It can make Redis Maintain high performance .
  • Faster recovery . Compared with AOF Mechanism ,RDB Faster recovery of , Better for data recovery , Especially when the data set is very large .

shortcoming

  • If you want to ensure high availability of data , That is to avoid data loss as much as possible , that RDB Will not be a good choice . Because once the system goes down before timing persistence , Data not written to disk before will be lost .

    therefore ,RDB Under the actual situation , Need and AOF Use it together .

  • because RDB It's through fork Sub processes to assist in data persistence , therefore , If the dataset is large , May cause the entire server to stop serving for hundreds of milliseconds , Even 1 Second .

    therefore ,RDB Suggest underestimating in business , For example, in the middle of the night .

14、 say something AOF Of Advantages and disadvantages

advantage

1、 This mechanism can bring higher data security

, Data persistence .Redis Provided in 3 A synchronization strategy , Synchronization per second 、 Every modification ( Execute an order ) Synchronous and asynchronous .

  • in fact , Synchronization per second is also done asynchronously , It's also very efficient , What's worse is that once the system goes down , Then the modified data will be lost in one second .
  • And every modification is synchronized , We can think of it as synchronous persistence , That is, every data change will be recorded in the disk immediately . Predictably, , This is the least efficient way .
  • As for out of sync , Needless to say , I think everyone can understand it correctly .

2、 Because this mechanism uses append Pattern , So even if there is downtime in the writing process , It will not destroy the existing content in the log file .

  • Because in order to append-only Mode writing , So there's no disk addressing overhead , Very high write performance .
  • in addition , If we only write half of the data in this operation, there will be a system crash , Never mind , stay Redis Before next startup , We can go through redis-check-aof Tools to help us solve data consistency problems .

3、 If AOF The log is too large ,Redis Can be enabled automatically  rewrite  Mechanism . Even if there is a background rewrite operation , It will not affect the client's reading and writing . Because in rewrite log When , The instructions in it will be compressed , Create a minimum log of data to be recovered . When creating a new log file , The old log file is still written as usual . When new merge Post log file ready When , Exchange old and new log files .

4、AOF Include a clear format 、 An easy to understand log file is used to record all data Modify the operating . in fact , We can also reconstruct the data through this file .

shortcoming

1、 For the same number of data sets ,AOF Files are usually larger than RDB file .RDB Speed ratio when recovering large data sets AOF It's faster to recover .

2、 According to different synchronization strategies ,AOF The operation efficiency is often slower than RDB . All in all , The efficiency of synchronization policy per second is relatively high , Efficiency and RDB As efficient as .

3、 before AOF It happened bug , It is through AOF The log of records , During data recovery , No data is recovered as like as two peas. . So , similar AOF This kind of complex command log /merge/ Playback mode , Bi Ji Yu RDB How to persist a complete data snapshot file at a time , More vulnerable , It's easy to have bug . however AOF Just to avoid rewrite The process leads to bug , So every time rewrite It is not based on the old instruction log merge Of , It is based on the data in memory at that time to rebuild the instruction , It's a lot more robust .

15、 How to choose the two persistence methods ?

bgsave Mirror full persistence ,AOF Do incremental persistence . because bgsave It will take a long time , Not real time enough , During shutdown, a lot of data will be lost , So we need to AOF To work with . stay Redis When the instance is restarted , Will use bgsave Persistent files rebuild memory , Reuse AOF Replay the recent operation instructions to fully recover the state before restart .

Generally speaking , If you want to achieve enough PostgreSQL Data security of , You should use both persistence functions at the same time . If you are very concerned about your data , But it can still withstand data loss within a few minutes , Then you can only use RDB Persistence .

Many users only use AOF Persistence , But it's not recommended : Because of timing generation RDB snapshot (snapshot) Very convenient for database backup , also RDB Data set recovery is also faster than AOF Fast recovery , besides , Use RDB You can also avoid the above AOF The problem of procedure .

16、 The interviewer asked what would happen if the machine suddenly turned off ?

Depending on AOF journal sync Configuration of properties , If performance is not required , Every time I write an instruction sync Here's the disk , There's no loss of data . But with high performance, every time sync It's unrealistic , Usually use timing sync , such as 1 second 1 Time , The most you can lose at this time 1 Second data . actually , In extreme cases , It's the most lost 2 Second data . because AOF Threads , Responsible for executing once per second fsync operation , After the operation is completed , Record the last synchronization time . The main thread , Responsible for comparing the last synchronization time , If exceeded 2 second , Blocking wait for success .

17、 The interviewer asked bgsave What is the principle of ?

fork and cow .fork Refer to Redis By creating subprocesses bgsave operation .cow refer to copy on write , After the subprocess is created , Parent child processes share data segments , The parent process continues to provide read and write Services , Writing dirty page data will gradually separate from subprocesses . here bgsave After the operation , Will produce RDB Snapshot file .

18、Redis There are several kinds of data “ Be overdue ” Strategy ?

Redis The expiration strategy of , It means when Redis Cached key Out of date ,Redis How to deal with it .

Redis Provides 3 A data expiration policy :

  • Passive delete : When reading / Write an expired key when , Will trigger lazy delete policy , Delete this expiration directly key .
  • Active delete : Due to the lazy delete strategy, the cold data can not be deleted in time , therefore Redis Will regularly take the initiative to eliminate a number of expired key .
  • Active delete : The memory used currently exceeds maxmemory When limited , Trigger active cleanup strategy .

stay Redis in , At the same time, the above 3 Strategies , That is, they are not mutually exclusive .

19、Redis What kinds of data are there “ Eliminate ” Strategy ?

Redis There are six data elimination strategies :

1.noeviction: Return error when memory limit reaches , And the client tries to execute commands that make more memory available .

2.allkeys-lru: Try to recycle the least used keys (LRU), Make room for new data .

3.volatile-lru: Try to recycle the least used keys (LRU), But only in expired sets of keys , Make room for new data .

4.allkeys-random: Recycle random keys to make room for new data .

5.volatile-random: Recycle random keys to make room for new data , But only in expired sets of keys .

6.volatile-ttl: Recycle keys in expired sets , And priority is given to recovery of survival time (TTL) Shorter keys , Make room for new data .

20、 What is the maximum storage capacity of a string type value ?

512M

21、 be familiar with Redis Which clients of ?

Jedis

Is the most familiar and commonly used client . Light weight , concise , Easy to integrate and transform .

Jedis When multiple threads use a connection, the thread is not safe . You can use connection pools , Create a different connection for each request , be based on Apache common pool Realization . It's like a database , Parameters such as the maximum number of connections can be set .Jedis There are many subclasses of connection pools in

Jedis Yes 4 Working mode : A single node 、 Fragmentation 、 sentry 、 colony .

3 Request mode :Client、Pipeline、 Business .Client Mode is that the client sends a command , Blocking waiting for the server to execute , Then read Return results .Pipeline Mode is to send multiple commands at once , The last time to retrieve all the returned results , This mode reduces the round-trip time of the network and io Number of reads and writes , Greatly improve communication performance . The third is the transaction mode .Transaction Mode is on Redis Transaction management of , When transaction mode is on , All orders ( except exec,discard,multi and watch) It will not be executed immediately after arriving at the server , Will enter a waiting line .

Luttece

And Jedis comparison ,Lettuce It completely overcomes its thread insecurity :Lettuce Is a scalable thread safe Redis client , Support synchronization 、 Asynchronous and reactive patterns (Reactive). Multiple threads can share a connection instance , You don't have to worry about multithreading .

A synchronous invocation : Asynchronous results use RedisFuture packing , Provides a number of callback methods .

Asynchronous call : It's based on Netty Frame building , Support Redis Advanced features of , Such as Pipeline、 Publish subscribe , Business 、Sentinel, colony , Support connection pool .Lettuce yes Spring Boot 2.x Default client , To replace the Jedis. After integration, we don't need to use it alone , Call directly Spring Of RedisTemplate operation , We don't need to worry about connecting and creating and closing .

Redisson

It's a Redis On the basis of implementation Java In memory data grid (In-Memory Data Grid), Provides distributed and scalable Java data structure .

characteristic

  1. be based on Netty Realization , Non blocking IO, High performance
  2. Supports asynchronous requests
  3. Support connection pool 、pipeline、LUA Scripting、Redis Sentinel、Redis Cluster Unsupported transaction , Official advice LUA Scripting Substitute transaction
  4. Master-slave 、 sentry 、 Clusters all support .Spring You can also configure and inject RedissonClient.

Implement distributed locks

stay Redisson It provides a simpler implementation of distributed locks .

22、 What is? Redis Business ?

Multiple commands can be executed at one time , It's essentially a set of commands . All commands in a transaction are serialized , And then serialize it in order , Instead of being inserted into other commands .

Redis The transaction related commands of are :

(1)DISCARD: Cancel the business , Abort execution of all commands in the transaction block

(2)EXEC: Execute the command in the transaction block

(3)MULTI: Mark the beginning of a transaction

(4)UNWATCH: Cancel WATCH Command to all key Surveillance

(5)WATCH key [key...]: Watch one ( Or more )key, If you execute this before the transaction ( Or these )key Altered by other orders , Then the transaction will be interrupted .

23、Redis What are the points for attention in business ?

  1. Rollback is not supported , If there is an incorrect operation in the transaction , Cannot roll back to the state before processing , It needs to be handled by the developer .
  2. Before executing all the instructions in the current transaction , Requests from other clients are not executed at the same time .

24、 Why? Redis Rollback is not supported for transactions ?

Redis Rollback is not supported for transactions , If there is a problem , Will continue with the rest of the order . This is not very consistent with relational databases . The reasons for this are :

  1. Only grammatical errors ,Redis To fail , For example, the assignment of the wrong type , That is to say, these problems can be captured and solved from the program level
  2. Supporting rollback requires a lot of work , Without support ,Redis It can be kept simple 、 Fast features

25、Redis What are the usage scenarios ?

common Redis The usage scenario of is as follows :

Cache data

Redis Key expiration is provided , It also provides a flexible key elimination strategy , therefore , Now? Redis It's used a lot for caching .

Ranking List

Many websites have leaderboard applications , Such as Jingdong's monthly sales list 、 The new ranking of goods by time, etc .Redis The provided ordered collection data class can realize all kinds of complex leaderboard applications .

Counter

For example, the number of visits to products on e-commerce websites 、 The number of videos played on video websites, etc . In order to ensure that the data is effective , Every time you browse, you have to give +1, When the amount of concurrency is high, it is undoubtedly a challenge and pressure to request database operations every time .

Distributed session

Cluster mode , In the case of few applications, it is generally used that the container comes with session Copy function can satisfy , As applications increase, in relatively complex systems , It's usually built to Redis And so on session service ,session No longer managed by containers , But by the session Service and memory database management .

Distributed lock

Distributed lock implementation scheme , There are three common types : database ,Redis、zookeepr.Redis Is one of them .

Such as the overall situation ID、 Reduce inventory 、 Second kill and other scenes , The pessimistic lock of the database can be used in the scenario with small amount of concurrency 、 Optimistic lock to achieve , But in the case of high concurrency , It is not ideal to use database lock to control concurrent access of resources , Greatly affected the performance of the database . You can use Redis Of setnx Function to write distributed locks , If set, return to 1 Indicates that the lock is obtained successfully , Otherwise, the lock acquisition fails , There are more details to consider in practical application .

Social networks

give the thumbs-up 、 Step on 、 Focus on / Be focus on 、 Mutual friends and so on are the basic functions of social networking sites , Social networking sites tend to get a lot of traffic , And traditional relational database types are not suitable for storing this type of data ,Redis Provided hash 、 Data structures such as collections can easily implement these functions .

Latest list

Redis List structure ,LPUSH You can insert a content in the head of the list ID As a keyword ,LTRIM Can be used to limit the number of lists , So the list is always N individual ID, No need to query the latest list , Direct basis ID Go to the corresponding content page .

The messaging system

Message queue is mainly used for business decoupling 、 Peak shaving and asynchronous processing of low real-time services .Redis Provides a release / Subscription and blocking queue function , Can realize a simple message queue system . but Redis Not a professional message queue . Other message queues are recommended :Kafka、RocketMQ、RabbitMQ etc. .

26、 How to use Redis Implement distributed locks ?

Use redis The idea of implementing distributed lock :

1、setnx(String key,String value)

If returns 1, Indicates that the setting is successful , Get lock ;

If returns 0, Description setting failed , I already have this key, Indicates that other threads hold locks , retry .

2、expire(String key, int seconds)

Get lock ( return 1) after , You also need to set the lifetime with , If it's not done in seconds , Like a machine failure 、 Network failure, etc , Key value pair expired , Release the lock , High availability .

3、del(String key)

The lock needs to be released after the business is completed . The release lock has 2 Ways of planting :del Delete key, perhaps expire Set the expiration date to 0( It will expire immediately ).

In the execution of business , If something unusual happens , You can't go on , You should release the lock right away .

If you have a project Redis It's multi machine deployment , Then try to use Redisson Implement distributed locks , This is a Redis Official Java Components .

27、 Implementation conditions of distributed lock ?

1、 Mutual exclusivity , Same as monomer application , Make sure that at any time , Only one client can hold the lock

2、 reliability , To ensure the stability of the system , No deadlock

3、 Uniformity , To ensure that the lock can only be unlocked by the lock , Can't produce A Of the lock is B User unlocking

28、Redis and Zookeeper What's the difference between implementing distributed locks ?

Different ways of implementation ,Redis The implementation is to insert a piece of occupied data , and ZK The implementation is to register a temporary node .

In case of downtime ,Redis Need to wait until the expiration time to automatically release the lock , and ZK Because it's a temporary node , At the time of downtime, the node has been deleted to release the lock .

Redis In the case of not seizing the lock, we usually spin to get the lock , It's a waste of performance , and ZK The way to get the listener is to register , In terms of performance, it is better than Redis.

No one is the best

For those with high performance requirements, it is recommended to use Redis To achieve , otherwise , It is recommended to use Zookeeper To achieve .

29、 How to use Redis Realize distributed current limiting ?

The purpose of throttling is through concurrent access / Request speed limit or request speed limit within a time window to protect the system , Once the limit rate is reached, the service can be rejected .

Redis There are several ways to realize current limiting 3 Kind of , Namely :

1、 be based on Redis Of setnx The operation of , To assign to key Set up expired practices ;

2、 be based on Redis Data structure of zset, Make the request into a zset Array ;

3、 be based on Redis The token bucket algorithm of , The output rate is greater than the input rate , We have to limit the current .

30、 How to use Redis Implement message queuing ?

Redis Of list( list ) Data structures are often used as asynchronous message queues , Use rpush/lpush Operation in queue , Use lpop and rpop Come out of the line .rpush and lpop combination perhaps lpush and rpop combination .

 picture


The client is queued pop Operation to get the message , And then deal with it . Get the message after processing , Reprocessing . And so on and so on , This is the life cycle of the client as a queue consumer .

31、Redis What are the high availability solutions ?

Redis Single copy

Redis Single copy , Using single Redis Node deployment architecture , There is no backup node to synchronize data in real time , No data persistence and backup strategy , It is applicable to pure cache business scenarios with low data reliability requirements .

Redis Multiple copies ( Master-slave )

Redis Multiple copies , Use master-slave (replication) Deployment structure , Compared with single copy, the biggest feature is real-time data synchronization between master and slave instances , And provide data persistence and backup strategy . The master and slave instances are deployed on different physical servers , According to the basic environment of the company , It can realize the strategy of providing external services and separating reading from writing at the same time .

Redis Sentinel( sentry )

Redis Sentinel Is a native high availability solution from the community version , Its deployment architecture mainly includes two parts :Redis Sentinel Clusters and Redis Data clustering .

among Redis Sentinel Clusters are made up of a number of Sentinel A distributed cluster of nodes , Fault detection can be realized 、 Auto fail over 、 Configuration center and client notification .Redis Sentinel The number of nodes should meet 2n+1(n>=1) The odd number of .

Redis Cluster

Redis Cluster It's the community version Redis Distributed clustering solution , Main solution Redis Distributed requirements , such as , When single machine memory is encountered , When there are bottlenecks such as concurrency and traffic ,Redis Cluster Can play a good load balancing purpose .

Redis Cluster Minimum configuration of cluster nodes 6 More than nodes (3 Lord 3 from ), The master node provides read and write operations , Slave as standby , Don't offer requests , For failover purposes only .

Redis Cluster Use virtual slot partition , All keys map to by hash function 0~16383 In an integer slot , Each node is responsible for maintaining a part of the slot and the key value data printed by the slot .

Redis Since the research

Redis Self developed high availability solution , Mainly reflected in the configuration center 、 Fault detection and failover On the handling mechanism of , Usually it needs to be customized according to the actual online environment of the enterprise business .

32、 What is? Redis Master slave synchronization ?

Redis Master slave synchronization of (replication) Mechanism , allow Slave from Master Where? , Copy to full data backup through network transmission , So as to achieve the master-slave mechanism .

  • The main database can be read and write , Automatically synchronize the data to the slave database when the write operation occurs , And the slave database is generally read-only , And receive the data synchronized from the main database .
  • A master database can have more than one slave database , A slave database can only have one master database .
  • The first synchronization , The master node does it once bgsave operation , At the same time, the subsequent modification operations are recorded in memory buffer , To be completed RDB Full file synchronization to replication node , When the replication node acceptance is complete, it will RDB The image is loaded into memory . After loading , Then inform the main node to synchronize the modified operation records to the replication node for replay, and the synchronization process is completed .

benefits

adopt Redis The copy work of , Can achieve a good database read and write separation , Increase the load capacity of the server . The main database is mainly written , And reading from the database .

actually , We don't recommend it very much Redis in , Using read write separation . There are two main reasons :

  • Redis Sentinel Only the failure transfer of the primary node is guaranteed , And for example Jedis library , It only monitors the changes of the master node , But from the case of node failure ,Jedis It's not handled . This will lead to ,Jedis Read access to the slave node , Cause problems . Of course ,Redisson The library is powerful , It supports the fault monitoring of slave nodes .
  • If you reach a volume that requires read-write separation , General write operations are not necessarily less , Consider Redis Cluster programme , More reliable .

33、 How to use Redis Sentinel High availability ?

Redis Of Sentinel The system is used to manage multiple Redis The server (instance), The system performs the following three tasks :

  • monitor (Monitoring):Sentinel Constantly check whether your master and slave servers are working properly .
  • remind (Notification): When someone is being monitored Redis When there's a problem with the server , Sentinel Can pass API Send notifications to administrators or other applications .
  • Automatic failover (Automatic failover): When a primary server doesn't work , Sentinel An automatic failover operation will start , It will upgrade one of the failed master servers from the server to the new master server , And let the other slave servers of the failed master server copy the new master server ; When a client tries to connect to a failed primary server , The cluster will also return the address of the new primary server to the client , So that the cluster can use the new master server instead of the failed server .

Redis Sentinel It's a distributed system , You can run multiple... In one architecture Sentinel process (progress), These processes use gossip protocol (gossip protocols) To receive information about whether the primary server is offline , And use voting protocols (agreement protocols) To decide whether to perform automatic failover , And which slave server to choose as the new master .

although Redis Sentinel Release as a separate executable redis-sentinel , But in fact, it's just a running in a special mode Redis The server , You can start a normal Redis When the server is given –sentinel Option to start Redis Sentinel .

34、 If you use Redis Cluster High availability ?

Redis Cluster It's the community version Redis Distributed clustering solution , Main solution Redis Distributed requirements , such as , When single machine memory is encountered , When there are bottlenecks such as concurrency and traffic ,Redis Cluster Can play a good load balancing purpose .

Redis Cluster Minimum configuration of cluster nodes 6 More than nodes (3 Lord 3 from ), The master node provides read and write operations , Slave as standby , Don't offer requests , For failover purposes only .

Redis Cluster Use virtual slot partition , All keys map to by hash function 0~16383 In an integer slot , Each node is responsible for maintaining a portion of the slots and the key-value data that the slots map to .

 picture


advantage

No central Architecture ;

Data according to slot Storage is distributed over multiple nodes , Data sharing between nodes , Data distribution can be adjusted dynamically ;

Extensibility : It can be linearly extended to 1000 Multiple nodes , Nodes can be added or removed dynamically ;

High availability : Some nodes are not available , The cluster is still available . By increasing the Slave do standby Copy of data , It can realize automatic fault failover, Nodes pass through gossip Protocol exchanges status information , By voting Slave To Master The role of ;

Lower maintenance cost , Improve the scalability and availability of the system .

shortcoming

Client The implementation is complex , The driver requires the implementation of Smart Client, cache slots mapping Information and update in time , It improves the difficulty of development , The immaturity of the client affects the stability of the business . At present, only JedisCluster Relatively mature , The exception handling part is not perfect , For example, the common “max redirect exception”.

Nodes will block for some reason ( The blocking time is longer than clutser-node-timeout), Be judged to be offline , such failover There is no need to .

Data is replicated asynchronously , Strong consistency of data is not guaranteed .

When multiple businesses use the same cluster , It's impossible to distinguish between hot and cold data according to statistics , Poor resource isolation , It is easy to interact with each other .

Slave Act as... In a cluster “ Cold standby ”, Can't relieve reading pressure , Yes, of course SDK To improve Slave Utilization of resources .

35、 say something Redis The concept of hash slot ?

Redis Cluster No use of consistency hash , It introduces the concept of hash slot .

Redis Cluster has 16384 Hash slot , Every key adopt CRC16 Check pair 16384 Take the mold to decide which slot to place , Each node in the cluster is responsible for part of hash Slot . Because the biggest is 16384 Hash slot , So consider Redis Each node in the cluster can be assigned a hash slot , So at most we support 16384 individual Redis node .

Why 16384 Well ?

We mainly consider the network bandwidth in the cluster , and 16384 just 2K Byte size .

36、Redis Cluster What is the master-slave replication model of ?

In order to make the cluster available even if some nodes fail or most nodes cannot communicate , So the cluster uses Master slave copy Model , Each node will have N-1 Replication nodes .

therefore ,Redis Cluster Can be said to be Redis Sentinel Enhanced version with split . It can also be said that :

  • Redis Sentinel Focus on high availability , stay master It will automatically slave Upgrade to master , Continued provision of services .
  • Redis Cluster Focus on scalability , In a single Redis When there is not enough memory , Use Cluster For fragmentation storage .

37、Redis What's the function of your sentry ?

The sentry is Redis A very important component of cluster architecture , The main functions are as follows :

  • Cluster monitoring , Responsible for monitoring Redis Master and Slave Is the process working ;
  • Notice of news , If a Redis The instance is faulty , Then the sentry is responsible for sending a message as an alarm to the Administrator ;
  • Fail over , If Master node Hang up , Will automatically transfer to Slave node On ;
  • Configuration center , If a fail over occurs , notice Client Client new Master Address .

38、Redis What's the difference between a sentry and a group ?

Redis Our sentinel role is to manage multiple Redis The server , Provides monitoring 、 Reminders and automatic failover capabilities . Sentinels can guarantee that when the main server is down , You can choose one from the slave server to be the master server , Transfer others from the server to the new host .Redis The main functions of sentinels are :

  • Cluster monitoring : Yes Redis The master and slave processes of the cluster are monitored , Judge whether it works properly .
  • Notice of news : If there is Redis The instance is faulty , Then the Sentry can send an alarm message to the administrator .
  • Fail over : If the host (master) The node is hung , Then it can be automatically transferred from (slave) Node .
  • Configuration center : When there is a fault , After the failure has been transferred , The configuration center will inform the client of the new host (master) Address .

Redis The function of cluster is to solve the problem of single machine Redis The problem of limited capacity , Allocate data to multiple machines according to certain rules , Access to memory per second is not limited to a single server , It can benefit from the high scalability of distributed clusters .

39、 What does cache hit ratio mean ?

Generally speaking , The higher the cache hit rate , The higher the benefit of caching , The better the performance of the application .

  • A cache hit :  You can get the required data from the same cache
  • ** Cache miss :** Unable to get the required data in the cache , You need to query the database or other data storage carrier again .

cache hit rate = The number of times data is retrieved from the cache / The total number of times to get data

40、 If Redis There are 1 One hundred million key, Among them is 10w individual key It starts with a fixed known prefix , If you find them all ?

Use  keys  The command can scan out the key list .

  • The other side then asked : If this Redis Providing services for online business , That use  keys  What's wrong with the instructions ?
  • At this time, you have to answer Redis A key feature :Redis Of single thread .keys  Instructions can cause threads to block for a period of time , Online services will pause , Until the command is executed , Service can be restored . It can be used at this time  scan  Instructions ,scan  The instruction can extract the specified pattern without blocking key list , But there is a certain probability of repetition , In the client to do a duplicate can , But the overall time spent will be more than the direct use of  keys  Commander .

41、 Please talk about... In your production environment Redis How it was deployed ?

Please feel confident to taste , As long as we have this problem ,98% I'll hang up with you , Here are the reference answers :

  • Redis Cluster ,10 Taiwan machine ,5 Machines deployed Redis Main instance , in addition 5 Machines deployed Redis From the example of , Each master instance has a slave instance ,5 Nodes provide external reading and writing services , Read write peak of each node qps May be up to per second 5 ten thousand ,5 The maximum number of machines is 25 Ten thousand read and write requests per second .
  • What is the configuration of the machine ?32G Memory + 8 nucleus CPU + 1T disk , But assigned to Redis The process is 10G Memory , General online production environment ,Redis Try not to exceed 10G, exceed 10G There may be problems . that ,5 Machines provide external reading and writing , Altogether 50G Memory .
  • Because each primary instance has a slave instance attached , So it's highly available , Any primary instance goes down , Will automatically fail over ,Redis The slave instance will automatically become the master instance and continue to provide read and write services .
  • What data do you write into memory ? What is the size of each piece of data ? Commodity data , Each data is 10kb .100 The data is 1mb ,10 Ten thousand pieces of data 1G . Memory resident 200 10000 item data , Occupied memory is 20G , Less than the total memory 50% . The current peak is 3500 Left and right requests .
  • In fact, large companies , There will be infrastructure Team Responsible for the operation and maintenance of the cache cluster .

42、 You know what Redis Partition implementation scheme ?

Redis Partition plan , There are two main types :

  • Client partition , It is on the client side that the data will be stored Redis Node or from which Redis Node read . Most clients have implemented client partition . Case study :Redis Cluster And client partition .
  • Agent partition , Means that the client sends the request to the agent , Then the agent decides which node to write or read data to . The agent determines which requests are based on partition rules Redis example , And then according to Redis The response result of is returned to the client . Case study :Twemproxy and Codis .

Query route (Query routing) It means , It's the client that randomly requests any Redis example , Then from Redis Forward the request to the correct Redis node .Redis Cluster A hybrid query route is implemented , But not directly from a request to Redis The node forwards to another Redis node , But with the help of the client Redirect To the right Redis node .

43、 How to improve Redis shooting ?

Provides cache hit ratio , There are usually the following ways :

  • Cache preload
  • Increase cache storage
  • Adjust cache storage data type
  • Increase cache update frequency

44、 How to optimize the Redis Memory footprint ?

There are six ways to do this Redis Memory optimization :

  • redisObject object
  • Reduce key value object
  • Shared object pool
  • String optimization
  • Coding optimization
  • control key The number of

45、 What is cache penetration ? How to solve ?

A lot of requests rush into the system , And the data is Redis Does not exist in the , All the requests go to the database. Kill the database . The reason for this is that the system design is unreasonable 、 Cache data update is not timely , Or reptiles and other malicious ***. The solution is :

1. Using the bloon filter

Store the parameters of the query in a bitmap in , Before the query cache , Find a new one bitmap, Verify the parameters inside . If it's verified bitmap in , Then query the underlying cache data , If bitmap There are no query parameters in , Then intercept , No more cached data queries .

2. Caching empty objects

If the result of the query from the database is empty , Still cache this result , Then use it key When getting data , Even if the data doesn't exist ,Redis You can also return the result directly , Avoid multiple access to the database .

But the disadvantage of caching null values is :

  • If there is *** Malicious random access , Causing too many null values in the cache , That may cause a lot of memory space waste . But you can also set a short expiration time for these data to control ;
  • If the inquiry key Corresponding Redis The cache null value has not expired , The database now has new data , There will be inconsistency between the database and the cache data . But it can be guaranteed that when the database has data, the cache will be updated to solve the problem .

46、 What is a cache avalanche ? How to solve ?

Cache avalanche is when a large number of caches fail , A large number of requests to access the direct request database , The database server cannot resist the request or hang up . At this time, websites often appear 502 error , Causes the website to be unavailable .

When preventing cache avalanches , The following principles are recommended :

  • Plan the cache expiration time reasonably , You can add a random number to the cache time , Prevent uniform time expiration ;
  • Reasonably evaluate the load pressure of the database , This is conducive to a reasonable range of partial cache loss , The database can also be accessed normally ;
  • Overload protection or application layer current limiting for database , In this case, the website is generally in a big flow 、 High concurrency , When the whole server can't bear it , A current limiting protection measure that can be adopted ;
  • Finally, we can also consider the design of multi-level cache , Achieve high availability of cache .

47、 Yes Redis Performance optimization , Some suggestions ?

1、Master It's best not to do any persistence work , Such as RDB Memory snapshot and AOF Log files .

2、Master call BGREWRITEAOF rewrite AOF file ,AOF It's going to take up a lot of when you rewrite it CPU And memory resources , Lead to service load Too high , Temporary suspension of service .

3、 Try to avoid adding too many slaves to the stressed master library .

4、 Master and slave replication do not use graphical structures , One - way linked list structure is more stable , namely :Master <- Slave1 <- Slave2 <- Slave3... .

5、Redis Performance issues with master-slave replication , For master-slave replication speed and connection stability ,Slave and Master Preferably on the same LAN .

Reference resources

http://redis.cn/、 https://www.jianshu.com/p/fb499fbbccbd、 https://segmentfault.com/a/1190000014507534、 https://www.imooc.com/article/36399

summary

This kind of chain fire , It's only used when people are in a hurry ,“ Temporary sharpening , Not fast will also shine ”.

If you want long-term development 、 Want to learn more 、 Want to be more DIAO Words , I suggest buying this Redis E-books or paper books , Look at the official website , Look for some good information , More practical , Fang is king .

It's not easy to code words , I'm looking forward to your praise , forward , Thank you very much !