Redis foundation - understand how redis implements data persistence

DetectiveHLH 2020-11-10 10:44:19
redis foundation understand redis implements

The previous article introduced Redis The related use and underlying principles of simple data structure of , Let's talk about this article Redis How to ensure high availability .

Data persistence

We know that although the single machine Redis Although the performance is excellent , A single machine can carry it 10w Of QPS, This is due to its memory based fast read and write operations , Well, if at some time Redis What should I do if I hang up suddenly ? We need a Persistence The mechanism of , To save data in memory , Otherwise, the data will be directly lost .

Redis There are two ways to achieve data persistence , Namely RDB(Redis Database) and AOF(Append Only File), You can simply put RDB Understood as a moment Redis Data snapshot in memory , and AOF It is a collection of all instructions that record all changes to memory data ( That is to say Redis Set of instructions ), And these two methods will generate the corresponding file to the disk , Data persistence , Convenient for the next time .

Let's talk about these two persistence schemes respectively .


stay redis In the middle of RDB There are two ways to take a snapshot , One is to use save, The other is bgsave, But the underlying implementation , It calls the same function , It's called rdbsave, It's just that it's called in a different way .

Generation method


save The command calls directly rdbsave Method , This will Blocking Redis The main process , Until the snapshot file is generated .

void saveCommand(client *c) {
if (server.rdb_child_pid != -1) {
addReplyError(c,"Background save already in progress");
rdbSaveInfo rsi, *rsiptr;
rsiptr = rdbPopulateSaveInfo(&rsi);
if (rdbSave(server.rdb_filename,rsiptr) == C_OK) {
} else {


bgsave Orders will fork Make a sub process , from fork Out of the subprocess call rdbsave. The parent process will continue to respond to read and write requests from the client . Subprocess complete RDB After the file is generated, it will send a signal to the parent process , Notifies the parent process that the save is complete .

void bgsaveCommand(client *c) {
int schedule = 0;
/* The SCHEDULE option changes the behavior of BGSAVE when an AOF rewrite
* is in progress. Instead of returning an error a BGSAVE gets scheduled. */
if (c->argc > 1) {
if (c->argc == 2 && !strcasecmp(c->argv[1]->ptr,"schedule")) {
schedule = 1;
} else {
rdbSaveInfo rsi, *rsiptr;
rsiptr = rdbPopulateSaveInfo(&rsi);
if (server.rdb_child_pid != -1) {
addReplyError(c,"Background save already in progress");
} else if (hasActiveChildProcess()) {
if (schedule) {
server.rdb_bgsave_scheduled = 1;
addReplyStatus(c,"Background saving scheduled");
} else {
"Another child process is active (AOF?): can't BGSAVE right now. "
"Use BGSAVE SCHEDULE in order to schedule a BGSAVE whenever "
} else if (rdbSaveBackground(server.rdb_filename,rsiptr) == C_OK) {
addReplyStatus(c,"Background saving started");
} else {

That's why Redis It's single threaded , But it can be generated RDB Provide external services at the same time .fork yes unix Main methods of creating process on system , All data of the parent process will be copied to the child process , Shared memory space between parent and child processes .

fork after , The operating system kernel sets all memory in the parent process as read-only , Only when a write occurs , Page break will occur , The kernel will copy the corresponding memory page , The parent and child processes each hold one , So it's generating RDB In the process , Due to the use of COW, Dirty memory pages are gradually separated from subprocesses .

Is it possible to call bgsave In the process of , I call again save Orders , Is it not that the time to generate two copies RDB file ?

Actually calling save On command ,Redis Will judge bgsave Is execution in progress , If you are executing, the server can no longer call the underlying rdbsave Function , This avoids resource competition between two commands .

for example , stay save In command , There are the following judgments :

if (server.rdb_child_pid != -1) {
addReplyError(c,"Background save already in progress");

And in the bgsave There are the following judgments :

if (server.rdb_child_pid != -1) {
addReplyError(c,"Background save already in progress");
} else if (hasActiveChildProcess()) {

You can see that they are all judgments of the same variable , as follows :

pid_t rdb_child_pid; /* PID of RDB saving child */

let me put it another way , Calling save、bgsave When ordered , Will judge ahead of time bgsave Is it still running , If it's running , Will not continue bgsave command . and save The command itself is blocked , If there are other commands coming, they will be blocked , until save completion of enforcement , To deal with it .

Then I will RDB How to use the file after it is generated ?

Redis Called when the server is started rdbLoad function , It will generate RDB The file is loaded into memory , During loading , Every load 1000 A key will handle a request that has arrived , But only deal with it publish、subscribe、psubscribe、unsubscribe、punsubscribe These five orders . All other requests return errors , Until the load is complete .

You play so well ,RDB What are the advantages and disadvantages of ?


RDB Policies can flexibly configure cycles , It depends on what kind of backup strategy you want . for example :

  • Generate the latest every hour 24 Hours of data
  • Generate data for the last week every day
  • Generate data for the last month every day

Based on this strategy , Can quickly restore the data of a certain period of time before .

secondly ,RDB It's a great fit Cold backup , You can take RDB Files are transferred to other storage media after storage . It can even do Cross cloud storage , For example, put OSS At the same time , Put it again S3 On , Cross cloud storage makes data backup more robust .

and , be based on RDB Recovery rate ratio of mode AOF faster , because AOF It's one by one Redis Instructions ,RDB It's what the data looks like in the end . If there is a large amount of data AOF All instructions replay than RDB More slowly .


RDB As a Data persistence The scheme is feasible , But if you want to pass RDB Achieve Redis Of High availability ,RDB It's not so appropriate .

Because if Redis At this time, there is no time to generate the data in memory RDB file , Just hang up , So, since the last successful generation RDB The newly added data will be lost , And I can't get it back .

and , If there is a large amount of data in memory ,RDB Even through fork Child process , But it also needs to be occupied by the machine CPU resources , There may also be a lot of abnormal interruptions , It can also cause the whole thing Redis Stop responding for hundreds of milliseconds .


As mentioned above RDB Can't meet Redis High availability . Because in some cases , Will permanently lose data for a period of time , So let's talk about another solution AOF. First of all, we have to have a concept , That's it RDB Is right for the current Redis Server Data snapshot in , and AOF It is a record of the change order ( All fetch operations are not logged , Because of the current Redis The data didn't change ).

But that's why ,AOF Documents are better than RDB The file is bigger . Let's talk about one Redis Command request from client to AOF Documentation process .

AOF Record the process

First Redis Communication is required between the client and the server , The client is not sending the string we wrote , It's specialized Agreement text . If you can be familiar with Thrift perhaps Protobuf You should be able to understand this agreement .

For example, execute command SET KEY VALUE, When it reaches the server, it becomes "*3\r\n$3\r\nSET\r\n$3\r\nKEY\r\n$5\r\nVALUE\r\n".

then Redis The server will be based on the content of the protocol text , Choose the right handler To deal with . When the client sends instructions to Redis After the server , As long as the command is executed successfully , This command will be propagated to AOF In the program .

Be careful , Spread to AOF The program will not write to disk immediately after the program , Because frequent IO The operation will bring huge cost , It will greatly reduce Redis Performance of , The text of the agreement will be written to Redis In the server aof_buf In the middle , Also called AOF Of Write buffer .

You've written all this to the buffer , When does it land ?

whenever serverCron( First there is one. Timing task The concept of , I'll talk about it in a minute serverCron What is it ) When executed ,flushAppendOnlyFile This function is called .

This command will call write Writes data written to the buffer to AOF In file , But it's still at this time No, real Drop to disk . This is a OS In order to improve the efficiency of writing files , Data will be temporarily written to OS In the buffer , Wait until the buffer is filled or exceeds the specified time , Will call fsync perhaps sdatasync Actually write the contents of the buffer to the disk .

But if the machine goes down in the meantime , that Data will still be lost . So if you want to really AOF The file is saved on disk , You have to call the two functions mentioned above .



Now let's talk about it in detail serverCron function , It is mainly used for processing Redis Medium Routine tasks .

What is a routine task ?

As mentioned above AOF Write buffer , Every time serverCron When it is executed, the AOF write file ( Of course ,OS Will write their own buffer in ). The rest is like AOF and RDB Persistent operation of , Master slave synchronization and cluster related operations , Clean up invalid clients 、 Expiration keys and so on .

So this one cron How often is it executed ?

Many blogs are direct conclusions ,100ms Do it once , Words alone are no proof , We directly copy the source code . Here is serverCron Function definition of .

/* This is our timer interrupt, called server.hz times per second.
* .............
int serverCron(struct aeEventLoop *eventLoop, long long id, void *clientData) {
server.hz = server.config_hz;

In order to avoid affecting people's thinking , I've omitted code and comments that are temporarily useless to us . You can see that there are called server.hz times per second. It means serverCron This function will be called in every second. server.hz Time , So this one server.hz What is it ?


I'm sure you all know that HZ( Hertz ) This unit , It is the SI unit of frequency , Represents the number of times each periodic event occurs . therefore , We know that this configuration item is used to control the frequency of periodic events .

The assignment is given in the above function , You can see that the initial value is derived from redis.conf Configuration file for . Let's take a look at the configuration .

# Redis calls an internal function to perform many background tasks, like
# closing connections of clients in timeout, purging expired keys that are
# never requested, and so forth.
# Not all tasks are performed with the same frequency, but Redis checks for
# tasks to perform according to the specified "hz" value.
# By default "hz" is set to 10. Raising the value will use more CPU when
# Redis is idle, but at the same time will make Redis more responsive when
# there are many keys expiring at the same time, and timeouts may be
# handled with more precision.
# The range is between 1 and 500, however a value over 100 is usually not
# a good idea. Most users should use the default of 10 and raise this up to
# 100 only in environments where very low latency is required.
hz 10

Simply extract useful information ,Redis It will call functions internally to perform many background tasks , And the frequency of calling these functions depends on this hz To decide , The default value is 10. That means , As mentioned above serverCron The function is executed in one second 10 Time , So, on average, it's per cent 100ms(1000ms/10) Call once .

Write strategy

The above said , If Redis Of AOF Already located in OS In the buffer of , If it goes down at this time , that AOF Data will also be lost .

You can't do that , What's the point of persistence ? How can data not be lost ?

I need to talk about it AOF Log writing strategy , It has three strategies , They are as follows :

  • always Each command writes a file and synchronizes to disk
  • everysec Synchronize data to disk every second
  • no Do not force writing , wait for OS Decide for yourself when to write

Obviously always This strategy is not desirable in a real production environment , Every command writes a file , It's going to make a huge difference IO expenses , Will occupy Redis Many resources of the server , Reduce Redis Service efficiency of .

And if you use everysec In terms of strategy , Even if there is a power failure , The machine is down , I'll lose a second of data at most .

and no Then the system will be fully delivered to the operating system for scheduling , More data may be lost .

, What about this AOF How to use the document , How to recover ?

As mentioned above ,AOF The file records all write commands from the client , So the server just needs to read in and replay it again Redis State recovery for .

however ,Redis The command can only be executed in the context of the client , therefore Redis A pseudo client with no network connection was created to execute the command , Until the command is executed .

Buddy , You can't do that , In case AOF There is a large amount of log data , Don't you want to recover for a long time , Isn't the service unavailable ?

You bet , With the server running ,AOF The amount of data will be larger and larger , Playback will take more and more time . therefore Redis There is one rewrite (AOF Rewrite) Mechanism , To achieve the right AOF Of documents Slimming .

Although the name is right AOF The reduction of documents , But the actual operation to be done is the same as that already generated AOF The document has nothing to do with a dime .

So-called Slimming By reading Redis The current data status of the server , Of course , This is when the server is running normally . In fact, you can also understand it as a snapshot , It's just not a real binary , It's a command that directly sets the snapshot value .

Let me give you an example , Suppose you Redis There's a key called test, The history of its value change is 1 -> 3 -> 5 -> 7 -> 9 such , So if it's normal AOF The file will be recorded 5 strip Redis Instructions . and AOF Rewrite Now step in , Only one will be recorded test=9 Data like this .

And before AOF The file is still written as usual , When new AOF Replace the file after it is generated .

you tm I'm kidding ? you are here rewrite At the same time , The server is still processing normal requests , At this point, if you change the state of the server , You've been slimming down AOF File data is not consistent ?

This will happen , however Redis Through one AOF Rewrite buffer To solve this problem .

When rewrite After start ,Redis Meeting fork A subprocess , Let the child process to implement AOF Thin operation of , The parent process can handle the request normally .AOF The override buffer will be in the rewrite Start creating child processes, and then start using , here Redis The server will send the written instructions to two places at the same time :

  1. aof_buf, That's what I mentioned above AOF Write buffer for file
  2. AOF Rewrite buffer

You may ask , Why do you record in two places ? As mentioned above ,Redis perform Slimming In operation , The conventional AOF The file is still generated normally , So the new Redis The instruction must be sent to the write buffer .

And sent to AOF The buffer is rewritten to replay the Slimming During the operation, the Redis State changes , such Slimming After that AOF The file status can guarantee the Redis In the same state . in general , Just to make sure Slimming Of AOF Data status in file and Redis The memory state at that time keeps the data consistent .


About Redis Data persistence , So much to talk about first , The plan for the next issue should be to have a chat Redis The mechanism of high availability , Those who are interested can search through wechat 「SH The whole stack of notes 」 Continuous attention , The official account will be pushed first than other platforms. .

Past articles :


  1. 【计算机网络 12(1),尚学堂马士兵Java视频教程
  2. 【程序猿历程,史上最全的Java面试题集锦在这里
  3. 【程序猿历程(1),Javaweb视频教程百度云
  4. Notes on MySQL 45 lectures (1-7)
  5. [computer network 12 (1), Shang Xuetang Ma soldier java video tutorial
  6. The most complete collection of Java interview questions in history is here
  7. [process of program ape (1), JavaWeb video tutorial, baidu cloud
  8. Notes on MySQL 45 lectures (1-7)
  9. 精进 Spring Boot 03:Spring Boot 的配置文件和配置管理,以及用三种方式读取配置文件
  10. Refined spring boot 03: spring boot configuration files and configuration management, and reading configuration files in three ways
  11. 精进 Spring Boot 03:Spring Boot 的配置文件和配置管理,以及用三种方式读取配置文件
  12. Refined spring boot 03: spring boot configuration files and configuration management, and reading configuration files in three ways
  13. 【递归,Java传智播客笔记
  14. [recursion, Java intelligence podcast notes
  15. [adhere to painting for 386 days] the beginning of spring of 24 solar terms
  16. K8S系列第八篇(Service、EndPoints以及高可用kubeadm部署)
  17. K8s Series Part 8 (service, endpoints and high availability kubeadm deployment)
  18. 【重识 HTML (3),350道Java面试真题分享
  19. 【重识 HTML (2),Java并发编程必会的多线程你竟然还不会
  20. 【重识 HTML (1),二本Java小菜鸟4面字节跳动被秒成渣渣
  21. [re recognize HTML (3) and share 350 real Java interview questions
  22. [re recognize HTML (2). Multithreading is a must for Java Concurrent Programming. How dare you not
  23. [re recognize HTML (1), two Java rookies' 4-sided bytes beat and become slag in seconds
  24. 造轮子系列之RPC 1:如何从零开始开发RPC框架
  25. RPC 1: how to develop RPC framework from scratch
  26. 造轮子系列之RPC 1:如何从零开始开发RPC框架
  27. RPC 1: how to develop RPC framework from scratch
  28. 一次性捋清楚吧,对乱糟糟的,Spring事务扩展机制
  29. 一文彻底弄懂如何选择抽象类还是接口,连续四年百度Java岗必问面试题
  30. Redis常用命令
  31. 一双拖鞋引发的血案,狂神说Java系列笔记
  32. 一、mysql基础安装
  33. 一位程序员的独白:尽管我一生坎坷,Java框架面试基础
  34. Clear it all at once. For the messy, spring transaction extension mechanism
  35. A thorough understanding of how to choose abstract classes or interfaces, baidu Java post must ask interview questions for four consecutive years
  36. Redis common commands
  37. A pair of slippers triggered the murder, crazy God said java series notes
  38. 1、 MySQL basic installation
  39. Monologue of a programmer: despite my ups and downs in my life, Java framework is the foundation of interview
  40. 【大厂面试】三面三问Spring循环依赖,请一定要把这篇看完(建议收藏)
  41. 一线互联网企业中,springboot入门项目
  42. 一篇文带你入门SSM框架Spring开发,帮你快速拿Offer
  43. 【面试资料】Java全集、微服务、大数据、数据结构与算法、机器学习知识最全总结,283页pdf
  44. 【leetcode刷题】24.数组中重复的数字——Java版
  45. 【leetcode刷题】23.对称二叉树——Java版
  46. 【leetcode刷题】22.二叉树的中序遍历——Java版
  47. 【leetcode刷题】21.三数之和——Java版
  48. 【leetcode刷题】20.最长回文子串——Java版
  49. 【leetcode刷题】19.回文链表——Java版
  50. 【leetcode刷题】18.反转链表——Java版
  51. 【leetcode刷题】17.相交链表——Java&python版
  52. 【leetcode刷题】16.环形链表——Java版
  53. 【leetcode刷题】15.汉明距离——Java版
  54. 【leetcode刷题】14.找到所有数组中消失的数字——Java版
  55. 【leetcode刷题】13.比特位计数——Java版
  56. oracle控制用户权限命令
  57. 三年Java开发,继阿里,鲁班二期Java架构师
  58. Oracle必须要启动的服务
  59. 万字长文!深入剖析HashMap,Java基础笔试题大全带答案
  60. 一问Kafka就心慌?我却凭着这份,图灵学院vip课程百度云