Previous review

CommitLog piece ——【RocketMQ Source code analysis 】 Deep into message storage (1)

ConsumeQueue piece ——【RocketMQ Source code analysis 】 Deep into message storage (2)

The previous two articles have talked about how messages are stored in CommitLog, as well as ConsumeQueue Construction process of , In chapter three , We have a hurdle we have to cross ,MappedFile —— Memory file mapping .

MappedFile The existence of is RocketMQ The key to choosing to store messages directly to disk , In the first chapter CommitLog In the beginning of the storage process , I just wrote a thought .

  1. Both memory and local disk are used
  2. Filling and swapping
  3. Files mapped to memory
  4. Random read interface to access

Here are some key sentences , We can't do without what this article is going to say MappedFile.

RocketMQ Since you want to store files interactively with disk , Different IO Methods vary widely in performance gap , How to connect with disk efficiently / Memory interaction , It is an important sign of the strength of many middleware related to storage .

Implement an in-process queue based message persistence storage engine

This is the topic of Tianchi middleware competition a few years ago , The goal is to design a system that uses limited memory 、 More disk space to implement a message queue , In fact, the thinking has been mentioned in the first article , The point is that he requires the queue to support aggregation operations .

It reminds me of ElasticSearch The aggregation scene of , If you want to implement such a complex aggregation function , It's too Southern .

But the good thing is that the topic only requires adding messages in a specified period of time , This is nothing more than maintaining a message storage offset and time storage .

To learn more about memory file mapping , We can read the source code , This is relative to CommitLog、ConsumeQueue It's much lower , More about IO、Buffer、PageCache Such as knowledge .

From page table to zero copy

When I used to learn assembly language , There are two addressing related registers .

Segment register 、 Index register .

stay 8086 The s , The address bus is 20 position , But registers 16 position , Addressing capacity is limited , In order to ensure 1M The ability to address , It's going to be two 16 Bit registers are used together , In the form of segment base address and offset address , achieve 1M Addressing capability .

The idea is the same in OS protected mode , If we had one 32 Bit operating system , Memory 4GB.

Let's think about its memory layout , Kernel space and user space are well-known concepts , If the memory space doesn't do anything , In order, let's visit , The first big problem is memory isolation , How can two processes avoid memory pollution , And that leads to Java A problem of memory allocation in virtual machines , The allocated memory space is cleaned up by the garbage collector , The rest of the space may not be continuous , The next object that needs to occupy large memory may not be able to be stored ,JVM You can choose to recycle - Clean up in such a way that there is no debris , This is because there are references on the stack pointing to the heap , Even if a large object is moved, don't worry , But the operating system is different , If you want to use something like JVM Recycling - Clean up the way to reduce fragmented memory , The first problem we have to face is Change of address , Subsequent processes may not find the target when addressing .

Notice here Change of address , Because we'll also talk about , Operating system PageCache Improper operation can also cause this problem .

There's another problem , This kind of sequential space is not safe , All processes can access each other's addresses , This is a common tool for some modifiers .

Based on the above questions , The operating system is mapped into protected mode , Adjust memory space to virtual memory based on page table , Separate from actual physical memory .

The current page table is usually a secondary page table , The so-called two-level page table is to paginate the page table again , All page table items in a page table are stored continuously , Page tables are essentially a bunch of data , It is also stored in memory in pages .

The first level is called the page table of contents . The physical address of each page table is shown in the page directory table as a page directory entry (PDE) In the form of ,4MB Page table can be divided into 1K(4MB/4KB) A page , The description of each page requires 4 Bytes , So the page table of contents takes up 4K size , It's exactly the size of a standard page , It points to the second level table . The high of the linear address 10 Bit produces the first level index , In the table entries obtained from the index , Designated and selected 1K One of the secondary tables is a page table .

The second level is called the page table , Store in a 4K The size of the page , contain 1K Table items , Each table entry contains the physical base address of a page . In the middle of a linear address 10 Bits produce a secondary index , You can get a page table entry that contains the physical address of the page . The height of this physical address 20 Bit and linear address of low 12 Bits form the final physical address .

With the page table can be a good division of process space , And reduce debris space , For a process , Theoretically, the maximum usable space is 4GB. Based on this , Most of the memory operations of the operating system are based on pages (4KB).

The mapping of virtual memory makes it more convenient for the operating system to manage and divide memory , The unit that actually maps the virtual address to the physical address is MMU,mmap Memory file mapping is the same , adopt MMU Map to file .

To solve the problem of disk IO The problem of inefficiency , The operating system adds space to the process space , Used for address mapping with disk files , This part of memory is also the virtual memory address , Operate this part of memory through the pointer , The system will automatically write the processed page back to the corresponding disk file location , You don't have to call the system read、write Such as function , Kernel space changes to this area also directly reflect user space , So we can share files among different processes .

This part of memory mapping needs to maintain a page table , For managing memory —— The mapping of file addresses , If the current virtual memory address cannot find the corresponding physical address , There will be so-called missing pages , When the page is missing, the system will set the page number in... According to the address offset PageCache Check whether the destination address has been cached in , If there is, point directly to the PageCache Address , If not, you need to load the target file into PageCache in .

adopt mmap The mapping function of , Can avoid IO operation , Direct access to memory , This is called zero copy technology .

I'll start with a few pictures IO To zero copy .

This is the most common file server file transfer process , First, read the file from the physical device to the kernel space in the kernel state , This is a direct memory copy , Then the user process needs to read the data from the kernel into the user process space , Complete the process of reading , This is a time CPU Copy , thus , The process of reading is complete , The process needs to send data to the client , At this time, it is necessary to put the data into the kernel space socket It's about , And then it's sent out through the protocol layer .

The whole process takes two CPU Copy 、 Two direct memory copies , You also need to switch between kernel mode and user mode .( The first one is : The four time )

The second model introduces mmap, Mapping between kernel space and user space , You can make socket The copy function can be completed by operating kernel space directly , There's no need to switch between kernel mode and user mode ,write The system call causes the kernel to copy data from the original kernel buffer to the kernel buffer associated with the socket .

This is the way to use mmap Instead of read, Although it seems to reduce the number of copies , But there are risks . When mapping a file to memory , And then call write, In another process write The same file , There will be a system error .( The second kind : Three times )

The third model , be based on Linux The introduction of sendfile system call , Not only does it reduce file copies , It also reduces system switching ,sendfile Can directly complete the copy process of kernel space , Copy from kernel space to socket space , Thus skipping user space .( The third kind of : Three times )

The fourth model , In kernel version 2.4 in , Yes sendfile optimized , Data can be sent directly from kernel space to the protocol processor , It also eliminates data copies to the socket area , No change for user level applications .( A fourth : two )

Sum up , In the process of data transmission, the data will not result in redundant copies , There will be no redundant backup in the kernel and user mode space , This is called zero copy technology , be based on sendfile And mmap.

Back to RocketMQ

MQ yes IO Big users ,MMap、FileChannel、RandomAccessFile yes MQ The most commonly used method of file manipulation .

RocketMQ Support MMap And FileChannel, By default MMap, stay PageCache When busy , Will use FileChannel, It can also be avoided PageCache Competitive lock .

stay MappedFile Class , You can see FileChannel And MappedByteBuffer Two variables , stay Java Code can be passed through FileChannel Of map Method to map the file to virtual memory .

stay MappedFile Of init It can also be seen in the method mmap Initialization process .

In the actual writing process , Operation of the buffer May be mmap It could be TransientStorePool Apply for direct memory , Avoid pages being swapped out to the swap area .

TransientStorePool Whether to enable depends on TransientStorePoolEnable determine , When opened , It means to use the memory outside the heap to store data preferentially , adopt Commit Thread to memory mapping Buffer in .

TransientStorePool Is a simple pooling class , It includes the size of the pool , The size of each cell , Storage unit queue and storage configuration class . Specific initialization operations can be performed in init Methods see recycling allocateDirect apply JVM Memory space outside , Compared with allocate Applied for JVM Memory in , Faster off heap memory operation , The process of copying data from outside the heap to inside the heap is eliminated .

After applying to memory , Got the memory address of the application .

Pointer pointer = new Pointer(address);
LibC.INSTANCE.mlock(pointer, new NativeLong(fileSize));

When you get the address , Create a pointer to that , Call the local link library method , Lock the memory of the address , Prevent release .

Sum up , I believe you've made a list of pages 、 file system IO I have a certain understanding of the operation .

【RocketMQ Source code analysis 】 Deep into message storage (3) More articles about

  1. RocketMQ Source code analysis —— Message send and receive

    1. summary Producer  Send a message . Mainly send message source code synchronously , involves asynchronous /Oneway Send a message , Transaction message will be skipped . Broker  receive messages .( Store messages in <RocketMQ Source code analysis —— Mes ...

  2. RocketMQ Source code analysis from official examples :RocketMQ The basic idea of transaction message implementation

    Abstract : RocketMQ Source code analysis from official examples RocketMQ The basic idea of transaction message implementation . Before reading this article , If you are on RocketMQ Technical interest , Please join RocketMQ Technology exchange group RocketMQ4.3.0 edition ...

  3. RocketMQ Source code analysis RocketMQ Transaction message implementation principle part 1 ( Two stage submission )

    Before reading this article , If you are on RocketMQ Technical interest , Please join  RocketMQ Technology exchange group According to the above description , The entry to send the transaction message is : TransactionMQProducer#sendMessageInTra ...

  4. ROCKETMQ Source code analysis notes 1:tools

    rocketmq Source code analysis notes Hello everyone , Let's start with Amway , My male ,35 year , married . Currently working in petty bourgeoisie ( Beijing ), The position is development director . full name DaneBrown Okay . I promise that this article will never be a eunuch ! Please attach the above Amway information when reprinting . ...

  5. ROCKETMQ Source code analysis notes 2:client

    CLIENT Talked about before tools There are a lot of calls in it client Things that are . In order to understand from the source level rocket, Decided to nibble at client This bone . pom First look at pom, have a look CLIENT On whom . After watching it, it turned out to be dependence commo ...

  6. RocketMQ Source code analysis of the routing center (NameServer)

    You may not have seen RocketMQ The architecture of the figure , No problem , Let's study together ,RocketMQ The architecture is as follows : stay RocketMQ in , There are four roles : Producer: Message producer , Every MQ Middleware has it all . C ...

  7. RocketMQ Source code analysis RocketMQ Transaction message implementation principle part 2 ---- Transaction message status lookup

    The last section has been sorted out RocketMQ The process of sending transaction messages ( Based on a two-stage submission ), In this section, you'll continue to learn more about transaction status message lookback , We know , When you first submit to the message server, the subject of the message is replaced with RMQ_SYS_TRANS_HALF_TOP ...

  8. RocketMQ Source code analysis RocketMQ Transaction message implementation ( Transaction commit or rollback )

    Abstract :  The implementation principle of transaction message commit or rollback is based on commitlogOffset Message found , If it's a submit action , Restore the subject and queue of the original message , Deposit again commitlog The file then goes to the message consumption queue , For consumer consumption , And then ...

  9. RocketMQ Source code analysis RouteInfoManager( Four )

    In the last chapter, we analyzed NamesrvController When the constructor for , Will generate a RouteInfoManager object , This object stores the related messages of the whole message cluster , So here's a separate analysis . Just think about it namesrv The work of ...

  10. rocketmq Source code analysis 2-broker Message reception for

    broker Message reception , Suppose you receive a normal message ( That is, there is no transaction ), The analysis here only analyzes master On the action logic , Don't involve ha. 1. How to find the message receiving and processing entry Can pass broker Listening port of 10911 It's like following the vine ...

Random recommendation

  1. Tomcat Internal structure and working principle study

    Tomcat It was Servlet/JSP A debugging tool for , Later, it developed into a Servlet/JSP The container of . Tomcat As Servlet Containers , Responsible for handling customer requests , Send the request to Servlet And return the results to the guest ...

  2. One :Shell Basics

    1.shell summary   shell It's a command line interpreter , It provides a way for users to Linux The kernel sends requests to run the program's interface system level program , Users can use shell To start up , Hang up , Stop even writing some programs :  shell still ...

  3. Redis Common command summary

    Redis HGET Gets the value associated with the key hash stored in the field D:\web\JH2016\RedisV3.2\2MasterOpenAPI-15698\redis-cli.exe -h 127.0.0.1 - ...

  4. 【 course 】 Simulated landing on Baidu Java Code version

    [ background ] I've written a tutorial before , Analyze and simulate the logic of landing on Baidu : [ course ] Hand in hand teaching you how to use tools (IE9 Of F12) To analyze the simulated landing website ( Baidu Homepage ) The internal logical process of And then use different languages : Python Of : [ course ] model ...

  5. easyui layout Contracted bug

    easyui layout Provide collapse Method to collapse the specified panel,'region' The possible values of the parameter are :'north'.'south'.'east'.'west', But in IE6 Under the environment of , Call this ...

  6. poj2001 Shortest Prefixes (trie)

    Read in and build a letter tree , And each node will increase the coverage number of this node . And then do it again , Once you come across a coverage number of 1 It's the shortest prefix for this word . I don't know why the following program always has bug…… I don't know if it's about reading in ? type nod ...

  7. Deep learning algorithm practice 15--- Stack de-noising automatic coder (SdA) Principle and Implementation

    We discussed the de-noising automatic coder (dA), And discussed Theano The details of the framework implementation . In this section , We will discuss the de-noising automatic coder (dA) The main application of , That is to say, the stack automatic encoder is formed (SdA), We're going to MNIST Take handwritten letter recognition as an example , Stack from ...

  8. python itchat Crawling wechat friend information

    Link to the original text :https://mp.weixin.qq.com/s/4EXgR4GkriTnAzVxluJxmg 「itchat」 An open source wechat personal interface , We'll use it today itchat Crawling wechat friend information , No picture, no words ...

  9. FZU The 11th operation of soft worker - Software product case analysis

    Catalog Preface : The first part . research , evaluating : 1.1. For the first time : 1.2. enterprise bug: 1.3. Why do you think people in this product group didn't find these bug: 1.4. Suppose your team needs to develop this system , What should we pay attention to : 2. ...

  10. .Net Asynchronous programming z

    1. introduction Studying recently Abp frame , Find out Abp A lot of framework Api Both provide synchronous and asynchronous writing . Asynchronous programming says , You may say that asynchronous programming has good performance . But the good thing is , What kind of problems are introduced , And how to use it , I don't think I can answer it . since ...