Flink + Kafka: construction practice of Netease cloud music real time data warehouse

InfoQ 2021-01-21 07:36:35
flink kafka construction practice netease


{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" at present , Netease cloud music is based on Kafka+Flink The real-time task of has reached 500+,Kafka The total number of nodes in the cluster reaches 200+, single Kafka Peak value QPS 400W+. This paper is shared by Yue Meng, R & D Engineer of Netease cloud music real-time computing platform , We will introduce from the following four parts Flink + Kafka In the application of Netease cloud music :"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":" background "}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"Flink + Kafka Platform design "}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"Kafka Application in real-time data warehouse "}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":" problem & improvement "}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":" Background introduction "}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1. General framework of streaming platform "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" At present, the common architecture of streaming platform generally includes message queue 、 Computing engine and storage , The general architecture is shown in the figure below . Client or web Of log Logs are collected to the message queue ; The computing engine calculates the data of the message queue in real time ; The real-time calculation results are as follows Append perhaps Update In a real-time storage system ."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" at present , Our common message queue is Kafka, The computing engine we started with was Spark Streaming, With Flink The advantage of stream computing engine is more and more obvious , We finally decided Flink As our unified real-time computing engine ."}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/41\/41d6c12393e8d49cbb304329898abbcb.png","alt":" picture ","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2. Why Kafka?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kafka It's an early message queue , But it's a very stable message queue , There are many user groups , One of them is Netease . We consider the Kafka The main reasons for our message middleware are as follows :"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" High throughput , Low latency : Hundreds of thousands per second QPS And millisecond delay ;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" High concurrency : Support thousands of clients to read and write at the same time ;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Fault tolerance , Can be high : Support data backup , Allow nodes to be lost ;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Extensibility : Support for hot expansion , It will not affect the current online business ."}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3. Why choose Flink?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Apache Flink It is an open source big data stream computing engine which is more and more popular in recent years , It supports both batch and stream processing , consider Flink The main factor that makes us a stream computing engine is :"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" High throughput , Low latency , High performance ;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Highly flexible streaming window ;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" State calculated Exactly-once semantics ;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Lightweight fault tolerance mechanism ;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Support EventTime And disorderly Events ;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Stream batch unified engine ."}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"4. Kafka + Flink Stream computing system "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" be based on Kafka and Flink In message middleware and streaming computing , So there was a circle around Kafka And Flink Based on the stream computing platform system , As shown in the figure below : be based on APP、web The way to collect logs will be generated in real time Kafka, And then leave it to Flink To do the usual ETL, Global aggregation and Window Real time computing such as aggregation ."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/49\/494603ae231cc70f5efb7c34586b7959.png","alt":" picture ","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"5. Netease cloud music use Kafka The status quo of "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" At present, we have 10+ individual Kafka colony , The main tasks of each cluster are different , Some as business clusters , Some as mirror clusters , Some as computing clusters and so on . At present Kafka The total number of nodes in the cluster reaches 200+, single Kafka Peak value QPS 400W+. at present , Netease cloud music is based on Kafka+Flink The real-time task of has reached 500+."}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Flink+Kafka Platform design "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Based on the above situation , We want to be right Kafka+Flink Do a platform development , Reduce the development cost and operation and maintenance cost of users . In fact, 2018 We started based on Flink Make a real-time computing platform ,Kafka It plays an important role in it , This year, , In order to make users more convenient 、 It's easier to use Flink and Kafka, We refactored ."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" be based on Flink 1.0 We made a version of Magina Version refactoring , stay API Hierarchy we provide Magina SQL and Magina SDK through DataStream and SQL operation ; Then by customizing Magina SQL Parser They will SQL convert to Logical Plan, Will be LogicalPlan Convert to physical execution code , In the process, we will go through catalog Connect to the metadata management center to get some metadata information . We are Kafka In the course of using , Will Kafka Metadata information is registered to the metadata center , Access to real-time data is in the form of stream tables . stay Magina We are right Kafka Three parts of the work have been done :"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" colony catalog turn ;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Topic Flow representation ;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Message Schema turn ."}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/be\/beb829cdff7893ef86366bd00b89b19c.png","alt":" picture ","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Users can register different table information or catalog Information, etc , It can also be in DB Create and maintain Kafka Table of , Users only need to use the corresponding table according to their personal needs . Below is the Kafka The main reference logic of flow table ."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/47\/4779d9ccc8a07dc38f52a533e345136a.png","alt":" picture ","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Kafka Application in real-time data warehouse "}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1. Develop in problem solving "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kafka In the process of using real-time data warehouse , We have different problems , Different solutions have been tried in the middle ."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" In the early days of the platform , At first, there were only two clusters used for real-time computing , And there's a collection cluster , single Topic Very large amount of data ; Different real-time tasks will consume the same amount of data Topic,Kafka colony IO The pressure is abnormally high ;"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" therefore , In the process of using, we found Kafka The pressure is very high , There are often delays 、I\/O soaring ."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" We think of big ones Topic Real time distribution to solve the above problem , be based on Flink 1.5 The program of data distribution is designed as shown in the figure below , This is the prototype of real-time data warehouse . Based on this, it will be big Topic Distribute it into small ones Topic Methods , It greatly reduces the pressure on the cluster , Improved performance , in addition , Initially, static distribution rules were used , When you need to add rules later, restart the task , It has a great impact on the business , Then we considered using dynamic rules to accomplish the task of data distribution ."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/71\/71cfe13f35a0488ac931bce530baef9b.png","alt":" picture ","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" After solving the initial problems of the platform , In the process of platform upgrading Kafka There are new problems :"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Although the cluster has been expanded , But the amount of work is also increasing ,Kafka Cluster pressure is still rising ;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Cluster pressure rises sometimes I\/O Related issues , Consumption tasks are easy to interact with each other ;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Users consume different Topic There is no intermediate data landing in the process , It is easy to cause repeated consumption ;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Task migration Kafka difficult ."}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" For the above problems , We did the following Kafka Cluster isolation and data layering . The process is simple , Divide the cluster into DS colony 、 Log collection cluster 、 Distributed cluster , Data is distributed to Flink To deal with , And then through data cleaning, you can enter DW colony , At the same time DW During the write process, it will be synchronized to the mirror cluster , In this process, we will also use Flink Statistics and splicing of real-time calculation , And will generate ADS Data written online ADS Clustering and Statistics ADS colony . Through the above process , It ensures that tasks with high requirements for real-time calculation will not be affected by statistical reports ."}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/4c\/4c4d4814841c7d02e88309c72119123c.jpeg","alt":" picture ","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Through the above process , It ensures that tasks with high requirements for real-time calculation will not be affected by statistical reports . But after we distribute different clusters, we will inevitably face new problems :"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" How to perceive Kafka State of the cluster ?"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" How to quickly analyze Job Consumption is abnormal ?"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" In view of the above two problems , We made a Kafka The monitoring system , Its monitoring is divided into the following two dimensions , In this way, when an exception occurs, you can make a specific judgment on the details of the problem :"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Monitoring of cluster profile : You can see the corresponding Topic Number and number of running tasks , And each Topic Consumption task data amount 、 Data inflow 、 Total inflow and average size of each data ;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Indicator monitoring : You can see Flink Tasks and corresponding Topic、GroupID、 The cluster 、 Starting time 、 Input bandwidth 、InTPS、OutTPS、 Consumer delays and Lag situation ."}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2. Flink + Kafka stay Lambda The application under the framework "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Stream batch unification is a very popular concept at present , Many companies are also considering this application , At present, the commonly used architecture is either Lambda framework , Or Kappa framework . For streaming batch unification, we need to consider storage unification and computing engine unification , Because we don't have unified storage for infrastructure , So we have to choose Lamda framework ."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" The figure below is based on Flink and Kafka Of Lambda The specific practice of architecture in cloud music , The upper layer is real-time computing , The lower layer is offline computing , Horizontal is divided by computing engine , Vertical is based on real-time data warehouse to distinguish ."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/5f\/5fa088d4908a41de9ef26343a29dd846.jpeg","alt":" picture ","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":" problem & improvement "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" In the specific application process , We also have a lot of problems , The two main problems are :"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" many Sink Next Kafka Source The problem of repeated consumption ;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" The same switch traffic surge consumption computing delay problem ."}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1. many Sink Next Kafka Source The problem of repeated consumption "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Magina There are many supports on the platform Sink, In other words, any result in the middle can be inserted into different storage in the process of operation . There will be a problem in the process , Like the same intermediate result , We insert different parts into different stores , Then there will be more than one DAG, Although it's all temporary results , But it can also cause Kafka Source Repeated consumption of , Huge waste of performance and resources ."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" So we thought , Whether it is possible to avoid multiple consumption of temporary intermediate results . stay 1.9 Before the release , We did StreamGraph Reconstruction of , Will three DataSource Of DAG We have a merger ; stay 1.9 edition ,Magina I also provide a query and Source Optimization of the merger ; But we found that if it was in the same data update There are multiple... For the same table in Source References to , It will merge itself , But if it's not in the same data update in , It's not going to merge immediately , So in 1.9 After the version, we have to modifyOperations Made a buffer To solve this problem ."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/6c\/6c57638fa56a57310a2d76539161f7a9.png","alt":" picture ","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2. The same switch traffic surge consumption computing delay problem "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" This is a recent problem , It may not be just the same switch , The situation in the same computer room may also be . We deployed a lot of machines under the same switch , Some of the machines are deployed Kafka colony , There are also some deployed Hadoop colony . stay Hadoop We might be doing this up there Spark、Hive Offline computing and Flink In real time ,Flink They also consume Kafka Do real-time calculations . In the process of running, we find that there will be overall delay in a certain task , After checking, no other abnormalities were found , In addition to the switch browsing at a certain point in time , Further investigation found that there was a surge in offline computing browsing , Because of the bandwidth limitation of the same switch , It's affecting Flink In real time ."}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/da\/da9b2ff93ac071221448eb8bf7c56606.png","alt":" picture ","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" To solve this problem , We consider to avoid the interaction between offline cluster and real-time cluster , To optimize switch deployment or machine deployment , For example, an offline cluster uses a single switch ,Kafka and Flink The cluster also uses a separate switch , From the hardware level to ensure that there is no interaction between the two ."}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Q & A"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q1:Kafka Is the data in the real-time data warehouse reliable ?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"A1: The answer to this question depends more on the definition of data accuracy , Different criteria may lead to different answers . First of all, you should define the conditions under which the data is reliable , There's another fault tolerance mechanism to deal with ."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q2: When we are learning, how to learn the problems encountered in these enterprises ? How to accumulate these problems ?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"A2: I think the process of learning is problem driven , When you have a problem, think about it and solve it , In the process of solving, to accumulate experience and their own shortcomings ."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q3: You're dealing with Kafka In the process of , How to deal with abnormal data , Is there a detection mechanism ?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"A3: In the process of running, we have a distributed service , In the process of distribution, we will detect which data is abnormal according to certain rules , What is normal , An exception is then distributed to a single exception Topic To make inquiries and so on , Later users in the use of the process can be based on the relevant indicators and keywords to abnormal Topic To see the data ."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Reprinted from :DataFunTalk(ID:datafuntalk)"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Link to the original text :"},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/jCQ6lUejGPzsrHDaNTy6sw","title":"xxx","type":null},"content":[{"type":"text","text":"Flink+Kafka: Netease cloud music real-time data warehouse construction practice "}]}]}]}
版权声明
本文为[InfoQ]所创,转载请带上原文链接,感谢
https://javamana.com/2021/01/20210121073549829W.html

  1. 【计算机网络 12(1),尚学堂马士兵Java视频教程
  2. 【程序猿历程,史上最全的Java面试题集锦在这里
  3. 【程序猿历程(1),Javaweb视频教程百度云
  4. Notes on MySQL 45 lectures (1-7)
  5. [computer network 12 (1), Shang Xuetang Ma soldier java video tutorial
  6. The most complete collection of Java interview questions in history is here
  7. [process of program ape (1), JavaWeb video tutorial, baidu cloud
  8. Notes on MySQL 45 lectures (1-7)
  9. 精进 Spring Boot 03:Spring Boot 的配置文件和配置管理,以及用三种方式读取配置文件
  10. Refined spring boot 03: spring boot configuration files and configuration management, and reading configuration files in three ways
  11. 精进 Spring Boot 03:Spring Boot 的配置文件和配置管理,以及用三种方式读取配置文件
  12. Refined spring boot 03: spring boot configuration files and configuration management, and reading configuration files in three ways
  13. 【递归,Java传智播客笔记
  14. [recursion, Java intelligence podcast notes
  15. [adhere to painting for 386 days] the beginning of spring of 24 solar terms
  16. K8S系列第八篇(Service、EndPoints以及高可用kubeadm部署)
  17. K8s Series Part 8 (service, endpoints and high availability kubeadm deployment)
  18. 【重识 HTML (3),350道Java面试真题分享
  19. 【重识 HTML (2),Java并发编程必会的多线程你竟然还不会
  20. 【重识 HTML (1),二本Java小菜鸟4面字节跳动被秒成渣渣
  21. [re recognize HTML (3) and share 350 real Java interview questions
  22. [re recognize HTML (2). Multithreading is a must for Java Concurrent Programming. How dare you not
  23. [re recognize HTML (1), two Java rookies' 4-sided bytes beat and become slag in seconds
  24. 造轮子系列之RPC 1:如何从零开始开发RPC框架
  25. RPC 1: how to develop RPC framework from scratch
  26. 造轮子系列之RPC 1:如何从零开始开发RPC框架
  27. RPC 1: how to develop RPC framework from scratch
  28. 一次性捋清楚吧,对乱糟糟的,Spring事务扩展机制
  29. 一文彻底弄懂如何选择抽象类还是接口,连续四年百度Java岗必问面试题
  30. Redis常用命令
  31. 一双拖鞋引发的血案,狂神说Java系列笔记
  32. 一、mysql基础安装
  33. 一位程序员的独白:尽管我一生坎坷,Java框架面试基础
  34. Clear it all at once. For the messy, spring transaction extension mechanism
  35. A thorough understanding of how to choose abstract classes or interfaces, baidu Java post must ask interview questions for four consecutive years
  36. Redis common commands
  37. A pair of slippers triggered the murder, crazy God said java series notes
  38. 1、 MySQL basic installation
  39. Monologue of a programmer: despite my ups and downs in my life, Java framework is the foundation of interview
  40. 【大厂面试】三面三问Spring循环依赖,请一定要把这篇看完(建议收藏)
  41. 一线互联网企业中,springboot入门项目
  42. 一篇文带你入门SSM框架Spring开发,帮你快速拿Offer
  43. 【面试资料】Java全集、微服务、大数据、数据结构与算法、机器学习知识最全总结,283页pdf
  44. 【leetcode刷题】24.数组中重复的数字——Java版
  45. 【leetcode刷题】23.对称二叉树——Java版
  46. 【leetcode刷题】22.二叉树的中序遍历——Java版
  47. 【leetcode刷题】21.三数之和——Java版
  48. 【leetcode刷题】20.最长回文子串——Java版
  49. 【leetcode刷题】19.回文链表——Java版
  50. 【leetcode刷题】18.反转链表——Java版
  51. 【leetcode刷题】17.相交链表——Java&python版
  52. 【leetcode刷题】16.环形链表——Java版
  53. 【leetcode刷题】15.汉明距离——Java版
  54. 【leetcode刷题】14.找到所有数组中消失的数字——Java版
  55. 【leetcode刷题】13.比特位计数——Java版
  56. oracle控制用户权限命令
  57. 三年Java开发,继阿里,鲁班二期Java架构师
  58. Oracle必须要启动的服务
  59. 万字长文!深入剖析HashMap,Java基础笔试题大全带答案
  60. 一问Kafka就心慌?我却凭着这份,图灵学院vip课程百度云