In recent business , I received a need to process about 100000 pieces of data . These data are given in the form of strings , And the steps to deal with them are asynchronous and time-consuming （ Average processing of a piece of data requires 25s Time for ）. If it is implemented in serial mode , It takes quite a long time ：
Total time = Data volume × Single data processing time
T = N * t (N = 100,000; t = 25s)
Total time = 2,500,000 second ≈ 695 Hours ≈ 29 God
obviously , We can't simply process the data one by one . Is there any way to reduce the processing time ？ After investigation, we found that , It's a good idea to use asynchronous task queues .
The following will share with you Redis + NodeJS The idea and method of realizing an asynchronous task queue system which can deal with massive data , Hope to communicate with you . The authors ：jrain, Tencent application R & D Engineer .
We can “ Processing single data ” It is understood as an asynchronous task , So the processing of these 100000 pieces of data , It turns out that there are 100000 asynchronous tasks waiting to be done . We can put these 100000 pieces of data into a queue , Let the task processor get and complete from the queue spontaneously .
Task processors can have multiple , They also take the task out of the queue and process it . When the task queue is empty , All tasks have been claimed ; When all task processors complete the task , All tasks have been processed .
Its basic principle is shown in the figure below ：
Link to the original text :【https://www.infoq.cn/article/wgE5N8zN4OYveDiDUJ71】. Without the permission of the author , Prohibited reproduced .