6 month 19 The day begins , We will cache the blog site from memcached Instead of redis, Stable operation 3 God , There was a sudden problem during the visit peak this morning , stay 11:00-12:30 During this period, the normal access of the website was affected , This brings you trouble , Request for understanding .

During the breakdown , There are a lot of timeout errors waiting for response in the log

StackExchange.Redis.RedisTimeoutException: Timeout awaiting response

One of the typical logs :

StackExchange.Redis.RedisTimeoutException: Timeout awaiting response (outbound=0KiB, inbound=6697KiB, 2937ms elapsed, timeout is 2000ms), command=GET, next: GET blog_v2_BlogPosts-291733, inst: 0, qu: 0, qs: 31, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, serverEndpoint: 192.168.12.28:6379, mc: 1/1/0, mgr: 5 of 10 available, clientName: blog-web-c845b6686-zpl4s(SE.Redis-v2.5.61.22961), PerfCounterHelperkeyHashSlot: 11935, IOCP: (Busy=0,Free=1000,Min=20,Max=1000), WORKER: (Busy=19,Free=32748,Min=50,Max=32767), POOL: (Threads=73,QueuedItems=5,CompletedItems=2170212), v: 2.5.61.22961 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

It is confirmed that redis After the failure caused by , We immediately took a stupid approach , Will be used before memcached The cached part switches back to memcached, After switching, in addition to the personal blog blog list page , Other access has been restored .

The first level cache of personal blog posts list has been switched to memcached, But L2 cache is still in use redis, So I still feel redis Problem impact .

At this time , We restarted redis, But the problem remains after the restart . We thought it was a problem of high concurrency trigger , But we were wrong . We went back to thinking , Since it is not the reason for too much concurrency , That may be the reason why there are so many data ,flushall Empty redis After data in , It soon returned to normal .

Next, we will further analyze the cause of the problem , Optimize redis Deployment of , It's in k8s Non high availability deployed on the cluster redis colony (3 Nodes ).

【 Fault announcement 】 replace memcached Of redis There are more related articles about the website failure caused by problems

  1. 【 Fault announcement 】 database server CPU near 100% The breakdown caused

    I'm sorry , This morning, 10:48 ~ 10:33 period , The database service we use ( Alibaba cloud RDS example SQL Server 2016  The standard version ) again CPU near 100% problem , This brings you trouble , Please understand ...

  2. 【 Fault announcement 】 Upgrade alicloud RDS SQL Server Instance failure process

    Last night, , Alibaba cloud we use RDS SQL Server 2008 R2 The instance suddenly appears and continues CPU 100% problem , Later, we reset the instance to normal ( See fault announcement for details ). But after returning to normal, we found new problems , The machine ...

  3. 【 Fault announcement 】 The database server again CPU 100% Cause the whole station failure

    It's early this may day 5:50-6:30 period , Alibaba cloud we use RDS SQL Server The database instance appears again CPU 100% problem , Cause the whole station failure , This brings you trouble , Please understand . After we found the fault ...

  4. 【 Fault announcement 】Memcached Of “ Make trouble ”, I don't know for whom

    stay .NET 5.0 Back pot . Memcached To cause trouble . After the cache avalanche , We didn't find the real cause of the problem , We know that failures that don't find the root cause will always come back again , Either this week or next week , Maybe it's in pairs 11 Before and after . Just today ...

  5. 【 Fault announcement 】 database server CPU near 100% The breakdown caused ( From the .NET Core 3.0 One of the bug)

    So sorry , This trouble has brought you trouble , Please understand . This morning, 10:54 about , The database service we use ( Alibaba cloud RDS example SQL Server 2016  The standard version )CPU Suddenly soared to 90% above , Application day ...

  6. 【 Fault announcement 】docker swarm Failures caused by cluster problems

    Our production environment deployed 2 individual docker swarm colony , colony swarm-a Used to run blog sites , colony swarm-b For running other sites ,manager The node uses the Alibaba cloud monthly server ,worker section ...

  7. About 【【 Fault announcement 】 database server CPU near 100% The breakdown caused ( From the .NET Core 3.0 One of the bug)】IS NOT NULL test

    The test is shown in the figure ,Core_Users Of PhoneNumber May be empty , There is an index , Record number 1500000+ increase is not null, Query plan consumption has been increased by 0% Filter consumption of ,IO The consumption is exactly the same If it is IS N ...

  8. 【 Fault announcement 】redis Server downtime caused blog site failure

    So sorry , This afternoon, 17:10~17:40 about , Because the blog system uses redis Server down , Causes the blog site to be unable to visit normally , This brings you a lot of trouble , Please understand . We will improve on this fault redis The server ...

  9. Talk about Memcached And Redis

    1. Memcached brief introduction Memcached In order to LiveJurnal its Danga Interactive The company's Bard Fitzpatric High performance distributed memory cache server . It's essentially a ...

  10. memcached And redis The differences and specific application scenarios of

    1. Memcached brief introduction Memcached In order to LiveJurnal its Danga Interactive The company's Bard Fitzpatric High performance distributed memory cache server . It's essentially a ...

Random recommendation

  1. Jquery Medium $().each() Method

    Let me give you an example , Output each li The text of the element : <html> <head> <script type="text/javascript" src=" ...

  2. Table View It's wrong

    Learning table view (Table View) Application time , I wrote a demo, Finally, the table came out , But I'm wrong , An error is as follows : This is me ViewController.m Part of the code : #import "ViewC ...

  3. CentOS 6.4 Installation of structures, Scrapy 0.22 Environmental Science

    One . install Python2.7.6 to update CentOS lib The library files yum -y update Install development kit yum groupinstall -y development Install expansion pack yum inst ...

  4. In-depth analysis of Java Of ClassLoader Mechanism ( Source level )

    Write it at the front :Java All classes in , Must be loaded into jvm In order to run , This loading is done by jvm Class loader in , The essence of class loader is to read class files from hard disk to memory ,JVM When loading classes , It's all through ClassLoa ...

  5. 337. House Robber III

    Second brush .. do not know why house robbery Series I can't find notes , But I have done it several times . Not very hard , With post-order do bottom-up Arithmetic . For one Node Come on , Yes 2 In this case , One is to choose ( ...

  6. turn : Ten good programming ideas .( Source : cool shell – CoolShell.cn)

    stay Stack Overflow There is such a post on <What’s your most controversial programming opinion?>, Translated into Chinese “ What you think is the most controversial ...

  7. composer stay ubuntu Lower installation

    adopt curl download composer Installation package curl -sS https://getcomposer.org/installer | php Move the installation package to the global installation folder mv composer.pha ...

  8. Smart Framework

    Smart Framework: Lightweight Java Web frame Published in 2 Years ago (2013-09-01 08:39)   read (48569) |  Comment on (188) 544 People collect this article ,  I want to collect Fabulous 83 o ...

  9. [js Master's Road ] html5 canvas Series of tutorials - arc Drawing curves ( curve , Arc , circular )

    draw a curve , The knowledge of path is often used , If you have questions about the path , Please refer to my article [js Master's Road ] html5 canvas Series of tutorials - Start path beginPath And close the path closePath Detailed explanation . arc: draw ...

  10. JavaScript Design patterns Item 2 -- Interface implementation

    1. Interface Overview 1. What is an interface ? An interface is a means to specify which methods an object should have . Although it can indicate the semantics of these methods , But it does not specify how these methods should be implemented . 2. Interface benefits Promote code reuse . Interfaces can tell ...