Remember a Java Web A situation in which service requests are dead , The reason is that the request thread pool is full of deadlocks , Unable to receive new request . The following is the troubleshooting process of this problem ：
1、 An instance failed to respond to the website alarm , Find the corresponding instance , Manual
curl Request interface , Unable to respond .
jps Command to find the process number of the corresponding process .
jstat see GC Is it still in normal operation , Full command ：
jstat -gcutil 1624 1s, every other 1s Print once memory and gc situation , Here's the picture ：
As you can see from the figure above, there is basically no change in memory , Not now GC, On the one hand, it shows that the process basically does not carry out any tasks ; On the other hand, it can also reflect JVM It's not completely dead , Still alive （ There's a small change in memory ）. Take a look at the back gc situation , Here's the picture ：
Basically, we can see that we had a GC, Column YGC One more time than the last picture .
jmap -heap 1647 confirm JVM There's no pressure on memory , Here's the picture ：
5、 Take a look at the threads
jstack -l 1647 > jstack-20201224.log , And output the results to a file , Easy to check . Open the file and find that there are a lot of qtp The thread named for the beginning , Statistics , Here's the picture ：
After the check , Because the project uses jetty Containers , This is the name of the thread in the request thread pool , Obviously , Threads are full , And don't release . Take a look at the normal JVM What are examples like ？
Number of threads 28 individual . Confirm the state of the exception thread again ：
6、 Open the file of view thread just exported , Find out the specific reason why the thread cannot be released , Here's the picture ：
You can see that it's because the thread is blocked in
BasicFuture.get() method , And then you can see that your code is called in
ProcessTaskQueryResultService.java Of 198 That's ok , Here's the picture ：
You can see that the code uses Apache Of httpasyncclient Asynchronous calls to Libraries , return Future, If the request is abnormal , Here is the request for a relatively large response result , Interfaces are resource intensive , It's not responding , Will always be blocked in
Future.get() It's about , The best way is to set the timeout for getting the results , As shown in the example above .