Why does a seemingly correct code cause the Dubbo thread pool to be full

Java front line 2021-05-04 14:54:58
seemingly correct code cause dubbo


Welcome everyone to follow the official account 「JAVA Front 」 See more wonderful sharing articles , Including source code analysis 、 The practical application 、 Structure thinking 、 Workplace sharing 、 Product thinking and so on , At the same time, welcome to wechat 「java_front」 Exchange and study together

1 A formula

In a previous article, we used a formula to analyze why DUBBO Why is the thread pool full , At the beginning of this article, we might as well review this formula : A company has 7200 Employees , The clock in time every day is in the morning 8 Point to 8 spot 30 branch , It takes time for the system to punch in every time 5 second . Excuse me, RT、QPS、 What are the concurrency quantities ?

RT Response time , The question has told us the answer :

RT = 5

QPS Represents the number of queries per second , Suppose that the check-in behavior is evenly distributed :

QPS = 7200 / (30 * 60) = 4

Concurrency represents the number of requests that the system processes at the same time :

Concurrency = QPS x RT = 4 x 5 = 20

According to the above examples, the following formula is derived :

Concurrency = QPS x RT

If the system allocates a processing thread for each request , Then the concurrency can be approximately equal to the number of threads . Based on the above formula, it is not difficult to see that concurrency is affected by QPS and RT influence , The increase of either of these two indicators will lead to the increase of concurrency .

But it's just an ideal situation , Because the amount of concurrency is limited by the ability of the system, it can not continue to rise , for example DUBBO The thread pool limits the number of threads , If the maximum number of threads is exceeded, the rejection policy will be executed , The reject policy will indicate that the thread pool is full , This is it. DUBBO The root of the thread pool full problem .


2 A piece of code

Now let's analyze why the right piece of code leads to DUBBO Thread pool full :MyCache It's a caching tool , Read data from many files to memory during initialization , Get it directly from memory .

public class MyCache {
private static Map<String, String> cacheMap = new HashMap<String, String>();
static {
initCacheFromFile();
}
private static void initCacheFromFile() {
try {
long start = System.currentTimeMillis();
System.out.println("init start");
// It takes time to simulate reading files 
Thread.sleep(10000L);
cacheMap.put("K1", "V1");
System.out.println("init end cost " + (System.currentTimeMillis() - start));
} catch (Exception ex) {
}
}
public static String getValueFromCache(String key) {
return cacheMap.get(key);
}
}
 Copy code 

2.1 producer

(1) Service statement

public interface HelloService {
public String getValueFromCache(String key);
}
@Service("helloService")
public class HelloServiceImpl implements HelloService {
@Override
public String getValueFromCache(String key) {
return MyCache.getValueFromCache(key);
}
}
 Copy code 

(2) The configuration file

<beans>
<dubbo:application name="java-front-provider" />
<dubbo:registry address="zookeeper://127.0.0.1:2181" />
<dubbo:protocol name="dubbo" port="9999" />
<dubbo:service interface="com.java.front.dubbo.demo.provider.HelloService" ref="helloService" />
</beans>
 Copy code 

(3) Service release

public class Provider {
public static void main(String[] args) throws Exception {
String path = "classpath*:META-INF/spring/dubbo-provider.xml";
ClassPathXmlApplicationContext context = new ClassPathXmlApplicationContext(path);
System.out.println(context);
context.start();
System.in.read();
}
}
 Copy code 

2.2 consumer

(1) The configuration file

<beans>
<dubbo:application name="java-front-consumer" />
<dubbo:registry address="zookeeper://127.0.0.1:2181" />
<dubbo:reference id="helloService" interface="com.java.front.dubbo.demo.provider.HelloService" timeout="10000" />
</beans>
 Copy code 

(2) Service consumption

public class Consumer {
public static void main(String[] args) throws Exception {
ClassPathXmlApplicationContext context = new ClassPathXmlApplicationContext(new String[] { "classpath*:META-INF/spring/dubbo-consumer.xml" });
context.start();
System.out.println(context);
// Simulate a large number of requests 
for (int i = 0; i < 1000; i++) {
new Thread(new Runnable() {
@Override
public void run() {
HelloService helloService = (HelloService) context.getBean("helloService");
String result = helloService.getValueFromCache("K1");
System.out.println(result);
}
}).start();
}
}
}
 Copy code 

2.3 Running results

We observe the log content and find that the thread pool is full :

NettyServerWorker-5-1 WARN support.AbortPolicyWithReport:
[DUBBO] Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-1.1.1.1:9999, Pool Size: 200 (active: 200, core: 200, max: 200, largest: 200), Task: 201 (completed: 1),
Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://1.1.1.1:9999!, dubbo version: 2.7.0-SNAPSHOT, current host: 1.1.1.1
 Copy code 

3 A tool

According to the formula and code fragment introduced in the first chapter , It's not hard to speculate that it's probably because RT Rising causes the thread pool to be full , But if we need to analyze the detailed reasons, we can't stop here , It also needs to be analyzed in combination with thread snapshot . Another function of thread snapshot is that if the online server suddenly reports a thread pool full error , We can't immediately locate the problem code , This needs to be analyzed through thread snapshot .


3.1 jstack

The first way to get a thread snapshot is jstack command , This order can be based on JAVA Process number print thread snapshot , There are three steps to use it , The first is to confirm JAVA Process number , Second, print thread snapshot , Third, analyze thread snapshot .

(1) determine JAVA Process number

jps -l
 Copy code 

(2) Print thread snapshot

Suppose the first step is to get JAVA The process number is 5678

jstack 5678 > dump.log
 Copy code 

(3) Analyze thread snapshots

Now we're going to analyze the snapshot file dump.log, Of course, we can directly open the snapshot file for analysis , You can also use tools to analyze , I usually have a IBM Development of free thread snapshot analysis tool :

IBM Thread and Monitor Dump Analyzer for Java
 Copy code 

(a) How to download

https://public.dhe.ibm.com/software/websphere/appserv/support/tools/jca/jca469.jar
 Copy code 

(b) How to run

java -jar jca469.jar
 Copy code 

(c) How to analyze

We use this tool to open dump.log file , Select the toolbar pie icon to analyze the thread state :

1.jpg

We found a lot of thread blocking in HelloServiceImpl The first 48 That's ok , Find the corresponding code location :

public class HelloServiceImpl implements HelloService {
// Omit code ......
@Override
public String getValueFromCache(String key) {
return MyCache.getValueFromCache(key); // The first 48 That's ok 
}
}
 Copy code 

Let's assume that if MyCache.getValueFromCache There are time-consuming operations in this method , Then the thread should be blocked on one line of this method , But in the end it got stuck in HelloServiceImpl This class , This means that the blockage happened in MyCache On this class initialization . Let's review MyCache Code , It turns out that the initialization method really takes a lot of time , Prove the correctness of thread snapshot analysis .

public class MyCache {
private static Map<String, String> cacheMap = new HashMap<String, String>();
static {
initCacheFromFile();
}
private static void initCacheFromFile() {
try {
long start = System.currentTimeMillis();
System.out.println("init start");
// It takes time to simulate reading files 
Thread.sleep(10000L);
cacheMap.put("K1", "V1");
System.out.println("init end cost " + (System.currentTimeMillis() - start));
} catch (Exception ex) {
}
}
}
 Copy code 

3.2 DUBBO Thread snapshot

The second way to get thread snapshot is in DUBBO Thread pool rejection policy source code , We analyze the source code and know that whenever the thread pool is full DUBBO Will print thread snapshot .

public class AbortPolicyWithReport extends ThreadPoolExecutor.AbortPolicy {
protected static final Logger logger = LoggerFactory.getLogger(AbortPolicyWithReport.class);
private final String threadName;
private final URL url;
private static volatile long lastPrintTime = 0;
private static Semaphore guard = new Semaphore(1);
public AbortPolicyWithReport(String threadName, URL url) {
this.threadName = threadName;
this.url = url;
}
@Override
public void rejectedExecution(Runnable r, ThreadPoolExecutor e) {
String msg = String.format("Thread pool is EXHAUSTED!" +
" Thread Name: %s, Pool Size: %d (active: %d, core: %d, max: %d, largest: %d), Task: %d (completed: %d)," +
" Executor status:(isShutdown:%s, isTerminated:%s, isTerminating:%s), in %s://%s:%d!",
threadName, e.getPoolSize(), e.getActiveCount(), e.getCorePoolSize(), e.getMaximumPoolSize(), e.getLargestPoolSize(),
e.getTaskCount(), e.getCompletedTaskCount(), e.isShutdown(), e.isTerminated(), e.isTerminating(),
url.getProtocol(), url.getIp(), url.getPort());
logger.warn(msg);
// Print thread snapshot 
dumpJStack();
throw new RejectedExecutionException(msg);
}
private void dumpJStack() {
long now = System.currentTimeMillis();
// Every time 10 Minute output thread snapshot 
if (now - lastPrintTime < 10 * 60 * 1000) {
return;
}
if (!guard.tryAcquire()) {
return;
}
ExecutorService pool = Executors.newSingleThreadExecutor();
pool.execute(() -> {
String dumpPath = url.getParameter(Constants.DUMP_DIRECTORY, System.getProperty("user.home"));
System.out.println("AbortPolicyWithReport dumpJStack directory=" + dumpPath);
SimpleDateFormat sdf;
String os = System.getProperty("os.name").toLowerCase();
// linux file location /home/xxx/Dubbo_JStack.log.2021-01-01_20:50:15
// windows file location /user/xxx/Dubbo_JStack.log.2020-01-01_20-50-15
if (os.contains("win")) {
sdf = new SimpleDateFormat("yyyy-MM-dd_HH-mm-ss");
} else {
sdf = new SimpleDateFormat("yyyy-MM-dd_HH:mm:ss");
}
String dateStr = sdf.format(new Date());
try (FileOutputStream jStackStream = new FileOutputStream(new File(dumpPath, "Dubbo_JStack.log" + "." + dateStr))) {
JVMUtil.jstack(jStackStream);
} catch (Throwable t) {
logger.error("dump jStack error", t);
} finally {
guard.release();
}
lastPrintTime = System.currentTimeMillis();
});
pool.shutdown();
}
}
 Copy code 

From the thread snapshot file below, we can see ,200 individual DUBBO Threads are also executed in HelloServiceImpl The first 48 That's ok , So we can also locate the problem code . however DUBBO Printing thread snapshots is not jstack A standard format , So it can't be used IBM Tools for analysis .

DubboServerHandler-1.1.1.1:9999-thread-200 Id=230 RUNNABLE
at com.java.front.dubbo.demo.provider.HelloServiceImpl.getValueFromCache(HelloServiceImpl.java:48)
at org.apache.dubbo.common.bytecode.Wrapper1.invokeMethod(Wrapper1.java)
at org.apache.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:56)
at org.apache.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:85)
at org.apache.dubbo.config.invoker.DelegateProviderMetaDataInvoker.invoke(DelegateProviderMetaDataInvoker.java:56)
 Copy code 

4 Some thoughts

MyCache The tool modification method is not complicated , You can give it to Spring management , adopt PostConstruct Annotation , And declare the get cache method as an object method .

In fact, we found that MyCache There are no errors in class Syntax , It is not impossible to perform initialization operations in static code blocks . But because of the large flow of callers , It happened. MyCache It is called a lot without initialization , Causing a large number of threads to block on initialization methods , Eventually, the thread pool is full . So as the flow increases , Quantitative change causes qualitative change , The problems that are not the original problems have also been exposed , It needs our attention , I hope this article can help you .

Welcome everyone to follow the official account 「JAVA Front 」 See more wonderful sharing articles , Including source code analysis 、 The practical application 、 Structure thinking 、 Workplace sharing 、 Product thinking and so on , At the same time, welcome to wechat 「java_front」 Exchange and study together

版权声明
本文为[Java front line]所创,转载请带上原文链接,感谢
https://javamana.com/2021/05/20210504144301106D.html

  1. Help, Java how to get all the current processes of the system
  2. Has anyone ever used JMeter or written tests in httpUnit????
  3. Living in a mountain village in late spring
  4. Partridge day, spring of HKUST
  5. JavaScript异步编程4——Promise错误处理
  6. 海康摄像SDK开发笔记(一):海康威视网络摄像头SDK介绍与模块功能
  7. JavaScript asynchronous programming 4 -- promise error handling
  8. Haikang video SDK development notes (1): introduction and module functions of Hikvision webcam SDK
  9. JOP:用于FPGA的嵌入式实时系统中的Java优化处理器内核
  10. Spring Boot源码:使用MongoDB MongoTemplate公开REST在几分钟内实现CRUD功能
  11. Spring Boot应用程序事件教程 - reflectoring
  12. 带有Resilience4j断路器的Spring云网关 - rome
  13. 经验分享:Apache Kafka的缺点与陷阱 - Emil Koutanov
  14. 通过Spring Boot Webflux实现Reactor Kafka
  15. 从Java 8升级到Java 11应该注意的问题
  16. Jop: Java optimized processor core for FPGA embedded real time system
  17. Spring boot source code: use mongodb mongotemplate to open rest to realize crud function in a few minutes
  18. Spring boot application event tutorial - reflecting
  19. Spring cloud gateway with resilience4j circuit breaker - ROM
  20. Experience sharing: shortcomings and pitfalls of Apache Kafka - Emil koutanov
  21. Realization of reactor Kafka through spring boot Webflux
  22. RPC框架设计----Socket与I/0模型
  23. Problems in upgrading from Java 8 to Java 11
  24. RPC framework design -- socket and I / 0 model
  25. RPC框架设计----I/0模型
  26. RPC framework design: I / 0 model
  27. RPC框架设计----NIO编程缓冲区Buffer
  28. RPC框架设计----NIO编程缓冲区Buffer
  29. RPC framework design -- NiO programming buffer
  30. RPC framework design -- NiO programming buffer
  31. Java多线程基础
  32. Java multithreading Foundation
  33. 码农飞升记-00-Java发展历程
  34. Development history of coder-00-java
  35. 码农飞升记-00-Java发展历程
  36. Development history of coder-00-java
  37. Spring and Autumn Moon
  38. Node.js与Spring Boot比较? - Ryan Gleason
  39. Spring WebFlux的明显陷阱 - ŁukaszKyć
  40. Spring创始人Rod大叔对YAML的真实想法
  41. Compare node.js with spring boot- Ryan Gleason
  42. Obvious pitfalls of spring Webflux- Ł ukaszKy ć
  43. Spring founder uncle rod's real thoughts on yaml
  44. 码农飞升记-02-OracleJDK是什么?OracleJDK的版本怎么选择?
  45. What is manong feisheng-02-oracle JDK? How to choose the version of Oracle JDK?
  46. Spring tide surging Xinanjiang
  47. Linux内核软中断
  48. Linux kernel soft interrupt
  49. Linux内核软中断
  50. Linux kernel soft interrupt
  51. Java multithreading Foundation
  52. The construction of Maven private library nexus
  53. I / O stream in Java
  54. JDK 16:Java 16的新功能 - InfoWorld
  55. 在Java中本地进行线程间数据传输的三种方式和源码展示
  56. jdon导致cpu 99%最后tomcat死掉---banq给予回复
  57. 用领域事件模拟AOP注入
  58. JDK 16: new function of Java 16 - InfoWorld
  59. Cartoon: from JVM lock to redis distributed lock
  60. Spring 3.1 终于加入了Cache支持