Java shell command execution and log collection pitfall Guide

When you come back 2020-11-08 14:33:58
java shell command execution log


Sometimes we need to call system commands to execute something , Maybe it's for convenience , There may be no way to call . Something that involves executing system commands , You can't cross platform , This sum java The original intention of language is opposite .

I don't say much nonsense ,java How to execute shell command ? Nature calls java The interface provided by language class library API 了 .

 

1. java perform shell Of api

perform shell command , It can be said that system level calls , Programming languages will naturally provide the corresponding api Operation . stay java in , There are two api For calling :Runtime.exec(), Process API. Simple use as follows :

1.1. Runtime.exec() Realization

The call implementation is as follows :

import java.io.InputStream;
public class RuntimeExecTest {
@Test
public static void testRuntimeExec() {
try {
Process process = Runtime.getRuntime()
.exec("cmd.exe /c dir");
process.waitFor();
}
catch (Exception e) {
e.printStackTrace();
}
}
}

In short, there is only one line to call :Runtime.getRuntime().exec("cmd.exe /c dir") ; It looks very simple .

 

1.2. ProcessBuilder Realization

Use ProcessBuilder You need to operate more by yourself , So you can set more things on your own .( But in fact, the bottom and Runtime It's the same ), The use cases are as follows :

public class ProcessBuilderTest {
@Test
public void testProcessBuilder() {
ProcessBuilder processBuilder = new ProcessBuilder();
processBuilder.command("ipconfig");
// Merge standard input stream with error input stream , Read information through a standard input stream 
processBuilder.redirectErrorStream(true);
try {
// Start the process 
Process start = processBuilder.start();
// Get input stream 
InputStream inputStream = start.getInputStream();
// Convert to character input stream 
InputStreamReader inputStreamReader = new InputStreamReader(inputStream, "gbk");
int len = -1;
char[] c = new char[1024];
StringBuffer outputString = new StringBuffer();
// Read the contents of the process input stream 
while ((len = inputStreamReader.read(c)) != -1) {
String s = new String(c, 0, len);
outputString.append(s);
System.out.print(s);
}
inputStream.close();
}
catch (IOException e) {
e.printStackTrace();
}
}
}

It looks like it's going to be a little more trouble , But it's almost the same , It's just that the last use case didn't handle the output log . But on the whole ProcessBuilder More controllable , So it's more free to use this in general .

following Runtime.exec() The implementation of the :

 // java.lang.Runtime#exec
public Process exec(String[] cmdarray, String[] envp, File dir)
throws IOException {
// Only for ProcessBuilder An encapsulation of 
return new ProcessBuilder(cmdarray)
.environment(envp)
.directory(dir)
.start();
}

 

2. call shell Think about things

From above , To call the system command , It's not hard . Does that mean that we can call the ready-made scheme for processing at will ? Of course not. , We should consider a few questions ?

1. Calling system commands is a process level call ;
The difference between process and thread is well understood , More heavyweight , It's more expensive . stay java in , We use multithreading for concurrency . But if it's used for system calls , That's process level concurrency , And external processes are no longer subject to jvm control , If something goes wrong, it's not fun . therefore , It's a good practice not to call system commands .
2. Calling system commands is a hardware related call ;
java The idea of language is to write once , Use everywhere . But if you use the system call , It's not easy to handle , Because the commands supported by each system are not exactly the same , Your code will be inconsistent because of the environment . The robustness comes down , therefore , It's better to use less .
3. Is there enough memory ?
Generally we jvm Run as a stand-alone process , Will be allocated enough memory , To ensure smooth and efficient operation . At this time , Maybe there won't be much space left for the system , At this time, the system process is called to run the business , We have to predict in advance .
4. When the process stops ?
When I set up a system process , How do we do it later ? For example, if it is called asynchronously , Maybe the results are ignored . And if it's a synchronous call , The current thread must wait for the process to exit , This will greatly simplify our business . Because there are so many things to think about asynchrony .
5. How to get process log information ?
One shell Process call , It may be a time-consuming operation , This should be as long as any progress , It should be reported , So that the outside doesn't seem to be responding all the time , It is impossible to determine whether it is dead or running . And the communication of external processes , It's not like an ordinary io Call to , Output the result information directly . This often requires us to capture through two output streams . And how to read the data of these two output streams , It becomes the key to get log information .ProcessBuilder It's using inputStream and errStream To represent two output streams , It corresponds to the standard output stream and error output stream of the operating system respectively . But both streams are blocked io flow , If not handled properly , It will cause the risk of system feign death .
6. How to catch the exception of a process ?
stay jvm Exception generated in the thread , It can be easily used directly try...catch... Capture , and shell What about the exception called ? It doesn't actually throw exceptions directly , We can determine whether an exception has occurred by the return code of the process , These error codes generally follow the error definition specification of the operating system , But if we wrote it ourselves shell Or other students wrote shell There's no guarantee . therefore , Often, in addition to catching mistakes , At least, it should be stipulated that 0 For the correct return code . Try not to misuse other error codes . secondly , We should also when something goes wrong , From the error output stream information , Get some clues , So that we can quickly debug .

 

Above questions , If you can handle it properly , So I think , This call is safe . On the contrary, there are risks .

however , There are many problems , But it's all about refinement , Don't worry too much about . Basically , We use the thread pool to control the expansion of the process ; By reading the io Stream to solve the problem of abnormal information ; Planning memory and usage through call type ;

 

3. complete shell Call reference

So many theories , It's better to be practical .don't bb, show me the code! 

import com.my.mvc.app.common.exception.ShellProcessExecException;
import com.my.mvc.app.common.helper.NamedThreadFactory;
import lombok.extern.log4j.Log4j2;
import org.apache.commons.io.FileUtils;
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.InputStreamReader;
import java.nio.charset.Charset;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
/**
* Function description : Shell Command run tool class encapsulation
*
*/
@Log4j2
public class ShellCommandExecUtil {
/**
* @see #runShellCommandSync(String, String[], Charset, String)
*/
public static int runShellCommandSync(String baseShellDir, String[] cmd,
Charset outputCharset) throws IOException {
return runShellCommandSync(baseShellDir, cmd, outputCharset, null);
}
/**
* Really run shell command
*
* @param baseShellDir The directory where the command is running ( Switch to this directory before running the command )
* @param cmd Command array
* @param outputCharset Log output character set , commonly windows by GBK, linux by utf8
* @param logFilePath Log output file path , If it is empty, it will be output directly to the current application log , Otherwise write to the file
* @return Process exit code , 0: success , other : Failure
* @throws IOException Throw when executing an exception
*/
public static int runShellCommandSync(String baseShellDir, String[] cmd,
Charset outputCharset, String logFilePath)
throws IOException {
long startTime = System.currentTimeMillis();
boolean needReadProcessOutLogStreamByHand = true;
log.info("【cli】receive new Command. baseDir: {}, cmd: {}, logFile:{}",
baseShellDir, String.join(" ", cmd), logFilePath);
ProcessBuilder pb = new ProcessBuilder(cmd);
pb.directory(new File(baseShellDir));
initErrorLogHolder(logFilePath, outputCharset);
int exitCode = 0;
try {
if(logFilePath != null) {
ensureFilePathExists(logFilePath);
// String redirectLogInfoAndErrCmd = " > " + logFilePath + " 2>&1 ";
// cmd = mergeTwoArr(cmd, redirectLogInfoAndErrCmd.split("\\s+"));
pb.redirectErrorStream(true);
pb.redirectOutput(new File(logFilePath));
needReadProcessOutLogStreamByHand = false;
}
Process p = pb.start();
if(needReadProcessOutLogStreamByHand) {
readProcessOutLogStream(p, outputCharset);
}
try {
p.waitFor();
}
catch (InterruptedException e) {
log.error(" The process was interrupted ", e);
setProcessLastError(" Interrupt exception :" + e.getMessage());
}
finally {
exitCode = p.exitValue();
log.info("【cli】process costTime:{}ms, exitCode:{}",
System.currentTimeMillis() - startTime, exitCode);
}
if(exitCode != 0) {
throw new ShellProcessExecException(exitCode,
" Process returns exception information , returnCode:" + exitCode
+ ", lastError:" + getProcessLastError());
}
return exitCode;
}
finally {
removeErrorLogHolder();
}
}
/**
* Use Runtime.exec() function shell
*/
public static int runShellWithRuntime(String baseShellDir,
String[] cmd,
Charset outputCharset) throws IOException {
long startTime = System.currentTimeMillis();
initErrorLogHolder(null, outputCharset);
Process p = Runtime.getRuntime().exec(cmd, null, new File(baseShellDir));
readProcessOutLogStream(p, outputCharset);
int exitCode;
try {
p.waitFor();
}
catch (InterruptedException e) {
log.error(" The process was interrupted ", e);
setProcessLastError(" Interrupt exception :" + e.getMessage());
}
catch (Throwable e) {
log.error(" Other anomalies ", e);
setProcessLastError(e.getMessage());
}
finally {
exitCode = p.exitValue();
log.info("【cli】process costTime:{}ms, exitCode:{}",
System.currentTimeMillis() - startTime, exitCode);
}
if(exitCode != 0) {
throw new ShellProcessExecException(exitCode,
" Process returns exception information , returnCode:" + exitCode
+ ", lastError:" + getProcessLastError());
}
return exitCode;
}
/**
* Make sure the folder exists
*
* @param filePath File path
* @throws IOException Create folder exception throw
*/
public static void ensureFilePathExists(String filePath) throws IOException {
File path = new File(filePath);
if(path.exists()) {
return;
}
File p = path.getParentFile();
if(p.mkdirs()) {
log.info(" Create a directory for the file : {} success ", p.getPath());
return;
}
log.warn(" Create directory :{} Failure ", p.getPath());
}
/**
* Merge two array data
*
* @param arrFirst Array on the left
* @param arrAppend Array to add
* @return Combined array
*/
public static String[] mergeTwoArr(String[] arrFirst, String[] arrAppend) {
String[] merged = new String[arrFirst.length + arrAppend.length];
System.arraycopy(arrFirst, 0,
merged, 0, arrFirst.length);
System.arraycopy(arrAppend, 0,
merged, arrFirst.length, arrAppend.length);
return merged;
}
/**
* Delete a character that ends with a character
*
* @param originalStr Original characters
* @param toTrimChar The word to be detected
* @return The clipped string
*/
public static String trimEndsWith(String originalStr, char toTrimChar) {
char[] value = originalStr.toCharArray();
int i = value.length - 1;
while (i > 0 && value[i] == toTrimChar) {
i--;
}
return new String(value, 0, i + 1);
}
/**
* Error log read thread pool ( No upper limit )
*/
private static final ExecutorService errReadThreadPool = Executors.newCachedThreadPool(
new NamedThreadFactory("ReadProcessErrOut"));
/**
* Last exception message
*/
private static final Map<Thread, ProcessErrorLogDescriptor>
lastErrorHolder = new ConcurrentHashMap<>();
/**
* Actively read the standard output log of the process
*
* @param process Process entities
* @param outputCharset Log character set
* @throws IOException Throw when reading an exception
*/
private static void readProcessOutLogStream(Process process,
Charset outputCharset) throws IOException {
try (BufferedReader stdInput = new BufferedReader(new InputStreamReader(
process.getInputStream(), outputCharset))) {
Thread parentThread = Thread.currentThread();
// Start another thread to read the error message , The thread must be started first 
errReadThreadPool.submit(() -> {
try {
try (BufferedReader stdError = new BufferedReader(
new InputStreamReader(process.getErrorStream(), outputCharset))) {
String err;
while ((err = stdError.readLine()) != null) {
log.error("【cli】{}", err);
setProcessLastError(parentThread, err);
}
}
}
catch (IOException e) {
log.error(" An exception occurred while reading the process error log output ", e);
setProcessLastError(parentThread, e.getMessage());
}
});
// External thread reads standard output message 
 String stdOut;
while ((stdOut = stdInput.readLine()) != null) {
log.info("【cli】{}", stdOut);
}
}
}
/**
* Create a new process error information container
*
* @param logFilePath Log file path , If not, it is null
*/
private static void initErrorLogHolder(String logFilePath, Charset outputCharset) {
lastErrorHolder.put(Thread.currentThread(),
new ProcessErrorLogDescriptor(logFilePath, outputCharset));
}
/**
* Remove error log listening
*/
private static void removeErrorLogHolder() {
lastErrorHolder.remove(Thread.currentThread());
}
/**
* Get the last error message of the process
*
* Be careful : This method is called only in the parent thread
*/
private static String getProcessLastError() {
Thread thread = Thread.currentThread();
return lastErrorHolder.get(thread).getLastError();
}
/**
* Set the last error message description
*
* Use current thread or custom
*/
private static void setProcessLastError(String lastError) {
lastErrorHolder.get(Thread.currentThread()).setLastError(lastError);
}
private static void setProcessLastError(Thread thread, String lastError) {
lastErrorHolder.get(thread).setLastError(lastError);
}
/**
* Determine whether the current system is windows
*/
public static boolean isWindowsSystemOs() {
return System.getProperty("os.name").toLowerCase()
.startsWith("win");
}
/**
* Process error information describes the encapsulation class
*/
private static class ProcessErrorLogDescriptor {
/**
* Error message log file
*/
private String logFile;
/**
* The last line of error messages
*/
private String lastError;
private Charset charset;
ProcessErrorLogDescriptor(String logFile, Charset outputCharset) {
this.logFile = logFile;
charset = outputCharset;
}
String getLastError() {
if(lastError != null) {
return lastError;
}
try{
if(logFile == null) {
return null;
}
List<String> lines = FileUtils.readLines(
new File(logFile), charset);
StringBuilder sb = new StringBuilder();
for (int i = lines.size() - 1; i >= 0; i--) {
sb.insert(0, lines.get(i) + "\n");
if(sb.length() > 200) {
break;
}
}
return sb.toString();
}
catch (Exception e) {
log.error("【cli】 Failed to read the last error message ", e);
}
return null;
}
void setLastError(String err) {
if(lastError == null) {
lastError = err;
return;
}
lastError = lastError + "\n" + err;
if(lastError.length() > 200) {
lastError = lastError.substring(lastError.length() - 200);
}
}
}
}

The above implementation , It's done. We're at 2 Some of the issues discussed in point :

1. The main use of ProcessBuilder It's done shell Call to ;
2. Supports reading all the output information of the process , And when necessary , Support the use of separate files for receiving output logs ;
3. When the process execution exception , Support throwing corresponding exception , And give a certain amount of errMessage describe ;
4. If you want to control the number of calling processes , When the external call is made ;
5. Use two threads to receive two output streams , Avoid the use of suspended animation , Use newCachedThreadPool Thread pools avoid creating threads too quickly ;

Next , Let's do the next unit test :

public class ShellCommandExecUtilTest {
@Test
public void testRuntimeShell() throws IOException {
int errCode;
errCode = ShellCommandExecUtil.runShellWithRuntime("E:\\tmp",
new String[] {"cmd", "/c", "dir"}, Charset.forName("gbk"));
Assert.assertEquals(" Incorrect process return code ", 0, errCode);
}
@Test(expected = ShellProcessExecException.class)
public void testRuntimeShellWithErr() throws IOException {
int errCode;
errCode = ShellCommandExecUtil.runShellWithRuntime("E:\\tmp",
new String[] {"cmd", "/c", "dir2"}, Charset.forName("gbk"));
Assert.fail("dir2 It's supposed to fail , But it passed , Please find out why ");
}
@Test
public void testProcessShell1() throws IOException {
int errCode;
errCode = ShellCommandExecUtil.runShellCommandSync("/tmp",
new String[]{"cmd", "/c", "dir"}, Charset.forName("gbk"));
Assert.assertEquals(" Incorrect process return code ", 0, errCode);
String logPath = "/tmp/cmd.log";
errCode = ShellCommandExecUtil.runShellCommandSync("/tmp",
new String[]{"cmd", "/c", "dir"}, Charset.forName("gbk"), logPath);
Assert.assertTrue(" As a result, the log file does not exist ", new File(logPath).exists());
}
@Test(expected = ShellProcessExecException.class)
public void testProcessShell1WithErr() throws IOException {
int errCode;
errCode = ShellCommandExecUtil.runShellCommandSync("/tmp",
new String[]{"cmd", "/c", "dir2"}, Charset.forName("gbk"));
Assert.fail("dir2 It's supposed to fail , But it passed , Please find out why ");
}
@Test(expected = ShellProcessExecException.class)
public void testProcessShell1WithErr2() throws IOException {
int errCode;
String logPath = "/tmp/cmd2.log";
try {
errCode = ShellCommandExecUtil.runShellCommandSync("/tmp",
new String[]{"cmd", "/c", "dir2"}, Charset.forName("gbk"), logPath);
}
catch (ShellProcessExecException e) {
e.printStackTrace();
throw e;
}
Assert.assertTrue(" As a result, the log file does not exist ", new File(logPath).exists());
}
}

thus , One of our safe and reliable shell The running function is finished .

 

版权声明
本文为[When you come back]所创,转载请带上原文链接,感谢

  1. 【计算机网络 12(1),尚学堂马士兵Java视频教程
  2. 【程序猿历程,史上最全的Java面试题集锦在这里
  3. 【程序猿历程(1),Javaweb视频教程百度云
  4. Notes on MySQL 45 lectures (1-7)
  5. [computer network 12 (1), Shang Xuetang Ma soldier java video tutorial
  6. The most complete collection of Java interview questions in history is here
  7. [process of program ape (1), JavaWeb video tutorial, baidu cloud
  8. Notes on MySQL 45 lectures (1-7)
  9. 精进 Spring Boot 03:Spring Boot 的配置文件和配置管理,以及用三种方式读取配置文件
  10. Refined spring boot 03: spring boot configuration files and configuration management, and reading configuration files in three ways
  11. 精进 Spring Boot 03:Spring Boot 的配置文件和配置管理,以及用三种方式读取配置文件
  12. Refined spring boot 03: spring boot configuration files and configuration management, and reading configuration files in three ways
  13. 【递归,Java传智播客笔记
  14. [recursion, Java intelligence podcast notes
  15. [adhere to painting for 386 days] the beginning of spring of 24 solar terms
  16. K8S系列第八篇(Service、EndPoints以及高可用kubeadm部署)
  17. K8s Series Part 8 (service, endpoints and high availability kubeadm deployment)
  18. 【重识 HTML (3),350道Java面试真题分享
  19. 【重识 HTML (2),Java并发编程必会的多线程你竟然还不会
  20. 【重识 HTML (1),二本Java小菜鸟4面字节跳动被秒成渣渣
  21. [re recognize HTML (3) and share 350 real Java interview questions
  22. [re recognize HTML (2). Multithreading is a must for Java Concurrent Programming. How dare you not
  23. [re recognize HTML (1), two Java rookies' 4-sided bytes beat and become slag in seconds
  24. 造轮子系列之RPC 1:如何从零开始开发RPC框架
  25. RPC 1: how to develop RPC framework from scratch
  26. 造轮子系列之RPC 1:如何从零开始开发RPC框架
  27. RPC 1: how to develop RPC framework from scratch
  28. 一次性捋清楚吧,对乱糟糟的,Spring事务扩展机制
  29. 一文彻底弄懂如何选择抽象类还是接口,连续四年百度Java岗必问面试题
  30. Redis常用命令
  31. 一双拖鞋引发的血案,狂神说Java系列笔记
  32. 一、mysql基础安装
  33. 一位程序员的独白:尽管我一生坎坷,Java框架面试基础
  34. Clear it all at once. For the messy, spring transaction extension mechanism
  35. A thorough understanding of how to choose abstract classes or interfaces, baidu Java post must ask interview questions for four consecutive years
  36. Redis common commands
  37. A pair of slippers triggered the murder, crazy God said java series notes
  38. 1、 MySQL basic installation
  39. Monologue of a programmer: despite my ups and downs in my life, Java framework is the foundation of interview
  40. 【大厂面试】三面三问Spring循环依赖,请一定要把这篇看完(建议收藏)
  41. 一线互联网企业中,springboot入门项目
  42. 一篇文带你入门SSM框架Spring开发,帮你快速拿Offer
  43. 【面试资料】Java全集、微服务、大数据、数据结构与算法、机器学习知识最全总结,283页pdf
  44. 【leetcode刷题】24.数组中重复的数字——Java版
  45. 【leetcode刷题】23.对称二叉树——Java版
  46. 【leetcode刷题】22.二叉树的中序遍历——Java版
  47. 【leetcode刷题】21.三数之和——Java版
  48. 【leetcode刷题】20.最长回文子串——Java版
  49. 【leetcode刷题】19.回文链表——Java版
  50. 【leetcode刷题】18.反转链表——Java版
  51. 【leetcode刷题】17.相交链表——Java&python版
  52. 【leetcode刷题】16.环形链表——Java版
  53. 【leetcode刷题】15.汉明距离——Java版
  54. 【leetcode刷题】14.找到所有数组中消失的数字——Java版
  55. 【leetcode刷题】13.比特位计数——Java版
  56. oracle控制用户权限命令
  57. 三年Java开发,继阿里,鲁班二期Java架构师
  58. Oracle必须要启动的服务
  59. 万字长文!深入剖析HashMap,Java基础笔试题大全带答案
  60. 一问Kafka就心慌?我却凭着这份,图灵学院vip课程百度云