arthas服务集成
启动
arthas.sh脚本
创建脚本arthas.sh
#!/bin/bash
if [ -n "$1" ]; then
echo "start arthas for $1"
java -jar arthas-boot.jar $(ps -ef | grep java | grep -v grep | awk '{print $1}') -c "$1"
else
echo "start arthas"
java -jar arthas-boot.jar $(ps -ef | grep java | grep -v grep | awk '{print $1}')
fi
- arthas的启动方式是
java -jar arthas-boot.jar - 在服务中通过ps获取进程号，传入arthas监控该java进程
- 上述脚本执行传入的参数，如
./arthas.sh 'thread -n 1'
如果在运行中报错，无tools.jar；将该系统对应的jdk lib包下的tools.jar保存到该服务jdk -> lib下；注意系统需一致，比如服务是linux环境不能将windows对应的jdk tools.jar保存到该服务下
dump.sh脚本
#!/bin/bash
echo "==============================================" >> /opt/pinpoint-agent/logs/monitor.log
echo "begin: $(date) " >> /opt/pinpoint-agent/logs/monitor.log
/opt/pinpoint-agent/arthas/arthas.sh 'dashboard -n 1;thread -n 100;heapdump /opt/pinpoint-agent/logs/dump.hprof' >> /opt/pinpoint-agent/logs/monitor.log
echo "end: $(date) " >> /opt/pinpoint-agent/logs/monitor.log
echo "==============================================" >> /opt/pinpoint-agent/logs/monitor.log
- 执行
dashboard -n 1;thread -n 100;heapdump三个命令，并将结果保存到文件中
结合k8s
添加preStop
lifecycle:
preStop:
exec:
command:
- /opt/pinpoint-agent/arthas/dump.sh
发现收集的内容为空
======================begin========================
======================begin========================
======================begin========================
begin: Mon Apr 27 18:06:56 CST 2020
start arthas for dashboard -n 1;thread -n 100;heapdump /opt/pinpoint-agent/logs/dump.hprof
end: Mon Apr 27 18:06:56 CST 2020
======================end========================
======================end========================
======================end========================
修改dump.sh脚本为
#!/bin/bash
echo "======================begin========================" >> /opt/pinpoint-agent/logs/monitor.log
echo "======================begin========================" >> /opt/pinpoint-agent/logs/monitor.log
echo "======================begin========================" >> /opt/pinpoint-agent/logs/monitor.log
echo "begin: $(date) " >> /opt/pinpoint-agent/logs/monitor.log
java -jar /opt/pinpoint-agent/arthas/arthas-boot.jar $(ps -ef | grep java | grep -v grep | awk '{print $1}') -c "dashboard -n 1;thread -n 100;heapdump /opt/pinpoint-agent/logs/dump.hprof" >> /opt/pinpoint-agent/logs/monitor.log
echo "end: $(date) " >> /opt/pinpoint-agent/logs/monitor.log
echo "======================end========================" >> /opt/pinpoint-agent/logs/monitor.log
echo "======================end========================" >> /opt/pinpoint-agent/logs/monitor.log
echo "======================end========================" >> /opt/pinpoint-agent/logs/monitor.log
如果仍然未记录全尝试调整k8s deploy terminationGracePeriodSeconds的值
重点命令
dashboard-jvm的线程、内存、gc、vm、tomcat信息概述
[arthas@18]$ dashboard -n 1
······
heapdump-存储堆的信息到dump.111.hprof
[arthas@18]$ heapdump /tmp/dump.111.hprof
Dumping heap to /tmp/dump.111.hprof...
Heap dump file created
thread-显示线程信息，线程堆栈
[arthas@18]$ thread -n 2
"as-command-execute-daemon" Id=484 cpuUsage=86% RUNNABLE
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:448)
at com.taobao.arthas.core.command.monitor200.ThreadCommand.processTopBusyThreads(ThreadCommand.java:179)
at com.taobao.arthas.core.command.monitor200.ThreadCommand.process(ThreadCommand.java:100)
at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.process(AnnotatedCommandImpl.java:82)
at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.access$100(AnnotatedCommandImpl.java:18)
at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl$ProcessHandler.handle(AnnotatedCommandImpl.java:111)
at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl$ProcessHandler.handle(AnnotatedCommandImpl.java:108)
at com.taobao.arthas.core.shell.system.impl.ProcessImpl$CommandProcessTask.run(ProcessImpl.java:371)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@3397ef06
"Pinpoint-TcpDataSender(Default)-Timer(13-0)" Id=50 cpuUsage=6% TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:445)
at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:364)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at java.lang.Thread.run(Thread.java:745)
Affect(row-cnt:0) cost in 154 ms.
stack-显示指定类和方法的堆栈跟踪
[arthas@18]$ stack *SyncIndexServiceImpl -n 1
Press Q or Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:7) cost in 155 ms.
ts=2020-04-27 16:03:11;thread_name=org.springframework.amqp.rabbit.RabbitListenerEndpointContainer#1-1;id=87;is_daemon=false;priority=5;TCCL=org.springframework.boot.loader.LaunchedURLClassLoader@5be067de
@com.xxx.search.service.impl.SyncIndexServiceImpl.syncGroup()
at com.xxx.search.mq.ConsumerClient.processMessage(ConsumerClient.java:80)
at sun.reflect.GeneratedMethodAccessor212.invoke(null:-1)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:171)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:120)
at org.springframework.amqp.rabbit.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:50)
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:196)
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:129)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1552)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.actualInvokeListener(AbstractMessageListenerContainer.java:1478)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:1466)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:1461)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:1410)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:870)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:854)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1600(SimpleMessageListenerContainer.java:78)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.mainLoop(SimpleMessageListenerContainer.java:1137)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1043)
at java.lang.Thread.run(Thread.java:745)
Command execution times exceed limit: 1, so command will exit. You can set it with -n option.
热更新代码
# 反编译
$ jad --source-only com.example.demo.arthas.user.UserController > /tmp/UserController.java
# 修改文件
$ vim /tmp/UserController.java
# 查找加载的 ClassLoader
$ $ sc -d *UserController | grep classLoaderHash
classLoaderHash 6bc28484
# 编译
$ mc -c 6bc28484 /tmp/UserController.java -d /tmp
# 热更新
$ redefine /tmp/com/example/demo/arthas/user/UserController.class
redefine success, size: 1
帮助文档
查看arthar-boot.jar帮助文档
$java -jar arthas-boot.jar -h
······
Options and Arguments:
--versions List local and remote arthas versions
--use-http Enforce use http to download, default use https
--attach-only Attach target process only, do not connect
-c,--command <value> Command to execute, multiple commands separated
by ;
-f,--batch-file <value> The batch file to execute
--height <value> arthas-client terminal height
--width <value> arthas-client terminal width
-v,--verbose Verbose, print debug info.
--tunnel-server <value> The tunnel server url
--agent-id <value> The agent id register to tunnel server
--stat-url <value> The report stat url
-h,--help Print usage
--target-ip <value> The target jvm listen ip, default 127.0.0.1
--telnet-port <value> The target jvm listen telnet port, default 3658
--http-port <value> The target jvm listen http port, default 8563
--session-timeout <value> The session timeout seconds, default 1800
(30min)
--arthas-home <value> The arthas home
--use-version <value> Use special version arthas
--repo-mirror <value> Use special maven repository mirror, value is
center/aliyun or http repo url.
<pid> Target pid
······
比如可直接执行java -jar arthas-boot.jar 18 -c 'dashboard -n 1'查看dashboard内容，但是值得注意的是通过-c执行dashboard与进入arthas命令行执行dashboard展示的结果是有区别的，如通过-c执行dashboard是无metadata部分信息的
查看arthas帮助文档
[arthas@18]$help
NAME DESCRIPTION
help Display Arthas Help
keymap Display all the available keymap for the specified connection.
sc Search all the classes loaded by JVM
sm Search the method of classes loaded by JVM
classloader Show classloader info
jad Decompile class
getstatic Show the static field of a class
monitor Monitor method execution statistics, e.g. total/success/failure count, average rt, fail rate, etc.
stack Display the stack trace for the specified class and method
thread Display thread info, thread stack
trace Trace the execution time of specified method invocation.
watch Display the input/output parameter, return object, and thrown exception of specified method invocation
tt Time Tunnel
jvm Display the target JVM information
perfcounter Display the perf counter infornation.
ognl Execute ognl expression.
mc Memory compiler, compiles java files into bytecode and class files in memory.
redefine Redefine classes. @see Instrumentation#redefineClasses(ClassDefinition...)
dashboard Overview of target jvm's thread, memory, gc, vm, tomcat info.
dump Dump class byte array from JVM
heapdump Heap dump
options View and change various Arthas options
cls Clear the screen
reset Reset all the enhanced classes
version Display Arthas version
session Display current session information
sysprop Display, and change the system properties.
sysenv Display the system env.
vmoption Display, and update the vm diagnostic options.
logger Print logger info, and update the logger level
history Display command history
cat Concatenate and print files
echo write arguments to the standard output
pwd Return working directory name
mbean Display the mbean information
grep grep command for pipes.
tee tee command for pipes.
profiler Async Profiler. https://github.com/jvm-profiling-tools/async-profiler
stop Stop/Shutdown Arthas server and exit the console.
查看具体命令帮助文档
[arthas@18]$ sysenv -h
USAGE:
sysenv [-h] [env-name]
SUMMARY:
Display the system env.
EXAMPLES:
sysenv
sysenv USER
WIKI:
https://alibaba.github.io/arthas/sysenv
OPTIONS:
-h, --help this help
<env-name> env name