arthas服务集成

启动

arthas.sh脚本

创建脚本arthas.sh

#!/bin/bash

if [ -n "$1" ]; then
	echo "start arthas for $1"
	java -jar arthas-boot.jar $(ps -ef | grep java | grep -v grep | awk '{print $1}') -c "$1"
else
	echo "start arthas"
	java -jar arthas-boot.jar $(ps -ef | grep java | grep -v grep | awk '{print $1}')
fi
  • arthas的启动方式是java -jar arthas-boot.jar
  • 在服务中通过ps获取进程号,传入arthas监控该java进程
  • 上述脚本执行传入的参数,如./arthas.sh 'thread -n 1'

如果在运行中报错,无tools.jar;将该系统对应的jdk lib包下的tools.jar保存到该服务jdk -> lib下;注意系统需一致,比如服务是linux环境不能将windows对应的jdk tools.jar保存到该服务下

dump.sh脚本

#!/bin/bash

echo "==============================================" >> /opt/pinpoint-agent/logs/monitor.log

echo "begin: $(date) "  >> /opt/pinpoint-agent/logs/monitor.log

/opt/pinpoint-agent/arthas/arthas.sh 'dashboard -n 1;thread -n 100;heapdump /opt/pinpoint-agent/logs/dump.hprof' >> /opt/pinpoint-agent/logs/monitor.log

echo "end: $(date) " >> /opt/pinpoint-agent/logs/monitor.log

echo "==============================================" >> /opt/pinpoint-agent/logs/monitor.log
  • 执行dashboard -n 1;thread -n 100;heapdump三个命令,并将结果保存到文件中

结合k8s

添加preStop

lifecycle:
  preStop:
    exec:
      command:
      - /opt/pinpoint-agent/arthas/dump.sh

发现收集的内容为空

======================begin========================
======================begin========================
======================begin========================
begin: Mon Apr 27 18:06:56 CST 2020
start arthas for dashboard -n 1;thread -n 100;heapdump /opt/pinpoint-agent/logs/dump.hprof
end: Mon Apr 27 18:06:56 CST 2020
======================end========================
======================end========================
======================end========================

修改dump.sh脚本为

#!/bin/bash

echo "======================begin========================" >> /opt/pinpoint-agent/logs/monitor.log
echo "======================begin========================" >> /opt/pinpoint-agent/logs/monitor.log
echo "======================begin========================" >> /opt/pinpoint-agent/logs/monitor.log

echo "begin: $(date) "  >> /opt/pinpoint-agent/logs/monitor.log

java -jar /opt/pinpoint-agent/arthas/arthas-boot.jar $(ps -ef | grep java | grep -v grep | awk '{print $1}') -c "dashboard -n 1;thread -n 100;heapdump /opt/pinpoint-agent/logs/dump.hprof" >> /opt/pinpoint-agent/logs/monitor.log

echo "end: $(date) " >> /opt/pinpoint-agent/logs/monitor.log

echo "======================end========================" >> /opt/pinpoint-agent/logs/monitor.log
echo "======================end========================" >> /opt/pinpoint-agent/logs/monitor.log
echo "======================end========================" >> /opt/pinpoint-agent/logs/monitor.log

如果仍然未记录全尝试调整k8s deploy terminationGracePeriodSeconds的值

重点命令

dashboard-jvm的线程、内存、gc、vm、tomcat信息概述

[arthas@18]$ dashboard -n 1
······

heapdump-存储堆的信息到dump.111.hprof

[arthas@18]$ heapdump /tmp/dump.111.hprof
Dumping heap to /tmp/dump.111.hprof...
Heap dump file created

thread-显示线程信息,线程堆栈

[arthas@18]$ thread -n 2
"as-command-execute-daemon" Id=484 cpuUsage=86% RUNNABLE
    at sun.management.ThreadImpl.dumpThreads0(Native Method)
    at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:448)
    at com.taobao.arthas.core.command.monitor200.ThreadCommand.processTopBusyThreads(ThreadCommand.java:179)
    at com.taobao.arthas.core.command.monitor200.ThreadCommand.process(ThreadCommand.java:100)
    at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.process(AnnotatedCommandImpl.java:82)
    at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.access$100(AnnotatedCommandImpl.java:18)
    at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl$ProcessHandler.handle(AnnotatedCommandImpl.java:111)
    at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl$ProcessHandler.handle(AnnotatedCommandImpl.java:108)
    at com.taobao.arthas.core.shell.system.impl.ProcessImpl$CommandProcessTask.run(ProcessImpl.java:371)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

    Number of locked synchronizers = 1
    - java.util.concurrent.ThreadPoolExecutor$Worker@3397ef06


"Pinpoint-TcpDataSender(Default)-Timer(13-0)" Id=50 cpuUsage=6% TIMED_WAITING
    at java.lang.Thread.sleep(Native Method)
    at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:445)
    at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:364)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at java.lang.Thread.run(Thread.java:745)


Affect(row-cnt:0) cost in 154 ms.

stack-显示指定类和方法的堆栈跟踪

[arthas@18]$ stack *SyncIndexServiceImpl -n 1
Press Q or Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:7) cost in 155 ms.
ts=2020-04-27 16:03:11;thread_name=org.springframework.amqp.rabbit.RabbitListenerEndpointContainer#1-1;id=87;is_daemon=false;priority=5;TCCL=org.springframework.boot.loader.LaunchedURLClassLoader@5be067de
    @com.xxx.search.service.impl.SyncIndexServiceImpl.syncGroup()
        at com.xxx.search.mq.ConsumerClient.processMessage(ConsumerClient.java:80)
        at sun.reflect.GeneratedMethodAccessor212.invoke(null:-1)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:171)
        at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:120)
        at org.springframework.amqp.rabbit.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:50)
        at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:196)
        at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:129)
        at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1552)
        at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.actualInvokeListener(AbstractMessageListenerContainer.java:1478)
        at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:1466)
        at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:1461)
        at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:1410)
        at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:870)
        at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:854)
        at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1600(SimpleMessageListenerContainer.java:78)
        at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.mainLoop(SimpleMessageListenerContainer.java:1137)
        at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1043)
        at java.lang.Thread.run(Thread.java:745)

Command execution times exceed limit: 1, so command will exit. You can set it with -n option.

热更新代码

# 反编译
$ jad --source-only com.example.demo.arthas.user.UserController > /tmp/UserController.java
# 修改文件
$ vim /tmp/UserController.java
# 查找加载的 ClassLoader
$ $ sc -d *UserController | grep classLoaderHash
 classLoaderHash   6bc28484
# 编译
$ mc -c 6bc28484 /tmp/UserController.java -d /tmp
# 热更新
$ redefine /tmp/com/example/demo/arthas/user/UserController.class
redefine success, size: 1

帮助文档

查看arthar-boot.jar帮助文档

$java -jar arthas-boot.jar -h

······
Options and Arguments:
    --versions                  List local and remote arthas versions
    --use-http                  Enforce use http to download, default use https
    --attach-only               Attach target process only, do not connect
 -c,--command <value>           Command to execute, multiple commands separated
                                by ;
 -f,--batch-file <value>        The batch file to execute
    --height <value>            arthas-client terminal height
    --width <value>             arthas-client terminal width
 -v,--verbose                   Verbose, print debug info.
    --tunnel-server <value>     The tunnel server url
    --agent-id <value>          The agent id register to tunnel server
    --stat-url <value>          The report stat url
 -h,--help                      Print usage
    --target-ip <value>         The target jvm listen ip, default 127.0.0.1
    --telnet-port <value>       The target jvm listen telnet port, default 3658
    --http-port <value>         The target jvm listen http port, default 8563
    --session-timeout <value>   The session timeout seconds, default 1800
                                (30min)
    --arthas-home <value>       The arthas home
    --use-version <value>       Use special version arthas
    --repo-mirror <value>       Use special maven repository mirror, value is
                                center/aliyun or http repo url.
 <pid>                          Target pid
······

比如可直接执行java -jar arthas-boot.jar 18 -c 'dashboard -n 1'查看dashboard内容,但是值得注意的是通过-c执行dashboard与进入arthas命令行执行dashboard展示的结果是有区别的,如通过-c执行dashboard是无metadata部分信息的

查看arthas帮助文档

[arthas@18]$help
NAME         DESCRIPTION
 help         Display Arthas Help
 keymap       Display all the available keymap for the specified connection.
 sc           Search all the classes loaded by JVM
 sm           Search the method of classes loaded by JVM
 classloader  Show classloader info
 jad          Decompile class
 getstatic    Show the static field of a class
 monitor      Monitor method execution statistics, e.g. total/success/failure count, average rt, fail rate, etc.
 stack        Display the stack trace for the specified class and method
 thread       Display thread info, thread stack
 trace        Trace the execution time of specified method invocation.
 watch        Display the input/output parameter, return object, and thrown exception of specified method invocation
 tt           Time Tunnel
 jvm          Display the target JVM information
 perfcounter  Display the perf counter infornation.
 ognl         Execute ognl expression.
 mc           Memory compiler, compiles java files into bytecode and class files in memory.
 redefine     Redefine classes. @see Instrumentation#redefineClasses(ClassDefinition...)
 dashboard    Overview of target jvm's thread, memory, gc, vm, tomcat info.
 dump         Dump class byte array from JVM
 heapdump     Heap dump
 options      View and change various Arthas options
 cls          Clear the screen
 reset        Reset all the enhanced classes
 version      Display Arthas version
 session      Display current session information
 sysprop      Display, and change the system properties.
 sysenv       Display the system env.
 vmoption     Display, and update the vm diagnostic options.
 logger       Print logger info, and update the logger level
 history      Display command history
 cat          Concatenate and print files
 echo         write arguments to the standard output
 pwd          Return working directory name
 mbean        Display the mbean information
 grep         grep command for pipes.
 tee          tee command for pipes.
 profiler     Async Profiler. https://github.com/jvm-profiling-tools/async-profiler
 stop         Stop/Shutdown Arthas server and exit the console.

查看具体命令帮助文档

[arthas@18]$ sysenv -h
 USAGE:
   sysenv [-h] [env-name]
 SUMMARY:
   Display the system env.
 EXAMPLES:
   sysenv
   sysenv USER
 WIKI:
   https://alibaba.github.io/arthas/sysenv
 OPTIONS:
 -h, --help                 this help
 <env-name>                 env name

arthas官网