今天同事去厂商部署项目的时候发现项目启动一会便会奔溃,但是项目已经在其他的厂商和自己公司的云服务器上稳定运行了很长时间,且经过了大量严格的功能、并发测试,这让我有些诧异,通过了了解,知道了对方机器的操作系统为SUSE,同事发回了jvm奔溃时生成的hs_err_pid文件。通过查看hs_err_pid文件,发现以下信息较为关键。
Stack: [0x00007f823bcfd000,0x00007f823bdfe000], sp=0x00007f823bdf7a90, free space=1002k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libresolv.so.2+0x7e7d] __libc_res_nquery+0x4dd
C [libresolv.so.2+0x816d] __libc_res_nquerydomain+0xcd
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j java.net.Inet6AddressImpl.lookupAllHostAddr(Ljava/lang/String😉[Ljava/net/InetAddress;+0
j java.net.InetAddress\(<span class="hljs-number">1</span>.lookupAllHostAddr(Ljava<span class="hljs-operator">/</span>lang<span class="hljs-operator">/</span><span class="hljs-keyword">String</span>;)[Ljava<span class="hljs-operator">/</span>net<span class="hljs-operator">/</span>InetAddress;<span class="hljs-operator">+</span><span class="hljs-number">4</span>
<span class="hljs-comment">j jav</span>a.net.InetAddress.getAddressesFromNameService(Ljava<span class="hljs-operator">/</span>lang<span class="hljs-operator">/</span><span class="hljs-keyword">String</span>;Ljava<span class="hljs-operator">/</span>net<span class="hljs-operator">/</span>InetAddress;)[Ljava<span class="hljs-operator">/</span>net<span class="hljs-operator">/</span>InetAddress;<span class="hljs-operator">+</span><span class="hljs-number">51</span>
<span class="hljs-comment">j jav</span>a.net.InetAddress.getLocalHost()Ljava<span class="hljs-operator">/</span>net<span class="hljs-operator">/</span>InetAddress;<span class="hljs-operator">+</span><span class="hljs-number">90</span>
<span class="hljs-comment">j com</span>.mchange.v<span class="hljs-number">2</span>.uid.UidUtils.generateVmId()Ljava<span class="hljs-operator">/</span>lang<span class="hljs-operator">/</span><span class="hljs-keyword">String</span>;<span class="hljs-operator">+</span><span class="hljs-number">30</span>
<span class="hljs-comment">j com</span>.mchange.v<span class="hljs-number">2</span>.uid.UidUtils.<span class="hljs-operator"><</span>clinit<span class="hljs-operator">></span>()V<span class="hljs-operator">+</span><span class="hljs-number">9</span>
<span class="hljs-comment">v </span>~StubRoutines<span class="hljs-operator">::</span><span class="hljs-keyword">call</span>_stub
<span class="hljs-comment">j com</span>.mchange.v<span class="hljs-number">2</span>.c<span class="hljs-number">3</span>p<span class="hljs-number">0</span>.impl.C<span class="hljs-number">3</span>P<span class="hljs-number">0</span>ImplUtils.<span class="hljs-operator"><</span>clinit<span class="hljs-operator">></span>()V<span class="hljs-operator">+</span><span class="hljs-number">77</span>
<span class="hljs-comment">v </span>~StubRoutines<span class="hljs-operator">::</span><span class="hljs-keyword">call</span>_stub
<span class="hljs-comment">j com</span>.mchange.v<span class="hljs-number">2</span>.c<span class="hljs-number">3</span>p<span class="hljs-number">0</span>.impl.DriverManagerDataSourceBase.<span class="hljs-operator"><</span>init<span class="hljs-operator">></span>(Z)V<span class="hljs-operator">+</span><span class="hljs-number">69</span>
<span class="hljs-comment">j com</span>.mchange.v<span class="hljs-number">2</span>.c<span class="hljs-number">3</span>p<span class="hljs-number">0</span>.DriverManagerDataSource.<span class="hljs-operator"><</span>init<span class="hljs-operator">></span>(Z)V<span class="hljs-operator">+</span><span class="hljs-number">2</span>
<span class="hljs-comment">j com</span>.mchange.v<span class="hljs-number">2</span>.c<span class="hljs-number">3</span>p<span class="hljs-number">0</span>.DriverManagerDataSource.<span class="hljs-operator"><</span>init<span class="hljs-operator">></span>()V<span class="hljs-operator">+</span><span class="hljs-number">2</span>
<span class="hljs-comment">j com</span>.mchange.v<span class="hljs-number">2</span>.c<span class="hljs-number">3</span>p<span class="hljs-number">0</span>.DataSources.unpooledDataSource(Ljava<span class="hljs-operator">/</span>lang<span class="hljs-operator">/</span><span class="hljs-keyword">String</span>;Ljava<span class="hljs-operator">/</span>util<span class="hljs-operator">/</span>Properties;)Ljavax<span class="hljs-operator">/</span>sql<span class="hljs-operator">/</span>DataSource;<span class="hljs-operator">+</span><span class="hljs-number">4</span>
<span class="hljs-comment">j org</span>.hibernate.c<span class="hljs-number">3</span>p<span class="hljs-number">0</span>.internal.C<span class="hljs-number">3</span>P<span class="hljs-number">0</span>ConnectionProvider.configure(Ljava<span class="hljs-operator">/</span>util<span class="hljs-operator">/</span>Map;)V<span class="hljs-operator">+</span><span class="hljs-number">414</span>
<span class="hljs-comment">j org</span>.hibernate.boot.registry.internal.StandardServiceRegistryImpl.configureService(Lorg<span class="hljs-operator">/</span>hibernate<span class="hljs-operator">/</span>service<span class="hljs-operator">/</span>spi<span class="hljs-operator">/</span>ServiceBinding;)V<span class="hljs-operator">+</span><span class="hljs-number">24</span>
<span class="hljs-comment">j org</span>.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(Lorg<span class="hljs-operator">/</span>hibernate<span class="hljs-operator">/</span>service<span class="hljs-operator">/</span>spi<span class="hljs-operator">/</span>ServiceBinding;)Lorg<span class="hljs-operator">/</span>hibernate<span class="hljs-operator">/</span>service<span class="hljs-operator">/</span>Service;<span class="hljs-operator">+</span><span class="hljs-number">55</span>
<span class="hljs-comment">j org</span>.hibernate.service.internal.AbstractServiceRegistryImpl.getService(Ljava<span class="hljs-operator">/</span>lang<span class="hljs-operator">/</span><span class="hljs-keyword">Class</span>;)Lorg<span class="hljs-operator">/</span>hibernate<span class="hljs-operator">/</span>service<span class="hljs-operator">/</span>Service;<span class="hljs-operator">+</span><span class="hljs-number">30</span>
<span class="hljs-comment">j org</span>.hibernate.engine.jdbc.internal.JdbcServicesImpl.buildJdbcConnectionAccess(Ljava<span class="hljs-operator">/</span>util<span class="hljs-operator">/</span>Map;)Lorg<span class="hljs-operator">/</span>hibernate<span class="hljs-operator">/</span>engine<span class="hljs-operator">/</span>jdbc<span class="hljs-operator">/</span>spi<span class="hljs-operator">/</span>JdbcConnectionAccess;<span class="hljs-operator">+</span><span class="hljs-number">20</span>
<span class="hljs-comment">j org</span>.hibernate.engine.jdbc.internal.JdbcServicesImpl.configure(Ljava<span class="hljs-operator">/</span>util<span class="hljs-operator">/</span>Map;)V<span class="hljs-operator">+</span><span class="hljs-number">2</span>
<span class="hljs-comment">j org</span>.hibernate.boot.registry.internal.StandardServiceRegistryImpl.configureService(Lorg<span class="hljs-operator">/</span>hibernate<span class="hljs-operator">/</span>service<span class="hljs-operator">/</span>spi<span class="hljs-operator">/</span>ServiceBinding;)V<span class="hljs-operator">+</span><span class="hljs-number">24</span>
<span class="hljs-comment">j org</span>.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(Lorg<span class="hljs-operator">/</span>hibernate<span class="hljs-operator">/</span>service<span class="hljs-operator">/</span>spi<span class="hljs-operator">/</span>ServiceBinding;)Lorg<span class="hljs-operator">/</span>hibernate<span class="hljs-operator">/</span>service<span class="hljs-operator">/</span>Service;<span class="hljs-operator">+</span><span class="hljs-number">55</span>
<span class="hljs-comment">j org</span>.hibernate.service.internal.AbstractServiceRegistryImpl.getService(Ljava<span class="hljs-operator">/</span>lang<span class="hljs-operator">/</span><span class="hljs-keyword">Class</span>;)Lorg<span class="hljs-operator">/</span>hibernate<span class="hljs-operator">/</span>service<span class="hljs-operator">/</span>Service;<span class="hljs-operator">+</span><span class="hljs-number">30</span>
<span class="hljs-comment">j org</span>.hibernate.cfg.<span class="hljs-keyword">Configuration</span>.buildTypeRegistrations(Lorg<span class="hljs-operator">/</span>hibernate<span class="hljs-operator">/</span>service<span class="hljs-operator">/</span>ServiceRegistry;)V<span class="hljs-operator">+</span><span class="hljs-number">13</span>
<span class="hljs-comment">j org</span>.hibernate.cfg.<span class="hljs-keyword">Configuration</span>.buildSessionFactory(Lorg<span class="hljs-operator">/</span>hibernate<span class="hljs-operator">/</span>service<span class="hljs-operator">/</span>ServiceRegistry;)Lorg<span class="hljs-operator">/</span>hibernate<span class="hljs-operator">/</span>SessionFactory;<span class="hljs-operator">+</span><span class="hljs-number">17</span>
<span class="hljs-comment">j com</span>.dear.simpler.db.utils.HibernateUtil.init()V<span class="hljs-operator">+</span><span class="hljs-number">35</span>
<span class="hljs-comment">j com</span>.dear.simpler.db.utils.HibernateUtil.<span class="hljs-operator"><</span>clinit<span class="hljs-operator">></span>()V<span class="hljs-operator">+</span><span class="hljs-number">18</span>
<span class="hljs-comment">v </span>~StubRoutines<span class="hljs-operator">::</span><span class="hljs-keyword">call</span>_stub
<span class="hljs-comment">j com</span>.dear.simpler.dao.impl.BaseDaoImpl.getSession()Lorg<span class="hljs-operator">/</span>hibernate<span class="hljs-operator">/</span>Session;<span class="hljs-operator">+</span><span class="hljs-number">0</span>
<span class="hljs-comment">j com</span>.dear.simpler.dao.impl.BaseDaoImpl.<span class="hljs-keyword">get</span>(Ljava<span class="hljs-operator">/</span>io<span class="hljs-operator">/</span>Serializable;)Ljava<span class="hljs-operator">/</span>lang<span class="hljs-operator">/</span><span class="hljs-keyword">Object</span>;<span class="hljs-operator">+</span><span class="hljs-number">8</span>
<span class="hljs-comment">j com</span>.dear.simpler.dbrpc.service.SmbDBOperatorImpl.dbcache_getSpLock(Ljava<span class="hljs-operator">/</span>lang<span class="hljs-operator">/</span><span class="hljs-keyword">String</span>;)Lcom<span class="hljs-operator">/</span>dear<span class="hljs-operator">/</span>simpler<span class="hljs-operator">/</span>dbrpc<span class="hljs-operator">/</span>thrift<span class="hljs-operator">/</span>Response;<span class="hljs-operator">+</span><span class="hljs-number">25</span>
<span class="hljs-comment">v </span>~StubRoutines<span class="hljs-operator">::</span><span class="hljs-keyword">call</span>_stub
<span class="hljs-comment">j sun</span>.reflect.NativeMethodAccessorImpl.<span class="hljs-keyword">invoke</span><span class="hljs-number">0</span>(Ljava<span class="hljs-operator">/</span>lang<span class="hljs-operator">/</span>reflect<span class="hljs-operator">/</span><span class="hljs-keyword">Method</span>;Ljava<span class="hljs-operator">/</span>lang<span class="hljs-operator">/</span><span class="hljs-keyword">Object</span>;[Ljava<span class="hljs-operator">/</span>lang<span class="hljs-operator">/</span><span class="hljs-keyword">Object</span>;)Ljava<span class="hljs-operator">/</span>lang<span class="hljs-operator">/</span><span class="hljs-keyword">Object</span>;<span class="hljs-operator">+</span><span class="hljs-number">0</span>
<span class="hljs-comment">j sun</span>.reflect.NativeMethodAccessorImpl.<span class="hljs-keyword">invoke</span>(Ljava<span class="hljs-operator">/</span>lang<span class="hljs-operator">/</span><span class="hljs-keyword">Object</span>;[Ljava<span class="hljs-operator">/</span>lang<span class="hljs-operator">/</span><span class="hljs-keyword">Object</span>;)Ljava<span class="hljs-operator">/</span>lang<span class="hljs-operator">/</span><span class="hljs-keyword">Object</span>;<span class="hljs-operator">+</span><span class="hljs-number">87</span>
<span class="hljs-comment">j sun</span>.reflect.DelegatingMethodAccessorImpl.<span class="hljs-keyword">invoke</span>(Ljava<span class="hljs-operator">/</span>lang<span class="hljs-operator">/</span><span class="hljs-keyword">Object</span>;[Ljava<span class="hljs-operator">/</span>lang<span class="hljs-operator">/</span><span class="hljs-keyword">Object</span>;)Ljava<span class="hljs-operator">/</span>lang<span class="hljs-operator">/</span><span class="hljs-keyword">Object</span>;<span class="hljs-operator">+</span><span class="hljs-number">6</span>
<span class="hljs-comment">j jav</span>a.lang.reflect.<span class="hljs-keyword">Method</span>.<span class="hljs-keyword">invoke</span>(Ljava<span class="hljs-operator">/</span>lang<span class="hljs-operator">/</span><span class="hljs-keyword">Object</span>;[Ljava<span class="hljs-operator">/</span>lang<span class="hljs-operator">/</span><span class="hljs-keyword">Object</span>;)Ljava<span class="hljs-operator">/</span>lang<span class="hljs-operator">/</span><span class="hljs-keyword">Object</span>;<span class="hljs-operator">+</span><span class="hljs-number">57</span>
<span class="hljs-comment">j com</span>.dear.simpler.dbrpc.processor.SMBDBServiceImpl.invokeMethod(Ljava<span class="hljs-operator">/</span>lang<span class="hljs-operator">/</span><span class="hljs-keyword">String</span>;Ljava<span class="hljs-operator">/</span>lang<span class="hljs-operator">/</span><span class="hljs-keyword">String</span>;)Lcom<span class="hljs-operator">/</span>dear<span class="hljs-operator">/</span>simpler<span class="hljs-operator">/</span>dbrpc<span class="hljs-operator">/</span>thrift<span class="hljs-operator">/</span>Response;<span class="hljs-operator">+</span><span class="hljs-number">44</span>
<span class="hljs-comment">j com</span>.dear.simpler.dbrpc.processor.SMBDBServiceImpl.getService(Lcom<span class="hljs-operator">/</span>dear<span class="hljs-operator">/</span>simpler<span class="hljs-operator">/</span>dbrpc<span class="hljs-operator">/</span>thrift<span class="hljs-operator">/</span>Request;)Lcom<span class="hljs-operator">/</span>dear<span class="hljs-operator">/</span>simpler<span class="hljs-operator">/</span>dbrpc<span class="hljs-operator">/</span>thrift<span class="hljs-operator">/</span>Response;<span class="hljs-operator">+</span><span class="hljs-number">228</span>
<span class="hljs-comment">j com</span>.dear.simpler.dbrpc.thrift.SMBDBService\)Processor\(getService.getResult(Lcom<span class="hljs-operator">/</span>dear<span class="hljs-operator">/</span>simpler<span class="hljs-operator">/</span>dbrpc<span class="hljs-operator">/</span>thrift<span class="hljs-operator">/</span>SMBDBService\)Iface;Lcom/dear/simpler/dbrpc/thrift/SMBDBService\(getService_args;)Lcom<span class="hljs-operator">/</span>dear<span class="hljs-operator">/</span>simpler<span class="hljs-operator">/</span>dbrpc<span class="hljs-operator">/</span>thrift<span class="hljs-operator">/</span>SMBDBService\)getService_result;+14
j com.dear.simpler.dbrpc.thrift.SMBDBService\(Processor\)getService.getResult(Ljava/lang/Object;Lorg/apache/thrift/TBase;)Lorg/apache/thrift/TBase;+9
j org.apache.thrift.ProcessFunction.process(ILorg/apache/thrift/protocol/TProtocol;Lorg/apache/thrift/protocol/TProtocol;Ljava/lang/Object;)V+86
j org.apache.thrift.TBaseProcessor.process(Lorg/apache/thrift/protocol/TProtocol;Lorg/apache/thrift/protocol/TProtocol;)Z+126
j org.apache.thrift.server.AbstractNonblockingServer\(FrameBuffer.<span class="hljs-keyword">invoke</span>()V<span class="hljs-operator">+</span><span class="hljs-number">77</span>
<span class="hljs-comment">j org</span>.apache.thrift.server.Invocation.<span class="hljs-keyword">run</span>()V<span class="hljs-operator">+</span><span class="hljs-number">4</span>
<span class="hljs-comment">j jav</span>a.util.concurrent.ThreadPoolExecutor.runWorker(Ljava<span class="hljs-operator">/</span>util<span class="hljs-operator">/</span>concurrent<span class="hljs-operator">/</span>ThreadPoolExecutor\)Worker;)V+95
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub
通过观察,觉得问题应该就出现在java.net.Inet6AddressImpl.lookupAllHostAddr(Ljava/lang/String;)[Ljava/net/InetAddress;+0的地方,于是开始google 这个方法在SUSE中的问题,看看有没有人其他人遇到过。发现了这个有用信息:
The problem is that the method InetAddress.getLocalHost() does not work in SUSE environment for due to following reason. Wso2ESB uses this method.
SUSE ignores the /etc/resolv.conf but only looks at the /etc/hosts file (where you hadn't specified anything besides localhost)
http://stackoverflow.com/questions/1881546/inetaddress-getlocalhost-throws-unknownhostexception
原来,Java的InetAddressImpl的getLocalHostName的方法是native的,是本地系统的一个实现。它是依赖系统库来完成主机名,在大多数的linux系统中,都是以/etc/hosts中的配置来查找主机的,但在suse中却用/etc/hosts文件中的配置做主机名,所以会出现问题。
解决方案:在/etc/hosts中加入ip名与主机名的映射即可。问题解决