同事说crm登录hang住了，tomcat后台应用报错如下：
2016-06-28 10:30:11,214 [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0] WARN [com.mchange.v2.resourcepool.BasicResourcePool] - com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@3930f3cc -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: java.sql.SQLException: Io exception: Message file 'oracle.net.mesg.Message' is missing. at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112) at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146) at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:255) at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:387) at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:414) at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165) at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35) at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801) at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:134) at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182) at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137) at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014) at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32) at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810) at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547) 2016-06-28 10:30:56,213 [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0] WARN [com.mchange.v2.resourcepool.BasicResourcePool] - com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@7127f4c0 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: java.sql.SQLException: Io exception: Message file 'oracle.net.mesg.Message' is missing. at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112) at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146) at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:255) at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:387) at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:414) at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165) at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35) at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801) at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:134) at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182) at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137) at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014) at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32) at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810) at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Jun 28, 2016 10:33:59 AM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run SEVERE: Socket accept failed java.net.SocketException: Too many open files at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.implAccept(ServerSocket.java:530) at java.net.ServerSocket.accept(ServerSocket.java:498) at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60) at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:222) at java.lang.Thread.run(Thread.java:745)
|
看到有信息java.sql.SQLException: Ioexception: Message file 'oracle.net.mesg.Message' is missing. 先去查看数据库是否正常：通过plsql连接oracle数据库，显示数据库正常：
SQL> create table zz_bak (id number);
Table created
SQL> insert into zz_bak(id) values(1);
1 row inserted
SQL> commit;
Commit complete
SQL> select * from zz_bak;
ID
----------
1
SQL> drop table zz_bak;
Table dropped
SQL>-- 用dba权限的用户查看数据库都有哪些锁，没有查询到记录
selectt2.username,t2.sid,t2.serial#,t2.logon_time from v$locked_object t1,v$sessiont2 where t1.session_id=t2.sid order by t2.logon_time;
-- 查看session等待，也没用记录
SELECT/*+ rule */ lpad(' ', decode(l.xidusn, 0, 3, 0)) || l.oracle_usernameUser_name,o.owner,o.object_name,o.object_type,s.sid,s.serial#,p.spid
FROMv$locked_object l, dba_objects o, v$session s, v$process p
WHEREl.object_id = o.object_id
AND l.session_id = s.sid and s.paddr =p.addr
ORDERBY o.object_id, xidusn DESC
看到有“java.net.SocketException:Too many open files”信息，查看打开文件数：
[tomcat@crmtestlogs]$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 112257 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 65535 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 65535 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited [tomcat@crmtestlogs]$
查看系统句柄文件数，当前系统文件句柄的最大数目，只用于查看，不能设置修改
# cat /proc/sys/fs/file-max [tomcat@crmtestlogs]$ cat/proc/sys/fs/file-max 1427648 [tomcat@crmtestlogs]$
如果需要查看所有进程的文件打开数，如下图命令lsof |wc –l：
[tomcat@crmtestlogs]$ lsof |wc -l 1508 [tomcat@crmtestlogs]$
分析得出：文件句柄数量有些多，但是打开文件数还好不算太多
文件句柄数有些多，所以采用解决办法是：重启tomcat应用服务器，释放打开的文件句柄，之后观察5分钟，再也不会报错了。