springboot下,JedisPool getResource导致大量线程WAITING,服务假死

springboot 同时被 2 个专栏收录
16 篇文章 0 订阅
87 篇文章 1 订阅

环境:

springboot版本2.1.4.RELEASE、jedis连接池

服务配置:

设置了tomcat最大线程数为1000:

server:
  port: 9090
  tomcat:
    uri-encoding: utf-8
    max-threads: 1000

jedis连接池配置:

      pool:
        max-active: 300  # 连接池最大连接数(使用负值表示没有限制)
        max-wait: -1  # 连接池最大阻塞等待时间(使用负值表示没有限制)
        max-idle: 100  # 连接池中的最大空闲连接
        min-idle: 20   # 连接池中的最小空闲连接

现象:

1、浏览器访问业务接口链接,报超时,且服务日志没有接口相关日志输出

2、Eureka控制台上能够看到该服务节点,且状态正常

3、打开服务日志查看,每隔5分钟,和Eureka通讯一次

23:47:36.206 INFO  [trap-executor-0] c.n.d.s.r.a.ConfigClusterResolver - Resolving eureka endpoints via configuration
23:52:36.206 INFO  [trap-executor-0] c.n.d.s.r.a.ConfigClusterResolver - Resolving eureka endpoints via configuration
23:57:36.206 INFO  [trap-executor-0] c.n.d.s.r.a.ConfigClusterResolver - Resolving eureka endpoints via configuration

排查:

jps

top -Hp pid

发现 Threads:  1017 total
这么多的线程说明服务不正常,正常来说,线程数也就80多,100以内。

继续排查

jstack pid

发现有大量线程处于WAITING状态:

"http-nio-9084-exec-1059" #18281 daemon prio=5 os_prio=0 tid=0x00007f951cc17000 nid=0x75de waiting on condition [0x00007f947d1cf000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000000801d03f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:590)
    at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:425)
    at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:346)
    at redis.clients.util.Pool.getResource(Pool.java:49)
    at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
    at redis.clients.jedis.JedisSlotBasedConnectionHandler.getConnectionFromSlot(JedisSlotBasedConnectionHandler.java:70)
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:113)
    at redis.clients.jedis.JedisClusterCommand.runBinary(JedisClusterCommand.java:58)
    at redis.clients.jedis.BinaryJedisCluster.hget(BinaryJedisCluster.java:373)
    at org.springframework.data.redis.connection.jedis.JedisClusterHashCommands.hGet(JedisClusterHashCommands.java:95)

而且都在等待锁定同一个地址<0x00000000801d03f8>,统计了一下这样的线程有1000个,和上面设置的tomcat max-threads刚好相等。

猜测可能就是这个原因导致后续请求过来,tomcat由于线程数到达上限而拒绝响应。但是tomcat并没有死掉,因为它还在跟Eureka正常通讯, Eureka控制台也能看到这个节点。

结合上面的堆栈,可以看出JedisPool在getResource的时候被阻塞住了。为什么会这样呢?可能某个时间JedisPool中的线程获取Redis连接超时,而连接池max-wait配置的值是-1,也就是一直阻塞等待下去,导致了线程越积越多,最后超过Tomcat设置的max-threads,而无法响应后续请求。

方案:

给max-wait设置一个正值,当超过这个时间,就会抛出异常:

max-wait: 1000

观察一段时间再说,看看管用不管用!

附JedisPool源码:

@Override
public Jedis getResource() {
  Jedis jedis = super.getResource();
  jedis.setDataSource(this);
  return jedis;
}


public T getResource() {
  try {
    return internalPool.borrowObject();
  } catch (Exception e) {
    throw new JedisConnectionException("Could not get a resource from the pool", e);
  }
}



public T borrowObject(long borrowMaxWaitMillis) throws Exception {
    assertOpen();

    AbandonedConfig ac = this.abandonedConfig;
    if (ac != null && ac.getRemoveAbandonedOnBorrow() &&
            (getNumIdle() < 2) &&
            (getNumActive() > getMaxTotal() - 3) ) {
        removeAbandoned(ac);
    }

    PooledObject<T> p = null;

    // Get local copy of current config so it is consistent for entire
    // method execution
    boolean blockWhenExhausted = getBlockWhenExhausted();

    boolean create;
    long waitTime = 0;

    while (p == null) {
        create = false;
        if (blockWhenExhausted) {
            p = idleObjects.pollFirst();
            if (p == null) {
                create = true;
                p = create();
            }
            if (p == null) {
                if (borrowMaxWaitMillis < 0) {
                    p = idleObjects.takeFirst();
                } else {
                    waitTime = System.currentTimeMillis();
                    p = idleObjects.pollFirst(borrowMaxWaitMillis,
                            TimeUnit.MILLISECONDS);
                    waitTime = System.currentTimeMillis() - waitTime;
                }
            }
            if (p == null) {
                throw new NoSuchElementException(
                        "Timeout waiting for idle object");
            }
            if (!p.allocate()) {
                p = null;
            }
        } else {
            p = idleObjects.pollFirst();
            if (p == null) {
                create = true;
                p = create();
            }
            if (p == null) {
                throw new NoSuchElementException("Pool exhausted");
            }
            if (!p.allocate()) {
                p = null;
            }
        }

        if (p != null) {
            try {
                factory.activateObject(p);
            } catch (Exception e) {
                try {
                    destroy(p);
                } catch (Exception e1) {
                    // Ignore - activation failure is more important
                }
                p = null;
                if (create) {
                    NoSuchElementException nsee = new NoSuchElementException(
                            "Unable to activate object");
                    nsee.initCause(e);
                    throw nsee;
                }
            }
            if (p != null && getTestOnBorrow()) {
                boolean validate = false;
                Throwable validationThrowable = null;
                try {
                    validate = factory.validateObject(p);
                } catch (Throwable t) {
                    PoolUtils.checkRethrow(t);
                    validationThrowable = t;
                }
                if (!validate) {
                    try {
                        destroy(p);
                        destroyedByBorrowValidationCount.incrementAndGet();
                    } catch (Exception e) {
                        // Ignore - validation failure is more important
                    }
                    p = null;
                    if (create) {
                        NoSuchElementException nsee = new NoSuchElementException(
                                "Unable to validate object");
                        nsee.initCause(validationThrowable);
                        throw nsee;
                    }
                }
            }
        }
    }

    updateStatsBorrow(p, waitTime);

    return p.getObject();
}
    /**
     * Unlinks the first element in the queue, waiting until there is an element
     * to unlink if the queue is empty.
     *
     * @return the unlinked element
     * @throws InterruptedException if the current thread is interrupted
     */
    public E takeFirst() throws InterruptedException {
        lock.lock();
        try {
            E x;
            while ( (x = unlinkFirst()) == null) {
                notEmpty.await();
            }
            return x;
        } finally {
            lock.unlock();
        }
    }

参考链接:

https://www.codetd.com/article/4661280

https://blog.csdn.net/u012998254/article/details/78305866

https://www.jianshu.com/p/c4a75ca20abe

  • 1
    点赞
  • 3
    评论
  • 1
    收藏
  • 一键三连
    一键三连
  • 扫一扫,分享海报

相关推荐
<p style="color:#666666;"> <span style="font-size:14px;">本门课程重实战将基础知识拆解到项目里让你在项目情境里学知识。</span> </p> <p style="color:#666666;"> <span style="font-size:14px;">这样的学习方式能让你保持兴趣、充满动力时刻知道学的东西能用在哪、能怎么用。</span> </p> <p style="color:#666666;"> <span style="font-size:14px;">平时不明白的知识点放在项目里去理解就恍然大悟了。</span> </p> <p style="color:#666666;"> <span></span> </p> <p style="color:#666666;"> <span style="font-size:14px;"> </span> </p> <p style="color:#666666;"> <span style="color:#FF0000;font-size:14px;"><strong>一、融汇贯通</strong></span> </p> <p style="color:#666666;"> <span style="font-size:14px;">本视频采用了前后端分离的开发模式前端使用Vue.js+Element UI实现了Web页面的呈现后端使用Python 的Django框架实现了数据访问的接口前端通过Axios访问后端接口获得数据。在学习完本章节后真正理解前后端的各自承担的工作。</span> </p> <p style="color:#666666;"> <span style="font-size:14px;"> </span> </p> <p style="color:#666666;"> <span style="color:#FF0000;font-size:14px;"><strong>二、贴近实战</strong></span> </p> <p style="color:#666666;"> <span style="font-size:14px;">本系列课程为练手项目实战:学生管理系统v4.0的开发项目包含了如几个内容:项目的总体介绍、基本功能的演示、Vuejs的初始化、Element UI的使用、在Django中实现针对数据的增删改查的接口、在Vuejs中实现前端增删改查的调用、实现文件的上传、实现表格的分页、实现导出数据到Excel、实现通过Excel导入数据、实现针对表格的批量化操作等等所有的功能都通过演示完成、贴近了实战</span> </p> <p style="color:#666666;"> <span style="font-size:14px;"> </span> </p> <p style="color:#666666;"> <span style="color:#FF0000;font-size:14px;"><strong>三、课程亮点</strong></span> </p> <p style="color:#666666;"> <span style="font-size:14px;">在本案例中最大的亮点在于前后端做了分离真正理解前后端的各自承担的工作。前端如何和后端交互</span> </p> <p style="color:#666666;"> <span style="font-size:14px;"> </span> </p> <p style="color:#666666;"> <span style="color:#FF0000;font-size:14px;"><strong>适合人群:</strong></span> </p> <p style="color:#666666;"> <span style="font-size:14px;">1、有Python语言基础、web前端基础想要深入学习Python Web框架的朋友;</span> </p> <p style="color:#666666;"> <span style="font-size:14px;">2、有Django基础但是想学习企业级项目实战的朋友;</span> </p> <p style="color:#666666;"> <span style="font-size:14px;">3、有MySQL数据库基础的朋友</span> </p> <p style="color:#666666;"> <span style="font-size:14px;"> </span> </p> <p style="color:#666666;"> <span style="font-size:14px;"><img alt="" src="https://img-bss.csdnimg.cn/202009070752197496.png" /><br /> </span> </p> <p style="color:#666666;"> <span style="font-size:14px;"><br /> </span> </p>
©️2020 CSDN 皮肤主题: 编程工作室 设计师:CSDN官方博客 返回首页
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、C币套餐、付费专栏及课程。

余额充值