环境

<dependency>
            <groupId>com.alibaba.cloud</groupId>
            <artifactId>spring-cloud-starter-alibaba-nacos-config</artifactId>
           <version>2.2.0.RELEASE</version>
</dependency>

现象

修改nacos配置文件后，保存后，一段时间后服务cpu告警。

jstack 日志

查看了一下 jstack 日志，有200个线程都是阻塞在同一个接口上，该接口访问量非常高。

可以看到都阻塞到了 LockedScopedProxyFactoryBean 的 invoke 方法

Thread 15906: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=836 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(int) @bci=83, line=967 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(int) @bci=10, line=1283 (Compiled frame)
 - java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock() @bci=5, line=727 (Compiled frame)
 - org.springframework.cloud.context.scope.GenericScope$LockedScopedProxyFactoryBean.invoke(org.aopalliance.intercept.MethodInvocation) @bci=132, line=494 (Compiled frame)
 - org.springframework.aop.framework.ReflectiveMethodInvocation.proceed() @bci=120, line=186 (Compiled frame)
 - org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed() @bci=1, line=747 (Compiled frame)
 - org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(java.lang.Object, java.lang.reflect.Method, java.lang.Object[], org.springframework.cglib.proxy.MethodProxy) @bci=133, line=689 (Compiled frame)

原因

可以从如下的源码中看到阻塞到了 lock.lock() 方法，这是一个ReadWriteLock 获取readLock并尝试lock。

org.springframework.cloud.context.scope.GenericScope.LockedScopedProxyFactoryBean#invoke

    public Object invoke(MethodInvocation invocation) throws Throwable {
            Method method = invocation.getMethod();
            if (AopUtils.isEqualsMethod(method) || AopUtils.isToStringMethod(method)
                    || AopUtils.isHashCodeMethod(method)
                    || isScopedObjectGetTargetObject(method)) {
                return invocation.proceed();
            }
            Object proxy = getObject();
            ReadWriteLock readWriteLock = this.scope.getLock(this.targetBeanName);
            if (readWriteLock == null) {
                if (logger.isDebugEnabled()) {
                    logger.debug("For bean with name [" + this.targetBeanName
                            + "] there is no read write lock. Will create a new one to avoid NPE");
                }
                readWriteLock = new ReentrantReadWriteLock();
            }
            Lock lock = readWriteLock.readLock();
            lock.lock();
            try {
                if (proxy instanceof Advised) {
                    Advised advised = (Advised) proxy;
                    ReflectionUtils.makeAccessible(method);
                    return ReflectionUtils.invokeMethod(method,
                            advised.getTargetSource().getTarget(),
                            invocation.getArguments());
                }
                return invocation.proceed();
            }
            // see gh-349. Throw the original exception rather than the
            // UndeclaredThrowableException
            catch (UndeclaredThrowableException e) {
                throw e.getUndeclaredThrowable();
            }
            finally {
                lock.unlock();
            }
        }

我们知道 ReadWriteLock 如果都是readLock，那不阻塞，可以直接lock。这个阻塞住了，那应该是被此时有个writeLock导致的

从jstack中找到了如下的writeLock 堆栈，可以看到，获取这个writeLock 是由于nacos更新了配置需要销毁这个bean，所以需要获取writeLock 。

org.springframework.cloud.context.scope.GenericScope.destroy()

Thread 9488: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=836 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.util.concurrent.locks.AbstractQueuedSynchronizer$Node, int) @bci=67, line=870 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int) @bci=17, line=1199 (Compiled frame)
 - java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock() @bci=5, line=943 (Compiled frame)
 - org.springframework.cloud.context.scope.GenericScope.destroy() @bci=69, line=140 (Interpreted frame)
 - org.springframework.cloud.context.scope.refresh.RefreshScope.refreshAll() @bci=1, line=154 (Interpreted frame)
 - org.springframework.cloud.context.refresh.ContextRefresher.refresh() @bci=9, line=86 (Interpreted frame)
 - org.springframework.cloud.endpoint.event.RefreshEventListener.handle(org.springframework.cloud.endpoint.event.RefreshEvent) @bci=44, line=72 (Interpreted frame)
 - org.springframework.cloud.endpoint.event.RefreshEventListener.onApplicationEvent(org.springframework.context.ApplicationEvent) @bci=30, line=61 (Interpreted frame)
 - org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(org.springframework.context.ApplicationListener, org.springframework.context.ApplicationEvent) @bci=2, line=172 (Compiled frame)
 - org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(org.springframework.context.ApplicationListener, org.springframework.context.ApplicationEvent) @bci=34, line=165 (Compiled frame)
 - org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(org.springframework.context.ApplicationEvent, org.springframework.core.ResolvableType) @bci=83, line=139 (Compiled frame)
 - org.springframework.context.support.AbstractApplicationContext.publishEvent(java.lang.Object, org.springframework.core.ResolvableType) @bci=70, line=403 (Compiled frame)
 - org.springframework.context.support.AbstractApplicationContext.publishEvent(org.springframework.context.ApplicationEvent) @bci=3, line=360 (Compiled frame)
 - com.alibaba.cloud.nacos.refresh.NacosContextRefresher$1.innerReceive(java.lang.String, java.lang.String, java.lang.String) @bci=34, line=133 (Interpreted frame)

这个writeLock 为啥被阻塞住了，我们知道 writeLock 和 （readLock，writeLock ）都是互斥的，继续从jstack查看，没有发现其他的writeLock ，但找到了一个 readLock ，该readLock是由于Http请求外部服务导致的阻塞。

读写锁阻塞链路

解决方法

由于该bean中写了动态刷新的属性，并添加了@RefreshScope注解，所以该bean在nacos配置更新时需要销毁重新创建。

将该bean的属性单独抽出一个类，该配置类在nacos配置时刷新，并移除该bean的注解@RefreshScope。

总结

跟随nacos刷新的配置最好单独抽出一个类。