来源

同事

问题场景

• 同步任务开始的时候设置一个标记，放到redis，结束的时候删除；进程被kiil -9之后，redis缓存没有删除；重启应用之后，一直认为在同步中

思考

• redis缓存设置时间
• 如何优雅的停机

研究优雅停机时的一点思考

研究优雅停机时的一点思考

项目实践

停止脚本 使用kill -15；而不是kill -9

• kill -15 献给应用一段时间善后，这时候Spring容器会正常关闭，执行 @PreDestroy 等类似方法
  原先

stop_application() {
    echo "stop jar"
    if [ -f "$PID_FILE" ]; then
        kill -9 `cat $PID_FILE`
        rm $PID_FILE
    else
        echo "pid file $PID_FILE does not exist, do noting"
    fi
}

修改后

stop_application() {
    echo "stop jar"
    if [ -f "$PID_FILE" ]; then
        kill -15 `cat $PID_FILE`
    else
        echo "pid file $PID_FILE does not exist, do noting"
    fi

  SLEEP=5
  FORCE=1
  if [ ! -z "$PID_FILE" ]; then
    if [ -f "$PID_FILE" ]; then
      while [ $SLEEP -ge 0 ]; do
        kill -0 `cat "$PID_FILE"` >/dev/null 2>&1
        if [ $? -gt 0 ]; then
          rm -f "$PID_FILE" >/dev/null 2>&1
          if [ $? != 0 ]; then
            if [ -w "$PID_FILE" ]; then
              cat /dev/null > "$PID_FILE"
              # If application has stopped don't try and force a stop with an empty PID file
              FORCE=0
            else
              echo "The PID file could not be removed or cleared."
            fi
          fi
          echo "application stopped."
          break
        fi
        if [ $SLEEP -gt 0 ]; then
          sleep 1
        fi
        if [ $SLEEP -eq 0 ]; then
          echo "application did not stop in time."
          if [ $FORCE -eq 0 ]; then
            echo "PID file was not removed."
          fi
          echo "To aid diagnostics a thread dump has been written to standard out."
          kill -3 `cat "$PID_FILE"`
        fi
        SLEEP=`expr $SLEEP - 1 `
      done
    fi
    else
      echo "application normal stopped."
      rm -f "$PID_FILE" >/dev/null 2>&1
  fi

  ##强制杀死进程

  KILL_SLEEP_INTERVAL=5
    if [ $FORCE -eq 1 ]; then
      if [ -z "$PID_FILE" ]; then
        echo "Kill failed: \$PID_FILE not set"
      else
        if [ -f "$PID_FILE" ]; then
          PID=`cat "$PID_FILE"`
          echo "killing application with the PID: $PID force"
          kill -9 $PID
          while [ $KILL_SLEEP_INTERVAL -ge 0 ]; do
              kill -0 `cat "$PID_FILE"` >/dev/null 2>&1
              if [ $? -gt 0 ]; then
                  rm -f "$PID_FILE" >/dev/null 2>&1
                  if [ $? != 0 ]; then
                      if [ -w "$PID_FILE" ]; then
                          cat /dev/null > "$PID_FILE"
                      else
                          echo "The PID file could not be removed."
                      fi
                  fi
                  echo "The application process has been killed."
                  break
              fi
              if [ $KILL_SLEEP_INTERVAL -gt 0 ]; then
                  sleep 1
              fi
              KILL_SLEEP_INTERVAL=`expr $KILL_SLEEP_INTERVAL - 1 `
          done
          if [ $KILL_SLEEP_INTERVAL -lt 0 ]; then
              echo "application has not been killed completely yet. The process might be waiting on some system call or might be UNINTERRUPTIBLE."
          fi
        fi
      fi
    fi
}

JAVA代码

Spring容器关闭的钩子

1. Spring已经自动帮我们加了钩子

org.springframework.context.support.AbstractApplicationContext#registerShutdownHook
    @Override
    public void registerShutdownHook() {
        if (this.shutdownHook == null) {
            // No shutdown hook registered yet.
            this.shutdownHook = new Thread() {
                @Override
                public void run() {
                    synchronized (startupShutdownMonitor) {
                        doClose();
                    }
                }
            };
            Runtime.getRuntime().addShutdownHook(this.shutdownHook);
        }
    }

2. 实现下面这些，Sping容器关闭的时候会自动处理

• @PreDestroy注解
• destory-method方法
• DisposableBean接口

也可以自己加其他的钩子

Runtime.getRuntime().addShutdownHook(this.shutdownHook);

触发了 linux OOM KILLER，还会不会执行钩子？

应该不会，属于强杀

• 这是linux oom kill 的代码

https://github.com/torvalds/linux/blob/master/mm/oom_kill.c

强行翻译：
为了防止OOM受害者耗尽 在它控制下的 从用户空间那保留的 内存，我们应该 在授权它访问保留的内存 之前发送SIGKILL

    /*
     * We should send SIGKILL before granting access to memory reserves
     * in order to prevent the OOM victim from depleting the memory
     * reserves from the user space under its control.
     */
        do_send_sig_info(SIGKILL, SEND_SIG_PRIV, victim, PIDTYPE_TGID);

• 代码中的SIGKILL

https://linux.cn/article-8771-1.html

image.png

扩展

Spring钩子方法和钩子接口的使用

Spring钩子方法和钩子接口的使用详解

参考

Spring优雅关闭之：ShutDownHook

理解和配置 Linux 下的 OOM Killer