美文网首页
使用docker-compose安装mlflow,mysql,m

使用docker-compose安装mlflow,mysql,m

作者: 万州客 | 来源:发表于2022-05-28 17:30 被阅读0次

    国内好像就一个CSDN有文档，还要花费下载。于是，从github找了一个仓库，针对国内pip作了一下改造，很快就跑起来了。

    一，找到仓库地址下载

    https://github.com/Toumash/mlflow-docker

    二，小改造

    注意，不能只要docker-compose.yml文件，还需要.env及mlflow目录下的Dockerfile及wait-for-it.sh文件。我们要更改的是Dockerfile文件，增加国内安排pip。docker-compose.yml我也修改了一下，因为version版本太高，我降了降，为了每次都是新测试，我把volume也删除了，但因为可能minio的时区不对的话，dvc push是会出403错误的，我又改了挂载了时区。
    三、重要文件的内容
    内容分别如下：

    .env

    AWS_ACCESS_KEY_ID=admin
AWS_SECRET_ACCESS_KEY=adminadmin
AWS_REGION=us-east-1
AWS_BUCKET_NAME=mlflow
MYSQL_DATABASE=mlflow
MYSQL_USER=mlflow_user
MYSQL_PASSWORD=mlflow_password
MYSQL_ROOT_PASSWORD=toor
MLFLOW_S3_ENDPOINT_URL=http://localhost:9000
MLFLOW_TRACKING_URI=http://localhost:5000

    docker-compose.yml

    version: "3.3"
services:
  s3:
    image:  minio/minio:RELEASE.2021-11-24T23-19-33Z
    restart: unless-stopped
    ports:
      - "9000:9000"
      - "9001:9001"
    environment:
      - MINIO_ROOT_USER=${AWS_ACCESS_KEY_ID}
      - MINIO_ROOT_PASSWORD=${AWS_SECRET_ACCESS_KEY}
      - SET_CONTAINER_TIMEZONE=true
      - CONTAINER_TIMEZONE=Asia/Shanghai
    volumes:
      - /etc/localtime:/etc/localtime:ro
    command: server /data --console-address ":9001" --address ":9000"
    networks:
      - internal
      - public
  db:
    image: mysql/mysql-server:5.7.28
    restart: unless-stopped
    container_name: mlflow_db
    expose:
      - "3306"
    environment:
      - MYSQL_DATABASE=${MYSQL_DATABASE}
      - MYSQL_USER=${MYSQL_USER}
      - MYSQL_PASSWORD=${MYSQL_PASSWORD}
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
    networks:
      - internal
  mlflow:
    container_name: tracker_mlflow
    image: tracker_ml
    restart: unless-stopped
    build:
      context: ./mlflow
      dockerfile: Dockerfile
    ports:
      - "5000:5000"
    environment:
      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
      - AWS_DEFAULT_REGION=${AWS_REGION}
      - MLFLOW_S3_ENDPOINT_URL=http://s3:9000
    networks:
      - public
      - internal
    entrypoint: bash ./wait-for-it.sh db:3306 -t 90 -- mlflow server --backend-store-uri mysql+pymysql://${MYSQL_USER}:${MYSQL_PASSWORD}@db:3306/${MYSQL_DATABASE} --default-artifact-root s3://${AWS_BUCKET_NAME}/ -h 0.0.0.0
  create_s3_buckets:
    image: minio/mc
    depends_on:
      - "s3"
    entrypoint: >
      /bin/sh -c "
      until (/usr/bin/mc alias set minio http://s3:9000 '${AWS_ACCESS_KEY_ID}' '${AWS_SECRET_ACCESS_KEY}') do echo '...waiting...' && sleep 1; done;
      /usr/bin/mc mb minio/mlflow;
      exit 0;
      "
    networks:
      - internal
networks:
  internal:
  public:
    driver: bridge
volumes:
  db_volume:
  minio_volume:

    Dockerfile

    FROM continuumio/miniconda3:latest

RUN pip install mlflow boto3 pymysql -i https://pypi.tuna.tsinghua.edu.cn/simple

ADD . /app
WORKDIR /app

COPY wait-for-it.sh wait-for-it.sh 
RUN chmod +x wait-for-it.sh

    wait-for-it.sh

    #!/usr/bin/env bash
# Use this script to test if a given TCP host/port are available

WAITFORIT_cmdname=${0##*/}

echoerr() { if [[ $WAITFORIT_QUIET -ne 1 ]]; then echo "$@" 1>&2; fi }

usage()
{
    cat << USAGE >&2
Usage:
    $WAITFORIT_cmdname host:port [-s] [-t timeout] [-- command args]
    -h HOST | --host=HOST       Host or IP under test
    -p PORT | --port=PORT       TCP port under test
                                Alternatively, you specify the host and port as host:port
    -s | --strict               Only execute subcommand if the test succeeds
    -q | --quiet                Don't output any status messages
    -t TIMEOUT | --timeout=TIMEOUT
                                Timeout in seconds, zero for no timeout
    -- COMMAND ARGS             Execute command with args after the test finishes
USAGE
    exit 1
}

wait_for()
{
    if [[ $WAITFORIT_TIMEOUT -gt 0 ]]; then
        echoerr "$WAITFORIT_cmdname: waiting $WAITFORIT_TIMEOUT seconds for $WAITFORIT_HOST:$WAITFORIT_PORT"
    else
        echoerr "$WAITFORIT_cmdname: waiting for $WAITFORIT_HOST:$WAITFORIT_PORT without a timeout"
    fi
    WAITFORIT_start_ts=$(date +%s)
    while :
    do
        if [[ $WAITFORIT_ISBUSY -eq 1 ]]; then
            nc -z $WAITFORIT_HOST $WAITFORIT_PORT
            WAITFORIT_result=$?
        else
            (echo -n > /dev/tcp/$WAITFORIT_HOST/$WAITFORIT_PORT) >/dev/null 2>&1
            WAITFORIT_result=$?
        fi
        if [[ $WAITFORIT_result -eq 0 ]]; then
            WAITFORIT_end_ts=$(date +%s)
            echoerr "$WAITFORIT_cmdname: $WAITFORIT_HOST:$WAITFORIT_PORT is available after $((WAITFORIT_end_ts - WAITFORIT_start_ts)) seconds"
            break
        fi
        sleep 1
    done
    return $WAITFORIT_result
}

wait_for_wrapper()
{
    # In order to support SIGINT during timeout: http://unix.stackexchange.com/a/57692
    if [[ $WAITFORIT_QUIET -eq 1 ]]; then
        timeout $WAITFORIT_BUSYTIMEFLAG $WAITFORIT_TIMEOUT $0 --quiet --child --host=$WAITFORIT_HOST --port=$WAITFORIT_PORT --timeout=$WAITFORIT_TIMEOUT &
    else
        timeout $WAITFORIT_BUSYTIMEFLAG $WAITFORIT_TIMEOUT $0 --child --host=$WAITFORIT_HOST --port=$WAITFORIT_PORT --timeout=$WAITFORIT_TIMEOUT &
    fi
    WAITFORIT_PID=$!
    trap "kill -INT -$WAITFORIT_PID" INT
    wait $WAITFORIT_PID
    WAITFORIT_RESULT=$?
    if [[ $WAITFORIT_RESULT -ne 0 ]]; then
        echoerr "$WAITFORIT_cmdname: timeout occurred after waiting $WAITFORIT_TIMEOUT seconds for $WAITFORIT_HOST:$WAITFORIT_PORT"
    fi
    return $WAITFORIT_RESULT
}

# process arguments
while [[ $# -gt 0 ]]
do
    case "$1" in
        *:* )
        WAITFORIT_hostport=(${1//:/ })
        WAITFORIT_HOST=${WAITFORIT_hostport[0]}
        WAITFORIT_PORT=${WAITFORIT_hostport[1]}
        shift 1
        ;;
        --child)
        WAITFORIT_CHILD=1
        shift 1
        ;;
        -q | --quiet)
        WAITFORIT_QUIET=1
        shift 1
        ;;
        -s | --strict)
        WAITFORIT_STRICT=1
        shift 1
        ;;
        -h)
        WAITFORIT_HOST="$2"
        if [[ $WAITFORIT_HOST == "" ]]; then break; fi
        shift 2
        ;;
        --host=*)
        WAITFORIT_HOST="${1#*=}"
        shift 1
        ;;
        -p)
        WAITFORIT_PORT="$2"
        if [[ $WAITFORIT_PORT == "" ]]; then break; fi
        shift 2
        ;;
        --port=*)
        WAITFORIT_PORT="${1#*=}"
        shift 1
        ;;
        -t)
        WAITFORIT_TIMEOUT="$2"
        if [[ $WAITFORIT_TIMEOUT == "" ]]; then break; fi
        shift 2
        ;;
        --timeout=*)
        WAITFORIT_TIMEOUT="${1#*=}"
        shift 1
        ;;
        --)
        shift
        WAITFORIT_CLI=("$@")
        break
        ;;
        --help)
        usage
        ;;
        *)
        echoerr "Unknown argument: $1"
        usage
        ;;
    esac
done

if [[ "$WAITFORIT_HOST" == "" || "$WAITFORIT_PORT" == "" ]]; then
    echoerr "Error: you need to provide a host and port to test."
    usage
fi

WAITFORIT_TIMEOUT=${WAITFORIT_TIMEOUT:-15}
WAITFORIT_STRICT=${WAITFORIT_STRICT:-0}
WAITFORIT_CHILD=${WAITFORIT_CHILD:-0}
WAITFORIT_QUIET=${WAITFORIT_QUIET:-0}

# Check to see if timeout is from busybox?
WAITFORIT_TIMEOUT_PATH=$(type -p timeout)
WAITFORIT_TIMEOUT_PATH=$(realpath $WAITFORIT_TIMEOUT_PATH 2>/dev/null || readlink -f $WAITFORIT_TIMEOUT_PATH)

WAITFORIT_BUSYTIMEFLAG=""
if [[ $WAITFORIT_TIMEOUT_PATH =~ "busybox" ]]; then
    WAITFORIT_ISBUSY=1
    # Check if busybox timeout uses -t flag
    # (recent Alpine versions don't support -t anymore)
    if timeout &>/dev/stdout | grep -q -e '-t '; then
        WAITFORIT_BUSYTIMEFLAG="-t"
    fi
else
    WAITFORIT_ISBUSY=0
fi

if [[ $WAITFORIT_CHILD -gt 0 ]]; then
    wait_for
    WAITFORIT_RESULT=$?
    exit $WAITFORIT_RESULT
else
    if [[ $WAITFORIT_TIMEOUT -gt 0 ]]; then
        wait_for_wrapper
        WAITFORIT_RESULT=$?
    else
        wait_for
        WAITFORIT_RESULT=$?
    fi
fi

if [[ $WAITFORIT_CLI != "" ]]; then
    if [[ $WAITFORIT_RESULT -ne 0 && $WAITFORIT_STRICT -eq 1 ]]; then
        echoerr "$WAITFORIT_cmdname: strict mode, refusing to execute subprocess"
        exit $WAITFORIT_RESULT
    fi
    exec "${WAITFORIT_CLI[@]}"
else
    exit $WAITFORIT_RESULT
fi

    四，运行及效果
    docker-compose up

    [root@127 mlflow]# docker-compose up
Recreating mlflow_db    ... done
Starting tracker_mlflow ... done
Recreating mlflow_s3_1  ... done
Recreating mlflow_create_s3_buckets_1 ... done
Attaching to tracker_mlflow, mlflow_db, mlflow_s3_1, mlflow_create_s3_buckets_1
db_1                 | [Entrypoint] MySQL Docker Image 5.7.28-1.1.13
tracker_mlflow       | wait-for-it.sh: waiting 90 seconds for db:3306
db_1                 | [Entrypoint] Starting MySQL 5.7.28-1.1.13
create_s3_buckets_1  | mc: <ERROR> Unable to initialize new alias from the provided credentials. Get "http://s3:9000/probe-bucket-sign-efnl02v5rjxb/?location=": dial tcp 172.26.0.4:9000: connect: connection refused.
create_s3_buckets_1  | ...waiting...
tracker_mlflow       | wait-for-it.sh: db:3306 is available after 1 seconds
s3_1                 | 
s3_1                 |  You are running an older version of MinIO released 6 months ago 
s3_1                 |  Update: Run `mc admin update` 
s3_1                 | 
s3_1                 | 
s3_1                 | API: http://172.26.0.4:9000  http://172.27.0.3:9000  http://127.0.0.1:9000 
s3_1                 | 
s3_1                 | Console: http://172.26.0.4:9001 http://172.27.0.3:9001 http://127.0.0.1:9001 
s3_1                 | 
s3_1                 | Documentation: https://docs.min.io
create_s3_buckets_1  | Added `minio` successfully.
create_s3_buckets_1  | Bucket created successfully `minio/mlflow`.
mlflow_create_s3_buckets_1 exited with code 0
tracker_mlflow       | [2022-05-27 15:30:21 +0000] [26] [INFO] Starting gunicorn 20.1.0
tracker_mlflow       | [2022-05-27 15:30:21 +0000] [26] [INFO] Listening at: http://0.0.0.0:5000 (26)
tracker_mlflow       | [2022-05-27 15:30:21 +0000] [26] [INFO] Using worker: sync
tracker_mlflow       | [2022-05-27 15:30:21 +0000] [27] [INFO] Booting worker with pid: 27
tracker_mlflow       | [2022-05-27 15:30:21 +0000] [28] [INFO] Booting worker with pid: 28
tracker_mlflow       | [2022-05-27 15:30:21 +0000] [29] [INFO] Booting worker with pid: 29
tracker_mlflow       | [2022-05-27 15:30:21 +0000] [30] [INFO] Booting worker with pid: 30
    2022-05-28 17_28_23-MessageCenterUI.png 2022-05-28 17_28_07-MessageCenterUI.png

    相关文章

      网友评论

          本文标题：使用docker-compose安装mlflow,mysql,m

          本文链接：https://www.haomeiwen.com/subject/hkaiprtx.html

          延伸阅读

          深度阅读

          • 您也可以注册成为美文阅读网的作者，发表您的原创作品、分享您的心情！

          栏目导航

          热点阅读

          关于我们|服务条款|联系我们|使用docker-compose安装mlflow,mysql,m|投稿指南|网站地图|RSS订阅|排版工具|手机版

          提供经典美文摘抄,优美散文欣赏,现代诗歌精选,短篇小说,心情随笔,表白情书范文,故事会在线阅读欣赏

          Copyright © 2014-2023 Haomeiwen.com All Rights Reserved. 好美文阅读网 版权所有

          备案信息：桂公网安备 45052102000051号 · 桂ICP备13007215号-3

          本站所收录作品、热点评论等信息部分来源互联网，目的只是为了系统归纳学习和传递资讯

          所有作品版权归原创作者所有，与本站立场无关，如不慎侵犯了你的权益，请联系我们告知，我们将做删除处理！