AWS Lambda笔记-自动部署-5

作者: lazy_zhu | 来源:发表于2020-06-06 10:15 被阅读0次

本章使用CloudFormation来实现AWS Lambda函数的自动部署。主要包括：

编译打包

创建S3存储，打包程序文件上传

创建函数

工程目录结构

准备3个java文件

代码同 Serverless-HelloWorld-3可以直接copy即可。

2. 准备Gradle的2个文件

setting.build 文件：gradle指定子目录项目

include 'lambda-test'

build.gradle 文件

task wrapper(type: Wrapper) {
    gradleVersion = '3.0'
}

// 全工程配置，子工程也生效
allprojects {
    // 产品ID
    group 'com.serverlessbook'
    // 项目的版本号
    version '1.0'
    // 添加Gradle java插件
    apply plugin: 'java'
    // java 1.8 
    sourceCompatibility = 1.8
}

// 添加 maven库，jCenter，jitpack
allprojects {
    repositories {
        mavenCentral()
        jcenter()
        maven {
            url "https://jitpack.io"
        }
    }
}

buildscript {
    repositories {
        mavenCentral()
        jcenter()
        maven {
            url "https://jitpack.io"
        }
    }

    //添加各种依赖
    dependencies {
        classpath "com.github.jengelman.gradle.plugins:shadow:1.2.3"
        //classmethod公司的AWS插件，在构建脚本里调用AWS的API
        classpath "jp.classmethod.aws:gradle-aws-plugin:0.37"
    }
}

//制定插件应用于所有子项目。所有的子项目部署到区中
//${aws.region} 取的是 ~/.aws/config 中的region
allprojects {
    apply plugin: "jp.classmethod.aws"
    aws {
        region = "${aws.region}"
    }
}

// 定义Bucket的名字，传递给cloudFormation.template 文件
def bucketRegionName = "${aws.region}"
//定义bucket的名词变量，aws要求所有的bucket名称唯一性
def deploymentBucketName = "serverless-book-${aws.region}"
//定义一个时间变量
def deploymentTime = new java.text.SimpleDateFormat("yyyyMMddHHmmss").format(new Date());

//创建一个bucket，如果存在就不创建
//可以运行 ./gradlew createDeploymentBucket 测试下, 运行 aws s3 ls 查看创建的bucket
allprojects {
    apply plugin: "jp.classmethod.aws.s3"
    task createDeploymentBucket(type: jp.classmethod.aws.gradle.s3.CreateBucketTask) {
        bucketName deploymentBucketName
        region bucketRegionName
        ifNotExists true
    }
}

//lambda-开头的子项目，编译lambda工程
configure(subprojects.findAll { it.name.startsWith("lambda-") }) {
    dependencies {
        compile 'com.amazonaws:aws-lambda-java-core:1.1.0'
        compile 'com.fasterxml.jackson.core:jackson-databind:2.6.+'
    }
    //添加插件shadow，编译成fat-jar
    apply plugin: "com.github.johnrengelman.shadow"
    build.finalizedBy shadowJar

    //获取 shadowJar 归档的路径
    def producedJarFilePath = it.tasks.shadowJar.archivePath
    //根据编译的时间命名jar，每次都是一个新jar，要拿过来抱着后续lambda调用的都是最新的jar
    //it 代表闭包的参数,version在最上面已定义为1.0
    def s3Key = "artifacts/${it.name}/${it.version}/${deploymentTime}.jar"

    //上传jar到s3，有两个依赖，build ，createDeploymentBucket 两个task
    //测试：./gradlew uploadArtifactsToS3
    //查看：aws s3 ls s3://serverless-book-us-east-2/artifacts/lambda-test/1.0/
    task uploadArtifactsToS3(type: jp.classmethod.aws.gradle.s3.AmazonS3FileUploadTask,
            dependsOn: [build, createDeploymentBucket]) {
        bucketName deploymentBucketName
        file producedJarFilePath
        key s3Key
    }
}

// 添加cloudformation插件
apply plugin: "jp.classmethod.aws.cloudformation"

cloudFormation {
    //创建IAM资源，由于我们需要创建IAM角色和IAM策略，所以同时必须创建IAM资源，否则cloudformation无法执行。
    capabilityIam true
    //cloudformation 模版文件的位置
    templateFile project.file('cloudformation.template')
    //上传模版文件的bucket名称使用定义的名称：deploymentBucketName = "serverless-book1-${aws.region}" 
    templateBucket deploymentBucketName
    //S3桶名称前缀，可以任意指定,
    templateKeyPrefix "cfn-templates"
    //cloudformation伐的名称
    stackName "serverlessbook"
    //gradle 和 cloudformation模版传参赋值，cloudformation的三个参数
    conventionMapping.stackParams = {
        return [
                DeploymentBucket: deploymentBucketName,
                ProjectVersion  : project.version,
                DeploymentTime  : deploymentTime
        ]
    }
}

//awsCfnMigrateStackAndWaitCompleted：创建或更新CloudFormation伐兵等待创建完成
//awsCfnMigrateStack：同上唯一区别就是不等待操作完成。
//awsCfnUploadTemplate：将本地cloudformation模版上传到S3云存储桶。
awsCfnMigrateStack.dependsOn awsCfnUploadTemplate

//构建和上传程序文件，部署cloudformation模版
task deploy {
    configure(subprojects.findAll { it.name.startsWith("lambda-") }) {
        //依赖 uploadArtifactsToS3 任务（编译打包，创建bucket，上传程序到bucket）
        dependsOn it.uploadArtifactsToS3
    }
    //部署的最后一个操作
    finalizedBy awsCfnMigrateStackAndWaitCompleted
}

cloudformation.template 文件说明

定义IAM角色，让Lambda函数执行期间的身份，例如函数访问S3资源，是否可读写等权限。

{
    "Resources": {
    "LambdaExecutionRole": {
      "Type": "AWS::IAM::Role",
      "Properties": {
        "Path": "/",
        "AssumeRolePolicyDocument": { 
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Principal": {
                "Service": [
                  "lambda.amazonaws.com"
                ]
              },
              "Action": [
                "sts:AssumeRole"
              ]
            }
          ]
        },
        "ManagedPolicyArns": [
          "arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole"
        ]
      }
    }
}

属性说明：
AssumeRolePolicyDocument：与此角色关联的信任策略。信任策略定义哪些实体可以代入该角色。只能将一个信任策略与一个角色关联。这里表示关联的服务由lambda函数来执行。
ManagedPolicyArns：附加到用户的 IAM 托管策略的 Amazon 资源名称 (ARN) 的列表。其中内建IAM策略 AWSLambdaVPCAccessExecutionRole，赋予lambda函数在vpc环境中运行以及在cloudWatch中记录日志的权限。几乎所有的函数都需要这两个权限。
在Gradle里deploy的task运行完成后，在IAM控制台可以看到类似 serverlessbook-LambdaExecutionRole-WMME8CT809HQ的角色。角色里面有策略“LambdaCustomPolicy”，策略里面有S3，S3里面有个ListBuckets操作权限。

创建自定义IAM策略，并给上面创建的IAM角色（LambdaExecutionRole）

"LambdaCustomPolicy": {
  "Type": "AWS::IAM::Policy",
  "Properties": {
    "PolicyName": "LambdaCustomPolicy",
    "PolicyDocument": {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "s3:ListBuckets"
          ],
          "Resource": "*"
        }
      ]
    },
    "Roles": [
      {
        "Ref": "LambdaExecutionRole"
      }
    ]
  }
}

该策略添加一个S3 listBuckets 权限，他使 Lambda函数有获取S3bucket list 的权限。
Roles：关联内建角色时使用{“Ref”:"RESOURCE_NAME"}

cloudformation 参数

"Parameters": {
    "DeploymentBucket": {
      "Type": "String",
      "Description": "S3 bucket name where built artifacts are deployed"
    },
    "ProjectVersion": {
      "Type": "String",
      "Description": "Project Version"
    },
    "DeploymentTime": {
      "Type": "String",
      "Description": "It is a timestamp value which shows the deployment time. Used to rotate sources."
    }
}

定义了三个参数，DeploymentBucket，ProjectVersion，DeploymentTime，这三个参数会在哪儿赋值呢？
这三个参数的值是从 build.gradle传递的参数，对应文件如下：

//gradle 和 cloudformation模版传参赋值，cloudformation的三个参数
    conventionMapping.stackParams = {
        return [
                DeploymentBucket: deploymentBucketName,
                ProjectVersion  : project.version,
                DeploymentTime  : deploymentTime
        ]
    }

创建Lambda函数

"TestLambda": {
  "Type": "AWS::Lambda::Function",
  "Properties": {
    "Handler": "com.serverlessbook.lambda.test.Handler",
    "Runtime": "java8",
    "Timeout": "300",
    "MemorySize": "128",
    "Description": "Test lambda",
    "Role": {
      "Fn::GetAtt": [
        "LambdaExecutionRole",
        "Arn"
      ]
    },
    "Code": {
      "S3Bucket": {
        "Ref": "DeploymentBucket"
      },
      "S3Key": {
        "Fn::Sub": "artifacts/lambda-test/${ProjectVersion}/${DeploymentTime}.jar"
      }
    }
  }
}

Handler：函数的入口方法，注意必须基础LambdaHandle<I,O>
Role：运行Lambda函数的IAM角色，这边引用了LambdaExecutionRole角色，并返回对应的ARN，类似在执行lambda函数创建aws lambda create-function --role arn:aws:iam::083845341320:role/lambada-execution-role
Code：Jar包在S3中和对应的路径(S3Key)

完整的 cloudformation.template 文件

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Parameters": {
        "DeploymentBucket": {
          "Type": "String",
          "Description": "S3 bucket name where built artifacts are deployed"
        },
        "ProjectVersion": {
          "Type": "String",
          "Description": "Project Version"
        },
        "DeploymentTime": {
          "Type": "String",
          "Description": "It is a timestamp value which shows the deployment time. Used to rotate sources."
        }
    },
    "Resources": {
        "LambdaExecutionRole": {
          "Type": "AWS::IAM::Role",
          "Properties": {
            "Path": "/",
            "AssumeRolePolicyDocument": {
              "Version": "2012-10-17",
              "Statement": [
                {
                  "Effect": "Allow",
                  "Principal": {
                    "Service": [
                      "lambda.amazonaws.com"
                    ]
                  },
                  "Action": [
                    "sts:AssumeRole"
                  ]
                }
              ]
            },
            "ManagedPolicyArns": [
              "arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole"
            ]
          }
        },
        "LambdaCustomPolicy": {
          "Type": "AWS::IAM::Policy",
          "Properties": {
            "PolicyName": "LambdaCustomPolicy",
            "PolicyDocument": {
              "Version": "2012-10-17",
              "Statement": [
                {
                  "Effect": "Allow",
                  "Action": [
                    "s3:ListBuckets"
                  ],
                  "Resource": "*"
                }
              ]
            },
            "Roles": [
              {
                "Ref": "LambdaExecutionRole"
              }
            ]
          }
        },
        "TestLambda": {
          "Type": "AWS::Lambda::Function",
          "Properties": {
            "Handler": "com.serverlessbook.lambda.test.Handler",
            "Runtime": "java8",
            "Timeout": "300",
            "MemorySize": "1024",
            "Description": "Test lambda",
            "Role": {
              "Fn::GetAtt": [
                "LambdaExecutionRole",
                "Arn"
              ]
            },
            "Code": {
              "S3Bucket": {
                "Ref": "DeploymentBucket"
              },
              "S3Key": {
                "Fn::Sub": "artifacts/lambda-test/${ProjectVersion}/${DeploymentTime}.jar"
              }
            }
          }
        }
    }
}

上面的代码转换成三者的关系，通过CloudFormation设计器可以很明显看出。

Role，Policy，Fuction 关系图

一键部署Lambda函数（完成所有工作最爽的一步）

./gradlew deploy

查看整个部署情况:

aws cloudformation describe-stack-resources \
--stack-name serverlessbook \
--region us-east-2

输出结果：

{
    "StackResources": [
        {
            "StackName": "serverlessbook",
            "StackId": "arn:aws:cloudformation:us-east-2:083845954160:stack/serverlessbook/79bc0a10-a573-11ea-8763-0a3e4885fce2",
            "LogicalResourceId": "LambdaCustomPolicy",
            "PhysicalResourceId": "serve-Lamb-17L0RKTV9AKCD",
            "ResourceType": "AWS::IAM::Policy",
            "Timestamp": "2020-06-03T08:24:47.772Z",
            "ResourceStatus": "CREATE_COMPLETE",
            "DriftInformation": {
                "StackResourceDriftStatus": "NOT_CHECKED"
            }
        },
        {
            "StackName": "serverlessbook",
            "StackId": "arn:aws:cloudformation:us-east-2:083845954160:stack/serverlessbook/79bc0a10-a573-11ea-8763-0a3e4885fce2",
            "LogicalResourceId": "LambdaExecutionRole",
            "PhysicalResourceId": "serverlessbook-LambdaExecutionRole-11Y8OL55973LC",
            "ResourceType": "AWS::IAM::Role",
            "Timestamp": "2020-06-03T08:24:31.012Z",
            "ResourceStatus": "CREATE_COMPLETE",
            "DriftInformation": {
                "StackResourceDriftStatus": "NOT_CHECKED"
            }
        },
        {
            "StackName": "serverlessbook",
            "StackId": "arn:aws:cloudformation:us-east-2:083845954160:stack/serverlessbook/79bc0a10-a573-11ea-8763-0a3e4885fce2",
            "LogicalResourceId": "TestLambda",
            "PhysicalResourceId": "serverlessbook-TestLambda-OYCXMPOYMQXE",
            "ResourceType": "AWS::Lambda::Function",
            "Timestamp": "2020-06-03T08:24:33.630Z",
            "ResourceStatus": "CREATE_COMPLETE",
            "DriftInformation": {
                "StackResourceDriftStatus": "NOT_CHECKED"
            }
        }
    ]
}

命令行调用Lambda函数

aws lambda invoke \
--invocation-type RequestResponse \
--region us-east-2 \
--function-name serverlessbook-TestLambda-OYCXMPOYMQXE \
--payload '{"value":"test"}' \
--log-type Tail  \
/tmp/test.txt

查看输出结果：

tail -f /tmp/test.txt

异常一：创建Bucket 名词有冲突

//当 region "us-east-21" 写错了，会提示：Status Code: 409 ，主要是区的不存在或bucket名词冲突，改下名词即可。
allprojects {
    apply plugin: "jp.classmethod.aws.s3"
    task createDeploymentBucket(type: jp.classmethod.aws.gradle.s3.CreateBucketTask) {
        bucketName "serverless-book-us-east-2"
        region "us-east-2"
        ifNotExists true
    }
}

:createDeploymentBucket FAILED
FAILURE: Build failed with an exception.

What went wrong:
Execution failed for task ':createDeploymentBucket'.
A conflicting conditional operation is currently in progress against this resource. Please try again. (Service: Amazon S3; Status Code: 409; Error Code: OperationAborted; Request ID: 57D7C696E1B13245; S3 Extended Request ID: eWOVTiGFTjeuXdd/ZPVdJDNId044kU65jSQjaMszpIRrAifFFe81Y7zjXGPiY76KuuyieLUNeJ0=)

Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
BUILD FAILED
Total time: 2.51 secs
解决：修改“bucketName”的名词，即可解决冲突问题。

异常二： Region 指定的区域不在 aws-java-sdk-s3-xxx.jar 包中。

What went wrong:
Execution failed for task ':createDeploymentBucket'.
Cannot create enum from us-east-1 value!

Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
BUILD FAILED
Total time: 7.015 secs
我们来看下两个不同版本之间com.amazonaws.services.s3.model.Region的区别，通过命令可以找到本地的jar：

cd ~/.gradle/caches/modules-2/files-2.1/com.amazonaws/aws-java-sdk-s3

aws-java-sdk-s3-1.11.27.jar

aws-java-sdk-s3-1.11.538.jar

那么，这些jar到底是上面Gradel的哪些配置依赖呢？
是 classpath "jp.classmethod.aws:gradle-aws-plugin:0.37" ，版本不一样依赖的aws-java-sdk-s3有区别。

异常二： Region 指定的区域为默认区域us-east-1。

What went wrong:
Execution failed for task ':lambda-test:createDeploymentBucket'.
The specified location-constraint is not valid (Service: Amazon S3; Status Code: 400; Error Code: >InvalidLocationConstraint; Request ID: A661AAB282AA92BC; S3 Extended Request ID:4GpSQ1bksHt8TcXge2WS/vAh+6weh0gtYYCcM5Nhlf0tHL/MGAPhjXOvwLzDcU7lGI31wEjH7mg=)

Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

注释region即可

allprojects {
    apply plugin: "jp.classmethod.aws.s3"
    task createDeploymentBucket(type: jp.classmethod.aws.gradle.s3.CreateBucketTask) {
        bucketName deploymentBucketName
        //region "us-east-1" //默认us-east-1 不需要指定
        ifNotExists true
    }
}

AWS Lambda笔记-自动部署-5

工程目录结构

准备3个java文件

2. 准备Gradle的2个文件

cloudformation.template 文件说明

完整的 cloudformation.template 文件

一键部署Lambda函数（完成所有工作最爽的一步）

查看整个部署情况:

命令行调用Lambda函数

异常一：创建Bucket 名词有冲突

异常二： Region 指定的区域不在 aws-java-sdk-s3-xxx.jar 包中。

异常二： Region 指定的区域为默认区域us-east-1。

相关文章

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读

AWS Serverless 学习

AWS Lambda笔记-自动部署-5

工程目录结构

准备3个java文件

2. 准备Gradle的2个文件

cloudformation.template 文件说明

完整的 cloudformation.template 文件

一键部署Lambda函数（完成所有工作最爽的一步）

查看整个部署情况:

命令行调用Lambda函数

异常一： 创建Bucket 名词有冲突

异常二： Region 指定的区域不在 aws-java-sdk-s3-xxx.jar 包中。

异常二： Region 指定的区域为默认区域us-east-1。

相关文章

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读

AWS Serverless 学习

异常一：创建Bucket 名词有冲突