美文网首页
Spark Rest API 使用

Spark Rest API 使用

作者: 晨磊的微博 | 来源:发表于2021-05-13 21:41 被阅读0次

    一、地址和端口问题

    spark官网 说默认端口是 4040,使用spark.ui.port 进行配置spark配置

    我们使用的是CDH,使用 spark.ui.port 查看没有找到该配置

    image.png
    • 搜索 port,找到如下几个端口,看着也就 18088比较像

      image.png
    • 使用如下命令访问

    curl http://xxx.xxx.xxx.xxx:18088/api/v1/applications?limit=2
    

    得到如下结果,看着挺像那么回事,仔细一看,怎么都是历史的任务?

    [ {
      "id" : "application_1618400751843_125338",
      "name" : "cat3-title-vec",
      "attempts" : [ {
        "startTime" : "2021-05-13T13:33:24.965GMT",
        "endTime" : "2021-05-13T13:34:41.551GMT",
        "lastUpdated" : "2021-05-13T13:34:46.298GMT",
        "duration" : 76586,
        "sparkUser" : "supdev",
        "completed" : true,
        "appSparkVersion" : "2.4.0-cdh6.2.0",
        "startTimeEpoch" : 1620912804965,
        "lastUpdatedEpoch" : 1620912886298,
        "endTimeEpoch" : 1620912881551
      } ]
    }, {
      "id" : "application_1618400751843_125318",
      "name" : "re-list-demo",
      "attempts" : [ {
        "startTime" : "2021-05-13T13:26:58.297GMT",
        "endTime" : "2021-05-13T13:28:58.500GMT",
        "lastUpdated" : "2021-05-13T13:28:58.570GMT",
        "duration" : 120203,
        "sparkUser" : "supdev",
        "completed" : true,
        "appSparkVersion" : "2.4.0-cdh6.2.0",
        "startTimeEpoch" : 1620912418297,
        "lastUpdatedEpoch" : 1620912538570,
        "endTimeEpoch" : 1620912538500
      } ]
    } ]
    
    • 加上状态
    curl "http://sj-20-207-14:18088/api/v1/applications?limit=2&status=running"
    

    还是历史

    [ {
      "id" : "application_1618400751843_99465",
      "name" : "Hive on Spark (hiveSessionId = 4f556621-4e68-40d5-a416-8f1fc6a3ea9e)",
      "attempts" : [ {
        "attemptId" : "1",
        "startTime" : "2021-05-08T09:50:00.273GMT",
        "endTime" : "1969-12-31T23:59:59.999GMT",
        "lastUpdated" : "2021-05-08T10:36:31.654GMT",
        "duration" : 0,
        "sparkUser" : "jiahang.liu",
        "completed" : false,
        "appSparkVersion" : "2.4.0-cdh6.2.0",
        "startTimeEpoch" : 1620467400273,
        "lastUpdatedEpoch" : 1620470191654,
        "endTimeEpoch" : -1
      } ]
    }, {
      "id" : "application_1618400751843_93669",
      "name" : "Hive on Spark (hiveSessionId = f31a0dec-abbe-4418-bf7f-98428a7cb44a)",
      "attempts" : [ {
        "attemptId" : "1",
        "startTime" : "2021-05-07T13:30:08.725GMT",
        "endTime" : "1969-12-31T23:59:59.999GMT",
        "lastUpdated" : "2021-05-07T14:58:18.949GMT",
        "duration" : 0,
        "sparkUser" : "supdev",
        "completed" : false,
        "appSparkVersion" : "2.4.0-cdh6.2.0",
        "startTimeEpoch" : 1620394208725,
        "lastUpdatedEpoch" : 1620399498949,
        "endTimeEpoch" : -1
      } ]
    } ]
    

    未完待续

    相关文章

      网友评论

          本文标题:Spark Rest API 使用

          本文链接:https://www.haomeiwen.com/subject/lhoxjltx.html