美文网首页
EBAS观测数据下载

EBAS观测数据下载

作者: 别有路 | 来源:发表于2022-10-02 09:36 被阅读0次

记录一下 下载EBAS-Thredds数据的流程, 以全站点的ozone为例


1. 下载ebas-thredds-harvest的脚本

https://git.nilu.no/ebas/ebas-thredds-harvest

2. 获取所有的文件名列表 => EBAS.filelist.txt

这里复制过来, 并掐头去尾, 保持每一行格式固定,都是文件名

3. 获取含ozone的文件名列表 => EBAS.filelist.ozone.txt

# in wsl
grep ozone EBAS.filelist.txt > EBAS.filelist.ozone.txt

4. 获取站点列表 => EBAS.sites.txt

# in wsl
awk -F. '{print $1}' EBAS.filelist.ozone.txt | sort | uniq > EBAS.sites.txt

5. python配置必要环境

netCDF4包, thredds包等等, 按需来,

6. 执行脚本获取所有站点对应的urls => urls.txt

执行下面的powershell脚本, 循环调用python脚本, 获取http下载链接, 并保存至urls.txt

这一步比较慢

# get_ozone_urls.ps1

$sites = "AM0001R", "AR0001R", "AR0002G", "AR0004R", "AR0005R", "AT0002R", "AT0003R", "AT0004R", "AT0005R", "AT0030R", "AT0032R", "AT0033R", "AT0034G", "AT0037R", "AT0038R", "AT0040R", "AT0041R", "AT0042R", "AT0043R", "AT0044R", "AT0045R", "AT0046R", "AT0047R", "AT0048R", "AT0049R", "AT0050R", "AU0002G", "BB0001R", "BE0001R", "BE0032R", "BE0035R", "BG0053R", "BM0001R", "BO0001R", "CA0002R", "CA0003R", "CA0006R", "CA0010R", "CA0011R", "CA0012R", "CA0023R", "CA0103R", "CA0420G", "CH0001G", "CH0002R", "CH0003R", "CH0004R", "CH0005R", "CH0010U", "CH0031R", "CH0053R", "CL0001R", "CV0001G", "CY0002R", "CZ0001R", "CZ0003R", "CZ0005R", "DE0001R", "DE0002R", "DE0003R", "DE0004R", "DE0005R", "DE0006R", "DE0007R", "DE0008R", "DE0009R", "DE0011R", "DE0012R", "DE0013R", "DE0014R", "DE0017R", "DE0018R", "DE0026R", "DE0031R", "DE0035R", "DE0038R", "DE0039R", "DE0042R", "DE0043G", "DE0045R", "DE0046R", "DE0047R", "DE0054R", "DE0057G", "DE0060G", "DK0005R", "DK0009R", "DK0010G", "DK0012R", "DK0025G", "DK0031R", "DK0032R", "DK0041R", "EE0009R", "EE0011R", "EG0001U", "EG0002U", "EG0004R", "ES0001R", "ES0002R", "ES0003R", "ES0004R", "ES0005R", "ES0006R", "ES0007R", "ES0008R", "ES0009R", "ES0010R", "ES0011R", "ES0012R", "ES0013R", "ES0014R", "ES0015R", "ES0016R", "ES0017R", "ES0018G", "FI0004R", "FI0009R", "FI0017R", "FI0018R", "FI0022R", "FI0037R", "FI0096G", "FR0008R", "FR0009R", "FR0010R", "FR0011R", "FR0012R", "FR0013R", "FR0014R", "FR0015R", "FR0016R", "FR0017R", "FR0018R", "FR0019R", "FR0020R", "FR0023R", "FR0024R", "FR0025R", "FR0026R", "FR0028R", "FR0030R", "FR0031R", "FR0032R", "FR0034G", "GB0002R", "GB0006R", "GB0013R", "GB0014R", "GB0015R", "GB0031R", "GB0033R", "GB0035R", "GB0036R", "GB0037R", "GB0038R", "GB0039R", "GB0041R", "GB0043R", "GB0044R", "GB0045R", "GB0048R", "GB0049R", "GB0050R", "GB0051R", "GB0052R", "GB0053R", "GB0059G", "GB1055R", "GR0001R", "GR0002R", "GR0003R", "GR0101R", "HU0002R", "HU0003R", "HU0017R", "ID1013R", "IE0001R", "IE0031R", "IS0091R", "IT0001R", "IT0004R", "IT0009R", "IT0014R", "IT0015U", "IT0016R", "IT0018R", "IT0019R", "JP0002G", "JP0003U", "JP1020R", "JP1028G", "JP1029R", "KE0001G", "KG0002U", "KR0100R", "KR0101R", "LT0015R", "LV0010R", "LV0016R", "LV2000U", "MK0007R", "MT0001R", "NL0007R", "NL0009R", "NL0010R", "NL0011R", "NL0091R", "NL0644R", "NO0001R", "NO0002R", "NO0015R", "NO0030R", "NO0039R", "NO0041R", "NO0042G", "NO0043R", "NO0044R", "NO0045R", "NO0047R", "NO0048R", "NO0052R", "NO0055R", "NO0056R", "NO0058G", "NO0059G", "NO0488R", "NO0489R", "NO0492R", "NO0762R", "NO0977R", "NO1007R", "NO1011R", "NO1083R", "NO1200R", "NP0001G", "NZ0001R", "NZ0002R", "NZ0003G", "PE0001R", "PL0002R", "PL0003R", "PL0004R", "PL0005R", "PT0004R", "RO0003R", "RO0008R", "RS0005R", "RU0001R", "RU0013R", "RU0016R", "RU0018R", "RU0100R", "SE0002R", "SE0003R", "SE0005R", "SE0011R", "SE0012R", "SE0013R", "SE0014R", "SE0018R", "SE0019R", "SE0020R", "SE0022R", "SE0032R", "SE0033R", "SE0034R", "SE0035R", "SE0039R", "SE0094R", "SI0008R", "SI0031R", "SI0032R", "SI0033R", "SK0002R", "SK0004R", "SK0005R", "SK0006R", "SK0007R", "US0008R", "US0039G", "US0142R", "US0204R", "US0602R", "US0901R", "US1200R", "US6001R", "US6004G", "US6005G", "US6006R", "VN0001R", "ZA0001G"

foreach ($s in $sites){
    echo $s
    python ebas-thredds-harvest/get_tds_data.py -s "ozone" -c "$s" -p "http" >> urls.txt
}

7. 下载这些链接

wget, curl之类的去下载urls.txt里的http链接即可



后续的处理逻辑记在EBAS观测数据处理

相关文章

  • 【泰阁志-数据分析】作业4:区间估计

    观测到的数据,可以在一定程度上认为接近总体均值 观测多个数据,取样本均值,比观测一个数据更接近总体均值,观测数据越...

  • 观测数据

    数据工具 选择数据工具的核心逻辑:业务问题(需求)⇔数据工具(套路)根据业务问题的核心需求,匹配适当的分析套路,选...

  • 观测数据

    World Ozone and Ultraviolet Radiation Data Centrehttps://...

  • Stata系列-如何处理极端数据

    什么是“极端数据”呢?如果样本数据中的少数观测值离大多数观测值很远,这些数据称为“极端观测值”或“高影响力数据”。...

  • OpenTelemetry 简析

    OpenTelemetry 是 CNCF 的一个可观测性项目,旨在提供可观测性领域的标准化方案,解决观测数据的数据...

  • 网络单频RTK用处

    RTK技术的关键在于数据处理技术和数据传输技术,RTK定位时要求基准站接收机实时地把观测数据(伪距观测值,相位观测...

  • SAS数据的导入(3)

    每个观测分布在多行或者一行有多个观测数据的读取 1、每个观测分布在多行的数据 上期提到列指针用 @40 或者 @'...

  • IGS数据下载地址

    IGS站观测数据和相关产品下载地址,主要采用ftp方式,其中个别IGS站,可能仅在个别机构提供,例如这个ulab站...

  • 计量经济学

    第一章 三个数据 截面数据: 在同一时点上,对不同观测单位观测得到的多个数据。 时间序列数据: 对同一个观测单位,...

  • 表达矩阵三元组(长数据)与宽数据转换

    长数据与宽数据的转换: 1、长数据:观测和测量值独立一列;常作为分析时数据的读取格式。2、宽数据:每个观测占一行,...

网友评论

      本文标题:EBAS观测数据下载

      本文链接:https://www.haomeiwen.com/subject/rmteartx.html