Google Cluster Data Download

GCloud在ubuntu 初始化的问题解决

sudo chown shootime -R /home/shootime/.config/gcloud

1. Install the latest Cloud Tools version (154.0.1)

https://cloud.google.com/sdk/docs/#deb

2. 下载task_events部分数据[http://www.cnblogs.com/instant7/p/4102818.html]

importurllib2

url='https://commondatastorage.googleapis.com/clusterdata-2011-1/'f= open('C:\\SHA256SUM')

l=f.readlines()

f.close()foriinl:ifi.count('task_events')>0:

fileAddr= i.split()[1][1:]

fileName= fileAddr.split('/')[1]print'downloading', fileName

data= urllib2.urlopen(url+fileAddr).read()print'saving', fileName

fileDown= open('C:\\task_events\\'+fileName,'wb')

fileDown.write(data)

fileDown.close()

3. 云服务器ssh远程下载到本地文件夹

在本地终端打开下列命令

sudo scp cluster@13.94.41.xxx:~/data/clusterdata-2011-2/task_usage/part-00000-of-00500.csv.gz ~/Downloads/

sudo scp -r cluster@13.94.41.xxx:~/data/clusterdata-2011-2/task_usage /media/shootime/TFV/cluster

4. 下载数据集

gsutil cp -R gs://clusterdata-2011-2/task_usage ~/data/