【生物学家用R做图】Lesson_2:数据输入与数据下载

课程作者是美国Cold Spring Harbor 研究所的Maria Nattestad。这个课程适合初学bioinformatics 和 computational biology的同学。R编程语言非常适合数据分析，统计和科学制图。这个课程本打算是付费课程，后来作者改成免费资源，但是欢迎打赏，我这里是记笔记学习，如果有人觉得打赏过来我会转捐给原作者，届时会把转钱信息公开。
课程里提到的DATA/脚本下载。链接:http://pan.baidu.com/s/1bpaZ9Rx 密码:c439如果有Youtube看不到的请留言给我发你其他链接，清晰度没有Youtube好。
课程内容(往期内容)
Lesson 1: A quick start guide — From data to plot with a few magic words

课程内容（本次课程）

Lesson 2: Importing and downloading data — From Excel, text files, or publicly available data, this lesson covers how to get all of it into R and addresses a number of common problems with data formatting issues.

# ==========================================================
#
#      Lesson 2 -- Importing and downloading data
#      •   Importing data from Excel
#      •   Downloading from UCSC
#      •   Downloading from ENSEMBL
#      •   Downloading from ENCODE
#
# ==========================================================

# Getting data from Excel
# Get the excel file from this paper: "Gene expression profiling of breast cell lines identifies potential new basal markers". Supplementary table 1
# Go into excel and save it as "Tab Delimited Text (.txt)"

filename <- "Lesson-02/micro_array_results_table1.txt"

my_data <- read.csv(filename, sep="\t", header=TRUE)
head(my_data)

# Where to find publicly available big data
# UCSC -- RefSeq genes from table browser
# Ensembl -- Mouse regulatory features MultiCell
# ENCODE -- HMM: wgEncodeBroadHmmGm12878HMM.bed

genes <- read.csv("Lesson-02/RefSeq_Genes.dms", sep="\t", header=TRUE)
head(genes)
dim(genes)

regulatory_features <- read.csv("Lesson-02/homo_sapiens.GRCh38.Fetal_Muscle_Leg.Regulatory_Build.regulatory_activity.20161111.gff", sep="\t", header=FALSE)
head(regulatory_features)
dim(regulatory_features)



chromHMM <- read.csv("Lesson-02/wgEncodeBroadHmmGm12878HMM.bed", sep="\t", header=FALSE)
head(chromHMM)
dim(chromHMM)

最后补充一下，各个基因组的版本对应关系，找了些，感觉生信菜鸟团的比较好，如下：

首先是NCBI对应UCSC，对应ENSEMBL数据库：

• GRCh36 (hg18): ENSEMBL release_52.
• GRCh37 (hg19): ENSEMBL release_59/61/64/68/69/75.
• GRCh38 (hg38): ENSEMBL release_76/77/78/80/81/82.
  可以看到ENSEMBL的版本特别复杂！！！很容易搞混！
  但是UCSC的版本就简单了，就hg18,19,38, 常用的是hg19，但是我推荐大家都转为hg38
  看起来NCBI也是很简单，就GRCh36,37,38，但是里面水也很深！
  Feb 13 2014 00:00 Directory April_14_2003
  Apr 06 2006 00:00 Directory BUILD.33
  Apr 06 2006 00:00 Directory BUILD.34.1
  Apr 06 2006 00:00 Directory BUILD.34.2
  Apr 06 2006 00:00 Directory BUILD.34.3
  Apr 06 2006 00:00 Directory BUILD.35.1
  Aug 03 2009 00:00 Directory BUILD.36.1
  Aug 03 2009 00:00 Directory BUILD.36.2
  Sep 04 2012 00:00 Directory BUILD.36.3
  Jun 30 2011 00:00 Directory BUILD.37.1
  Sep 07 2011 00:00 Directory BUILD.37.2
  Dec 12 2012 00:00 Directory BUILD.37.3

可以看到，有37.1, 37.2， 37.3 等等，不过这种版本一般指的是注释在更新，基因组序列一般不会更新！！！
反正你记住hg19基因组大小是3G，压缩后八九百兆即可！！！

如果要下载GTF注释文件，基因组版本尤为重要！！！

对NCBI：ftp://ftp.ncbi.nih.gov/genomes/H_sapiens/GFF/ ##最新版（hg38）
ftp://ftp.ncbi.nlm.nih.gov/genomes/Homo_sapiens/ARCHIVE/ ## 其它版本

对于ensembl：
ftp://ftp.ensembl.org/pub/release-75/gtf/homo_sapiens/Homo_sapiens.GRCh37.75.gtf.gz
变幻中间的release就可以拿到所有版本信息：ftp://ftp.ensembl.org/pub/
对于UCSC，那就有点麻烦了：
需要选择一系列参数：
http://genome.ucsc.edu/cgi-bin/hgTables

1. Navigate to http://genome.ucsc.edu/cgi-bin/hgTables

2. Select the following options:clade: Mammalgenome: Humanassembly: Feb. 2009 (GRCh37/hg19)group: Genes and Gene Predictionstrack: UCSC Genestable: knownGeneregion: Select "genome" for the entire genome.output format: GTF - gene transfer formatoutput file: enter a file name to save your results to a file, or leave blank to display results in the browser

3. Click 'get output'.

现在重点来了，搞清楚版本关系了，就要下载呀！
UCSC里面下载非常方便，只需要根据基因组简称来拼接url即可：
http://hgdownload.cse.ucsc.edu/goldenPath/mm10/bigZips/chromFa.tar.gz
http://hgdownload.cse.ucsc.edu/goldenPath/mm9/bigZips/chromFa.tar.gz
http://hgdownload.cse.ucsc.edu/goldenPath/hg19/bigZips/chromFa.tar.gz
http://hgdownload.cse.ucsc.edu/goldenPath/hg38/bigZips/chromFa.tar.gz

或者用shell脚本指定下载的染色体号：

for i in $(seq 1 22) X Y M;
do echo $i;
wget http://hgdownload.cse.ucsc.edu/goldenPath/hg19/chromosomes/chr${i}.fa.gz; 
## 这里也可以用NCBI的：ftp://ftp.ncbi.nih.gov/genomes/M_musculus/ARCHIVE/MGSCv3_Release3/Assembled_Chromosomes/chr前缀
done
gunzip *.gz
for i in $(seq 1 22) X Y M;
do cat chr${i}.fa >> hg19.fasta;
done
rm -fr chr*.fasta