基因数据处理22之对GRCH38全基因建立BWA索引

原创

KeepLearningAI 2023-01-04 10:58:28 博主文章分类：基因数据处理 ©著作权

文章标签 基因数据处理22之对GRCH38全基因建 sed hadoop Real 文章分类 Spark 大数据

©著作权归作者所有：来自51CTO博客作者KeepLearningAI的原创作品，请联系作者获取转载授权，否则将追究法律责任

环境：

ubuntu 14.04

内存 6G

bwa 0.7.12

结论：

建立索引大概4500秒左右

节点2运行：

hadoop@Mcnode2:~/cloud/adam/xubo/data/test20160422$ cp ../test20160310/GCA_000001405.15_GRCh38/GCA_000001405.15_GRCh38_full_analysis_set.fna .
hadoop@Mcnode2:~/cloud/adam/xubo/data/test20160422$ ls
GCA_000001405.15_GRCh38_full_analysis_set.fna
hadoop@Mcnode2:~/cloud/adam/xubo/data/test20160422$ bwa index GCA_000001405.15_GRCh38_full_analysis_set.fna 
[bwa_index] Pack FASTA... 34.05 sec
[bwa_index] Construct BWT for the packed sequence...
[BWTIncCreate] textLength=6418915856, availableWord=463658232
[BWTIncConstructFromPacked] 10 iterations done. 100000000 characters processed.
[BWTIncConstructFromPacked] 20 iterations done. 200000000 characters processed.
[BWTIncConstructFromPacked] 30 iterations done. 300000000 characters processed.
[BWTIncConstructFromPacked] 40 iterations done. 400000000 characters processed.
[BWTIncConstructFromPacked] 50 iterations done. 500000000 characters processed.
[BWTIncConstructFromPacked] 60 iterations done. 600000000 characters processed.
[BWTIncConstructFromPacked] 70 iterations done. 700000000 characters processed.
[BWTIncConstructFromPacked] 80 iterations done. 800000000 characters processed.
[BWTIncConstructFromPacked] 90 iterations done. 900000000 characters processed.
[BWTIncConstructFromPacked] 100 iterations done. 1000000000 characters processed.
[BWTIncConstructFromPacked] 110 iterations done. 1100000000 characters processed.
[BWTIncConstructFromPacked] 120 iterations done. 1200000000 characters processed.
[BWTIncConstructFromPacked] 130 iterations done. 1300000000 characters processed.
[BWTIncConstructFromPacked] 140 iterations done. 1400000000 characters processed.
[BWTIncConstructFromPacked] 150 iterations done. 1500000000 characters processed.
[BWTIncConstructFromPacked] 160 iterations done. 1600000000 characters processed.
[BWTIncConstructFromPacked] 170 iterations done. 1700000000 characters processed.
[BWTIncConstructFromPacked] 180 iterations done. 1800000000 characters processed.
[BWTIncConstructFromPacked] 190 iterations done. 1900000000 characters processed.
[BWTIncConstructFromPacked] 200 iterations done. 2000000000 characters processed.
[BWTIncConstructFromPacked] 210 iterations done. 2100000000 characters processed.
[BWTIncConstructFromPacked] 220 iterations done. 2200000000 characters processed.
[BWTIncConstructFromPacked] 230 iterations done. 2300000000 characters processed.
[BWTIncConstructFromPacked] 240 iterations done. 2400000000 characters processed.
[BWTIncConstructFromPacked] 250 iterations done. 2500000000 characters processed.
[BWTIncConstructFromPacked] 260 iterations done. 2600000000 characters processed.
[BWTIncConstructFromPacked] 270 iterations done. 2700000000 characters processed.
[BWTIncConstructFromPacked] 280 iterations done. 2800000000 characters processed.
[BWTIncConstructFromPacked] 290 iterations done. 2900000000 characters processed.
[BWTIncConstructFromPacked] 300 iterations done. 3000000000 characters processed.
[BWTIncConstructFromPacked] 310 iterations done. 3100000000 characters processed.
[BWTIncConstructFromPacked] 320 iterations done. 3200000000 characters processed.
[BWTIncConstructFromPacked] 330 iterations done. 3300000000 characters processed.
[BWTIncConstructFromPacked] 340 iterations done. 3400000000 characters processed.
[BWTIncConstructFromPacked] 350 iterations done. 3500000000 characters processed.
[BWTIncConstructFromPacked] 360 iterations done. 3600000000 characters processed.
[BWTIncConstructFromPacked] 370 iterations done. 3700000000 characters processed.
[BWTIncConstructFromPacked] 380 iterations done. 3800000000 characters processed.
[BWTIncConstructFromPacked] 390 iterations done. 3900000000 characters processed.
[BWTIncConstructFromPacked] 400 iterations done. 4000000000 characters processed.
[BWTIncConstructFromPacked] 410 iterations done. 4100000000 characters processed.
[BWTIncConstructFromPacked] 420 iterations done. 4200000000 characters processed.
[BWTIncConstructFromPacked] 430 iterations done. 4300000000 characters processed.
[BWTIncConstructFromPacked] 440 iterations done. 4400000000 characters processed.
[BWTIncConstructFromPacked] 450 iterations done. 4500000000 characters processed.
[BWTIncConstructFromPacked] 460 iterations done. 4600000000 characters processed.
[BWTIncConstructFromPacked] 470 iterations done. 4700000000 characters processed.
[BWTIncConstructFromPacked] 480 iterations done. 4800000000 characters processed.
[BWTIncConstructFromPacked] 490 iterations done. 4900000000 characters processed.
[BWTIncConstructFromPacked] 500 iterations done. 5000000000 characters processed.
[BWTIncConstructFromPacked] 510 iterations done. 5100000000 characters processed.
[BWTIncConstructFromPacked] 520 iterations done. 5200000000 characters processed.
[BWTIncConstructFromPacked] 530 iterations done. 5300000000 characters processed.
[BWTIncConstructFromPacked] 540 iterations done. 5400000000 characters processed.
[BWTIncConstructFromPacked] 550 iterations done. 5500000000 characters processed.
[BWTIncConstructFromPacked] 560 iterations done. 5600000000 characters processed.
[BWTIncConstructFromPacked] 570 iterations done. 5700000000 characters processed.
[BWTIncConstructFromPacked] 580 iterations done. 5798188880 characters processed.
[BWTIncConstructFromPacked] 590 iterations done. 5886472096 characters processed.
[BWTIncConstructFromPacked] 600 iterations done. 5964934432 characters processed.
[BWTIncConstructFromPacked] 610 iterations done. 6034667936 characters processed.
[BWTIncConstructFromPacked] 620 iterations done. 6096643264 characters processed.
[BWTIncConstructFromPacked] 630 iterations done. 6151723072 characters processed.
[BWTIncConstructFromPacked] 640 iterations done. 6200674128 characters processed.
[BWTIncConstructFromPacked] 650 iterations done. 6244177920 characters processed.
[BWTIncConstructFromPacked] 660 iterations done. 6282840176 characters processed.
[BWTIncConstructFromPacked] 670 iterations done. 6317199264 characters processed.
[BWTIncConstructFromPacked] 680 iterations done. 6347733664 characters processed.
[BWTIncConstructFromPacked] 690 iterations done. 6374868704 characters processed.
[BWTIncConstructFromPacked] 700 iterations done. 6398982368 characters processed.
[BWTIncConstructFromPacked] 710 iterations done. 6418915856 characters processed.
[bwt_gen] Finished constructing BWT in 710 iterations.
[bwa_index] 3102.10 seconds elapse.
[bwa_index] Update BWT... 23.50 sec
[bwa_index] Pack forward-only FASTA... 21.21 sec
[bwa_index] Construct SA from BWT and Occ... 1087.39 sec
[main] Version: 0.7.12-r1039
[main] CMD: bwa index GCA_000001405.15_GRCh38_full_analysis_set.fna
[main] Real time: 4444.462 sec; CPU: 4268.248 sec

hadoop@Mcnode2:~/cloud/adam/xubo/data/test20160422$ ll -h
total 8.3G
drwxrwxr-x 2 hadoop hadoop 4.0K  4月 22 16:36 ./
drwxrwxr-x 4 hadoop hadoop 4.0K  4月 22 15:20 ../
-rw------- 1 hadoop hadoop 3.1G  4月 22 15:22 GCA_000001405.15_GRCh38_full_analysis_set.fna
-rw-rw-r-- 1 hadoop hadoop  20K  4月 22 16:18 GCA_000001405.15_GRCh38_full_analysis_set.fna.amb
-rw-rw-r-- 1 hadoop hadoop  72K  4月 22 16:18 GCA_000001405.15_GRCh38_full_analysis_set.fna.ann
-rw-rw-r-- 1 hadoop hadoop 3.0G  4月 22 16:17 GCA_000001405.15_GRCh38_full_analysis_set.fna.bwt
-rw-rw-r-- 1 hadoop hadoop 766M  4月 22 16:18 GCA_000001405.15_GRCh38_full_analysis_set.fna.pac
-rw-rw-r-- 1 hadoop hadoop 1.5G  4月 22 16:37 GCA_000001405.15_GRCh38_full_analysis_set.fna.sa

节点3运行：

hadoop@Mcnode3:~/cloud/adam/xubo/data/test20160422$ cp ../test20160310/GCA_000001405.15_GRCh38/GCA_000001405.15_GRCh38_full_analysis_set.fna .
hadoop@Mcnode3:~/cloud/adam/xubo/data/test20160422$ free -m
             total       used       free     shared    buffers     cached
Mem:          5960       5851        109          0        149       4482
-/+ buffers/cache:       1218       4742
Swap:         6133        314       5819
hadoop@Mcnode3:~/cloud/adam/xubo/data/test20160422$ bwa index GCA_000001405.15_GRCh38_full_analysis_set.fna 
[bwa_index] Pack FASTA... 33.06 sec
[bwa_index] Construct BWT for the packed sequence...
[BWTIncCreate] textLength=6418915856, availableWord=463658232
[BWTIncConstructFromPacked] 10 iterations done. 100000000 characters processed.
[BWTIncConstructFromPacked] 20 iterations done. 200000000 characters processed.
[BWTIncConstructFromPacked] 30 iterations done. 300000000 characters processed.
[BWTIncConstructFromPacked] 40 iterations done. 400000000 characters processed.
[BWTIncConstructFromPacked] 50 iterations done. 500000000 characters processed.
[BWTIncConstructFromPacked] 60 iterations done. 600000000 characters processed.
[BWTIncConstructFromPacked] 70 iterations done. 700000000 characters processed.
[BWTIncConstructFromPacked] 80 iterations done. 800000000 characters processed.
[BWTIncConstructFromPacked] 90 iterations done. 900000000 characters processed.
[BWTIncConstructFromPacked] 100 iterations done. 1000000000 characters processed.
[BWTIncConstructFromPacked] 110 iterations done. 1100000000 characters processed.
[BWTIncConstructFromPacked] 120 iterations done. 1200000000 characters processed.
[BWTIncConstructFromPacked] 130 iterations done. 1300000000 characters processed.
[BWTIncConstructFromPacked] 140 iterations done. 1400000000 characters processed.
[BWTIncConstructFromPacked] 150 iterations done. 1500000000 characters processed.
[BWTIncConstructFromPacked] 160 iterations done. 1600000000 characters processed.
[BWTIncConstructFromPacked] 170 iterations done. 1700000000 characters processed.
[BWTIncConstructFromPacked] 180 iterations done. 1800000000 characters processed.
[BWTIncConstructFromPacked] 190 iterations done. 1900000000 characters processed.
[BWTIncConstructFromPacked] 200 iterations done. 2000000000 characters processed.
[BWTIncConstructFromPacked] 210 iterations done. 2100000000 characters processed.
[BWTIncConstructFromPacked] 220 iterations done. 2200000000 characters processed.
[BWTIncConstructFromPacked] 230 iterations done. 2300000000 characters processed.
[BWTIncConstructFromPacked] 240 iterations done. 2400000000 characters processed.
[BWTIncConstructFromPacked] 250 iterations done. 2500000000 characters processed.
[BWTIncConstructFromPacked] 260 iterations done. 2600000000 characters processed.
[BWTIncConstructFromPacked] 270 iterations done. 2700000000 characters processed.
[BWTIncConstructFromPacked] 280 iterations done. 2800000000 characters processed.
[BWTIncConstructFromPacked] 290 iterations done. 2900000000 characters processed.
[BWTIncConstructFromPacked] 300 iterations done. 3000000000 characters processed.
[BWTIncConstructFromPacked] 310 iterations done. 3100000000 characters processed.
[BWTIncConstructFromPacked] 320 iterations done. 3200000000 characters processed.
[BWTIncConstructFromPacked] 330 iterations done. 3300000000 characters processed.
[BWTIncConstructFromPacked] 340 iterations done. 3400000000 characters processed.
[BWTIncConstructFromPacked] 350 iterations done. 3500000000 characters processed.
[BWTIncConstructFromPacked] 360 iterations done. 3600000000 characters processed.
[BWTIncConstructFromPacked] 370 iterations done. 3700000000 characters processed.
[BWTIncConstructFromPacked] 380 iterations done. 3800000000 characters processed.
[BWTIncConstructFromPacked] 390 iterations done. 3900000000 characters processed.
[BWTIncConstructFromPacked] 400 iterations done. 4000000000 characters processed.
[BWTIncConstructFromPacked] 410 iterations done. 4100000000 characters processed.
[BWTIncConstructFromPacked] 420 iterations done. 4200000000 characters processed.
[BWTIncConstructFromPacked] 430 iterations done. 4300000000 characters processed.
[BWTIncConstructFromPacked] 440 iterations done. 4400000000 characters processed.
[BWTIncConstructFromPacked] 450 iterations done. 4500000000 characters processed.
[BWTIncConstructFromPacked] 460 iterations done. 4600000000 characters processed.
[BWTIncConstructFromPacked] 470 iterations done. 4700000000 characters processed.
[BWTIncConstructFromPacked] 480 iterations done. 4800000000 characters processed.
[BWTIncConstructFromPacked] 490 iterations done. 4900000000 characters processed.
[BWTIncConstructFromPacked] 500 iterations done. 5000000000 characters processed.
[BWTIncConstructFromPacked] 510 iterations done. 5100000000 characters processed.
[BWTIncConstructFromPacked] 520 iterations done. 5200000000 characters processed.
[BWTIncConstructFromPacked] 530 iterations done. 5300000000 characters processed.
[BWTIncConstructFromPacked] 540 iterations done. 5400000000 characters processed.
[BWTIncConstructFromPacked] 550 iterations done. 5500000000 characters processed.
[BWTIncConstructFromPacked] 560 iterations done. 5600000000 characters processed.
[BWTIncConstructFromPacked] 570 iterations done. 5700000000 characters processed.
[BWTIncConstructFromPacked] 580 iterations done. 5798188880 characters processed.
[BWTIncConstructFromPacked] 590 iterations done. 5886472096 characters processed.
[BWTIncConstructFromPacked] 600 iterations done. 5964934432 characters processed.
[BWTIncConstructFromPacked] 610 iterations done. 6034667936 characters processed.
[BWTIncConstructFromPacked] 620 iterations done. 6096643264 characters processed.
[BWTIncConstructFromPacked] 630 iterations done. 6151723072 characters processed.
[BWTIncConstructFromPacked] 640 iterations done. 6200674128 characters processed.
[BWTIncConstructFromPacked] 650 iterations done. 6244177920 characters processed.
[BWTIncConstructFromPacked] 660 iterations done. 6282840176 characters processed.
[BWTIncConstructFromPacked] 670 iterations done. 6317199264 characters processed.
[BWTIncConstructFromPacked] 680 iterations done. 6347733664 characters processed.
[BWTIncConstructFromPacked] 690 iterations done. 6374868704 characters processed.
[BWTIncConstructFromPacked] 700 iterations done. 6398982368 characters processed.
[BWTIncConstructFromPacked] 710 iterations done. 6418915856 characters processed.
[bwt_gen] Finished constructing BWT in 710 iterations.
[bwa_index] 3115.88 seconds elapse.
[bwa_index] Update BWT... 24.15 sec
[bwa_index] Pack forward-only FASTA... 21.30 sec
[bwa_index] Construct SA from BWT and Occ... 1092.00 sec
[main] Version: 0.7.12-r1039
[main] CMD: bwa index GCA_000001405.15_GRCh38_full_analysis_set.fna
[main] Real time: 4647.870 sec; CPU: 4286.403 sec

hadoop@Mcnode3:~/cloud/adam/xubo/data/test20160422$ ll -h
total 8.3G
drwxrwxr-x 2 hadoop hadoop 4.0K  4月 22 16:42 ./
drwxrwxr-x 4 hadoop hadoop 4.0K  4月 22 15:22 ../
-rw------- 1 hadoop hadoop 3.1G  4月 22 15:24 GCA_000001405.15_GRCh38_full_analysis_set.fna
-rw-rw-r-- 1 hadoop hadoop  20K  4月 22 16:21 GCA_000001405.15_GRCh38_full_analysis_set.fna.amb
-rw-rw-r-- 1 hadoop hadoop  72K  4月 22 16:21 GCA_000001405.15_GRCh38_full_analysis_set.fna.ann
-rw-rw-r-- 1 hadoop hadoop 3.0G  4月 22 16:19 GCA_000001405.15_GRCh38_full_analysis_set.fna.bwt
-rw-rw-r-- 1 hadoop hadoop 766M  4月 22 16:21 GCA_000001405.15_GRCh38_full_analysis_set.fna.pac
-rw-rw-r-- 1 hadoop hadoop 1.5G  4月 22 16:42 GCA_000001405.15_GRCh38_full_analysis_set.fna.sa