Adam学习10之集群节点的内存对运行时间的影响

原创

KeepLearningAI 2023-01-04 11:01:05 博主文章分类：adam ©著作权

文章标签 Adam学习10之集群节点的内存对运行时 hadoop spark 待解决 文章分类 运维

©著作权归作者所有：来自51CTO博客作者KeepLearningAI的原创作品，请联系作者获取转载授权，否则将追究法律责任

问题：将单节点的内存从512M加到4096M时运行时间不减少反而增加，不解，待解决。是不是单个测试用例的偶然性？

hadoop@Master:~/cloud/testByXubo/spark/hs38DH/package$ ./cluster.sh 
fq0.count:105887                                                                
Method 1=> Length:2971 sum:7888989 time:26936ms
Method 2=> Length:2971 sum2:7888989 time:2367ms
Method 3=> Length:2971 sum3:7888989.0 time:4200ms
hadoop@Master:~/cloud/testByXubo/spark/hs38DH/package$ cp cluster.sh submit.sh
hadoop@Master:~/cloud/testByXubo/spark/hs38DH/package$ mv submit.sh cluster4G.sh 
hadoop@Master:~/cloud/testByXubo/spark/hs38DH/package$ vi cluster4G.sh 
hadoop@Master:~/cloud/testByXubo/spark/hs38DH/package$ ./cluster4G.sh 
fq0.count:105887                                                                
Method 1=> Length:2971 sum:7888989 time:34320ms
Method 2=> Length:2971 sum2:7888989 time:3078ms
Method 3=> Length:2971 sum3:7888989.0 time:4136ms
hadoop@Master:~/cloud/testByXubo/spark/hs38DH/package$ 
hadoop@Master:~/cloud/testByXubo/spark/hs38DH/package$ 
hadoop@Master:~/cloud/testByXubo/spark/hs38DH/package$ ./cluster4G.sh 
fq0.count:105887                                                                
Method 1=> Length:2971 sum:7888989 time:27065ms
Method 2=> Length:2971 sum2:7888989 time:2764ms
Method 3=> Length:2971 sum3:7888989.0 time:1292ms
hadoop@Master:~/cloud/testByXubo/spark/hs38DH/package$ ./cluster
-bash: ./cluster: No such file or directory
hadoop@Master:~/cloud/testByXubo/spark/hs38DH/package$ ./cluster.sh 
fq0.count:105887                                                                
Method 1=> Length:2971 sum:7888989 time:24095ms
Method 2=> Length:2971 sum2:7888989 time:4263ms
Method 3=> Length:2971 sum3:7888989.0 time:4006ms
hadoop@Master:~/cloud/testByXubo/spark/hs38DH/package$ ./cluster4G.sh 
fq0.count:105887                                                                
Method 1=> Length:2971 sum:7888989 time:26671ms
Method 2=> Length:2971 sum2:7888989 time:3404ms
Method 3=> Length:2971 sum3:7888989.0 time:4196ms
hadoop@Master:~/cloud/testByXubo/spark/hs38DH/package$

集群：

Application ID Name Cores Memory per Node Submitted Time User State Duration 
app-20160417144715-0006  readFileFromHs38DH  12  4.0 GB  2016/04/17 14:47:15 hadoop FINISHED 47 s 
app-20160417144235-0005  readFileFromHs38DH  12  512.0 MB  2016/04/17 14:42:35 hadoop FINISHED 40 s 
app-20160417143730-0004  readFileFromHs38DH  12  4.0 GB  2016/04/17 14:37:30 hadoop FINISHED 44 s 
app-20160417143619-0003  readFileFromHs38DH  12  4.0 GB  2016/04/17 14:36:19 hadoop FINISHED 51 s 
app-20160417143459-0002  readFileFromHs38DH  12  512.0 MB  2016/04/17 14:34:59 hadoop FINISHED 42 s