sqoop 架构 sqoop gc overhead

转载

gulaotou 2023-06-27 23:52:45

文章标签 sqoop 架构字段 hive sqoop 文章分类 架构后端开发

事故现场：Error: GC overhead limit exceeded

第一反应：数据量是否猛增？ select dt,count(*) from ods.ods_x where dt>='2018-03-01' group by dt order by dt desc; 然而并没有

第二反应：split字段分布不均？因为sqoop 抽取的原理是，先求min(split 字段)，max(split 字段), 然后平均分，每个map抽相应范围的数据量。如果某一天

split字段有个跳跃式的增长，比如一个量级，虽然数据量没有变化，但是导致split分布不均，有些map分到的数据量过大，有些却几乎没有。

现场还原1：

18/03/14 02:05:32 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN("id"), MAX("id") FROM "x"

18/03/14 02:05:32 INFO db.IntegerSplitter: split 条件 1=1 and 2=513601701


18/03/13 02:05:34 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN("id"), MAX("id") FROM "x"

18/03/13 02:05:34 INFO db.IntegerSplitter: split 条件 1=1 and 2=10219211

现场还原2:

hive (temp)> dfs -du -h /data/external_table/ods/ods_x/dt=2018-03-12;
0       0        /data/external_table/ods/ods_x/dt=2018-03-12/_SUCCESS
38.0 M  113.9 M  /data/external_table/ods/ods_x/dt=2018-03-12/part-m-00000.snappy
30.9 M  92.7 M   /data/external_table/ods/ods_x/dt=2018-03-12/part-m-00001.snappy
47.2 M  141.7 M  /data/external_table/ods/ods_x/dt=2018-03-12/part-m-00002.snappy
45.6 M  136.9 M  /data/external_table/ods/ods_x/dt=2018-03-12/part-m-00003.snappy
43.8 M  131.4 M  /data/external_table/ods/ods_x/dt=2018-03-12/part-m-00004.snappy
40.0 M  120.0 M  /data/external_table/ods/ods_x/dt=2018-03-12/part-m-00005.snappy
41.5 M  124.6 M  /data/external_table/ods/ods_x/dt=2018-03-12/part-m-00006.snappy
33.6 M  100.9 M  /data/external_table/ods/ods_x/dt=2018-03-12/part-m-00007.snappy
38.5 M  115.6 M  /data/external_table/ods/ods_x/dt=2018-03-12/part-m-00008.snappy
41.0 M  123.1 M  /data/external_table/ods/ods_x/dt=2018-03-12/part-m-00009.snappy


hive (temp)> dfs -du -h /data/external_table/ods/ods_x/dt=2018-03-13;
0        0        /data/external_table/ods/ods_x/dt=2018-03-13/_SUCCESS
93.3 M   280.0 M  /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00000.snappy
113.3 M  340.0 M  /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00001.snappy
100.4 M  301.3 M  /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00002.snappy
95.4 M   286.1 M  /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00003.snappy
106.6 K  319.8 K  /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00004.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00005.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00006.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00007.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00008.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00009.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00010.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00011.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00012.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00013.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00014.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00015.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00016.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00017.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00018.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00019.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00020.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00021.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00022.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00023.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00024.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00025.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00026.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00027.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00028.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00029.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00030.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00031.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00032.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00033.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00034.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00035.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00036.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00037.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00038.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00039.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00040.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00041.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00042.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00043.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00044.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00045.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00046.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00047.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00048.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00049.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00050.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00051.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00052.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00053.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00054.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00055.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00056.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00057.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00058.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00059.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00060.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00061.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00062.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00063.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00064.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00065.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00066.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00067.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00068.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00069.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00070.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00071.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00072.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00073.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00074.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00075.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00076.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00077.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00078.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00079.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00080.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00081.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00082.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00083.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00084.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00085.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00086.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00087.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00088.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00089.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00090.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00091.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00092.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00093.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00094.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00095.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00096.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00097.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00098.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00099.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00100.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00101.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00102.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00103.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00104.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00105.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00106.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00107.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00108.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00109.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00110.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00111.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00112.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00113.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00114.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00115.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00116.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00117.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00118.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00119.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00120.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00121.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00122.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00123.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00124.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00125.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00126.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00127.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00128.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00129.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00130.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00131.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00132.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00133.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00134.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00135.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00136.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00137.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00138.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00139.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00140.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00141.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00142.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00143.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00144.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00145.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00146.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00147.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00148.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00149.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00150.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00151.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00152.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00153.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00154.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00155.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00156.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00157.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00158.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00159.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00160.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00161.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00162.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00163.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00164.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00165.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00166.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00167.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00168.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00169.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00170.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00171.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00172.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00173.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00174.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00175.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00176.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00177.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00178.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00179.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00180.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00181.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00182.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00183.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00184.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00185.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00186.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00187.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00188.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00189.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00190.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00191.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00192.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00193.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00194.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00195.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00196.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00197.snappy
4        12       /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00198.snappy
260      780      /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00199.snappy

解决方案：

1，替换split字段(该字段要有索引，不然抽取异常慢)

2，split字段应该选择没有业务含义的自增字段

3，与业务开发沟通，寻求解决方案。

本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。