Sqoop从Hive导出表到Mysql报错处理

精选原创

江南独孤客 2022-05-13 09:22:40 博主文章分类：大数据篇 ©著作权

文章标签 java mysql apache 文章分类 Hive 大数据

©著作权归作者所有：来自51CTO博客作者江南独孤客的原创作品，请联系作者获取转载授权，否则将追究法律责任

1.调度中的报错信息

[INFO] 2022-05-13 08:44:19.995  - [taskAppId=TASK-29-71229-742765]:[138] -  -> 22/05/13 08:44:19 INFO mapreduce.Job:  map 100% reduce 0%
  22/05/13 08:44:19 INFO mapreduce.Job: Job job_1640763627991_314918 failed with state FAILED due to: Task failed task_1640763627991_314918_m_000000
  Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0
  
  22/05/13 08:44:19 INFO mapreduce.Job: Counters: 9
    Job Counters 
      Failed map tasks=1
      Killed map tasks=1
      Launched map tasks=2
      Rack-local map tasks=2
      Total time spent by all maps in occupied slots (ms)=110892
      Total time spent by all reduces in occupied slots (ms)=0
      Total time spent by all map tasks (ms)=55446
      Total vcore-milliseconds taken by all map tasks=55446
      Total megabyte-milliseconds taken by all map tasks=113553408
  22/05/13 08:44:19 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
  22/05/13 08:44:19 INFO mapreduce.ExportJobBase: Transferred 0 bytes in 46.8801 seconds (0 bytes/sec)
  22/05/13 08:44:19 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
  22/05/13 08:44:19 INFO mapreduce.ExportJobBase: Exported 0 records.
  22/05/13 08:44:19 ERROR mapreduce.ExportJobBase: Export job failed!
  22/05/13 08:44:19 ERROR tool.ExportTool: Error during export: 
  Export job failed!
    at org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:444)
    at org.apache.sqoop.manager.SqlManager.exportTable(SqlManager.java:930)
    at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:93)
    at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:112)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:146)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:182)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:233)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:242)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:251)

2.yarn任务的报错信息

2022-05-13 08:44:17,658 ERROR [IPC Server handler 7 on 45227] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1640763627991_314918_m_000000_0 - exited : java.io.IOException: Can't export data, please check failed map task logs
  at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:121)
  at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:38)
  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
  at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:422)
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: java.io.IOException: java.sql.BatchUpdateException: Data truncation: Out of range value for column 'watch_n' at row 1
  at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWriter.java:233)
  at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWriter.java:46)
  at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:670)
  at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
  at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
  at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:89)
  ... 10 more
Caused by: java.sql.BatchUpdateException: Data truncation: Out of range value for column 'watch_n' at row 1
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
  at com.mysql.jdbc.Util.getInstance(Util.java:408)
  at com.mysql.jdbc.SQLError.createBatchUpdateException(SQLError.java:1163)
  at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1823)
  at com.mysql.jdbc.PreparedStatement.executeBatchInternal(PreparedStatement.java:1307)
  at com.mysql.jdbc.StatementImpl.executeBatch(StatementImpl.java:970)
  at org.apache.sqoop.mapreduce.AsyncSqlOutputFormat$AsyncSqlExecThread.run(AsyncSqlOutputFormat.java:231)
Caused by: com.mysql.jdbc.MysqlDataTruncation: Data truncation: Out of range value for column 'watch_n' at row 1
  at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3976)
  at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3914)
  at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2530)
  at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2683)
  at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2495)
  at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1903)
  at com.mysql.jdbc.PreparedStatement.executeUpdateInternal(PreparedStatement.java:2124)
  at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1801)
  ... 3 more

2022-05-13 08:44:17,658 INFO [IPC Server handler 7 on 45227] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1640763627991_314918_m_000000_0: Error: java.io.IOException: Can't export data, please check failed map task logs
  at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:121)
  at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:38)
  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
  at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:422)
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: java.io.IOException: java.sql.BatchUpdateException: Data truncation: Out of range value for column 'watch_n' at row 1
  at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWriter.java:233)
  at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWriter.java:46)
  at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:670)
  at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
  at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
  at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:89)
  ... 10 more
Caused by: java.sql.BatchUpdateException: Data truncation: Out of range value for column 'watch_n' at row 1
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
  at com.mysql.jdbc.Util.getInstance(Util.java:408)
  at com.mysql.jdbc.SQLError.createBatchUpdateException(SQLError.java:1163)
  at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1823)
  at com.mysql.jdbc.PreparedStatement.executeBatchInternal(PreparedStatement.java:1307)
  at com.mysql.jdbc.StatementImpl.executeBatch(StatementImpl.java:970)
  at org.apache.sqoop.mapreduce.AsyncSqlOutputFormat$AsyncSqlExecThread.run(AsyncSqlOutputFormat.java:231)
Caused by: com.mysql.jdbc.MysqlDataTruncation: Data truncation: Out of range value for column 'watch_n' at row 1
  at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3976)
  at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3914)
  at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2530)
  at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2683)
  at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2495)
  at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1903)
  at com.mysql.jdbc.PreparedStatement.executeUpdateInternal(PreparedStatement.java:2124)
  at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1801)
  ... 3 more

2022-05-13 08:44:17,660 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1640763627991_314918_m_000000_0: Error: java.io.IOException: Can't export data, please check failed map task logs
  at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:121)
  at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:38)
  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
  at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:422)
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: java.io.IOException: java.sql.BatchUpdateException: Data truncation: Out of range value for column 'watch_n' at row 1
  at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWriter.java:233)
  at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWriter.java:46)
  at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:670)
  at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
  at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
  at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:89)
  ... 10 more
Caused by: java.sql.BatchUpdateException: Data truncation: Out of range value for column 'watch_n' at row 1
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
  at com.mysql.jdbc.Util.getInstance(Util.java:408)
  at com.mysql.jdbc.SQLError.createBatchUpdateException(SQLError.java:1163)
  at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1823)
  at com.mysql.jdbc.PreparedStatement.executeBatchInternal(PreparedStatement.java:1307)
  at com.mysql.jdbc.StatementImpl.executeBatch(StatementImpl.java:970)
  at org.apache.sqoop.mapreduce.AsyncSqlOutputFormat$AsyncSqlExecThread.run(AsyncSqlOutputFormat.java:231)
Caused by: com.mysql.jdbc.MysqlDataTruncation: Data truncation: Out of range value for column 'watch_n' at row 1
  at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3976)
  at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3914)
  at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2530)
  at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2683)
  at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2495)
  at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1903)
  at com.mysql.jdbc.PreparedStatement.executeUpdateInternal(PreparedStatement.java:2124)
  at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1801)
  ... 3 more
2022-05-13 08:44:17,662 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1640763627991_314918_m_000000_0 TaskAttempt Transitioned from RUNNING to FAIL_FINISHING_CONTAINER