真的是一步一个坑啊!!!!

 


初始化数据库


sudo mysql_secure_installation


 


 




hadoop与数据库 hadoop mysql 数据库解决方案_hadoop与数据库


 



hadoop与数据库 hadoop mysql 数据库解决方案_hadoop_02



这条赋权命令分三部分写


grant all on *.* to hive@localhost identified by 'hive';

 



hadoop与数据库 hadoop mysql 数据库解决方案_hive_03



hadoop与数据库 hadoop mysql 数据库解决方案_大数据_04


 


hadoop与数据库 hadoop mysql 数据库解决方案_java_05


 


创建两个文件输入内容



hadoop与数据库 hadoop mysql 数据库解决方案_hadoop与数据库_06


 


问题一,multiple SLF4J bindings


 



hadoop与数据库 hadoop mysql 数据库解决方案_hadoop_07


   


hive 和 hadoop 中的 SLF4J 重复了


在两个中的位置分别如下:


- /usr/local/hive/lib/


- /usr/local/hadoop/share/hadoop/common/lib/


解决办法:

删除任意一个 log4j-slf4j-impl.jar 即可,


建议保留版本高版本。


删除任意一个jar包即可,建议 保留高版本的


把hive下的那个删了


问题2.

Exception in thread" main" java. Lang . NoSuchMethodError: 。。。。


hadoop与数据库 hadoop mysql 数据库解决方案_hadoop与数据库_08


Hadoop的和hive Guava的版本不一致保留复制最高


cp /usr/Local/hadoop/share/hadoop/common/lib/guava-27.0-jre.jar /usr/local/hive/lib

 



hadoop与数据库 hadoop mysql 数据库解决方案_hive_09


问题3.拒绝连接



hadoop与数据库 hadoop mysql 数据库解决方案_java_10


 


错误原因:

 


1.没开hadoop


2.配置文件错误,连接mysql的配置以及hadoop的配置错误


问题4.多处语句错误


 



hadoop与数据库 hadoop mysql 数据库解决方案_hive_11


 


原因:版本、配置不一致。


解决方法 :使用 schematool 工具。 Hive 现在包含一个用于 Hive Metastore 架构操控的脱机工具,名为 schematool. 此工具可用于初始化当前 Hive 版本的 Metastore 架构。此外,其还可处理从较旧版本到新版本的架构升级

 


cd /usr/local/hive/bin
./schematool -initSchema -dbType mysql

hadoop与数据库 hadoop mysql 数据库解决方案_大数据_12



hadoop与数据库 hadoop mysql 数据库解决方案_java_13


 


问题5.Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient


 


FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException:  Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

解决办法:


  进入所安装的 Hive 的 conf 目录,找到 hive-site.xml ,(若没修改,则是 hive-default.xml.template )。


<property>


  <name>hive.metastore.schema.verification</name>


  <value>true</value>


   <description>


   Enforce metastore schema version consistency.


   True: Verify that version information stored in metastore matches with one from Hive jars.  Also disable automatic


         schema migration attempt. Users are required to manully migrate schema after Hive upgrade which ensures


         proper metastore schema migration. (Default)


   False: Warn if the version information stored in metastore doesn't match with one from in Hive jars.


   </description>


</property>


 


改为


<property>


  <name>hive.metastore.schema.verification</name>


  <value> false </value>


   <description>


   Enforce metastore schema version consistency.


   True: Verify that version information stored in metastore matches with one from Hive jars.  Also disable automatic


         schema migration attempt. Users are required to manully migrate schema after Hive upgrade which ensures


         proper metastore schema migration. (Default)


   False: Warn if the version information stored in metastore doesn't match with one from in Hive jars.


   </description>


</property>


 


  然后,再重启 hive ,进入即可。


或者


首先进入hive/bin目录下


执行命令    ./hive --service metastore &

 


问题6:Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient


[hadoop@HadoopSlave1 hive-1.2.1]$ bin/hive


 


Logging initialized using configuration in jar:file:/home/hadoop/app/hive-1.2.1/lib/hive-common-1.2.1.jar!/hive-log4j.properties


Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient


at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)


at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)


at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)


at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)


at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)


at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)


at java.lang.reflect.Method.invoke(Method.java:606)


at org.apache.hadoop.util.RunJar.run(RunJar.java:221)


at org.apache.hadoop.util.RunJar.main(RunJar.java:136)


Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient


at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523)


at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86)


at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)


at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)


at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005)


at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024)


at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)


... 8 more


Caused by: java.lang.reflect.InvocationTargetException


at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)


at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)


at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)


at java.lang.reflect.Constructor.newInstance(Constructor.java:526)


at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)


... 14 more


Caused by: MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: hfds)


at org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:1213)


at org.apache.hadoop.hive.metastore.Warehouse.getFs(Warehouse.java:106)


at org.apache.hadoop.hive.metastore.Warehouse.getDnsPath(Warehouse.java:140)


at org.apache.hadoop.hive.metastore.Warehouse.getDnsPath(Warehouse.java:146)


at org.apache.hadoop.hive.metastore.Warehouse.getWhRoot(Warehouse.java:159)


at org.apache.hadoop.hive.metastore.Warehouse.getDefaultDatabasePath(Warehouse.java:177)


at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB_core(HiveMetaStore.java:600)


at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:620)


at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:461)


at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:66)


at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:72)


at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5762)


at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:199)


at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)


... 19 more


[hadoop@HadoopSlave1 hive-1.2.1]$


 

将hadoop中share/hadoop/yarn/lib路径下jline包换成hive中匹配的jar包

问题7.启动运行hive在ctl输出过多info


配置好hive后,输入hive命令启动时会出现许多INFO和WARN信息。如下: 至今仍未解决


 


15/04/21 11:31:45 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive


15/04/21 11:31:45 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize


15/04/21 11:31:45 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize


15/04/21 11:31:45 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack


15/04/21 11:31:45 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node


15/04/21 11:31:45 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces


15/04/21 11:31:45 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative


15/04/21 11:31:45 WARN conf.HiveConf: DEPRECATED: hive.metastore.ds.retry.* no longer has any effect.  Use hive.hmshandler.retry.* instead


Logging initialized using configuration in file:/home/hbase/install/hive-0.12.0-cdh5.1.0/conf/hive-log4j.properties


SLF4J: Class path contains multiple SLF4J bindings.


SLF4J: Found binding in [jar:file:/home/hbase/install/hadoop-2.3.0-cdh5.1.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]


SLF4J: Found binding in [jar:file:/home/hbase/install/hive-0.12.0-cdh5.1.0/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]


SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.


SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]


 


原因是在新的Hadoop版本中,很多参数都废弃了,所以这些信息一定会显示出来。解决的办法是在$HADOOP_HOME/etc/hadoop/log4j.properties中将 log4j.logger.org.apache.hadoop.conf.Configuration.deprecation打开。如下:


# Uncomment the following line to turn off configuration deprecation warnings.


log4j.logger.org.apache.hadoop.conf.Configuration.deprecation=WARN


 


log4j.properties修改为:hive.root.logger=WARN,console 即可


 


或者   在 hive 安装目录下的 bin 目录下, ./hive -hiveconf hive.root.logger=DEBUG,console

 


问题8.词频统计找不到文件


hadoop与数据库 hadoop mysql 数据库解决方案_hive_14


 


分析原因:


When using the JDBC driver, the command executes on the HiveServer2 side. The file is evaluated to locally exist on the server, which is not true in your case (it exists on the local client program machine).


 


Try instead to load the file to HDFS first, and use a HDFS URI in the LOAD DATA statement to make the server find it.


从上面的解释可知, hive导入数据语句  load data [local] inpath ,是一个服务器端的指令,它是在服务器端执行。因此指定local时表明加载的文件为本地文件,但是这里的local,在hive中指的是 hiveserver 服务所在的机器,而不是hivecli 或 beeline客户端所在的机器(生产环境大都是 hiveserver 和 hivecli不在同一个机器)。


总结:

    解决方法一:把要加载的文件上传到 hiveserver所在的服务器(这一般是不可能的事情),然后执行  load data local inpath [path] [overwrite] into table table_name.




    解决方法二:先将本地文件上传至hdfs,然后使用 load data inpath [hdfspath] [overwrite] into table table_name.


 


方法二为推荐做法。