文章目录

  • 目标
  • 一、hadoop如何在Linux环境下使用opencv4.5.1？
  • 二、安装opencv4.5.1
  • 1.安装所需依赖
  • 2.编写mapreduce代码
  • 总结


目标

目前正在学习hadoop及flink，想尝试用flink处理图片，但是能用的资料太少，于是打算把hadoop处理图片吃透再去探索flink处理图片，现在有个要求，要在hadoop上将mapreduce与opencv结合达到hadoop+opencv处理图片，网上相关资料很少加上自己很菜，所以走了很多弯路

提示：所用的系统Ubuntu16.04，hadoop集群安装了hadoop版本是3.2.1，jdk是1.8.0_271

一、hadoop如何在Linux环境下使用opencv4.5.1？

与windows不同，在Linux环境下使用opencv，尤其是编写mapreduce代码中使用opencv，需要将opencv的源码编译，安装到Linux上，然后将生成的libopencv_java451.so文件导入到$HADOOP_HOME/lib/native下

二、安装opencv4.5.1

1.安装所需依赖

代码如下：（当然，肯定是在hadoop已经搭建完成的情况下，所以安装jdk的部分不再赘述，我安装的是1.8版本）：

//下载apache ant1.9.15，这个是官网提到的需要安装的，很多安装opencv3.x的教程都没提到apache ant，我编译opencv4.5.1、安装后，发现没有生成libopencv_xxx.so。直到安装了apache ant以后才成功
iwget  https://mirrors.tuna.tsinghua.edu.cn/apache//ant/binaries/apache-ant-1.9.15-bin.zip
//解压所得的apache-ant-1.9.15-bin.zip
unzip ~/apache-ant-1.9.7-bin.zip
//设置ANT_HOME，并添加$ANT_HOME/bin到PATH
export ANT_HOME=/home/ubuntu/Hadoop/opencv/ApacheAnt
export PATH=$PATH:$ANT_HOME/bin
//下载opencv4.5.1的源码，通过github或者官网下载都行，官网安装方式是推荐在github上下载，我是通过官网下载的，都行。
wget -O opencv.zip https://github.com/opencv/opencv/archive/master.zip
//解压opencv.zip，得到opencv.4.5.1，并在里面新建一个build文件夹并进入，待会的编译都要在build文件夹下
unzip opencv.zip
mkdir -p build && cd build

Hadoop 存储照片 hadoop处理图片_opencv

//安装编译所需要的依赖
sudo apt-get install cmake
sudo apt-get install build-essential libgtk2.0-dev libavcodec-dev libavformat-dev libjpeg.dev libtiff4.dev libswscale-dev libjasper-dev
//安装好依赖以后进行编译，这里是将opencv安装在build/install文件夹中，不在系统目录里。如果需要安装在/usr/local中，则需要把cmake那句中的./install 更改成为/usr/local即可。
cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=./install ..
//稍等,出现以下即为成功：
-- Configuring done
-- Generating done
-- Build files have been written to: 此处省略/opencv4.2.0/build
//往上翻，查看有关java的编译信息,看到两个YES就是ok的，我没有遇到过NO，如果出现NO应该是要检测jdk
--   Java:                          
--     ant:                         /usr/bin/ant (ver 1.10.5)
--     JNI:                         /usr/lib/jvm/java-8-openjdk-amd64/include /usr/lib/jvm/java-8-openjdk-amd64/include/linux /usr/lib/jvm/java-8-openjdk-amd64/include
--     Java wrappers:               YES
--     Java tests:                  YES
//进行编译，make -j8，其中的8代表编译的并行度，看自己Ubuntu系统的cpu是几核的，我的是老机子，就设置了-j2，比较慢。
make -j2
//等待100%
[ 99%] Linking CXX executable ../../bin/opencv_test_video 
[100%] Linking CXX executable ../../bin/opencv_perf_video 
[100%] Linking CXX executable ../../bin/opencv_test_gapi 
[100%] Built target opencv_test_video [100%] Built target 
opencv_perf_video [100%] Built target opencv_test_gapi
//编译成功后，即可看到build出现了很多文件夹，然后进行安装，由于编译的时候设置了安装在build/install的信息，所以执行按照后，build下会多出一个install的文件夹
sudo make install

进入build/install就可以看到安装好的opencv4.5.1

Hadoop 存储照片 hadoop处理图片_大数据_02


我们所要的libopencv_java451.so文件就在

build/install/share/java/opencv4下（同时还可以看到opencv-451.jar）

Hadoop 存储照片 hadoop处理图片_opencv_03


接下来就是怎么与hadoop联系起来，在这里我是把整个opencv的文件夹导入到各个节点上，我没有尝试只拷贝libopencv_java451.so，哎，不懂。

Hadoop 存储照片 hadoop处理图片_Hadoop 存储照片_04


然后将libopencv_java451.so，通过scp到各个节点，在这里我的hadoop集群安装的是hadoop3.2.1版本

最后，在各个节点的$HADOOP_HOME/lib/native下要保证都有libopencv_java451.so


在这里可以查看一下libopencv_java451.so文件是否可用

root@hadoop4:/home/ubuntu/Hadoop/hadoop-3.2.1/lib/native# ldd libopencv_java451.so 
	linux-vdso.so.1 =>  (0x00007ffe784fd000)
	libopencv_ml.so.4.5 => /home/ubuntu/Hadoop/opencv/opencv-4.5.1/build/install/lib/libopencv_ml.so.4.5 (0x00007fc4df6c8000)
	libopencv_photo.so.4.5 => /home/ubuntu/Hadoop/opencv/opencv-4.5.1/build/install/lib/libopencv_photo.so.4.5 (0x00007fc4df41f000)
	libopencv_objdetect.so.4.5 => /home/ubuntu/Hadoop/opencv/opencv-4.5.1/build/install/lib/libopencv_objdetect.so.4.5 (0x00007fc4df1a6000)
	libopencv_video.so.4.5 => /home/ubuntu/Hadoop/opencv/opencv-4.5.1/build/install/lib/libopencv_video.so.4.5 (0x00007fc4def2a000)
	libopencv_videoio.so.4.5 => /home/ubuntu/Hadoop/opencv/opencv-4.5.1/build/install/lib/libopencv_videoio.so.4.5 (0x00007fc4decc2000)
	libopencv_imgcodecs.so.4.5 => /home/ubuntu/Hadoop/opencv/opencv-4.5.1/build/install/lib/libopencv_imgcodecs.so.4.5 (0x00007fc4de7b4000)
	libopencv_dnn.so.4.5 => /home/ubuntu/Hadoop/opencv/opencv-4.5.1/build/install/lib/libopencv_dnn.so.4.5 (0x00007fc4de168000)
	libopencv_calib3d.so.4.5 => /home/ubuntu/Hadoop/opencv/opencv-4.5.1/build/install/lib/libopencv_calib3d.so.4.5 (0x00007fc4ddd7b000)
	libopencv_features2d.so.4.5 => /home/ubuntu/Hadoop/opencv/opencv-4.5.1/build/install/lib/libopencv_features2d.so.4.5 (0x00007fc4ddaa9000)
	libopencv_flann.so.4.5 => /home/ubuntu/Hadoop/opencv/opencv-4.5.1/build/install/lib/libopencv_flann.so.4.5 (0x00007fc4dd848000)
	libopencv_imgproc.so.4.5 => /home/ubuntu/Hadoop/opencv/opencv-4.5.1/build/install/lib/libopencv_imgproc.so.4.5 (0x00007fc4dcfb9000)
	libopencv_core.so.4.5 => /home/ubuntu/Hadoop/opencv/opencv-4.5.1/build/install/lib/libopencv_core.so.4.5 (0x00007fc4dc954000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fc4dc5d2000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc4dc2c9000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fc4dc0b3000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc4dbce9000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc4dbacc000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fc4db8c8000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fc4db6c0000)
	libavcodec-ffmpeg.so.56 => /usr/lib/x86_64-linux-gnu/libavcodec-ffmpeg.so.56 (0x00007fc4da28c000)
	libavformat-ffmpeg.so.56 => /usr/lib/x86_64-linux-gnu/libavformat-ffmpeg.so.56 (0x00007fc4d9e8d000)
	libavutil-ffmpeg.so.54 => /usr/lib/x86_64-linux-gnu/libavutil-ffmpeg.so.54 (0x00007fc4d9c1e000)
	libswscale-ffmpeg.so.3 => /usr/lib/x86_64-linux-gnu/libswscale-ffmpeg.so.3 (0x00007fc4d998f000)
	libjpeg.so.8 => /usr/lib/x86_64-linux-gnu/libjpeg.so.8 (0x00007fc4d9736000)
	libpng12.so.0 => /lib/x86_64-linux-gnu/libpng12.so.0 (0x00007fc4d9511000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fc4d92f7000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fc4dfca7000)
	libswresample-ffmpeg.so.1 => /usr/lib/x86_64-linux-gnu/libswresample-ffmpeg.so.1 (0x00007fc4d90da000)
	libva.so.1 => /usr/lib/x86_64-linux-gnu/libva.so.1 (0x00007fc4d8ebe000)
	libzvbi.so.0 => /usr/lib/x86_64-linux-gnu/libzvbi.so.0 (0x00007fc4d8c33000)
	libxvidcore.so.4 => /usr/lib/x86_64-linux-gnu/libxvidcore.so.4 (0x00007fc4d891f000)
	libx265.so.79 => /usr/lib/x86_64-linux-gnu/libx265.so.79 (0x00007fc4d7d00000)
	libx264.so.148 => /usr/lib/x86_64-linux-gnu/libx264.so.148 (0x00007fc4d795c000)
	libwebp.so.5 => /usr/lib/x86_64-linux-gnu/libwebp.so.5 (0x00007fc4d7700000)
	libwavpack.so.1 => /usr/lib/x86_64-linux-gnu/libwavpack.so.1 (0x00007fc4d74d7000)
	libvpx.so.3 => /usr/lib/x86_64-linux-gnu/libvpx.so.3 (0x00007fc4d70b3000)
	libvorbisenc.so.2 => /usr/lib/x86_64-linux-gnu/libvorbisenc.so.2 (0x00007fc4d6e0a000)
	libvorbis.so.0 => /usr/lib/x86_64-linux-gnu/libvorbis.so.0 (0x00007fc4d6bdf000)
	libtwolame.so.0 => /usr/lib/x86_64-linux-gnu/libtwolame.so.0 (0x00007fc4d69bc000)
	libtheoraenc.so.1 => /usr/lib/x86_64-linux-gnu/libtheoraenc.so.1 (0x00007fc4d677d000)
	libtheoradec.so.1 => /usr/lib/x86_64-linux-gnu/libtheoradec.so.1 (0x00007fc4d6563000)
	libspeex.so.1 => /usr/lib/x86_64-linux-gnu/libspeex.so.1 (0x00007fc4d634a000)
	libsnappy.so.1 => /usr/lib/x86_64-linux-gnu/libsnappy.so.1 (0x00007fc4d6142000)
	libshine.so.3 => /usr/lib/x86_64-linux-gnu/libshine.so.3 (0x00007fc4d5f35000)
	libschroedinger-1.0.so.0 => /usr/lib/x86_64-linux-gnu/libschroedinger-1.0.so.0 (0x00007fc4d5c60000)
	libopus.so.0 => /usr/lib/x86_64-linux-gnu/libopus.so.0 (0x00007fc4d5a16000)
	libopenjpeg.so.5 => /usr/lib/x86_64-linux-gnu/libopenjpeg.so.5 (0x00007fc4d57f3000)
	libmp3lame.so.0 => /usr/lib/x86_64-linux-gnu/libmp3lame.so.0 (0x00007fc4d557e000)
	libgsm.so.1 => /usr/lib/x86_64-linux-gnu/libgsm.so.1 (0x00007fc4d5370000)
	libcrystalhd.so.3 => /usr/lib/x86_64-linux-gnu/libcrystalhd.so.3 (0x00007fc4d5155000)
	liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fc4d4f33000)
	libssh-gcrypt.so.4 => /usr/lib/x86_64-linux-gnu/libssh-gcrypt.so.4 (0x00007fc4d4ce9000)
	librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007fc4d4acd000)
	libmodplug.so.1 => /usr/lib/x86_64-linux-gnu/libmodplug.so.1 (0x00007fc4d4742000)
	libgme.so.0 => /usr/lib/x86_64-linux-gnu/libgme.so.0 (0x00007fc4d44f4000)
	libbluray.so.1 => /usr/lib/x86_64-linux-gnu/libbluray.so.1 (0x00007fc4d42ab000)
	libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007fc4d3f7b000)
	libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007fc4d3d6b000)
	libsoxr.so.0 => /usr/lib/x86_64-linux-gnu/libsoxr.so.0 (0x00007fc4d3b06000)
	libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007fc4d38fb000)
	libogg.so.0 => /usr/lib/x86_64-linux-gnu/libogg.so.0 (0x00007fc4d36f2000)
	liborc-0.4.so.0 => /usr/lib/x86_64-linux-gnu/liborc-0.4.so.0 (0x00007fc4d3472000)
	libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007fc4d3191000)
	libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007fc4d2f47000)
	libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007fc4d2d14000)
	libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007fc4d2ade000)
	libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007fc4d285e000)
	libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007fc4d24a3000)
	libfontconfig.so.1 => /usr/lib/x86_64-linux-gnu/libfontconfig.so.1 (0x00007fc4d2260000)
	libfreetype.so.6 => /usr/lib/x86_64-linux-gnu/libfreetype.so.6 (0x00007fc4d1fb6000)
	libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007fc4d1d52000)
	libidn.so.11 => /usr/lib/x86_64-linux-gnu/libidn.so.11 (0x00007fc4d1b1f000)
	libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007fc4d190c000)
	libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007fc4d16ea000)
	libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007fc4d14d6000)
	libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007fc4d1204000)
	libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007fc4d0fd5000)
	libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007fc4d0dd1000)
	libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007fc4d0bc6000)
	libicuuc.so.55 => /usr/lib/x86_64-linux-gnu/libicuuc.so.55 (0x00007fc4d0832000)
	libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007fc4d0609000)
	libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007fc4d0401000)
	libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007fc4d01fd000)
	libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007fc4cffe2000)
	libicudata.so.55 => /usr/lib/x86_64-linux-gnu/libicudata.so.55 (0x00007fc4ce52b000)

没有出现not found的情况就是ok的，至于为什么要这么做，可以查看一下.so文件的作用，我不太懂。
将所有节点都拷贝了libopencv_java451.so以后就可以编写mapreduce的代码了，

2.编写mapreduce代码

在mapreduce的编写过程中，三个自定义类（Runner类，Mapper类，Reduce类）在一开始就要添加一句：

//以下语句十分重要，若是不填，则在Linux下会Error: org.opencv.core.Mat.n_Mat(III)J
        System.loadLibrary("opencv_java451");

Hadoop 存储照片 hadoop处理图片_大数据_05


如果reduce类不添加System.loadLibrary(“opencv_java451”);会报错

即：

Hadoop 存储照片 hadoop处理图片_mapreduce_06

在windows下还是要安装opencv（这部分内容可以寻找其他资料，很多，包括引入jar包，导入xxx.dll，比在Linux环境下部署要简单的多），然后导入到IntelliJ IDEA 里才能顺利编写代码、如果不出意外的话，mapreduce代码编写完毕，打成jar包，传到集群上运行后即可。

Hadoop 存储照片 hadoop处理图片_大数据_07


在这里我是利用opencv将hdfs上的图片灰度化，并存在hdfs上

总结

很少有人用hadoop结合opencv去处理图片，以至于找不到什么hadoop+opencv的部署方法，我也是尝试了好久才将mapreduce与opencv联系起来，总而言之还是自己太菜，对很多东西还不理解，很多东西还没熟悉，以上说的很多东西如果有出错的地方，等我遇到问题解决后再改。