前面已经简单介绍了,在Android系统中open camera的流程,但是,它又是怎么预览、怎么配置流,如何最终操作到camera HAL的呢。接下来以android原生相机应用,android9,API2,camera HAL3为例,继续阅读代码,看看预览过程到底进行了什么操作?

阅读代码前,先看图。

android 相机预览界面矩形内文件自动成像 手机相机预览界面在哪_预览

configureStreams 操作

APP — 从open camera说起

在 Camera2 相机应用,打开相机应用将会调用到 Camera2OneCameraOpenerImpl 类 [packages/apps/Camera2/src/com/android/camera/one/v2/Camera2OneCameraOpenerImpl.java] 的 open() 方法,在该方法中,将会通过 CameraManager 对象的 openCamera() 方法打开camera设备。打开流程就不再述说了,可以参考 Android openCamera流程 ,在这里主要看看该方法的第二个参数 CameraDevice.StateCallback()。当camera成功打开之后,将会调用到 StateCallback 的 onOpened() 方法,而该方法将调用 OneCameraCreator 类的 create() 方法。在 OneCameraCreator::create() 中,将会根据应用层的设置,填充 PictureSizeCalculator.Configuration,将包含分辨率、格式、裁剪的大小等信息。最后在 onOpened() 方法中将会通过 openCallback.onCameraOpened(oneCamera) ,继续将成功打开设备的信息回调到上层。这里将调到 CaptureModule 类 [packages/apps/Camera2/src/com/android/camera/CaptureModule.java] 的 onCameraOpened() 方法。在该方法中,将会进行预览操作 camera.startPreview()。camera.startPreview() 方法将调用 GenericOneCameraImpl 类的 startPreview() 方法,在该方法中,又将通过 mPreviewStarter.startPreview(surface) 调用 PreviewStarter 类的 startPreview() 方法进行预览。在 PreviewStarter::startPreview() 方法中,将通过 mCaptureSessionCreator.createCaptureSession(surfaceList) 调用 CaptureSessionCreator::createCaptureSession() 方法,而在 createCaptureSession() 方法中,又将调用 AndroidCameraDeviceProxy::createCaptureSession(),最后在该方法中,通过 mCameraDevice.createCaptureSession() 调用到 Framework 代码。

Framework

CameraDeviceImpl

在上述的 mCameraDevice.createCaptureSession() 接口将会调用到哪里呢?看看open camera时是如何赋值的。

在open camera的流程中,将会调用到 CameraManager::openCameraDeviceUserAsync() 方法,该方法如下:

private CameraDevice openCameraDeviceUserAsync(String cameraId,
            CameraDevice.StateCallback callback, Executor executor, final int uid) 
            throws CameraAccessException {
        CameraDevice device = null;

            android.hardware.camera2.impl.CameraDeviceImpl deviceImpl =
                    new android.hardware.camera2.impl.CameraDeviceImpl(
                        cameraId,
                        callback,
                        executor,
                        characteristics,
                        mContext.getApplicationInfo().targetSdkVersion);
        ...
            device = deviceImpl;
        return device;
    }

从 openCameraDeviceUserAsync() 的实现可以知道返回给应用的是 CameraDeviceImpl 实例对象,所以当应用通过 mCameraDevice.createCaptureSession() 将会调用到 CameraDeviceImpl::createCaptureSession() 方法,从这里,进入了Framework代码。

而 CameraDeviceImpl::createCaptureSession() 方法[android/frameworks/base/core/java/android/hardware/camera2/impl/CameraDeviceImpl.java],是用来创建预览session,输出位置是 Surface,会有两个 Surface,一个现实预览画面的 Surface,一个接收预览数据的 Surface。

@Override
    public void createCaptureSession(List<Surface> outputs,
            CameraCaptureSession.StateCallback callback, Handler handler)
            throws CameraAccessException {
        List<OutputConfiguration> outConfigurations = new ArrayList<>(outputs.size());
        for (Surface surface : outputs) {
            outConfigurations.add(new OutputConfiguration(surface));
        }
        createCaptureSessionInternal(null, outConfigurations, callback,
                checkAndWrapHandler(handler), /*operatingMode*/ICameraDeviceUser.NORMAL_MODE,
                /*sessionParams*/ null);
    }

在 createCaptureSessionInternal() 方法中,将会通过 configureStreamsChecked(inputConfig, outputConfigurations, operatingMode, sessionParams) 配置流。在 configureStreamsChecked() 方法中,主要是通过 mRemoteDevice 操作到 CameraService 中的CameraDeviceClient。(同样的在 openCameraDeviceUserAsync() 方法中,connectDevice()返回的CameraDeviceClient,通过CameraDeviceImpl::setRemoteDevice()封装保存到 mRemoteDevice,所以,通过 mRemoteDevice 将会操作到CameraService中的具体函数。)

而 configureStreamsChecked() 方法的操作,可以概括如下:

public boolean configureStreamsChecked(InputConfiguration inputConfig,
            List<OutputConfiguration> outputs, int operatingMode, CaptureRequest sessionParams)
                    throws CameraAccessException {
		...
                mRemoteDevice.beginConfigure();
        ...
		/* 注意传递进来的方法参数,inputConfig为null,sessionParams为null */
                // Add all new streams
                for (OutputConfiguration outConfig : outputs) {
                    if (addSet.contains(outConfig)) {
                        /* 将会在这里创建并配置输出流,该流输出显示,
                         * 注意 outConfig 中有显示 surface
                         */
                        int streamId = mRemoteDevice.createStream(outConfig);
                        mConfiguredOutputs.put(streamId, outConfig);
                    }
                }
        ...
                if (sessionParams != null) {
                    mRemoteDevice.endConfigure(operatingMode, sessionParams.getNativeCopy());
                } else {
                    /* 上述只是保存配置,这里才会真正的设置到 HAL */
                    mRemoteDevice.endConfigure(operatingMode, null);
                }

我们主要分析 mRemoteDevice.createStream(outConfig) 的操作,在这里,将会通过Binder将相应的配置信息传输到 CameraService 端的CameraDeviceClient。

CameraService — frameworks/av/services/camera/libcameraservice

在通过Binder之后,CameraDeviceImpl 操作的 createStream() 将会由CameraService调用 CameraDeviceClient::createStream() 函数并返回,注意 CameraDeviceClient::createStream() 有两个函数参数,但是第二个可以默认为NULL。

binder::Status CameraDeviceClient::createStream(
        const hardware::camera2::params::OutputConfiguration &outputConfiguration,
        /*out*/
        int32_t* newStreamId) {
    const std::vector<sp<IGraphicBufferProducer>>& bufferProducers =
            outputConfiguration.getGraphicBufferProducers();

    ...

    OutputStreamInfo streamInfo;
    bool isStreamInfoValid = false;
    
    ...

        sp<Surface> surface;
    	/* 将OutputConfiguration中的配置信息转换保存到streamInfo */
        res = createSurfaceFromGbp(streamInfo, isStreamInfoValid, surface, bufferProducer);
    ...
        /* 将 outputConfiguration 中的 surface 提取出来并添加到 surfaces */
        surfaces.push_back(surface);
    ...

    int streamId = camera3::CAMERA3_STREAM_ID_INVALID;
    std::vector<int> surfaceIds;
    /* 这里将传递 surfaces 作为参数创建流 */
    /* 这里又调用device的 createStream() 函数,那么这里又是调用到了哪里呢,看下面的解析 */
    err = mDevice->createStream(surfaces, deferredConsumer, streamInfo.width,
            streamInfo.height, streamInfo.format, streamInfo.dataSpace,
            static_cast<camera3_stream_rotation_t>(outputConfiguration.getRotation()),
            &streamId, physicalCameraId, &surfaceIds, outputConfiguration.getSurfaceSetID(),
            isShared);
    
    ...
		/* 保存相应的信息 */
        mStreamMap.add(binder, StreamSurfaceId(streamId, surfaceIds[i]));
    
        mConfiguredOutputs.add(streamId, outputConfiguration);
        mStreamInfoMap[streamId] = streamInfo;
}

上面的 mDevice->createStream() 将会调用到哪里呢?

CameraDeviceClient 类是继承模板类 Camera2ClientBase,看 mDevice 的定义,我们可以知道是 sp<CameraDeviceBase> mDevice,mDevice 的赋值是在模板类Camera2ClientBase 的构造函数中,mDevice = new Camera3Device(cameraId),通过其构造函数可以知道 mDevice 指向 Camera3Device 实例对象。所以,mDevice->createStream() 将调用到 Camera3Device::createStream() 函数。

status_t Camera3Device::createStream(const std::vector<sp<Surface>>& consumers,
        bool hasDeferredConsumer, uint32_t width, uint32_t height, int format,
        android_dataspace dataSpace, camera3_stream_rotation_t rotation, int *id,
        const String8& physicalCameraId,
        std::vector<int> *surfaceIds, int streamSetId, bool isShared, uint64_t consumerUsage) {
	...
    } else {
    	/* 创建 Camera3OutputStream 实例对象,注意第二个参数,它是 Surface,
    	 * 将保存在 Camera3OutputStream 的 mConsumer 成员 */
        newStream = new Camera3OutputStream(mNextStreamId, consumers[0],
                width, height, format, dataSpace, rotation,
                mTimestampOffset, physicalCameraId, streamSetId);
    }

	...

	/* newStream 实例对象保存到 mOutputStreams */
    res = mOutputStreams.add(mNextStreamId, newStream);

	mNeedConfig = true;

	...
}

到这里为止,似乎 CameraDeviceClient::createStream() 函数已经跟踪完了,最后只是看到保存一些stream的配置信息,并没有设置到 HAL,那么,导致是在哪里设置的呢,我们回头看看 CameraDeviceImpl::configureStreamsChecked() 方法,在该方法中,通过 mRemoteDevice 操作影响到 CameraService,在 createStream() 之后,还将会调用 endConfigure() 操作,既然 createStream() 并没有设置到 HAL,那么,有没有可能是 endConfigure() 操作呢,我们来看看 CameraDeviceClient::endConfigure() 实现。

binder::Status CameraDeviceClient::endConfigure(int operatingMode,
        const hardware::camera2::impl::CameraMetadataNative& sessionParams) {
    ...
	/* 和上述一样,将会调用到 Camera3Device::configureStreams() */
    status_t err = mDevice->configureStreams(sessionParams, operatingMode);
    ...
}

而在 Camera3Device::configureStreams() 中,最终又将是调用到 Camera3Device::configureStreamsLocked() 函数。而在 Camera3Device::configureStreamsLocked() 函数主要进行了以下操作:

  • 将 Camera3Device::createStream() 保存到 mOutputStreams 中的 stream 添加到 config.streams;
  • 通过 mInterface->configureStreams(sessionBuffer, &config, bufferSizes) 操作到HAL配置流;
  • 通知上层,配置完成;

上述的 mInterface 变量是在 Camera3Device::initialize() 时,保存 HalInterface 实例对象的,所以,mInterface->configureStreams() 将调用到 Camera3Device::HalInterface::configureStreams()。

Camera3Device::HalInterface::configureStreams() 将stream config转换为 HIDL 接口使用的之后,将按照 HAL 版本调用不同的接口函数,下面我们以 HAL 3.3 为例进行跟踪,将调用 mHidlSession_3_3->configureStreams_3_3()。注:mHidlSession_3_3 是在 HalInterface 构造函数中赋值的,而 mHidlSession_3_3 是 ICameraDeviceSession转换得来的,ICameraDeviceSession 则是Camera3Device::initialize() 中通过 manager->openSession() 获取的。

在上述的 mHidlSession_3_3->configureStreams_3_3() 调用,将会调用到 BpHwCameraDeviceSession::configureStreams_3_3() [android/out/soong/.intermediates/hardware/interfaces/camera/device/3.3/android.hardware.camera.device@3.3_genc++/gen/android/hardware/camera/device/3.3/CameraDeviceSessionAll.cpp],最终调用到 CameraProvider 中的 CameraDeviceSession::configureStreams_3_3()函数。具体怎么调用到这里的就不详细跟踪,知道是通过HIDL影响到这边即可,感兴趣是怎么调用到这边的可以继续跟踪下去。

所以,以上,在 Camera3Device::configureStreamsLocked() 中,将会通过 mInterface->configureStreams(sessionBuffer, &config, bufferSizes) 调用到 CameraProvider,从而配置流操作进入 CameraProvider。

CameraProvider — hardware/interfaces/camera

上述已经说到,在CameraService 通过 mInterface->configureStreams(sessionBuffer, &config, bufferSizes) 函数将配置信息通过HIDL传递到 CameraProvider,将调用 CameraDeviceSession::configureStreams_3_3() 函数。

Return<void> CameraDeviceSession::configureStreams_3_3(
        const StreamConfiguration& requestedConfiguration,
        ICameraDeviceSession::configureStreams_3_3_cb _hidl_cb)  {
	...
	/* 在完成函数参数的转换之后,将通过 mDevice 操作到 HAL 的 configure_streams() */
    status_t ret = mDevice->ops->configure_streams(mDevice, &stream_list);
    ...
}

至于为什么会可以通过 mDevice->ops->configure_streams() 操作到HAL呢?

首先,通过代码,我们了解到 mDevice 是在 CameraDeviceSession 构造函数中初始化的,而 CameraDeviceSession 是在 CameraDevice::createSession() 中实例化的。而 createSession() 则是在 open阶段,在 CameraDevice::open() 函数中调用的,跟踪到这里,可以知道,mDevice 是在 CameraDevice::open() 函数中,通过 mModule->open() 获取再赋值的。而 mModule->open() 函数返回的 device 则是在 HAL 中的 Camera::openDevice() 赋值的,该 ops 为 camera3_device_ops_t。

所以,到这里,完成了 configure streams 操作。完成了流的配置之后,又是怎么预览的呢,继续看。

预览操作

回到 APP 的 PreviewStarter

从APP的代码我们可以知道,在创建 PreviewStarter 实例对象时,有一个 onCameraCaptureSessionCreated() 回调函数,当session创建成功时,将会执行这个函数,而从这个函数,最终将进行预览操作,流程如下:

onCameraCaptureSessionCreated()
	cameraStarter.startCamera()
		SimpleOneCameraFactory::startCamera()
			BasicPreviewCommandFactory() preview命令
				CameraDeviceRequestBuilderFactory::create()
					AndroidCameraDeviceProxy::createCaptureRequest()

同理,在APP中,通过 mCameraDevice.createCaptureRequest() 将会调用到 CameraDeviceImpl::createCaptureRequest() 方法,从这里,进入了Framework代码。

Framework

CameraDeviceImpl
@Override
    public CaptureRequest.Builder createCaptureRequest(int templateType)
            throws CameraAccessException {
        synchronized(mInterfaceLock) {
            checkIfCameraClosedOrInError();
    
            CameraMetadataNative templatedRequest = null;
    
            /* 一样的,也是通过 mRemoteDevice 代理操作 CameraDeviceClient,
             * 将调用到 CameraDeviceClient::createDefaultRequest() */
            templatedRequest = mRemoteDevice.createDefaultRequest(templateType);
    
            ...
    
            return builder;
        }
    }
CameraService — frameworks/av/services/camera/libcameraservice
CameraDeviceClient
// Create a request object from a template.
binder::Status CameraDeviceClient::createDefaultRequest(int templateId,
        /*out*/
        hardware::camera2::impl::CameraMetadataNative* request)
{
    ...
 
    CameraMetadata metadata;
    status_t err; 
    /* 与 createStream() 相同,将会调用到 Camera3Device::createDefaultRequest() */
    if ( (err = mDevice->createDefaultRequest(templateId, &metadata) ) == OK &&                                                                                                                              
        request != NULL) {
 
        request->swap(metadata);
    }
    ... 
}
Camera3Device
status_t Camera3Device::createDefaultRequest(int templateId,
        CameraMetadata *request) {
	...
    camera_metadata_t *rawRequest;
    /* 又将通过 mInterface 调用到 CameraProvider*/
    status_t res = mInterface->constructDefaultRequestSettings(
            (camera3_request_template_t) templateId, &rawRequest);
    
	{
		...
		/* 保存这个类型请求的信息 */
        set_camera_metadata_vendor_id(rawRequest, mVendorTagId);
        mRequestTemplateCache[templateId].acquire(rawRequest);
        ...
    }
}
CameraProvider — hardware/interfaces/camera

上述的操作,最后将按照HAL版本的差异,调用不同的HAL接口函数,HAL3.3将调用 device::V3_3::CameraDeviceSession::constructDefaultRequestSettings(),这个也将调用到 device::V3_2::CameraDeviceSession::constructDefaultRequestSettings()。

// Methods from ::android::hardware::camera::device::V3_2::ICameraDeviceSession follow.
Return<void> CameraDeviceSession::constructDefaultRequestSettings(                      
        RequestTemplate type, ICameraDeviceSession::constructDefaultRequestSettings_cb _hidl_cb)  {
    CameraMetadata outMetadata;
    Status status = constructDefaultRequestSettingsRaw( (int) type, &outMetadata);
    _hidl_cb(status, outMetadata);
    return Void();
}

Status CameraDeviceSession::constructDefaultRequestSettingsRaw(int type, CameraMetadata *outMetadata) {
	...
       /* 这里将调用到相应平台 HAL 代码,完成这个操作之后,相机设备硬件已经开始输出图像数据了 */ 
        rawRequest = mDevice->ops->construct_default_request_settings(mDevice, (int) type);
    ...
}

图像开始输出,又是怎么通过 SurfaceFlinger 最终显示出来的呢?

从APP的预览命令看起
public class PreviewCommand implements CameraCommand {

	public void run() throws InterruptedException, CameraAccessException,
            CameraCaptureSessionClosedException, ResourceAcquisitionFailedException {
        try (FrameServer.Session session = mFrameServer.createExclusiveSession()) {
            /* 这个就是之前的 CameraDeviceRequestBuilderFactory::create() 调用处 */
            RequestBuilder photoRequest = mBuilderFactory.create(mRequestType);
            /* 请求图像数据 */
            session.submitRequest(Arrays.asList(photoRequest.build()),
                    FrameServer.RequestType.REPEATING);
        }       
    } 
}

上述的 session.submitRequest() 操作,最终将经过 FrameServerImpl::submitRequest() 调用到 Framework 的 ICameraDeviceUserWrapper::submitRequest()。

Framework
public class ICameraDeviceUserWrapper {
    ...
	public SubmitInfo submitRequest(CaptureRequest request, boolean streaming)
            throws CameraAccessException  {
        try {
            /* 调用到 CameraService 的 CameraDeviceClient */
            return mRemoteDevice.submitRequest(request, streaming);
        } catch (Throwable t) {
            CameraManager.throwAsPublicException(t);
            throw new UnsupportedOperationException("Unexpected exception", t);                                                                                                                              
        }   
    } 
}
CameraService
[frameworks/av/services/camera/libcameraservice/api2/CameraDeviceClient.cpp]
binder::Status CameraDeviceClient::submitRequest(           
        const hardware::camera2::CaptureRequest& request,
        bool streaming,
        /*out*/
        hardware::camera2::utils::SubmitInfo *submitInfo) {
    std::vector<hardware::camera2::CaptureRequest> requestList = { request };
    return submitRequestList(requestList, streaming, submitInfo);
}

binder::Status CameraDeviceClient::submitRequestList(
        const std::vector<hardware::camera2::CaptureRequest>& requests,
        bool streaming,
        /*out*/
        hardware::camera2::utils::SubmitInfo *submitInfo) {
	...
        } else {
            for (size_t i = 0; i < request.mStreamIdxList.size(); i++) {
                int streamId = request.mStreamIdxList.itemAt(i);
                int surfaceIdx = request.mSurfaceIdxList.itemAt(i);

                ssize_t index = mConfiguredOutputs.indexOfKey(streamId);
                
                const auto& gbps = mConfiguredOutputs.valueAt(index).getGraphicBufferProducers();
                
                /* 获取 surfaceMap 等信息 */
                res = insertGbpLocked(gbps[surfaceIdx], &surfaceMap, &outputStreamIds, nullptr);
                ...
		}
    	...
		/* 添加到 surfaceMapList */
        surfaceMapList.push_back(surfaceMap);
    	...
    } else {
    	/* 调用到 CameraDevice3 的 captureList() */
        err = mDevice->captureList(metadataRequestList, surfaceMapList,
                &(submitInfo->mLastFrameNumber));
	}
}

[frameworks/av/services/camera/libcameraservice/device3/Camera3Device.cpp]
status_t Camera3Device::captureList(const List<const PhysicalCameraSettingsList> &requestsList,
                                    const std::list<const SurfaceMap> &surfaceMaps,
                                    int64_t *lastFrameNumber) {

    return submitRequestsHelper(requestsList, surfaceMaps, /*repeating*/false, lastFrameNumber);
}

status_t Camera3Device::submitRequestsHelper(
        const List<const PhysicalCameraSettingsList> &requests,
        const std::list<const SurfaceMap> &surfaceMaps,
        bool repeating,
        /*out*/
        int64_t *lastFrameNumber) {
	...
	/* 转换信息,并指定 surface id */
    res = convertMetadataListToRequestListLocked(requests, surfaceMaps,
            repeating, /*out*/&requestList);
    ...
    } else {
    	/* 添加信息到 mRequestThread 线程的 mRequestQueue 队列,
    	 * 该线程在open camera时已经运行,在该函数中,将调用
         * Camera3Device::RequestThread::unpauseForNewRequests()
         * 函数发送信号量告知 mRequestThread 线程 */
        res = mRequestThread->queueRequestList(requestList, lastFrameNumber);
    }
	...
}

/* 通过该函数将会获取得到 surface id */
status_t Camera3Device::convertMetadataListToRequestListLocked(
        const List<const PhysicalCameraSettingsList> &metadataList,
        const std::list<const SurfaceMap> &surfaceMaps,
        bool repeating,
        RequestList *requestList) {
	...
    List<const PhysicalCameraSettingsList>::const_iterator metadataIt = metadataList.begin();
    std::list<const SurfaceMap>::const_iterator surfaceMapIt = surfaceMaps.begin();
    /* 逐个转换处理 */
    for (; metadataIt != metadataList.end() && surfaceMapIt != surfaceMaps.end();
            ++metadataIt, ++surfaceMapIt) {
        /* 将会通过 setUpRequestLocked() 创建 CaptureRequest */
        sp<CaptureRequest> newRequest = setUpRequestLocked(*metadataIt, *surfaceMapIt);
        ...
        requestList->push_back(newRequest);
        ...
    }
    ...
}

sp<Camera3Device::CaptureRequest> Camera3Device::setUpRequestLocked(
        const PhysicalCameraSettingsList &request, const SurfaceMap &surfaceMap) {
    ...
	/* 实质是调用 createCaptureRequest() 创建的请求 */
    sp<CaptureRequest> newRequest = createCaptureRequest(request, surfaceMap);
    return newRequest;
}

sp<Camera3Device::CaptureRequest> Camera3Device::createCaptureRequest(
        const PhysicalCameraSettingsList &request, const SurfaceMap &surfaceMap) {
    sp<CaptureRequest> newRequest = new CaptureRequest;
    
    ...
    for (size_t i = 0; i < streams.count; i++) {
        sp<Camera3OutputStreamInterface> stream =
                mOutputStreams.editValueAt(idx);

        // It is illegal to include a deferred consumer output stream into a request
        auto iter = surfaceMap.find(streams.data.i32[i]);
        if (iter != surfaceMap.end()) {
            const std::vector<size_t>& surfaces = iter->second;
            ...
			/* 这里,获取到这次请求的 surfaces id 并保存到
			 * mOutputSurfaces,接下来将会用到 */
            newRequest->mOutputSurfaces[i] = surfaces;
            ...
        }
    }
}

在 mRequestThread 线程中,操作如下:

bool Camera3Device::RequestThread::threadLoop() {
    /* 等待请求信息 */
    waitForNextRequestBatch();
    
    ...
        
    /* 处理请求并申请output buffer,以待 HAL 填充图像数据 */
    res = prepareHalRequests();
    
    ...
        
    if (mInterface->supportBatchRequest()) {
        submitRequestSuccess = sendRequestsBatch();
    } else {
        /* 发送请求到 HAL */
        submitRequestSuccess = sendRequestsOneByOne();
    }
    ...
}

那么 prepareHalRequests() 进行了什么操作呢,接着往下看。

status_t Camera3Device::RequestThread::prepareHalRequests() {
    for (size_t i = 0; i < mNextRequests.size(); i++) {
        /* 获取上层传递下来的请求信息 */
        auto& nextRequest = mNextRequests.editItemAt(i);
        sp<CaptureRequest> captureRequest = nextRequest.captureRequest;
        Vector<camera3_stream_buffer_t>* outputBuffers = &nextRequest.outputBuffers;
        
        ...
        for (size_t j = 0; j < captureRequest->mOutputStreams.size(); j++) {
            sp<Camera3OutputStreamInterface> outputStream = captureRequest->mOutputStreams.editItemAt(j);
            ...
			/* 这里将会根据当前请求配置的 surfaces id 获取surfaces buffer */
            res = outputStream->getBuffer(&outputBuffers->editItemAt(j),
                    captureRequest->mOutputSurfaces[j]);
            ...
        }
    }
}

status_t Camera3Stream::getBuffer(camera3_stream_buffer *buffer,
        const std::vector<size_t>& surface_ids) {
    ...
    res = getBufferLocked(buffer, surface_ids);
    ...
}

status_t Camera3OutputStream::getBufferLocked(camera3_stream_buffer *buffer,
        const std::vector<size_t>&) {
    ...
    res = getBufferLockedCommon(&anb, &fenceFd);
    ...
}

status_t Camera3OutputStream::getBufferLockedCommon(ANativeWindowBuffer** anb, int* fenceFd) {
    ...
        /* 这个 mConsumer 是在创建 Camera3OutputStream 实例对象时传递进来的参数,
         * 而这个实例对象是在 Camera3Device::createStream() 时创建的,可以看看上面
         * 的章节,mConsumer 是在创建流时传递下来的 Surface */
        sp<ANativeWindow> currentConsumer = mConsumer;
    ...
        /* 这里将申请buf,待HAL填充数据 */
		/* 直接调用 ANativeWindow 对象的 dequeueBuffer 方法将申请 buffer,
		 * ANativeWindow 是 OpenGL 定义的图形接口,在 Android上 的实现就是
     	 * Surface 和 SurfaceFlinger,一个用于生产 buffer ,一个用于消费 buffer */
        res = currentConsumer->dequeueBuffer(currentConsumer.get(), anb, fenceFd);
    ...
}

所以,通过上述的跟踪,了解到 prepareHalRequests() 是进行请求信息填充以及buffer申请等。

在 mRequestThread 线程中,获取请求信息之后,经过转换,将会通过 submitRequestSuccess = sendRequestsOneByOne() 经过CameraProvider提交请求信息到 HAL。

bool Camera3Device::RequestThread::sendRequestsOneByOne() {
    ...
    for (auto& nextRequest : mNextRequests) {
        /* 将请求信息传递到 CameraProvider 处理 */
        res = mInterface->processCaptureRequest(&nextRequest.halRequest);
        ...
    }
}
CameraProvider
Return<void> CameraDeviceSession::processCaptureRequest(
        const hidl_vec<CaptureRequest>& requests,
        const hidl_vec<BufferCache>& cachesToRemove,
        ICameraDeviceSession::processCaptureRequest_cb _hidl_cb)  {
    updateBufferCaches(cachesToRemove);
 
    uint32_t numRequestProcessed = 0; 
    Status s = Status::OK;
    for (size_t i = 0; i < requests.size(); i++, numRequestProcessed++) {
        s = processOneCaptureRequest(requests[i]);
        if (s != Status::OK) {
            break;
        }
    }
 
    ....
        
    return Void();
}

Status CameraDeviceSession::processOneCaptureRequest(const CaptureRequest& request)  {

    /* 在该函数中,经过前面一系列的参数转换之后,将调用 HAL 的
     * process_capture_request() 函数,将请求添加到 HAL 队列
     */
    status_t ret = mDevice->ops->process_capture_request(mDevice, &halRequest);
  
}
从源头HAL回调

首先的,Camera HAL通过 VIDIOC_DQBUF 拿到图像数据之后,将会通过 mCallbackOps->notify()、mCallbackOps->process_capture_result() 回调通知 CameraProvider,而 mCallbackOps 是 CameraProvider 在调用 CameraDeviceSession::initialize() 时传递 this 指针下来赋值的,而 process_capture_result() 指针指向implementation 的 CameraDeviceSession::sProcessCaptureResult(),notify() 指向implementation 的 CameraDeviceSession::sNotify()。

typedef struct camera3_capture_result {

    uint32_t frame_number;

    const camera_metadata_t *result;

    uint32_t num_output_buffers;

    const camera3_stream_buffer_t *output_buffers;

    const camera3_stream_buffer_t *input_buffer;

    uint32_t partial_result;

    uint32_t num_physcam_metadata;

    const char **physcam_ids;

    const camera_metadata_t **physcam_metadata;

} camera3_capture_result_t;

一般的,camera HAL中,在获取图像数据之后,图像数据信息拷贝到 request->output_buffers[0] ,接着将调用 mCallbackOps->notify() 发送请求完成的消息到 CameraProvider,再调用 mCallbackOps->process_capture_result() 发送请求返回的相应数据信息。

void Camera::notifyShutter(uint32_t frame_number, uint64_t timestamp)
{
    camera3_notify_msg_t message;
    memset(&message, 0, sizeof(message));
    message.type = CAMERA3_MSG_SHUTTER;
    message.message.shutter.frame_number = frame_number;
    message.message.shutter.timestamp = timestamp;
    mCallbackOps->notify(mCallbackOps, &message);
}

void Camera::sendResult(std::shared_ptr<CaptureRequest> request) {
    // Fill in the result struct
    // (it only needs to live until the end of the framework callback).
    camera3_capture_result_t result {
        request->frame_number,
        request->settings.getAndLock(),
        static_cast<uint32_t>(request->output_buffers.size()),
        request->output_buffers.data(),
        request->input_buffer.get(),
        1  // Total result; only 1 part.
    };
    // Make the framework callback.
    mCallbackOps->process_capture_result(mCallbackOps, &result);
}

将会通过 process_capture_result() 函数指针调用 CameraProvider 中的 CameraDeviceSession::sProcessCaptureResult() 函数,而传递的函数参数就是 camera3_capture_result_t,从 camera3_capture_result_t 的定义我们可以知道,传递下来的信息有帧序号、图像数据信息等。

CameraProvider
CameraDeviceSession::sNotify()
void CameraDeviceSession::sNotify(
        const camera3_callback_ops *cb,
        const camera3_notify_msg *msg) {
    CameraDeviceSession *d =
            const_cast<CameraDeviceSession*>(static_cast<const CameraDeviceSession*>(cb));
    NotifyMsg hidlMsg;
    convertToHidl(msg, &hidlMsg);
    ...
	/* 在进行一些参数检查之后,将调用回调的 notify() 函数,
     * mResultBatcher 实际是在open camera时创建 CameraDeviceSession 
     * 实例对象时传递下来的回调函数,而通过阅读代码可以知道,这个mResultBatcher
     * 实质是 CameraService 端的 Camera3Device */
    d->mResultBatcher.notify(hidlMsg);
}

通过上面代码的理解,我们知道,CameraDeviceSession::sNotify() 最终将会把这个消息传递到 CameraService 的 Camera3Device 实例对象进行处理。

CameraDeviceSession::sProcessCaptureResult()
/**
 * Static callback forwarding methods from HAL to instance
 */
void CameraDeviceSession::sProcessCaptureResult(
        const camera3_callback_ops *cb,
        const camera3_capture_result *hal_result) {
    CameraDeviceSession *d =
            const_cast<CameraDeviceSession*>(static_cast<const CameraDeviceSession*>(cb));

    CaptureResult result = {};
    camera3_capture_result shadowResult;
    bool handlePhysCam = (d->mDeviceVersion >= CAMERA_DEVICE_API_VERSION_3_5);
    std::vector<::android::hardware::camera::common::V1_0::helper::CameraMetadata> compactMds;
    std::vector<const camera_metadata_t*> physCamMdArray;
    sShrinkCaptureResult(&shadowResult, hal_result, &compactMds, &physCamMdArray, handlePhysCam);

    /* 检查数据的完成性以及擦除之前提交请求信息时的一些参数信息 */
    status_t ret = d->constructCaptureResult(result, &shadowResult);
    if (ret == OK) {
        /* 一样的,调用到CameraService 的 Camera3Device */
        d->mResultBatcher.processCaptureResult(result);
    }
}
CameraService
notify()

在CameraProvider 中通过 notify() 传递到CameraService 之后,最终将调用到以下函数:

void Camera3Device::notify(const camera3_notify_msg *msg) {
    ATRACE_CALL();
    sp<NotificationListener> listener;
    {
        Mutex::Autolock l(mOutputLock);
        listener = mListener.promote();
    }

    switch (msg->type) {
        ...
        case CAMERA3_MSG_SHUTTER: {
            notifyShutter(msg->message.shutter, listener);
            break;
        }
        ...
    }
}

void Camera3Device::notifyShutter(const camera3_shutter_msg_t &msg,
        sp<NotificationListener> listener) {
        idx = mInFlightMap.indexOfKey(msg.frame_number);
        if (idx >= 0) {
            InFlightRequest &r = mInFlightMap.editValueAt(idx);
            
		...
            
            r.shutterTimestamp = msg.timestamp;
            if (r.hasCallback) {

                // Call listener, if any
                if (listener != NULL) {
                    /* CameraDeviceClient::notifyShutter() */
                    listener->notifyShutter(r.resultExtras, msg.timestamp);
                }
                // send pending result and buffers
                sendCaptureResult(r.pendingMetadata, r.resultExtras,
                    r.collectedPartialResult, msg.frame_number,
                    r.hasInputBuffer, r.physicalMetadatas);
            }
            returnOutputBuffers(r.pendingOutputBuffers.array(),
                r.pendingOutputBuffers.size(), r.shutterTimestamp);
            r.pendingOutputBuffers.clear();

            removeInFlightRequestIfReadyLocked(idx);

}

在 Camera3Device::notifyShutter() 中,将调用到 CameraDeviceClient::notifyShutter(),在该函数中,将会逐步经过 JAVA 层代码传递到 APP。

processCaptureResult()

在CameraProvider 中通过 processCaptureResult() 传递到CameraService 之后,最终将调用到以下函数:

/**
 * Camera HAL device callback methods
 */

void Camera3Device::processCaptureResult(const camera3_capture_result *result) {
    /* 在该函数中,还是进行取帧号的操作,isPartialResult是指部分,
     * 目前对Partial的具体含义还不是很了解,可能的情况比如拍HDR,需要
     * 采集三帧,三帧的FrameNumber相同,这三帧一起才能解析合成一帧图片,
     * 所以三帧中的每一帧就是Partial的意思了。(源自参考文章的理解)
     */
    ...
        if (shutterTimestamp == 0) {
            request.pendingOutputBuffers.appendArray(result->output_buffers,
                result->num_output_buffers);
        } else {
            /* 在这里,将会返回buf到 Surface 进行显示 */
            returnOutputBuffers(result->output_buffers,
                result->num_output_buffers, shutterTimestamp);
        }
    ...
            if (shutterTimestamp == 0) {
                request.pendingMetadata = result->result;
                request.collectedPartialResult = collectedPartialResult;
           } else if (request.hasCallback) {
                CameraMetadata metadata;
                metadata = result->result;
                /* 返回结果到 APP */
                sendCaptureResult(metadata, request.resultExtras,
                    collectedPartialResult, frameNumber,
                    hasInputBufferInRequest, request.physicalMetadatas);
            }
}
Camera3Device::returnOutputBuffers()

上面介绍到,在 Camera3Device::processCaptureResult() 中将调用到 returnOutputBuffers() 函数,而该函数的调用流程将如下:

Camera3Device::processCaptureResult() ---> stream->returnBuffer()
    Camera3Stream::returnBuffer() ---> returnBufferLocked()
    	Camera3OutputStream::returnBufferLocked() ---> returnAnyBufferLocked( , , true)
    		Camera3IOStreamBase::returnAnyBufferLocked() ---> returnBufferCheckedLocked()
    			Camera3OutputStream::returnBufferCheckedLocked()

下面我们来看看 Camera3OutputStream::returnBufferCheckedLocked() 进行了什么操作。

status_t Camera3OutputStream::returnBufferCheckedLocked(
            const camera3_stream_buffer &buffer,
            nsecs_t timestamp,
            bool output,
            /*out*/
            sp<Fence> *releaseFenceOut) {

    ...

    /* 这个 mConsumer 是在创建 Camera3OutputStream 实例对象时传递进来的参数,
     * 而这个实例对象是在 Camera3Device::createStream() 时创建的,可以看看上面
     * 的章节,mConsumer 是在创建流时传递下来的 Surface */
    sp<ANativeWindow> currentConsumer = mConsumer;
    mLock.unlock();

    ANativeWindowBuffer *anwBuffer = container_of(buffer.buffer, ANativeWindowBuffer, handle);
    /**
     * Return buffer back to ANativeWindow
     */
    ...
    } else {
        ...

        /* 将调用 queueBufferToConsumer() 归还 buffer,这时
         * buffer 已经经过ISP HAL等处理、填充完毕,可以进行显示了 */
        res = queueBufferToConsumer(currentConsumer, anwBuffer, anwReleaseFence);
        if (res != OK) {
            ALOGE("%s: Stream %d: Error queueing buffer to native window: "
                  "%s (%d)", __FUNCTION__, mId, strerror(-res), res);
        }
    }
    ...

    return res;
}

status_t Camera3OutputStream::queueBufferToConsumer(sp<ANativeWindow>& consumer,
            ANativeWindowBuffer* buffer, int anwReleaseFence) {
    /* 直接调用 ANativeWindow 对象的 queueBuffer 方法将 buffer 归还回去,
     * ANativeWindow 是 OpenGL 定义的图形接口,在 Android上 的实现就是
     * Surface 和 SurfaceFlinger,一个用于生产 buffer ,一个用于消费 buffer
     */
    return consumer->queueBuffer(consumer.get(), buffer, anwReleaseFence);
}

这样在 queueBuffer() 之后,buffer 将可以通过 Surface 显示出来了。

Camera3Device::sendCaptureResult()

了解到显示buffer的填充之后,我们再来看看,CameraService 又是如何通过 sendCaptureResult() 告知 APP 的。

void Camera3Device::sendCaptureResult(CameraMetadata &pendingMetadata,
        CaptureResultExtras &resultExtras,
        CameraMetadata &collectedPartialResult,
        uint32_t frameNumber,
        bool reprocess,
        const std::vector<PhysicalCaptureResultInfo>& physicalMetadatas) {

    ...
    /* 在完成 captureResult 的填充之后,将会把结果通过
     * insertResultLocked() 添加到 mResultQueue 队列中 */
    insertResultLocked(&captureResult, frameNumber);
}

void Camera3Device::insertResultLocked(CaptureResult *result,
        uint32_t frameNumber) {
    ...

    // Valid result, insert into queue
    List<CaptureResult>::iterator queuedResult =
            mResultQueue.insert(mResultQueue.end(), CaptureResult(*result));

    /* 添加完毕之后,发送信号 */
    mResultSignal.signal();
}

在 Camera3Device::insertResultLocked() 中的 mResultSignal 信号到底是发给谁呢?

通过搜索代码可以知道,在 FrameProcessorBase 帧处理线程中将会接收该信号。而帧处理线程是在 CameraDeviceClient 对象的 CameraDeviceClient::initializeImpl() 中启动的(open camera阶段调用到该函数)。

看看帧处理线程进行什么操作?

bool FrameProcessorBase::threadLoop() {
    status_t res;

    sp<CameraDeviceBase> device;
    {
        device = mDevice.promote();
        if (device == 0) return false;
    }

    /* 就是在这里等待 mResultSignal 信号 */
    res = device->waitForNextFrame(kWaitDuration);
    if (res == OK) {
        /* 处理帧 */
        processNewFrames(device);
    }
    
    ...
        
    return true;
}

void FrameProcessorBase::processNewFrames(const sp<CameraDeviceBase> &device) {
    CaptureResult result;
    
    /* 读取队列中的 captureResult 信息 */
    while ( (res = device->getNextResult(&result)) == OK) {
        ...
        if (!processSingleFrame(result, device)) {
            break;
        }
        ...
}

bool FrameProcessorBase::processSingleFrame(CaptureResult &result,
                                            const sp<CameraDeviceBase> &device) {

    return processListeners(result, device) == OK;
}
    
status_t FrameProcessorBase::processListeners(const CaptureResult &result,
        const sp<CameraDeviceBase> &device) {
    ...
    List<sp<FilteredListener> > listeners;
    {
        /* 获取监听并添加到 listeners,mRangeListeners 是通过
         * FrameProcessorBase::registerListener() 登记监听的,
         * 在 CameraDeviceClient::initializeImpl() 中创建
         * FrameProcessorBase 实例时,将会把 CameraDeviceClient
         * 作为监听者进行注册登记到 mRangeListeners */
        List<RangeListener>::iterator item = mRangeListeners.begin(); 
        while (item != mRangeListeners.end()) {
            ...
                            listeners.push_back(listener);
        }
    }
    
    List<sp<FilteredListener> >::iterator item = listeners.begin();
    for (; item != listeners.end(); item++) {
        /* 针对监听者,将会调用 onResultAvailable(),这里将调
         * 用到 CameraDeviceClient::onResultAvailable() */
        (*item)->onResultAvailable(result);
    }
    return OK;
}
    
/** Device-related methods */
void CameraDeviceClient::onResultAvailable(const CaptureResult& result) {

    /* 这里的 mRemoteCallback 就是在CameraService 创建 CameraDeviceClient
     * 实例时传递进来的 remoteCb,自然就是 Camera APP当中的成员了,它是
     * CameraDeviceImpl 类的内部类CameraDeviceCallbacks对象 */
    sp<hardware::camera2::ICameraDeviceCallbacks> remoteCb = mRemoteCallback;
    if (remoteCb != NULL) {
        remoteCb->onResultReceived(result.mMetadata, result.mResultExtras,
                result.mPhysicalMetadatas);
    }
}

上述的这个camera数据的流转,可以使用下图简单描述:

android 相机预览界面矩形内文件自动成像 手机相机预览界面在哪_android_02

APP

上述已经说到,在 CameraDeviceClient::onResultAvailable() 中将会将结果返回到 CameraDeviceImpl 中,在这里不再详细的跟踪,粗略看了一下代码,主要也是 onCaptureCompleted() 回调函数,一路设置到APP进行操作。


我们来看一下回调回来的两个参数,分别是CameraMetadataNative、CaptureResultExtras类型,其中根本没有buffer数据,那我们怎么取预览或者拍照数据呢?这就是API2架构的修改了,API2的架构已经不像API1那样直接在回调接口中支持buffer数据的回调了,而我们要想取到预览或者拍照结果的buffer数据,可以通过ImageReader来实现,关于使用ImageReader获取预览和拍照的buffer数据的,网上有很多教程,这里就不讲了,大家可以自己去查,主要的重点就是将构造好的ImageReader的Surface通过createCaptureSession下发下去,然后重写OnImageAvailableListener类的onImageAvailable方法,当我们前面讲过的returnBuffer将buffer数据回填给Surface之后,显示系统就会回调onImageAvailable方法,我们就可以取到想要的buffer了。

至此,camera预览流程整理完成。

附上android camera官方网站模型图。

android 相机预览界面矩形内文件自动成像 手机相机预览界面在哪_预览_03