Javacv FFmpegFrameGrabber设置GPU解码 ffmpeg opencl解码

AudioTrackTest-opensles

目录

• AudioTrackTest-opensles

• OpenSL使用流程分析

• 整体流程：

• 源码分析

• 1. setAudioDataSource

• 1. soundService->initSongDecoder
• 2. soundService->initSoundTrack

• 1. 初始化engine
• 2. 初始化outputmix
• 3. 初始化audioplayer
• 4. 注册player的回调

• 2. play开始播放

• 1. play开始播放
• 2. producePacket回调获取buffer数据

• 3. stop停止播放

• 1. stop停止播放

• 问题

• 1. SL_RESULT_BUFFER_INSUFFICIENT错误
• 2. play函数中出现段错误

• 参考

OpenSL使用流程分析

OpenSL ES的API都是基于对象和接口的方式来提供的。

（1）先create Engine的ObjectItf，然后Realize Engine的ObjectItf，然后Engine的ObjectItf通过GetInterface创建EngineItf

（2）EngineItf通过CreateAudioPlayer接口来创建audio player的ObjectItf，通过CreateOutputMix接口来创建outputmix的ObjectItf

（3）audio player的ObjectItf通过GetInterface来创建bufferqueue和play Itf

整体流程：

（1）ffmpeg解析mp3，得到PCM，打包成AudioPacket，放到packet_pool中。
（2）opensl从packet_pool中读取AudioPacket，通过Enqueue函数，压进buffer中

源码分析

1. setAudioDataSource

进行ffmpeg音频解码初始化操作和openSLES初始化操作

JNIEXPORT jboolean JNICALL Java_com_example_audiotracktest_opensles_SoundTrackController_setAudioDataSource
        (JNIEnv *env, jobject obj, jstring acPathParam) {
    const char* acPath = env->GetStringUTFChars(acPathParam, NULL);
    soundService = SoundService::GetInstance();
    // 1. 初始化ffmpeg相关的东西
    soundService->initSongDecoder(acPath);
    // 2. 初始化OpenSLES相关的东西
    SLresult result = soundService->initSoundTrack();
    env->ReleaseStringUTFChars(acPathParam, acPath);
    return true;
}

1. soundService->initSongDecoder

void SoundService::initSongDecoder(const char *acPath) {
    LOGI("enter SoundService::initSongDecoder");
    mDecoderController = new AccompanyDecoderController();
    // 对ffmpeg进行初始化操作
    mDecoderController->init(acPath);
    // 设置buffer count的数量
    this->mBufferNums = 2;
    this->mCurrentFrame = 0;
    // 设置一个包的buffer大小，channel * sample rate * bytes per sample
    mPacketBufferSize = 2 * 44100 * 2;
    mBufferSize = mPacketBufferSize * mBufferNums;
}

2. soundService->initSoundTrack

SLresult SoundService::initSoundTrack() {
    LOGI("enter SoundService::initSoundTrack");
    SLresult result;
    OpenSLESContext* openSLESContext = OpenSLESContext::GetInstance();
    // 1. 创建engine对象
    mEngine = openSLESContext->getEngine();

    // 2. 创建outputmix对象
    result = CreateOutputMix();
    if (SL_RESULT_SUCCESS != result) {
        return result;
    }

    // 3. 实例化outputmix对象
    result = RealizeObject(mOutputMixObject);
    if (SL_RESULT_SUCCESS != result) {
        return result;
    }

    // 4. 创建两个buffer，mBuffer和mTarget
    InitPlayerBuffer();
    // 5. 创建AudioPlayer Itf
    result = CreateBufferQueueAudioPlayer();
    if (SL_RESULT_SUCCESS != result) {
        return result;
    }

    // 6. 实例化AudioPlayer对象
    result = RealizeObject(mAudioPlayerObject);
    if (SL_RESULT_SUCCESS != result) {
        return result;
    }
    // 7. 获取AudioPlayer对象的接口
    result = GetAudioPlayerBufferQueueInterface();
    if (SL_RESULT_SUCCESS != result) {
        return result;
    }
    // 8. 注册player的回调
    result = RegisterPlayerCallback();
    if (SL_RESULT_SUCCESS != result) {
        return result;
    }
    // 9. 获取audioplayer的接口
    result = GetAudioPlayerInterface();
    if (SL_RESULT_SUCCESS != result) {
        return result;
    }
    LOGI("leave init");
    return SL_RESULT_SUCCESS;
}

OpenSLES的对象初始化的过程：

（1）创建SLObjectItf（create）

（2）实例化SLObjectItf（Realize）

（3）通过实例化后的对象的函数GetInterface来获取接口/其他对象（GetInterface）

1. 初始化engine

1. 创建SLObjectItf mEngineObject对象
	slCreateEngine(&mEngineObject, ARRAY_LEN(engineOptions),engineOptions,0,0,0);
2. 实例化SLObjectItf mEngineObject对象
	(*mEngineObject)->Realize(mEngineObject, SL_BOOLEAN_FALSE)
3. 通过mEngineObject获取SLEngineItf mEngine对象
	(*mEngineObject)->GetInterface(mEngineObject, SL_IID_ENGINE, &mEngine);

2. 初始化outputmix

1. 通过mEngine对象来创建SLObjectItf mOutputMixObject
	(*mEngine)->CreateOutputMix(mEngine, &mOutputMixObject, 0, 0, 0);
2. 实例化SLObjectItf mOutputMixObject对象

3. 初始化audioplayer

1. 通过mEngine对象来创建SLObjectItf mAudioPlayerObject
    SLresult CreateBufferQueueAudioPlayer() {
        SLDataLocator_AndroidSimpleBufferQueue dataSourceLocator = {
                SL_DATALOCATOR_ANDROIDSIMPLEBUFFERQUEUE, // locator type
            	// 注意，这里设置的数字应该和后面play函数中调用的Enqueue函数的次数一致
            	// 不然的话，就会出现SL_RESULT_BUFFER_INSUFFICIENT错误。
                1  // buffer count
        };
        SLDataFormat_PCM dataSourceFormat = {
                SL_DATAFORMAT_PCM,  // format type
                2,  // channel count
                SL_SAMPLINGRATE_44_1, // samples per second in millihertz
                SL_PCMSAMPLEFORMAT_FIXED_16, // bits per sample
                SL_PCMSAMPLEFORMAT_FIXED_16, // container size
                SL_SPEAKER_FRONT_LEFT | SL_SPEAKER_FRONT_RIGHT, // channel mask
                SL_BYTEORDER_LITTLEENDIAN // endianness
        };
        SLDataSource dataSource = {
                &dataSourceLocator,
                &dataSourceFormat
        };
        SLDataLocator_OutputMix dataSinkLocator = {
                SL_DATALOCATOR_OUTPUTMIX,
                mOutputMixObject
        };
        SLDataSink dataSink = {
                &dataSinkLocator,
                0
        };
        SLInterfaceID interfaceIds[] = { SL_IID_BUFFERQUEUE };
        SLboolean requiredInterfaces[] { SL_BOOLEAN_TRUE };

        return (*mEngine)->CreateAudioPlayer(mEngine, &mAudioPlayerObject,
                &dataSource, &dataSink, ARRAY_LEN(interfaceIds),
                interfaceIds, requiredInterfaces);
    };
（1）配置SLDataSource
    <1> 配置SLDataLocator_AndroidSimpleBufferQueue buffer类型和数量
    <2> 配置SLDataFormat_PCM PCM的类型，通道数，采样率，定点，扬声器，小端模式
（2）配置SLDataSink：就是音频输出
（3）需要请求的接口：SLInterfaceID SL_IID_BUFFERQUEUE为缓冲队列，SL_IID_VOLUME为音量；然后对应的requiredInterfaces设为SL_BOOLEAN_TRUE

2. 实例化SLObjectItf mAudioPlayerObject对象
3. 通过mAduioPlayerObject对象来获取SLPlayItf mAudioPlayerPlay对象
4. 通过mAduioPlayerObject对象来获取SLAndroidSimpleBufferQueueItf mAudioPlayerBufferQueue对象
    (*mAudioPlayerObject)->GetInterface(mAudioPlayerObject, SL_IID_BUFFERQUEUE,&mAudioPlayerBufferQueue);

综上看：mAudioPlayerPlay对象和mAudioPlayerBufferQueue对象隶属于mAudioPlayerObject对象

4. 注册player的回调

1. 通过mAudioPlayerBufferQueue对象来注册PlayerCallback回调函数，来向buffer中填充数据
	(*mAudioPlayerBufferQueue)->RegisterCallback(mAudioPlayerBufferQueue, PlayerCallback, this);

2. play开始播放

JNIEXPORT void JNICALL Java_com_example_audiotracktest_opensles_SoundTrackController_play
(JNIEnv * env, jobject obj) {
    if (NULL != soundService) {
        // 1. 开始播放
        soundService->play();
    }
}

1. play开始播放

SLresult SoundService::play() {
    LOGI("enter SoundService::play()...");
    // 1. 设置AudioPlayer的状态为playing
    // (*mAudioPlayerPlay)->SetPlayState(mAudioPlayerPlay, SL_PLAYSTATE_PLAYING);
    SLresult result = SetAudioPlayerStatePlaying();
    if (SL_RESULT_SUCCESS != result) {
        return result;
    }
    mPlayingState = PLAYING_STATE_PLAYING;

        int ret = 0;
        if (NULL != mDecoderController) {
            /*for (int j = 0; j < 33; j++) {
                int pSize = -1;
                pSize = mDecoderController->readSamples(mTarget + ret, mPacketBufferSize/2, NULL);
                if (0 < pSize) {
                    ret += pSize;
                } else {
                    break;
                }
                LOGI("ret = %d", ret);
            } */
            // 2. 从ffmpeg中读取数据
            ret = mDecoderController->readSamples(mTarget + ret, mPacketBufferSize/2, NULL);
        }
        if (0 < ret) {
            convertByteArrayFromShortArray(mTarget, ret, mBuffer);
            // 3. 然后将数据通过AudioPlayerBufferQueue填充到buffer中
            (*mAudioPlayerBufferQueue)->Enqueue(mAudioPlayerBufferQueue, mBuffer, ret * 2);
        }
    LOGI("out SoundService::play()...");
    // 4. 然后play函数进入到死循环中，不能退出，退出应用就会闪退了。
    while(mPlayingState != PLAYING_STATE_STOPPED) {};
}

2. producePacket回调获取buffer数据

void SoundService::producePacket() {
    LOGI("SoundService::producePacket() audio player call back method...");
    byte *audioBuffer = mBuffer + (mCurrentFrame % mBufferNums) * mPacketBufferSize;
    int result = 0;
    if (NULL != mDecoderController) {
        /*for (int j = 0; j < 33; j++) {
            int pSize = -1;
            pSize = mDecoderController->readSamples(mTarget + result, mPacketBufferSize/2, NULL);
            if (0 < pSize) {
                result += pSize;
            } else {
                break;
            }
        } */
        // 读取数据
        result = mDecoderController->readSamples(mTarget + result, mPacketBufferSize/2, NULL);
        LOGI("enter SoundService::producePacket() PLAYING_STATE_PLAYING packetBufferSize=%d, result=%d", mPacketBufferSize, result);
    }
    if (0 < result) {
        convertByteArrayFromShortArray(mTarget, result, audioBuffer);
        // 入栈
        (*mAudioPlayerBufferQueue)->Enqueue(mAudioPlayerBufferQueue, audioBuffer, result * 2);
    }
    mCurrentFrame = (mCurrentFrame + 1) % mBufferNums;
}

3. stop停止播放

JNIEXPORT void JNICALL Java_com_example_audiotracktest_opensles_SoundTrackController_stop
(JNIEnv * env, jobject obj) {
    if (NULL != soundService) {
        // 1. 停止播放
        soundService->stop();
        soundService = NULL;
    }
}

1. stop停止播放

SLresult SoundService::stop() {
    LOGI("enter SoundService::stop()");
    mPlayingState = PLAYING_STATE_STOPPED;
    LOGI("Set the audio player state paused");
    // 1. 设置audioplayer的状态为stoped
    // (*mAudioPlayerPlay)->SetPlayState(mAudioPlayerPlay, SL_PLAYSTATE_PAUSED);
    SLresult result = SetAudioPlayerStateStoped();
    if (SL_RESULT_SUCCESS != result) {
        LOGI("Set the audio player state paused return false");
        return result;
    }
    // 2. 销毁object
    DestroyContext();
    LOGI("out SoundService::stop()");
}

void SoundService::DestroyContext() {
    LOGI("enter SoundService::DestroyContext");
    DestroyObject(mAudioPlayerObject);
    FreePlayerBuffer();
    DestroyObject(mOutputMixObject);
    if (NULL != mDecoderController) {
        mDecoderController->destroy();
        delete mDecoderController;
        mDecoderController = NULL;
    }

    LOGI("leave SoundService::DestroyContext");
}

问题

1. SL_RESULT_BUFFER_INSUFFICIENT错误

SLDataLocator_AndroidSimpleBufferQueue配置的buffer数和play的时候Enqueue的buffer数目对不上

2. play函数中出现段错误

play函数不应该退出，不然就会出现段错误。在最后面加个while死循环。

参考

（1）Android音频开发（7）：使用 OpenSL ES API（下）：
https://blog.51cto.com/ticktick/1771239
（2）Android Open SL ES — 官方Demo解析native-audio
https://www.jianshu.com/p/5308ab38a3d0
（3）源码位置：https://github.com/mashenlyl/AudioTrackTest