腾讯云TCB云函数趣应用：巧用 puppeteer 五分钟实现一个云加社区个人成就爬虫...

原创

薛定喵君 2023-08-04 22:12:28 ©著作权

文章标签 java python linux vue spring 文章分类 代码人生

©著作权归作者所有：来自51CTO博客作者薛定喵君的原创作品，请联系作者获取转载授权，否则将追究法律责任

技术选型
应用思路
核心代码
参数介绍
HTTP访问
注意事项
参考资料
最后奉上几个社区大佬的成就

# 技术选型

之前一直想用云函数来做爬虫，无奈小程序云开发的云函数还未完全支持 puppeteer ，调用会报错。但是最近发现 TCB 的云函数支持 puppeteer 依赖，就决定用它了。
另外一个原因就是它也有免费额度 🥴

云开发(Tencent CloudBase，TCB)是腾讯云提供的云原生一体化开发环境和工具平台，为开发者提供高可用、自动弹性扩缩的后端云服务，包含计算、存储、托管等 Serverless 化能力，可用于云端一体化开发多种端应用(小程序、公众号、Web 应用、Flutter 客户端等)，帮助开发者统一构建和管理后端服务和云资源，避免了应用开发过程中繁琐的服务器搭建及运维，开发者可以专注于业务逻辑的实现，开发门槛更低，效率更高。

# 应用思路

使用puppeteer打开社区个人首页截取指定区域并进行截图后上传至云存储，下载文件后直接返回图片数据

# 核心代码

index.js

'use strict';
const puppeteer = require('puppeteer')
exports.main = async (event, context) => {
    let param = {}
    if (event.queryStringParameters) {
        param = { ...event.queryStringParameters }
    } else {
        param = { ...event }
    }
    let uid = param.uid || '1683232'
    let isImg = param.img || 0
    let force = param.force || 0
    console.log(uid, isImg, force)
    const browser = await puppeteer.launch({
        headless: true,
        args: ['--no-sandbox', '--disable-setuid-sandbox'],
        dumpio: false,
    })
    const page = await browser.newPage()
    page.setUserAgent(
        'Mozilla/5.0 (Linux; Android 10; Redmi K30 Pro Build/QKQ1.191117.002; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/78.0.3904.62 XWEB/2581 MMWEBSDK/200801 Mobile Safari/537.36'
    )
    await page.goto('https://cloud.tencent.com/developer/user/' + uid, {
        waitUntil: 'networkidle0',
    })

    // 返回图片
    if (isImg) {

        await page.$eval（
            '.uc-achievements',
            (el, value) => el.setAttribute('style', value),
            'width:200px;padding-bottom:10px;'
        )
        await page.$eval（
            '.uc-achievement:nth-child(1)',
            (el, value) => el.setAttribute('style', value),
            'float:none;width:100%;'
        )
        await page.$eval（
            '.uc-achievement:nth-child(2)',
            (el, value) => el.setAttribute('style', value),
            'float:none;width:100%;'
        )
        await page.$eval（
            '.uc-achievement:nth-child(3)',
            (el, value) => el.setAttribute('style', value),
            'float:none;width:100%;'
        )
        let rec = await page.$('.uc-achievement:nth-child(4)')
        if (rec) {
            await page.$eval（
                '.uc-achievement:nth-child(4)',
                (el, value) => el.setAttribute('style', value),
                'float:none;width:100%;'
            )
        }

        let clip = await page.evaluate(() => {
            let { x, y, width, height } = document
                .querySelector('.uc-achievements')
                .getBoundingClientRect()
            x = x - 5
            y = y - 130
            height = height + 130
            return {
                x,
                y,
                width,
                height,
            }
        })

        await page.screenshot({
            path: '/tmp/achievement.png',
            clip: clip,
        })

        await browser.close()

        const tcb = require("@cloudbase/node-sdk")
        const fs = require("fs")
        const app = tcb.init({ env: 'your env' })

        // 判断是否已经存在文件
        try {
            let oldPic = await app.downloadFile({
                fileID: 'cloud://xxx/achievement/' + uid + '.png'
            })
            // 存在
            console.log('oldPic:' + oldPic)
            // 是否覆盖上传
            if (force) {
                let res = await app.uploadFile({
                    cloudPath: "achievement/" + uid + ".png",
                    fileContent: fs.createReadStream('/tmp/achievement.png')
                })
                oldPic = await app.downloadFile({
                    fileID: res.fileID
                })
            }
            return {
                isBase64Encoded: true,
                statusCode: 200,
                headers: {
                    "content-type": "image/png"
                },
                body: oldPic.fileContent.toString('base64')
            }
        } catch (e) {
            // 不存在的情况
            console.log('e:' + e)
            const res = await app.uploadFile({
                cloudPath: "achievement/" + uid + ".png",
                fileContent: fs.createReadStream('/tmp/achievement.png')
            })
            // console.log(res)
            const result = await app.downloadFile({
                fileID: res.fileID
            })
            console.log('result:' + result)
            return {
                isBase64Encoded: true,
                statusCode: 200,
                headers: {
                    "content-type": "image/png"
                },
                body: result.fileContent.toString('base64')
            }
        }
    } else {
        const staticInfo = await page.evaluate(() => {
            const element = document.querySelector('.uc-achievements')
            let rank = element.getElementsByClassName('uc-achievement-text').item(0)
                .innerHTML
            let like = element.getElementsByClassName('uc-achievement-text').item(1)
                .innerHTML
            let read = element.getElementsByClassName('uc-achievement-text').item(2)
                .innerHTML
            return { rank: rank, like: like, read: read }
        })
        await browser.close()
        return staticInfo
    }
};