python 上传大量文件 ftp python如何上传文件

转载

mob64ca140a8e67 2024-08-27 19:11:46

文章标签 python 上传大量文件 ftp 服务器 python linux googlecloud 文章分类 Python 后端开发

用服务器批量生成的文件是HTML，每次同步到本地查看尊嘟好麻烦，所以想了个办法直接一键上传谷歌云，然后到本地Drive客户端文件夹里直接打开就行了。

步骤如下，可供诸君参考~

Step 1 注册Google Cloud 获得秘钥

进入Google Cloud Console.
注册新工程“New Project”, 输入你的Project Name.
点击“ENABLE API AND SERVICE”, 进入新页面.
在新页面的搜索栏目里搜索“Google Drive”，选择下图中这个，将会进入新页面。
在新的页面中点击ENABLE按钮（这步忘记截图了QAQ）。此时会弹出一个页面让你创建一个新API，有两种类型，只要记得选择“OAuth 2.0 Client IDs”就行。后面的表单按照你的需求填写。
都填写完毕点击确定后会获得一个下图这样的界面，点击下图下载按钮获得一个.json文件，里面存储的就是你的API和相关秘钥信息。
把下载好的秘钥文件上传到你的server的某个地方，秘钥下载成就达成！

Step 2 下载PyDrive

你需要在你的server的环境中安装一个Python包‘PyDrive’，并通过它来交互你的Google Drive。安装命令如下：

pip install pydrive

OK， Python相关包下载成就达成！

Step 3 撰写Python脚本

在你的服务器上上传下列Python脚本‘submit2drive.py’，修改Main函数部分的三个相关参数，包括你的秘钥存储地址、本地文件地址、目标存储地址等。
注意：

我写了两个函数，第一个是上传单个文件，第二个是批量上传文件夹里的所有文件，按需选择嘻嘻~
每个函数的思路是1-用秘钥通过谷歌用户认证获取进入你的Drive的权限；2-获取你的目标云存储的具体地址链接；3-读取本地文件地址，是单个文件就上传到目标链接中！是文件夹就遍历，然后挨个上传！
gauth.CommandLineAuth() 这一步特别重要，它设置认证方式为命令行而非图形界面（毕竟大部分Linux Server不支持图形界面QAQ）。

import os
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive


def get_folder_id(drive, parent_id, folder_name):
    # List folders in the parent folder
    file_list = drive.ListFile(
        {
            'q': f"'{parent_id}' in parents and mimeType='application/vnd.google-apps.folder' and trashed=false"}).GetList()

    for file in file_list:
        if file['title'] == folder_name:
            return file['id']

    # Folder not found, create it
    folder_metadata = {
        'title': folder_name,
        'mimeType': 'application/vnd.google-apps.folder',
        'parents': [{'id': parent_id}]
    }
    folder = drive.CreateFile(folder_metadata)
    folder.Upload()

    return folder['id']


def upload_file_to_google_drive(local_path, drive_path, client_secrets_path):
    # Authenticate and initialize PyDrive
    gauth = GoogleAuth()
    gauth.LoadClientConfigFile(client_secrets_path)
    gauth.CommandLineAuth()  # Use command-line authentication
    drive = GoogleDrive(gauth)

    # Split the drive path into components
    path_parts = drive_path.strip('/').split('/')

    # Set up folder structure
    parent_id = 'root'  # Start from the root folder
    for part in path_parts[:-1]:  # Go through each part of the path except the last one
        parent_id = get_folder_id(drive, parent_id, part)

    # Upload file
    file_title = path_parts[-1]  # The last part of the path is the file title
    file_to_upload = drive.CreateFile({'title': file_title, 'parents': [{'id': parent_id}]})
    file_to_upload.SetContentFile(local_path)
    file_to_upload.Upload()

    print(f'File uploaded successfully to {drive_path}')


def upload_folder_to_google_drive(local_folder, drive_path, client_secrets_path):
    # Authenticate and initialize PyDrive
    gauth = GoogleAuth()
    gauth.LoadClientConfigFile(client_secrets_path)
    gauth.CommandLineAuth()  # Use command-line authentication
    drive = GoogleDrive(gauth)

    # Split the drive path into components
    path_parts = drive_path.strip('/').split('/')

    # Set up folder structure
    parent_id = 'root'  # Start from the root folder
    for part in path_parts:  # Go through each part of the path
        parent_id = get_folder_id(drive, parent_id, part)

    # Iterate over all files in the local folder
    for root, _, files in os.walk(local_folder):
        for file_name in files:
            file_path = os.path.join(root, file_name)
            file_to_upload = drive.CreateFile({'title': file_name, 'parents': [{'id': parent_id}]})
            file_to_upload.SetContentFile(file_path)
            file_to_upload.Upload()
            print(f'File {file_name} uploaded successfully to {drive_path}')


if __name__ == "__main__":
    client_secrets_path = '.../client_secrets.json'  # Replace with the correct path
    local_path = '.../result' # Local Path
    drive_path = '.../test/'  # Desired path on Google Drive

    upload_folder_to_google_drive(local_path, drive_path, client_secrets_path)