gitlab备份文件上传腾讯云COS

  • 脚本说明 
    1
    2
    3
    4
    5
    6
    7
    8
    9
    脚本名称：upload.py
    假设gitlab备份文件目录：/opt/gitlab/backups
    gitlab备份文件格式：1706922037_2024_02_06_14.2.1_gitlab_backup.tar

    1.脚本需和gitlab备份文件同级目录
    2.根据备份文件中的日期判断是否上传，如今天的日期存在于备份文件名列表中，则上传今天备份文件，反之不上传。
    3.如需上传此目录下所有文件，则去掉日期判断逻辑即可
      upload_cos(file) # ===> 上传单个文件
      upload_cos(file_list) # ===> 上传目录下所有文件
  • 源码如下 
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    # -*- coding: utf-8 -*-
    # /usr/bin/env python3
    # file_name : upload.py
    # 依赖安装：pip3 install -U cos-python-sdk-v5
    import os
    import sys
    import time
    import datetime
    import logging
    from qcloud_cos import CosConfig
    from qcloud_cos import CosS3Client
    from qcloud_cos.cos_exception import CosClientError, CosServiceError

    logging.basicConfig(level=logging.INFO, stream=sys.stdout)

    secret_id = 'SecretId'
    secret_key = 'SecretKey'
    region = 'ap-guangzhou'
    token = None

    config = CosConfig(Region=region, SecretId=secret_id, SecretKey=secret_key, Token=token)
    client = CosS3Client(config)


    def upload_cos(file):
        current = os.getcwd()
        # 线程上传
        for i in range(0, 10):
            try:
                client.upload_file(
                    Bucket='backup-1145114',
                    Key=f'gitlab/{file}',
                    LocalFilePath=current + '/' + file,
                )
                break
            except CosClientError or CosServiceError as e:
                print(e)


    def get_files():
    	# 切换到gitlab备份目录
        os.chdir('/opt/gitlab/backups')
        current_dir = os.getcwd()
        file_list = os.listdir(current_dir)
        # 脚本文件排除上传
        if 'upload.py' in file_list:
            file_list.remove('upload.py')
        # print(file_list)

        current_day = datetime.datetime.now().strftime("%Y_%m_%d")
        for file in file_list:
            if current_day in file:
                print('file exist ===>', file)
                # window下用 \\ ,linux下用 /
                print(current_dir + "/" + file)
                upload_cos(file)


    if __name__ == '__main__':
        start_time = int(time.time() * 1000)
        get_files()
        end_Time = int(time.time() * 1000)
        allCostTime = end_Time - start_time
        print(f"上传耗时：{allCostTime}ms")