转至元数据结尾
转至元数据起始

正在查看旧版本。 查看 当前版本.

与当前比较 查看页面历史

« 上一页 版本 2 下一步 »

概述

经过差不多两周的迁移终于将AWS账号(576184071779)下的数据迁移到DigitalOcean，包含以下几个项目S3桶、前端项目、ES服务数据（main和barnard）、Postgresql数据库这几部分

s3迁移

目标是将AWS中的ES数据迁移到DigitalOcean，使用工具 rclone （参考https://github.com/rclone/rclone ，该工具支持一些市面上一些主流的存储服务的相互迁移）

分为以下步骤

  1. 首先安装rclone成功后，进行配置，我们需要配置源和目标，有两种配置方法即通过命令行配置和配置文件（实际上命令行配置后也会写入到该配置文件，文件路径为~/.config/rclone/rclone.conf），文件内容如下

    # AWS 配置
[s3]
type = s3
provider = AWS
env_auth = false
access_key_id = XXXXXXXX # 获取方式自行查阅相关文档
secret_access_key = XXXXXXXX # 获取方式自行查阅相关文档
region = ap-northeast-1
location_constraint = ap-northeast-1
acl = private

# DigitalOcean 配置
[spaces]
type = s3
provider = DigitalOcean
env_auth = false
access_key_id = XXXXXXXX # 获取方式自行查阅相关文档
secret_access_key = XXXXXXXX # 获取方式自行查阅相关文档
endpoint = sgp1.digitaloceanspaces.com
acl = private

  2. 执行以下同步命令，将开始同步指定的AWS s3桶，这里举例`flamegraph.starcoin.org`，需要开一个窗口等待执行完成（这里建议在远程机器或者在本地上面开一个可被detach的命令行，否则当前session会被锁定在这里直到同步执行完成），请确保在执行该命令前，DigitalOcean中Space中对应的桶已经存在

    # 同步s3 的 cw-to-feishu-westar  桶中的内容到 DigitalOcean中对应的桶
rclone --no-check-certificate sync s3:cw-to-feishu-westar spaces:cw-to-feishu-westar -vv

# 同步完成后执行该命令做检查
rclone check s3:cw-to-feishu-westar spaces:cw-to-feishu-westar -vv

ElasticSearch数据迁移

目标是将AWS中的ES数据（源集群，下同）迁移到DigitalOcean（目标集群，下同），这里选择了采用基于快照的迁移方式，基于以下优点考虑：

  1. 侵入式比较少且数据也会比较完整

  2. 可以选择自己想要的部分同步

  3. 采用挂载方式省去快照数据拷贝的过程

主要的流程为

  1. 先在源集群上创建一个快照，并存入到对应的s3存储桶

  2. 在目标集群挂载s3

  3. 从快照中进行数据同步

1. 在源的集群上创建快照

AWS这里比较麻烦，需要先去配置权限，（这里参考文档https://docs.aws.amazon.com/zh_cn/opensearch-service/latest/developerguide/managedomains-snapshots.html ）如果这里不太熟悉AWS的权限，那么先建议阅读一下相关文档，我这里简要说明一下，AWS中有角色和账户，通常可以通过创建角色来为该角色配置安全策略。关于创建策略和角色参考以下文档：

https://docs.aws.amazon.com/zh_cn/IAM/latest/UserGuide/access_policies_create-console.html

https://docs.aws.amazon.com/zh_cn/IAM/latest/UserGuide/id_roles_create_for-user.html

  1. 首先需要创建一个角色专门为快照进行同步，且赋予它两个IAM策略（上面创建快照文档中均有说明）

    1. 角色本身有个信任源，需要修改角色信任关系，让其信任OpenSearch服务

      {
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "",
    "Effect": "Allow",
    "Principal": {
      "Service": "es.amazonaws.com"
    },
    "Action": "sts:AssumeRole"
  }]
}

    2. 允许读写桶的策略，这里的策略资源名称为arn:aws:iam::576184071779:policy/es-snapshot-s3-access，

      {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Action": [
                    "s3:ListBucket"
                ],
                "Effect": "Allow",
                "Resource": [
                    "arn:aws:s3:::elasticserch-snapshot-backup"
                ]
            },
            {
                "Action": [
                    "s3:GetObject",
                    "s3:PutObject",
                    "s3:DeleteObject"
                ],
                "Effect": "Allow",
                "Resource": [
                    "arn:aws:s3:::elasticserch-snapshot-backup/*"
                ]
            }
        ]
    }

    3. 是将这个角色的读写桶的权限交给ElasticSearch（AWS里面叫OpenSearch），这里我们称为权限传递

      {
    "Version": "2012-10-17",
    "Statement": [
            {
                "Effect": "Allow",
                "Action": "iam:PassRole",
                "Resource": "arn:aws:iam::576184071779:role/es-snapshot"
            },
            {
                "Effect": "Allow",
                "Action": "es:ESHttpPut",
                "Resource": "arn:aws:es:ap-northeast-1:576184071779:domain/starcoin-es2/*"
            }
        ]
    }

  2. 使用 awscurl 发起请求创建快照库（这里一定要用awscurl，否则OpenSearch服务不知道请求者的身份），如果已经配置了 AWS CLI，awscurl 可以使用相同的凭证文件（通常位于 ~/.aws/credentials）

    # 创建s3快照库
awscurl --service es --region ap-northeast-1 -XPUT 'https://search-starcoin-es2-47avtmhexhbg7qtynzebcnnu64.ap-northeast-1.es.amazonaws.com/_snapshot/my-snapshot-repo?pretty' -H 'Content-Type: application/json' -d '{"type": "s3", "settings": {"role_arn": "arn:aws:iam::576184071779:role/es-snapshot", "region": "ap-northeast-1", "bucket": "elasticserch-snapshot-backup"}}
{
  "acknowledge": true
}

# 创建快照
PUT _snapshot/my-snapshot-repo/snapshot-20240917
{
  "acknowledge": true
}

  3. 在kibana的devtool中查看快照的创建进度，若状态为SUCCESS 说明创建成功

    GET _snapshot/my-snapshot-repo/snapshot-20240917

{
  "snapshots" : [ {
    "snapshot" : "snapshot-20240917",
    "uuid" : "TVlHLRoMSXupw60xQgsWcA",
    "version_id" : 7100299,
    "version" : "7.10.2",
    "indices" : [ "halley.0727.transfer_journal", "vega.0727.block_ids", "vega.0727.txn_events", "vega.0727.dag_inspector_block", "vega.0727.pending_txns", "halley.0727.block_ids", ".opendistro-anomaly-detector-jobs", "halley.0727.token_info", "barnard.0727.blocks", ".tasks", "proxima.0727.pending_txns", "barnard.0727.txn_events", "vega.0727.dag_inspector_height_group", "main.0727.market_cap", "barnard.0727.txn_infos", "txn_infos", "barnard.0727.market_cap_bak", "opendistro-sample-http-responses", "halley.0727.txn_events", "main.0727.pending_txns", "vega.0727.txn_infos", "proxima.0727.transfer_journal", "proxima.0727.address_holder", "halley.0727.txn_infos", "barnard.0727.txn_payloads", "vega.0727.transfer_journal", "barnard.0727.918.address_holder", ".opendistro-anomaly-detectors", "barnard.0914.txn_infos", ".opendistro-reports-definitions", ".opendistro_security", "main.0727.txn_payloads", "main.0727.token_info", ".opendistro-job-scheduler-lock", "halley.0727.txn_payloads", "main.0727.txn_infos", ".opendistro-anomaly-results-history-2021.05.07-1", "proxima.0727.token_info", "barnard.0727.market_cap", ".opendistro-reports-instances", "barnard.0727.block_ids", "main.0727.transfer_journal", "halley.0727.transfer", "vega.0727.txn_payloads", "halley.0727.address_holder", "vega.0727.market_cap", "proxima.0727.transfer", "vega.0727.uncle_blocks", "vega.0727.address_holder", ".opendistro-anomaly-checkpoints", "vega.0727.token_info", "halley.0727.blocks", "barnard.0727.txn_infos_0915", "main.0727.transfer", "halley.0727.uncle_blocks", ".kibana_1", "barnard.0727.address_holder", "proxima.0727.txn_infos", "proxima.0727.blocks", "halley.0727.market_cap", "proxima.0727.uncle_blocks", "barnard.0727.transfer_journal", "barnard.0727.token_info", "main.0727.uncle_blocks", "barnard.0727.uncle_blocks", "main.0727.block_ids", "vega.0727.blocks", "proxima.0727.market_cap", "barnard.0401.txn_infos", "halley.0727.pending_txns", ".opendistro-anomaly-detection-state", "vega.0727.transfer", "proxima.0727.txn_payloads", "barnard.0727.pending_txns", "main.0727.txn_events", "test_index", "main.0727.blocks", "barnard.0727.transfer", "proxima.0727.block_ids", "main.0727.address_holder", ".kibana_-1666338091_elastic_1", "vega.0727.dag_inspector_edge", "proxima.0727.txn_events" ],
    "data_streams" : [ ],
    "include_global_state" : true,
    "state" : "SUCCESS",
    "start_time" : "2024-09-17T05:04:52.562Z",
    "start_time_in_millis" : 1726549492562,
    "end_time" : "2024-09-17T07:08:33.370Z",
    "end_time_in_millis" : 1726556913370,
    "duration_in_millis" : 7420808,
    "failures" : [ ],
    "shards" : {
      "total" : 381,
      "failed" : 0,
      "successful" : 381
    }
  } ]
}

2. 在目标集群挂载S3

  • 无标签