Terraform 集成Ansible 实战(remote)

前言

Terraform 和 Ansible 是两个独立的工具，有各自的用途，但它们可以集成以解决典型用例以及它们相互补充的方式，这使得它们更加受欢迎。

一 工具简介

1.1 Terraform

它是作为代码(IaC)软件工具的基础设施，用于构建、更改和版本控制基础设施。它与500多个供应商合作，这些供应商的资源可用于提供基础设施。

1.2 Ansible

它是一种配置管理工具，可在已设置的基础架构上配置和部署应用程序时派上用场。在本文中，我们使用 Ansible 配置了 terraform 部署的基础设施。

本文将使用Terraform对腾讯云基础设施资源进行编排管理，使用Ansible对主机实例进行配置管理。

二 架构及流程

总体方案使用terraform来进行云上基础设施编排，使用ansible进行主机或其他设施的配置管理。

编排出一台独立的远程ansible manager 主机作为ansible 配置管理的管理服务器，需要后续在其上进行ansible 配置、密钥下发，及生成管控目标服务器的invertory主机清单，对需要管控的node节点需要在该服务器进行操作。

三 实战

3.1 编排Ansible manager节点

利用本地密钥编排ansible节点

• 安装ansible
• 将ansible的配置文件传输到ansible manger的特定目录

locals {
  cidr_block_vpc_name    = "10.0.0.0/16"
  cidr_block_subnet_name = "10.0.1.0/24"
  count                  = 1
  username = "root"
  local_key_file = "id_rsa"
  local_ansible_config = "ansible.cfg"
  ansible_dir = "/opt/ansibleworkspace/"
  key_file = "/Users/xuel/.ssh/id_rsa"
}

data "tencentcloud_images" "image" {
  image_type = ["PUBLIC_IMAGE"]
  os_name    = "centos 7.5"
}

data "tencentcloud_instance_types" "instanceType" {
  cpu_core_count = 1
  memory_size    = 1
  filter {
    name   = "instance-family"
    values = ["S3"]
  }
}

data "tencentcloud_availability_zones" "zone" {}


resource "tencentcloud_vpc" "vpc" {
  cidr_block = local.cidr_block_vpc_name
  name       = "xuel_tf_vpc"
}

resource "tencentcloud_subnet" "subnet" {
  availability_zone = data.tencentcloud_availability_zones.zone.zones.0.name
  cidr_block        = local.cidr_block_subnet_name
  name              = "xuel_tf_subnet"
  vpc_id            = tencentcloud_vpc.vpc.id
}

resource "tencentcloud_key_pair" "xuel-tf-key" {
  key_name   = "xuel_tf_ansible_manager"
  public_key = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCcJxzfjqQDikRRShExu22kvNOfMKeWrn++s5nQMTm+9SNsQISEk+P15JhFVachgjumupMcOIrfJQAAQcnzNmxoRCTCJQfmegGpDZVpE1cyHUhGMA5kwu67PmK1Snm8hkg2Xzlduhr1xysL2mRn3+6o5tsFXhGrYOcSSXnf5SpTPgMjqo339ksH0iv8kvu3NaZRueygLYaVEMjixJvsnUisL3uY8LQ+4cm2Zu5mdQamhWhN0kkSdlfbjPgzxexL4AglD9YDy4I9Q80vKzy33Ubwo17a2aNCF3uPpYvCKiV0H9z2XtMxisKDfsQQA01Q1vpccUIK6L48xSbersxxxxxxxxxxxxxxxxxi1SGabjYLsv23ki6EMGjM/AK+fq+vj3pIPUMpscX3xVDGmz/zusq6v1KfOtQw7B/Dg8c2cxKUlEWZqqC3A7rt3JO/RVEbeqSe5mlRm2yngINVemmhkcfZNs= xuel@kaliarchmacbookpro"
}


resource "tencentcloud_instance" "xuel_tf_ansible" {
  availability_zone = data.tencentcloud_availability_zones.zone.zones[0].name
  image_id          = data.tencentcloud_images.image.images[0].image_id
  instance_type     = data.tencentcloud_instance_types.instanceType.instance_types[0].instance_type
  vpc_id            = tencentcloud_vpc.vpc.id
  subnet_id         = tencentcloud_subnet.subnet.id
  //  allocate_public_ip = true
  system_disk_size           = 50
  system_disk_type           = "CLOUD_PREMIUM"
  key_name                   = tencentcloud_key_pair.xuel-tf-key.id
  allocate_public_ip         = true
  internet_max_bandwidth_out = 2

  data_disks {
    data_disk_size = 50
    data_disk_type = "CLOUD_PREMIUM"
  }
  hostname             = format("xuel-tf-ansible-manager-%d", count.index)
  instance_charge_type = "POSTPAID_BY_HOUR"
  count                = local.count
  instance_name        = format("xuel-tf-server-%d", count.index)

  tags = {
    tagkey = "xuel-ansible-manager"
  }
}

// ansible 服务器部署ansible
resource "null_resource" "shell" {
  depends_on = [tencentcloud_instance.xuel_tf_ansible]
  count = local.count

  triggers = {
    instance_ids = element(tencentcloud_instance.xuel_tf_ansible.*.id, count.index)
  }
  provisioner "remote-exec" {
    connection {
      host        = element(tencentcloud_instance.xuel_tf_ansible.*.public_ip, count.index)
      type        = "ssh"
      user        = "root"
      private_key = file("${local.key_file}")
    }

    inline = [
      // 安装nginx
      "echo index.html > /xueltf.txt",
      "yum -y install nginx",
      "echo tf-nginx > /usr/share/nginx/html/index.html",
      "systemctl start nginx",
      "systemctl status nginx",

      // 安装ansible
      "yum install epel-release -y",
      "yum install ansible -y",

      // 配置ansible目录
      format("[ ! -d %s ] && mkdir -pv %s || echo 'dir create ok'", local.ansible_dir,local.ansible_dir)

    ]
  }
}

// 拷贝ansible文件至ansible 管理节点
resource "null_resource" "file_copy" {
  depends_on = [null_resource.shell]

  connection {
    host = tencentcloud_instance.xuel_tf_ansible.0.public_ip
    user = local.username
    type = "ssh"
    private_key = file("${local.key_file}")
  }

  // ansible 配置文件
  provisioner "file" {
    source = local.local_ansible_config
    destination = format("${local.ansible_dir}/%s", local.local_ansible_config)
  }
}

output "summary" {
  value = {
    //    image = {for k,v in data.tencentcloud_images.image:k=>v}
    //    instanceType = {for k,v in data.tencentcloud_instance_types
    //    .instanceType:k=>v}
    //    zone = {for k,v in data.tencentcloud_availability_zones.zone: k=>v}
    instance = { for k, v in tencentcloud_instance.xuel_tf_ansible : k => v }

    ip = tencentcloud_instance.xuel_tf_ansible.0.public_ip
  }
}

3.2 编排node节点

• 利用本地密钥编排多台node节点
• 从ansible_manager 的 terraform_remote_state 获取ansible manger的ip地址信息，用于传输node节点的IP.txt 列表文件至ansible
• 将本地编排node节点的ansible playbook脚本上次至目标服务器。
• 将创建node节点的私钥传输到ansible，供后续ansible免密登录node节点执行ansible playbook

locals {
  cidr_block_vpc_name    = "10.0.0.0/16"
  cidr_block_subnet_name = "10.0.1.0/24"
  count                  = 3
  local_ip_file = "ip.txt"
  ansible_dir = "/opt/ansibleworkspace/"

  // 本地ansible
  play_bookfile = "instance.yaml"
  local_ansible_playbook_dir = "ansible_playbook"
  key_file = "id_rsa"

}

// 获取ansible_manager 的公网IP
data "terraform_remote_state" "ansible_manager_ip" {
  backend = "cos"
  config = {
    region = "ap-beijing"
    bucket = "tfproject-1253329830"
    prefix = "ansible_manager/default"
  }
}

data "tencentcloud_images" "image" {
  image_type = ["PUBLIC_IMAGE"]
  os_name    = "centos 7.5"
}

data "tencentcloud_instance_types" "instanceType" {
  cpu_core_count = 1
  memory_size    = 1
  filter {
    name   = "instance-family"
    values = ["S3"]
  }
}

data "tencentcloud_availability_zones" "zone" {}


resource "tencentcloud_vpc" "vpc" {
  cidr_block = local.cidr_block_vpc_name
  name       = "xuel_tf_vpc"
}

resource "tencentcloud_subnet" "subnet" {
  availability_zone = data.tencentcloud_availability_zones.zone.zones.0.name
  cidr_block        = local.cidr_block_subnet_name
  name              = "xuel_tf_subnet"
  vpc_id            = tencentcloud_vpc.vpc.id
}

// 定义公钥
resource "tencentcloud_key_pair" "xuel-tf-key" {
  key_name   = "xuel_node_key"
  public_key = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCcJxzfjqQDikRRShExu22kvNOfMKeWrn++s5nQMTm+9SNsQISEk+P15JhFVachgjumupMcOIrfJQAAQcnzNmxoRCTCJQfmegGpDZVpE1cyHUhGMA5kwu67PmK1Snm8hkg2Xzlduhr1xysL2mRn3+6o5tsFXhGrYOcSSXnf5SpTPgMjqo339ksH0iv8kvu3NaZRueygLYaVEMjixJvsnUisL3uY8LQ+4cm2Zu5mdQamhWhN0kkSdlfbjPgzxexL4AglD9YDy4I9Q80vKzy33Ubwo17a2aNCF3uPpYvCKiV0H9z2XtMxisKDfsQQA01Q1vpccUIK6L48xSbers8hV2xxpSEWzEuoZg18eG2ikAencA6mhGjFWcp9A1dllY2rUhcEdrjcjXji1SGabjYLsv23ki6EMGjM/AK+fq+vj3pIPUMpscX3xVDGmz/zusq6v1KfOtQw7B/Dg8c2cxKUlEWZqqC3A7rt3JO/RVEbeqSe5mlRm2yngINVemmhkcfZNs= xuel@kaliarchmacbookpro"
}


resource "tencentcloud_instance" "xuel_tf_ansible_node" {
  availability_zone = data.tencentcloud_availability_zones.zone.zones[0].name
  image_id          = data.tencentcloud_images.image.images[0].image_id
  instance_type     = data.tencentcloud_instance_types.instanceType.instance_types[0].instance_type
  vpc_id            = tencentcloud_vpc.vpc.id
  subnet_id         = tencentcloud_subnet.subnet.id
  //  allocate_public_ip = true
  system_disk_size = 50
  system_disk_type = "CLOUD_PREMIUM"
  key_name         = tencentcloud_key_pair.xuel-tf-key.id
  allocate_public_ip = true
  internet_max_bandwidth_out = 2

  data_disks {
    data_disk_size = 50
    data_disk_type = "CLOUD_PREMIUM"
  }
  hostname             = format("xuel-tf-server-%d", count.index)
  instance_charge_type = "POSTPAID_BY_HOUR"
  count                = local.count
  instance_name        = format("xuel-tf-server-%d", count.index)

  tags = {
    name = "xuel_tf_ansibles"
  }
}

// 生成IP地址文件
resource "local_file" "ip_local_file" {
  filename = local.local_ip_file
  content = join("\n", tencentcloud_instance.xuel_tf_ansible_node.*.public_ip)
}

// 第一步，将生产的公网IP地址发送给远程ansible服务器
resource "null_resource" "copy_file" {
  depends_on = [tencentcloud_instance.xuel_tf_ansible_node]

  connection {
    type = "ssh"
    host = data.terraform_remote_state.ansible_manager_ip.outputs.summary.ip
    user = var.ansible_user
    private_key = file("/Users/xuel/.ssh/id_rsa")
  }


  // 生成的目标服务器的ip地址
  provisioner "file" {
    source = local.local_ip_file
    destination = format("${local.ansible_dir}/%s", local.local_ip_file)
  }

  // 到目标服务器执行的ansible playbook
  provisioner "file" {
    source = format("%s/%s", local.local_ansible_playbook_dir,local.play_bookfile )
    destination = format("${local.ansible_dir}/%s", local.play_bookfile)
  }

  // 到目标服务器的密钥
  provisioner "file" {
    source = "/Users/xuel/.ssh/id_rsa"
    destination = "${local.ansible_dir}/${local.key_file}"
  }

  // 目标服务器进行授权
  provisioner "remote-exec" {
    inline = [
      "chmod 600 ${local.ansible_dir}/${local.key_file}"
    ]
  }

}

// 第二步，登录ansible 控制节点执行ansible playbook
resource "null_resource" "exec_ansible_playbook" {
  depends_on = [null_resource.copy_file]

  provisioner "remote-exec" {

    connection {
      type = "ssh"
      host = data.terraform_remote_state.ansible_manager_ip.outputs.summary.ip
      user = var.ansible_user
      private_key = file("/Users/xuel/.ssh/id_rsa")
    }

    inline = [
      "cd ${local.ansible_dir}",
      "ansible-playbook ${local.play_bookfile}"
    ]
  }
}

在对node节点编排出来后，使用以下ansible playbook进行对服务器执行配置管理，主要内容为安装http，git并启动http服务，对数据盘/dev/vdb进行格式化挂在，并在其写入html文件

- name: integration of terraform and ansible
  hosts: all

  tasks:
    - name: installing httpd
      package:
        name: httpd
        state: present
    - name: installing php
      package:
        name: php
        state: present
    - name: starting httpd service
      service:
        name: httpd
        state: started
    - name: installing git
      package:
        name: git
        state: present
    - name: formatting storage
      filesystem:
        fstype: ext4
        dev : /dev/vdb
    - name: making folder
      file:
        path: /var/www/html/web
        state: directory

    - name: mounting storage
      mount:
        fstype: ext4
        src: /dev/vdb
        path: /var/www/html
        state: mounted

    - name: create html file
      ansible.builtin.shell:
        cmd: echo "xuel tf ansible page" > index.html
        chdir: /var/www/html

四 测试

本实战将backend存储在腾讯云cos中，workspace使用default

4.1 编排ansible manger节点

进行编排ansible manger节点

查看存储在cos中的state文件

登录编排出来的ansible manger查看信息

4.2 编排node节点

再次编排三台node节点。

查看存储在cos中的state状态文件

登录任意服务器进行查看情况，http已经启动，并且磁盘已经格式化并挂载。

查看web页面

其他

• 在写入多个ip列表时

content = join("\n", tencentcloud_instance.xuel_tf_ansible_node.*.public_ip)

• ansible 的配置文件中定义了inventory 和 private_key_file 的路径

[defaults]
inventory = /opt/ansibleworkspace/ip.txt
host_key_checking= False
private_key_file=/opt/ansibleworkspace/id_rsa
remote_user=root
[privilege_escalation]
become=true
become_method=sudo
become_user=root
become_ask_pass=false

后期可以结合将编排出的主机进行分组，并配合git流程实现基于Gitlab 的DevOPS流程。