티스토리 뷰
16 Terraform을 사용하여 Azure Kubernetes Service에 Application Gateway Ingress Controller 만들기
zerobig-k8s 2020. 1. 28. 08:07<참조>
AKS(Azure Kubernetes Service)는 호스트된 Kubernetes 환경을 관리한다. AKS를 사용하면 컨테이너 오케스트레이션에 대한 전문 지식 없이도 컨테이너화된 애플리케이션을 쉽고 빠르게 배포 및 관리할 수 있다. AKS는 운영 및 유지 관리 작업을 위해 애플리케이션을 오프라인으로 전환해야 하는 부담을 없애준다. AKS를 사용하면 이러한 작업(예: 리소스 프로비저닝, 업그레이드 및 크기 조정)을 On-Demand(주문형) 요청 방식으로 수행할 수 있다.
Ingress Controller(수신 컨트롤러)는 Kubernetes 서비스에 대해 다양한 기능을 제공한다. 이러한 기능에는 역방향 프록시, 구성 가능한 트래픽 라우팅 및 TLS 종료가 포함된다. Kubernetes 수신 리소스는 개별 Kubernetes 서비스에 대한 수신 규칙을 구성하는 데 사용된다. 수신 컨트롤러 및 수신 규칙을 사용하면 단일 IP 주소를 사용하여 Kubernetes 클러스터의 여러 서비스에 트래픽을 라우팅할 수 있다. 이러한 모든 기능이 Azure Application Gateway에서 제공되기 때문에 Azure의 Kubernetes에 이상적인 수신 컨트롤러다.
이 자습서에서는 다음 작업을 수행하는 방법을 알아본다.
- Application Gateway를 수신 컨트롤러로 가지는 AKS를 사용하여 Kubernetes 클러스터 만들기
- HCL(HashiCorp Language)을 사용하여 Kubernetes 클러스터 정의
- Terraform을 사용하여 Application Gateway 리소스 만들기
- Terraform 및 AKS를 사용하여 Kubernetes 클러스터 만들기
- kubectl 도구를 사용하여 Kubernetes 클러스터의 가용성 테스트
들어가기에 앞서
본 실습내용은 다소 복잡하고 쉽지 않았다. 더우기 원문(영문포함)에 오류가 있어 어렵게 어렵게 완성한 실습이다. 초보자들이 따라하기에는 만만치 않을 것이다. 그럼에도 불구하고 중요한 요소기술들이 복합적으로 구성되어 있기에 적지않게 유용하고 보람이 있을 것이다.
사전 요구 사항
-
Azure 구독: Azure 구독이 아직 없는 경우 시작하기 전에 체험 계정을 만든다.
-
Terraform 구성: 이전 실습 Azure의 Terraform을 확인한다.
- Azure 리소스 그룹: 데모에 사용할 Azure 리소스 그룹이 없는 경우 Azure 리소스 그룹을 만든다. 해당 값이 데모에 사용되므로 리소스 그룹 이름과 위치를 기록해 둔다. (다음 섹션 참조)
-
Azure 서비스 주체: 이전 실습 Azure의 Terraform을 확인한다. appId, displayName, 암호, 테넌트를 기록해 둔다. (아래 Kubernetes 클러스터 만들기 섹션 참조)
- 서비스 사용자 개체(Object) ID 가져오기: Cloud Shell에서 다음 명령을 실행한다. (아래 Kubernetes 클러스터 만들기 섹션 참조)
디렉터리 구조 만들기
실습할 디렉토리를 생성하고 마우스 오른쪽 버튼을 클릭하여 "Code(으)로 열기"를 선택한다.
생성한 실습 디렉토리 이하에 terraform-aks-appgw-ingress 디렉토리를 생성한 뒤 해당 디렉토리로 이동한다.
먼저 사전 요구 사항에서 언급했던 리소스 그룹 생성을 위해 az login으로 로그인 하여 az group create 명령을 이용하여 리소스 그룹을 생성한다.
zerobig@ZEROBIG-NT800:/mnt/d/Azure_DevOps_Study/20200127/terraform-aks-appgw-ingress$$ az login
Not able to launch a browser to log you in, falling back to device code...
To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code HKWB7ACUX to authenticate.
[
{
"cloudName": "AzureCloud",
"id": "f8764f39-xxxx-xxxx-xxxx-77b286ed971b",
"isDefault": true,
"name": "Visual Studio Enterprise 201907",
"state": "Enabled",
"tenantId": "917bfe84-0ca6-488d-ad3a-236e41ceafe9",
"user": {
"name": "zerobig.kim@gmail.com",
"type": "user"
}
}
]
zerobig@ZEROBIG-NT800:/mnt/d/Azure_DevOps_Study/20200127/terraform-aks-appgw-ingress$ az group create -n demo0127 -l koreacentral
{
"id": "/subscriptions/f8764f39-xxxx-xxxx-xxxx-77b286ed971b/resourceGroups/demo0127",
"location": "koreacentral",
"managedBy": null,
"name": "demo0127",
"properties": {
"provisioningState": "Succeeded"
},
"tags": null,
"type": "Microsoft.Resources/resourceGroups"
}리소스 그룹 이름과 위치(location)을 기록해 둔다.
Azure 공급자 선언
Azure 공급자를 선언하는 Terraform 구성 파일을 만든다.
1. main.tf 라는 파일을 만든다.
2. 다음 코드를 붙여 넣는다.
|
provider "azurerm" { version = "~>1.18" }
terraform { backend "azurerm" {} } |
3. 파일을 저장한다.
입력 변수 정의
이 배포에 필요한 모든 변수를 나열하는 Terraform 구성 파일을 만든다.
1. variables.tf라는 파일을 만든다.
2. 다음 코드를 붙여 넣는다.
- aks_dns_prefix, aks_name은 각자가 원하는 이름으로 수정한다.
- aks_agent_count(워커 노드)를 1~3개 값 중 각자 상황에 맞게 수정한다. (실습목적이므로 2개 정도가 적당하다.)
|
variable "resource_group_name" { description = "Name of the resource group." }
variable "location" { description = "Location of the cluster." }
variable "aks_service_principal_app_id" { description = "Application ID/Client ID of the service principal. Used by AKS to manage AKS related resources on Azure like vms, subnets." }
variable "aks_service_principal_client_secret" { description = "Secret of the service principal. Used by AKS to manage Azure." }
variable "aks_service_principal_object_id" { description = "Object ID of the service principal." }
variable "virtual_network_name" { description = "Virtual network name" default = "aksVirtualNetwork" }
variable "virtual_network_address_prefix" { description = "Containers DNS server IP address." default = "15.0.0.0/8" }
variable "aks_subnet_name" { description = "AKS Subnet Name." default = "kubesubnet" }
variable "aks_subnet_address_prefix" { description = "Containers DNS server IP address." default = "15.0.0.0/16" }
variable "app_gateway_subnet_address_prefix" { description = "Containers DNS server IP address." default = "15.1.0.0/16" }
variable "app_gateway_name" { description = "Name of the Application Gateway." default = "ApplicationGateway1" }
variable "app_gateway_sku" { description = "Name of the Application Gateway SKU." default = "Standard_v2" }
variable "app_gateway_tier" { description = "Tier of the Application Gateway SKU." default = "Standard_v2" }
variable "aks_name" { description = "Name of the AKS cluster." default = "zero-aks-cluster1" } variable "aks_dns_prefix" { description = "Optional DNS prefix to use with hosted Kubernetes API server FQDN." default = "zero-aks" }
variable "aks_agent_os_disk_size" { description = "Disk size (in GB) to provision for each of the agent pool nodes. This value ranges from 0 to 1023. Specifying 0 applies the default disk size for that agentVMSize." default = 40 }
variable "aks_agent_count" { description = "The number of agent nodes for the cluster." default = 2 }
variable "aks_agent_vm_size" { description = "The size of the Virtual Machine." default = "Standard_D3_v2" }
variable "kubernetes_version" { description = "The version of Kubernetes." default = "1.11.5" }
variable "aks_service_cidr" { description = "A CIDR notation IP range from which to assign service cluster IPs." default = "10.0.0.0/16" }
variable "aks_dns_service_ip" { description = "Containers DNS server IP address." default = "10.0.0.10" }
variable "aks_docker_bridge_cidr" { description = "A CIDR notation IP for Docker bridge." default = "172.17.0.1/16" }
variable "aks_enable_rbac" { description = "Enable RBAC on the AKS cluster. Defaults to false." default = "false" }
variable "vm_user_name" { description = "User name for the VM" default = "vmuser1" }
variable "public_ssh_key_path" { description = "Public key path for SSH." default = "~/.ssh/id_rsa.pub" }
variable "tags" { type = "map"
default = { source = "terraform" } } |
3. 파일을 저장한다.
리소스 정의
모든 리소스를 만드는 Terraform 구성 파일을 만든다.
1. resources.tf 라는 파일을 만든다.
2. 다음 코드 블록을 붙여넣어 계산(computed) 변수가 재사용할 로컬 블록을 만든다.
|
# Locals block for hardcoded names. locals { backend_address_pool_name = "${azurerm_virtual_network.test.name}-beap" frontend_port_name = "${azurerm_virtual_network.test.name}-feport" frontend_ip_configuration_name = "${azurerm_virtual_network.test.name}-feip" http_setting_name = "${azurerm_virtual_network.test.name}-be-htst" listener_name = "${azurerm_virtual_network.test.name}-httplstn" request_routing_rule_name = "${azurerm_virtual_network.test.name}-rqrt" app_gateway_subnet_name = "appgwsubnet" } |
3. 다음 코드 블록을 붙여넣어 리소스 그룹, 새 사용자 ID에 대한 데이터 원본을 만든다.
|
data "azurerm_resource_group" "rg" { name = var.resource_group_name }
# User Assigned Identities resource "azurerm_user_assigned_identity" "testIdentity" { resource_group_name = data.azurerm_resource_group.rg.name location = data.azurerm_resource_group.rg.location
name = "identity1"
tags = var.tags } |
4. 다음 코드 블록을 붙여넣어 기본 네트워킹 리소스를 만든다.
|
resource "azurerm_virtual_network" "test" { name = var.virtual_network_name location = data.azurerm_resource_group.rg.location resource_group_name = data.azurerm_resource_group.rg.name address_space = [var.virtual_network_address_prefix]
subnet { name = var.aks_subnet_name address_prefix = var.aks_subnet_address_prefix }
subnet { name = "appgwsubnet" address_prefix = var.app_gateway_subnet_address_prefix }
tags = var.tags }
data "azurerm_subnet" "kubesubnet" { name = var.aks_subnet_name virtual_network_name = azurerm_virtual_network.test.name resource_group_name = data.azurerm_resource_group.rg.name }
data "azurerm_subnet" "appgwsubnet" { name = "appgwsubnet" virtual_network_name = azurerm_virtual_network.test.name resource_group_name = data.azurerm_resource_group.rg.name }
# Public Ip resource "azurerm_public_ip" "test" { name = "publicIp1" location = data.azurerm_resource_group.rg.location resource_group_name = data.azurerm_resource_group.rg.name allocation_method = "Static" sku = "Standard"
tags = var.tags } |
5. 다음 코드 블록을 붙여넣어 Application Gateway 리소스를 만든다.
|
resource "azurerm_application_gateway" "network" { name = var.app_gateway_name resource_group_name = data.azurerm_resource_group.rg.name location = data.azurerm_resource_group.rg.location
sku { name = var.app_gateway_sku tier = "Standard_v2" capacity = 2 }
gateway_ip_configuration { name = "appGatewayIpConfig" subnet_id = data.azurerm_subnet.appgwsubnet.id }
frontend_port { name = local.frontend_port_name port = 80 }
frontend_port { name = "httpsPort" port = 443 }
frontend_ip_configuration { name = local.frontend_ip_configuration_name public_ip_address_id = azurerm_public_ip.test.id }
backend_address_pool { name = local.backend_address_pool_name }
backend_http_settings { name = local.http_setting_name cookie_based_affinity = "Disabled" port = 80 protocol = "Http" request_timeout = 1 }
http_listener { name = local.listener_name frontend_ip_configuration_name = local.frontend_ip_configuration_name frontend_port_name = local.frontend_port_name protocol = "Http" }
request_routing_rule { name = local.request_routing_rule_name rule_type = "Basic" http_listener_name = local.listener_name backend_address_pool_name = local.backend_address_pool_name backend_http_settings_name = local.http_setting_name }
tags = var.tags
depends_on = ["azurerm_virtual_network.test", "azurerm_public_ip.test"] } |
6. 다음 코드 블록을 붙여넣어 역할 할당을 만든다.
|
resource "azurerm_role_assignment" "ra1" { scope = data.azurerm_subnet.kubesubnet.id role_definition_name = "Network Contributor" principal_id = var.aks_service_principal_object_id
depends_on = ["azurerm_virtual_network.test"] }
resource "azurerm_role_assignment" "ra2" { scope = azurerm_user_assigned_identity.testIdentity.id role_definition_name = "Managed Identity Operator" principal_id = var.aks_service_principal_object_id depends_on = ["azurerm_user_assigned_identity.testIdentity"] }
resource "azurerm_role_assignment" "ra3" { scope = azurerm_application_gateway.network.id role_definition_name = "Contributor" principal_id = azurerm_user_assigned_identity.testIdentity.principal_id depends_on = ["azurerm_user_assigned_identity.testIdentity", "azurerm_application_gateway.network"] }
resource "azurerm_role_assignment" "ra4" { scope = data.azurerm_resource_group.rg.id role_definition_name = "Reader" principal_id = azurerm_user_assigned_identity.testIdentity.principal_id depends_on = ["azurerm_user_assigned_identity.testIdentity", "azurerm_application_gateway.network"] } |
7. 다음 코드 블록을 붙여넣어 Kubernetes 클러스터를 만든다.
|
resource "azurerm_kubernetes_cluster" "k8s" { name = var.aks_name location = data.azurerm_resource_group.rg.location dns_prefix = var.aks_dns_prefix
resource_group_name = data.azurerm_resource_group.rg.name
linux_profile { admin_username = var.vm_user_name
ssh_key { key_data = file(var.public_ssh_key_path) } }
addon_profile { http_application_routing { enabled = false } }
agent_pool_profile { name = "agentpool" count = var.aks_agent_count vm_size = var.aks_agent_vm_size os_type = "Linux" os_disk_size_gb = var.aks_agent_os_disk_size vnet_subnet_id = data.azurerm_subnet.kubesubnet.id }
service_principal { client_id = var.aks_service_principal_app_id client_secret = var.aks_service_principal_client_secret }
network_profile { network_plugin = "azure" dns_service_ip = var.aks_dns_service_ip docker_bridge_cidr = var.aks_docker_bridge_cidr service_cidr = var.aks_service_cidr }
depends_on = ["azurerm_virtual_network.test", "azurerm_application_gateway.network"] tags = var.tags } |
8. 파일을 저장한다.
이 섹션에 제시된 코드는 클러스터의 이름, 위치 및 resource_group_name을 설정한다. 클러스터에 액세스하는 데 사용되는 FQDN(정규화된 도메인 이름)의 일부를 형성하는 dns_prefix 값이 설정된다.
linux_profile 레코드를 사용하면 SSH를 사용하여 워커노드에 로그인 할 수 있는 설정을 구성할 수 있다.
AKS를 사용하면 워커노드에 대해서만 지불한다. agent_pool_profile 레코드는 이러한 워커노드에 대한 세부정보를 구성한다. agent_pool_profile record에는 만들 워커노드 수와 워커노드 유형이 포함된다. 나중에 클러스터를 확장하거나 축소해야 하는 경우, 이 레코드에서 count 값을 수정한다.
Terraform 출력 파일 만들기
Terraform 출력을 사용하면 Terraform이 계획을 적용할 때 사용자에게 강조 표시되는 값 및 terraform output 명령을 사용하여 쿼리할 수 있는 값을 정의할 수 있다. 이 섹션에서는 kubectl 도구를 사용하여 클러스터 액세스를 허용하는 출력 파일을 만든다.
1. output.tf 라는 파일을 만든다.
2. 다음 코드를 붙여 넣는다.
|
output "client_key" { value = azurerm_kubernetes_cluster.k8s.kube_config.0.client_key }
output "client_certificate" { value = azurerm_kubernetes_cluster.k8s.kube_config.0.client_certificate }
output "cluster_ca_certificate" { value = azurerm_kubernetes_cluster.k8s.kube_config.0.cluster_ca_certificate }
output "cluster_username" { value = azurerm_kubernetes_cluster.k8s.kube_config.0.username }
output "cluster_password" { value = azurerm_kubernetes_cluster.k8s.kube_config.0.password }
output "kube_config" { value = azurerm_kubernetes_cluster.k8s.kube_config_raw }
output "host" { value = azurerm_kubernetes_cluster.k8s.kube_config.0.host }
output "identity_resource_id" { value = azurerm_user_assigned_identity.testIdentity.id }
output "identity_client_id" { value = azurerm_user_assigned_identity.testIdentity.client_id } |
3. 파일을 저장한다.
Terraform 상태를 저장하도록 Azure Storage 설정
Terraform은 terraform.tfstate 파일을 통해 로컬로 상태를 추적한다. 이 패턴은 단일 작업자 환경에서 잘 작동한다. 다중 사용자 환경에서 Azure 스토리지는 상태를 추적하는 데 사용된다.
이 섹션에서는 다음 작업을 수행하는 방법을 보여준다.
- 스토리지 계정 정보 검색(계정 이름 및 계정 키)
- Terraform 상태 정보가 저장되는 스토리지 컨테이너를 만든다.
1. Azure Portal에서 메뉴 중 스토리지 계정(Storage accounts)을 선택한다.
2. 스토리지 계정 탭에서 Terraform가 상태를 저장하도록 설정할 스토리지 계정을 하나 새로 만들기 위해 +Add를 선택한다.
Storage account name 필드에 고유한 적절값을 입력하고 Review + Create를 누른다.
선택한 스토리지 계정을 기록해 둔다. 이 값은 나중에 필요하다.
3. [스토리지 계정] 탭에서 액세스 키를 선택하여 key1 key 값을 기록해 둔다. 이 값은 나중에 필요하다.
4. Azure 스토리지 계정에 컨테이너를 만든다. 자리 표시자(placeholders)를 사용자 환경에 적합한 값으로 바꾼다.
| az storage container create --name tfstate --account-name <YourAzureStorageAccountName> --account-key <YourAzureStorageAccountKey> |
zerobig@ZEROBIG-NT800:/mnt/d/Azure_DevOps_Study/20200127/terraform-aks-appgw-ingress$ az storage container create --name tfstate --account-name 4tfstate --account-key WnZ6AMBb1AKUqMEsFX9Fqs5YjRRDCAtg6WRqYhFsyJ....
{
"created": true
}
Kubernetes 클러스터를 만들기
이 섹션에서는 terraform init 명령을 사용하여 이전 섹션에서 만든 구성 파일을 정의하는 리소스를 만드는 방법을 보여 준다.
1.Terraform을 초기화 한다. 자리 표시자를 사용자 환경에 적합한 값으로 바꾼다.
| terraform init -backend-config="storage_account_name=<YourAzureStorageAccountName>" -backend-config="container_name=tfstate" -backend-config="access_key=<YourStorageAccountAccessKey>" -backend-config="key=codelab.microsoft.tfstate" |
zerobig@ZEROBIG-NT800:/mnt/d/Azure_DevOps_Study/20200127/terraform-aks-appgw-ingress$ terraform init -backend-config="storage_account_name=4tfstate" -backend-config="container_name=tfstate" -backend-config="access_key=WnZ6AMB....HDO+yKjXQ==" -backend-config="key=codelab.microsoft.tfstate"
Initializing the backend...
Successfully configured the backend "azurerm"! Terraform will automatically
use this backend unless the backend configuration changes.
Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "azurerm" (hashicorp/azurerm) 1.41.0...
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
2. terraform.tfvars인 파일을 만든다.
|
resource_group_name = "<Name of the Resource Group already created>" location = "<Location of the Resource Group>" aks_service_principal_app_id = "<Service Principal AppId>" aks_service_principal_client_secret = "<Service Principal Client Secret>" aks_service_principal_object_id = "<Service Principal Object Id>" |
위 서비스 주체에 대한 값들을 구하기 위해 다음 명령을 수행한다.
|
# Azure 서비스 주체 생성 az ad sp create-for-rbac --role="Contributor" --scopes="/subscriptions/<구독 ID>"
# Azure 사용자 개체(Object) ID 가져오기 az ad sp list --display-name <displayname> | grep -i objectid |
더불어 Object ID를 취하기 위해 "az ad sp list --display-name azure-cli-2020-01-26-06-59-15 | grep -i objectid" 명령을 수행한다.
zerobig@ZEROBIG-NT800:/mnt/d/Azure_DevOps_Study/20200127/terraform-aks-appgw-ingress$ az ad sp create-for-rbac --role="Contributor" --scopes="/subscriptions/f8764f39-xxxx-xxxx-xxxx-77b286ed971b"
Creating a role assignment under the scope of "/subscriptions/f8764f39-xxxx-xxxx-xxxx-77b286ed971b"
Retrying role assignment creation: 1/36
Retrying role assignment creation: 2/36
{
"appId": "c2ff6c89-ab7c-48aa-a7de-afb1484f872c",
"displayName": "azure-cli-2020-01-26-06-59-15",
"name": "http://azure-cli-2020-01-26-06-59-15",
"password": "472bbed2-xxxx-xxxx-xxxx-33ffd499d7e5",
"tenant": "917bfe84-0ca6-488d-ad3a-236e41ceafe9"
}
zerobig@ZEROBIG-NT800:/mnt/d/Azure_DevOps_Study/20200127/terraform-aks-appgw-ingress$ az ad sp list --display-name azure-cli-2020-01-26-06-59-15 | grep -i objectid
"objectId": "9d262acd-83f3-40d6-8c24-4f2a28af0f5f",
3. 출력된 값으로 terraform.tfvars에 입력해주고 파일을 저장한다.
| resource_group_name = "demo0127" location = "koreacentral" aks_service_principal_app_id = "c2ff6c89-ab7c-48aa-a7de-afb1484f872c" aks_service_principal_client_secret = "472bbed2-xxxx-xxxx-xxxx-33ffd499d7e5" aks_service_principal_object_id = "9d262acd-83f3-40d6-8c24-4f2a28af0f5f" |
4. terraform plan 명령을 실행하여 인프라 요소를 정의하는 Terraform 계획을 만든다.
| terraform plan -out out.plan |
zerobig@ZEROBIG-NT800:/mnt/d/Azure_DevOps_Study/20200127/terraform-aks-appgw-ingress$ terraform plan -out out.plan
Acquiring state lock. This may take a few moments...
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
data.azurerm_resource_group.rg: Refreshing state...
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
<= read (data resources)
Terraform will perform the following actions:
# data.azurerm_subnet.appgwsubnet will be read during apply
# (config refers to values not yet known)
<= data "azurerm_subnet" "appgwsubnet" {
+ address_prefix = (known after apply)
+ enforce_private_link_endpoint_network_policies = (known after apply)
+ enforce_private_link_service_network_policies = (known after apply)
+ id = (known after apply)
+ ip_configurations = (known after apply)
+ name = "appgwsubnet"
+ network_security_group_id = (known after apply)
+ resource_group_name = "demo0127"
+ route_table_id = (known after apply)
+ service_endpoints = (known after apply)
+ virtual_network_name = "aksVirtualNetwork"
+ timeouts {
+ read = (known after apply)
}
}
# data.azurerm_subnet.kubesubnet will be read during apply
# (config refers to values not yet known)
<= data "azurerm_subnet" "kubesubnet" {
+ address_prefix = (known after apply)
+ enforce_private_link_endpoint_network_policies = (known after apply)
+ enforce_private_link_service_network_policies = (known after apply)
+ id = (known after apply)
+ ip_configurations = (known after apply)
+ name = "kubesubnet"
+ network_security_group_id = (known after apply)
+ resource_group_name = "demo0127"
+ route_table_id = (known after apply)
+ service_endpoints = (known after apply)
+ virtual_network_name = "aksVirtualNetwork"
+ timeouts {
+ read = (known after apply)
}
}
# azurerm_application_gateway.network will be created
+ resource "azurerm_application_gateway" "network" {
+ disabled_ssl_protocols = (known after apply)
+ id = (known after apply)
+ location = "koreacentral"
+ name = "ApplicationGateway1"
+ resource_group_name = "demo0127"
+ tags = {
+ "source" = "terraform"
}
+ backend_address_pool {
+ fqdn_list = (known after apply)
+ fqdns = (known after apply)
+ id = (known after apply)
+ ip_address_list = (known after apply)
+ ip_addresses = (known after apply)
+ name = "aksVirtualNetwork-beap"
}
+ backend_http_settings {
+ cookie_based_affinity = "Disabled"
+ id = (known after apply)
+ name = "aksVirtualNetwork-be-htst"
+ pick_host_name_from_backend_address = false
+ port = 80
+ probe_id = (known after apply)
+ protocol = "Http"
+ request_timeout = 1
}
+ frontend_ip_configuration {
+ id = (known after apply)
+ name = "aksVirtualNetwork-feip"
+ private_ip_address = (known after apply)
+ private_ip_address_allocation = (known after apply)
+ public_ip_address_id = (known after apply)
+ subnet_id = (known after apply)
}
+ frontend_port {
+ id = (known after apply)
+ name = "aksVirtualNetwork-feport"
+ port = 80
}
+ frontend_port {
+ id = (known after apply)
+ name = "httpsPort"
+ port = 443
}
+ gateway_ip_configuration {
+ id = (known after apply)
+ name = "appGatewayIpConfig"
+ subnet_id = (known after apply)
}
+ http_listener {
+ frontend_ip_configuration_id = (known after apply)
+ frontend_ip_configuration_name = "aksVirtualNetwork-feip"
+ frontend_port_id = (known after apply)
+ frontend_port_name = "aksVirtualNetwork-feport"
+ id = (known after apply)
+ name = "aksVirtualNetwork-httplstn"
+ protocol = "Http"
+ ssl_certificate_id = (known after apply)
}
+ identity {
+ identity_ids = (known after apply)
+ type = (known after apply)
}
+ request_routing_rule {
+ backend_address_pool_id = (known after apply)
+ backend_address_pool_name = "aksVirtualNetwork-beap"
+ backend_http_settings_id = (known after apply)
+ backend_http_settings_name = "aksVirtualNetwork-be-htst"
+ http_listener_id = (known after apply)
+ http_listener_name = "aksVirtualNetwork-httplstn"
+ id = (known after apply)
+ name = "aksVirtualNetwork-rqrt"
+ redirect_configuration_id = (known after apply)
+ rewrite_rule_set_id = (known after apply)
+ rule_type = "Basic"
+ url_path_map_id = (known after apply)
}
+ sku {
+ capacity = 2
+ name = "Standard_v2"
+ tier = "Standard_v2"
}
+ ssl_policy {
+ cipher_suites = (known after apply)
+ disabled_protocols = (known after apply)
+ min_protocol_version = (known after apply)
+ policy_name = (known after apply)
+ policy_type = (known after apply)
}
}
# azurerm_kubernetes_cluster.k8s will be created
+ resource "azurerm_kubernetes_cluster" "k8s" {
+ dns_prefix = "zero-aks"
+ enable_pod_security_policy = (known after apply)
+ fqdn = (known after apply)
+ id = (known after apply)
+ kube_admin_config = (known after apply)
+ kube_admin_config_raw = (sensitive value)
+ kube_config = (known after apply)
+ kube_config_raw = (sensitive value)
+ kubernetes_version = (known after apply)
+ location = "koreacentral"
+ name = "zero-aks-cluster1"
+ node_resource_group = (known after apply)
+ private_fqdn = (known after apply)
+ resource_group_name = "demo0127"
+ tags = {
+ "source" = "terraform"
}
+ addon_profile {
+ http_application_routing {
+ enabled = false
+ http_application_routing_zone_name = (known after apply)
}
}
+ agent_pool_profile {
+ count = 2
+ dns_prefix = (known after apply)
+ fqdn = (known after apply)
+ max_pods = (known after apply)
+ name = "agentpool"
+ os_disk_size_gb = 40
+ os_type = "Linux"
+ type = "AvailabilitySet"
+ vm_size = "Standard_D3_v2"
+ vnet_subnet_id = (known after apply)
}
+ linux_profile {
+ admin_username = "vmuser1"
+ ssh_key {
+ key_data = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCttszvheNvmlgaKGf9t2L1mT2bDMhGHCPbGvo9Ys1lSgB~~~~~wbHZgGp+ApBxuU35g0OAzXVhYR8fToT zerobig.kim@gmail.com\n"
}
}
+ network_profile {
+ dns_service_ip = "10.0.0.10"
+ docker_bridge_cidr = "172.17.0.1/16"
+ load_balancer_sku = "basic"
+ network_plugin = "azure"
+ network_policy = (known after apply)
+ pod_cidr = (known after apply)
+ service_cidr = "10.0.0.0/16"
}
+ role_based_access_control {
+ enabled = (known after apply)
+ azure_active_directory {
+ client_app_id = (known after apply)
+ server_app_id = (known after apply)
+ server_app_secret = (sensitive value)
+ tenant_id = (known after apply)
}
}
+ service_principal {
+ client_id = "c2ff6c89-ab7c-48aa-a7de-afb1484f872c"
+ client_secret = (sensitive value)
}
}
# azurerm_public_ip.test will be created
+ resource "azurerm_public_ip" "test" {
+ allocation_method = "Static"
+ fqdn = (known after apply)
+ id = (known after apply)
+ idle_timeout_in_minutes = 4
+ ip_address = (known after apply)
+ ip_version = "IPv4"
+ location = "koreacentral"
+ name = "publicIp1"
+ public_ip_address_allocation = (known after apply)
+ resource_group_name = "demo0127"
+ sku = "Standard"
+ tags = {
+ "source" = "terraform"
}
}
# azurerm_role_assignment.ra1 will be created
+ resource "azurerm_role_assignment" "ra1" {
+ id = (known after apply)
+ name = (known after apply)
+ principal_id = "9d262acd-83f3-40d6-8c24-4f2a28af0f5f"
+ principal_type = (known after apply)
+ role_definition_id = (known after apply)
+ role_definition_name = "Network Contributor"
+ scope = (known after apply)
+ skip_service_principal_aad_check = (known after apply)
}
# azurerm_role_assignment.ra2 will be created
+ resource "azurerm_role_assignment" "ra2" {
+ id = (known after apply)
+ name = (known after apply)
+ principal_id = "9d262acd-83f3-40d6-8c24-4f2a28af0f5f"
+ principal_type = (known after apply)
+ role_definition_id = (known after apply)
+ role_definition_name = "Managed Identity Operator"
+ scope = (known after apply)
+ skip_service_principal_aad_check = (known after apply)
}
# azurerm_role_assignment.ra3 will be created
+ resource "azurerm_role_assignment" "ra3" {
+ id = (known after apply)
+ name = (known after apply)
+ principal_id = (known after apply)
+ principal_type = (known after apply)
+ role_definition_id = (known after apply)
+ role_definition_name = "Contributor"
+ scope = (known after apply)
+ skip_service_principal_aad_check = (known after apply)
}
# azurerm_role_assignment.ra4 will be created
+ resource "azurerm_role_assignment" "ra4" {
+ id = (known after apply)
+ name = (known after apply)
+ principal_id = (known after apply)
+ principal_type = (known after apply)
+ role_definition_id = (known after apply)
+ role_definition_name = "Reader"
+ scope = "/subscriptions/f8764f39-xxxx-xxxx-xxxx-77b286ed971b/resourceGroups/demo0127"
+ skip_service_principal_aad_check = (known after apply)
}
# azurerm_user_assigned_identity.testIdentity will be created
+ resource "azurerm_user_assigned_identity" "testIdentity" {
+ client_id = (known after apply)
+ id = (known after apply)
+ location = "koreacentral"
+ name = "identity1"
+ principal_id = (known after apply)
+ resource_group_name = "demo0127"
+ tags = {
+ "source" = "terraform"
}
}
# azurerm_virtual_network.test will be created
+ resource "azurerm_virtual_network" "test" {
+ address_space = [
+ "15.0.0.0/8",
]
+ id = (known after apply)
+ location = "koreacentral"
+ name = "aksVirtualNetwork"
+ resource_group_name = "demo0127"
+ tags = {
+ "source" = "terraform"
}
+ subnet {
+ address_prefix = "15.0.0.0/16"
+ id = (known after apply)
+ name = "kubesubnet"
}
+ subnet {
+ address_prefix = "15.1.0.0/16"
+ id = (known after apply)
+ name = "appgwsubnet"
}
}
Plan: 9 to add, 0 to change, 0 to destroy.
Warning: "agent_pool_profile": [DEPRECATED] This has been replaced by `default_node_pool` and will be removed in version 2.0 of the AzureRM Provider
on resources.tf line 160, in resource "azurerm_kubernetes_cluster" "k8s":
160: resource "azurerm_kubernetes_cluster" "k8s" {
------------------------------------------------------------------------
This plan was saved to: out.plan
To perform exactly these actions, run the following command to apply:
terraform apply "out.plan"
5. terraform apply 명령을 실행하여 Kubernetes 클러스터를 만들 계획에 적용한다. Kubernetes 클러스터를 만드는 프로세스는 몇 분 정도 소요될 수도 있다.
| terraform apply out.plan |
zerobig@ZEROBIG-NT800:/mnt/d/Azure_DevOps_Study/20200127/terraform-aks-appgw-ingress$ terraform apply out.plan
Acquiring state lock. This may take a few moments...
azurerm_user_assigned_identity.testIdentity: Creating...
azurerm_virtual_network.test: Creating...
azurerm_public_ip.test: Creating...
azurerm_user_assigned_identity.testIdentity: Creation complete after 2s [id=/subscriptions/f8764f39-xxxx-xxxx-xxxx-77b286ed971b/resourcegroups/demo0127/providers/Microsoft.ManagedIdentity/userAssignedIdentities/identity1]
azurerm_role_assignment.ra2: Creating...
azurerm_public_ip.test: Creation complete after 3s [id=/subscriptions/f8764f39-xxxx-xxxx-xxxx-77b286ed971b/resourceGroups/demo0127/providers/Microsoft.Network/publicIPAddresses/publicIp1]
azurerm_role_assignment.ra2: Creation complete after 6s [id=/subscriptions/f8764f39-xxxx-xxxx-xxxx-77b286ed971b/resourcegroups/demo0127/providers/Microsoft.ManagedIdentity/userAssignedIdentities/identity1/providers/Microsoft.Authorization/roleAssignments/07ad0b93-f9e9-268a-8f99-e222839de8c8]
azurerm_virtual_network.test: Still creating... [10s elapsed]
azurerm_virtual_network.test: Creation complete after 11s [id=/subscriptions/f8764f39-xxxx-xxxx-xxxx-77b286ed971b/resourceGroups/demo0127/providers/Microsoft.Network/virtualNetworks/aksVirtualNetwork]
data.azurerm_subnet.appgwsubnet: Refreshing state...
data.azurerm_subnet.kubesubnet: Refreshing state...
azurerm_role_assignment.ra1: Creating...
azurerm_application_gateway.network: Creating...
azurerm_role_assignment.ra1: Creation complete after 6s [id=/subscriptions/f8764f39-xxxx-xxxx-xxxx-77b286ed971b/resourceGroups/demo0127/providers/Microsoft.Network/virtualNetworks/aksVirtualNetwork/subnets/kubesubnet/providers/Microsoft.Authorization/roleAssignments/4d143196-092a-a1ea-7520-d8b88f8859cc]
azurerm_application_gateway.network: Still creating... [10s elapsed]
azurerm_application_gateway.network: Still creating... [20s elapsed]
azurerm_application_gateway.network: Still creating... [30s elapsed]
azurerm_application_gateway.network: Still creating... [40s elapsed]
azurerm_application_gateway.network: Still creating... [50s elapsed]
azurerm_application_gateway.network: Still creating... [1m0s elapsed]
azurerm_application_gateway.network: Still creating... [1m10s elapsed]
azurerm_application_gateway.network: Still creating... [1m20s elapsed]
azurerm_application_gateway.network: Still creating... [1m30s elapsed]
azurerm_application_gateway.network: Still creating... [1m40s elapsed]
azurerm_application_gateway.network: Still creating... [1m50s elapsed]
azurerm_application_gateway.network: Still creating... [2m0s elapsed]
azurerm_application_gateway.network: Still creating... [2m10s elapsed]
azurerm_application_gateway.network: Still creating... [2m20s elapsed]
azurerm_application_gateway.network: Still creating... [2m30s elapsed]
azurerm_application_gateway.network: Still creating... [2m40s elapsed]
azurerm_application_gateway.network: Still creating... [2m50s elapsed]
azurerm_application_gateway.network: Still creating... [3m0s elapsed]
azurerm_application_gateway.network: Still creating... [3m10s elapsed]
azurerm_application_gateway.network: Still creating... [3m20s elapsed]
azurerm_application_gateway.network: Still creating... [3m30s elapsed]
azurerm_application_gateway.network: Still creating... [3m40s elapsed]
azurerm_application_gateway.network: Still creating... [3m50s elapsed]
azurerm_application_gateway.network: Still creating... [4m0s elapsed]
azurerm_application_gateway.network: Still creating... [4m10s elapsed]
azurerm_application_gateway.network: Still creating... [4m20s elapsed]
azurerm_application_gateway.network: Still creating... [4m30s elapsed]
azurerm_application_gateway.network: Creation complete after 4m31s [id=/subscriptions/f8764f39-xxxx-xxxx-xxxx-77b286ed971b/resourceGroups/demo0127/providers/Microsoft.Network/applicationGateways/ApplicationGateway1]
azurerm_role_assignment.ra4: Creating...
azurerm_role_assignment.ra3: Creating...
azurerm_kubernetes_cluster.k8s: Creating...
azurerm_role_assignment.ra3: Creation complete after 5s [id=/subscriptions/f8764f39-xxxx-xxxx-xxxx-77b286ed971b/resourceGroups/demo0127/providers/Microsoft.Network/applicationGateways/ApplicationGateway1/providers/Microsoft.Authorization/roleAssignments/eff279b0-5e58-75cf-a99d-ab5276d26265]
azurerm_role_assignment.ra4: Creation complete after 5s [id=/subscriptions/f8764f39-xxxx-xxxx-xxxx-77b286ed971b/resourceGroups/demo0127/providers/Microsoft.Authorization/roleAssignments/57b761b5-812e-118b-2761-6c69f05365ee]
azurerm_kubernetes_cluster.k8s: Still creating... [10s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [20s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [30s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [40s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [50s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [1m0s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [1m10s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [1m20s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [1m30s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [1m40s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [1m50s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [2m0s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [2m10s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [2m20s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [2m30s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [2m40s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [2m50s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [3m0s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [3m10s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [3m20s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [3m30s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [3m40s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [3m50s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [4m0s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [4m10s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [4m20s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [4m30s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [4m40s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [4m50s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [5m0s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [5m10s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [5m20s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [5m30s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [5m40s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [5m50s elapsed]
azurerm_kubernetes_cluster.k8s: Still creating... [6m0s elapsed]
azurerm_kubernetes_cluster.k8s: Creation complete after 6m6s [id=/subscriptions/f8764f39-xxxx-xxxx-xxxx-77b286ed971b/resourcegroups/demo0127/providers/Microsoft.ContainerService/managedClusters/zero-aks-cluster1]
Apply complete! Resources: 9 added, 0 changed, 0 destroyed.
Outputs:
client_certificate = LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk~~~~hhdHdMbwotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
client_key = LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSU~~~UWFaL3BDRHVzdXZmRDBNMUJ2NTFuZ3ByTkNLQT0KLS0tLS1FTkQgUlNBIFBSSVZBVEUgS0VZLS0tLS0K
cluster_ca_certificate = LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tL~~~~dmU2eHg5clptdTlqaGh5SEFZPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
cluster_password = 89146c5b49e4c0688022056ecd3cb63b5b5b867c2bcead752260bc783e2be0c85c8c3d125ed09cb34a0833f1a6e4e60b91c3c686b0156e5c8d0ad628ed04a4b9
cluster_username = clusterUser_demo0127_zero-aks-cluster1
host = https://zero-aks-2087fed3.hcp.koreacentral.azmk8s.io:443
identity_client_id = 400c0423-0650-4308-b0b5-ed9c3fc03f40
identity_resource_id = /subscriptions/f8764f39-xxxx-xxxx-xxxx-77b286ed971b/resourcegroups/demo0127/providers/Microsoft.ManagedIdentity/userAssignedIdentities/identity1
kube_config = apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1~~~~EFZPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
server: https://zero-aks-2087fed3.hcp.koreacentral.azmk8s.io:443
name: zero-aks-cluster1
contexts:
- context:
cluster: zero-aks-cluster1
user: clusterUser_demo0127_zero-aks-cluster1
name: zero-aks-cluster1
current-context: zero-aks-cluster1
kind: Config
preferences: {}
users:
- name: clusterUser_demo0127_zero-aks-cluster1
user:
client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0t~~~~eXZicWJJYzQzYThWa0hhdHdMbwotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlKS0FJQkFBS0NBZ0VBMVVMN0MxZkRnQy84K1FaUGErNGZaQmpRUjRy~~~~rL1F2cVYrMEhEaVY4UWFaL3BDRHVzdXZmRDBNMUJ2NTFuZ3ByTkNLQT0KLS0tLS1FTkQgUlNBIFBSSVZBVEUgS0VZLS0tLS0K
token: 89146c5b49e4c0688022056ecd3cb63b5b5b867c2bcead752260bc783e2be0c85c8c3d125ed09cb34a0833f1a6e4e60b91c3c686b0156e5c8d0ad628ed04a4b9
Kubernetes 클러스터 테스트
새로 만든 클러스터를 Kubernetes 도구를 사용해 확인할 수 있다.
1. Terraform 상태에서 Kubernetes 구성을 가져오고 이를 kubectl이 읽을 수 있는 파일에 저장한다.
| echo "$(terraform output kube_config)" > ./azurek8s |
zerobig@ZEROBIG-NT800:/mnt/d/Azure_DevOps_Study/20200127/terraform-aks-appgw-ingress$ echo "$(terraform output kube_config)" > ./azurek8s
2. kubectl이 올바른 구성을 선택할 수 있도록 환경 변수를 설정한다.
| export KUBECONFIG=./azurek8s |
zerobig@ZEROBIG-NT800:/mnt/d/Azure_DevOps_Study/20200127/terraform-aks-appgw-ingress$ export KUBECONFIG=./azurek8s
3. 클러스터 상태를 확인한다.
| kubectl get nodes |
zerobig@ZEROBIG-NT800:/mnt/d/Azure_DevOps_Study/20200127/terraform-aks-appgw-ingress$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
aks-agentpool-38651548-0 Ready agent 2m49s v1.14.8
aks-agentpool-38651548-1 Ready agent 2m40s v1.14.8워커 노드의 세부 정보가 표시된다. 노드는 다음 이미지에 나온 대로 모두 Ready 상태여야 한다.
Azure AD Pod ID 설치
Azure Active Directory Pod ID는 Azure Resource Manager에 대한 토큰 기반 액세스를 제공한다.
Azure AD Pod ID는 Kubernetes 클러스터에 다음 구성 요소를 추가한다.
- Kubernetes CRD: AzureIdentity, AzureAssignedIdentity, AzureIdentityBinding
- MIC(Managed Identity Controller) 구성 요소
- NMI(Node Managed Identity) 구성 요소
RBAC를 사용하도록 설정한 경우 다음 명령을 실행하여 Azure AD Pod ID를 클러스터에 설치한다.
| kubectl create -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/deployment-rbac.yaml |
zerobig@ZEROBIG-NT800:/mnt/d/Azure_DevOps_Study/20200127/terraform-aks-appgw-ingress$ kubectl create -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/deployment-rbac.yaml
serviceaccount/aad-pod-id-nmi-service-account created
customresourcedefinition.apiextensions.k8s.io/azureassignedidentities.aadpodidentity.k8s.io created
customresourcedefinition.apiextensions.k8s.io/azureidentitybindings.aadpodidentity.k8s.io created
customresourcedefinition.apiextensions.k8s.io/azureidentities.aadpodidentity.k8s.io created
customresourcedefinition.apiextensions.k8s.io/azurepodidentityexceptions.aadpodidentity.k8s.io created
clusterrole.rbac.authorization.k8s.io/aad-pod-id-nmi-role created
clusterrolebinding.rbac.authorization.k8s.io/aad-pod-id-nmi-binding created
daemonset.apps/nmi created
serviceaccount/aad-pod-id-mic-service-account created
clusterrole.rbac.authorization.k8s.io/aad-pod-id-mic-role created
clusterrolebinding.rbac.authorization.k8s.io/aad-pod-id-mic-binding created
deployment.apps/mic created
만약, RBAC를 사용하지 않도록 설정한 경우 다음 명령을 실행하여 Azure AD Pod ID를 클러스터에 설치한다.
| kubectl create -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/deployment.yaml |
Helm 설치
이 섹션의 코드는 Helm(Kubernetes 패키지 관리자)을 사용하여 application-gateway-kubernetes-ingress 패키지를 설치한다.
1. RBAC를 사용하도록 설정한 경우 다음 명령 집합을 실행하여 Helm을 설치하고 구성한다.
| kubectl create serviceaccount --namespace kube-system tiller-sa kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller-sa helm init --tiller-namespace kube-system --service-account tiller-sa |
zerobig@ZEROBIG-NT800:/mnt/d/Azure_DevOps_Study/20200127/terraform-aks-appgw-ingress$ kubectl create serviceaccount --namespace kube-system tiller-sa
serviceaccount/tiller-sa created
zerobig@ZEROBIG-NT800:/mnt/d/Azure_DevOps_Study/20200127/terraform-aks-appgw-ingress$ kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller-sa
clusterrolebinding.rbac.authorization.k8s.io/tiller-cluster-rule created
zerobig@ZEROBIG-NT800:/mnt/d/Azure_DevOps_Study/20200127/terraform-aks-appgw-ingress$ helm init --tiller-namespace kube-system --service-account tiller-sa
$HELM_HOME has been configured at /home/zerobig/.helm.
Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.
Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
2. RBAC를 사용하지 않도록 설정한 경우 다음 명령을 실행하여 Helm을 설치하고 구성한다.
| helm init |
3. AGIC 리포지토리를 추가한다.
| helm repo add application-gateway-kubernetes-ingress https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/ helm repo update |
zerobig@ZEROBIG-NT800:/mnt/d/Azure_DevOps_Study/20200127/terraform-aks-appgw-ingress$ helm repo add application-gateway-kubernetes-ingress https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/
"application-gateway-kubernetes-ingress" has been added to your repositories
zerobig@ZEROBIG-NT800:/mnt/d/Azure_DevOps_Study/20200127/terraform-aks-appgw-ingress$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Successfully got an update from the "application-gateway-kubernetes-ingress" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete.
수신 컨트롤러 Helm 차트 설치
1. helm-config.yaml을 다운로드하여 AGIC를 구성한다.
| wget https://raw.githubusercontent.com/Azure/application-gateway-kubernetes-ingress/master/docs/examples/sample-helm-config.yaml -O helm-config.yaml |
zerobig@ZEROBIG-NT800:/mnt/d/Azure_DevOps_Study/20200127/terraform-aks-appgw-ingress$ wget https://raw.githubusercontent.com/Azure/application-gateway-kubernetes-ingress/master/docs/examples/sample-helm-config.yaml -O helm-config.yaml
Will not apply HSTS. The HSTS database must be a regular and non-world-writable file.
ERROR: could not open HSTS store at '/home/zerobig/.wget-hsts'. HSTS will be disabled.
--2020-01-27 11:01:23-- https://raw.githubusercontent.com/Azure/application-gateway-kubernetes-ingress/master/docs/examples/sample-helm-config.yaml
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.40.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.40.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1997 (2.0K) [text/plain]
Saving to: ‘helm-config.yaml’
helm-config.yaml 100%[=======================================================================================================>] 1.95K --.-KB/s in 0s
2020-01-27 11:01:26 (7.08 MB/s) - ‘helm-config.yaml’ saved [1997/1997]
2. helm-config.yaml을 편집하고 appgw와 armAuth 섹션에 적절한 값을 입력한다.
# This file contains the essential configs for the ingress controller helm chart
# Verbosity level of the App Gateway Ingress Controller
verbosityLevel: 3
################################################################################
# Specify which application gateway the ingress controller will manage
#
appgw:
subscriptionId: f8764f39-xxxx-xxxx-xxxx-77b286ed971b
resourceGroup: demo0127
name: ApplicationGateway1
usePrivateIP: false
# Setting appgw.shared to "true" will create an AzureIngressProhibitedTarget CRD.
# This prohibits AGIC from applying config for any host/path.
# Use "kubectl get AzureIngressProhibitedTargets" to view and change this.
shared: false
################################################################################
# Specify which kubernetes namespace the ingress controller will watch
# Default value is "default"
# Leaving this variable out or setting it to blank or empty string would
# result in Ingress Controller observing all acessible namespaces.
#
# kubernetes:
# watchNamespace: <namespace>
################################################################################
# Specify the authentication with Azure Resource Manager
#
# Two authentication methods are available:
# - Option 1: AAD-Pod-Identity (https://github.com/Azure/aad-pod-identity)
armAuth:
type: aadPodIdentity
identityResourceID: /subscriptions/f8764f39-xxxx-xxxx-xxxx-77b286ed971b/resourcegroups/demo0127/providers/Microsoft.ManagedIdentity/userAssignedIdentities/identity1
identityClientID: 400c0423-0650-4308-b0b5-ed9c3fc03f40
## Alternatively you can use Service Principal credentials
# armAuth:
# type: servicePrincipal
# secretJSON: <<Generate this value with: "az ad sp create-for-rbac --subscription <subscription-uuid> --sdk-auth | base64 -w0" >>
################################################################################
# Specify if the cluster is RBAC enabled or not
rbac:
enabled: false # true/false
# Specify aks cluster related information. THIS IS BEING DEPRECATED.
aksClusterConfiguration:
apiServerAddress: zero-aks-2087fed3.hcp.koreacentral.azmk8s.io값에 대한 설명은 다음과 같다.
- verbosityLevel: AGIC 로깅 인프라의 세부 정보 표시 수준을 설정한다. 가능한 값은 로깅 수준을 참조한다.
- appgw.subscriptionId: App Gateway의 Azure 구독 ID 다. 예: a123b234-a3b4-557d-b2df-a0bc12de1234
- appgw.resourceGroup: App Gateway가 생성된 Azure Resource Group의 이름이다.
- appgw.name: Application Gateway의 이름이다. 예: applicationgateway1.
- appgw.shared: 이 부울 플래그는 기본적으로 false로 설정되어야 한다. 공유 App Gateway가 필요하면 true로 설정한다.
- kubernetes.watchNamespace: AGIC가 조사할 네임스페이스를 지정한다. 네임스페이스는 단일 문자열 값 또는 쉼표로 구분된 네임스페이스 목록일 수 있다. 이 변수를 주석으로 처리하거나 공백 또는 빈 문자열로 설정하면 수신 컨트롤러에서 액세스 가능한 모든 네임스페이스를 관찰한다.
- armAuth.type: 값은 aadPodIdentity 또는 servicePrincipal이다.
- armAuth.identityResourceID: 관리 ID의 리소스 ID이다.
- armAuth.identityClientId: ID의 클라이언트 ID이다.
- armAuth.secretJSON: 서비스 주체 비밀 유형을 선택한 경우(armAuth.type을 servicePrincipal로 설정한 경우)에만 필요하다.
주요 참고 사항:
- identityResourceID 값은 terraform 스크립트에서 생성되고 echo "$(terraform output identity_resource_id)"를 실행하여 찾을 수 있다.
- identityClientID 값은 terraform 스크립트에서 생성되고 echo "$(terraform output identity_client_id)"를 실행하여 찾을 수 있다.
- <resource-group> 값은 App Gateway의 리소스 그룹이다.
- <identity-name> 값은 생성된 ID의 이름이다.
- 지정된 구독에 대한 모든 ID는 az identity list를 사용하여 나열할 수 있다.
3. Application Gateway 수신 컨트롤러 패키지 설치
| helm install -f helm-config.yaml application-gateway-kubernetes-ingress/ingress-azure |
zerobig@ZEROBIG-NT800:/mnt/d/Azure_DevOps_Study/20200127/terraform-aks-appgw-ingress$ helm install -f helm-config.yaml application-gateway-kubernetes-ingress/ingress-azure
NAME: wandering-jackal
LAST DEPLOYED: Mon Jan 27 11:04:43 2020
NAMESPACE: default
STATUS: DEPLOYED
RESOURCES:
==> v1/AzureIdentity
NAME AGE
wandering-jackal-azid-ingress-azure 1s
==> v1/AzureIdentityBinding
NAME AGE
wandering-jackal-azidbinding-ingress-azure 1s
==> v1/ConfigMap
NAME AGE
wandering-jackal-cm-ingress-azure 1s
==> v1/ServiceAccount
NAME AGE
wandering-jackal-sa-ingress-azure 1s
==> v1beta2/Deployment
NAME AGE
wandering-jackal-ingress-azure 1s
NOTES:
Thank you for installing ingress-azure:1.0.0.
Your release is named wandering-jackal.
The controller is deployed in deployment wandering-jackal-ingress-azure.
Configuration Details:
----------------------
* AzureRM Authentication Method:
- Use AAD-Pod-Identity
* Application Gateway:
- Subscription ID : f8764f39-xxxx-xxxx-xxxx-77b286ed971b
- Resource Group : demo0127
- Application Gateway Name : ApplicationGateway1
* Kubernetes Ingress Controller:
- Watching All Namespaces
- Verbosity level: 3
Please make sure the associated aadpodidentity and aadpodidbinding is configured.
For more information on AAD-Pod-Identity, please visit https://github.com/Azure/aad-pod-identity
배포 상태를 확인해 본다. quarrelsome-rat-ingress-azure-7787d5b596-hr8fs가 Running 상태이면 정상이다.
zerobig@ZEROBIG-NT800:/mnt/d/Azure_DevOps_Study/20200127/terraform-aks-appgw-ingress$ kubectl get pods
NAME READY STATUS RESTARTS AGE
mic-5446c87ff5-ghgkw 1/1 Running 0 6m
mic-5446c87ff5-zfk6c 1/1 Running 0 6m
nmi-7zmwc 1/1 Running 0 6m
nmi-t4ldq 1/1 Running 0 6m
wandering-jackal-ingress-azure-d867d586b-2wstw 0/1 Running 0 20s
zerobig@ZEROBIG-NT800:/mnt/d/Azure_DevOps_Study/20200127/terraform-aks-appgw-ingress$ kubectl get pods
NAME READY STATUS RESTARTS AGE
mic-5446c87ff5-ghgkw 1/1 Running 0 6m4s
mic-5446c87ff5-zfk6c 1/1 Running 0 6m4s
nmi-7zmwc 1/1 Running 0 6m4s
nmi-t4ldq 1/1 Running 0 6m4s
wandering-jackal-ingress-azure-d867d586b-2wstw 1/1 Running 0 24s
샘플 앱 설치
App Gateway, AKS 및 AGIC가 설치되면, Azure Cloud Shell을 통해 샘플 앱을 설치할 수 있다.
1. curl 명령을 사용하여 YAML 파일을 다운로드한다.
| curl https://raw.githubusercontent.com/Azure/application-gateway-kubernetes-ingress/master/docs/examples/aspnetapp.yaml -o aspnetapp.yaml |
zerobig@ZEROBIG-NT800:/mnt/d/Azure_DevOps_Study/20200127/terraform-aks-appgw-ingress$ curl https://raw.githubusercontent.com/Azure/application-gateway-kubernetes-ingress/master/docs/examples/aspnetapp.yaml -o aspnetapp.yaml
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 686 100 686 0 0 831 0 --:--:-- --:--:-- --:--:-- 831
2. YAML 파일을 적용한다.
| kubectl apply -f aspnetapp.yaml |
zerobig@ZEROBIG-NT800:/mnt/d/Azure_DevOps_Study/20200127/terraform-aks-appgw-ingress$ kubectl apply -f aspnetapp.yaml
pod/aspnetapp created
service/aspnetapp created
ingress.extensions/aspnetapp created
3. Ingress 배포 상태를 확인한다.
| kubectl get ingress |
zerobig@ZEROBIG-NT800:/mnt/d/Azure_DevOps_Study/20200127/terraform-aks-appgw-ingress$ kubectl get ingress
NAME HOSTS ADDRESS PORTS AGE
aspnetapp * 40.82.154.143 80 19s
참고로 부여받은 Public IP는 Application Gateway의 Public IP임을 알 수 있다.
4. 브라우저 창에서 결과를 확인한다.
리소스 정리
더 이상 필요하지 않은 경우 이 문서에서 만든 리소스를 삭제햔다.
자리 표시자를 적절한 값으로 바꾼다. 지정된 리소스 그룹 내의 모든 리소스가 삭제된다.
| az group delete -n <resource-group> |
zerobig@ZEROBIG-NT800:/mnt/d/Azure_DevOps_Study/20200127/terraform-aks-appgw-ingress$ az group delete -n demo0127
Are you sure you want to perform this operation? (y/n): y
'Azure와 함께 하는 DevOps' 카테고리의 다른 글
| 18 Terraform 및 HCL를 사용하여 Azure VM 클러스터 만들기 (0) | 2020.02.10 |
|---|---|
| 17 모듈 레지스트리를 사용하여 Terraform으로 Azure VM 클러스터 만들기 (0) | 2020.02.03 |
| 15 Terraform을 사용하여 Azure Kubernetes Service로 Kubernetes 클러스터 만들기 (0) | 2020.01.20 |
| 14 Terraform을 사용하여 Azure 배포 슬롯으로 인프라 프로비전 (0) | 2020.01.13 |
| 13 Terraform 설정과 구성 - 상태를 Azure Storage에 저장 (0) | 2020.01.06 |