27 GCP와 Ansible을 활용한 쿠버네티스 클러스터 한방에 구성하기 2편

3. Ansible을 활용한 자동 배포 

 

먼저 관련 소스를 다운로드 받기 위해 git을 설치가 필요하다.

[root@zero-gcp-vmc ~]# yum install git
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.usc.edu
 * epel: mirror.layeronline.com
 * extras: mirror.mobap.edu
 * updates: mirror.chpc.utah.edu
Resolving Dependencies
<중략>
Installed:
  git.x86_64 0:1.8.3.1-20.el7                                                                                                                                                                                                               
 
Dependency Installed:
  perl-Error.noarch 1:0.17020-2.el7                                            perl-Git.noarch 0:1.8.3.1-20.el7                                            perl-TermReadKey.x86_64 0:2.30-20.el7                                          
 
Complete!

 

https://github.com/zer0big/ansible-k8s-cluster.git으로부터 설치 관련 소스를 clone 하고 해당 디렉토리로 이동한 뒤,

1편에서 생성해 둔 hosts 인벤토리 파일을 복사해 넣고

다운받은 소스 디렉토리 내 building-k8s-cluster.sh 파일에 실행권한을 준다.

[root@zero-gcp-vmc ~]# git clone https://github.com/zer0big/ansible-k8s-cluster.git
Cloning into 'ansible-k8s-cluster'...
remote: Enumerating objects: 11, done.
remote: Counting objects: 100% (11/11), done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 11 (delta 1), reused 11 (delta 1), pack-reused 0
Unpacking objects: 100% (11/11), done.
[root@zero-gcp-vmc ~]# cd ansible-k8s-cluster/
[root@zero-gcp-vmc ansible-k8s-cluster]# ls
building-k8s-cluster.sh  kube-dependencies.yml  master.yml  workers.yml
[root@zero-gcp-vmc ansible-k8s-cluster]# cp ~/hosts ./
[root@zero-gcp-vmc ansible-k8s-cluster]# chmod a+x building-k8s-cluster.sh

 

building-k8s-cluster.sh의 43라인에서 Master에 대한 호스트 네임을 자신의 상황에 맞게 수정한 뒤 쉘을 실행한다.

[root@zero-gcp-vmc ansible-k8s-cluster]# ./building-k8s-cluster.sh
==================================================================================================================================
Execute Ansible Playbook kube-dependencies.yml
==================================================================================================================================
 
PLAY [all] **********************************************************************************************************************************************************************************************************************************
 
TASK [Gathering Facts] **********************************************************************************************************************************************************************************************************************
ok: [worker2]
ok: [worker1]
ok: [master]
 
TASK [install Docker] ***********************************************************************************************************************************************************************************************************************
changed: [worker1]
changed: [worker2]
changed: [master]
 
TASK [start Docker] *************************************************************************************************************************************************************************************************************************
changed: [worker1]
changed: [worker2]
changed: [master]
 
TASK [disable SELinux] **********************************************************************************************************************************************************************************************************************
changed: [master]
changed: [worker2]
changed: [worker1]
 
TASK [disable SELinux on reboot] ************************************************************************************************************************************************************************************************************
 [WARNING]: SELinux state change will take effect next reboot
changed: [worker2]
changed: [worker1]
changed: [master]
 
TASK [ensure net.bridge.bridge-nf-call-ip6tables is set to 1] *******************************************************************************************************************************************************************************
changed: [worker1]
changed: [master]
changed: [worker2]
 
TASK [ensure net.bridge.bridge-nf-call-iptables is set to 1] ********************************************************************************************************************************************************************************
changed: [worker1]
changed: [worker2]
changed: [master]
 
TASK [add Kubernetes' YUM repository] *******************************************************************************************************************************************************************************************************
changed: [master]
changed: [worker1]
changed: [worker2]
 
TASK [install kubelet] **********************************************************************************************************************************************************************************************************************
changed: [worker2]
changed: [worker1]
changed: [master]
 
TASK [install kubeadm] **********************************************************************************************************************************************************************************************************************
changed: [worker1]
changed: [worker2]
changed: [master]
 
TASK [start kubelet] ************************************************************************************************************************************************************************************************************************
changed: [worker1]
changed: [worker2]
changed: [master]
 
PLAY [master] *******************************************************************************************************************************************************************************************************************************
 
TASK [Gathering Facts] **********************************************************************************************************************************************************************************************************************
ok: [master]
 
PLAY RECAP **********************************************************************************************************************************************************************************************************************************
master                     : ok=12   changed=10   unreachable=0    failed=0  
worker1                    : ok=11   changed=10   unreachable=0    failed=0  
worker2                    : ok=11   changed=10   unreachable=0    failed=0  
 
==================================================================================================================================
Execute Ansible Playbook master.yml
==================================================================================================================================
 
PLAY [master] *******************************************************************************************************************************************************************************************************************************
 
TASK [Gathering Facts] **********************************************************************************************************************************************************************************************************************
ok: [master]
 
TASK [initialize the cluster] ***************************************************************************************************************************************************************************************************************
changed: [master]
 
TASK [create .kube directory] ***************************************************************************************************************************************************************************************************************
changed: [master]
 
TASK [copy admin.conf to user's kube config] ************************************************************************************************************************************************************************************************
changed: [master]
 
TASK [install Pod network] ******************************************************************************************************************************************************************************************************************
changed: [master]
 
PLAY RECAP **********************************************************************************************************************************************************************************************************************************
master                     : ok=5    changed=4    unreachable=0    failed=0  
 
==================================================================================================================================
Install kubectl to local
==================================================================================================================================
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.usc.edu
 * epel: mirror.layeronline.com
 * extras: mirror.mobap.edu
 * updates: mirror.chpc.utah.edu
kubernetes/signature                                                                                                                                                                                                  |  454 B  00:00:00    
Retrieving key from https://packages.cloud.google.com/yum/doc/yum-key.gpg
Importing GPG key 0xA7317B0F:
 Userid     : "Google Cloud Packages Automatic Signing Key <gc-team@google.com>"
 Fingerprint: d0bc 747f d8ca f711 7500 d6fa 3746 c208 a731 7b0f
 From       : https://packages.cloud.google.com/yum/doc/yum-key.gpg
Retrieving key from https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
kubernetes/signature                                                                                                                                                                                                  | 1.4 kB  00:00:00 !!!
kubernetes/primary                                                                                                                                                                                                    |  47 kB  00:00:00    
kubernetes                                                                                                                                                                                                                           336/336
Resolving Dependencies
--> Running transaction check
---> Package kubectl.x86_64 0:1.14.0-0 will be installed
--> Finished Dependency Resolution
 
Dependencies Resolved
 
=============================================================================================================================================================================================================================================
 Package                                                Arch                                                  Version                                                  Repository                                                       Size
=============================================================================================================================================================================================================================================
Installing:
 kubectl                                                x86_64                                                1.14.0-0                                                 google-cloud-sdk                                                9.5 M
 
Transaction Summary
=============================================================================================================================================================================================================================================
Install  1 Package
 
Total download size: 9.5 M
Installed size: 41 M
Downloading packages:
2b52e839216dfc620bd1429cdb87d08d00516eaa75597ad4491a9c1e7db3c392-kubectl-1.14.0-0.x86_64.rpm                                                                                                                          | 9.5 MB  00:00:00    
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : kubectl-1.14.0-0.x86_64                                                                                                                                                                                                   1/1
  Verifying  : kubectl-1.14.0-0.x86_64                                                                                                                                                                                                   1/1
 
Installed:
  kubectl.x86_64 0:1.14.0-0                                                                                                                                                                                                                 
 
Complete!
==================================================================================================================================
Copy K8S Master kube config to local
==================================================================================================================================
admin.conf                                                                                                                                                                                                 100% 5446     2.8MB/s   00:00   
==================================================================================================================================
Check the cluster status
==================================================================================================================================
NAME             STATUS     ROLES    AGE   VERSION
k8s-master-gcp   NotReady   master   24s   v1.14.0
==================================================================================================================================
Execute Ansible Playbook worker.yml
==================================================================================================================================
 
PLAY [master] *******************************************************************************************************************************************************************************************************************************
 
TASK [get join command] *********************************************************************************************************************************************************************************************************************
changed: [master]
 
TASK [set join command] *********************************************************************************************************************************************************************************************************************
ok: [master]
 
PLAY [workers] ******************************************************************************************************************************************************************************************************************************
 
TASK [Gathering Facts] **********************************************************************************************************************************************************************************************************************
ok: [worker2]
ok: [worker1]
 
TASK [join cluster] *************************************************************************************************************************************************************************************************************************
changed: [worker2]
changed: [worker1]
 
PLAY RECAP **********************************************************************************************************************************************************************************************************************************
master                     : ok=2    changed=1    unreachable=0    failed=0  
worker1                    : ok=2    changed=1    unreachable=0    failed=0  
worker2                    : ok=2    changed=1    unreachable=0    failed=0  
 
==================================================================================================================================
Check the cluster status
==================================================================================================================================
NAME                STATUS     ROLES    AGE   VERSION
k8s-master-gcp      Ready      master   35s   v1.14.0
k8s-worker-01-gcp   NotReady   <none>   4s    v1.14.0
k8s-worker-02-gcp   NotReady   <none>   4s    v1.14.0

 

정상적으로 설치가 이루어 졌다면 최종 화면이 위와 같을 것이다. 

혹시 위와 같이 STATUS가  "NotReady" 인 경우, 잠시 후 다시 확인 해 보면 Ready로 업데이트 될 것이다.

[root@zero-gcp-vmc ansible-k8s-cluster]# kubectl get nodes
NAME                STATUS   ROLES    AGE     VERSION
k8s-master-gcp      Ready    master   3m14s   v1.14.0
k8s-worker-01-gcp   Ready    <none>   2m43s   v1.14.0
k8s-worker-02-gcp   Ready    <none>   2m43s   v1.14.0
[root@zero-gcp-vmc ansible-k8s-cluster]# kubectl version --short
Client Version: v1.14.0
Server Version: v1.14.0
[root@zero-gcp-vmc ansible-k8s-cluster]# kubectl cluster-info
Kubernetes master is running at https://10.146.0.8:6443
KubeDNS is running at https://10.146.0.8:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
 
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
[root@zero-gcp-vmc ansible-k8s-cluster]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                     READY   STATUS    RESTARTS   AGE
kube-system   coredns-fb8b8dccf-m6sx4                  1/1     Running   0          10m
kube-system   coredns-fb8b8dccf-z2tw7                  1/1     Running   0          10m
kube-system   etcd-k8s-master-gcp                      1/1     Running   0          9m26s
kube-system   kube-apiserver-k8s-master-gcp            1/1     Running   0          9m43s
kube-system   kube-controller-manager-k8s-master-gcp   1/1     Running   0          9m34s
kube-system   kube-flannel-ds-amd64-5ppkb              1/1     Running   0          10m
kube-system   kube-flannel-ds-amd64-lznml              1/1     Running   1          10m
kube-system   kube-flannel-ds-amd64-tjwq5              1/1     Running   0          10m
kube-system   kube-proxy-7jcrw                         1/1     Running   0          10m
kube-system   kube-proxy-l829n                         1/1     Running   0          10m
kube-system   kube-proxy-njg2p                         1/1     Running   0          10m
kube-system   kube-scheduler-k8s-master-gcp            1/1     Running   0          9m28s
[root@zero-gcp-vmc ansible-k8s-cluster]#

축하한다~!!!

드디어 최신 버전의 쿠버네티스가 정상적으로 설치된 것이다.

내친김에 K8S 대시보드도 띄어 보도록 하자.

 

 

4. K8S 대시보드 띄우기

[root@zero-gcp-vmc ansible-k8s-cluster]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                     READY   STATUS    RESTARTS   AGE
default       zero-nginx-7cc7fbcc5b-7znnn              1/1     Running   1          2d19h
default       zero-nginx-7cc7fbcc5b-n5dms              1/1     Running   1          2d19h
kube-system   coredns-fb8b8dccf-m6sx4                  1/1     Running   1          3d12h
kube-system   coredns-fb8b8dccf-z2tw7                  1/1     Running   1          3d12h
kube-system   etcd-k8s-master-gcp                      1/1     Running   1          3d11h
kube-system   kube-apiserver-k8s-master-gcp            1/1     Running   1          3d11h
kube-system   kube-controller-manager-k8s-master-gcp   1/1     Running   1          3d11h
kube-system   kube-flannel-ds-amd64-5ppkb              1/1     Running   1          3d12h
kube-system   kube-flannel-ds-amd64-lznml              1/1     Running   2          3d12h
kube-system   kube-flannel-ds-amd64-tjwq5              1/1     Running   1          3d12h
kube-system   kube-proxy-7jcrw                         1/1     Running   1          3d12h
kube-system   kube-proxy-l829n                         1/1     Running   1          3d12h
kube-system   kube-proxy-njg2p                         1/1     Running   1          3d12h
kube-system   kube-scheduler-k8s-master-gcp            1/1     Running   1          3d11h
[root@zero-gcp-vmc ansible-k8s-cluster]# curl -O https://raw.githubusercontent.com/kubernetes/dashboard/master/aio/deploy/recommended/kubernetes-dashboard.yaml  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4784  100  4784    0     0   8032      0 --:--:-- --:--:-- --:--:--  8053
[root@zero-gcp-vmc ansible-k8s-cluster]# kubectl create -f kubernetes-dashboard.yaml
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
serviceaccount/kubernetes-dashboard created
role.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
deployment.apps/kubernetes-dashboard created
service/kubernetes-dashboard created
[root@zero-gcp-vmc ansible-k8s-cluster]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                     READY   STATUS    RESTARTS   AGE
default       zero-nginx-7cc7fbcc5b-7znnn              1/1     Running   1          2d19h
default       zero-nginx-7cc7fbcc5b-n5dms              1/1     Running   1          2d19h
kube-system   coredns-fb8b8dccf-m6sx4                  1/1     Running   1          3d12h
kube-system   coredns-fb8b8dccf-z2tw7                  1/1     Running   1          3d12h
kube-system   etcd-k8s-master-gcp                      1/1     Running   1          3d12h
kube-system   kube-apiserver-k8s-master-gcp            1/1     Running   1          3d12h
kube-system   kube-controller-manager-k8s-master-gcp   1/1     Running   1          3d12h
kube-system   kube-flannel-ds-amd64-5ppkb              1/1     Running   1          3d12h
kube-system   kube-flannel-ds-amd64-lznml              1/1     Running   2          3d12h
kube-system   kube-flannel-ds-amd64-tjwq5              1/1     Running   1          3d12h
kube-system   kube-proxy-7jcrw                         1/1     Running   1          3d12h
kube-system   kube-proxy-l829n                         1/1     Running   1          3d12h
kube-system   kube-proxy-njg2p                         1/1     Running   1          3d12h
kube-system   kube-scheduler-k8s-master-gcp            1/1     Running   1          3d12h
kube-system   kubernetes-dashboard-5f7b999d65-n9cf4    1/1     Running   0          9s

 

localhost에서 대시보드를 접속할 수 있도록 xrdp를 설치한다. 

※ 1편 방화벽 허용 절차를 참고하여 zero-gcp-vmc에 대해 3389 포트 허용이 선행되어야 함!

[root@zero-gcp-vmc ansible-k8s-cluster]# rpm -Uvh http://mirror.centos.org/centos/7/extras/x86_64/Packages/epel-release-7-11.noarch.rpm
Retrieving http://mirror.centos.org/centos/7/extras/x86_64/Packages/epel-release-7-11.noarch.rpm
Preparing...                          ################################# [100%]
        package epel-release-7-11.noarch is already installed
[root@zero-gcp-vmc ansible-k8s-cluster]# rpm -Uvh http://li.nux.ro/download/nux/dextop/el7/x86_64/nux-dextop-release-0-1.el7.nux.noarch.rpm
Retrieving http://li.nux.ro/download/nux/dextop/el7/x86_64/nux-dextop-release-0-1.el7.nux.noarch.rpm
warning: /var/tmp/rpm-tmp.01NN5y: Header V4 RSA/SHA1 Signature, key ID 85c6cd8a: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:nux-dextop-release-0-1.el7.nux   ################################# [100%]
[root@zero-gcp-vmc ansible-k8s-cluster]# yum update
[root@zero-gcp-vmc ansible-k8s-cluster]# yum groupinstall "GNOME Desktop" "Graphical Administration Tools"
Loaded plugins: fastestmirror
There is no installed groups file.
Maybe run: yum groups mark convert (see man yum)
Loading mirror speeds from cached hostfile
<중략>
Complete!
[root@zero-gcp-vmc ansible-k8s-cluster]# ln -sf /lib/systemd/system/runlevel5.target /etc/systemd/system/default.target
[root@zero-gcp-vmc ansible-k8s-cluster]# yum -y install xrdp tigervnc-server
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirrordenver.fdcservers.net
 * epel: d2lzkl7pfhq30w.cloudfront.net
 * extras: centos.mirror.lstn.net
 * nux-dextop: mirror.li.nux.ro
 * updates: mirrors.umflint.edu
Resolving Dependencies
--> Running transaction check
---> Package tigervnc-server.x86_64 0:1.8.0-13.el7 will be installed
---> Package xrdp.x86_64 1:0.9.9-1.el7 will be installed
--> Processing Dependency: xrdp-selinux = 1:0.9.9-1.el7 for package: 1:xrdp-0.9.9-1.el7.x86_64
--> Processing Dependency: xorgxrdp for package: 1:xrdp-0.9.9-1.el7.x86_64
--> Running transaction check
---> Package xorgxrdp.x86_64 0:0.2.9-1.el7 will be installed
---> Package xrdp-selinux.x86_64 1:0.9.9-1.el7 will be installed
--> Finished Dependency Resolution
 
Dependencies Resolved
<중략>
Installed:
  tigervnc-server.x86_64 0:1.8.0-13.el7                                                                   xrdp.x86_64 1:0.9.9-1.el7                                                                 
 
Dependency Installed:
  xorgxrdp.x86_64 0:0.2.9-1.el7                                                                   xrdp-selinux.x86_64 1:0.9.9-1.el7                                                                 
 
Complete!
[root@zero-gcp-vmc ansible-k8s-cluster]# systemctl start xrdp.service
[root@zero-gcp-vmc ansible-k8s-cluster]# systemctl enable xrdp.service
Created symlink from /etc/systemd/system/multi-user.target.wants/xrdp.service to /usr/lib/systemd/system/xrdp.service.

 

터미널 창에 kubectl proxy를 입력하여 대시 보드에 액세스 할 수 있도록 해준다.

[root@zero-vmc-az ansible-k8s-cluster]# kubectl proxy
Starting to serve on 127.0.0.1:8001

 

원격데스크톱 연결을 통해 서버에 접속한다.

 

 

대시보드 접속을 위한 Token을 생성한다. (원격접속 터미널 상에서 시행)

 

브라우저 창에서 다음을 입력한다.

http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/ 

 

 

대시보드 접속을 위한 Token을 생성한다. (원격접속 터미널 상에서 시행)

 

[root@zero-gcp-vmc ~]# kubectl apply -f https://gist.githubusercontent.com/chukaofili/9e94d966e73566eba5abdca7ccb067e6/raw/0f17cd37d2932fb4c3a2e7f4434d08bc64432090/k8s-dashboard-admin-user.yaml
serviceaccount/admin-user created
clusterrolebinding.rbac.authorization.k8s.io/admin-user created
[root@zero-gcp-vmc ~]# kubectl get sa admin-user -n kube-system
NAME         SECRETS   AGE
admin-user   1         6s
[root@zero-gcp-vmc ~]# kubectl describe sa admin-user -n kube-system
Name:                admin-user
Namespace:           kube-system
Labels:              <none>
Annotations:         kubectl.kubernetes.io/last-applied-configuration:
                       {"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{},"name":"admin-user","namespace":"kube-system"}}
Image pull secrets:  <none>
Mountable secrets:   admin-user-token-tsrxb
Tokens:              admin-user-token-tsrxb
Events:              <none>
[root@zero-gcp-vmc ~]# kubectl describe secret admin-user-token-tsrxb -n kube-system
Name:         admin-user-token-tsrxb
Namespace:    kube-system
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: admin-user
              kubernetes.io/service-account.uid: d6de1afd-5407-11e9-a883-42010a920008
 
Type:  kubernetes.io/service-account-token
 
Data
====
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLXRzcnhiIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJkNmRlMWFmZC01NDA3LTExZTktYTg4My00MjAxMGE5MjAwMDgiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06YWRtaW4tdXNlciJ9.DCudHXmDCR6UAJUwLIHr77Fob9kA29xPn9mW9l-sZGiUlOywbOb-2OL2PDs01uGYd0AcvJCo2jlOpHlCYffPEXP7anS8wTfxEEQOsBbPsjCyhRuYFdKvtlzJwPcI4EtBU6OHBW07D9ApuwtetMfDes0PzaPTbYWiPC_drfqBHm0zZe39AhFQYdOw6LarvtTF-Y4kSbrdiJRgJqcRTMR7uyDTa131wokPMv3LOWfj8JrUEPUFRqVYYxpMRfH60H9P6uoyzN7Yh94tmg4hiv9CtATQ46WXDv3gj7dbwdu0HkoL8V2Dh_xv6T3UTKK_2wSR2PjVNMpu6Eti5tI_A1nOuA
ca.crt:     1025 bytes
namespace:  11 bytes

 

획득한 토큰값을 입력하여 로그인 한다.

 

 

정상적으로 대시보드가 나타남을 확인한다.

 

 

축하한다~~!!!

이제 원하는 나만의 쿠버네티스 클러스터가 구성되었으니, 마음 껏 K8S를 가지고 놀아보자~~~!^^