[Azure Tips] AKS 생성 명령 수행 시 Status Code = '400' 에러 발생 현상 해결책

<현상>

 

New-AzAks 명령(PowerShell)이나 az aks create 명령(bash)을 사용하여 AKS 생성 명령을 수행 시에 종종 서비스 주체 내 자격증명의 유효성 문제로 Status Code = '400'가 발생한다. 

az ad sp create-for-rbac 명령을 사용하여 이래 저래 시도하지만 잘 해결이 되지 않는 경우가 많다.

PS C:\AzureDevOpsDemo\20200229_dockerizing_demo> Write-Output "Creating new Azure Kubernetes Service cluster: $aks" -Verbose
New-AzAks `
    -Name $aks `
    -Location $location `
    -ResourceGroupName $rg `
    -NodeCount 2 `
    -KubernetesVersion $kubeVersion `
    -NodeVmSize $nodeVMSize `
    -Verbose
Write-Output "New Azure Kubernetes Service cluster: ($aks) created" -Verbose
Creating new Azure Kubernetes Service cluster: akszerodemo
VERBOSE: Cluster does not exist.
VERBOSE: Cluster is new.
VERBOSE: Performing the operation "Creating a managed Kubernetes cluster." on target "akszerodemo in rgzerodemo".
VERBOSE: Preparing for deployment of your managed Kubernetes cluster.
VERBOSE: Using DNS name prefix akszerodem-rgzerodemo-a791ad.
VERBOSE: Fetching SSH public key from file C:\Users\zerobig\.ssh\id_rsa.pub
VERBOSE: No Service Principal found in C:\Users\zerobig\.azure\acsServicePrincipal.json for this subscription. Creating a new Service Principal.
VERBOSE: Created a new Service Principal and assigned the contributor role for this subscription.
VERBOSE: Deploying your managed Kubernetes cluster.
New-AzAks : The credentials in ServicePrincipalProfile were invalid. Please see https://aka.ms/aks-sp-help for more details. (Details: adal: Refresh request failed. Status Code = '400'. Response body: {"error":"unauthorized_client","error_description":"AADSTS700016: Application with identifier '8aa1a27c-eaeb-463d-8d3a-0c269caa9f33' was not found in the directory 'bb43721a-680c-4c9f-b234-72c8fe6c8e3c'. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You may have sent your authentication request to the wrong tenant.\r\nTrace ID: 2a9b93b0-4b09-4c59-ba19-c91c0875b000\r\nCorrelation ID: 059a956c-06b4-4f88-a940-4ac932d68604\r\nTimestamp: 2020-02-29 04:24:44Z","error_codes":[700016],"timestamp":"2020-02-29 04:24:44Z","trace_id":"2a9b93b0-4b09-4c59-ba19-c91c0875b000","correlation_id":"059a956c-06b4-4f88-a940-4ac932d68604","error_uri":"https://login.microsoftonline.com/error?code=700016"})
At line:2 char:1
+ New-AzAks `
+ ~~~~~~~~~~~
+ CategoryInfo          : CloseError: (:) [New-AzAks], PSInvalidOperationException
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.Aks.NewAzureRmAks
 
New Azure Kubernetes Service cluster: (akszerodemo) created

 

 

 

<해결과정>

자신의 홈 디렉토리 내 az.sess이라는 파일을 삭제한다. 

PS C:\AzureDevOpsDemo\20200229_dockerizing_demo> dir C:\Users\zerobig\.Azure\

    -Name $aks `
    Directory: C:\Users\zerobig\.Azure
    -ResourceGroupName $rg `
Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----         2/29/2020  1:12 PM                commands
d-----         2/29/2020  1:24 PM                ErrorRecords
d-----         2/28/2020  8:48 AM                logs$aks) created" -Verbose
d-----         2/28/2020  8:48 AM                telemetry
-a----         2/28/2020  7:05 PM          14442 accessTokens.json
-a----         2/29/2020  1:24 PM            144 acsServicePrincipal.json
-a----         2/28/2020  8:48 AM              5 az.json
-a----         2/29/2020  1:12 PM              5 az.sess
-a----         2/29/2020  1:07 PM             35 AzInstallationChecks.json
-a----         2/28/2020  7:02 PM            792 azureProfile.json
-a----         2/28/2020  8:44 AM             34 AzurePSDataCollectionProfile.json
-a----         2/29/2020  1:20 PM           6334 AzureRmContext.json
-a----         2/28/2020  8:44 AM            195 AzureRmContextSettings.json
-a----         2/28/2020  7:02 PM             69 clouds.config
-a----         2/28/2020  8:48 AM             57 config
-a----         2/29/2020  1:12 PM             19 telemetry.txt
-a----         2/29/2020  1:22 PM          19333 TokenCache.dat

PS C:\AzureDevOpsDemo\20200229_dockerizing_demo> rm C:\Users\zerobig\.Azure\az.sess
PS C:\AzureDevOpsDemo\20200229_dockerizing_demo>

참고로 az.sess 파일을 삭제해도 마찬가지거나 이 파일 자체가 없을 경우, acsServicePrincipal.json 파일을 삭제한다. 그리고 1분(?) 내외로 시간이 지나서 다시 명령을 시도하면 정상적으로 배포가 이루어진다. 이렇게 해도 동일 현상이 반복되면 az login을 다시 수행하고 잠시 후 다시 생성 명령을 시도한다.

 

약 1분 정도 지난 후 다시 명령을 수행하면 정상적으로 처리되는 것을 확인할 수 있다.

PS C:\AzureDevOpsDemo\20200229_dockerizing_demo> Write-Output "Creating new Azure Kubernetes Service cluster: $aks" -Verbose
New-AzAks `
    -Name $aks `
    -Location $location `
    -ResourceGroupName $rg `
    -NodeCount 2 `
    -KubernetesVersion $kubeVersion `
    -NodeVmSize $nodeVMSize `
    -Verbose
Write-Output "New Azure Kubernetes Service cluster: ($aks) created" -Verbose
Creating new Azure Kubernetes Service cluster: akszerodemo
VERBOSE: Cluster does not exist.
VERBOSE: Cluster is new.
VERBOSE: Performing the operation "Creating a managed Kubernetes cluster." on target "akszerodemo in rgzerodemo".
VERBOSE: Preparing for deployment of your managed Kubernetes cluster.
VERBOSE: Using DNS name prefix akszerodem-rgzerodemo-a791ad.
VERBOSE: Fetching SSH public key from file C:\Users\zerobig\.ssh\id_rsa.pub
VERBOSE: Deploying your managed Kubernetes cluster.

ProvisioningState       : Succeeded
DnsPrefix               : akszerodem-rgzerodemo-a791ad
Fqdn                    : akszerodem-rgzerodemo-a791ad-347f3ee9.hcp.koreacentral.azmk8s.io
KubernetesVersion       : 1.14.8
AgentPoolProfiles       : {default}
LinuxProfile            : Microsoft.Azure.Commands.Aks.Models.PSContainerServiceLinuxProfile
ServicePrincipalProfile : Microsoft.Azure.Commands.Aks.Models.PSContainerServiceServicePrincipalProfile
Id                      : /subscriptions/a791ad38-xxxx-xxxx-xxxx-460ee318d917/resourcegroups/rgzerodemo/providers/Microsoft.ContainerService/managedClusters/akszerodemo
Name                    : akszerodemo
Type                    : Microsoft.ContainerService/ManagedClusters
Location                : koreacentral
Tags                    : {}

New Azure Kubernetes Service cluster: (akszerodemo) created

PS C:\AzureDevOpsDemo\20200229_dockerizing_demo>