Setting up Azure AD Application from Azure DevOps pipeline Powershell task, Part 2 of 2: The scripts

This one is about Microsoft Azure Cloud platform. Specifically about Azure DevOps Services. And specifically about accessing Azure Active Directory with appropriate permissions and changing settings there. For anybody knowing their way around Azure and Azure AD, there is nothing special about it. Tinkering with AD and what it contains is more or less business-as-usual. Doing the same without a mouse and keyboard is an another story.

Given the complexity of this topic, this part is for technical DevOps personnel. The previous part of this blog post was mostly about getting a reader aligned what the heck I'm talking about.

Disclaimer: If you are reading this and you're thinking this is some kind of gibberish magic, don't worry. You're not alone. This is about a detail of a detail of a detail most people will never need to know about. Those who do, might think of this as something too difficult to even attempt.

Access problem

In order to setup an own application to authenticate against Azure AD, a pre-announcement with parameters specific to this application needs to be done to AD. As a result, AD will assign set of IDs to identify this particular application and its chosen method of authentication. The Azure AD terminology is "App Registration" and "Enterprise application", both of which are effectively the same thing, your application from AD's point-of-view. Also both entries can be found from Azure Portal, Azure Active Directory menu with those words. As mentioned in part 1, all this setup can be done with a mouse and keyboard from Azure Portal. However, this time we choose to do it in an automated way.

To access Azure AD from a random machine via PowerShell, first you user account needs to unsurprisingly authenticate with a suitable user having enough permissions to do the change. For auth, you can use Connect-AzureAD -cmdlet from AzureAD-module. Module information is at https://docs.microsoft.com/en-us/powershell/module/azuread/?view=azureadps-2.0, Connect-AzureAD -cmdlet documentation is at https://docs.microsoft.com/en-us/powershell/module/azuread/connect-azuread?view=azureadps-2.0.

Investigation reveals, that for Connect-AzureAD -call to succeed, it requires one of these:

• -Credential -argument, that translates as username and password. However, service principal users used by Azure DevOps pipeline don't have an username to use. Service principals can have a password, but these accounts are not available for regular credential-based logins.
• -AccountId -argument having "documentation" of Specifies the ID of an account. You must specify the UPN of the user when authenticating with a user access token.
• -AadAccessToken -argument having "documentation" of Specifies a Azure Active Directory Graph access token.

The documentation is very vague at best. By testing we can learn, that a logged in user (DevOps service principal) running Azure PowerShell does have an Azure context. The context has the required Account ID, but not the required UPN, as it is not a regular user. When you login from a PowerShell prompt of your own workstation, a context can be used. Again, you're a human, not service principal.

Two out of three authentication options are gone. The last one of using an access token remains. Microsoft Graph API documents obtaining such an access token in https://docs.microsoft.com/en-us/azure/active-directory/develop/v1-protocols-openid-connect-code. A very short version of that can be found from my StackOverflow comment https://stackoverflow.com/a/54480804/1548275.

Briefly: On top of your Azure tenant ID, if you know your AD application's ID and client secret, getting the token can be done. The triplet in detail:

1. Tenant ID: Azure Portal, Azure Active Directory, Properties, Directory ID.
   • For an already logged in Azure user (Login-AzureRmAccount, in PowerShell up to 5.x)
   • PowerShell up to 5.x using AzureRM-library: Get-AzureRmSubscription
2. AD application ID: A service principal used for your Azure DevOps service connection.
   • You can list all available service principals and their details. The trick is to figure out the name of your Azure DevOps service connection service principal.
   • PowerShell up to 5.x using AzureRM-library: Get-AzureRmADServicePrincipal
   • To get the actual Application ID, affix the command with a:
     | Sort-Object -Property Displayname | Select-Object -Property Displayname,ApplicationId
   • Note: Below in this article, there is a separate chapter about getting your IDs, they are very critical in this operation.
3. Client secret: The password of the service principal used for your Azure DevOps service connection.

Problem:
First two can be easily found. What the client secret is, nobody knows. If you created our service connection like I did from Azure DevOps, the wizard creates everything automatically. On Azure-side, all user credentials are hashed beyond recovery. On Azure DevOps-side user credentials are encrypted and available for pipeline extensions using Azure SDK in environment variables. An Azure PowerShell task is NOT an extension and doesn't enjoy the privilege of receving much of the required details as free handouts.

To be complete, it is entirely possible to use Azure DevOps service connection wizard to create the service principal automatically and let it generate a random password for it. What you can do is reset the password on Azure AD to something of your own choosing. Just go reset the password for service connection in Azure DevOps too, and you can write your scripts using the known secret from some secret place. What that secret stash would be is up to you. Warning: Having the password as plain text in your pipeline wouldn't be a good choice.

Links

See, what other people in The Net have been doing and talking about their doings to overcome the described problem:

• Create AD application with VSTS task (in an Azure PowerShell task)
• VSTS Build and PowerShell and AzureAD Authentication (in an Azure PowerShell task, or in an Azure DevOps extension)
• Use AzureAD PowerShell cmdlets on VSTS agent (in an Azure DevOps extension)
• Azure AD Application Management -extension in Azure Marketplace

What you can do with the Azure AD Application Management -extension

In a pipeline, you can create, update, delete or get the application. Whoa! That's exactly what I need to do!

To create a new AD application, you need to specify the Name and Sign-on URL. Nothing more. Also, there is the problem. Any realistic application need to setup bunch of other settings to manifest and/or authentication.

A get AD application -operation will return following pipeline variables:

• ObjectId
• ApplicationId
• Name
• AppIdUri
• HomePageUrl
• ServicePrincipalObjectId

My approach

When eyeballing the App registration from Azure AD -side, it would look something like this:

The password (or secret) is hashed and gone. But what's the second option there? A certificate!

Documentation for Microsoft identity platform access tokens is at https://docs.microsoft.com/en-us/azure/active-directory/develop/access-tokens. When combined with previous link of Graph API documentation of OpenID Connect and Certificate credentials for application authentication from https://docs.microsoft.com/en-us/azure/active-directory/develop/active-directory-certificate-credentials we have all required information how to make this happen.

What you need for this to work

• PowerShell 5.x prompt with AzureRM and AzureAD modules installed
• Azure account
• Azure DevOps account
• Azure user with lots of access, I'm using a Global Admin role for my setups
• Azure DevOps user with lots of access, I'm using an Organization Owner role for my setups
• Azure Key Vault service up & running
• Azure DevOps pipeline almost up & almost running

Create a certificate into your Azure Key Vault

X.509 certificate

In PowerShell 5.x there is PKIClient-module, and New-SelfSignedCertificate cmdlet. In PowerShell 6.x, that particular module hasn't been ported yet. A few very useful cmdlets are using Windows-specific tricks to get the thing done. Since PowerShell 6.x (or PowerShell Core), is a multi-platform thing, the most complex ones have not been ported to macOS and Linux, so no Windows-version is available. However, connecting into Azure AD is done with AzureAD-module, which doesn't work with PowerShell 6.x, sorry. As much you and me both would love to go with the latest one, using 5.x is kinda mandatory for this operation.

Also, if you would have a self-signed certificate, what then? How would you use it in Azure DevOps pipeline? You wouldn't, the problem remains: you need to be able to pass the certificate to the pipeline task. Same as a password would be.

There is a working solution for this: Azure Key Vault. It is an encrypted storage for shared secrets, where you as an Azure admin can control on a fine-grained level who can access and what.

Your Azure DevOps service principal

If you haven't already done so, make sure you have a logged-in user in your PowerShell 5.x prompt. Hint: Login-AzureRmAccount cmdlet with your personal credentials for Azure Portal will get you a long way.

Next, you need to connect your logged in Azure administrator to a specific Azure AD for administering it. Run following spell to achieve that:

$currentAzureContext = Get-AzureRmContext;
$tenantId = $currentAzureContext.Tenant.Id;
$accountId = $currentAzureContext.Account.Id;
Connect-AzureAD -TenantId $tenantId -AccountId $accountId;

Now your user is connected to its "home" Azure AD. This seems bit over-complex, but you can actually connect to other Azure ADs where you might have permission to log into, so this complexity is needed.

As everything in Azure revolves around IDs. Your service principal has two very important IDs, which will be needed in various operations while granting permissions. The required IDs can be read from your Azure AD Application registration pages. My preference is to work from a PowerShell-session, I will be writing my pipeline tasks in PowerShell, so I choose to habit that realm.

As suggested earlier, run something like:

Get-AzureRmADServicePrincipal | Sort-Object -Property Displayname | Select-Object -Property Displayname,ApplicationId

will get you rolling. Carefully choose your Azure DevOps service principal from the list and capture it into a variable:

$adApp = Get-AzureADApplication -Filter "AppId eq '-Your-DevOps-application-ID-GUID-here-'"

Key Vault Access Policy

Here, I'm assuming you already have an Azure Key Vault service setup done and running Get-AzureRMKeyVault would return something useful for you (in Powershell 5.x).

To allow your DevOps service principal access the Key Vault, run:

$kv = Get-AzureRMKeyVault -Name
Set-AzureRmKeyVaultAccessPolicy -VaultName $kv.VaultName `
  -ServicePrincipalName $adApp.AppId `
  -PermissionsToKeys Get `
  -PermissionsToSecrets Get,Set `
  -PermissionsToCertificates Get,Create

X.509 certificate in a Key Vault

Azure Key Vault can generate self-signed certificates for you. Unlike a certificate and private key generated by you on a command-line, this one can be accessed remotely. You can read the certificate, set it as authentication mechanism for your DevOps service principal, and here comes the kicker: on an Azure DevOps pipeline task, you don't need to know the actual value of the certificate, all you need is a method for accessing it, when needed.

Create a certificate renewal policy into Azure Key Vault with:

$devOpsSpnCertificateName​ = "My cool DevOps auth cert";
$policy = New-AzureKeyVaultCertificatePolicy -SubjectName "CN=My DevOps SPN cert" `
  -IssuerName "Self" `
  -KeyType "RSA" `
  -KeyUsage "DigitalSignature" `
  -ValidityInMonths 12 `
  -RenewAtNumberOfDaysBeforeExpiry 60 `
  -KeyNotExportable:$False `
  -ReuseKeyOnRenewal:$False

Add-AzureKeyVaultCertificate -VaultName $kv.VaultName `
  -Name $devOpsSpnCertificateName `
  -CertificatePolicy $policy

This will instruct Key Vault to create a self-signed certificate by name My cool DevOps auth cert, and have it expire in 12 months. Also, it will auto-renew 60 days before expiry. At that point, it is advisable to set the new certificate into Azure AD App registration.

Now you have established a known source for certificates. You as an admin can access it, also your pipeline can access it.

Allow certificate authentication

To use this shiny new certificate for authentication, following spell needs to be run to first get the X.509 certificate, extract the required details out of it and allow using it as login credential:

$pfxSecret = Get-AzureKeyVaultSecret -VaultName $kv.VaultName `
  -Name $devOpsSpnCertificateName;
$pfxUnprotectedBytes = [Convert]::FromBase64String($pfxSecret.SecretValueText);
$pfx = New-Object Security.Cryptography.X509Certificates.X509Certificate2 -ArgumentList `
  $pfxUnprotectedBytes, $null,  `
  [Security.Cryptography.X509Certificates.X509KeyStorageFlags]::Exportable;

$validFrom = [datetime]::Parse($pfx.GetEffectiveDateString());
$validFrom = [System.TimeZoneInfo]::ConvertTimeBySystemTimeZoneId($validFrom, `
  [System.TimeZoneInfo]::Local.Id, 'GMT Standard Time');
$validTo = [datetime]::Parse($pfx.GetExpirationDateString());
$validTo = [System.TimeZoneInfo]::ConvertTimeBySystemTimeZoneId($validTo, `
  [System.TimeZoneInfo]::Local.Id, 'GMT Standard Time');

$base64Value = [System.Convert]::ToBase64String($pfx.GetRawCertData());
$base64Thumbprint = [System.Convert]::ToBase64String($pfx.GetCertHash());
$cred = New-AzureADApplicationKeyCredential -ObjectId $adApp.ObjectId `
  -CustomKeyIdentifier $base64Thumbprint `
  -Type AsymmetricX509Cert `
  -Usage Verify `
  -Value $base64Value `
  -StartDate $validFrom `
  -EndDate $validTo;

That's it! Now you're able to use the X.509 certificate from Azure Key Vault for logging in as Azure DevOps service principal.

Grant permissions to administer AD

The obvious final step is to allow the DevOps service principal to make changes in Azure AD. As default, it has no admin rights into Azure AD at all.

About level of access: I chose to go with Company Administrator, as I'm doing a setup for custom AD-domain. Without that requirement, an Application administrator would do the trick. Docs for different roles are at https://docs.microsoft.com/en-us/azure/active-directory/users-groups-roles/directory-assign-admin-roles.

So, the spell goes:

$roleName ='Company Administrator';
$role = Get-AzureADDirectoryRole | where-object {$_.DisplayName -eq $roleName};

Depending on your setup and what you've been doing in your AD before this, it is possible, that getting the role fails. Enable it with this one:

$roleTemplate = Get-AzureADDirectoryRoleTemplate | ? { $_.DisplayName -eq $roleName };
Enable-AzureADDirectoryRole -RoleTemplateId $roleTemplate.ObjectId;

Get the service principal, and grant the designated role for it:

$devOpsSpn = Get-AzureRmADServicePrincipal | `
  where-object {$_.ApplicationId -eq $adApp.AppId};
Add-AzureADDirectoryRoleMember -ObjectId $role.Objectid -RefObjectId $devOpsSpn.Id;

Now you're good to go.

What to do in a task

Lot of setup done already, now we're ready to go for the actual business.

Here is some code. First get the certificate from Key Vault, then connect the DevOps service principal into Azure AD:

$pfxSecret = Get-AzureKeyVaultSecret -VaultName $kv.VaultName `
  -Name $devOpsSpnCertificateName;
$pfxUnprotectedBytes = [Convert]::FromBase64String($pfxSecret.SecretValueText);
$pfx = New-Object Security.Cryptography.X509Certificates.X509Certificate2 -ArgumentList `   $pfxUnprotectedBytes, $null,  `   [Security.Cryptography.X509Certificates.X509KeyStorageFlags]::Exportable;

ConnectCurrentSessionToAzureAD $pfx​;

function ConnectCurrentSessionToAzureAD($cert) {
    $clientId = (Get-AzureRmContext).Account.Id;
    $tenantId = (Get-AzureRmSubscription).TenantId;
    $adTokenUrl = "https://login.microsoftonline.com/$tenantId/oauth2/token";
    $resource = "https://graph.windows.net/";

    $now = (Get-Date).ToUniversalTime();
    $nowTimeStamp = [System.Math]::Truncate((Get-Date -Date $now -UFormat %s -Millisecond 0));

    $thumbprint = [System.Convert]::ToBase64String($pfx.GetCertHash());
    $headerJson = @{
        alg = "RS256"
        typ = "JWT"
        x5t = $thumbprint
    } | ConvertTo-Json;
    $payloadJson = @{
        aud = $adTokenUrl
        nbf = $nowTimeStamp
        exp = ($nowTimeStamp + 3600)
        iss = $clientId
        jti = [System.Guid]::NewGuid().ToString()
        sub = $clientId
    } | ConvertTo-Json;
    $jwt = New-Jwt -Cert $cert -Header $headerJson -PayloadJson $payloadJson;

    $body = @{
        grant_type = "client_credentials"
        client_id = $clientId
        client_assertion_type = "urn:ietf:params:oauth:client-assertion-type:jwt-bearer"
        client_assertion = $jwt
        resource = $resource
    }

    $response = Invoke-RestMethod -Method 'Post' -Uri $adTokenUrl `
        -ContentType "application/x-www-form-urlencoded" -Body $body;
    $token = $response.access_token

    Connect-AzureAD -AadAccessToken $token -AccountId $clientId -TenantId $tenantId | Out-Null
}

Now, you're good to go with cmdlets like New-AzureADApplication and Set-AzureADApplication, or whatever you wanted to do with your Azure AD. Suggestions can be found from docs at https://docs.microsoft.com/en-us/powershell/module/azuread/?view=azureadps-2.0.

About PowerShell-modules in a task

This won't be a surprise to you: not all the modules you'll be needing are there in an Azure DevOps agent running your tasks. As an example, AzureAD won't be there, also the required JWT-module won't be there. What I'm doing in my pipeline task, is to install the requirements like this:

Install-Module -Name JWT -Scope CurrentUser -Force

Now the task won't ask anything (-Force) and it won't require administrator privileges to do the installation to a system-wide location (-Scope CurrentUser).

Finally

Phew! That was a lot to chew on. But now my entire appication is maintained via Azure DevOps release pipeline. If something goes wrong, I can always run a deployment in my environment setup pipeline and it will guarantee all the settings are as they should.