How to renew a client secret without any downtime?

As an API consumer (web site, batch, ...), in order to invoke the APIs, you need a client oAuth2 application.

This oAuth2 application is used to obtain a JWT from an Authorization Server, by sending the client_id, client_secret and the requested scopes.

To reinforce the security of your credentials, you might like to regularly change ("rotate" hereafter) the client_secret of your oAuth2 applications and you wonder how to do this without any downtime? This document is here to help.

Introduction

Beware that, when you renew a client_secret, all applications that rely on the old client_secret will fail when they will try to retrieve the JWT from the Authorization server. When not properly dealt with, this situation will most certainly lead to an interruption of service for your application.

Indeed, in most consumer application configuration, each secret is stored as a string and directly used to retrieve the JWT from the Authorization server. To handle properly the secret rotation, you will need to review:

• The management of the configuration of the credentials
• The JWT retrieval strategy

Update the configuration of the credentials

To handle rotating secret without downtime, each oAuth2 application you want to prepare for rotating secret must update its configuration in order to:

• Have client_secret stored within an array of string (string[]) instead of simple string
• Have the release pipeline (CI/CD) updated so that it can feed an array of secrets, instead of just one. Of course, for now, this array will only contain one element.

As you made some modification to your configuration, your code needs to be updated to cope with it. At this stage, the simplest possible change is to simply use the first element of the array.

Update the JWT retrieval strategy

Now that your application credential configuration management have been updated to cope with an array of string, you will have to revisit the strategy to retrieve the JWT from the Authorization server. To do this we advise that:

• Your oAuth2 application will try to use the first client_secret from the array
  • Would the Authorization Server refuse to generate a JWT and return an authentication error, the application must try with the next client_secret from the array, until one of the request succeeded, or there is no more client_secret to try.
    Depending on the Authorization Server you rely on (Auth0...), the way authentication errors are described may be different, and so would be the way to identify them. For troubleshooting purposes, it is highly recommended to feed the log with a clear warning message when falling back to the next one to be tried.
  • Although this is not strictly part of the "rotating" strategy, your code should be ready to cope with a misconfiguration corner case: when all client_secrets have been tested and not one has been able to successfully generate a JWT. In that case, the service requiring this JWT will suffer from a downtime. Update your code to ensure this is properly logged and tracked.
• To ensure that your JWT retrieval strategy is working as expected, we suggest you to add automated tests to protect your codebase from potential regressions. Using a mock oAuth2 server such as oauth2-mock-server will make this pretty easy to set up. You could kick start this with the three following use cases:
  • The first secret in the array is valid
  • The first secret in the array is invalid and another secret in the array is valid
  • None of the secrets in the array are valid
• Deploy the behavioral update through all stages up to Production

A journey to rotate your client secret

Now that your configuration and strategy for the client rotation is ready, you will need to manually apply the client_secret change.

To guide you on this journey, we will consider that you will want to apply it on the Production client_id like KkyT0XKW2VfqS6Aw7 whose current client_secret is SeCr3t_1 and whose next client_secret should be SeCr3t_2. For this example you should:

1. Generate a new client secret compliant with the following rules:
   • Length = 64 characters
   • Shannon entropy score > 100
   • Contains lowercase characters (a-z)
   • Contains uppercase characters (A-Z)
   • Contains following special characters: -, _, +, = and .
   Example: 2cerfNrK7C_3fH1CB5a4pbgnrEpyOGeioopFoGomE1BmzjBjh7-VFU9eCDDNBo9b
2. Add this new secret to the configuration array storing the secrets for the given client_id
3. Deploy the change up to Production environment
   At this stage, your system should still be able to properly retrieve a JWT using the old secret.
4. Request to your AXA Partners business referent to change the secret of oAuth2 application from the old secret to new secret into the Authorization Server
   Email is the recommended channel, make sure to not put into the same communication thread the client_id and the client_secret (old and new) At this stage, your system should fail to retrieve the JWT using the old secret, fall back to the new secret and succeed.
5. Remove the old secret from the configuration array storing the secrets for the given client_id
6. Deploy the change up to Production environment
   At this stage, your system should properly retrieve the JWT using the secret new secret.