The Jenkins credential store in most enterprises is becoming a potential attack vector. It’s generally filled with long lived credentials, sometimes even to production systems.
In comes Hashicorp’s Vault, a Secret Management solution that enables the secure store of secrets, and dynamic generation of credentials for your job. Looks like a great match right? Look at the demo, certainly looks promising (specially with Jenkins beautiful new BlueOcean UI):
Interested? Let’s dive into it:
What is Hashicorp Vault?
Quite simply, is a tool for managing secrets. What’s really innovative about Vault is that it has methods for establishing both user and machine identity (through Auth Backends), so secrets can be consumed programatically. Identity is ultimately established by a (short lived) token. It also generates dynamic secrets on a number of backends, such as Cassandra, MySQL, PostgreSQL, SQL Server, MongoDB, etc. … I’m not going to cover here how to configure Vault, feel free to head out to the Vault documentation. For all intent and purposes of this document, you can download Vault and run it in “dev mode” for a quick test:
Please don’t run a Vault cluster in dev mode on production, feel free to reachout through the usual means if you need help.
AppRole is a secure introduction method to establish machine identity. In AppRole, in order for the application to get a token, it would need to login using a Role ID (which is static, and associated with a policy), and a Secret ID (which is dynamic, one time use, and can only be requested by a previously authenticated user/system. In this case, we have two options:
- Store the Role IDs in Jenkins
- Store the Role ID in the Jenkinsfile of each project
So let’s generate a policy for this role:
In this case, tokens assigned to the java-example policy would have permission to read a secret on the secret/hello path.
Now we have to create a Role that will generate tokens associated with that policy, and retrieve the token:
Note that in this case, the tokens generated through this policy have a time-to-live of 60 minutes. That means that after an hour, that token is expired and can’t be used anymore. If you’re Jenkins jobs are shorted, you can adjust that time to live now to increase security.
Let’s write a secret that our application will consume:
Now Jenkins will need permissions to retrieve Secret IDs for our newly created role. Jenkins shouldn’t be able to access the secret itself, list other Secret IDs, or even the Role ID.
And generate a token for Jenkins to login into Vault. This token should have a relatively large TTL, but will have to be rotated:
In this way we’re minimizing attack vectors:
Jenkins only knows it’s Vault Token (and potentially the Role ID) but doesn’t know the Secret ID, which is generated at pipeline runtime and it’s for one time use only.
The Role ID can be stored in the Jenkinsfile. Without a token and a Secret ID has no use.
The Secret ID is dynamic and one time use only, and only lives for a short period of time while it’s requested and a login process is carried out to obtain a token for the role.
The role token is short lived, and it will be useless once the pipeline finishes. It can even be revoked once you’re finished with your pipeline.
Jenkins pipeline and configuration
A full example for the project is available here. The Jenkinsfile will be using is this one:
Lot’s of stuff here, including certain Maven tasks, but we will be focusing on the Integration Tests stage (the last one), what we’re doing here is:
Downloading the Vault binary (in my case the ARM linux one, I’ll brag about that in a different blog post :D)
Reading credentials from the Jenkins credential store. In this case I’m storing both the ROLE_ID and the VAULT_TOKEN I’ve generated for Jenkins. As mentioned before if you want to split them for more security, you can just use the ROLE_ID as a variable in the Jenkinsfile.
I’m doing a set +x to disable verbosity in the shell in order not to leak credentials, although even with -x, in this case I’m just showing the Secret ID (which is useless after I’ve already used it), and the VAULT_TOKEN that I’m going to use to consume credentials, which is short lived, and can be revoked at the end of this runtime just adding a
I’m retrieving a SECRET_ID using Jenkins administrative token (that I’ve manually generated before, that’s the only one that would be relatively longed lived, but can only generate SECRET_IDs).
I’m doing an AppRole login with the ROLE_ID and the SECRET_ID, and storing that token (short lived).
My Java Process is reading VAULT_TOKEN and VAULT_ADDR to contact Vault and retrieve the secret we stored.
The VAULT_TOKEN (And optionally ROLE_ID) are stored in the Credential store in Jenkins: