Securing Automated Database Access – The New Stack

In my years as a software engineer, I’ve seen quite a few different approaches to managing CI/CD credentials, ranging from security to extremely bad.

A common approach I’ve come across is the “vault and forget” approach. In this approach, a team of engineers uses a credential manager like lastpass or 1Password to store their long-lived shared credentials that their various automation services use. These can be passwords, API tokens, certificates, SSH keys, etc., all residing in a central location. Then, when the automation service accounts are deployed, they send a request to the vault server, which then returns the key.

This way, no credentials need to be stored in git repositories or, heaven forbid, on sticky notes stuck on the monitor. While at first glance this might seem like a safe practice, storing these credentials in a vault introduces a litany of problems.

The problem with shared access

Kenneth DuMez

Kenneth joined Teleport in April 2022, after working at Pivotal and VMware developing Kubernetes build solutions. Currently, he is focused on getting developers to adopt Teleport, an open source secure access control scheme. He mainly spends his time producing written and visual content as well as traveling across the United States to attend different conferences, learning and training on various cloud security technologies.

Perhaps the biggest problem with sharing credentials is that these tokens are almost always long-lived. In some cases, they are even permanent, having no automatic expiration date.

This makes credential management much easier for engineers because they don’t have to rotate those keys and redeploy automation pipelines, causing production downtime.

However easy, it is NOT GOOD. Due to their long lifespan, if any of these credentials are leaked, attackers can use them indefinitely to grant access to secure systems and wreak havoc.

Without frequent rotation, former employees may still retain these credentials, leaving you open to potential leaks. And without proper monitoring and auditing of your various automated services, it can be difficult to determine exactly what credibility has been exposed, increasing the response time to put out the fire.

Type However Strength
Legacy authentication method
  • As a general rule, always use long-term credits
  • May not enforce RBAC roles
  • May not have a central audit log
  • Usually very heavy and resource-intensive
It’s better than managing passwords yourself. Only recommend it if you already use one and the cost of switching is too prohibitive. In other cases, you should probably upgrade.
Strong password encryption for authentication
  • Simple (no overhead)
  • Resistant to brute force attacks
  • Large blast radius in the event of an ID leak
  • Passwords should be rotated frequently to maintain a secure posture
Good for isolated, non-shared resources that have a small attack surface.
Certificate-based authentication
  • Short-lived, even if credibility is leaked, the time it is useful to attackers is minimal
  • Able to embed authorization and configuration directly into the credential itself
  • CA setup and maintenance can be complicated and expensive
  • Difficult to do ad hoc
Ideal for diverse and large-scale distributed infrastructures. Requires a certificate authority. Teleport is an open source product that provides a certificate authority as part of its access plan.

Rethinking automated access

Of course, this problem is not unique to automation; human engineers face the same issues and also have shared credentials to manage. For humans, these problems have largely been solved with identity-based access controls: binding a human engineer, known as “Alice Smith”, to a set of role-based access control (RBAC) roles , forcing him to authenticate with a single sign-on (SSO) like Okta for example to access sets of shared resources. At Teleport, while developing open source secure access tools for humans, we thought, “What if we could bring the same identity-based access to every microservice, bot, and resource in our CI/CD environment? And so Machine ID was born.

With MachineID, we aim to change the way you think about automation access, giving every resource and bot in your infrastructure a unique ID mapped to specific RBAC roles and authenticating every request with X certificates. .509 cartridges that automatically rotate in a finely configurable manner. Since each resource has its own identity, this makes it possible to audit all activities of different bots on a variety of resources in one central location.

Previously, if you really wanted to secure a database, you could set up your own certificate authority (CA) that would sign and issue SSL certificates whenever you wanted to add a service or user. It’s extremely expensive, however, and I’ve seen companies with entire teams dedicated to maintaining and servicing such systems. In contrast, Teleport acts as its own certificate authority, allowing automated service accounts to connect to the database using short-lived SSL certificates with database credentials and configuration. integrated.

Machine ID Demo

As an example of setting up an on-premises MySQL database with Teleport MachineID, we have created a small demo here. This demo is powered by Instruqt, which sets you up with a sandbox environment right from your browser.

Let’s go over what you can expect in the demo.

For this example, we’ll assume we have three nodes: the Teleport proxy, our MySQL database server, and our machine where we’ll be running our automated service account.

Step 1: Create the Teleport bot user

In this step, we will create our Teleport bot user. We will use this bot user to assign an identity to our automated service, configure RBAC, and allow us to audit all bot activity. Once we have created this user, we will then configure our bot service with MachineID using a short-lived token to connect to the Teleport proxy.

Step 2: Configure the database

Once we have our bot user, we will then configure our MySQL database server to connect to Teleport using SSL certificates generated from the Teleport proxy. You must first export the SSL database certificates generated by the proxy to the database host. Then, to configure MySQL to use these certificates, add the paths of the exported certificates to your MySQL configuration file, mysql.cnf Thus:

Additionally, your MySQL/MariaDB database user accounts should be configured to require a valid X509 client certificate by adding the REQUIRE SUBJECT field on the database user. This will later allow us to access our database remotely with our automated robot account.

Step 3: Start the MachineID service and connect to the database

Finally, we’ll start our MachineID service on the bot machine, open a proxy tunnel through the Teleport node to our database, and retrieve some of our data. Once this configuration is complete, you can configure automated services to securely access data in the database and modify it as you see fit. And all of this activity is logged centrally in the teleport node, giving you an easy-to-follow audit trail for your bot accounts.

It works through a lightweight agent called tbot. tbot is a small executable that comes with a standard Teleport installation. This binary communicates with the Teleport proxy to facilitate automatic reissuance of database certificates. These certificates contain both the identity of the bot user and the database credentials that are directly embedded in them once the bot user logs in with the single sign-on token as soon as the first step. Once tbot is authenticated, it will continuously receive reissues of these Teleport Proxy End User Access Certificates.

Conclusion

When thinking about automated database access, it’s really important to find a solution that minimizes the explosion radius when things go wrong. Whether that solution is something you build in-house from scratch or an open-source Teleport MachineID, having the same level of accountability for your human developers as your robotic developers is critical. 🤖

Characteristic picture Going through Unsplash.

Maria H. Underwood