BigQuery Column-level security with Terraform

I’ve been working with one of Google’s newest additions to BigQuery — Column Level Security, a way to restrict access control all the way down to individual columns in a BigQuery Table. I couldn’t find anything online about how to easily do this with terraform, so here’s my go at it.

I’d suggest anyone interested in this topic to first take a read of the the official GCP documentation on this, it will cover the concepts behind this topic in greater detail.

Throughout this article I will be referring to the column level access control policies as “Policy Tags” and “CLS” for Column Level Security.

For my implementation I’ve used yaml configurations to allow users to create Policy Tags in one project, and Tables schemas with CLS in another. This implementation ensures slightly more security & governance controls but isn’t entirely necessary for use.

In GCP, the CLS starts with Data Catalogue, here you will create a Taxonomy and a Policy tag— the following code allows for multiple taxonomies within a project, multiple policy tags within that Taxonomy and multiple accesses within the Policy Tag.

You’ll also need to include the variable for using as a default & building upon.

This implementation doesn't account for any subtag Policy Tags (of which GCP allows 4 levels of subtags), and is built on an assumed file structure of alpha/beta/zeta project policy configurations.

Now you can create a new yaml file in the directory structure listed above, the Taxonomy takes a name, description and region; the policy tag takes a name, description and accesses. For accesses, you will need to use either roles of roles/datacatalog.categoryFineGrainedReader or roles/datacatalog.categoryAdmin to apply to whichever accounts are necessary to have access to your policies.

Here you can see I have created two Taxonomies, one for Finance in EU and one for HR in US, these policies have access control restricted to the User/Group/Service Account shown in the example. To use these new Policy Tags, you will need to take the Resource Name, and add it into a BigQuery table Schema.

Now that you have your Policy Tag and appropriate accesses created, you will need to apply the Policy Tag onto the BigQuery table. In my implementation I’ve also chosen to apply Table level access control — so that repository admins can determine if a given User/Group/Service account gets both CLS and Table access, or Table access only.

This implementation adds the tables to an existing dataset, and I’ve included additional security protections, by enforcing deletion_protection as default and requiring kms encryption.

I’ve also chosen to create a 1 json file= 1 BigQuery Dataset relationship in implementation, the json file follows a standard BigQuery schema format, an example json file showing multiple tables, and a CLS Policy Tag can be seen here:

Finally, you create a storage project configuration yaml to provide your table level access control, and list the tables within the dataset that you want to create.

Now you can apply Column Level Security to sensitive columns within your BigQuery data with Terraform — enjoy!