Automated EMR Rehydration for Security Remediation

Vulnerability findings on EMR clusters required remediation across multiple teams and AWS accounts. Patching long-lived clusters in place was slow, inconsistent and hard to verify.

Several teams run EMR in their own AWS accounts with different EMR versions and environments. Remediation needed to be repeatable, and teams needed to run it without platform-team hand-holding.

1. 1.Chose immutable rehydration over in-place patching: rebuild on a known-good image instead of mutating running clusters.
2. 2.Built Packer templates that bake all required packages into custom AMIs.
3. 3.Provisioned EMR through Terraform Enterprise, so cluster configuration is versioned and consistent.
4. 4.Integrated Jenkins, Packer and TFE into one parameterized pipeline that teams run themselves.

• ▸Remediation is built into the image, and every cluster launches from a patched baseline.
• ▸Consistent images across accounts reduce configuration drift.
• ▸Pipeline runs leave an auditable record of what was built and deployed.

• ▸End-to-end pipeline from image build to cluster provisioning.
• ▸Parameterized for EMR version, environment and build variables.
• ▸Used by multiple teams across multiple AWS accounts.

• ▸Self-service EMR rehydration adopted by multiple teams across multiple AWS accounts.
• ▸Remediation became repeatable and standardized instead of manual and cluster-by-cluster.
• ▸Less manual work and less operational risk during remediation.

• ▸Immutable infrastructure turns remediation from an operation into a build.
• ▸Parameterize the pipeline for real variation (version, environment) and standardize everything else.