Add AWS EC2 NitroTPM attestation and guest OS support#753
Conversation
d6fb1cd to
504e342
Compare
504e342 to
420ea41
Compare
420ea41 to
41f9b7e
Compare
41f9b7e to
41a4aa1
Compare
fe08404 to
4062083
Compare
Simplification pass (operator decisions)Rebased onto master (includes #777 empty TPM key-provider id) and applied:
Commit: Still follow-up as needed: Yocto UKI recipe stop embedding |
Follow-up commit
|
ca7909f to
c49b7cf
Compare
Add AWS EC2 NitroTPM as a first-class attestation platform under the account-admin-untrusted threat model, integrated like SEV-SNP rather than with AWS-specific auth surface. Measurement model (aligned with bare TDX): - Boot: PCR4/7/12 for AMI/UKI identity - Config: shared-disk MrConfigV3 measured into PCR8 (no UKI cmdline pin) - Events: all dstack events extend SHA384 PCR14 (RTMR3 analogue; no PCR23 launch/runtime split) - os_image_hash: prefer unified sha256(sha256sum.txt) via VmConfig.aws_measurement + reference PCR bind; legacy PCR-hash fallback - KMS GetAppKey: RA-TLS only (api_version=1); no recipient encryption - GetQuote/Attest: report_data only (no platform attestation_options) - check_kms / auth-simple: pin early mrAggregated only (boot-mr-done) Includes Yocto AWS kernel/image support, verifier/KMS paths, SDKs, docs, and release/smoke helpers. Empty TPM key-provider id (already on master) keeps early MR precomputable.
c49b7cf to
d8f4871
Compare
Local TPM sealing on AwsEc2 now binds only SHA384 PCR4/7/12 (boot/OS identity). PCR23 is unused after the single PCR14 event-lane model and is resettable, so it must not gate sealed app keys.
Local NitroTPM key-provider sealing now uses sha384:4,7,8,12,14 — boot path (4/7/12), shared-disk MrConfigV3 (8), and the dstack event lane at seal time (14). Seal runs after PCR8 measure and key-provider events, so reboot unseal matches the same intermediate PCR14 state.
Wire unified os_image_hash + aws_measurement end-to-end: - dstack-mr aws-measurement-cbor and assemble measurement.aws.cbor - guest quote embeds sys-config vm_config when aws_measurement binds attested PCR4/7/12 (legacy PCR-hash fallback otherwise) - dstack-cloud: platform=aws, prepare sys-config with aws_measurement, full EC2 lifecycle (AMI import, shared snapshot, run/stop/start/logs) - remove standalone aws-ec2-import/smoke/attach scripts; docs point to dstack-cloud. Keep image audit + release-manifest helpers only.
UKI assemble always precomputes NitroTPM PCR4/7/12 into measurement.aws.cbor and lists it in sha256sum.txt so digest.txt is fixed for the release. Host nitro-tpm-pcr-compute is preferred; Docker AL2023 is fallback; missing both fails the build. dstack-cloud prepare only embeds the release material and refuses packages that lack a checksum-committed measurement.aws.cbor.
AwsOsImageMeasurement now holds sha256(PCR4||PCR7||PCR12) instead of the three raw SHA384 PCR values, matching the legacy image-hash composition. hex encoding uses serde_human_bytes (hex_bytes alias). Verifier and guest quote compare digests; dstack-mr still takes PCR4/7/12 hex and folds them.
Drop panic from cbor_to_vec and the awkward context string parameter.
Callers attach context with .context("…") instead. to_cbor_vec,
from_cbor_slice, measurement_hash, from_measurement, decode_*, and
document verify now use anyhow::Result for a consistent error model.
4fe61c8 to
f6ed3c4
Compare
The pre-key-release check added no security guarantee: key release for an unauthorized compose is already blocked by KMS via the PCR14 event lane, and key use under a mismatched config is still rejected by the full mr_config check in verify_app before any key is used or persisted. The early check only duplicated the static-field validation for AWS, diverging from the other platforms. Remove it and let AWS follow the same single post-key verification path as TDX and SEV-SNP.
On the AWS EC2 platform the attestation mode must be NitroTPM only. Exposing TDX or SEV-SNP devices on an AWS instance must not silently switch the attestation mode away from the AWS platform path.
The extra sha384(config_id) before the PCR8 extend added no security (PCR extend already hashes) and hid the config id version byte behind a second, non-obvious hash step for verifiers. Extend the raw 48-byte config id directly so PCR8 = sha384(0^48 || config_id), letting a verifier parse the claimed config id's version byte and recompute the register in one step. Also fix stale docs that still described the shared-disk MrConfigV3 / UKI-cmdline model: the runbook and security-model checklist now state the guest-computed MrConfig V2 id, and the dated AMI reproducibility manifest gets a supersession note instead of rewriting its evidence.
the wrapper was a pure delegation to emit_runtime_event; call sites now use the original name again, matching main.
Restore pre-existing, non-AWS code that had been refactored or churned but was not required by the NitroTPM feature: - nsm-qvl: revert cert-chain refactor (verify.rs) and lib.rs to base, keeping only the AWS-required #[serde(alias = "nitrotpm_pcrs")] on pcrs. - tpm-attest/gcp_ak.rs: revert GCP-only quote refactor to base. - kms proto: drop never-shipped GetAppKey v2 fields (9/10); revert AppKeyResponse construction to the direct form. - kms/dstack-util: drop unused deps (aes-gcm, prost x2). - dstack-types/mr_config: drop unused MrConfigV3::to_mr_config_id / mr_config_id_from_document scaffolding; keep the tdx_-named API. - auth-simple: revert behavior-neutral empty-allowlist guard. - SDK js/python: revert attestation-payload helper extraction; keep the new 'attestation' response field. - dstack-auth: drop stray comment. Kept intentionally: dstack-types fallible CBOR refactor and the verifier --verify-cert mode (per request). Skipped: tpm2 internal comment/test-rename churn (interleaved with required AWS additions) and dstack-cloud (genuinely AWS-refactored, help text updated for --platform).
decode_mr_aws_nitro_tpm_from_pcrs bound the quoted PCR14 against a boot-mr-done-truncated replay when use_boottime_mr was set, so a full runtime quote (PCR14 covering the whole event log) always failed with 'PCR14 mismatch'. This made KMS SignCert (the only caller passing use_boottime_mr=true) unusable on AWS NitroTPM. Align with the TDX (decode_mr_tdx_from_quote) and GCP (decode_mr_gcp_tpm) architecture: the event-log-to-register binding always uses the full replay, and the boot-time snapshot boundary is applied only to the MR value. aws_nitro_tpm_verify_event_pcr now always binds the full replay (boottime param dropped); decode replays the truncated snapshot separately for mr_aggregated. The decode-side binding is kept as defense-in-depth (KMS reject tests exercise it), matching AWS's existing extra check rather than dropping it like TDX/GCP. Rewrite the full-log replay test for the corrected semantics and add a KMS regression test that a full runtime quote decoded with use_boottime_mr=true succeeds and yields the boot-time snapshot MR.
…hortcut PCR8 is not part of dstack's own verification path: the authoritative AWS NitroTPM app-identity binding is the PCR14 event-log replay. PCR8 carries a MrConfig V2 config commitment only so a lightweight third-party verifier can confirm the compose hash and key-provider by recomputing sha384(0^48 || config_id) instead of replaying the event log. Several docs described PCR8 as a required verifier check. Reword them (and the auth-simple README's mr_config_id mention) to present it as the optional shortcut it is, and note that dstack's verifier and KMS do not check it.
run_cert_oneshot previously emitted app_info.os_image_hash under is_valid:true without checking that it is bound to the attested boot measurement, so a relying party reading the output could trust a host claim. Add verify_cert_os_image_hash, which runs the existing CvmVerifier::verify_os_image_hash binding and reports the result as app_info.os_image_hash_verified. The binding is self-contained (no image download) for AWS NitroTPM, SEV-SNP, Nitro Enclave, GCP TDX, and TDX lite. TDX legacy would require an image download that this oneshot cert mode does not perform, so it is reported as os_image_hash_verified: false; use /verify for that path. Document the new field in the verifier README and require os_image_hash_verified == true in the AWS production runbook's jq check.
The FreshnessPolicy / verify_freshness_policy / details.freshness_verified machinery was added by this PR and has no in-tree consumer: KMS key release is replay-protected by the live RA-TLS handshake and timestamped signatures, and the gateway authorizes over live RA-TLS connections. Freshness for the external /verify path was an opt-in advisory field that nothing else depended on. Remove the request FreshnessPolicy, the freshness_verified detail field, verify_freshness_policy and its helpers (aws_nitro_tpm_report, attestation_timestamp_ms, ensure_expected_bytes, ensure_attestation_max_age, FRESHNESS_CLOCK_SKEW), and the four freshness tests. External /verify callers that need liveness embed their own report_data challenge and check it against the returned boot_info. Update the verifier README, verification.md, the AWS production runbook, and the dated evaluation doc's supersession note.
reuse policy_tcb_fields when filling VerificationDetails so details.tcb_status matches boot_info.tcbStatus (notably AWS NitroTPM, which is normalized to "UpToDate" for the auth gate but surfaced as null at the top level). also picks up the pending rustfmt import reorder that broke CI.
the "legacy paths without aws_measurement use the value directly" path no longer exists; aws_measurement is mandatory on AWS.
the KMS key-release pipeline gates non-SNP quotes through CvmVerifier::verify_os_image_hash; pin down that the AWS path rejects a vm_config without aws_measurement, accepts a unified vm_config bound to the quoted PCRs, and rejects a boot_pcr_digest mismatch.
- audit-aws-ec2-image-hardening.sh: a supplied --rootfs-squashfs with no unsquashfs available is now a FAIL instead of WARN + exit 0 - generate-aws-ec2-release-manifest.sh: capture the audit exit status explicitly so a failing audit prints its output instead of silently aborting the manifest run under set -e
hop limit 2 lets processes behind the docker bridge fetch IMDSv2 tokens, exposing instance-role credentials to app containers when iam_instance_profile is set. dstack itself never talks to IMDS, so default metadata_hop_limit to 1 and leave 2 as an explicit opt-in for deployments whose containers legitimately need the instance role.
generate_tpm_app_keys now fails closed on platforms whose TPM is not part of the platform trust model; document that in the app-compose key_provider field description.
drop the unpinned docker amazonlinux:2023 + dnf fallback from assemble.sh: the computed PCRs feed measurement.aws.cbor -> sha256sum.txt -> os_image_hash, i.e. the measurement the KMS and verifiers enforce, so only a version-pinned operator-installed nitro-tpm-pcr-compute may produce it.
release images and their evidence (sha256sum.txt, digest.txt, measurement.aws.cbor, aws-pcrs.json) are produced by the unified os/build.sh flow now, so drop generate-aws-ec2-release-manifest.sh and the reproducibility-manifest doc written around it. rework the production verifier runbook around the build.sh release package (reproducible rebuild + pinned host tools for reference measurements) and note the removal in the dated security evaluation. the hardening audit script stays.
Trim the superseded security evaluation to the live threat model and property table, document the actual UKI archive and PCR sidecar layout, and remove stale recipient/fallback claims. Also simplify the certificate output and build fragments referenced by the updated runbook.
Drop the write-only AppInfo.attestation field and its backend, simulator, SDK, and documentation plumbing. GetQuoteResponse.attestation remains the platform-agnostic evidence field.
Parameterize GCP/AWS vm-config generation, reuse one shared-disk builder with required-file checks, and remove dead config serialization and save helpers. Correct all project-file diagnostics to app.json.
Restore the established string-error measurement API while using or_panic for invariant-only CBOR encoding. Consolidate TCB and key-release policy helpers, move AWS image-binding tests to the verifier, and remove redundant tests, dead accessors, and unrelated TPM/NSM diff noise.
There was a problem hiding this comment.
Pull request overview
Adds end-to-end AWS EC2 NitroTPM support across the monorepo (guest OS build/measurement, attestation generation, verifier/KMS policy normalization, SDK surface updates, and operational documentation) under the account-admin-untrusted threat model.
Changes:
- Introduces AWS NitroTPM attestation collection/verification and AWS-specific OS-image binding (
aws_measurement, boot PCR digest) across attest/verifier/KMS. - Extends Yocto guest OS/kernel configuration and image assembly to produce AWS measurement artifacts and hardening audit tooling.
- Updates SDKs and docs to expose/use a platform-agnostic
attestationfield and to document AWS verification/operations.
Reviewed changes
Copilot reviewed 60 out of 62 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| sdk/python/src/dstack_sdk/dstack_client.py | Adds attestation field to quote response model. |
| sdk/js/src/index.ts | Adds attestation field to JS SDK quote response type. |
| sdk/go/dstack/client.go | Adds attestation field and decoding helper to Go SDK. |
| sdk/curl/api.md | Documents attestation in /GetQuote and cloud fields in /Info. |
| os/yocto/tools/README.md | Clarifies Yocto host tools scope and AWS tooling location. |
| os/yocto/tools/aws/audit-aws-ec2-image-hardening.sh | Adds AWS EC2 image hardening audit script. |
| os/yocto/layers/meta-dstack/recipes-kernel/linux/linux-yocto%.bbappend | Adds AWS kernel feature config sources and feature inclusion. |
| os/yocto/layers/meta-dstack/recipes-kernel/linux/files/dstack-aws.scc | Introduces Yocto kernel feature definition for AWS. |
| os/yocto/layers/meta-dstack/recipes-kernel/linux/files/dstack-aws.cfg | Adds kernel config needed for EC2 Nitro boot devices/console. |
| os/yocto/layers/meta-dstack/recipes-core/images/dstack-uki.bb | Documents AWS app/config binding expectations for UKI cmdline. |
| os/yocto/layers/meta-dstack/recipes-core/dstack-guest/dstack-guest.bb | Adds tpm2-tss build dependency for guest recipe. |
| os/image/README.md | Documents AWS measurement artifact and tool requirements. |
| os/image/assemble.sh | Generates and includes measurement.aws.cbor in image digest/tar flows. |
| dstack/vmm/src/app.rs | Initializes aws_measurement field in VM config assembly path. |
| dstack/verifier/src/verification.rs | Adds AWS NitroTPM OS-image verification and policy boot info construction. |
| dstack/verifier/src/types.rs | Adds PolicyBootInfo and includes it in verification details. |
| dstack/verifier/src/main.rs | Adds --verify-cert oneshot mode for RA-TLS cert verification. |
| dstack/verifier/src/lib.rs | Re-exports PolicyBootInfo and policy_tcb_fields. |
| dstack/verifier/README.md | Documents AWS mode, policy boot info, and --verify-cert workflow. |
| dstack/verifier/Cargo.toml | Gates binary build on binary feature (default-enabled). |
| dstack/tpm2/src/types.rs | Adds RSA EK template and configurable name algorithm for templates/NV. |
| dstack/tpm2/src/device.rs | Adds raw session command helpers and flexible auth-area builders. |
| dstack/tpm2/src/commands.rs | Adds raw execution, NV name read, auth NV read/write, capabilities helpers. |
| dstack/tpm-types/src/lib.rs | Adds SHA384 PCR selection helper. |
| dstack/tpm-qvl/src/lib.rs | Rejects AWS EC2 platform for GCP AK quote root retrieval. |
| dstack/tpm-attest/tests/tpm_quote_sample.README.md | Documents TPM quote parsing fixture semantics/provenance. |
| dstack/tpm-attest/src/lib.rs | Adds AWS NitroTPM PCR policy and single-PCR read helper. |
| dstack/ra-tls/src/attestation.rs | Adds RA-TLS cert verification APIs returning verified certificate evidence. |
| dstack/ra-tls/Cargo.toml | Adds tokio dev-dependency for async tests. |
| dstack/nsm-qvl/tests/nitro_attestation.README.md | Documents NSM attestation fixture capture/verification model. |
| dstack/nsm-qvl/src/lib.rs | Adds alias for NitroTPM PCR field name in attestation document. |
| dstack/kms/src/onboard_service.rs | Adds platform evidence mapping for AWS NitroTPM. |
| dstack/kms/src/main_service/upgrade_authority.rs | Reuses verifier’s canonical policy boot info and TCB normalization. |
| dstack/kms/src/main_service.rs | Adds AWS NitroTPM key-release gate and extensive AWS-specific tests. |
| dstack/kms/src/config.rs | Adds aws_nitro_tpm_key_release config flag. |
| dstack/kms/kms.toml | Documents and defaults AWS NitroTPM key-release gate to disabled. |
| dstack/kms/Cargo.toml | Adds cc-eventlog dev-dependency for AWS-related tests. |
| dstack/kms/auth-simple/README.md | Updates allowlisting guidance and adds AWS NitroTPM example. |
| dstack/kms/auth-simple/index.test.ts | Normalizes attestationMode casing and updates tests/messages. |
| dstack/kms/auth-simple/auth-config.example.json | Updates example KMS MR label to “early/boot-mr-done”. |
| dstack/dstack-util/src/system_setup/config_id_verifier.rs | Skips AWS PCR8 config-id cross-check (guest-measured). |
| dstack/dstack-util/src/system_setup.rs | Adds AWS instance binding and AWS-specific PCR measurements/policy. |
| dstack/dstack-util/src/main.rs | Allows emitting raw AWS NitroTPM attestation doc in quote command path. |
| dstack/dstack-util/src/host_api.rs | Skips host notify on AWS EC2 platform. |
| dstack/dstack-util/src/crypto.rs | Rejects all-zero X25519 shared secret during DH decrypt. |
| dstack/dstack-util/Cargo.toml | Adds tpm2 dependency to support AWS NitroTPM operations. |
| dstack/dstack-types/src/lib.rs | Adds AWS platform detection and AWS OS image measurement document types. |
| dstack/dstack-mr/src/main.rs | Adds AWS measurement encoding/inspection support. |
| dstack/dstack-attest/src/v1.rs | Adds AWS NitroTPM platform evidence variant. |
| dstack/dstack-attest/src/lib.rs | Extends measured-event emission to TPM PCRs (sha256/sha384) and adds PCR8 measure helper. |
| dstack/dstack-attest/src/aws_nitro_tpm.rs | Implements NitroTPM vendor-command flow to obtain attestation documents. |
| dstack/dstack-attest/src/attestation.rs | Adds AWS NitroTPM mode, quote/report verification, and MR derivation from PCRs. |
| dstack/dstack-attest/Cargo.toml | Adds AWS NitroTPM dependencies under quote feature. |
| dstack/Cargo.lock | Locks new dependencies for AWS NitroTPM functionality. |
| docs/verification.md | Adds platform-specific verification guidance including AWS NitroTPM. |
| docs/tutorials/attestation-verification.md | Clarifies TDX-specific flow and documents platform differences incl. AWS. |
| docs/security/security-model.md | Updates security model and measurement semantics for AWS NitroTPM mode. |
| docs/security/cvm-boundaries.md | Documents TPM key provider constraints and AWS measurement lane semantics. |
| docs/quickstart.md | Extends quickstart to include AWS NitroTPM option and config example. |
| docs/aws-ec2-production-verifier-runbook.md | New runbook for AWS EC2 NitroTPM production verification and ops. |
| docs/aws-attested-instance-security-evaluation.md | New AWS NitroTPM threat-model security evaluation document. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
GCP TDX boot regression — root-caused and fixed in 46fc3cfReal-machine testing of the unified 0.6.0 image found it reboot-looping on GCP TDX (c3-standard-4) before reaching Root cause: Bisect evidence — same failing image, only the UKI
Failure point: serial console shows the 16 GiB zero-on-free pass itself completes; the guest then dies silently (no panic, straight back to UEFI BdsDxe) inside The fix drops the option from |
|
128db22 restores the empty- |
Summary
Adds the complete AWS EC2 NitroTPM platform path under the account-admin-untrusted threat model. This PR now combines the former dstack#753 implementation with the Yocto image changes from Dstack-TEE/meta-dstack#79.
Attestation and key release
Guest OS
dstack-awsYocto kernel feature with built-in NVMe, ENA, UEFI/GPT, serial-console, and TPM CRB support;tpm2-tssto the dstack guest recipe build dependencies;os/yocto/tools/aws/and formal documentation underdocs/.Operations and documentation
--repo) rather than separate dstack/meta-dstack checkouts.Validation
cargo test -p dstack-attest aws_nitro_tpmcargo test -p dstack-kms aws_nitro_tpmcargo test -p dstack-verifier awscargo test -p dstack-guest-agent -p dstack-guest-agent-simulator -p ra-tls -p tpm-attest -p tpm-qvlnpm run test:runandnpm run lintindstack/kms/auth-simplebash -n/shellcheckthroughprek run --all-filesreuse lintforge test --ffiwas not rerun locally because Foundry is not installed in this environment; CI covers the Foundry suite.