Skip to content

Add AWS EC2 NitroTPM attestation and guest OS support#753

Open
h4x3rotab wants to merge 39 commits into
masterfrom
aws-attestation-platform-spec
Open

Add AWS EC2 NitroTPM attestation and guest OS support#753
h4x3rotab wants to merge 39 commits into
masterfrom
aws-attestation-platform-spec

Conversation

@h4x3rotab

@h4x3rotab h4x3rotab commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds the complete AWS EC2 NitroTPM platform path under the account-admin-untrusted threat model. This PR now combines the former dstack#753 implementation with the Yocto image changes from Dstack-TEE/meta-dstack#79.

Attestation and key release

  • parse and verify AWS NitroTPM Attestation Documents against the AWS NitroTPM PKI;
  • replay PCR14 launch measurements and use PCR23 only for runtime telemetry;
  • bind boot PCRs, OS image hash, measured config/app identity, freshness, and recipient keys;
  • support AWS-aware KMS, verifier, RA-TLS, auth-simple, auth-eth, guest-agent, and SDK flows;
  • expose canonical AWS platform details without depending on AWS KMS.

Guest OS

  • add the dstack-aws Yocto kernel feature with built-in NVMe, ENA, UEFI/GPT, serial-console, and TPM CRB support;
  • add tpm2-tss to the dstack guest recipe build dependencies;
  • keep AWS image/release helpers under os/yocto/tools/aws/ and formal documentation under docs/.

Operations and documentation

  • add production verifier, reproducibility, image hardening, AMI promotion, and EC2 live-smoke documentation/tooling;
  • adapt release-manifest generation to the unified dstack monorepo (--repo) rather than separate dstack/meta-dstack checkouts.

Validation

  • cargo test -p dstack-attest aws_nitro_tpm
  • cargo test -p dstack-kms aws_nitro_tpm
  • cargo test -p dstack-verifier aws
  • cargo test -p dstack-guest-agent -p dstack-guest-agent-simulator -p ra-tls -p tpm-attest -p tpm-qvl
  • npm run test:run and npm run lint in dstack/kms/auth-simple
  • bash -n/shellcheck through prek run --all-files
  • reuse lint
  • the original branch also recorded a live EC2 smoke test against its promoted Attestable AMI

forge test --ffi was not rerun locally because Foundry is not installed in this environment; CI covers the Foundry suite.

Comment thread dstack/dstack-attest/src/aws_nitro_tpm.rs Fixed
@h4x3rotab h4x3rotab force-pushed the aws-attestation-platform-spec branch 6 times, most recently from d6fb1cd to 504e342 Compare July 8, 2026 03:48
@h4x3rotab h4x3rotab force-pushed the aws-attestation-platform-spec branch from 504e342 to 420ea41 Compare July 8, 2026 22:32
@kvinwang kvinwang force-pushed the aws-attestation-platform-spec branch from 420ea41 to 41f9b7e Compare July 13, 2026 04:14
@kvinwang kvinwang changed the title [codex] Add AWS NitroTPM attestation platform support Add AWS EC2 NitroTPM attestation and guest OS support Jul 13, 2026
@kvinwang kvinwang force-pushed the aws-attestation-platform-spec branch from 41f9b7e to 41a4aa1 Compare July 13, 2026 04:16
Comment thread dstack/dstack-attest/src/aws_nitro_tpm.rs Fixed
@kvinwang kvinwang force-pushed the aws-attestation-platform-spec branch from fe08404 to 4062083 Compare July 14, 2026 04:37
@kvinwang

Copy link
Copy Markdown
Collaborator

Simplification pass (operator decisions)

Rebased onto master (includes #777 empty TPM key-provider id) and applied:

Decision Implementation
Single event lane (TDX-like) All dstack events → SHA384 PCR14; dropped PCR23 runtime split + emit_launch domain
Config not via UKI cmdline Shared-disk MrConfigV3 measured into PCR8; guest verifies PCR8 vs document
Unified os_image_hash Optional VmConfig.aws_measurement (sha256(sha256sum.txt) + ref PCR4/7/12); legacy PCR-hash still works
Drop complexity Removed GetAppKey v2 recipient encryption, GetQuote attestation_options, aws-pcr14-fault-injection
check_kms Only mrAggregated (no kms.composeHashes); pin early/boot-mr-done style like bare TDX

Commit: 40620832 simplify AWS NitroTPM: single PCR14, PCR8 config, drop v2 complexity

Still follow-up as needed: Yocto UKI recipe stop embedding dstack.mr_config_id in cmdline; production docs/smoke; auth-simple README examples; optional SDK cleanup if any client exposed attestation_options.

@kvinwang

Copy link
Copy Markdown
Collaborator

Follow-up commit ca7909f4

Completed the remaining cleanup called out after the simplify pass:

  • UKI / Yocto tools: document that UKI cmdline must not embed per-app dstack.mr_config_id (shared-disk MrConfigV3 → PCR8)
  • auth-simple: KMS allowlist is mrAggregated only; examples, schema, and tests updated (app composeHashes unchanged)
  • SDKs: removed attestation_options / AWS NitroTPM options from Rust, Python, JS, Go, and sdk/curl/api.md
  • Docs: production verifier runbook + security-model PCR map; security-evaluation banner that historical cmdline/PCR23/recipient/composeHashes sections are superseded

@kvinwang kvinwang force-pushed the aws-attestation-platform-spec branch from ca7909f to c49b7cf Compare July 14, 2026 04:46
Add AWS EC2 NitroTPM as a first-class attestation platform under the
account-admin-untrusted threat model, integrated like SEV-SNP rather than
with AWS-specific auth surface.

Measurement model (aligned with bare TDX):
- Boot: PCR4/7/12 for AMI/UKI identity
- Config: shared-disk MrConfigV3 measured into PCR8 (no UKI cmdline pin)
- Events: all dstack events extend SHA384 PCR14 (RTMR3 analogue; no PCR23
  launch/runtime split)
- os_image_hash: prefer unified sha256(sha256sum.txt) via
  VmConfig.aws_measurement + reference PCR bind; legacy PCR-hash fallback
- KMS GetAppKey: RA-TLS only (api_version=1); no recipient encryption
- GetQuote/Attest: report_data only (no platform attestation_options)
- check_kms / auth-simple: pin early mrAggregated only (boot-mr-done)

Includes Yocto AWS kernel/image support, verifier/KMS paths, SDKs, docs,
and release/smoke helpers. Empty TPM key-provider id (already on master)
keeps early MR precomputable.
@kvinwang kvinwang force-pushed the aws-attestation-platform-spec branch from c49b7cf to d8f4871 Compare July 14, 2026 04:51
kvinwang added 7 commits July 13, 2026 22:22
Local TPM sealing on AwsEc2 now binds only SHA384 PCR4/7/12 (boot/OS
identity). PCR23 is unused after the single PCR14 event-lane model and
is resettable, so it must not gate sealed app keys.
Local NitroTPM key-provider sealing now uses sha384:4,7,8,12,14 —
boot path (4/7/12), shared-disk MrConfigV3 (8), and the dstack event
lane at seal time (14). Seal runs after PCR8 measure and key-provider
events, so reboot unseal matches the same intermediate PCR14 state.
Wire unified os_image_hash + aws_measurement end-to-end:

- dstack-mr aws-measurement-cbor and assemble measurement.aws.cbor
- guest quote embeds sys-config vm_config when aws_measurement binds
  attested PCR4/7/12 (legacy PCR-hash fallback otherwise)
- dstack-cloud: platform=aws, prepare sys-config with aws_measurement,
  full EC2 lifecycle (AMI import, shared snapshot, run/stop/start/logs)
- remove standalone aws-ec2-import/smoke/attach scripts; docs point to
  dstack-cloud. Keep image audit + release-manifest helpers only.
UKI assemble always precomputes NitroTPM PCR4/7/12 into measurement.aws.cbor
and lists it in sha256sum.txt so digest.txt is fixed for the release.
Host nitro-tpm-pcr-compute is preferred; Docker AL2023 is fallback; missing
both fails the build. dstack-cloud prepare only embeds the release material
and refuses packages that lack a checksum-committed measurement.aws.cbor.
AwsOsImageMeasurement now holds sha256(PCR4||PCR7||PCR12) instead of the
three raw SHA384 PCR values, matching the legacy image-hash composition.
hex encoding uses serde_human_bytes (hex_bytes alias). Verifier and guest
quote compare digests; dstack-mr still takes PCR4/7/12 hex and folds them.
Drop panic from cbor_to_vec and the awkward context string parameter.
Callers attach context with .context("…") instead. to_cbor_vec,
from_cbor_slice, measurement_hash, from_measurement, decode_*, and
document verify now use anyhow::Result for a consistent error model.
@kvinwang kvinwang force-pushed the aws-attestation-platform-spec branch from 4fe61c8 to f6ed3c4 Compare July 14, 2026 10:58
kvinwang added 2 commits July 14, 2026 08:16
The pre-key-release check added no security guarantee: key release for
an unauthorized compose is already blocked by KMS via the PCR14 event
lane, and key use under a mismatched config is still rejected by the
full mr_config check in verify_app before any key is used or persisted.
The early check only duplicated the static-field validation for AWS,
diverging from the other platforms.

Remove it and let AWS follow the same single post-key verification path
as TDX and SEV-SNP.
On the AWS EC2 platform the attestation mode must be NitroTPM only.
Exposing TDX or SEV-SNP devices on an AWS instance must not silently
switch the attestation mode away from the AWS platform path.
kvinwang added 20 commits July 14, 2026 09:50
The extra sha384(config_id) before the PCR8 extend added no security
(PCR extend already hashes) and hid the config id version byte behind a
second, non-obvious hash step for verifiers. Extend the raw 48-byte
config id directly so PCR8 = sha384(0^48 || config_id), letting a
verifier parse the claimed config id's version byte and recompute the
register in one step.

Also fix stale docs that still described the shared-disk MrConfigV3 /
UKI-cmdline model: the runbook and security-model checklist now state
the guest-computed MrConfig V2 id, and the dated AMI reproducibility
manifest gets a supersession note instead of rewriting its evidence.
the wrapper was a pure delegation to emit_runtime_event; call sites now
use the original name again, matching main.
Restore pre-existing, non-AWS code that had been refactored or churned but
was not required by the NitroTPM feature:

- nsm-qvl: revert cert-chain refactor (verify.rs) and lib.rs to base, keeping
  only the AWS-required #[serde(alias = "nitrotpm_pcrs")] on pcrs.
- tpm-attest/gcp_ak.rs: revert GCP-only quote refactor to base.
- kms proto: drop never-shipped GetAppKey v2 fields (9/10); revert
  AppKeyResponse construction to the direct form.
- kms/dstack-util: drop unused deps (aes-gcm, prost x2).
- dstack-types/mr_config: drop unused MrConfigV3::to_mr_config_id /
  mr_config_id_from_document scaffolding; keep the tdx_-named API.
- auth-simple: revert behavior-neutral empty-allowlist guard.
- SDK js/python: revert attestation-payload helper extraction; keep the new
  'attestation' response field.
- dstack-auth: drop stray comment.

Kept intentionally: dstack-types fallible CBOR refactor and the verifier
--verify-cert mode (per request). Skipped: tpm2 internal comment/test-rename
churn (interleaved with required AWS additions) and dstack-cloud (genuinely
AWS-refactored, help text updated for --platform).
decode_mr_aws_nitro_tpm_from_pcrs bound the quoted PCR14 against a
boot-mr-done-truncated replay when use_boottime_mr was set, so a full
runtime quote (PCR14 covering the whole event log) always failed with
'PCR14 mismatch'. This made KMS SignCert (the only caller passing
use_boottime_mr=true) unusable on AWS NitroTPM.

Align with the TDX (decode_mr_tdx_from_quote) and GCP (decode_mr_gcp_tpm)
architecture: the event-log-to-register binding always uses the full
replay, and the boot-time snapshot boundary is applied only to the MR
value. aws_nitro_tpm_verify_event_pcr now always binds the full replay
(boottime param dropped); decode replays the truncated snapshot
separately for mr_aggregated. The decode-side binding is kept as
defense-in-depth (KMS reject tests exercise it), matching AWS's existing
extra check rather than dropping it like TDX/GCP.

Rewrite the full-log replay test for the corrected semantics and add a
KMS regression test that a full runtime quote decoded with
use_boottime_mr=true succeeds and yields the boot-time snapshot MR.
…hortcut

PCR8 is not part of dstack's own verification path: the authoritative AWS
NitroTPM app-identity binding is the PCR14 event-log replay. PCR8 carries a
MrConfig V2 config commitment only so a lightweight third-party verifier can
confirm the compose hash and key-provider by recomputing
sha384(0^48 || config_id) instead of replaying the event log.

Several docs described PCR8 as a required verifier check. Reword them (and the
auth-simple README's mr_config_id mention) to present it as the optional
shortcut it is, and note that dstack's verifier and KMS do not check it.
run_cert_oneshot previously emitted app_info.os_image_hash under
is_valid:true without checking that it is bound to the attested boot
measurement, so a relying party reading the output could trust a host
claim. Add verify_cert_os_image_hash, which runs the existing
CvmVerifier::verify_os_image_hash binding and reports the result as
app_info.os_image_hash_verified.

The binding is self-contained (no image download) for AWS NitroTPM,
SEV-SNP, Nitro Enclave, GCP TDX, and TDX lite. TDX legacy would require
an image download that this oneshot cert mode does not perform, so it is
reported as os_image_hash_verified: false; use /verify for that path.

Document the new field in the verifier README and require
os_image_hash_verified == true in the AWS production runbook's jq check.
The FreshnessPolicy / verify_freshness_policy / details.freshness_verified
machinery was added by this PR and has no in-tree consumer: KMS key release is
replay-protected by the live RA-TLS handshake and timestamped signatures, and
the gateway authorizes over live RA-TLS connections. Freshness for the external
/verify path was an opt-in advisory field that nothing else depended on.

Remove the request FreshnessPolicy, the freshness_verified detail field,
verify_freshness_policy and its helpers (aws_nitro_tpm_report,
attestation_timestamp_ms, ensure_expected_bytes, ensure_attestation_max_age,
FRESHNESS_CLOCK_SKEW), and the four freshness tests. External /verify callers
that need liveness embed their own report_data challenge and check it against
the returned boot_info. Update the verifier README, verification.md, the AWS
production runbook, and the dated evaluation doc's supersession note.
reuse policy_tcb_fields when filling VerificationDetails so
details.tcb_status matches boot_info.tcbStatus (notably AWS NitroTPM,
which is normalized to "UpToDate" for the auth gate but surfaced as
null at the top level). also picks up the pending rustfmt import
reorder that broke CI.
the "legacy paths without aws_measurement use the value directly"
path no longer exists; aws_measurement is mandatory on AWS.
the KMS key-release pipeline gates non-SNP quotes through
CvmVerifier::verify_os_image_hash; pin down that the AWS path rejects
a vm_config without aws_measurement, accepts a unified vm_config bound
to the quoted PCRs, and rejects a boot_pcr_digest mismatch.
- audit-aws-ec2-image-hardening.sh: a supplied --rootfs-squashfs with
  no unsquashfs available is now a FAIL instead of WARN + exit 0
- generate-aws-ec2-release-manifest.sh: capture the audit exit status
  explicitly so a failing audit prints its output instead of silently
  aborting the manifest run under set -e
hop limit 2 lets processes behind the docker bridge fetch IMDSv2
tokens, exposing instance-role credentials to app containers when
iam_instance_profile is set. dstack itself never talks to IMDS, so
default metadata_hop_limit to 1 and leave 2 as an explicit opt-in for
deployments whose containers legitimately need the instance role.
generate_tpm_app_keys now fails closed on platforms whose TPM is not
part of the platform trust model; document that in the app-compose
key_provider field description.
drop the unpinned docker amazonlinux:2023 + dnf fallback from
assemble.sh: the computed PCRs feed measurement.aws.cbor ->
sha256sum.txt -> os_image_hash, i.e. the measurement the KMS and
verifiers enforce, so only a version-pinned operator-installed
nitro-tpm-pcr-compute may produce it.
release images and their evidence (sha256sum.txt, digest.txt,
measurement.aws.cbor, aws-pcrs.json) are produced by the unified
os/build.sh flow now, so drop generate-aws-ec2-release-manifest.sh and
the reproducibility-manifest doc written around it. rework the
production verifier runbook around the build.sh release package
(reproducible rebuild + pinned host tools for reference measurements)
and note the removal in the dated security evaluation. the hardening
audit script stays.
Trim the superseded security evaluation to the live threat model and property table, document the actual UKI archive and PCR sidecar layout, and remove stale recipient/fallback claims. Also simplify the certificate output and build fragments referenced by the updated runbook.
Drop the write-only AppInfo.attestation field and its backend, simulator, SDK, and documentation plumbing. GetQuoteResponse.attestation remains the platform-agnostic evidence field.
Parameterize GCP/AWS vm-config generation, reuse one shared-disk builder with required-file checks, and remove dead config serialization and save helpers. Correct all project-file diagnostics to app.json.
Restore the established string-error measurement API while using or_panic for invariant-only CBOR encoding. Consolidate TCB and key-release policy helpers, move AWS image-binding tests to the verifier, and remove redundant tests, dead accessors, and unrelated TPM/NSM diff noise.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds end-to-end AWS EC2 NitroTPM support across the monorepo (guest OS build/measurement, attestation generation, verifier/KMS policy normalization, SDK surface updates, and operational documentation) under the account-admin-untrusted threat model.

Changes:

  • Introduces AWS NitroTPM attestation collection/verification and AWS-specific OS-image binding (aws_measurement, boot PCR digest) across attest/verifier/KMS.
  • Extends Yocto guest OS/kernel configuration and image assembly to produce AWS measurement artifacts and hardening audit tooling.
  • Updates SDKs and docs to expose/use a platform-agnostic attestation field and to document AWS verification/operations.

Reviewed changes

Copilot reviewed 60 out of 62 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
sdk/python/src/dstack_sdk/dstack_client.py Adds attestation field to quote response model.
sdk/js/src/index.ts Adds attestation field to JS SDK quote response type.
sdk/go/dstack/client.go Adds attestation field and decoding helper to Go SDK.
sdk/curl/api.md Documents attestation in /GetQuote and cloud fields in /Info.
os/yocto/tools/README.md Clarifies Yocto host tools scope and AWS tooling location.
os/yocto/tools/aws/audit-aws-ec2-image-hardening.sh Adds AWS EC2 image hardening audit script.
os/yocto/layers/meta-dstack/recipes-kernel/linux/linux-yocto%.bbappend Adds AWS kernel feature config sources and feature inclusion.
os/yocto/layers/meta-dstack/recipes-kernel/linux/files/dstack-aws.scc Introduces Yocto kernel feature definition for AWS.
os/yocto/layers/meta-dstack/recipes-kernel/linux/files/dstack-aws.cfg Adds kernel config needed for EC2 Nitro boot devices/console.
os/yocto/layers/meta-dstack/recipes-core/images/dstack-uki.bb Documents AWS app/config binding expectations for UKI cmdline.
os/yocto/layers/meta-dstack/recipes-core/dstack-guest/dstack-guest.bb Adds tpm2-tss build dependency for guest recipe.
os/image/README.md Documents AWS measurement artifact and tool requirements.
os/image/assemble.sh Generates and includes measurement.aws.cbor in image digest/tar flows.
dstack/vmm/src/app.rs Initializes aws_measurement field in VM config assembly path.
dstack/verifier/src/verification.rs Adds AWS NitroTPM OS-image verification and policy boot info construction.
dstack/verifier/src/types.rs Adds PolicyBootInfo and includes it in verification details.
dstack/verifier/src/main.rs Adds --verify-cert oneshot mode for RA-TLS cert verification.
dstack/verifier/src/lib.rs Re-exports PolicyBootInfo and policy_tcb_fields.
dstack/verifier/README.md Documents AWS mode, policy boot info, and --verify-cert workflow.
dstack/verifier/Cargo.toml Gates binary build on binary feature (default-enabled).
dstack/tpm2/src/types.rs Adds RSA EK template and configurable name algorithm for templates/NV.
dstack/tpm2/src/device.rs Adds raw session command helpers and flexible auth-area builders.
dstack/tpm2/src/commands.rs Adds raw execution, NV name read, auth NV read/write, capabilities helpers.
dstack/tpm-types/src/lib.rs Adds SHA384 PCR selection helper.
dstack/tpm-qvl/src/lib.rs Rejects AWS EC2 platform for GCP AK quote root retrieval.
dstack/tpm-attest/tests/tpm_quote_sample.README.md Documents TPM quote parsing fixture semantics/provenance.
dstack/tpm-attest/src/lib.rs Adds AWS NitroTPM PCR policy and single-PCR read helper.
dstack/ra-tls/src/attestation.rs Adds RA-TLS cert verification APIs returning verified certificate evidence.
dstack/ra-tls/Cargo.toml Adds tokio dev-dependency for async tests.
dstack/nsm-qvl/tests/nitro_attestation.README.md Documents NSM attestation fixture capture/verification model.
dstack/nsm-qvl/src/lib.rs Adds alias for NitroTPM PCR field name in attestation document.
dstack/kms/src/onboard_service.rs Adds platform evidence mapping for AWS NitroTPM.
dstack/kms/src/main_service/upgrade_authority.rs Reuses verifier’s canonical policy boot info and TCB normalization.
dstack/kms/src/main_service.rs Adds AWS NitroTPM key-release gate and extensive AWS-specific tests.
dstack/kms/src/config.rs Adds aws_nitro_tpm_key_release config flag.
dstack/kms/kms.toml Documents and defaults AWS NitroTPM key-release gate to disabled.
dstack/kms/Cargo.toml Adds cc-eventlog dev-dependency for AWS-related tests.
dstack/kms/auth-simple/README.md Updates allowlisting guidance and adds AWS NitroTPM example.
dstack/kms/auth-simple/index.test.ts Normalizes attestationMode casing and updates tests/messages.
dstack/kms/auth-simple/auth-config.example.json Updates example KMS MR label to “early/boot-mr-done”.
dstack/dstack-util/src/system_setup/config_id_verifier.rs Skips AWS PCR8 config-id cross-check (guest-measured).
dstack/dstack-util/src/system_setup.rs Adds AWS instance binding and AWS-specific PCR measurements/policy.
dstack/dstack-util/src/main.rs Allows emitting raw AWS NitroTPM attestation doc in quote command path.
dstack/dstack-util/src/host_api.rs Skips host notify on AWS EC2 platform.
dstack/dstack-util/src/crypto.rs Rejects all-zero X25519 shared secret during DH decrypt.
dstack/dstack-util/Cargo.toml Adds tpm2 dependency to support AWS NitroTPM operations.
dstack/dstack-types/src/lib.rs Adds AWS platform detection and AWS OS image measurement document types.
dstack/dstack-mr/src/main.rs Adds AWS measurement encoding/inspection support.
dstack/dstack-attest/src/v1.rs Adds AWS NitroTPM platform evidence variant.
dstack/dstack-attest/src/lib.rs Extends measured-event emission to TPM PCRs (sha256/sha384) and adds PCR8 measure helper.
dstack/dstack-attest/src/aws_nitro_tpm.rs Implements NitroTPM vendor-command flow to obtain attestation documents.
dstack/dstack-attest/src/attestation.rs Adds AWS NitroTPM mode, quote/report verification, and MR derivation from PCRs.
dstack/dstack-attest/Cargo.toml Adds AWS NitroTPM dependencies under quote feature.
dstack/Cargo.lock Locks new dependencies for AWS NitroTPM functionality.
docs/verification.md Adds platform-specific verification guidance including AWS NitroTPM.
docs/tutorials/attestation-verification.md Clarifies TDX-specific flow and documents platform differences incl. AWS.
docs/security/security-model.md Updates security model and measurement semantics for AWS NitroTPM mode.
docs/security/cvm-boundaries.md Documents TPM key provider constraints and AWS measurement lane semantics.
docs/quickstart.md Extends quickstart to include AWS NitroTPM option and config example.
docs/aws-ec2-production-verifier-runbook.md New runbook for AWS EC2 NitroTPM production verification and ops.
docs/aws-attested-instance-security-evaluation.md New AWS NitroTPM threat-model security evaluation document.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread dstack/verifier/src/main.rs
@kvinwang

Copy link
Copy Markdown
Collaborator

GCP TDX boot regression — root-caused and fixed in 46fc3cf

Real-machine testing of the unified 0.6.0 image found it reboot-looping on GCP TDX (c3-standard-4) before reaching /init, while the same image boots fine on AWS EC2 and local QEMU TDX.

Root cause: CONFIG_INIT_ON_FREE_DEFAULT_ON=y, added to dstack.cfg as part of the kernel hardening baseline in 77df428.

Bisect evidence — same failing image, only the UKI .cmdline section patched with runtime overrides, booted on GCP TDX:

init_on_alloc init_on_free result
on on reboot loop (5/5 boots)
off on reboot loop (4/4 boots)
on off boots to initramfs
off off boots to initramfs

init_on_free=1 is the sole trigger; CONFIG_INIT_ON_ALLOC_DEFAULT_ON=y is kept.

Failure point: serial console shows the 16 GiB zero-on-free pass itself completes; the guest then dies silently (no panic, straight back to UEFI BdsDxe) inside mem_encrypt_init() — the 1 GiB SWIOTLB private→shared conversion. An accept_memory=eager variant also fails, ruling out an unaccepted/lazy-accept interaction. The same kernel + cmdline + 16 GiB + 1 GiB SWIOTLB conversion succeeds on a local QEMU/KVM TDX host, so this is a GCP host-side interaction with a fully dirtied guest memory image, not a generic kernel bug.

The fix drops the option from dstack.cfg (with a comment documenting the incompatibility) and removes the matching recommend_config line from the AWS hardening audit script so it doesn't nudge anyone into re-adding it.

@kvinwang kvinwang marked this pull request as ready for review July 16, 2026 05:50
@kvinwang

Copy link
Copy Markdown
Collaborator

128db22 restores the empty-/dev/vdb auto-initialization path (create_data_partition + step 3.3) that 77df428 removed together with the multi-disk blank-device scan. Only the AWS-era "pick the first blank disk" heuristic stays removed; the long-standing behavior of initializing a blank /dev/vdb with a dstack-data GPT partition is back to the exact code on master. Without it, a fresh dstack-vmm VM (blank qcow2 data disk) fail-closes with "Empty /dev/vdb has no 'dstack-data' partition" and reboot-loops — caught while booting a local TDX instance end-to-end (nginx HTTP 200 after manually pre-labeling the disk).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants