Skip to content

Fix stale _dd.base_service on GraalVM/Mandrel native-image builds - #12571

Open
Yeison2020 wants to merge 1 commit into
masterfrom
fix/apms-20492-native-image-base-service
Open

Yeison2020 wants to merge 1 commit into
masterfrom
fix/apms-20492-native-image-base-service

Conversation

@Yeison2020

@Yeison2020 Yeison2020 commented Sep 18, 2026

Copy link
Copy Markdown

What Does This Do

On a GraalVM/Mandrel native-image build, every span was tagged with a spurious _dd.base_service pointing at the native-image builder tool's own process identity (org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner), even when DD_SERVICE was set correctly and every span's actual service tag resolved correctly. On live traffic this surfaces as the correct application name only appearing as a Service Override, never as the Base Service, for the affected spans.

TagsPostProcessorFactory caches its tag-processor chain - including InternalTagsAdder, which stamps _dd.base_service - in a nested holder class (Lazy) that computes itself from Config.get().getServiceName() the first time it's touched. Under GraalVM native-image's default class-initialization policy, that first touch happens during the native-image build itself, before DD_SERVICE exists at all. At that point the running process genuinely is the native-image builder tool, and Config's fallback resolves to that process's own identity, which gets frozen into the resulting binary and never recomputed.

Config/Platform are already marked :rerun in this tracer's native-image class-initialization directives specifically to avoid this class of staleness, but TagsPostProcessorFactory was missed.

This adds a call to TagsPostProcessorFactory.reset() inside TracerInstaller.installGlobalTracer(CoreTracer), guarded by Platform.isNativeImage(). That call site only executes at real application startup (the native-image build-time path in Agent.start() never reaches tracer installation, it only applies bytecode instrumentation and returns early), so it runs as plain, ordering-safe Java method-call execution rather than relying on GraalVM's class-initialization ordering — by the time it runs, Config has already re-resolved against the real runtime environment.

Motivation

APMS-20593

Additional Notes

  • Considered marking TagsPostProcessorFactory/its Lazy holder as :rerun via the existing -H:ClassInitialization directive instead. Rejected: GraalVM's rerun mechanism re-executes each marked class's static initializer independently at startup with no guaranteed ordering relative to other rerun classes, so Lazy could still run before Config has re-resolved, reproducing the same staleness.
  • The installGlobalTracer call site avoids this ordering ambiguity entirely.
  • Verified locally against a Quarkus/Mandrel native-image ix, every span carried the spurious _dd.base_service;after the fix, no _dd.base_service tag appears, matching a plain-JVM build of the same app/config.

Testing - dd-trace-core: new TagsPostProcessorFactoryTest — loally rebuilds the cached eager/lazy processor chains (object identity), so the fix can't silently regress into a no-op. - dd-smoke-tests/quarkus-native: new assertion in Quarkes real trace payloads from an actual native-image build and asserts every span has the correct service and no _dd.base_service tag. This is the only test type that can actually exercise the GraalVM native-image class-initialization timing this bug depends

  • Ran :dd-java-agent:agent-installer:test and :dd-trace-core:test for the touched classes - all pass.
  • ./gradlew spotlessApply run on all touched modules.

…MS-20492)

TagsPostProcessorFactory caches its tag-processor chain (including
InternalTagsAdder, which stamps _dd.base_service) in a nested holder
class that GraalVM native-image initializes at build time by default.
That first touch can happen during the native-image build itself,
before DD_SERVICE exists, freezing Config's build-time fallback
service name (the native-image builder's own process identity) into
the resulting binary. Config/Platform are already marked :rerun for
this exact reason, but TagsPostProcessorFactory was missed, so its
cached chain never gets recomputed at real runtime.

Call TagsPostProcessorFactory.reset() once from
TracerInstaller.installGlobalTracer(CoreTracer), guarded by
Platform.isNativeImage(). That method only actually runs at real
application startup (the native-image build-time path in Agent.start()
never reaches tracer installation), as plain, ordering-safe Java
method-call execution, by which point Config has already re-resolved
against the real runtime environment.

Tests:
- dd-trace-core: new TagsPostProcessorFactoryTest locks in that
  reset() rebuilds the cached eager/lazy processor chains, so the fix
  can't silently regress into a no-op.
- dd-smoke-tests/quarkus-native: new assertion in
  QuarkusNativeSmokeTest verifies every span in a real native-image
  build carries the correct service and no _dd.base_service tag, since
  this class of bug can only be observed against an actual
  GraalVM/Mandrel native-image build, not a plain JVM unit test.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@Yeison2020
Yeison2020 requested review from a team as code owners September 18, 2026 18:03
@Yeison2020
Yeison2020 requested review from sarahchen6 and vandonr and removed request for a team September 18, 2026 18:03
@dd-octo-sts dd-octo-sts Bot added the tag: ai generated Largely based on code generated by an AI or LLM label Sep 18, 2026
@dd-octo-sts

dd-octo-sts Bot commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

  • Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

@datadog-prod-us1-6

This comment has been minimized.

@Yeison2020 Yeison2020 added type: bug fix Bug fix comp: native-image GraalVM native-image labels Sep 18, 2026

@datadog-prod-us1-6 datadog-prod-us1-6 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: PASS

More details

The reset runs after native-image runtime configuration is available. It rebuilds the cached processor chains at the correct time.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Datadog Autotest · Commit 4813143 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

@dd-octo-sts

dd-octo-sts Bot commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 13.97 s 13.93 s [-0.5%; +1.2%] (no difference)
startup:insecure-bank:tracing:Agent 12.93 s 13.06 s [-1.9%; -0.1%] (maybe better)
startup:petclinic:appsec:Agent 17.04 s 16.92 s [-0.3%; +1.7%] (no difference)
startup:petclinic:iast:Agent 16.95 s 16.47 s [-1.4%; +7.2%] (no difference)
startup:petclinic:profiling:Agent 16.70 s 16.30 s [-1.9%; +6.8%] (no difference)
startup:petclinic:sca:Agent 16.96 s 16.87 s [-0.6%; +1.8%] (no difference)
startup:petclinic:tracing:Agent 16.19 s 16.21 s [-1.0%; +0.8%] (no difference)

Commit: 4813143d · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@sarahchen6 sarahchen6 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! worth another eye from IDM though

// to a build-time fallback service name (typically the native-image builder's own
// process identity), not the real DD_SERVICE the app is run with. Force the cache to
// recompute here, now that Config reflects the actual runtime environment.
TagsPostProcessorFactory.reset();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems like we could be more precise than recomputing the whole thing, if we only need to refresh a single tag adder ? But given this only runs once, maybe it's fine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: native-image GraalVM native-image tag: ai generated Largely based on code generated by an AI or LLM type: bug fix Bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants