Fix stale _dd.base_service on GraalVM/Mandrel native-image builds - #12571
Yeison2020 wants to merge 1 commit into
Conversation
…MS-20492) TagsPostProcessorFactory caches its tag-processor chain (including InternalTagsAdder, which stamps _dd.base_service) in a nested holder class that GraalVM native-image initializes at build time by default. That first touch can happen during the native-image build itself, before DD_SERVICE exists, freezing Config's build-time fallback service name (the native-image builder's own process identity) into the resulting binary. Config/Platform are already marked :rerun for this exact reason, but TagsPostProcessorFactory was missed, so its cached chain never gets recomputed at real runtime. Call TagsPostProcessorFactory.reset() once from TracerInstaller.installGlobalTracer(CoreTracer), guarded by Platform.isNativeImage(). That method only actually runs at real application startup (the native-image build-time path in Agent.start() never reaches tracer installation), as plain, ordering-safe Java method-call execution, by which point Config has already re-resolved against the real runtime environment. Tests: - dd-trace-core: new TagsPostProcessorFactoryTest locks in that reset() rebuilds the cached eager/lazy processor chains, so the fix can't silently regress into a no-op. - dd-smoke-tests/quarkus-native: new assertion in QuarkusNativeSmokeTest verifies every span in a real native-image build carries the correct service and no _dd.base_service tag, since this class of bug can only be observed against an actual GraalVM/Mandrel native-image build, not a plain JVM unit test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
More details
The reset runs after native-image runtime configuration is available. It rebuilds the cached processor chains at the correct time.
🤖 Datadog Autotest · Commit 4813143 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
sarahchen6
left a comment
There was a problem hiding this comment.
LGTM! worth another eye from IDM though
| // to a build-time fallback service name (typically the native-image builder's own | ||
| // process identity), not the real DD_SERVICE the app is run with. Force the cache to | ||
| // recompute here, now that Config reflects the actual runtime environment. | ||
| TagsPostProcessorFactory.reset(); |
There was a problem hiding this comment.
seems like we could be more precise than recomputing the whole thing, if we only need to refresh a single tag adder ? But given this only runs once, maybe it's fine.
What Does This Do
On a GraalVM/Mandrel native-image build, every span was tagged with a spurious
_dd.base_servicepointing at the native-image builder tool's own process identity (org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner), even whenDD_SERVICEwas set correctly and every span's actualservicetag resolved correctly. On live traffic this surfaces as the correct application name only appearing as a Service Override, never as the Base Service, for the affected spans.TagsPostProcessorFactorycaches its tag-processor chain - includingInternalTagsAdder, which stamps_dd.base_service- in a nested holder class (Lazy) that computes itself fromConfig.get().getServiceName()the first time it's touched. Under GraalVM native-image's default class-initialization policy, that first touch happens during the native-image build itself, beforeDD_SERVICEexists at all. At that point the running process genuinely is the native-image builder tool, andConfig's fallback resolves to that process's own identity, which gets frozen into the resulting binary and never recomputed.Config/Platformare already marked:rerunin this tracer's native-image class-initialization directives specifically to avoid this class of staleness, butTagsPostProcessorFactorywas missed.This adds a call to
TagsPostProcessorFactory.reset()insideTracerInstaller.installGlobalTracer(CoreTracer), guarded byPlatform.isNativeImage(). That call site only executes at real application startup (the native-image build-time path inAgent.start()never reaches tracer installation, it only applies bytecode instrumentation and returns early), so it runs as plain, ordering-safe Java method-call execution rather than relying on GraalVM's class-initialization ordering — by the time it runs,Confighas already re-resolved against the real runtime environment.Motivation
APMS-20593
Additional Notes
TagsPostProcessorFactory/itsLazyholder as:rerunvia the existing-H:ClassInitializationdirective instead. Rejected: GraalVM'srerunmechanism re-executes each marked class's static initializer independently at startup with no guaranteed ordering relative to otherrerunclasses, soLazycould still run beforeConfighas re-resolved, reproducing the same staleness.installGlobalTracercall site avoids this ordering ambiguity entirely._dd.base_service;after the fix, no_dd.base_servicetag appears, matching a plain-JVM build of the same app/config.Testing -
dd-trace-core: newTagsPostProcessorFactoryTest— loally rebuilds the cached eager/lazy processor chains (object identity), so the fix can't silently regress into a no-op. -dd-smoke-tests/quarkus-native: new assertion in Quarkes real trace payloads from an actual native-image build and asserts every span has the correctserviceand no_dd.base_servicetag. This is the only test type that can actually exercise the GraalVM native-image class-initialization timing this bug depends:dd-java-agent:agent-installer:testand:dd-trace-core:testfor the touched classes - all pass../gradlew spotlessApplyrun on all touched modules.