Preserve Spark application failures after Spark stops on YARN

[aboitreaud]

What Does This Do

Wait for ApplicationMaster.finish() to finalize Spark application spans in YARN cluster mode, allowing the existing instrumentation to capture the driver's exit code and failure diagnostics after SparkContext.stop().

Motivation

Fixes a Spark application status gap between what the tracer reports (OK) and the expected ERROR.
A Python driver can create a Spark session, fail before submitting any jobs, and stop the session during cleanup before exiting nonzero. The application-end event previously finalized and flushed an "ok" span; the later YARN failure was ignored because the span had already ended.

The change applies only when spark.master=yarn and spark.submit.deployMode=cluster. OpenLineage notification and the shutdown fallback are preserved.

Contributor Checklist

• Imperative title and required labels.
• Formatting and targeted tests passed.
• No source files added, moved, or deleted; no CODEOWNERS update needed.
• No new configuration flags.

[dd-octo-sts]

🟡 Java Benchmark SLOs — Performance SLO warning (near threshold)

Suite	Status
Startup	🟡 warning

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.79 s	14.72 s	[-0.3%; +1.3%] (no difference)
startup:insecure-bank:tracing:Agent	13.54 s	13.78 s	[-2.5%; -1.0%] (significantly better)
startup:petclinic:appsec:Agent	17.54 s	17.40 s	[-0.1%; +1.8%] (no difference)
startup:petclinic:iast:Agent	17.45 s	17.61 s	[-1.9%; +0.1%] (no difference)
startup:petclinic:profiling:Agent	17.43 s	17.42 s	[-1.2%; +1.3%] (no difference)
startup:petclinic:sca:Agent	17.56 s	16.78 s	[+0.4%; +9.0%] (maybe worse)
startup:petclinic:tracing:Agent	16.50 s	16.66 s	[-1.9%; -0.1%] (maybe better)

Commit: a87b8177 · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

[datadog-prod-us1-4]

More details

The change delays the Spark application span only in YARN cluster mode until ApplicationMaster.finish() supplies the exit status. Other modes keep application-end completion.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit a87b817 · @DataDog review to ask questions}