performance/flaky-benchmarks-monitoring

[igoragoli]

No description provided.

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass
Load	🟢 pass
DaCapo	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Variant	Metric	Candidate	master	Δ (95% CI of mean)
dacapo:biojava:tracing	datadog	execution_time	14.23 s	14.43 s	[-2.4%; -0.3%] (maybe better)
dacapo:tomcat:tracing	datadog	execution_time	1.67 ms	1.69 ms	[-2.6%; +0.1%] (no difference)
startup:insecure-bank:iast:Agent	datadog	execution_time	14.00 s	13.95 s	[-0.4%; +1.2%] (no difference)
startup:insecure-bank:tracing:Agent	datadog	execution_time	12.98 s	13.02 s	[-1.4%; +0.7%] (no difference)
webserver--insecure-bank--tracing--high_load	load	agg_http_req_duration_p50	805.98 µs	777.88 µs	[+1.0%; +6.2%] (maybe worse)
startup:petclinic:appsec:Agent	datadog	execution_time	17.10 s	16.22 s	[+1.2%; +9.6%] (significantly worse)
startup:petclinic:iast:Agent	datadog	execution_time	17.00 s	17.07 s	[-1.2%; +0.4%] (no difference)
startup:petclinic:profiling:Agent	datadog	execution_time	16.59 s	16.76 s	[-2.2%; +0.1%] (no difference)
startup:petclinic:sca:Agent	datadog	execution_time	16.89 s	16.92 s	[-1.3%; +0.9%] (no difference)
startup:petclinic:tracing:Agent	datadog	execution_time	16.08 s	16.15 s	[-1.5%; +0.5%] (no difference)
webserver--spring-petclinic--sca--high_load	load	agg_http_req_duration_p50	25.32 ms	24.59 ms	[-3.2%; +9.1%] (unstable)
webserver--spring-petclinic--tracing--high_load	load	agg_http_req_duration_p50	25.52 ms	25.70 ms	[-7.0%; +5.6%] (unstable)

Commit: e9bad03e · CI Pipeline · Benchmarking Platform UI

[pr-commenter]

Kafka / producer-benchmark

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	performance/flaky-benchmarks-monitoring
git_commit_date	1790025511	1790033537
git_commit_sha	2425f43	e9bad03

See matching parameters

	Baseline	Candidate
ci_job_date	1790035043	1790035043
ci_job_id	2065548828	2065548828
ci_pipeline_id	139046870	139046870
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
jdkVersion	11.0.31	11.0.31
jmhVersion	1.36	1.36
jvm	/usr/lib/jvm/java-11-openjdk-amd64/bin/java	/usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs	-Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=15002 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=15002 -Dhttp.nonProxyHosts=localhost	*.localhost
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-xy7w8jr4 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-xy7w8jr4 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
vmName	OpenJDK 64-Bit Server VM	OpenJDK 64-Bit Server VM
vmVersion	11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu	11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results

scenario	Δ mean throughput
scenario:not-instrumented/KafkaProduceBenchmark.benchProduce	same
scenario:only-tracing-dsm-disabled-benchmarks/KafkaProduceBenchmark.benchProduce	same
scenario:only-tracing-dsm-enabled-benchmarks/KafkaProduceBenchmark.benchProduce	same

[pr-commenter]

Kafka / consumer-benchmark

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	performance/flaky-benchmarks-monitoring
git_commit_date	1790025511	1790033537
git_commit_sha	2425f43	e9bad03

See matching parameters

	Baseline	Candidate
ci_job_date	1790035086	1790035086
ci_job_id	2065548829	2065548829
ci_pipeline_id	139046870	139046870
cpu_model	Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
jdkVersion	11.0.31	11.0.31
jmhVersion	1.36	1.36
jvm	/usr/lib/jvm/java-11-openjdk-amd64/bin/java	/usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs	-Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=15002 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=15002 -Dhttp.nonProxyHosts=localhost	*.localhost
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-y4adbshi 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-y4adbshi 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
vmName	OpenJDK 64-Bit Server VM	OpenJDK 64-Bit Server VM
vmVersion	11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu	11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results

scenario	Δ mean throughput
scenario:not-instrumented/KafkaConsumerBenchmark.benchConsume	same
scenario:only-tracing-dsm-disabled-benchmarks/KafkaConsumerBenchmark.benchConsume	same
scenario:only-tracing-dsm-enabled-benchmarks/KafkaConsumerBenchmark.benchConsume	same

[pr-commenter]

Debugger benchmarks

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
ci_job_date	1790034535	1790034892
end_time	2026-09-21T23:50:22	2026-09-21T23:56:19
git_branch	master	performance/flaky-benchmarks-monitoring
git_commit_sha	2425f43	e9bad03
start_time	2026-09-21T23:48:56	2026-09-21T23:54:53

See matching parameters

	Baseline	Candidate
ci_job_id	2065548830	2065548830
ci_pipeline_id	139046870	139046870
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
git_commit_date	1790033537	1790033537

Summary

Found 5 performance improvements and 0 performance regressions! Performance is the same for 4 metrics, 6 unstable metrics.

scenario	Δ mean agg_http_req_duration_min	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p75	Δ mean agg_http_req_duration_p99	Δ mean throughput
scenario:loop	better [-1.859ms; -1.519ms] or [-19.125%; -15.623%]	better [-1.875ms; -1.532ms] or [-19.150%; -15.648%]	better [-1.871ms; -1.530ms] or [-19.046%; -15.574%]	better [-1.910ms; -1.566ms] or [-19.186%; -15.733%]	better [+15.783op/s; +21.513op/s] or [+15.388%; +20.975%]

See unchanged results

scenario	Δ mean agg_http_req_duration_min	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p75	Δ mean agg_http_req_duration_p99	Δ mean throughput
scenario:noprobe	unstable [-29.606µs; +35.549µs] or [-9.383%; +11.267%]	unstable [-42.805µs; +49.464µs] or [-12.221%; +14.122%]	unstable [-52.087µs; +62.066µs] or [-14.268%; +17.002%]	unstable [-142.321µs; +109.465µs] or [-11.980%; +9.214%]	same
scenario:basic	same	same	same	unstable [-141.584µs; +92.756µs] or [-13.151%; +8.616%]	unstable [-130.578op/s; +130.578op/s] or [-5.615%; +5.615%]

Request duration reports for reports

gantt
    title reports - request duration [CI 0.99] : candidate=None, baseline=None
    dateFormat X
    axisFormat %s
section baseline
noprobe (350.259 µs) : 313, 388
.   : milestone, 350,
basic (322.85 µs) : 320, 325
.   : milestone, 323,
loop (9.793 ms) : 9577, 10010
.   : milestone, 9793,
section candidate
noprobe (353.589 µs) : 304, 403
.   : milestone, 354,
basic (324.895 µs) : 319, 330
.   : milestone, 325,
loop (8.089 ms) : 8027, 8152
.   : milestone, 8089,

Loading

• baseline results

Scenario	Request median duration [CI 0.99]
noprobe	350.259 µs [312.84 µs, 387.679 µs]
basic	322.85 µs [320.265 µs, 325.434 µs]
loop	9.793 ms [9.577 ms, 10.01 ms]

• candidate results

Scenario	Request median duration [CI 0.99]
noprobe	353.589 µs [304.147 µs, 403.03 µs]
basic	324.895 µs [319.341 µs, 330.448 µs]
loop	8.089 ms [8.027 ms, 8.152 ms]