Fix Exception Replay in Lambda by nhulston · Pull Request #8849 · DataDog/dd-trace-java

nhulston · 2025-05-19T17:26:48Z

What Does This Do

In Lambda, the inner stack trace is truncated -- stack frames from fileName='AWSLambda.java' are removed. Therefore, in the original code, we would end up with a negative currentIdx, resulting in an error occurring in sanityCheckSnapshotAssignment.

This change adds a simple fallback:

      int currentIdx = innerTrace.length - snapshot.getStack().size();

      if (currentIdx < 0) {
        // This means the innerTrace was truncated by the underlying environment.
        // This is known to happen in AWS Lambda, but may also happen elsewhere.
        currentIdx = i;
      }

From manual testing in Lambda, this correctly gets the index in Lambda for all snapshots.

Motivation

https://datadoghq.atlassian.net/browse/SVLS-6855

Additional Notes

Exception Replay requires the exception to occur twice in Java: the first occurrence transforms the class, and the second occurrence captures the local variables
In Java, currently method parameters are captured by Exception Replay but other local variables are not captured yet

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, move, or deletion
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

pr-commenter · 2025-05-19T17:38:46Z

Debugger benchmarks

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
ci_job_date	1747845985	1747846355
end_time	2025-05-21T16:47:46	2025-05-21T16:53:56
git_branch	master	nicholas.hulston/fix-exception-replay-in-lambda
git_commit_sha	`58089f3`	`1f12c25`
start_time	2025-05-21T16:46:26	2025-05-21T16:52:36

See matching parameters

	Baseline	Candidate
ci_job_id	948550027	948550027
ci_pipeline_id	65761122	65761122
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
git_commit_date	1747845400	1747845400

Summary

Found 0 performance improvements and 5 performance regressions! Performance is the same for 4 metrics, 6 unstable metrics.

scenario	Δ mean agg_http_req_duration_min	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p75	Δ mean agg_http_req_duration_p99	Δ mean throughput
scenario:loop	worse [+579.873µs; +614.558µs] or [+5.767%; +6.112%]	worse [+594.154µs; +651.668µs] or [+5.824%; +6.387%]	worse [+553.132µs; +651.974µs] or [+5.373%; +6.333%]	worse [+500.061µs; +843.173µs] or [+4.727%; +7.971%]	worse [-6.440op/s; -4.149op/s] or [-6.665%; -4.294%]

See unchanged results

scenario	Δ mean agg_http_req_duration_min	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p75	Δ mean agg_http_req_duration_p99	Δ mean throughput
scenario:noprobe	unstable [-30.568µs; +43.765µs] or [-10.925%; +15.642%]	unstable [-44.257µs; +58.112µs] or [-13.640%; +17.910%]	unstable [-54.772µs; +68.325µs] or [-16.113%; +20.100%]	unstable [-140.565µs; +409.950µs] or [-14.919%; +43.509%]	same
scenario:basic	same	same	same	unstable [-38.673µs; +141.429µs] or [-4.807%; +17.581%]	unstable [-231.458op/s; +115.314op/s] or [-9.490%; +4.728%]

Request duration reports for reports

gantt
    title reports - request duration [CI 0.99] : candidate=None, baseline=None
    dateFormat X
    axisFormat %s
section baseline
noprobe (324.475 µs) : 284, 365
.   : milestone, 324,
basic (315.173 µs) : 305, 325
.   : milestone, 315,
loop (10.203 ms) : 10172, 10233
.   : milestone, 10203,
section candidate
noprobe (331.402 µs) : 276, 387
.   : milestone, 331,
basic (316.213 µs) : 306, 326
.   : milestone, 316,
loop (10.826 ms) : 10804, 10848
.   : milestone, 10826,

baseline results

Scenario	Request median duration [CI 0.99]
noprobe	324.475 µs [284.051 µs, 364.899 µs]
basic	315.173 µs [305.194 µs, 325.151 µs]
loop	10.203 ms [10.172 ms, 10.233 ms]

candidate results

Scenario	Request median duration [CI 0.99]
noprobe	331.402 µs [275.595 µs, 387.21 µs]
basic	316.213 µs [306.048 µs, 326.377 µs]
loop	10.826 ms [10.804 ms, 10.848 ms]

pr-commenter · 2025-05-19T17:41:07Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nicholas.hulston/fix-exception-replay-in-lambda
git_commit_date	1747843428	1747845400
git_commit_sha	`58089f3`	`1f12c25`
release_version	1.50.0-SNAPSHOT~58089f324f	1.50.0-SNAPSHOT~1f12c25da2

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1747847720	1747847720
ci_job_id	948550021	948550021
ci_pipeline_id	65761122	65761122
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-pw3vcngw-project-304-concurrent-2-mr4c9knq 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-pw3vcngw-project-304-concurrent-2-mr4c9knq 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 1 performance regressions! Performance is the same for 56 metrics, 14 unstable metrics.

scenario	Δ mean execution_time	candidate mean execution_time	baseline mean execution_time
scenario:startup:insecure-bank:iast:Remote Config	worse [+14.368µs; +59.830µs] or [+2.487%; +10.356%]	614.835µs	577.736µs

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.50.0-SNAPSHOT~1f12c25da2, baseline=1.50.0-SNAPSHOT~58089f324f

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.026 s) : 0, 1026092
Total [baseline] (10.455 s) : 0, 10455475
Agent [candidate] (1.022 s) : 0, 1021593
Total [candidate] (10.455 s) : 0, 10455242
section appsec
Agent [baseline] (1.16 s) : 0, 1160093
Total [baseline] (10.639 s) : 0, 10638653
Agent [candidate] (1.16 s) : 0, 1159737
Total [candidate] (10.628 s) : 0, 10628407
section iast
Agent [baseline] (1.165 s) : 0, 1164850
Total [baseline] (10.925 s) : 0, 10924915
Agent [candidate] (1.147 s) : 0, 1147400
Total [candidate] (10.93 s) : 0, 10930161
section profiling
Agent [baseline] (1.266 s) : 0, 1265807
Total [baseline] (10.816 s) : 0, 10816311
Agent [candidate] (1.273 s) : 0, 1272961
Total [candidate] (10.86 s) : 0, 10860176

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.026 s	-
Agent	appsec	1.16 s	134.001 ms (13.1%)
Agent	iast	1.165 s	138.757 ms (13.5%)
Agent	profiling	1.266 s	239.715 ms (23.4%)
Total	tracing	10.455 s	-
Total	appsec	10.639 s	183.178 ms (1.8%)
Total	iast	10.925 s	469.44 ms (4.5%)
Total	profiling	10.816 s	360.836 ms (3.5%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.022 s	-
Agent	appsec	1.16 s	138.145 ms (13.5%)
Agent	iast	1.147 s	125.808 ms (12.3%)
Agent	profiling	1.273 s	251.369 ms (24.6%)
Total	tracing	10.455 s	-
Total	appsec	10.628 s	173.165 ms (1.7%)
Total	iast	10.93 s	474.919 ms (4.5%)
Total	profiling	10.86 s	404.934 ms (3.9%)

gantt
    title petclinic - break down per module: candidate=1.50.0-SNAPSHOT~1f12c25da2, baseline=1.50.0-SNAPSHOT~58089f324f

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (684.611 ms) : 0, 684611
BytebuddyAgent [candidate] (682.427 ms) : 0, 682427
GlobalTracer [baseline] (242.585 ms) : 0, 242585
GlobalTracer [candidate] (240.721 ms) : 0, 240721
AppSec [baseline] (55.616 ms) : 0, 55616
AppSec [candidate] (55.173 ms) : 0, 55173
Debugger [baseline] (7.611 ms) : 0, 7611
Debugger [candidate] (6.857 ms) : 0, 6857
Remote Config [baseline] (697.175 µs) : 0, 697
Remote Config [candidate] (702.882 µs) : 0, 703
Telemetry [baseline] (11.34 ms) : 0, 11340
Telemetry [candidate] (12.116 ms) : 0, 12116
section appsec
BytebuddyAgent [baseline] (699.866 ms) : 0, 699866
BytebuddyAgent [candidate] (699.86 ms) : 0, 699860
GlobalTracer [baseline] (236.36 ms) : 0, 236360
GlobalTracer [candidate] (236.605 ms) : 0, 236605
AppSec [baseline] (175.114 ms) : 0, 175114
AppSec [candidate] (175.007 ms) : 0, 175007
Debugger [baseline] (5.938 ms) : 0, 5938
Debugger [candidate] (5.899 ms) : 0, 5899
Remote Config [baseline] (630.746 µs) : 0, 631
Remote Config [candidate] (634.404 µs) : 0, 634
Telemetry [baseline] (7.779 ms) : 0, 7779
Telemetry [candidate] (7.372 ms) : 0, 7372
IAST [baseline] (21.73 ms) : 0, 21730
IAST [candidate] (21.753 ms) : 0, 21753
section iast
BytebuddyAgent [baseline] (814.416 ms) : 0, 814416
BytebuddyAgent [candidate] (801.041 ms) : 0, 801041
GlobalTracer [baseline] (233.097 ms) : 0, 233097
GlobalTracer [candidate] (230.372 ms) : 0, 230372
AppSec [baseline] (53.004 ms) : 0, 53004
AppSec [candidate] (51.114 ms) : 0, 51114
Debugger [baseline] (5.918 ms) : 0, 5918
Debugger [candidate] (5.779 ms) : 0, 5779
Remote Config [baseline] (593.746 µs) : 0, 594
Remote Config [candidate] (577.072 µs) : 0, 577
Telemetry [baseline] (7.861 ms) : 0, 7861
Telemetry [candidate] (7.799 ms) : 0, 7799
IAST [baseline] (26.197 ms) : 0, 26197
IAST [candidate] (26.493 ms) : 0, 26493
section profiling
BytebuddyAgent [baseline] (675.572 ms) : 0, 675572
BytebuddyAgent [candidate] (678.634 ms) : 0, 678634
GlobalTracer [baseline] (359.634 ms) : 0, 359634
GlobalTracer [candidate] (362.027 ms) : 0, 362027
AppSec [baseline] (61.801 ms) : 0, 61801
AppSec [candidate] (62.264 ms) : 0, 62264
Debugger [baseline] (6.251 ms) : 0, 6251
Debugger [candidate] (6.36 ms) : 0, 6360
Remote Config [baseline] (659.852 µs) : 0, 660
Remote Config [candidate] (657.077 µs) : 0, 657
Telemetry [baseline] (8.195 ms) : 0, 8195
Telemetry [candidate] (8.313 ms) : 0, 8313
ProfilingAgent [baseline] (102.814 ms) : 0, 102814
ProfilingAgent [candidate] (103.589 ms) : 0, 103589
Profiling [baseline] (102.838 ms) : 0, 102838
Profiling [candidate] (103.613 ms) : 0, 103613

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.50.0-SNAPSHOT~1f12c25da2, baseline=1.50.0-SNAPSHOT~58089f324f

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.021 s) : 0, 1020732
Total [baseline] (8.652 s) : 0, 8652168
Agent [candidate] (1.017 s) : 0, 1017118
Total [candidate] (8.607 s) : 0, 8607095
section iast
Agent [baseline] (1.15 s) : 0, 1149518
Total [baseline] (9.228 s) : 0, 9228251
Agent [candidate] (1.148 s) : 0, 1147968
Total [candidate] (9.228 s) : 0, 9228182
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.151 s) : 0, 1151419
Total [baseline] (9.221 s) : 0, 9220792
Agent [candidate] (1.15 s) : 0, 1150253
Total [candidate] (9.199 s) : 0, 9199083
section iast_TELEMETRY_OFF
Agent [baseline] (1.145 s) : 0, 1145382
Total [baseline] (9.193 s) : 0, 9192773
Agent [candidate] (1.155 s) : 0, 1154517
Total [candidate] (9.232 s) : 0, 9232212

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.021 s	-
Agent	iast	1.15 s	128.786 ms (12.6%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.151 s	130.687 ms (12.8%)
Agent	iast_TELEMETRY_OFF	1.145 s	124.65 ms (12.2%)
Total	tracing	8.652 s	-
Total	iast	9.228 s	576.083 ms (6.7%)
Total	iast_HARDCODED_SECRET_DISABLED	9.221 s	568.624 ms (6.6%)
Total	iast_TELEMETRY_OFF	9.193 s	540.605 ms (6.2%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.017 s	-
Agent	iast	1.148 s	130.85 ms (12.9%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.15 s	133.136 ms (13.1%)
Agent	iast_TELEMETRY_OFF	1.155 s	137.4 ms (13.5%)
Total	tracing	8.607 s	-
Total	iast	9.228 s	621.086 ms (7.2%)
Total	iast_HARDCODED_SECRET_DISABLED	9.199 s	591.988 ms (6.9%)
Total	iast_TELEMETRY_OFF	9.232 s	625.117 ms (7.3%)

gantt
    title insecure-bank - break down per module: candidate=1.50.0-SNAPSHOT~1f12c25da2, baseline=1.50.0-SNAPSHOT~58089f324f

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (682.628 ms) : 0, 682628
BytebuddyAgent [candidate] (680.3 ms) : 0, 680300
GlobalTracer [baseline] (240.517 ms) : 0, 240517
GlobalTracer [candidate] (239.421 ms) : 0, 239421
AppSec [baseline] (54.458 ms) : 0, 54458
AppSec [candidate] (54.329 ms) : 0, 54329
Debugger [baseline] (9.079 ms) : 0, 9079
Debugger [candidate] (8.968 ms) : 0, 8968
Remote Config [baseline] (700.994 µs) : 0, 701
Remote Config [candidate] (683.736 µs) : 0, 684
Telemetry [baseline] (9.807 ms) : 0, 9807
Telemetry [candidate] (9.797 ms) : 0, 9797
section iast
BytebuddyAgent [baseline] (802.082 ms) : 0, 802082
BytebuddyAgent [candidate] (800.67 ms) : 0, 800670
GlobalTracer [baseline] (230.996 ms) : 0, 230996
GlobalTracer [candidate] (230.939 ms) : 0, 230939
AppSec [baseline] (50.269 ms) : 0, 50269
AppSec [candidate] (49.5 ms) : 0, 49500
Debugger [baseline] (5.907 ms) : 0, 5907
Debugger [candidate] (5.919 ms) : 0, 5919
Remote Config [baseline] (577.736 µs) : 0, 578
Remote Config [candidate] (614.835 µs) : 0, 615
Telemetry [baseline] (7.883 ms) : 0, 7883
Telemetry [candidate] (7.859 ms) : 0, 7859
IAST [baseline] (27.544 ms) : 0, 27544
IAST [candidate] (28.999 ms) : 0, 28999
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (801.813 ms) : 0, 801813
BytebuddyAgent [candidate] (802.5 ms) : 0, 802500
GlobalTracer [baseline] (231.424 ms) : 0, 231424
GlobalTracer [candidate] (231.166 ms) : 0, 231166
AppSec [baseline] (52.889 ms) : 0, 52889
AppSec [candidate] (51.149 ms) : 0, 51149
Debugger [baseline] (5.995 ms) : 0, 5995
Debugger [candidate] (5.959 ms) : 0, 5959
Remote Config [baseline] (628.716 µs) : 0, 629
Remote Config [candidate] (595.533 µs) : 0, 596
Telemetry [baseline] (7.972 ms) : 0, 7972
Telemetry [candidate] (7.965 ms) : 0, 7965
IAST [baseline] (27.275 ms) : 0, 27275
IAST [candidate] (27.387 ms) : 0, 27387
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (799.26 ms) : 0, 799260
BytebuddyAgent [candidate] (805.686 ms) : 0, 805686
GlobalTracer [baseline] (230.174 ms) : 0, 230174
GlobalTracer [candidate] (231.77 ms) : 0, 231770
AppSec [baseline] (54.287 ms) : 0, 54287
AppSec [candidate] (55.681 ms) : 0, 55681
Debugger [baseline] (5.957 ms) : 0, 5957
Debugger [candidate] (6.005 ms) : 0, 6005
Remote Config [baseline] (587.194 µs) : 0, 587
Remote Config [candidate] (623.31 µs) : 0, 623
Telemetry [baseline] (7.755 ms) : 0, 7755
Telemetry [candidate] (7.804 ms) : 0, 7804
IAST [baseline] (23.809 ms) : 0, 23809
IAST [candidate] (23.242 ms) : 0, 23242

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2025-05-21T16:47:30	2025-05-21T16:55:15
git_branch	master	nicholas.hulston/fix-exception-replay-in-lambda
git_commit_date	1747843428	1747845400
git_commit_sha	`58089f3`	`1f12c25`
release_version	1.50.0-SNAPSHOT~58089f324f	1.50.0-SNAPSHOT~1f12c25da2
start_time	2025-05-21T16:47:16	2025-05-21T16:55:00

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1747846912	1747846912
ci_job_id	948550022	948550022
ci_pipeline_id	65761122	65761122
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-3qkfhwhr-project-304-concurrent-0-ex7sp2gz 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-3qkfhwhr-project-304-concurrent-0-ex7sp2gz 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 18 unstable metrics.

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.50.0-SNAPSHOT~1f12c25da2, baseline=1.50.0-SNAPSHOT~58089f324f
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.366 ms) : 1345, 1386
.   : milestone, 1366,
appsec (1.731 ms) : 1709, 1754
.   : milestone, 1731,
appsec_no_iast (1.735 ms) : 1712, 1758
.   : milestone, 1735,
code_origins (1.677 ms) : 1649, 1704
.   : milestone, 1677,
iast (1.505 ms) : 1480, 1529
.   : milestone, 1505,
profiling (1.516 ms) : 1492, 1540
.   : milestone, 1516,
tracing (1.498 ms) : 1473, 1523
.   : milestone, 1498,
section candidate
no_agent (1.345 ms) : 1326, 1364
.   : milestone, 1345,
appsec (1.729 ms) : 1705, 1752
.   : milestone, 1729,
appsec_no_iast (1.722 ms) : 1698, 1746
.   : milestone, 1722,
code_origins (1.661 ms) : 1634, 1687
.   : milestone, 1661,
iast (1.519 ms) : 1495, 1542
.   : milestone, 1519,
profiling (1.526 ms) : 1501, 1551
.   : milestone, 1526,
tracing (1.496 ms) : 1474, 1519
.   : milestone, 1496,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.366 ms [1.345 ms, 1.386 ms]	-
appsec	1.731 ms [1.709 ms, 1.754 ms]	365.75 µs (26.8%)
appsec_no_iast	1.735 ms [1.712 ms, 1.758 ms]	369.749 µs (27.1%)
code_origins	1.677 ms [1.649 ms, 1.704 ms]	310.987 µs (22.8%)
iast	1.505 ms [1.48 ms, 1.529 ms]	139.004 µs (10.2%)
profiling	1.516 ms [1.492 ms, 1.54 ms]	150.313 µs (11.0%)
tracing	1.498 ms [1.473 ms, 1.523 ms]	132.232 µs (9.7%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.345 ms [1.326 ms, 1.364 ms]	-
appsec	1.729 ms [1.705 ms, 1.752 ms]	383.554 µs (28.5%)
appsec_no_iast	1.722 ms [1.698 ms, 1.746 ms]	376.675 µs (28.0%)
code_origins	1.661 ms [1.634 ms, 1.687 ms]	315.461 µs (23.5%)
iast	1.519 ms [1.495 ms, 1.542 ms]	173.606 µs (12.9%)
profiling	1.526 ms [1.501 ms, 1.551 ms]	180.613 µs (13.4%)
tracing	1.496 ms [1.474 ms, 1.519 ms]	151.12 µs (11.2%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.50.0-SNAPSHOT~1f12c25da2, baseline=1.50.0-SNAPSHOT~58089f324f
    dateFormat X
    axisFormat %s
section baseline
no_agent (380.8 µs) : 361, 401
.   : milestone, 381,
iast (522.769 µs) : 501, 545
.   : milestone, 523,
iast_FULL (738.32 µs) : 715, 762
.   : milestone, 738,
iast_GLOBAL (569.882 µs) : 547, 592
.   : milestone, 570,
iast_HARDCODED_SECRET_DISABLED (521.483 µs) : 499, 544
.   : milestone, 521,
iast_INACTIVE (460.825 µs) : 440, 482
.   : milestone, 461,
iast_TELEMETRY_OFF (515.777 µs) : 493, 539
.   : milestone, 516,
tracing (464.261 µs) : 442, 487
.   : milestone, 464,
section candidate
no_agent (378.1 µs) : 358, 398
.   : milestone, 378,
iast (523.531 µs) : 500, 547
.   : milestone, 524,
iast_FULL (730.646 µs) : 709, 753
.   : milestone, 731,
iast_GLOBAL (573.889 µs) : 551, 597
.   : milestone, 574,
iast_HARDCODED_SECRET_DISABLED (518.315 µs) : 495, 542
.   : milestone, 518,
iast_INACTIVE (463.142 µs) : 441, 485
.   : milestone, 463,
iast_TELEMETRY_OFF (517.545 µs) : 494, 541
.   : milestone, 518,
tracing (459.539 µs) : 437, 482
.   : milestone, 460,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	380.8 µs [360.861 µs, 400.739 µs]	-
iast	522.769 µs [501.016 µs, 544.523 µs]	141.969 µs (37.3%)
iast_FULL	738.32 µs [714.793 µs, 761.847 µs]	357.52 µs (93.9%)
iast_GLOBAL	569.882 µs [547.362 µs, 592.401 µs]	189.082 µs (49.7%)
iast_HARDCODED_SECRET_DISABLED	521.483 µs [498.989 µs, 543.977 µs]	140.682 µs (36.9%)
iast_INACTIVE	460.825 µs [439.525 µs, 482.124 µs]	80.024 µs (21.0%)
iast_TELEMETRY_OFF	515.777 µs [492.548 µs, 539.006 µs]	134.977 µs (35.4%)
tracing	464.261 µs [441.753 µs, 486.769 µs]	83.461 µs (21.9%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	378.1 µs [358.202 µs, 397.999 µs]	-
iast	523.531 µs [500.093 µs, 546.969 µs]	145.431 µs (38.5%)
iast_FULL	730.646 µs [708.757 µs, 752.536 µs]	352.546 µs (93.2%)
iast_GLOBAL	573.889 µs [550.687 µs, 597.09 µs]	195.788 µs (51.8%)
iast_HARDCODED_SECRET_DISABLED	518.315 µs [495.1 µs, 541.53 µs]	140.214 µs (37.1%)
iast_INACTIVE	463.142 µs [440.916 µs, 485.368 µs]	85.041 µs (22.5%)
iast_TELEMETRY_OFF	517.545 µs [493.977 µs, 541.114 µs]	139.445 µs (36.9%)
tracing	459.539 µs [436.908 µs, 482.169 µs]	81.438 µs (21.5%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nicholas.hulston/fix-exception-replay-in-lambda
git_commit_date	1747843428	1747845400
git_commit_sha	`58089f3`	`1f12c25`
release_version	1.50.0-SNAPSHOT~58089f324f	1.50.0-SNAPSHOT~1f12c25da2

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1747847409	1747847409
ci_job_id	948550023	948550023
ci_pipeline_id	65761122	65761122
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-pw3vcngw-project-304-concurrent-3-huawmceg 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-pw3vcngw-project-304-concurrent-3-huawmceg 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~1f12c25da2, baseline=1.50.0-SNAPSHOT~58089f324f
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.473 ms) : 1462, 1485
.   : milestone, 1473,
appsec (2.394 ms) : 2345, 2442
.   : milestone, 2394,
iast (2.174 ms) : 2113, 2235
.   : milestone, 2174,
iast_GLOBAL (2.211 ms) : 2150, 2272
.   : milestone, 2211,
profiling (2.032 ms) : 1982, 2082
.   : milestone, 2032,
tracing (2.004 ms) : 1956, 2051
.   : milestone, 2004,
section candidate
no_agent (1.478 ms) : 1467, 1490
.   : milestone, 1478,
appsec (2.388 ms) : 2340, 2437
.   : milestone, 2388,
iast (2.175 ms) : 2114, 2237
.   : milestone, 2175,
iast_GLOBAL (2.212 ms) : 2151, 2273
.   : milestone, 2212,
profiling (2.028 ms) : 1979, 2077
.   : milestone, 2028,
tracing (1.998 ms) : 1951, 2046
.   : milestone, 1998,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.473 ms [1.462 ms, 1.485 ms]	-
appsec	2.394 ms [2.345 ms, 2.442 ms]	920.421 µs (62.5%)
iast	2.174 ms [2.113 ms, 2.235 ms]	700.915 µs (47.6%)
iast_GLOBAL	2.211 ms [2.15 ms, 2.272 ms]	737.525 µs (50.1%)
profiling	2.032 ms [1.982 ms, 2.082 ms]	558.637 µs (37.9%)
tracing	2.004 ms [1.956 ms, 2.051 ms]	530.355 µs (36.0%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.478 ms [1.467 ms, 1.49 ms]	-
appsec	2.388 ms [2.34 ms, 2.437 ms]	910.161 µs (61.6%)
iast	2.175 ms [2.114 ms, 2.237 ms]	697.035 µs (47.2%)
iast_GLOBAL	2.212 ms [2.151 ms, 2.273 ms]	733.614 µs (49.6%)
profiling	2.028 ms [1.979 ms, 2.077 ms]	549.884 µs (37.2%)
tracing	1.998 ms [1.951 ms, 2.046 ms]	520.198 µs (35.2%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~1f12c25da2, baseline=1.50.0-SNAPSHOT~58089f324f
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.961 s) : 14961000, 14961000
.   : milestone, 14961000,
appsec (14.851 s) : 14851000, 14851000
.   : milestone, 14851000,
iast (18.796 s) : 18796000, 18796000
.   : milestone, 18796000,
iast_GLOBAL (18.328 s) : 18328000, 18328000
.   : milestone, 18328000,
profiling (15.29 s) : 15290000, 15290000
.   : milestone, 15290000,
tracing (14.993 s) : 14993000, 14993000
.   : milestone, 14993000,
section candidate
no_agent (15.291 s) : 15291000, 15291000
.   : milestone, 15291000,
appsec (14.893 s) : 14893000, 14893000
.   : milestone, 14893000,
iast (18.725 s) : 18725000, 18725000
.   : milestone, 18725000,
iast_GLOBAL (17.728 s) : 17728000, 17728000
.   : milestone, 17728000,
profiling (15.106 s) : 15106000, 15106000
.   : milestone, 15106000,
tracing (14.71 s) : 14710000, 14710000
.   : milestone, 14710000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.961 s [14.961 s, 14.961 s]	-
appsec	14.851 s [14.851 s, 14.851 s]	-110.0 ms (-0.7%)
iast	18.796 s [18.796 s, 18.796 s]	3.835 s (25.6%)
iast_GLOBAL	18.328 s [18.328 s, 18.328 s]	3.367 s (22.5%)
profiling	15.29 s [15.29 s, 15.29 s]	329.0 ms (2.2%)
tracing	14.993 s [14.993 s, 14.993 s]	32.0 ms (0.2%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.291 s [15.291 s, 15.291 s]	-
appsec	14.893 s [14.893 s, 14.893 s]	-398.0 ms (-2.6%)
iast	18.725 s [18.725 s, 18.725 s]	3.434 s (22.5%)
iast_GLOBAL	17.728 s [17.728 s, 17.728 s]	2.437 s (15.9%)
profiling	15.106 s [15.106 s, 15.106 s]	-185.0 ms (-1.2%)
tracing	14.71 s [14.71 s, 14.71 s]	-581.0 ms (-3.8%)

jpbempel · 2025-05-20T07:25:04Z

...gent-debugger/src/test/java/com/datadog/debugger/exception/DefaultExceptionDebuggerTest.java

+    List<Snapshot> snapshots = state.getSnapshots();
+    StackTraceElement ste = exception.getStackTrace()[0];
+    CapturedStackFrame dummyFrame = CapturedStackFrame.from(ste);
+    for (Snapshot snap : snapshots) {
+      snap.getStack().add(0, dummyFrame);
+    }


Can we mock instead the exception that is provided to handleException instead to return a truncated stack like what it is done by AWS lambda?

LMK what you think -- i ended up having to mock the snapshot, probe, exception state, and exception probe manager

i ended up having to mock the snapshot, probe, exception state, and exception probe manager

I am not sure it is necessary to do all of that, collecting real exception in snapshot is what happen anyway.
What matters is the truncated stacktrace for main exception

…ambda

github-actions · 2025-11-01T03:19:31Z

This pull request has been marked as stale because it has not had activity over the past quarter. It will be closed in 7 days if no further activity occurs. Feel free to reopen the PR if you are still working on it.

github-actions · 2025-12-01T03:31:01Z

This pull request has been closed because it has not had activity over the past quarter. Feel free to reopen the PR if you are still working on it.

nhulston added 2 commits May 19, 2025 13:13

add fallback for lambda

c4a17c6

unit test

a60af39

nhulston added type: enhancement Enhancements and improvements comp: debugger Dynamic Instrumentation labels May 19, 2025

nhulston marked this pull request as ready for review May 19, 2025 18:32

nhulston requested a review from a team as a code owner May 19, 2025 18:32

nhulston requested review from jpbempel and removed request for a team May 19, 2025 18:32

jpbempel reviewed May 20, 2025

View reviewed changes

hghotra approved these changes May 20, 2025

View reviewed changes

nhulston and others added 2 commits May 20, 2025 10:45

mock exception to return truncated stack like Lambda

03689f9

Merge branch 'master' into nicholas.hulston/fix-exception-replay-in-l…

130556a

…ambda

nhulston requested review from hghotra and jpbempel and removed request for hghotra May 20, 2025 14:48

nhulston added 2 commits May 20, 2025 13:28

Merge branch 'master' into nicholas.hulston/fix-exception-replay-in-l…

a968384

…ambda

Merge branch 'master' into nicholas.hulston/fix-exception-replay-in-l…

1f12c25

…ambda

nhulston marked this pull request as draft July 23, 2025 19:40

github-actions bot added the tag: stale Stale pull requests label Nov 1, 2025

github-actions bot closed this Dec 1, 2025

shreyamalpani mentioned this pull request Jan 30, 2026

Fix Exception Replay in Lambda #10380

Closed

jpbempel mentioned this pull request Feb 2, 2026

Fix Exception Replay in Lambda #10505

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix Exception Replay in Lambda#8849

Fix Exception Replay in Lambda#8849
nhulston wants to merge 6 commits intomasterfrom
nicholas.hulston/fix-exception-replay-in-lambda

nhulston commented May 19, 2025 •

edited

Loading

Uh oh!

pr-commenter bot commented May 19, 2025 •

edited

Loading

Uh oh!

pr-commenter bot commented May 19, 2025 •

edited

Loading

Uh oh!

jpbempel May 20, 2025

Uh oh!

nhulston May 20, 2025

Uh oh!

jpbempel May 20, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Nov 1, 2025

Uh oh!

github-actions bot commented Dec 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

Conversation

nhulston commented May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

pr-commenter bot commented May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Debugger benchmarks

Parameters

Summary

Uh oh!

pr-commenter bot commented May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

jpbempel May 20, 2025

Choose a reason for hiding this comment

Uh oh!

nhulston May 20, 2025

Choose a reason for hiding this comment

Uh oh!

jpbempel May 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Nov 1, 2025

Uh oh!

github-actions bot commented Dec 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

nhulston commented May 19, 2025 •

edited

Loading

pr-commenter bot commented May 19, 2025 •

edited

Loading

pr-commenter bot commented May 19, 2025 •

edited

Loading

jpbempel May 20, 2025 •

edited

Loading