Skip to content

Add-claude-3.7 to repro journal #222

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
recursix merged 4 commits into from
Mar 13, 2025
Merged

Add-claude-3.7 to repro journal #222

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
c9d2ef9
Add AGENT_37_SONNET support and update agent configurations
jardinetsouffleton Feb 25, 2025
2d8d4de
add new results to repro journal
jardinetsouffleton Feb 25, 2025
24f48f3
Revert changes to main script
jardinetsouffleton Mar 13, 2025
463e0cf
Merge branch 'main' into add-claude-3.7
recursix Mar 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,10 @@
RANDOM_SEARCH_AGENT,
AGENT_4o,
AGENT_4o_MINI,
AGENT_o3_MINI,
AGENT_o1_MINI,
AGENT_37_SONNET,
AGENT_CLAUDE_SONNET_35,
)
from agentlab.experiments.study import Study

Expand Down
1 change: 1 addition & 0 deletions reproducibility_journal.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,3 +73,4 @@ Leo Boisvert,GenericAgent-openai_o1-mini-2024-09-12,workarena_l1,0.4.1,2025-02-0
M: src/agentlab/agents/generic_agent/agent_configs.py
M: src/agentlab/analyze/agent_xray.py
M: src/agentlab/llm/llm_configs.py",0.13.3,1d2d7160e5b7ec9954ecb48988f71eb56288dd29,"
Leo Boisvert,GenericAgent-anthropic_claude-3.7-sonnet,workarena_l1,0.4.1,2025-02-25_02-32-09,d4f900c2-1de1-4e4b-a3ab-495ff2675fff,0.515,0.028,0,330/330,None,Linux (#68-Ubuntu SMP Mon Oct 7 14:34:20 UTC 2024),3.12.3,1.44.0,v0.4.0,c9d2ef9648435ef1119950ecb1a0734497ccc33b,,0.13.3,1d2d7160e5b7ec9954ecb48988f71eb56288dd29,
2 changes: 2 additions & 0 deletions src/agentlab/agents/generic_agent/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
AGENT_4o,
AGENT_4o_MINI,
AGENT_CLAUDE_SONNET_35,
AGENT_37_SONNET,
AGENT_CLAUDE_SONNET_35_VISION,
AGENT_4o_VISION,
AGENT_4o_MINI_VISION,
Expand All @@ -36,6 +37,7 @@
"RANDOM_SEARCH_AGENT",
"AGENT_CUSTOM",
"AGENT_CLAUDE_SONNET_35",
"AGENT_37_SONNET",
"AGENT_4o_VISION",
"AGENT_4o_MINI_VISION",
"AGENT_CLAUDE_SONNET_35_VISION",
Expand Down
5 changes: 4 additions & 1 deletion src/agentlab/agents/generic_agent/agent_configs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -264,7 +264,10 @@
chat_model_args=CHAT_MODEL_ARGS_DICT["openrouter/anthropic/claude-3.5-sonnet:beta"],
flags=FLAGS_GPT_4o,
)

AGENT_37_SONNET = GenericAgentArgs(
chat_model_args=CHAT_MODEL_ARGS_DICT["openrouter/anthropic/claude-3.7-sonnet"],
flags=FLAGS_GPT_4o,
)
Comment on lines +267 to +270
Copy link

@korbit-ai korbit-ai bot Feb 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Invalid Claude Model Version category Functionality

Tell me more
What is the issue?

The code references a non-existent Claude 3.7 model. Claude 3 only has 3.0 (Opus/Sonnet), 3.1, and 3.2 variants.

Why this matters

Using a non-existent model ID will cause runtime failures when the agent attempts to make API calls.

Suggested change ∙ Feature Preview

Correct the model version to use an existing Claude model. For example:

AGENT_3_SONNET = GenericAgentArgs(
    chat_model_args=CHAT_MODEL_ARGS_DICT["openrouter/anthropic/claude-3-sonnet"],
    flags=FLAGS_GPT_4o,
)

Report a problem with this comment

💬 Chat with Korbit by mentioning @korbit-ai.

AGENT_o3_MINI = GenericAgentArgs(
chat_model_args=CHAT_MODEL_ARGS_DICT["openai/o3-mini-2025-01-31"],
flags=FLAGS_GPT_4o,
Expand Down
8 changes: 8 additions & 0 deletions src/agentlab/llm/llm_configs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -184,6 +184,14 @@
max_new_tokens=2_000,
temperature=1e-1,
),
"openrouter/anthropic/claude-3.7-sonnet": OpenRouterModelArgs(
model_name="anthropic/claude-3.7-sonnet",
max_total_tokens=200_000,
max_input_tokens=200_000,
max_new_tokens=8_192,
temperature=1e-1,
Copy link

@korbit-ai korbit-ai bot Feb 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-intuitive number format category Readability

Tell me more
What is the issue?

Scientific notation 1e-1 is used repeatedly for temperature values, which is less immediately readable than its decimal equivalent 0.1.

Why this matters

Using scientific notation for a simple decimal value adds unnecessary cognitive load when reading the configuration.

Suggested change ∙ Feature Preview

Replace all instances of temperature=1e-1 with temperature=0.1

Report a problem with this comment

💬 Chat with Korbit by mentioning @korbit-ai.

vision_support=True,
),
"openrouter/openai/o1-mini-2024-09-12": OpenRouterModelArgs(
model_name="openai/o1-mini-2024-09-12",
max_total_tokens=128_000,
Expand Down
Loading