feat: add custom output directory support to vf-eval

[ericlau98]

Summary

• Add optional path argument to -s/--save-results flag in vf-eval
• When path provided, results save to custom location instead of default ./outputs
• Adds output_dir field to EvalConfig and updates path_utils.py

Usage

vf-eval gsm8k -s                 # save to default location
vf-eval gsm8k -s /custom/path    # save to custom location

---

Note

Enables saving eval outputs to a custom directory and adds a new RL config.

• CLI: --save-results/-s now optionally accepts a PATH; without a path it behaves as a boolean flag. The chosen path is propagated as output_dir.
• Plumb output_dir through the stack: add output_dir to EvalConfig, pass it from verifiers/scripts/eval.py, and make get_eval_results_path in verifiers/utils/path_utils.py honor it while preserving default ./outputs behavior.
• Add configs/vf-rl/math-python.toml with env.id and run_name set to math-python.

^{Written by Cursor Bugbot for commit cd4c172. This will update automatically on new commits. Configure here.}

[CLAassistant]

All committers have signed the CLA.

[cursor]

Config file references wrong environment ID

The math-python.toml configuration file sets id = "primeintellect/wiki-search" and run_name = "wiki-search", but it should reference the math-python environment instead. The environments/math_python/ directory exists with proper metadata indicating the environment ID is math-python. This appears to be a copy-paste error from another config file, causing the math-python config to load an entirely different environment when used.

configs/vf-rl/math-python.toml#L3-L4

verifiers/configs/vf-rl/math-python.toml

Lines 3 to 4 in 0e0e872

	[env]
	id = "primeintellect/wiki-search"

configs/vf-rl/math-python.toml#L17-L18

verifiers/configs/vf-rl/math-python.toml

Lines 17 to 18 in 0e0e872

	[trainer.args]
	run_name = "wiki-search"

 

---

[ericlau98]

Updated the math-python.toml based purely on what the Bugbot said, otherwise didn't touch that file. All existing functionality is still built in, nothing changes about default behaviour. I added the ability to define a custom output path using the -s/--save-results flag to allow for streamlining custom workflows. @willccbb Please let me know if you need anything else from me, would love to have this merged.

[mikasenghaas]

nice, lgtm

[willccbb]

Can ignore the bugbot comment, not related to the PR.

What's the main use case for you for custom save paths? I get that it's maybe nice to have sometimes, but this might conflict with some other planned changes + not sure we're gonna want to maintain the plumbing here as-is. Definitely a nice feature idea though, wanna maybe open as an issue? Can make sure we incorporate something like this into upcoming updates.

[ericlau98]

Can ignore the bugbot comment, not related to the PR.

What's the main use case for you for custom save paths? I get that it's maybe nice to have sometimes, but this might conflict with some other planned changes + not sure we're gonna want to maintain the plumbing here as-is. Definitely a nice feature idea though, wanna maybe open as an issue? Can make sure we incorporate something like this into upcoming updates.

@willccbb The use case is for automation! The change maintains existing functionality should no custom path be specified. Are there any specific concerns I'm not thinking of?

[willccbb]

we're planning a more comprehensive update to vf-eval / related features to support better configurability + more robustness for building on top of it; currently it's intended as a pretty lightweight manual helper script

will leave this open + think about it a bit more

for a stopgap, would prefer if we can do this without the dual-use of -s and just have a more explicit flag