Chore/small improvements - #2200
Conversation
`Env.init` declared `*kwargs`, which binds arbitrary positional arguments rather than keyword arguments. Since callers and subclasses pass configuration by keyword, the base signature should be `**kwargs`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Vj1rThtWbAUSYd4vuuWLa
The `bool` and `int/float` branches both coerced to `float`, and `bool` is already a subclass of `int`, so the two branches were redundant. Collapse into a single `isinstance` check and use the imported `List` type for consistency with the module's other annotations. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Vj1rThtWbAUSYd4vuuWLa
`hasattr(...) and getattr(...) is not None` did two attribute lookups to express a single "present and non-null" test. `getattr` with a `None` default collapses both into one lookup with identical behavior. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Vj1rThtWbAUSYd4vuuWLa
Three boolean environment variables each repeated the same
`str(os.environ.get(...)).lower() in ("true", "1", "yes")` incantation.
Factor it into a single `_env_flag` helper so the truthy set is defined
once and each flag reads as a one-liner.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Vj1rThtWbAUSYd4vuuWLa
`load_env_creator` unpacked `name.split(":")` directly, so a string
without exactly one colon failed with an opaque "not enough values to
unpack" ValueError. Validate the format up front and raise a message
that names the offending value and shows the expected form.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Vj1rThtWbAUSYd4vuuWLa
`_import_object` unpacked `path.rsplit(".", 1)` directly, so a path with
no dot raised an opaque unpack ValueError. Check for a dot first and
raise a message that names the bad value and the expected format.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Vj1rThtWbAUSYd4vuuWLa
There was a problem hiding this comment.
Code Review
This pull request introduces several robustness improvements, bug fixes, and refactorings across the codebase, including validating import paths and entry points, fixing the signature of the init method in skyrl_gym/core.py from *kwargs to **kwargs, simplifying metric aggregation, and introducing a helper to parse boolean environment variables. The review feedback suggests hardening the new validation logic in auto.py and registration.py to handle edge cases where the input string starts or ends with the delimiter, and refactoring the _env_flag helper to avoid unnecessary string conversions of the default value.
| if "." not in path: | ||
| raise ValueError( | ||
| f"Invalid import path '{path}'. Expected a dotted 'module.path.ClassName', e.g. 'skyrl_agent.agents.runner.AgentRunner'." | ||
| ) |
There was a problem hiding this comment.
The validation only checks if . is in path. However, if path starts or ends with a dot (e.g., ".ClassName" or "module.path."), rsplit(".", 1) will produce empty strings for module_path or class_name. This can lead to unhandled ValueError (from import_module("")) or AttributeError (from getattr(..., "")).
We can make this more robust by also checking that neither split part is empty.
| if "." not in path: | |
| raise ValueError( | |
| f"Invalid import path '{path}'. Expected a dotted 'module.path.ClassName', e.g. 'skyrl_agent.agents.runner.AgentRunner'." | |
| ) | |
| if "." not in path or not all(path.rsplit(".", 1)): | |
| raise ValueError( | |
| f"Invalid import path '{path}'. Expected a dotted 'module.path.ClassName', e.g. 'skyrl_agent.agents.runner.AgentRunner'." | |
| ) |
| if name.count(":") != 1: | ||
| raise error.Error( | ||
| f"Malformed entry point '{name}'. Expected the form '(import path):(environment name)', e.g. 'my_pkg.envs:MyEnv'." | ||
| ) |
There was a problem hiding this comment.
The validation checks if there is exactly one : in name. However, if name starts or ends with : (e.g., ":MyEnv" or "my_pkg:"), name.split(":") will produce empty strings for mod_name or attr_name. This can lead to unhandled ValueError (from import_module("")) or AttributeError (from getattr(..., "")).
We can make this more robust by also checking that neither split part is empty.
| if name.count(":") != 1: | |
| raise error.Error( | |
| f"Malformed entry point '{name}'. Expected the form '(import path):(environment name)', e.g. 'my_pkg.envs:MyEnv'." | |
| ) | |
| if name.count(":") != 1 or not all(name.split(":")): | |
| raise error.Error( | |
| f"Malformed entry point '{name}'. Expected the form '(import path):(environment name)', e.g. 'my_pkg.envs:MyEnv'." | |
| ) |
|
|
||
| def _env_flag(name: str, default: bool = False) -> bool: | ||
| """Parse a boolean environment variable, treating "true"/"1"/"yes" as True.""" | ||
| return os.environ.get(name, str(default)).strip().lower() in _TRUTHY |
There was a problem hiding this comment.
Converting the boolean default to a string ("True" or "False") and then parsing it back via _TRUTHY is a bit indirect and less readable. It is cleaner and more idiomatic to check if the environment variable is present first, and if not, directly return the boolean default.
| return os.environ.get(name, str(default)).strip().lower() in _TRUTHY | |
| val = os.environ.get(name) | |
| if val is None: | |
| return default | |
| return val.strip().lower() in _TRUTHY |
|
| raise NotImplementedError | ||
|
|
||
| def init(self, *kwargs) -> Tuple[ObsType, Dict[str, Any]]: | ||
| def init(self, **kwargs) -> Tuple[ObsType, Dict[str, Any]]: |
There was a problem hiding this comment.
The base Env.init method now accepts only keyword arguments, but concrete environments define init(self, prompt), callers pass prompt positionally, and the environment-authoring guide documents that positional form. This gives extension authors and static-analysis tools an inaccurate public contract; the base signature should declare the actual prompt parameter.
| def init(self, **kwargs) -> Tuple[ObsType, Dict[str, Any]]: | |
| def init(self, prompt: ObsType) -> Tuple[ObsType, Dict[str, Any]]: |
Note
Low Risk
Changes are mostly validation, typing, and DRY refactors; the
Env.init**kwargschange is a minor signature fix with existing subclasses using named parameters.Overview
This PR tightens configuration and API ergonomics with clearer validation and small refactors, plus clearer GSM8K test documentation.
Validation:
_import_objectin the agent auto-loader now rejects import paths without a dot with an explicit error.load_env_creatorrejects malformed registry entry points unless they contain exactly one:.API / behavior: The base
Env.initsignature is updated from*kwargsto**kwargs. Metric aggregation indefault_aggregate_metricsis simplified to average anyint/float(including bools) and skip other types.check_is_vlmuses a singlegetattr(..., None)check.Refactor: Boolean SkyRL env vars (
SKYRL_LD_LIBRARY_PATH_EXPORT,SKYRL_PYTHONPATH_EXPORT,SKYRL_DUMP_INFRA_LOG_TO_STDOUT) share a new_env_flaghelper. GSM8K scoring tests gain comments explaining format vs correctness expectations.Reviewed by Cursor Bugbot for commit 3426c47. Bugbot is set up for automated code reviews on this repo. Configure here.