✨ Use Pydantic for parameter validation

[svlandeg]

Description

Make Pydantic a required dependency and rely on it for parameter validation.

Click's ParamType hierarchy is now replaced by a three-layer design:

• TypeDescriptor objects store static facts about a parameter's type (in the new module coercion.py)
• Pydantic TypeAdapter objects are defined in the new module adapters.py
• RuntimeParam subclasses own coercion (also defined in coercion.py)

Further type-specific functionality is bundled in the new module param_types.py.

How to review this

• Probably read all of the below first 👇
• Note that _click/types.py and typer/_types.py are deleted
• First look at typer/param_types.py, then typer/adapters.py, then typer/coercion.py
• Look at the changes in typer/core.py
• Check all other changes in typer/*.py
• Check all changes in typer/_click/*.py
• Check all changes in the test suite & documentation

Extended/improved functionality

• More date time formats are now supported by default, instead of just the three [%Y-%m-%d|%Y-%m-%dT%H:%M:%S|%Y-%m-%d %H:%M:%S]
• Started moving towards a TyperParameter, not quite finished yet with some ugly imports from within _click but we'll deal with those in follow-up work

Tests with same behaviour on master

• Added test_bool_convert_valid to ensure that the "bool" conversion has remained the same for all kinds of inputs.
• Rewrote several tests to use Typer public API instead of internals
• test_path_resolves, cf point 1) in the Fable review 👇

Breaking changes

(more or less in order of breakingness from most to least)

• Removed support for click_type and open bounds through min_open and max_open. This greatly simplifies the code base while still providing an alternative by setting parser instead.
• A list of choices is now shown as <list[Eggs|Bacon|Cheese]> (before there was no visual distinction between a single choice, or a list of them).
• When not providing a default for DateTime, only one example is shown in the metavar column (<%Y-%m-%d> instead of <%Y-%m-%d|%Y-%m-%dT%H:%M:%S|%Y-%m-%d %H:%M:%S>) even though all three (and more) will work. It's not just those three anymore, so it felt arbitrary to show 3 out of n. We can decide to reverse the change and keep showing those three formats though. Alternatively, we could just display something like <datetime> but then the user might be unsure on how to exactly provide that.
• Validation error messages have changed from Click's to Pydantic's phrasing. This mostly requires updating test suites and otherwise shouldn't impact users too badly.
• Removed repr functionality of param.type. This wasn't really used except for the tests that were recently added in ➖ Vendor Click and streamline Typer's functionality and code base #1774.
• A parameter typed as Any does NOT raise RuntimeError: Type not yet supported anymore, but instead just falls through and gets a generic TypeAdapter(annotation)

Bug fixes (also breaking bwd compat)

• def main(age: int = typer.Option(15.3)) will now throw a validation error by Pydantic instead of int(15.3) converting it to 15. I consider this a bug fix instead of a regression, and have added a test test_int_rejects_float_default for it.
• Types are now inferred from the default if the parameter is untyped. Fixes Default value of `False` cast to `'False'` and is thus True #942, cf. new test test_default_infers_param_type.

Decision points

• _parse_cli_bool was created to ensure we still parse "" as False and to strip whitespace from something like " True ". This is just to mimic old Click behaviour. If we don't do this preprocessing (cleaner code base), the empty string and the non-stripped strings won't pass Pydantic validation and it would be slightly breaking.
• How to display the datetime type if no formats are provided by the user (cf ☝️ "Breaking changes")

AI Disclaimer

Cursor was used as a micro-managed junior. Every edit was reviewed & understood by me.

TODO

• Mention Pydantic dependency in the docs
• Update the tutorial, focus on (changed) error messages
• Fix coverage
• Merge in the fix from 🏷️ Make Parameter.name typed as str instead of str | None #1878
• I added tests for the change.
• Coverage stays at 100%.
• Metavar printing that came up in this PR has already been fixed in tests & documentation in 💥 Update metavar printing #1863.
• Self review with Fable

[svlandeg]

This test currently mimics master behaviour 100%.

[github-actions]

📝 Docs preview

Last commit eb0d6cc at: https://9b76f1e5.typertiangolo.pages.dev

Modified Pages

• https://9b76f1e5.typertiangolo.pages.dev/ - (before)
• https://9b76f1e5.typertiangolo.pages.dev/tutorial/install/ - (before)
• https://9b76f1e5.typertiangolo.pages.dev/tutorial/multiple-values/options-with-multiple-values/ - (before)
• https://9b76f1e5.typertiangolo.pages.dev/tutorial/parameter-types/ - (before)
• https://9b76f1e5.typertiangolo.pages.dev/tutorial/parameter-types/datetime/ - (before)
• https://9b76f1e5.typertiangolo.pages.dev/tutorial/parameter-types/enum/ - (before)
• https://9b76f1e5.typertiangolo.pages.dev/tutorial/parameter-types/number/ - (before)
• https://9b76f1e5.typertiangolo.pages.dev/tutorial/parameter-types/path/ - (before)
• https://9b76f1e5.typertiangolo.pages.dev/tutorial/parameter-types/uuid/ - (before)

[svlandeg]

Removed the note that "most are tiny" as that feels a bit untrue now that Pydantic is added to the list.

[svlandeg]

WIP: I'm now self reviewing this with Fable, and it's got opinions 🥲

Review by Fable 5, transcribed & commented by me

Positives

• The direction is right and the destination is good
• Threading ctx/param through Pydantic's validation context is elegant
• the parity tests (test_bool_convert_valid, the new envvar-list-splitting tests, test_coercion_tuple_files) show real care about preserving behavior

Issues

1) Path-specific kwargs for non-Path annotations

Fable got confused and then confused me as well, so allow me to recap the issues involved here.

• on master, a str parameter with something like resolve_path=True would silently ignore the Path-specific setting and just resolve as str, as types.STRING would be returned before checking anything Path-like.

  • As a consequence, we have the test pytest.param(Annotated[str, typer.Option(..., resolve_path=True)], "<str>") which means this resolves to str, unchanged on this PR. So far so good, IMO.
  • One thing to note is that those Path-like args are being ignored silently, which is something to be reconsidered, but not in this PR.

• on master, a parameter typed as Any WOULD get parsed as a Path if it has something like allow_dash orso (cf here). And as a consequence, something like tmp_path / "first_dir" / "second_dir" / ".." / "file.txt" (notice the ..) will be resolved correctly to tmp_path / "first_dir" / "file.txt". I've edited this PR to also succeed here to avoid breaking behaviour, but it's something we can reconsider in the future. Some of the necessary functions for this are marked as "defined as such for bwd-compat".

• Both the str and Any behaviour is now captured in test_path_resolves, which behaves exactly the same on master as on this PR. Note that the test test_path_coerced should have actually tested this properly but doesn't, and I kept it unchanged in this PR for now (i.e. future work).

2) Heterogeneous tuples

Fable had a comment about heterogeneous tuples, especially with paths in them, not being processed correctly. There were indeed some edge cases not covered correctly, so I wrote some additional unit tests for them: test_coercion_tuple_file_and_str and test_coercion_tuple_file_modes, that both succeed on master. The code on this PR was fixed so both tests now succeed here as well.

3) None parsing in a list/tuple

Fable pointed out at an issue in the code where None values in a list or tuple would bypass Pydantic validation. That didn't seem right, so I fixed the code and wrote up two tests to ensure this can't happen: test_list_rejects_none and test_tuple_rejects_none.

As a consequence, we should use just None for the default value of a tuple, rather than (None, None, None), which would now go through coercion. Cf. The edits in docs_src\multiple_values\options_with_multiple\values\tutorial001_*_py310.py.

x) Startup-cost structure

• needs numbers