Parametric self and default arguments

[dvarrazzo]

Hello,

Psycopg objects are parametrized with a Row type, which is the type of object returned by the fetch*() methods. Normally it is TupleRow = Tuple[Any, ...], but it can be customized using a row_factory argument to the constructor method Connection.connect(). For instance, in order to return dicts instead of tuples, one can use Connection.connect(..., row_factory=dict_row). At the moment, the connect() signature, simplified, is:

class Connection(Generic[Row]):

    @overload
    @classmethod
    def connect(cls, conninfo: str, row_factory: RowFactory[Row]) -> "Connection[Row]":
        ...

    @overload
    @classmethod
    def connect(cls, conninfo: str) -> "Connection[TupleRow]":
        ...

    @classmethod
    def connect(cls, conninfo: str, row_factory: Optional[RowFactory[Row]] = None) -> "Connection[Any]":
        ...  # real implementation

Because now we need to subclass Connection, I would like to annotate connect() using a parametric Self and avoid the need to redeclare all of the above in each subclass (yes, I am aware of PEP 673, but I'm trying to make this work with the current mypy implementation).

I don't seem able to declare correctly the default TupleRow together with a parametric self. The current implementation I have attempted is, in a self-contained example:

from typing import Any, cast, Dict, Generic, Optional, Protocol, Sequence
from typing import Tuple, Type, TypeVar, overload

Row = TypeVar("Row", covariant=True)
TupleRow = Tuple[Any, ...]
DictRow = Dict[str, Any]

class RowMaker(Protocol[Row]):
    def __call__(self, __values: Sequence[Any]) -> Row:
        ...

class RowFactory(Protocol[Row]):
    def __call__(self) -> RowMaker[Row]:
        ...

def tuple_row() -> "RowMaker[TupleRow]":
    ...

def dict_row() -> "RowMaker[DictRow]":
    ...

class Connection(Generic[Row]):
    _Self = TypeVar("_Self", bound="Connection[Row]")

    @overload
    @classmethod
    def connect(cls: Type[_Self], row_factory: RowFactory[Row]) -> _Self:
        ...

    @overload
    @classmethod
    def connect(cls: Type[_Self]) -> _Self:
        ...

    @classmethod
    def connect(cls: Type[_Self], row_factory: Optional[RowFactory[Row]] = None) -> _Self:
        ...

c = Connection.connect()
reveal_type(c)

ct = Connection.connect(row_factory=tuple_row)
reveal_type(ct)

cd = Connection.connect(row_factory=dict_row)
reveal_type(cd)

With this implementation, connect() with no argument is reported by mypy 0.950 as needing annotation and of type Connection[Any]:

testmypy.py:47: error: Need type annotation for "c"  [var-annotated]
testmypy.py:48: note: Revealed type is "testmypy.Connection[Any]"
testmypy.py:51: note: Revealed type is "testmypy.Connection[builtins.tuple[Any, ...]]"
testmypy.py:54: note: Revealed type is "testmypy.Connection[builtins.dict[builtins.str, Any]]"

One way I wish it was possible to implement this would be with a parametrized self, aka returning _Self[Row] in the first variant and _Self[TupleRow] in the second. It doesn't seem possible because error: Type variable "_Self" used with arguments, which is legit. I have then tried to leave _Self generic, by defining it as _Self = TypeVar("_Self", bound="Connection"), but this gives fails with error: Missing type parameters for generic type "Connection".

An alternative attempt, which I think would be even better than using @overload, would be if the default value for row_factory was able to give information about the parameter type. However it doesn't seem to work either:

class ConnectionNoOverload(Generic[Row]):
    _Self = TypeVar("_Self", bound="ConnectionNoOverload[Row]")

    @classmethod
    def connect(cls: Type[_Self], row_factory: RowFactory[Row] = cast(RowFactory[TupleRow], tuple_row)) -> _Self:
        ...


cno = ConnectionNoOverload.connect()
reveal_type(cno)

cnot = ConnectionNoOverload.connect(row_factory=tuple_row)
reveal_type(cnot)

cnod = ConnectionNoOverload.connect(row_factory=dict_row)
reveal_type(cnod)

In this test, the row_factory default is also reported to be incompatible with the parameter type. cno, again, has parameter type Any:

testmypy.py:67: error: Incompatible default for argument "row_factory" (default has type "RowFactory[Tuple[Any, ...]]", argument has type "RowFactory[Row]")  [assignment]
testmypy.py:72: error: Need type annotation for "cno"  [var-annotated]
testmypy.py:73: note: Revealed type is "testmypy.ConnectionNoOverload[Any]"
testmypy.py:76: note: Revealed type is "testmypy.ConnectionNoOverload[builtins.tuple[Any, ...]]"
testmypy.py:79: note: Revealed type is "testmypy.ConnectionNoOverload[builtins.dict[builtins.str, Any]]"

My questions:

• can this be done with the current state of mypy and of type annotations?
• would it be a good idea if it was possible to do this?
• would this be possible when PEP 673 will be implemented in Mypy?

Thank you very much!

P.S. Original issue at psycopg/psycopg#308

[erictraut]

Parameterized type variables are not currently supported in the Python type system. You cannot use a generic type as a bound for a TypeVar. (Mypy doesn't currently emit an error if a generic type is used in a bound, but it should.) A parameterized TypeVar is known as a "higher-kinded type" (HKT). There have been some discussions about introducing HKTs into the Python type system, but these discussions haven't yet led to any concrete proposals.

The Self type introduced in PEP 673 is probably the best approach here. While it is not yet supported in mypy, it is supported in pyright and pyre if you want to give it a try.

Here's a solution that works with the current version of mypy. You can use an explicit specialization when calling the connect() class method with no arguments.

c = Connection[Any].connect()

[dvarrazzo]

This question is now 2Y old and the code illustrated is quite out of date compared to the state of the Python typing system. Closing it in favour of:

• Adding a limited form of parametrized Self #1555
• Inferring Generic parameters from function parameters defaults #1556