Skip to content

Possible option: use ParamSpec for lru_cache#6986

Closed
sobolevn wants to merge 2 commits intopython:masterfrom
sobolevn:patch-98
Closed

Possible option: use ParamSpec for lru_cache#6986
sobolevn wants to merge 2 commits intopython:masterfrom
sobolevn:patch-98

Conversation

@sobolevn
Copy link
Member

@sobolevn sobolevn commented Jan 21, 2022 •
edited
Loading

This one might be controversal. Why?

  • When adding ParamSpec we cannot ensure that args are Hashable, but signature remains the same
  • When not using ParamSpec, we can ensure that args are Hashable, but signature changes to *Any, **Any

What do others think? In my opinion better signature is more important 🤔

Local testing with the latest mypy shows that my version works fine:

from typing import TypeVar, Callable, ParamSpec, Generic

_T = TypeVar("_T")
_P = ParamSpec("_P")

class _lru_cache_wrapper(Generic[_P, _T]):
    __wrapped__: Callable[_P, _T]
    def __call__(self, *args: _P.args, **kwargs: _P.kwargs) -> _T: ...

def lru_cache(maxsize: Callable[_P, _T]) -> _lru_cache_wrapper[_P, _T]:
    pass

@lru_cache
def some(arg: int) -> str:
    pass

reveal_type(some)  # N: Revealed type is "ex._lru_cache_wrapper[def (arg: builtins.int), builtins.str*]"

@sobolevn
Copy link
Member Author

sobolevn commented Jan 21, 2022 •
edited
Loading

This looks like a mypy bug:

 stdlib/functools.pyi:27: error: Argument 1 to "final" has incompatible type "Type[_lru_cache_wrapper[Any, Any]]"; expected "Type[_lru_cache_wrapper[Any, Any]]"  [arg-type]

Update

Yes, I can reproduce it locally. Going to report to mypy (and probably fix it).

@sobolevn sobolevn mentioned this pull request Jan 21, 2022
Closed
@github-actions

This comment has been minimized.

Sign in to view
@sobolevn
Copy link
Member Author

sobolevn commented Jan 21, 2022 •
edited
Loading

Looks like we have another problem, here's an example from flake8:

@functools.lru_cache(maxsize=None)
def style_guide_for(self, filename: str) -> "StyleGuide": pass

It is then used as: guide = self.style_guide_for(filename)
So, it reports:

+ src/flake8/style_guide.py:275: error: Missing positional argument "filename" in call to "__call__" of "_lru_cache_wrapper"
+ src/flake8/style_guide.py:275: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "StyleGuideManager"

I guess that self arg breaks it 😒

@AlexWaygood
Copy link
Member

AlexWaygood commented Jan 21, 2022

We had a looong discussion about how to type lru_cache over in #6347, but didn't really come to a conclusion ://

@sobolevn
Copy link
Member Author

sobolevn commented Jan 21, 2022

Going to try @staticmethod approach!

AlexWaygood
AlexWaygood reviewed Jan 21, 2022
View reviewed changes
stdlib/functools.pyi
def lru_cache(maxsize: int | None = ..., typed: bool = ...) -> Callable[[Callable[_P, _T]], _lru_cache_wrapper[_P, _T]]: ...
@overload
def lru_cache(maxsize: Callable[..., _T], typed: bool = ...) -> _lru_cache_wrapper[_T]: ...
def lru_cache(maxsize: Callable[_P, _T]) -> _lru_cache_wrapper[_P, _T]: ...
Copy link
Member

@AlexWaygood AlexWaygood Jan 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why'd the typed arg get deleted? :)

Copy link
Member Author

@sobolevn sobolevn Jan 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeap, my bad. It is allowed: https://github.com/python/cpython/blob/54610bb448a9cf5be77d53b66169fca4c11be6cb/Lib/functools.py#L508

@github-actions
Copy link
Contributor

github-actions bot commented Jan 21, 2022

Diff from mypy_primer, showing the effect of this PR on open source code:

rich (https://github.com/willmcgugan/rich)
+ rich/theme.py:25: error: Missing positional argument "style_definition" in call to "__call__" of "_lru_cache_wrapper"
+ rich/theme.py:25: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[Style]"
+ rich/theme.py:54: error: Missing positional argument "style_definition" in call to "__call__" of "_lru_cache_wrapper"
+ rich/theme.py:54: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[Style]"
+ rich/style.py:148: error: Missing positional argument "color" in call to "__call__" of "_lru_cache_wrapper"
+ rich/style.py:148: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[Color]"
+ rich/style.py:393: error: Missing positional argument "system" in call to "__call__" of "_lru_cache_wrapper"
+ rich/style.py:393: error: Missing positional argument "self" in call to "__call__" of "_lru_cache_wrapper"
+ rich/style.py:393: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "ColorSystem"; expected "Color"
+ rich/style.py:396: error: Missing positional argument "system" in call to "__call__" of "_lru_cache_wrapper"
+ rich/style.py:396: error: Missing positional argument "self" in call to "__call__" of "_lru_cache_wrapper"
+ rich/style.py:396: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "ColorSystem"; expected "Color"
+ rich/style.py:416: error: Missing positional argument "style_definition" in call to "__call__" of "_lru_cache_wrapper"
+ rich/style.py:416: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[Style]"
+ rich/style.py:542: error: Missing positional argument "color" in call to "__call__" of "_lru_cache_wrapper"
+ rich/style.py:542: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[Color]"
+ rich/style.py:569: error: Missing positional argument "color" in call to "__call__" of "_lru_cache_wrapper"
+ rich/style.py:569: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[Color]"
+ rich/segment.py:158: error: Missing positional argument "cut" in call to "__call__" of "_lru_cache_wrapper"
+ rich/segment.py:158: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "Segment"; expected "Type[Segment]"
+ rich/segment.py:158: error: Argument 2 to "__call__" of "_lru_cache_wrapper" has incompatible type "int"; expected "Segment"
+ rich/markup.py:155: error: Missing positional argument "style" in call to "__call__" of "_lru_cache_wrapper"
+ rich/markup.py:155: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[Style]"
+ rich/markup.py:212: error: Missing positional argument "style" in call to "__call__" of "_lru_cache_wrapper"
+ rich/markup.py:212: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[Style]"
+ rich/jupyter.py:63: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "TerminalTheme"; expected "Style"
+ rich/console.py:1393: error: Missing positional argument "style_definition" in call to "__call__" of "_lru_cache_wrapper"
+ rich/console.py:1393: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[Style]"
+ rich/console.py:2089: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "TerminalTheme"; expected "Style"
+ rich/console.py:2101: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "TerminalTheme"; expected "Style"
+ rich/color.py:513: error: Missing positional argument "color" in call to "__call__" of "_lru_cache_wrapper"
+ rich/color.py:513: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "ColorTriplet"; expected "Palette"
+ rich/color.py:526: error: Missing positional argument "color" in call to "__call__" of "_lru_cache_wrapper"
+ rich/color.py:526: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "ColorTriplet"; expected "Palette"
+ rich/ansi.py:163: error: Missing positional argument "style_definition" in call to "__call__" of "_lru_cache_wrapper"
+ rich/ansi.py:163: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[Style]"
+ rich/progress_bar.py:142: error: Missing positional argument "no_color" in call to "__call__" of "_lru_cache_wrapper"
+ rich/progress_bar.py:143: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "Style"; expected "ProgressBar"
+ rich/progress_bar.py:143: error: Argument 3 to "__call__" of "_lru_cache_wrapper" has incompatible type "Optional[str]"; expected "Style"
+ rich/progress_bar.py:143: error: Argument 4 to "__call__" of "_lru_cache_wrapper" has incompatible type "bool"; expected "str"

flake8 (https://github.com/pycqa/flake8)
+ src/flake8/style_guide.py:275: error: Missing positional argument "filename" in call to "__call__" of "_lru_cache_wrapper"
+ src/flake8/style_guide.py:275: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "StyleGuideManager"
+ src/flake8/style_guide.py:308: error: Missing positional argument "filename" in call to "__call__" of "_lru_cache_wrapper"
+ src/flake8/style_guide.py:308: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "StyleGuideManager"
+ tests/unit/test_style_guide.py:144: error: Missing positional argument "filename" in call to "__call__" of "_lru_cache_wrapper"

dedupe (https://github.com/dedupeio/dedupe)
+ dedupe/training.py:93: error: Missing positional argument "iterable" in call to "__call__" of "_lru_cache_wrapper"
+ dedupe/training.py:93: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "FrozenSet[int]"; expected "Resampler"

manticore (https://github.com/trailofbits/manticore)
+ manticore/core/smtlib/solver.py:505: error: Signature of "can_be_true" incompatible with supertype "Solver"
+ manticore/core/smtlib/solver.py:605: error: Signature of "get_all_values" incompatible with supertype "Solver"

isort (https://github.com/pycqa/isort)
+ isort/settings.py:335: error: Argument 2 to "__call__" of "_lru_cache_wrapper" has incompatible type "Tuple[str, ...]"; expected "Tuple[str]"
+ isort/settings.py:769: error: Argument 2 to "__call__" of "_lru_cache_wrapper" has incompatible type "Tuple[str, ...]"; expected "Tuple[str]"
+ isort/settings.py:807: error: Argument 2 to "__call__" of "_lru_cache_wrapper" has incompatible type "Tuple[str, ...]"; expected "Tuple[str]"
+ isort/deprecated/finders.py:302: error: Missing positional argument "path" in call to "__call__" of "_lru_cache_wrapper"
+ isort/deprecated/finders.py:302: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[RequirementsFinder]"
+ isort/deprecated/finders.py:333: error: Missing positional argument "path" in call to "__call__" of "_lru_cache_wrapper"
+ isort/deprecated/finders.py:333: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[RequirementsFinder]"

edgedb (https://github.com/edgedb/edgedb)
+ edb/schema/expr.py:258:18: error: Missing positional argument "data" in call to "__call__" of "_lru_cache_wrapper"
+ edb/schema/expr.py:258:53: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "Tuple[str, Union[Tuple[type, ...], type, None], Tuple[UUID, ...], Tuple[Tuple[str, Any], ...]]"; expected "Type[ObjectCollection[Object_T]]"
+ edb/schema/schema.py:1146:16: error: Missing positional argument "stype" in call to "__call__" of "_lru_cache_wrapper"
+ edb/schema/schema.py:1146:32: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "Type"; expected "FlatSchema"
+ edb/schema/schema.py:1156:16: error: Missing positional argument "stype" in call to "__call__" of "_lru_cache_wrapper"
+ edb/schema/schema.py:1156:32: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "Type"; expected "FlatSchema"
+ edb/schema/schema.py:1166:16: error: Missing positional argument "scls" in call to "__call__" of "_lru_cache_wrapper"
+ edb/schema/schema.py:1167:13: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "Object"; expected "FlatSchema"
+ edb/schema/schema.py:1208:5: error: Signature of "get_referrers_ex" incompatible with supertype "Schema"
+ edb/schema/schema.py:1254:19: error: Missing positional argument "data" in call to "__call__" of "_lru_cache_wrapper"
+ edb/schema/schema.py:1254:44: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "Tuple[str, UUID]"; expected "Type[Object]"
+ edb/schema/schema.py:1791:16: error: Missing positional argument "scls" in call to "__call__" of "_lru_cache_wrapper"
+ edb/schema/schema.py:1791:51: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "Object"; expected "FlatSchema"
+ edb/schema/schema.py:1792:15: error: Missing positional argument "scls" in call to "__call__" of "_lru_cache_wrapper"
+ edb/schema/schema.py:1792:49: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "Object"; expected "FlatSchema"
+ edb/schema/schema.py:1793:14: error: Missing positional argument "scls" in call to "__call__" of "_lru_cache_wrapper"
+ edb/schema/schema.py:1793:51: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "Object"; expected "FlatSchema"
+ edb/schema/scalars.py:132:55: error: Argument 2 to "__call__" of "_lru_cache_wrapper" has incompatible type "Object"; expected "Type"
+ edb/schema/scalars.py:132:61: error: Argument 3 to "__call__" of "_lru_cache_wrapper" has incompatible type "Object"; expected "Type"
+ edb/schema/delta.py:1860:19: error: Missing positional argument "field_name" in call to "__call__" of "_lru_cache_wrapper"
+ edb/schema/delta.py:1860:51: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[ObjectCommand[so.Object_T]]"
+ edb/schema/delta.py:3272:53: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "Name"; expected "QualName"
+ edb/schema/indexes.py:74:40: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "Name"; expected "QualName"

pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/dtypes/cast.py:514: error: Unused "type: ignore" comment
+ pandas/core/groupby/ops.py:199: error: Missing positional argument "is_numeric" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ pandas/core/groupby/ops.py:199: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "Type[WrappedCythonOp]"  [arg-type]
+ pandas/core/groupby/ops.py:199: error: Argument 4 to "__call__" of "_lru_cache_wrapper" has incompatible type "bool"; expected "dtype[Any]"  [arg-type]

mypy (https://github.com/python/mypy)
+ mypy/find_sources.py:161: error: Missing positional argument "dir" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ mypy/find_sources.py:161: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "SourceFinder"  [arg-type]
+ mypy/find_sources.py:203: error: Missing positional argument "dir" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ mypy/find_sources.py:203: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "SourceFinder"  [arg-type]

core (https://github.com/home-assistant/core)
+ homeassistant/components/mqtt/__init__.py:1069: error: Missing positional argument "topic" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]

paroxython (https://github.com/laowantong/paroxython)
+ paroxython/assess_costs.py:128: error: Missing positional argument "taxon" in call to "__call__" of "_lru_cache_wrapper"
+ paroxython/assess_costs.py:128: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "TaxonName"; expected "LearningCostAssessor"
+ paroxython/recommend_programs.py:290: error: Missing positional argument "taxon" in call to "__call__" of "_lru_cache_wrapper"
+ paroxython/recommend_programs.py:290: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "TaxonName"; expected "LearningCostAssessor"
+ paroxython/map_taxonomy.py:243: error: Missing positional argument "label_name" in call to "__call__" of "_lru_cache_wrapper"
+ paroxython/map_taxonomy.py:243: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "LabelName"; expected "Taxonomy"

pydantic (https://github.com/samuelcolvin/pydantic)
+ pydantic/tools.py:37: error: Unused "type: ignore" comment

pip (https://github.com/pypa/pip)
+ src/pip/_internal/req/req_install.py:489: error: Missing positional argument "self" in call to "__call__" of "_lru_cache_wrapper"
+ src/pip/_internal/req/req_install.py:514: error: Missing positional argument "self" in call to "__call__" of "_lru_cache_wrapper"
+ src/pip/_internal/index/package_finder.py:863: error: Missing positional argument "project_name" in call to "__call__" of "_lru_cache_wrapper"
+ src/pip/_internal/index/package_finder.py:863: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "PackageFinder"
+ src/pip/_internal/index/package_finder.py:881: error: Missing positional argument "project_name" in call to "__call__" of "_lru_cache_wrapper"
+ src/pip/_internal/index/package_finder.py:882: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "PackageFinder"
+ src/pip/_internal/self_outdated_check.py:145: error: Missing positional argument "project_name" in call to "__call__" of "_lru_cache_wrapper"
+ src/pip/_internal/self_outdated_check.py:145: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "PackageFinder"
+ src/pip/_internal/wheel_builder.py:75: error: Missing positional argument "self" in call to "__call__" of "_lru_cache_wrapper"
+ src/pip/_internal/distributions/sdist.py:100: error: Missing positional argument "self" in call to "__call__" of "_lru_cache_wrapper"
+ src/pip/_internal/resolution/resolvelib/factory.py:269: error: Missing positional argument "self" in call to "__call__" of "_lru_cache_wrapper"
+ src/pip/_internal/resolution/resolvelib/factory.py:581: error: Missing positional argument "project_name" in call to "__call__" of "_lru_cache_wrapper"
+ src/pip/_internal/resolution/resolvelib/factory.py:581: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "NormalizedName"; expected "PackageFinder"
+ src/pip/_internal/commands/list.py:231: error: Missing positional argument "project_name" in call to "__call__" of "_lru_cache_wrapper"
+ src/pip/_internal/commands/list.py:231: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "NormalizedName"; expected "PackageFinder"
+ src/pip/_internal/commands/index.py:118: error: Missing positional argument "project_name" in call to "__call__" of "_lru_cache_wrapper"

pylint (https://github.com/pycqa/pylint)
+ pylint/utils/file_state.py:100: error: Missing positional argument "msgid_or_symbol" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ pylint/utils/file_state.py:100: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "MessageDefinitionStore"  [arg-type]
+ pylint/message/message_definition_store.py:69: error: Missing positional argument "msgid_or_symbol" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ pylint/message/message_definition_store.py:69: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "MessageDefinitionStore"  [arg-type]
+ pylint/message/message_definition_store.py:78: error: Missing positional argument "msgid_or_symbol" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ pylint/message/message_definition_store.py:79: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "MessageDefinitionStore"  [arg-type]

@sobolevn
Copy link
Member Author

sobolevn commented Jan 21, 2022 •
edited
Loading

Ok, it didn't work out. I am going to close it and at least wait for python/mypy#12033 to be merged and Concatenate to be supported.

@sobolevn sobolevn closed this Jan 21, 2022
@tamird
Copy link
Contributor

tamird commented Feb 23, 2024

@sobolevn is it time to revisit this?

@sobolevn
Copy link
Member Author

sobolevn commented Feb 23, 2024

@tamird please, go ahead and try!

@tamird
Copy link
Contributor

tamird commented Feb 23, 2024

I'm out of my depth on trying, just another frustrated user suffering bad typings :(

@sobolevn
Copy link
Member Author

sobolevn commented Feb 23, 2024

Looking at some recent discussion, this seems to be rather hard / impossible right now: #10523

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@AlexWaygood AlexWaygood AlexWaygood left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@sobolevn @AlexWaygood @tamird