diff --git a/conformance/README.md b/conformance/README.md index f4510de2..f8cb346e 100644 --- a/conformance/README.md +++ b/conformance/README.md @@ -80,7 +80,7 @@ Note that some type checkers may not run on some platforms. If a type checker fa Different type checkers report errors in different ways (with different wording in error messages and different line numbers or character ranges for errors). This variation makes it difficult to fully automate test validation given that tests will want to check for both false positive and false negative type errors. Some level of manual inspection will therefore be needed to determine whether a type checker is fully conformant with all tests in any given test file. This "scoring" process is required only when the output of a test changes — e.g. when a new version of that type checker is released and the tests are rerun. We assume that the output of a type checker will be the same from one run to the next unless/until a new version is released that fixes or introduces a bug. In this case, the output will need to be manually inspected and the conformance results re-scored for those tests whose output has changed. -Conformance results are reported and summarized for each supported type checker. Currently, results are reported for mypy, pyrefly, pyright, and zuban. It is the goal and desire to add additional type checkers over time. +Conformance results are reported and summarized for each supported type checker. Currently, results are reported for mypy, pyrefly, pyright, zuban and ty. It is the goal and desire to add additional type checkers over time. ## Adding a New Test Case diff --git a/conformance/pyproject.toml b/conformance/pyproject.toml index 9b511933..7483444b 100644 --- a/conformance/pyproject.toml +++ b/conformance/pyproject.toml @@ -9,6 +9,7 @@ dependencies = [ "tomli", "tomlkit", "zuban", + "ty", ] [tool.uv] diff --git a/conformance/results/results.html b/conformance/results/results.html index 058d7e55..f6f30a4f 100644 --- a/conformance/results/results.html +++ b/conformance/results/results.html @@ -180,8 +180,10 @@
Does not report error for a forward reference that is not enclosed in quotes.
Does not report error for use of quoted type with "|" operator (runtime error).
Incorrectly generates error for quoted type defined in class scope.
Incorrectly generates error for quoted type defined in class scope.
Types in quotes incorrectly refer to shadowing class member.
Does not reject some type forms that require quotes.
Does not detect runtime errors from partially stringified PEP-604 unions.
Resolves references in type annotations as referring to end-of-scope types (https://discuss.python.org/t/annotation-string-references-in-class-scope-in-conformance-tests/105439, https://github.com/python/typing/pull/2144)
Does not report incompatible Generator type in `yield from` statement.
Does not detect that invalid yield is unreachable
Type evaluation differs from other type checkers because of ambiguity in the spec related to method bindings.
Type evaluation differs from other type checkers because of ambiguity in the spec related to method bindings.
Does not treat `type` same as `type[Any]` for assert_type.
Does not allow access to unknown attributes from object of type `type[Any]`.
Allows arbitrary attributes to be accessed on `TA` where `TA = typing.Type[typing.Any]` or `TA = type[typing.Any]`.
Treats `type` equivalently to `type[object]` rather than `type[typing.Any]`.
Incorrect rejects + between two AnyStr
Constrained type var resolves to subtype instead of explcitly listed constraint
Incorrectly allows constrained type variables to be solved to a union of their constraints.
Does not forbid a `TypeVar` immediately following a `TypeVarTuple` in a parameter list from having a default.
Does not support `TypeVarTuple`.
Does not fully support defaults for `ParamSpec`s.
Does not reject subscription of an already-specialized generic class.
Does not reject using a bare `ParamSpec` as a type alias value.
Does not support `Concatenate`.
Does not reject usage of args/kwargs for out-of-scope ParamSpec
Incorrectly allows using `*args: P.args` and `**kwargs: P.kwargs` when `P` has not been put into scope by any other parameter annotation or enclosing scope.
Does not support `Concatenate`.
Constraint solver doesn't find common type for two signatures captured by a single ParamSpec (allowed).
Does not support `Concatenate`.
Does not implement several scoping checks/restrictions for generics
Does not reject `list[T]()` in the global scope, where `T` is an unbound type variable.
Does nto reject `alias: TypeAlias = list[T]` in the body scope of a class generic over a type variable `T`.
Does not infer the type of an unannotated `self` parameter to be type `Self`.
Does not retain `Self` when calling method that returns `Self`.
Does not infer the type of an unannotated `cls` parameter to be type `type[Self]`.
Does not retain `Self` when accessing attribute through `type[Self]`.
Doesn't allow accessing `Self` in a classmethod
Treats attributes not initialized on the class as instance-only
Return annotation of Self allows returning the concrete instance of the current class.
Does not implement some restrictions on where Self can be used
Does not reject `Self` used in a return annotation when `self` is annotated using another type variable.
Does not reject `Self` used in staticmethods or metaclasses.
Type parameter syntax not yet supported.
Does not following runtime scoping rules for type parameters in all cases.
Infers Node[Never] instead of Node[Any] when argument is not provided.
False negative on instance attribute access on type(node).
Does not reject access of generic instance variable from the class object.
Does not enforce that tuples captured by TypeVarTuple are same type.
Supports PEP-646 unpacked tuples but not TypeVarTuple.
Does not enforce that tuples captured by TypeVarTuple are same length.
Does not enforce that tuples captured by TypeVarTuple are same type.
TypeVarTuple is pinned too early when calling generic function
Incorrectly specializes generic alias that includes a TypeVar and TypeVarTuple if no type arguments are provided.
Rejects specialization of generic type alias defined as a tuple containing a TypeVar.
Sometimes specializes to tuple[Any, ...] instead of empty tuple
Does not reject use of type variable within an upper bound.
Cannot find a common supertype of `list[int]` and `set[int]` in order to solve a type variable bound to `Sized`.
Does not reject use of class-scoped TypeVar used in a base class when variance is incompatible.
Allows Annotated in some contexts where it should not be allowed
Does not treat use of Final name as if it was replaced by the literal in NamedTuple definition.
Does not allow conditional assignment of Final instance variable in __init__ method.
Does not allow redefinition of private class variable that is marked Final in parent class.
Does not report modification of local Final variable via "for" statement.
Final attributes not initialized on the class can be assigned to
Does not forbid `Final`-annotated assignments to attributes on `self` in non-`__init__` methods.
Does not flag modifications of `Final`-annotated attributes via augmented assignment.
`Concatenate` in type aliases is currently unsupported
Does not reject invalid syntax in implicit type aliases.
Does not reject variables with `Any` or `Unknown` types when used as implicit type aliases.
Does not support `Concatenate` in type aliases.
`NewType`s are incorrectly considered to be classes.
Does not reject type alias defined in function scope.
Does not reject use of the `type` statement inside functions.
Does not reject circular definitions of type aliases.
Does not reject redeclarations of type aliases.
Does not support `type` statements generic over `TypeVarTuple`s.
Incorrectly rejects some recursive type aliases using TypeAliasType.
Incorrectly rejects the use of a class-scoped TypeVar in a TypeAliasType definition.
Does not reject specializing a type parameter in a generic type alias with a type inconsistent with the parameter's upper bound.
Does not reject declaring a type alias with a type variable that is not in scope.
Does not reject declaring a type alias with a non-literal tuple passed to the `type_params` parameter.
Does not reject cyclically defined type aliases.
Deliberately does not allow `str` to be narrowed to literal string types through equality or containment checks due to the possibility of `str` subclasses that could have unexpected equality semantics.
Support for `LiteralString` is not implemented.
Does not reject tuple within Literal.
`type[Proto]` is not yet supported.
`ClassVar` protocol members are not yet supported.
`@property` protocol members are only partially supported.
A class object `C` is only considered to inhabit a protocol type with a method member `f` if `f` exists as an attribute on the metaclass of `C`.
Does not detect protocol mismatch if concrete method is missing annotations.
Does not detect protocol mismatch if concrete method's parameters are position-only.
Does not reject implicit instance attributes in `Protocol` methods.
Does not support `ClassVar` protocol members.
Incorrectly considers `ClassVar` attributes on concrete classes as satisfying non-`ClassVar` attribute members on protocols.
Only has partial support for `@property` protocol members.
Does not report unimplemented attributes for class that explicitly derives from protocol until it is instantiated.
Allows implicitly abstract protocol methods to be called via `super()` on a protocol subclass.
Allows instantiation of abstract subclasses of protocol classes.
Only partially supports `@property` protocol members.
Does not reject attempted instantiation of abstract subclasses of protocols.
Fails one subtyping example of protocol modules
Never considers a module as satisfying a protocol with a method member due to the fact that the method will never exist on the class `types.ModuleType`, only on a given specific module instance.
Fails to solve a type variable involving a recursive generic protocol.
Does not report unsafe overlap for runtime_checkable protocol.
Does not reject `isinstance()` or `issubclass()` calls against runtime-checkable protocols where there is an unsafe overlap between the type of the first argument and the protocol.
Parameter names are lost when resolving ParamSpec
Does not support `Concatenate`.
Infers a callback protocol as being a gradual type if the callback has signature `__call__[T](self, *args: T, **kwargs: T)` and `T` has been explicitly specialized to `Any`.
Does not infer a callback protocol as being a gradual type if the callback has signature `__call__(self, a: int, /, *args: Any, **kwargs: Any)`.
Allows callable without kwargs to be assigned to callable with unpacked kwargs
Does not reject invalid argument types to an inherited constructor in a specialized subclass of a generic superclass.
Does not reject class-scoped type variables used in the `self` annotation.
Does not support inferring type variables for generic classes where the `__init__` method uses method-scoped type variables.
Does not honor metaclass __call__ method when evaluating constructor call.
Does not skip evaluation of __new__ and __init__ if custom metaclass call returns non-class.
Does not support __new__ return type that is not a subclass of the class being constructed.
Does not skip evaluation of __init__ based on __new__ return type.
Does not report errors during binding to cls parameter of __new__ method.
Does not validate call to custom metaclass __call__ method through type[T].
Does not support metaclass `__call__`.
Has overly lenient handling of calls to `type[T]` if `T` is a type variable without an upper bound.
Does not generate a union type for __new__ and __init__ when converting class to callable.
Does not ignore __init__ based on __new__ return type when converting class to callable.
Does not support __new__ return type that is different from class being constructed.
Converting constructor to callable does not preserve class-scoped type params.
Converting constructor to callable does not substitute Self in __new__
Converting constructor to callable uses __new__ signature instead of __init__
Does not include `__init__` when `__new__` returns `Self`.
Does not respect `NoReturn` return type on metaclass `__call__`.
Does not ignore `__init__` when `__new__` returns `Any`.
Unions overload return types.
Does not report inconsistency between __new__ and __init__ (optional).
Allows @override to be on all overloads and implementation, instead of just implementation.
Allows @override to appear in a stub file not on the first overload.
Does not expand boolean arguments to Literal[True] and Literal[False].
Does not expand enum arguments to literal variants.
Does not expand tuple arguments to possible combinations.
Does not evaluate Any in some cases where overload is ambiguous.
Evaluates Any in some cases where overload is not ambiguous.
Does not evaluate Any in some cases where overload is ambiguous.
Some error suppressing context managers are not detected
* Assumes descriptor behavior only when field is assigned in class body
* Doesn't allow non-data descriptors or data descriptors with differing `__get__` and `__set__` types
Only infers a descriptor `__get__` method as being called when a descriptor attribute is accessed on an instance if the descriptor attribute is present in the class namespace.
Wrongly requires a Final dataclass field to be initialized at class level.
Doesn't support Final nested inside ClassVar.
Does not synthesize `__hash__ = None` as a class attribute for unhashable dataclasses.
Does not report when an unhashable dataclass has `__hash__` called directly on an instance.
Does not report when dataclass is not compatible with Hashable protocol.
Does not synthesize a `__hash__ = None` class attribute for unhashable dataclasses.
Understands the `Hashable` protocol as equivalent to `object`.
Currently only enforces the Liskov Substitution Principle for methods.
Infers `__match_args__` as having type `tuple[()] | Unknown` for `__match_args__ = ()`
Does not reject write to instance variable that is not defined in __slots__.
__slots__ is generated but not checked during attribute assignment
Synthesizes a `__slots__` attribute but does not validate attribute assignments against `__slots__`.
Converter parameter not yet supported.
Does not properly handle field constructor that has default value for `kw_only` or `init` parameter.
Does not handle `kw_only=False` override when `kw_only_default=True`.
Does not report error when `order=False` and comparison operators are used.
Does not detect unannotated usage of `dataclasses.field()`.
Not supported.
Does not validate overrides of `TypedDict` fields in subclasses.
Supports `ReadOnly`, but not the functional syntax for `TypedDict`s currently, leading to one assertion failing.
Incorrectly rejects non-ReadOnly override of ReadOnly item.
Incorrectly rejects override of ReadOnly item with another ReadOnly item with narrower type.
Incorrectly rejects override of NotRequired ReadOnly item with a Required ReadOnly item.
Does not validate overrides of `TypedDict` keys.
Incorrectly allows update of ReadOnly item.
Incorrectly rejects update involving an item with Never type.
Incorrectly marks a match case as unreachable.
"More than one unpack" error is missing in some cases.
Permits subclasses of `NamedTuple` classes to override read-only attributes in the class body of the subclass.
Does not reject attempt to delete named tuple field by name.
Does not detect runtime errors from attempting to delete namedtuple attributes.
Improperly applies narrowing to Flag subclass.
Does not support `enum.Flag`.
Does not support special-cased handling of member name literal types in some cases (optional).
Does not enforce declared type of `_value_`.
Does not enforce assigned tuple types for enum members (optional).
Does not treat attribute with annotation and no assignment as non-member.
Does not treat callables as non-members.
Does not honor `enum.member` as method decorator.
Does not properly handle aliased enum members.
Does not support `_ignore_` mechanism (optional).
Does not treat attributes with private names as non-members.
Does not support `_ignore_` mechanism (optional).
Intersects the pre-existing type with the top materialization of the bracketed type rather than the bracketed type itself.
Does not detect calls to deprecated overloads.
Does not detect implicit calls to deprecated dunder methods, for example via operators.
Does not detect accesses of, or attempts to set, deprecated properties.
Does not honor `@no_type_check` class decorator (allowed).
Does not reject invalid call of `@no_type_check` function.
Does not honor `@no_type_check` class decorator (allowed).
Attempts to detect some errors even in blocks it determines to be unreachable, including in `if not TYPE_CHECKING` blocks.
Does not honor "# type: ignore" comment if comment includes additional text.
Does not honor "# type: ignore" comment if comment includes additional text.
Does not understand three-element form of sys.version checks.
Does not understand os.name checks.