Support for recursive datatypes

[abhi-glitchhg]

since release of mypy 0.981 recursive types are supported; i have just removed the # as per suggestion in the TODO comment and have changed the mypy version in the ci jobs.

This PR is to check if ci is breaking with the changes are not. fixes #640

Let me know if any further changes need to be made
thanks..

[NivekT]

@ejguan Any issue form upgrading mypy to a newer version? Aside from fixing the lints?

[abhi-glitchhg]

Hmm, mypy is raising one error on this line

returning TypeVar should receive at least one argument containing the same
Typevar  [type-var]
Found 1 error in 1 file (checked 76 source files)
        def draw(self) -> T:

data/torchdata/datapipes/iter/util/randomsplitter.py

Lines 103 to 105 in b1b3406

	def draw(self) -> T:
	selected_key = self._rng.choices(self.keys, self.weights)[0]

How can i fix this? can anyone help me out here?

[NivekT]

Based on this issue, it seems to be warning that a single TypeVar will not do anything. I think you can either remove -> T or suppress the warning.

[facebook-github-bot]

@NivekT has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[ejguan]

Thank you for your contribution. The PR looks good. However, I am sorry that I have to block this PR for now until PyTorch Core has updated their pinned mypy version from 0.960 to 0.981.

The main problem is that torchdata is still partially in the tree of PyTorch. And PyTorch has strong opinion on the version of mypy. See: https://github.com/pytorch/pytorch/blob/6bb7f4f29f0a36ce410ce53d824f531eaf74c76e/mypy.ini#L5
If we make this PR landed, in order to develop torchdata without mypy complain on my local environment, I have to update mypy to 0.981. However, if I need to work on something in PyTorch Core, the updated mypy will prevent me running static type checking on PyTorch.

IMHO, to land this PR, we accomplish one of the following 2 options:

• Make TorchData fully out of PyTorch
• Update PyTorch's mypy requirement

LMK if it makes sense. And, please correct me if anything wrong or we have other options.

[abhi-glitchhg]

hello @ejguan! Thanks for the feedback.

Make TorchData fully out of PyTorch

Honestly, I do not know the consequences and the amount of work required for this but the second option seems easier than the first.

Anyway, there is no hurry from my side; I just noticed the TODO issue and was aware of the recent mypy version release.

We could try updating the pytorch's mypy version. Are there any existing blockers in doing this that you are aware of?

Would love to hear your thoughts.
Thanks!

[abhi-glitchhg]

I would also like to point out that, --install-types --non-interactive will install the missing package stubs hence slower.

Quoting relevant part from the above documentation link :- This is somewhat slower than explicitly installing stubs before running mypy, since it may type check your code twice – the first time to find the missing stubs, and the second time to type check your code properly after mypy has installed the stubs.

I don't think downloading the package stubs would be a bottleneck; but if it is, then we could think of using cached installations. (just a suggestion)

[ejguan]

Thank you for providing the idea. I think it's fine for now as torchdata is still a small package. We might want to investigate if we have noticed significant overhead.

[ejguan]

BTW, regarding the recursive type hints, 0.981 is not the best version as it's an experimental feature. 0.990 is the version that officially supports recursive type hints.

[ejguan]

Let's wait until PyTorch has officially updated mypy

[abhi-glitchhg]

BTW, regarding the recursive type hints, 0.981 is not the best version as it's an experimental feature. 0.990 is the version that officially supports recursive type hints.

I was just going to mention this.