Replace the blib2to3 tokenizer with pytokens

[tusharsadhwani]

Description

Replaces black's tokenizer with a from-scratch rewrite done by me. We could vendor the code into black itself, but either pinning it or keeping it as-is would be my recommendation, the tokenizer can be used by multiple tools for perfect compatibility.

Resolves #4520
Resolves #970
Resolves #3700

Tests passing so far: 381/381 (!)

What's left

• ERRORTOKENs are not produced just yet. And there are no tests for ERRORTOKENs. Let's add those.
• Fix any leftover primer regressions.

[tusharsadhwani]

@JelleZijlstra with this, the test suite is fully passing. Primer is failing (mostly just because some file in hypothesis failed to parse), I'll be working on that, but the code should be good for a first review.

[MeGaGiGaGon]

I don't think we currently have any tests for this, but I just linked the above two issues here because they are the same bug in the parser where \rs cause issues. The most minimal reproduction is {\r}, and since this is a parser rewrite hopefully it can be solved. Note that this is currently only observable by directly calling internal methods due to how black reads input.

[tusharsadhwani]

Thanks for linking this, I'll make sure these parse identically to how CPython does it.

[tusharsadhwani]

Okay, primer is fixed, and all tests are green.