Inability to specify EOF in TokenParsers.Parser

[peteraldous]

As discussed in #397 by @martingd, scanners use their atEnd predicate instead of reporting EofCh or the built-in Tokens.EOF:

Note about EofCh

First of all, it seems scala.util.parsing.combinator.lexical.Scanners.Scanner will never read past the end of the its in: Reader[Char] because it stops at in.atEnd so the token and whitespace parsers in scala.util.parsing.combinator.lexical.StdLexical will never encounter the EofCh character. However, we have left the mentions of EofCh in the token and whitespace parsers to avoid breaking something we might not fully understand.

The fixes implemented for strings and such work for those cases, but it seems like it's impossible for a combinator defined in a TokenParsers.Parser object to determine if its scanner has reached the end-of-file without having access to the Scanner instance; in particular, any combinator that asserts the presence of EOF will fail because the token can never emit.

It is, of course, possible that I'm using the library in a way different from how it was intended to be used. If it would be helpful, I can modify and publish my code (I'm working on a reference solution, which I don't want to post so my students can't plagiarize it). On that note, would you mind if I worked on some examples that could help with documentation? How stable is the API (#236, for example, suggested that there might be another way to represent data types)?

[peteraldous]

I see now that phrase() is intended to solve this problem. It's still fairly confusing that EOF is a thing. I'd like to see improvements to the API and/or documentation to clear this up and am willing to help but am unsure which direction would be best.