✨ Upgrade autocompletion functionality for compatibility with shell_complete

[bckohan]

Cf issue: #949

Note, this includes (and supersedes) the changes from #974 and implements these suggestions #949 (comment), with a few differences.

I attempted to push these changes to #974 but was asked to open a new PR here instead.

This PR does 3 things:

1. Allows autocompletion functions to accept a click.core.Parameter argument. I think this is necessary because without it you can't define generic completer functions that don't know what CLI options or arguments they may be attached to and that need to alter their behavior based on how click has processed the arguments before the incomplete argument. Concrete example: you have an option or argument that accepts multiple values but you do not wish those values to be repeated.

2. Allows CompletionItems to be returned in addition to strings and 2-tuples from autocompletion functions.

3. Fixes arg: List[str] parameter to be passed what it used to be given in click 7 (i.e. the raw command line string minus the script and incomplete string). In click 7 this parameter was being set to the arguments passed in the environment variables via the completion scripts installed on the system that are invoked when you hit tab. These parameters do not show up in ctx.args or sys.argv. Click has a general purpose obj attribute on the context that is available for downstream use. This PR sets that object to a dictionary in the completer classes and attaches the args from the environment variables to it. If obj is already defined it'll just not do that and nothing will break accept the args will be passed as an empty list to the autocompletion functions - which is what the original PR was doing anyway even though the tutorial docs said otherwise.

I think numbers 1 and 2 are important. Number 3 is mostly for backwards compatibility but I don't think its that critical - I just wanted to show that there is a way to do it using public Click 8 interfaces.

A reusable completer function tutorial has been added to the autocompletion tutorials

[github-actions]

📝 Docs preview for commit 249b70e at: https://6ae37f21.typertiangolo.pages.dev

Modified Pages

• https://6ae37f21.typertiangolo.pages.dev/contributing/
• https://6ae37f21.typertiangolo.pages.dev/tutorial/options-autocompletion/

[github-actions]

📝 Docs preview for commit c5aab0a at: https://9a5f770f.typertiangolo.pages.dev

Modified Pages

• https://9a5f770f.typertiangolo.pages.dev/contributing/
• https://9a5f770f.typertiangolo.pages.dev/tutorial/options-autocompletion/

[svlandeg]

Thanks for the PR! We'll get this reviewed and get back to you 🙏

[github-actions]

📝 Docs preview for commit 340c459 at: https://985aa99d.typertiangolo.pages.dev

Modified Pages

• https://985aa99d.typertiangolo.pages.dev/contributing/
• https://985aa99d.typertiangolo.pages.dev/tutorial/options-autocompletion/

[svlandeg]

@bckohan: the tests are failing because there isn't a 100% code coverage with the new commits. Could you look into this? 🙏

[github-actions]

📝 Docs preview for commit 26581c0 at: https://9afeb588.typertiangolo.pages.dev

Modified Pages

• https://9afeb588.typertiangolo.pages.dev/contributing/
• https://9afeb588.typertiangolo.pages.dev/tutorial/options-autocompletion/

[github-actions]

📝 Docs preview for commit 1037702 at: https://c58d58a9.typertiangolo.pages.dev

Modified Pages

• https://c58d58a9.typertiangolo.pages.dev/contributing/
• https://c58d58a9.typertiangolo.pages.dev/tutorial/options-autocompletion/

[github-actions]

📝 Docs preview for commit 8b300ba at: https://9a9692fe.typertiangolo.pages.dev

Modified Pages

• https://9a9692fe.typertiangolo.pages.dev/contributing/
• https://9a9692fe.typertiangolo.pages.dev/tutorial/options-autocompletion/

[github-actions]

📝 Docs preview for commit 00e6af5 at: https://b52048c2.typertiangolo.pages.dev

Modified Pages

• https://b52048c2.typertiangolo.pages.dev/contributing/
• https://b52048c2.typertiangolo.pages.dev/tutorial/options-autocompletion/

[bckohan]

@svlandeg I've updated the docs and added the necessary tests. Should be good to go!

[svlandeg]

Great, thanks! I've got this on my TODO list to review in detail, but realistically it might take me 1 or 2 more weeks to get to it 🙏

[github-actions]

📝 Docs preview for commit 8668bb7 at: https://2142af69.typertiangolo.pages.dev

Modified Pages

• https://2142af69.typertiangolo.pages.dev/contributing/
• https://2142af69.typertiangolo.pages.dev/tutorial/options-autocompletion/

[github-actions]

📝 Docs preview for commit da62b3b at: https://09f6ce46.typertiangolo.pages.dev

Modified Pages

• https://09f6ce46.typertiangolo.pages.dev/contributing/
• https://09f6ce46.typertiangolo.pages.dev/tutorial/options-autocompletion/

[svlandeg]

Ok so looking at the diffs of these two PRs (this one and #949) it appears that all the changes from #949 are still included here verbatim. As such I think it's actually easier to review these changes separately, i.e. one PR at a time. I've also pulled out the typo changes to avoid cluttering the diff: #1077 (I should have done that from the get-go).

@bckohan: do you see any reason not to merge #949 first, then continue working on the functionality from this PR?

[bckohan]

@svlandeg as long as the two PRs are merged in the same release that should be ok. Otherwise #949 alone will break downstream software.

[github-actions]

📝 Docs preview for commit 365e374 at: https://e719418f.typertiangolo.pages.dev

Modified Pages

• https://e719418f.typertiangolo.pages.dev/tutorial/options-autocompletion/ - (before)

[tiangolo]

@bckohan in which way would it break downstream software?

[bckohan]

@tiangolo Its not exactly broken - but deprecation warnings are being thrown with no recourse to fix them because autocompletion does not pass the Parameter objects. There are also downstream completers that expect to be able to pass a CompletionItem back because they're setting the type field.

[tiangolo]

I would think now there would be fewer warnings as the autocompletion parameter is now undeprecated. 😅

And the other shell_complete parameter was never documented, and the implementation was not completed, so I would expect people would not be using it yet. 🤔

There are also downstream completers that expect to be able to pass a CompletionItem back because they're setting the type field.

I'm not sure I got this, could you share an example?

It might also be useful to start in a GitHub Discussion with a minimal example we can copy paste that shows the problem (or if there are multiple problems, multiple examples).

[github-actions]

📝 Docs preview for commit 876bcab at: https://989516f8.typertiangolo.pages.dev

Modified Pages

• https://989516f8.typertiangolo.pages.dev/tutorial/options-autocompletion/ - (before)

[bckohan]

HI!

I think its reasonable to expect significant usage of shell_complete for the following reasons:

1. The deprecation message for autocompletion pointed to it.
2. It's documented by Click and was in the named parameter lists of Argument/Option.
3. Without the Parameter object there are things you simply can't do in generic completer functions (i.e. functions that are written without knowing which parameter they are going to be asked to complete).
4. It's much more reliable to fetch the parsed argument values for variadic arguments from the Context using the parameter name. The autocompletion interface as documented would have you manually parse the list of argument strings to determine if you need to exclude previously provided argument values, but see 5.
5. The argument list passed to autocompletion is usually empty because it's derived from ctx.args which only includes unparsed leftover arguments. This PR fixes the behavior to match the docs and also the test that is currently erroneously passing.

I'm extensively using shell_complete with Parameter objects downstream. And there's evidence here of others trying. I also added example/explanation of why this is needed to the tutorial in the docs. And here's real world production code where I'm using Parameter to filter out previously provided model objects for a generic model field completer.

CompletionItem is somewhat less important than all of the above. Click uses it because it's shell specific scripts are using inbuilt file/directory completer logic. To trigger these you have to be able to pass back meta info. I can think of 4 reasons to support this:

1. Click does, so there are likely generic completer functions in the wild that people might want to reuse on typer apps.
2. Its a trivial addition to the code.
3. Gives future implementors of other shells a hook to add meta information that may be useful for enabling whatever super awesome shell specific completer feature they want to use. The arbitrariness of type is important. It could be something other than file or directory.
4. Existing code is probably already passing these objects back (mine is at least!)

[github-actions]

📝 Docs preview for commit 46c406b at: https://402b6ae8.typertiangolo.pages.dev

Modified Pages

• https://402b6ae8.typertiangolo.pages.dev/tutorial/options-autocompletion/ - (before)

[bckohan]

I rebased - hopefully it will be easier to review!

[github-actions]

📝 Docs preview for commit 031a59e at: https://21b1706e.typertiangolo.pages.dev

Modified Pages

• https://21b1706e.typertiangolo.pages.dev/tutorial/options-autocompletion/ - (before)

[github-actions]

📝 Docs preview for commit 05cb6f0 at: https://fb6de8b0.typertiangolo.pages.dev

Modified Pages

• https://fb6de8b0.typertiangolo.pages.dev/tutorial/options-autocompletion/ - (before)