Create distributed.md

[mikekgfb]

Initial documentation for use of distributed inference w/ torchchat. @mreso @lessw2020 @kwen2501 please review and update as appropriate.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchchat/1438

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[kwen2501]

Looks great! Thanks so much for writing it up!

[mreso]

Thanks @mikekgfb looks good, the response of the server is a bit different but thats true for the non distributed case too.

[mreso]

Suggested change

	<!--
	[skip default]: begin
	## Generate output (requires testing and review by mreso)
	<!--
	[skip default]: begin
	## Generate output

[mreso]

yes, that works

[mikekgfb]

@mreso @kwen2501 @lessw2020 do you have a perspective what list of models will run with distributed? I know we had a separate model dictionary at some point?

Or should we just sweep all the models from README.md and see which work? With this integration this will be pretty easy and cool to do. (yay!) Is there a minimum size for models to run distributed? I know below a certain size it doesn't make sense to run distributed inference, but just for kicks (or a test that's quick to download?)... can it run stories 15M on distributed?

Any attempts at 405B? Wer should def add some of the super large models to the models.Json so they're accessible for distributed inference?

Also, should we create a spare table here, or we could put a second column in the main readme for "distributed inference"? Next to "mobile" column?

Since I'm on a roll asking questions, I'll assume it works only on GPUs (only on CUDA?!), not CPU? Should we add that somewhere? Wanna suggest wording to this effect?