Convert to parsing project files using MSBuild

[rprouse]

Hand editing the new CSPROJ format is now very common and a scenario we need to take into account going forward especially for users working outside of Visual Studio. @OsirisTerje mentioned an idea for this yesterday. He has some experience parsing the project files using the MSBuild assemblies from the Microsoft.Build nuget package.

It would be a total rework of this project, but would help us understand the many projects.

[ChrisMaddock]

I've had my eye on this for a long time, I think @jnm2 even got further than I did, and actually tried something! 😄 I seem to remember he hit a sticking point however...

Big issue is of course that it's .NET 4.6 or .NET Standard - I don't know what logic the engine currently has for loading extensions where the runtime might not be available, but we should definitely look at that.

I think this would be a much better solution. We could even keep both old and new solutions - package old for .NET 2-4, and new for .NET 4.6 onwards.

[CharliePoole]

@ChrisMaddock Bear in mind that this particular extension only runs in the primary test engine. That is, any engines created in an agent process neither need nor have the extension.

[CharliePoole]

We had some earlier discussion where various people considered or were suggested as being the leads for this project. It seems like such a big push requires a lead.

IIRC both @ChrisMaddock and @ManfredLange thought about it but felt they didn't have time to do it right. Just sayin' if nobody is in a position right now to take overall responsibility for the project, maybe we should own up to the fact that we can't support it as fully as we would like until somebody pops up.

This project could end up being a pretty full job for a few people.

[ChrisMaddock]

Sure - this is a task that needs someone to take it on, but I don't see the harm in getting the idea out there anyway. 😄 It would be a fun little project!

[jnm2]

See https://github.com/rainersigwald/visualstudio-docs/blob/b281deb021636ae4e9cd57c1c2147db7c8af7e3f/docs/msbuild/updating-an-existing-application.md#using-microsoftbuildlocator for brand-new documentation on how to load MSBuild from the correct location rather than bundling it.

[adamkadzban]

The VS Project Loader currently fails to detect configuration that lives in Directory.Build.props files - I'm assuming that's because the project file is parsed as XML, as opposed to parsed via MSBuild like this issue suggests.

This issue is 4 years old...Any chance it gets picked up? Should I open a separate issue specific to the Directory.Build.props issue?

[CharliePoole]

@adamkadzban I'm not sure what you mean... vs-project-loader doesn't make use of Directory.Build.props because I'm not a big fan of it for a small project like this one - I prefer to see all the settings in the project files.

But in any case, a Directory.Build.props file would impact the effective content of the extension's project files at the time they are built. It would have no bearing on the running of the extension or the parsing of the project files from which it creates test packages for execution.

One definite goal for this project is to be able to handle all modern vs project formats. In this issue, Rob was suggesting that we use some underlying code present in MS-Build to parse the projects rather than the hand-coded approach we have taken in the past.

[jnm2]

Any time you are parsing a .csproj file for any reason, you will have to locate the nearest Directory.Build.props and pretend it's part of that .csproj file or risk missing things that were moved out of the csproj into the props.

[CharliePoole]

Ah... thanks for pointing that out. Of course, a not-great option would be to document that we ignore it, but I'd rather make it work. Do you know when Directory.Build.props came in? We have code for parsing ancient layouts, to which it may not apply.

And, of course, if we used the approach Rob suggests in this issue, I imagine MSBuild would take care of that for us.

FWIW, if we can't make the latest project formats work without doing all our own parsing, I'd be inclined to retire the extension and tell people they have to use dotnet CLI to run from projects or solutions. That way I could devote my time to the console runner and engine.

[jnm2]

I've heard that Directory.Build.props has been a thing for a very long time but it's only come into vogue in the last few years. https://docs.microsoft.com/en-us/visualstudio/msbuild/customize-your-build#directorybuildprops-and-directorybuildtargets documents it.

[CharliePoole]

Looks like it came in VS2017 - so fairly recent. See https://docs.microsoft.com/en-us/visualstudio/msbuild/what-s-new-in-msbuild-15-0?view=vs-2017

The project loader supports back to at least vs2005... maybe even 2003. We should probably think about dropping support for the really old stuff if we continue to provide the extension.

[rprouse]

The project loader supports back to at least vs2005... maybe even 2003. We should probably think about dropping support for the really old stuff if we continue to provide the extension.

I know I'm not going to install anything that old to test changes 😄

How far back should we go though? VS2017 is mainstream EOL next month, extended until 2027. 2012 through 2015 are in extended support and 2010 and earlier are out of support.

As I've stated elsewhere, I think we should follow the EOL lifecycle of the frameworks and tools we support. I tend to lean towards mainstream support because we are a small team with limited resources. I know @CharliePoole is usually in favor of being more lenient which is admirable if we can support it.

Nobody has picked this up though.

[CharliePoole]

The old stuff is actually the least of our problems. As it stands now that support is stable, and the tests are built-in, using projects, which were created back in the day and are embedded in the test assembly. It's the new stuff, which is problematic. As well as the fact that editing the project file is now common.

I just noticed that this is still an "idea" which may partly explain why nobody has volunteered to work on it. I'm changing it to a feature, which is how I've been thinking of it since I took on the extensions.

Here's my logic for keeping this...

We (i.e. I) can't keep up with Microsoft's introduction of new features in the project file structure as we have in the past - by reverse-engineering the changes and parsing the file ourselves. By definition, any new wrinkle invented by Microsoft will have to be supported by MSBuild. For that reason, your idea seems hopefull to me, which is why I've kept it here. In fact, if this approach can't be made to work, I'll be proposing we drop the extension itself.

I'm a "team" of one person wrt this project, as nobody else in the organization has wanted to work on it. It would be great if somebody were to volunteer to contribute to this issue, even by doing some preliminary experiments with msbuild so we can decide whether the approach is feasible.

PS: With respect, I don't think it's "lenient" to support our users. The word is unfortunate because it implies that there is something wrong with folks who continue to use old versions of software rather than upgrading. It puts the blame on them rather than simply saying, when we have to, that we are not able to support some feature.

[CharliePoole]

I have added issue #65 so we can try to make a decision about what would need to be supported in a new 4.0 version of the extension. @nunit/engine-team @nunit/core-team

[jifb]

Just for completeness (and the search engine): Using a project which does not contain any "normal" .csproj files, following error raises:

Errors, Failures and Warnings

1) Invalid : X:\Path\to\Complete.sln
File type is not supported

If the feature will be implemented, this will vanish. If the feature will not come (or if MSBuild will be optional), I wish there was a better error message there: "This solution/project(s) could not be loaded because they contain no parsable Configuration/OutputPath. Maybe the files were custom-written? This is not supported yet"