-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Test names are aliased (different tests misreported as instances of the same test) when using long assembly names. #15
Comments
We have a set of issues Also we can't change rule because backward compatibility will be broken. Try to use additional NUnit command line argument "--test-name-format". See examples: ` [TestCase("{m}", ExpectedResult = "TestMethod")] [TestCase("{n}", ExpectedResult = "NUnit.Framework.Internal.Tests")] [TestCase("{c}", ExpectedResult = "TestNameGeneratorTests")] [TestCase("{C}", ExpectedResult = "NUnit.Framework.Internal.Tests.TestNameGeneratorTests")] [TestCase("{M}", ExpectedResult = "NUnit.Framework.Internal.Tests.TestNameGeneratorTests.TestMethod")] [TestCase("{m}_SpecialCase", ExpectedResult = "TestMethod_SpecialCase")] [TestCase("{n}.{c}.{m}", ExpectedResult = "NUnit.Framework.Internal.Tests.TestNameGeneratorTests.TestMethod")] [TestCase("{x}", ExpectedResult = "{x}")] [TestCase("{n}.{c.{m}", ExpectedResult = "NUnit.Framework.Internal.Tests.{c.{m}")] [TestCase("{m}{a}", ExpectedResult = "TestMethod")] [TestCase("{i}", ExpectedResult="THE_ID")]` I hope it will help. |
@NikolayPianikov That looks very promising, but bizarrely enough this seems to only affect the very end part of the name; the test case itself. Running with NUnit 3.4.1 (and the --teamcity argument), in the ##teamcity service messages, the full namespace is still included regardless (and in addition to the format here). The output is So, if I specify the format Providing Back to NUnit support then? |
Yes, I've checked it is not working unfortunately. As workaround you could use this nunit wrapper, which replaces the console output: Just update nunit3-console.exe.config And specify TC nunit3-console path to your nunitWrapper Src code is here: https://github.com/DevTeam/wrapper |
OK, thanks a lot. I will have a look and see whether this could do the trick, and whether I am able to shoehorn it into our repository structure and nuget-based setup. |
@NikolayPianikov What about dropping the assembly name and using [n] instead, where n = the part of the test id that precedes the hyphen? |
@CharliePoole I am not sure that I understand your idea. TC extension uses the "fullName" attribute to create message. The TC extension does not add the name of assembly directly. The TC creates full test names using 2 messages: So TC creates name "MyAssemblyName:MyNamespace.TestFixture.MyTest" and user can't change this rule on the TC. The argument "--test-name-format" changes "name" and "fullname" attributes. The "fullname" attribute is a combination of a full class name and a "name" attribute. So when I add the argument --test-name-format={m}_aaaa:
|
@NikolayPianikov I was not suggesting the user could do this, but that you could. I assumed the extension create the concatenated name. If TC does it then of course you can't change the format. Regarding the test name format, the argument is only designed to change test names. It doesn't affect the name of the class. |
…nces of the same test) when using long assembly names.
The following is a representative example of a test name as captured from nunit3-console output by TeamCity:
[Deepish.Namespaces.Due.To.Established.Project.Practices.EngineeringModulinator.UnitTest.dll] Deepish.Namespaces.Due.To.Established.Project.Practices.EngineeringModulinator.UnitTest.RelatedTestsGroup.ConfigureContainer_Always_RegisterType(Deepish.Namespaces.Due.To.Established.Project.Practices.EngineeringModulinator.IEngineeringModulinator)
The bit in brackets is the assembly, which serves as the suite name. Next up is the fully qualified name of the test, with a fully qualified parameter. Note that the test is already "scoped" to the assembly, so in this case, the full namespace is not necessarily required to uniquely identify the test.
When this comes up against the hard 256-character limit for test names, the name is inevitably truncated.
Counting to 256 gets us almost to the last 's' in the final "Namespaces", so any reported summaries of test results will be subject to aliasing, i.e. other instances of this parametrized test that check registration of other types in the same namespace will count as a repeat run of the same test, rather than a separate test.
Even excluding the "suite" identifier, the fully qualified test name with parameter alone is stretching the limits a bit, at 248 characters.
Barring a change that takes away the 256-character limit, is there any way to control how test names are identified in TeamCity? As it stands, the specifics at the end of the test name (i.e. expectations, parameters) are clipped for the benefit of the (in my case, somewhat more redundant) specifics at the start of the name.
The text was updated successfully, but these errors were encountered: