MS Test 2.2.4 - 2.2.10: Tests fail because DynamicData gets only structs with default values

[gdir22]

Description

Hi,

I have a strange problem with MS Test versions 2.2.4 - 2.2.10. In my actual project a have a lot of unit tests that use DynamicData to fetch the test cases from a method. Up to MS Test 2.2.3 this worked fine. With MS Test 2.2.4 - 2.2.10 some (not all) of these tests fail, because the test cases are mixed up because all structs have default values. This seems to happen when structs are involved, but I'm not sure about that.

Steps to reproduce

I have prepared a minimal solution here: https://github.com/gdir22/Vector

There a four branches in the project:

Branch	Vector2D is	MS Test version	Problem occurs
struct-mstest-2.2.3	struct	2.2.3	no
struct-mstest-2.2.10	struct	2.2.10	yes
class-mstest-2.2.3	class	2.2.3	no
class-mstest-2.2.10	class	2.2.10	no

Just switch the branches, build the solution and run the unit tests.

Expected behavior

All tests should pass.

Actual behavior

If Vector2D is a struct and MS Test version 2.2.10 (or any version after 2.2.3) is used, some of the test cases of the test method Vector2D_op_Multiplication_Vector2D_Vector2D___valid_Vector2D___double_scalar_product fail. They fail deterministically (always the same cases fail). If you debug the failing tests you will notice that the arguments of the test cases are mixed all structs have default values.

The arguments that the factory method sends are not the arguments that the test method receives.

The two other test methods, that are structured in the same way, don't show this behaviour. They pass in all combinations as expected. The same happens here.

Environment

Windows 10 21H2, Visual Studio 2019 or 2022, MS Test 2.2.10

[gdir22]

[gdir22]

Test results with MS Test 2.2.10:

[gdir22]

Test results with MS Test 2.2.3:

[gdir22]

After further investigation my initial suspicion about the test cases getting mixed seems to be wrong. It seems like the structs are not passed correctly between the factory and the test method. All Vector2D structs that the test method receives, get the default value Vector(0.0, 0.0).

[gdir22]

I found the solution here: #905 (comment). Adding

[assembly: TestDataSourceDiscovery(TestDataSourceDiscoveryOption.DuringExecution)]

restores the old behaviour.

[gdir22]

Added a comment at #905 (comment).

[Haplois]

Sorry for the inconvenience caused by this, the solution you found is the correct one.

[dgasaway]

I just tried to upgrade to v3, and at the same time took out TestDataSourceDiscoveryOption.DuringExecution since #1028 got fixed. I got mysterious test failures with uninitialized structs, and ended up here. I suppose I'll be going back to DuringExecution, but I'm wondering if the problem with DuringDiscovery will get fixed?

[Evangelink]

Hi @dgasaway,

Would you mind opening a new issue with a repro of what's not working for you please?

[dgasaway]

@Evangelink My problem is what @gdir22 described so well in this very issue. The only solution suggested here so far is to use DuringExecution instead of DuringDiscovery, which is fine, but leaves open the question whether DuringDiscovery can be made to work in the situation described. After since looking around at some other issues in the tracker, I understand that DuringDiscovery performs a serialization and de-serialization of the structs, but if the struct is not decorated with SerializableAttribute it quietly passes a default struct to the test method. At the very least, it would seem wise to have the test framework warn the user if the argument array contains a non-serializable struct, with a link to documentation with more information, such has how and why to switch to DuringExecution. As it stands, people are left to wonder why their tests have failed (as evidenced by multiple similar issues in the tracker). Ideally, though, it would be nice if DuringDiscovery could work without a serialization round-trip.

[Evangelink]

@dgasaway I don't think it will be possible to have DuringDiscovery working without serialization as we get the data and transfer it over tcp to the Test Platform which transfers it to the Test Window and then during the execution it's transferred back down to MSTest. But I haven't spent much time on the subject at the moment.

I have created a separate task to track the fact that we need to use a better serialization mechanism for this need as the current solution causes too many issues (see #1462).