From 6d08e71326956f29812287bc3048a33b407b1618 Mon Sep 17 00:00:00 2001 From: mariam-abdulla Date: Thu, 12 Dec 2024 11:17:43 +0100 Subject: [PATCH 01/31] Create draft proposal for running dotnet test using Microsoft.Testing.Platform --- docs/RFCs/011-Sdk-Testing-Platform.md | 41 +++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100644 docs/RFCs/011-Sdk-Testing-Platform.md diff --git a/docs/RFCs/011-Sdk-Testing-Platform.md b/docs/RFCs/011-Sdk-Testing-Platform.md new file mode 100644 index 0000000000..0357e41a40 --- /dev/null +++ b/docs/RFCs/011-Sdk-Testing-Platform.md @@ -0,0 +1,41 @@ +# RFC 011 - Run dotnet test with Microsoft.Testing.Platform + +- [x] Approved in principle +- [x] Under discussion +- [ ] Implementation +- [ ] Shipped + +## Current State + +Currently, when we run `dotnet test` in CLI, we use vstest as a test runner/driver to run tests in test projects. + +## Motivation + +With `dotnet test`, users should be able to use [Microsoft testing platform](https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-platform-intro?tabs=dotnetcli#microsofttestingplatform-pillars) to run their tests for the sake of improving their experience. They should have the option to opt-in/out this new experience. + +## Note + +We want to design in a way we won't break users and it should be backwards compatible. + +### Proposed solution + +Make this option configurable in global.json. + +Here is a global.json sample: + +```json +{ +"testSdk" : + { + "useTestingPlatform": true + } +} +``` + +### Defaults + +- If no test runner was provided in global.json, the default is set to vstest. + +## Unresolved questions + +None. From f269d198e6929408ff9fc2964bb1dcfe91f10e80 Mon Sep 17 00:00:00 2001 From: mariam-abdulla Date: Thu, 12 Dec 2024 11:29:21 +0100 Subject: [PATCH 02/31] add more suggestions --- docs/RFCs/011-Sdk-Testing-Platform.md | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/docs/RFCs/011-Sdk-Testing-Platform.md b/docs/RFCs/011-Sdk-Testing-Platform.md index 0357e41a40..e6bfcd9b5f 100644 --- a/docs/RFCs/011-Sdk-Testing-Platform.md +++ b/docs/RFCs/011-Sdk-Testing-Platform.md @@ -21,8 +21,9 @@ We want to design in a way we won't break users and it should be backwards compa Make this option configurable in global.json. -Here is a global.json sample: +Here are some global.json samples: +1. ```json { "testSdk" : @@ -32,6 +33,23 @@ Here is a global.json sample: } ``` +What if we want to support another test runner? +We simply can't, with this approach we either use the testing platform, or fallback to vstest if this property was set to false. + +2. +```json +{ +"testSdk" : + { + "testRunner": "vstest/testingplatform" + } +} +``` + +What if we decide to extract the testing platform as an external tool? +We still could support more options. +But if, for some reason, the latest version was broken, we will break as well. + ### Defaults - If no test runner was provided in global.json, the default is set to vstest. From 413b972a110da0c28278c0c87b0c0c808e19f064 Mon Sep 17 00:00:00 2001 From: mariam-abdulla Date: Thu, 12 Dec 2024 11:31:11 +0100 Subject: [PATCH 03/31] add empty lines --- docs/RFCs/011-Sdk-Testing-Platform.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/RFCs/011-Sdk-Testing-Platform.md b/docs/RFCs/011-Sdk-Testing-Platform.md index e6bfcd9b5f..8dfe107420 100644 --- a/docs/RFCs/011-Sdk-Testing-Platform.md +++ b/docs/RFCs/011-Sdk-Testing-Platform.md @@ -34,6 +34,7 @@ Here are some global.json samples: ``` What if we want to support another test runner? + We simply can't, with this approach we either use the testing platform, or fallback to vstest if this property was set to false. 2. @@ -47,7 +48,9 @@ We simply can't, with this approach we either use the testing platform, or fallb ``` What if we decide to extract the testing platform as an external tool? + We still could support more options. + But if, for some reason, the latest version was broken, we will break as well. ### Defaults From 5ff765671ccb0715f6390fe4758154e56c6952bb Mon Sep 17 00:00:00 2001 From: mariam-abdulla Date: Thu, 12 Dec 2024 11:49:27 +0100 Subject: [PATCH 04/31] update doc --- docs/RFCs/011-Sdk-Testing-Platform.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/RFCs/011-Sdk-Testing-Platform.md b/docs/RFCs/011-Sdk-Testing-Platform.md index 8dfe107420..a36e9164ce 100644 --- a/docs/RFCs/011-Sdk-Testing-Platform.md +++ b/docs/RFCs/011-Sdk-Testing-Platform.md @@ -53,6 +53,21 @@ We still could support more options. But if, for some reason, the latest version was broken, we will break as well. +3. +```json +{ +"testSdk" : + { + "testRunner": "vstest/testingplatform/...", + "version": "1.5.0" + } +} +``` + +Users are allowed to force install a specific version of the tool. + +If not specified then we will fallback to the latest version. + ### Defaults - If no test runner was provided in global.json, the default is set to vstest. From a2c9e359078dbe51bb782d1699bd709f45fff487 Mon Sep 17 00:00:00 2001 From: mariam-abdulla Date: Thu, 12 Dec 2024 11:57:04 +0100 Subject: [PATCH 05/31] fix errors --- docs/RFCs/011-Sdk-Testing-Platform.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/RFCs/011-Sdk-Testing-Platform.md b/docs/RFCs/011-Sdk-Testing-Platform.md index a36e9164ce..c59741851a 100644 --- a/docs/RFCs/011-Sdk-Testing-Platform.md +++ b/docs/RFCs/011-Sdk-Testing-Platform.md @@ -24,6 +24,7 @@ Make this option configurable in global.json. Here are some global.json samples: 1. + ```json { "testSdk" : @@ -38,6 +39,7 @@ What if we want to support another test runner? We simply can't, with this approach we either use the testing platform, or fallback to vstest if this property was set to false. 2. + ```json { "testSdk" : @@ -54,6 +56,7 @@ We still could support more options. But if, for some reason, the latest version was broken, we will break as well. 3. + ```json { "testSdk" : From ce31e519532c5ad2fecc09ba3aa6855c57533b9d Mon Sep 17 00:00:00 2001 From: mariam-abdulla Date: Thu, 12 Dec 2024 12:20:57 +0100 Subject: [PATCH 06/31] apply comments --- docs/RFCs/011-Sdk-Testing-Platform.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/RFCs/011-Sdk-Testing-Platform.md b/docs/RFCs/011-Sdk-Testing-Platform.md index c59741851a..7827b67034 100644 --- a/docs/RFCs/011-Sdk-Testing-Platform.md +++ b/docs/RFCs/011-Sdk-Testing-Platform.md @@ -13,6 +13,10 @@ Currently, when we run `dotnet test` in CLI, we use vstest as a test runner/driv With `dotnet test`, users should be able to use [Microsoft testing platform](https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-platform-intro?tabs=dotnetcli#microsofttestingplatform-pillars) to run their tests for the sake of improving their experience. They should have the option to opt-in/out this new experience. +The reason for opting-in/out this experience is +1. if we were to auto-infer if test projects were using vstest or the testing platform is we will end up with many vulnerabilities, thus making it hard to handle this correctly. +2. Mixed mode (i.e. having projects with vstest and testing platform in the same solution) will never work as the two platform have different command line options and different features. + ## Note We want to design in a way we won't break users and it should be backwards compatible. @@ -71,7 +75,7 @@ Users are allowed to force install a specific version of the tool. If not specified then we will fallback to the latest version. -### Defaults +### Default - If no test runner was provided in global.json, the default is set to vstest. From 6d61fb0cfdee6001cf0ba11e95c7c87f4069c168 Mon Sep 17 00:00:00 2001 From: mariam-abdulla Date: Thu, 12 Dec 2024 12:22:26 +0100 Subject: [PATCH 07/31] update doc --- docs/RFCs/011-Sdk-Testing-Platform.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/RFCs/011-Sdk-Testing-Platform.md b/docs/RFCs/011-Sdk-Testing-Platform.md index 7827b67034..da5e437a1d 100644 --- a/docs/RFCs/011-Sdk-Testing-Platform.md +++ b/docs/RFCs/011-Sdk-Testing-Platform.md @@ -15,7 +15,7 @@ With `dotnet test`, users should be able to use [Microsoft testing platform](htt The reason for opting-in/out this experience is 1. if we were to auto-infer if test projects were using vstest or the testing platform is we will end up with many vulnerabilities, thus making it hard to handle this correctly. -2. Mixed mode (i.e. having projects with vstest and testing platform in the same solution) will never work as the two platform have different command line options and different features. +2. Mixed mode (i.e. having projects using vstest and testing platform in the same solution) will never work as the two platforms have different command line options and different features, thus the mapping will not work as expected. ## Note From dd544a6feb1ccd94aef052bf63a3c8e3b50a5e3f Mon Sep 17 00:00:00 2001 From: Mariam Abdullah <122357303+mariam-abdulla@users.noreply.github.com> Date: Thu, 12 Dec 2024 13:31:12 +0100 Subject: [PATCH 08/31] Update docs/RFCs/011-Sdk-Testing-Platform.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Jakub Jareš --- docs/RFCs/011-Sdk-Testing-Platform.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/RFCs/011-Sdk-Testing-Platform.md b/docs/RFCs/011-Sdk-Testing-Platform.md index da5e437a1d..72065347c3 100644 --- a/docs/RFCs/011-Sdk-Testing-Platform.md +++ b/docs/RFCs/011-Sdk-Testing-Platform.md @@ -14,7 +14,7 @@ Currently, when we run `dotnet test` in CLI, we use vstest as a test runner/driv With `dotnet test`, users should be able to use [Microsoft testing platform](https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-platform-intro?tabs=dotnetcli#microsofttestingplatform-pillars) to run their tests for the sake of improving their experience. They should have the option to opt-in/out this new experience. The reason for opting-in/out this experience is -1. if we were to auto-infer if test projects were using vstest or the testing platform is we will end up with many vulnerabilities, thus making it hard to handle this correctly. +1. autodetecting if test projects are using vstest or the testing platform is complex, and we would end up with many false positives, making it hard to deliver a working and consistent experience. 2. Mixed mode (i.e. having projects using vstest and testing platform in the same solution) will never work as the two platforms have different command line options and different features, thus the mapping will not work as expected. ## Note From 20539bd6733fb43e3e3ba6a07f5039c445234f81 Mon Sep 17 00:00:00 2001 From: mariam-abdulla Date: Thu, 12 Dec 2024 13:33:20 +0100 Subject: [PATCH 09/31] apply comments --- docs/RFCs/011-Sdk-Testing-Platform.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/RFCs/011-Sdk-Testing-Platform.md b/docs/RFCs/011-Sdk-Testing-Platform.md index 72065347c3..0b95e6942e 100644 --- a/docs/RFCs/011-Sdk-Testing-Platform.md +++ b/docs/RFCs/011-Sdk-Testing-Platform.md @@ -14,12 +14,12 @@ Currently, when we run `dotnet test` in CLI, we use vstest as a test runner/driv With `dotnet test`, users should be able to use [Microsoft testing platform](https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-platform-intro?tabs=dotnetcli#microsofttestingplatform-pillars) to run their tests for the sake of improving their experience. They should have the option to opt-in/out this new experience. The reason for opting-in/out this experience is -1. autodetecting if test projects are using vstest or the testing platform is complex, and we would end up with many false positives, making it hard to deliver a working and consistent experience. +1. Autodetecting if test projects are using vstest or the testing platform is complex, and we would end up with many false positives, making it hard to deliver a working and consistent experience. 2. Mixed mode (i.e. having projects using vstest and testing platform in the same solution) will never work as the two platforms have different command line options and different features, thus the mapping will not work as expected. ## Note -We want to design in a way we won't break users and it should be backwards compatible. +We want to design in a way that is future proof and easy to keep backwards compatible. ### Proposed solution @@ -48,7 +48,7 @@ We simply can't, with this approach we either use the testing platform, or fallb { "testSdk" : { - "testRunner": "vstest/testingplatform" + "tool": "vstest/testingplatform" } } ``` @@ -65,7 +65,7 @@ But if, for some reason, the latest version was broken, we will break as well. { "testSdk" : { - "testRunner": "vstest/testingplatform/...", + "tool": "vstest/testingplatform/...", "version": "1.5.0" } } From 67189b49a2378b1c9f934979467de7e9e93adfc9 Mon Sep 17 00:00:00 2001 From: mariam-abdulla Date: Thu, 12 Dec 2024 13:50:23 +0100 Subject: [PATCH 10/31] apply comments --- docs/RFCs/011-Sdk-Testing-Platform.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/RFCs/011-Sdk-Testing-Platform.md b/docs/RFCs/011-Sdk-Testing-Platform.md index 0b95e6942e..9882448e47 100644 --- a/docs/RFCs/011-Sdk-Testing-Platform.md +++ b/docs/RFCs/011-Sdk-Testing-Platform.md @@ -23,7 +23,7 @@ We want to design in a way that is future proof and easy to keep backwards compa ### Proposed solution -Make this option configurable in global.json. +Make this option configurable in global.json. We chose global.json because it's located on the current directory level or its parent directories and is picked up by the dotnet sdk. Here are some global.json samples: From 9bd88bc43ef3645b7cc731b6267923ff316043d4 Mon Sep 17 00:00:00 2001 From: mariam-abdulla Date: Thu, 12 Dec 2024 13:55:11 +0100 Subject: [PATCH 11/31] update doc --- docs/RFCs/011-Sdk-Testing-Platform.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/docs/RFCs/011-Sdk-Testing-Platform.md b/docs/RFCs/011-Sdk-Testing-Platform.md index 9882448e47..84a37d5815 100644 --- a/docs/RFCs/011-Sdk-Testing-Platform.md +++ b/docs/RFCs/011-Sdk-Testing-Platform.md @@ -27,7 +27,7 @@ Make this option configurable in global.json. We chose global.json because it's Here are some global.json samples: -1. +1. Boolean ```json { @@ -42,7 +42,7 @@ What if we want to support another test runner? We simply can't, with this approach we either use the testing platform, or fallback to vstest if this property was set to false. -2. +2. Specify the Tool ```json { @@ -57,23 +57,24 @@ What if we decide to extract the testing platform as an external tool? We still could support more options. -But if, for some reason, the latest version was broken, we will break as well. +But if, for some reason, the latest version of the testing platform was broken, we will break as well. -3. +3. Specify the Tool and Version ```json { "testSdk" : { "tool": "vstest/testingplatform/...", - "version": "1.5.0" + "version": "1.5.0", + "allowPrerelease": false } } ``` Users are allowed to force install a specific version of the tool. -If not specified then we will fallback to the latest version. +If it's not specified, then we will fallback to the latest version. ### Default From 712be43f0c122effe5f92cc3ce1f6addda45a5d1 Mon Sep 17 00:00:00 2001 From: mariam-abdulla Date: Thu, 12 Dec 2024 13:58:36 +0100 Subject: [PATCH 12/31] update doc --- docs/RFCs/011-Sdk-Testing-Platform.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/RFCs/011-Sdk-Testing-Platform.md b/docs/RFCs/011-Sdk-Testing-Platform.md index 84a37d5815..cb9b165982 100644 --- a/docs/RFCs/011-Sdk-Testing-Platform.md +++ b/docs/RFCs/011-Sdk-Testing-Platform.md @@ -13,7 +13,8 @@ Currently, when we run `dotnet test` in CLI, we use vstest as a test runner/driv With `dotnet test`, users should be able to use [Microsoft testing platform](https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-platform-intro?tabs=dotnetcli#microsofttestingplatform-pillars) to run their tests for the sake of improving their experience. They should have the option to opt-in/out this new experience. -The reason for opting-in/out this experience is +The reason for opting-in/out this experience is + 1. Autodetecting if test projects are using vstest or the testing platform is complex, and we would end up with many false positives, making it hard to deliver a working and consistent experience. 2. Mixed mode (i.e. having projects using vstest and testing platform in the same solution) will never work as the two platforms have different command line options and different features, thus the mapping will not work as expected. @@ -27,7 +28,7 @@ Make this option configurable in global.json. We chose global.json because it's Here are some global.json samples: -1. Boolean +1. Enable/Disable Testing Platform ```json { From fd32cb144eae8623734f39a14eed1b5a01a4c2ca Mon Sep 17 00:00:00 2001 From: mariam-abdulla Date: Thu, 12 Dec 2024 14:14:33 +0100 Subject: [PATCH 13/31] Replace 'samples' with 'suggestions' --- docs/RFCs/011-Sdk-Testing-Platform.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/RFCs/011-Sdk-Testing-Platform.md b/docs/RFCs/011-Sdk-Testing-Platform.md index cb9b165982..096ae40053 100644 --- a/docs/RFCs/011-Sdk-Testing-Platform.md +++ b/docs/RFCs/011-Sdk-Testing-Platform.md @@ -26,7 +26,7 @@ We want to design in a way that is future proof and easy to keep backwards compa Make this option configurable in global.json. We chose global.json because it's located on the current directory level or its parent directories and is picked up by the dotnet sdk. -Here are some global.json samples: +Here are some global.json suggestions: 1. Enable/Disable Testing Platform From ee6b117631a4dd58df420ee5c05086d604d7be57 Mon Sep 17 00:00:00 2001 From: mariam-abdulla Date: Thu, 12 Dec 2024 14:26:42 +0100 Subject: [PATCH 14/31] Update titles' suggestions --- docs/RFCs/011-Sdk-Testing-Platform.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/RFCs/011-Sdk-Testing-Platform.md b/docs/RFCs/011-Sdk-Testing-Platform.md index 096ae40053..eaeb75dc23 100644 --- a/docs/RFCs/011-Sdk-Testing-Platform.md +++ b/docs/RFCs/011-Sdk-Testing-Platform.md @@ -22,13 +22,13 @@ The reason for opting-in/out this experience is We want to design in a way that is future proof and easy to keep backwards compatible. -### Proposed solution +## Proposed solution Make this option configurable in global.json. We chose global.json because it's located on the current directory level or its parent directories and is picked up by the dotnet sdk. Here are some global.json suggestions: -1. Enable/Disable Testing Platform +### 1. Enable/Disable Testing Platform ```json { @@ -43,7 +43,7 @@ What if we want to support another test runner? We simply can't, with this approach we either use the testing platform, or fallback to vstest if this property was set to false. -2. Specify the Tool +### 2. Specify the Test Runner Tool ```json { @@ -60,7 +60,7 @@ We still could support more options. But if, for some reason, the latest version of the testing platform was broken, we will break as well. -3. Specify the Tool and Version +### 3. Specify the Test Runner Tool and Version ```json { From ee6e8920af4514199131a7c5be23c8fee947a920 Mon Sep 17 00:00:00 2001 From: mariam-abdulla Date: Thu, 12 Dec 2024 14:41:05 +0100 Subject: [PATCH 15/31] update doc --- docs/RFCs/011-Sdk-Testing-Platform.md | 20 +++++--------------- 1 file changed, 5 insertions(+), 15 deletions(-) diff --git a/docs/RFCs/011-Sdk-Testing-Platform.md b/docs/RFCs/011-Sdk-Testing-Platform.md index eaeb75dc23..dd4e6cda2a 100644 --- a/docs/RFCs/011-Sdk-Testing-Platform.md +++ b/docs/RFCs/011-Sdk-Testing-Platform.md @@ -39,9 +39,7 @@ Here are some global.json suggestions: } ``` -What if we want to support another test runner? - -We simply can't, with this approach we either use the testing platform, or fallback to vstest if this property was set to false. +What if we want to support another test runner? We simply can't, with this approach we either use the testing platform, or fallback to vstest if this property was set to false. ### 2. Specify the Test Runner Tool @@ -54,9 +52,7 @@ We simply can't, with this approach we either use the testing platform, or fallb } ``` -What if we decide to extract the testing platform as an external tool? - -We still could support more options. +What if we decide to extract the testing platform as an external tool? We still could support more options. But if, for some reason, the latest version of the testing platform was broken, we will break as well. @@ -66,21 +62,15 @@ But if, for some reason, the latest version of the testing platform was broken, { "testSdk" : { - "tool": "vstest/testingplatform/...", + "tool": "vstest/testingplatform", "version": "1.5.0", "allowPrerelease": false } } ``` -Users are allowed to force install a specific version of the tool. - -If it's not specified, then we will fallback to the latest version. +Users are allowed to force install a specific version of the tool. If it's not specified, then we will fallback to the latest version. ### Default -- If no test runner was provided in global.json, the default is set to vstest. - -## Unresolved questions - -None. +- If no test runner was provided in global.json, the default is set to vstest. \ No newline at end of file From d24bb0e4eb899d0edb9193664127d8231509c155 Mon Sep 17 00:00:00 2001 From: mariam-abdulla Date: Thu, 12 Dec 2024 14:58:53 +0100 Subject: [PATCH 16/31] Add empty line at the end --- docs/RFCs/011-Sdk-Testing-Platform.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/RFCs/011-Sdk-Testing-Platform.md b/docs/RFCs/011-Sdk-Testing-Platform.md index dd4e6cda2a..d9dad6821f 100644 --- a/docs/RFCs/011-Sdk-Testing-Platform.md +++ b/docs/RFCs/011-Sdk-Testing-Platform.md @@ -73,4 +73,4 @@ Users are allowed to force install a specific version of the tool. If it's not s ### Default -- If no test runner was provided in global.json, the default is set to vstest. \ No newline at end of file +- If no test runner was provided in global.json, the default is set to vstest. From ca95ed58d7d78c1fa0fe9c9e72bbcf785568c7b2 Mon Sep 17 00:00:00 2001 From: mariam-abdulla Date: Thu, 12 Dec 2024 15:02:28 +0100 Subject: [PATCH 17/31] Update doc --- docs/RFCs/011-Sdk-Testing-Platform.md | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/docs/RFCs/011-Sdk-Testing-Platform.md b/docs/RFCs/011-Sdk-Testing-Platform.md index d9dad6821f..4b71e34702 100644 --- a/docs/RFCs/011-Sdk-Testing-Platform.md +++ b/docs/RFCs/011-Sdk-Testing-Platform.md @@ -22,7 +22,7 @@ The reason for opting-in/out this experience is We want to design in a way that is future proof and easy to keep backwards compatible. -## Proposed solution +## Proposals Make this option configurable in global.json. We chose global.json because it's located on the current directory level or its parent directories and is picked up by the dotnet sdk. @@ -43,11 +43,23 @@ What if we want to support another test runner? We simply can't, with this appro ### 2. Specify the Test Runner Tool +#### Examples of Usage ```json { "testSdk" : { - "tool": "vstest/testingplatform" + "tool": "testingplatform" + } +} +``` + +or + +```json +{ +"testSdk" : + { + "tool": "vstest" } } ``` @@ -62,7 +74,7 @@ But if, for some reason, the latest version of the testing platform was broken, { "testSdk" : { - "tool": "vstest/testingplatform", + "tool": "testingplatform", "version": "1.5.0", "allowPrerelease": false } From 0f483be13ba974c7f7c5f1cd4615fea79ccca4d4 Mon Sep 17 00:00:00 2001 From: mariam-abdulla Date: Thu, 12 Dec 2024 15:06:23 +0100 Subject: [PATCH 18/31] Update doc --- docs/RFCs/011-Sdk-Testing-Platform.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/RFCs/011-Sdk-Testing-Platform.md b/docs/RFCs/011-Sdk-Testing-Platform.md index 4b71e34702..2f8f2c2ea8 100644 --- a/docs/RFCs/011-Sdk-Testing-Platform.md +++ b/docs/RFCs/011-Sdk-Testing-Platform.md @@ -44,6 +44,7 @@ What if we want to support another test runner? We simply can't, with this appro ### 2. Specify the Test Runner Tool #### Examples of Usage + ```json { "testSdk" : @@ -70,6 +71,8 @@ But if, for some reason, the latest version of the testing platform was broken, ### 3. Specify the Test Runner Tool and Version +#### Example of Usage + ```json { "testSdk" : @@ -81,7 +84,7 @@ But if, for some reason, the latest version of the testing platform was broken, } ``` -Users are allowed to force install a specific version of the tool. If it's not specified, then we will fallback to the latest version. +This configuration specifies that the testing platform tool should be used with version 1.5.0, and it does not allow pre-release versions. This provides more control over the test runner tool and ensures compatibility with specific versions. If it's not specified, then we will fallback to the latest version. ### Default From e9a9a1aff23f981077c22b0b19a7928bb21eb130 Mon Sep 17 00:00:00 2001 From: mariam-abdulla Date: Thu, 12 Dec 2024 15:16:29 +0100 Subject: [PATCH 19/31] Update doc --- docs/RFCs/011-Sdk-Testing-Platform.md | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/docs/RFCs/011-Sdk-Testing-Platform.md b/docs/RFCs/011-Sdk-Testing-Platform.md index 2f8f2c2ea8..799aaecfe3 100644 --- a/docs/RFCs/011-Sdk-Testing-Platform.md +++ b/docs/RFCs/011-Sdk-Testing-Platform.md @@ -30,6 +30,11 @@ Here are some global.json suggestions: ### 1. Enable/Disable Testing Platform +- `testSdk`: Represents the configuration settings for the test SDK. + - `useTestingPlatform`: A boolean value that determines whether to use the Microsoft Testing Platform. If set to `true`, the testing platform will be used; if set to `false`, vstest will be used. + +#### Example of Usage + ```json { "testSdk" : @@ -39,10 +44,15 @@ Here are some global.json suggestions: } ``` +#### Unresolved Questions + What if we want to support another test runner? We simply can't, with this approach we either use the testing platform, or fallback to vstest if this property was set to false. ### 2. Specify the Test Runner Tool +- `testSdk`: Represents the configuration settings for the test SDK. + - `tool`: Specifies the testing tool to be used. In this case, "testingplatform" is the tool being used. + #### Examples of Usage ```json @@ -65,12 +75,21 @@ or } ``` +#### Unresolved Questions + What if we decide to extract the testing platform as an external tool? We still could support more options. But if, for some reason, the latest version of the testing platform was broken, we will break as well. ### 3. Specify the Test Runner Tool and Version +- `testSdk`: This is the main object that contains the configuration for the testing SDK. + - `tool`: Specifies the name of the testing tool being used. In this case, it is "testingplatform". + - `version`: Indicates the version of the testing tool. Here, it is set to "1.5.0". + - `allowPrerelease`: A boolean value that determines whether pre-release versions of the testing tool are allowed. It is set to false, meaning pre-release versions are not permitted. + +This provides more control over the test runner tool and ensures compatibility with specific versions. If it's not specified, then we will fallback to the latest version. + #### Example of Usage ```json @@ -84,8 +103,6 @@ But if, for some reason, the latest version of the testing platform was broken, } ``` -This configuration specifies that the testing platform tool should be used with version 1.5.0, and it does not allow pre-release versions. This provides more control over the test runner tool and ensures compatibility with specific versions. If it's not specified, then we will fallback to the latest version. - ### Default - If no test runner was provided in global.json, the default is set to vstest. From 7c414d39816a7dc3eb0523860e3da2cab5413f35 Mon Sep 17 00:00:00 2001 From: mariam-abdulla Date: Thu, 12 Dec 2024 15:19:02 +0100 Subject: [PATCH 20/31] Update --- docs/RFCs/011-Sdk-Testing-Platform.md | 35 +++++++++------------------ 1 file changed, 12 insertions(+), 23 deletions(-) diff --git a/docs/RFCs/011-Sdk-Testing-Platform.md b/docs/RFCs/011-Sdk-Testing-Platform.md index 799aaecfe3..bc6e83179f 100644 --- a/docs/RFCs/011-Sdk-Testing-Platform.md +++ b/docs/RFCs/011-Sdk-Testing-Platform.md @@ -30,9 +30,6 @@ Here are some global.json suggestions: ### 1. Enable/Disable Testing Platform -- `testSdk`: Represents the configuration settings for the test SDK. - - `useTestingPlatform`: A boolean value that determines whether to use the Microsoft Testing Platform. If set to `true`, the testing platform will be used; if set to `false`, vstest will be used. - #### Example of Usage ```json @@ -44,15 +41,15 @@ Here are some global.json suggestions: } ``` +- `testSdk`: Represents the configuration settings for the test SDK. + - `useTestingPlatform`: A boolean value that determines whether to use the Microsoft Testing Platform. If set to `true`, the testing platform will be used; if set to `false`, vstest will be used. + #### Unresolved Questions What if we want to support another test runner? We simply can't, with this approach we either use the testing platform, or fallback to vstest if this property was set to false. ### 2. Specify the Test Runner Tool -- `testSdk`: Represents the configuration settings for the test SDK. - - `tool`: Specifies the testing tool to be used. In this case, "testingplatform" is the tool being used. - #### Examples of Usage ```json @@ -64,16 +61,8 @@ What if we want to support another test runner? We simply can't, with this appro } ``` -or - -```json -{ -"testSdk" : - { - "tool": "vstest" - } -} -``` +- `testSdk`: Represents the configuration settings for the test SDK. + - `tool`: Specifies the testing tool to be used (`vstest` or `testingplatform`). In this case, `testingplatform` is the tool being used. #### Unresolved Questions @@ -83,13 +72,6 @@ But if, for some reason, the latest version of the testing platform was broken, ### 3. Specify the Test Runner Tool and Version -- `testSdk`: This is the main object that contains the configuration for the testing SDK. - - `tool`: Specifies the name of the testing tool being used. In this case, it is "testingplatform". - - `version`: Indicates the version of the testing tool. Here, it is set to "1.5.0". - - `allowPrerelease`: A boolean value that determines whether pre-release versions of the testing tool are allowed. It is set to false, meaning pre-release versions are not permitted. - -This provides more control over the test runner tool and ensures compatibility with specific versions. If it's not specified, then we will fallback to the latest version. - #### Example of Usage ```json @@ -103,6 +85,13 @@ This provides more control over the test runner tool and ensures compatibility w } ``` +- `testSdk`: Represents the configuration settings for the test SDK. + - `tool`: Specifies the name of the testing tool being used. In this case, it is "testingplatform". + - `version`: Indicates the version of the testing tool. Here, it is set to "1.5.0". + - `allowPrerelease`: A boolean value that determines whether pre-release versions of the testing tool are allowed. It is set to false, meaning pre-release versions are not permitted. + +This provides more control over the test runner tool and ensures compatibility with specific versions. If it's not specified, then we will fallback to the latest version. + ### Default - If no test runner was provided in global.json, the default is set to vstest. From adf0550c296a8686c95310e687782d0a6ecd8f68 Mon Sep 17 00:00:00 2001 From: mariam-abdulla Date: Thu, 12 Dec 2024 15:20:57 +0100 Subject: [PATCH 21/31] Update doc --- docs/RFCs/011-Sdk-Testing-Platform.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/RFCs/011-Sdk-Testing-Platform.md b/docs/RFCs/011-Sdk-Testing-Platform.md index bc6e83179f..b8b798e642 100644 --- a/docs/RFCs/011-Sdk-Testing-Platform.md +++ b/docs/RFCs/011-Sdk-Testing-Platform.md @@ -66,11 +66,11 @@ What if we want to support another test runner? We simply can't, with this appro #### Unresolved Questions -What if we decide to extract the testing platform as an external tool? We still could support more options. +What if we decide to use the testing platform as an external tool/service? We still could support more options. But if, for some reason, the latest version of the testing platform was broken, we will break as well. -### 3. Specify the Test Runner Tool and Version +### 3. Specify the Test Runner Tool and Other Options #### Example of Usage @@ -86,7 +86,7 @@ But if, for some reason, the latest version of the testing platform was broken, ``` - `testSdk`: Represents the configuration settings for the test SDK. - - `tool`: Specifies the name of the testing tool being used. In this case, it is "testingplatform". + - `tool`: Specifies the name of the testing tool being used. In this case, it is `testingplatform`. - `version`: Indicates the version of the testing tool. Here, it is set to "1.5.0". - `allowPrerelease`: A boolean value that determines whether pre-release versions of the testing tool are allowed. It is set to false, meaning pre-release versions are not permitted. From ce37955e544906d2c1c689ea237d05d3c35629a3 Mon Sep 17 00:00:00 2001 From: mariam-abdulla Date: Thu, 12 Dec 2024 15:26:22 +0100 Subject: [PATCH 22/31] Solve indentation errors --- docs/RFCs/011-Sdk-Testing-Platform.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/RFCs/011-Sdk-Testing-Platform.md b/docs/RFCs/011-Sdk-Testing-Platform.md index b8b798e642..7bafb31dea 100644 --- a/docs/RFCs/011-Sdk-Testing-Platform.md +++ b/docs/RFCs/011-Sdk-Testing-Platform.md @@ -11,7 +11,7 @@ Currently, when we run `dotnet test` in CLI, we use vstest as a test runner/driv ## Motivation -With `dotnet test`, users should be able to use [Microsoft testing platform](https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-platform-intro?tabs=dotnetcli#microsofttestingplatform-pillars) to run their tests for the sake of improving their experience. They should have the option to opt-in/out this new experience. +With `dotnet test`, users should be able to use [Microsoft Testing Platform](https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-platform-intro?tabs=dotnetcli#microsofttestingplatform-pillars) to run their tests for the sake of improving their experience. They should have the option to opt-in/out this new experience. The reason for opting-in/out this experience is @@ -62,7 +62,7 @@ What if we want to support another test runner? We simply can't, with this appro ``` - `testSdk`: Represents the configuration settings for the test SDK. - - `tool`: Specifies the testing tool to be used (`vstest` or `testingplatform`). In this case, `testingplatform` is the tool being used. + - `tool`: Specifies the testing tool to be used (`vstest` or `testingplatform`). In this case, `testingplatform` is the tool being used. #### Unresolved Questions From 8f5308d3f6c5f77b69196731f45243db600bd1de Mon Sep 17 00:00:00 2001 From: mariam-abdulla Date: Thu, 12 Dec 2024 15:29:55 +0100 Subject: [PATCH 23/31] Update title --- docs/RFCs/011-Sdk-Testing-Platform.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/RFCs/011-Sdk-Testing-Platform.md b/docs/RFCs/011-Sdk-Testing-Platform.md index 7bafb31dea..c5b74b0f32 100644 --- a/docs/RFCs/011-Sdk-Testing-Platform.md +++ b/docs/RFCs/011-Sdk-Testing-Platform.md @@ -28,7 +28,7 @@ Make this option configurable in global.json. We chose global.json because it's Here are some global.json suggestions: -### 1. Enable/Disable Testing Platform +### 1. Enable/Disable the Testing Platform #### Example of Usage From 97bf74193efa1f3c23de6550fa9862b4ef4f2fbe Mon Sep 17 00:00:00 2001 From: mariam-abdulla Date: Fri, 13 Dec 2024 09:10:09 +0100 Subject: [PATCH 24/31] Update doc --- docs/RFCs/011-Sdk-Testing-Platform.md | 74 +++++++++------------------ 1 file changed, 24 insertions(+), 50 deletions(-) diff --git a/docs/RFCs/011-Sdk-Testing-Platform.md b/docs/RFCs/011-Sdk-Testing-Platform.md index c5b74b0f32..8ec245d63d 100644 --- a/docs/RFCs/011-Sdk-Testing-Platform.md +++ b/docs/RFCs/011-Sdk-Testing-Platform.md @@ -22,75 +22,49 @@ The reason for opting-in/out this experience is We want to design in a way that is future proof and easy to keep backwards compatible. -## Proposals +## Proposal Make this option configurable in global.json. We chose global.json because it's located on the current directory level or its parent directories and is picked up by the dotnet sdk. -Here are some global.json suggestions: +Here is a global.json sample: -### 1. Enable/Disable the Testing Platform +### Examples of Usage -#### Example of Usage +1. Example 1 ```json { -"testSdk" : - { - "useTestingPlatform": true + "test" : { + "runner": { + "name": "MicrosoftTestingPlatform" + } } } ``` -- `testSdk`: Represents the configuration settings for the test SDK. - - `useTestingPlatform`: A boolean value that determines whether to use the Microsoft Testing Platform. If set to `true`, the testing platform will be used; if set to `false`, vstest will be used. - -#### Unresolved Questions - -What if we want to support another test runner? We simply can't, with this approach we either use the testing platform, or fallback to vstest if this property was set to false. - -### 2. Specify the Test Runner Tool - -#### Examples of Usage +/** + * JSON Configuration for the testing platform. + * + * Properties: + * - test: Contains configuration related to the test settings. + * - runner: Specifies the test runner details. + * - name: The name of the test runner to be used, in this case, "MicrosoftTestingPlatform". + */ + +2. Example 2 ```json { -"testSdk" : - { - "tool": "testingplatform" + "test" : { + "runner": { + "name": "VSTest" + } } } ``` -- `testSdk`: Represents the configuration settings for the test SDK. - - `tool`: Specifies the testing tool to be used (`vstest` or `testingplatform`). In this case, `testingplatform` is the tool being used. - -#### Unresolved Questions - -What if we decide to use the testing platform as an external tool/service? We still could support more options. - -But if, for some reason, the latest version of the testing platform was broken, we will break as well. - -### 3. Specify the Test Runner Tool and Other Options - -#### Example of Usage - -```json -{ -"testSdk" : - { - "tool": "testingplatform", - "version": "1.5.0", - "allowPrerelease": false - } -} -``` - -- `testSdk`: Represents the configuration settings for the test SDK. - - `tool`: Specifies the name of the testing tool being used. In this case, it is `testingplatform`. - - `version`: Indicates the version of the testing tool. Here, it is set to "1.5.0". - - `allowPrerelease`: A boolean value that determines whether pre-release versions of the testing tool are allowed. It is set to false, meaning pre-release versions are not permitted. - -This provides more control over the test runner tool and ensures compatibility with specific versions. If it's not specified, then we will fallback to the latest version. +This design is extendable. If later on we decided to support dotnet test as an external dotnet tool. +We can simply add the type/source property and other options as well. ### Default From 04e470b517c3a52b017ca9390991a197594765ec Mon Sep 17 00:00:00 2001 From: mariam-abdulla Date: Fri, 13 Dec 2024 09:16:55 +0100 Subject: [PATCH 25/31] Fix placement of comments --- docs/RFCs/011-Sdk-Testing-Platform.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/RFCs/011-Sdk-Testing-Platform.md b/docs/RFCs/011-Sdk-Testing-Platform.md index 8ec245d63d..d28d52fc32 100644 --- a/docs/RFCs/011-Sdk-Testing-Platform.md +++ b/docs/RFCs/011-Sdk-Testing-Platform.md @@ -32,7 +32,7 @@ Here is a global.json sample: 1. Example 1 -```json +```json { "test" : { "runner": { @@ -40,7 +40,6 @@ Here is a global.json sample: } } } -``` /** * JSON Configuration for the testing platform. @@ -50,7 +49,8 @@ Here is a global.json sample: * - runner: Specifies the test runner details. * - name: The name of the test runner to be used, in this case, "MicrosoftTestingPlatform". */ - +``` + 2. Example 2 ```json From 3b389f2d3b9f009948024f894bf7f7fa296f1aa0 Mon Sep 17 00:00:00 2001 From: mariam-abdulla Date: Fri, 13 Dec 2024 09:17:30 +0100 Subject: [PATCH 26/31] Update doc --- docs/RFCs/011-Sdk-Testing-Platform.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/RFCs/011-Sdk-Testing-Platform.md b/docs/RFCs/011-Sdk-Testing-Platform.md index d28d52fc32..2ddedb0a11 100644 --- a/docs/RFCs/011-Sdk-Testing-Platform.md +++ b/docs/RFCs/011-Sdk-Testing-Platform.md @@ -30,7 +30,7 @@ Here is a global.json sample: ### Examples of Usage -1. Example 1 +#### 1. Example 1 ```json { @@ -51,7 +51,7 @@ Here is a global.json sample: */ ``` -2. Example 2 +#### 2. Example 2 ```json { From 9aae287fe6837364bdb32cfe739e1ac43a349a43 Mon Sep 17 00:00:00 2001 From: mariam-abdulla Date: Fri, 13 Dec 2024 09:21:48 +0100 Subject: [PATCH 27/31] Fix errors --- docs/RFCs/011-Sdk-Testing-Platform.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/RFCs/011-Sdk-Testing-Platform.md b/docs/RFCs/011-Sdk-Testing-Platform.md index 2ddedb0a11..2281dcc6e8 100644 --- a/docs/RFCs/011-Sdk-Testing-Platform.md +++ b/docs/RFCs/011-Sdk-Testing-Platform.md @@ -32,7 +32,7 @@ Here is a global.json sample: #### 1. Example 1 -```json +```json { "test" : { "runner": { @@ -55,7 +55,7 @@ Here is a global.json sample: ```json { - "test" : { + "test" : { "runner": { "name": "VSTest" } From c9a141da33a23b1c5d960323554b7e8461048415 Mon Sep 17 00:00:00 2001 From: mariam-abdulla Date: Fri, 13 Dec 2024 09:29:35 +0100 Subject: [PATCH 28/31] Fix error --- docs/RFCs/011-Sdk-Testing-Platform.md | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/docs/RFCs/011-Sdk-Testing-Platform.md b/docs/RFCs/011-Sdk-Testing-Platform.md index 2281dcc6e8..c39a80c1e2 100644 --- a/docs/RFCs/011-Sdk-Testing-Platform.md +++ b/docs/RFCs/011-Sdk-Testing-Platform.md @@ -34,23 +34,19 @@ Here is a global.json sample: ```json { - "test" : { + "test" : { "runner": { "name": "MicrosoftTestingPlatform" } } } - -/** - * JSON Configuration for the testing platform. - * - * Properties: - * - test: Contains configuration related to the test settings. - * - runner: Specifies the test runner details. - * - name: The name of the test runner to be used, in this case, "MicrosoftTestingPlatform". - */ ``` +It contains the properties below: + - test: Contains configuration related to the test settings. + - runner: Specifies the test runner details. + - name: The name of the test runner to be used, in this case, "MicrosoftTestingPlatform". + #### 2. Example 2 ```json From 6a86c668e6f8247eb992095483034dece2c0e34b Mon Sep 17 00:00:00 2001 From: mariam-abdulla Date: Fri, 13 Dec 2024 09:41:52 +0100 Subject: [PATCH 29/31] Fix errors --- docs/RFCs/011-Sdk-Testing-Platform.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/RFCs/011-Sdk-Testing-Platform.md b/docs/RFCs/011-Sdk-Testing-Platform.md index c39a80c1e2..148fddbcbe 100644 --- a/docs/RFCs/011-Sdk-Testing-Platform.md +++ b/docs/RFCs/011-Sdk-Testing-Platform.md @@ -43,9 +43,9 @@ Here is a global.json sample: ``` It contains the properties below: - - test: Contains configuration related to the test settings. - - runner: Specifies the test runner details. - - name: The name of the test runner to be used, in this case, "MicrosoftTestingPlatform". +- test: Contains configuration related to the test settings. +- runner: Specifies the test runner details. +- name: The name of the test runner to be used, in this case, "MicrosoftTestingPlatform". #### 2. Example 2 From f13f0b342c09d1471cc3ceaba55d9352025ef66b Mon Sep 17 00:00:00 2001 From: mariam-abdulla Date: Fri, 13 Dec 2024 10:02:13 +0100 Subject: [PATCH 30/31] fix errors --- docs/RFCs/011-Sdk-Testing-Platform.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/RFCs/011-Sdk-Testing-Platform.md b/docs/RFCs/011-Sdk-Testing-Platform.md index 148fddbcbe..129d53b4ba 100644 --- a/docs/RFCs/011-Sdk-Testing-Platform.md +++ b/docs/RFCs/011-Sdk-Testing-Platform.md @@ -43,10 +43,11 @@ Here is a global.json sample: ``` It contains the properties below: + - test: Contains configuration related to the test settings. - runner: Specifies the test runner details. - name: The name of the test runner to be used, in this case, "MicrosoftTestingPlatform". - + #### 2. Example 2 ```json From 43c3b4b3ab61dd99a7675b93462a2a620a12fca1 Mon Sep 17 00:00:00 2001 From: Mariam Abdullah <122357303+mariam-abdulla@users.noreply.github.com> Date: Fri, 13 Dec 2024 16:08:41 +0100 Subject: [PATCH 31/31] Update docs/RFCs/011-Sdk-Testing-Platform.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Amaury Levé --- docs/RFCs/011-Sdk-Testing-Platform.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/RFCs/011-Sdk-Testing-Platform.md b/docs/RFCs/011-Sdk-Testing-Platform.md index 129d53b4ba..23fb6e8647 100644 --- a/docs/RFCs/011-Sdk-Testing-Platform.md +++ b/docs/RFCs/011-Sdk-Testing-Platform.md @@ -11,7 +11,7 @@ Currently, when we run `dotnet test` in CLI, we use vstest as a test runner/driv ## Motivation -With `dotnet test`, users should be able to use [Microsoft Testing Platform](https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-platform-intro?tabs=dotnetcli#microsofttestingplatform-pillars) to run their tests for the sake of improving their experience. They should have the option to opt-in/out this new experience. +With `dotnet test`, users should be able to use [Microsoft Testing Platform](https://learn.microsoft.com/dotnet/core/testing/unit-testing-platform-intro?tabs=dotnetcli#microsofttestingplatform-pillars) to run their tests for the sake of improving their experience. They should have the option to opt-in/out this new experience. The reason for opting-in/out this experience is