New test runner design #130

mk-mxp · 2024-08-10T15:49:53Z

As suggested in #110 a re-implementation of all current use cases (test cases) using PHPUnit events. This reduces the complexity a lot.

@exercism/maintainers-admin I changed code ownership, as this replaces the existing test-runner completely.

@homersimpsons Do you want to take a look, also?

I also optimized the files remaining in the Docker image:

#23 [runtime 7/7] RUN ls -la && ls -la bin
#23 0.057 total 136
#23 0.057 drwxr-xr-x    5 root     root          4096 Aug 10 09:02 .
#23 0.057 drwxr-xr-x    1 root     root          4096 Aug 10 09:02 ..
#23 0.057 -rw-r--r--    1 root     root          4345 Aug 10 09:02 CODE_OF_CONDUCT.md
#23 0.057 -rw-r--r--    1 root     root         34522 Aug 10 09:02 LICENSE
#23 0.057 -rw-r--r--    1 root     root          1936 Aug 10 09:02 README.md
#23 0.057 drwxr-xr-x    2 root     root          4096 Aug 10 09:02 bin
#23 0.057 -rw-r--r--    1 root     root           262 Aug 10 09:02 composer.json
#23 0.057 -rw-r--r--    1 root     root         59562 Aug 10 09:02 composer.lock
#23 0.057 -rw-r--r--    1 root     root           372 Aug 10 09:02 phpunit.xml
#23 0.057 drwxr-xr-x    2 root     root          4096 Aug 10 09:02 src
#23 0.057 drwxr-xr-x   10 root     root          4096 Aug 10 09:02 vendor
#23 0.058 total 12
#23 0.058 drwxr-xr-x    2 root     root          4096 Aug 10 09:02 .
#23 0.058 drwxr-xr-x    5 root     root          4096 Aug 10 09:02 ..
#23 0.058 -rwxr-xr-x    1 root     root          2172 Aug 10 09:02 run.sh
#23 DONE 0.1s

- Have a Tracer class to track test runs - Extract required data from Passed events - Produce JSON from Result objects - Inject the output file name by environment variable, use fallback name for local runs

This only changes the test names from method names to what PHPUnit sees as prettified test name.

- Use PHPUnit from vendor/* - Provide env var with result file path - Strip out no longer required log files and post-processor The Dockerfile must be written further. composer dependencies should be installed in build step and only "vendor/*" transfered to the runtime image. bin/run.sh still contains debug output.

We don't have to disallow output on the command line anymore, the event fires always.

Also must fail total result on one failed test in list

This must have the absolute exercise path, but cannot get it from PHPUnit. So a new env var provides this.

homersimpsons

Thank you very much for this implementation! I find it way less hacky than the previous one. It also removes a lot of code which means it will be easier to maintain.

Thank you for improving the Dockerfile too. This would probably improve the execution time of the test runner.

.dockerignore

.gitignore

Dockerfile

README.md

src/Result.php

src/Tracer.php

bin/run.sh

src/Tracer.php

homersimpsons

Thanks for all those improvements!

Note: I let some threads open, but they are hidden, I think it would be great to make those changes, but if you don't want to feel free to skip those.

ErikSchierboom

LGTM I'll let the PHP-specific review for the other reviewers

mk-mxp · 2024-08-13T07:51:15Z

Thanks to everyone! I'll merge on Thursday, as I have more time that day in case of unexpected things happening.

mk-mxp added 30 commits August 4, 2024 17:06

Make project with PHPUnit as no-dev requirement

90f8915

Configure PHPUnit to load our extension stub

0a965e4

Produce a valid success result JSON

8775bd5

- Have a Tracer class to track test runs - Extract required data from Passed events - Produce JSON from Result objects - Inject the output file name by environment variable, use fallback name for local runs

Adjust success test case to match output produced

c66dc5c

This only changes the test names from method names to what PHPUnit sees as prettified test name.

Capture user output on success into result JSON

8d0b3ab

We don't have to disallow output on the command line anymore, the event fires always.

Process partial fail into result JSON

536ebf4

Also must fail total result on one failed test in list

Move reflection method finding to code extraction

c576b0e

Remove unused method name from Result

2d57974

Remove PHP extension ds from README

013e8da

Add run-locally.sh to run PHPUnit + diff directly

6793415

Process fail-with-user-output into result JSON

4b1a0e7

Normalize object id locally as in run-tests.sh

fccb20d

Missing test case update from previous commit

fef631e

Replace all path occurences in output

673d80e

Remove dead code

c06052d

Process all-error into result JSON

dc811c1

Process error-with-user-output into result JSON

f4d43ae

Process empty-file into result JSON

95fad6c

Process runtime-error-in-global-code

dc103ca

This must have the absolute exercise path, but cannot get it from PHPUnit. So a new env var provides this.

Process dataProvider-success-with-user-output

172e4d6

Process dataProvider-partial-fail-with-user-output

9bc8c86

Process dataProvider-fail-with-user-output

be037d3

Process dataProvider-error

ca8e7b4

Process dataProvider-error-with-user-output

b2a299e

Finalize the Docker image and scripts

15974db

Copy only run.sh into Docker image

270727e

Remove old junit-handler

fbc644c

Use non-deprecated env var to get no build summary

d6f4d6a

Prevent composer super user warning

455b17a

mk-mxp changed the title ~~Draft: New test runner design~~ New test runner design Aug 10, 2024

Process success-with-task-id

c09ff92

mk-mxp self-assigned this Aug 10, 2024

homersimpsons reviewed Aug 10, 2024

View reviewed changes

bin/run.sh Show resolved Hide resolved

homersimpsons reviewed Aug 10, 2024

View reviewed changes

src/Tracer.php Show resolved Hide resolved

neenjaw removed their request for review August 10, 2024 18:37

mk-mxp mentioned this pull request Aug 11, 2024

Tooling and CI for code quality #131

Open

mk-mxp added 6 commits August 11, 2024 11:01

Use allow list for .dockerignore

15b4845

Add ds extension, drop apk update, missing comma

ead7784

Slice instead of filter the code line array

2c47b05

Make Result->task_id readonly

fb4f930

Use return of preg_match for value detection

560768f

Add type hint to foreach(), where it may matter

5e59544

mk-mxp requested a review from homersimpsons August 11, 2024 09:52

homersimpsons approved these changes Aug 11, 2024

View reviewed changes

mk-mxp added 4 commits August 11, 2024 19:03

Document array format and string options

6627642

Add debug flags

dccd6ce

Add version limit to ds extension

e483d0c

Update README

a237cf9

ErikSchierboom approved these changes Aug 13, 2024

View reviewed changes

mk-mxp merged commit ddd15b3 into exercism:main Aug 15, 2024
1 check passed

mk-mxp deleted the new-test-runner-design branch August 15, 2024 14:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New test runner design #130

New test runner design #130

mk-mxp commented Aug 10, 2024 •

edited

Loading

homersimpsons left a comment

homersimpsons left a comment •

edited

Loading

ErikSchierboom left a comment

mk-mxp commented Aug 13, 2024

New test runner design #130

New test runner design #130

Conversation

mk-mxp commented Aug 10, 2024 • edited Loading

homersimpsons left a comment

Choose a reason for hiding this comment

homersimpsons left a comment • edited Loading

Choose a reason for hiding this comment

ErikSchierboom left a comment

Choose a reason for hiding this comment

mk-mxp commented Aug 13, 2024

mk-mxp commented Aug 10, 2024 •

edited

Loading

homersimpsons left a comment •

edited

Loading