HADOOP-19278. S3A: Remove option to delete directory markers entirely #7052

steveloughran · 2024-09-18T11:53:11Z

The options to delete markers everywhere or under authoritative paths have been removed.

Markers on parent paths will never be deleted after any file creation.

Removes a lot of production code
testing can cut out a lot of parameterized tests, especially on costed operation.
Lines up for pushing more code down into S3AStore (but does not do this)

How was this patch tested?

Marker tool tests all failing.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

steveloughran · 2024-10-29T11:50:04Z

This is good for review. Nice purge of outdated stuff, simpler docs, and with lots of tests deparameterized: faster test runs. What not to like?

@mukund-thakur @ahmarsuhail @shameersss1 @saikatroy038 @HarshitGupta11

shameersss1 · 2024-10-29T13:10:13Z

Great effort @steveloughran for taking this one out. Removing legacy code and making the code base clean.

hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/directory_markers.md

shameersss1 · 2024-10-29T13:15:51Z

hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/directory_markers.md

@@ -306,7 +290,6 @@ line of bucket policies via the `-marker` option



Should we edit this table as well ? delete and authoritative are not valid anymore

shameersss1 · 2024-10-29T13:16:29Z

hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md

@@ -77,7 +77,7 @@ and compatible implementations.

 * Directly reads and writes S3 objects.
 * Compatible with standard S3 clients.
-* Compatible with files created by the older `s3n://` client and Amazon EMR's `s3://` client.
+* Compatible with files created by Amazon EMR's `s3://` client.


Better to call it out as Amazon EMR's s3:// client (EMRFS).

ahmarsuhail · 2024-11-04T14:49:43Z

Thanks Steve, LGTM

- Removes a lot of code and testing - Lines up for pushing more code down into S3AStore (but does not do this) - Marker tool tests all failing. Change-Id: I975dfb22376acd9021c18c4ecdb4b9d47f8be586

Change-Id: I8f30359068e464eb49810bbe964e2048c5962455

Change-Id: I9f41965bad7d5b9189f67593b067b42cdec5f605

It's been years since anyone created a file with the $folder$ reference. Cut all support and mention of it. Change-Id: I4f51b687d4669a9d43006e365c7f85fba1addb43

Change-Id: Ifb06ef183e245b29e68f92bdcdb608997bd767aa

Change-Id: Ia5e927ddd86b3cfef63dc321b41b4b273379878d

identify some tests which were now failing, fix. Change-Id: I8fe55c40372605548c301ad97ab0a325cd424430

steveloughran · 2025-01-07T17:46:12Z

revisting this; missed the approval. some bit of resync needed and I've done a full doc review/rework.

couple of test failures cropped up

hadoop-yetus · 2025-01-07T19:11:19Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	6m 56s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+0 🆗	markdownlint	0m 1s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 37 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	23m 16s		trunk passed
+1 💚	compile	0m 26s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 22s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	checkstyle	0m 22s		trunk passed
+1 💚	mvnsite	0m 28s		trunk passed
+1 💚	javadoc	0m 28s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 19s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	0m 48s		trunk passed
+1 💚	shadedclient	19m 16s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 17s		the patch passed
+1 💚	compile	0m 19s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 19s		the patch passed
+1 💚	compile	0m 16s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	javac	0m 16s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 3 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️	checkstyle	0m 11s	/results-checkstyle-hadoop-tools_hadoop-aws.txt	hadoop-tools/hadoop-aws: The patch generated 1 new + 13 unchanged - 6 fixed = 14 total (was 19)
+1 💚	mvnsite	0m 20s		the patch passed
+1 💚	javadoc	0m 18s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 17s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	0m 41s		the patch passed
+1 💚	shadedclient	19m 5s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 12s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 25s		The patch does not generate ASF License warnings.
		78m 5s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7052/7/artifact/out/Dockerfile
GITHUB PR	#7052
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle markdownlint
uname	Linux 6e22c74da0cd 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `f250f4c`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7052/7/testReport/
Max. process+thread count	554 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7052/7/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

* yetus complaints * search source for "marker" and identify what needs to change (docs, some test comments and messages) * same for "keeping" Change-Id: I78988d33a15be2fbc48c7bc309391f8a7e52f63c

steveloughran · 2025-01-08T13:24:45Z

latest test: s3 london.

One failure, unrelated, seen before

ITestS3APrefetchingLruEviction.testSeeksWithLruEviction:161 » TestTimedOut

hadoop-yetus · 2025-01-08T14:36:22Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 21s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 38 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	21m 44s		trunk passed
+1 💚	compile	0m 27s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 19s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	checkstyle	0m 22s		trunk passed
+1 💚	mvnsite	0m 29s		trunk passed
+1 💚	javadoc	0m 28s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 21s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	0m 44s		trunk passed
+1 💚	shadedclient	19m 9s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 20s		the patch passed
+1 💚	compile	0m 19s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 19s		the patch passed
+1 💚	compile	0m 14s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	javac	0m 14s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 12s		hadoop-tools/hadoop-aws: The patch generated 0 new + 14 unchanged - 6 fixed = 14 total (was 20)
+1 💚	mvnsite	0m 17s		the patch passed
+1 💚	javadoc	0m 17s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 17s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	0m 44s		the patch passed
+1 💚	shadedclient	21m 30s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 59s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 21s		The patch does not generate ASF License warnings.
		71m 46s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7052/8/artifact/out/Dockerfile
GITHUB PR	#7052
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle markdownlint
uname	Linux c8cee6637a7e 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `cb5a7f4`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7052/8/testReport/
Max. process+thread count	553 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7052/8/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Change-Id: I40445f2b5209dc1a8fc2aa3d695618fa4b6ecfe7

hadoop-yetus · 2025-01-08T17:31:26Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+0 🆗	markdownlint	0m 1s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 38 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	22m 2s		trunk passed
+1 💚	compile	0m 26s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 24s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	checkstyle	0m 22s		trunk passed
+1 💚	mvnsite	0m 29s		trunk passed
+1 💚	javadoc	0m 28s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 21s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	0m 47s		trunk passed
+1 💚	shadedclient	19m 18s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 18s		the patch passed
+1 💚	compile	0m 23s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 23s		the patch passed
+1 💚	compile	0m 17s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	javac	0m 17s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 13s		hadoop-tools/hadoop-aws: The patch generated 0 new + 14 unchanged - 6 fixed = 14 total (was 20)
+1 💚	mvnsite	0m 18s		the patch passed
+1 💚	javadoc	0m 17s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 18s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	0m 44s		the patch passed
+1 💚	shadedclient	19m 11s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 11s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 25s		The patch does not generate ASF License warnings.
		70m 17s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7052/9/artifact/out/Dockerfile
GITHUB PR	#7052
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle markdownlint
uname	Linux c3ead398332f 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `736ccd9`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7052/9/testReport/
Max. process+thread count	737 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7052/9/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran marked this pull request as draft September 18, 2024 11:53

github-actions bot added build trunk TOOLS AWS labels Sep 18, 2024

This comment was marked as outdated.

Sign in to view

steveloughran force-pushed the s3/HADOOP-19278-remove-marker-delete branch from 04844d5 to c02d320 Compare October 9, 2024 16:29

steveloughran force-pushed the s3/HADOOP-19278-remove-marker-delete branch from c02d320 to eadce2a Compare October 28, 2024 19:20

This comment was marked as outdated.

Sign in to view

apache deleted a comment from hadoop-yetus Oct 29, 2024

steveloughran marked this pull request as ready for review October 29, 2024 11:47

steveloughran changed the title ~~HADOOP-19278. remove marker delete option~~ HADOOP-19278. S3A: Remove option to delete directory markers entirely Oct 29, 2024

shameersss1 reviewed Oct 29, 2024

View reviewed changes

ahmarsuhail approved these changes Nov 4, 2024

View reviewed changes

This comment was marked as outdated.

Sign in to view

steveloughran added 6 commits January 7, 2025 14:37

HADOOP-19278. remove marker delete option

c8fd158

- Removes a lot of code and testing - Lines up for pushing more code down into S3AStore (but does not do this) - Marker tool tests all failing. Change-Id: I975dfb22376acd9021c18c4ecdb4b9d47f8be586

HADOOP-19278. fix the build, tune PutObjectOptions use

647b3b5

Change-Id: I8f30359068e464eb49810bbe964e2048c5962455

HADOOP-19278. yetus feedback

92b67c3

Change-Id: I9f41965bad7d5b9189f67593b067b42cdec5f605

HADOOP-19278 Cut all legacy s3n support

f810fb4

It's been years since anyone created a file with the $folder$ reference. Cut all support and mention of it. Change-Id: I4f51b687d4669a9d43006e365c7f85fba1addb43

HADOOP-19278 make sure bucket-info docs are in sync with code

3e4b6df

Change-Id: Ifb06ef183e245b29e68f92bdcdb608997bd767aa

HADOOP-19278. markers: doc review

93baadc

Change-Id: Ia5e927ddd86b3cfef63dc321b41b4b273379878d

steveloughran force-pushed the s3/HADOOP-19278-remove-marker-delete branch from 48ba2c7 to 93baadc Compare January 7, 2025 14:37

This comment was marked as outdated.

Sign in to view

HADOOP-19278. fix trunk compilation.

f250f4c

identify some tests which were now failing, fix. Change-Id: I8fe55c40372605548c301ad97ab0a325cd424430

HADOOP-19278. Final pre-merge review

cb5a7f4

* yetus complaints * search source for "marker" and identify what needs to change (docs, some test comments and messages) * same for "keeping" Change-Id: I78988d33a15be2fbc48c7bc309391f8a7e52f63c

HADOOP-19278. review comment

736ccd9

Change-Id: I40445f2b5209dc1a8fc2aa3d695618fa4b6ecfe7

steveloughran merged commit d44ac28 into apache:trunk Jan 8, 2025
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-19278. S3A: Remove option to delete directory markers entirely #7052

HADOOP-19278. S3A: Remove option to delete directory markers entirely #7052

steveloughran commented Sep 18, 2024 •

edited

Loading

This comment was marked as outdated.

This comment was marked as outdated.

steveloughran commented Oct 29, 2024

shameersss1 commented Oct 29, 2024

shameersss1 Oct 29, 2024

shameersss1 Oct 29, 2024

ahmarsuhail commented Nov 4, 2024

This comment was marked as outdated.

This comment was marked as outdated.

steveloughran commented Jan 7, 2025

hadoop-yetus commented Jan 7, 2025

steveloughran commented Jan 8, 2025

hadoop-yetus commented Jan 8, 2025

hadoop-yetus commented Jan 8, 2025

		@@ -306,7 +290,6 @@ line of bucket policies via the `-marker` option

HADOOP-19278. S3A: Remove option to delete directory markers entirely #7052

HADOOP-19278. S3A: Remove option to delete directory markers entirely #7052

Conversation

steveloughran commented Sep 18, 2024 • edited Loading

How was this patch tested?

For code changes:

This comment was marked as outdated.

This comment was marked as outdated.

steveloughran commented Oct 29, 2024

shameersss1 commented Oct 29, 2024

shameersss1 Oct 29, 2024

Choose a reason for hiding this comment

shameersss1 Oct 29, 2024

Choose a reason for hiding this comment

ahmarsuhail commented Nov 4, 2024

This comment was marked as outdated.

This comment was marked as outdated.

steveloughran commented Jan 7, 2025

hadoop-yetus commented Jan 7, 2025

steveloughran commented Jan 8, 2025

hadoop-yetus commented Jan 8, 2025

hadoop-yetus commented Jan 8, 2025

steveloughran commented Sep 18, 2024 •

edited

Loading