Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDFS-17384. [FGL] HDFS NameNode FGL Phase I #6762

Open
wants to merge 28 commits into
base: trunk
Choose a base branch
from
Open

Conversation

ZanderXu
Copy link
Contributor

@ZanderXu ZanderXu commented Apr 23, 2024

What's the purpose of this feature?

  1. This PR introduces a Fine-Grained Lock Mode, replacing the global lock with a Global FS Lock and a Global BM Lock while retaining the Global Lock Mode as the default
  2. This PR does not modify any processing logic
  3. The Fine-Grained Lock Mode lays the groundwork for Phase II, where the Global FS Lock will be replaced with fine-grained IIPLocks.

What changes were proposed in this pull request?

  • HDFS-17394. [FGL] Remove unused WriteHoldCount of FSNamesystemLock
  • HDFS-17387. [FGL] Abstract the configuration locking mode
  • HDFS-17398. [FGL] Implement the FGL lock for FSNLockManager
  • HDFS-17405. [FGL] Using different metric name to trace performance for FGL and Global lock
  • HDFS-17390. [FGL] FSDirectory supports this fine-grained locking
  • HDFS-17395. [FGL] Use FSLock to protect ErasureCodingPolicy related operations
  • HDFS-17413. [FGL] CacheReplicationMonitor supports fine-grained lock
  • HDFS-17414. [FGL] RPCs in DatanodeProtocol support fine-grained lock
  • HDFS-17416. [FGL] Monitor threads in BlockManager.class support fine-grained lock
  • HDFS-17423. [FGL] BlockManagerSafeMode supports fine-grained lock
  • HDFS-17410. [FGL] Client RPCs that changes file attributes supports fine-grained lock
  • HDFS-17415. [FGL] RPCs in NamenodeProtocol support fine-grained lock
  • HDFS-17417. [FGL] HeartbeatManager and DatanodeAdminMonitor support fine-grained locking
  • HDFS-17412. [FGL] Client RPCs involving maintenance supports fine-grained lock
  • HDFS-17420. [FGL] EditLogTailer and FSEditLogLoader support FGL
  • HDFS-17388. [FGL] Client RPCs involving write process supports fine-grained lock
  • HDFS-17389. [FGL] Client RPCs involving read process supports fine-grained lock
  • HDFS-17411. [FGL] Client RPCs involving snapshot support fine-grained lock
  • HDFS-17445. [FGL] some operations support fine-grained locking
  • HDFS-17424. [FGL] DelegationTokenSecretManager supports fine-grained lock
  • HDFS-17472. [FGL] gcDeletedSnapshot and getDelegationToken support FGL
  • HDFS-17459. [FGL] Add documentation
  • HDFS-17457. [FGL] UTs support fine-grained locking
  • HDFS-17506. [FGL] Performance for phase 1
  • HDFS-17691. [FGL] Move FSNamesystemLockMode to org.apache.hadoop.hdfs.util package
  • HDFS-17692. [FGL] fix bug in getContentSummary method

How was this patch tested?

  • All changes are thoroughly covered by unit tests
  • We have deployed this feature in our production environment for half a year, and all NameNodes work well

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
_ Prechecks _
+1 💚 dupname 0m 14s No case conflicting files found.
+0 🆗 spotbugs 0m 01s spotbugs executables are not available.
+0 🆗 codespell 0m 01s codespell was not available.
+0 🆗 detsecrets 0m 01s detect-secrets was not available.
+0 🆗 xmllint 0m 01s xmllint was not available.
+0 🆗 markdownlint 0m 01s markdownlint was not available.
+1 💚 @author 0m 00s The patch does not contain any @author tags.
+1 💚 test4tests 0m 00s The patch appears to include 47 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 2m 18s Maven dependency ordering for branch
+1 💚 mvninstall 87m 56s trunk passed
+1 💚 compile 39m 10s trunk passed
+1 💚 checkstyle 7m 54s trunk passed
+1 💚 mvnsite 15m 54s trunk passed
+1 💚 javadoc 14m 59s trunk passed
+1 💚 shadedclient 174m 23s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 2m 14s Maven dependency ordering for patch
+1 💚 mvninstall 9m 51s the patch passed
+1 💚 compile 36m 45s the patch passed
+1 💚 javac 36m 44s the patch passed
+1 💚 blanks 0m 01s The patch has no blanks issues.
+1 💚 checkstyle 5m 53s the patch passed
+1 💚 mvnsite 16m 04s the patch passed
+1 💚 javadoc 14m 57s the patch passed
+1 💚 shadedclient 183m 47s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 asflicense 6m 03s The patch does not generate ASF License warnings.
552m 56s
Subsystem Report/Notes
GITHUB PR #6762
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint
uname MINGW64_NT-10.0-17763 087b28dd158d 3.4.10-87d57229.x86_64 2024-02-14 20:17 UTC x86_64 Msys
Build tool maven
Personality /c/hadoop/dev-support/bin/hadoop.sh
git revision trunk / 14153f0
Default Java Azul Systems, Inc.-1.8.0_332-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6762/2/testReport/
modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf hadoop-tools/hadoop-fs2img U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6762/2/console
versions git=2.44.0.windows.1
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Copy link
Contributor

@ferhui ferhui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 from me.
I think we can merge it after getting the performance benchmark if no other comments.

if (lockMode.equals(FSNamesystemLockMode.GLOBAL)) {
if (this.fsLock.isWriteLockedByCurrentThread()) {
// The bm writeLock should be held by the current thread.
assert this.bmLock.isWriteLockedByCurrentThread();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found that hasWriteLock has two way to used:

  • (1) Make sure we hold a lock. like assert fsn.hasReadLock(FSNamesystemLockMode.GLOBAL);
  • (2) Check whether there is a lock to determine whether to perform some action. like boolean hadFsnWriteLock = fsn.hasWriteLock(FSNamesystemLockMode.GLOBAL);

Here add assert, may throw AssertionError. So I think the new hasWriteLock only have the semantics described in (1).
I think we need to separate two approaches to handle (1) and (2).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @zhengchenyu , for your review.
Regarding point (2), are you suggesting that the hasWriteLock(RwLockMode lockMode) method should strictly return true or false and never throw an AssertionError?

Indeed, hasWriteLock(RwLockMode lockMode) is intended to always return true or false. However, I included the check to identify potential issue with lock ordering, because the caller often expects to receive a true result in NameNode.

If this additional check is deemed unnecessary, the code could simply be:

if (lockMode.equals(RwLockMode.GLOBAL)) {
      return this.fsLock.isWriteLockedByCurrentThread() && this.bmLock.isWriteLockedByCurrentThread();
}

Looking forward your ideas, thanks.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#7250 is used to fix this case. @zhengchenyu please help review it. Thanks

@zhengchenyu
Copy link
Contributor

I know this PR is phase 1 work: split global lock into fslock and bmlock.
We should ensure the lock order: bmlock cannot occur in front of fslock. Can we add some restrictions or check mechanisms to avoid getting a bmlock before fslock?

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 33s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 2s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 xmllint 0m 0s xmllint was not available.
+0 🆗 markdownlint 0m 0s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 49 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 5m 54s Maven dependency ordering for branch
+1 💚 mvninstall 29m 50s trunk passed
+1 💚 compile 16m 23s trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 compile 15m 7s trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 checkstyle 4m 20s trunk passed
+1 💚 mvnsite 3m 13s trunk passed
+1 💚 javadoc 3m 0s trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 3m 27s trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 spotbugs 5m 42s trunk passed
+1 💚 shadedclient 33m 2s branch has no errors when building and testing our client artifacts.
-0 ⚠️ patch 33m 28s Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 32s Maven dependency ordering for patch
+1 💚 mvninstall 1m 54s the patch passed
+1 💚 compile 15m 51s the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javac 15m 51s the patch passed
+1 💚 compile 15m 6s the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 javac 15m 6s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 4m 15s root: The patch generated 0 new + 1233 unchanged - 16 fixed = 1233 total (was 1249)
+1 💚 mvnsite 3m 11s the patch passed
-1 ❌ javadoc 1m 19s /results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 13 new + 3117 unchanged - 0 fixed = 3130 total (was 3117)
+1 💚 javadoc 3m 24s the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 spotbugs 6m 15s the patch passed
+1 💚 shadedclient 33m 6s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 227m 34s hadoop-hdfs in the patch passed.
+1 💚 unit 30m 15s hadoop-hdfs-rbf in the patch passed.
+1 💚 unit 0m 59s hadoop-fs2img in the patch passed.
+1 💚 asflicense 1m 12s The patch does not generate ASF License warnings.
474m 1s
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6762/6/artifact/out/Dockerfile
GITHUB PR #6762
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint
uname Linux 77e0fea14a86 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / be1bcd6
Default Java Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6762/6/testReport/
Max. process+thread count 3722 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf hadoop-tools/hadoop-fs2img U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6762/6/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@apache apache deleted a comment from hadoop-yetus Dec 26, 2024
@apache apache deleted a comment from hadoop-yetus Dec 26, 2024
@apache apache deleted a comment from hadoop-yetus Dec 26, 2024
@apache apache deleted a comment from hadoop-yetus Dec 26, 2024
Copy link
Contributor

@hfutatzhanghb hfutatzhanghb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sir, leave some small comments.

@@ -2058,7 +2080,7 @@ BatchedListEntries<OpenFileEntry> listOpenFiles(long prevId,

public BatchedListEntries<OpenFileEntry> getFilesBlockingDecom(long prevId,
String path) {
assert hasReadLock();
assert hasReadLock(RwLockMode.FS);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should use GLOBAL mode here, because below codes contain iterate blockManager.getDatanodeManager().getDatanodes()

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @hfutatzhanghb for your review.
Both dataNode.getLeavingServiceStatus().getOpenFiles() and blockManager.getDatanodeManager().getDatanodes()) are already thread-safe, so the BMLock is unnecessary here.

@@ -2031,7 +2053,7 @@ BatchedListEntries<OpenFileEntry> listOpenFiles(long prevId,
BatchedListEntries<OpenFileEntry> batchedListEntries;
String normalizedPath = new Path(path).toString(); // normalize path.
try {
readLock();
readLock(RwLockMode.FS);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also use GLOBAL mode here, because it invoke getFilesBlockingDecom method.

@ZanderXu
Copy link
Contributor Author

I know this PR is phase 1 work: split global lock into fslock and bmlock. We should ensure the lock order: bmlock cannot occur in front of fslock. Can we add some restrictions or check mechanisms to avoid getting a bmlock before fslock?

Thanks, @zhengchenyu . This is a good idea.
The check mechanism is theoretically feasible (verifying whether the BMLock is already held when acquiring the FSLock). However, due to lock reentry, this approach introduces complexity and negatively impacts the performance.

Let's briefly consider some common scenarios:

  1. A thread acquires the FS read lock for the first time.
  2. A thread acquires the FS write lock for the first time.
  3. A thread holding the Global read lock reenters the FS read lock (already holding the BM read lock).
  4. A thread holding the Global write lock reenters the FS read lock (already holding the BM write lock).
  5. A thread holding the Global write lock reenters the FS write lock (already holding the BM write lock).

In scenarios 3, 4, and 5, the BM lock already held is normal when acquiring FS lock. However, in scenarios 1 and 2, the BM lock already held is abnormal when acquiring FS lock.

Indeed, lock reentry can also be checked, but I think it’s unnecessary as it’s overly complicated. We can use other methods to detect deadlock issues.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 31s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 2s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 xmllint 0m 0s xmllint was not available.
+0 🆗 markdownlint 0m 0s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 49 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 7m 24s Maven dependency ordering for branch
+1 💚 mvninstall 30m 20s trunk passed
+1 💚 compile 16m 31s trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 compile 14m 42s trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 checkstyle 4m 20s trunk passed
+1 💚 mvnsite 3m 12s trunk passed
+1 💚 javadoc 3m 3s trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 3m 22s trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 spotbugs 5m 44s trunk passed
+1 💚 shadedclient 33m 8s branch has no errors when building and testing our client artifacts.
-0 ⚠️ patch 33m 34s Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 32s Maven dependency ordering for patch
+1 💚 mvninstall 1m 56s the patch passed
+1 💚 compile 15m 45s the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javac 15m 45s the patch passed
+1 💚 compile 15m 13s the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 javac 15m 13s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 4m 20s root: The patch generated 0 new + 1233 unchanged - 16 fixed = 1233 total (was 1249)
+1 💚 mvnsite 3m 9s the patch passed
+1 💚 javadoc 1m 19s hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 0 new + 3113 unchanged - 4 fixed = 3113 total (was 3117)
+1 💚 javadoc 0m 54s hadoop-hdfs-rbf in the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.
+1 💚 javadoc 0m 45s hadoop-fs2img in the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.
+1 💚 javadoc 3m 28s the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 spotbugs 6m 19s the patch passed
+1 💚 shadedclient 33m 30s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 224m 24s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 unit 30m 22s hadoop-hdfs-rbf in the patch passed.
+1 💚 unit 0m 59s hadoop-fs2img in the patch passed.
+1 💚 asflicense 1m 13s The patch does not generate ASF License warnings.
473m 10s
Reason Tests
Failed junit tests hadoop.hdfs.TestDecommissionWithBackoffMonitor
hadoop.hdfs.TestDecommission
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6762/8/artifact/out/Dockerfile
GITHUB PR #6762
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint
uname Linux f76b4751795d 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / c2a0c61
Default Java Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6762/8/testReport/
Max. process+thread count 3863 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf hadoop-tools/hadoop-fs2img U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6762/8/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

ZanderXu and others added 21 commits January 7, 2025 11:10
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 42s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+0 🆗 xmllint 0m 1s xmllint was not available.
+0 🆗 markdownlint 0m 1s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 49 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 9m 18s Maven dependency ordering for branch
+1 💚 mvninstall 30m 41s trunk passed
+1 💚 compile 16m 43s trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 compile 14m 56s trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 checkstyle 4m 18s trunk passed
+1 💚 mvnsite 3m 10s trunk passed
+1 💚 javadoc 2m 59s trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 3m 24s trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 spotbugs 5m 47s trunk passed
+1 💚 shadedclient 33m 1s branch has no errors when building and testing our client artifacts.
-0 ⚠️ patch 33m 27s Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 32s Maven dependency ordering for patch
+1 💚 mvninstall 1m 55s the patch passed
+1 💚 compile 16m 10s the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javac 16m 10s the patch passed
+1 💚 compile 15m 16s the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 javac 15m 16s the patch passed
+1 💚 blanks 0m 1s The patch has no blanks issues.
+1 💚 checkstyle 4m 21s root: The patch generated 0 new + 1233 unchanged - 16 fixed = 1233 total (was 1249)
+1 💚 mvnsite 3m 11s the patch passed
+1 💚 javadoc 1m 18s hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 0 new + 3113 unchanged - 4 fixed = 3113 total (was 3117)
+1 💚 javadoc 0m 53s hadoop-hdfs-rbf in the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.
+1 💚 javadoc 0m 44s hadoop-fs2img in the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.
+1 💚 javadoc 3m 15s the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 spotbugs 6m 21s the patch passed
+1 💚 shadedclient 33m 51s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 230m 47s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
-1 ❌ unit 30m 59s /patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt hadoop-hdfs-rbf in the patch passed.
+1 💚 unit 0m 59s hadoop-fs2img in the patch passed.
+1 💚 asflicense 1m 14s The patch does not generate ASF License warnings.
483m 32s
Reason Tests
Failed junit tests hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
hadoop.hdfs.server.federation.store.driver.TestStateStoreFileSystem
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6762/9/artifact/out/Dockerfile
GITHUB PR #6762
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint
uname Linux 6a739942335f 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / e055021
Default Java Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6762/9/testReport/
Max. process+thread count 3949 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf hadoop-tools/hadoop-fs2img U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6762/9/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@kokon191
Copy link
Contributor

kokon191 commented Jan 8, 2025

Failed tests seem to be unrelated to FGL. In particular, hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped failure was caused by HDFS-17080 and hadoop.hdfs.server.federation.store.driver.TestStateStoreFileSystem by HDFS-17496

edit: created HDFS-17706 and HDFS-17707 to track the failed UTs

@ZanderXu ZanderXu changed the title HDFS-17384. [FGL] Replace the global lock with global FS Lock and global BM lock HDFS-17384. [FGL] HDFS NameNode FGL Phase I Jan 8, 2025
Copy link
Contributor

@haiyang1987 haiyang1987 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. +1.

@hfutatzhanghb
Copy link
Contributor

LGTM. +1.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 31s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 2s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 xmllint 0m 0s xmllint was not available.
+0 🆗 markdownlint 0m 0s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 49 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 6m 43s Maven dependency ordering for branch
+1 💚 mvninstall 30m 19s trunk passed
+1 💚 compile 16m 26s trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 compile 15m 5s trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 checkstyle 4m 26s trunk passed
+1 💚 mvnsite 3m 11s trunk passed
+1 💚 javadoc 3m 3s trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 3m 28s trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 spotbugs 5m 48s trunk passed
+1 💚 shadedclient 33m 32s branch has no errors when building and testing our client artifacts.
-0 ⚠️ patch 33m 59s Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 30s Maven dependency ordering for patch
+1 💚 mvninstall 1m 54s the patch passed
+1 💚 compile 15m 51s the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javac 15m 51s the patch passed
+1 💚 compile 15m 26s the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 javac 15m 26s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 4m 20s root: The patch generated 0 new + 1233 unchanged - 16 fixed = 1233 total (was 1249)
+1 💚 mvnsite 3m 12s the patch passed
+1 💚 javadoc 1m 21s hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 0 new + 3113 unchanged - 4 fixed = 3113 total (was 3117)
+1 💚 javadoc 0m 52s hadoop-hdfs-rbf in the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.
+1 💚 javadoc 0m 45s hadoop-fs2img in the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.
+1 💚 javadoc 3m 21s the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 spotbugs 6m 22s the patch passed
+1 💚 shadedclient 33m 16s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 232m 8s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
-1 ❌ unit 30m 58s /patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt hadoop-hdfs-rbf in the patch passed.
+1 💚 unit 0m 59s hadoop-fs2img in the patch passed.
+1 💚 asflicense 1m 12s The patch does not generate ASF License warnings.
481m 42s
Reason Tests
Failed junit tests hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
hadoop.hdfs.server.federation.store.driver.TestStateStoreFileSystem
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6762/10/artifact/out/Dockerfile
GITHUB PR #6762
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint
uname Linux 732af2e331f0 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / e055021
Default Java Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6762/10/testReport/
Max. process+thread count 3683 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf hadoop-tools/hadoop-fs2img U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6762/10/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants