HBASE-30049 RestoreSnapshotHelper creates StoreFileTracker with wrong…#8013
HBASE-30049 RestoreSnapshotHelper creates StoreFileTracker with wrong…#8013bjomobo wants to merge 2 commits into
Conversation
bjomobo
force-pushed
the
HBASE-30049-fix-restore-sft
branch
2 times, most recently
from
e3e3c55 to
6af91ac
Compare
… config causing no-op filelist updates RestoreSnapshotHelper.restoreRegion() creates a StoreFileTracker using the raw Master Configuration object, which does not contain table-level settings like hbase.store.file-tracker.impl=FILE. This causes DefaultStoreFileTracker to be instantiated, whose doSetStoreFiles() is a complete no-op. The .filelist is never updated after the restore moves HFiles to the archive and creates link files for the snapshot's HFiles. When a region subsequently opens, the stale .filelist references HFiles that were moved to the archive, resulting in FileNotFoundException and the region getting stuck in OPENING state indefinitely. This is a regression introduced by HBASE-28564, which refactored reference file creation to go through the StoreFileTracker interface. The cloneRegion() method in the same commit correctly merges the table descriptor config via StoreUtils.createStoreConfiguration() before creating the tracker, but restoreRegion() was missed. The fix applies the same pattern: merge the table descriptor and column family descriptor configuration into the Configuration object before passing it to StoreFileTrackerFactory.create(). This ensures the correct StoreFileTracker implementation is resolved based on the table-level setting. Both locations in restoreRegion() are fixed: 1. For existing families already on disk 2. For new families added from the snapshot
bjomobo
force-pushed
the
HBASE-30049-fix-restore-sft
branch
from
6af91ac to
282757c
Compare
| } | ||
|
|
||
| /** | ||
| * Test that reproduces the scenario where restore fails with FileNotFoundException: 1. Create |
There was a problem hiding this comment.
Looks like this test and TestRestoreSnapshotFileTrackerTableLevel.java have some overlapping test scenarios, can you check once, or am I missing something here?
There was a problem hiding this comment.
TestRestoreSnapshotProcedureFileBasedSFT tests the end-to-end procedure path (single region, simple data change + compaction scenarios), while TestRestoreSnapshotFileTrackerTableLevel uses pre-split regions (4 regions), proper compaction wait via UTIL.waitFor(), and covers the multi-family restore path (add/remove column families). Happy to consolidate if you feel strongly, but I kept them separate since they exercise slightly different things
There was a problem hiding this comment.
admin.restoreSnapshot(snapshotName); in TestRestoreSnapshotFileTrackerTableLevel also creates procedure know? but its a nit only
| } | ||
|
|
||
| /** | ||
| * Test that reproduces the scenario where restore fails with FileNotFoundException: 1. Create |
There was a problem hiding this comment.
admin.restoreSnapshot(snapshotName); in TestRestoreSnapshotFileTrackerTableLevel also creates procedure know? but its a nit only
| // Family exists in the snapshot, create tracker with merged table descriptor config | ||
| // so that table-level settings are picked up. |
There was a problem hiding this comment.
nit: Let's make clear that tableDesc here means the one recovered from the snapshot manifest. This means if current SFT config in the table is different from the snapshot one, it will be overwritten.
|
@bjomobo , looks like this needs some code style polishing:
Can you check on the spotless, checkstyles and spotbugs errors? |
3 participants
Description
RestoreSnapshotHelper.restoreRegion() creates a StoreFileTracker using the raw Master conf which lacks table-level settings like
hbase.store.file-tracker.impl=FILE. This causes DefaultStoreFileTracker to be used, whosedoSetStoreFiles()is a no-op. The.filelistis never updated after restore, leading toFileNotFoundExceptionwhen regions try to open files that were archived.Regression introduced by HBASE-28564.
Changes
StoreUtils.createStoreConfiguration()before creating the tracker inrestoreRegion()snapshotFamilyFiles != nullcheck to avoidNullPointerExceptionon families being removedwithColumnFamilyDescriptor()to the "Add families not present in the table" code pathNew Tests
TestRestoreSnapshotProcedureFileBasedSFT— end-to-end restore with FILE trackerTestRestoreSnapshotHelperWithFileBasedSFT— unit-level.filelistverificationTestRestoreSnapshotFileTrackerTableLevel— table-level FILE tracker with compaction and multi-family restoreJira: https://issues.apache.org/jira/browse/HBASE-30049