Latest patch from Amith should address this issue.
I assume you're referring to
I've asked for a description before of how it solves the problem, because that was not obvious from the discussion nor from reading the diff. I'm disappointed that nobody responded to that request, so I've gone and read the patch in detail, and I think I can finally explain the approach 7.patch is using.
The problem 7.patch is solving is: currently when we finish scanning a BP, we unconditionally rotate the log. We keep two logs, the previous and the current logs. We sometimes re-scan a BP before the scanPeriod completes. If rescan happens twice within a single scanPeriod, the logs will rotate away and we will forget which blocks we previously scanned, so we will scan the first blocks again.
To fix this, 7.patch delays rotating the logs until the log has reached a predetermined size, rather than rotating when the scan completes.
+ static final int verficationLogLimit = 5;
What does this constant do? It seems to govern the block verification log size, but I don't understand why we want to keep 5 log entries for every block in blockMap.
+ private static long BLOCK_SCAN_PERIOD_UNIT = 3600 * 1000;
- this.scanPeriod = hours * 3600 * 1000;
+ this.scanPeriod = hours * BLOCK_SCAN_PERIOD_UNIT;
I don't think adding a named constant here is an improvement, but if you feel that it helps, please use a more descriptive name for this constant, like MS_PER_HOUR or something similar.
Uma, Amith – Have you tested 7.patch with multiple block pools and a full cluster restart? I think the changed code will leave a dncp_block_verification_log.prev in multiple BP directories, and I suspect that the BLockPoolSliceScanner might resume from the wrong place if there are multiple verification_logs in the data directories.
Per Eli's request, I'm going to close this Jira as resolved by my one-line patch which resolves the "Block scanner runs too frequently" bug, and open a new Jira to track the "Block scanner repeatedly rescans blocks" bug which is addressed by