Last week I scheduled a 2 hour downtime to remount some large JFS2 file systems. When I inherited this server a year ago most of the data was contained in a few 32TB JFS2 filesystems. Fearing a week long fsck looming, I began breaking these in to smaller, more manageable chunks of no more than 6TB each.

When my downtime began I unmounted my filesystems, and issued the mount commands to make them available again. As luck would have it, all 3 needed to be fscked before they would mount.

The first was a brand new jfs2 filesystem 4TB. Fsck is frustrating across platforms because there is no clear indication of how long it will take. Luckily this first filesystem check was less than 10 minutes.

I'm glad I did because last week I ended up with required fsck before I was able to mount 3 of them.
I spent a nervous 2 hours last week fscking some large (6TB) JFS2 filesystems

The current volume is: /dev/img3lv
Primary superblock is valid.
J2_LOGREDO:log redo processing for /dev/img3lv
j2_logredo.cpp: ERROR in j2_logredo line 672, log already redone!


Primary superblock is valid.
*** Phase 1 - Initial inode scan
*** Phase 2 - Process remaining directories
*** Phase 3 - Process remaining files
*** Phase 4 - Check and repair inode allocation map
*** Phase 5 - Check and repair block allocation map
File system is clean.
Superblock is marked dirty; FIX? y
All observed inconsistencies have been repaired.


I then went on to mount img4 and img5 and discovered both needed fscking. Both had slightly different error output when I tried the mount command. Unfortunately I did not capture the exact message.

I ran fsck concurrently on each file system. Each one took an hour to complete. Img5 finished first and displayed a few errors.


The current volume is: /dev/img5lv
Primary superblock is valid.
J2_LOGREDO:log redo processing for /dev/img5lv
j2_logredo.cpp: ERROR in j2_logredo line 672, log already redone!


Primary superblock is valid.
*** Phase 1 - Initial inode scan
*** Phase 2 - Process remaining directories
Inode 541893 has incorrect link count; FIX? y
*** Phase 3 - Process remaining files
*** Phase 4 - Check and repair inode allocation map
File system inode map is corrupt; FIX? y
Superblock marked dirty because repairs are about to be written.
*** Phase 5 - Check and repair block allocation map
Inodes not connected to the root directory
tree have been detected. Will reconnect.
File system is dirty.


I was concerned that although fsck had completed, img5 was still dirty and could not be mounted.

A few minutes later img4 completed. It was also dirty, but offered a useful message.

The current volume is: /dev/img4lv
Primary superblock is valid.
J2_LOGREDO:log redo processing for /dev/img4lv
j2_logredo.cpp: ERROR in j2_logredo line 672, log already redone!


Primary superblock is valid.
*** Phase 1 - Initial inode scan
*** Phase 2 - Process remaining directories
*** Phase 3 - Process remaining files
*** Phase 4 - Check and repair inode allocation map
*** Phase 5 - Check and repair block allocation map
File system is clean.
Superblock is marked dirty; FIX? y
File system is clean but is marked dirty.
Run fsck with -p parameter to fix.


Only one hour left in my maintenance window I concurrently ran fsck again on each file system, this time using the -p (fix minor problems) and -y flags. It took another hour but this time all inconsistencies were repaired.


# fsck -p -y /dev/img4lv


The current volume is: /dev/img4lv
Primary superblock is valid.
J2_LOGREDO:log redo processing for /dev/img4lv
j2_logredo.cpp: ERROR in j2_logredo line 672, log already redone!


Primary superblock is valid.
*** Phase 1 - Initial inode scan
*** Phase 2 - Process remaining directories
*** Phase 3 - Process remaining files
*** Phase 4 - Check and repair inode allocation map
*** Phase 5 - Check and repair block allocation map
File system is clean.
Superblock is marked dirty (FIXED)
All observed inconsistencies have been repaired.


# fsck -p -y /dev/img5lv


The current volume is: /dev/img5lv
Primary superblock is valid.
J2_LOGREDO:log redo processing for /dev/img5lv
j2_logredo.cpp: ERROR in j2_logredo line 672, log already redone!


Primary superblock is valid.
*** Phase 1 - Initial inode scan
*** Phase 2 - Process remaining directories
Inode 541893 has incorrect link count (FIXED)
Superblock marked dirty because repairs are about to be written.
*** Phase 3 - Process remaining files
*** Phase 4 - Check and repair inode allocation map
*** Phase 5 - Check and repair block allocation map
File system is clean.
Superblock is marked dirty (FIXED)
All observed inconsistencies have been repaired


In all cases the 'Phase 1' message marked about 90% of the time. Phase 2,3,4,5 completed much more quickly.