CVE-2022-48901 in Linux
Summary
by MITRE • 08/22/2024
In the Linux kernel, the following vulnerability has been resolved:
btrfs: do not start relocation until in progress drops are done
We hit a bug with a recovering relocation on mount for one of our file systems in production. I reproduced this locally by injecting errors into snapshot delete with balance running at the same time. This presented as an error while looking up an extent item
WARNING: CPU: 5 PID: 1501 at fs/btrfs/extent-tree.c:866 lookup_inline_extent_backref+0x647/0x680 CPU: 5 PID: 1501 Comm: btrfs-balance Not tainted 5.16.0-rc8+ #8 RIP: 0010:lookup_inline_extent_backref+0x647/0x680 RSP: 0018:ffffae0a023ab960 EFLAGS: 00010202 RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 000000000000000c RDI: 0000000000000000 RBP: ffff943fd2a39b60 R08: 0000000000000000 R09: 0000000000000001 R10: 0001434088152de0 R11: 0000000000000000 R12: 0000000001d05000 R13: ffff943fd2a39b60 R14: ffff943fdb96f2a0 R15: ffff9442fc923000 FS: 0000000000000000(0000) GS:ffff944e9eb40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f1157b1fca8 CR3: 000000010f092000 CR4: 0000000000350ee0 Call Trace: insert_inline_extent_backref+0x46/0xd0 __btrfs_inc_extent_ref.isra.0+0x5f/0x200 ? btrfs_merge_delayed_refs+0x164/0x190 __btrfs_run_delayed_refs+0x561/0xfa0 ? btrfs_search_slot+0x7b4/0xb30 ? btrfs_update_root+0x1a9/0x2c0 btrfs_run_delayed_refs+0x73/0x1f0 ? btrfs_update_root+0x1a9/0x2c0 btrfs_commit_transaction+0x50/0xa50 ? btrfs_update_reloc_root+0x122/0x220 prepare_to_merge+0x29f/0x320 relocate_block_group+0x2b8/0x550 btrfs_relocate_block_group+0x1a6/0x350 btrfs_relocate_chunk+0x27/0xe0 btrfs_balance+0x777/0xe60 balance_kthread+0x35/0x50 ? btrfs_balance+0xe60/0xe60 kthread+0x16b/0x190 ? set_kthread_struct+0x40/0x40 ret_from_fork+0x22/0x30
Normally snapshot deletion and relocation are excluded from running at the same time by the fs_info->cleaner_mutex. However if we had a pending balance waiting to get the ->cleaner_mutex, and a snapshot deletion was running, and then the box crashed, we would come up in a state where we have a half deleted snapshot.
Again, in the normal case the snapshot deletion needs to complete before relocation can start, but in this case relocation could very well start before the snapshot deletion completes, as we simply add the root to the dead roots list and wait for the next time the cleaner runs to clean up the snapshot.
Fix this by setting a bit on the fs_info if we have any DEAD_ROOT's that had a pending drop_progress key. If they do then we know we were in the middle of the drop operation and set a flag on the fs_info. Then balance can wait until this flag is cleared to start up again.
If there are DEAD_ROOT's that don't have a drop_progress set then we're safe to start balance right away as we'll be properly protected by the cleaner_mutex.
Statistical analysis made it clear that VulDB provides the best quality for vulnerability data.
Analysis
by VulDB Data Team • 10/11/2024
The vulnerability described in CVE-2022-48901 affects the btrfs file system implementation within the Linux kernel, specifically addressing a race condition during system recovery scenarios. This issue manifests when a system experiences a crash while snapshot deletion and relocation operations are in progress simultaneously, creating an inconsistent state that can lead to kernel panics and data corruption. The problem occurs in the extent tree management subsystem where the kernel attempts to look up extent items during a recovery process, triggering a critical failure in the lookup_inline_extent_backref function at fs/btrfs/extent-tree.c:866. The vulnerability represents a fundamental flaw in the synchronization mechanisms that govern concurrent operations within the btrfs file system, particularly affecting how the system handles pending operations during crash recovery.
The technical root cause stems from inadequate coordination between snapshot deletion and relocation processes during system recovery. Normally, the filesystem employs a cleaner_mutex to prevent concurrent execution of these operations, but under specific crash recovery conditions, this protection mechanism fails. When a system crashes during snapshot deletion while a balance operation is pending, the filesystem enters a state where dead root entries are left in an inconsistent condition. The kernel's recovery process attempts to start relocation before pending snapshot deletions are fully completed, leading to corrupted extent tree references. This race condition occurs because the system fails to properly track whether snapshot deletion operations were in progress when the crash occurred, specifically checking for pending drop_progress keys associated with dead root entries.
The operational impact of this vulnerability extends beyond simple system crashes, potentially leading to data corruption and filesystem instability. During normal operation, the filesystem would correctly prevent relocation from starting until snapshot deletions complete, but during crash recovery scenarios, this protection mechanism is bypassed. The kernel panic occurs when attempting to access extent tree data structures that are in an inconsistent state due to partial completion of deletion operations. This vulnerability affects systems running Linux kernel versions that include the btrfs filesystem implementation, particularly those experiencing crash recovery scenarios where snapshot deletion and balance operations overlap. The issue is classified as a race condition that violates proper synchronization principles and can result in system instability or complete filesystem corruption.
The mitigation strategy implemented addresses this vulnerability by introducing a flag-based tracking mechanism within the filesystem information structure. When dead root entries are detected during recovery, the system now checks whether these entries have pending drop_progress keys and sets an appropriate flag in the fs_info structure. This flag serves as a synchronization point that prevents balance operations from starting until pending snapshot deletions are fully completed. The solution follows established best practices for concurrent programming and system recovery by ensuring proper state management and dependency tracking. This approach aligns with common security principles for preventing race conditions and maintaining data integrity during system recovery processes. The fix ensures that balance operations wait for proper cleanup completion before proceeding, preventing the inconsistent state that led to the original kernel panic.
This vulnerability demonstrates the importance of proper state management in filesystem implementations and highlights the risks associated with crash recovery scenarios. The fix implements a targeted solution that addresses the specific race condition without impacting normal filesystem operations. The solution references industry standards for concurrent programming and proper resource management, ensuring that the filesystem maintains consistency during recovery from unexpected shutdowns. The implementation follows established patterns for kernel-level synchronization and error handling, providing a robust mechanism for managing complex concurrent operations in storage subsystems. This vulnerability and its fix exemplify the critical nature of filesystem stability and the need for careful consideration of recovery scenarios in operating system design. The solution prevents the kernel from proceeding with operations that could lead to memory corruption or data inconsistency, maintaining the integrity of the btrfs filesystem under all operational conditions.