Known LVM issue - Kernel panic Red Hat Linux Enterprise 7.9
Important notification by Red Hat
All Red Hat Linux Enterprise 7.9 (RHEL 7.9) images which running a kernel version greater than 3.10.0-1160.10.1.el7 are affected by a kernel panic during execution of a pvmove command!
Please find the latest official information and workaround link down below:
Bug 1916407
All public available OTC RHEL 7.9 images - Enterprise_RedHat_7_latest and Enterprise_RedHat_7_prev - are currently affected by this bug. A patch is planned to be released in mid of March 2021!
Customer advice
This advice is only valid for customers which currently use an existing VM with LVM or plan to use LVM with a new VM!
Existing RHEL 7.9 customers with LVM usage
We strongly recommend to not upgrade kernel version above 3.10.0-1160.10.1.el7 until a patch has been released. Currently this is the latest stable version declared by Red Hat Engineering Team.
New RHEL 7.9 customers which plan to use LVM
We strongly recommend to use an image with the following image-id and kernel version 3.10.0-1160.6.1.el7:
7a98484f-72a1-4d71-b087-b3d3bf7c4915
Background information
Please find down below a use case which is causing the kernel panic with kernel greater version 3.10.0-1160.10.1.el7
- Create and attach two data disks (10GB)
/dev/vdb and /dev/vdc - Create first physical volume
$ sudo pvcreate /dev/vdb - Create a volume group name test
$ sudo vgcreate test /dev/vdb - Create a logical volume testlv in volume group test
$ sudo lvcreate -l 2559 -n testlv test - Create xfs file system on logical volume testlv
$ sudo mkfs.xfs /dev/test/testlv - Mount
$ sudo mount /dev/test/testlv /mnt/ - Create a 4GB file
$ cd /mnt/ $ sudo dd if=/dev/urandom of=image count=1000 bs=4M status=progress - Create second physical volume
$ sudo pvcreate /dev/vdc - Extend volume group test by adding second physical volume
$ sudo vgextend test /dev/vdc - Move physical volume
$ sudo pvmove /dev/vdb /dev/vdc - Kernel panic
[ 115.326688] BUG: unable to handle kernel NULL pointer dereference at (null) [ 115.328784] IP: [< (null)>] (null) [ 115.330102] PGD 0 [ 115.330665] Oops: 0010 [#1] SMP [ 115.331577] Modules linked in: dm_mirror dm_region_hash dm_log xfs dm_mod ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack libcrc32c ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc nfit libnvdimm iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw ppdev gf128mul glue_helper ablk_helper cryptd pcspkr virtio_balloon joydev i2c_piix4 parport_pc parport ip_tables ext4 mbcache jbd2 virtio_net net_failover virtio_console failover virtio_blk ata_generic pata_acpi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw virtio_pci virtio_ring virtio drm_panel_orientation_quirks floppy [ 115.357024] CPU: 0 PID: 1884 Comm: systemd-udevd Not tainted 3.10.0-1160.15.2.el7.x86_64 #1 [ 115.359093] Hardware name: OpenStack Foundation OpenStack Nova, BIOS rel-1.10.2-0-g5f4c7b1-20181220_000000-szxrtosci10000 04/01/2014 [ 115.362025] task: ffff884d392e8000 ti: ffff884d39ed0000 task.ti: ffff884d39ed0000 [ 115.363865] RIP: 0010:[<0000000000000000>] [< (null)>] (null) [ 115.365775] RSP: 0018:ffff884d39ed38a0 EFLAGS: 00010202 [ 115.367103] RAX: 0000000000000000 RBX: ffff884d39ed3988 RCX: ffff884d3dc249c0 [ 115.368904] RDX: ffff884d36648d88 RSI: ffff884d39ed38a8 RDI: ffff884d36b657c0 [ 115.370683] RBP: ffff884d39ed38f8 R08: 0000000000000000 R09: ffff884d3a3a7c08 [ 115.372449] R10: 0000000000000000 R11: ffff884d3a3a7c08 R12: 0000000000000000 [ 115.374240] R13: 0000000000010000 R14: ffff884d36648d88 R15: ffff884d3a3a7400 [ 115.376024] FS: 00007f738beba8c0(0000) GS:ffff884d3fc00000(0000) knlGS:0000000000000000 [ 115.378034] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 115.379470] CR2: 0000000000000000 CR3: 0000000039ece000 CR4: 00000000003606f0 [ 115.381255] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 115.383013] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 115.384766] Call Trace: [ 115.385380] [<ffffffffc07e4155>] ? mirror_merge+0xb5/0x100 [dm_mirror] [ 115.386995] [<ffffffffc07be847>] dm_merge_bvec+0x87/0xd0 [dm_mod] [ 115.388533] [<ffffffffc07c57b4>] linear_merge+0x54/0x60 [dm_mod] [ 115.390056] [<ffffffffc07be847>] dm_merge_bvec+0x87/0xd0 [dm_mod] [ 115.391604] [<ffffffffb568b59b>] __bio_add_page+0x20b/0x2b0 [ 115.393034] [<ffffffffb568b6cd>] bio_add_page+0x6d/0x70 [ 115.394931] [<ffffffffb56944a0>] do_mpage_readpage+0x2f0/0x760 [ 115.396918] [<ffffffffb568ded0>] ? set_init_blocksize+0x90/0x90 [ 115.398908] [<ffffffffb5694a22>] mpage_readpages+0x112/0x170 [ 115.400832] [<ffffffffb568ded0>] ? set_init_blocksize+0x90/0x90 [ 115.402903] [<ffffffffb568ded0>] ? set_init_blocksize+0x90/0x90 [ 115.404909] [<ffffffffb568e6cd>] blkdev_readpages+0x1d/0x20 [ 115.406808] [<ffffffffb55cb3bf>] __do_page_cache_readahead+0x1cf/0x260 [ 115.408930] [<ffffffffb55cb909>] force_page_cache_readahead+0x99/0xe0 [ 115.411017] [<ffffffffb55cb99b>] page_cache_sync_readahead+0x4b/0x80 [ 115.413194] [<ffffffffb55bf0cc>] do_generic_file_read.constprop.52+0xdc/0x510 [ 115.415432] [<ffffffffb5640fc6>] ? mem_cgroup_update_page_stat+0x16/0x50 [ 115.417556] [<ffffffffb55bfea9>] generic_file_aio_read+0x1c9/0x290 [ 115.419537] [<ffffffffb55ee659>] ? do_read_fault.isra.63+0x139/0x1b0 [ 115.421580] [<ffffffffb568eadc>] blkdev_aio_read+0x4c/0x70 [ 115.423401] [<ffffffffb564d163>] do_sync_read+0x93/0xe0 [ 115.425212] [<ffffffffb564db9f>] vfs_read+0x9f/0x170 [ 115.426929] [<ffffffffb564ea1f>] SyS_read+0x7f/0xf0 [ 115.428639] [<ffffffffb5b94f92>] system_call_fastpath+0x25/0x2a [ 115.430517] Code: Bad RIP value. [ 115.431864] RIP [< (null)>] (null) [ 115.433572] RSP <ffff884d39ed38a0> [ 115.434829] CR2: 0000000000000000 [ 115.436058] ---[ end trace fe9c90f8ddf27049 ]--- [ 115.437580] Kernel panic - not syncing: Fatal exception [ 115.439790] Kernel Offset: 0x34400000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)