by Schischke, Johannes
<h2>Important notification by Red Hat</h2>
<p>All Red Hat Linux Enterprise 7.9 (RHEL 7.9) images which running a kernel version greater than <strong>3.10.0-1160.<code>10.1</code>.el7</strong> are affected by a kernel panic during execution of a <strong>pvmove</strong> command! </p>
<p>Please find the latest official information and workaround link down below:</p>
<blockquote>
<p>Bug <a href="https://access.redhat.com/solutions/5704731">1916407</a></p>
</blockquote>
<p>All public available <strong>OTC RHEL 7.9</strong> images - <strong>Enterprise_RedHat_7_latest</strong> and <strong>Enterprise_RedHat_7_prev</strong> - are currently affected by this bug. A patch is planned to be released in mid of <strong>March 2021</strong>!</p>
<h2>Customer advice</h2>
<div class="notices red">
<p>This advice is only valid for customers which currently use <strong>an existing VM</strong> with LVM or plan to use LVM with a <strong>new VM</strong>!</p>
</div>
<p><strong>Existing RHEL 7.9 customers with LVM usage</strong></p>
<p>We strongly recommend to <strong>not</strong> upgrade kernel version above <strong>3.10.0-1160.<code>10.1</code>.el7</strong> until a patch has been released. Currently this is the latest stable version declared by Red Hat Engineering Team.</p>
<p><strong>New RHEL 7.9 customers which plan to use LVM</strong></p>
<p>We strongly recommend to use an image with the following <strong>image-id</strong> and kernel version <strong>3.10.0-1160.<code>6.1</code>.el7</strong>:</p>
<blockquote>
<p>7a98484f-72a1-4d71-b087-b3d3bf7c4915</p>
</blockquote>
<h2>Background information</h2>
<p>Please find down below a use case which is causing the kernel panic with kernel greater version <strong>3.10.0-1160.<code>10.1</code>.el7</strong></p>
<ol>
<li>Create and attach two data disks (10GB)
<pre><code>/dev/vdb and /dev/vdc</code></pre></li>
<li>Create first physical volume
<pre><code>$ sudo pvcreate /dev/vdb</code></pre></li>
<li>Create a volume group name <strong>test</strong>
<pre><code>$ sudo vgcreate test /dev/vdb</code></pre></li>
<li>Create a logical volume <strong>testlv</strong> in volume group test
<pre><code>$ sudo lvcreate -l 2559 -n testlv test</code></pre></li>
<li>Create xfs file system on logical volume <strong>testlv</strong>
<pre><code>$ sudo mkfs.xfs /dev/test/testlv</code></pre></li>
<li>Mount
<pre><code>$ sudo mount /dev/test/testlv /mnt/</code></pre></li>
<li>Create a 4GB file
<pre><code>$ cd /mnt/
$ sudo dd if=/dev/urandom of=image count=1000 bs=4M status=progress</code></pre></li>
<li>Create second physical volume
<pre><code>$ sudo pvcreate /dev/vdc</code></pre></li>
<li>Extend volume group <strong>test</strong> by adding second physical volume
<pre><code>$ sudo vgextend test /dev/vdc</code></pre></li>
<li>Move physical volume
<pre><code>$ sudo pvmove /dev/vdb /dev/vdc</code></pre></li>
<li>Kernel panic
<pre><code>[ 115.326688] BUG: unable to handle kernel NULL pointer dereference at (null)
[ 115.328784] IP: [< (null)>] (null)
[ 115.330102] PGD 0
[ 115.330665] Oops: 0010 [#1] SMP
[ 115.331577] Modules linked in: dm_mirror dm_region_hash dm_log xfs dm_mod ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack libcrc32c ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc nfit libnvdimm iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw ppdev gf128mul glue_helper ablk_helper cryptd pcspkr virtio_balloon joydev i2c_piix4 parport_pc parport ip_tables ext4 mbcache jbd2 virtio_net net_failover virtio_console failover virtio_blk ata_generic pata_acpi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw virtio_pci virtio_ring virtio drm_panel_orientation_quirks floppy
[ 115.357024] CPU: 0 PID: 1884 Comm: systemd-udevd Not tainted 3.10.0-1160.15.2.el7.x86_64 #1
[ 115.359093] Hardware name: OpenStack Foundation OpenStack Nova, BIOS rel-1.10.2-0-g5f4c7b1-20181220_000000-szxrtosci10000 04/01/2014
[ 115.362025] task: ffff884d392e8000 ti: ffff884d39ed0000 task.ti: ffff884d39ed0000
[ 115.363865] RIP: 0010:[<0000000000000000>] [< (null)>] (null)
[ 115.365775] RSP: 0018:ffff884d39ed38a0 EFLAGS: 00010202
[ 115.367103] RAX: 0000000000000000 RBX: ffff884d39ed3988 RCX: ffff884d3dc249c0
[ 115.368904] RDX: ffff884d36648d88 RSI: ffff884d39ed38a8 RDI: ffff884d36b657c0
[ 115.370683] RBP: ffff884d39ed38f8 R08: 0000000000000000 R09: ffff884d3a3a7c08
[ 115.372449] R10: 0000000000000000 R11: ffff884d3a3a7c08 R12: 0000000000000000
[ 115.374240] R13: 0000000000010000 R14: ffff884d36648d88 R15: ffff884d3a3a7400
[ 115.376024] FS: 00007f738beba8c0(0000) GS:ffff884d3fc00000(0000) knlGS:0000000000000000
[ 115.378034] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 115.379470] CR2: 0000000000000000 CR3: 0000000039ece000 CR4: 00000000003606f0
[ 115.381255] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 115.383013] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 115.384766] Call Trace:
[ 115.385380] [<ffffffffc07e4155>] ? mirror_merge+0xb5/0x100 [dm_mirror]
[ 115.386995] [<ffffffffc07be847>] dm_merge_bvec+0x87/0xd0 [dm_mod]
[ 115.388533] [<ffffffffc07c57b4>] linear_merge+0x54/0x60 [dm_mod]
[ 115.390056] [<ffffffffc07be847>] dm_merge_bvec+0x87/0xd0 [dm_mod]
[ 115.391604] [<ffffffffb568b59b>] __bio_add_page+0x20b/0x2b0
[ 115.393034] [<ffffffffb568b6cd>] bio_add_page+0x6d/0x70
[ 115.394931] [<ffffffffb56944a0>] do_mpage_readpage+0x2f0/0x760
[ 115.396918] [<ffffffffb568ded0>] ? set_init_blocksize+0x90/0x90
[ 115.398908] [<ffffffffb5694a22>] mpage_readpages+0x112/0x170
[ 115.400832] [<ffffffffb568ded0>] ? set_init_blocksize+0x90/0x90
[ 115.402903] [<ffffffffb568ded0>] ? set_init_blocksize+0x90/0x90
[ 115.404909] [<ffffffffb568e6cd>] blkdev_readpages+0x1d/0x20
[ 115.406808] [<ffffffffb55cb3bf>] __do_page_cache_readahead+0x1cf/0x260
[ 115.408930] [<ffffffffb55cb909>] force_page_cache_readahead+0x99/0xe0
[ 115.411017] [<ffffffffb55cb99b>] page_cache_sync_readahead+0x4b/0x80
[ 115.413194] [<ffffffffb55bf0cc>] do_generic_file_read.constprop.52+0xdc/0x510
[ 115.415432] [<ffffffffb5640fc6>] ? mem_cgroup_update_page_stat+0x16/0x50
[ 115.417556] [<ffffffffb55bfea9>] generic_file_aio_read+0x1c9/0x290
[ 115.419537] [<ffffffffb55ee659>] ? do_read_fault.isra.63+0x139/0x1b0
[ 115.421580] [<ffffffffb568eadc>] blkdev_aio_read+0x4c/0x70
[ 115.423401] [<ffffffffb564d163>] do_sync_read+0x93/0xe0
[ 115.425212] [<ffffffffb564db9f>] vfs_read+0x9f/0x170
[ 115.426929] [<ffffffffb564ea1f>] SyS_read+0x7f/0xf0
[ 115.428639] [<ffffffffb5b94f92>] system_call_fastpath+0x25/0x2a
[ 115.430517] Code: Bad RIP value.
[ 115.431864] RIP [< (null)>] (null)
[ 115.433572] RSP <ffff884d39ed38a0>
[ 115.434829] CR2: 0000000000000000
[ 115.436058] ---[ end trace fe9c90f8ddf27049 ]---
[ 115.437580] Kernel panic - not syncing: Fatal exception
[ 115.439790] Kernel Offset: 0x34400000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)</code></pre></li>
</ol>