Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

regression on linux 6.3.1: 'vmalloc error' during crawl #257

Open
tytgatlieven opened this issue May 12, 2023 · 16 comments
Open

regression on linux 6.3.1: 'vmalloc error' during crawl #257

tytgatlieven opened this issue May 12, 2023 · 16 comments

Comments

@tytgatlieven
Copy link

Dear
I upgraded my system from linux 6.2.13 to 6.3.1. This resulted in the error messages as below in my logs. There are no crashes.

This has also been reported here

In the reply to above it is mentioned that a out-of-memory condition could trigger the issue. In my setup I have about 30GB of free RAM memory, so this shouldn't be the case.

Feel free to contact me for more info/tests.

crawl_4157_6261[2629]: WORKAROUND: abandoned toxic match for hash 0x3b0cd01625bf10af addr 0x71c0f81d000tz1c000 matching bbd BeesBlockData { 4K 0x4e43000 fd = 14 'FILEPATH1', address = 0x35010ff4000z3000, hash = 0x3b0cd01625bf10af, data[4096] }
mei 12 15:04:22 ltytgat-desktop beesd[2174]: crawl_24398_6380[2631]: WORKAROUND: abandoned toxic match for hash 0x37c47b46be0dbd80 addr 0x63c37a49000t matching bbd BeesBlockData { 4K 0x71a7000 fd = 25 'FILEPATH2', address = 0x94fd5d4000z7000, hash = 0x37c47b46be0dbd80, data[4096] }
mei 12 15:04:22 ltytgat-desktop kernel: warn_alloc: 37 callbacks suppressed
mei 12 15:04:22 ltytgat-desktop kernel: crawl_4157_6261: vmalloc error: size 8388608, page order 9, failed to allocate pages, mode:0xcc2(GFP_KERNEL|__GFP_HIGHMEM), nodemask=(null),cpuset=system-beesd.slice,mems_allowed=0
mei 12 15:04:22 ltytgat-desktop kernel: CPU: 0 PID: 2629 Comm: crawl_4157_6261 Tainted: G           OE      6.3.1-060301-generic #202304302031
mei 12 15:04:22 ltytgat-desktop kernel: Hardware name: MSI MS-7760/X79A-GD45 Plus (MS-7760), BIOS V17.9 12/08/2014
mei 12 15:04:22 ltytgat-desktop kernel: Call Trace:
mei 12 15:04:22 ltytgat-desktop kernel:  <TASK>
mei 12 15:04:22 ltytgat-desktop kernel:  dump_stack_lvl+0x48/0x70
mei 12 15:04:22 ltytgat-desktop kernel:  dump_stack+0x10/0x20
mei 12 15:04:22 ltytgat-desktop kernel:  warn_alloc+0x174/0x1f0
mei 12 15:04:22 ltytgat-desktop kernel:  __vmalloc_area_node+0x4f7/0x610
mei 12 15:04:22 ltytgat-desktop kernel:  __vmalloc_node_range+0xda/0x220
mei 12 15:04:22 ltytgat-desktop kernel:  kvmalloc_node+0xbf/0x100
mei 12 15:04:22 ltytgat-desktop kernel:  ? init_data_container+0x2a/0x70 [btrfs]
mei 12 15:04:22 ltytgat-desktop kernel:  init_data_container+0x2a/0x70 [btrfs]
mei 12 15:04:22 ltytgat-desktop kernel:  btrfs_ioctl_logical_to_ino+0xdf/0x190 [btrfs]
mei 12 15:04:22 ltytgat-desktop kernel:  btrfs_ioctl+0xb3f/0x14f0 [btrfs]
mei 12 15:04:22 ltytgat-desktop kernel:  ? __x64_sys_futex+0x95/0x200
mei 12 15:04:22 ltytgat-desktop kernel:  ? __fget_light+0xa5/0x120
mei 12 15:04:22 ltytgat-desktop kernel:  __x64_sys_ioctl+0xa0/0xe0
mei 12 15:04:22 ltytgat-desktop kernel:  do_syscall_64+0x5b/0x90
mei 12 15:04:22 ltytgat-desktop kernel:  ? exit_to_user_mode_prepare+0x42/0x110
mei 12 15:04:22 ltytgat-desktop kernel:  ? syscall_exit_to_user_mode+0x1b/0x50
mei 12 15:04:22 ltytgat-desktop kernel:  ? do_syscall_64+0x67/0x90
mei 12 15:04:22 ltytgat-desktop kernel:  ? do_syscall_64+0x67/0x90
mei 12 15:04:22 ltytgat-desktop kernel:  entry_SYSCALL_64_after_hwframe+0x72/0xdc
mei 12 15:04:22 ltytgat-desktop kernel: RIP: 0033:0x7f6f7e31aaff
mei 12 15:04:22 ltytgat-desktop kernel: Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <41> 89 c0 3d 00 f0 ff ff 77 1f 48 8b 44 24 18 64 48 2b 04 25 28 00
mei 12 15:04:22 ltytgat-desktop kernel: RSP: 002b:00007f6f7b1f5150 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
mei 12 15:04:22 ltytgat-desktop kernel: RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00007f6f7e31aaff
mei 12 15:04:22 ltytgat-desktop kernel: RDX: 00007f6f7b1f51d0 RSI: 00000000c038943b RDI: 0000000000000003
mei 12 15:04:22 ltytgat-desktop kernel: RBP: 00007f6b54008fe0 R08: 0000555dd60e8408 R09: 0000000000000000
mei 12 15:04:22 ltytgat-desktop kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00007f6f7b1f54f0
mei 12 15:04:22 ltytgat-desktop kernel: R13: 00007f6f7b1f5888 R14: 0000555dd7bb7ad0 R15: 00007f6f7b1f5620
mei 12 15:04:22 ltytgat-desktop kernel:  </TASK>
mei 12 15:04:22 ltytgat-desktop kernel: Mem-Info:
mei 12 15:04:22 ltytgat-desktop kernel: active_anon:481898 inactive_anon:401017 isolated_anon:0
                                         active_file:554499 inactive_file:173686 isolated_file:0
                                         unevictable:5242926 dirty:11851 writeback:696
                                         slab_reclaimable:1069472 slab_unreclaimable:327280
                                         mapped:115937 shmem:26784 pagetables:32053
                                         sec_pagetables:0 bounce:0
                                         kernel_misc_reclaimable:0
                                         free:5565117 free_pcp:1937 free_cma:0

@kakra
Copy link
Contributor

kakra commented May 15, 2023

Show cat /proc/buddyinfo before and after the problem occurs. I'm seeing similar behavior since some of the latest updates for kernel 6.1. While I don't get the exact same behavior, I'm seeing the kernel generating memory pressure, flushing cache and sending memory to swap despite multiple gigabytes of memory announced as free, resulting in IO thrashing.

buddyinfo shows how many pages of which size are free (column 1 = order 0 = 4k * 2^0, column 2 = 4k * 2^1 = 8k, then 16k, 32k...), and if the kernel cannot find a free block that can hold the allocation, it will swap data out or fail the allocation. This is because the kernel itself allocates physical pages not virtual memory (thus, it cannot split big blocks of allocations and merge them virtually into one continuous range of memory: if something doesn't fit, it doesn't fit, this is called memory fragmentation).

Disabling the memory cgroup controller (and using generational LRU instead) somewhat mitigated that for me, setting transparent huge pages to madvise also helped, but it is still not fully fixed for me.

What you want to see in buddyinfo is high numbers in the high order columns. If it peaks in the low order columns, the kernel should (and will) try to compact movable pages into larger blocks, thus defragmenting free space. This can take a few seconds, so you may want to take multiple snapshots of buddyinfo. If pages do not migrate to higher order free space, you should check if your system has a lot of non-movable pages or huge pages.

User-space pages are movable (because they are addressed indirectly through page table lookups). Buffers for hardware are usually not movable. Page cache is, I think, neither movable but reclaimable.

Your error indicates it's trying to get an order 9 allocation (4k * 2^9 = 2M) so buddyinfo probably doesn't have anything beyond the 9th column.

Something bad is going on with memory allocations in the kernel since 6.1.

@tytgatlieven
Copy link
Author

Sorry for the delay.
It is hard to accurately snap the buddyinfo before and after the event. I have done cat /proc/buddyinfo continuously while doing a tail -f on syslog and this is the buddyinfo before and after the error. Is there a more accurate way of getting this info?

/proc/buddyinfo before:
Node 0, zone DMA 0 0 0 0 0 0 0 0 1 1 2
Node 0, zone DMA32 61 50 341 879 741 548 368 171 85 22 1
Node 0, zone Normal 51878 58470 54194 56426 40574 24602 14169 8418 9425 0 0

/proc/buddyinfo after:
Node 0, zone DMA 0 0 0 0 0 0 0 0 1 1 2
Node 0, zone DMA32 62 48 341 877 740 549 367 171 84 7 0
Node 0, zone Normal 60753 58348 54154 56373 40536 24567 14150 8407 9409 5 0

@kakra
Copy link
Contributor

kakra commented May 26, 2023

That indicates that both before and after the event, the memory is already very fragmented - and that it is fragmented before is probably why it is happening in the first place.

Could you look at it after a fresh reboot, then look how it develops while using the system? Maybe you can identify an action or behavior on your system that is causing this behavior.

As a first counter measure you could try disabling huge pages after a fresh reboot (while buddyinfo still shows low numbers on the left side and high numbers on the right side):

echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
echo always | /sys/kernel/mm/transparent_hugepage/defrag

If this helps but you feel like you want to use huge pages (because it lowers TLB cache misses and can increase performance for some workloads by up to 10%), try this as a next step (I am using these settings, it causes around 1 GB of unused memory on my desktop system under memory pressure when memory is partially fragmented, instead of 4-8 GB with huge pages always turned on):

echo 64 | sudo tee /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none
echo 8 | sudo tee /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_swap
echo 32 | sudo tee /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_shared

echo within_size | sudo tee /sys/kernel/mm/transparent_hugepage/shmem_enabled
echo defer+madvise | sudo tee /sys/kernel/mm/transparent_hugepage/defrag

echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled

It tells transparent huge pages to only create huge pages for madvise memory regions (thus, when an application explicitly asks for it, bees does this for the hash table). It also tells to defer defragmenting huge pages for better latency (but this tends to delay seeing the immediate effects of bad memory layout). Depending on your workload, you may have better results with defrag = always at the cost of higher memory allocation latency (fine for servers, not so for desktops). within_size tells to use huge pages only for shared memory if the allocation is at least 2 MB. Again, you may have better result when setting it to never.

The max_ptes_{none,swap,shared} tell the defragger when to combine 4k pages into huge pages during compaction: Only if less than 64x 4k pages of a 1x 2M page candidate are NOT YET allocated (max_ptes_none), compaction will combine those pages into one 2M page (sacrificing up to 64*4=256 kB of RAM). Similar for swap: Compaction to 2M will only occur if no more than 8 pages must be swapped in. And for shared: Compaction to 2M will only occur, if no more than 32x 4k pages would be unshared in the process.

You can cat each sysfs file to see the current and possible settings so you can experiment with it.

Background: 2M pages create less more fragmented free memory because memory holes for buddy allocation tend to be smaller. It also creates more memory pressure often causing the kernel to flush out cache early and create "seemingly free" memory which actually cannot be used because it is too fragmented. I think exactly this is what you initially described.

Using and tuning huge pages is a question of cost vs benefit: Memory loads become up to 10% at the cost of reduced usable memory. If your system allocates memory in bad patterns, the cost easily becomes very high in which case you may want to disable huge pages completely or identify the process which is causing it. Btrfs itself seems to spike buddy memory allocations quite often which increases the cost of huge pages.

BTW: 2M pages cannot be swapped. They need to be broken up back into 4k pages for swapping. I'm not sure if the kernel does this by default or if there's a tunable for when this should happen.

@tytgatlieven
Copy link
Author

tytgatlieven commented May 31, 2023

@kakra

my setup
56GB memory btrfs filesystems:

  1. 1*2TB ssd
  2. 4*16TB raid5 ssd

Tests

  • I have disabled transparent huge pages immediately after a reboot and disabled most background processes (3GB actual memory consumption, almost no fs usage) and the issue persists, on both filesystems

  • I have upgraded my kernel to 6.4.0-rc4: issue persists

  • I have reduced the bees db on my raid from 16GB to 4GB: issue persists

  • I have verified bees using a 4GB db on my ssd: issue persists

  • I did notice that a btrfs scrub makes buddyinfo behave similarly as bees, though no vmalloc errors occur

How can we determine if this is a regression in bees when using newer kernels or if it is the kernel's btrfs code used by crawl/dedup itself?

A new traceback:
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334233] warn_alloc: 52 callbacks suppressed
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334239] crawl_258_10978: vmalloc error: size 6291456, page order 9, failed to allocate pages, mode:0xcc2(GFP_KERNEL|__GFP_HIGHMEM), nodemask=(null),cpuset=system-beesd.slice,mems_allowed=0
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334257] CPU: 11 PID: 12028 Comm: crawl_258_10978 Tainted: G OE 6.4.0-060400rc4-generic #202305281232
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334261] Hardware name: MSI MS-7760/X79A-GD45 Plus (MS-7760), BIOS V17.9 12/08/2014
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334263] Call Trace:
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334266]
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334271] dump_stack_lvl+0x48/0x70
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334280] dump_stack+0x10/0x20
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334284] warn_alloc+0x174/0x1f0
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334291] __vmalloc_area_node+0x50f/0x630
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334298] __vmalloc_node_range+0xda/0x220
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334303] kvmalloc_node+0xbf/0x100
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334310] ? init_data_container+0x2a/0x70 [btrfs]
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334438] init_data_container+0x2a/0x70 [btrfs]
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334528] btrfs_ioctl_logical_to_ino+0xdf/0x190 [btrfs]
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334528] btrfs_ioctl+0xb4f/0x1510 [btrfs]
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334528] ? __seccomp_filter+0x37b/0x560
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334528] ? __fget_light+0xa5/0x120
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334528] __x64_sys_ioctl+0xa0/0xe0
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334528] do_syscall_64+0x5b/0x90
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334528] ? do_futex+0x10f/0x200
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334528] ? __x64_sys_futex+0x95/0x200
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334528] ? __secure_computing+0x89/0xf0
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334528] ? syscall_trace_enter.constprop.0+0xae/0x1c0
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334528] ? exit_to_user_mode_prepare+0x42/0x110
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334528] ? syscall_exit_to_user_mode+0x1b/0x50
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334528] ? do_syscall_64+0x67/0x90
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334528] ? exit_to_user_mode_prepare+0x42/0x110
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334528] ? syscall_exit_to_user_mode+0x1b/0x50
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334528] ? do_syscall_64+0x67/0x90
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334528] ? do_syscall_64+0x67/0x90
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334528] ? sysvec_thermal+0x4b/0xd0
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334528] entry_SYSCALL_64_after_hwframe+0x72/0xdc
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334528] RIP: 0033:0x7fdd0eb1aaff
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334528] Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <41> 89 c0 3d 00 f0 ff ff 77 1f 48 8b 44 24 18 64 48 2b 04 25 28 00
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334528] RSP: 002b:00007fdd0d1f8150 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334528] RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00007fdd0eb1aaff
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334528] RDX: 00007fdd0d1f81d0 RSI: 00000000c038943b RDI: 0000000000000003
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334528] RBP: 00007fdbd0003660 R08: 000056391b45b408 R09: 0000000000000000
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334528] R10: 0000000000000000 R11: 0000000000000246 R12: 00007fdd0d1f84f0
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334528] R13: 00007fdd0d1f8888 R14: 000056391ca4caf0 R15: 00007fdd0d1f8620
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334528]
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334942] Mem-Info:
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334945] active_anon:376486 inactive_anon:635379 isolated_anon:0
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334945] active_file:5119053 inactive_file:1570948 isolated_file:0
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334945] unevictable:1048604 dirty:8653 writeback:0
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334945] slab_reclaimable:963828 slab_unreclaimable:282463
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334945] mapped:169849 shmem:46555 pagetables:13320
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334945] sec_pagetables:0 bounce:0
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334945] kernel_misc_reclaimable:0
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334945] free:4138751 free_pcp:287 free_cma:0
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334954] Node 0 active_anon:1505944kB inactive_anon:2541516kB active_file:20476212kB inactive_file:6283792kB unevictable:4194416kB isolated(anon):0kB isolated(file):0kB mapped:679396kB dirty:34612kB writeback:0kB shmem:186220kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 4857856kB writeback_tmp:0kB kernel_stack:23408kB pagetables:53280kB sec_pagetables:0kB all_unreclaimable? no
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334961] Node 0 DMA free:11264kB boost:0kB min:16kB low:28kB high:40kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334970] lowmem_reserve[]: 0 2590 55806 55806 55806
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334977] Node 0 DMA32 free:217896kB boost:0kB min:3136kB low:5788kB high:8440kB reserved_highatomic:0KB active_anon:286968kB inactive_anon:100580kB active_file:1688012kB inactive_file:304236kB unevictable:0kB writepending:24kB present:3041136kB managed:2776480kB mlocked:0kB bounce:0kB free_pcp:88kB local_pcp:0kB free_cma:0kB
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334987] lowmem_reserve[]: 0 0 53215 53215 53215
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.334994] Node 0 Normal free:16325844kB boost:2048kB min:66476kB low:120968kB high:175460kB reserved_highatomic:0KB active_anon:1218976kB inactive_anon:2440692kB active_file:18786992kB inactive_file:5981308kB unevictable:4194416kB writepending:34588kB present:55574528kB managed:54501388kB mlocked:4194416kB bounce:0kB free_pcp:1056kB local_pcp:0kB free_cma:0kB
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.335002] lowmem_reserve[]: 0 0 0 0 0
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.335010] Node 0 DMA: 04kB 08kB 016kB 032kB 064kB 0128kB 0256kB 0512kB 11024kB (U) 12048kB (M) 24096kB (M) = 11264kB
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.335031] Node 0 DMA32: 1426
4kB (UME) 11638kB (UME) 38916kB (UME) 134732kB (UME) 82064kB (UME) 182128kB (UME) 15256kB (ME) 3512kB (ME) 31024kB (UME) 202048kB (U) 74096kB (U) = 218192kB
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.335061] Node 0 Normal: 268814kB (UME) 728098kB (UME) 7168916kB (UME) 4846432kB (UME) 3807164kB (UME) 19470128kB (UME) 8633256kB (UME) 4662512kB (UE) 33321024kB (UM) 02048kB 0*4096kB = 16325532kB
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.335088] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.335091] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.335094] 6742283 total pagecache pages
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.335096] 6067 pages in swap cache
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.335098] Free swap = 104119548kB
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.335099] Total swap = 104857596kB
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.335101] 14657915 pages RAM
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.335102] 0 pages HighMem/MovableOnly
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.335103] 334608 pages reserved
May 31 14:16:37 ltytgat-desktop kernel: [ 1460.335105] 0 pages hwpoisoned
May 31 14:16:37 ltytgat-desktop beesd[12001]: crawl_258_1097879[12028]: dedup: src 8 [0x1000..0x1008] {0x725568a3000uza000} /run/bees/mnt/21568dec-71e3-44d5-8cf4-fe736c87f481/@home/.sxbackup/sx-20230531-104003-utc/ltytgat/tymeli-technical/internal/embedded/openwrt/openwrt-cm4/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/linux-5.10.161/drivers/crypto/ux500/cryp/cryp_irqp.h
May 31 14:16:37 ltytgat-desktop beesd[12001]: crawl_258_1097879[12028]: dst 8 [0x3000..0x3008] {0x779d1198000uz3000} /run/bees/mnt/21568dec-71e3-44d5-8cf4-fe736c87f481/@home/ltytgat/tymeli-technical/internal/embedded/carambola2/buildenv/carambola2.github/build_dir.old/toolchain-mips_34kc_gcc-5.3.0_musl-1.1.14/linux-4.1.23/include/dt-bindings/clock/qcom,gcc-msm8974.h
May 31 14:16:37 ltytgat-desktop beesd[12001]: crawl_258_1097879[12028]: scan: 8 0x3000 [D] 0x3008 /run/bees/mnt/21568dec-71e3-44d5-8cf4-fe736c87f481/@home/ltytgat/tymeli-technical/internal/embedded/carambola2/buildenv/carambola2.github/build_dir.old/toolchain-mips_34kc_gcc-5.3.0_musl-1.1.14/linux-4.1.23/include/dt-bindings/clock/qcom,gcc-msm8974.h

@kakra
Copy link
Contributor

kakra commented May 31, 2023

How can we determine if this is a regression in bees when using newer kernels or if it is the kernel's btrfs code used by crawl/dedup itself?

By definition, user space software must never be able to create kernel oops or traces [1] - so this is a kernel regression. Does it work fine with an older kernel then?

[1]: bees does some efforts to work around such issues, tho - but that doesn't make it bees fault

@tytgatlieven
Copy link
Author

It does work fine with kernel 6.2.13 (no kernel traces). The buddyinfo does behave identical to the buddyinfo in the non-working 6.3.1 case

I do agree that userspace should never be able to create a oops or trace, so indeed it should not bees's fault.

@kakra
Copy link
Contributor

kakra commented Jun 7, 2023

Actually, I feel like memory fragmentation is becoming a bigger issue when running btrfs with each kernel cycle. I'm currently running 6.1 and see very high order 0 values in buddyinfo, and get oops'es or IO thrashing - while it worked fine in the previous LTS kernel (and thus I never looked at buddyinfo). Using memory cgroups seems to worsen the problem but that may be an effect of using bees and how cache ownership works in memory cgroups.

One of our servers running 6.1 had buddyinfo with order 0 in the millions - and increasing RAM for it only worsened the problem for some reason. This hasn't been an issue with the previous 5.19. With transparent hugepages completely turned off it now behaves mostly as expected but the order 0 numbers are still very high.

There's another metric you could look at: /proc/pagetypeinfo - but I'm not yet sure how to properly read that.

@tytgatlieven
Copy link
Author

tytgatlieven commented Jun 13, 2023

I have noticed that even without bees running vmalloc errors occur after some time due to other services.
Hence, It becomes clear that this is either:

  • a BTRFS issue with memory fragmentation
  • or a more generic vmalloc issue.

If I do a echo 1 > /proc/sys/vm/drop_caches the low order numbers jump up and are reduced after some time. The order 9 numbers increase, and the vmalloc errors are removed for some time, until the order 9 number are reduced to 0 again.

Does anybody have a contact in the BTRFS development community where this could be triggered?

These are the errors I get:

Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198519] kded5: vmalloc error: size 10485760, page order 9, failed to allocate pages, mode:0x400cc2(GFP_KERNEL_ACCOUNT|__GFP_HIGHMEM), nodemask=(null),cpuset=user.slice,mems_allowed=0

Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198530] CPU: 7 PID: 147280 Comm: kded5 Tainted: G W OE 6.3.7-060307-generic #202306090936
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198532] Hardware name: MSI MS-7760/X79A-GD45 Plus (MS-7760), BIOS V17.9 12/08/2014
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198533] Call Trace:
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198535]
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198538] dump_stack_lvl+0x48/0x70
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198543] dump_stack+0x10/0x20
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198544] warn_alloc+0x174/0x1f0
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198548] ? __mod_memcg_state+0x47/0xd0
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198551] __vmalloc_area_node+0x50f/0x630
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198554] __vmalloc_node_range+0xda/0x220
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198557] kvmalloc_node+0xbf/0x100
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198560] ? seq_read_iter+0x1a8/0x4a0
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198562] seq_read_iter+0x1a8/0x4a0
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198564] seq_read+0xcd/0x110
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198567] vfs_read+0xac/0x330
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198569] ? do_statx+0x78/0xc0
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198571] ? __fget_light+0xa5/0x120
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198574] ksys_read+0x73/0x100
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198576] __x64_sys_read+0x19/0x30
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198577] do_syscall_64+0x5b/0x90
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198581] ? syscall_exit_to_user_mode+0x29/0x50
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198584] ? do_syscall_64+0x67/0x90
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198586] ? do_syscall_64+0x67/0x90
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198588] ? do_syscall_64+0x67/0x90
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198590] entry_SYSCALL_64_after_hwframe+0x72/0xdc
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198593] RIP: 0033:0x7f25c8b149cc
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198620] Code: ec 28 48 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 b9 c0 f7 ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 34 44 89 c7 48 89 44 24 08 e8 ff c0 f7 ff 48
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198621] RSP: 002b:00007ffc8d21b0f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198623] RAX: ffffffffffffffda RBX: 000055ecf19c3f30 RCX: 00007f25c8b149cc
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198624] RDX: 0000000000004000 RSI: 000055ecf20961d8 RDI: 000000000000001e
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198625] RBP: 0000000000004000 R08: 0000000000000000 R09: 000055ecf20961c0
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198626] R10: 0000000000004000 R11: 0000000000000246 R12: 0000000000000000
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198627] R13: 0000000000004000 R14: 000000000000001e R15: 000055ecf20961d8
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198630]
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198631] Mem-Info:
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198633] active_anon:436966 inactive_anon:4133461 isolated_anon:0
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198633] active_file:3707141 inactive_file:1486304 isolated_file:0
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198633] unevictable:69 dirty:3808 writeback:0
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198633] slab_reclaimable:1288983 slab_unreclaimable:382481
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198633] mapped:2454753 shmem:174506 pagetables:27905
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198633] sec_pagetables:0 bounce:0
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198633] kernel_misc_reclaimable:0
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198633] free:215076 free_pcp:125 free_cma:0
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198637] Node 0 active_anon:1747864kB inactive_anon:16533844kB active_file:14828564kB inactive_file:5945216kB unevictable:276kB isolated(anon):0kB isolated(file):0kB mapped:9819012kB dirty:15232kB writeback:0kB shmem:698024kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 344064kB writeback_tmp:0kB kernel_stack:30336kB pagetables:111620kB sec_pagetables:0kB all_unreclaimable? no
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198641] Node 0 DMA free:11264kB boost:0kB min:16kB low:28kB high:40kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198645] lowmem_reserve[]: 0 2591 55807 55807 55807
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198648] Node 0 DMA32 free:218136kB boost:0kB min:3136kB low:5788kB high:8440kB reserved_highatomic:2048KB active_anon:148160kB inactive_anon:679556kB active_file:615728kB inactive_file:134388kB unevictable:0kB writepending:44kB present:3041136kB managed:2776480kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198652] lowmem_reserve[]: 0 0 53215 53215 53215
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198655] Node 0 Normal free:630904kB boost:0kB min:64428kB low:118920kB high:173412kB reserved_highatomic:2048KB active_anon:1599960kB inactive_anon:15854144kB active_file:14213120kB inactive_file:5811172kB unevictable:276kB writepending:15196kB present:55574528kB managed:54501480kB mlocked:276kB bounce:0kB free_pcp:500kB local_pcp:0kB free_cma:0kB
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198659] lowmem_reserve[]: 0 0 0 0 0
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198662] Node 0 DMA: 04kB 08kB 016kB 032kB 064kB 0128kB 0256kB 0512kB 11024kB (U) 12048kB (M) 24096kB (M) = 11264kB
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198671] Node 0 DMA32: 9478
4kB (UME) 33468kB (UME) 78316kB (UME) 18232kB (UMEH) 38364kB (UMEH) 158128kB (UMEH) 39256kB (UEH) 43512kB (UH) 131024kB (UE) 122048kB (UME) 54096kB (UM) = 218136kB
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198684] Node 0 Normal: 377854kB (UMEH) 230548kB (UMEH) 1000616kB (UMEH) 254932kB (UMEH) 66864kB (UEH) 75128kB (UEH) 7256kB (UM) 0512kB 01024kB 02048kB 04096kB = 631380kB
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198694] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198696] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198697] 5367802 total pagecache pages
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198698] 2132 pages in swap cache
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198698] Free swap = 102937084kB
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198700] Total swap = 104857596kB
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198700] 14657915 pages RAM
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198701] 0 pages HighMem/MovableOnly
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198702] 334585 pages reserved
Jun 13 15:00:30 ltytgat-desktop kernel: [110969.198703] 0 pages hwpoisoned
Jun 13 15:01:08 ltytgat-desktop kernel: [111007.138124] input: RustDesk UInput Keyboard as /devices/virtual/input/input91
Jun 13 15:01:08 ltytgat-desktop kernel: [111007.190175] input: mouce-library-fake-mouse as /devices/virtual/input/input92
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243013] warn_alloc: 3 callbacks suppressed
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243017] kded5: vmalloc error: size 12582912, page order 9, failed to allocate pages, mode:0x400cc2(GFP_KERNEL_ACCOUNT|__GFP_HIGHMEM), nodemask=(null),cpuset=user.slice,mems_allowed=0
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243027] CPU: 9 PID: 147280 Comm: kded5 Tainted: G W OE 6.3.7-060307-generic #202306090936
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243029] Hardware name: MSI MS-7760/X79A-GD45 Plus (MS-7760), BIOS V17.9 12/08/2014
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243030] Call Trace:
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243032]
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243035] dump_stack_lvl+0x48/0x70
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243040] dump_stack+0x10/0x20
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243042] warn_alloc+0x174/0x1f0
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243045] ? __mod_memcg_state+0x47/0xd0
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243048] __vmalloc_area_node+0x50f/0x630
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243051] __vmalloc_node_range+0xda/0x220
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243054] kvmalloc_node+0xbf/0x100
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243057] ? seq_read_iter+0x1a8/0x4a0
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243060] seq_read_iter+0x1a8/0x4a0
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243062] seq_read+0xcd/0x110
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243064] vfs_read+0xac/0x330
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243066] ? __fget_light+0xa5/0x120
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243069] ksys_read+0x73/0x100
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243071] __x64_sys_read+0x19/0x30
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243073] do_syscall_64+0x5b/0x90
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243077] ? syscall_exit_to_user_mode+0x29/0x50
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243079] ? do_syscall_64+0x67/0x90
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243081] ? do_syscall_64+0x67/0x90
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243083] ? syscall_exit_to_user_mode+0x29/0x50
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243084] ? do_syscall_64+0x67/0x90
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243086] ? do_syscall_64+0x67/0x90
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243088] ? do_syscall_64+0x67/0x90
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243091] entry_SYSCALL_64_after_hwframe+0x72/0xdc
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243095] RIP: 0033:0x7f25c8b149cc
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243122] Code: ec 28 48 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 b9 c0 f7 ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 34 44 89 c7 48 89 44 24 08 e8 ff c0 f7 ff 48
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243124] RSP: 002b:00007ffc8d21b0f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243126] RAX: ffffffffffffffda RBX: 000055ecf1951cf0 RCX: 00007f25c8b149cc
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243127] RDX: 0000000000004000 RSI: 000055ecf20961d8 RDI: 000000000000001e
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243128] RBP: 0000000000004000 R08: 0000000000000000 R09: 000055ecf20961c0
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243129] R10: 0000000000004000 R11: 0000000000000246 R12: 0000000000000000
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243130] R13: 0000000000004000 R14: 000000000000001e R15: 000055ecf20961d8
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243133]
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243134] Mem-Info:
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243136] active_anon:435502 inactive_anon:4163524 isolated_anon:81
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243136] active_file:3882752 inactive_file:1275393 isolated_file:53
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243136] unevictable:69 dirty:7566 writeback:0
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243136] slab_reclaimable:1288966 slab_unreclaimable:382278
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243136] mapped:2454468 shmem:174674 pagetables:27929
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243136] sec_pagetables:0 bounce:0
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243136] kernel_misc_reclaimable:0
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243136] free:221094 free_pcp:46 free_cma:0
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243140] Node 0 active_anon:1742008kB inactive_anon:16654096kB active_file:15531008kB inactive_file:5101572kB unevictable:276kB isolated(anon):324kB isolated(file):212kB mapped:9817872kB dirty:30264kB writeback:0kB shmem:698696kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 339968kB writeback_tmp:0kB kernel_stack:30336kB pagetables:111716kB sec_pagetables:0kB all_unreclaimable? no
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243143] Node 0 DMA free:11264kB boost:0kB min:16kB low:28kB high:40kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243147] lowmem_reserve[]: 0 2591 55807 55807 55807
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243151] Node 0 DMA32 free:219512kB boost:0kB min:3136kB low:5788kB high:8440kB reserved_highatomic:2048KB active_anon:148180kB inactive_anon:680616kB active_file:621360kB inactive_file:127188kB unevictable:0kB writepending:260kB present:3041136kB managed:2776480kB mlocked:0kB bounce:0kB free_pcp:184kB local_pcp:0kB free_cma:0kB
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243155] lowmem_reserve[]: 0 0 53215 53215 53215
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243157] Node 0 Normal free:653600kB boost:0kB min:64428kB low:118920kB high:173412kB reserved_highatomic:2048KB active_anon:1594212kB inactive_anon:15972832kB active_file:14909648kB inactive_file:4974060kB unevictable:276kB writepending:30004kB present:55574528kB managed:54501480kB mlocked:276kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243161] lowmem_reserve[]: 0 0 0 0 0
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243164] Node 0 DMA: 0
4kB 08kB 016kB 032kB 064kB 0128kB 0256kB 0512kB 11024kB (U) 12048kB (M) 24096kB (M) = 11264kB
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243174] Node 0 DMA32: 93264kB (UME) 33608kB (UME) 79616kB (UME) 19232kB (UMEH) 38664kB (UMEH) 159128kB (UMEH) 39256kB (UEH) 43512kB (UH) 141024kB (UE) 122048kB (UME) 54096kB (UM) = 219512kB
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243186] Node 0 Normal: 31170
4kB (UMEH) 242128kB (UMEH) 1065116kB (UMEH) 288432kB (UMEH) 77264kB (UMEH) 94128kB (UMEH) 34256kB (U) 1512kB (U) 21024kB (U) 02048kB 04096kB = 653784kB
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243197] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243199] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243200] 5332808 total pagecache pages
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243201] 2361 pages in swap cache
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243202] Free swap = 102919932kB
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243202] Total swap = 104857596kB
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243203] 14657915 pages RAM
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243204] 0 pages HighMem/MovableOnly
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243205] 334585 pages reserved
Jun 13 15:10:30 ltytgat-desktop kernel: [111569.243206] 0 pages hwpoisoned
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.436709] warn_alloc: 3 callbacks suppressed
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.436717] kioslave5: vmalloc error: size 0, page order 9, failed to allocate pages, mode:0xdc2(GFP_KERNEL|__GFP_HIGHMEM|__GFP_ZERO), nodemask=(null),cpuset=user.slice,mems_allowed=0
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.436732] CPU: 8 PID: 902068 Comm: kioslave5 Tainted: G W OE 6.3.7-060307-generic #202306090936
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.436735] Hardware name: MSI MS-7760/X79A-GD45 Plus (MS-7760), BIOS V17.9 12/08/2014
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.436737] Call Trace:
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.436739]
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.436743] dump_stack_lvl+0x48/0x70
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.436750] dump_stack+0x10/0x20
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.436752] warn_alloc+0x174/0x1f0
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.436757] __vmalloc_area_node+0x50f/0x630
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.436761] __vmalloc_node_range+0xda/0x220
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.436764] kvmalloc_node+0xbf/0x100
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.436771] ? amdgpu_bo_create+0xdc/0x4c0 [amdgpu]
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.437105] amdgpu_bo_create+0xdc/0x4c0 [amdgpu]
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.437311] amdgpu_bo_create_vm+0x3a/0x90 [amdgpu]
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.437521] amdgpu_vm_pt_create+0xe9/0x2a0 [amdgpu]
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.437739] amdgpu_vm_init+0x2d6/0x510 [amdgpu]
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.437957] amdgpu_driver_open_kms+0x9c/0x250 [amdgpu]
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438159] drm_file_alloc+0x1a1/0x260 [drm]
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438208] drm_open_helper+0x77/0x130 [drm]
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438240] ? drm_minor_acquire+0xb1/0x180 [drm]
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438270] drm_open+0x7f/0x170 [drm]
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438302] drm_stub_open+0xa9/0xe0 [drm]
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438332] chrdev_open+0xc5/0x240
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438335] ? fsnotify_perm.part.0+0x79/0x190
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438339] ? __pfx_chrdev_open+0x10/0x10
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438340] do_dentry_open+0x1f8/0x430
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438343] vfs_open+0x2d/0x40
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438345] do_open+0x2ea/0x460
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438349] path_openat+0x135/0x2c0
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438352] do_filp_open+0xaf/0x170
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438356] do_sys_openat2+0xbf/0x180
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438358] __x64_sys_openat+0x6c/0xa0
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438360] do_syscall_64+0x5b/0x90
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438365] ? do_syscall_64+0x67/0x90
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438367] ? __f_unlock_pos+0x12/0x20
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438370] ? __x64_sys_getdents64+0x10e/0x130
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438372] ? __pfx_filldir64+0x10/0x10
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438374] ? exit_to_user_mode_prepare+0x30/0xb0
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438377] ? syscall_exit_to_user_mode+0x29/0x50
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438379] ? do_syscall_64+0x67/0x90
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438381] ? do_syscall_64+0x67/0x90
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438383] ? do_syscall_64+0x67/0x90
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438385] ? do_syscall_64+0x67/0x90
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438387] entry_SYSCALL_64_after_hwframe+0x72/0xdc
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438391] RIP: 0033:0x7f280a914764
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438424] Code: 24 20 eb 8f 66 90 44 89 54 24 0c e8 26 c3 f7 ff 44 8b 54 24 0c 44 89 e2 48 89 ee 41 89 c0 bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 77 34 44 89 c7 89 44 24 0c e8 68 c3 f7 ff 8b 44
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438425] RSP: 002b:00007ffdce33a5b0 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438427] RAX: ffffffffffffffda RBX: 000055c0ceb889e0 RCX: 00007f280a914764
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438429] RDX: 0000000000080002 RSI: 000055c0ceb89620 RDI: 00000000ffffff9c
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438430] RBP: 000055c0ceb89620 R08: 0000000000000000 R09: 0000000000000004
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438431] R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000080002
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438432] R13: 000055c0ceb89620 R14: 00007f28056d6340 R15: 00007f28056d6110
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438434]
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438435] Mem-Info:
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438438] active_anon:435525 inactive_anon:4166387 isolated_anon:0
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438438] active_file:3886595 inactive_file:1234456 isolated_file:81
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438438] unevictable:69 dirty:4887 writeback:0
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438438] slab_reclaimable:1288937 slab_unreclaimable:382584
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438438] mapped:2456589 shmem:174403 pagetables:27990
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438438] sec_pagetables:0 bounce:0
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438438] kernel_misc_reclaimable:0
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438438] free:245824 free_pcp:726 free_cma:0
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438442] Node 0 active_anon:1742100kB inactive_anon:16665548kB active_file:15546380kB inactive_file:4937824kB unevictable:276kB isolated(anon):0kB isolated(file):324kB mapped:9826356kB dirty:19548kB writeback:0kB shmem:697612kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 339968kB writeback_tmp:0kB kernel_stack:30464kB pagetables:111960kB sec_pagetables:0kB all_unreclaimable? no
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438446] Node 0 DMA free:11264kB boost:0kB min:16kB low:28kB high:40kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438450] lowmem_reserve[]: 0 2591 55807 55807 55807
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438453] Node 0 DMA32 free:215712kB boost:0kB min:3136kB low:5788kB high:8440kB reserved_highatomic:2048KB active_anon:148176kB inactive_anon:679496kB active_file:613520kB inactive_file:130324kB unevictable:0kB writepending:4268kB present:3041136kB managed:2776480kB mlocked:0kB bounce:0kB free_pcp:540kB local_pcp:0kB free_cma:0kB
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438457] lowmem_reserve[]: 0 0 53215 53215 53215
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438461] Node 0 Normal free:756320kB boost:0kB min:64428kB low:118920kB high:173412kB reserved_highatomic:2048KB active_anon:1593924kB inactive_anon:15986760kB active_file:14932980kB inactive_file:4808200kB unevictable:276kB writepending:15276kB present:55574528kB managed:54501480kB mlocked:276kB bounce:0kB free_pcp:2336kB local_pcp:0kB free_cma:0kB
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438464] lowmem_reserve[]: 0 0 0 0 0
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438467] Node 0 DMA: 04kB 08kB 016kB 032kB 064kB 0128kB 0256kB 0512kB 11024kB (U) 12048kB (M) 24096kB (M) = 11264kB
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438477] Node 0 DMA32: 10067
4kB (UME) 35778kB (UME) 83516kB (UME) 19632kB (UMEH) 38664kB (UMEH) 159128kB (UMEH) 29256kB (UEH) 42512kB (UH) 161024kB (UE) 142048kB (UME) 24096kB (UM) = 215748kB
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438491] Node 0 Normal: 371704kB (UMEH) 288578kB (UMEH) 1211916kB (UMEH) 360532kB (UMEH) 104364kB (UMEH) 19128kB (UMH) 1256kB (U) 0512kB 01024kB 02048kB 04096kB = 758240kB
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438502] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438503] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438505] 5295487 total pagecache pages
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438505] 2198 pages in swap cache
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438506] Free swap = 102918396kB
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438507] Total swap = 104857596kB
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438508] 14657915 pages RAM
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438508] 0 pages HighMem/MovableOnly
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438509] 334585 pages reserved
Jun 13 15:12:13 ltytgat-desktop kernel: [111672.438510] 0 pages hwpoisoned
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.641938] kioslave5: vmalloc error: size 0, page order 9, failed to allocate pages, mode:0xdc2(GFP_KERNEL|__GFP_HIGHMEM|__GFP_ZERO), nodemask=(null),cpuset=user.slice,mems_allowed=0
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.641953] CPU: 0 PID: 902881 Comm: kioslave5 Tainted: G W OE 6.3.7-060307-generic #202306090936
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.641956] Hardware name: MSI MS-7760/X79A-GD45 Plus (MS-7760), BIOS V17.9 12/08/2014
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.641957] Call Trace:
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.641959]
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.641962] dump_stack_lvl+0x48/0x70
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.641967] dump_stack+0x10/0x20
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.641969] warn_alloc+0x174/0x1f0
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.641973] __vmalloc_area_node+0x50f/0x630
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.641977] __vmalloc_node_range+0xda/0x220
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.641980] kvmalloc_node+0xbf/0x100
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.641985] ? amdgpu_bo_create+0xdc/0x4c0 [amdgpu]
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.642273] amdgpu_bo_create+0xdc/0x4c0 [amdgpu]
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.642489] amdgpu_bo_create_vm+0x3a/0x90 [amdgpu]
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.642705] amdgpu_vm_pt_create+0xe9/0x2a0 [amdgpu]
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.642931] amdgpu_vm_init+0x2d6/0x510 [amdgpu]
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643159] amdgpu_driver_open_kms+0x9c/0x250 [amdgpu]
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643372] drm_file_alloc+0x1a1/0x260 [drm]
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643424] drm_open_helper+0x77/0x130 [drm]
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643458] ? drm_minor_acquire+0xb1/0x180 [drm]
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643490] drm_open+0x7f/0x170 [drm]
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643523] drm_stub_open+0xa9/0xe0 [drm]
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643555] chrdev_open+0xc5/0x240
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643558] ? fsnotify_perm.part.0+0x79/0x190
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643562] ? __pfx_chrdev_open+0x10/0x10
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643564] do_dentry_open+0x1f8/0x430
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643566] vfs_open+0x2d/0x40
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643569] do_open+0x2ea/0x460
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643572] path_openat+0x135/0x2c0
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643575] do_filp_open+0xaf/0x170
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643580] do_sys_openat2+0xbf/0x180
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643582] __x64_sys_openat+0x6c/0xa0
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643585] do_syscall_64+0x5b/0x90
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643589] ? exit_to_user_mode_prepare+0x30/0xb0
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643592] ? syscall_exit_to_user_mode+0x29/0x50
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643594] ? do_syscall_64+0x67/0x90
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643596] ? do_syscall_64+0x67/0x90
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643598] entry_SYSCALL_64_after_hwframe+0x72/0xdc
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643603] RIP: 0033:0x7f9582314764
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643635] Code: 24 20 eb 8f 66 90 44 89 54 24 0c e8 26 c3 f7 ff 44 8b 54 24 0c 44 89 e2 48 89 ee 41 89 c0 bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 77 34 44 89 c7 89 44 24 0c e8 68 c3 f7 ff 8b 44
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643637] RSP: 002b:00007ffcd2b718b0 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643639] RAX: ffffffffffffffda RBX: 000055ac5b759310 RCX: 00007f9582314764
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643641] RDX: 0000000000080002 RSI: 000055ac5b759b40 RDI: 00000000ffffff9c
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643642] RBP: 000055ac5b759b40 R08: 0000000000000000 R09: 0000000000000004
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643643] R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000080002
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643645] R13: 000055ac5b759b40 R14: 00007f957d0ac340 R15: 00007f957d0ac110
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643647]
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643648] Mem-Info:
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643650] active_anon:436786 inactive_anon:4235828 isolated_anon:0
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643650] active_file:3790917 inactive_file:1213402 isolated_file:0
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643650] unevictable:69 dirty:3708 writeback:0
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643650] slab_reclaimable:1288758 slab_unreclaimable:382638
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643650] mapped:2472015 shmem:186174 pagetables:28180
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643650] sec_pagetables:0 bounce:0
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643650] kernel_misc_reclaimable:0
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643650] free:284604 free_pcp:581 free_cma:0
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643655] Node 0 active_anon:1747144kB inactive_anon:16943312kB active_file:15163668kB inactive_file:4853608kB unevictable:276kB isolated(anon):0kB isolated(file):0kB mapped:9888060kB dirty:14832kB writeback:0kB shmem:744696kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 339968kB writeback_tmp:0kB kernel_stack:30800kB pagetables:112720kB sec_pagetables:0kB all_unreclaimable? no
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643659] Node 0 DMA free:11264kB boost:0kB min:16kB low:28kB high:40kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643663] lowmem_reserve[]: 0 2591 55807 55807 55807
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643666] Node 0 DMA32 free:214592kB boost:0kB min:3136kB low:5788kB high:8440kB reserved_highatomic:2048KB active_anon:148728kB inactive_anon:679356kB active_file:608712kB inactive_file:124796kB unevictable:0kB writepending:36kB present:3041136kB managed:2776480kB mlocked:0kB bounce:0kB free_pcp:44kB local_pcp:0kB free_cma:0kB
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643671] lowmem_reserve[]: 0 0 53215 53215 53215
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643674] Node 0 Normal free:912560kB boost:0kB min:64428kB low:118920kB high:173412kB reserved_highatomic:2048KB active_anon:1598416kB inactive_anon:16264336kB active_file:14554304kB inactive_file:4728528kB unevictable:276kB writepending:14796kB present:55574528kB managed:54501480kB mlocked:276kB bounce:0kB free_pcp:2404kB local_pcp:0kB free_cma:0kB
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643678] lowmem_reserve[]: 0 0 0 0 0
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643681] Node 0 DMA: 0
4kB 08kB 016kB 032kB 064kB 0128kB 0256kB 0512kB 11024kB (U) 12048kB (M) 24096kB (M) = 11264kB
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643691] Node 0 DMA32: 55384kB (UME) 39968kB (UME) 189516kB (UME) 69732kB (UMEH) 36964kB (UMEH) 157128kB (UMEH) 31256kB (UMEH) 32512kB (UH) 71024kB (UE) 162048kB (UME) 04096kB = 214712kB
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643703] Node 0 Normal: 48693
4kB (UMEH) 355668kB (UMEH) 1400716kB (UMEH) 404232kB (UMEH) 117064kB (UMEH) 41128kB (UMEH) 0256kB 0512kB 01024kB 02048kB 04096kB = 912884kB
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643714] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643716] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643717] 5190552 total pagecache pages
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643718] 2369 pages in swap cache
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643719] Free swap = 102882300kB
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643720] Total swap = 104857596kB
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643721] 14657915 pages RAM
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643721] 0 pages HighMem/MovableOnly
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643722] 334585 pages reserved
Jun 13 15:13:59 ltytgat-desktop kernel: [111778.643723] 0 pages hwpoisoned
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203362] kded5: vmalloc error: size 71303168, page order 9, failed to allocate pages, mode:0x400cc2(GFP_KERNEL_ACCOUNT|__GFP_HIGHMEM), nodemask=(null),cpuset=user.slice,mems_allowed=0
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203375] CPU: 10 PID: 147280 Comm: kded5 Tainted: G W OE 6.3.7-060307-generic #202306090936
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203377] Hardware name: MSI MS-7760/X79A-GD45 Plus (MS-7760), BIOS V17.9 12/08/2014
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203378] Call Trace:
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203380]
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203383] dump_stack_lvl+0x48/0x70
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203388] dump_stack+0x10/0x20
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203389] warn_alloc+0x174/0x1f0
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203393] ? __mod_memcg_state+0x47/0xd0
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203396] __vmalloc_area_node+0x50f/0x630
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203399] __vmalloc_node_range+0xda/0x220
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203402] kvmalloc_node+0xbf/0x100
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203405] ? seq_read_iter+0x1a8/0x4a0
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203407] seq_read_iter+0x1a8/0x4a0
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203409] ? _copy_to_user+0x25/0x60
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203412] seq_read+0xcd/0x110
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203415] vfs_read+0xac/0x330
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203417] ? __fget_light+0xa5/0x120
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203420] ksys_read+0x73/0x100
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203422] __x64_sys_read+0x19/0x30
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203423] do_syscall_64+0x5b/0x90
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203428] ? putname+0x5d/0x80
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203430] ? exit_to_user_mode_prepare+0x30/0xb0
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203433] ? syscall_exit_to_user_mode+0x29/0x50
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203435] ? do_syscall_64+0x67/0x90
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203437] ? syscall_exit_to_user_mode+0x29/0x50
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203438] ? do_syscall_64+0x67/0x90
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203441] ? syscall_exit_to_user_mode+0x29/0x50
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203442] ? do_syscall_64+0x67/0x90
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203444] ? do_syscall_64+0x67/0x90
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203446] entry_SYSCALL_64_after_hwframe+0x72/0xdc
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203450] RIP: 0033:0x7f25c8b149cc
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203477] Code: ec 28 48 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 b9 c0 f7 ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 34 44 89 c7 48 89 44 24 08 e8 ff c0 f7 ff 48
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203479] RSP: 002b:00007ffc8d21b0f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203481] RAX: ffffffffffffffda RBX: 000055ecf1955920 RCX: 00007f25c8b149cc
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203482] RDX: 0000000000004000 RSI: 000055ecf20961d8 RDI: 000000000000001e
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203483] RBP: 0000000000004000 R08: 0000000000000000 R09: 000055ecf20961c0
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203484] R10: 0000000000004000 R11: 0000000000000246 R12: 0000000000000000
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203485] R13: 0000000000004000 R14: 000000000000001e R15: 000055ecf20961d8
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203488]
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203489] Mem-Info:
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203490] active_anon:427507 inactive_anon:4110105 isolated_anon:324
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203490] active_file:3824105 inactive_file:528463 isolated_file:0
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203490] unevictable:69 dirty:5538 writeback:0
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203490] slab_reclaimable:1285550 slab_unreclaimable:382777
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203490] mapped:2457981 shmem:174494 pagetables:27688
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203490] sec_pagetables:0 bounce:0
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203490] kernel_misc_reclaimable:0
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203490] free:875566 free_pcp:336 free_cma:0
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203495] Node 0 active_anon:1710028kB inactive_anon:16440420kB active_file:15296420kB inactive_file:2113852kB unevictable:276kB isolated(anon):1296kB isolated(file):0kB mapped:9831924kB dirty:22152kB writeback:0kB shmem:697976kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 337920kB writeback_tmp:0kB kernel_stack:32000kB pagetables:110752kB sec_pagetables:0kB all_unreclaimable? no
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203499] Node 0 DMA free:11264kB boost:0kB min:16kB low:28kB high:40kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203503] lowmem_reserve[]: 0 2591 55807 55807 55807
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203506] Node 0 DMA32 free:219164kB boost:0kB min:3136kB low:5788kB high:8440kB reserved_highatomic:2048KB active_anon:147564kB inactive_anon:642672kB active_file:612568kB inactive_file:35604kB unevictable:0kB writepending:36kB present:3041136kB managed:2776480kB mlocked:0kB bounce:0kB free_pcp:452kB local_pcp:0kB free_cma:0kB
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203510] lowmem_reserve[]: 0 0 53215 53215 53215
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203513] Node 0 Normal free:3271836kB boost:0kB min:64428kB low:118920kB high:173412kB reserved_highatomic:2048KB active_anon:1562464kB inactive_anon:15798360kB active_file:14683852kB inactive_file:2078552kB unevictable:276kB writepending:22104kB present:55574528kB managed:54501480kB mlocked:276kB bounce:0kB free_pcp:1020kB local_pcp:0kB free_cma:0kB
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203517] lowmem_reserve[]: 0 0 0 0 0
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203520] Node 0 DMA: 04kB 08kB 016kB 032kB 064kB 0128kB 0256kB 0512kB 11024kB (U) 12048kB (M) 24096kB (M) = 11264kB
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203529] Node 0 DMA32: 9792
4kB (UME) 60128kB (UME) 240116kB (UME) 90932kB (UMEH) 39864kB (UMEH) 135128kB (UMEH) 17256kB (UMH) 4512kB (UMH) 31024kB (UM) 62048kB (UM) 04096kB = 219280kB
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203541] Node 0 Normal: 1170444kB (UMEH) 935868kB (UMEH) 5094616kB (UMEH) 2242732kB (UMEH) 701864kB (UMEH) 236128kB (UMEH) 120256kB (UM) 9512kB (U) 71024kB (U) 02048kB 0*4096kB = 3271520kB
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203552] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203554] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203555] 4527253 total pagecache pages
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203556] 2513 pages in swap cache
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203558] Free swap = 102839292kB
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203559] Total swap = 104857596kB
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203560] 14657915 pages RAM
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203561] 0 pages HighMem/MovableOnly
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203562] 334585 pages reserved
Jun 13 15:20:32 ltytgat-desktop kernel: [112171.203563] 0 pages hwpoisoned

Jun 13 15:24:24 ltytgat-desktop kernel: [112402.962588] bash (161844): drop_caches: 3

@kakra
Copy link
Contributor

kakra commented Jun 14, 2023

This is a good find. Maybe hop over to IRC #btrfs then, there are quite some btrfs devs active.

@Zygo
Copy link
Owner

Zygo commented Jul 6, 2023

The bug is:

        v6.3-rc6: f349b15e183d mm: vmalloc: avoid warn_alloc noise caused by fatal signal

The fixes are:

        v6.4: 95a301eefa82 mm/vmalloc: do not output a spurious warning when huge vmalloc() fails
        v6.3.10: c189994b5dd3 mm/vmalloc: do not output a spurious warning when huge vmalloc() fails

The bug has been backported to LTS, but the fix has not:

        v6.2.11: 61334bc29781 mm: vmalloc: avoid warn_alloc noise caused by fatal signal
        v6.1.24: ef6bd8f64ce0 mm: vmalloc: avoid warn_alloc noise caused by fatal signal
        v5.15.107: a184df0de132 mm: vmalloc: avoid warn_alloc noise caused by fatal signal

So kernel 6.3.10 and 6.4 are good to go, but now the LTS kernels are broken.

@tytgatlieven
Copy link
Author

I was just going to mention that in 6.5rc1 this issue seems to have been tackled, but then I saw the comment above from @Zygo

Do I understand it correctly that in essence this was not an error, only an unfortunate kernel log message?

@Zygo
Copy link
Owner

Zygo commented Jul 17, 2023

in essence this was not an error, only an unfortunate kernel log message?

Yes. The underlying error condition behind the message is expected, and the btrfs code already handles the error cases. The recent kernel code changes are all related to when the message should appear in the log.

@kakra
Copy link
Contributor

kakra commented Jul 17, 2023

There's still a problem with memory fragmentation, no matter the error log.

@Zygo
Copy link
Owner

Zygo commented Jul 17, 2023

There's still a problem with memory fragmentation, no matter the error log.

Yes, that's issue #260. Let's keep #257 about the kernel message, and #260 about the thing that is triggering it.

@kakra
Copy link
Contributor

kakra commented Jul 17, 2023

But in #260 (comment) you explicitly mention that this is a bug that was backported but the fix wasn't backported yet. Unless I got something wrong...

@Zygo
Copy link
Owner

Zygo commented Jul 17, 2023

But in #260 (comment) you explicitly mention that this is a bug that was backported but the fix wasn't backported yet.

Also here in #257 (comment). That means current LTS kernels 5.15 and 6.1 now have the 'vmalloc error' kernel messages that were fixed in 6.4.

The kernel changes would not affect any VM behavior, other than emitting the log message or not.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants