Adding asan sanitizer support for hip #795

kab163 · 2022-11-23T23:19:39Z

Cmake command on rzvernal:

cmake -DROCM_ROOT_DIR=/opt/rocm-5.3.0 -DHIP_ROOT_DIR=/opt/rocm-5.3.0/hip -DHIP_PATH=/opt/rocm-5.3.0/llvm/bin -DENABLE_HIP=On -DENABLE_OPENMP=Off -DENABLE_CUDA=Off -DUMPIRE_ENABLE_ASAN=On -DCMAKE_CXX_COMPILER=/opt/rocm-5.3.0/llvm/bin/amdclang++ -DCMAKE_C_COMPILER=/opt/rocm-5.3.0/llvm/bin/amdclang -DCMAKE_CXX_FLAGS="-fsanitize=address" -DUMPIRE_ENABLE_SANITIZER_TESTS=ON -DCMAKE_HIP_ARCHITECTURES=gfx90a:xnack+ ../

Warning while compiling: clang-15: warning: ignoring '-fsanitize=address' option for offload arch 'gfx90a' as it is not currently supported there. Use it with an offload arch containing 'xnack+' instead [-Woption-ignored]

Output of sanitizer_tests.cpp

[belcher6@rzvernal11:build]$ ./bin/sanitizer_tests QuickPool write
data[INDEX] = 100
=================================================================
==79729==ERROR: AddressSanitizer: use-after-poison on address 0x154d16201530 at pc 0x0000004f5eb4 bp 0x7fffffff8b10 sp 0x7fffffff8b08
WRITE of size 8 at 0x154d16201530 thread T0
    #0 0x4f5eb3 in test_write_after_free() (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x4f5eb3)
    #1 0x4f67cb in main (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x4f67cb)
    #2 0x15555267acf2 in __libc_start_main (/lib64/libc.so.6+0x3acf2) (BuildId: 64aa558dcdda2d8b0d7b04cef33ddbb2d9d8b8b4)
    #3 0x404e4d in _start (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x404e4d)

Address 0x154d16201530 is a wild pointer inside of access range of size 0x000000000008.
SUMMARY: AddressSanitizer: use-after-poison (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x4f5eb3) in test_write_after_free()
Shadow bytes around the buggy address:
  0x02aa22c38250: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c38260: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c38270: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c38280: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c38290: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
=>0x02aa22c382a0: f7 f7 f7 f7 f7 f7[f7]f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c382b0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c382c0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c382d0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c382e0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c382f0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==79729==ABORTING
[belcher6@rzvernal11:build]$ ./bin/sanitizer_tests QuickPool read
data[INDEX] = 100
=================================================================
==79736==ERROR: AddressSanitizer: use-after-poison on address 0x154d16200800 at pc 0x0000004f5c22 bp 0x7fffffff8b30 sp 0x7fffffff8b28
READ of size 8 at 0x154d16200800 thread T0
    #0 0x4f5c21 in test_read_after_free() (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x4f5c21)
    #1 0x4f679e in main (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x4f679e)
    #2 0x15555267acf2 in __libc_start_main (/lib64/libc.so.6+0x3acf2) (BuildId: 64aa558dcdda2d8b0d7b04cef33ddbb2d9d8b8b4)
    #3 0x404e4d in _start (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x404e4d)

Address 0x154d16200800 is a wild pointer inside of access range of size 0x000000000008.
SUMMARY: AddressSanitizer: use-after-poison (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x4f5c21) in test_read_after_free()
Shadow bytes around the buggy address:
  0x02aa22c380b0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c380c0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c380d0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c380e0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c380f0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
=>0x02aa22c38100:[f7]f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c38110: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c38120: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c38130: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c38140: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c38150: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==79736==ABORTING

kab163 · 2022-12-06T18:03:55Z

Updated cmake command:

cmake -DROCM_ROOT_DIR=/opt/rocm-5.4.0 -DHIP_ROOT_DIR=/opt/rocm-5.4.0/hip -DHIP_PATH=/opt/rocm-5.4.0/llvm/bin -DENABLE_HIP=On -DENABLE_OPENMP=Off -DENABLE_CUDA=Off -DUMPIRE_ENABLE_ASAN=On -DCMAKE_CXX_COMPILER=/opt/rocm-5.4.0/llvm/bin/amdclang++ -DCMAKE_C_COMPILER=/opt/rocm-5.4.0/llvm/bin/amdclang -DCMAKE_CXX_FLAGS="-fsanitize=address" -DUMPIRE_ENABLE_SANITIZER_TESTS=ON -DCMAKE_HIP_ARCHITECTURES="gfx90a:xnack+" -DGPU_TARGETS="gfx90a:xnack+" -DAMDGPU_TARGETS="gfx90a:xnack+" ../

Updated Output (for hip kernel sanitizer test):

bash-4.4$ ./bin/sanitizer_tests QuickPool read
AddressSanitizer:DEADLYSIGNAL
=================================================================
==333094==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000020 (pc 0x155553a4b9cd bp 0x7fffffff7ab0 sp 0x7fffffff7a50 T0)
==333094==The signal is caused by a READ memory access.
==333094==Hint: address points to the zero page.
    #0 0x155553a4b9cd  (/opt/rocm-5.4.0/lib/libamdhip64.so.5+0xd59cd) (BuildId: a4e47799322466785f920489031c315137965c08)
    #1 0x155553a4bb26  (/opt/rocm-5.4.0/lib/libamdhip64.so.5+0xd5b26) (BuildId: a4e47799322466785f920489031c315137965c08)
    #2 0x155553a4c9fe  (/opt/rocm-5.4.0/lib/libamdhip64.so.5+0xd69fe) (BuildId: a4e47799322466785f920489031c315137965c08)
    #3 0x155553a0a5d6  (/opt/rocm-5.4.0/lib/libamdhip64.so.5+0x945d6) (BuildId: a4e47799322466785f920489031c315137965c08)
    #4 0x155553b5de15  (/opt/rocm-5.4.0/lib/libamdhip64.so.5+0x1e7e15) (BuildId: a4e47799322466785f920489031c315137965c08)
    #5 0x155553b39881  (/opt/rocm-5.4.0/lib/libamdhip64.so.5+0x1c3881) (BuildId: a4e47799322466785f920489031c315137965c08)
    #6 0x155553b3af93 in hipLaunchKernel (/opt/rocm-5.4.0/lib/libamdhip64.so.5+0x1c4f93) (BuildId: a4e47799322466785f920489031c315137965c08)
    #7 0x543a21 in __device_stub__test_for_hip(double**, unsigned long) (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x543a21)
    #8 0x544cdf in main (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x544cdf)
    #9 0x15555267dcf2 in __libc_start_main (/lib64/libc.so.6+0x3acf2) (BuildId: 64aa558dcdda2d8b0d7b04cef33ddbb2d9d8b8b4)
    #10 0x45285d in _start (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x45285d)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/opt/rocm-5.4.0/lib/libamdhip64.so.5+0xd59cd) (BuildId: a4e47799322466785f920489031c315137965c08)
==333094==ABORTING

Note: Will not work if HSA_XNACK is set to 1, do not export... explanation TBD

kab163 · 2022-12-07T23:40:59Z

Issue: it seems like running the test with ./bin/sanitizer_tests QuickPool read HOST and ./bin/sanitizer_tests QuickPool write HOST both give an error that detects a write access....
(Whether it is build with hip support or without it)

kab163 · 2022-12-08T00:28:32Z

The way this test is ran in make test might need to be changed...

kab163 · 2022-12-21T17:22:57Z

See AMD bug report (277) - made reproducer for AMD to look in to. Seems like this does not yet work as expected in rocm 5.4

kab163 · 2023-03-02T22:49:36Z

Update: A fix is supposed to be ready in rocm 6.0

…an-poisoning-for-hip

kab163 · 2023-04-03T16:07:23Z

Edited out...

…an-poisoning-for-hip

…-1020-asan-poisoning-for-hip

kab163 · 2024-01-22T18:23:23Z

The fix is in rocm/6.0.0, but we have to add a "-Wl,-rpath=/opt/rocm-6.0.0/llvm/lib/clang/17.0.0/lib/linux/" flag. Once we update the corona/tioga jobs to rocm-6.0.0 and get that in, it should be good to go.

kab163 · 2024-01-24T16:12:28Z

This has been a long time coming! @adrienbernede - once the blt/camp releases come out and we can update umpire to rocm/6.0.0 we should add this test! (No updates yet for the reason behind why the device allocator test is failing with 6.0.0 - ticket has been "submitted to vendor", so it is being looked into at least...)

adrienbernede · 2024-06-07T13:05:43Z

@davidbeckingsale +tools currently conflicts with +rocm. Depending on what you want to achieve, you’ll want to either turn off tools or remove the conflict.

adrienbernede · 2024-06-10T14:10:14Z

@kab163 @davidbeckingsale now that the allocation is not limiting anymore I’ll let you deal with the rest ;).

adding asan sanitizer support for hip

a7e72b3

kab163 requested a review from davidbeckingsale November 23, 2022 23:20

github-actions and others added 3 commits November 23, 2022 23:20

Apply style updates

f90a7af

WIP adding a hip kernel sanitizer test

00e3f43

Apply style updates

ac5cd62

kab163 and others added 7 commits December 7, 2022 11:39

adding usage checkers to sanitizer test

522a4a9

Apply style updates

3ac6a7c

more updates to sanitizer test for error checking

a4f007a

merging...

a1c0180

Apply style updates

8b9e194

adding hip define macros

6617aba

Apply style updates

4c17280

getting rid of double pointer

5379807

kab163 and others added 2 commits December 21, 2022 09:31

updating sanitizer test

4e4804f

Apply style updates

4b92502

Merge branch 'develop' of github.com:LLNL/Umpire into task/UM-1020-as…

276f15b

…an-poisoning-for-hip

kab163 added 2 commits September 18, 2023 08:06

Merge branch 'develop' of github.com:LLNL/Umpire into task/UM-1020-as…

986c5a1

…an-poisoning-for-hip

Merge branch 'develop' of https://github.com/LLNL/Umpire into task/UM…

3858c0f

…-1020-asan-poisoning-for-hip

kab163 added 3 commits January 22, 2024 10:40

adding flag for hip asan

e9bf9f6

updates for rocm test

0232f2c

had to update the test runner to handle new paramter

f154c08

kab163 marked this pull request as ready for review January 24, 2024 16:10

kab163 requested a review from adrienbernede January 24, 2024 16:13

adrienbernede mentioned this pull request Mar 12, 2024

[WIP] [Woptim] [email protected] #874

Closed

This was referenced Jun 3, 2024

Use rocm 6.1.0 in CI on Tioga #883

Closed

Use [email protected] and update Spack #885

Merged

davidbeckingsale added 2 commits June 6, 2024 10:46

Merge branch 'develop' into task/UM-1020-asan-poisoning-for-hip

2019270

Remove umpire namespace from fmt call

417b66d

Allow +tools+rocm in Umpire package

43f0875

rhornung67 previously approved these changes Jun 7, 2024

View reviewed changes

davidbeckingsale and others added 4 commits June 7, 2024 11:01

Add amdgpu_target for new memleak job

f76f5be

remove extra linker flags

f2eb907

Add xnack+ to target for memleak job

1332301

Increase allocation on tioga due to memleak job

9022631

Merge branch 'develop' into task/UM-1020-asan-poisoning-for-hip

c458f02

kab163 dismissed rhornung67’s stale review via c458f02 July 16, 2024 17:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding asan sanitizer support for hip #795

Adding asan sanitizer support for hip #795

kab163 commented Nov 23, 2022 •

edited

Loading

kab163 commented Dec 6, 2022 •

edited

Loading

kab163 commented Dec 7, 2022 •

edited

Loading

kab163 commented Dec 8, 2022

kab163 commented Dec 21, 2022

kab163 commented Mar 2, 2023

kab163 commented Apr 3, 2023 •

edited

Loading

kab163 commented Jan 22, 2024 •

edited

Loading

kab163 commented Jan 24, 2024 •

edited

Loading

adrienbernede commented Jun 7, 2024 •

edited

Loading

adrienbernede commented Jun 10, 2024

Adding asan sanitizer support for hip #795

Are you sure you want to change the base?

Adding asan sanitizer support for hip #795

Conversation

kab163 commented Nov 23, 2022 • edited Loading

kab163 commented Dec 6, 2022 • edited Loading

kab163 commented Dec 7, 2022 • edited Loading

kab163 commented Dec 8, 2022

kab163 commented Dec 21, 2022

kab163 commented Mar 2, 2023

kab163 commented Apr 3, 2023 • edited Loading

kab163 commented Jan 22, 2024 • edited Loading

kab163 commented Jan 24, 2024 • edited Loading

adrienbernede commented Jun 7, 2024 • edited Loading

adrienbernede commented Jun 10, 2024

kab163 commented Nov 23, 2022 •

edited

Loading

kab163 commented Dec 6, 2022 •

edited

Loading

kab163 commented Dec 7, 2022 •

edited

Loading

kab163 commented Apr 3, 2023 •

edited

Loading

kab163 commented Jan 22, 2024 •

edited

Loading

kab163 commented Jan 24, 2024 •

edited

Loading

adrienbernede commented Jun 7, 2024 •

edited

Loading