JVM crashes on Windows when using CRT client (AwsCrtAsyncHttpClient) for S3AsyncClient

[accelariosoftware]

Describe the bug

I am downloading files from S3 to local server using the AWS java SDK.
When using standard NettyNioAsyncHttpClient- everything works.
When using AwsCrtAsyncHttpClient- the JVM crashes

    SdkAsyncHttpClient httpClient = NettyNioAsyncHttpClient.builder()
      .maxConcurrency(parallel)
      .build();

    SdkAsyncHttpClient  crtHttpClient = AwsCrtAsyncHttpClient.builder()
      .maxConcurrency(parallel)
      .build();

    S3AsyncClient s3AsyncClient =
      S3AsyncClient.builder()
        .multipartEnabled(true)
        .credentialsProvider(() -> AwsBasicCredentials.create(awsAccessKey, awsSecretKey))
        .region(Region.of(region))
        .multipartConfiguration(MultipartConfiguration.builder()
          .minimumPartSizeInBytes(10 * MB)
          .apiCallBufferSizeInBytes(32 * MB)
          .build())
        .httpClient(crtHttpClient)    --- when using crtHttpClient then JVM craches, when using httpClient everything works
        .build();

    S3TransferManager transferManager =
      S3TransferManager.builder()
        .s3Client(s3AsyncClient)
        .build();

    DownloadFileRequest downloadFileRequest=  DownloadFileRequest.builder()
        .destination(Paths.get(destinationPath))
          .getObjectRequest(getObjectRequest -> getObjectRequest
            .bucket(bucket)
            .key(file))
        .build();

    transferManager.downloadFile(downloadFileRequest).completionFuture().join();

The error that is shown :

Fatal error condition occurred in C:\Program Files (x86)\Jenkins\workspace\aws-crt-java-build-dll-win64\aws-crt-java\crt\aws-c-io\source\windows\secure_channel_tls_handler.c:1105: !append_failed
Exiting Application
at 0x7FFA0DEA41BA: Java_software_amazon_awssdk_crt_auth_credentials_StsCredentialsProvider_stsCredentialsProviderNew
at 0x7FFA0DEA41BA: Java_software_amazon_awssdk_crt_auth_credentials_StsCredentialsProvider_stsCredentialsProviderNew
at 0x7FFA0DEA41BA: Java_software_amazon_awssdk_crt_auth_credentials_StsCredentialsProvider_stsCredentialsProviderNew
at 0x7FFA0DEA41BA: Java_software_amazon_awssdk_crt_auth_credentials_StsCredentialsProvider_stsCredentialsProviderNew
at 0x7FFA0DEA41BA: Java_software_amazon_awssdk_crt_auth_credentials_StsCredentialsProvider_stsCredentialsProviderNew
at 0x7FFA0DEA41BA: Java_software_amazon_awssdk_crt_auth_credentials_StsCredentialsProvider_stsCredentialsProviderNew
at 0x7FFA0DEA41BA: Java_software_amazon_awssdk_crt_auth_credentials_StsCredentialsProvider_stsCredentialsProviderNew
at 0x7FFA0DEA41BA: Java_software_amazon_awssdk_crt_auth_credentials_StsCredentialsProvider_stsCredentialsProviderNew
at 0x7FFA0DEA41BA: Java_software_amazon_awssdk_crt_auth_credentials_StsCredentialsProvider_stsCredentialsProviderNew
at 0x7FFA39F984C0: BaseThreadInitThunk
at 0x7FFA3A851770: RtlUserThreadStart

The trace for CRT is attached
crt_trace.log

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

CRT client works and JVM does not crash

Current Behavior

JVM crashes

Reproduction Steps

Run the code specified above

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

0.33.0

JDK version used

java version "1.8.0_421" Java(TM) SE Runtime Environment (build 1.8.0_421-b09) Java HotSpot(TM) 64-Bit Server VM (build 25.421-b09, mixed mode)

Operating System and version

Windows Server 2016

[debora-ito]

@accelariosoftware thank you for the CRT logs.

• Can you reproduce the issue reliably, or is it intermittent?
• What is the file size?
• Can you generate the JVM crash report?

[accelariosoftware]

I can reproduce the issue reliably. The JVM crashes 9 times out of 10.

The real size of the file on S3 is 390MB
When JVM crashes the downloaded file has different size after each failure - sometimes 10MB, sometimes 30MB .

There is no JVM crash report file generated (no hs_err_pid... files at all)
I have tried to force its generation by using -XX:+CreateMinidumpOnCrash, -XX:ErrorFile=..., registry HKLM\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps - no luck.
The only thing is the event viewer entry:

Faulting application name: java.exe, version: 8.0.4320.6, time stamp: 0x67124093
Faulting module name: ucrtbase.dll, version: 10.0.14393.3659, time stamp: 0x5e9140a1
Exception code: 0xc0000409
Fault offset: 0x000000000006de4e
Faulting process id: 0x1984
Faulting application start time: 0x01db3e5310b7f85d
Faulting application path: C:\Program Files\OpenLogic\jdk-8.0.432.06-hotspot\bin\java.exe
Faulting module path: C:\Windows\System32\ucrtbase.dll
Report Id: 8c5ec996-9f9c-440f-9a13-bbd3428a4fe0
Faulting package full name: 
Faulting package-relative application ID: 

Also, this dialog box pops up on every JVM crash

[nedy13]

A tip from me. We also use JNI and JNA functions in our software and sometimes we need more information in the event of a crash.

For this we use the tool “procdump (sysinternals)” with the parameters procdump -ma -t <pid>. Minus t creates a dump on exit which happened on a crash ;-)
Then we have a memory dump of the crash and can read out the stacktrace of the faulty thread, etc.

[debora-ito]

@nedy13 to the rescue!

@accelariosoftware we'll need more logs to troubleshoot this further. I don't see anything standing out in the CRT logs. Can you try to generate a memory dump with procdump?

[accelariosoftware]

@debora-ito Here it is: dump file