Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

unreliable test: crossFileContextUtil, for t1 group, should return repomap + opentabs context #6079

Open
Hweinstock opened this issue Nov 21, 2024 · 1 comment
Open

unreliable test: crossFileContextUtil, for t1 group, should return repomap + opentabs context #6079

Hweinstock opened this issue Nov 21, 2024 · 1 comment
Labels
tests-ci-cd

Comments

@Hweinstock
Copy link
Contributor

Hweinstock commented Nov 21, 2024 •
edited
Loading

Test Details

  • Name of test: crossFileContextUtil, fetchSupplementalContextForSrc, for t1 group, should return repomap + opentabs context:
  • OS of failure: Windows
  • VSCode version of failure (minimum/insider/stable): Insider
  • Link to failing run: https://github.com/aws/aws-toolkit-vscode/actions/runs/11938979186/job/33278472299
  • Link to failing test:

    aws-toolkit-vscode/packages/amazonq/test/unit/codewhisperer/util/crossFileContextUtil.test.ts

    Lines 64 to 97 in 2844554

    it('for t1 group, should return repomap + opentabs context', async function () {
    await toTextEditor(aStringWithLineCount(200), 'CrossFile.java', tempFolder, { preview: false })
    const myCurrentEditor = await toTextEditor('', 'TargetFile.java', tempFolder, {
    preview: false,
    })
    await assertTabCount(2)
    sinon.stub(FeatureConfigProvider.instance, 'getProjectContextGroup').returns('t1')
    sinon
    .stub(LspController.instance, 'queryInlineProjectContext')
    .withArgs(sinon.match.any, sinon.match.any, 'codemap')
    .resolves([
    {
    content: 'foo',
    score: 0,
    filePath: 'q-inline',
    },
    ])
    const actual = await crossFile.fetchSupplementalContextForSrc(myCurrentEditor, fakeCancellationToken)
    assert.ok(actual)
    assert.strictEqual(actual.supplementalContextItems.length, 4)
    assert.strictEqual(actual?.strategy, 'codemap')
    assert.deepEqual(actual?.supplementalContextItems[0], {
    content: 'foo',
    score: 0,
    filePath: 'q-inline',
    })
    assert.strictEqual(actual.supplementalContextItems[1].content.split('\n').length, 50)
    assert.strictEqual(actual.supplementalContextItems[2].content.split('\n').length, 50)
    assert.strictEqual(actual.supplementalContextItems[3].content.split('\n').length, 50)
    })

Log of Test Failure

1) crossFileContextUtil
       fetchSupplementalContextForSrc
         for t1 group, should return repomap + opentabs context:

      AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:

0 !== 4

      + expected - actual

      -0
      +4
      
      at Context.<anonymous> (D:\a\aws-toolkit-vscode\aws-toolkit-vscode\packages\amazonq\test\unit\codewhisperer\util\crossFileContextUtil.test.ts:86:20)

deleteTestTempDirs: deleted 103 test temp dirs
Error: 1 tests failed.
    at D:\a\aws-toolkit-vscode\aws-toolkit-vscode\packages\core\src\test\testRunner.ts:124:28
    at MultiReporters.done (D:\a\aws-toolkit-vscode\aws-toolkit-vscode\node_modules\mocha-multi-reporters\lib\MultiReporters.js:124:16)
    at done (D:\a\aws-toolkit-vscode\aws-toolkit-vscode\node_modules\mocha\lib\mocha.js:1007:16)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
Extension host test runner error Error: 1 tests failed.
    at D:\a\aws-toolkit-vscode\aws-toolkit-vscode\packages\core\src\test\testRunner.ts:124:28
Asking native host service to exit with code 1.
    at MultiReporters.done (D:\a\aws-toolkit-vscode\aws-toolkit-vscode\node_modules\mocha-multi-reporters\lib\MultiReporters.js:124:16)
    at done (D:\a\aws-toolkit-vscode\aws-toolkit-vscode\node_modules\mocha\lib\mocha.js:1007:16)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

Additional Information

Observed 3+ times.
@Hweinstock Hweinstock mentioned this issue Nov 21, 2024
Closed
@Will-ShaoHua Will-ShaoHua mentioned this issue Nov 21, 2024
Merged
jpinkney-aws pushed a commit that referenced this issue Nov 22, 2024
@Will-ShaoHua
test(amazonq): skip flaky crossfileContextUtil.test (#6085)
0e070f6 
## Problem
#6079

## Solution
Ignore the flaky test cases for now until we root caused why it failed
occasionally in some CI machines
@Hweinstock
Copy link
Contributor Author

Currently skipped in #6085

@justinmk3 justinmk3 mentioned this issue Dec 16, 2024
Closed
@Will-ShaoHua Will-ShaoHua mentioned this issue Dec 16, 2024
Merged
Will-ShaoHua added a commit that referenced this issue Dec 16, 2024
@Will-ShaoHua
fix(inline-suggestion): replace vscode.cancellation with waitUntil fo…
ad52466 
…r timeout (#6256)

## Problem

related issue: #6079,
#6252

caller
```
function main () {
       // init vscode cancellation token
       const cancellationToken
       setTimeout(100, () => {
              cancellationToken.cancel()
       })

      highlevelWrapperFetchSupplementalContext(editor, cancellationToken)
}

```

```
export function highlevelWrapperFetchSupplementalContext(editor, cancellationToken) {
       const supplementalContext = waitUntil(100, () => {
            // here always timeout and throw TimeoutException
            const opentabs =  await fetchOpenTabsContext(...)
            const projectContext = await fetchProjectContext()

           const result = []
           if (projectContext not empty) {
                    // push project context
           }

           if (opentabs not empty) {}
                   // push openttabs
           })  
          

         return result
}


async function fetchOpenTabsContext(editor, cancellationToken) {
      ....
      // VSC api call
}

async function fetchProjectContext() {
     ....
     // LSP call
}

```



After investigation, it looks like mix use of `vscode.CancellationToken`
and `waitUntil()` will likely cause cancellation token to be cancelled
prematurely (might be because another layer of waitUntil will run the
fetchOpenTabsContext asynchronously thus causing it to timeout
prematurely) therefore `fetchOpebtabsContext(..)` will return null in
this case and hence causing test cases failing.

Therefore, the issue here is actually not the test case itself and
they're failing due to race condition

## Solution
remove usage of cancellation token and only use waitUntil for timeout
purpose








## Functional testing

retrieved sup context as expected





### Case 1: repomap is available (there are local imports)
```
2024-12-16 13:10:15.616 [debug] CodeWhispererSupplementalContext:
    isUtg: false,
    isProcessTimeout: false,
    contentsLength: 14436,
    latency: 16.67179101705551
    strategy: codemap
    Chunk 0:
        Path: q-inline
        Length: 10209
        Score: 0
    Chunk 1:
        Path: /Volumes/workplace/ide/aws-toolkit-vscode-staging/packages/core/src/codewhisperer/service/serviceContainer.ts
        Length: 1486
        Score: 22.60257328587725
    Chunk 2:
        Path: /Volumes/workplace/ide/aws-toolkit-vscode-staging/packages/core/src/codewhisperer/tracker/lineTracker.ts
        Length: 1649
        Score: 19.106700952807103
    Chunk 3:
        Path: /Volumes/workplace/ide/aws-toolkit-vscode-staging/packages/core/src/codewhisperer/tracker/lineTracker.ts
        Length: 1092
        Score: 10.334690655691002
```

### Case 2: No repomap, should fallback to opentabs only

![image](https://github.com/user-attachments/assets/f59c11cf-0e34-40b8-8162-34b4d057673f)

```
2024-12-16 13:11:29.738 [debug] CodeWhispererSupplementalContext:
    isUtg: false,
    isProcessTimeout: false,
    contentsLength: 5046,
    latency: 16.311500012874603
    strategy: opentabs
    Chunk 0:
        Path: /Volumes/workplace/ide/aws-toolkit-vscode-staging/packages/core/src/codewhisperer/tracker/lineTracker.ts
        Length: 1564
        Score: 0
    Chunk 1:
        Path: /Volumes/workplace/ide/aws-toolkit-vscode-staging/packages/core/src/codewhisperer/tracker/lineTracker.ts
        Length: 1649
        Score: 0
    Chunk 2:
        Path: /Volumes/workplace/ide/aws-toolkit-vscode-staging/packages/core/src/codewhisperer/tracker/lineTracker.ts
        Length: 1833
        Score: 0
```





---

- Treat all work as PUBLIC. Private `feature/x` branches will not be
squash-merged at release time.
- Your code changes must meet the guidelines in
[CONTRIBUTING.md](https://github.com/aws/aws-toolkit-vscode/blob/master/CONTRIBUTING.md#guidelines).

License: I confirm that my contribution is made under the terms of the
Apache 2.0 license.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
tests-ci-cd
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Hweinstock