[QUESTION] Models not using GPU as expected

[theTov]

Question and Issue

Installed Nexa SDK using option 2 : GPU ( vulcan)

Followed these instructions using $env:CMAKE_ARGS="-DGGML_VULKAN=on"; pip install nexaai --prefer-binary --index-url https://nexaai.github.io/nexa-sdk/whl/vulkan --extra-index-url https://pypi.org/simple --no-cache-dir

However when running the script my CPU peaks at 100% while my GPU is idle. Script stops running processing the 122222_77.docx file throwing the below errors.

changes made to Main.py
n_ctx=4096, # Increase context window
backend="vulkan" # Ensure Vulkan is used

no dice, some help would be welcomed.

Processing c8fed35720671596882b0c6a2ad9586d.gif ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:03:35
Processing IMG_0967.PNG ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:01:38
Processing logo.png ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:01:11
Processing animal.jpg ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:01:26
Processing 1.xlsx ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:40
Processing 12222_777.docx ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:02:32
Traceback (most recent call last):
File "C:\Home_lab\Repos\Local-File-Organizer-main\main.py", line 339, in
main()
File "C:\Home_lab\Repos\Local-File-Organizer-main\main.py", line 254, in main
data_texts = process_text_files(text_tuples, text_inference, silent=silent_mode, log_file=log_file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Home_lab\Repos\Local-File-Organizer-main\text_data_processing.py", line 60, in process_text_files
data = process_single_text_file(args, text_inference, silent=silent, log_file=log_file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Home_lab\Repos\Local-File-Organizer-main\text_data_processing.py", line 37, in process_single_text_file
foldername, filename, description = generate_text_metadata(text, file_path, progress, task_id, text_inference)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Home_lab\Repos\Local-File-Organizer-main\text_data_processing.py", line 93, in generate_text_metadata
filename_response = text_inference.create_completion(filename_prompt)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Home_lab\miniconda3\envs\local_file_organizer\Lib\site-packages\nexa\gguf\nexa_inference_text.py", line 266, in create_completion
return self.model.create_completion(prompt=prompt, **params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Home_lab\miniconda3\envs\local_file_organizer\Lib\site-packages\nexa\gguf\llama\llama.py", line 1746, in create_completion
completion: Completion = next(completion_or_chunks) # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Home_lab\miniconda3\envs\local_file_organizer\Lib\site-packages\nexa\gguf\llama\llama.py", line 1190, in _create_completion
raise ValueError(
ValueError: Requested tokens (2153) exceed context window of 2048
Script execution failed. Please check your script.
Press any key to continue . . .

OS

windows 11

Python Version

3.12.7

Nexa SDK Version

0.0.9.2

GPU (if using one)

RX 5700 XT

[Davidqian123]

Hi, when using vulkan backend, make sure you have alreday installed vulkan sdk in advancd https://vulkan.lunarg.com/sdk/home

[theTov]

Thanks @Davidqian123 , yes I did install the Sdk prior to installing the Nexa SDK

also updated AMD drivers

[Davidqian123]

Could you provide vulkan sdk version and AMD driver version? Or I'm not sure if RX 5700 XT support ROCM backend?