Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Post-processor crashing makes the worker stall #88

Open
db0 opened this issue Jan 16, 2024 · 0 comments
Open

Post-processor crashing makes the worker stall #88

db0 opened this issue Jan 16, 2024 · 0 comments

Comments

@db0
Copy link
Member

db0 commented Jan 16, 2024

From the logs of cozmyc, I noticed some weird OOM errors about Post-processors even though there should be more than enough VRAM. I don't quite understand why since I thought the CPU uses the RAM.

(Cozmyc has a very old CPU, so this is probably relevant)

2024-01-16 09:43:59.153 | ERROR    | hordelib.comfy_horde:send_sync:666 - execution_error, {'prompt_id': 'b0749e50-0fc3-423e-b157-d72a8511b395', 'node_id': 'face_restore_with_model', 'node_type': 'FaceRestoreWithModel', 'executed': ['model_loader', 'image_loader'], 'exception_message': 'Unable to allocate 384. MiB for an array with shape (4096, 4096, 3) and data type float64', 'exception_type': 'numpy.core._exceptions._ArrayMemoryError', 'traceback': ['  File "C:\\Users\\santiago\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\site-packages\\hordelib\\_comfyui\\execution.py", line 154, in recursive_execute\n    output_data, output_ui = get_output_data(obj, input_data_all)\n', '  File "C:\\Users\\santiago\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\site-packages\\hordelib\\_comfyui\\execution.py", line 84, in get_output_data\n    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)\n', '  File "C:\\Users\\santiago\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\site-packages\\hordelib\\_comfyui\\execution.py", line 77, in map_node_over_list\n    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))\n', '  File "C:\\Users\\santiago\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\site-packages\\hordelib\\nodes\\facerestore\\__init__.py", line 180, in restore_face\n    restored_img = face_helper.paste_faces_to_input_image()\n', '  File "C:\\Users\\santiago\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\site-packages\\hordelib\\nodes\\facerestore\\facelib\\utils\\face_restoration_helper.py", line 527, in paste_faces_to_input_image\n    inv_soft_mask * pasted_face + (1 - inv_soft_mask) * upsample_img\n'],

We should look at our error crashing in the PP process to make it fail more gracefully and inform the user.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant