-
Notifications
You must be signed in to change notification settings - Fork 120
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
is this only for linux? #957
Comments
it might be the torchao version is too low: "torchao==0.1", we introduced quantize_ in 0.4.0 I think: https://github.com/pytorch/ao/releases, but in the meantime our packages are only available in linux and mac right now I think |
Can you try updating torchao? I don't think the top level But from what I understand, torch.compile() does not work on windows because triton is lacking windows support, which we use to codegen our quantization kernels, so I wouldn't expect this to work on windows. |
i am about to test latest @jerryzh168 thank you will write here |
what pip finds latest version is ERROR: Could not find a version that satisfies the requirement torchao==0.4.0 (from versions: 0.0.1, 0.0.3, 0.1) how can i install latest on windows? python 3.10 windows 10 if you have wheel link i can directly install |
we don't have a windows build today I think, cc @atalman can you provide a pointer to support a build in windows as well? |
awesome waiting to test thank you so much |
You can probably install torchao from source. If you don't need the CUDA extensions, you can do USE_CPP=0 pip install git+https://github.com/pytorch/ao But again, since torch.compile() doesn't work in windows, it's not very useful. |
is there a way to make the quantizations work on windows+nvidia gpu without torch.compile and inductor backend? I am mostly concerned about inference speedups. |
I'm also in need of the wheel for torchao on Windows to get Quantization working for Flux, CogVideoX, etc. in my app. I'm fine without compile, but the other features are really needed to optimize vram. Tried installing from the github and running setup.py install from clone, but gave me errors. Hoping we can run something newer than v0.1 soon.. Thanks. |
so true from this logic (that use linux not windows) why do we even have Python on Windows?, PyTorch on Windows? xFormers on Windows? if such stuff is not necessary on Windows? I don't get logic of forcing people to use Linux. If we follow this mindset, why we have all these on Windows? |
If you don't need the CUDA extensions (right now they are only for backing FPx and sparse marlin kernels I think), and you don't mind the lack of torch.compile() support, you can install torchao from source on Windows like I mentioned previously set USE_CPP=0
pip install git+https://github.com/pytorch/ao I don't have access to a Windows machine right now, so I just googled how to set environment variable on Windows here. You might need to adjust accordingly. You are welcome to improve torchao experience on Windows. In fact, there are past PRs by the community, including me, that help build torchao successfully on Windows, including CUDA extension support. |
Thanks for the reply. I have a couple of clarifying questions - It seems that previously one was able to build torchao with cuda extension support on windows. What changed since then? Also, since torch.compile is not available on windows, what kind of speedups(if any) on gpu- can we expect for normal pytorch models quantized by torchao |
@abhi-vandit Since there is no Windows CI, there is no guarantee that new CUDA extensions in torchao can be built correctly on Windows. However, most of the errors usually come from Unix-specific features, thus the fix is usually simple e.g. #951 #396. I think torchao welcome small fixes like these. I mentioned not building CUDA extensions previously since usually it's quite involved to set up C++ and CUDA compiler on Windows. So if you don't need the CUDA extensions, it's not really worth the efforts.
I think most likely you will only see slow down. Perhaps you can still get some memory savings. |
@gau-nernst Thanks for the prompt reply. Hope this changes in near future and we are able to utilize quantization for inference time speedups on windows as well. |
this worked
|
I installed on windows and failing
from torchao.quantization import quantize_
pip freeze
The text was updated successfully, but these errors were encountered: