AttributeError: 'Parameter' object has no attribute 'grad_sample' in inception v3

[mr-lz]

🐛 Bug

I manually replaced batch_norm with group_norm in inception_v3, but there is still a problem: AttributeError: 'Parameter' object has no attribute 'grad_sample'.
And this is the trace:

Traceback (most recent call last):
  File "e:\jupyter\LearnTorch\deep learning with privacy\cifar10_inception3_dp.py", line 139, in <module>
    main()
  File "e:\jupyter\LearnTorch\deep learning with privacy\cifar10_inception3_dp.py", line 126, in main
    train(model, DEVICE, test_loader, optimizer, epoch)
  File "e:\jupyter\LearnTorch\deep learning with privacy\cifar10_inception3_dp.py", line 65, in train
    optimizer.step()
  File "F:\Anaconda3\envs\pytorch_gpu\lib\site-packages\opacus\privacy_engine.py", line 197, in dp_step
    self.privacy_engine.step()
  File "F:\Anaconda3\envs\pytorch_gpu\lib\site-packages\opacus\privacy_engine.py", line 280, in step
    self.clipper.clip_and_accumulate()
  File "F:\Anaconda3\envs\pytorch_gpu\lib\site-packages\opacus\per_sample_gradient_clip.py", line 182, in clip_and_accumulate
    flat=not self.norm_clipper.is_per_layer,
  File "F:\Anaconda3\envs\pytorch_gpu\lib\site-packages\opacus\utils\tensor_utils.py", line 38, in calc_sample_norms
    norms = [param.view(len(param), -1).norm(2, dim=-1) for name, param in named_params]
  File "F:\Anaconda3\envs\pytorch_gpu\lib\site-packages\opacus\utils\tensor_utils.py", line 38, in <listcomp>
    norms = [param.view(len(param), -1).norm(2, dim=-1) for name, param in named_params]
  File "F:\Anaconda3\envs\pytorch_gpu\lib\site-packages\opacus\per_sample_gradient_clip.py", line 263, in <genexpr>
    if p.requires_grad
AttributeError: 'Parameter' object has no attribute 'grad_sample'

github repository

Sorry, I haven't used coLab.
here is my code: https://github.com/mr-lz/inception_v3_dp/tree/master , you can run it with python cifar10_inception3_dp.py

Environment

PyTorch version: 1.6.0
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 家庭版
GCC version: (x86_64-posix-seh-rev0, Built by MinGW-W64 project) 8.1.0
Clang version: Could not collect
CMake version: Could not collect

Python version: 3.6 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 10.2.89
GPU models and configuration: GPU 0: GeForce GTX 960M
Nvidia driver version: 451.67
cuDNN version: F:\Program Files\NVDIA GPU Computing Toolkit\CUDA\v10.2\bin\cudnn_ops_train64_8.dll
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.1
[pip3] torch==1.6.0
[pip3] torchcsprng==0.1.2
[pip3] torchtext==0.7.0
[pip3] torchvision==0.7.0
[conda] blas 1.0 mkl https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
[conda] captum 0.3.0 0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch
[conda] cpuonly 1.0 0 pytorch
[conda] cudatoolkit 10.2.89 h74a9793_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
[conda] mkl 2020.2 256 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
[conda] mkl-service 2.3.0 py36hb782905_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
[conda] mkl_fft 1.1.0 py36h45dec08_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
[conda] mkl_random 1.1.1 py36h47e9c7a_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
[conda] numpy 1.19.1 py36h5510c5b_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
[conda] numpy-base 1.19.1 py36ha3acd2a_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
[conda] pytorch 1.6.0 py3.6_cuda102_cudnn7_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch
[conda] torchcsprng 0.1.2 py36_cpu [cpuonly] pytorch
[conda] torchtext 0.7.0 py36 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch
[conda] torchvision 0.7.0 py36_cu102 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch

[alexandresablayrolles]

Thanks for raising this. We are able to repro the bug, it looks like the grad_sample is not computed in AuxLogits, we are currently looking into it.

[alexandresablayrolles]

So the main problem seems to be that the AuxLogits do not receive any gradients. I would suggest the following changes: output = model(data) line 62 and model = Inception3(num_classes=10, aux_logits=False).to(DEVICE) line 110.

[mr-lz]

It is now working properly, thank you!

[sword-king1]

@alexandresablayrolles I'm also having the same issue：AttributeError: 'Parameter' object has no attribute 'grad_sample'.
I've done my best to find the problem, but I still have this error, could you help me?

[karthikprasad]

Hello @sword-king1 , could you provide repro code on colab for us to help debug this? Also, could you open a new issue? This issue references a very old version of Opacus and a lot has changed since then.

[sword-king1]

@karthikprasad Thank you for replying to me.And sorry for the time it took so long to get back to you. I open a new issue.#548