Running huggingface model benchmark

[jerryzh168]

with the same PR: #2519

Running python run_benchmark.py torchao --performance --inference --bfloat16 --inductor-compile-mode max-autotune --huggingface --ci

Gives error:

  File "/home/jerryzh/anaconda3/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 948, in inline_user_function_return
    return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  File "/home/jerryzh/anaconda3/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 3103, in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
  File "/home/jerryzh/anaconda3/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 3231, in inline_call_
    tracer.run()
  File "/home/jerryzh/anaconda3/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1095, in run
    while self.step():
  File "/home/jerryzh/anaconda3/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1007, in step
    self.dispatch_table[inst.opcode](self, inst)
  File "/home/jerryzh/anaconda3/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1856, in LOAD_ATTR
    self._load_attr(inst)
  File "/home/jerryzh/anaconda3/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1846, in _load_attr
    result = BuiltinVariable(getattr).call_function(
  File "/home/jerryzh/anaconda3/envs/ao/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 1000, in call_function
    return handler(tx, args, kwargs)
  File "/home/jerryzh/anaconda3/envs/ao/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 848, in builtin_dispatch
    rv = fn(tx, args, kwargs)
  File "/home/jerryzh/anaconda3/envs/ao/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 766, in call_self_handler
    result = self_handler(tx, *args, **kwargs)
  File "/home/jerryzh/anaconda3/envs/ao/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 1716, in call_getattr
    return obj.var_getattr(tx, name)
  File "/home/jerryzh/anaconda3/envs/ao/lib/python3.10/site-packages/torch/_dynamo/variables/base.py", line 244, in var_getattr
    value = self.const_getattr(tx, name)
  File "/home/jerryzh/anaconda3/envs/ao/lib/python3.10/site-packages/torch/_dynamo/variables/constant.py", line 130, in const_getattr
    member = getattr(self.value, name)
torch._dynamo.exc.InternalTorchDynamoError: AttributeError: 'str' object has no attribute 'size'

from user code:
   File "/home/jerryzh/anaconda3/envs/ao/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 978, in forward
    outputs = self.albert(
  File "/home/jerryzh/anaconda3/envs/ao/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 692, in forward
    input_shape = input_ids.size()

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information

[anijain2305]

cc @williamwen42 @mlazos if you have time to go through this.

[williamwen42]

Did a little debugging - error still occurs even with torch.compile disabled (using set_stance("force_eager")).

I think the bug is caused here: https://github.com/jerryzh168/benchmark/blob/ec965d54ec52414af8edd5b009da5a64f22fb812/userbenchmark/dynamo/dynamobench/torchao_backend.py#L66 inputs is a dict but benchmark_models assumes a tuple input (https://github.com/pytorch/ao/blob/ae77f4066fe4966153f769625cba32cbef5cad51/torchao/utils.py#L56). So inputs is being unpacked incorrectly.

[jerryzh168]

Did a little debugging - error still occurs even with torch.compile disabled (using set_stance("force_eager")).

I think the bug is caused here: jerryzh168/benchmark@ec965d5/userbenchmark/dynamo/dynamobench/torchao_backend.py#L66 inputs is a dict but benchmark_models assumes a tuple input (pytorch/ao@ae77f40/torchao/utils.py#L56). So inputs is being unpacked incorrectly.

Ah I see, makes sense, thanks!

[jerryzh168]

verified that this one is because of dictionary input, it's resolved now, thanks for your help! @williamwen42