Problem Categories

[normster]

Hi,

In the report some GPT-4 annotated categories of problems were shown. Would it be possible to share these categories?

Thanks!

[terryyz]

Hi @normster,

Sorry for forgetting to upload the categories. You should be able to see them here. Note that the problem categories are based on the manually labeled library domain categories.

You may also notice that lib2domain.json contains more libraries than the ones covered by BigCodeBench. The list is based on the BigCodeBench, ODEX, and DS-1000.

Cheers

[terryyz]

Ah, I realised that you may be looking for the categorisations in the preference selection phase. I'll share it with you shortly.

[terryyz]

Hi @normster, please check the zip file under https://github.com/bigcode-project/bigcodebench-annotation/blob/main/data_collection/r0.zip, which should cover all the scripts and raw data.

[terryyz]

FYI, I plan to classify the BigCodeBench tasks into several application categories and release the breakdown analysis shortly.