[python] Atom slicing issues

[HendrikSchrieber]

Hi again,

I am having issues with atom slicing not working when using the official cdxgen-deno Docker Image:

$ docker run --rm -v /tmp:/tmp -e CDXGEN_DEBUG_MODE=debug -v $PWD:/app:rw ghcr.io/cyclonedx/cdxgen-deno:v10.8.1 -o /app/sbom.json -r 
/app -t python
cdxgen will now attempt to generate an SBOM for 'build' lifecycle phase for Python. This would take some time ...
To speed up this step, invoke cdxgen from within a virtual environment with all the dependencies installed.
Alternatively, pass the argument '--lifecycle pre-build' to generate a faster but less precise SBOM.
Using the virtual environment /tmp/cdxgen-venv-c6qqi7
About to construct the pip dependency tree based on /app/requirements.txt. Please wait ...
Using the virtual environment /tmp/cdxgen-venv-c6qqi7
About to construct the pip dependency tree based on /app/requirements-dev.txt. Please wait ...
Executing atom parsedeps -l python -o /tmp/atom-deps-5E2Gr4/app.atom --slice-outfile /deps.slices.json /app
Slicing was not successful. For large projects (> 1 million lines of code), try running atom cli externally in Java mode. Please refer to the instructions in https://github.com/CycloneDX/cdxgen/blob/master/ADVANCED.md.
NOTE: Atom is in detached mode and will continue to run in the background with max CPU and memory unless it's killed.

The project is very small and creating the slices is working when calling atom on the generated sbom without Docker:

$ atom parsedeps -l python -o app.atom --slice-outfile deps.slices.json .
Slices have been successfully written to /app/deps.slices.json

I also tried the unofficial python3.10 image which also seems to work concerning the slicing but is producing a different issue with schema validation:

$ docker run --rm -e CDXGEN_DEBUG_MODE=debug -v /tmp:/tmp -v $(pwd):/app:rw -t ghcr.io/appthreat/cdxgen-python310:v10 -r /app -o /app
/sbom.json -t python
cdxgen will now attempt to generate an SBOM for 'build' lifecycle phase for Python. This would take some time ...
To speed up this step, invoke cdxgen from within a virtual environment with all the dependencies installed.
Alternatively, pass the argument '--lifecycle pre-build' to generate a faster but less precise SBOM.
Using the virtual environment /tmp/cdxgen-venv-aSWPjl
Executing /tmp/cdxgen-venv-aSWPjl/bin/python -m pip install --disable-pip-version-check -r /app/requirements.txt
About to construct the pip dependency tree based on /app/requirements.txt. Please wait ...
Using the python executable /tmp/cdxgen-venv-aSWPjl/bin/python
Using the virtual environment /tmp/cdxgen-venv-aSWPjl
Executing /tmp/cdxgen-venv-aSWPjl/bin/python -m pip install --disable-pip-version-check -r /app/requirements-dev.txt
About to construct the pip dependency tree based on /app/requirements-dev.txt. Please wait ...
Using the python executable /tmp/cdxgen-venv-aSWPjl/bin/python
Executing node /usr/lib/node_modules/@cyclonedx/cdxgen/node_modules/@appthreat/atom/index.js parsedeps -l python -o /tmp/atom-deps-jQLgs0/app.atom --slice-outfile /deps.slices.json /app
Slices have been successfully written to /deps.slices.json

Schema validation failed for app
[
  {
    instancePath: '/formulation/0/workflows/0/inputs/0',
    schemaPath: '#/oneOf/0/required',
    keyword: 'required',
    params: { missingProperty: 'resource' },
    message: "must have required property 'resource'",
    schema: [ 'resource' ],
    parentSchema: { required: [Array] },
    data: {}
  },
  {
    instancePath: '/formulation/0/workflows/0/inputs/0',
    schemaPath: '#/oneOf/1/required',
    keyword: 'required',
    params: { missingProperty: 'parameters' },
    message: "must have required property 'parameters'",
    schema: [ 'parameters' ],
    parentSchema: { required: [Array] },
    data: {}
  },
  {
    instancePath: '/formulation/0/workflows/0/inputs/0',
    schemaPath: '#/oneOf/2/required',
    keyword: 'required',
    params: { missingProperty: 'environmentVars' },
    message: "must have required property 'environmentVars'",
    schema: [ 'environmentVars' ],
    parentSchema: { required: [Array] },
    data: {}
  },
  {
    instancePath: '/formulation/0/workflows/0/inputs/0',
    schemaPath: '#/oneOf/3/required',
    keyword: 'required',
    params: { missingProperty: 'data' },
    message: "must have required property 'data'",
    schema: [ 'data' ],
    parentSchema: { required: [Array] },
    data: {}
  },
  {
    instancePath: '/formulation/0/workflows/0/inputs/0',
    schemaPath: '#/oneOf',
    keyword: 'oneOf',
    params: { passingSchemas: null },
    message: 'must match exactly one schema in oneOf',
    schema: [ [Object], [Object], [Object], [Object] ],
    parentSchema: {
      title: 'Input type',
      description: 'Type that represents various input data types and formats.',
      type: 'object',
      oneOf: [Array],
      additionalProperties: false,
      properties: [Object]
    },
    data: {}
  }
]

Here are my requirements files:
requirements-dev.txt
requirements.txt

Any ideas or anything else I should try?

Thanks for your help!

[prabhu]

Thank you for the detailed report! Will check the atom failure in the official image. Regarding the validation failures, formulation block is created automatically in 10.8.x; it is failing due to an empty list of environment variables here

cdxgen/index.js

Line 504 in 17e18f9

const sourceInput = environmentVars ? { environmentVars } : {};

You can run cdxgen with --no-include-formulation as a workaround till we fix this.

[prabhu]

Looks like the atom command is never installed in the deno image.

cdxgen/ci/Dockerfile-deno

Line 79 in 17e18f9

&& deno install --allow-read --allow-env --allow-run --allow-sys=uid,systemMemoryInfo,gid,homedir --allow-write --allow-net -n cdxgen "npm:@cyclonedx/cdxgen/cdxgen" \

[HendrikSchrieber]

Works with 10.8.4 now, thank you!

[prabhu]

Thanks for confirming!