Removing whitespace in WDL doesn't work for overindented code blocks

[stxue1]

The spec seems to want runners to support stripping whitespace from overindented python code blocks:
https://github.com/openwdl/wdl/blob/9c0b9cf4586508a9e6260cc5c5e562e21f625aac/SPEC.md?plain=1#L3590-L3624

The current issue is the amount of whitespace to remove is done at the first seen line:

toil/src/toil/wdl/wdltoil.py

Line 241 in 4cb5bb3

common_whitespace_prefix = line_whitespace_prefix

In the spec example's case, it finds 2 spaces at python <<CODE but needs to remove 4 at with open("~{infile}") as fp:

┆Issue is synchronized with this Jira Story
┆Issue Number: TOIL-1596

[adamnovak]

Does Cromwell handle this somehow?

I would think the spec is wrong here; I don't think that that shell command with just its internal indentation can work. You might want to indent inside the heredoc, but if Bash doesn't make allowances for it, and Python doesn't let you indent the entire script, I don't know why it makes sense for the WDL interpreter to provide special logic to give you this feature.

[adamnovak]

The spec doesn't actually say that the given example works, just that it produces a particular Bash script:

python <<CODE
  with open("/path/to/file") as fp:
    for line in fp:
      if not line.startswith('#'):
        print(line.strip())
CODE

That Bash script runs Python with an input where the first line starts with two spaces. Python then raises an error.

[adamnovak]

I've reported a bug on the spec: openwdl/wdl#668

[adamnovak]

The WDL spec is going to change with openwdl/wdl#669 to no longer suggest over-indenting heredocs in its examples, so I am going to close this.