Issue finding age in months from participants.tsv for v24.0.0

[lundq163]

What happened?

When submitting recent processing jobs with v24.0.0 I have been running into an error when trying to collect the age in months information from the participants.tsv that I've been using. Specifying the age in months with the --age-months flag and not using a participants.tsv can get around in the error in the meantime.

What command did you use?

singularity run --cleanenv \
 -B ${data_dir}/bids_dir/sub-${subj_id}_ses-${ses_id}:/bids_dir \
 -B ${data_dir}/processed/nibabies/sub-${subj_id}_ses-${ses_id}:/output_dir \
 -B ${data_dir}/work_dir/nibabies/sub-${subj_id}_ses-${ses_id}:/wd \
 -B ${data_dir}/processed/bibsnet/sub-${subj_id}_ses-${ses_id}:/derivatives \
 -B ${run_dir}/license.txt:/license.txt \
 /home/faird/shared/code/external/pipelines/nibabies/nibabies_24.0.0.sif \
 /bids_dir /output_dir participant \
--output-spaces MNI152NLin6Asym:res-2 \
--fs-license-file /license.txt \
--omp-nthreads 8 \
--nprocs 8 \
--cifti-output 91k \
-vv \
--project-goodvoxels \
--fd-radius 45 \
--derivatives /derivatives \
--surface-recon-method ${recon} \
-w /wd


and here is the first 5 lines of the participants.tsv I was using:
participant_id	session	age
sub-100619	ses-17mo	17
sub-100619	ses-23mo	23
sub-102606	ses-10mo	10
sub-102606	ses-19mo	19

What version of NiBabies are you using?

v24.0.0

Relevant log output

Traceback (most recent call last):
  File "/opt/conda/envs/nibabies/bin/nibabies", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/conda/envs/nibabies/lib/python3.11/site-packages/nibabies/cli/run.py", line 62, in main
    retval = build_workflow(config_file)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/nibabies/lib/python3.11/site-packages/nibabies/cli/workflow.py", line 76, in build_workflow
    retval['workflow'] = init_nibabies_wf(config.execution.unique_labels)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/nibabies/lib/python3.11/site-packages/nibabies/workflows/base.py", line 124, in init_nibabies_wf
    age = parse_bids_for_age_months(config.execution.bids_dir, subject_id, session_id)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/nibabies/lib/python3.11/site-packages/nibabies/utils/bids.py", line 261, in parse_bids_for_age_months
    age = _get_age_from_tsv(participants_tsv, index_column='participant_id', index_val=subject)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/nibabies/lib/python3.11/site-packages/nibabies/utils/bids.py", line 284, in _get_age_from_tsv
    idx = df.index[df[index_column] == index_val].item()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/nibabies/lib/python3.11/site-packages/pandas/core/base.py", line 418, in item
    raise ValueError("can only convert an array of size 1 to a Python scalar")
ValueError: can only convert an array of size 1 to a Python scalar

Add any additional information or context about the problem here.

If any more information is needed please let me know. Happy to help with testing/troubleshooting as well, thanks!

[scott-huberty]

I've also had some issues using TSV files for this. @lundq163 Can you by chance share the actual participants.tsv file?

[lundq163]

participants.txt
looks like tsv files werent supported so I had to switch it to be a txt to share it here. hopefully this still helps though @scott-huberty

important to note that this participants.tsv had worked for me when using previous versions but runs into the above error when using v24.0.0

[scott-huberty]

Sorry @lundq163 can you send the TSV file to [email protected] ?

I want to test this on my side, but I'd like to minimize the differences between our runtime environments (e.g. using txt instead of TSV)

[scott-huberty]

important to note that this participants.tsv had worked for me when using previous versions but runs into the above error when using v24.0.0

OK thanks @lundq163 , that is good to know.

I can replicate this locally. Will look into a fix.

[mgxd]

Seems like the problem is multiple results are returned when querying the participants.tsv, since we aren't dual indexing with <subject, session>. We could add a fallback to handle this case, though I think using a sessions.tsv is the usual way this information is saved.

The heuristic when searching for age is:

1. sub-*/ses-*/*scans.tsv
2. sub-*/sub-*sessions.tsv
3. participants.tsv

If you do have sessions.tsv, then it's possible the results are being skipped (and thus the next step of the heuristic is being used) after the changes in #376. This change was motivated by longitudinal studies potentially changing age units through the study's lifespan. In short, you can either be explicit and denote age within the tsv as age_months, age_years, etc., or use age and include a sidecar file to the tsv describing the column's units.

We definitely should make some noise when this happens though, I think adding a warning here would be very helpful

nibabies/nibabies/utils/bids.py

Lines 296 to 297 in 8e2415f

	if age_units is False:
	return None

[scott-huberty]

Continued conversation at #389