refineTagsAndSort settings for Cell + UMI with partial whitelist

[racng]

How can we handle UMI correction if we have whitelist for only one portion of the barcode?

For example, for the tag pattern: (CELL:N{6})atgc(UMI:N{5}N{6}N{5}) I only have the white list sequence for the middle N{6}. I want guided barcode correction for the cell barcode and middle six nucleotides of UMI, and then the other nucleotides in the UMI to have unguided UMI correction.

Lets say I expect the middle part of UMIs to be GGGCCC or GGGTTT. Should I make their entries in the umi_whitelist.txt file to be NNNNNGGGCCCNNNNN and NNNNNGGGTTTNNNNN?

refineTagsAndSort:
    whitelists:
      CELL: cell_whitelist.txt
      UMI: umi_whitelist.txt

Alternatively, should I break the UMI in the tag pattern like the following(CELL:N{6})atgc(UMI1:N{5})(UMI2:N{6})(UMI3:N{5}) ? And then have a whitelist file made of GGGCCC and GGGTTT?

refineTagsAndSort:
    whitelists:
      CELL: cell_whitelist.txt
      UMI2: umi_whitelist.txt

I was worried that by not providing a whitelist for UMI1 and UMI3, that they don't get corrected.

I was also looking for presets to process both cell barcodes and UMIs. I found your preset for BD and 10x. They look similar for the first half:

  refineTagsAndSort:
    whitelists:
      CELL: builtin:737K-august-2016
    runCorrection: true
    parameters:
      correctionPower: 0.001
      backgroundSubstitutionRate: 0.001
      backgroundIndelRate: 1.0e-05
      minQuality: 12
      maxSubstitutions: 2
      maxIndels: 2
      maxTotalErrors: 3
      postFilter:
        type: filter_and
        operands:
          - type: filter_groups
            groupingKeys:
              - allTags:Cell
              - allTags:Molecule
            predicates:
              - metrics:
                  - type: "group_metric_sum_weight"
                    reportHist:
                      log: true
                      binNumber: 0
                      minBinWidth: 0.2
                operator:
                  type: group_operator_lowest_threshold
                  operators:
                    - type: group_operator_advanced_thresholding
                      algo:
                        type: otsu
                      histTransformers:
                        # Artificially adds 15% of singletons to the histogram before auto-thresholding algorithm
                        - type: inflate
                          fraction: 0.15
                      logX: true
                      minimalSample: 20
                      fallbackThreshold: 1.0
                    - type: group_operator_cumtop
                      share: 0.85
                      round: Down

But then BD continues with

          - type: filter_groups
            groupingKeys:
              - allTags:Cell
            predicates:
              - metrics:
                  - type: group_metric_nunique
                    reportHist:
                      log: true
                      binNumber: 0
                      minBinWidth: 0.2
                    keys:
                      - allTags:Molecule
                operator:
                  type: group_operator_range
                  lower: 3.0

and 10x continues with

          - type: filter_groups
            groupingKeys:
              - allTags:Cell
            predicates:
              - metrics:
                  - type: "group_metric_nunique"
                    reportHist:
                      log: true
                      binNumber: 0
                      minBinWidth: 0.2
                    keys:
                      - allTags:Molecule
                operator:
                  type: group_operator_advanced_thresholding
                  algo:
                    type: otsu
                  histTransformers:
                    # Artificially adds 15% of singletons to the histogram before auto-thresholding algorithm
                    - type: inflate
                      fraction: 0.15
                  logX: true
                  minimalSample: 20
                  fallbackThreshold: 1.0

What are the implications for these different postFilter parameters?

[Alex-Davydov]

Hi,
Regarding the UMI whitelist, the right way is the second variant, which involves dividing the UMI into three groups (e.g., UMI1, UMI2, and UMI3) and providing a whitelist for the middle UMI group (UMI2). When supplying a whitelist locally, it is necessary to include the path to the file that contains the barcodes.

refineTagsAndSort:
    whitelists:
      CELL: file:path_to_whitelist/cell_whitelist.txt
      UMI2: file:path_to_whitelist/umi_whitelist.txt

It is important to note that the presence or absence of a whitelist does not affect the correction step.

With regards to the filters, they are used to remove errorneous cells based on the UMI count. For BD data, a strict threshold of 3 UMI per cell is applied, whereas for 10x data, an automatic thresholding approach is used, which employs a modified Otsu algorithm. However, the automatic thresholding approach does not work properly for BD data, which is why the strict threshold is used instead.

[racng]

Thanks for the clarification.
Also what is the purpose of the first part of postFilter setting?

postFilter:
        type: filter_and
        operands:
          - type: filter_groups
            groupingKeys:
              - allTags:Cell
              - allTags:Molecule
            predicates:
              - metrics:
                  - type: "group_metric_sum_weight"
                    reportHist:
                      log: true
                      binNumber: 0
                      minBinWidth: 0.2
                operator:
                  type: group_operator_lowest_threshold
                  operators:
                    - type: group_operator_advanced_thresholding
                      algo:
                        type: otsu
                      histTransformers:
                        # Artificially adds 15% of singletons to the histogram before auto-thresholding algorithm
                        - type: inflate
                          fraction: 0.15
                      logX: true
                      minimalSample: 20
                      fallbackThreshold: 1.0
                    - type: group_operator_cumtop
                      share: 0.85
                      round: Down

What happens if we don't put any settings for postFilter?

[Alex-Davydov]

Hi,
This filter automatically computes the number of reads per UMI threshold.