tolerance for new word within cell is reversed

[bellma-lilly]

close_text_redacted.pdf
In the example pdf, many of the cells combine elements that should be separate (XX, XNS,etc) because they are too close vertically.:

                          0       1                   2       3     4     5     6     7     8     9     10    11
0      Analytical\nPropertya  Method  Storage\nCondition  Months  None  None  None  None  None  None  None  None
1                       None    None                None       I     1     3     6     9    12    18   23b   24c
2                       CCId                                   X    NS    NS    NS    NS     X    NS    NS     X
3                         pH                                   X    XX    XX    XX   XNS   XNS   XNS   XNS   XNS
4         Particulate Matter                                   X    XX    XX    XX   XNS   XNS   XNS   XNS   XNS
5         Particulate Matter                                   X    XX    XX    XX   XNS   XNS   XNS   XNS   XNS
6         Particulate Matter                                   X    XX    XX    XX   XNS   XNS   XNS   XNS   XNS
7                      Color                                   X    XX    XX    XX   XNS   XNS   XNS   XNS   XNS
8                    Clarity                                   X    XX    XX    XX   XNS   XNS   XNS   XNS   XNS
9         Break-Loose Forced                                   X    XX    XX    XX   XNS   XNS   XNS   XNS   XNS
10              Glide Forced                                   X    XX    XX    XX   XNS   XNS   XNS   XNS   XNS
11         VisualInspectione                                   X    XX    XX    XX   XNS   XNS   XNS   XNS   XNS
12            Dose Accuracye                                   X    XX    XX    XX   XNS   XNS   XNS   XNS   XNS
13  Injection Process\nTimef                                   X    XX    XX    XX   XNS   XNS   XNS   XNS   XNS

Expected result (\n separation):

                          0       1                   2       3     4     5     6      7      8      9      10     11
0      Analytical\nPropertya  Method  Storage\nCondition  Months  None  None  None   None   None   None   None   None
1                       None    None                None       I     1     3     6      9     12     18    23b    24c
2                       CCId                                   X    NS    NS    NS     NS      X     NS     NS      X
3                         pH                                   X  X\nX  X\nX  X\nX  X\nNS  X\nNS  X\nNS  X\nNS  X\nNS
4         Particulate Matter                                   X  X\nX  X\nX  X\nX  X\nNS  X\nNS  X\nNS  X\nNS  X\nNS
5         Particulate Matter                                   X  X\nX  X\nX  X\nX  X\nNS  X\nNS  X\nNS  X\nNS  X\nNS
6         Particulate Matter                                   X  X\nX  X\nX  X\nX  X\nNS  X\nNS  X\nNS  X\nNS  X\nNS
7                      Color                                   X  X\nX  X\nX  X\nX  X\nNS  X\nNS  X\nNS  X\nNS  X\nNS
8                    Clarity                                   X  X\nX  X\nX  X\nX  X\nNS  X\nNS  X\nNS  X\nNS  X\nNS
9         Break-Loose Forced                                   X  X\nX  X\nX  X\nX  X\nNS  X\nNS  X\nNS  X\nNS  X\nNS
10              Glide Forced                                   X  X\nX  X\nX  X\nX  X\nNS  X\nNS  X\nNS  X\nNS  X\nNS
11         VisualInspectione                                   X  X\nX  X\nX  X\nX  X\nNS  X\nNS  X\nNS  X\nNS  X\nNS
12            Dose Accuracye                                   X  X\nX  X\nX  X\nX  X\nNS  X\nNS  X\nNS  X\nNS  X\nNS
13  Injection Process\nTimef                                   X  X\nX  X\nX  X\nX  X\nNS  X\nNS  X\nNS  X\nNS  X\nNS

I believe this is because the tolerances are reversed in utils/text.py char_begins_new_word:

        return bool(
            (next_char["x0"] > word_x1 + intraline_tol)
            or (next_char["x1"] < word_x0 - intraline_tol)
            or (next_char["top"] > word_bottom + interline_tol)
            or (next_char["bottom"] < word_top - interline_tol)
        )

In this code, as the tolerance increases, it actually makes it harder for items to be separated. I understand that the same tolerance value is used to decide if words are on the same line, but that is a comparison between the tops of the elements. In this case, it is a difference between the space of the top and bottom element. I solved the problem by flipping the sign of the tolerance (I have a PR ready for this). Alternatively, a new tolerance variable is needed for this function.

[jsvine]

Hmmm, without seeing the actual code/settings you're using, it's hard to assess exactly what's producing your output. But if I set text_y_tolerance to 1, it seems to produce the output you wanted:

In this code, as the tolerance increases, it actually makes it harder for items to be separated.

That's actually the intent! Perhaps it's not worded as clearly as it could in the documentation? Basically: the tolerance represents how much space between characters you'll tolerate before deciding there's a new word.

[bellma-lilly]

@jsvine Sorry I didn't include code; I was just using the defaults:

for page in pdf.pages:
    tables = page.extract_tables()

Your result actually shows the issue. With text_y_tolerance = 1, it fixes the vertical cell issue, but it breaks the header:
"23b" becomes "b\n23". That's because the same y_tolerance is used in different ways.
In the header, it compares the doctop for both elements because 23b has already been decided to be on the same line, but the b is a superscript slightly higher than 23. In the cell, it's comparing top vs. bottom in a different function to decide if they are on the same line or not.

So, changing y_tolerance fixes one and breaks the other no matter which way it is tuned.

[jsvine]

Ah, I see. Two options that come to mind:

• Fiddle with text_y_tolerance until you find a value that distinguishes between the things you do and don't want treated as separate. In this case, 1.5 seems to do the trick, but there's no guarantee you'll find one for other PDFs.
• Use page.crop(...) in combination with page.find_tables()[0].cells, to iterate through each cell in the table, calling this_cropped_cell.extract_text(...) with the extraction parameters you want for that particular cell.

[bellma-lilly]

@jsvine Once again, I'm impressed that there is a possible solution that doesn't require pdfplumber code changes.

In this case, though, I think a separate tolerance parameter is in order for the difference uses. That is, of course, just my opinion. pdfplumber is an awesome resource and you have clearly made good decisions on trade offs.

[jsvine]

Thanks, that's helpful! Let me ruminate on this a little bit and run some experiments. Would also be curious for @samkit-jain's thoughts.

[jsvine]

Did some investigating and testing, and came to the conclusion that 3 should be defined the same way 1 and 2 are — and, in fact, was behaving that way because of to_textmap's clustering. In the just-released v0.9.0, I've cleaned up the code and logic for char_begins_new_word, reflecting this and, hopefully, making the whole operation clearer. Thank you for the suggestion and nudge!

But, I would assume this code was intentionally written differently to clearly understand when one line of text is on a completely different line.

That was indeed the idea originally, many moons ago, but, per above, it wasn't really working that way. I think it's conceptually simpler to have just one rule/threshold for separating lines, and to keep it consistent across methods.

[bellma-lilly]

@jsvine How well does this work with subscripts?

[jsvine]

The changes shouldn't change how subscripts are handled, iiiiiiif I understand my own code correctly. Although in isolation char_begins_new_word would have handled them differently before/after the changes, the pre-clustering done by to_textmap meant never reaching that situation. Which is a long way of saying: The handling of subscripts will depend on (a) the distance of top of the subscript character's bounding box from the top of the rest of the word's bounding box, and (b) the value of y_tolerance. By default y_tolerance is fairly small (3 units), so handling subscripts will likely often require adjusting that parameter.

I ought to add an example and test of this! Do you have a particular PDF you'd be interested in contributing?

[bellma-lilly]

@jsvine That all sounds good. I don't have a specific pdf to test with at the moment aside from the one I provided here.
Because I have a locally implemented solution, I am on to some other work for a while, but when I return to this, I will try out the new version with the changes you have made.