Extracting table rows with cells that have partially missing borders

[pdille]

In this example pdf, the last row has the final horizontal border missing for some of the cells (first 4 and last 6 columns)
20230413-test.pdf

With default settings, the last row pdfplumber extracts looks like:

[None, None, None, None, '23:45', '63', '0.0051007', '5.1006799', '23:45', 'N/A', '0.0530657', None, None, None, None, None, None]

If I instead tweak the table settings and use an explicit horizontal strategy of page.curves + page.edges with a larger join_x_tolerance

table_settings = {
    "horizontal_strategy": "explicit",
    "explicit_horizontal_lines": page.curves + page.edges,
    "join_x_tolerance" : 260,
}
table = page.extract_table(table_settings=table_settings)

Then I get the following output for the last row of the pdf:

['23:45', '56', '0.0266173', '26.617339', '23:45', '63', '0.0051007', '5.1006799', '23:45', 'N/A', '0.0530657', '53.0657', '23:45', '48', '0.010573', '10.572958', '']

which is what I expect.

It seems I can also use table settings with the same horizontal strategy as above, but instead tweaking the intersection_x_tolerance to a fairly large value.

table_settings = {
    "horizontal_strategy": "explicit",
    "explicit_horizontal_lines": page.curves + page.edges,
    "intersection_x_tolerance" : 700,
}

Is there a better way to ensure that last row gets extracted properly? What's the suggested path of minimal false positive/incorrect extraction based on the strategies I tried above if I want it used in an as general way as possible? (I know, easier said than done, heh)

Thanks!!

[jsvine]

With a table like this one, where you don't have to worry about interfering with other tables on the page, you should be fine with:

table_settings = {
  "explicit_horizontal_lines": [ edge["top"] for edge in page.edges ]
}

(In v0.9.0 curve-edges are now automatically incorporated into page.edges, although this is moot for your particular PDF, which appears to encode the table lines as rects rather than curves.)

Passing a list of tops tells pdfplumber to make those explicit lines the full width of the page.

[pdille]

Ah, useful to know this, thanks! I will say I did simplify the example here, since the last page often has an additional short table that I am not interested in extracting.

Because I am using "extract_table" (singular) I think your warning of your edge tops example interfering with other tables on a page isn't an issue then? (since I'm only extracting the biggest table on the page)

[jsvine]

Because I am using "extract_table" (singular) I think your warning of your edge tops example interfering with other tables on a page isn't an issue then? (since I'm only extracting the biggest table on the page)

The main sort of interference I'd be worried about is if there were multiple tables horizontally side-by-side, in which case the extension of the horizontal lines to the full page width would cause problems.

The layout you describe, though, seems fine! (Famous last words!)

[pdille]

Ah certainly no multiple horizontally side-by-side tables in the pdfs I'm working through. The tables, if more than 1 on a page, would be underneath each other vertically.

Thanks again for taking the time to answer all these specific pdf questions. Your willingness to help beyond maintaining the library is deeply appreciated!