Some of my PDF pages are completely unreadable

[bpugnaire]

Hello,

As part of my job, I am trying to extract many informations from a large amount of PDF files and the pdfplumber library seems very promising.

Nonetheless,
I have discovered that some pages (exemple below) are completely unreadable when displaying them with .to_image().

Do you have an idea of what could be happening here ?

Thank you

[jsvine]

Hi @bpugnaire, and very interesting. The PDF->image conversion is largely handled by ImageMagick (plus a small amount of post-processing), so you could try experimenting with that library directly to see if there's an extra setting that works. Or if you can share the PDF itself here, I can poke around as well.

[bpugnaire]

Thank you for looking into it. Unfortunately, I can't provide you with the original PDF for privacy reasons.

This rendering/image-processing issue is a real issue since in order to find the "description" of a figure I am using this piece of code :

def find_figure_legend_object(page):
    search_result = page.search('XXXX') + page.search('XXXX')
    result = []
    for legend in search_result:
        bbox = legend['x0'],legend['top'],legend['x1'],legend['bottom']
        n = 0
        while len(page.within_bbox(bbox, relative=False).extract_text()) > n : #Shift right
            n = len(page.within_bbox(bbox, relative=False).extract_text())
            bbox = bbox[0],bbox[1],bbox[2]+8,bbox[3]
        n = 0
        while len(page.within_bbox(bbox, relative=False).extract_text()) > n : # Shift left (should not happen often)
            n = len(page.within_bbox(bbox, relative=False).extract_text())
            bbox = bbox[0]-8,bbox[1],bbox[2],bbox[3]
        n = 0
        while len(page.within_bbox(bbox, relative=False).extract_text()) > n : # Shift down
            n = len(page.within_bbox(bbox, relative=False).extract_text())
            bbox = bbox[0],bbox[1],bbox[2],bbox[3]+12
            
        bbox = bbox[0]+5,bbox[1]-2,bbox[2]-5,bbox[3]-8  # Eye candy
        keys = ['x0', 'top', 'x1', 'bottom']
        dict_bbox = dict(zip(keys,bbox))
        result.append(dict_bbox)
    return result

Which iteratively try to find the right bbox by doing multiple crops but when the crops are made on a "poorly" rendered image, my algorithm fails.

This is a workaround because I would have liked that page.extract_text to have a "bbox" argument that tells it to only look for text that falls in this specific bounding box (or rectangle).

Thank you again for your time and please keep me updated if you find the origin of the issue.

[jsvine]

Hmm, now I'm slightly confused: The image-rendering/visual-debugging component of this library shouldn't have any bearing on the text-extraction; they operate entirely independently. (The Page.crop(...)/etc. methods do not invoke Page.to_image(...) at any point.) It's possible that the image issue you've noticed above is just a red herring, or that there's some sort of underlying factor causing a correlation between the image issue and the text-extraction issue.

[bpugnaire]

I've investigated and I think the error is from my code, it just happened to be on the exact page that had a rendering issue which led me to believe there was a correlation.
The fact that I am sequentially searching text and expanding the bounding box first from left top right then from top to bottom makes it fails at being accurate when the first line is shorter than its successors.

[jsvine]

Ah, good to know — thanks for the update.

[bpugnaire]

Thank you for the insights :)