Detect/Extract Project Website Information

[Max-at-Vlaanderen]

To further enrich the metadata, project websites can be a good source of clearly communicated info. In our current harvested data, the reference to such websites could already be found, making it possible to find the project website more accurately (as opposed to querying via google). This would require some further developments to be added to the Harvester.

Desired Behavior:

1. Detection of URLs: Implement functionality to detect URLs in the metadata that link to project websites.
2. URL Following: Automatically follow these URLs to access the linked websites.
3. Detection of projectwebsite: not all URLs will lead to the projectwebsite. Finding a way to distinguish a project website from another will be necessary.
4. Scraping Additional Information: Extract additional text from these project website
5. Store URLs