Code in this repository is created within the LIACi project by SINTEF Digital. It is published in the paper "Fusion of Multi-Modal Data from Underwater Ship Inspection with Knowledge Graphs" and is also content of the master thesis by Joseph Hirsch with the same name.
The LIACi project aims at improving decision support systems in underwater ship inspections. One big goal is to make use of the onging development in computer vision. The inference results of machine learning models are contextualized with the available data from the inspection vehicles. This enables a semantic browsability of the data and also big data analytics for example in a business intelligence context of a whole fleet of ships. Semantic similarities and clusters complete this feature set.
This repository contains a data contextualization pipeline for underwater ship inspection videos where also telemetry data is available in .ass format and a demonstrator web application that conatains different visualizations for this data.
Please refer to the readmes of the respective directories for further information about the different elements.
| Directory | Content |
|---|---|
| /data | - domain model for underwater ship inspections - data access scripts for objects from the domain model - data store that handles connections to the neo4j database - Place for thumbnails of video frames and mosaics accessible from both the demonstrator as well as the pipeline |
| /demonstrator | Web application containing different kinds of visulizations on the data of the knowledge graph |
| /pipeline | - Inference pipeline for data enrichment with computer vision models - Contextualization pipeline to find similarities and semantic clusters in the inspection video frames |
The architecture follows a four layer achritecture as suggested for knowledge graph data contextualization applications in this paper by Waszak et al. It consists of the following layers
| Layer | task(s) | technology | where in this repo |
|---|---|---|---|
| Presentation | Presenting the data to the user such that additional knowledge and value is created |  |
/demonstrator |
| Application | API for data storage and retrieval |   |
/data/access |
| Business | Business logic of the inference pipeline, machine learning models, computer vision, data contextualization |   |
/pipeline |
| Data | Data persistande, domain model |  |
/data/vismodel, /data/inspection and the neo4j database |