Cannot run kvdb prometheus exporter as sidecar for paritydb due to filesystem lock

[PierreBesson]

I am trying to run kvdb introspector in prometheus mode alongside a node using paritydb but running it crashes my node with the following Lock error:

2022-09-09 13:06:15.837  INFO main sc_cli::runner: 👤 Role: AUTHORITY    
2022-09-09 13:06:15.837  INFO main sc_cli::runner: 💾 Database: ParityDb at /data/chains/versi_v1_9/paritydb/full    
2022-09-09 13:06:15.837  INFO main sc_cli::runner: ⛓  Native runtime: rococo-9270 (parity-rococo-v2.0-0.tx1.au0)    
2022-09-09 13:06:19.317 DEBUG main parachain::chain-selection: Using dispute aware relay-chain selection algorithm
versi-validator-profiled-node-0 versi 2022-09-09 13:00:04.372  INFO main sub-libp2p: 🏷  Local node identity is: 12D3KooWSGB5yT5eLdCD4jdnTXpQP34dtk8cHfEf3zGgoLu5Q8F2    
versi-validator-profiled-node-0 versi Error: 
versi-validator-profiled-node-0 versi    0: Locked(Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" })
versi-validator-profiled-node-0 versi 
versi-validator-profiled-node-0 versi   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
versi-validator-profiled-node-0 versi                                 ⋮ 5 frames hidden ⋮                               
versi-validator-profiled-node-0 versi    6: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual::h34a2788a33e8e67c
versi-validator-profiled-node-0 versi       at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/result.rs:2105
versi-validator-profiled-node-0 versi    7: polkadot::main::h97ef844da7c8dc77
versi-validator-profiled-node-0 versi       at /builds/parity/mirrors/polkadot/src/main.rs:25
versi-validator-profiled-node-0 versi    8: core::ops::function::FnOnce::call_once::h3c266f22b41c124d
versi-validator-profiled-node-0 versi       at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/ops/function.rs:248
versi-validator-profiled-node-0 versi                                 ⋮ 14 frames hidden ⋮                              
versi-validator-profiled-node-0 versi 
versi-validator-profiled-node-0 versi Run with COLORBT_SHOW_HIDDEN=1 environment variable to disable frame filtering.
versi-validator-profiled-node-0 versi Run with RUST_BACKTRACE=full to include source snippets.
- versi-validator-profiled-node-0 › versi

• I am running the exporter with: polkadot-introspector kvdb --db /data/chains/versi_v1_9/paritydb/full --db-type paritydb prometheus --port 9620
• I am starting the introspector and the node at the same time in the same pod so it crashes the node as the db files are locked by the introspector
• If I start the introspector with a delay (eg. sleep 60; polkadot-introspector kvdb --db /data/chains/versi_v1_9/paritydb/full --db-type paritydb prometheus) then the node runs fine but the introspector crashes on startup:

Error: 
   0: Database file is in use. (Resource temporarily unavailable (os error 11))
   1: Resource temporarily unavailable (os error 11)

Location:
   src/kvdb/paritydb.rs:38

[sandreim]

Thanks for logging this. This is due to the fact that paritydb doesn't truly support opening the db in readonly mode. We'll have to push a fix in paritydb to solve it.

[PierreBesson]

@sandreim Do you have any idea if paritydb will ever support the kvdb exporter ?

[sandreim]

It can support what we need, but it seems not really easy to implement. There is no urgency here as we can use rocksdb for monitoring successfuly