Prometheus service discovery

[benbro]

Prometheus supports dynamic service discovery.
Is there a way Prometheus could automatically discover all registered nodes from a lighthouse? The certificate groups could be returned as labels to help filter and relabel config.
Something like consul_sd_config will be great.

[nbrownus]

You could accomplish this today with a script that could ssh to the nebula process on your lighthouse and issue a list-hostmap -json and prepare a file_sd_config based on the host list returned.

[benbro]

I'll try it. thanks.

[benbro]

Using sshd will work but it seems to be the wrong tool for this task. Docs says "sshd enables nebula's built-in debugging console". It exposes functions I don't want to expose like change-remote and profiling and it requires managing a new set of ssh keys in addition to the nebula certificates.
Is it possible to expose an http server on the nebula overlay that will output metadata or at least a command line interface that won't require ssh.

[nbrownus]

I agree the ssh interface exposes too much for you goals here. We have talked about adding an allow list config option to limit what a given user can accomplish via the ssh interface, would that suffice?

[benbro]

The ssh interface is also a little more cumbersome to use because it requires additional ssh keys. Can't it use the existing nebula overlay? I don't think I'll use an ssh interface to get static metadata about the network so feel free to ignore my feature request. Thank you for taking the time to consider it.

[nbrownus]

The main trouble is helping folks avoid accidental information disclosure. Exposing a hostmap list over http with the current capabilities of nebula would at best allow any user on the computer where its enabled to access the hostmap list. It could give a would-be attacker a treasure map for lateral movement within the environment. The ssh system bulks this up by requiring authentication prior to use, an attacker has to gain root access to the system or a private key for an already authorized ssh key.

That's not to say there isn't value in the idea and we won't do it. We also have #913 to consider which brings about similar challenges and if we determine to merge it then I would imagine we would desire to add more capability to the http server. The trick will be striking the right balance between usability and security.

[benbro]

Can't I already discover all IPs when lighthouse dns is enabled?

nmap -sL --dns-server=100.64.64.9 100.64.64.0/24
dig @100.64.64.9 +short "100.64.64.10" TXT

Another option is to expose this http interface just to a specific nebula group.