Add net/http/pprof profiling endpoint to http port

[timvaillancourt]

This PR adds the ability to gather pprof profiles from Nebula over HTTP in order to understand/improve performance and understand the high CPU usage we see in Slack's production Vitess platform

This change requires making the http server opened by stats.go to become more generic than "just stats", so some logic has been shuffled into http.go and a new, generic config variable http.listen has been added to define this port

For backwards compatibility, http.listen will fallback to using the value for stats.listen if http.listen is undef. I suggest the stats.listen var (and the fallback) is removed in the future because the http port will not be just stats after this PR

[wadey]

Thanks! We will consider this, but I'm not sure if you are aware there is already a method to get pprof traces using the SSH debug interface.

• https://nebula.defined.net/docs/config/sshd/

Once you connect you can run help to see the methods, but the ones you might care about are start-cpu-profile, stop-cpu-profile.

Defined here in the code: https://github.com/slackhq/nebula/blob/v1.7.2/ssh.go#L228

[timvaillancourt]

Thanks! We will consider this, but I'm not sure if you are aware there is already a method to get pprof traces using the SSH debug interface.

• https://nebula.defined.net/docs/config/sshd/

Once you connect you can run help to see the methods, but the ones you might care about are start-cpu-profile, stop-cpu-profile.

Defined here in the code: https://github.com/slackhq/nebula/blob/v1.7.2/ssh.go#L228

Thanks @wadey! I didn't notice that when I started this branch, but saw mentions of it in issues later 👍

It seems we don't enable the sshd server in Prod (although we could), so this isn't currently an option. That said, I'd much prefer to use the standard net/http/pprof approach because it would not require adding sshd to all our hosts, it is easier to plug-in to our automation that already supports fetching profiles via the standard http approach, and /debug/pprof provides access to more information than CPU profiles, such as heap and mutex usage, tracing, etc