Retry OOM killed jobs

[cmelone]

Automatically retries jobs if they are OOM killed after gantry underallocates memory.

---

• Implements a pipeline webhook handler. Checks if any jobs failed due to being OOM killed and inserts them into the db.
• Creates a new pipeline following spackbot's example and gitlab's api

As discussed in #74, we would prefer to restart jobs directly by supplying new variables, but this feature is not supported by gitlab. When a new pipeline is created for a ref, successful builds in the previous pipeline will be pruned in the generate/concretization step, minimizing wasted cycles.

During the generate step, gantry will receive a request for resource allocations for a job that was recently OOM killed. The program will look for an exact spec match in the database and return these modified variables:

• KUBERNETES_MEMORY_LIMIT * 0.2 -- bump the past request by 20%
• GANTRY_RETRY_COUNT +=1 -- maintain a count of how many times this spec has been retried
• cpu request/limit and memory limit will remain unmodified
• maybe: GANTRY_RETRY_ID -- gitlab id for the original job to link between retries...not sure if necessary

To ensure we don't fall into an infinite loop of increasing memory limits, gantry will not bump the limit if the retry count will exceed three. Additionally, it will not restart a pipeline if all OOM killed jobs also exceed the retry limit. This means we are allowing certain jobs to fail. If we investigate and it seems like an increase is warranted, how do we ensure that this gets communicated to gantry?

New optional columns in the jobs table:

• oomed -- failed job OOM killed
• retry_count -- number of times the job has been retried

TODO:

• fix oom detection in k8s cluster. see kitware-llnl channel
• tests
• figure out how you'll deal with missing data issue -- use the last timestamp as a datapoint? (what happens during missed webhooks?)
• get api permissions for restarting pipelines in spack-infra terraform config
• add new variables to annotations in spack-infra k8s config

Questions:

• Do we need to weigh the most current build higher after has been retried and bumped up?
  • No, we will allow the genetic algorithm to learn. If subsequent jobs are OOM killed, they will be given more memory and retried, which will eventually lead to an optimal memory limit.

[cmelone]

@alecbcs the pipeline webhook aspect of this PR is ready for review. This first step essentially detects OOM killed jobs and inserts them into the db.

Like I mentioned in the kitware channel, OOM detection is broken atm, so I will update the prometheus.job.is_oom method once we have that cleared up