[GENERAL SUPPORT]: Plotting Pareto fronts / Posterior mean model

[renaissanceM4n]

Question

I plotted my evaluated data with the interact_pareto_frontier() method and did not use the posterior mean model. Is there a simple way to set the evaluated values that do not belong to the Pareto set in contrast to the Pareto front?
I am also generally interested in what factors could be responsible for the fact that the posterior mean model extrapolates strongly in its prediction.

Please provide any relevant code snippet if applicable.

No response

Code of Conduct

• I agree to follow this Ax's Code of Conduct

[danielcohenlive]

Hi @renaissanceM4n , I'm assuming you're using https://ax.dev/tutorials/visualizations.html#3.-Tradeoff-plots. I don't know what you mean by "posterior mean model" here though. I'd be happy to help you if you could give me a minimal repro that I can run and help you improve. Also, when you refer to it extrapolating strongly, I need some visuals, perhaps raw effects vs predicted effects plots, to see what you mean by that.

[renaissanceM4n]

Hi @danielcohenlive , thank you for the response!
I use these imports to plot the pareto front based only on evaluations and not on the Gaussian Process Model (=p.m.m mentioned before) for now
from ax.plot.pareto_utils import get_observed_pareto_frontiers
from ax.plot.pareto_frontier import interact_pareto_frontier
more information: https://ax.dev/api/_modules/ax/plot/pareto_utils.html
get_observed_pareto_frontiers(ax_client.experiment)
with render(interact_paretofrontier(frontier)) i can generate the following graph.

Before that i tried to plot the pareto frontier based on the GP-model with this structure
frontier = compute_posterior_pareto_frontier(
experiment=ax_client.experiment,
data=ax_client.experiment.fetch_data(),
primary_objective=objectives[1].metric,
secondary_objective=objectives[0].metric,
num_points=20,
)
render(plot_pareto_frontier(frontier, CI_level=0.90))

and the following happened

Although the reference point for the objective on the y-scale is set at 1, the model yields values which are far away from this point. And the smallest possible value of the objective on the y-scale is 0, yet the model predicts large negative values. Thats what i meant with strong extrapolation.

Two things would be interesting for me now:
Is there a way to clearly plot the pareto set in contrast to the dominated points?
Or is it possible to restrict the extrapolation to specific bounds?
Unfortunately, there is no possibility to specify outcome_constraints for objectives. That would be an interesting last point why it cannot be implemented.

[danielcohenlive]

@renaissanceM4n

reference point for the objective on the y-scale is set at 1

What do you mean by reference point here?

Gaussian Process Model (=p.m.m mentioned before) for now

I'm still unsure which model you're referring to. A code pointer would clear it up.

there is no possibility to specify outcome_constraints for objectives

The model thinks it can do better on the at expanding the hypervolume on a different area of the curve. If you want to restrict it you can set objective thresholds. These are basically outcome_constraints for objectives. I don't know if you're using the dev API or service API, so I can't show you how to add them without a code repro.

the smallest possible value of the objective on the y-scale is 0, yet the model predicts large negative values.

The smallest possible value of the objective being 0 is a result of the implementation of your metric or evaluation function, which the model doesn't have access to. While I the nature of the model is to extrapolate, you may be able to change what it will predict by setting priors. cc @Balandat

I would caution against trying to fix this problem with outcome constraints. The model will still predict negative values, but the constraint will eliminate the candidates, creating the false impression it worked.

Is there a way to clearly plot the pareto set in contrast to the dominated points?

We don't currently have that plot. I'd suggest using both plot_pareto_frontier and plot_objective_vs_constraints .

[renaissanceM4n]

Thank you for the quick response @danielcohenlive .

What do you mean by reference point here?

The reference point is equivalent to the defined objective_thresholds for each objective. It is a bad reference value depending on the optimisation aim (min, max) for calculating the Hypervolume Improvement mentioned in this paper: https://arxiv.org/abs/2006.05078

I'm still unsure which model you're referring to. A code pointer would clear it up.

https://ax.dev/api/_modules/ax/plot/pareto_utils.html
compute_posterior_pareto_frontier() computes "the Pareto frontier between two objectives. For experiments
with batch trials, a trial index or data object must be provided. This is done by fitting a GP and finding the pareto front according to the GP posterior mean." (description of the method)
Its basically the surrogate model used to determine "good" points.

These are basically outcome_constraints for objectives. I don't know if you're using the dev API or service API, so I can't show you how to add them without a code repro.

I am using the Service-API and also defined objective_thresholds. As already mentioned, the reference point is equivalent to the objective_thresholds. Sorry for the confusion. Despite the set objective_threshold of 1, points beyond this are added to the Pareto set.

While I the nature of the model is to extrapolate, you may be able to change what it will predict by setting priors

If this is a possibilty to face this issue i would definiately try it.

I would caution against trying to fix this problem with outcome constraints. The model will still predict negative values, but the constraint will eliminate the candidates, creating the false impression it worked.

I was wondering if there is a useful implementation to specify a lower and upper bound. But in principle, the set objective_threshold should influence the predictions of the model in such a way that the range beyond this is ignored?

We don't currently have that plot. I'd suggest using both plot_pareto_frontier and plot_objective_vs_constraints .

Thank you very much for this information!

[danielcohenlive]

@renaissanceM4n About the negative metric value being impossible: it's strange that the model predicts a negative value when it's not possible, but I don't know if it's actually a problem you have to fix. It's just a predicted value, so it's really only important that it's directionally accurate. It won't affect your observed values later.

Despite the set objective_threshold of 1, points beyond this are added to the Pareto set.

It sounds like something isn't working right here. If you send me a minimal repro I can run, I can try to help you fix it.

[renaissanceM4n]

Hi @danielcohenlive
after a long time a response from my side, so far I haven't got round to it :) sorry but unfortunately I can't provide the objective function. However, the problem with the objective_thresholds/refPoints should be easy to generate if you simply carry out a multi-objective optimisation and simply take a few Sobol points for initialisation. The predictions of compute_posterior_model() are then not filtered in relation to the reference points. The problem with the negative values can probably only be solved with a higher resolution in the data points. If possible, I would like to add the restriction by the thresholds/reference points for the predictions of the Pareto front using the trained model. I have looked into the different plot functions again (highlighted).

Over several steps pareto_frontier_evaluator() is called for the observed values.
(get_observed_pareto_frontiers → observed_pareto_frontier → pareto_frontier → get_pareto_frontier_and_configs → pareto_frontier_evaluator).
This function filters the pareto points depending on the thresholds/reference points set.

With compute_posterior_pareto_frontier() there is no filtering according to the objective_thresholds.
If there is another way to filter the predictions of the Gaussian Process according to the thresholds, please let me know.
If not, this issue can be closed.