best_parameters as optimal ranges instead of single values

[Mikemraz]

I read about Ax in a post and I found it to be very powful. But I do have a question to raise. I found the analytical model booth funciton is adopted as evaluation function in the Getting Started section. I wonder if we could use other customized models such as neural networks, GBM etc. with the goal to find the feature vector that can maximize the model output. If we can, is there a way that we directly call these models in the optimize API?

Thanks for your time.

[Jakepodell]

Hi @Mikemraz!

To start I recommend you take a look at the CNN tutorial on the ax website: https://ax.dev/tutorials/tune_cnn.html.

We only use the Booth function for a simple example, but one can optimize almost any function.

Ax does not support feature selection, so if that's what you're looking to do, that might not work. We can say for sure if you give us more information on what your search space looks like (e.g. parameters and metrics/outcomes that you hope to optimize). Feel free to also take a look at the Service API tutorial and Building blocks of Ax tutorials to better understand the kins of optimization setups we support and what we mean by search space and parameters.

Let us know if this helps!

[Mikemraz]

Thanks for you response @Jakepodell !

I realized I could use any ML modles to optimize my feature space. I had a test on sklearn's implementation of logstic regression and the results looks good to me.

FYI I was tring to find a feature vector (design choice) that can maximize the model output (user experience in this case). A ML model is already trained. So I mananged to use Ax to find the best design choice for me so we can make recommendations.

Here is my new question, is there a way we can get each feature's value range that can still keep target high enough? You know, sometimes a fixed design choice is not feasible, and we need more flexibility. I just wonder if Ad has this feature.

Thanks!

[Balandat]

Here is my new question, is there a way we can get each feature's value range that can still keep target high enough? You know, sometimes a fixed design choice is not feasible, and we need more flexibility. I just wonder if Ad has this feature.

Sorry @Mikemraz, I have a hard time following this question. Are you interested in exploring the tradeoffs between multiple outcomes of interest? I.e., perform Multi-objective_optimization?

[Mikemraz]

Hi @Balandat

Sorry about the confusion. Imagine a scenario where the user experience is a function of many features, and I have already build a ML model to capture this relationship. Here I am trying to use Ax to find the feature vector(hopefully it is not a vector of fixed value but a vector of intervals) that can maximize the user experience variable.

Thanks for your patience！

[Balandat]

So your ML model is f: x -> y and you just want to solve max_x f(x)? Sure, you could use Ax for that. But how costly is it to evaluate your model? If that's cheap then you might be able to just use a standard off-the-shelf optimization method (e.g CMA-ES, mulit-start L-BFGS if you have gradients, ...).

[Mikemraz]

@Balandat Thanks for your valuable feedback! You are right about what I was trying to do. I have already get it run with the basic optimize API, here are my concern:
is it possible to obtain a less restricted best_parameters soluton? Now the best_parameters is a vector of fixed value that can maximize/minimize the objective. What if we prefer the predicted feature to be a range like [lower bound, upper bound] instead of a fixed value? At the same time, it would satisfy if the objective is bigger/lower than some predetermined value.

[ldworkin]

Ah, I understand the question now, thanks for clarifying! That's really interesting, I've never considered that use case before, but I see no reason why Ax couldn't be generalized to support this. Our models are learning the relationship between parameter values and objective values, so it seems reasonable that we could determine the parameter value range resulting in an objective value between [min, max], rather than the parameter value that optimizes the objective. Curious to hear @Balandat 's thoughts though!

[lena-kashtelyan]

We will now be tracking wishlist items / feature requests in a master issue for improved visibility: #566. Of course please feel free to still open new feature requests issues; we'll take care of thinking them through and adding them to the master issue.

[Mikemraz]

Great to hear! thanks!