OptimizationFunction cannot return multiple values

[vpuri3]

The API used to allow the OptimizationFunction to return multiple objects where the first one needs to be the loss value.

https://github.com/SciML/SciMLBase.jl/blob/master/src/scimlfunctions.jl#L1813-L1815

This MWE shows that all of the outputs of the optimization function are passed down to the AD backend.

julia> using OptimizationOptimJL, Zygote                                                                                                                                                                                                                             
                                                                                                                              
julia> f(x, p) = sum(x), (;)                                                                                                                                                                                                                                 
f (generic function with 1 method)                                                                                            
                                                                                                                                                                                                                                                             
julia> optf = OptimizationFunction(f, AutoZygote())                                                                                                                                                                                                          
                                                                                                                              
julia> optprob = OptimizationProblem(optf, rand(4))                                                                                                                                                                                                          
OptimizationProblem. In-place: true                                                                                           
u0: 4-element Vector{Float64}:                                                                                                                                                                                                                               
 0.7025958336860735                                                                                                           
 0.5420026600629022                                                                                                                                                                                                                                          
 0.6272098967494587
 0.1342462910974671

julia> optsol = solve(optprob, Optim.BFGS())
ERROR: Output should be scalar; gradients are not defined for output (2.0060546815959017, NamedTuple())
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] sensitivity(y::Tuple{Float64, @NamedTuple{}})
    @ Zygote ~/.julia/packages/Zygote/Tt5Gx/src/compiler/interface.jl:114
  [3] gradient(::Function, ::Vector{Float64}, ::Vararg{Any})
    @ Zygote ~/.julia/packages/Zygote/Tt5Gx/src/compiler/interface.jl:148
  [4] gradient
    @ ~/.julia/packages/DifferentiationInterface/QK77S/ext/DifferentiationInterfaceZygoteExt/DifferentiationInterfaceZygoteExt.jl:100 [inlined]

[SebastianM-C]

#835 (comment) suggests that this is no longer supported.