Skip to contents
library(diseasy)
#> Loading required package: diseasystore
#> 
#> Attaching package: 'diseasy'
#> The following object is masked from 'package:diseasystore':
#> 
#>     diseasyoption
im <- DiseasyImmunity$new()

Motivation

From our testing, we have learned that our method used to express the waning immunity in a compartmental model1 is a hard optimisation problem. Depending on the choice of optimisation algorithm, the quality of the approximation will vary wildly as well as the time it takes for the algorithm to converge.

To mitigate this issue, we investigate the effect of several optimisation algorithms to identify the best performing one, evaluating both run time and approximation accuracy of the waning immunity target.

?DiseasyImmunity can internally employ the optimisation algorithms of stats, nloptr and optimx::optimr(), which means that we have around 30 different algorithms to test.

Note that the nature of the optimisation problem also changes dependent on the method used to approximate (i.e. “free_delta”, “free_gamma”, “all_free” - see ?DiseasyImmunity documentation), which means that the algorithm that performs well for one method will not necessarily perform well on the other methods.

Setup

Since we are searching for a “general” best choice of optimisation algorithm, we will define a setup that tests a wide range of waning immunity targets.

We define a “family” of functions to test the optimisation algorithms against. These start at 1 and go to 0 within a time scale, τ\tau.

These targets are:

Function Functional form
Exponential exp⁡(−t/τ)\exp\left(-t / \tau\right)
Sigmoidal exp⁡(−(t−τ)/6)/(1+exp(−(t−τ)/6))\exp\left(-(t - \tau) / 6)\; /\; (1 + \exp\!\!(-(t - \tau) / 6)\right)
Sum of exponentials exp⁡(−0.5t/τ)+exp⁡(−2t/τ)2\frac{\exp(-0.5 t / \tau) + \exp(-2 t / \tau)}{2}

We then construct more target functions from this family of functions:

  • The unaltered functions (f(t)f(t))
  • The functions but with non-zero asymptote (g(t)=0.8⋅f(t)+0.2g(t) = 0.8 \cdot f(t) + 0.2)
  • The functions but with longer time scales (h(t)=f(t/2)h(t) = f(t / 2))

The optimisation

As stated above, the optimisation algorithms vary wildly in the time it takes to complete the optimisation. To conduct this test in a reasonable time frame (and to determine algorithms that are reasonably efficient), we use a testing schema consisting of a number of “rounds” where we incrementally increase the number of compartments in the problem (and thereby the number of degrees of freedom to optimise).

In addition, we allocate a time limit to each algorithm in each round. If the execution time exceeds this time limit, the algorithm is “eliminated” and no more computation is done for that algorithm. Furthermore, if the value (error) of the objective function for the optimiser is equal or greater than 1e3 it is also “eliminated”. Note that this is done on per-method basis as we know the optimisation algorithms fare differently for the different methods for approximation.

Once the round is complete, we update the list of eliminated algorithms and then we run the optimisation of M+1M + 1 with the reduced set of algorithms.

The entire optimisation process is run for three configurations of monotonous and individual_level: - monotonous = 0 and individual_level = 0 - monotonous = 1 and individual_level = 0 - monotonous = 1 and individual_level = 1 see vignette("diseasy-immunity") for monotonous and individual_level definitions. The results of the optimisation round are stored in the ?diseasy_immunity_optimiser_results data-set.

results <- diseasy_immunity_optimiser_results |>
  dplyr::mutate(
    "monotonous" = dplyr::if_else(.data$monotonous == 1, "yes", "no"),
    "individual_level" = dplyr::if_else(.data$individual_level == 1, "yes", "no")
  )

results
#> # A tibble: 73,093 × 14
#>    target  variation  method strategy monotonous individual_level     M    value
#>    <chr>   <chr>      <chr>  <chr>    <chr>      <chr>            <int>    <dbl>
#>  1 exp_sum Base       all_f… combina… no         no                   2 4.20e- 1
#>  2 exp_sum Base       all_f… naive    no         no                   2 4.20e- 1
#>  3 exp_sum Base       all_f… recursi… no         no                   2 4.20e- 1
#>  4 exp_sum Base       free_… naive    no         no                   2 4.50e+15
#>  5 exp_sum Base       free_… recursi… no         no                   2 4.50e+15
#>  6 exp_sum Base       free_… naive    no         no                   2 4.20e- 1
#>  7 exp_sum Base       free_… recursi… no         no                   2 4.20e- 1
#>  8 exp_sum Non-zero … all_f… combina… no         no                   2 3.36e- 1
#>  9 exp_sum Non-zero … all_f… naive    no         no                   2 3.36e- 1
#> 10 exp_sum Non-zero … all_f… recursi… no         no                   2 3.36e- 1
#> # ℹ 73,083 more rows
#> # ℹ 6 more variables: execution_time <dbl>, optim_method <chr>,
#> #   target_label <chr>, n_dof <dbl>, walltime <dbl>, config <list>

Global results

To get an overview of the best performing optimisers, we present an aggregated view of the results in the table below. Here we present the top 5 “best” optimiser/strategy combination for each method and monotonous/individual_level setting. To define what it means to be the “best” we simply take the total integral difference between the approximation and the target function across all target functions and problem sizes (MM).

In addition, we only consider problems up to size M=5M = 5 for the general results.

Table 1: Global results (excluding heaviside and linear targets)
monotonous individual_level
free_delta
free_gamma
all_free
strategy optimiser value strategy optimiser value strategy optimiser value
no no recursive auglag_lbfgs 14.46 recursive subplex 11.61 recursive subplex 10.05
no no recursive auglag_slsqp 14.46 naive subplex 11.73 combination subplex 10.26
no no recursive bfgs 14.46 recursive hjkb 12.00 recursive neldermead 10.28
no no recursive bobyqa 14.46 recursive hjn 12.05 combination pracmanm 10.35
no yes naive auglag_lbfgs 15.80 recursive pracmanm 18.43 recursive neldermead 14.17
no yes recursive auglag_lbfgs 15.80 recursive subplex 18.43 recursive nvm 14.17
no yes naive auglag_slsqp 15.80 recursive neldermead 18.81 recursive rvmmin 14.17
no yes recursive auglag_slsqp 15.80 naive neldermead 18.86 recursive slsqp 14.17
yes no recursive auglag_lbfgs 14.46 recursive subplex 11.98 combination subplex 10.33
yes no recursive auglag_slsqp 14.46 recursive hjkb 12.35 naive auglag_slsqp 10.42
yes no recursive bfgs 14.46 recursive hjn 12.38 naive lbfgs 10.42
yes no recursive bobyqa 14.46 naive subplex 12.44 naive ncg 10.42
yes yes naive auglag_lbfgs 15.80 recursive subplex 18.53 recursive neldermead 14.17
yes yes recursive auglag_lbfgs 15.80 recursive pracmanm 18.54 recursive nvm 14.17
yes yes naive auglag_slsqp 15.80 recursive sbplx 18.90 recursive rvmmin 14.17
yes yes recursive auglag_slsqp 15.80 recursive neldermead 18.91 recursive ucminf 14.17
auglag_lbfgs: auglag with LBFGS as local solver.
auglag_lbfgs: auglag with SLSQP as local solver.

To better choose the default optimisers for DiseasyImmunity, we will also consider how fast the different optimisation methods are. For that purpose, we have also measured the time to run the optimisation for each problem.

# Define the defaults for DiseasyImmunity
chosen_defaults <- im$optimiser_defaults |>
  dplyr::mutate(
    "monotonous" = dplyr::if_else(.data$monotonous == 1, "yes", "no"),
    "individual_level" = dplyr::if_else(.data$individual_level == 1, "yes", "no")
  )

Total error vs execution time for the free_delta method.

Total error vs execution time for the free_gamma method.

Total error vs execution time for the all_free method.

In total, we find that there is no clear choice for best, general optimiser/strategy combination.

For the defaults, we have chosen the optimiser/strategy combination dependent on the method of parametrisation and the values of monotonous and individual_level.

The following optimiser/strategy combinations act as the currently selected defaults for the ?DiseasyImmunity class. No default is assigned yet for monotonous = yes, individual_level = no; the new results are included in the comparisons above so that this choice can be made from the benchmark.

method strategy monotonous individual_level optim_method priority config
all_free combination no no subplex 1 subplex
all_free naive no no ncg 2 ncg
all_free recursive no no bfgs 3 BFGS
all_free combination yes no subplex 4 subplex
all_free naive yes no ucminf 5 ucminf
all_free recursive yes no bfgs 6 BFGS
free_gamma recursive no no neldermead 7 neldermead
free_gamma naive no no uobyqa 8 uobyqa
free_gamma recursive yes no neldermead 9 neldermead
free_gamma naive yes no uobyqa 10 uobyqa
all_free combination yes yes nlminb 11 nlminb
all_free naive no yes slsqp 12 slsqp
all_free naive yes yes slsqp 13 slsqp
all_free recursive no yes neldermead 14 neldermead
all_free recursive yes yes nlminb 15 nlminb
free_delta recursive yes no auglag_lbfgs 16 auglag, LBFGS
free_delta recursive no no auglag_lbfgs 17 auglag, LBFGS
free_delta naive yes no mma 18 mma
free_delta naive no no mma 19 mma
free_delta naive yes yes mma 20 mma
free_delta naive no yes mma 21 mma
free_delta recursive no yes auglag_lbfgs 22 auglag, LBFGS
free_delta recursive yes yes auglag_lbfgs 23 auglag, LBFGS
free_gamma recursive yes yes subplex 24 subplex
free_gamma recursive no yes pracmanm 25 pracmanm
free_gamma naive no yes uobyqa 26 uobyqa
free_gamma naive yes yes tnewton 27 tnewton
all_free combination no yes anms 28 anms

However, as we see in the Per target results section below, the choice of optimiser/strategy can be improved when accounting for the specific target function.

Per target results

Here we dive deeper into the performance of the optimisers on a per target basis. As before, we present best performing optimisers for the given targets in an aggregated view.

We present the top 3 best optimiser/strategy combination for each method and penalty setting.

Table 2: Results for Exponential target
monotonous individual_level
free_delta
free_gamma
all_free
strategy optimiser value strategy optimiser value strategy optimiser value
no no naive auglag_lbfgs 0.00 naive auglag_slsqp 0.00 naive auglag_lbfgs 0.00
no no naive auglag_slsqp 0.00 naive auglag_lbfgs 0.00 naive auglag_slsqp 0.00
no no naive bfgs 0.00 naive bfgs 0.00 naive bfgs 0.00
no yes naive varmetric 0.37 naive bobyqa 1.56 naive slsqp 0.37
no yes naive lbfgs 0.37 recursive bobyqa 1.57 naive ucminf 0.37
no yes naive auglag_lbfgs 0.37 recursive hjkb 1.58 naive rvmmin 0.37
yes no naive auglag_lbfgs 0.00 naive auglag_slsqp 0.00 naive auglag_lbfgs 0.00
yes no naive auglag_slsqp 0.00 naive auglag_lbfgs 0.00 naive lbfgs 0.00
yes no naive bfgs 0.00 naive lbfgs 0.00 naive auglag_slsqp 0.00
yes yes naive varmetric 0.37 recursive hjkb 1.51 naive ucminf 0.37
yes yes naive lbfgs 0.37 naive bobyqa 1.56 naive bfgs 0.37
yes yes naive nlminb 0.37 naive hjkb 1.56 combination ucminf 0.37
auglag_lbfgs: auglag with LBFGS as local solver.
auglag_lbfgs: auglag with SLSQP as local solver.

Table 3: Results for Sum of exponentials target
monotonous individual_level
free_delta
free_gamma
all_free
strategy optimiser value strategy optimiser value strategy optimiser value
no no recursive nlminb 1.63 recursive subplex 2.55 recursive subplex 0.99
no no recursive ucminf 1.63 recursive pracmanm 2.55 recursive neldermead 1.15
no no recursive lbfgs 1.63 naive subplex 2.66 combination subplex 1.20
no yes naive ucminf 2.26 recursive subplex 5.17 recursive nlminb 2.16
no yes naive lbfgs 2.26 recursive pracmanm 5.17 recursive auglag_lbfgs 2.16
no yes naive bfgs 2.26 naive hjkb 5.43 recursive auglag_slsqp 2.16
yes no recursive nlminb 1.63 recursive subplex 2.91 recursive subplex 0.99
yes no recursive ucminf 1.63 recursive pracmanm 2.91 recursive neldermead 1.15
yes no recursive lbfgsb3c 1.63 recursive sbplx 2.97 naive neldermead 1.20
yes yes naive ucminf 2.26 recursive pracmanm 5.27 recursive nlminb 2.16
yes yes naive lbfgs 2.26 recursive subplex 5.27 recursive auglag_slsqp 2.16
yes yes naive bfgs 2.26 recursive sbplx 5.57 recursive auglag_lbfgs 2.16
auglag_lbfgs: auglag with LBFGS as local solver.
auglag_lbfgs: auglag with SLSQP as local solver.

Table 4: Results for Sigmoidal target
monotonous individual_level
free_delta
free_gamma
all_free
strategy optimiser value strategy optimiser value strategy optimiser value
no no naive nlminb 12.83 naive auglag_lbfgs 9.06 naive ucminf 9.06
no no naive ucminf 12.83 naive lbfgs 9.06 naive nlminb 9.06
no no naive lbfgs 12.83 naive nlminb 9.06 naive nlm 9.06
no yes naive nlminb 13.16 naive nlminb 11.62 naive auglag_slsqp 11.62
no yes naive ucminf 13.16 naive lbfgs 11.62 naive nvm 11.62
no yes naive lbfgs 13.16 naive ucminf 11.62 naive rvmmin 11.62
yes no naive nlminb 12.83 naive auglag_lbfgs 9.07 naive ucminf 9.07
yes no naive ucminf 12.83 naive lbfgs 9.07 naive nlminb 9.07
yes no naive lbfgs 12.83 naive nlminb 9.07 naive lbfgs 9.07
yes yes naive nlminb 13.16 naive nlminb 11.62 combination ucminf 11.62
yes yes naive ucminf 13.16 naive lbfgs 11.62 combination nlminb 11.62
yes yes naive lbfgs 13.16 naive ucminf 11.62 combination auglag_lbfgs 11.62
auglag_lbfgs: auglag with LBFGS as local solver.
auglag_lbfgs: auglag with SLSQP as local solver.