,
Diederick Vermetten [cre, aut]
,
Carola Doerr [aut] ,
Thomas Bäck [aut] | Title: | Data Analysis Part of 'IOHprofiler' |
|---|---|
| Description: | The data analysis module for the Iterative Optimization Heuristics Profiler ('IOHprofiler'). This module provides statistical analysis methods for the benchmark data generated by optimization heuristics, which can be visualized through a web-based interface. The benchmark data is usually generated by the experimentation module, called 'IOHexperimenter'. 'IOHanalyzer' also supports the widely used 'COCO' (Comparing Continuous Optimisers) data format for benchmarking. |
| Authors: | Hao Wang [aut] ,
Diederick Vermetten [cre, aut]
,
Carola Doerr [aut] ,
Thomas Bäck [aut] |
| Maintainer: | Diederick Vermetten <[email protected]> |
| License: | BSD_3_clause + file LICENSE |
| Version: | 0.1.8.11 |
| Built: | 2025-02-10 05:30:31 UTC |
| Source: | https://github.com/iohprofiler/iohanalyzer |
S3 extraction function for DataSetList
## S3 method for class 'DataSetList' x[i, drop = FALSE]## S3 method for class 'DataSetList' x[i, drop = FALSE]
x |
The DataSetList to use |
i |
The indices to extract |
drop |
Currently unused parameter |
The DataSetList of the DataSets at indices i of DataSetList x
dsl[c(1, 3)]dsl[c(1, 3)]
S3 generic == operator for DataSets
## S3 method for class 'DataSet' dsL == dsR## S3 method for class 'DataSet' dsL == dsR
dsL |
A 'DataSet' object |
dsR |
A 'DataSet' object |
True if the DataSets contain the same function, dimension and algorithm, and have the exact same attributes
dsl[[1]] == dsl[[2]]dsl[[1]] == dsl[[2]]
Sorts a DataSetList based on the custom specified attributes ('algId', 'DIM' or 'funcId'). Default is as ascending, can be made descending by adding a - in front of the attribute. Sorting accross multiple attributes is supported, in the order they are specified.
arrange(dsl, ...) ## S3 method for class 'DataSetList' arrange(dsl, ...)arrange(dsl, ...) ## S3 method for class 'DataSetList' arrange(dsl, ...)
dsl |
The DataSetList to sort |
... |
attribute by which 'dsl' is sorted. Multiple attributes can be specified. |
arrange(dsl, DIM, -funcId, algId)arrange(dsl, DIM, -funcId, algId)
S3 generic as.character operator for DataSet
## S3 method for class 'DataSet' as.character(x, verbose = F, ...)## S3 method for class 'DataSet' as.character(x, verbose = F, ...)
x |
A DataSet object |
verbose |
Verbose mode, currently not implemented |
... |
Arguments passed to other methods |
A short description of the DataSet
as.character(dsl[[1]])as.character(dsl[[1]])
Area Under Curve (Empirical Cumulative Dsitribution Function)
AUC(fun, from = NULL, to = NULL) ## S3 method for class 'ECDF' AUC(fun, from = NULL, to = NULL)AUC(fun, from = NULL, to = NULL) ## S3 method for class 'ECDF' AUC(fun, from = NULL, to = NULL)
fun |
A ECDF object. |
from |
double. Starting point of the area on x-axis |
to |
double. Ending point of the area on x-axis |
a object of type 'ECDF'
ecdf <- ECDF(dsl,c(12,14)) AUC(ecdf, 0, 100)ecdf <- ECDF(dsl,c(12,14)) AUC(ecdf, 0, 100)
Bootstrapping for running time samples
bootstrap_RT(x, max_eval, bootstrap.size)bootstrap_RT(x, max_eval, bootstrap.size)
x |
A numeric vector. A sample of the running time. |
max_eval |
A numeric vector, containing the maximal running time in each run. It should have the same size as x |
bootstrap.size |
integer, the size of the bootstrapped sample |
A numeric vector of the bootstrapped running time sample
ds <- dsl[[1]] x <- get_RT_sample(ds, ftarget = 16, output = 'long') max_eval <- get_maxRT(dsl, output = 'long') bootstrap_RT(x$RT, max_eval$maxRT, bootstrap.size = 30)ds <- dsl[[1]] x <- get_RT_sample(ds, ftarget = 16, output = 'long') max_eval <- get_maxRT(dsl, output = 'long') bootstrap_RT(x$RT, max_eval$maxRT, bootstrap.size = 30)
Concatenation for DataSets. Combines multiple runs from separate DataSets into a single DataSet object if all provided arguments have the same dimension, function ID and algorithm ID, and each contains only a single run. Currently does not support parameter tracking
## S3 method for class 'DataSet' c(...)## S3 method for class 'DataSet' c(...)
... |
The DataSets to concatenate |
A new DataSet
c(dsl[[1]], dsl[[1]])c(dsl[[1]], dsl[[1]])
S3 concatenation function for DataSetList
## S3 method for class 'DataSetList' c(...)## S3 method for class 'DataSetList' c(...)
... |
The DataSetLists to concatenate |
A new DataSetList
c(dsl[1], dsl[3])c(dsl[1], dsl[3])
S3 generic cat operator for DataSet
cat.DataSet(x)cat.DataSet(x)
x |
A DataSet object |
A short description of the DataSet
cat.DataSet(dsl[[1]])cat.DataSet(dsl[[1]])
Note that this function returns a new DataSetList object, since a split into new datasetlist has to be done to ensure each dataset has exactly one unique identifier. Note that only static attributes (see 'get_static_attributes') can be used to create unique identifiers.
change_id(dsl, attrs)change_id(dsl, attrs)
dsl |
The DataSetList |
attrs |
The list of attributes to combine into a unique identifier |
A new DataSetList object where the split has been done based on the provided attributes, and the unique identifier has been added.
change_id(dsl, c('instance'))change_id(dsl, c('instance'))
This uses the keyring package to store and load credentials. If the keyring package does not exists, it will default to look for a config-file in the 'repository'-folder, under your home directory. This can be changed by setting the option IOHprofiler.config_dir If you already have an account, please call 'set_DSC_credentials' with the corresponding username and password. If you don't have an account, you can register for one using 'register_DSC'
check_dsc_configured()check_dsc_configured()
check_dsc_configured()check_dsc_configured()
Throws a warning when multiple formats are found in the same folder.
check_format(path)check_format(path)
path |
The path to the folder to check |
The format of the data in the given folder. Either 'COCO', 'IOHprofiler', 'NEVERGRAD' or 'SOS'.
path <- system.file("extdata", "ONE_PLUS_LAMDA_EA", package = "IOHanalyzer") check_format(path)path <- system.file("extdata", "ONE_PLUS_LAMDA_EA", package = "IOHanalyzer") check_format(path)
Concatenates all DataSets with the same ID, algid, function id and dimension
clean_DataSetList(dsList)clean_DataSetList(dsList)
dsList |
The DataSetList object to clean |
clean_DataSetList(dsl)clean_DataSetList(dsl)
DataSet contains the following attributes * funId * DIM * algId * datafile * instance * maxEvals * finalFunEvals * comment * Additional attributes based on the original format
DataSet( info, verbose = F, maximization = NULL, format = IOHprofiler, subsampling = FALSE, full_sampling = FALSE )DataSet( info, verbose = F, maximization = NULL, format = IOHprofiler, subsampling = FALSE, full_sampling = FALSE )
info |
A List. Contains a set of in a *.info file. |
verbose |
Logical. |
maximization |
Logical. Whether the underlying optimization algorithm performs a maximization? Set to NULL to determine automatically based on format |
format |
A character. The format of data source, either 'IOHProfiler', 'COCO' or 'TWO_COL" |
subsampling |
Logical. Whether *.cdat files are subsampled? |
full_sampling |
Logical. Whether the raw (unaligned) FV matrix should be stored. Currenlt only useful when a correlation plot between function values and parameters should be made |
A S3 object 'DataSet'
path <- system.file('extdata', 'ONE_PLUS_LAMDA_EA', package = 'IOHanalyzer') info <- read_index_file(file.path(path, 'IOHprofiler_f1_i1.info')) DataSet(info[[1]])path <- system.file('extdata', 'ONE_PLUS_LAMDA_EA', package = 'IOHanalyzer') info <- read_index_file(file.path(path, 'IOHprofiler_f1_i1.info')) DataSet(info[[1]])
Attributes funId DIM algId
DataSetList( path = NULL, verbose = T, print_fun = NULL, maximization = NULL, format = IOHprofiler, subsampling = FALSE, full_aggregation = TRUE )DataSetList( path = NULL, verbose = T, print_fun = NULL, maximization = NULL, format = IOHprofiler, subsampling = FALSE, full_aggregation = TRUE )
path |
Path to the data files. Will look for all .info-files in this directory and use the corresponding datafiles to create the DataSetList |
verbose |
Logical. |
print_fun |
Function used to print output when in verbose mode |
maximization |
Logical. Whether the underlying optimization algorithm performs a maximization? |
format |
A character. The format of data source, options are:
These formats are specified in more detail in our github wiki. |
subsampling |
Logical. Whether *.cdat files are subsampled? |
full_aggregation |
If True, individual DataSets are aggregated as much as possible: all DataSets with the same algorithmname, function id and dimension are combined together. This leads to information loss related to static variables, so only use if that information is not required. |
A DataSetList object
path <- system.file("extdata", "ONE_PLUS_LAMDA_EA", package = "IOHanalyzer") DataSetList(path)path <- system.file("extdata", "ONE_PLUS_LAMDA_EA", package = "IOHanalyzer") DataSetList(path)
A DataSetList containing DataSets on 2 IOHProfiler functions from 2 algorithms in 16D
dsldsl
DataSetList
summary(dsl)summary(dsl)
A DataSetList containing DataSets on all IOHProfiler functions from 11 algorithms in 100D
dsl_largedsl_large
DataSetList
summary(dsl_large)summary(dsl_large)
Empirical Cumulative Dsitribution Function of Runtime of a single data set
ECDF(ds, ftarget, ...) ## S3 method for class 'DataSet' ECDF(ds, ftarget, ...) ## S3 method for class 'DataSetList' ECDF(ds, ftarget, ...)ECDF(ds, ftarget, ...) ## S3 method for class 'DataSet' ECDF(ds, ftarget, ...) ## S3 method for class 'DataSetList' ECDF(ds, ftarget, ...)
ds |
A DataSet or DataSetList object. |
ftarget |
A Numerical vector. Function values at which runtime values are consumed |
... |
Arguments passed to other methods |
a object of type 'ECDF'
ECDF(dsl,c(12,14)) ECDF(dsl[[1]],c(12,14))ECDF(dsl,c(12,14)) ECDF(dsl[[1]],c(12,14))
Function to get just the RT samples needed, without any formatting to improve speed
fast_RT_samples(RT_mat, target, maximization = F)fast_RT_samples(RT_mat, target, maximization = F)
RT_mat |
A matrix containing the RT-values of a dataset |
target |
Which target-value to use |
maximization |
Whether maximization is needed or not |
This function generates a dataframe which can be easily plotted using the 'plot_general_data'-function
generate_data.Aggr(dsList, aggr_on = "funcId", targets = NULL, which = "by_RT")generate_data.Aggr(dsList, aggr_on = "funcId", targets = NULL, which = "by_RT")
dsList |
The DataSetList object |
aggr_on |
Which attribute to use for aggregation. Either 'funcId' or 'DIM' |
targets |
Optional list of target values (Runtime or target value) |
which |
Whether to use a fixed-target 'by_RT' perspective or fixed-budget 'by_FV' |
generate_data.Aggr(dsl)generate_data.Aggr(dsl)
This function generates a dataframe which can be easily plotted using the 'plot_general_data'-function
generate_data.AUC( dsList, targets, scale_log = F, which = "by_RT", dt_ecdf = NULL, multiple_x = FALSE, normalize = T )generate_data.AUC( dsList, targets, scale_log = F, which = "by_RT", dt_ecdf = NULL, multiple_x = FALSE, normalize = T )
dsList |
The DataSetList object |
targets |
A list or data.table containing the targets per function / dimension. If this is a data.table, it needs columns 'target', 'DIM' and 'funcId' |
scale_log |
Whether to use logarithmic scaling or not |
which |
Whether to use a fixed-target 'by_RT' perspective or fixed-budget 'by_FV' |
dt_ecdf |
A data table of the ECDF to avoid needless recomputations. Will take preference if it is provided together with dsList and targets |
multiple_x |
Boolean, whether to get only the total AUC or get stepwise AUC values |
normalize |
Whether to normalize the resulting AUC values to [0,1] or not |
generate_data.AUC(dsl, get_ECDF_targets(dsl)) generate_data.AUC(NULL, NULL, dt_ecdf = generate_data.ECDF(dsl, get_ECDF_targets(dsl)))generate_data.AUC(dsl, get_ECDF_targets(dsl)) generate_data.AUC(NULL, NULL, dt_ecdf = generate_data.ECDF(dsl, get_ECDF_targets(dsl)))
This function generates a dataframe that can be used to generate the 'cumulative_difference_plot'.
generate_data.CDP( dsList, runtime_or_target_value, isFixedBudget, alpha = 0.05, EPSILON = 1e-80, nOfBootstrapSamples = 1000 )generate_data.CDP( dsList, runtime_or_target_value, isFixedBudget, alpha = 0.05, EPSILON = 1e-80, nOfBootstrapSamples = 1000 )
dsList |
The DataSetList object. Note that the 'cumulative_difference_plot' can only compare two algorithms in a single problem of dimension one. |
runtime_or_target_value |
The target runtime or the target value |
isFixedBudget |
Should be TRUE when target runtime is used. False otherwise. |
alpha |
1 minus the confidence level of the confidence band. |
EPSILON |
If abs(x-y) < EPSILON, then we assume that x = y. |
nOfBootstrapSamples |
The number of bootstrap samples used in the estimation. |
A dataframe with the data to generate the cumulative difference plot.
dsl_sub <- subset(dsl, funcId == 1) generate_data.CDP(dsl_sub, 15, TRUE, nOfBootstrapSamples = 10)dsl_sub <- subset(dsl, funcId == 1) generate_data.CDP(dsl_sub, 15, TRUE, nOfBootstrapSamples = 10)
This function generates a dataframe which can be easily plotted using the 'plot_eaf_data'-function
generate_data.EAF( dsList, n_sets = 11, subsampling = 100, scale_xlog = F, xmin = "", xmax = "" )generate_data.EAF( dsList, n_sets = 11, subsampling = 100, scale_xlog = F, xmin = "", xmax = "" )
dsList |
The DataSetList object |
n_sets |
The number of level sets to calculate |
subsampling |
Level of subsampling to use for runtime-values (number of runtimes to consider). Setting to 0 will make the calculations more precise at the cost of potentially much longer exectution times |
scale_xlog |
Only has effect when 'subsampling' is True. The scaling of the subsampled runtimes When true, these are equally spaced in log-space, when false they are linearly spaced. |
xmin |
Minimum runtime value |
xmax |
Maximum runtime value |
generate_data.EAF(subset(dsl, funcId == 1))generate_data.EAF(subset(dsl, funcId == 1))
This is an approximation of “, since the number of required polygons can quickly become problematic for plotly. This function uses discretized contour matrices instead, which trades off accuracy for scalability.
generate_data.EAF_diff_Approximate( dsList, xmin, xmax, ymin, ymax, x.log = T, y.log = T )generate_data.EAF_diff_Approximate( dsList, xmin, xmax, ymin, ymax, x.log = T, y.log = T )
dsList |
The DataSetList object, containing at least 2 IDs |
xmin |
Minimum runtime to consider |
xmax |
Maximum runtime to consider |
ymin |
Minimum f(x) to consider |
ymax |
Maximum f(x) to consider |
x.log |
Whether to scale the y-space logarithmically |
y.log |
Whether to scale the y-space logarithmically |
generate_data.EAF_diff_Approximate(subset(dsl, funcId == 1), 1, 16, 1, 16)generate_data.EAF_diff_Approximate(subset(dsl, funcId == 1), 1, 16, 1, 16)
This function uses the 'eaf' package to calculate eaf differences
generate_data.EAF_Difference(dsList1, dsList2)generate_data.EAF_Difference(dsList1, dsList2)
dsList1 |
The first DataSetList object |
dsList2 |
The second DataSetList object |
generate_data.EAF_Difference(dsl[1], dsl[3])generate_data.EAF_Difference(dsl[1], dsl[3])
This function generates a dataframe which can be easily plotted using the 'plot_general_data'-function
generate_data.ECDF( dsList, targets, scale_log = F, which = "by_RT", use_full_range = TRUE )generate_data.ECDF( dsList, targets, scale_log = F, which = "by_RT", use_full_range = TRUE )
dsList |
The DataSetList object |
targets |
A list or data.table containing the targets per function / dimension. If this is a data.table, it needs columns 'target', 'DIM' and 'funcId' |
scale_log |
Wheterh to use logarithmic scaling or not |
which |
Whether to use a fixed-target 'by_RT' perspective or fixed-budget 'by_FV' |
use_full_range |
Whether or not to use the full range of the x-axis or cut it off as soon as all algorithms reach 98% success (+10% buffer). Only supported in the case of one function and dimension |
generate_data.ECDF(subset(dsl, funcId == 1), c(10, 15, 16))generate_data.ECDF(subset(dsl, funcId == 1), c(10, 15, 16))
This function uses EAF-data to calculate a target-independent version of the ECDF
generate_data.ECDF_From_EAF( eaf_table, min_val, max_val, maximization = F, scale_log = F, normalize = T )generate_data.ECDF_From_EAF( eaf_table, min_val, max_val, maximization = F, scale_log = F, normalize = T )
eaf_table |
Datatable resulting from the 'generate_data.EAF' function |
min_val |
Minimum value to use for y-space |
max_val |
Maximum value to use for y-space |
maximization |
Whether the data resulted from maximization or not |
scale_log |
Whether to use logarithmic scaling in y-space before calculating the partial integral |
normalize |
Whether to normalize the resulting integrals to [0,1] (Based on 'min_val' and 'max_va') |
generate_data.ECDF_From_EAF(generate_data.EAF(subset(dsl, funcId == 1)), 1, 16, maximization = TRUE)generate_data.ECDF_From_EAF(generate_data.EAF(subset(dsl, funcId == 1)), 1, 16, maximization = TRUE)
This provides an unaggregated version of the function 'generate_data.ECDF'.
generate_data.ECDF_raw(dsList, targets, scale_log = F)generate_data.ECDF_raw(dsList, targets, scale_log = F)
dsList |
The DataSetList object |
targets |
A list or data.table containing the targets per function / dimension. If this is a data.table, it needs columns 'target', 'DIM' and 'funcId' |
scale_log |
Wheterh to use logarithmic scaling or not |
generate_data.ECDF_raw(subset(dsl, funcId == 1), c(10, 15, 16))generate_data.ECDF_raw(subset(dsl, funcId == 1), c(10, 15, 16))
This procedure calculates the fraction of times algorithm A is better than algorithm B according to their mean on each function,dimension,target tuple
generate_data.Heatmaps(dsList, which = "by_FV", target_dt = NULL)generate_data.Heatmaps(dsList, which = "by_FV", target_dt = NULL)
dsList |
The DataSetList, can contain multiple functions and dimensions, but should have the same algorithms for all of them. For functions/dimensions where this is not the case, all algorithms are considered tied. |
which |
Whether to use fixed-target ('by_FV') or fixed-budget ('by_RT') perspective |
target_dt |
Custom data.table target value to use. When NULL, this is selected automatically. |
A matrix containing the pairwise win-ratios.
generate_data.Heatmaps(dsl) generate_data.Heatmaps(dsl, which = 'by_RT')generate_data.Heatmaps(dsl) generate_data.Heatmaps(dsl, which = 'by_RT')
This function generates a dataframe which can be easily plotted using the 'plot_general_data'-function
generate_data.hist(dsList, target, use.equal.bins = F, which = "by_RT")generate_data.hist(dsList, target, use.equal.bins = F, which = "by_RT")
dsList |
The DataSetList object |
target |
The target value (Runtime or target value) |
use.equal.bins |
Whether all bins should be equal size for each algorithm or not |
which |
Whether to use a fixed-target 'by_RT' perspective or fixed-budget 'by_FV' |
generate_data.hist(subset(dsl, funcId == 1), target = 15, which = 'by_RT')generate_data.hist(subset(dsl, funcId == 1), target = 15, which = 'by_RT')
This function generates a dataframe which can be easily plotted using the 'plot_general_data'-function
generate_data.Parameter_correlation(dsList, par1, par2)generate_data.Parameter_correlation(dsList, par1, par2)
dsList |
The DataSetList object |
par1 |
The first parameter. Either a parameter name or 'f(x)' |
par2 |
The second parameter. Either a parameter name or 'f(x)' |
generate_data.Parameter_correlation(subset(dsl, funcId == 1), 'f(x)', 'f(x)')generate_data.Parameter_correlation(subset(dsl, funcId == 1), 'f(x)', 'f(x)')
This function generates a dataframe which can be easily plotted using the 'plot_general_data'-function
generate_data.Parameters(dsList, which = "by_RT", scale_log = F)generate_data.Parameters(dsList, which = "by_RT", scale_log = F)
dsList |
The DataSetList object |
which |
Whether to use a fixed-target 'by_RT' perspective or fixed-budget 'by_FV' |
scale_log |
Wheterh to use logarithmic scaling or not |
generate_data.Parameters(subset(dsl, funcId == 1))generate_data.Parameters(subset(dsl, funcId == 1))
This function generates a dataframe which can be easily plotted using the 'plot_general_data'-function
generate_data.PMF(dsList, target, which = "by_RT")generate_data.PMF(dsList, target, which = "by_RT")
dsList |
The DataSetList object |
target |
The target value (Runtime or target value) |
which |
Whether to use a fixed-target 'by_RT' perspective or fixed-budget 'by_FV' |
generate_data.PMF(subset(dsl, funcId == 1), target = 15, which = 'by_RT')generate_data.PMF(subset(dsl, funcId == 1), target = 15, which = 'by_RT')
This function generates a dataframe which can be easily plotted using the 'plot_general_data'-function
generate_data.Single_Function( dsList, start = NULL, stop = NULL, scale_log = F, which = "by_RT", include_opts = F, budget = NULL, include_geom_mean = F )generate_data.Single_Function( dsList, start = NULL, stop = NULL, scale_log = F, which = "by_RT", include_opts = F, budget = NULL, include_geom_mean = F )
dsList |
The DataSetList object |
start |
Optional start value (Runtime or target value) |
stop |
Optional end value (Runtime or target value) |
scale_log |
Wheterh to use logarithmic scaling or not |
which |
Whether to use a fixed-target 'by_RT' perspective or fixed-budget 'by_FV' |
include_opts |
Whether or not to also include the best value hit by each algorithm to the generated datapoints |
budget |
Optional; overwrites the budget of each individual algorithm when doing ERT calculations. Only works in fixed_target mode. |
include_geom_mean |
Boolean to indicate whether to include the geometric mean. Only works in fixed_budget mode. Negative values cause NaN, zeros cause output to be completely 0. Defaults to False. |
generate_data.Single_Function(subset(dsl, funcId == 1), which = 'by_RT')generate_data.Single_Function(subset(dsl, funcId == 1), which = 'by_RT')
Get all algorithm ids present in a DataSetList
get_algId(dsList)get_algId(dsList)
dsList |
The DataSetLsit |
A sorted list of all unique algorithm ids which occur in the DataSetList
get_algId(dsl)get_algId(dsl)
Get colors according to the current colorScheme of the IOHanalyzer
get_color_scheme(ids_in)get_color_scheme(ids_in)
ids_in |
List of algorithms (or custom ids, see 'change_id') for which to get colors |
get_color_scheme(get_algId(dsl))get_color_scheme(get_algId(dsl))
Get datatable of current color (and linestyle) scheme to file
get_color_scheme_dt()get_color_scheme_dt()
data.table object with 3 columns: ids, colors, linestyles
get_color_scheme_dt()get_color_scheme_dt()
Generate ECDF targets for a DataSetList
get_default_ECDF_targets(data, format_func = as.integer)get_default_ECDF_targets(data, format_func = as.integer)
data |
A DataSetList |
format_func |
function to format the targets |
a vector of targets
get_default_ECDF_targets(dsl)get_default_ECDF_targets(dsl)
Get all dimensions present in a DataSetList
get_dim(dsList)get_dim(dsList)
dsList |
The DataSetLsit |
A sorted list of all unique dimensions which occur in the DataSetList
get_dim(dsl)get_dim(dsl)
Perform omnibus statistical tests on the matrix of rankings from the DSCtool api
get_dsc_omnibus(res, method = NULL, alpha = 0.05)get_dsc_omnibus(res, method = NULL, alpha = 0.05)
res |
The result of a call to the 'get_dsc_rank' |
method |
Which method to use to do the tests. Has be be one of the allowed ones in 'res$valid_methods'. When NULL, the first valid option is chosen by default |
alpha |
Threshold value for statistical significance |
A named list containing the algorithm means
get_dsc_omnibus(get_dsc_rank(dsl, na.correction = 'PAR-10'))get_dsc_omnibus(get_dsc_rank(dsl, na.correction = 'PAR-10'))
Perform post-hoc processing on data from DSCtool
get_dsc_posthoc( omni_res, nr_algs, nr_problems, base_algorithm = NULL, method = "friedman", alpha = 0.05 )get_dsc_posthoc( omni_res, nr_algs, nr_problems, base_algorithm = NULL, method = "friedman", alpha = 0.05 )
omni_res |
The result from a call to 'get_dsc_omnibus' |
nr_algs |
The number of algorithms present in 'omni_res' |
nr_problems |
The number of problems present in 'omni_res' |
base_algorithm |
The base algorithm to which the other are compared. This has to be present in 'omni_res$algorithm_means' as an 'algorithm' property |
method |
Either 'friedman' or 'friedman-aligned-rank' |
alpha |
Threshold value for statistical significance |
A named list containing 4 types of analyses: * Zvalue * UnadjustedPValue * Holm * Hochberg
get_dsc_posthoc(get_dsc_omnibus(get_dsc_rank(dsl, na.correction = 'PAR-10')), 2, 2)get_dsc_posthoc(get_dsc_omnibus(get_dsc_rank(dsl, na.correction = 'PAR-10')), 2, 2)
Get the matrix of rankings using the DSCtool api for a DataSetList
get_dsc_rank( dsList, targets = NULL, which = "by_RT", test_type = "AD", alpha = 0.05, epsilon = 0, monte_carlo_iterations = 0, na.correction = NULL )get_dsc_rank( dsList, targets = NULL, which = "by_RT", test_type = "AD", alpha = 0.05, epsilon = 0, monte_carlo_iterations = 0, na.correction = NULL )
dsList |
The DataSetList object |
targets |
Optional list of target values (Runtime or target value) |
which |
Whether to use a fixed-target 'by_RT' perspective or fixed-budget 'by_FV' |
test_type |
Either 'AD' for Anderson-Darling or KS for Kolmogorov-Smirnov tests |
alpha |
Threshold value for statistical significance |
epsilon |
Minimum threshold to have practical difference between algorithms (eDSC) |
monte_carlo_iterations |
How many monte-carlo-simulations to perform (set to 0 to use regular DSC) |
na.correction |
How to deal with missing values. Only used in fixed-target perspective. Options are: - 'NULL': No correction is done. This will likely result in an error, as the DSCtool does not allow for na values - 'PAR-1' Replace missing values with Budget (budget taken from relevant DataSet) - 'PAR-10' Replace missing values with 10*Budget (budget taken from relevant DataSet) - 'ERT' Replace NA values with the Expected Running Time. If all values are NA, this reverts to nr_runs * budget - 'Remove-na' Removes all NA values |
A named list containing a ranked-matrix which has the rankin of each algorithm on each problem, as well as a list of which omnibus tests can be used to further process this data. This can be further analyzed using 'get_dsc_omnibus'
get_dsc_rank(dsl, na.correction = 'PAR-10')get_dsc_rank(dsl, na.correction = 'PAR-10')
Generation of default ECDF-targets
get_ECDF_targets(dsList, type = "log-linear", number_targets = 10)get_ECDF_targets(dsList, type = "log-linear", number_targets = 10)
dsList |
The DataSetList object for which to generate the targets |
type |
The way to generate the targets. Either 'log-linear', 'linear' or 'bbob' (51 fixed targets, equal for all functions / dimensions) |
number_targets |
The amount of targets to generate |
A data.table with 3 columns: funcId, DIM and target
get_ECDF_targets(dsl, 'linear', 10)get_ECDF_targets(dsl, 'linear', 10)
Get Expected RunTime
get_ERT(ds, ftarget, budget, ...) ## S3 method for class 'DataSet' get_ERT(ds, ftarget, budget = NULL, ...) ## S3 method for class 'DataSetList' get_ERT(ds, ftarget, budget = NULL, algorithm = "all", ...)get_ERT(ds, ftarget, budget, ...) ## S3 method for class 'DataSet' get_ERT(ds, ftarget, budget = NULL, ...) ## S3 method for class 'DataSetList' get_ERT(ds, ftarget, budget = NULL, algorithm = "all", ...)
ds |
A DataSet or DataSetList object |
ftarget |
The function target(s) for which to get the ERT |
budget |
Optional; overwrites the budget found in ds for ERT-calculation |
... |
Arguments passed to other methods |
algorithm |
DEPRECATED, will be removed in next release. Which algorithms in the DataSetList to consider. |
A data.table containing the runtime samples for each provided target function value
get_ERT(dsl, 14) get_ERT(dsl[[1]], 14)get_ERT(dsl, 14) get_ERT(dsl[[1]], 14)
Get all function ids present in a DataSetList
get_funcId(dsList)get_funcId(dsList)
dsList |
The DataSetLsit |
A sorted list of all unique function ids which occur in the DataSetList
get_funcId(dsl)get_funcId(dsl)
Get all function names present in a DataSetList
get_funcName(dsList)get_funcName(dsList)
dsList |
The DataSetLsit |
A list of all unique function names which occur in the DataSetList
get_funcName(dsl)get_funcName(dsl)
Get all function values present in a DataSetList
get_funvals(dsList)get_funvals(dsList)
dsList |
The DataSetLsit |
A list matrices of all function values which occur in the DataSetList
get_funvals(dsl)get_funvals(dsl)
To be used instead of accessing ds$FV directly, since in the case of constrained problems, the violation handling should be applied before using the function values Constraint penalty function should be set in global options, as IOHanalyzer.Violation_Function
get_FV(ds, ...) ## S3 method for class 'DataSet' get_FV(ds, ...)get_FV(ds, ...) ## S3 method for class 'DataSet' get_FV(ds, ...)
ds |
The DataSet |
... |
Arguments passed to other methods |
The matrix of FV values in the dataset, penalized if applicable.
get_FV(dsl[[1]])get_FV(dsl[[1]])
Get Function Value condensed overview
get_FV_overview(ds, ...) ## S3 method for class 'DataSet' get_FV_overview(ds, ...) ## S3 method for class 'DataSetList' get_FV_overview(ds, algorithm = "all", ...)get_FV_overview(ds, ...) ## S3 method for class 'DataSet' get_FV_overview(ds, ...) ## S3 method for class 'DataSetList' get_FV_overview(ds, algorithm = "all", ...)
ds |
A 'DataSet' or 'DataSetList' object |
... |
Arguments passed to other methods |
algorithm |
DEPRECATED, will be removed in next release. Which algorithms in the DataSetList to consider. |
A data.table containing the algorithm ID, best, worst and mean reached function values, the number of runs and available budget for the DataSet
get_FV_overview(dsl) get_FV_overview(dsl[[1]]) get_FV_overview(dsl, algorithm = '(1+1)_greedy_hill_climber_1')get_FV_overview(dsl) get_FV_overview(dsl[[1]]) get_FV_overview(dsl, algorithm = '(1+1)_greedy_hill_climber_1')
Get Funtion Value Samples
get_FV_sample(ds, ...) ## S3 method for class 'DataSet' get_FV_sample(ds, runtime, output = "wide", ...) ## S3 method for class 'DataSetList' get_FV_sample(ds, runtime, algorithm = "all", ...)get_FV_sample(ds, ...) ## S3 method for class 'DataSet' get_FV_sample(ds, runtime, output = "wide", ...) ## S3 method for class 'DataSetList' get_FV_sample(ds, runtime, algorithm = "all", ...)
ds |
A DataSet or DataSetList object |
... |
Arguments passed to other methods |
runtime |
A Numerical vector. Runtimes at which function values are reached |
output |
A String. The format of the output data: 'wide' or 'long' |
algorithm |
DEPRECATED, will be removed in next release. Which algorithms in the DataSetList to consider. |
A data.table containing the function value samples for each provided target runtime
get_FV_sample(dsl, 100) get_FV_sample(dsl[[1]], 100)get_FV_sample(dsl, 100) get_FV_sample(dsl[[1]], 100)
Get Function Value Summary
get_FV_summary(ds, ...) ## S3 method for class 'DataSet' get_FV_summary(ds, runtime, include_geom_mean = F, ...) ## S3 method for class 'DataSetList' get_FV_summary(ds, runtime, algorithm = "all", include_geom_mean = F, ...)get_FV_summary(ds, ...) ## S3 method for class 'DataSet' get_FV_summary(ds, runtime, include_geom_mean = F, ...) ## S3 method for class 'DataSetList' get_FV_summary(ds, runtime, algorithm = "all", include_geom_mean = F, ...)
ds |
A DataSet or DataSetList object |
... |
Arguments passed to other methods |
runtime |
A Numerical vector. Runtimes at which function values are reached |
include_geom_mean |
Boolean to indicate whether to include the geometric mean. Only works in fixed_budget mode. Negative values cause NaN, zeros cause output to be completely 0. Defaults to False. |
algorithm |
DEPRECATED, will be removed in next release. Which algorithms in the DataSetList to consider. |
A data.table containing the function value statistics for each provided target runtime value
get_FV_summary(dsl, 100) get_FV_summary(dsl[[1]], 100)get_FV_summary(dsl, 100) get_FV_summary(dsl[[1]], 100)
Get the unique identifiers for each DataSet in the provided DataSetList
get_id(ds, ...) ## S3 method for class 'DataSet' get_id(ds, ...) ## S3 method for class 'DataSetList' get_id(ds, ...)get_id(ds, ...) ## S3 method for class 'DataSet' get_id(ds, ...) ## S3 method for class 'DataSetList' get_id(ds, ...)
ds |
The DataSetList |
... |
Arguments passed to other methods |
If no unique identifier is set (using 'change_id' or done in DataSet construction from 1.6.0 onwards), this function falls back on returning the algorith id (from 'get_aldId')to ensure backwards compatibility
The list of unique identiefiers present in dsl
get_id(dsl) get_id(dsl[[1]])get_id(dsl) get_id(dsl[[1]])
Get line styles according to the current styleScheme of the IOHanalyzer
get_line_style(ids_in)get_line_style(ids_in)
ids_in |
List of algorithms (or custom ids, see 'change_id') for which to get linestyles |
get_line_style(get_algId(dsl))get_line_style(get_algId(dsl))
Based on the contribution to the ECDF-curve of the VBS of the portfolio
get_marg_contrib_ecdf(id, perm, j, dt)get_marg_contrib_ecdf(id, perm, j, dt)
id |
The id for which to get the contribution |
perm |
The permutation of algorithms to which is being contributed |
j |
At which point in the permutation the contribution should be measured |
dt |
The datatable in which the raw ecdf-values are stored (see 'generate_data.ECDF_raw') |
dt <- generate_data.ECDF_raw(dsl, get_ECDF_targets(dsl)) get_marg_contrib_ecdf(get_id(dsl)[[1]], get_id(dsl), 1, dt)dt <- generate_data.ECDF_raw(dsl, get_ECDF_targets(dsl)) get_marg_contrib_ecdf(get_id(dsl)[[1]], get_id(dsl), 1, dt)
Get the maximal running time
get_maxRT(ds, ...) ## S3 method for class 'DataSet' get_maxRT(ds, output = "wide", ...) ## S3 method for class 'DataSetList' get_maxRT(ds, algorithm = "all", ...)get_maxRT(ds, ...) ## S3 method for class 'DataSet' get_maxRT(ds, output = "wide", ...) ## S3 method for class 'DataSetList' get_maxRT(ds, algorithm = "all", ...)
ds |
A DataSet or DataSetList object |
... |
Arguments passed to other methods |
output |
The format of the outputted table: 'wide' or 'long' |
algorithm |
DEPRECATED, will be removed in next release. Which algorithms in the DataSetList to consider. |
A data.table object containing the algorithm ID and the running time when the algorithm terminates in each run
get_maxRT(dsl) get_maxRT(dsl[[1]])get_maxRT(dsl) get_maxRT(dsl[[1]])
Get the list of available options for data from the OPTION ontology
get_ontology_data( datasource, fids, dims, algs, iids = NULL, funcsuites = NULL, min_target = NULL, max_target = NULL, min_budget = NULL, max_budget = NULL )get_ontology_data( datasource, fids, dims, algs, iids = NULL, funcsuites = NULL, min_target = NULL, max_target = NULL, min_budget = NULL, max_budget = NULL )
datasource |
The datasource: either BBOB or Nevergrad |
fids |
The function names as given by 'get_ontology_var' |
dims |
The dimensionalities as given by 'get_ontology_var' |
algs |
The algorithm names as given by 'get_ontology_var' |
iids |
The instances as given by 'get_ontology_var' (only for BBOB data) |
funcsuites |
The function suite as given by 'get_ontology_var' (only for Nevergrad data) |
min_target |
The minimum target value for which to return data |
max_target |
The maximum target value for which to return data |
min_budget |
The minimum budget value for which to return data |
max_budget |
The maximum budget value for which to return data |
a DataSetList object matching the selected attributes.
get_ontology_data("BBOB", "f5", 5, "IPOP400D", 1)get_ontology_data("BBOB", "f5", 5, "IPOP400D", 1)
Get the list of available options for data from the OPTION ontology
get_ontology_var(varname, datasource = NULL, study = NULL, algs = NULL, ...)get_ontology_var(varname, datasource = NULL, study = NULL, algs = NULL, ...)
varname |
The variable for which to get the options. Restricted to [Fid, Iid, DIM, AlgId, Suite] |
datasource |
The datasource for which to get the attributes. Either BBOB or Nevergrad, or NULL if looking at a specific 'study' argument |
study |
Which study to load the requested variables for (NULL if no study is considered) |
algs |
Which algorithms to get the requested variables for. Required for varnames in [Fid, Iid, DIM] |
... |
Additional arguments to the OPTION call. Currently only supports 'Suite' for nevergrad. |
the options of varname given the specified datasource
get_ontology_var("Fid", "BBOB")get_ontology_var("Fid", "BBOB")
Get condensed overview of datasets
get_overview(ds, ...) ## S3 method for class 'DataSet' get_overview(ds, ...) ## S3 method for class 'DataSetList' get_overview(ds, ...)get_overview(ds, ...) ## S3 method for class 'DataSet' get_overview(ds, ...) ## S3 method for class 'DataSetList' get_overview(ds, ...)
ds |
A DataSet or DataSetList object |
... |
Arguments passed to other methods |
A data.table containing some basic information about the provided DataSet(List)
get_overview(dsl) get_overview(dsl[[1]])get_overview(dsl) get_overview(dsl[[1]])
Get the parameter names of the algorithm
get_PAR_name(ds, which) ## S3 method for class 'DataSet' get_PAR_name(ds, which = "by_FV")get_PAR_name(ds, which) ## S3 method for class 'DataSet' get_PAR_name(ds, which = "by_FV")
ds |
A DataSet object |
which |
a string takes it value in ‘c(’by_FV', 'by_RT')', indicating the parameters aligned against the running time (RT) or function value (FV). ''by_FV'' is the default value. |
a character list of paramter names, if recorded in the data set
get_PAR_name(dsl[[1]])get_PAR_name(dsl[[1]])
Get Parameter Value Samples
get_PAR_sample(ds, idxValue, ...) ## S3 method for class 'DataSet' get_PAR_sample( ds, idxValue, parId = "all", which = "by_FV", output = "wide", ... ) ## S3 method for class 'DataSetList' get_PAR_sample(ds, idxValue, algorithm = "all", ...)get_PAR_sample(ds, idxValue, ...) ## S3 method for class 'DataSet' get_PAR_sample( ds, idxValue, parId = "all", which = "by_FV", output = "wide", ... ) ## S3 method for class 'DataSetList' get_PAR_sample(ds, idxValue, algorithm = "all", ...)
ds |
A DataSet or DataSetList object |
idxValue |
A Numerical vector. Index values at which parameter values are observed. The index value can either take its value in the range of running times, or function values. Such a value type is signified by 'which' parameter. |
... |
Arguments passed to other methods |
parId |
A character vector. Either 'all' or the name of parameters to be retrieved |
which |
A string takes values in ‘c(’by_FV', 'by_RT')', indicating the parameters to be retrieved are aligned against the running time (RT) or function value (FV). ''by_FV'' is the default value. |
output |
A character. The format of the output data: 'wide' or 'long' |
algorithm |
DEPRECATED, will be removed in next release. Which algorithms in the DataSetList to consider. |
A data.table object containing parameter values aligned at each given target value
get_PAR_sample(dsl, 14) get_PAR_sample(dsl[[1]], 14)get_PAR_sample(dsl, 14) get_PAR_sample(dsl[[1]], 14)
Get Parameter Value Summary
get_PAR_summary(ds, idxValue, ...) ## S3 method for class 'DataSet' get_PAR_summary(ds, idxValue, parId = "all", which = "by_FV", ...) ## S3 method for class 'DataSetList' get_PAR_summary(ds, idxValue, algorithm = "all", ...)get_PAR_summary(ds, idxValue, ...) ## S3 method for class 'DataSet' get_PAR_summary(ds, idxValue, parId = "all", which = "by_FV", ...) ## S3 method for class 'DataSetList' get_PAR_summary(ds, idxValue, algorithm = "all", ...)
ds |
A DataSet or DataSetList object |
idxValue |
A Numerical vector. Index values at which parameter values are observed. The index value can either take its value in the range of running times, or function values. Such a value type is signified by 'which' parameter. |
... |
Arguments passed to other methods |
parId |
A character vector. Either 'all' or the name of parameters to be retrieved |
which |
A string takes values in ‘c(’by_FV', 'by_RT')', indicating the parameters to be retrieved are aligned against the running time (RT) or function value (FV). ''by_FV'' is the default value. |
algorithm |
DEPRECATED, will be removed in next release. Which algorithms in the DataSetList to consider. |
A data.table object containing basic statistics of parameter values aligned at each given target value
get_PAR_summary(dsl, 14) get_PAR_summary(dsl[[1]], 14)get_PAR_summary(dsl, 14) get_PAR_summary(dsl[[1]], 14)
Get all parameter ids present in a DataSetList
get_parId(dsList, which = "by_FV")get_parId(dsList, which = "by_FV")
dsList |
The DataSetList |
which |
A string takes values in ‘c(’by_FV', 'by_RT')'. To choose the parameters aligned by the running time (RT) or the function value (FV). Note that parameters in each case are not necessary the same. |
A sorted list of all unique parameter ids which occur in the DataSetList
get_parId(dsl)get_parId(dsl)
Extract the position information from a datasetlist object
get_position_dsl(dsList, iid)get_position_dsl(dsList, iid)
dsList |
The DataSetList object |
iid |
the Instance Id from which to get the position history (can be a list) |
get_position_dsl(subset(dsl, funcId == 1), 1)get_position_dsl(subset(dsl, funcId == 1), 1)
To be used instead of accessing ds$RT directly, since in the case of constrained problems, the violation handling should be applied before using the function values Constraint penalty function should be set in global options, as IOHanalyzer.Violation_Function
get_RT(ds, ...) ## S3 method for class 'DataSet' get_RT(ds, ...)get_RT(ds, ...) ## S3 method for class 'DataSet' get_RT(ds, ...)
ds |
The DataSet |
... |
Arguments passed to other methods |
The matrix of FV values in the dataset, penalized if applicable.
get_RT(dsl[[1]])get_RT(dsl[[1]])
Get Runtime Value condensed overview
get_RT_overview(ds, ...) ## S3 method for class 'DataSet' get_RT_overview(ds, ...) ## S3 method for class 'DataSetList' get_RT_overview(ds, algorithm = "all", ...)get_RT_overview(ds, ...) ## S3 method for class 'DataSet' get_RT_overview(ds, ...) ## S3 method for class 'DataSetList' get_RT_overview(ds, algorithm = "all", ...)
ds |
A DataSet or DataSetList object |
... |
Arguments passed to other methods |
algorithm |
DEPRECATED, will be removed in next release. Which algorithms in the DataSetList to consider. |
A data.table containing the algorithm ID, minimum and maximum used evaluations, number of runs and available budget for the DataSet
get_RT_overview(dsl) get_RT_overview(dsl[[1]])get_RT_overview(dsl) get_RT_overview(dsl[[1]])
Get RunTime Sample
get_RT_sample(ds, ftarget, ...) ## S3 method for class 'DataSet' get_RT_sample(ds, ftarget, output = "wide", ...) ## S3 method for class 'DataSetList' get_RT_sample(ds, ftarget, algorithm = "all", ...)get_RT_sample(ds, ftarget, ...) ## S3 method for class 'DataSet' get_RT_sample(ds, ftarget, output = "wide", ...) ## S3 method for class 'DataSetList' get_RT_sample(ds, ftarget, algorithm = "all", ...)
ds |
A DataSet or DataSetList object |
ftarget |
A Numerical vector. Function values at which runtime values are consumed |
... |
Arguments passed to other methods |
output |
A character determining the format of output data.table: 'wide' or 'long' |
algorithm |
DEPRECATED, will be removed in next release. Which algorithms in the DataSetList to consider. |
A data.table containing the runtime samples for each provided target function value
get_RT_sample(dsl, 14) get_RT_sample(dsl[[1]], 14)get_RT_sample(dsl, 14) get_RT_sample(dsl[[1]], 14)
Get RunTime Summary
get_RT_summary(ds, ftarget, budget, ...) ## S3 method for class 'DataSet' get_RT_summary(ds, ftarget, budget = NULL, ...) ## S3 method for class 'DataSetList' get_RT_summary(ds, ftarget, budget = NULL, ...)get_RT_summary(ds, ftarget, budget, ...) ## S3 method for class 'DataSet' get_RT_summary(ds, ftarget, budget = NULL, ...) ## S3 method for class 'DataSetList' get_RT_summary(ds, ftarget, budget = NULL, ...)
ds |
A DataSet or DataSetList object |
ftarget |
The function target(s) for which to get the runtime summary |
budget |
Optional; overwrites the budget found in ds for ERT-calculation |
... |
Arguments passed to other methods |
A data.table containing the runtime statistics for each provided target function value
get_RT_summary(dsl, 14) get_RT_summary(dsl[[1]], 14)get_RT_summary(dsl, 14) get_RT_summary(dsl[[1]], 14)
Get all runtime values present in a DataSetList
get_runtimes(dsList)get_runtimes(dsList)
dsList |
The DataSetLsit |
A list matrices of all runtime values which occur in the DataSetList
get_runtimes(dsl)get_runtimes(dsl)
Based on the contribution to the ECDF-curve of the VBS of the portfolio
get_shapley_values( dsList, targets, scale.log = T, group_size = 5, max_perm_size = 10, normalize = T )get_shapley_values( dsList, targets, scale.log = T, group_size = 5, max_perm_size = 10, normalize = T )
dsList |
The DataSetList object |
targets |
A list or data.table containing the targets per function / dimension. If this is a data.table, it needs columns 'target', 'DIM' and 'funcId' |
scale.log |
Whether to use logarithmic scaling for the runtimes at which the ecdf will be sampled or not |
group_size |
How many permutation groups will be considered |
max_perm_size |
The maximum limit for permutations to be considered |
normalize |
Whether or not to ensure the resulting values will be in [0,1] |
dsl_sub <- subset(dsl, funcId == 1) get_shapley_values(dsl_sub, get_ECDF_targets(dsl_sub), group_size = 2)dsl_sub <- subset(dsl, funcId == 1) get_shapley_values(dsl_sub, get_ECDF_targets(dsl_sub), group_size = 2)
This is a more generic version of the existing 'get_dim', 'get_funcId' and 'get_algId' functions. Note the only attributes returned by 'get_static_attributes' are supported in this funcion
get_static_attribute_values(dsl, attribute)get_static_attribute_values(dsl, attribute)
dsl |
The DataSetList |
attribute |
the name of the attribute for which to get the available options in dsl |
The list of options for the specified attribute
get_static_attribute_values(dsl, 'funcId')get_static_attribute_values(dsl, 'funcId')
Get all attributes which can be used to subset a DataSetList
get_static_attributes(dsl)get_static_attributes(dsl)
dsl |
The DataSetList |
The list of available attributes
get_static_attributes(dsl)get_static_attributes(dsl)
Only one target is generated per (function, dimension)-pair, as opposed to the function 'get_default_ECDF_targets', which generates multiple targets.
get_target_dt(dsList, which = "by_RT")get_target_dt(dsList, which = "by_RT")
dsList |
A DataSetList |
which |
Whether to generate fixed-target ('by_FV') or fixed-budget ('by_RT') targets |
a data.table of targets
get_target_dt(dsl)get_target_dt(dsl)
This procedure ranks algorithms based on a glicko2-procedure. Every round (total nr_rounds), for every function and dimension of the datasetlist, each pair of algorithms competes. This competition samples a random runtime for the provided target (defaults to best achieved target). Whichever algorithm has the lower runtime wins the game. Then, from these games, the glicko2-rating is determined.
glicko2_ranking(dsl, nr_rounds = 100, which = "by_FV", target_dt = NULL)glicko2_ranking(dsl, nr_rounds = 100, which = "by_FV", target_dt = NULL)
dsl |
The DataSetList, can contain multiple functions and dimensions, but should have the same algorithms for all of them |
nr_rounds |
The number of rounds to run. More rounds leads to a more accurate ranking. |
which |
Whether to use fixed-target ('by_FV') or fixed-budget ('by_RT') perspective |
target_dt |
Custom data.table target value to use. When NULL, this is selected automatically. |
A dataframe containing the glicko2-ratings and some additional info
glicko2_ranking(dsl, nr_round = 25) glicko2_ranking(dsl, nr_round = 25, which = 'by_RT')glicko2_ranking(dsl, nr_round = 25) glicko2_ranking(dsl, nr_round = 25, which = 'by_RT')
Template for creating plots in the IOHanalyzer-style
IOH_plot_ly_default(title = NULL, x.title = NULL, y.title = NULL)IOH_plot_ly_default(title = NULL, x.title = NULL, y.title = NULL)
title |
Title for the plot |
x.title |
X-axis label |
y.title |
Y-axis label |
IOH_plot_ly_default("Example plot","x-axis","y-axis")IOH_plot_ly_default("Example plot","x-axis","y-axis")
The data analysis module for the Iterative Optimization Heuristics Profiler (IOHprofiler). This module provides statistical analysis methods for the benchmark data generated by optimization heuristics, which can be visualized through a web-based interface. The benchmark data is usually generated by the experimentation module, called IOHexperimenter. IOHanalyzer also supports the widely used COCO (Comparing Continuous Optimisers) data format for benchmarking.
The IOHanalyzer consists of 3 main functionalities:
Reading and alligning data from different heuristics, such as IOHExperimenter.
This is done using the DataSet and DataSetList functions
Processing and summarizing this data
Creating various plots
Maintainer: Diederick Vermetten [email protected] (ORCID)
Authors:
Hao Wang [email protected] (ORCID)
Carola Doerr [email protected] (ORCID)
Thomas Bäck [email protected] (ORCID)
Useful links:
Report bugs at https://github.com/IOHprofiler/IOHAnalyzer/issues
path <- system.file("extdata", "ONE_PLUS_LAMDA_EA", package="IOHanalyzer") dsList <- DataSetList(path) summary(dsList) Plot.RT.Single_Func(dsList[1]) ## Not run: runServer() ## End(Not run)path <- system.file("extdata", "ONE_PLUS_LAMDA_EA", package="IOHanalyzer") dsList <- DataSetList(path) summary(dsList) Plot.RT.Single_Func(dsList[1]) ## Not run: runServer() ## End(Not run)
Reduce the size of the data set by evenly subsampling the records
limit.data(df, n)limit.data(df, n)
df |
The data to subsample |
n |
The amount of samples |
A smaller data.frame
Get the ERT-values for all DataSets in a DataSetList at certain targets
max_ERTs(dsList, aggr_on = "funcId", targets = NULL, maximize = T) ## S3 method for class 'DataSetList' max_ERTs(dsList, aggr_on = "funcId", targets = NULL, maximize = T)max_ERTs(dsList, aggr_on = "funcId", targets = NULL, maximize = T) ## S3 method for class 'DataSetList' max_ERTs(dsList, aggr_on = "funcId", targets = NULL, maximize = T)
dsList |
The DataSetLsit |
aggr_on |
Whether to aggregate on 'funcId' or 'DIM'. |
targets |
Predifined target function-values. Should be one for each function/dimension |
maximize |
Whether the DataSetList is from a maximization or minimization problem |
A data.table containing ERT-values
max_ERTs(dsl)max_ERTs(dsl)
Get the expected function-values for all DataSets in a DataSetList at certain runtimes
mean_FVs(dsList, aggr_on = "funcId", runtimes = NULL) ## S3 method for class 'DataSetList' mean_FVs(dsList, aggr_on = "funcId", runtimes = NULL)mean_FVs(dsList, aggr_on = "funcId", runtimes = NULL) ## S3 method for class 'DataSetList' mean_FVs(dsList, aggr_on = "funcId", runtimes = NULL)
dsList |
The DataSetLsit |
aggr_on |
Whether to aggregate on 'funcId' or 'DIM'. |
runtimes |
Predifined target runtimes-values. Should be one for each function/dimension |
A data.table containing expected fucntion-values
mean_FVs(dsl)mean_FVs(dsl)
This function performs a Kolmogorov-Smirnov test on each pair of algorithms in the input x to determine which algorithm gives a significantly smaller running time. The resulting p-values are arranged in a matrix, where each cell (i, j) contains a p-value from the test with alternative hypothesis: the running time of algorithm i is smaller (thus better) than that of j.
pairwise.test(x, ...) ## S3 method for class 'list' pairwise.test(x, max_eval, bootstrap.size = 30, ...) ## S3 method for class 'DataSetList' pairwise.test(x, ftarget, bootstrap.size = 0, which = "by_FV", ...)pairwise.test(x, ...) ## S3 method for class 'list' pairwise.test(x, max_eval, bootstrap.size = 30, ...) ## S3 method for class 'DataSetList' pairwise.test(x, ftarget, bootstrap.size = 0, which = "by_FV", ...)
x |
either a list that contains running time sample for each algorithm as sub-lists, or a DataSetList object |
... |
all other options |
max_eval |
list that contains the maximal running time for each algorithm as sub-lists |
bootstrap.size |
integer, the size of the bootstrapped sample. Set to 0 to disable bootstrapping |
ftarget |
float, the target value used to determine the running / hitting |
which |
wheter to do fixed-target ('by_FV') or fixed-budget ('by_RT') comparison time |
A matrix containing p-values of the test
pairwise.test(subset(dsl, funcId == 1), 16)pairwise.test(subset(dsl, funcId == 1), 16)
Create EAF-based polygon plots
plot_eaf_data( df, maximization = F, scale.xlog = F, scale.ylog = F, scale.reverse = F, p = NULL, x_title = NULL, xmin = NULL, xmax = NULL, ymin = NULL, ymax = NULL, y_title = NULL, plot_title = NULL, subplot_attr = NULL, show.colorbar = F, subplot_shareX = F, dt_overlay = NULL, ... )plot_eaf_data( df, maximization = F, scale.xlog = F, scale.ylog = F, scale.reverse = F, p = NULL, x_title = NULL, xmin = NULL, xmax = NULL, ymin = NULL, ymax = NULL, y_title = NULL, plot_title = NULL, subplot_attr = NULL, show.colorbar = F, subplot_shareX = F, dt_overlay = NULL, ... )
df |
The dataframe containing the data to plot. This should come from 'generate_data.EAF' |
maximization |
Whether the data comes from maximization or minimization |
scale.xlog |
Logarithmic scaling of x-axis |
scale.ylog |
Logarithmic scaling of y-axis |
scale.reverse |
Decreasing or increasing x-axis |
p |
A previously existing plot on which to add traces. If NULL, a new canvas is created |
x_title |
Title of x-axis. Defaults to x_attr |
xmin |
Minimum value for the x-axis |
xmax |
Maximum value for the x-axis |
ymin |
Minimum value for the y-axis |
ymax |
Maximum value for the y-axis |
y_title |
Title of x-axis. Defaults to x_attr |
plot_title |
Title of x-axis. Defaults to no title |
subplot_attr |
Which attribute of the dataframe to use for creating subplots |
show.colorbar |
Whether or not to include a colorbar |
subplot_shareX |
Whether or not to share X-axis when using subplots |
dt_overlay |
Dataframe containing additional data (e.g. quantiles) to plot on top of the EAF. This should have a column labeled 'runtime'. The other columsn will all be plotted as function values. |
... |
Additional parameters for the add_trace function |
An EAF plot
## Not run: plot_eaf_data(generate_data.EAF(subset(dsl, ID==get_id(dsl)[[1]])), maximization=T) ## End(Not run)## Not run: plot_eaf_data(generate_data.EAF(subset(dsl, ID==get_id(dsl)[[1]])), maximization=T) ## End(Not run)
Create EAF-difference contour plots
plot_eaf_differences( matrices, scale.xlog = T, scale.ylog = F, zero_transparant = F, show_negatives = F )plot_eaf_differences( matrices, scale.xlog = T, scale.ylog = F, zero_transparant = F, show_negatives = F )
matrices |
The dataframes containing the data to plot. This should come from 'generate_data.EAF_diff_Approximate' |
scale.xlog |
Logarithmic scaling of x-axis |
scale.ylog |
Logarithmic scaling of y-axis |
zero_transparant |
Whether values of 0 should be made transparant or not |
show_negatives |
Whether to also show negative values or not |
EAF difference plots
## Not run: plot_eaf_differences(generate_data.EAF_diff_Approximate(subset(dsl, funcId == 1), 1, 50, 1, 16)) ## End(Not run)## Not run: plot_eaf_differences(generate_data.EAF_diff_Approximate(subset(dsl, funcId == 1), 1, 50, 1, 16)) ## End(Not run)
General function for plotting within IOHanalyzer
plot_general_data( df, x_attr = "ID", y_attr = "vals", type = "violin", legend_attr = "ID", scale.xlog = F, scale.ylog = F, scale.reverse = F, p = NULL, x_title = NULL, y_title = NULL, plot_title = NULL, upper_attr = NULL, lower_attr = NULL, subplot_attr = NULL, show.legend = F, inf.action = "none", violin.showpoints = F, frame_attr = "frame", symbol_attr = "run_nr", subplot_shareX = F, line.step = F, ... )plot_general_data( df, x_attr = "ID", y_attr = "vals", type = "violin", legend_attr = "ID", scale.xlog = F, scale.ylog = F, scale.reverse = F, p = NULL, x_title = NULL, y_title = NULL, plot_title = NULL, upper_attr = NULL, lower_attr = NULL, subplot_attr = NULL, show.legend = F, inf.action = "none", violin.showpoints = F, frame_attr = "frame", symbol_attr = "run_nr", subplot_shareX = F, line.step = F, ... )
df |
The dataframe containing the data to plot. It should contain at least two columns: 'x_attr' and 'y_attr' |
x_attr |
The column to specify the x_axis. Default is 'algId' |
y_attr |
The column to specify the y_axis |
type |
The type of plot to use. Currently available: 'violin', 'line', 'radar', 'bar', hist' and 'ribbon' |
legend_attr |
Default is 'algId' This is also used for the selection of colorschemes |
scale.xlog |
Logarithmic scaling of x-axis |
scale.ylog |
Logarithmic scaling of y-axis |
scale.reverse |
Decreasing or increasing x-axis |
p |
A previously existing plot on which to add traces. If NULL, a new canvas is created |
x_title |
Title of x-axis. Defaults to x_attr |
y_title |
Title of x-axis. Defaults to x_attr |
plot_title |
Title of x-axis. Defaults to no title |
upper_attr |
When using ribbon-plot, this can be used to create a shaded area. Only works in combination with‘lower_attr' and 'type' == ’ribbon' |
lower_attr |
When using ribbon-plot, this can be used to create a shaded area. Only works in combination with‘upper_attr' and 'type' == ’ribbon' |
subplot_attr |
Which attribute of the dataframe to use for creating subplots |
show.legend |
Whether or not to include a legend |
inf.action |
How to deal with infinite values. Can be 'none', 'overlap' or 'jitter' |
violin.showpoints |
Wheteher or not to show individual points when making a violinplot |
frame_attr |
Which attribute of the dataframe to use for the time element of the animation |
symbol_attr |
Which attribute of the dataframe to use for the scatter symbol |
subplot_shareX |
Whether or not to share X-axis when using subplots |
line.step |
Whether to plot lines as a step-function (T) or as linear interpolation (F, default) |
... |
Additional parameters for the add_trace function |
Plot a heatmap according to the specifications from the Nevergrad dashboard
Plot.Comparison.Heatmap(dsList, target_dt, which = "by_FV") ## S3 method for class 'DataSetList' Plot.Comparison.Heatmap(dsList, target_dt = NULL, which = "by_FV")Plot.Comparison.Heatmap(dsList, target_dt, which = "by_FV") ## S3 method for class 'DataSetList' Plot.Comparison.Heatmap(dsList, target_dt = NULL, which = "by_FV")
dsList |
A DataSetList (should consist of only one function and dimension). |
target_dt |
A data-table containing the targets to condider on each function/dimension pair |
which |
Whether to use fixed-target ('by_FV') or fixed-budget ('by_RT') perspective |
A heatmap showing the fraction of times algorithm A beats algorithm B
Plot.Comparison.Heatmap(dsl)Plot.Comparison.Heatmap(dsl)
Plot the cumulative difference plot given a DataSetList.
Plot.cumulative_difference_plot( dsList, runtime_or_target_value, isFixedBudget, alpha = 0.05, EPSILON = 1e-80, nOfBootstrapSamples = 1000, dataAlreadyComputed = FALSE, precomputedData = NULL )Plot.cumulative_difference_plot( dsList, runtime_or_target_value, isFixedBudget, alpha = 0.05, EPSILON = 1e-80, nOfBootstrapSamples = 1000, dataAlreadyComputed = FALSE, precomputedData = NULL )
dsList |
A DataSetList (should consist of only one function and dimension and two algorithms). |
runtime_or_target_value |
The target runtime or the target value |
isFixedBudget |
Should be TRUE when target runtime is used. False otherwise. |
alpha |
1 minus the confidence level of the confidence band. |
EPSILON |
If abs(x-y) < EPSILON, then we assume that x = y. |
nOfBootstrapSamples |
The number of bootstrap samples used in the estimation. |
dataAlreadyComputed |
If false, 'generate_data.CDP' will be called to process the data. |
precomputedData |
only needed when dataAlreadyComputed=TRUE. The result of 'generate_data.CDP'. |
A cumulative difference plot.
dsl dsl_sub <- subset(dsl, funcId == 1) target <- 15 Plot.cumulative_difference_plot(dsl_sub, target, FALSE)dsl dsl_sub <- subset(dsl, funcId == 1) target <- 15 Plot.cumulative_difference_plot(dsl_sub, target, FALSE)
Plot expected function value-based comparison over multiple functions or dimensions
Plot.FV.Aggregated( dsList, aggr_on = "funcId", runtimes = NULL, plot_mode = "radar", use_rank = F, scale.ylog = T, fvs = NULL ) ## S3 method for class 'DataSetList' Plot.FV.Aggregated( dsList, aggr_on = "funcId", runtimes = NULL, plot_mode = "radar", use_rank = F, scale.ylog = T, fvs = NULL )Plot.FV.Aggregated( dsList, aggr_on = "funcId", runtimes = NULL, plot_mode = "radar", use_rank = F, scale.ylog = T, fvs = NULL ) ## S3 method for class 'DataSetList' Plot.FV.Aggregated( dsList, aggr_on = "funcId", runtimes = NULL, plot_mode = "radar", use_rank = F, scale.ylog = T, fvs = NULL )
dsList |
A DataSetList (should consist of only one function OR dimension). |
aggr_on |
Whether to compare on functions ('funcId') or dimensions ('DIM') |
runtimes |
Custom list of function-value targets, one for each function or dimension. |
plot_mode |
How the plots should be created. Can be 'line' or 'radar' |
use_rank |
Wheter to use a ranking system. If False, the actual expected function- values will be used. |
scale.ylog |
Whether or not to scale the y-axis logaritmically |
fvs |
Pre-calculated expected function-values for the provided runtimes Created by the max_ERTs function of DataSetList. Can be provided to prevent needless computation in recalculating ERTs when recreating this plot. |
A plot of expected function value-based comparison on the provided functions or dimensions of the DataSetList
Plot.FV.Aggregated(dsl)Plot.FV.Aggregated(dsl)
Radarplot of the area under the aggregated ECDF-curve of a DataSetList.
Plot.FV.ECDF_AUC(dsList, rt_min = NULL, rt_max = NULL, rt_step = NULL) ## S3 method for class 'DataSetList' Plot.FV.ECDF_AUC(dsList, rt_min = NULL, rt_max = NULL, rt_step = NULL)Plot.FV.ECDF_AUC(dsList, rt_min = NULL, rt_max = NULL, rt_step = NULL) ## S3 method for class 'DataSetList' Plot.FV.ECDF_AUC(dsList, rt_min = NULL, rt_max = NULL, rt_step = NULL)
dsList |
A DataSetList (should consist of only one function and dimension). |
rt_min |
The starting runtime |
rt_max |
The final runtime |
rt_step |
The spacing between starting and final runtimes |
A radarplot of the area under the aggregated ECDF-curve of the DataSetList
Plot.FV.ECDF_AUC(subset(dsl, funcId == 1))Plot.FV.ECDF_AUC(subset(dsl, funcId == 1))
Plot the empirical cumulative distriburtion as a function of the target values of a DataSetList at certain target runtimes
Plot.FV.ECDF_Per_Target(dsList, runtimes, scale.xlog = F, scale.reverse = F) ## S3 method for class 'DataSetList' Plot.FV.ECDF_Per_Target(dsList, runtimes, scale.xlog = F, scale.reverse = F)Plot.FV.ECDF_Per_Target(dsList, runtimes, scale.xlog = F, scale.reverse = F) ## S3 method for class 'DataSetList' Plot.FV.ECDF_Per_Target(dsList, runtimes, scale.xlog = F, scale.reverse = F)
dsList |
A DataSetList (should consist of only one function and dimension). |
runtimes |
The target runtimes |
scale.xlog |
Whether or not to scale the x-axis logaritmically |
scale.reverse |
Whether or not to reverse the x-axis (when using minimization) |
A plot of the empirical cumulative distriburtion as a function of the fucntion values of the DataSetList at the target runtimes
Plot.FV.ECDF_Per_Target(subset(dsl, funcId == 1), 10)Plot.FV.ECDF_Per_Target(subset(dsl, funcId == 1), 10)
Plot the aggregated empirical cumulative distriburtion as a function of the function values of a DataSetList.
Plot.FV.ECDF_Single_Func( dsList, rt_min = NULL, rt_max = NULL, rt_step = NULL, scale.xlog = F, show.per_target = F, scale.reverse = F ) ## S3 method for class 'DataSetList' Plot.FV.ECDF_Single_Func( dsList, rt_min = NULL, rt_max = NULL, rt_step = NULL, scale.xlog = F, show.per_target = F, scale.reverse = F )Plot.FV.ECDF_Single_Func( dsList, rt_min = NULL, rt_max = NULL, rt_step = NULL, scale.xlog = F, show.per_target = F, scale.reverse = F ) ## S3 method for class 'DataSetList' Plot.FV.ECDF_Single_Func( dsList, rt_min = NULL, rt_max = NULL, rt_step = NULL, scale.xlog = F, show.per_target = F, scale.reverse = F )
dsList |
A DataSetList (should consist of only one function and dimension). |
rt_min |
The starting runtime |
rt_max |
The final runtime |
rt_step |
The spacing between starting and final runtimes |
scale.xlog |
Whether or not to scale the x-axis logaritmically |
show.per_target |
Whether or not to show the individual ECDF-curves for each runtime |
scale.reverse |
Whether or not to reverse the x-axis (when using minimization) |
A plot of the empirical cumulative distriburtion as a function of the function values of the DataSetList
Plot.FV.ECDF_Single_Func(subset(dsl, funcId == 1))Plot.FV.ECDF_Single_Func(subset(dsl, funcId == 1))
Plot histograms of the function values of a DataSetList at a certain target runtime
Plot.FV.Histogram(dsList, runtime, plot_mode = "overlay", use.equal.bins = F) ## S3 method for class 'DataSetList' Plot.FV.Histogram(dsList, runtime, plot_mode = "overlay", use.equal.bins = F)Plot.FV.Histogram(dsList, runtime, plot_mode = "overlay", use.equal.bins = F) ## S3 method for class 'DataSetList' Plot.FV.Histogram(dsList, runtime, plot_mode = "overlay", use.equal.bins = F)
dsList |
A DataSetList (should consist of only one function and dimension). |
runtime |
The target runtime |
plot_mode |
How to plot the different hisograms for each algorithm. Can be either 'overlay' to show all algorithms on one plot, or 'subplot' to have one plot per algorithm. |
use.equal.bins |
Whether to determine one bin size for all plots or have individual bin sizes for each algorithm |
A plot of the histograms of the function values at a the target runtime of the DataSetList
Plot.FV.Histogram(subset(dsl, funcId == 1), 100)Plot.FV.Histogram(subset(dsl, funcId == 1), 100)
Plot FV-plots for multiple functions or dimensions
Plot.FV.Multi_Func(dsList, scale.xlog = F, scale.ylog = F, backend = NULL) ## S3 method for class 'DataSetList' Plot.FV.Multi_Func(dsList, scale.xlog = F, scale.ylog = F, backend = NULL)Plot.FV.Multi_Func(dsList, scale.xlog = F, scale.ylog = F, backend = NULL) ## S3 method for class 'DataSetList' Plot.FV.Multi_Func(dsList, scale.xlog = F, scale.ylog = F, backend = NULL)
dsList |
A DataSetList (should consist of only one function OR dimension). |
scale.xlog |
Whether or not to scale the x-axis logaritmically |
scale.ylog |
Whether or not to scale the y-axis logaritmically |
backend |
Which plotting library to use. Either 'plotly' or 'ggplot2'. |
A plot of Function-values of the DataSetList
Plot.FV.Multi_Func(dsl)Plot.FV.Multi_Func(dsl)
Plot the parameter values recorded in a DataSetList (aligned by budget)
Plot.FV.Parameters( dsList, rt_min = NULL, rt_max = NULL, algids = "all", par_name = NULL, scale.xlog = F, scale.ylog = F, show.mean = T, show.median = F, show.CI = F ) ## S3 method for class 'DataSetList' Plot.FV.Parameters( dsList, rt_min = NULL, rt_max = NULL, algids = "all", par_name = NULL, scale.xlog = F, scale.ylog = F, show.mean = T, show.median = F, show.CI = F )Plot.FV.Parameters( dsList, rt_min = NULL, rt_max = NULL, algids = "all", par_name = NULL, scale.xlog = F, scale.ylog = F, show.mean = T, show.median = F, show.CI = F ) ## S3 method for class 'DataSetList' Plot.FV.Parameters( dsList, rt_min = NULL, rt_max = NULL, algids = "all", par_name = NULL, scale.xlog = F, scale.ylog = F, show.mean = T, show.median = F, show.CI = F )
dsList |
A DataSetList (should consist of only one function and dimension). |
rt_min |
The starting budget value. |
rt_max |
The final budget value. |
algids |
Which algorithms from dsList to use |
par_name |
Which parameters to create plots for; set to NULL to use all parameters found in dsList. |
scale.xlog |
Whether or not to scale the x-axis logaritmically |
scale.ylog |
Whether or not to scale the y-axis logaritmically |
show.mean |
Whether or not to show the mean parameter values |
show.median |
Whether or not to show the median parameter values |
show.CI |
Whether or not to show the standard deviation |
A plot of for every recorded parameter in the DataSetList
Plot.FV.Parameters(subset(dsl, funcId == 1))Plot.FV.Parameters(subset(dsl, funcId == 1))
Plot probablity density function of the function values of a DataSetList at a certain target runtime
Plot.FV.PDF(dsList, runtime, show.sample = F, scale.ylog = F) ## S3 method for class 'DataSetList' Plot.FV.PDF(dsList, runtime, show.sample = F, scale.ylog = F)Plot.FV.PDF(dsList, runtime, show.sample = F, scale.ylog = F) ## S3 method for class 'DataSetList' Plot.FV.PDF(dsList, runtime, show.sample = F, scale.ylog = F)
dsList |
A DataSetList (should consist of only one function and dimension). |
runtime |
The target runtime |
show.sample |
Whether or not to show the individual function value samples |
scale.ylog |
Whether or not to scale the y-axis logaritmically |
A plot of the probablity density function of the runtimes at a the target function value of the DataSetList
Plot.FV.PDF(subset(dsl, funcId == 1), 100)Plot.FV.PDF(subset(dsl, funcId == 1), 100)
Plot lineplot of the expected function values of a DataSetList
Plot.FV.Single_Func( dsList, RTstart = NULL, RTstop = NULL, show.CI = F, show.mean = T, show.median = F, backend = NULL, scale.xlog = F, scale.ylog = F, scale.reverse = F ) ## S3 method for class 'DataSetList' Plot.FV.Single_Func( dsList, RTstart = NULL, RTstop = NULL, show.CI = F, show.mean = T, show.median = F, backend = NULL, scale.xlog = F, scale.ylog = F, scale.reverse = F )Plot.FV.Single_Func( dsList, RTstart = NULL, RTstop = NULL, show.CI = F, show.mean = T, show.median = F, backend = NULL, scale.xlog = F, scale.ylog = F, scale.reverse = F ) ## S3 method for class 'DataSetList' Plot.FV.Single_Func( dsList, RTstart = NULL, RTstop = NULL, show.CI = F, show.mean = T, show.median = F, backend = NULL, scale.xlog = F, scale.ylog = F, scale.reverse = F )
dsList |
A DataSetList (should consist of only one function and dimension). |
RTstart |
The starting runtime value. |
RTstop |
The final runtime value. |
show.CI |
Whether or not to show the standard deviations |
show.mean |
Whether or not to show the mean runtimes |
show.median |
Whether or not to show the median runtimes |
backend |
Which plotting library to use. Can be 'plotly' or 'ggplot2' |
scale.xlog |
Whether or not to scale the x-axis logaritmically |
scale.ylog |
Whether or not to scale the y-axis logaritmically |
scale.reverse |
Wheter or not to reverse the x-axis (when using minimization) |
A plot of ERT-values of the DataSetList
Plot.FV.Single_Func(subset(dsl, funcId == 1))Plot.FV.Single_Func(subset(dsl, funcId == 1))
From the paper:
Plot.Performviz(DSC_rank_result)Plot.Performviz(DSC_rank_result)
DSC_rank_result |
The result from a call to DSCtool rank service ('get_dsc_rank') |
A performviz plot
## Not run: Plot.Performviz(get_dsc_rank(dsl)) ## End(Not run)## Not run: Plot.Performviz(get_dsc_rank(dsl)) ## End(Not run)
Plot ERT-based comparison over multiple functions or dimensions
Plot.RT.Aggregated( dsList, aggr_on = "funcId", targets = NULL, plot_mode = "radar", use_rank = F, scale.ylog = T, maximize = T, erts = NULL, inf.action = "overlap" ) ## S3 method for class 'DataSetList' Plot.RT.Aggregated( dsList, aggr_on = "funcId", targets = NULL, plot_mode = "radar", use_rank = F, scale.ylog = T, maximize = T, erts = NULL, inf.action = "overlap" )Plot.RT.Aggregated( dsList, aggr_on = "funcId", targets = NULL, plot_mode = "radar", use_rank = F, scale.ylog = T, maximize = T, erts = NULL, inf.action = "overlap" ) ## S3 method for class 'DataSetList' Plot.RT.Aggregated( dsList, aggr_on = "funcId", targets = NULL, plot_mode = "radar", use_rank = F, scale.ylog = T, maximize = T, erts = NULL, inf.action = "overlap" )
dsList |
A DataSetList (should consist of only one function OR dimension). |
aggr_on |
Whether to compare on functions ('funcId') or dimensions ('DIM') |
targets |
Custom list of function-value targets, one for each function or dimension. |
plot_mode |
How the plots should be created. Can be 'line' or 'radar' |
use_rank |
Wheter to use a ranking system. If False, the actual ERT-values will be used. |
scale.ylog |
Whether or not to scale the y-axis logaritmically |
maximize |
Wheter or not to the data is of a maximization problem |
erts |
Pre-calculated ERT-values for the provided targets. Created by the max_ERTs function of DataSetList. Can be provided to prevent needless computation in recalculating ERTs when recreating this plot. |
inf.action |
How to handle infinite ERTs ('overlap' or 'jitter') |
A plot of ERT-based comparison on the provided functions or dimensions of the DataSetList
Plot.RT.Aggregated(dsl)Plot.RT.Aggregated(dsl)
Radarplot of the area under the aggregated ECDF-curve of a DataSetList.
Plot.RT.ECDF_AUC( dsList, fstart = NULL, fstop = NULL, fstep = NULL, fval_formatter = as.integer ) ## S3 method for class 'DataSetList' Plot.RT.ECDF_AUC( dsList, fstart = NULL, fstop = NULL, fstep = NULL, fval_formatter = as.integer )Plot.RT.ECDF_AUC( dsList, fstart = NULL, fstop = NULL, fstep = NULL, fval_formatter = as.integer ) ## S3 method for class 'DataSetList' Plot.RT.ECDF_AUC( dsList, fstart = NULL, fstop = NULL, fstep = NULL, fval_formatter = as.integer )
dsList |
A DataSetList (should consist of only one function and dimension). |
fstart |
The starting function value |
fstop |
The final function value |
fstep |
The spacing between starting and final function values |
fval_formatter |
Function to format the function-value labels |
A radarplot of the area under the aggregated ECDF-curve of the DataSetList
Plot.RT.ECDF_AUC(subset(dsl, funcId == 1))Plot.RT.ECDF_AUC(subset(dsl, funcId == 1))
Plot the aggregated empirical cumulative distriburtion as a function of the running times of a DataSetList. Aggregated over multiple functions or dimensions.
Plot.RT.ECDF_Multi_Func(dsList, targets = NULL, scale.xlog = F) ## S3 method for class 'DataSetList' Plot.RT.ECDF_Multi_Func(dsList, targets = NULL, scale.xlog = F)Plot.RT.ECDF_Multi_Func(dsList, targets = NULL, scale.xlog = F) ## S3 method for class 'DataSetList' Plot.RT.ECDF_Multi_Func(dsList, targets = NULL, scale.xlog = F)
dsList |
A DataSetList. |
targets |
The target function values. Specified in a data.frame, as can be generated |
scale.xlog |
Whether or not to scale the x-axis logaritmically by the function 'get_ECDF_targets' |
A plot of the empirical cumulative distriburtion as a function of the running times of the DataSetList
Plot.RT.ECDF_Multi_Func(dsl)Plot.RT.ECDF_Multi_Func(dsl)
Plot the empirical cumulative distriburtion as a function of the running times of a DataSetList at certain target function values
Plot.RT.ECDF_Per_Target(dsList, ftargets, scale.xlog = F) ## S3 method for class 'DataSetList' Plot.RT.ECDF_Per_Target(dsList, ftargets, scale.xlog = F)Plot.RT.ECDF_Per_Target(dsList, ftargets, scale.xlog = F) ## S3 method for class 'DataSetList' Plot.RT.ECDF_Per_Target(dsList, ftargets, scale.xlog = F)
dsList |
A DataSetList (should consist of only one function and dimension). |
ftargets |
The target function values |
scale.xlog |
Whether or not to scale the x-axis logaritmically |
A plot of the empirical cumulative distriburtion as a function of the running times of the DataSetList at the target function values
Plot.RT.ECDF_Per_Target(subset(dsl, funcId == 1), 14)Plot.RT.ECDF_Per_Target(subset(dsl, funcId == 1), 14)
Plot the aggregated empirical cumulative distriburtion as a function of the running times of a DataSetList.
Plot.RT.ECDF_Single_Func( dsList, fstart = NULL, fstop = NULL, fstep = NULL, show.per_target = F, scale.xlog = F ) ## S3 method for class 'DataSetList' Plot.RT.ECDF_Single_Func( dsList, fstart = NULL, fstop = NULL, fstep = NULL, show.per_target = F, scale.xlog = F )Plot.RT.ECDF_Single_Func( dsList, fstart = NULL, fstop = NULL, fstep = NULL, show.per_target = F, scale.xlog = F ) ## S3 method for class 'DataSetList' Plot.RT.ECDF_Single_Func( dsList, fstart = NULL, fstop = NULL, fstep = NULL, show.per_target = F, scale.xlog = F )
dsList |
A DataSetList (should consist of only one function and dimension). |
fstart |
The starting function value |
fstop |
The final function value |
fstep |
The spacing between starting and final function values |
show.per_target |
Whether or not to show the individual ECDF-curves for each target |
scale.xlog |
Whether or not to scale the x-axis logaritmically |
A plot of the empirical cumulative distriburtion as a function of the running times of the DataSetList
Plot.RT.ECDF_Single_Func(subset(dsl, funcId == 1))Plot.RT.ECDF_Single_Func(subset(dsl, funcId == 1))
Plot histograms of the runtimes of a DataSetList at a certain target function value
Plot.RT.Histogram(dsList, ftarget, plot_mode = "overlay", use.equal.bins = F) ## S3 method for class 'DataSetList' Plot.RT.Histogram(dsList, ftarget, plot_mode = "overlay", use.equal.bins = F)Plot.RT.Histogram(dsList, ftarget, plot_mode = "overlay", use.equal.bins = F) ## S3 method for class 'DataSetList' Plot.RT.Histogram(dsList, ftarget, plot_mode = "overlay", use.equal.bins = F)
dsList |
A DataSetList (should consist of only one function and dimension). |
ftarget |
The target function value. |
plot_mode |
How to plot the different hisograms for each algorithm. Can be either 'overlay' to show all algorithms on one plot, or 'subplot' to have one plot per algorithm. |
use.equal.bins |
Whether to determine one bin size for all plots or have individual bin sizes for each algorithm |
A plot of the histograms of the runtimes at a the target function value of the DataSetList
Plot.RT.Histogram(subset(dsl, funcId == 1), 14)Plot.RT.Histogram(subset(dsl, funcId == 1), 14)
Plot ERT-plots for multiple functions or dimensions
Plot.RT.Multi_Func( dsList, scale.xlog = F, scale.ylog = F, scale.reverse = F, backend = NULL ) ## S3 method for class 'DataSetList' Plot.RT.Multi_Func( dsList, scale.xlog = F, scale.ylog = F, scale.reverse = F, backend = NULL )Plot.RT.Multi_Func( dsList, scale.xlog = F, scale.ylog = F, scale.reverse = F, backend = NULL ) ## S3 method for class 'DataSetList' Plot.RT.Multi_Func( dsList, scale.xlog = F, scale.ylog = F, scale.reverse = F, backend = NULL )
dsList |
A DataSetList (should consist of only one function OR dimension). |
scale.xlog |
Whether or not to scale the x-axis logaritmically |
scale.ylog |
Whether or not to scale the y-axis logaritmically |
scale.reverse |
Wheter or not to reverse the x-axis (when using minimization) |
backend |
Which plotting library to use. Either 'plotly' or 'ggplot2'. |
A plot of ERT-values of the DataSetList
Plot.RT.Multi_Func(dsl)Plot.RT.Multi_Func(dsl)
Plot the parameter values recorded in a DataSetList (aligned by funcion value)
Plot.RT.Parameters( dsList, f_min = NULL, f_max = NULL, algids = "all", par_name = NULL, scale.xlog = F, scale.ylog = F, show.mean = T, show.median = F, show.CI = F ) ## S3 method for class 'DataSetList' Plot.RT.Parameters( dsList, f_min = NULL, f_max = NULL, algids = "all", par_name = NULL, scale.xlog = F, scale.ylog = F, show.mean = T, show.median = F, show.CI = F )Plot.RT.Parameters( dsList, f_min = NULL, f_max = NULL, algids = "all", par_name = NULL, scale.xlog = F, scale.ylog = F, show.mean = T, show.median = F, show.CI = F ) ## S3 method for class 'DataSetList' Plot.RT.Parameters( dsList, f_min = NULL, f_max = NULL, algids = "all", par_name = NULL, scale.xlog = F, scale.ylog = F, show.mean = T, show.median = F, show.CI = F )
dsList |
A DataSetList (should consist of only one function and dimension). |
f_min |
The starting function value. |
f_max |
The final function value. |
algids |
Which algorithms from dsList to use |
par_name |
Which parameters to create plots for; set to NULL to use all parameters found in dsList. |
scale.xlog |
Whether or not to scale the x-axis logaritmically |
scale.ylog |
Whether or not to scale the y-axis logaritmically |
show.mean |
Whether or not to show the mean parameter values |
show.median |
Whether or not to show the median parameter values |
show.CI |
Whether or not to show the standard deviation |
A plot of for every recorded parameter in the DataSetList
Plot.RT.Parameters(subset(dsl, funcId == 1))Plot.RT.Parameters(subset(dsl, funcId == 1))
Plot probablity mass function of the runtimes of a DataSetList at a certain target function value
Plot.RT.PMF(dsList, ftarget, show.sample = F, scale.ylog = F, backend = NULL) ## S3 method for class 'DataSetList' Plot.RT.PMF(dsList, ftarget, show.sample = F, scale.ylog = F, backend = NULL)Plot.RT.PMF(dsList, ftarget, show.sample = F, scale.ylog = F, backend = NULL) ## S3 method for class 'DataSetList' Plot.RT.PMF(dsList, ftarget, show.sample = F, scale.ylog = F, backend = NULL)
dsList |
A DataSetList (should consist of only one function and dimension). |
ftarget |
The target function value. |
show.sample |
Whether or not to show the individual runtime samples |
scale.ylog |
Whether or not to scale the y-axis logaritmically |
backend |
Which plotting library to use. Can be 'plotly' or 'ggplot2' |
A plot of the probablity mass function of the runtimes at a the target function value of the DataSetList
Plot.RT.PMF(subset(dsl, funcId == 1), 14)Plot.RT.PMF(subset(dsl, funcId == 1), 14)
Plot lineplot of the ERTs of a DataSetList
Plot.RT.Single_Func( dsList, Fstart = NULL, Fstop = NULL, show.ERT = T, show.CI = F, show.mean = F, show.median = F, backend = NULL, scale.xlog = F, scale.ylog = F, scale.reverse = F, includeOpts = F, p = NULL ) ## S3 method for class 'DataSetList' Plot.RT.Single_Func( dsList, Fstart = NULL, Fstop = NULL, show.ERT = T, show.CI = T, show.mean = F, show.median = F, backend = NULL, scale.xlog = F, scale.ylog = F, scale.reverse = F, includeOpts = F, p = NULL )Plot.RT.Single_Func( dsList, Fstart = NULL, Fstop = NULL, show.ERT = T, show.CI = F, show.mean = F, show.median = F, backend = NULL, scale.xlog = F, scale.ylog = F, scale.reverse = F, includeOpts = F, p = NULL ) ## S3 method for class 'DataSetList' Plot.RT.Single_Func( dsList, Fstart = NULL, Fstop = NULL, show.ERT = T, show.CI = T, show.mean = F, show.median = F, backend = NULL, scale.xlog = F, scale.ylog = F, scale.reverse = F, includeOpts = F, p = NULL )
dsList |
A DataSetList (should consist of only one function and dimension). |
Fstart |
The starting function value. |
Fstop |
The final function value. |
show.ERT |
Whether or not to show the ERT-values |
show.CI |
Whether or not to show the standard deviations |
show.mean |
Whether or not to show the mean hitting times |
show.median |
Whether or not to show the median hitting times |
backend |
Which plotting library to use. Can be 'plotly' or 'ggplot2' |
scale.xlog |
Whether or not to scale the x-axis logaritmically |
scale.ylog |
Whether or not to scale the y-axis logaritmically |
scale.reverse |
Wheter or not to reverse the x-axis (when using minimization) |
includeOpts |
Whether or not to include all best points reached by each algorithm |
p |
Existing plot to which to add the current data |
A plot of ERT-values of the DataSetList
Plot.RT.Single_Func(subset(dsl, funcId == 1))Plot.RT.Single_Func(subset(dsl, funcId == 1))
Create a candlestick plot of Glicko2-rankings
Plot.Stats.Glicko2_Candlestick( dsList, nr_rounds = 100, glicko2_rank_df = NULL, which = "by_FV", target_dt = NULL ) ## S3 method for class 'DataSetList' Plot.Stats.Glicko2_Candlestick( dsList, nr_rounds = 100, glicko2_rank_df = NULL, which = "by_FV", target_dt = NULL )Plot.Stats.Glicko2_Candlestick( dsList, nr_rounds = 100, glicko2_rank_df = NULL, which = "by_FV", target_dt = NULL ) ## S3 method for class 'DataSetList' Plot.Stats.Glicko2_Candlestick( dsList, nr_rounds = 100, glicko2_rank_df = NULL, which = "by_FV", target_dt = NULL )
dsList |
A DataSetList |
nr_rounds |
The number of rounds in the tournament |
glicko2_rank_df |
Optional. Dataframe containing the glicko2 rating to avoid needless recalculation. |
which |
Whether to use fixed-target ('by_FV') or fixed-budget ('by_RT') perspective |
target_dt |
Optional: data table containing the targets for each function and dimension |
Plot.Stats.Glicko2_Candlestick(dsl, nr_rounds=2)Plot.Stats.Glicko2_Candlestick(dsl, nr_rounds=2)
Plot a network graph showing the statistically different algorithms
Plot.Stats.Significance_Graph( dsList, ftarget, alpha = 0.01, bootstrap.size = 30, which = "by_FV" ) ## S3 method for class 'DataSetList' Plot.Stats.Significance_Graph( dsList, ftarget, alpha = 0.01, bootstrap.size = 30, which = "by_FV" )Plot.Stats.Significance_Graph( dsList, ftarget, alpha = 0.01, bootstrap.size = 30, which = "by_FV" ) ## S3 method for class 'DataSetList' Plot.Stats.Significance_Graph( dsList, ftarget, alpha = 0.01, bootstrap.size = 30, which = "by_FV" )
dsList |
A DataSetList (should consist of only one function and dimension). |
ftarget |
The target function value to use |
alpha |
The cutoff for statistical significance |
bootstrap.size |
The amound of bootstrapped samples used |
which |
Whether to use fixed-target ('by_FV') or fixed-budget ('by_RT') perspective |
A graph showing the statistical significance between algorithms
Plot.Stats.Significance_Graph(subset(dsl, funcId == 2), 16)Plot.Stats.Significance_Graph(subset(dsl, funcId == 2), 16)
Plot a heatmap showing the statistically different algorithms
Plot.Stats.Significance_Heatmap( dsList, ftarget, alpha = 0.01, bootstrap.size = 30, which = "by_FV" ) ## S3 method for class 'DataSetList' Plot.Stats.Significance_Heatmap( dsList, ftarget, alpha = 0.01, bootstrap.size = 30, which = "by_FV" )Plot.Stats.Significance_Heatmap( dsList, ftarget, alpha = 0.01, bootstrap.size = 30, which = "by_FV" ) ## S3 method for class 'DataSetList' Plot.Stats.Significance_Heatmap( dsList, ftarget, alpha = 0.01, bootstrap.size = 30, which = "by_FV" )
dsList |
A DataSetList (should consist of only one function and dimension). |
ftarget |
The target function value to use |
alpha |
The cutoff for statistical significance |
bootstrap.size |
The amound of bootstrapped samples used |
which |
Whether to use fixed-target ('by_FV') or fixed-budget ('by_RT') perspective |
A heatmap showing the statistical significance between algorithms
Plot.Stats.Significance_Heatmap(subset(dsl, funcId == 2), 16)Plot.Stats.Significance_Heatmap(subset(dsl, funcId == 2), 16)
S3 generic print operator for DataSet
## S3 method for class 'DataSet' print(x, ...)## S3 method for class 'DataSet' print(x, ...)
x |
A DataSet object |
... |
Arguments passed to other methods |
A short description of the DataSet
print(dsl[[1]])print(dsl[[1]])
S3 print function for DataSetList
## S3 method for class 'DataSetList' print(x, ...)## S3 method for class 'DataSetList' print(x, ...)
x |
The DataSetList to print |
... |
Arguments for underlying print function? |
print(dsl)print(dsl)
Read .info files and extract information
read_index_file(fname)read_index_file(fname)
fname |
The path to the .info file |
The data contained in the .info file
path <- system.file("extdata", "ONE_PLUS_LAMDA_EA", package="IOHanalyzer") info <- read_index_file(file.path(path,"IOHprofiler_f1_i1.info"))path <- system.file("extdata", "ONE_PLUS_LAMDA_EA", package="IOHanalyzer") info <- read_index_file(file.path(path,"IOHprofiler_f1_i1.info"))
Read .csv files in arbitrary format
read_IOH_v1plus(info, full_sampling = FALSE)read_IOH_v1plus(info, full_sampling = FALSE)
info |
A List containing all meta-data about the dataset to create |
full_sampling |
Logical. Whether the raw (unaligned) FV matrix should be stored. Currently only useful when a correlation plot between function values and parameters should be made |
The DataSetList extracted from the .csv file provided
Read .csv files in arbitrary format
read_pure_csv( path, neval_name, fval_name, fname_name, algname_name, dim_name, run_name, maximization = F, static_attrs = NULL )read_pure_csv( path, neval_name, fval_name, fname_name, algname_name, dim_name, run_name, maximization = F, static_attrs = NULL )
path |
The path to the .csv file |
neval_name |
The name of the column to use for the evaluation count. If NULL, will be assumed to be sequential |
fval_name |
The name of the column to use for the function values |
fname_name |
The name of the column to use for the function name |
algname_name |
The name of the column to use for the algorithm name |
dim_name |
The name of the column to use for the dimension |
run_name |
The name of the column to use for the run number |
maximization |
Boolean indicating whether the data is resulting from maximization or minimization |
static_attrs |
Named list containing the static values for missing columns. When a parameter is not present in the csv file, its name-parameter should be set to NULL, and the static value should be added to this static_attrs list. |
The DataSetList extracted from the .csv file provided
This uses the keyring package to store and load credentials. If you already have an account, please call 'set_DSC_credentials' instead
register_DSC(name, username, affiliation, email, password = NULL)register_DSC(name, username, affiliation, email, password = NULL)
name |
Your name |
username |
A usename to be identified with. Will be stored on keyring under 'DSCtool_name' |
affiliation |
Your affiliation (university / company) |
email |
Your email adress |
password |
The password to use. If NULL, this will be generated at random. Will be stored on keyring under 'DSCtool' |
## Not run: register_DSC('John Doe', 'jdoe', 'Sample University', "j.doe.sample.com") ## End(Not run)## Not run: register_DSC('John Doe', 'jdoe', 'Sample University', "j.doe.sample.com") ## End(Not run)
Create a shiny-server GUI to interactively use the IOHanalyzer
runServer(port = getOption("shiny.port"), open_browser = TRUE, orca_gpu = TRUE)runServer(port = getOption("shiny.port"), open_browser = TRUE, orca_gpu = TRUE)
port |
Optional; which port the server should be opened at. Defaults to the option set for 'shiny.port' |
open_browser |
Whether or not to open a browser tab with the IOHanalyzer GUI. Defaults to TRUE. |
orca_gpu |
Whether or not orca will be allowed to use gpu-accelleration for saving figures to file. |
## Not run: runServer(6563, TRUE) ## End(Not run)## Not run: runServer(6563, TRUE) ## End(Not run)
NOTE: This function requires orca to be installed
save_plotly(p, file, width = NULL, height = NULL, ...)save_plotly(p, file, width = NULL, height = NULL, ...)
p |
plotly object. The plot to be saved |
file |
String. The name of the figure file, with the extension of the required file-format |
width |
Optional. Width of the figure |
height |
Optional. Height of the figure |
... |
Additional arguments for orca |
## Not run: p <- Plot.RT.Single_Func(dsl[1]) save_plotly(p, 'example_file.png') ## End(Not run)## Not run: p <- Plot.RT.Single_Func(dsl[1]) save_plotly(p, 'example_file.png') ## End(Not run)
Save DataTable in multiple formats
save_table(df, file, format = NULL)save_table(df, file, format = NULL)
df |
The DataTable to store |
file |
String. The name of the figure file, with the extension of the required file-format |
format |
Optional, string. Overwrites the extension of the 'file' parameter. If not specified while file does not have an extension, it defaults to csv |
df <- generate_data.Single_Function(subset(dsl, funcId == 1), which = 'by_RT') save_table(df, tempfile(fileext = ".md"))df <- generate_data.Single_Function(subset(dsl, funcId == 1), which = 'by_RT') save_table(df, tempfile(fileext = ".md"))
Scan *.info files for IOHProfiler or COCO
scan_index_file(folder)scan_index_file(folder)
folder |
The folder containing the .info or .json files |
The paths to all found .info and .json-files
This automatically filetrs our files of size 0
path <- system.file("extdata", "ONE_PLUS_LAMDA_EA", package="IOHanalyzer") scan_index_file(path)path <- system.file("extdata", "ONE_PLUS_LAMDA_EA", package="IOHanalyzer") scan_index_file(path)
Function for generating sequences of function values
seq_FV( FV, from = NULL, to = NULL, by = NULL, length.out = NULL, scale = NULL, force_limits = FALSE )seq_FV( FV, from = NULL, to = NULL, by = NULL, length.out = NULL, scale = NULL, force_limits = FALSE )
FV |
A list of function values |
from |
Starting function value. Will be replaced by min(FV) if it is NULL or too small |
to |
Stopping function value. Will be replaced by max(FV) if it is NULL or too large |
by |
Stepsize of the sequence. Will be replaced if it is too small |
length.out |
Number of values in the sequence. 'by' takes preference if both it and length.out are provided. |
scale |
Scaling of the sequence. Can be either 'linear' or 'log', indicating a linear or log-linear spacing respectively. If NULL, the scale will be predicted based on FV |
force_limits |
Whether the from and to values are hard, or should be modified based on detected FV values (default False) |
A sequence of function values
FVall <- get_runtimes(dsl) seq_FV(FVall, 10, 16, 1, scale='linear')FVall <- get_runtimes(dsl) seq_FV(FVall, 10, 16, 1, scale='linear')
Function for generating sequences of runtime values
seq_RT( RT, from = NULL, to = NULL, by = NULL, length.out = NULL, scale = "linear" )seq_RT( RT, from = NULL, to = NULL, by = NULL, length.out = NULL, scale = "linear" )
RT |
A list of runtime values |
from |
Starting runtime value. Will be replaced by min(RT) if it is NULL or too small |
to |
Stopping runtime value. Will be replaced by max(RT) if it is NULL or too large |
by |
Stepsize of the sequence. Will be replaced if it is too small |
length.out |
Number of values in the sequence. 'by' takes preference if both it and length.out are provided. |
scale |
Scaling of the sequence. Can be either 'linear' or 'log', indicating a linear or log-linear spacing respectively. |
A sequence of runtime values
RTall <- get_runtimes(dsl) seq_RT(RTall, 0, 500, length.out=10, scale='log')RTall <- get_runtimes(dsl) seq_RT(RTall, 0, 500, length.out=10, scale='log')
Set the colorScheme of the IOHanalyzer plots
set_color_scheme(schemename, ids, path = NULL)set_color_scheme(schemename, ids, path = NULL)
schemename |
Three default colorschemes are implemented:
And it is also possible to select "Custom", which allows uploading of a custom set of colors |
ids |
The names of the algorithms (or custom ids, see 'change_id') for which to set the colors |
path |
The path to the file containing the colors to use. Only used if schemename is "Custom" |
set_color_scheme("Default", get_algId(dsl))set_color_scheme("Default", get_algId(dsl))
This uses the keyring package to store and load credentials. If you already have an account, please call 'add_DSC_credentials' instead
set_DSC_credentials(username, password)set_DSC_credentials(username, password)
username |
The usename you use on DSCtool. Will be stored on keyring under 'DSCtool_name' |
password |
The password you use on DSCtool. Will be stored on keyring under 'DSCtool' |
## Not run: set_DSC_credentials('jdoe', 'monkey123')## Not run: set_DSC_credentials('jdoe', 'monkey123')
Estimator 'SP' for the Expected Running Time (ERT)
SP(data, max_runtime)SP(data, max_runtime)
data |
A dataframe or matrix. Each row stores the runtime sample points from several runs |
max_runtime |
The budget to use for calculating ERT. If this is a vector, the largest value is taken. Using this as a vector is being deprecated, and will be removed in a future update |
A list containing ERTs, number of succesfull runs and the succes rate
SP(dsl[[1]]$RT, max(dsl[[1]]$RT))SP(dsl[[1]]$RT, max(dsl[[1]]$RT))
Subset for DataSets. Based on the provided mask, the relevant data is taken from the given DataSet and turned into a new DataSet object.
## S3 method for class 'DataSet' subset(x, mask, ...)## S3 method for class 'DataSet' subset(x, mask, ...)
x |
The DataSet from which to get a subset |
mask |
The mask (as boolean list) to use when subsetting. The length should be equal to the number of runs present in the provided dataset object x. |
... |
Arguments passed to underlying subset method (not yet supported) |
A new DataSet
subset(dsl[[1]], c(0,1,1,1,0,0,0,0,0,0,0))subset(dsl[[1]], c(0,1,1,1,0,0,0,0,0,0,0))
Filter a DataSetList by some criteria
## S3 method for class 'DataSetList' subset(x, ...)## S3 method for class 'DataSetList' subset(x, ...)
x |
The DataSetList |
... |
The conditions to filter on. Can be any expression which assigns True or False to a DataSet object, such as DIM == 625 or funcId == 2. Usage of && and || is only supported on default attributes (funcId, algId, DIM), not on combinations of with other attributes (e.g. instance). In those cases, & and | should be used respectively. Alternatively, this can be used as a keyword argument named 'text', with the condition as a string to be parsed. This allows exectution of subset commands on arbitrary variables in code. |
The filtered DataSetList
subset(dsl, funcId == 1) subset(dsl, funcId == 1 && DIM == 16) # Can use && and || for default attributes subset(dsl, instance == 1) subset(dsl, instance == 1 & funcId == 1) # Can use & and | for all attributes subset(dsl, instance == 1, funcId == 1) # Comma-seperated conditions are treated as ANDsubset(dsl, funcId == 1) subset(dsl, funcId == 1 && DIM == 16) # Can use && and || for default attributes subset(dsl, instance == 1) subset(dsl, instance == 1 & funcId == 1) # Can use & and | for all attributes subset(dsl, instance == 1, funcId == 1) # Comma-seperated conditions are treated as AND
S3 generic summary operator for DataSet
## S3 method for class 'DataSet' summary(object, ...)## S3 method for class 'DataSet' summary(object, ...)
object |
A DataSet object |
... |
Arguments passed to other methods |
A summary of the DataSet containing both function-value and runtime based statistics.
summary(dsl[[1]])summary(dsl[[1]])
Prints the Function ID, Dimension, Algorithm Id, datafile location and comment for every DataSet in the DataSetList
## S3 method for class 'DataSetList' summary(object, ...)## S3 method for class 'DataSetList' summary(object, ...)
object |
The DataSetList to print |
... |
Arguments for underlying summary function? |
summary(dsl)summary(dsl)