Package: IOHanalyzer 0.1.8.11

Diederick Vermetten

IOHanalyzer: Data Analysis Part of 'IOHprofiler'

The data analysis module for the Iterative Optimization Heuristics Profiler ('IOHprofiler'). This module provides statistical analysis methods for the benchmark data generated by optimization heuristics, which can be visualized through a web-based interface. The benchmark data is usually generated by the experimentation module, called 'IOHexperimenter'. 'IOHanalyzer' also supports the widely used 'COCO' (Comparing Continuous Optimisers) data format for benchmarking.

Authors:Hao Wang [aut], Diederick Vermetten [cre, aut], Carola Doerr [aut], Thomas Bäck [aut]

IOHanalyzer_0.1.8.11.tar.gz
IOHanalyzer_0.1.8.11.zip(r-4.5)IOHanalyzer_0.1.8.11.zip(r-4.4)IOHanalyzer_0.1.8.11.zip(r-4.3)
IOHanalyzer_0.1.8.11.tgz(r-4.4-x86_64)IOHanalyzer_0.1.8.11.tgz(r-4.4-arm64)IOHanalyzer_0.1.8.11.tgz(r-4.3-x86_64)IOHanalyzer_0.1.8.11.tgz(r-4.3-arm64)
IOHanalyzer_0.1.8.11.tar.gz(r-4.5-noble)IOHanalyzer_0.1.8.11.tar.gz(r-4.4-noble)
IOHanalyzer_0.1.8.11.tgz(r-4.4-emscripten)IOHanalyzer_0.1.8.11.tgz(r-4.3-emscripten)
IOHanalyzer.pdf |IOHanalyzer.html✨
IOHanalyzer/json (API)

# Install 'IOHanalyzer' in R:
install.packages('IOHanalyzer', repos = c('https://iohprofiler.r-universe.dev', 'https://cloud.r-project.org'))

Peer review:

Bug tracker:https://github.com/iohprofiler/iohanalyzer/issues

Uses libs:
  • c++– GNU Standard C++ Library v3
Datasets:
  • dsl - Example DataSetList used in tests / examples
  • dsl_large - Larger example DataSetList used in tests / examples

On CRAN:

5.37 score 15 stars 13 scripts 460 downloads 111 exports 86 dependencies

Last updated 6 months agofrom:c749d9a48a. Checks:OK: 3 NOTE: 6. Indexed: yes.

TargetResultDate
Doc / VignettesOKNov 12 2024
R-4.5-win-x86_64OKNov 12 2024
R-4.5-linux-x86_64OKNov 12 2024
R-4.4-win-x86_64NOTENov 12 2024
R-4.4-mac-x86_64NOTENov 12 2024
R-4.4-mac-aarch64NOTENov 12 2024
R-4.3-win-x86_64NOTENov 12 2024
R-4.3-mac-x86_64NOTENov 12 2024
R-4.3-mac-aarch64NOTENov 12 2024

Exports:arrangeAUCbootstrap_RTcat.DataSetchange_idcheck_dsc_configuredcheck_formatclean_DataSetListDataSetDataSetListECDFfast_RT_samplesgenerate_data.Aggrgenerate_data.AUCgenerate_data.CDPgenerate_data.EAFgenerate_data.EAF_diff_Approximategenerate_data.EAF_Differencegenerate_data.ECDFgenerate_data.ECDF_From_EAFgenerate_data.ECDF_rawgenerate_data.Heatmapsgenerate_data.histgenerate_data.Parameter_correlationgenerate_data.Parametersgenerate_data.PMFgenerate_data.Single_Functionget_algIdget_color_schemeget_color_scheme_dtget_default_ECDF_targetsget_dimget_dsc_omnibusget_dsc_posthocget_dsc_rankget_ECDF_targetsget_ERTget_funcIdget_funcNameget_funvalsget_FVget_FV_overviewget_FV_sampleget_FV_summaryget_idget_line_styleget_marg_contrib_ecdfget_maxRTget_ontology_dataget_ontology_varget_overviewget_PAR_nameget_PAR_sampleget_PAR_summaryget_parIdget_position_dslget_RTget_RT_overviewget_RT_sampleget_RT_summaryget_runtimesget_shapley_valuesget_static_attribute_valuesget_static_attributesget_target_dtglicko2_rankingIOH_plot_ly_defaultmax_ERTsmean_FVspairwise.testplot_eaf_dataplot_eaf_differencesplot_general_dataPlot.Comparison.HeatmapPlot.cumulative_difference_plotPlot.FV.AggregatedPlot.FV.ECDF_AUCPlot.FV.ECDF_Per_TargetPlot.FV.ECDF_Single_FuncPlot.FV.HistogramPlot.FV.Multi_FuncPlot.FV.ParametersPlot.FV.PDFPlot.FV.Single_FuncPlot.PerformvizPlot.RT.AggregatedPlot.RT.ECDF_AUCPlot.RT.ECDF_Multi_FuncPlot.RT.ECDF_Per_TargetPlot.RT.ECDF_Single_FuncPlot.RT.HistogramPlot.RT.Multi_FuncPlot.RT.ParametersPlot.RT.PMFPlot.RT.Single_FuncPlot.Stats.Glicko2_CandlestickPlot.Stats.Significance_GraphPlot.Stats.Significance_Heatmapread_index_fileread_IOH_v1plusread_pure_csvregister_DSCrunServersave_plotlysave_tablescan_index_fileseq_FVseq_RTset_color_schemeset_DSC_credentialsSP

Dependencies:askpassbase64encbslibcachemclicolorspacecommonmarkcpp11crayoncrosstalkcurldata.tabledigestdplyreafevaluatefansifarverfastmapfontawesomefsgenericsggplot2gluegridExtragtablehighrhtmltoolshtmlwidgetshttpuvhttrisobandjquerylibjsonliteknitrlabelinglaterlatticelazyevallifecyclemagrittrMASSMatrixmatrixStatsmemoisemgcvmimemodeltoolsmunsellnlmeopensslpillarpkgconfigplotlyplyrpromisespurrrR6rappdirsrbibutilsRColorBrewerRcppRdpackreshape2rjsonrlangrmarkdownsassscalesshinysourcetoolsstringistringrsystibbletidyrtidyselecttinytexutf8vctrsviridisviridisLitewithrxfunxtableyaml

Readme and manuals

Help Manual

Help pageTopics
S3 extraction function for DataSetList[.DataSetList
S3 generic == operator for DataSets==.DataSet
S3 sort function for DataSetListarrange arrange.DataSetList
S3 generic as.character operator for DataSetas.character.DataSet
Area Under Curve (Empirical Cumulative Dsitribution Function)AUC AUC.ECDF
Bootstrapping for running time samplesbootstrap_RT
S3 concatenation function for DataSetc.DataSet
S3 concatenation function for DataSetListc.DataSetList
S3 generic cat operator for DataSetcat.DataSet
Add unique identifiers to each DataSet in the provided DataSetList based on static attributeschange_id
Verify that the credentials for DSCtool have been setcheck_dsc_configured
Check the format of datacheck_format
Clean DataSetList object by concatenating DataSetsclean_DataSetList
Constructor of S3 class 'DataSet'DataSet
S3 constructor of the 'DataSetList'DataSetList
Example DataSetList used in tests / examplesdsl
Larger example DataSetList used in tests / examplesdsl_large
Empirical Cumulative Dsitribution Function of Runtime of a single data setECDF ECDF.DataSet ECDF.DataSetList
Function to get just the RT samples needed, without any formatting to improve speedfast_RT_samples
Generate dataframe of a single function/dimension pairgenerate_data.Aggr
Generate dataframe containing the AUC for any ECDF-curvesgenerate_data.AUC
Generate data for the cumulative difference plot.generate_data.CDP
Generate dataframe consisting of the levelsets of the EAFgenerate_data.EAF
Generate EAF-differences between each function and the remaining portfoliogenerate_data.EAF_diff_Approximate
Generate differences between two EAFsgenerate_data.EAF_Difference
Generate dataframe of a single function/dimension pairgenerate_data.ECDF
Generate dataframe consisting of the ECDF-equivalent based on the EAFgenerate_data.ECDF_From_EAF
Generate dataframe of a the unaggregated values of individual algorithms. Stripped-down version ofgenerate_data.ECDF_raw
Nevergrad-dashboard based algorithm comparisongenerate_data.Heatmaps
Generate dataframe of a single function/dimension pairgenerate_data.hist
Generate dataframe of exactly 2 parameters, matched by running timegenerate_data.Parameter_correlation
Generate dataframe of a single function/dimension pairgenerate_data.Parameters
Generate dataframe of a single function/dimension pair for creating PDF or PMF plotsgenerate_data.PMF
Generate dataframe of a single function/dimension pairgenerate_data.Single_Function
Get all algorithm ids present in a DataSetListget_algId
Get colors according to the current colorScheme of the IOHanalyzerget_color_scheme
Get datatable of current color (and linestyle) scheme to fileget_color_scheme_dt
Generate ECDF targets for a DataSetListget_default_ECDF_targets
Get all dimensions present in a DataSetListget_dim
Perform omnibus statistical tests on the matrix of rankings from the DSCtool apiget_dsc_omnibus
Perform post-hoc processing on data from DSCtoolget_dsc_posthoc
Get the matrix of rankings using the DSCtool api for a DataSetListget_dsc_rank
Generation of default ECDF-targetsget_ECDF_targets
Get Expected RunTimeget_ERT get_ERT.DataSet get_ERT.DataSetList
Get all function ids present in a DataSetListget_funcId
Get all function names present in a DataSetListget_funcName
Get all function values present in a DataSetListget_funvals
Get function value matrix of the used dataset.get_FV get_FV.DataSet
Get Function Value condensed overviewget_FV_overview get_FV_overview.DataSet get_FV_overview.DataSetList
Get Funtion Value Samplesget_FV_sample get_FV_sample.DataSet get_FV_sample.DataSetList
Get Function Value Summaryget_FV_summary get_FV_summary.DataSet get_FV_summary.DataSetList
Get condensed overview of datasetsget_id get_id.DataSet get_id.DataSetList
Get line styles according to the current styleScheme of the IOHanalyzerget_line_style
Get the marginal contribution of an algorithm to a portfolioget_marg_contrib_ecdf
Get the maximal running timeget_maxRT get_maxRT.DataSet get_maxRT.DataSetList
Get the list of available options for data from the OPTION ontologyget_ontology_data
Get the list of available options for data from the OPTION ontologyget_ontology_var
Get condensed overview of datasetsget_overview get_overview.DataSet get_overview.DataSetList
Get the parameter names of the algorithmget_PAR_name get_PAR_name.DataSet
Get Parameter Value Samplesget_PAR_sample get_PAR_sample.DataSet get_PAR_sample.DataSetList
Get Parameter Value Summaryget_PAR_summary get_PAR_summary.DataSet get_PAR_summary.DataSetList
Get all parameter ids present in a DataSetListget_parId
Extract the position information from a datasetlist objectget_position_dsl
Get runtime matrix of the used dataset.get_RT get_RT.DataSet
Get Runtime Value condensed overviewget_RT_overview get_RT_overview.DataSet get_RT_overview.DataSetList
Get RunTime Sampleget_RT_sample get_RT_sample.DataSet get_RT_sample.DataSetList
Get RunTime Summaryget_RT_summary get_RT_summary.DataSet get_RT_summary.DataSetList
Get all runtime values present in a DataSetListget_runtimes
Get the shapley-values of a portfolio of algorithmsget_shapley_values
Get all options for a specific attribute which can be used to subset a DataSetListget_static_attribute_values
Get all attributes which can be used to subset a DataSetListget_static_attributes
Generate datatables of runtime or function value targets for a DataSetListget_target_dt
Glicko2 raning of algorithmsglicko2_ranking
Template for creating plots in the IOHanalyzer-styleIOH_plot_ly_default
IOHanalyzer: Data Analysis Part of IOHprofilerIOHanalyzer-package IOHanalyzer
Reduce the size of the data set by evenly subsampling the recordslimit.data
Get the ERT-values for all DataSets in a DataSetList at certain targetsmax_ERTs max_ERTs.DataSetList
Get the expected function-values for all DataSets in a DataSetList at certain runtimesmean_FVs mean_FVs.DataSetList
Performs a pairwise Kolmogorov-Smirnov test on the bootstrapped running times among a data setpairwise.test pairwise.test.DataSetList pairwise.test.list
Create EAF-based polygon plotsplot_eaf_data
Create EAF-difference contour plotsplot_eaf_differences
General function for plotting within IOHanalyzerplot_general_data
Plot a heatmap according to the specifications from the Nevergrad dashboardPlot.Comparison.Heatmap Plot.Comparison.Heatmap.DataSetList
Plot the cumulative difference plot given a DataSetList.Plot.cumulative_difference_plot
Plot expected function value-based comparison over multiple functions or dimensionsPlot.FV.Aggregated Plot.FV.Aggregated.DataSetList
Radarplot of the area under the aggregated ECDF-curve of a DataSetList.Plot.FV.ECDF_AUC Plot.FV.ECDF_AUC.DataSetList
Plot the empirical cumulative distriburtion as a function of the target values of a DataSetList at certain target runtimesPlot.FV.ECDF_Per_Target Plot.FV.ECDF_Per_Target.DataSetList
Plot the aggregated empirical cumulative distriburtion as a function of the function values of a DataSetList.Plot.FV.ECDF_Single_Func Plot.FV.ECDF_Single_Func.DataSetList
Plot histograms of the function values of a DataSetList at a certain target runtimePlot.FV.Histogram Plot.FV.Histogram.DataSetList
Plot FV-plots for multiple functions or dimensionsPlot.FV.Multi_Func Plot.FV.Multi_Func.DataSetList
Plot the parameter values recorded in a DataSetList (aligned by budget)Plot.FV.Parameters Plot.FV.Parameters.DataSetList
Plot probablity density function of the function values of a DataSetList at a certain target runtimePlot.FV.PDF Plot.FV.PDF.DataSetList
Plot lineplot of the expected function values of a DataSetListPlot.FV.Single_Func Plot.FV.Single_Func.DataSetList
Create the PerformViz plotPlot.Performviz
Plot ERT-based comparison over multiple functions or dimensionsPlot.RT.Aggregated Plot.RT.Aggregated.DataSetList
Radarplot of the area under the aggregated ECDF-curve of a DataSetList.Plot.RT.ECDF_AUC Plot.RT.ECDF_AUC.DataSetList
Plot the aggregated empirical cumulative distriburtion as a function of the running times of a DataSetList. Aggregated over multiple functions or dimensions.Plot.RT.ECDF_Multi_Func Plot.RT.ECDF_Multi_Func.DataSetList
Plot the empirical cumulative distriburtion as a function of the running times of a DataSetList at certain target function valuesPlot.RT.ECDF_Per_Target Plot.RT.ECDF_Per_Target.DataSetList
Plot the aggregated empirical cumulative distriburtion as a function of the running times of a DataSetList.Plot.RT.ECDF_Single_Func Plot.RT.ECDF_Single_Func.DataSetList
Plot histograms of the runtimes of a DataSetList at a certain target function valuePlot.RT.Histogram Plot.RT.Histogram.DataSetList
Plot ERT-plots for multiple functions or dimensionsPlot.RT.Multi_Func Plot.RT.Multi_Func.DataSetList
Plot the parameter values recorded in a DataSetList (aligned by funcion value)Plot.RT.Parameters Plot.RT.Parameters.DataSetList
Plot probablity mass function of the runtimes of a DataSetList at a certain target function valuePlot.RT.PMF Plot.RT.PMF.DataSetList
Plot lineplot of the ERTs of a DataSetListPlot.RT.Single_Func Plot.RT.Single_Func.DataSetList
Create a candlestick plot of Glicko2-rankingsPlot.Stats.Glicko2_Candlestick Plot.Stats.Glicko2_Candlestick.DataSetList
Plot a network graph showing the statistically different algorithmsPlot.Stats.Significance_Graph Plot.Stats.Significance_Graph.DataSetList
Plot a heatmap showing the statistically different algorithmsPlot.Stats.Significance_Heatmap Plot.Stats.Significance_Heatmap.DataSetList
S3 generic print operator for DataSetprint.DataSet
S3 print function for DataSetListprint.DataSetList
Read .info files and extract informationread_index_file
Read Nevergrad dataread_IOH_v1plus
Read Nevergrad dataread_pure_csv
Register an account to the DSCtool APIregister_DSC
Create a shiny-server GUI to interactively use the IOHanalyzerrunServer
Save plotly figure in multiple formatsave_plotly
Save DataTable in multiple formatssave_table
Scan *.info files for IOHProfiler or COCOscan_index_file
Function for generating sequences of function valuesseq_FV
Function for generating sequences of runtime valuesseq_RT
Set the colorScheme of the IOHanalyzer plotsset_color_scheme
Register an account to the DSCtool APIset_DSC_credentials
Estimator 'SP' for the Expected Running Time (ERT)SP
S3 subset function for DataSetsubset.DataSet
Filter a DataSetList by some criteriasubset.DataSetList
S3 generic summary operator for DataSetsummary.DataSet
S3 summary function for DataSetListsummary.DataSetList