Permutation-Based Testing
Source:vignettes/permutation_based_testing.Rmd
permutation_based_testing.RmdBy default, CaDrA performs both forward and backward
search algorithm to look for a subset of features whose union is
maximally associated with an outcome of interest, based on (currently)
one of four scoring functions (Kolmogorov-Smirnov,
Conditional Mutual Information,
Wilcoxon, and custom-defined). To test
whether the strength of the association between the set of features and
the observed input scores (e.g., pathway activity, drug sensitivity,
etc.) is greater than it would be expected by chance,
CaDrA supports permutation-based significance testing.
Importantly, the permutation test iterates over the entire search
procedure (e.g., if top_N = 7, each permutation iteration
will consist of running the search over the top 7 features).
Load required datasets
- A
binary features matrixalso known asFeature Set(such as somatic mutations, copy number alterations, chromosomal translocations, etc.) The 1/0 row vectors indicate the presence/absence of ‘omics’ features in the samples. TheFeature Setcan be a matrix or an object of class SummarizedExperiment from SummarizedExperiment package) - A vector of continuous scores (or
Input Scores) representing a functional response of interest (such as protein expression, pathway activity, etc.)
Find a subset of features that maximally associated with a given outcome of interest
Here we are using Kolmogorow-Smirnow (KS) scoring method to search for best features
candidate_search_res <- CaDrA::candidate_search(
FS = sim_FS,
input_score = sim_Scores,
method = "ks_pval", # Use Kolmogorow-Smirnow scoring function
method_alternative = "less", # Use one-sided hypothesis testing
weights = NULL, # If weights is provided, perform a weighted-KS test
search_method = "both", # Apply both forward and backward search
top_N = 7, # Number of top features to kick start the search
max_size = 10, # Allow at most 10 features in meta-feature matrix
best_score_only = FALSE # Return all results from the search
)Visualize best meta-features result
# Extract the best meta-feature result
topn_best_meta <- CaDrA::topn_best(topn_list = candidate_search_res)
# Visualize meta-feature result
CaDrA::meta_plot(topn_best_list = topn_best_meta)
Perform permutation-based testing
# Set seed for permutation-based testing
set.seed(123)
perm_res <- CaDrA::CaDrA(
FS = sim_FS,
input_score = sim_Scores,
method = "ks_pval", # Use Kolmogorow-Smirnow scoring function
method_alternative = "less", # Use one-sided hypothesis testing
weights = NULL, # If weights is provided, perform a weighted-KS test
search_method = "both", # Apply both forward and backward search
top_N = 7, # Repeat the search with the top N features
max_size = 10, # Allow at most 10 features in the meta-feature matrix
n_perm = 100, # Number of permutations to perform
perm_alternative = "one.sided", # One-sided permutation-based p-value alternative type
plot = FALSE, # We will plot later
ncores = 2 # Number of cores to perform parallelization
)
SessionInfo
sessionInfo()
R version 4.4.1 (2024-06-14)
Platform: x86_64-pc-linux-gnu
Running under: Ubuntu 22.04.4 LTS
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so; LAPACK version 3.10.0
locale:
[1] LC_CTYPE=C.UTF-8 LC_NUMERIC=C LC_TIME=C.UTF-8
[4] LC_COLLATE=C.UTF-8 LC_MONETARY=C.UTF-8 LC_MESSAGES=C.UTF-8
[7] LC_PAPER=C.UTF-8 LC_NAME=C LC_ADDRESS=C
[10] LC_TELEPHONE=C LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C
time zone: UTC
tzcode source: system (glibc)
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] CaDrA_1.0.2 testthat_3.2.1.1 devtools_2.4.5 usethis_3.0.0
loaded via a namespace (and not attached):
[1] bitops_1.0-8 tcltk_4.4.1
[3] remotes_2.5.0 rlang_1.1.4
[5] magrittr_2.0.3 matrixStats_1.3.0
[7] compiler_4.4.1 systemfonts_1.1.0
[9] callr_3.7.6 vctrs_0.6.5
[11] reshape2_1.4.4 stringr_1.5.1
[13] profvis_0.3.8 pkgconfig_2.0.3
[15] crayon_1.5.3 fastmap_1.2.0
[17] XVector_0.44.0 ellipsis_0.3.2
[19] labeling_0.4.3 caTools_1.18.2
[21] utf8_1.2.4 promises_1.3.0
[23] rmarkdown_2.28 sessioninfo_1.2.2
[25] UCSC.utils_1.0.0 ps_1.7.7
[27] ragg_1.3.2 purrr_1.0.2
[29] xfun_0.47 zlibbioc_1.50.0
[31] cachem_1.1.0 GenomeInfoDb_1.40.1
[33] jsonlite_1.8.8 highr_0.11
[35] later_1.3.2 DelayedArray_0.30.1
[37] parallel_4.4.1 R6_2.5.1
[39] bslib_0.8.0 stringi_1.8.4
[41] pkgload_1.4.0 brio_1.1.5
[43] GenomicRanges_1.56.1 jquerylib_0.1.4
[45] Rcpp_1.0.13 SummarizedExperiment_1.34.0
[47] iterators_1.0.14 knitr_1.48
[49] R.utils_2.12.3 IRanges_2.38.1
[51] Matrix_1.7-0 httpuv_1.6.15
[53] R.cache_0.16.0 rstudioapi_0.16.0
[55] abind_1.4-5 yaml_2.3.10
[57] doParallel_1.0.17 gplots_3.1.3.1
[59] codetools_0.2-20 miniUI_0.1.1.1
[61] misc3d_0.9-1 processx_3.8.4
[63] pkgbuild_1.4.4 lattice_0.22-6
[65] tibble_3.2.1 plyr_1.8.9
[67] shiny_1.9.1 Biobase_2.64.0
[69] withr_3.0.1 evaluate_0.24.0
[71] desc_1.4.3 urlchecker_1.0.1
[73] pillar_1.9.0 MatrixGenerics_1.16.0
[75] KernSmooth_2.23-24 foreach_1.5.2
[77] stats4_4.4.1 rprojroot_2.0.4
[79] S4Vectors_0.42.1 ggplot2_3.5.1
[81] munsell_0.5.1 scales_1.3.0
[83] gtools_3.9.5 xtable_1.8-4
[85] glue_1.7.0 ppcor_1.1
[87] tools_4.4.1 fs_1.6.4
[89] grid_4.4.1 colorspace_2.1-1
[91] GenomeInfoDbData_1.2.12 cli_3.6.3
[93] textshaping_0.4.0 fansi_1.0.6
[95] S4Arrays_1.4.1 gtable_0.3.5
[97] R.methodsS3_1.8.2 sass_0.4.9
[99] digest_0.6.37 BiocGenerics_0.50.0
[101] SparseArray_1.4.8 farver_2.1.2
[103] htmlwidgets_1.6.4 memoise_2.0.1
[105] htmltools_0.5.8.1 pkgdown_2.1.0
[107] R.oo_1.26.0 lifecycle_1.0.4
[109] httr_1.4.7 mime_0.12
[111] MASS_7.3-60.2