Refactor test modules

[ganow]

Summary

• Fixed one testing module such that it can work on the current implementation
  • 4ce8a2f: changed the behavior of a test code itself since the behavior of ModelTraining/ModelTest seemed to be changed (model.pkl -> model.pkl.gz)
• Resolved dependencies on data files not contained in the repository.
  • Almost all commits are done by just adding necessary data in this repository
  • 2505d61: removed the dependency on pre-computed Features data by creating a mock function that generates a fake random data
• reduce the file size -> leave files uncommitted due to lack of time
  • epi.mat
• The structure of the test files was reorganized to the typical form used in many OSS.
  • Changed the test directory from test to tests
  • Test files in the same directory hierarchy (tests/path/to/test_<src>.py) are now responsible for their corresponding source file (bdpy/path/to/<src>.py).

[ganow]

I found the test code that assumes the existence of some mat files, which are not included in this repository. Do you know what type of dataset we need? > @ShuntaroAoki

[ganow]

ref. to the original code:

bdpy/test/test_dataform_sparse.py

Lines 65 to 74 in 4e87ee1

	def test_load_array_jl(self):
	data = np.array([[1, 0, 0, 0],
	[2, 2, 0, 0],
	[3, 3, 3, 0]])
	testdata = load_array('data/array_jl_dense_v1.mat', key='a')
	np.testing.assert_array_equal(data, testdata)
	testdata = load_array('data/array_jl_sparse_v1.mat', key='a')
	np.testing.assert_array_equal(data, testdata)

[ganow]

Moved /test/test_ml.py to /test/ml/test_learning.py and prepared the model parameter files for testing ModelTest class.
commit hash: 4ce8a2f

Current problem
Since *.mat is specified in .gitignore, we cannot commit the data files. (how should we do?)

[ganow]

Moved /test/test_ml.py to /test/ml/test_learning.py and prepared the model parameter files for testing ModelTest class. commit hash: 4ce8a2f

Current problem Since *.mat is specified in .gitignore, we cannot commit the data files. (how should we do?)

I have used the script as follows to prepare model parameter files:

import os

import numpy as np
from sklearn.linear_model import LinearRegression
from fastl2lir import FastL2LiR

from bdpy.ml import ModelTraining


X = np.random.rand(100, 500)
Y1dim = np.random.rand(100, 50)
Y4dim = np.random.rand(100, 8, 4, 4)


def run(model_type, format_, chunked):
    key = f'{model_type}-{"chunk" if chunked else "nochunk"}-{format_}'

    if model_type == 'lir':
        model = LinearRegression()
    elif model_type == 'fastl2lir':
        model = FastL2LiR()
    else:
        raise ValueError(f'Unknown model type: {model_type}')

    Y = Y4dim if chunked else Y1dim

    train = ModelTraining(model, X, Y)
    train.id = key
    if model_type == 'fastl2lir':
        train.model_parameters = {'alpha': 100, 'n_feat': 100}
    train.dtype = np.float32
    if chunked:
        train.chunk_axis = 1
    train.save_format = 'bdmodel' if format_ == 'bd' else 'pickle'
    train.save_path = os.path.join('test/data/test_models', key)

    train.run()


run('lir', 'pkl', False)

model_formats = ['pkl', 'bd']
chunked_options = [False, True]

for model_format in model_formats:
    for chunked in chunked_options:
        run('fastl2lir', model_format, chunked)

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
bdpy/bdata
bdata.py	398	194	51%	79, 104, 109, 113, 118, 122, 132–135, 193, 237–243, 258–268, 283–284, 321, 325, 330–368, 418–424, 432–433, 438–439, 456–463, 481–482, 488, 522, 554, 565, 578, 611–620, 632, 647, 683, 706–714, 721–754, 764, 776–783, 788–794, 799–827, 832–853, 859–893, 897–899, 903–905, 910–919
featureselector.py	64	12	81%	62–67, 69–74
metadata.py	67	1	99%	84
utils.py	113	37	67%	71, 82, 85–86, 95, 127–173, 201, 246, 258, 263
bdpy/dataform
datastore.py	107	85	21%	59–75, 90–93, 97–98, 102–113, 116–119, 122–127, 131–132, 137–158, 190–197, 222–259, 262–265
features.py	292	161	45%	29–32, 43–46, 90–92, 101–103, 107, 111, 115, 119, 154–158, 165–194, 211–212, 226–230, 268, 282, 299–313, 317, 321, 325, 329, 333, 337, 341, 345, 349, 353, 358–385, 389–409, 413–453, 456, 461–468, 482–484, 487–490, 493–496, 499–503, 506–507, 527–540
pd.py	9	5	44%	25–27, 43–44
sparse.py	67	7	90%	29, 52–58, 74, 109, 123
utils.py	12	12	0%	3–18
bdpy/dataset
utils.py	45	45	0%	3–98
bdpy/distcomp
distcomp.py	92	18	80%	33, 35, 49, 53, 55, 66–70, 74, 76, 81–82, 89–93, 97
bdpy/dl
caffe.py	60	60	0%	4–129
bdpy/dl/torch
base.py	43	24	44%	31–41, 48, 54, 60, 63, 73–83, 90, 96, 102, 105
models.py	332	226	32%	147–168, 296–315, 326–330, 344–349, 441–493, 514–516, 527–586, 610–613, 624–683, 707–710, 721–770, 789–792, 803–852, 871–874
torch.py	109	60	45%	49, 60, 81, 100, 107, 110, 172–202, 205, 208–220, 223–258
bdpy/evals
metrics.py	95	67	29%	49–53, 59–61, 82–112, 118–159, 172–179
bdpy/feature
feature.py	30	2	93%	69–70
bdpy/fig
__init__.py	4	4	0%	6–9
draw_group_image_set.py	90	90	0%	3–182
fig.py	88	88	0%	16–164
makeplots.py	336	336	0%	1–729
tile_images.py	59	59	0%	1–193
bdpy/ml
crossvalidation.py	59	27	54%	47–48, 113–114, 117–118, 138, 164–196
learning.py	308	96	69%	43–44, 48, 52, 59, 91–104, 109–125, 128, 158–170, 184–209, 293, 309, 313–315, 318–319, 329, 339–340, 345–346, 356–364, 367–368, 376, 411–418, 439, 452, 460, 469, 501–503, 542, 554, 557, 565, 573, 578, 599
model.py	140	120	14%	29–39, 54–70, 86–144, 156–169, 184–222, 225, 230–250, 254–258, 271–285
searchlight.py	16	13	19%	32–51
bdpy/mri
fmriprep.py	497	451	9%	25–34, 38, 44–62, 65–75, 78–89, 92–160, 163–194, 230–360, 367–380, 384, 388–390, 394, 398–400, 410–434, 437–454, 457–464, 471–472, 475–491, 494, 498, 502–815, 819–831, 842–862
glm.py	40	36	10%	46–95
image.py	24	19	21%	29–54
load_epi.py	28	18	36%	36–50, 56–63, 82–88
load_mri.py	19	16	16%	16–36
roi.py	248	234	6%	37–100, 122–148, 165–235, 241–314, 320–387, 399–466, 473–499
spm.py	158	139	12%	26–155, 162–166, 170, 174–179, 183–300
bdpy/opendata
__init__.py	1	1	0%	1
openneuro.py	210	210	0%	1–329
bdpy/pipeline
config.py	36	2	94%	37–38
bdpy/preproc
interface.py	52	16	69%	111–123, 148–157
preprocessor.py	129	69	47%	35, 44, 112–114, 121–128, 138–189, 196–227
select_top.py	22	1	95%	56
bdpy/recon
utils.py	55	55	0%	4–146
bdpy/recon/torch
__init__.py	1	1	0%	1
icnn.py	161	161	0%	15–478
bdpy/stats
corr.py	43	3	93%	57, 68, 102
bdpy/util
info.py	47	36	23%	19–79
utils.py	36	8	78%	60, 116–121, 140–142
TOTAL	4919	3325	32%

Tests	Skipped	Failures	Errors	Time
114	0 💤	5 ❌	0 🔥	9.859s ⏱️

[ganow]

Question

The following four files were not found in the current bdpy repository.

• data/array_jl_dense_v1.mat
• data/testdata-2d.pkl.gz
• data/testdata-2d-nan.pkl.gz
• ./data/mri/epi.mat

@ShuntaroAoki Could you commit them to this branch (or could you tell me how to create these files)? I can handle the modifications to make the current test codes work with the files you commit.

Log of the testing

==================================================================== short test summary info =====================================================================
FAILED tests/dataform/test_sparse.py::TestSparse::test_load_array_jl - FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = 'data/array_jl_dense_v1.mat', errno = 2, error message = 'No such file or direc...
FAILED tests/evals/test_metrics.py::TestMetrics::test_2d - FileNotFoundError: [Errno 2] No such file or directory: 'data/testdata-2d.pkl.gz'
FAILED tests/evals/test_metrics.py::TestMetrics::test_2d_nan - FileNotFoundError: [Errno 2] No such file or directory: 'data/testdata-2d-nan.pkl.gz'
ERROR tests/test_mri.py::TestMri::test_add_load_epi_pass0001 - FileNotFoundError: [Errno 2] No such file or directory: './data/mri/epi.mat'
ERROR tests/test_mri.py::TestMri::test_get_roiflag_pass0001 - FileNotFoundError: [Errno 2] No such file or directory: './data/mri/epi.mat'
ERROR tests/test_mri.py::TestMri::test_get_roiflag_pass0002 - FileNotFoundError: [Errno 2] No such file or directory: './data/mri/epi.mat'
===================================================== 3 failed, 104 passed, 70 warnings, 3 errors in 32.52s ======================================================

[ganow]

Working log on 6/27

• merged the current dev branch (ad3f588) to use GitHub workflow.

[ganow]

@ShuntaroAoki I've finished all of the refactoring of the test modules except the epi files. I would appreciate a review when you have time. Thank you.