exporting to phy for manual curation

[kshtjkumar]

Hi,

I’ve run spike sorting for several files and have the corresponding waveforms stored. Below is the code I used to display the folder contents:

import os
folder = r"C:\Users\garim\waveforms_kshtj_96lko_23_feb_tetrode_day_0_240223_161515311079"
print(os.listdir(folder))
waveform_folder = os.path.join(folder, 'waveforms')
print(os.listdir(waveform_folder))

this is the output:

['params.json', 'principal_components', 'quality_metrics', 'recording.json', 'recording_info', 'sorting.pickle', 'spike_amplitudes', 'templates_average.npy', 'templates_percentile_1.npy', 'templates_percentile_25.npy', 'templates_percentile_75.npy', 'templates_percentile_98.npy', 'waveforms']
['sampled_index_1.npy', 'sampled_index_2.npy', 'sampled_index_3.npy', 'sampled_index_4.npy', 'waveforms_1.npy', 'waveforms_2.npy', 'waveforms_3.npy', 'waveforms_4.npy']

I have several such folders, and I’d like to perform manual curation since the waveforms and SNR are good, but the ISI ratio is high. I’m still new to manual curation, so any resources or guidance would be appreciated.

Is it possible to load the existing results and convert them to export_to_phy ? I’d prefer not to rerun the spike sorting for each file, as it’s time-consuming. What would be the best way to approach this?

Thanks!

[zm711]

Hey @kshtjkumar,

which version of spikeinterface are you using? You should be to reload your data either as a waveform extractor (version < 0.101.0) or sorting analyzer (version >= 0.101.0) and then use the export_to_phy. we also offer manual curation through the spikeinterface-gui as well as through sortingview.

[kshtjkumar]

i am currently using 0.100.6 version, can you help on how to load it in phy or spikeinterface GUI ?

[zm711]

Could you post your script for analyzing your data? Basically you just need your waveform extractor and that can be used as an input to export_to_phy. If you don't already have spikeinterface-gui you might need @samuelgarcia to comment on the version you'll need since the current version of the gui only works with version > 0.101.0.

[kshtjkumar]

I used mountainsort 5 to sort the data using scheme 2. The waveform folder of each recording has these file :

['params.json', 'principal_components', 'quality_metrics', 'recording.json', 'recording_info', 'sorting.pickle', 'spike_amplitudes', 'templates_average.npy', 'templates_percentile_1.npy', 'templates_percentile_25.npy', 'templates_percentile_75.npy', 'templates_percentile_98.npy', 'waveforms']
['sampled_index_1.npy', 'sampled_index_2.npy', 'sampled_index_3.npy', 'sampled_index_4.npy', 'waveforms_1.npy', 'waveforms_2.npy', 'waveforms_3.npy', 'waveforms_4.npy']

[samuelgarcia]

For spikeinterface-gui you need a sorting analyzer.
You have some very basic example how to use it here on the readme:
https://github.com/SpikeInterface/spikeinterface-gui

The easiest way is to use spikeinterface 0.101.0 and this

from spikeinterface.widgets import plot_sorting_summary
plot_sorting_summary(sorting_analyzer, curation=True, backend="spikeinterface_gui")

[kshtjkumar]

@zm711 ?

[kshtjkumar]

Hi, so i created a new environment installing 0.101.0. Now i tried the codes on this: https://github.com/SpikeInterface/spikeinterface-gui
but it requires reading and sorting from scratch, I don't want to re sort my recordings because it takes a lot of time.

is it possible to do any of this based on the pre sorted recordings ? i have a folder called waveforms from each recording generated after sorting and the files in it are the same that i mentioned above.

Could you please give a clarity on if the manual sorting is possible based on these files or do i resort every file again !
@samuelgarcia @zm711

[zm711]

Well there are two components to this. You would never need to "resort" between version. If you are okay with the sorting results we have various read_x functions and those all work between versions. For example if you sorted with any version of kilosort or did phy stuff we have both read_kilosort and read_phy functions that will work for you :)

I think what you worried about is if the waveform extractor data will still be useful and it is. We have a backward compatibility function just for this case. Do you want to read the docs on it and then if you have more questions let us know!

https://spikeinterface.readthedocs.io/en/stable/tutorials/waveform_extractor_to_sorting_analyzer.html

Once you've converted your waveform extractor into a sorting analyzer you'll be able to use the spikeinterface-gui :)

[kshtjkumar]

hi @zm711, this was really helpful, i used spikeinterface 0.100.6 and ran the following, it worked perfectly:

waveform_folder_path = r"C:\Users\garim\waveforms_kshtj_961_aug_2023_lko_230801_151701347713"
new_sorting_analyzer_path = r"C:\Users\garim\testing_phy"
wvf_extractor = si.load_waveforms(folder=waveform_folder_path)
the_recording = wvf_extractor.recording
the_sorting = wvf_extractor.sorting
channel_locations =wvf_extractor.get_channel_locations()

from spikeinterface.exporters import export_to_phy
we = si.extract_waveforms(recording=the_recording, sorting=the_sorting, folder=waveform_folder_path+str('xyz'))
_  = computespike_amplitudes(waveform_extractor=we)
_  = compute_principal_components(waveform_extractor=we, n_components=3, mode='by_channel_global')
export_to_phy(waveform_extractor=we, output_folder=new_sorting_analyzer_path+str("abc"))

this opens perfectly in phy (but ofc not in spikeinterface gui!) and now am having trouble with understanding phy.