Clarify docstrings in neuropixels_utils

[alejoe91]

Fixes #3894

@billkarsh Do you mind taking a look? I believe that the main error and confusion point was in the use of num_channels_per_adc and its description

[chrishalcrow]

Hello, is there some NeuroPixels documentation/manual which describes this stuff in more detail? Could we link to it?

[alejoe91]

Hello, is there some NeuroPixels documentation/manual which describes this stuff in more detail? Could we link to it?

I couldn't find any good reference. Maybe @jsiegle and @billkarsh could provide some resources we could point at for the estimation of the channel inter sample shifts?

[billkarsh]

The operations are documented here: https://github.com/billkarsh/CatGT/blob/main/Build/Tool.cpp::FFT::init(), FFT::timeShiftChannel(). The mux tables are in the public probeTable repo: https://github.com/billkarsh/ProbeTable. The description of the mux table is here: https://billkarsh.github.io/SpikeGLX/Sgl_help/Metadata_30.html#imec. The mux table is organized as (nADC,nGrp)(grp)(grp)… Each (grp) are the (acquired) channel indices that are digitized together. The shift amount delT(igrp) is the time difference between group-igrp and group-0. Group-0 does not need to be shifted. delT(igrp) = igrp/(nGrp*sample_rate). Note that for 2.0-like probes (that are full band only), nGrp should be used as is from the table. In these cases the multiplexer clock runs at rate nGrp*sample_rate. That is, nGrp sets of channels are digitized during each effective sample period (1/sample_rate). However, for 1.0-like probes that have an AP and an LF band, the multiplexer runs at (nGrp+1)*ap_sample_rate. In all cases so far, nGrp=12. During each sample period, the multiplexer digitizes {nGrp=12 AP channel groups + 1 LF group}. The LF groups are the same groups as the AP groups in the table, however, it takes 12 AP sample periods to digitize all of the LF groups. That’s how the LF sample rate becomes exactly 1/12 of the AP sample rate. Bill From: Alessio Buccino ***@***.***> Sent: Wednesday, April 30, 2025 3:01 AM To: SpikeInterface/spikeinterface ***@***.***> Cc: Karsh, Bill ***@***.***>; Mention ***@***.***> Subject: Re: [SpikeInterface/spikeinterface] Clarify docstrings in neuropixels_utils (PR #3895) External Email: Use Caution [Image removed by sender.]alejoe91 left a comment (SpikeInterface/spikeinterface#3895)<https://urldefense.com/v3/__https:/github.com/SpikeInterface/spikeinterface/pull/3895*issuecomment-2841454844__;Iw!!Eh6p8Q!Hqg1KY42YHpobog4CDU4gqFx4dXCHkQj-WjIKjFmQOQIy3FK2MVDyjh8f276OsqT3WeE15d_HjcRQBduMXgQE3nZ6Es$> Hello, is there some NeuroPixels documentation/manual which describes this stuff in more detail? Could we link to it? I couldn't find any good reference. Maybe @jsiegle<https://urldefense.com/v3/__https:/github.com/jsiegle__;!!Eh6p8Q!Hqg1KY42YHpobog4CDU4gqFx4dXCHkQj-WjIKjFmQOQIy3FK2MVDyjh8f276OsqT3WeE15d_HjcRQBduMXgQ9Ge8pYk$> and @billkarsh<https://urldefense.com/v3/__https:/github.com/billkarsh__;!!Eh6p8Q!Hqg1KY42YHpobog4CDU4gqFx4dXCHkQj-WjIKjFmQOQIy3FK2MVDyjh8f276OsqT3WeE15d_HjcRQBduMXgQsgi4S-Y$> could provide some resources we could point at for the estimation of the channel inter sample shifts? — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/SpikeInterface/spikeinterface/pull/3895*issuecomment-2841454844__;Iw!!Eh6p8Q!Hqg1KY42YHpobog4CDU4gqFx4dXCHkQj-WjIKjFmQOQIy3FK2MVDyjh8f276OsqT3WeE15d_HjcRQBduMXgQE3nZ6Es$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/ABWEDUAPSGL2UDXLRM3HM5L24CNNLAVCNFSM6AAAAAB4E4SXSCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQNBRGQ2TIOBUGQ__;!!Eh6p8Q!Hqg1KY42YHpobog4CDU4gqFx4dXCHkQj-WjIKjFmQOQIy3FK2MVDyjh8f276OsqT3WeE15d_HjcRQBduMXgQmrnqi2E$>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[billkarsh]

I also put these comments on github issue, and edited as follows: However, for 1.0-like probes that have an AP and an LF band, the multiplexer runs at (nGrp+1) * ap_sample_rate. In all cases so far, nGrp=12. During each sample period, the multiplexer digitizes {nGrp=12 AP channel groups + 1 LF group}. The LF groups are the same groups as the AP groups in the table, however, it takes 12 AP sample periods to digitize all of the LF groups. That’s how the LF sample rate becomes exactly 1/12 of the AP sample rate. So when doing a 1.0-like AP file, delT(igrp) = igrp/(13 * ap_sample_rate). When doing an LF file, delT(igrp) = igrp/(12 * lf_sample_rate), as if this was single-band data. From: Alessio Buccino ***@***.***> Sent: Wednesday, April 30, 2025 3:01 AM To: SpikeInterface/spikeinterface ***@***.***> Cc: Karsh, Bill ***@***.***>; Mention ***@***.***> Subject: Re: [SpikeInterface/spikeinterface] Clarify docstrings in neuropixels_utils (PR #3895) External Email: Use Caution [Image removed by sender.]alejoe91 left a comment (SpikeInterface/spikeinterface#3895)<https://urldefense.com/v3/__https:/github.com/SpikeInterface/spikeinterface/pull/3895*issuecomment-2841454844__;Iw!!Eh6p8Q!Hqg1KY42YHpobog4CDU4gqFx4dXCHkQj-WjIKjFmQOQIy3FK2MVDyjh8f276OsqT3WeE15d_HjcRQBduMXgQE3nZ6Es$> Hello, is there some NeuroPixels documentation/manual which describes this stuff in more detail? Could we link to it? I couldn't find any good reference. Maybe @jsiegle<https://urldefense.com/v3/__https:/github.com/jsiegle__;!!Eh6p8Q!Hqg1KY42YHpobog4CDU4gqFx4dXCHkQj-WjIKjFmQOQIy3FK2MVDyjh8f276OsqT3WeE15d_HjcRQBduMXgQ9Ge8pYk$> and @billkarsh<https://urldefense.com/v3/__https:/github.com/billkarsh__;!!Eh6p8Q!Hqg1KY42YHpobog4CDU4gqFx4dXCHkQj-WjIKjFmQOQIy3FK2MVDyjh8f276OsqT3WeE15d_HjcRQBduMXgQsgi4S-Y$> could provide some resources we could point at for the estimation of the channel inter sample shifts? — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/SpikeInterface/spikeinterface/pull/3895*issuecomment-2841454844__;Iw!!Eh6p8Q!Hqg1KY42YHpobog4CDU4gqFx4dXCHkQj-WjIKjFmQOQIy3FK2MVDyjh8f276OsqT3WeE15d_HjcRQBduMXgQE3nZ6Es$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/ABWEDUAPSGL2UDXLRM3HM5L24CNNLAVCNFSM6AAAAAB4E4SXSCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQNBRGQ2TIOBUGQ__;!!Eh6p8Q!Hqg1KY42YHpobog4CDU4gqFx4dXCHkQj-WjIKjFmQOQIy3FK2MVDyjh8f276OsqT3WeE15d_HjcRQBduMXgQmrnqi2E$>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[jsiegle]

1.0 probes have 32 ADCs (each of which receives multiplexed input from 12 channels), while 2.0 probes have 24 ADCs (each of which receives multiplexed input from 16 channels). This is not explained anywhere in IMEC's public-facing documentation, as far as I'm aware.

The key piece of info for determining the sample shifts is knowing the order in which channels are sampled by a given ADC, rather than the total number of ADCs. And also keeping in mind that there are 13 cycles of digitization for the 1.0 probes (12 AP + 1 LFP), as Bill pointed out.

We created an illustration for the Open Ephys GUI documentation that may be helpful. Here, simultaneously sampled channels have the same color:

[alejoe91]

Thanks @jsiegle and @billkarsh!

I reverted the naming to num_channels_per_adc, which is correctly set to 12/16 for NP1/2 probes. The number of cycles (13 for NP1 and 16 for NP2) is also correct (both for Open Ephys and SpikeGLX).

@billkarsh I think that the mux table is super clear, but it is not available for older SpikeGLX versions and for Open Ephys. I checked a couple of muxTables entries in our test files and they correspond to the schema that Josh provided, which also corresponds to the "logic" implemented in the function:

# these correspond to the ADC#
adc_indices = np.floor(np.arange(num_channels) / (num_channels_per_adc * 2)) * 2 + np.mod(np.arange(num_channels), 2)

for adc_index in adc_indices:
    sample_shifts[adc_indices == adc_index] = np.arange(num_channels_per_adc) / num_cycles

I think this is correct for standard configurations, since np.nonzero(adc_indices == adc_index) results in the same mux table entries as some test files (e.g, this one). I'm not sure though if this logic is robust against custom configurations by the user or when only a subset of electrodes is selected.

@billkarsh how is the mux table generated by SpikeGLX? Is it retrieved by the probe registers or is there some logic in SpikeGLX to compute them? For the latter case, we can port the logic here to make sure it works for different SpikeGLX versions and for Open Ephys.

On the probeinterface side, we should are planning to sync all the NP info with ProbeTable repo. @billkarsh would it be possible to add the nGrp (i.e., num_channels_per_adc) to each entry? So we can propagate it to the metadata and read it from spikeinterface for each probe without hardcoding it.

[billkarsh]

There’s a lot of info here. Read carefully. * Mux tables have nothing to do with OEP vs SpikeGLX, or with any user choices or parameters. They are the wiring plans of probes that imec makes. They describe which channel are digitized together and by which ADC. * The probe tables I publish here https://github.com/billkarsh/ProbeTable are imec data. These are not specific to SpikeGLX or OEP. From time to time imec will make more probe types and the tables will be updated. I maintain the tables for everybody. * All the mux tables are in the probe tables. They are keyed by ‘mux_table_format_type’. * To identify the appropriate mux table to refer to, you need to consult the probe table. * To obtain the proper entry in the probe table you need the probe part number. That will point you to the right mux table. * Always refer to the probe part number. * The probe ‘type’ is a SpikeGLX private internal indexing scheme that is not as meaningful or reliable as the probe part number. Use the part number. SpikeGLX will deprecate ‘type’ at some point. * If the probe table tag ‘databus_decoder_type’ contains “1.0” then your ‘cycles’, which is my nGrp, is {nGrp +1 for AP-band data, nGrp for LF-band data}. Be careful about that. Don’t add 1 for SpikeGLX 1.0 LF data files. * The mux tables list readout channels that are digitized together, hence, which multiplexing cycle they belong to. * SpikeGLX files need not save all readout channels. The metadata item snsSaveChanSubset indicates which readout channels compose the timepoints/samples. You must identify the readout channel indices to map to the correct cycle. * Yes you should access data from the probe table ASAP and remove hardcoded assumptions in code. Bill From: Alessio Buccino ***@***.***> Sent: Friday, May 2, 2025 4:42 AM To: SpikeInterface/spikeinterface ***@***.***> Cc: Karsh, Bill ***@***.***>; Mention ***@***.***> Subject: Re: [SpikeInterface/spikeinterface] Clarify docstrings in neuropixels_utils (PR #3895) External Email: Use Caution [https://avatars.githubusercontent.com/u/17097257?s=20&v=4]alejoe91 left a comment (SpikeInterface/spikeinterface#3895)<https://urldefense.com/v3/__https:/github.com/SpikeInterface/spikeinterface/pull/3895*issuecomment-2847013151__;Iw!!Eh6p8Q!C2Lh7d4WU_Tc6cadeP-XY-wNsqQ0_aMVK2p43dh7bh0942l7NBEYIt5xLWNlUK9ub66aw969TQvVWMG673tHtAcJktE$> Thanks @jsiegle<https://urldefense.com/v3/__https:/github.com/jsiegle__;!!Eh6p8Q!C2Lh7d4WU_Tc6cadeP-XY-wNsqQ0_aMVK2p43dh7bh0942l7NBEYIt5xLWNlUK9ub66aw969TQvVWMG673tHf87S7Mg$> and @billkarsh<https://urldefense.com/v3/__https:/github.com/billkarsh__;!!Eh6p8Q!C2Lh7d4WU_Tc6cadeP-XY-wNsqQ0_aMVK2p43dh7bh0942l7NBEYIt5xLWNlUK9ub66aw969TQvVWMG673tHbGpar1I$>! I reverted the naming to num_channels_per_adc, which is correctly set to 12/16 for NP1/2 probes. The number of cycles (13 for NP1 and 16 for NP2) is also correct (both for Open Ephys<https://urldefense.com/v3/__https:/github.com/SpikeInterface/spikeinterface/blob/main/src/spikeinterface/extractors/neoextractors/openephys.py*L242-L255__;Iw!!Eh6p8Q!C2Lh7d4WU_Tc6cadeP-XY-wNsqQ0_aMVK2p43dh7bh0942l7NBEYIt5xLWNlUK9ub66aw969TQvVWMG673tHmpNmFuA$> and SpikeGLX<https://urldefense.com/v3/__https:/github.com/SpikeInterface/spikeinterface/blob/main/src/spikeinterface/extractors/neoextractors/spikeglx.py*L92-L99__;Iw!!Eh6p8Q!C2Lh7d4WU_Tc6cadeP-XY-wNsqQ0_aMVK2p43dh7bh0942l7NBEYIt5xLWNlUK9ub66aw969TQvVWMG673tHIp9njHo$>). @billkarsh<https://urldefense.com/v3/__https:/github.com/billkarsh__;!!Eh6p8Q!C2Lh7d4WU_Tc6cadeP-XY-wNsqQ0_aMVK2p43dh7bh0942l7NBEYIt5xLWNlUK9ub66aw969TQvVWMG673tHbGpar1I$> I think that the mux table is super clear, but it is not available for older SpikeGLX versions and for Open Ephys. I checked a couple of muxTables entries in our test files and they correspond to the schema that Josh provided, which also corresponds to the "logic" implemented in the function: # these correspond to the ADC# adc_indices = np.floor(np.arange(num_channels) / (num_channels_per_adc * 2)) * 2 + np.mod(np.arange(num_channels), 2) for adc_index in adc_indices: sample_shifts[adc_indices == adc_index] = np.arange(num_channels_per_adc) / num_cycles I think this is correct for standard configurations, since np.nonzero(adc_indices == adc_index) results in the same mux table entries as some test files (e.g, this one<https://urldefense.com/v3/__https:/github.com/SpikeInterface/probeinterface/blob/main/tests/data/spikeglx/non_human_primate_long_staggered.imec0.ap.meta*L55__;Iw!!Eh6p8Q!C2Lh7d4WU_Tc6cadeP-XY-wNsqQ0_aMVK2p43dh7bh0942l7NBEYIt5xLWNlUK9ub66aw969TQvVWMG673tHQRyYq5c$>). I'm not sure though if this logic is robust against custom configurations by the user or when only a subset of electrodes is selected. @billkarsh<https://urldefense.com/v3/__https:/github.com/billkarsh__;!!Eh6p8Q!C2Lh7d4WU_Tc6cadeP-XY-wNsqQ0_aMVK2p43dh7bh0942l7NBEYIt5xLWNlUK9ub66aw969TQvVWMG673tHbGpar1I$> how is the mux table generated by SpikeGLX? Is it retrieved by the probe registers or is there some logic in SpikeGLX to compute them? For the latter case, we can port the logic here to make sure it works for different SpikeGLX versions and for Open Ephys. On the probeinterface side, we should are planning to sync all the NP info with ProbeTable repo. @billkarsh<https://urldefense.com/v3/__https:/github.com/billkarsh__;!!Eh6p8Q!C2Lh7d4WU_Tc6cadeP-XY-wNsqQ0_aMVK2p43dh7bh0942l7NBEYIt5xLWNlUK9ub66aw969TQvVWMG673tHbGpar1I$> would it be possible to add the nGrp (i.e., num_channels_per_adc) to each entry? So we can propagate it to the metadata and read it from spikeinterface for each probe without hardcoding it. — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/SpikeInterface/spikeinterface/pull/3895*issuecomment-2847013151__;Iw!!Eh6p8Q!C2Lh7d4WU_Tc6cadeP-XY-wNsqQ0_aMVK2p43dh7bh0942l7NBEYIt5xLWNlUK9ub66aw969TQvVWMG673tHtAcJktE$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/ABWEDUDL7MDGM7LRHV74HAT24NKZRAVCNFSM6AAAAAB4E4SXSCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQNBXGAYTGMJVGE__;!!Eh6p8Q!C2Lh7d4WU_Tc6cadeP-XY-wNsqQ0_aMVK2p43dh7bh0942l7NBEYIt5xLWNlUK9ub66aw969TQvVWMG673tHORNrFl8$>. You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>>

[alejoe91]

Thanks @billkarsh

If I understand correctly, then the mux tables are fixed and don't change if the user changes the selected electrodes.
Therefore:

• whatever the user chooses as configuration the at-most 384 channels will be sampled by the ADCs as in the z_mux_tables
• if a subset of channels are used, we can just slice the mux table values accordingly (e.g., if 100 channels are used, we'll take values until 100)

I think that then what I'll do is to refactor probeinterface to use this information and add the mux table index as an array annotation to the contact, so we can read it and compute the correct sample shift on the spikeinterface side.

Here is the issue on probeinterface: SpikeInterface/probeinterface#341

[billkarsh]

* Mux tables are fixed. * Probes like the 2020 have 1536 channels. There’s a 128-channel probe/headstage. So get unstuck from 384 channels. * You need to look up how many channels there are from the probe table. * Let’s say there are 384 channels on a given probe. OEP and SpikeGLX always acquire all 384 channels all the time, there is never a subset of channels that is acquired. * In OEP and SpikeGLX one can choose which electrodes on the probe surface are connected to the channels, but this has no effect on the digitizing and acquisition of the channels. The channels are fixed in number for that probe and you read out all the channels for each sample. They are all digitized according to the mux table. * In SpikeGLX you can choose which channels to save into a file. Suppose there are ten saved channels. Again, we will acquire all the channels from the hardware, but only write out ten. They may be acquired channels {2,30,33,34,67,88,90,201,300,301} or whatever the user desires. These indices are given by snsSaveChanSubset. The channel numbers indicate the location of these saved channels in the original mux table. You must look up mux group membership using the original acquisition channel identity. From: Alessio Buccino ***@***.***> Sent: Friday, May 2, 2025 6:30 AM To: SpikeInterface/spikeinterface ***@***.***> Cc: Karsh, Bill ***@***.***>; Mention ***@***.***> Subject: Re: [SpikeInterface/spikeinterface] Clarify docstrings in neuropixels_utils (PR #3895) External Email: Use Caution [https://avatars.githubusercontent.com/u/17097257?s=20&v=4]alejoe91 left a comment (SpikeInterface/spikeinterface#3895)<https://urldefense.com/v3/__https:/github.com/SpikeInterface/spikeinterface/pull/3895*issuecomment-2847206831__;Iw!!Eh6p8Q!GRlobACO6RnZqVq2_ZxQ2IBAv9LOk5JUYyo8lbNLILeRX3Kz0LPDiZ3vUkP0vQz_etbhTnz9DtHMvO255sjj9_25of4$> Thanks @billkarsh<https://urldefense.com/v3/__https:/github.com/billkarsh__;!!Eh6p8Q!GRlobACO6RnZqVq2_ZxQ2IBAv9LOk5JUYyo8lbNLILeRX3Kz0LPDiZ3vUkP0vQz_etbhTnz9DtHMvO255sjjAPw9glE$> If I understand correctly, then the mux tables are fixed and don't change if the user changes the selected electrodes. Therefore: * whatever the user chooses as configuration the at-most 384 channels will be sampled by the ADCs as in the z_mux_tables * if a subset of channels are used, we can just slice the mux table values accordingly (e.g., if 100 channels are used, we'll take values until 100) I think that then what I'll do is to refactor probeinterface to use this information and add the mux table index as an array annotation to the contact, so we can read it and compute the correct sample shift on the spikeinterface side. Here is the issue on probeinterface: SpikeInterface/probeinterface#341<https://urldefense.com/v3/__https:/github.com/SpikeInterface/probeinterface/issues/341__;!!Eh6p8Q!GRlobACO6RnZqVq2_ZxQ2IBAv9LOk5JUYyo8lbNLILeRX3Kz0LPDiZ3vUkP0vQz_etbhTnz9DtHMvO255sjjAxT6xBA$> — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/SpikeInterface/spikeinterface/pull/3895*issuecomment-2847206831__;Iw!!Eh6p8Q!GRlobACO6RnZqVq2_ZxQ2IBAv9LOk5JUYyo8lbNLILeRX3Kz0LPDiZ3vUkP0vQz_etbhTnz9DtHMvO255sjj9_25of4$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/ABWEDUEZ7OCHOF6NP2RCTZ324NXLVAVCNFSM6AAAAAB4E4SXSCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQNBXGIYDMOBTGE__;!!Eh6p8Q!GRlobACO6RnZqVq2_ZxQ2IBAv9LOk5JUYyo8lbNLILeRX3Kz0LPDiZ3vUkP0vQz_etbhTnz9DtHMvO255sjjLDPqCL8$>. You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>>