lazy=True in read_XXX and proxy object

[samuelgarcia]

Now, in read_segment() or read_block() or read_XXX(), lazy=True is use to load an object without its data but with all attributes and annotations.

The main use case is to open a dataset with a small memory footprint. Explore, filter, select some neo object and load a few of them or one by one.

The actual behavior is more or less this one:

reader = MyIO(filename='yep')
block = reader.read_block(lazy=True)

lazy_anasig = block.segments[15].analogsignals[128]
print(lazy_anasig.shape)
#this give (0, )
print(lazy_anasig.lazy_shape)
#this give (100000, 1)

anasig = reader.load_lazy_olbject(lazy_anasig)
print(anasig.shape)
#this give (100000, 1)

All IOs propose lazy=True but very few IOs propose load_lazy_olbject() because it is very hard to implement it (need to keep trace of id object).

Now with rawio, this could be easier to really implement it.
I propose to modify a bit the behavior to a better API.

The idea is to implement proxy object for thoses containing arrays : AnalogSognal, Event, Epoch, SpikeTrain and IrregularlySampledSignal. This obect would keep a link to the reader to be able to load the data when needed in a given time slice.

Example:

reader = MyIO(filename='yep')
block = reader.read_block(lazy=True)

proxy_anasig = block.segments[15].analogsignals[128]
print(proxy_anasig)
# this give ProxyAnalogSignal withh all attributes and annotations
print(proxy_anasig.t_start, proxy_anasig.sampling_rate)
print(proxy_anasig.annotations)

#put signals ion memory
anasig = proxy_anasig.load() 
print(lazy_anasig.shape)
#this give (100000, 1)

#put part of signals ion memory
chunk_anasig = proxy_anasig.load(time_slice=(10*pq.s, 15*pq.s))
print(anasig.shape)
#this give (5000, 1)


...
same idea for spike and events

Any idea on that ?

[JuliaSprenger]

Here are some usage examples of how we imagine loading data in the future might work using proxyobjects and the proposed object model (#456).

Example 1 - Reading some traces of an AnalogSignal

from neo import BlackrockIO
io = BlackrockIO(filename=filename, nsx_override=None, nev_override='my_spikesorted_file.nev', 
nsx_to_load=[2])

lazy_block = io.read_block(block_index=0, lazy=True) # this block contains only ProxyObjects including all metadata, but no data arrays. This should be very quick and cost few memory.

# selecting only every second trace of an AnalogSignalObject and loading all data of the trace
full_anasig = lazy_block.segments[0].analogsignals[0]
for channel_id in full_anasig.multiannotations['channel_id']:
     sig = neo.View(full_anasig, channel_id=[channel_id]).get_trace() # loads only selected data of proxyobject
    # do something with sig

Example 2 - Reading all traces in slides defined by Epochs # advanced example, mixed lazyness

from neo import BlackrockIO
io = BlackrockIO(filename=filename, nsx_override=None, nev_override='my_spikesorted_file.nev', 
nsx_to_load=[2,5,6])

lazy_block = io.read_block(block_index=0, lazy={'Event': True, 'AnalogSignal': False}) # lazy is by default False
# this block contains only AnalogSignals as ProxyObjects, but Event data are loaded.

# selecting only every second trace of all AnalogSignalObjects and loading all data of the trace
full_anasigs = lazy_block.segments[0].analogsignals
event = lazy_block.segments[0].events[0].times
for anasig in full_anasigs
   for t_start, duration in zip(event.times, event.durations):
       sig = full_anasig.load(time_slice=(t_start, t_start+duration))
       # do something with sig of dimensions

Example 3 - Non-lazy loading

from neo import BlackrockIO
io = BlackrockIO(filename=filename, nsx_override=None, nev_override=None, 
nsx_to_load=None) # this should still get header data / meta info of all nsx files

lazy_block = io.read_block(block_index=0, lazy=False) # loading everything except AnalogSignals

Example 4 - Loading slices based on Events

from private import ExperimentIO

io = ExperimentIO(filename=filename, nsx_override=None, nev_override='my_spikesorted_file.nev', 
nsx_to_load='all')

lazy_block = io.read_block(block_index=0, lazy=True)
event = lazy_block.segments[0].events[0].load(magnitude_mode='raw'/'?'/'?')


for t in event.times():
    anasig = lazy_block.segments[0].analogsignals[0].load(time_slice(t,t+10*pq.s), channel_index=[3], magnitude_mode='raw'/'rescaled') #raw = data as in file + compoundunit; rescaled = rescaled to specified preferred unit (as defined in rawio)
    st = lazy_block.segments[0].spiketrains[0].load(time_slice=(None,None), load_waveforms=False, magnitude_mode='raw'/'rescaled')

neo.load_lazy_automatically(replace=True)

This way there is a number of utility functions needed for loading and slicing on the segment and block level, so the code above can be simplified when loading a fixed set of data (no chunking of data). One example could be

def load_block_from_lazy_block(lazy_block,time_slices,**kwargs):
    bl = neo.Block()
    for time_slice in time_slices:
        for lazy_segment in lazy_block.segments:
             bl.append(lazy_segment.load(time_slice, **kwargs)

[samuelgarcia]

Implemented in #461.