Accessing recorded spikes via direct memory interface with NEURON

[iraikov]

Hello,
Is it currently possible to access the recorded spikes from within the direct memory interface with NEURON? This would be very convenient for parallel optimizations where different parameter instantiations of the same cell(s) are simulated on different ranks, and writing/reading the spike times to and from files for each evaluation would not be practical. Thanks!

[pramodk]

Yeah agree! Currently we read-in generated out.dat but that will be generated by all ranks together.

@nrnhines : Would it be possible to add new API to transfer spikes from CoreNEURON to NEURON so that pc.spike_record would work transparently for NEURON as well as CoreNEURON?

[nrnhines]

That is certainly possible and desirable. Spike transfer back from CoreNEURON to NEURON is one of the three next major steps for direct mode integration. The other two being transfer of all state data back to NEURON at the end of a CoreNEURON solve, and the inverse of that to allow NEURON to modify state after a CoreNEURON run and restart CoreNEURON to continue the run without rebuilding the CoreNEURON data structures. For the latter it would be beneficial to support the appropriate grain level for amount of transfer. ie. ranging from a single parameter value, to the entire state. It remains to be seen whether NEURON can just begin to use the CoreNEURON data directly rather than maintaining a copy but I think that is promising.

Anyway, for spikes, at the risk of being a bit of a "hack" prior to a complete API design, one would have to create a new method

ParallelContext.nrncore_spike_transfer(spiketimevec, gidvec)

which would do the job. I suppose we could use the neuron.coreneuron module to present this to the user but the implementation is a bit more indirect from a developer perspective and as on the foggy horizon there seem to be many images of other desireable methods, it may be useful to have an API discussion sooner rather than later.

[iraikov]

Hi @nrnhines @pramodk thanks for your comments. My understanding of the CoreNEURON direct memory test in the NEURON repository was that any vector recording a state variable in a section would get transferred to NEURON at the end of the simulation. This would appear to be the most seamless interface from a user's perspective. Having to explicitly call nrncore_spike_transfer would be slightly more overhead but would be ok as long as we can clear the spike recording vectors prior to starting a new simulation in the context of an optimization loop. (But I am guessing CoreNEURON already clears all recording vectors at the beginning of a simulation). The ability to stop and restart CoreNEURON would be very useful indeed, also for things like checkpoining and hotstart, but this is perhaps somewhat independent of shared memory buffers.

[pramodk]

Michael can comment on the other parts

This would appear to be the most seamless interface from a user's perspective. Having to explicitly call nrncore_spike_transfer would be slightly more overhead

I agree with this! I don't know all implementation aspects from NEURON but I would also prefer to not have different API for coreneuron.

The ability to stop and restart CoreNEURON would be very useful indeed, also for things like checkpoining and hotstart, but this is perhaps somewhat independent of shared memory buffers.

👍 We kind of "ignored" integration aspects with NEURIN for quite some time. But we are starting on this now. So hopefully we will get these use cases implemented soon.

By the way, if you haven't noticed, this is how current in-memory transfer usage looks like:

• https://github.com/neuronsimulator/nrn/blob/master/test/coreneuron/test_direct.py
• https://github.com/neuronsimulator/nrn/blob/master/test/coreneuron/test_direct.hoc

[nrnhines]

A NEURON + CoreNEURON direct API is actually more useful than I thought as getting several minds involved is likely to produce something better than the first thing that comes to mind. E.g. the ParallelContext.nrncore_spike_transfer(spiketimevec, gidvec) idea will certainly work. But one of the contextual details is that NEURON and CoreNEURON both know that they are participating in direct communication. So perhaps CoreNEURON should just turn off its creating an out.dat file and send back the spikes to the vectors that would have recorded them in a normal pc.solve(tstop). One nice side effect is that existing models that have an elaborate sequence of psolve followed by distributed parallel sorted output of spikes generated during the last psolve will just work when the simulation is integrated by CoreNEURON.

I wonder how far one can carry the principle of just neuron.coreneuron.enable = True and all other code remains the same.
And get to the point where it can be turned on or off at will.

[iraikov]

@nrnhines This sounds very general indeed. When you say the CoreNEURON will "send back" the spikes, does that mean that they will be stored in ParallelContext on the NEURON side, and then they can be retrieved at any time after the end of simulation? If yes, then I agree this will enable all sorts of complex workflows.

[nrnhines]

No. It means the spikes are stored in the two hoc Vector instances that are used in a non-coreneuron simulation. ie. ideally
https://www.neuron.yale.edu/neuron/static/py_doc/modelspec/programmatic/network/parcon.html#ParallelContext.spike_record in its

pc.spike_record(-1, tvec, idvec)

form or the more cumbersome
https://www.neuron.yale.edu/neuron/static/py_doc/modelspec/programmatic/network/netcon.html#NetCon.record in its

nc.record(tvec, idvec, nc.srcgid)

form.

[iraikov]

I see, yes that would be the most seamless option indeed.

[nrnhines]

@pramod what do you think about turning off the writing of out.dat by default during direct transfer mode. Or else a
default

neuron.coreneuron.spikeout = False

which would turn it off if there were a --spikeout 0 or --no-spikeout option.. We may have to eventually escape from the use of launch args to properly handle the direct mode, especially if the direct mode has a lot of specific options.

[alexsavulescu]

Hi @iraikov, we've just added this feature (#354). Would you consider trying it and giving us some feedback?

FYI, here is the unit test from NEURON : https://github.com/neuronsimulator/nrn/blob/master/test/coreneuron/test_spikes.py (via neuronsimulator/nrn#692)

[iraikov]

Hi @alexsavulescu thanks a lot! I will try it out and let you know how it goes.

[iraikov]

Hi @alexsavulescu now CoreNEURON does not seem able to load my mechanisms. (The same simulation works in the latest version of standard NEURON). Is there anything I can do to diagnose this error?