[NixIO] Object names, writing, and overwriting

[achilleas-k]

This is a copy of a thread I started on the NeuralEnsemble mailing list (https://groups.google.com/forum/#!topic/neuralensemble/NYpXcjlTzFE)

I'm rewriting some parts of the NixIO to make them more efficient. One thing that's very inefficient in the current IO is the function that resolves naming conflicts between objects. While rethinking the functionality, I've realised that there is some ambiguity in the way writing works, in particular when rewriting objects. I'm starting this thread to request comments on how people expect the IO to work and what should be considered the right way of doing rewrites.

First, some background

The NIX data format requires unique names between objects of the same type. Due to the mapping between Neo and NIX object types, object names in the following groups must be unique:

• AnalogSignal, IrregularlySampledSignal: Map to DataArray
• Event, Epoch, Spiketrain: Map to MultiTag
• ChannelIndex, Unit: Map to Source

Another important restriction of the NIX format is that object names are required (can't be empty string or None). I have noticed that it is not uncommon for objects, especially container types like Block and Segment, to be anonymous in Neo, so the NixIO automatically generates names for anonymous objects during writing.

Current state of NIX writer

Currently the NixIO has a method called 'resolve_name_conflicts' which accepts a list of objects and, as the name suggests, goes through and finds anonymous objects and objects with conflicting names and resolves the issues. Anonymous objects get a name based on their type and the number of objects of that type (e.g., neo.Block-2, neo.Block-3, etc) and conflicting names get a number appended to them.

Issues with the current state

There are two things that I don't like about the current function:

• It renames the user's objects. This could cause issues for users who attach meaning to the names of objects but more generally, I believe it's wrong for a "save" function to in-place modify the data it's been given. While it is possibly unavoidable to keep conflicting or invalid names after a user saves and later loads the data (more on this later), we can certainly avoid editing the objects that the user has passed to the writer and may still be working on after a successful save (during the same session or script).
• It's rather slow. Checking for conflicts can slow down writing and it's especially bad when there are no conflicts to resolve. An on-demand renaming would be preferable (I'm working on this now).

The bigger issue

The issues I mention above can be resolved in many ways, but each solution makes some assumption about how rewrites should work. This is where I could use some feedback. When writing an object to a file that already contains an object with the same name, should the new object overwrite the old, or should it be renamed and stored alongside? If we overwrite, how do we handle repeated writes of anonymous objects?

Consider the following example script:

nixfile = NixIO(filename, "rw")

block = neo.Block("datablock")  # a simple block
nixfile.write_block(block)

block.name = "datablock-B"  # same block renamed
nixfile.write_block(block)

print(len(nixfile.read_all_blocks()))
nixfile.close()

nixfile = NixIO(filename, "rw")
nixfile.write_block(block)  # same block written again
print(len(nixfile.read_all_blocks()))
nixfile.close()


nixfile = NixIO(filename, "rw")
block = neo.Block("datablock-B",  # new block, reused name
                  description="a new datablock with an old name")
nixfile.write_block(block)
print(len(nixfile.read_all_blocks()))
nixfile.close()

nixfile = NixIO(filename, "rw")
blks = [neo.Block() for _ in range(10)]  # 10 anonymous blocks
nixfile.write_all_blocks(blks)
print(len(nixfile.read_all_blocks()))
nixfile.write_all_blocks(blks)  # write them again without changing anything
print(len(nixfile.read_all_blocks()))

blks = [neo.Block() for _ in range(10)]  # 10 new anonymous blocks
nixfile.write_all_blocks(blks)
print(len(nixfile.read_all_blocks()))
nixfile.close()

nixfile = NixIO(filename, "rw")
for _ in range(10):  # write 10 anonymous blocks, one at a time
    nixfile.write_block(neo.Block())
print(len(nixfile.read_all_blocks()))
nixfile.close()

nixfile = NixIO(filename, "rw")
for idx in range(10):  # write 10 named blocks, one at a time
    nixfile.write_block(neo.Block(name=str(idx)))
print(len(nixfile.read_all_blocks()))
nixfile.close()

What should the number of blocks be each time we print the result of 'read_all_blocks'?

I don't know if this is comparable to how other IOs handle rewrites. This issue begins with the naming of objects, but it goes beyond that. For instance, the naming issue can be circumvented entirely by giving each object stored in NIX a unique identifier for a name and just storing the Neo object name as an attribute. This would preserve the name for loading and even keep anonymous objects anonymous after a save-load cycle.

I'm curious to hear from Neo devs, IO devs, and users. What should happen when you do a "write" multiple times on the same object? What about if you load data, modify it, and then write it within a single session?

I can probably work with any (reasonable) expected behaviour, but while thinking about all the alternatives I realised that I was assuming too much.

[apdavison]

the naming issue can be circumvented entirely by giving each object stored in NIX a unique identifier for a name and just storing the Neo object name as an attribute.

I think this is definitely the best way to solve the name problem.

[achilleas-k]

Sam Garcia's response to the original thread

A short answer:

• Yes, writting in neo.io is not well defined, few IO use write possibility and so API is fuzzy.
• To avoid magic behavior. If a unique name is required for an object, it should raise an error if name is not given when NixIO.write_block(...).
• I would add 2 args for each writting methods: generate_primary_key=False and overwrite_object_if_exists=False. So errors would be raised unless the user known what he is doing.

Primary keys for neo object was more or less discussed once. I remember correctly we decided to let this problem out of neo because a primary is unique but only in a given context. And the context can have different scale (file, set of files, database, set of databases).

I like the idea of leaving it up to the user to define the behaviour via args. This might cause issues with usability if users just want to read from one format and quickly save it to another.
It still kinda leaves open the issue of writing anonymous objects repeatedly. In other words, what would the following steps produce:

1. Load AnalogSignal with no name from a file in a format other than NIX.
2. Save AnalogSignal using NIXIO and set generate_primary_key=True.
3. Change some data in the AnalogSignal and save again with generate_primary_key=True (the value of overwrite_object may not be relevant, depending on whether the IO assumes certain things about the object's identity).

This will most likely duplicate the data in the file since there's no good way to determine that the previously saved DataArray is the same as the modified one. If this happens repeatedly, it might leave the user in a state where they don't know which of the stored DataArrays is the latest.

[achilleas-k]

@apdavison Since you like it, I'll get to work on that naming solution as a start.

[apdavison]

It looks to me as if consistent and intuitive overwriting behaviour will require NIXIO to give a unique ID to each object, so there should be no generate_primary_key option. In your example:

1. Load AnalogSignal with no name from a file in a format other than NIX.
2. Save AnalogSignal using NIXIO
3. Change some data in the AnalogSignal and save again

At stage 2 , NIXIO should create a unique ID for the AnalogSignal, and add it as an annotation to the AS object. At stage 3, you know it is the same object because the UID hasn't changed, so you can overwrite the DataArray if necessary.

[achilleas-k]

I see. Yes, I think that would work nicely.
To clarify (for myself as well as anyone else reading), the steps would be:

1. Load AnalogSignal with no name from a file in a format other than NIX.
2. Save AnalogSignal using NIXIO
   2.1. neo.AnalogSignal is converted to nix.DataArray
   2.2. DataArray name is assigned a unique ID (nix_id).
   2.3. AnalogSignal name is stored in DataArray's metadata as neo_name.
   2.4. nix_id from step 2.2 is assigned to AnalogSignal's annotations.
3. Change some data in the AnalogSignal and save again
   3.1. NIXIO reads the AnalogSignal nix_id annotation and overwrites the previous DataArray.
4. In a new session, the data is read from the NIX file
   4.1. The nix.DataArray is converted to a neo.AnalogSignal
   4.2. The AnalogSignal name is set using the neo_name property from the DataArray's metadata.
   4.3. The AnalogSignal nix_id annotation is set using DataArray's name.

[samuelgarcia]

I am back, to make things more complicated:

One neo object could belong to 2 NIX files with 2 diffrents ids (name here).
So the primary key must be "by context". Am I wrong ?

In that case the tuple (instance of NixIO, instance of neo object) give a unique key (name).

So 2 possiblities, using the function id() of python:

1. the neo object have dict of all NixIO instance (general case is one instance) to get the primary key.
2. the instance have dict of neo object to get the primary key.

I prefer the 2. In that case the nixIo instance is the context.

[apdavison]

I think that if you save the same Neo object to two different NIX files, it should have the same uid in both files.

[achilleas-k]

I'm not sure I understand the issue. The first time a Neo object is saved in a NIX file, it gets a key. As long as it has that key, if it's stored in a new NIX file, the key wont be generated; the NIX IO will just find the existing key and use it. So the object will retain its unique key throughout multiple saves and loads and across multiple files.

This might be an issue if an object is loaded from, say, a pickled file, stored in a NIX file, then loaded again, and stored in another NIX file. The two objects in the two NIX files will have different keys in this case.

[achilleas-k]

I came across an interesting case that I hadn't considered.
Consider the following:

block = neo.Block("blk")
st = neo.SpikeTrain(...)
for idx in range(10):
    seg = neo.Segment(str(idx))
    block.segments.append(seg)
    seg.spiketrains.append(st)

So we have one Block with one Segment and each Segment references the same SpikeTrain.
This isn't incompatible with nix, but what happens now, while writing the entire Block, is that the first time it comes across the SpikeTrain, it names it and stores it. The second time it comes across that same SpikeTrain, it finds the name and tries to write it again, instead of simply adding a reference to a Group (the analogue to a Segment). The current open Pull Request which implements this change doesn't cover it and will fail if this happens.

I'll update the branch of the open PR with a fix and add a test for it.