serialized data.tables from before cannot be read out of the box

[ben-schwen]

Since #7451 we see quite some revdep failures which are around serialization.

tmp = tempfile(fileext = ".rds")
download.file("https://raw.githubusercontent.com/cran/NMdata/master/inst/examples/data/xgxr2.rds", destfile = tmp, mode = "wb")
pk = readRDS(tmp)
pk[, trtact := NULL]
# Error in `[.data.table`(pk, , `:=`(trtact, NULL)) : 
#   Internal error in R_resizeVector_: attempt to resize a non-resizable vector. Please report to the data.table issues tracker.

pk = readRDS(tmp)
alloc.col(pk)
pk[, trtact := NULL]

I have not dived deeper into the issue, but I guess that these data.tables simply have no growable bit set and are therefore not recognized are resizable.

[aitap]

Yes, the old (pre-#7451) data.tables don't have it set at all. The new ones unserialize as "growable but TRUELENGTH=0" and work fine. Checking to see if adding a check for the growable bit to selfrefok will suffice.

Deleting columns used to be safe even if the data.table was not over-allocated, because SETLENGTH worked on all lists. R_resizeVector needs a resizable list now:

data.table/src/assign.c

Lines 665 to 666 in be67a4d

	R_resizeVector(dt, ndt-ndelete);
	R_resizeVector(names, ndt-ndelete);

Unserialized data.tables come out with TRUELENGTH=0; tables from the old version don't have the growable bit set; tables from the new version do.

This setalloccol:

data.table/R/data.table.R

Lines 1389 to 1423 in f6bbd5b

	if(!anyNA(m)) {
	# updates by reference to existing columns
	cols = as.integer(m)
	newnames = NULL
	} else {
	# Adding new column(s).
	newnames = setdiff(lhs, names_x)
	m[is.na(m)] = ncol(x) + seq_along(newnames)
	cols = as.integer(m)
	# ok <- selfrefok above called without verbose -- only activated when
	# ok=-1 which will trigger setalloccol with verbose in the next
	# branch, which again calls _selfrefok and returns the message then
	# !is.data.table for DF |> DT(,:=) tests 2212.16-19 (#5113) where a shallow copy is routine for data.frame
	if ((ok<1L) || (truelength(x) < ncol(x)+length(newnames))) {
	DT = x # in case getOption contains "ncol(DT)" as it used to. TODO: warn and then remove
	n = length(newnames) + eval(getOption("datatable.alloccol")) # TODO: warn about expressions and then drop the eval()
	# i.e. reallocate at the size as if the new columns were added followed by setalloccol().
	name = substitute(x)
	if (is.name(name) && ok && verbose) { # && NAMED(x)>0 (TO DO) # ok here includes -1 (loaded from disk)
	catf("Growing vector of column pointers from truelength %d to %d. A shallow copy has been taken, see ?setalloccol. Only a potential issue if two variables point to the same data (we can't yet detect that well) and if not you can safely ignore this. To avoid this message you could setalloccol() first, deep copy first using copy(), wrap with suppressWarnings() or increase the 'datatable.alloccol' option.\n", truelength(x), n)
	# #1729 -- copying to the wrong environment here can cause some confusion
	if (ok == -1L) catf("Note that the shallow copy will assign to the environment from which := was called. That means for example that if := was called within a function, the original table may be unaffected.\n")
	# Verbosity should not issue warnings, so cat rather than warning.
	# TO DO: Add option 'datatable.pedantic' to turn on warnings like this.
	# TO DO ... comments moved up from C ...
	# Note that the NAMED(dt)>1 doesn't work because .Call
	# always sets to 2 (see R-ints), it seems. Work around
	# may be possible but not yet working. When the NAMED test works, we can drop allocwarn argument too
	# because that's just passed in as FALSE from [<- where we know `*tmp*` isn't really NAMED=2.
	# Note also that this growing will happen for missing columns assigned NULL, too. But so rare, we
	# don't mind.
	}
	setalloccol(x, n, verbose=verbose) # always assigns to calling scope; i.e. this scope

...probably needs to happen when removing columns as well.

Test:

mtcars |> as.data.table() |> serialize(NULL) |> unserialize() |> _[, carb := NULL]