I have refered to :h diffview-config-view.x.layout and looked at the config, but there is no setting to disable the file panel by default.
As far as I can see a reasonable way is to create a hook when DiffView opens, and then run:
hooks = { -- See ':h diffview-config-hooks'
view_opened = function()
require("diffview.actions").toggle_files()
end,
},