MyXQL: allow use of INSERT IGNORE for on_conflict: :nothing

[mehiel]

The default on_conflict: :nothing behavior uses ON DUPLICATE KEY UPDATE x = x which has the following characteristics:

• Reports 1 row affected even when a row was skipped due to a duplicate key conflict, because the UPDATE clause still matches the existing row:
• Returns {1, nil} instead of {0, nil} for ignored duplicates
• The UPDATE clause could trigger unnecessary row-level operations

For users who need accurate row counts (0 when ignored, 1 when inserted), INSERT IGNORE can be explicitly enabled via:

Repo.insert_all(Post, posts,  on_conflict: :nothing,  insert_mode: :ignore_errors)

With INSERT IGNORE:

• Properly ignores duplicate key conflicts without affecting existing rows
• Returns accurate affected row count (0 for duplicates)
• The adapter handles num_rows: 0 by returning {:ok, []} since 0 rows is expected behavior when all rows are duplicates

Implementation
Following @josevalim's suggestion, this adds an opts parameter to the Connection behavior's insert/8 callback. The SQL.insert_all passes opts through to the connection, and the MyXQL adapter uses insert_mode: :ignore_errors to generate INSERT IGNORE. Other adapters (Postgres, TDS) accept and ignore this option.

Note
INSERT IGNORE has broader semantics in MySQL - it ignores certain type conversion errors in addition to duplicate key conflicts - which is why it's not the default behavior.

ORIGINAL POST

The previous implementation used ON DUPLICATE KEY UPDATE col = col which had incorrect semantics:

1. It reported 1 row affected even when a row was skipped due to a duplicate key conflict, because the UPDATE clause still matched the existing row
2. This caused insert_all to return {1, nil} instead of {0, nil} for ignored duplicates, misrepresenting the actual number of inserted records
3. The UPDATE clause could trigger unnecessary row-level operations

This change:

• Uses INSERT IGNORE INTO which properly ignores duplicate key conflicts without affecting existing rows or incrementing the affected row count
• Handles num_rows: 0 in the adapter by returning {:ok, []} for on_conflict: :nothing, since 0 rows is expected behavior when all rows are duplicates
• Updates tests to verify correct row counts: {0, nil} for all-duplicates, {N, nil} for N successfully inserted non-duplicate rows

[mehiel]

I guess this means at least a PATCH version. This should be considered a breaking change. Or we can somehow put it behind a configuration with the default being the existing behavior.

[greg-rychlewski]

I don't believe it is as simple as this. Please take a read here: https://dev.mysql.com/doc/refman/8.4/en/sql-mode.html#ignore-effect-on-execution

For example this part:

INSERT: With IGNORE, rows that duplicate an existing row on a unique key value are discarded. Rows set to values that would cause data conversion errors are set to the closest valid values instead.

For partitioned tables where no partition matching a given value is found, IGNORE causes the insert operation to fail silently for rows containing the unmatched value.

The issue seems to be that MySQL doesn't have any equivalent to conflict_target in Postgres. So now you have to choose between two suboptimal methods. The current one will at least still cause an error if an error unrelated to the conflict target happens.

[mehiel]

Yes, I read the docs but for some reason the data conversion part didn't trigger. There are 3 parts in this section:

1. ✔️ rows that duplicate an existing row on a unique key value are discarded - main reason to use it here
2. ❌ rows set to values that would cause data conversion errors are set to the closest valid values instead - this is sneaky
3. ✔️-ish for partitioned tables where no partition matching a given value is found, IGNORE causes the insert operation to fail silently - acceptable with on_conflict: :nothing

So if we have to choose our poison here we're indeed between 2 suboptimal solutions when we provide on_conflict: :nothing

1. accept wrong num_rows - we even have a unit test adding the same post twice and getting back num_rows 1 on the second insert
2. accept automatic type conversions - happens when we set on_conflict: :nothing and there is no conflict on a unique key and a bad query finds its way eg. inserting 'abc' to an integer column will make it with value 0 (ecto may prevent some of these but still)

I think preventing 2 is more important for a core piece like ecto_sql. I would love to have a way to configure it even on the query level because invalid num_rows is also a huge pain and usually related with use cases that need on_conflict: :nothing in the first place.

Thanks for pointing that out @greg-rychlewski.

[mehiel]

@greg-rychlewski did a last try to make this opt-in under a certain conflict_target value. This is the only option that allowed me to do this without changing Ecto internals.

I understand it looks like a magic value but check this out and let me know what you think.
It's completely fine to not like it 😄

[josevalim]

Yeah, :unsafe_fragment is meant to literally be a string that we insert, so I am a bit skeptical about special casing it. Perhaps the best way is to have a separate option that only MySQL handles on its insert, something like: insert_mode: :ignore_errors and then we document that it also affects the behaviour of ON CONFLICT NOTHING.

[mehiel]

Perhaps the best way is to have a separate option that only MySQL handles on its insert, something like: insert_mode: :ignore_errors

Yeah, tried that but if I read this correctly it requires change to the Connection behavior since it doesn't take opts there. I'll give it another try.

[josevalim]

You can probably add a default arguments opts \\ [] to MyXQL implementation only and invoke it.

[mehiel]

You can probably add a default arguments opts \\ [] to MyXQL implementation only and invoke it.

I couldn't really do that because insert_all is coming from SQL. Tried to make this as straight forward as possible and add opts to the behavior and then just ignore it on PG and TDS implementations.

[greg-rychlewski]

I think this can be a little bit misleading. It can effect the behaviour of any insert query because we are injecting it without looking at what conflict option was chosen.

So maybe rewriting this to say it changes the default sql for :nothing would be clearer. And probably having a specific example showing it with the option on and off to really drive the point home.

[greg-rychlewski]

Oh wait you are checking the conflict option when injecting it. Then I think there is a separate question if we should do that. Because IGNORE is an independent modifier in MySQL. We should probably allow it to be added whenever the user wants or raise if we want to only allow this one use case.

Personally I like the former. What do you think @josevalim ?

[josevalim]

Good call. We should probably have a section exclusively on the :insert_mode option and in there say it affects the behaviour of on_conflct: :nothing as well, as the mean feature here is no longer the upsert itself.

[josevalim]

We should probably allow it to be added whenever the user wants or raise if we want to only allow this one use case.

Yes, exactly. We should allow insert_mode everywhere, regardless of :on_conflict, and then document that it affects the on_conflict behaviour too.