Error `blknum: {"has already been taken", ...` while persisting blocks to DB

[unnawut]

Reported by a user running their own watcher.

2020-09-24 08:47:32.768 [info] module=Phoenix.Endpoint.CowboyAdapter function=start_link/3 ⋅Running OMG.WatcherRPC.Web.Endpoint with cowboy 1.1.2 at :::7534 (http)⋅,
2020-09-24 08:47:32.768 [info] module=Phoenix.Endpoint.Supervisor function=log_access_info/2 ⋅Access OMG.WatcherRPC.Web.Endpoint at http://ee3c68eb410f⋅,
2020-09-24 08:47:32.858 [info] module=OMG.Watcher.Monitor function=handle_cast/2 ⋅Got a health checkin... clearing alarm main_supervisor_halted.⋅,
2020-09-24 08:47:32.858 [info] module=OMG.Watcher.Monitor function=handle_event/2 ⋅Elixir.OMG.Watcher.Monitor got event: {:clear_alarm, {:main_supervisor_halted, %{node: :watcher_info@ee3c68eb410f, reporter: OMG.Watcher.Monitor}}}. Ignoring.⋅,
2020-09-24 08:47:32.858 [info] module=OMG.Eth.EthereumHeightMonitor.AlarmHandler function=handle_event/2 ⋅Elixir.OMG.Eth.EthereumHeightMonitor.AlarmHandler got event: {:clear_alarm, {:main_supervisor_halted, %{node: :watcher_info@ee3c68eb410f, reporter: OMG.Watcher.Monitor}}}. Ignoring.⋅,
2020-09-24 08:47:33.766 [info] module=OMG.WatcherInfo.DB.Block function=insert_from_pending_block/1 ⋅Block #3246000 not persisted in WatcherDB, done in 2.112ms⋅,
2020-09-24 08:47:33.766 [info] module=OMG.WatcherInfo.DB.Block function=insert_from_pending_block/1 ⋅Error in transaction current_block: [blknum: {"has already been taken", [constraint: :unique, constraint_name: "blocks_pkey"]}] %{}⋅,
2020-09-24 08:47:33.767 [error] module=gen_server function=error_info/7 ⋅GenServer OMG.WatcherInfo.PendingBlockProcessor terminating,
** (MatchError) no match of right hand side value: {:error, "current_block", #Ecto.Changeset<action: :insert, changes: %{blknum: 3246000, eth_height: 10821896, hash: <<73, 6, 3, 130, 194, 21, 18, 13, 163, 243, 162, 14, 231, 155, 221, 178, 239, 107, 221, 219, 252, 202, 191, 96, 60, 40, 12, 4, 75, 110, 40, 24>>, timestamp: 1599580079}, errors: [blknum: {"has already been taken", [constraint: :unique, constraint_name: "blocks_pkey"]}], data: #OMG.WatcherInfo.DB.Block<>, valid?: false>, %{}},
    (omg_watcher_info 1.0.3) lib/omg_watcher_info/pending_block_processor.ex:49: OMG.WatcherInfo.PendingBlockProcessor.handle_continue/2,
    (stdlib 3.12) gen_server.erl:637: :gen_server.try_dispatch/4,
    (stdlib 3.12) gen_server.erl:388: :gen_server.loop/7,
    (stdlib 3.12) proc_lib.erl:249: :proc_lib.init_p_do_apply/3,
Last message: {:continue, :process_block}⋅,

The user also mentioned that he was reaching Infura's daily quota limit prior to this error. So it may have something to do with the Eth call failing beforehand:

One more note that I want to add about the amount of calls to Infura, it may have to do with the watcher container constantly rebooting because of the error.