test_decoding transform: skip toast columns with unchanged values

[hanefi]

test_decoding plugin may not include the value of a TOAST column if it has not changed. In that case, instead of the value of the column the string "unchanged-toast-datum" is included in the output. In that case, we should remove the column from the range table entry list in the generated SQL queries instead of including it and setting that column to the value "unchanged-toast-datum".

There is one strange case that if a table has only toast columns, and an update query that does not update any value is executed, we end up filtering all the range table entries and skip executing any sql queries as it will be no-op.

Fixes: #350
Fixes: #710

[dimitri]

It looks like it could be worth adding a SQL query to the tests to show that the values are indeed “uncompressed external” or “compressed external”. I can't remember how to do that, but I think it is possible to check at run-time?

[hanefi]

It looks like it could be worth adding a SQL query to the tests to show that the values are indeed “uncompressed external” or “compressed external”. I can't remember how to do that, but I think it is possible to check at run-time?

Checking for storage type:

select att.attname,
       case att.attstorage
          when 'p' then 'plain'
          when 'm' then 'main'
          when 'e' then 'external'
          when 'x' then 'extended'
       end as attstorage
from pg_attribute att
  join pg_class tbl on tbl.oid = att.attrelid
  join pg_namespace ns on tbl.relnamespace = ns.oid
where tbl.relname = 'xpto'
  and ns.nspname = 'public'
  and not att.attisdropped
order by 1;

+--------------+------------+
|   attname    | attstorage |
+--------------+------------+
| cmax         | plain      |
| cmin         | plain      |
| ctid         | plain      |
| id           | plain      |
| rand1        | plain      |
| rand2        | plain      |
| tableoid     | plain      |
| toasted_col1 | extended   |
| toasted_col2 | extended   |
| xmax         | plain      |
| xmin         | plain      |
+--------------+------------+

#define  TYPSTORAGE_PLAIN		'p' /* type not prepared for toasting */
#define  TYPSTORAGE_EXTERNAL	'e' /* toastable, don't try to compress */
#define  TYPSTORAGE_EXTENDED	'x' /* fully toastable */
#define  TYPSTORAGE_MAIN		'm' /* like 'x' but try to store inline */

---

Checking for column compression:

select id,
       pg_column_compression(toasted_col1) toasted_col1_compression,
       pg_column_compression(toasted_col2) toasted_col2_compression
from xpto
order by 1;

+----+--------------------------+--------------------------+
| id | toasted_col1_compression | toasted_col2_compression |
+----+--------------------------+--------------------------+
|  1 | (null)                   | (null)                   |
|  2 | (null)                   | pglz                     |
+----+--------------------------+--------------------------+

It turns out that the comments were partially wrong. We have extended storage instead of external. I guess the 11 year old PG tests are also misleading as I copied the table definitions from them. https://github.com/postgres/postgres/blob/82b07eba9e8b863cc05adb7e53a86ff02b51d888/contrib/test_decoding/sql/toast.sql#L17-L21

[dimitri]

Merged as it is, please see if you want to add tests coverage for actual storage options of the columns in the unit tests. The goal I have in mind is to make sure that the actual Postgres behavior conforms with the unit tests expectations: limits could change in a new major version of Postgres in a way that we need to adjust the test data.