[Bug] Spark engine fails to generate +U events in changelog when using partial-update/aggregation and lookup changelog-producer

[david-shi-bi]

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

1.3.1

Compute Engine

Spark 3.5.2

Minimal reproduce step

CREATE TABLE test_partial_lookup (
id INT,
name STRING,
age INT
) TBLPROPERTIES (
'merge-engine' = 'partial-update',
'changelog-producer' = 'lookup',
'primary-key' = 'id'
);

spark.sql("insert into test_partial_lookup values(1, "test", 20)").show(10, False)

only +I rowkind

spark.sql("select * from test_partial_lookup$audit_log").show(10, False)

check compaction event

spark.sql("select * from test_partial_lookup$snapshots order by snapshot_id desc").show(10, False)

update the available record to get the updated event

spark.sql("update test_partial_lookup set name = 'test1' where id = 1").show(10, False)

check updated record

spark.sql("select * from test_partial_lookup").show(10, False)

only +I rowkind after updating

spark.sql("select * from test_partial_lookup$audit_log").show(10, False)

check compaction event

spark.sql("select * from test_partial_lookup$snapshots order by snapshot_id desc").show(10, False)

What doesn't meet your expectations?

Yes, I want to capture the updated event from the change log using spark

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[david-shi-bi]

Hello @JingsongLi , this issue seems to be quite important to us. Could you check whether this issue is possible to fix?