数据湖自身会提供一些元数据的信息，我们可以利用这些信息进行快速粗略的数据校验
分区数据数校验
以Iceberg为例，catalog.db.tb2.partitions可以得到分区的元数据，里面包括了分区下数据条数和文件大小等信息
select a.*, b.*
from (
select partition.date, sum(record_count) record_count, 'old' flag
from catalog.db.tb1.partitions
group by partition.date
) as a
full join (
select partition.date, sum(record_count) record_count, 'new' flag
from catalog.db.tb2.partitions
group by partition.date
) as b
on a.date = b.date
where a.record_count is null or b.record_count is null or a.record_count != b.record_count
网友评论