Data quality investigations
A CDP is only as good as what enters it. Enterprise clients push customer events from web SDKs, mobile apps, POS systems and batch files.
PROBLEM
Downstream symptoms — wrong segment sizes, personalisation that misfired, loyalty balances that disagreed with the client's own system — all traced back to ingest quality, but were reported as separate bugs.
HYPOTHESIS
Most 'wrong data' complaints originate at a small number of ingest paths with weak schema enforcement.
RESEARCH
Traced individual customer records end to end from source event through to activation, and compared record counts and field completeness at each hop.
DATA USED
Raw ingest batches · schema validation rejects · identity resolution output · trait tables
DECISION MAKING
Argued for validation at the boundary rather than correction downstream, since every downstream fix multiplies across consumers of the same record.
SOLUTION
Stricter ingest validation on the worst-offending paths, plus visibility into rejects so clients could see what was being dropped and why.
BUSINESS OUTCOME
A whole class of 'the data is wrong' escalations became self-explaining, and identity resolution accuracy improved on the affected accounts.
REFLECTION
Data quality is invisible until it isn't. Making rejects visible did more for trust than any dashboard.
SQL INVESTIGATION
SELECT source,
count(*) FILTER (WHERE email IS NULL) AS missing_email,
count(*) FILTER (WHERE phone IS NULL) AS missing_phone,
count(*) AS total
FROM raw_profiles
WHERE ingested_at >= now() - INTERVAL '30 days'
GROUP BY source
ORDER BY total DESC;