Phase 7: analytics rebuilt from the raw events every night
Dashboards now read aggregates only, and a nightly job reconciles those aggregates against the raw events so drift repairs itself.
Analytics screens now read pre-aggregated daily rows exclusively. No dashboard queries the raw scan or click tables.
Why aggregates
Reading millions of raw rows to draw a 30-day chart gets slower every week you succeed. Daily rows are written as events arrive, and the chart reads those.
The nightly rebuild
Live counters drift. A worker dies between two writes, a deploy lands mid-batch, and a number is quietly wrong from then on. Each night we:
- Recompute yesterday and today from the raw events
- Reconcile the counter on each QR code and link against the sum of its daily rows
- Roll finished days into monthly tables for long ranges
- Delete raw events past the retention window, keeping the aggregates
The reconciliation is the important part: the headline total on a code now always equals the sum of the chart below it, because if they disagree the job corrects the total.
Revenue stays in its own currency
INR and USD are reported side by side and never summed. Converting them for a single headline number would invent a figure that no bank statement will ever match.
Comparisons are the same length
Period-over-period growth compares against a window of exactly the same number of days, ending the day before your range. An off-by-one there quietly flatters every percentage on the page, which is worse than showing none.
Why not just fix the counter
The obvious alternative is to correct the running total when something changes: subtract the scans a newly-identified bot contributed, add back the ones wrongly excluded. It sounds cheaper, and it is, right up to the first time a correction is applied twice or dropped entirely.
Incremental corrections have to be perfect forever. Every one is a small mutation of a number nobody can independently verify, and an error made in March is silently carried through every report after it. There is no way to tell a correct total from a subtly wrong one by looking at it.
A rebuild has no such property. The raw event log is the record; the aggregates are a view derived from it. If the derivation is right, the numbers are right, and if it is wrong, fixing the code fixes every historical figure on the next run. Nothing accumulates.
What this costs
Storage, mostly. Raw events are kept for the whole retention window — 30 days on Free up to 730 on Premium — rather than being discarded once counted, and the rebuild is real work every night.
The trade is deliberate. Analytics that quietly drift are worse than no analytics, because people make decisions on them. Being able to say "these numbers were recomputed from the source last night" is the difference between a report and a guess.