The append-only log is the load-bearing part of this architecture. It is what lets a server-authoritative system answer "what happened, in what order, to whom" as a query rather than an investigation, and every argument on this site about incident response, audit and processing integrity rests on it. Removing rows from it is removing the thing that made those arguments true.
Meanwhile the GDPR says two things that sound fatal to it. Article 17 gives a data subject the right to have their personal data erased. Article 5(1)(e) says personal data must be kept in identifiable form no longer than is necessary for the purpose it was collected for. The first is a request you receive. The second applies whether or not anyone ever asks.
The second one is the sharp one
Most discussion of this topic fixes on the right to erasure, which is the wrong place to worry. Article 17 is qualified: paragraph 3 carves out processing necessary for compliance with a legal obligation, for the establishment or defence of legal claims, and for freedom of expression. A controller with a genuine, documented reason to keep a record has room to keep it, and an audit trail required by another regulation is a reasonable example of one.
Storage limitation has no such escape hatch, and it needs no trigger. If the purpose has ended and the data is still identifiable, you are already in breach, on a Tuesday, with no complaint filed and nobody looking. That is the principle an append-only log violates by default, because "append-only" and "forever" get treated as the same word.
What in a log is personal data
This is where most of the tension dissolves, and it dissolves because of a distinction people skip past. A row in an event log has two parts that matter here: an identifier that ties it to a person, and a payload describing what happened. Recital 30 is explicit that an online identifier — a cookie value, a session token — is personal data. So the row is personal data because of the identifier, before anyone has looked at the payload.
In a system that minimises properly, the payload on its own is frequently not personal data at all. "An item was added to a cart", "a slot was booked", "a workspace was created" describe events, not people. Strip the identifier and what remains is a statement about the system rather than about a data subject. Article 5(1)(e) asks for data to stop being kept in a form which permits identification, and that is a lower bar than deleting the row.
The append-only property that carries the weight is that history is never rewritten. That is a promise about revision, not about retention. A log can refuse to change what it recorded and still stop keeping the part that names someone.
Three ways to reconcile them
- A retention window
- Delete or de-identify rows once they pass an age you can defend. It is the least clever option and it addresses the principle that binds without a request. The work is choosing a period you can justify from the purpose rather than from convenience, and then actually running the job, on a schedule, in production. A retention policy nobody has automated is a document, not a control.
- Tombstones and a projection rebuild
- Append an erasure event and rebuild the read models so the subject disappears from everything a user or an operator can see. This is the option most often recommended in event-sourcing circles, and it deserves a warning: rebuilding projections removes the person from the views, and leaves every original row sitting in the log. The regulator's question is about the data you hold, not the data you display. Use tombstones for ordering and causality, and pair them with something that reaches the log itself.
- Crypto-shredding
- Encrypt each subject's payloads under a key held separately, and destroy the key to make the data unreadable while the rows stay physically in place. It is a real technique with a real cost: a key store to run, key loss to survive, and an argument to make about whether encrypted data whose key is gone counts as erased. Reach for it when the log genuinely must keep every row — a financial ledger, a regulated audit trail — rather than as a default.
What this site does
This site took the first option, because its log records demo interactions and written feedback rather than anything a regulator or a court will ask for later. A pass runs at start-up and every six hours. Events older than thirty days lose their session identifier or their row, depending on which they are.
-- Feedback survives; the part that named someone does not.
UPDATE events SET session_id = ''
WHERE occurred_at < ? AND session_id NOT IN ('', ?) AND type IN (?);
-- Everything else past the window goes.
DELETE FROM events
WHERE occurred_at < ? AND session_id NOT IN ('', ?) AND type NOT IN (?);Thirty days is defensible here because the session cookie expires after twenty-four hours. Past that, the visitor cannot reach their own rows and neither can anyone else on their behalf, so the identifier has outlived every purpose it was issued for. Feedback keeps its text because the argument someone took the trouble to write is the reason the form exists, and detaching it from a session costs that argument nothing.
The second clause in each statement excludes two things. Rows already detached are skipped, which makes the pass idempotent. Demo fixture rows are skipped too: the healthcare demo preseeds booked slots into the same table, and those describe no visitor at all. Finding that out was the useful part of writing the job — a retention pass built without looking would have quietly emptied a working demo thirty days after launch.
Where this still does not work
Clearing an identifier is not anonymisation. Recital 26 sets the bar at whether re-identification is reasonably likely by any means, and a free-text field clears that bar whenever somebody types their own name into it. This site's feedback box is exactly such a field. The honest description of what the pass achieves is that it removes the identifier the system attached, and leaves whatever the visitor attached themselves.
Backups are the second gap, and the one most often missed. A row deleted from the live database is still in last night's snapshot, and in the one before that, for as long as the backup retention says. A controller who answers an erasure request by running a DELETE has answered it for the primary copy only. The usual resolution is to document that backups are restored-then-re-erased rather than edited in place, which is a policy, not a query.
And the architecture decides none of the surrounding questions. It does not establish your lawful basis, set your retention period, or judge whether an Article 17(3) exception applies to a record you would rather keep. Those are decisions someone has to make and write down, and an event log is only ever the place they get enforced.
What survives all of this is a smaller claim than the one usually made for append-only storage. Never rewriting history is a property worth defending. Never removing anything is a habit that grew up next to it, and the two get confused because the same word covers both. A log that expires its identifiers on a schedule is still an audit trail. A log that keeps them because nobody wrote the job is a compliance finding with good intentions.