When a security hardening broke the data
QuorX 0.2.5 locked down its data folder so tightly that QuorX itself could no longer read it. Projects and chats seemed to vanish; nothing was lost. What happened, how 0.2.6 repaired it an hour later, and what every release does now.
QuorX 0.2.5 was called "Locked down by default". It made QuorX much harder to attack. It also locked QuorX out of its own data. For about an hour, people who updated saw their projects, chats and terminal seem to disappear. Nothing was lost, and 0.2.6 repaired it. This is what happened and what changed.
What went wrong
0.2.5 locked the data folder on Windows so only the user's account and the system can read it. That folder holds chats, projects, settings and keys. The rule was meant for the folder. It was applied with icacls to every file inside it as well (the /T switch), with inheritance flags that only make sense on a folder. On a file, those flags left no permission entries at all.
QuorX treated "can't read the file" like "no file yet", so the empty project list looked normal. That made it worse: it looked like data loss, not an error. And the update reached installed apps through auto-update, straight to everyone.
The fix, an hour later
- 1Probe before readingOpen app-key, projects, keys, access and one chat
- 2Any refused: repairReset the files to inherit from the folder, no administrator rights needed
- 3Start as usualAnd if the project list can't be read, say so
Only the folder is locked now, and its files inherit that.
The first version of the repair checked a single file. But a file rewritten since 0.2.5 already inherits the right entries, so one readable file doesn't mean the rest are. 0.2.6 probes several. A test now locks a data folder the 0.2.5 way, then checks that QuorX repairs it at start and that the new lock-down keeps files readable.
Why the tests missed it
No test locked a data folder and then read it back the way QuorX does at start. A lock-down that is too strict looks like success from the outside: the files were very private. Nobody could read them, QuorX included.
What every release does now
- A "can we still read it?" test for any lock-down change: lock it, then read it back as the app would.
- Staged releases. A new version is first a pre-release that installed apps don't take, and it reaches them only when promoted. A bad one can be pulled. See Shipping safer.
- Upgrades tested from real old data. For 0.2.7, the old version itself made a data folder with a project and three chats. 0.2.7 then started on it, and the project list and every chat's messages were byte for byte the same.
Pull request: #34 (0.2.6). The longer version is in docs/research/lockdown-postmortem.md.