All work

Threadvault

A chat migration left 7,022 messages with the wrong author and every thread rejecting replies. The repair became an open-source CLI.

Role
Incident lead, author of the tool
Stack
Node.js CLI, Azure Communication Services, Postgres
Status
Open source on npm

What happened

  1. The move

    A production chat estate moved from one Azure Communication Services resource to another. 7,200 threads were replayed onto the new resource.

  2. The damage

    7,022 messages showed the system account as their author, and every replayed thread rejected replies with CommunicationError Forbidden (HTTP 403).

  3. The confusion

    Reads came from our database, writes went to ACS. So threads looked fine to read but would not accept a reply, and the 403 looked like an auth problem. It was not.

  4. The repair

    Recoverable only because the extract had kept each message's original sender and every thread's participant list. Nobody lost a message.

Why it broke

ACS identities are scoped to the resource that minted them: 8:acs:<resourceGuid>_<userGuid>. Move resources and every identity you stored becomes a string that refers to nothing, all at once, with no error at write time.

The replay sent every message as the only identity that existed on the new resource, the system user, and ACS cannot send on behalf of someone else or backdate authorship. It also created threads without re-adding participants, and only participants may post.

Behind those: no dry run, no resume, no verification step, and no supported way to export chat history from ACS at all.

What I built

  1. extractACS to a portable file
  2. planfind gaps before writing
  3. rehearseprove it on a throwaway thread
  4. applyresumable, dry run until --commit
  5. verifyread back and compare

npx threadvault doctor audits a live resource and database for the five failure modes from the incident: stale identities, system-only threads, misattributed messages, a missing system identity and split-brain threads. It is read-only, never reads a message body, and exits 1 if it finds anything.

What changed

Got a production problem worth solving?