review · dynamic mode · 2026-07-26

Blindspot pass: RFC 214, incremental indexing for corpus-index

Kind review Mode dynamic Created 2026-07-26 Owner Ines Vargas Status Draft Read time ~9 min Source canonical-example

In plain terms

What this is: a blindspot pass over RFC 214 before anyone starts building it. A blindspot pass looks only for what the plan did not account for; it does not argue about whether the plan is a good idea.
What it found: seven gaps. Two of them make the search index wrong on the first day of incremental writes, and one of those two returns restricted documents to every search.
What is being asked: hold RFC 214 until findings one and two have an answer written into the document. The other five are build-order and alerting work, and they can be scheduled rather than argued about.
What would change this: findings one and two rest on the ACL filter and the delete path being read correctly from RFC 214 rather than from the running system. If the RFC is out of date on either, those two drop a band. What this pass did not cover lists what was not read.

Nothing below argues against incremental indexing. The findings are all about the same thing: the nightly rebuild was quietly doing four jobs, and RFC 214 only replaces one of them.

Band 1: stop The index is wrong on day one 2 findings

A restricted document reads as unrestricted until the next Sunday Bites from the first incremental write. Every restricted document touched between rebuilds.
Archived documents stay searchable for up to seven days Bites on the first archive after cutover. Everything archived between two Sundays.

Band 2: fix in build Known before cutover, fixed during it 3 findings

A ticket re-stamp fills the bulk queue and events start dropping after about four minutes Bites during the next workflow migration. Silent index drift with no counter.
A new field waits for the Sunday window Bites on the first mapping change after cutover. One change, delayed up to seven days.
A revert applied before its edit leaves the old title in the index Bites when two edits land inside one second. Single documents, no self-repair.

Band 3: accept Real, and worth taking on purpose 2 findings

Tombstoned segments add about 1.9 GB a week between rebuilds Does not bite at the current corpus size. Disk headroom only.
Rolling back means 5h40m without a correct index Bites only if a rollback happens. Search quality for most of a working day.

Band 0: clear Probed and came back clean 5 checks

Checked and found nothing Query path, rebuild window, topic retention, index fan-out, node sizing.
What this pass did not cover Twelve smaller connectors unread, and the burst figure is not from a load test.

Only the first band changes the decision. The other five findings change the build order and the alerting, which is why they are written down here instead of held back for the implementation review.

The change under review

The nightly rebuild and the alias flip are the parts everyone already works with, so this section only exists to fix the vocabulary the findings use. The single new mechanism in RFC 214 is a consumer that reads the change stream and writes into the live index.

Today, rebuild_all starts at 01:00 UTC, reads all fourteen sources, builds that night's index corpus_v0725, hands off to acl_expand, and flips the corpus_read alias when both finish. RFC 214 keeps that job but runs it once a week, on Sunday, and lets a consumer on the corpus.changes topic apply upserts to the live index in between.

Current vs proposed indexing path

Today: rebuild every night

graph TB A[14 sources] --> B[rebuild_all 01:00] B --> C[corpus_v0725] C --> D[acl_expand] D --> E[flip alias corpus_read]

RFC 214: stream in between

graph TB F[14 sources] --> G[corpus.changes] G --> H[index consumer] H --> I[live corpus_read] J[Sunday: rebuild_all
then acl_expand] -.weekly.-> I

Diagram

With JavaScript off, the two blocks above stay as their source text and still read as two node chains. The same two paths are set out in the pair of boxes below them.

Today

Every document in corpus_read was written by one job, in one pass, from a single read of the sources, and then had its permissions filled in by a second job before the alias moved. Freshness is up to 24 hours behind.

RFC 214

Most documents in corpus_read were written by the consumer, one at a time, in whatever order the topic delivered them, with no second pass. Freshness is seconds behind.

The findings all come from the same seam: the nightly rebuild was a full read of the sources, a delete sweep, a mapping refresh, and the trigger for the permissions pass. Moving it to Sunday keeps the first job and drops the other three for six days at a time.

Band 1: stop

Both of these produce a wrong index rather than a slow one, and neither shows up in a metric that is currently collected.

A restricted document reads as unrestricted until the next Sunday

Band 1: stop Bites: first incremental write Radius: every restricted document between rebuilds

Assumed. RFC 214 lists visible_to among the fields the consumer writes, which reads as though permissions travel with the document like any other field.

Actually. The indexer has never written visible_to. It is filled in by acl_expand, which runs after the rebuild finishes, reads group membership out of directory_snapshot, and updates the already built index in a second pass (jobs/acl_expand.py:118). It takes an index name, not a document id, and there is nothing to call for a single upsert.

What breaks. The search filter shortcuts an absent field to unrestricted: SearchQueryBuilder.java:212 adds the permissions clause only inside an exists guard, so a document with no visible_to matches every query from every account. A restricted page edited on Tuesday is returned to everyone who searches for it until Sunday. It looks healthy the whole time, because it is indexed, it is fresh, and no check compares the field against the source.

Fix now: make acl_expand callable for one document and have the consumer call it, or refuse the write when visible_to is absent. Roughly three days, nearly all of it in the first of those two. Fix after cutover: the same three days, plus reading the search logs to work out which restricted documents were returned and to whom, which is the part with no upper bound.

Archived documents stay searchable for up to seven days

Band 1: stop Bites: first archive after cutover Radius: everything archived between two Sundays

Assumed. Every change to a document arrives on corpus.changes, deletes included.

Actually. The wiki and tickets sources, which are 94% of rows, never hard-delete anything. They set archived = true in place, and the connector drops archived rows before they reach the topic (connectors/wiki.properties:22). An archive therefore produces no event at all. The nightly rebuild hid this completely, because it read the sources with where archived = false and an archived page simply failed to appear in the next index.

What breaks. After cutover the only thing that removes a document is a full rebuild, so a page archived at 09:00 on Monday keeps being returned until the Sunday alias flip: the job starts at 01:00 and takes 5h40m, which puts the flip 5 days and 22 hours after the archive. A page archived just after a flip waits the full seven days. The bad case is not staleness. It is a page that was archived precisely because it should stop being read.

Fix now: stop excluding archived rows and treat archived = true as a delete in the consumer. About two days, including one replay of the last week of events. Fix after cutover: the same code, but there is no event trail for the archives that were already dropped, so finding them means diffing every index id against the source tables, which took a week the last time it was done for a single source.

Band 2: fix in build

These three are all cases where a number in RFC 214 is an average and the failure lives in the tail.

A ticket re-stamp fills the bulk queue and events start dropping after about four minutes

Band 2: fix in build Bites: next workflow migration Radius: silent drift until Sunday

Assumed. The consumer is sized for 40,000 events a day, which is the daily average over the last quarter.

Actually. The tickets connector re-stamps every open ticket whenever a workflow migration runs. The last one, on 2026-04-19, put 2.1 million events on the topic in 25 minutes, about 1,400 a second, against a sizing figure that averages out to under half an event a second.

What breaks. At that rate the write queue on the three indexer nodes fills (it is at the default of 200), the cluster starts rejecting individual bulk items with 429, and RFC 214 step 6 handles a rejection with three attempts and a fixed 200 ms wait before logging the event and moving on. The queue fills in about four minutes. Every event dropped after that is a document that stays stale until Sunday, and no counter exists that would let anyone say how many there were.

Fix now: a dead letter topic and a consumer lag alarm, about two days, both useful whether or not a re-stamp ever happens again. Fix after cutover: the same two days, bought after an incident where the honest answer to "how far off is the index" is that nobody can tell.

A new field waits for the Sunday window

Band 2: fix in build Bites: first mapping change after cutover Radius: one change, up to seven days

Assumed. Mapping changes are out of scope for RFC 214.

Actually. They were in scope of the nightly rebuild. Every new field and every analyser change has shipped by merging a mapping and letting that night's rebuild pick it up. The cluster will not change the analyser on an existing field in place, so once the rebuild is weekly there is no path between Sundays.

What breaks. A filter that needs one new keyword field goes from "merge it and wait a night" to "merge it and wait up to seven days". The predictable response to that wait is writing the value into an existing text field, which is how tickets.status_text ended up holding three unrelated things.

Fix now: keep the build-a-new-index-and-flip path as a job someone can trigger rather than a cron entry. The code is already in jobs/rebuild_all.py; it needs a trigger and a lock, about a day. Fix after cutover: someone reinstates it under time pressure without the alias flip, and every search hits a half-filled index for the 5h40m the build takes.

A revert applied before its edit leaves the old title in the index

Band 2: fix in build Bites: two edits inside one second Radius: single documents, no self-repair

Assumed. Applying events in the order they arrive converges on the source state.

Actually. corpus.changes is partitioned on a hash of source and batch id, not on document id, so two edits to one document can sit on different partitions and be applied concurrently by different consumer tasks.

What breaks. A rename and a revert 300 ms apart can be applied in reverse order, leaving the intermediate title in the index. Both events were consumed successfully, so there is no retry, and no later event exists that would correct it. It repairs itself on Sunday and not before.

Fix now: write with the source row's updated_at as an external version, so the cluster rejects a stale write instead of the outcome depending on which task ran first. Half a day, plus adding updated_at to the connector projection. Fix after cutover: the same change, except a stale entry and a genuinely old document look identical from outside, so every report of a wrong title turns into a manual diff against the source row.

Band 3: accept

Neither of these needs code. They need a number written into RFC 214 so the trade is taken on purpose.

Tombstoned segments add about 1.9 GB a week between rebuilds

Band 3: accept Bites: not at the current size Radius: disk headroom

Assumed. Nothing. RFC 214 does not mention disk, which is what makes this a blindspot rather than a disagreement.

Actually. An in-place upsert marks the previous copy deleted and writes a new one. The space comes back when segments merge, or when the index is rebuilt. At 40,000 upserts a day against a 41 GB index, a week of drift is roughly 1.9 GB before Sunday reclaims it.

Why it stays here. The cluster sits at 62% of 1 TB, so 1.9 GB is noise, and a force merge is available if it stops being noise. The finding is not the gigabytes. It is that the disk alarm thresholds were set when reclaim happened nightly.

Fix now: one sentence in RFC 214 and a threshold that assumes weekly reclaim. An hour. Fix after cutover: also an hour, unless the corpus has tripled first, in which case the number to argue about is 5.7 GB a week and the answer may go the other way.

Rolling back means 5h40m without a correct index

Band 3: accept Bites: only during a rollback Radius: search quality for most of a working day

Assumed. The rollback in RFC 214 is "stop the consumer, run rebuild_all". As a procedure that is complete and correct.

Actually. rebuild_all takes 5h40m, the mean of the last four nightly runs, with a range of 5h21m to 5h58m. For that whole stretch the live index keeps whatever state the consumer left behind, including the wrong permissions from the first finding if that is what triggered the rollback.

Why it stays here. The alternative is running the nightly rebuild in parallel for the first month, which spends the maintenance window RFC 214 exists to reclaim. Taking a slow rollback looks like the better trade, at least at the current corpus size.

Fix now: put the number in the rollback step and agree what triggers it. Half a day of discussion, no code. Fix after cutover: the number gets discovered while a rollback is already running.

What fixing each finding costs

Finding	Band	Fix before cutover	Fix after cutover
Restricted document reads as unrestricted	Band 1: stop	3 days	3 days plus a search log audit of unknown size
Archived document stays searchable	Band 1: stop	2 days	2 days plus about a week of id diffing
Re-stamp drops events	Band 2: fix in build	2 days	2 days, after an incident with no drift counter
No mapping path between Sundays	Band 2: fix in build	1 day	1 day, or 40 minutes of half-filled index if improvised
Reverse-order edit sticks	Band 2: fix in build	0.5 day	0.5 day plus a manual diff per report
Weekly tombstone growth	Band 3: accept	1 hour	1 hour
Slow rollback	Band 3: accept	0.5 day of discussion	Discovered mid-rollback

Fixing the whole list before cutover is about nine days of work, and only five of those days sit in front of the decision; the rest can run alongside the build.

Checked and found nothing

A pass that reports only hits says nothing about where it looked, so these are the things that were probed and came back clean.

The query path needs no change. The search service reads through the corpus_read alias and never names a concrete index. All 19 call sites of indexName( were read, and the alias flip stays invisible to every one of them.
The Sunday window fits. A 5h40m rebuild inside the 8h maintenance window leaves 2h20m spare. Taking the run as linear in corpus size, the window holds to about 1.4 times the current corpus and stops holding before twice it.
Topic retention survives an outage. corpus.changes retains 7 days, checked against the live topic config rather than against the figure quoted in RFC 214. The longest outage the plan contemplates is 48 hours, so a restart replays instead of skipping ahead.
There is no index fan-out to break. One cluster, one alias, no per-source or per-account index. The delete and ordering findings therefore stay single-document problems and do not multiply.
Node sizing holds outside the burst case. At 40,000 events a day the consumer adds about 4% CPU on the existing three indexer nodes, so no new instances are needed unless the burst fix changes the shape of the work.

Coverage here is the query path, the window, retention, topology, and steady-state sizing; nothing in this list depends on the two stop-band findings being fixed, so it stays valid whichever way RFC 214 goes.

The checks, as run

$ rg -n 'indexName\(' services/search --stats     # 19 matches, 0 literal index names
$ curl -s "$OS/_cat/aliases/corpus_read?v"
$ curl -s "$OS/_cluster/settings?include_defaults" | jq '.defaults.thread_pool.write'
$ kafka-configs --describe --topic corpus.changes | grep retention.ms   # 604800000
$ psql -c "select count(*) from wiki_pages where archived"              # 1.4M archived rows

The 2026-04-19 burst figure comes from the indexer dashboard for that day, not from these commands.

What this pass did not cover

Only two connectors were read in full, wiki and tickets. They are 94% of rows, so the other twelve could still hold a delete-semantics surprise like the one in the second finding.
The 1,400 events a second figure is measured, but the four-minute time to failure is arithmetic against a queue depth of 200 and not a load test against the new consumer.
acl_expand was read, not run. Its per-document cost is unknown, so the three-day estimate on the first finding could be wrong in either direction.
Nothing here evaluates whether incremental indexing is worth doing. That question was settled before this pass started.

The ordering of the bands is judgment about how badly each finding bites, not a measurement, and two people ranking the same seven would probably swap the re-stamp finding and the mapping finding.