Building a Focused Search Index with YaCy

TL;DR: Small teams can make a local search index more useful by controlling which pages enter the crawl, how each page is scheduled, and how unfinished work resumes. While working on the coding project, I ran into this difficulty: turning a general search peer into a Canada-focused index while repeated URLs and memory pressure competed for limited capacity; the solution combined focused selection rules, a persistent crawl frontier, and bounded recovery.

From public peer to usable search

A public YaCy peer makes a locally collected index available as a search service. The project connected that peer to a separate search interface, then turned to the harder question: how should a small crawler choose which pages belong in a useful index? A broad crawl can keep discovering links long after an operator expects a collection to feel complete.

Project repository: yacy_search_server.

The goal became a Canadian-focused collection alongside the wider index. That required work at two layers. Search filters help readers choose a collection at query time; crawl profiles shape which pages enter each collection in the first place. A clear division between those jobs makes the system easier to tune: the crawler builds useful collections, while the search interface presents them to readers.

This distinction applies to many data systems. A dashboard filter can organize records already collected, while an intake policy controls which records enter the dataset. When those decisions are explicit, operators can explain why an item was collected, where it belongs, and when it should be revisited.

Making the index useful for Canadian search

A focused collection needs a repeatable way to classify discovered URLs. Here, a policy is the rule set that decides how a profile handles each URL. The YaCy work added rules that could evaluate a candidate, assign accepted pages to a collection, set priority, and plan a later recrawl. A recrawl is a scheduled revisit that checks for changes and keeps search results current.

One engineering lesson came from the order of decisions. A generic duplicate check can see a previously indexed page and stop it early. A focused profile may still need to reconsider that page under its own collection and refresh rules. Letting the profile policy evaluate those candidates preserves the chance to update relevant material while its rules control the added work.

The rollout also used a bounded pilot. A small test gives an operator a chance to inspect which candidates the policy accepts before widening a crawl. For any crawler, this creates a practical feedback loop: define the target, measure accepted work, review the result, then expand the scope with evidence.

Memory pressure appears across different crawling stacks, too. In a Crawlee discussion about a Playwright-based crawler, one operator described rising memory use, slowdown after roughly 1,000–1,500 sites, and an eventual freeze. That report shows a related operational concern in another system; YaCy's own investigation traced its stalls to its Java memory use and shared frontier locks.

Coverage limits create a separate risk. A GitLab issue about application-security crawling describes time, action-count, and depth limits that can stop exploration before adequate coverage while a job still appears successful. Together, the reports point to two useful checks for crawler operators: watch memory pressure during long runs, and track whether a crawl reached its planned scope.

Building focused crawling

The project turned these rules into focused-crawler profiles. A profile is a saved set of crawl goals and scheduling rules. Its policy can score a discovered page, assign it to a collection, select a priority lane (a queue group that decides which work runs sooner), and set a recrawl time. Multiple profiles can then pursue separate collections while using YaCy's existing fetching and indexing machinery.

Pull request: #839 — Focused Autocrawler Profiles was open in a GitHub API snapshot retrieved 2026-10-30. That PR record lists a last update of 2026-10-23, 74 commits, 118 changed files, and +11,388/−359. Its description reports 49 focused tests, 2 interruption-guard tests, 405 full-suite tests (0 failures, 0 errors, 1 skipped), a successful Docker build, a one-shot HTTP 200 smoke check.

Code history: focused crawl policy model and rule engine, weighted native crawl queue lanes, and persistent focused-crawl scheduler and recovery state.

The crawler's frontier is the list of pages waiting for a visit. The new persistent frontier records that work so a profile can continue after a restart. YaCy's native queue still handles downloading; the focused layer stores the profile's decisions and submits eligible work to that queue. This design extends an existing crawler while keeping its fetcher in charge of network work.

The data flow can be summarized as:

1. Discover a URL while processing a page.
2. Evaluate the URL against a profile's topic, region, and revisit rules.
3. Assign accepted work to a collection and priority lane, then record it in the persistent frontier.
4. Submit eligible work to YaCy's native queue and update the frontier as processing advances.

The implementation also handles recovery in bounded batches. A shared monitor is a lock that lets one operation at a time update protected frontier state. Full scans of a large queue can take time, so the project moved those scans outside the shared monitor, rechecked stale in-flight records before changing them, and recovered work in batches. That keeps queue admission and indexing callbacks responsive while maintenance examines a large frontier.

This structure echoes a broader research problem. A Microsoft Research paper on crawl scheduling studies how page importance, update frequency, request budgets, and per-host politeness shape decisions about what to revisit. Politeness means spacing requests so each website receives traffic at a manageable rate. The YaCy profiles provide a practical policy layer for making similar tradeoffs in a smaller, topic-focused collection.

Reliability, throughput, and what remained open

Long-running crawls need clear resource boundaries. The Java Virtual Machine (JVM) runs YaCy; its heap is the working memory available to the program. Session history traced stalls to JVM memory pressure and contention around the frontier. The project added resource guards and heap-recovery behavior so the crawler could pause during pressure and resume when capacity returned.

A soak test is an extended live-operation check that shows how a system behaves over time. Follow-up chronology (2026-10-21): the initial PR draft preceded the 24-hour soak. In a later update, the project reported completion of a 21-day soak and shared its results. The pull-request discussion records the operational follow-up. The persisted scheduler log covers about 28.5 calendar days and includes resource-guard pauses. The source records list the soak and log durations separately; pause accounting for the 21-day figure remains unspecified.

The PR's 2026-10-20 metrics snapshot counted 4,153,092 successful native-queue admissions (URLs accepted for work), 3,944,275 first-seen frontier discoveries (new links found), and 788,937 post-fetch policy decisions made after fetching and parsing, before storage in Solr. These are event counts; unique-URL totals and confirmed Solr writes call for separate measurements.

A separate read-only index audit on 2026-10-21 found 7,678,957 live documents, including 3,126,774 on .ca hosts (websites using Canada's .ca domain; 40.7%). The audit supplies an end-of-period snapshot; a starting baseline and per-document crawl attribution are unspecified. These snapshots describe corpus scale, while the profile-specific counters above describe crawl activity. A completed YaCy export discussed in the pull-request follow-up contained 7,534,100 total indexed records; the comment describes it as the full corpus, including earlier and other crawl work.

Code history: focused-frontier lock fix, and heap recovery before a crawl pause.

The branch also includes a separate search-safety change. A remote search can be cancelled while YaCy is committing metadata to Solr, the search index store. The code protects that commit from interruption, helping preserve the shared index writer that later crawler work depends on. Keeping this fix distinct from crawl policy makes the two reliability improvements easier to understand and review.

The wider lesson is to treat crawl selection, durable work tracking, resource limits, and measurement as one system. Start with a narrow profile, make its inclusion and revisit rules visible, record queue admissions separately from completed indexing, and report both active work and elapsed calendar time. That gives other operators a concrete way to build a focused collection, learn from its behavior, and expand it with confidence.