Task: Develop knowledge synchronization between websites and the central node
Develop knowledge synchronization between websites and the central node
Figure out how to keep the central information contour up to date without regular full re-reading of all websites.
The Problem
Even a good central index will quickly lose its meaning if updating it requires regularly re-reading all materials across all websites. When scaling from 10 to 100 websites, the synchronization cost should depend primarily on the volume of changes rather than the total size of the cluster.
Direction
We need to research incremental synchronization of local knowledge maps with the central node: how a website reports new, modified, and deleted information objects; how relevance is determined; what happens after temporary unavailability; and how desynchronization is detected.
The specific transport or protocol is not predefined. The goal is to establish reliable update semantics and state recovery.
Related to local map, central node, and cluster observability.
Direction Criterion
It should become normal for new or modified material to affect only a small part of the central state without forcing the system to re-explore the entire website.