Worklog for task "Add multilingual support"
Hypothesis: The Open Web Will Become a Competitive Advantage Again
Over the past fifteen years or so, the development of web infrastructure has moved toward increasingly restricting machine access.
The reasons were entirely rational. Bots generated load, scraped content, scanned for vulnerabilities, engaged in spam, copied databases, harvested prices, and created fake accounts. In response, websites gradually accumulated CDN, WAF, rate limits, JavaScript challenges, fingerprinting, CAPTCHA, anti-scraping, and behavioral analysis.
As a result, a paradox of the modern internet has emerged:
We created the World Wide Web for the free linking and dissemination of information, and then spent twenty years making that information as inconvenient as possible for automated reading.
For a Web where the primary consumer of a page was a human with a browser, this made sense.
I assume that with the advent of AI, this balance is beginning to shift.
The Bot Is Ceasing to Be Merely a Parasite
In the old economy of a public website, there was a fairly straightforward division:
Human → good visitorSearch crawler → tolerated because it brings humansOther bot → bad visitor
The latter had practically no economic value.
It fetched the page, consumed CPU and bandwidth, and bought nothing.
Therefore, the natural engineering strategy is:
do not let them in.
But AI creates a completely new class of machine consumer.
An AI crawler might read a site not to show its owner a stolen copy of the page, but to subsequently answer a human:
Where can I go on a day trip from Da Lat?
What is the difference between a porter and a stout?
How do you properly use a Finnish sauna?
And if a significant portion of human search indeed shifts from a list of links to a dialogue with AI, a fundamental shift occurs:
the bot becomes an intermediary between the publisher and the human.
This results in:
Old Web:Publisher ↓Google ↓SERP ↓Human ↓WebsiteAI Web:Publisher ↓Machine ↓understanding / synthesis ↓Human
And then an AI bot's request can no longer be automatically considered useless traffic.
Perhaps this is the top of a new user acquisition funnel.
Hence, the Paradox
The industry has spent huge amounts of money building infrastructure perfectly tailored for protecting information from machines, precisely at the moment when machines are becoming one of the primary ways of consuming information.
Moreover, the best and most commercially successful sites are often the most heavily protected.
Cloudflare, WAF, bot protection, dynamic rendering, authorization walls, rate limiting, JavaScript challenges.
In the old model, this is an advantage.
In the new model for a public information resource, part of this infrastructure potentially turns into a distribution handicap.
The most absurd situation might look like this:
A company has the best content in the industry, but the AI knows the competitor better because the competitor's site is easier to read.
Not because the competitor optimized keywords better.
Not because they have more backlinks.
But because their knowledge is physically accessible to a machine.
Second Paradox: Websites Have Learned to Deliver Cheap Information at Great Expense
There is another issue that I consider significant.
Modern web development has been optimized for human sessions for many years.
A single person opens a page, reads it for ten seconds, and clicks the next one.
Therefore, nobody is particularly bothered that fetching a single page triggers:
SSR↓application server↓5 API calls↓15 database queries↓personalization↓analytics↓third-party services↓render
A human is physically slow.
An AI crawler is not.
It is capable of telling the server:
GET page 1GET page 2GET page 3GET page 4GET page 5...
several times a second and keeping it up for hours.
And unexpectedly, it turns out that an architecture that served 1,000 humans brilliantly struggles to serve a single very inquisitive robot.
Therefore, simply saying:
"Fine, tomorrow we'll turn off Cloudflare protection and allow AI crawlers"
might turn out to be impossible.
Over years of a closed Web, many systems have lost the economic ability to be open.
My Bet Is the Opposite
For public knowledge/content projects, I consciously consider mass machine reading to be desirable behavior.
Therefore, the architecture must proceed from the assumption:
My site might be read not by thousands of humans, but by millions of machine requests. And that is a good thing.
Consequently, the marginal cost of serving public knowledge must tend toward zero.
If a crawler wants to read a thousand pages, let it read.
If several independent AI systems want to simultaneously download ten language versions of an encyclopedia, wonderful.
This is not a DDoS, as long as the behavior remains reasonable and the infrastructure can handle it.
This is distribution.
VietnamGuru — The First Experiment
On August 14, 2026, I published a new international domain vietnamguru.travel.
The domain is new.
At the same time, I consciously made it as simple as possible for machine discovery:
- standard indexable URLs;
- server-rendered HTML;
- proper
<a href>tags; - sitemap;
- canonical;
- hreflang;
- open language versions;
- connection to the old
vietnamguru.ru; - no artificial obstacles for normal crawlers.
And almost immediately after publication, various AI crawlers began exploring the new domain.
Some make multiple requests per second and systematically traverse related pages and language versions.
I am not trying to stop this behavior.
On the contrary, I see it as the first observable confirmation that the new distribution channel actually exists.
At the same time, the entire project runs on completely standard infrastructure: a small DigitalOcean server with 4 CPUs and 8 GB of RAM remains far from its performance limits under such crawling.
This is also part of the experiment.
My bet is not only that machines must be allowed to read.
It is also that:
a public knowledge resource must be cheap enough to maintain that it is economically viable to let machines read it aggressively.
But Crawling Proves Nothing by Itself
This is a fundamental caveat.
Today I am only observing:
Discovery.
I still have to test the following stages:
Discovery ↓Crawling ↓Understanding ↓Retrieval ↓Citation ↓Recommendation
The mere fact that GPTBot, PerplexityBot, or any other crawler arrives does not mean the site will gain an audience.
Therefore, the experiment must continue.
The next interesting question:
How soon after launching a brand new domain will independent AI systems be able to correctly answer questions using information from it?
An even stronger test:
When will they start using it for non-branded queries?
Not:
What is VietnamGuru?
but:
What waterfalls near Da Lat are worth visiting?
And finally, the most interesting level:
When will AI independently consider a resource useful enough to recommend it or use it as evidence among other sources?
If the Hypothesis Is Confirmed
Then the very concept of public website optimization will change.
In the previous generation, there was SEO:
help the search engine find the page so it brings a human to it.
In the next generation, a more fundamental task may emerge:
help the machine retrieve, understand, verify, and link your knowledge so it can use it when solving a human's problem.
This is no longer quite SEO.
And not even necessarily GEO/AEO in today's marketing sense.
This is machine accessibility as a property of an information system.
Then the competitive characteristics become:
accessibilitystructurabilityconnectednesssemantic clarityspeedstable URLscheap mass readingprovenancerefreshness
Meaning that many extremely boring engineering properties suddenly become properties of distribution.
And Here Another Paradox Appears
In the old Web, a site's value was partly measured by the number of people it managed to force to come to the site.
AI can destroy this metric.
A human may never open vietnamguru.travel.
They will ask their agent:
Is it worth going to Da Lat in August?
And the agent will read VietnamGuru, compare it with the weather, reviews, transport schedules, and five other sources, and give the human an answer.
From the perspective of Google Analytics:
0 visitors.
From the perspective of real impact:
VietnamGuru participated in the decision-making.
This results in a rather amusing situation:
the successful information site of the future could potentially become more influential while the share of people directly visiting its pages decreases.
And then many of today's Web metrics start measuring the wrong thing.
A More General Bet
My hypothesis is therefore not about VietnamGuru or specific AI crawlers.
It is this:
As we transition from the human-browsed Web to the AI-mediated Web, the ability of a public information system to be freely, massively, and cheaply read by machines will become a competitive advantage.
Today, a significant part of the industry out of inertia treats bot traffic as an expense or a threat.
I assume that for public knowledge resources, part of this traffic will become a distribution channel.
Therefore, a window of opportunity opens.
While other content owners are asking:
How do I ban AI from taking my content?
I want to test the opposite question:
What happens if you make your knowledge corpus one of the most convenient places for an AI wanting to understand your subject matter?
Maybe nothing.
Maybe AI platforms will build completely different mechanisms for acquiring knowledge.
Maybe publishers will indeed close up and a licensing economy will emerge.
Maybe today's crawlers will disappear altogether a year from now.
This is an experiment, not an established fact.
But my bet for August 2026 is simple:
If machines are becoming humanity's new interface to the internet, fighting every machine just because it is a machine is perhaps one of the last habits of the departing Web.
And I am consciously betting on the opposite:
For public knowledge, openness will once again become an advantage.