Worklog for task "Redesign the website structure, refine it, and rebuild the content"

Added a really cool AI feature :-) - content editing with internal cross-linking. The main highlight is passing the entire set of internal links to the agent, and it automatically replaces plain text with links within the content. But as usual, the AI doesn't work quite correctly at times and invents non-existent links. However, this is fixable - I simply added post-processing for the content, and it cleans up the content from broken internal links (fortunately, the content is entirely in Markdown, which makes this much easier than with pure HTML).

import { unified } from 'unified'
import remarkParse from 'remark-parse'
import remarkMdx from 'remark-mdx'
import remarkStringify from 'remark-stringify'
import { visit } from 'unist-util-visit'
import type { Link, Text } from 'mdast'
import type { Parent } from 'unist'

export interface ConceptLink {
  uri: string
}

export interface InvalidLink {
  url: string
  position?: {
    start: { line: number; column: number }
    end: { line: number; column: number }
  }
}

export interface ValidateInternalLinksResult {
  invalidLinks: InvalidLink[]
}

function isInternalLink(url: string): boolean {
  return url.startsWith('/')
}

export function buildValidUrisSet(concepts: ConceptLink[]): Set<string> {
  return new Set(concepts.map((c) => c.uri))
}

export async function removeInvalidLinks(
  content: string,
  validUris: Set<string>,
): Promise<string> {
  const tree = unified().use(remarkParse).use(remarkMdx).parse(content)

  visit(tree, 'link', (node: Link, index, parent: Parent | undefined) => {
    if (!parent || index === undefined) {
      return
    }

    const url = node.url

    if (isInternalLink(url) && !validUris.has(url)) {
      if (process.env.NODE_ENV === 'development') {
        console.error('Invalid url', url)
      }

      const textNode: Text = {
        type: 'text',
        value: node.children
          .filter((child): child is Text => child.type === 'text')
          .map((child) => child.value)
          .join(''),
      }

      parent.children.splice(index, 1, textNode)
    }
  })

  const result = unified()
    .use(remarkStringify, {
      bullet: '-',
      emphasis: '_',
      strong: '*',
    })
    .stringify(tree)

  return result
}

Here is an example of a processed article: https://vietnamguru.ru/content/reference/aktualniy-spisok-provintsiy-vetnama

It was originally written without a single internal link. Yet there should be many links there, as it is a list of provinces (and sometimes cities as well). Editing it manually by yourself is quite tedious (you not only need to find the links on your own website, but also check if such a province actually exists in our catalog). By the way, there is a side bonus here - we immediately got a list of missing provinces, making it instantly clear what needs to be added to the catalog. And so we ended up with content like this: