Worklog for task "Add local Markdown compatibility component"

New case: legacy HTML normalization before MDX/Markdown rendering

When importing content from old CKEditor, a new category of incompatibility emerged: even after eliminating nested <p> tags, some HTML remains syntactically valid for the browser, but survives MDX parsing and subsequent SSR rendering poorly.

Symptoms

The MDX parser crashed with errors like:

Error parsing markdown: Expected the closing tag `</p>` either after the end of `paragraph` ...
Error parsing markdown: Expected the closing tag `</div>` either after the end of `paragraph` ...

A typical legacy fragment looked like this:

<p style="text-align:justify"><img alt="" src="/ckeditor_assets/pictures/40/content_oil.png" style="border-style:solid; border-width:1px; height:503px; width:703px"><br>
<span style="font-family:trebuchet ms,helvetica,sans-serif; font-size:14px">Text...</span></p>

Here, several problems are combined simultaneously:

• <br> is written in HTML form, but the MDX/JSX pipeline expects a self-closing variant;
• the old WYSIWYG actively uses <p> as a universal container;
• line breaks and whitespace between HTML tags can be interpreted by MDX no longer as neutral formatting of the source, but as boundaries of markdown paragraphs.

What was tested

Initially, normalization via htmlparser2 + dom-serializer was considered, but this option turned out to be inconvenient for the task: self-closing tags were not normalized the way MDX requires, and text could be serialized with undesirable HTML entities.

A working option turned out to be rehype, which allows first bringing legacy HTML to a more stable serialization, and then passing the result to the general Markdown normalization pipeline.

Current pipeline

Normalization now consists of several sequential steps.

1. Legacy <p> are replaced with <div>

This eliminates invalid paragraph nesting and reduces the risk of hydration mismatch:

function replacePWithDiv(html: string): string {
  const $ = cheerio.load(html, { xml: false }, false)

  $('p').each((_, el) => {
    const $el = $(el)
    const div = $('<div></div>')

    div.html($el.html() || '')

    for (const attr of el.attributes) {
      div.attr(attr.name, attr.value)
    }

    $el.replaceWith(div)
  })

  return $.html()
}

2. HTML is passed through rehype

rehype is used as a parser/stringifier to fix and canonize legacy markup:

const normalized = await rehype()
  .data('settings', {
    fragment: true,
    closeSelfClosing: true,
  })
  .process(html)

The key effect is that self-closing elements like <br> are brought into a form that passes through MDX/JSX processing more safely.

3. Whitespace between adjacent tags is removed

After serialization, line breaks and spaces between HTML tags are additionally collapsed:

html = String(normalized).replace(/>\s+</g, '><')

This is important specifically for MDX: a line break between inline/block HTML nodes can affect markdown parsing and generate an unexpected paragraph boundary. In browser HTML, such whitespace is often irrelevant, but in a mixed Markdown+HTML format, it is no longer the case.

Final implementation

The logic is gathered into the cleanupOldContent helper, which works as a separate normalization layer between legacy CKEditor HTML and the new Concept/Markdown system:

import * as cheerio from 'cheerio'
import { rehype } from 'rehype'
import { normalizeMarkdownContent } from '../../KBConcept/helpers/normalizeMarkdownContent'

function replacePWithDiv(html: string): string {
  const $ = cheerio.load(html, { xml: false }, false)

  $('p').each((_, el) => {
    const $el = $(el)
    const div = $('<div></div>')
    div.html($el.html() || '')

    for (const attr of el.attributes) {
      div.attr(attr.name, attr.value)
    }

    $el.replaceWith(div)
  })

  return $.html()
}

export async function cleanupOldContent(html: string): Promise<string> {
  html = html.replaceAll(/<u>([^<]+)?<\/u>:/g, '<u>$1:</u>')
  html = replacePWithDiv(html)

  try {
    const normalized = await rehype()
      .data('settings', {
        fragment: true,
        closeSelfClosing: true,
      })
      .process(html)

    html = String(normalized).replace(/>\s+</g, '><')

    return await normalizeMarkdownContent(html)
  } catch (error) {
    console.error(error)
    console.error('cleanupOldContent html', html)
    throw error
  }
}

Architectural conclusion

This case shows that the local compatibility layer for BiznesHelper must normalize not only Markdown whitespace, but also embedded legacy HTML before content hits the MDX/SSR pipeline.

That is, the actual import path now looks like this:

legacy CKEditor HTML
→ DOM-level cleanup
→ HTML normalization via rehype
→ whitespace normalization
→ general normalizeMarkdownContent
→ Concept saving

This is a useful separation of concerns: browser-tolerant, but unstable legacy markup is fixed once at the entrance, rather than endlessly compensated for in the public renderer.

Status

The specific case with line breaks between tags and self-closing HTML is now closed by a local normalization helper. The general compatibility issue of MDXEditor / Markdown renderer remains a subject of a separate R&D task in haih-agent.