Custom Markdown Syntax

Area: Text formats and parsing
Related: Markdown, Regular Expressions, Sandboxed iframes, Web Workers

What is it?

Custom Markdown syntax is any construct your app understands that CommonMark doesn’t: [[wiki links]], ==highlights==, > [!warning] callouts, @mentions, :::tabs containers, {{ variables }}. Every notes app, docs site and chat product ends up adding a few.

There are four places you can hook in, from cheapest to most correct:

Before parsing: rewrite the text with regexes.
After parsing: walk the tree and rewrite nodes.
During parsing: teach the tokenizer a new construct.
At render time: change how existing nodes become HTML.

Add syntax at the layer that already knows the context. A regex doesn’t know it’s inside a code block. The tokenizer does.

Why it’s harder than it looks

Markdown has no error states and lots of context (see Markdown). A new construct has to answer the same questions the built-in ones do:

Does it work inside code spans and fenced code? (It shouldn’t.)
Can it contain emphasis, links or other inlines? Can it span lines?
What wins when it overlaps with existing syntax? [[x]] already means something in CommonMark: a bracket followed by a possible link reference.
What happens when it’s unclosed, like [[broken?
How does it look in tools that don’t know it?

All examples below were run on Node 22 with markdown-it 15.0.3, marked 18.1.1, micromark 4.0.3 and mdast-util-from-markdown. The test input was:

See [[Graph Theory|graphs]] and [[BFS]].
 
`[[not a link]]`
 
```
[[also not]]
```

1. Regex before parsing (and why it breaks)

const out = src.replace(
  /\[\[([^\]|]+)(?:\|([^\]]+))?\]\]/g,
  (_, target, alias) => `[${alias ?? target}](/notes/${encodeURIComponent(target)})`,
);

Real output:

See [graphs](/notes/Graph%20Theory) and [BFS](/notes/BFS).
 
`[not a link](/notes/not%20a%20link)`
 
```
[also not](/notes/also%20not)
```

The first line works. Then it rewrote inside a code span and a code block, which a note about wiki-link syntax will contain. The regex has no idea what context it’s in, and you can’t fix that without re-implementing code block detection, at which point you’re writing a parser.

It’s fine for throwaway scripts on text you control. Don’t ship it.

2. Transform the tree after parsing

Parse with a normal parser, then walk the AST and split text nodes that contain your pattern. With mdast (the remark tree):

import { fromMarkdown } from 'mdast-util-from-markdown';
import { visit, SKIP } from 'unist-util-visit';
 
const tree = fromMarkdown(src);
visit(tree, 'text', (node, index, parent) => {
  const re = /\[\[([^\]|]+)(?:\|([^\]]+))?\]\]/g;
  const parts = [];
  let last = 0, m;
  while ((m = re.exec(node.value))) {
    if (m.index > last) parts.push({ type: 'text', value: node.value.slice(last, m.index) });
    parts.push({
      type: 'link',
      url: '/notes/' + encodeURIComponent(m[1]),
      children: [{ type: 'text', value: m[2] ?? m[1] }],
    });
    last = re.lastIndex;
  }
  if (!parts.length) return;
  if (last < node.value.length) parts.push({ type: 'text', value: node.value.slice(last) });
  parent.children.splice(index, 1, ...parts);
  return [SKIP, index + parts.length]; // don't revisit the nodes we just inserted
});

Result for the first paragraph: text "See ", link /notes/Graph%20Theory, text " and ", link /notes/BFS, text ".". The code span and code block are inlineCode and code nodes, not text, so they’re left alone for free.

This is how a lot of remark plugins work, and it’s often good enough. Two real limits:

Your syntax gets parsed as Markdown first. [[A *b* c]] becomes three nodes: text, emphasis, text. No single text node contains [[...]], so the regex never matches.
Existing syntax can steal it. [[BFS]] on its own is one text node. But add [BFS]: /x anywhere in the document and the parser turns the inner [BFS] into a linkReference. Your wiki link is now text + link + text.

When either case matters, move down a layer.

3. Teach the tokenizer

This is the proper way. The new construct is recognised during inline parsing, with the same precedence rules as built-ins. All three big JS parsers support it, with very different APIs.

markdown-it: rule chains

markdown-it runs ordered lists of rules called rulers. You can see them (real output):

core:    normalize block strip_references inline linkify replacements smartquotes text_join
block:   table code fence blockquote hr list reference html_block heading lheading paragraph
inline:  text linkify newline escape backticks strikethrough emphasis link image autolink html_inline entity
inline2: balance_pairs strikethrough emphasis fragments_join

At each position the inline parser tries rules in order, and the first one that returns true wins. backticks is before link, which is why code spans are safe. Put a wiki-link rule before link so [[ isn’t grabbed as a normal bracket:

import MarkdownIt from 'markdown-it';
 
function wikiLinks(md: MarkdownIt) {
  md.inline.ruler.before('link', 'wikilink', (state, silent) => {
    const start = state.pos;
    if (state.src.charCodeAt(start) !== 0x5b /* [ */ ||
        state.src.charCodeAt(start + 1) !== 0x5b) return false;
 
    const end = state.src.indexOf(']]', start + 2);
    if (end < 0) return false;                       // unclosed: fall through to normal parsing
    const inner = state.src.slice(start + 2, end);
    if (!inner || inner.includes('\n')) return false;
 
    if (!silent) {
      const [target, alias] = inner.split('|');
      const open = state.push('wikilink_open', 'a', 1);
      open.attrSet('href', '/notes/' + encodeURIComponent(target.trim()));
      open.attrSet('class', 'wikilink');
      const text = state.push('text', '', 0);
      text.content = (alias ?? target).trim();       // text tokens are escaped by the renderer
      state.push('wikilink_close', 'a', -1);
    }
    state.pos = end + 2;
    return true;
  });
}
 
const md = MarkdownIt().use(wikiLinks);

Output for the test input:

<p>See <a href="/notes/Graph%20Theory" class="wikilink">graphs</a> and <a href="/notes/BFS" class="wikilink">BFS</a>.</p>
<p><code>[[not a link]]</code></p>
<pre><code>[[also not]]
</code></pre>

Things to know:

silent mode. markdown-it sometimes calls rules only to ask “would you match here?”, for example while finding where a link label ends. In silent mode you must advance state.pos and return true, but not push tokens.
Return false cleanly on anything you don’t fully match. Another rule (or plain text) then gets a turn.
Rendering is automatic for tokens with a tag. For custom output, add md.renderer.rules.wikilink_open = (tokens, idx, opts, env, self) => ....
Block syntax goes in md.block.ruler, and rules there work on lines (state.bMarks, state.eMarks), not characters. Plugins like markdown-it-container already handle ::: name blocks. Use them before writing your own.
Paired delimiters like ==mark== should use the delimiter mechanism (push to state.delimiters, resolve in a ruler2 rule), the way strikethrough does. That’s how ==a **b== c** nests correctly. markdown-it-mark is a working example to copy.

marked: extensions with `start`

marked’s extension object has a tokenizer and a renderer:

import { Marked } from 'marked';
 
const escape = (s: string) =>
  s.replace(/[&<>"']/g, c => ({ '&': '&amp;', '<': '&lt;', '>': '&gt;', '"': '&quot;', "'": '&#39;' }[c]!));
 
const marked = new Marked({
  extensions: [{
    name: 'wikilink',
    level: 'inline',
    start: src => src.indexOf('[['),          // where a match could begin
    tokenizer(src) {
      const m = /^\[\[([^\]|\n]+)(?:\|([^\]\n]+))?\]\]/.exec(src);
      if (m) return { type: 'wikilink', raw: m[0], target: m[1].trim(), text: (m[2] ?? m[1]).trim() };
    },
    renderer: tok =>
      `<a class="wikilink" href="/notes/${encodeURIComponent(tok.target)}">${escape(tok.text)}</a>`,
  }],
});

start matters. marked’s text tokenizer eats characters until something interesting might begin. start tells it where your construct could begin. I tested a ==highlight== extension:

without start:  <p>See ==this== now</p>
with start:     <p>See <mark>this</mark> now</p>

The wiki-link version happened to work without start, because [ already stops the text tokenizer. Don’t rely on that. Always provide start.

The renderer returns raw HTML. marked doesn’t escape what you return. Leave out escape() above and [[<img src=x onerror=alert(1)>]] is an XSS. This is the most common bug in custom Markdown plugins.

micromark: a state machine

micromark (the engine under remark and MDX) is a character-by-character state machine. An extension is a construct: a tokenize function that returns state functions. Each one gets the current character code and either consumes it and returns the next state, or says ok (matched) or nok (not this construct, try something else).

import { codes } from 'micromark-util-symbol';
 
const wikiLink = { name: 'wikiLink', tokenize };
 
function tokenize(effects, ok, nok) {
  let size = 0;
  return start;
 
  function start(code) {          // called on the first "["
    effects.enter('wikiLink');
    effects.enter('wikiLinkMarker');
    effects.consume(code);
    return open;
  }
  function open(code) {
    if (code !== codes.leftSquareBracket) return nok(code);
    effects.consume(code);
    effects.exit('wikiLinkMarker');
    effects.enter('wikiLinkTarget');
    return inside;
  }
  function inside(code) {
    if (code === codes.eof || code === codes.leftSquareBracket ||
        code === codes.lineFeed || code === codes.carriageReturn ||
        code === codes.carriageReturnLineFeed) return nok(code);
    if (code === codes.rightSquareBracket) {
      if (size === 0) return nok(code);
      effects.exit('wikiLinkTarget');
      effects.enter('wikiLinkMarker');
      effects.consume(code);
      return close;
    }
    size++;
    effects.consume(code);
    return inside;
  }
  function close(code) {
    if (code !== codes.rightSquareBracket) return nok(code);
    effects.consume(code);
    effects.exit('wikiLinkMarker');
    effects.exit('wikiLink');
    return ok;
  }
}
 
// Syntax extension: try this construct whenever inline text hits "["
export const wikiSyntax = { text: { [codes.leftSquareBracket]: wikiLink } };

nok rewinds automatically. micromark replays the characters for the next construct, so [[broken with no closing brackets just becomes text. You don’t write any backtracking yourself.

Syntax only produces events (enter/exit with positions). You then add a separate extension for each output:

// HTML, for micromark(src, { extensions: [wikiSyntax], htmlExtensions: [wikiHtml] })
const wikiHtml = {
  exit: {
    wikiLinkTarget(token) { this.setData('wikiTarget', this.sliceSerialize(token)); },
    wikiLink() {
      const t = this.getData('wikiTarget');
      this.tag(`<a class="wikilink" href="/notes/${encodeURIComponent(t)}">`);
      this.raw(this.encode(t));
      this.tag('</a>');
    },
  },
};
 
// mdast, for fromMarkdown(src, { extensions: [wikiSyntax], mdastExtensions: [wikiFromMarkdown] })
const wikiFromMarkdown = {
  enter: { wikiLink(token) { this.enter({ type: 'wikiLink', value: '' }, token); } },
  exit: {
    wikiLinkTarget(token) { this.stack[this.stack.length - 1].value = this.sliceSerialize(token); },
    wikiLink(token) { this.exit(token); },
  },
};

Real output:

micromark('See [[BFS & DFS]], [normal](/x), `[[code]]`, [[broken')
→ <p>See <a class="wikilink" href="/notes/BFS%20%26%20DFS">BFS &amp; DFS</a>, <a href="/x">normal</a>, <code>[[code]]</code>, [[broken</p>
 
fromMarkdown('a [[BFS]] b')
→ text "a ", wikiLink "BFS", text " b"

Two mistakes I made getting this working, both worth knowing:

My first HTML version used this.buffer() / this.resume() to grab the text, then encoded it again. That gave BFS &amp; DFS. The buffered text is already encoded. Read the raw source with sliceSerialize instead.
My first version wrapped the target in a chunkString token so escapes would be processed. The mdast compiler then crashed (Cannot read properties of undefined (reading 'length')), because the data events had no parent node with children. If your node has text content parsed as Markdown, give it children. If it’s a plain value, don’t emit nested content tokens.

micromark is the most work. In return you get exact CommonMark behaviour, positions on every node, and the same extension driving HTML, mdast, MDX and serialisation. For remark you’d also write a toMarkdown handler so the node prints back as [[BFS]].

4. Change only the renderer

If the syntax already parses, you just need different HTML. Common cases:

// markdown-it: open external links in a new tab
const defaultLinkOpen = md.renderer.rules.link_open
  ?? ((tokens, idx, opts, env, self) => self.renderToken(tokens, idx, opts));
md.renderer.rules.link_open = (tokens, idx, opts, env, self) => {
  const href = tokens[idx].attrGet('href') ?? '';
  if (/^https?:\/\//.test(href)) {
    tokens[idx].attrSet('target', '_blank');
    tokens[idx].attrSet('rel', 'noopener noreferrer');
  }
  return defaultLinkOpen(tokens, idx, opts, env, self);
};

Other uses: syntax highlighting in fence (render ```mermaid as a diagram), heading ids for anchors, lazy-loaded images. In React, react-markdown takes a components map that swaps any element for your own component.

Callouts are a good example of a tree transform plus a renderer. > [!warning] Careful is already a valid blockquote. A plugin checks whether a blockquote’s first paragraph starts with [!type], strips that marker and tags the node, and the renderer turns it into a styled <div>. No tokenizer work needed. In other apps it’s still a readable quote. That’s why Obsidian and GitHub both picked this syntax.

Choosing the layer

You want	Layer
Different HTML for existing syntax	Renderer
Decorate existing blocks (callouts, heading ids, link rewriting)	Tree transform
Simple inline pattern in plain text (`#tags`, `@mentions`), contents not Markdown	Tree transform, accepting the edge cases
New delimiters, precedence over links/emphasis, multi-line blocks	Tokenizer
Same syntax in several outputs (HTML, AST, editor, back to Markdown)	Tokenizer in micromark, or the editor’s own parser

Designing the syntax itself

The parsing is the easy part to change later. The syntax lives in users’ files forever.

Degrade gracefully. Ask how it looks on GitHub or in a plain text editor. [[Page]] and > [!note] read fine. %%{color:red}%% doesn’t.
Reuse existing conventions before inventing one. Obsidian’s wiki links and callouts, Pandoc’s {.class} attributes and ::: divs, and the generic directives proposal (:name[label]{attrs}, ::name, :::name) already have parsers.
Avoid characters that already mean something. [ starts links, * and _ emphasis, < HTML, ! images, ~ strikethrough in GFM, $ math in many tools.
Define the edge cases up front. Unclosed, empty, nested, multi-line, escaped (\[[not a link]]). Write them as fixtures, the way the CommonMark spec does.
Plan the round trip. If an editor saves the document, your node needs a serializer that prints it back the way it was typed.

Editors are a separate parser

A rich editor (ProseMirror, Tiptap, Lexical, CodeMirror) doesn’t use your HTML renderer. It has its own document model and usually its own Markdown import and export:

ProseMirror’s prosemirror-markdown parses with markdown-it tokens and maps them to editor nodes. Custom markdown-it tokens need matching entries there.
CodeMirror 6 uses @lezer/markdown, which has its own extension format (defineNodes, parseInline, parseBlock) and parses incrementally.

So one custom construct can mean three implementations: the editor, the renderer and the serializer. Keep one set of fixtures and run every implementation against it, or they’ll drift.

A real-world example

A collaborative notes app that stores notes as plain .md files:

Wiki links [[Page|alias]] are a tokenizer extension, so they’re ignored in code and work next to emphasis. Unknown targets render as “create page” links.
Callouts > [!note] are a blockquote transform, so the files still look right on GitHub.
Mentions @name are a tree transform over text nodes, with the actual user id kept in a link URL so a display name change doesn’t break them.
Diagrams are fenced blocks (```mermaid) handled in the renderer and drawn in an isolated frame.
The editor has its own node types for all four, and a serializer that writes them back in the same spelling.
Every plugin’s output goes through the sanitiser, and the shared fixture file includes XSS attempts for each construct.

Gotchas

A regex over the raw text matches inside code. Always.
Plugin order matters. In markdown-it, a rule after link never sees [[.
marked extensions need start or the text tokenizer swallows your syntax.
Custom renderers that build HTML strings must escape. Parsers don’t do it for you.
[[x]] is ambiguous in CommonMark. A reference definition [x]: ... changes how it parses before your tree transform runs.
Tree transforms only see text nodes, so syntax containing emphasis or code splits across nodes.
An unclosed construct should fall back to text, not throw and not swallow the rest of the paragraph.
Your plugin can make a linear-time parser quadratic. The markdown-it rule above calls indexOf(']]') on every [, which is O(n²) on a line of [. In my test native indexOf hid most of it (80,000 brackets: 114 ms plain, 250 ms with the plugin), but heavier per-position work won’t be that lucky. Limit how far a rule looks, and put a long run of your trigger character in your fuzz tests.
Editor, renderer and serializer each need the extension. Missing one means data loss on save.

Practice

Add ==highlight== to markdown-it using the delimiter API, and make sure ==a **b== c** doesn’t produce broken HTML.
Write #tag support as an mdast transform. Make sure # headings, URLs with #fragments and C# aren’t tags.
Port the wiki-link micromark construct to support [[Page#Heading|alias]] and write a toMarkdown handler so it round-trips.
Implement callouts as a remark plugin, then render them with react-markdown components.
Write a fixture file with 30 inputs (unclosed, nested, escaped, inside code, XSS). Run your markdown-it and micromark versions against it and diff the results.
Feed your plugin 100,000 [ characters and time it. Fix it if it’s quadratic.

Sources worth reading

markdown-it architecture doc and plugin list: https://github.com/markdown-it/markdown-it/blob/master/docs/architecture.md
markdown-it-mark source, for a delimiter-based inline rule
marked extensibility docs: https://marked.js.org/using_pro
micromark “Creating a micromark extension”: https://github.com/micromark/micromark#creating-a-micromark-extension
unified handbook and remark plugin list: https://unifiedjs.com/
Generic directives proposal: https://talk.commonmark.org/t/generic-directives-plugins-syntax/444
@lezer/markdown docs (extending the CodeMirror parser): https://github.com/lezer-parser/markdown

Things

Explorer

Custom Markdown Syntax

What is it?

Why it’s harder than it looks

1. Regex before parsing (and why it breaks)

2. Transform the tree after parsing

3. Teach the tokenizer

markdown-it: rule chains

marked: extensions with `start`

micromark: a state machine

4. Change only the renderer

Choosing the layer

Designing the syntax itself

Editors are a separate parser

A real-world example

Gotchas

Practice

Sources worth reading

Graph View

Table of Contents

Backlinks

Things

Explorer

Custom Markdown Syntax

What is it?

Why it’s harder than it looks

1. Regex before parsing (and why it breaks)

2. Transform the tree after parsing

3. Teach the tokenizer

markdown-it: rule chains

marked: extensions with start

micromark: a state machine

4. Change only the renderer

Choosing the layer

Designing the syntax itself

Editors are a separate parser

A real-world example

Gotchas

Practice

Sources worth reading

Graph View

Table of Contents

Backlinks

marked: extensions with `start`