Markdown

Area: Text formats and parsing
Related: Custom Markdown Syntax, Sandboxed iframes, Clipboard and DataTransfer, Web Workers

What is it?

Markdown is a plain-text format for writing documents that turn into HTML. You write **bold**, # Heading and - item, and a parser turns that into <strong>, <h1> and <ul><li>. The source stays readable as is, and that’s the whole design goal.

It isn’t one language. It’s a family of closely related dialects that agree on the common 95% and disagree on the edges. Most of the hard engineering around Markdown lives in those edges.

Markdown is an agreement about how plain text looks, and parsing it means guessing what the writer meant from how it looks.

That’s why the parser is harder than it seems. In a programming language a syntax error is an error. In Markdown there are no errors. Every string of characters is a valid document, so the parser always has to pick some meaning, even for **foo*.

Why it exists

In 2004 John Gruber (with input from Aaron Swartz) published Markdown as a syntax description plus a Perl script, Markdown.pl. The goal was text you could write in an email or a text editor that would still look right before conversion. It borrowed conventions people already used in plain-text email: *emphasis*, > for quoting, - for lists.

The problem: the description was prose, not a spec. It didn’t say what happens with nested lists indented by 3 spaces, or whether a list can interrupt a paragraph, or what *foo**bar* means. Every new implementation (PHP Markdown, Python-Markdown, Showdown, Redcarpet, and dozens more) made its own guess, and Markdown.pl itself had bugs people copied. The same document rendered differently on every site. The Babelmark tool exists purely to show you how badly 30+ implementations disagree on one input.

Then sites started adding features the original never had: tables, fenced code blocks, strikethrough, task lists, footnotes. Each one invented its own syntax.

The standards

Standard	What it is	Who uses it
Gruber’s original (2004)	Prose description + Markdown.pl	Historical. Nobody should target it today.
CommonMark (2014 on)	A precise spec with hundreds of numbered input/output examples, plus reference parsers cmark (C) and commonmark.js	The base of almost every modern parser
GFM (GitHub Flavored Markdown)	CommonMark + tables, task lists, strikethrough, extended autolinks, a raw HTML tag filter	GitHub, GitLab-ish, most dev tools
RFC 7763 / 7764 (2016)	Registers the text/markdown media type with a variant parameter	Content negotiation, rarely seen in practice
Pandoc Markdown	A big superset: footnotes, citations, definition lists, attributes, math	Academic writing, document conversion
MDX, Markdoc, MyST	Markdown plus a component or directive system	Docs sites

CommonMark is the one that matters. It started in 2014 as “Standard Markdown”, a project by John MacFarlane (who wrote Pandoc) with people from Stack Overflow, GitHub, Reddit and others. Gruber objected to the name and it became CommonMark. The spec is still versioned 0.x (0.31.2 at the time of writing) because it keeps resolving edge cases, but it has been stable in practice for years.

The spec’s value is the examples. Each one is a Markdown input and the exact HTML it must produce. A parser that passes them all is “CommonMark compliant”, and you can test any implementation against them mechanically.

GFM is a formal spec too, written as a diff on top of CommonMark. GitHub moved its rendering to cmark-gfm, a fork of the reference C parser, in 2017. Some existing READMEs broke, mostly ones that relied on #Heading with no space (CommonMark requires the space).

What isn’t standardised anywhere, even though everyone uses it:

• Front matter, the YAML block between --- lines at the top. A CommonMark parser sees a thematic break, then the YAML plus the closing --- as a setext heading. Tools strip it before parsing.
• Math ($x^2$), footnotes (only partly in GFM), wiki links ([[Page]]), callouts (> [!note]), heading ids, emoji shortcodes.

The syntax in two halves

Every Markdown parser thinks in two kinds of structure, and the split explains most of how parsing works.

Block structure is the page layout: paragraphs, headings, block quotes, lists, code blocks, tables, thematic breaks (---), HTML blocks. Blocks are decided line by line, mostly by what’s at the start of the line and how far it’s indented.

Inline structure is what happens inside a block’s text: emphasis, strong, code spans, links, images, autolinks, raw HTML, hard line breaks, entities like &.

Blocks can contain blocks (a list item containing a quote containing a code block). Inlines live only in “leaf” blocks like paragraphs and headings. A code block’s content is never inline-parsed.

How a CommonMark parser works

The spec describes a parsing strategy in an appendix, and most serious parsers follow its shape. It’s two passes.

Pass 1: blocks, one line at a time

The parser keeps a tree of open blocks: document → maybe a block quote → maybe a list → list item → paragraph. For each new line:

1. Match continuations. Walk down the open blocks and check whether the line continues each one. A block quote needs a >. A list item needs enough indentation. A paragraph continues if the line isn’t blank.
2. Open new blocks. Look at what’s left of the line. Does it start a heading (#), a fence (```), a quote (>), a list marker (-, 1.)? Open those.
3. Close the rest. Blocks that weren’t continued get closed.
4. Add the text to the deepest open leaf block. Paragraph text is stored raw. It isn’t inline-parsed yet.

Some cases this has to handle:

• Lazy continuation. In > a followed by b (no >), b still belongs to the quoted paragraph. You can be sloppy, and the spec allows it.
• Setext headings. Title followed by --- is an <h2>, but --- alone is a thematic break, and - -- might be a list. The meaning of a line depends on what came before it.
• Tabs count as tab stops of 4 columns, not as one character, so indentation maths isn’t simple string length.
• HTML blocks have seven different start conditions (a <script>, a comment, a known block tag like <div>, and so on), each with its own end condition.

Pass 2: inlines, per leaf block

Once the whole document’s blocks are known, the parser runs the inline parser over each paragraph and heading.

Why wait? Link reference definitions. This is legal:

See [the docs].
 
[the docs]: https://example.com

The definition comes after its use. You can’t know [the docs] is a link until you’ve seen the whole document. So pass 1 collects definitions, pass 2 resolves them.

The inline parser scans left to right. Code spans (backticks) win first, since nothing inside them is parsed. For emphasis and links it uses a delimiter stack:

1. When it sees a run of * or _, or a [, it pushes a delimiter entry and writes the characters as plain text for now.
2. When it sees ], it looks back for a matching [ and checks whether (url) or a reference follows. If so, the text in between becomes a link.
3. At the end, the “process emphasis” step pairs openers with closers and rewrites the text nodes into <em> and <strong>.

Whether a * run can open or close depends on its neighbours. A left-flanking run is followed by a non-space and can open. A right-flanking run is preceded by a non-space and can close. _ has extra rules so snake_case_names don’t get emphasised. That was a classic complaint about the original script.

There’s also the “rule of 3”: if a delimiter can both open and close, an opener and closer only match if the sum of their run lengths isn’t a multiple of 3 (unless both are). That rule is why this works:

*foo**bar*   →  <em>foo**bar</em>
**foo*       →  *<em>foo</em>

I ran those through commonmark.js, markdown-it, marked and micromark. All four agree, which is the point of a spec.

Output: events, tokens or a tree

Parsers differ most in what they hand you:

Shape	What you get	Examples
HTML string	Rendered output, nothing in between	marked.parse, micromark()
Flat token stream	heading_open, inline, heading_close… with nesting levels	markdown-it
Event stream (pull)	An iterator of start/end/text events	pulldown-cmark (Rust), micromark internally
AST	A real tree of nodes with positions	mdast (remark), commonmark.js, cmark

markdown-it’s tokens for # Hi *there*\n\n- a look like this (real output):

heading_open h1  map [0,1]
inline           map [0,1]  "Hi *there*"   ← children hold em_open, text, em_close
heading_close h1
bullet_list_open ul  map [2,3]
list_item_open li
paragraph_open p
inline               "a"
...

The map is the source line range. You need it for scroll sync between an editor and a preview.

An AST like mdast is easiest to transform, because you can walk and rewrite a tree. A token stream is easier to render fast. An event stream uses the least memory and suits huge documents.

The parsers you’ll meet

Parser	Language	Spec	Notes
cmark / cmark-gfm	C	CommonMark / GFM	Reference. Linear time by design. Bindings everywhere.
commonmark.js	JS	CommonMark	Reference JS port. AST + HTML renderer.
markdown-it	JS	CommonMark + plugins	Token stream. Big plugin ecosystem. Safe defaults.
marked	JS	Mostly GFM	Small and fast on normal input. Extension API. No sanitising.
micromark	JS	CommonMark exact	State machine. Base of remark and MDX.
remark (unified)	JS	via micromark	Markdown → mdast → plugins → HTML (rehype) or back to Markdown
@lezer/markdown	JS	CommonMark + GFM	Incremental. Powers Markdown in CodeMirror 6 editors.
pulldown-cmark, comrak	Rust	CommonMark / GFM	Pull parser / AST
goldmark	Go	CommonMark	Used by Hugo
Python-Markdown, mistune, markdown-it-py	Python	varies	Python-Markdown predates CommonMark and isn’t compliant
Pandoc	Haskell	its own superset	Converts between dozens of formats

Where parsers still disagree

I ran some inputs through four JS parsers (commonmark.js 0.31.2, markdown-it 15.0.3, marked 18.1.1, micromark 4.0.3 with GFM). The core syntax agreed everywhere. These didn’t:

Input	commonmark.js	markdown-it	marked	micromark
<div>\n*hi*\n</div>	raw HTML passed through	escaped, *hi* became <em>	raw HTML passed through	escaped as text
[a](javascript:alert(1))	href="javascript:..."	left as plain text	href="javascript:..."	href=""
~~gone~~ www.example.com	plain text (not CommonMark)	<s>, no autolink	<del> + link	<del> + link
text\n code?	indent stripped	indent stripped	indent kept	indent stripped

The first two rows are defaults, not bugs. markdown-it ships with html: false and a link validator that rejects javascript:, vbscript:, file: and most data: URLs. micromark has allowDangerousHtml and allowDangerousProtocol, both off. commonmark.js and marked trust the input. If you swap parsers, check these options first.

Security: Markdown is an HTML injection vector

The CommonMark spec requires raw HTML to pass through. So a Markdown renderer with default settings will happily output <img src=x onerror=alert(1)>. Links can carry javascript:. Image URLs can leak a reader’s IP to a tracker.

What to do:

• Turn raw HTML off if you don’t need it (html: false in markdown-it, the default in micromark).
• Sanitise the output anyway. Run the HTML through DOMPurify, or use rehype-sanitize in a unified pipeline. Sanitise after rendering, because that’s the HTML the browser actually sees. Sanitising the Markdown source misses things.
• marked has no sanitiser. It dropped its old sanitize option years ago, and its docs point you to DOMPurify.
• Plugins are a new hole. A custom renderer that builds HTML with string templates and forgets to escape is an XSS bug. See Custom Markdown Syntax.
• For full-page HTML that users paste, render it in a sandboxed iframe instead.

Performance: pathological input

Markdown parsers backtrack. Emphasis and brackets need look-back, and a naive implementation goes quadratic or worse on crafted input. cmark was designed to stay linear, and the CommonMark project has collected pathological test cases over the years. Other parsers have had ReDoS reports.

I timed three inputs on my laptop (Node 22):

Input	markdown-it	marked 18.1.1	micromark
20,000 [ then a then 20,000 ]	48 ms	18 ms	887 ms
"*a " repeated 30,000 times (90 KB)	31 ms	55 seconds	91 ms
5,000 > then a	1 ms	stack overflow	32 ms

90 KB of text froze marked for nearly a minute. If users can submit Markdown that you render on a server, that’s a denial-of-service vector. Defences:

• Cap input size.
• Render in a worker with a timeout (a Web Worker in the browser, a worker thread on the server).
• Pick a parser with a known linear-time design for untrusted input, and fuzz it with your own extensions turned on.

Editors and streaming

Live preview re-parses on every keystroke. For short notes, a full re-parse in a worker is fine. For long documents you want an incremental parser. @lezer/markdown reuses unchanged parts of the previous tree, which is why CodeMirror 6 can highlight a huge Markdown file as you type.

Streaming LLM output is the newer headache. Tokens arrive a few characters at a time, and the half-finished document is often misleading:

• An unclosed ``` turns everything after it into code until the closing fence arrives.
• **bol renders as literal asterisks, then snaps to bold.
• A half-written table flickers between a paragraph and a table.

Most chat UIs just re-parse the whole message on every chunk (throttled), sometimes after “closing” any open fence or emphasis in a copy of the text. Re-parsing everything per chunk is quadratic over the length of the reply, so throttle it.

Round-tripping is lossy

Parse to an AST and print it back and you don’t get your text back. With remark (mdast-util-to-markdown, default options), this is real output:

in:   * one\n* two\n\nSome _em_ and __strong__\n\nTitle\n=====\n
out:  * one\n* two\n\nSome *em* and **strong**\n\n# Title\n

The AST stores meaning (emphasis, heading depth 1), not spelling (_ vs *, setext vs ATX). That matters if you build an editor that edits a structured tree but saves .md files. Every save normalises the user’s style and makes noisy diffs. Options: configure the serializer to match the house style, or keep source positions and only rewrite the nodes that changed.

A real-world example

A collaborative notes app that stores each note as a .md file in the user’s cloud drive:

• The file is the source of truth, so other tools (GitHub, Obsidian, VS Code) can open it. That pushes the app towards CommonMark + GFM and away from private syntax.
• App-only features (wiki links, callouts, mentions) use syntax that degrades to readable text elsewhere: [[Page]] shows as brackets, > [!note] shows as a normal quote.
• The editor keeps an in-memory document model and converts to Markdown on save. The serializer is tuned so it doesn’t rewrite _ to * and churn every diff.
• Front matter is stripped before parsing and kept as metadata.
• Rendering happens in a worker, with raw HTML off and the output sanitised.
• Pasted HTML from the clipboard is converted to Markdown, not inserted raw (see Clipboard and DataTransfer).

Gotchas

• #Heading without a space is a paragraph in CommonMark. Old content breaks on migration.
• A list can’t interrupt a paragraph if it starts with a number other than 1. I was born in\n1984. ... stays a paragraph on purpose.
• Two trailing spaces mean a hard line break. Editors that trim trailing whitespace silently remove them. Use a trailing \ instead.
• 1. and 3) are different list types. Switching delimiter starts a new list.
• Raw HTML blocks end at a blank line, and Markdown inside an HTML block isn’t parsed unless there’s a blank line around it.
• Forget to strip front matter and ---\ntitle: x\n--- renders as an <hr> followed by <h2>title: x</h2>.
• Code spans with backticks inside need a longer fence: `` a`b ``.
• Default settings differ: raw HTML and javascript: links pass through in some parsers and not others.
• snake_case is safe in CommonMark, but * in the middle of words does emphasise: a*b*c → a<em>b</em>c.
• Round-tripping through an AST rewrites the user’s style.

Practice

1. Paste - a\n - b\n - c\n - d\n - e into Babelmark and compare. Then read the spec section on list items and predict the CommonMark output before checking.
2. Write a script that renders the same 20 tricky inputs with markdown-it and marked, and diffs the HTML.
3. Run the CommonMark spec tests (spec.json from the commonmark-spec repo) against a parser of your choice and count failures.
4. Build a renderer with markdown-it + DOMPurify, then try to get an XSS through it with raw HTML, links, image URLs and entities.
5. Write a block-only parser yourself for paragraphs, ATX headings, fenced code and > quotes, using the open-blocks algorithm. Add lazy continuation last.
6. Add a delimiter-stack inline parser for *em* and **strong** and make it pass the emphasis section of the spec. This is where you’ll learn the rule of 3.

Sources worth reading

• CommonMark spec, especially the appendix “A parsing strategy”: https://spec.commonmark.org/
• GitHub Flavored Markdown spec: https://github.github.com/gfm/
• John Gruber’s original syntax page: https://daringfireball.net/projects/markdown/syntax
• Babelmark 3, to compare implementations: https://babelmark.github.io/
• RFC 7763 and RFC 7764 (text/markdown)
• markdown-it docs and architecture: https://markdown-it.github.io/markdown-it/
• micromark readme (the state machine design): https://github.com/micromark/micromark
• mdast spec: https://github.com/syntax-tree/mdast