# HTML
`html/` parses and queries HTML documents — and CSS selectors against
them. The functions are native stdlib — compiled into every runtime,
available in every host without an import — backed by the vendored
lexbor HTML5 engine (the same parser PHP 8.4+'s DOM
extension uses).
Parsing follows the WHATWG algorithms exactly: unclosed and stray tags,
misnesting, and mangled tables recover the way a browser recovers, and
the full HTML5 character-reference set (named like `&`, numeric like
`😀`) is decoded in text and attributes during the parse. Where
`xml/` rejects anything not well-formed (see XML), `html/`
accepts the web and mail as it actually is.
Do not confuse this with the stdext `html.dft` module
(`deft/stdext/html`) — that one is generation-side only: `html-escape`
and the `[html … \html]` interpolation block for building pages. The
stdlib namespace here is the parsing side.
## The Tree Shape
`html/parse` and `html/select` produce one shape: elements are
%{ :tag "div" :attrs %{ :class "x" } :children @{ ... } }
`:children` holds the content in document order — text as strings,
child elements as maps, plus `%{ :comment "..." }` for comments and
`%{ :doctype "html" ... }` for the doctype (with optional
`:public-id` / `:system-id`). Adjacent text around a child element is
kept as separate strings:
set doc [html/parse "<p class='x'>Hello <b>world</b>!</p>" %{ :fragment true }] # => %{ :attrs %{ :class "x" } # :children @{ "Hello " %{ :attrs %{} :children @{ "world" } :tag "b" } "!" } # :tag "p" } $doc~tag # => "p" $doc~attrs~class # => "x" [nth $doc~children 0] # => "Hello " [nth $doc~children 1]~tag # => "b"
Root collapse rule: a parse with exactly one top-level element
returns that element's map directly; anything else returns a vector of
top-level nodes. The common full document (`<html>` with no doctype or
comments around it) collapses to the `<html>` map:
[html/parse "<div><p>hi</p></div>"]~tag # => "html" # a doctype makes it a vector: doctype map + html map set doc [html/parse "<!DOCTYPE html><p>x</p>"] [nth $doc 0] # => %{ :doctype "html" } [nth $doc 1]~tag # => "html"
Fragments (`%{ :fragment true }`) parse in `<body>` context —
innerHTML semantics — and return the fragment's own top-level nodes:
set doc [html/parse "a<b>c</b>d" %{ :fragment true }] [nth $doc 0] # => "a" [nth $doc 1]~tag # => "b"
## Functions
| Function | Purpose |
|---|---|
| `[html/parse $src ?opts]` | Parse into the tree shape above. Opts: `%{ :encoding string\|:auto :fragment bool }` |
| `[html/open $src ?opts]` | Parse once, return a `%HtmlDocument` handle for repeated queries |
| `[html/close $doc]` | Release a handle early (the GC finalizer does it too) |
| `[html/text $src\|$doc ?opts]` | Block-aware plain-text extraction (see below) |
| `[html/select $src\|$doc $sel]` | CSS selector query → vector of matching element trees |
| `[html/serialize $src\|$doc ?opts]` | Serialize back to HTML; `%{ :inner true }` gives body innerHTML |
| `[html/decode-entities $s]` | Decode character references in a plain string |
| `[html/to-markdown $src\|$doc ?opts]` | Convert to markdown in the markdown-viewer dialect (see below) |
| `[html/from-markdown $md ?opts]` | Render markdown to HTML (CommonMark-lite, escaped output) |
Fallible paths return `%Err{ :message "..." }`; a selector that matches
nothing is not an error — it is an empty vector.
### html/text
The reader's view: everything a mail client or fetch tool wants.
- `<script>`, `<style>`, `<head>`, `<template>`, form controls and other
non-content elements are dropped whole.
- Paragraph-level tags (`p`, `h1`–`h6`, lists, tables, `blockquote`,
`pre`, …) separate with a blank line; line-level tags (`div`,
`header`, `li`, …) with a single newline. `<br>` is a newline, `li`
items get a `- ` prefix, table cells join with ` | `.
- Whitespace collapses like a browser (runs of ASCII whitespace become
one space); `<pre>` keeps its text verbatim; ` ` survives as
U+00A0, not an ASCII space.
- Opts: `:links true` appends ` (href)` after anchor text, `:images`
true` renders `[image: alt]` for images, `:selector "css"` extracts`
text from the matching subtrees only.
[html/text "<h1>T</h1><p>Hello <b>world</b>!</p><ul><li>a<li>b</ul>"] # => "T\n\nHello world!\n\n- a\n- b" [html/text "see <a href='https://deft-lang.org'>docs</a> now" %{ :links true }] # => "see docs (https://deft-lang.org) now" [html/text "<div class=main>wanted</div><div>noise</div>" %{ :selector "div.main" }] # => "wanted"
### html/select
A CSS selector list (`.class`, `#id`, `a[href]`, `div > p`, `:has()`,
…) evaluated against the tree. A node matching several comma-list
selectors is returned once:
set hits [html/select "<div><p class=x>1</p><p id=y>2</p></div>" "p.x, p#y"] [len $hits] # => 2
For repeated queries, parse once with `html/open` and query the handle —
selectors run against the live document without re-parsing:
set d [html/open "<article><h2>One</h2><p>para</p><h2>Two</h2></article>"] set hs [html/select $d "h2"] set titles @{} each $hs {|h| $titles << [nth $h~children 0] } # titles => @{ "One" "Two" } [html/close $d]
Handles are GC-finalized, so `html/close` is an optimisation (prompt
teardown of large documents), not a requirement.
### html/serialize
Round-trips the document — the parser's entity decoding is re-encoded
on the way out:
[html/serialize "<p>a & b</p>" %{ :inner true }] # => "<p>a & b</p>"
### html/decode-entities
The full HTML5 named + numeric reference set on a bare string, without
parsing anything as markup — a literal `<` stays text, unknown
references stay as written, whitespace is preserved:
[html/decode-entities "a & b © A &bogus;"] # => "a & b © A &bogus;"
### html/to-markdown and html/from-markdown
The markdown pair turns HTML documents into something the TUI markdown
viewer renders — the pipeline behind the mail client's HTML bodies —
and turn markdown back into HTML for scripts and templates.
`html/to-markdown` emits the deft viewer dialect: ATX headings clamped
to `###`, flat `- ` / `N. ` lists (nesting flattens), pipe tables,
fenced code (language from `class="language-x"`), `> ` quotes, inline
`` `code` `` / `**bold**` / `[text](url)`, and `` images
(the viewer degrades unresolvable sources to an alt label itself).
Italic degrades to bold — the viewer has one emphasis style. Opts:
`:encoding` / `:fragment` / `:selector` like `html/text`.
[html/to-markdown "<h2>T</h2><p>a <b>b</b></p><ul><li>one</ul>"] # => "## T\n\na **b**\n\n- one"
`html/from-markdown` is a CommonMark-lite renderer: ATX h1–h6, fenced
code, flat and ordered lists, pipe tables (alignment kept), quotes,
`hr`, inline code/bold/italic/links/images, and backslash escapes.
All text is HTML-escaped on output; `%{ :full true }` wraps the result
in an html/head/body skeleton.
[html/from-markdown "- a **b** [c](https://d)"] # => "<ul>\n<li>a <strong>b</strong> <a href=\"https://d\">c</a></li>\n</ul>\n"
The two are designed to round-trip within the shared subset.
## Charsets
Input is treated as UTF-8 by default. Two opt forms re-decode first:
- `%{ :encoding "iso-8859-1" }` — an explicit charset, e.g. the mail
MIME `Content-Type` label. WHATWG label aliasing applies
(`iso-8859-1` is windows-1252, per the Encoding Standard).
- `%{ :encoding :auto }` — prescan the first 1024 bytes for a BOM or
`<meta charset>` declaration, per the HTML spec.
set latin1 [|printf "<p>caf\xe9</p>"] # raw 0xE9 byte [html/text $latin1 %{ :encoding "iso-8859-1" }] # => "café"
## Examples
# Pull every link target out of a page set d [html/open [ok! [fs/read "page.html"]]] set links @{} each [html/select $d "a[href]"] {|a| $links << $a~attrs~href } [html/close $d] # Readable text of an HTML mail body (what the mail client does) set text [html/text $html-body %{ :links true }] # Read a page's title without walking the whole tree set t [html/select $body "title"] # Entities in a feed snippet [html/decode-entities "Mac & Cheese — 12""] # => "Mac & Cheese — 12\""
Limits: 64 MiB input, 1024-node nesting depth (beyond returns `%Err`).
## See Also
- XML — the well-formed counterpart with its compact/tree shapes
- Data Formats — `json/` and `yaml/`
- Regex — for scraping the rare thing selectors cannot express