diff options
| author | Marshall Lochbaum <mwlochbaum@gmail.com> | 2021-08-11 14:06:02 -0400 |
|---|---|---|
| committer | Marshall Lochbaum <mwlochbaum@gmail.com> | 2021-08-11 14:06:02 -0400 |
| commit | 65eef4fade5eb426dae01d480f383b8a30b23071 (patch) | |
| tree | d5d98e372f68c928b7aaa2741f26cf89d5745ffc /docs/spec/token.html | |
| parent | e64ea0c5da34a59393e3b539cc1811ac63dfe3f4 (diff) | |
Change "BQN / main" in header to "(github) / BQN"
Diffstat (limited to 'docs/spec/token.html')
| -rw-r--r-- | docs/spec/token.html | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/docs/spec/token.html b/docs/spec/token.html index 8eb4bcee..e2b851a3 100644 --- a/docs/spec/token.html +++ b/docs/spec/token.html @@ -3,7 +3,7 @@ <link href="../style.css" rel="stylesheet"/> <title>Specification: BQN token formation</title> </head> -<div class="nav"><a href="https://github.com/mlochbaum/BQN">BQN</a> / <a href="../index.html">main</a> / <a href="index.html">spec</a></div> +<div class="nav">(<a href="https://github.com/mlochbaum/BQN">github</a>) / <a href="../index.html">BQN</a> / <a href="index.html">spec</a></div> <h1 id="specification-bqn-token-formation">Specification: BQN token formation</h1> <p>This page describes BQN's token formation rules (token formation is also called scanning). Most tokens in BQN are a single character long, but quoted characters and strings, identifiers, and numbers can consist of multiple characters, and comments, spaces, and tabs are discarded during token formation.</p> <p>BQN source code should be considered as a series of unicode code points, which we refer to as "characters". The separator between lines in a file is considered to be a single character, newline, even though some operating systems such as Windows typically represent it with a two-character CRLF sequence. Implementers should note that not all languages treat unicode code points as atomic, as exposing the UTF-8 or UTF-16 representation instead is common. For a language such as JavaScript that uses UTF-16, the double-struck characters <code><span class='Value'>𝕨</span><span class='Function'>𝕎</span><span class='Value'>𝕩</span><span class='Function'>𝕏</span><span class='Value'>𝕗</span><span class='Function'>𝔽</span><span class='Value'>𝕘</span><span class='Function'>𝔾</span></code> are represented as two 16-bit surrogate characters, but BQN treats them as a single unit.</p> |
