From 2afb23928e1984d475cc460e1672e8f6fa0e4dbe Mon Sep 17 00:00:00 2001 From: Marshall Lochbaum Date: Wed, 11 Aug 2021 17:21:31 -0400 Subject: Allow clicking on header to get fragment link --- docs/commentary/bacon.html | 2 +- docs/commentary/fanart.html | 2 +- docs/commentary/history.html | 28 ++--- docs/commentary/index.html | 2 +- docs/commentary/orchard.html | 2 +- docs/commentary/primitive.html | 12 +-- docs/commentary/problems.html | 152 +++++++++++++-------------- docs/doc/arithmetic.html | 12 +-- docs/doc/array.html | 14 +-- docs/doc/arrayrepr.html | 26 ++--- docs/doc/assert.html | 4 +- docs/doc/based.html | 10 +- docs/doc/block.html | 24 ++--- docs/doc/context.html | 10 +- docs/doc/control.html | 24 ++--- docs/doc/couple.html | 8 +- docs/doc/depth.html | 8 +- docs/doc/embed.html | 8 +- docs/doc/enclose.html | 12 +-- docs/doc/expression.html | 10 +- docs/doc/fill.html | 6 +- docs/doc/find.html | 4 +- docs/doc/fold.html | 14 +-- docs/doc/fromDyalog.html | 14 +-- docs/doc/fromJ.html | 14 +-- docs/doc/functional.html | 10 +- docs/doc/glossary.html | 18 ++-- docs/doc/group.html | 12 +-- docs/doc/identity.html | 10 +- docs/doc/index.html | 2 +- docs/doc/indices.html | 8 +- docs/doc/join.html | 6 +- docs/doc/leading.html | 18 ++-- docs/doc/lexical.html | 14 +-- docs/doc/logic.html | 10 +- docs/doc/map.html | 8 +- docs/doc/match.html | 6 +- docs/doc/namespace.html | 8 +- docs/doc/oop.html | 14 +-- docs/doc/order.html | 14 +-- docs/doc/paradigms.html | 6 +- docs/doc/pick.html | 8 +- docs/doc/prefixes.html | 8 +- docs/doc/primitive.html | 6 +- docs/doc/range.html | 6 +- docs/doc/repeat.html | 10 +- docs/doc/replicate.html | 10 +- docs/doc/reshape.html | 12 +-- docs/doc/reverse.html | 8 +- docs/doc/scan.html | 10 +- docs/doc/search.html | 12 +-- docs/doc/select.html | 8 +- docs/doc/selfcmp.html | 12 +-- docs/doc/shape.html | 6 +- docs/doc/shift.html | 12 +-- docs/doc/syntax.html | 22 ++-- docs/doc/take.html | 8 +- docs/doc/train.html | 10 +- docs/doc/transpose.html | 10 +- docs/doc/types.html | 18 ++-- docs/doc/undo.html | 8 +- docs/doc/windows.html | 12 +-- docs/editors/index.html | 18 ++-- docs/implementation/codfns.html | 10 +- docs/implementation/compile/dynamic.html | 22 ++-- docs/implementation/compile/index.html | 2 +- docs/implementation/index.html | 2 +- docs/implementation/kclaims.html | 10 +- docs/implementation/primitive/index.html | 2 +- docs/implementation/primitive/random.html | 6 +- docs/implementation/primitive/replicate.html | 18 ++-- docs/implementation/primitive/sort.html | 24 ++--- docs/implementation/vm.html | 28 ++--- docs/index.html | 16 +-- docs/keymap.html | 2 +- docs/running.html | 14 +-- docs/spec/evaluate.html | 8 +- docs/spec/grammar.html | 2 +- docs/spec/index.html | 2 +- docs/spec/inferred.html | 30 +++--- docs/spec/literal.html | 2 +- docs/spec/primitive.html | 34 +++--- docs/spec/scope.html | 12 +-- docs/spec/system.html | 22 ++-- docs/spec/token.html | 2 +- docs/spec/types.html | 2 +- docs/style.css | 13 +++ docs/tutorial/combinator.html | 24 ++--- docs/tutorial/expression.html | 16 +-- docs/tutorial/index.html | 2 +- docs/tutorial/list.html | 16 +-- docs/tutorial/variable.html | 14 +-- 92 files changed, 595 insertions(+), 582 deletions(-) (limited to 'docs') diff --git a/docs/commentary/bacon.html b/docs/commentary/bacon.html index d2c7ca27..d9af91a9 100644 --- a/docs/commentary/bacon.html +++ b/docs/commentary/bacon.html @@ -4,7 +4,7 @@ BQN: How to cook bacon -

How to cook bacon

+

How to cook bacon

In BQN, bacon is American, or side, bacon. The method described here works best with medium to thick cuts, and leaves the bacon uniformly cooked and crispy. If you prefer parts of your bacon chewy and underdone, seek help elsewhere.

Begin with a heated pan at least as wide as the length of the bacon. Prefer cast iron. Cook somewhat hotter than medium, but in order to avoid sticking, don't let the pan reach full heat before starting. However, there is never any need to lubricate the pan, as grease from the cooking bacon will soon serve this purpose.

Use metal tongs to hold and move strips of bacon during cooking. Lay three strips of bacon parallel in the center of the pan, and after a minute or two place three at a right angle to these, on top of them. After another few minutes, flip the first strips from the bottom. This is accomplished by pulling each one out from under the top strips by the end, then setting it back on top upside-down with that end on the opposite side of the pan. For more even cooking you might also rotate (1) the three strips, so that the center one ends up on one side and one of the side strips ends up in the faster-cooking center.

diff --git a/docs/commentary/fanart.html b/docs/commentary/fanart.html index 6b005248..d20de25a 100644 --- a/docs/commentary/fanart.html +++ b/docs/commentary/fanart.html @@ -4,7 +4,7 @@ BQN art -

BQN art

+

BQN art

Wezl has provided a view into the BQN programmer in 𝕩e's typical place of residence.

Depiction, remodel, animation.

Links contain the encoded art, which is CC-BY 4.0: see the "setup JS" pane.

diff --git a/docs/commentary/history.html b/docs/commentary/history.html index d4b01d0b..22197c09 100644 --- a/docs/commentary/history.html +++ b/docs/commentary/history.html @@ -4,14 +4,14 @@ BQN's development history -

BQN's development history

+

BQN's development history

I (Marshall Lochbaum) began designing BQN as a "fixed APL" in collaboration with my colleagues at Dyalog, and decided to take it on as a personal project when I chose to leave the company several months later in early 2020. BQN is influenced by my array language background, previous work in programming design, studies of APL history, and design discussions before and after starting work on the language. I developed most of the novel functionality in BQN, and am at the end of the day the one who writes the spec, but it includes significant contributions from collaborators, most notably dzaima and Adám Brudzewsky.

-

Background

+

Background

I learned J as my first computer programming language in 2009, and it's been my language of choice for personal projects from then until I started working with BQN. My first exposure to APL was Dyalog APL, which I began learning gradually after starting work at Dyalog in 2017; while I understand every primitive in detail (I've substantially reimplemented most of them), I've never written even a medium-sized script with it. I studied APL's history and many other APL dialects while helping to create the new APL Wiki in late 2019. In particular, I found A+ to be a very sound design and nominally took it as the starting point for BQN. As a result, BQN draws more from a general concept of APL than any particular dialect.

I have been working on programming language design since 2011 or earlier. The start of my first major language, I, might be placed in 2012 when I published a paper on its approach to mapping, unifying J's trains and function rank. By this time I had worked with several languages including Python and Factor, and implemented little interpreters in Java, Scala, and Haskell. There are many new ideas in I and some of them have made it to BQN, but the language has served mainly as a warning: its pure and simple syntax and approach to array and tacit programming lead to a rigidity that seems to take over from any human designer. And the end result is not usable. So I sought out constructs that would give back some control, like APL's two-layer function/operator syntax and explicit functions (although Dyalog has lexical scoping, it is crippled by the lack of closures and I ended up learning proper use of lexical scoping from Javascript—I've never done real work with any actual Lisp). Another language that I envisioned before BQN, called Iridescence, goes further in this direction, with Python-like syntax that is "noisy" relative to APL. It remains purely theoretical (I'll implement it some day) but has already had an influence on some BQN primitives.

The idea of a "fixed APL" is always percolating in the APL community, because of its long history and strong backwards compatibility requirements. BQN arose out of sessions by the Young APLers Group (YAG, unfortunately) inside Dyalog after I suggested that wild ideas for the future of APL might be a good topic for meetings (the previous order of business, right at YAG's formation, was creating the APL Wiki). At these meetings Adám, Richard Park, Nathan Rogers, and sometimes others discussed and proposed many ideas including the sketch I created that ended up as the blueprint of BQN. When I left Dyalog (putting an end to any notions of using those ideas at the company), I joined The APL Orchard forum, which most YAG members already used, to post about BQN. dzaima quickly began building his own BQN implementation using the existing dzaima/APL, and has discussed and contributed to most BQN design decisions since.

Shapecatcher is an essential resource for finding appropriate unicode characters. I've been using it heavily, and so has everyone else interested in glyph choices.

-

Timeline

+

Timeline

The table documents when I encountered features or interesting decisions in BQN. The "source" field indicates how I became aware of the idea while the "person" field gives, to the best of my knowledge, the original inventor (in blank entries and those that start with "w/", I am an inventor).

@@ -201,26 +201,26 @@
-

Features

+

Features

Discussion of specific features from the timeline above, with more detail.

-

Structural Under

+

Structural Under

I developed structural Under in 2017 and 2018. By Dyalog '17 I had figured out the basic concept, and sometime later I discovered it could be unified with J's Under operator, which was being considered for inclusion in Dyalog.

-

Prefix, Suffix, and Windows

+

Prefix, Suffix, and Windows

I discovered Prefix, Suffix, and Windows while thinking about Iridescence, probably in 2019. They are influenced by J's Prefix, Suffix, and Infix operators, but in Iridescence, with no distinction between functions and arrays, Prefix is just the Take function, and Suffix is Drop!

-

Array notation

+

Array notation

APL array notation has been developed mainly by Phil Last and later Adám Brudzewsky. The big difference from array literals in other languages is the idea that newline should be a separator equivalent to , as it is in ordinary APL execution including dfns. The changes I made for BQN, other than the ligature discussed below, were to use dedicated bracket pairs ⟨⟩ and [], and to allow , as a separator.

I picked out the ligature character between YAG meetings, but I think Richard Park was most responsible for the idea of a "shortcut" list notation.

-

Double-struck special names

+

Double-struck special names

There was a lot of discussion about names for arguments at YAG (no one liked alpha and omega); I think Nathan Rogers suggested using Unicode's mathematical variants of latin letters and I picked out the double-struck ones. My impression is that we were approaching a general concensus that "w" and "x" were the best of several bad choices of argument letters, but that I was the first to commit to them.

-

Assert primitive

+

Assert primitive

Nathan Rogers suggested that assertion should be made a primitive to elevate it to a basic part of the language. I used J's assert often enough for this idea to make sense immediately, but I think it was new to me. He suggested the dagger character; I changed this to the somewhat similar-looking !. The error-trapping modifier is identical to J's ::, but J only has the function [: to unconditionally throw an error, with no way to set a message.

-

Context-free grammar

+

Context-free grammar

In YAG meetings, I suggested adopting APL\iv's convention that variable case must match variable type in order to achieve a context-free grammar. Adám, a proponent of case-insensitive names, pointed out that the case might indicate the type the programmer wanted to use instead of the value's type, creating cross roles.

-

Headers

+

Headers

The idea of dfn headers is very common in the APL community, to the extent that it's hard to say which proposals lead to the form now used in BQN. A+ has headers which are similar but go outside the braces, and BQN headers aren't all that different from tradfn headers either. I found when creating BQN2NGN that ngn/apl allows dfns to include a monadic and dyadic case, separated by a semicolon. Some time later I realized that the ability to include multiple bodies is very powerful with headers because it enables a primitive sort of pattern matching, something I already wanted to squeeze into the language. I discussed this with dzaima, who added header support to dzaima/BQN almost immediately and was thus able to investigate the details of the format.

-

Group

+

Group

I've been fiddling with the idea of function or map inversion (preimage creation, really) for several years, and in fact independently discovered something very similar to K's Group function =, which is an excellent tool for languages that have dictionaries. I liked this approach as it didn't have all the ordering issues that J's Key has. However, I also didn't really want to introduce dictionaries to BQN, as they have a very strange relation to multidimensional arrays—are arrays like dictionaries with multiple keys, or dictionaries with a single vector key? I've been a proponent of / as a programming tool for much longer. I'd also developed a sophisticated view of partition representations while studying an extension to Dyalog's Partitioned Enclose proposed by Adám and included in Dyalog 18.0. I finally put all this together while fighting with Key to develop BQN's compiler: I realized that if the "key" argument was restricted to array indices, then it would make sense for the result to be an array, and that this was simply the "target indices" partition representation minus the requirement that those indices be nondecreasing.

-

Before and After

+

Before and After

It happens that BQN's Before () and After () modifiers are identical to I's hook (h) and backhook (H), but it took some time to arrive at this point. The hook function in I comes from J's 2-train, also called hook (I had probably seen Roger Hui's remarks that he would prefer hook to be a conjunction, with 2-trains indicating composition instead, but I don't think Roger has proposed a reverse hook). But the model for Before and After was initially APL's Compose () and the complement Reverse Compose that Adám created for Extended Dyalog APL. I noticed the similarity to Bind and decided to unify Binds and Composes at around the same time that I picked the symbols ⊸⟜. However, I kept the idea that the one-argument case should be simple composition unless the bound operand had a subject role. Eventually I decided the violation of syntactic type erasure was too inconsistent and settled on the current definition. Now I think these forms are better even ignoring constant functions, although I do occasionally run into cases where I'd like to use APL's Compose.

-

Constant modifier

+

Constant modifier

The idea of a constant function is nothing new; I named it k in I, taking influence from the SKI calculus. It was actually Adám who suggested adding it to Dyalog with the glyph , although I was the one who campaigned for it and introduced it to the public in version 18.0. It wasn't initially clear that a dedicated modifier was needed in BQN because the treatment of data types as constant functions seems to fill this role, but I eventually found that I needed a constant function returning a function too often to leave it out.

diff --git a/docs/commentary/index.html b/docs/commentary/index.html index c85c41fe..f5f3d2e0 100644 --- a/docs/commentary/index.html +++ b/docs/commentary/index.html @@ -4,7 +4,7 @@ BQN commentary -

BQN commentary

+

BQN commentary

Documents in this directory give context on how BQN was designed or remark on aspects of the language.

These extensions can be combined as well, so there are a lot of possibilities. A good picture to have in mind is cutting out a corner of the array 𝕩. This is because the result 𝕨𝕩 or 𝕨𝕩 always aligns with one side of 𝕩 along each axis, so it aligns with the corner where those sides meet.

The result d𝕩 is always the same as t𝕩 for some other argument t, but computing t wouldn't be too convenient. The reverse isn't true: only Take can insert fills, so results that include them can't come from Drop.

-

One axis

+

One axis

Let's start with a natural number 𝕨. Take gives the first 𝕨 major cells of 𝕩 (or elements of a list), while Drop gives all but the first 𝕨.

↗️
    4  "take and drop"
 "take"
@@ -76,7 +76,7 @@
     3  <"element"
 ⟨⟩
 
-

Negative argument

+

Negative argument

If 𝕨 is negative then wraps around the other side to take or drop from the end of 𝕩. It's a lot like negative indices in Select (), but while negative indices are asymmetric—0 is the first entry but ¯1 is the last—this case is symmetric. It's because the place to cut is always before the index 𝕨, cancelling out the negative index asymmetry.

↗️
    3  "abcdeEDCBA"
 "abc"
@@ -98,7 +98,7 @@
 ↗️
    ¯6  "xy"
 "    xy"
 
-

Multiple axes

+

Multiple axes

In the general case 𝕨 is a list of integers. They're matched with the leading axes of 𝕩, so that each affects one axis independently from the others.

↗️
     m  (10×↕5) + 7
 ┌─                      
diff --git a/docs/doc/train.html b/docs/doc/train.html
index b9c2f398..9675922c 100644
--- a/docs/doc/train.html
+++ b/docs/doc/train.html
@@ -4,10 +4,10 @@
   BQN: Function trains
 
 
-

Function trains

+

Function trains

Trains are an important aspect of BQN's tacit programming capabilities. In fact, a crucial one: with trains and the identity functions Left () and Right (), a fully tacit program can express any explicit function whose body is a statement with 𝕨 and 𝕩 used only as arguments (that is, there are no assignments and 𝕨 and 𝕩 are not used in operands or lists. Functions with assignments may have too many variables active at once to be directly translated but can be emulated by constructing lists. But it's probably a bad idea). Without trains it isn't possible to have two different functions that each use both arguments to a dyadic function. With trains it's perfectly natural.

BQN's trains are the same as those of Dyalog APL, except that Dyalog is missing the minor convenience of BQN's Nothing (·). There are many Dyalog-based documents and videos on trains you can view on the APL Wiki.

-

2-train, 3-train

+

2-train, 3-train

Trains are an adaptation of the mathematical convention that, for example, two functions F and G can be added to get a new function F+G that applies as (F+G)(x) = F(x)+G(x). With a little change to the syntax, we can do exactly this in BQN:

↗️
    (⊢+⌽) 5
 ⟨ 4 4 4 4 4 ⟩
@@ -25,7 +25,7 @@
 "fcdeab"
 

The three functions ∾⌽, ·∾⌽, and are completely identical: Join of Reverse. Why might we want three different ways to write the same thing? If we only want to define a function, there's hardly any difference. However, these three forms have different syntax, and might be easier or harder to use in different contexts. As we'll see, we can use inside a train without parenthesizing it, and string ·∾⌽ but not ∾⌽ together with other trains. Let's look at how the train syntax extends to longer expressions.

-

Longer trains

+

Longer trains

Function application in trains, as in other contexts, shares the lowest precedence level with assignment. Modifiers and strands (with ) have higher precedence, so they are applied before forming any trains. Once this is done, an expression is a subject expression if it ends with a subject and a function expression if it ends with a function (there are also modifier expressions, which aren't relevant here). A train is any function expression with multiple functions or subjects in it: while we've seen examples with two or three functions, any number are allowed.

Subject expressions are the domain of "old-school" APL, and just apply one function after another to a subject, possibly assigning some of the results (that's the top-level picture—anything can still happen within parentheses). Subjects other than the first appear only as left arguments to functions, which means that two subjects can't appear next to each other because the one on the left would have no corresponding function. Here's an example from the compiler (at one point), with functions and assignments numbered in the order they are applied and their arguments marked with «», and a fully-parenthesized version shown below.

cnpilt/𝕩civi+nv
@@ -40,7 +40,7 @@
 ⊢>(¯1»⌈`)
 

In a train, arguments alternate strictly with combining functions between them. Arguments can be either functions or subjects, except for the rightmost one, which has to be a function to indicate that the expression is a train. Trains tend to be shorter than subject expressions partly because to keep track of this alternation in a train of all functions, you need to know where each function is relative to the end of the train (subjects like the ¯1 above only occur as left arguments, so they can also serve as anchors).

-

Practice training

+

Practice training

The train ⊢>¯1»⌈` is actually a nice trick to get the result of Mark Firsts 𝕩 given the result of Classify 𝕩, without doing another search. Let's take a closer look, first by applying it mechanically. To do this, we apply each "argument" to the train's argument, and then combine them with the combining functions.

( > ¯1 » `) 𝕩
 (𝕩) > (¯1) » (`𝕩)
@@ -70,7 +70,7 @@
     (⊢>¯1»⌈`) sc
 ⟨ 1 1 1 1 0 0 1 0 0 1 1 ⟩
 
-

Composing trains

+

Composing trains

The example above uses a train with five functions: an odd number. Trains with an odd length are always composed of length-3 trains, and they themselves are composed the same way as subject expressions: an odd-length train can be placed in the last position of another train without parentheses, but it needs parentheses to go in any other position.

But we also saw the length-2 train ∾⌽ above. Even-length trains consist of a single function () applied to a function or odd-length train (); another perspective is that an even-length train is an odd-length train where the left argument of the final (leftmost) function is left out, so it's called with only a right argument. An even-length train always needs parentheses if it's used as one of the functions in another train. However, it can also be turned into an odd-length train by placing · at the left, making the implicit missing argument explicit. After this it can be used at the end of an odd-length train without parentheses. To get some intuition for even-length trains, let's look at an example of three functions used together: the unique () sorted () absolute values (|) of an argument list.

↗️
    ⍷∧| 34¯3¯20
diff --git a/docs/doc/transpose.html b/docs/doc/transpose.html
index 42d3dd71..b2319106 100644
--- a/docs/doc/transpose.html
+++ b/docs/doc/transpose.html
@@ -4,9 +4,9 @@
   BQN: Transpose
 
 
-

Transpose

+

Transpose

Transpose () is a tool for rearranging the axes of an array. BQN's version is tweaked relative to APL to align better with the leading axis model and make common operations easier.

-

Transpose basics

+

Transpose basics

The name for the primitive comes from the Transpose operation on matrices. Given a matrix as an array of rank 2, will transpose it:

↗️
     mat  23  6
 ┌─       
@@ -27,7 +27,7 @@
 3
 

With two axes the only interesting operation of this sort is to swap them (and with one or zero axes there's nothing interesting to do, and just returns the argument array). But a BQN programmer may well want to work with higher-rank arrays—although such a programmer might call them "tensors"—and this means there are many more ways to rearrange the axes. Transpose extends to high-rank arrays to allow some useful special cases as well as completely general axis rearrangement, as described below.

-

Monadic Transpose

+

Monadic Transpose

APL extends matrix transposition to any rank by reversing all axes for its monadic , but this generalization isn't very natural and is almost never used. The main reason for it is to maintain the equivalence a MP b ←→ b MP a, where MP +˝×1 is the generalized matrix product. But even here APL's Transpose is suspect. It does much more work than it needs to, as we'll see.

BQN's transpose takes the first axis of 𝕩 and moves it to the end.

↗️
     a23456  23456
@@ -88,7 +88,7 @@
 ⟨ 3 4 2 5 6 ⟩
 

In a case like this BQN's Dyadic transpose is much easier.

-

Dyadic Transpose

+

Dyadic Transpose

Transpose also allows a left argument that specifies a permutation of 𝕩's axes. For each index pi𝕨 in the left argument, axis i of 𝕩 is used for axis p of the result. Multiple argument axes can be sent to the same result axis, in which case that axis goes along a diagonal of 𝕩, and the result will have a lower rank than 𝕩.

↗️
     13204  a23456
 ⟨ 5 2 4 3 6 ⟩
@@ -109,7 +109,7 @@
 ⟨ 3 4 2 5 6 ⟩
 

Finally, it's worth noting that, as monadic Transpose moves the first axis to the end, it's equivalent to dyadic Transpose with a "default" left argument: (=-1˙).

-

Definitions

+

Definitions

Here we define the two valences of Transpose more precisely.

An atom right argument to either valence of Transpose is always enclosed to get an array before doing anything else.

Monadic transpose is identical to (=-1˙), except that if 𝕩 is a unit it is returned unchanged (after enclosing, if it's an atom) rather than giving an error.

diff --git a/docs/doc/types.html b/docs/doc/types.html index 1fffb7d1..075a0878 100644 --- a/docs/doc/types.html +++ b/docs/doc/types.html @@ -4,7 +4,7 @@ BQN: Types -

Types

+

Types

BQN supports the following fundamental types:

  • Number
  • @@ -44,11 +44,11 @@

    The reason operations and namespaces are called "mutable" is that the values obtained from them—by calling an operation on particular arguments or reading a field from a namespace—may change over the course of the program. This property is caused by variable modification , which can directly change a namespace field, or change the behavior of an operation that uses the modified variable. This means that a program that doesn't include won't have such changes in behavior. However, there will still be an observable difference between immutable data and the mutable types: code that creates a mutable value (for example, a block function {𝕩}) creates a different one each time, so that two different instances don't match () each other. Data values created at different times may match, but mutable values never will.

    An array is considered immutable because its shape, and what elements it contains, cannot change. An array has no identity outside these properties (and possibly its fill element), so an array with a different shape or different elements would simply be a different array. However, any element of an array could be mutable, in which case the behavior of the array would change with respect to the operation of selecting that element and calling it or accessing a field.

    -

    Data types

    +

    Data types

    Data types—numbers, characters, and arrays—are more like "things" than "actions". If called as a function, a value of one of these types simply returns itself. Data can be uniquely represented, compared for equality, and ordered using BQN's array ordering; in contrast, determining whether two functions always return the same result can be undecidable. For arrays, these properties apply only if there are no operations inside. We might say that "data" in BQN refers to numbers, characters, and arrays of data.

    -

    Numbers

    +

    Numbers

    The BQN spec allows for different numeric models to be used, but requires there to be only one numeric type from the programmer's perspective: while programs can often be executed faster by using limited-range integer types, there is no need to expose these details to the programmer. Existing BQN implementations are based on double-precision floats, like Javascript or Lua.

    -

    Characters

    +

    Characters

    A character in BQN is always a Unicode code point. BQN does not use encodings such as UTF-8 or UTF-16 for characters, although it would be possible to store arrays of integers or characters that correspond to data in these encodings. Because every code point corresponds to a single unit in UTF-32, BQN characters can be thought of as UTF-32 encoded.

    Addition and subtraction treat characters as an affine space relative to the linear space of numbers. This means that:

      @@ -56,18 +56,18 @@
    • Two characters can be subtracted to get the distance between them—a number.

    Other linear combinations such as adding two characters or negating a character are not allowed. You can check whether an application of + or - on numbers and characters is allowed by applying the same function to the "characterness" of each value: 0 for a number and 1 for a character. The result will be a number if this application gives 0 and a character if this gives 1, and otherwise the operation is not allowed.

    -

    Arrays

    +

    Arrays

    Full documentation here.

    A BQN array is a multidimensional arrangement of data. This means it has a certain shape, which is a finite list of natural numbers giving the length along each axis, and it contains an element for each possible index, which is a choice of one natural number that's less than each axis length in the shape. The total number of elements, or bound, is then the product of all the lengths in the shape. The shape may have any length including zero, and this shape is known as the array's rank. An array of rank 0, which always contains exactly one element, is called a unit, while an array of rank 1 is called a list and an array of rank 2 is called a table.

    Each array—empty or nonempty—has an inferred property called a fill. The fill either indicates what element should be used to pad an array, or that such an element is not known and an error should result. Fills can be used by Take (), the two Nudge functions (»«), Merge (>), and Reshape ().

    Arrays are value types (or immutable), so that there is no way to "change" the shape or elements of an array. An array with different properties is a different array. As a consequence, arrays are an inductive type, and it's not possible for an array to contain itself, or contain an array that contains itself, and so on. However, it is possible for an array to contain a function or other operation that has access to the array through a variable, and in this sense an array can "know about" itself.

    Different elements of an array should not influence each other. While some APLs force numbers placed in the same array to a common representation, which may have different precision properties, BQN values must not change behavior when placed in an array. However, this doesn't preclude changing the storage type of an array for better performance: for example, in a BQN implementation using 64-bit floats, an array whose elements are all integers that fit in 32-bit int range might be represented as an array of 32-bit ints.

    -

    Operation types

    +

    Operation types

    An operation is either a function or modifier, and can be applied to inputs—which are called arguments for functions and operands for modifiers—to obtain a result. During this application an operation might also change variables within its scope and call other operations, or cause an error, in which case it doesn't return a result. There is one type of call for each of the three operation types, and an operation will give an error if it is called in a way that doesn't match its type.

    In BQN syntax the result of a function has a subject role and the result of a modifier has a function role. However, the result can be any value at all: roles take place at the syntactic level, which has no bearing on types and values in the semantic level. This distinction is discussed further in Mixing roles.

    -

    Functions

    +

    Functions

    A function is called with one or two arguments. A data value (number, character, or array) can also be called the same way, but only a function takes any action when passed arguments, as data just returns itself. Both the one-argument and two-argument calls are considered function calls, and it's common for a function to allow both. A function that always errors in one case or the other might be called a one-argument or two-argument function, depending on which case is allowed.

    -

    Modifiers

    +

    Modifiers

    A 1-modifier is called with one operand, while a 2-modifier is called with two. In contrast to functions, these are distinct types, and it is impossible to have a value that can be called with either one or two operands. Also in contrast to functions, data values cannot be called as modifiers: they will cause an error if called this way.

    -

    Namespaces

    +

    Namespaces

    Functions and modifiers have internal scopes which they can manipulate (by defining and modifying variables) to save and update information. Namespaces let the programmer to expose this state more directly: identifiers in a namespace may be exported, allowing code outside the namespace to read their values. They are described in detail here.

    diff --git a/docs/doc/undo.html b/docs/doc/undo.html index dc3f9d37..5e7d6276 100644 --- a/docs/doc/undo.html +++ b/docs/doc/undo.html @@ -4,7 +4,7 @@ BQN: Undo -

    Undo

    +

    Undo

    Oh no, you've deleted a function after spending half an hour writing it! Well… hopefully your editor can help with that. But maybe you'll be interested to hear that BQN can reverse the action of some functions—ones that don't lose information. This capability is also used by Repeat () to repeat a negative number of times.

    ↗️
        2  "abcde"
     "cdeab"
    @@ -20,7 +20,7 @@
     "BQN"
     

    Here it undoes a function to decrement the last character by incrementing that character. In part this is enabled by the clean design of BQN primitives, because better-behaved functions like those using structural Under are easier to invert.

    -

    The rules

    +

    The rules

    If 𝔽 can be inverted exactly, then Undo just does that. However, there are also some other functions that BQN inverts. For example, the squaring function ט has both a positive and a negative inverse, and yet:

    ↗️
        ט ¯3
     9
    @@ -32,7 +32,7 @@
     ↗️
        6 - 6
     8.881784197001252e¯16
     
    -

    What's supported?

    +

    What's supported?

    For the full list, see the specification. An individual implementation might support a lot more functionality than is required, so if you're not concerned about portability just try out whatever function you're interested in.

    Arithmetic and simple combinators are usually invertible. A compound function that refers to its argument just once, like 6+⌽, can typically be undone, but one that uses the argument in two different ways, such as ⊢+⋆, probably can't.

    A few notable inverses are the logarithm , un-Transpose , and Indices inverse /. Enclose inverse, <, is an alternative to First that requires its argument to be a unit.

    @@ -42,7 +42,7 @@ ERROR 3 3 3
    -

    Undo headers

    +

    Undo headers

    Undo headers are currently supported only by dzaima/BQN.

    Of course BQN will never be able to invert all the functions you could write (if it could you could earn a lot of bitcoins, among other feats). But it does recognize some header forms that you can use to specify the inverse of a block function. BQN will trust you and won't verify the results your specified inverse gives.

    {
    diff --git a/docs/doc/windows.html b/docs/doc/windows.html
    index bc2aa263..77c40f19 100644
    --- a/docs/doc/windows.html
    +++ b/docs/doc/windows.html
    @@ -4,10 +4,10 @@
       BQN: Windows
     
     
    -

    Windows

    +

    Windows

    In BQN, it's strongly preferred to use functions, and not modifiers, for array manipulation. Functions are simpler as they have fewer moving parts. They are more concrete, since the array results can always be viewed right away. They are easier to implement with reasonable performance as well, since there is no need to recognize many possible function operands as special cases.

    The Window function replaces APL's Windowed Reduction, J's more general Infix operator, and Dyalog's Stencil, which is adapted from one case of J's Cut operator.

    -

    Definition

    +

    Definition

    We'll start with the one-axis case. Here Window's left argument is a number between 0 and 1+≠𝕩. The result is composed of slices of 𝕩 (contiguous sections of major cells) with length 𝕨, starting at each possible index in order.

    ↗️
        5"abcdefg"
     ┌─       
    @@ -24,7 +24,7 @@
     "cdefg"
     

    Windows differs from Prefixes and Suffixes in that it doesn't add a layer of nesting (it doesn't enclose each slice). This is possible because the slices have a fixed size.

    -

    Multiple dimensions

    +

    Multiple dimensions

    The above description applies to a higher-rank right argument. As an example, we'll look at two-row slices of a shape 34 array. For convenience, we will enclose each slice. Note that slices always have the same rank as the argument array.

    ↗️
        <2 2"0123""abcd""ABCD"
     ┌─                   
    @@ -49,10 +49,10 @@
     

    The slices are naturally arranged along multiple dimensions according to their starting index. Once again the equivalence ilx ←→ lix holds, provided i and l have the same length.

    If 𝕨 has length 0, then 𝕩 is not sliced along any dimensions. The only slice that results—the entire argument—is then arranged along an additional zero dimensions. In the end, the result is 𝕩, unchanged.

    -

    More formally

    +

    More formally

    𝕩 is an array. 𝕨 is a number, or numeric list or unit, with 𝕨≠≢𝕩. The result z has shape 𝕨∾¬𝕨((𝕨))𝕩, and element iz is 𝕩˜(𝕨)(↑+((𝕨)))i.

    Using Group we could also write iz ←→ 𝕩˜(𝕨()𝕩) +´¨ i.

    -

    Symmetry

    +

    Symmetry

    Let's look at an earlier example, along with its Transpose ().

    ↗️
        {𝕩,𝕩}5"abcdefg"
     ┌─                   
    @@ -77,7 +77,7 @@
     ↗️
        {((56¬22)𝕩)  23(22𝕩)} 567
     1
     
    -

    Applications

    +

    Applications

    Windows can be followed up with a reduction on each slice to give a windowed reduction. Here we take running sums of 3 values.

    ↗️
        +˝˘3 2,6,0,1,4,3
     ⟨ 8 7 5 8 ⟩
    diff --git a/docs/editors/index.html b/docs/editors/index.html
    index 882525c6..a787eeeb 100644
    --- a/docs/editors/index.html
    +++ b/docs/editors/index.html
    @@ -4,7 +4,7 @@
       BQN: Editor support
     
     
    -

    Editor support

    +

    Editor support

    Editor plugins and other tools for allowing BQN input are in this folder. Input is always performed with a backslash \ prefix by default, using the layout shown here. To type an actual backslash, hit the backslash key twice.

    This bookmarklet enables BQN input in any webpage in your browser.

    @@ -12,29 +12,29 @@

    For Android, this fork adds APL and BQN to Hacker's Keyboard.

    The file inputrc can be copied or appended to ~/.inputrc to enable backslash input in bash, BQN with rlwrap, and other software that uses GNU Readline.

    If you'd like to contribute files for another editor I'd gladly accept them!

    -

    System-wide

    -

    XKB (Unix)

    +

    System-wide

    +

    XKB (Unix)

    The file bqn is for configuring XKB on Linux, or other systems using X11. To use, copy it to /usr/share/X11/xkb/symbols/, then run

    $ setxkbmap -layout us,bqn -option grp:switch
     

    replacing us with your ordinary keyboard layout. switch indicates the right alt key and can be replaced with lswitch for left alt or other codes. The setting will go away on shutdown so you will probably want to configure it to run every time you start up. The way to do this depends on your desktop environment. For further discussion, see Wikipedia or the APL Wiki.

    Another XKB option, if you have a compose key enabled, is to place XCompose (possibly with adjustments) in ~/.XCompose.

    -

    Windows

    +

    Windows

    Folder autohotkey-win contains an AutoHotKey script and the generated .exe file. It runs as an ordinary program that recognizes BQN key combinations system-wide, using the right alt key (to change this, replace RAlt in the script and rebuild). Move it to the startup folder if you'd like to have it running all the time. You can right-click its icon in the system tray to disable it temporarily.

    The XCompose file, although created for XKB, should also be usable with WinCompose (but as far as I know this hasn't been tested).

    -

    Text editors

    -

    Vim

    +

    Text editors

    +

    Vim

    Copy or symlink all files into the corresponding directories in ~/.vim. Add the following two lines to ~/.vim/filetype.vim:

      au! BufRead,BufNewFile *.bqn setf bqn
       au! BufRead,BufNewFile * if getline(1) =~ '^#!.*bqn$' | setf bqn | endif
     

    Include syntax on in your .vimrc for syntax highlighting and filetype plugin on for keyboard input.

    -

    Emacs

    +

    Emacs

    Add the following two lines to init.el (usually ~/.emacs.d/init.el), replacing the path appropriately.

    (add-to-list 'load-path "/path/to/BQN/editors/emacs")
     (require 'gnu-apl-mode)
     
    -

    VS Code

    +

    VS Code

    See this repository.

    -

    Kakoune

    +

    Kakoune

    Copy or symlink kak/autoload/filetype/bqn.kak into autoload/filetype in your Kakoune config directory (probably .config/kak/).

    diff --git a/docs/implementation/codfns.html b/docs/implementation/codfns.html index 57cfa3d5..dcbaae30 100644 --- a/docs/implementation/codfns.html +++ b/docs/implementation/codfns.html @@ -4,16 +4,16 @@ Co-dfns versus BQN's implementation -

    Co-dfns versus BQN's implementation

    +

    Co-dfns versus BQN's implementation

    The BQN self-hosted compiler is directly inspired by the Co-dfns project, a compiler for a subset of Dyalog APL. I'm very grateful to Aaron for showing that array-oriented compilation is even possible! In addition to the obvious difference of target language, BQN differs from Co-dfns both in goals and methods.

    The shared goals of BQN and Co-dfns are to implement a compiler for an array language with whole-array operations. This provides the theoretical benefit of a short critical path, which in practice means that both compilers can make good use of a GPU or a CPU's vector instructions simply by providing an appropriate runtime (however, only Co-dfns has such a runtime—an ArrayFire program on the GPU and Dyalog APL on the CPU). The two implementations also share a preference for working "close to the metal" by passing around arrays of numbers rather than creating abstract types to work with data. Objects are right out. These choices lead to a compact source code implementation, and may have some benefits in terms of how easy it is to write and understand the compiler.

    -

    Compilation strategy

    +

    Compilation strategy

    Co-dfns development has primarily been focused on the core compiler, and not parsing, code generation, or the runtime. The associated Ph.D. thesis and famous 17 lines figure refer to this section, which transforms the abstract syntax tree (AST) of a program to a lower-level form, and resolves lexical scoping by linking variables to their definitions. While all of Co-dfns is written in APL, other sections aren't necessarily designed to be data-parallel and don't have the same performance guarantees. For example, the parser uses a parsing expression grammar (PEG), a sequential algorithm. In contrast, BQN is entirely written in a data-parallel style. It does not maintain the same clean separation between compiler sections: token formation and literal evaluation is separated into its own function, but parsing, AST manipulation, and code generation overlap.

    The core Co-dfns compiler is based on manipulating the syntax tree, which is mostly stored as parent and sibling vectors—that is, lists of indices of other nodes in the tree. BQN is less methodical, but generally treats the source tokens as a simple list. This list is reordered in various ways, allowing operations that behave like tree traversals to be performed with scans under the right ordering. This strategy allows BQN to be much stricter in the kinds of operations it uses. Co-dfns regularly uses (repeat until convergence) for iteration and creates nested arrays with (Key, like Group), but BQN has only a single instance of iteration to resolve quotes and comments, plus one complex but parallelizable scan for numeric literal processing, and only uses Group to extract identifiers and strings. This means that most primitives, if we count simple reductions and scans as single primitives, are executed a fixed number of times during execution, making complexity analysis even easier than in Co-dfns.

    -

    Backends and optimization

    +

    Backends and optimization

    Co-dfns was designed from the beginning to build GPU programs, and outputs code in ArrayFire (a C++ framework), which is then compiled. GPU programming is quite limiting, and as a result Co-dfns has strict limitations in functionality that are slowly being removed. It now has partial support for nested arrays and array ranks higher than 4. BQN is designed with performance in mind, but implementation effort focused on functionality first, so that arbitrary array structures as well as trains and lexical closures have been supported from the beginning. Rather than target a specific language, it outputs object code to be interpreted by a virtual machine. Another goal for BQN was to not only write the compiler in BQN but to use BQN for the runtime as much as possible. The BQN-based runtime uses a small number of basic array operations provided by the VM. The extra abstraction causes this runtime to be very slow, but this can be fixed by overwriting functions from the runtime with natively-implemented ones.

    Neither BQN nor Co-dfns significantly optimize their output at the time of writing (it could be said that Co-dfns relies on the ArrayFire backend to optimize). BQN does have one optimization, which is to compute variable lifetimes in functions so that the last access to a variable can clear it. Further optimizations often require finding properties such as reachability in a graph of expressions that probably can't be done efficiently in a strict array style. For this and other reasons it would probably be best to structure compiler optimization as a set of additional modules that can be provided during a given compilation.

    -

    Error reporting

    +

    Error reporting

    Co-dfns doesn't check for compilation errors, while BQN has complete error checking and good error messages, and includes source positions in compiler errors as well as in the compiled code for use in runtime errors. Position tracking and error checking add up to a little more than 20% overhead for the compiler, both in runtime and lines of code. And improving the way errors are reported once found has no cost for working programs, because reporting code only needs to be run if there's a compiler error. This leaves room for potentially very sophisticated error analysis to attempt to track down the root cause of a compilation error, but I haven't yet done any work along these lines.

    -

    Comments

    +

    Comments

    Aaron advocates the almost complete separation of code from comments (thesis) in addition to his very terse style as a general programming methodology. I find that this practice makes it hard to connect the documentation to the code, and is very slow in providing a summary or reminder of functionality that a comment might. One comment on each line makes a better balance of compactness and faster accessibility in my opinion. However, I do plan to write long-form material providing the necessary context and explanations required to understand the compiler.

    diff --git a/docs/implementation/compile/dynamic.html b/docs/implementation/compile/dynamic.html index 1edb73ef..0ffb6c46 100644 --- a/docs/implementation/compile/dynamic.html +++ b/docs/implementation/compile/dynamic.html @@ -4,12 +4,12 @@ BQN: Dynamic compilation -

    Dynamic compilation

    +

    Dynamic compilation

    This page discusses at a high level (without covering any specific source transformations) how compilation and interpretation might be structured in order to execute BQN faster. Effective strategies will compile parts of the program with more specialization or higher optimization as it runs and more information about it becomes known.

    To avoid confusion, an interpreter evaluates code to perform actions or produce a result; a CPU is a machine code interpreter. A compiler transforms code in one language or format to another. To run a program that does anything, it must eventually be interpreted; before this the program or parts of it might be compiled any number of times.

    The starting point for algorithms described here is bytecode for the BQN virtual machine. Producing this bytecode mostly amounts to parsing the program, which any implementation would have to do anyway, and it's fast enough that compiling an entire file before starting isn't a big concern. The bytecode can be interpreted directly, but compiler passes can reduce the overhead for each instruction or implement groups of instructions more effectively. Sophisticated compiler passes cost time, so the improvement must be balanced against time spent even though it might not be known in advance.

    -

    Considerations

    -

    What are we optimizing?

    +

    Considerations

    +

    What are we optimizing?

    How a program can be optimized depends on why it's taking time to run. Here we'll assume it's the core BQN syntax and primitives that are responsible, as things like system interaction are out of scope, and that the source file is not unreasonably large. The total time taken is the sum of the time taken by each action performed, so there are two main possibilities:

    • Primitives take a long time, because of large arrays.
    • @@ -18,7 +18,7 @@

      If many bytecode instructions are evaluated, it must be that blocks are repeated, because each instruction can only be run once by its containing block. A derived function might be run many times by a modifier, which typically involves a large array, but could also be because of Repeat ().

      This is an array-based viewpoint, because in a low-level language the large array case would just be considered one of several kinds of repetition. Traditionally APL focused exclusively on speeding up the large array case; BQN's compilation makes better block performance a reachable goal.

      The two conditions are routinely mixed in various ways: a program might split its time between manipulating small and large arrays, or it might work with large arrays but sometimes apply a block function to each element or small groups of elements. Array size or number of iterations could even differ between program runs. An evaluation strategy needs to adapt to these changes.

      -

      Hardware

      +

      Hardware

      It's easiest to get BQN running on a single-threaded scalar CPU, but all of the following improvements are commonly available, and can greatly increase speed for some programs:

      • SIMD instructions
      • @@ -27,30 +27,30 @@

      Each of the three is a good fit for array programming: any size array with SIMD and large arrays for multi-threading and GPU use. Multi-threading can also be useful for blocks that don't have side effects.

      SIMD instructions can be used at the primitive level (primitive implementation notes), with some improvement by fusing small operations together. Multi-threading is also possible at the primitive level, but no coordination between primitives will lead to cache coherency issues, and wasted time as some cores wait for others. It's probably better to analyze groups of primitives to allocate data and work. GPU execution is the fastest method for suitable programs, but can only handle certain kinds of code well and must use compiled kernels, which need to do a significant amount of work to justify the overhead.

      -

      Cached optimization

      +

      Cached optimization

      Expensive ahead-of-time optimizations can be saved somewhere in the filesystem (presumably the XDG cache directory for Linux). This of course introduces the problem of cache invalidation: cache files need to be versioned so an old cache isn't used by a new BQN, and they need to be thrown out when the file is changed and avoid saving information about user input or external files that could change.

      I think the data about a BQN source file that can be saved is generally pretty cheap to compute, and it's also important to make sure completely new code runs quickly. I'm hoping we don't have a reason to resort to caching.

      -

      Strategies

      -

      Hot paths

      +

      Strategies

      +

      Hot paths

      The basic strategy for JIT compilers like Java and Javascript (hey, they do actually have something in common!) is to track the number of times a block is called and perform more optimization as this number increases. It's also possible to record information about the inputs to do more specialized compilation, usually with a test in the compiled code to abort when the specialization is invalid.

      CBQN already includes a count to control native compilation; because this compilation is fast the best value is very low, between 1 and 5. More powerful optimization would generally use higher counts.

      Iteration modifiers ˘¨⌜´˝` can compute how many times they will call the operand quickly: it's usually based on the result size or argument length. The slowest case is probably monadic Scan `, where it's the size of 𝕩 minus the cell size, both values that already need to be computed. This number can be added to the existing count to find what level of optimization would be used, or even compared without adding, if false negatives are acceptable. This reduces the number of times the program runs blocks at the wrong optimization level, but slightly increases the overhead of mapping modifiers even on calls where the block to be run is already optimized at a high level. It can be restricted to only modifiers with a block operand because the modifier needs to inspect its operand anyway—the cost of running = or +´ without optimization is too high to skip this.

      Iteration modifiers on typed arrays often allow the argument types to be known with certainty, so that the operand can be specialized on that type with no testing.

      -

      Metadata

      +

      Metadata

      There's a lot of information about values that can be used to optimize, either in the general case if it's known for sure, or a specialization if it's suspected, or known under specific conditions. Type is the most important, then depth, rank, shape and element metadata for arrays and range or even value for numbers and characters. Other bulk properties like sortedness (to enable faster searches) or sum (for example, to find the shape of /𝕩) can be useful for arrays.

      Proofs that constrain the metadata in all cases are the most valuable, since the properties don't have to be tested or recomputed if they're already known. One way to get this information is to do an initial run of a block that only propagates known information about metadata. For example, if {+´𝕩} is run on an integer array, can return a list of natural numbers with the same type as 𝕩 or cause an error, and +´, given a list of natural numbers, returns a natural number. This approach can only help if the block will be run multiple times: clearly for a single run computing metadata along with data is the fastest. It runs at interpreted rather than native speed (because it's only run once) and could in fact take many calls to break even. For large array code, where the interpretive overhead is irrelevant, it could also be a step before a compilation pass that fuses and rearranges primitives.

      The same procedure can be run on local rather than global constraints, which might produce more specialized code at the cost of running through the block once per specialization.

      Saving metadata from the first run is another possibility, with very low overhead. This most naturally provides a guess as to what the metadata usually is, but it may also be possible to keep track of when metadata is "known" with a flag system.

      The desire to do metadata computations, or pure data ones once metadata is known suggests a system with a "wrapper" that computes type, shape, and so on, then selects and calls an "kernel" function for the computation. Specialized code could use a particular kernel, or a different wrapper that selects from a subset of the kernels.

      -

      Derived functions

      +

      Derived functions

      Like blocks, it can be valuable to optimize derived functions if they are run many times. Derived functions are often known at the program start by constant folding, but might also be constructed dynamically, particularly to bind an argument to a function.

      Compound arithmetic functions like +´, `, or = are essential to array programming, and have fast SIMD implementations, so they need to be recognized wherever they are found.

      In addition to these, there are patterns like ¨ that can be implemented faster than their components, and bindings like l where a computation (here, a hash table) on the left argument can be saved. These can be handled by inspecting the function. However, it's more robust to convert it to a canonical form, so this possibility should also be considered.

      Tacit code can be converted to SSA form very easily. To translate it into stack-based bytecode it would need a way to reuse values from the stack in multiple places; instructions to duplicate or extract a value from higher in the stack are an obvious candidate. Either of these forms is a natural step on the way to native compilation, and a bytecode representation would make it easier to optimize mixed tacit and explicit code—but it's easier to do the optimizations on SSA-form rather than stack-based code, so perhaps the right path is to convert both bytecode and derived functions to SSA.

      -

      Compile in another thread

      +

      Compile in another thread

      A simple and widely-used strategy to reduce slowdown due to dynamic compilation is to compile blocks in a separate thread from the one that runs them. The new code needs to be added in a thread-safe manner, which is not hard as the set of optimized implementations is a small lookup table of some sort with only one writer.

      If the implementation is able to make use of all available threads (possible when working with large arrays), then it's still important to minimize compilation time as that thread could be put to better use. If there are idle threads then the only costs of compilation overhead are minor: the optimized code can't be put to use as quickly, and there is more power draw and possible downclocking.

      -

      Anticipation

      +

      Anticipation

      The hot path strategy depends on targetting code for optimization based on history. Anticipation would identify in advance what code will take longer to run, and allocate a fraction of the time taken for optimizing that code. This is most useful for code that runs a small number of times on large arrays. An example where anticipation would be very important is for a programmer trying experimental one-off queries on a large dataset.

      The end result seems similar to that obtained by thunks as discussed at Dyalog '18 (video, slides). A thunk runs as part of a primitive, detecting that computing the result will be slow and outputting a deferred computation instead. Anticipation is more powerful because it can scan ahead in the bytecode instead of deciding as primitives are called whether or not to expand the thunk.

      Anticipation attempts to improve program speed while bounding the added overhead. For example, it might be constrained to add no more than 5% to the time to first program output, relative to base-level interpretation. The idea is to exit normal interpretation as soon as a large enough lower bound is established on this time, for example if an operation would create a large array. At this point it begins analysis, which will involve at least some shape propagation and probably increase the lower bound and optimization budget.

      diff --git a/docs/implementation/compile/index.html b/docs/implementation/compile/index.html index 4ec585f0..44424f90 100644 --- a/docs/implementation/compile/index.html +++ b/docs/implementation/compile/index.html @@ -4,7 +4,7 @@ BQN: Optimizing compilation notes -

      Optimizing compilation notes

      +

      Optimizing compilation notes

      Pages here discuss advanced compilation strategies for BQN, that is, steps that might take take place after compiling to bytecode or a similar intermediate representation.

      Most content here is currently speculative: C, Java, and Javascript backends are capable of compiling to native (x86, JVM, or Javascript) code in order to lower evaluation overhead but don't perform much if any analysis to improve this code. CBQN is likely to start making such optimizations in the future.

        diff --git a/docs/implementation/index.html b/docs/implementation/index.html index d0b73c59..8e46bc35 100644 --- a/docs/implementation/index.html +++ b/docs/implementation/index.html @@ -4,7 +4,7 @@ BQN implementation notes -

        BQN implementation notes

        +

        BQN implementation notes

        Notes about how BQN is or could be implemented.

        This repository's BQN implementation is written mainly in BQN: the bytecode compiler is completely self-hosted, and the majority of the runtime (r0, r1) is written in BQN except that it is allowed to define primitives; some preprocessing (pr) turns the primitives into identifiers.

        The remaining part, a Virtual Machine (VM), can be implemented in any language to obtain a version of BQN running in that language. The VM used for the online REPL is the Javascript implementation, while CBQN is a more advanced VM in C. There are platform-specific and generic tests in the test directory.

        diff --git a/docs/implementation/kclaims.html b/docs/implementation/kclaims.html index 4d028e6c..fcc0ddac 100644 --- a/docs/implementation/kclaims.html +++ b/docs/implementation/kclaims.html @@ -4,30 +4,30 @@ BQN: Wild claims about K performance -

        Wild claims about K performance

        +

        Wild claims about K performance

        Sometimes I see unsourced, unclear, vaguely mystical claims about K being the fastest array language. It happens often enough that I'd like to write a long-form rebuttal to these, and a demand that the people who make these do more to justify them.

        This isn't meant to put down the K language! K is in fact the only APL-family language other than BQN that I would recommend without reservations. And there's nothing wrong with the K community as a whole. Go to the k tree and meet them! What I want to fight is the myth of K, which is carried around as much by those who used K once upon a time, and no longer have any connection to it, as by active users.

        The points I argue here are narrow. To some extent I'm picking out the craziest things said about K to argue against. Please don't assume whoever you're talking to thinks these crazy things about K just because I wrote them here. Or, if they are wrong about these topics, that they're wrong about everything. Performance is a complicated and often counter-intuitive field and it's easy to be mislead.

        On that note, it's possible I've made mistakes, such as incorrectly designing or interpreting benchmarks. If you present me with concrete evidence against something I wrote below, I promise I'll revise this page to include it, even if I just have to quote verbatim because I don't understand a word of it.

        -

        The fastest array language

        +

        The fastest array language

        When you ask what the fastest array language is, chances are someone is there to answer one of k, kdb, or q. I can't offer benchmarks that contradict this, but I will argue that there's little reason to take these people at their word.

        The reason I have no measurements is that every contract for a commercial K includes an anti-benchmark clause. For example, Shakti's license says users cannot "distribute or otherwise make available to any third party any report regarding the performance of the Software benchmarks or any information from such a report". As I would be unable to share the results, I have not taken benchmarks of any commercial K. Or downloaded one for that matter. Shakti could publish benchmarks; they choose to publish a handful of comparisons with database software and none with array languages or frameworks. I do run tests with ngn/k, which is developed with goals similar to Whitney's K; the author says it's slower than Shakti but not by too much.

        The primary reason I don't give any credence to claims that K is the best is that they are always devoid of specifics. Most importantly, the same assertion is made across decades even though performance in J, Dyalog, and NumPy has improved by leaps and bounds in the meantime—I participated in advances of 26% and 10% in overall Dyalog benchmarks in the last two major versions. Has K4 (the engine behind kdb and Q) kept pace? Maybe it's fallen behind since Arthur left but Shakti K is better? Which other array languages has the poster used? Doesn't matter—they are all the same but K is better.

        A related theme I find is equivocating between different kinds of performance. I suspect that for interpreting scalar code K is faster than APL and J but slower than Javascript, and certainly any compiled language. For operations on arrays, maybe it beats Javascript and Java but loses to current Dyalog and tensor frameworks. Simple database queries, Shakti says it's faster than Spark and Postgres but is silent about newer in-memory databases. The most extreme K advocates sweep away all this complexity by comparing K to weaker contenders in each category. Just about any language can be "the best" with this approach.

        Before getting into array-based versus scalar code, here's a simpler case. It's well known that K works on one list at a time, that is, if you have a matrix—in K, a list of lists—then applying an operation (say sum) to each row works on each one independently. If the rows are short then there's function overhead for each one. In APL, J, and BQN, the matrix is stored as one unit with a stride. The sum can use one metadata computation for all rows, and there's usually special code for many row-wise functions. I measured that Dyalog is 30 times faster than ngn/k to sum rows of a ten-million by three float (double) matrix, for one fairly representative example. It's fine to say—as many K-ers do—that these cases don't matter or can be avoided in practice; it's dishonest (or ignorant) to claim they don't exist.

        -

        Scalar versus array code

        +

        Scalar versus array code

        I have a suspicion that users sometimes think K is faster than APL because they try out a Fibonacci function or other one-number-at-a-time code. Erm, your boat turns faster than a battleship, congratulations? Python beats these languages at interpreted performance. By like a factor of five. The only reason for anyone to think this is relevant is if they have a one-dimensional model where J is "better" than Python, so K is "better" than both.

        Popular APL and J implementations interpret source code directly, without even building an AST. This is very slow, and Dyalog has several other pathologies that get in the way as well. Like storing the execution stack in the workspace to prevent stack overflows, and the requirement that a user can save a workspace with paused code and resume it in a later version. But the overhead is per token executed, and a programmer can avoid the cost by working on large arrays where one token does a whole lot of work. If you want to show a language is faster than APL generally, this is the kind of code to look at.

        K is designed to be as fast as possible when interpreting scalar code, for example using a grammar that's much simpler than BQN's (speed isn't the only benefit of being simpler of course, but it's clearly a consideration). It succeeds at this, and K interpreters are very fast, even without bytecode compilation in advance.

        But K still isn't good at scalar code! It's an interpreter (if a good one) for a dynamically-typed language, and will be slower than compiled languages like C and Go, or JIT-compiled ones like Javascript and Java. A compiler generates code to do what you want, while an interpreter is code that reads data (the program) to do what you want. Once the code is compiled, the interpreter has an extra step and has to be slower. This is why BQN uses compiler-based strategies to speed up execution, first compiling to bytecode to make syntax overhead irrelevant and then usually post-processing that bytecode. Compilation is fast enough that it's perfectly fine to compile code every time it's run.

        -

        Instruction cache

        +

        Instruction cache

        A more specific claim about K is that the key to its speed is that the interpreter, or some part of it, fits in L1 cache. I know Arthur Whitney himself has said this; I can't find that now but here's some material from KX about the "L1/2 cache". Maybe this was a relevant factor in the early days of K around 2000—I'm doubtful. In the 2020s it's ridiculous to say that instruction caching matters.

        Let's clarify terms first. The CPU cache is a set of storage areas that are smaller and faster than RAM; memory is copied there when it's used so it will be faster to access it again later. L1 is the smallest and fastest level. On a typical CPU these days it might consist of 64KB of data cache for memory to be read and written, and 64KB of instruction cache for memory to be executed by the CPU. When I've seen it the L1 cache claim is specifically about the K interpreter (and not the data it works with) fitting in the cache, so it clearly refers to the instruction cache.

        (Unlike the instruction cache, the data cache is a major factor that makes array languages faster. It's what terms like "cache-friendly" typically refer to. I think the reason KX prefers to talk about the instruction cache is that it allows them to link this well-known consideration to the size of the kdb binary, which is easily measured and clearly different from other products. Anyone can claim to use cache-friendly algorithms.)

        A K interpreter will definitely benefit from the instruction cache. Unfortunately, that's where the truth of this claim runs out. Any other interpreter you use will get just about the same benefit, because the most used code will fit in the cache with plenty of room to spare. And the best case you get from a fast core interpreter loop is fast handling of scalar code—exactly the case that array languages typically ignore.

        So, 64KB of instruction cache. That would be small even for a K interpreter. Why is it enough? I claim specifically that while running a program might cause a cache miss once in a while, the total cost of these will only ever be a small fraction of execution time. This is because an interpreter is made of loops: a core loop to run the program as a whole and usually smaller loops for some specific instructions. These loops are small, with the core loop being on the larger side. In fact it can be pretty huge if the interpreter has a lot of exotic instructions, but memory is brought to the cache in lines of around 64 bytes, so that unused regions can be ignored. The active portions might take up a kilobyte or two. Furthermore, you've got the L2 and L3 caches as backup, which are many times larger than L1 and not much slower.

        So a single loop doesn't overflow the cache. And the meaning of a loop is that it's loaded once but run multiple times—for array operations, it could be a huge number. The body of an interpreter loop isn't likely to be fast either, typically performing some memory accesses or branches or both. An L1 instruction cache miss costs tens of cycles if it's caught by another cache layer and hundreds if it goes to memory. Twenty cycles would be astonishingly fast for a go around the core interpreter loop, and array operation loops are usually five cycles or more, plus a few tens in setup. It doesn't take many loops to overcome a cache miss, and interpreting any program that doesn't finish instantly will take millions of iterations or more, spread across various loops.

        -

        Measuring L1 with perf

        +

        Measuring L1 with perf

        Look, you can measure this stuff. Linux has a nice tool called perf that can track all sorts of hardware events related to your program, including cache misses. You can pass in a list of events with -e followed by the program to be run. It can even distinguish instruction from data cache misses! I'll be showing the following events:

        perf stat -e cycles,icache_16b.ifdata_stall,cache-misses,L1-dcache-load-misses,L1-icache-load-misses
         
        diff --git a/docs/implementation/primitive/index.html b/docs/implementation/primitive/index.html index b5256302..f139e5f4 100644 --- a/docs/implementation/primitive/index.html +++ b/docs/implementation/primitive/index.html @@ -4,7 +4,7 @@ BQN: Primitive implementation notes -

        Primitive implementation notes

        +

        Primitive implementation notes

        Commentary on the best methods I know for implementing various primitives. Often there are many algorithms that are viable in different situations, and in these cases I try to discuss the tradeoffs.

        • Replicate
        • diff --git a/docs/implementation/primitive/random.html b/docs/implementation/primitive/random.html index 7f660792..c907ee1c 100644 --- a/docs/implementation/primitive/random.html +++ b/docs/implementation/primitive/random.html @@ -4,12 +4,12 @@ BQN: Implementation of random stuff -

          Implementation of random stuff

          +

          Implementation of random stuff

          Not a primitive, but CBQN's •MakeRand initializes a random number generator that has some built-in utilities. For clarity we'll call a result of this initialization rand in the text below.

          -

          Random number generation

          +

          Random number generation

          CBQN is currently using wyrand, part of the wyhash library. It's extremely fast, passes the expected test suites, and no one's raised any concerns about it yet (but it's very new). It uses only 64 bits of state and doesn't have extra features like jump ahead.

          Other choices are xoshiro++ and PCG. The authors of these algorithms (co-author for xoshiro) hate each other very much and have spent quite some time slinging mud at each other. As far as I can tell they both have the normal small bias in favor of their own algorithms but are wildly unfair towards the other side, choosing misleading examples and inflating minor issues. I think both generators are good but find the case for xoshiro a little more convincing, and I think it's done better in third-party benchmarks.

          -

          Simple random sample

          +

          Simple random sample

          A simple random sample from a set is a subset with a specified size, chosen so that each subset of that size has equal probability. rand.Deal gets a sample of size 𝕨 from the set 𝕩 with elements in a uniformly random order, and rand.Subset does the same but sorts the elements.

          Deal uses a Knuth shuffle, stopping after the first 𝕨 elements have been shuffled, as the algorithm won't touch them again. Usually it creates 𝕩 explicitly for this purpose, but if 𝕨 is very small then initializing it is too slow. In this case we initialize 𝕨, but use a "hash" table with an identity hash—the numbers are already random—for 𝕨↓↕𝕩. The default is that every value in the table is equal to its key, so that only entries where a swap has happened need to be stored. The hash table is the same design as for self-search functions, with open addressing and linear probing.

          Subset uses Floyd's method, which is sort of a modification of shuffling where only the selected elements need to be stored, not what they were swapped with. This requires a lookup structure that can be updated efficiently and output all elements in sorted order. The choices are a bitset for large 𝕨 and another not-really-hash table for small 𝕨. The table uses a right shift—that is, division by a power of two—as a hash so that hashing preserves the ordering, and inserts like an insertion sort: any larger entries are pushed forward. Really this is an online sorting algorithm, that works because we know the input distribution is well-behaved (it degrades to quadratic performance only in very unlikely cases). When 𝕨>𝕩÷2, we always use a bitset, but select 𝕩-𝕨 elements and invert the selection.

          diff --git a/docs/implementation/primitive/replicate.html b/docs/implementation/primitive/replicate.html index 9d539d40..9e07a754 100644 --- a/docs/implementation/primitive/replicate.html +++ b/docs/implementation/primitive/replicate.html @@ -4,23 +4,23 @@ BQN: Implementation of Indices and Replicate -

          Implementation of Indices and Replicate

          +

          Implementation of Indices and Replicate

          The replicate family of functions contains not just primitives but powerful tools for implementing other functionality. The most important is converting bits to indices: AVX-512 extensions implement this natively for various index sizes, and even with no SIMD support at all there are surprisingly fast table-based algorithms for it.

          General replication is more complex. Branching will slow many useful cases down considerably when using the obvious solution. However, branch-free techniques introduce overhead for larger replication amounts. Hybridizing these seems to be the only way, but it's finicky.

          Replicate by a constant amount (so 𝕨 is a single number) is not too common in itself, but it's notable because it can be the fastest way to implement outer products and scalar dyadics with prefix agreement.

          -

          Indices

          +

          Indices

          Branchless algorithms are fastest, but with unbounded values in 𝕨 a fully branchless algorithm is impossible because you can't write an arbitrary amount of memory without branching. So the best algorithms depend on bounding 𝕨. Fortunately the most useful case is that 𝕨 is boolean.

          -

          Booleans to indices

          +

          Booleans to indices

          Indices (/) on a boolean 𝕩 of 256 or fewer bits can be made very fast on generic 64-bit hardware using a lookup table on 8 bits at a time. This algorithm can write past the end by up to 8 bytes (7 if trailing 0s are excluded), but never writes more than 256 bytes total. This means it's suitable for writing to an overallocated result array or a 256-byte buffer.

          To generate indices, use a 256×8-byte lookup table that goes from bytes to 8-byte index lists, and either a popcount instruction or another lookup table to get the sum of each byte. For each byte in 𝕨, get the corresponding indices, add an increment, and write them to the current index in the output. Then incease the output index by the byte's sum. The next indices will overlap the 8 bytes written, with the actual indices kept and junk values at the end overwritten. The increment added is an 8-byte value where each byte contains the current input index (always a multiple of 8); it can be added or bitwise or-ed with the lookup value.

          Some other methods discussed by Langdale and Lemire. I think very large lookup tables are not good for an interpreter because they cause too much cache pressure if used occasionally on smaller arrays. This rules out many of these strategies.

          -

          Non-booleans to indices

          +

          Non-booleans to indices

          If the maximum value in 𝕩 is, say, 8, then generating indices is fairly fast: for each element, write 8 indices and then move the output pointer forward by that much. This is much like the lookup table algorithm above, minus the lookup table. If the indices need to be larger than one byte, it's fine to expand them, and possibly add an offset, after generation (probably in chunks).

          There are two ways I know to fill in the gaps that this method would leave with elements that are too large. First is to stop after such an element and fill remaining space branchfully (maybe with memset). This is maximally efficient if 𝕩 is dominated by large elements—particularly for 2-byte indices when it skips index expansion—but not good if there are a lot of elements near the threshold. Second, initialize the buffer with 0 and perform ` afterwards, or other variations. This eliminates all but a fixed amount of branching, but it's a lot of overhead and I think unless a more sophisticated strategy arises it's best to stick with the first method.

          Indices is half of a counting sort: for sparse values, it's the slower half. Making it fast makes counting sort viable for much larger range-to-length ratios.

          -

          Replicate

          +

          Replicate

          For the most part, understanding Indices is the best way to implement Replicate quickly. But this is not the case if 𝕩 is boolean because then its elements are smaller than any useful index, and faster methods are available.

          -

          Compress

          +

          Compress

          Most of the methods listed below can be performed in place.

          For booleans, use BMI2's PEXT (parallel bits extract) instruction, or an emulation of it. The result can be built recursively alongside the also-required popcount using masked shifts.

          The generally best method for small elements seems to be to generate 1-byte indices into a buffer 256 at a time and select with those. There's a branchless method on one bit at a time which is occasionally better, but I don't think the improvement is enough to justify using it.

          @@ -28,11 +28,11 @@

          Odd-sized cells could be handled with an index buffer like small elements, using oversized writes and either overallocating or handling the last element specially.

          For medium-sized cells copying involves partial writes and so is somewhat inefficient. It's better to split 𝕨 into groups of 1s in order to copy larger chunks from 𝕩 at once. So the algorithm repeatedly searches 𝕨 for the next 1, then the next 0, then copies the corresponding value from 𝕩 to the result. This might be better for small odd-sized cells as well; I haven't implemented the algorithm with oversized writes to compare.

          The grouped algorithm, as well as a simpler sparse algorithm that just finds each 1 in 𝕨, can also better for small elements. Whether to use these depends on the value of +´𝕨 (sparse) or +´»<𝕨 (clumped). The checking is fast and these cases are common, but the general case is also fast enough that this is not a particularly high priority.

          -

          Replicate

          +

          Replicate

          Like Compress I think the best algorithm is often to generate small indices in a buffer and then select. But this is inefficient when 𝕨 contains large values, so those need to be detected and handled. Very tricky.

          -

          Constant replicate

          +

          Constant replicate

          Useful for outer products and leading-axis extension. See Expanding Bits in Shrinking Time for the boolean case. C compilers will generate decent code for constant small numbers and variable large ones, but I think specialized code with shuffle would be better for small numbers.

          -

          Higher ranks

          +

          Higher ranks

          When replicating along the first axis only, additional axes only change the element size (these are the main reason why a large element method is given). Replicating along a later axis offers a few opportunities for improvement relative to replicating each cell individually.

          Particularly for boolean 𝕨, Select is usually faster than Replicate (a major exception is for a boolean 𝕩). Simply replacing / with /¨ (after checking conformability) could be an improvement. It's probably best to compute the result shape first to avoid doing any work if it's empty. Similarly, if early result axes are small then the overhead of separating out Indices might make it worse than just doing the small number of Replicates.

          A technique when 𝕨 processed with one or more bytes at a time, and applies to many rows, is to repeat it up to an even number of bytes and combine rows of 𝕩 into longer virtual rows (the last one can be short). I think this only ends up being useful when 𝕩 is boolean.

          diff --git a/docs/implementation/primitive/sort.html b/docs/implementation/primitive/sort.html index 2a417770..5c6c3d70 100644 --- a/docs/implementation/primitive/sort.html +++ b/docs/implementation/primitive/sort.html @@ -4,35 +4,35 @@ BQN: Implementation of ordering functions -

          Implementation of ordering functions

          +

          Implementation of ordering functions

          The ordering functions are Sort (∧∨), Grade (⍋⍒), and Bins (⍋⍒). Although these are well-studied—particularly sorting, and then binary search or "predecessor search"—there are many recent developments, as well as techniques that I have not found in the literature. The three functions are closely related but have important differences in what algorithms are viable. Sorting is a remarkably deep problem with different algorithms able to do a wide range of amazing things, and sophisticated ways to combine those. It is by no means solved. In comparison, Bins is pretty tame.

          There's a large divide between ordering compound data and simple data. For compound data comparisons are expensive, and the best algorithm will generally be the one that uses the fewest comparisons. For simple data they fall somewhere between cheap and extremely cheap, and fancy branchless and vectorized algorithms are the best.

          -

          On quicksort versus merge sort

          +

          On quicksort versus merge sort

          Merge sort is better. It is deterministic, stable, and has optimal worst-case performance. Its pattern handling is better: while merge sort handles "horizontal" patterns and quicksort does "vertical" ones, merge sort gets useful work out of any sequence of runs but in-place quicksort will quickly mangle its analogue until it may as well be random.

          But that doesn't mean merge sort is always faster. Quicksort seems to work a little better branchlessly. For sorting, quicksort's partitioning can reduce the range of the data enough to use an extremely quick counting sort. Partitioning is also a natural fit for binary search, where it's mandatory for sensible cache behavior with large enough arguments. So it can be useful. But it doesn't merge, and can't easily be made to merge, and that's a shame.

          The same applies to the general categories of partitioning sorts (quicksort, radix sort, samplesort) and merging sorts (mergesort, timsort, multimerges). Radix sorts are definitely the best for some types and lengths, although the scattered accesses make their performance unpredictable and I think overall they're not worth it. A million uniformly random 4-byte integers is nearly the best possible case for radix sort, so the fact that this seems to be the go-to sorting benchmark means radix sorting looks better than it is.

          - +

          Binary searches are very easy to get wrong. Do not write (hi+lo)/2: it's not safe from overflows. I always follow the pattern given in the first code block here. This code will never access the value *base, so it should be considered a search on the n-1 values beginning at base+1 (the perfect case is when the number of values is one less than a power of two, which is in fact how it has to go). It's branchless and always takes the same number of iterations. To get a version that stops when the answer is known, subtract n%2 from n in the case that *mid < x.

          -

          Compound data

          +

          Compound data

          Array comparisons are expensive. The goal here is almost entirely to minimize the number of comparisons. Which is a much less complex goal than to get the most out of modern hardware, so the algorithms here are simpler.

          For Sort and Grade, use Timsort. It's time-tested and shows no signs of weakness (but do be sure to pick up a fix for the bug discovered in 2015 in formal verification). Hardly different from optimal comparison numbers on random data, and outstanding pattern handling. Grade can be done either by selecting from the original array to order indices or by moving the data around in the same order as the indices. I think the second of these ends up being substantially better for small-ish elements.

          For Bins, use a branching binary search: see On binary search above. But there are also interesting (although, I expect, rare) cases where only one argument is compound. Elements of this argument should be reduced to fit the type of the other argument, then compared to multiple elements. For the right argument, this just means reducing before doing whatever binary search is appropriate to the left argument. If the left argument is compound, its elements should be used as partitions. Then switch back to binary search only when the partitions get very small—probably one element.

          -

          Simple data

          +

          Simple data

          The name of the game here is "branchless".

          For sorting, the fastest algorithms for generic data and generic hardware are branchless quicksorts. Fluxsort is new and very exciting because it's a stable algorithm that's substantially faster than runner-up pdqsort on random arrays. However, it's immature and is missing a lot of the specialized strategies pdqsort has. I'm working on adapting these improvements to work for stable sorting and also on hybridizing with counting/bucket sort.

          A branchless binary search is adequate for Bins but in many cases—very small or large 𝕨, and small range—there are better methods.

          -

          Counting and bucket sort

          +

          Counting and bucket sort

          Both counting and bucket sort are small-range algorithms that begin by counting the number of each possible value. Bucket sort, as used here, means that the counts are then used to place values in the appropriate position in the result in another pass. Counting sort does not read from the initial values again and instead reconstructs them from the counts. It might be written (/≠¨)(-min) in BQN, with ¨ as a single efficient operation.

          Bucket sort can be used for Grade or sort-by (), but counting sort only works for sorting itself. It's not-even-unstable: there's no connection between result values and the input values except that they are constructed to be equal. But with fast Indices, Counting sort is vastly more powerful, and is effective with a range four to eight times the argument length. This is large enough that it might pose a memory usage problem, but the memory use can be made arbitrarily low by partitioning.

          -

          Quicksort

          +

          Quicksort

          Fluxsort attains high performance with a branchless stable partition that places one half on top of existing data and the other half somewhere else. One half ends up in the appropriate place in the sorted array. The other is in swap memory, and will be shifted back by subsequent partitions and base-case sorting. Aside from the partitioning strategy, Fluxsort makes a number of other decisions differently from pdqsort, including a fairly complicated merge sort (Quadsort) as the base case. I haven't fully evaluated these.

          This paper gives a good description of pdqsort. I'd start with the Rust version, which has some advantages but can still be improved further. The subsections below describe improved partitioning and an initial pass with several benefits. I also found that the pivot randomization methods currently used are less effective because they swap elements that won't become pivots soon; the pivot candidates and randomization targets need to be chosen to overlap. The optimistic insertion sort can also be improved: when a pair of elements is swapped the smaller one should be inserted as usual but the larger one can also be pushed forward at little cost, potentially saving many swaps and handling too-large elements as gracefully as too-small ones.

          While the stable partitioning for Fluxsort seems to be an overall better choice, pdqsort's unstable partitioning is what I've worked with in the past. The following sections are written from the perspective of pdqsort and will be rewritten for Fluxsort as the methods are adapted.

          -

          Partitioning

          +

          Partitioning

          In-place quicksort relies on a partitioning algorithm that exchanges elements in order to split them into two contiguous groups. The Hoare partition scheme does this, and BlockQuicksort showed that it can be performed quickly with branchless index generation; this method was then adopted by pdqsort. But the bit booleans to indices method is faster and fits well with vectorized comparisons.

          It's simplest to define an operation P that partitions a list 𝕩 according to a boolean list 𝕨. Partitioning permutes 𝕩 so that all elements corresponding to 0 in 𝕨 come before those corresponding to 1. The quicksort partition step, with pivot t, is (t𝕩)P𝕩, and the comparison can be vectorized. Interleaving comparison and partitioning in chunks would save memory (a fraction of the size of 𝕩, which should have 32- or 64-bit elements because plain counting sort is best for smaller ones) but hardly speeds things up: only a few percent, and only for huge lists with hundreds of millions of elements. The single-step P is also good for Bins, where the boolean 𝕨 will have to be saved.

          For binary search 𝕨𝕩, partitioning allows one pivot element t from 𝕨 to be compared to all of 𝕩 at once, instead of the normal strategy of working with one element from 𝕩 at a time. 𝕩 is partitioned according to t𝕩, then result values are found by searching the first half of 𝕨 for the smaller elements and the second half for the larger ones, and then they are put back in the correct positions by reversing the partitioning. Because Hoare partitioning works by swapping independent pairs of elements, P is a self inverse, identical to P. So the last step is simple, provided the partitioning information t𝕩 is saved.

          -

          Initial pass

          +

          Initial pass

          An initial pass for pdqsort (or another in-place quicksort) provides a few advantages:

          • Recognize sorted and reverse-sorted arrays as fast as possible
          • @@ -44,15 +44,15 @@

            Finding an initial run is fast as well. Compare the first two elements to determine direction, then search for the first pair that have the opposite direction (this can be vectorized because overreading is fine). This run can be used as the first range block, because the maximum and minimum are the two elements at the ends of the run.

            At the start of sorting, swap the smallest element to the beginning and the largest to the end, and shrink the size of the array by one in each direction. Now the element before the array is a lower bound and the one after is an upper bound. This property can also be maintained as the array is partitioned, by placing a pivot element between the two halves (swap it to one side of the array before partitioning and to the middle afterwards). As a result, it's always safe to use unguarded insertion sort, and an upper bound for the range of the array can always be found using the difference between the elements before and after it. Now finding the range is fast enough to check for counting sort at every recursion.

            This is a very simple initial pass; a more sophisticated one might be beneficial. If the array starts with a large run then there could be more of them. There may also be sampling-based tests to find when merge sort is better, even if the runs aren't perfect (but is this actually common?).

            -

            Other sorting algorithms

            +

            Other sorting algorithms

            IPS⁴o is a horrifyingly complicated samplesort thing. Unstable, but there's also a stable not-in-place version PS⁴o. For very large arrays it probably has the best memory access patterns, so a few samplesort passes could be useful.

            Vergesort has another useful first-pass strategy, which spends an asymptotically small amount of time searching for runs before sorting. Since it only detects perfect runs it won't give the full adaptivity of a good merge sort.

            Sorting networks compare and swap elements in a fixed pattern, and so can be implemented with branchless or even vectorized code. They're great for sorting many small arrays of the same size, but the limit before insertion sort beats it will be pretty small without hardware specialization.

            -

            SIMD sorting

            +

            SIMD sorting

            A few people have done some work on merge sorting with AVX2 or AVX-512: two examples. Pretty complicated, and still mostly in the proof of concept stage, but the benchmarks on uniform random arrays are good. Can these be made adaptive?

            ChipSort seems further along than those. It uses sorting networks, comb sort, and merging, which all fit nicely with SIMD and should work well together.

            Or AVX can speed up quicksort. I suspect this is more of a marginal improvement (over BlockQuicksort/pdqsort discussed below) relative to merge sort. If partitioning is fast enough it might make stable quicksort viable.

            - +

            Reminder that we're talking about simple, not compound data. The most important thing is just to have a good branchless binary search (see above), but there are other possible optimizations.

            If 𝕨 is extremely small, use a vector binary search as described in "Sub-nanosecond Searches" (video, slides). For 1-byte elements there's also a vectorized method that works whenever 𝕨 has no duplicates: create two lookup tables that go from multiples of 8 (5-bit values, after shifting) to bytes. One is a bitmask of 𝕨, so that a lookup gives 8 bits indicating which possible choices of the remaining 3 bits are in 𝕨. The other gives the number of values in 𝕨 less than the multiple of 8. To find the result of Bins, look up these two bytes. Mask off the bitmask to include only bits for values less than the target, and sum it (each of these steps can be done with another lookup, or other methods depending on instruction set). The result is the sum of these two counts.

            It's cheap and sometimes worthwhile to trim 𝕨 down to the range of 𝕩. After finding the range of 𝕩, binary cut 𝕨 to a smaller list that contains the range. Stop when the middle element fits inside the range, and search each half of 𝕨 for the appropriate endpoint of the range.

            diff --git a/docs/implementation/vm.html b/docs/implementation/vm.html index 1ecac46c..b7dd287f 100644 --- a/docs/implementation/vm.html +++ b/docs/implementation/vm.html @@ -4,14 +4,14 @@ The BQN virtual machine and runtime -

            The BQN virtual machine and runtime

            +

            The BQN virtual machine and runtime

            BQN's self-hosted compiler and runtime mean that only a small amount of native code is needed to run BQN on any given platform. The Javascript environment requires about 500 lines of Javascript code including system functions and performance improvements; probably around 250 would be required just to run the core language. This makes it fairly easy to port BQN to new platforms, allowing BQN to be embedded within other programming languages and interact with arrays or functions in those languages.

            The way data is represented is part of the VM implementation: it can use native arrays or a custom data structure, depending on what the language supports. An initial implementation will be very slow, but can be improved by replacing functions from the BQN-based runtime with native code. As the VM system can be hard to work with if you're not familiar with it, I advise you to contact me to discuss implementing a VM if you are interested.

            -

            Bytecode

            +

            Bytecode

            The BQN implementation here and dzaima/BQN share a stack-based object code format used to represent compiled code. This format is a list of numbers of unspecified precision (small precision will limit the length of list literals and number of locals per block, blocks, and constants). Previously it was encoded as bytes with the LEB128 format; while it no longer has anything to do with bytes it's called a "bytecode" because this is shorter than "object code".

            The self-hosted compiler uses a simpler, and less capable, format for block and variable data than dzaima/BQN. Only this format is described here; dc.bqn adapts it to be compatible with dzaima/BQN.

            dzaima/BQN can interpret bytecode or convert it to JVM bytecode, while the Javascript VM previously interpreted bytecode but now always compiles it. Since interpretation is a simpler strategy, it may be helpful to use the old Javascript bytecode interpreter as a reference (for bytecode execution only) when implementing a BQN virtual machine.

            -

            Components

            +

            Components

            The complete bytecode for a program consists of the following:

            • A bytecode sequence code
            • @@ -21,7 +21,7 @@
            • Optionally, source locations for each instruction
            • Optionally, tokenization information
            -

            Blocks

            +

            Blocks

            Each entry in blocks is a list of the following properties:

            • Block type: (0) function/immediate, (1) 1-modifier, (2) 2-modifier
            • @@ -31,7 +31,7 @@

              Compilation separates blocks so that they are not nested in bytecode. A block consists of bodies, so that all compiled code is contained in some body of a block. The self-hosted compiler compiles the entire program into an immediate block, and the program is run by evaluating this block. Bodies are terminated with a RETN or RETD instruction.

              When the block is evaluated depends on its type and immediateness. An immediate block (0,1) is evaluated as soon as it is pushed; a function (0,0) is evaluated when called on arguments, an immediate modifier (1 or 2, 1) is evaluated when called on operands, and a deferred modifier (1 or 2, 0) creates a derived function when called on operands and is evaluated when this derived function is called on arguments.

              The last property can be a single number, or, if it's a deferred block, might be a pair of lists. For a single number the block is always evaluated by evaluating the body with the given index. For a pair, the first element gives the monadic case and the second the dyadic one. A given valence should begin at the first body in the appropriate list, moving to the next one if a header test (SETH instruction) fails.

              -

              Bodies

              +

              Bodies

              Bodies in a block are separated by ;. Each entry in bodies is a list containing:

              • Starting index in code
              • @@ -41,7 +41,7 @@

              The starting index refers to the position in bytecode where execution starts in order to evaluate the block. Different bodies will always have the same set of special names, but the variables they define are unrelated, so of course they can have different counts. The given number of variables includes special names, but list of names and export mask don't.

              The program's symbol list is included in the tokenization information t: it is 02t. Since the entire program (the source code passed in one compiler call) uses this list, namespace field accesses can be performed with indices alone within a program. The symbol list is needed for cross-program access, for example if •BQN returns a namespace.

              -

              Instructions

              +

              Instructions

              The following instructions are defined by dzaima/BQN. The ones emitted by the self-hosted BQN compiler are marked in the "used" column. Instructions marked NS are used only in programs with namespaces, and so are not needed to support the compiler or self-hosted runtime.

              @@ -447,23 +447,23 @@

              Many instructions just call functions or modifiers or otherwise have fairly obvious implementations. Instructions to handle variables and blocks are more complicated (although very typical of bytecode representations for lexically-scoped languages) and are described in more detail below.

              -

              Local variables: DFND LOCO LOCU LOCM RETN

              +

              Local variables: DFND LOCO LOCU LOCM RETN

              The bytecode representation is designed with the assumption that variables will be stored in frames, one for each time an instance of a block is run. dzaima/BQN has facilities to give frame slots names, in order to support dynamic execution, but self-hosted BQN doesn't. A new frame is created when the block is evaluated (see #blocks) and in general has to be cleaned up by garbage collection, because a lexical closure might need to refer to the frame even after the corresponding block finishes. Lexical closures can form loops, so simple reference counting can leak memory, but it could be used in addition to less frequent tracing garbage collection or another strategy.

              A frame is a mutable list of slots for variable values. It has slots for any special names that are available during the blocks execution followed by the local variables it defines. Special names use the ordering 𝕤𝕩𝕨𝕣𝕗𝕘; the first three of these are available in non-immediate blocks while 𝕣 and 𝕗 are available in modifiers and 𝕘 in 2-modifiers specifically.

              When a block is pushed with DFND, an instance of the block is created, with its parent frame set to be the frame of the currently-executing block. Setting the parent frame when the block is first seen, instead of when it's evaluated, is what distinguishes lexical from dynamic scoping. If it's an immediate block, it's evaluated immediately, and otherwise it's pushed onto the stack. When the block is evaluated, its frame is initialized using any arguments passed to it, the next instruction's index is pushed onto the return stack, and execution moves to the first instruction in the block. When the RETN instruction is encountered, an index is popped from the return stack and execution returns to this location. As an alternative to maintaining an explicit return stack, a block can be implemented as a native function that creates a new execution stack and returns the value in it when the RETN instruction is reached. This approach uses the implementation language's call stack for the return stack.

              Local variables are manipulated with the LOCO (or LOCU) and LOCM instructions, which load the value of a variable and a reference to it (see the next section) respectively. These instructions reference variables by frame depth and slot index. The frame depth indicates in which frame the variable is found: the current frame has depth 0, its block's parent frame has depth 1, and so on. The slot index is an index within that frame.

              Slots should be initialized with some indication they are not yet defined. The variable can be defined with SETN only if it hasn't been defined yet, and can be accessed with LOCO or LOCU or modified with SETU or SETM only if it has been defined.

              -

              Variable references: ARRM LOCM SETN SETU SETM

              +

              Variable references: ARRM LOCM SETN SETU SETM

              A variable reference indicates a particular frame slot in a way that's independent of the execution context. For example, it could be a pointer to the slot, or a reference to the frame along with the index of the slot. LOCM pushes a variable reference to the stack.

              A reference list is a list of variable references or reference lists. It's created with the ARRM instruction. In the Javascript VM there's no difference between a reference list and an ordinary BQN list other than the contents.

              The SETN, SETU, and SETM instructions set a value for a reference. If the reference is to a variable, they simply set its value. For a reference list, the value needs to be destructured. It must be a list of the same length, and each reference in the reference list is set to the corresponding element of the value list.

              SETM additionally needs to get the current value of a reference. For a variable reference this is its current value (with an error if it's not defined yet); for a reference list it's a list of the values of each reference in the list.

              -

              Namespaces: FLDO FLDM NSPM RETD

              +

              Namespaces: FLDO FLDM NSPM RETD

              A namespace is the collection of variables in a particular scope, along with an association mapping some exported symbols (case- and underscore-normalized strings) to a subset of these. It can be represented using a frame for the variables, plus the variable name list and mask of exported variables from that block's properties in the bytecode. RETD finishes executing the block and returns the namespace for that execution.

              The variable name list is split into a program-level list of names and a list of indices into these names: within-program field accesses can be done with the indices only while cross-program ones must use names. One way to check whether an access is cross-program is to compare the accessor's program-level name list with the one for the accessed namespace as references or pointers. Then a lookup should be performed as appropriate. A persistent lookup table is needed to make this efficient.

              FLDO reads a field from a namespace. The parameter I is a program-level identifier index for this field. The VM must ensure that the field is exported, possibly by leaving unexported identifiers out of the namespace's lookup table. FLDM does the same but pushes a reference to the field, to be modified by assignment.

              NSPM is used for aliased assignment such as abcns. It tags a reference with a namespace field, identified with a program-level index. A value assigned to the tagged reference must be a namespace. The relevant field is extracted, and then stored in the original reference.

              -

              Runtime

              +

              Runtime

              Primitive functions and modifiers used in a program are stored in its consts array. The compiler needs to be passed a runtime with the value of every primitive so that these functions and modifiers are available.

              While it's perfectly possible to implement the runtime from scratch, the pseudo-BQN files r0.bqn and r1.bqn implement the full runtime in terms of a core runtime consisting of a smaller number of much simpler functions. pr.bqn converts this file so that it can be compiled. It changes values in the core runtime to primitives and primitives to generated identifiers, so that the first 22 values in the output's consts array are exactly the core runtime, and no other primitives are required. The result is a list of two elements: first the list of all primitive values, and then a function that can be called to pass in two additional core functions used for inferred properties.

              The contents of a core runtime are given below. The names given are those used in r1.bqn; the environment only provides a list of values and therefore doesn't need to use the same names. For named functions a description is given. For primitives, the implementation should match the BQN specification for that primitive on the specified domain, or the entire domain if left empty. They won't be called outside that domain except if there are bugs in the BQN sources.

              @@ -600,7 +600,7 @@

              To define the final two functions, call the second returned element as a function, with argument DecomposePrimInd. The function PrimInd gives the index of 𝕩 in the list of all primitives (defined as glyphs in pr.bqn), or the length of the runtime if 𝕩 is not a primitive. The two functions are only needed for computing inferred properties, and are defined by default so that every value is assumed to be a primitive, and PrimInd performs a linear search over the returned runtime. If the runtime is used directly, then this means that without setting DecomposePrimInd, function inferred properties will work slowly and for primitives only; if values from the runtime are wrapped then function inferred properties will not work at all.

              Remember that + and - can also work on characters in some circumstances, under the rules of affine characters. Other arithmetic functions should only accept numbers. = must work on any atoms including functions and modifiers. must work on numbers and characters.

              -

              GroupLen and GroupOrd

              +

              GroupLen and GroupOrd

              GroupLen and GroupOrd, short for Group length and Group order, are used to implement Group () and also to grade small-range lists and invert permutations (the inverse of a permutation p is 1¨GroupOrd p). Each of these only needs to work on lists of integers. Shown below are efficient implementations using BQN extended with the notation (il) Fn x meaning l Fnx(i) l, where Fn is if omitted. Since these special assignments only change one element of l, each can be a fast constant-time operation.

              GroupLen  {
                 l  ¯1 ´ 𝕩
              @@ -617,10 +617,10 @@
                 r
               }
               
              -

              Assembly

              +

              Assembly

              The full BQN implementation is made up of the two components above—virtual machine and core runtime—and the compiled runtime, compiler, and formatter. Since the compiler unlikely to work right away, I suggest initially testing the virtual machine on smaller pieces of code compiled by an existing, working, BQN implementation.

              BQN sources are compiled with cjs.bqn, which runs under dzaima/BQN as a Unix-style script. It has two modes. If given a command-line argument r, c, or fmt, it compiles one of the source files. With any other command-line arguments, it will compile each one, and format it as a single line of output. The output is in a format designed for Javascript, but it can be adjusted to work in other languages either by text replacement on the output or changes to the formatting functions in cjs.bqn.

              -

              Structure

              +

              Structure

              The following steps give a working BQN system, assuming a working VM and core runtime:

              • Evaluate the bytecode $ src/cjs.bqn r, passing the core runtime provide in the constants array. The result is a BQN list of a full runtime, and a function SetPrims.
              • @@ -630,7 +630,7 @@

              The compiler takes the runtime as 𝕨 and source code as 𝕩. To evaluate BQN source code, convert it into a BQN string (rank-1 array of characters), pass this string and runtime to the compiler, and evaluate the result as bytecode. Results can be formatted with the formatter for use in a REPL, or used from the implementation language.

              Two formatter arguments Glyph and FmtNum are not part of the runtime. Glyph assumes 𝕩 is a primitive and returns the character (not string) that represents it, and FmtNum assumes 𝕩 is a number and returns a string representing it.

              -

              Testing

              +

              Testing

              I recommend roughly the following sequence of tests to get everything working smoothly. It can be very difficult to figure out where in a VM things went wrong, so it's important to work methodically and make sure each component is all right before moving to the next.

              Because the compiler works almost entirely with lists of numbers, a correct fill implementation is not needed to run the compiler. Instead, you can define Fill as 0 and _fillBy_ as {𝔽} to always use a fill element of 0.

              -

              What kind of name is "BQN"?

              +

              What kind of name is "BQN"?

              It's three letters, that happen to match the capitals in "Big Questions Notation". You can pronounce it "bacon", but are advised to avoid this unless there's puns.

              -

              What does BQN look like?

              +

              What does BQN look like?

              Rather strange, most likely:

              ↗️
                  ⊑+`122  # The 12th Fibonacci number
               144
               

              More snippets are programmed into the live demo at the top of the page: hit the arrow at the right of the code window to see them. For longer samples, you can gaze into the abyss that is the self-hosted compiler, or the shallower but wider abyss of the runtime, or take a look at the friendlier markdown processor used to format and highlight documentation files. This repository also has some translations from "A History of APL in 50 Functions".

              -

              How do I work with the character set?

              +

              How do I work with the character set?

              Right at the beginning, you can use the bar above the online REPL to enter BQN code: hover over a character to see a short description, and click to insert it into the editor. But you'll soon want to skip the clicking and use keyboard input. I type the special characters using a backslash escape, so that, for example, typing \ then z writes (the backslash character itself is not used by BQN). The online REPL supports this method out of the box, and the editor plugins include or link to ways to enable it for editors, browsers, shells, and so on.

              The font comparison page shows several fonts that support BQN (including the one used on this site, BQN386). Most other monospace fonts are missing some BQN characters, such as double-struck letters 𝕨, 𝕩 and so on, which will cause these characters to be rendered with a fallback font and possibly have the wrong width or look inconsistent. The double-struck characters also require two bytes in UTF-16, which breaks rendering in many Windows terminals. If you have this problem, VS Code and wsl-terminal with an appropriate font definitely support them.

              -

              Why would I use it?

              +

              Why would I use it?

              There are plenty of clean, modern languages out there, and a good number of array languages. I don't think any other language fits both descriptions quite so well as BQN, and I find the combination lets me write powerful and reliable programs quickly. What you find in the language will depend on your background.

              If you haven't yet used an array language, BQN will present you with new ways of thinking that can streamline the way you work with data and algorithms. There's no denying that array programming has begun to creep into the mainstream, and you might be wondering if BQN has anything to offer when you can hack reduces and filters with the best of them. It does: real array programming is different in character, with more and better array operations on immutable multidimensional arrays, and syntax better suited to them. Performance that resembles a low-level compiled language more than a high-level dynamic one. Primitives flow together and compose better—one aspect that sets BQN apart from other array languages is a set of combinators that's more intuitive than previous attempts. I also happen to think BQN's character arithmetic system would improve just about any language.

              If your favorite language is J, you are missing out even more! Array programmers never seem willing to accept that good ideas can come from people other than Iverson and that legends like John McCarthy and Barbara Liskov advanced human knowledge of how to express computation. They did, and being able to casually pass around first-class functions and mutable closures, with namespaces keeping everything organized, is a huge quality of life improvement. Writing APL again is claustrophobic, the syntax worries and constraints in functionality suddenly rushing back. BQN's mutable objects make methods such as graph algorithms that just don't have a good array implementation (no, your O(n³) matrix method doesn't scale) possible, even natural. With bytecode compilation and NaN-boxing, a natural fit for the based array model, it evaluates that scalar code many times faster than APL or J. The Unix-oriented scripting system stretches seamlessly from quick sketch to multi-file program.

              BQN has no intention of being the last word in programming, but could be a practical and elegant tool in your kit—even if only used to inform your use of another language. Give it a try!

              -

              How do I get started?

              +

              How do I get started?

              The documentation still has some pages missing (not many now), while the tutorials are probably less than half complete. I don't think this is much of an impediment any more. Ask about anything you find confusing on the forums.

              BQN's tutorials are intended as an introduction to array programming with BQN. They assume only knowledge of elementary mathematics, but will probably be hard to follow if you have no programming experience. BQN has a lot in common with dynamically-typed functional languages like Lisp, Julia, or Javascript, so knowledge of these languages will be particularly helpful. The tutorials end abruptly right now, so you'll have to switch to the documentation, which is less structured.

              If you're already an array programmer, you might start with the documentation right away, using the BQN-Dyalog APL or BQN-J dictionary as a quick reference where appropriate. Be aware of two key differences between BQN and existing array languages beyond just the changes of primitives—if these differences don't seem important to you then you don't understand them! BQN's based array model is different from both a flat array model like J and a nested one like APL2, Dyalog, or GNU APL in that it has true non-array values (plain numbers and characters) that are different from depth-0 scalars. BQN also uses syntactic roles rather than dynamic type to determine how values interact, that is, what's an argument or operand and so on. This system, along with lexical closures, means BQN fully supports Lisp-style functional programming.

              A useful tool for both beginners and experienced users is BQNcrate, a searchable collection of BQN snippets to solve particular tasks. If you have a question about how you might approach a problem, give it a try by typing in a relevant keyword or two.

              -

              Where can I find BQN users?

              +

              Where can I find BQN users?

              There's a BQN Matrix channel at #bqn:matrix.org, which you can see in the Element web client with this link, and one on Discord that you can join with this invite. The two channels are bridged so that comments in one appear in both. It's among the most active array programming forums, with conversations nearly every day, so you can definitely get your questions answered fast.

              BQNBot will run your code from chat! Begin your message with bqn) and our friend (designation B-QN) will evaluate the rest and show the output. While putting your code in blocks `like this` is easier to read, the bot just operates on plain text and doesn't require it.

              In addition to these forums, you can contact me personally via Github issues or with the email address shown in my Github profile.

              -

              Can I help out?

              +

              Can I help out?

              Certainly! There are never enough hours in the day and contributors from beginner to advanced programmers are all welcome. If you're interested I recommend you ask on the forums first to get a feel for what exactly is needed.

              You will certainly feel an urge to skip this paragraph and get to the fun stuff, but the most important resource for implementing a language is testing and the most valuable one for building a language community is accessible introductions to the language and learning materials. These are both very demanding, but if you're willing to put in the work to advance BQN in the most effective way then this is it. One form of documentation that many users would appreciate is short descriptions—a sentence or two with examples—of the primitives for each glyph that can be displayed as help in the REPL. To be honest I'm lousy at making these and would prefer for someone else to do it.

              Work on BQN's implementation generally requires a high level of programming skill. We are focusing development effort on CBQN. It's true that the entire compiler and a (decreasing) portion of the runtime is written in BQN, but it would be hard to do much with these as they are very nearly complete, and work satisfactorily. On the other hand, there is a lot that needs to be written in C, including system values and higher-performance primitives. And there's no need to work on our implementation! BQN's specification and tests are there to make sure you can write a conforming implementation, and extend it with your own ideas. It's also possible to take advantage of the self-hosted BQN sources by writing a virtual machine that allows you to embed BQN in the language of your choice.

              diff --git a/docs/keymap.html b/docs/keymap.html index 5b70ec05..fdb26f6a 100644 --- a/docs/keymap.html +++ b/docs/keymap.html @@ -13,7 +13,7 @@ -

              BQN keymap

              +

              BQN keymap

              The standard BQN keymap is shown below. Note that characters ⍎⍕↙↖⍳, included for historical or speculative reasons, are not used by BQN. And don't miss spacebar for at the bottom!

              ┌────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬─────────┐
               │~ ¬! @ #  │$  │%  │^  │& ⍎ │* ⍕ │( )  │_ +  │Backspace│
              diff --git a/docs/running.html b/docs/running.html
              index c69ec64d..4d1aac71 100644
              --- a/docs/running.html
              +++ b/docs/running.html
              @@ -4,10 +4,10 @@
                 How to run BQN
               
               
              -

              How to run BQN

              +

              How to run BQN

              We are currently working quickly to make CBQN into the definitive offline implementation. Compilation speed (self-hosted) is good, the only significant core language feature missing is block headers and multiple bodies, and the essential system functions are there. Unless you need to start heavy number crunching right away, I recommend that you use CBQN and make system function or performance requests on Github or the BQN forums.

              A lot of development to date has been done in dzaima/BQN and uses features (mainly headers) that aren't in CBQN yet. Scripts in this repository use bqn in the #! line if self-hosted or dzaima/BQN can run them, and dbqn if only dzaima/BQN works.

              -

              Self-hosted BQN

              +

              Self-hosted BQN

              See the subsections below for instructions on specific implementations.

              This version of BQN is implemented mainly in BQN itself, but a host language supplies basic functionality and can also replace primitives for better performance. This also allows embedding, where programs in the host language can include BQN code. It fully supports all primitives except a few cases of structural Under (), but is still missing some advanced features: block headers and multiple body syntax, derived 1-modifiers, and block returns.

              Support in the following languages has been implemented:

              @@ -17,13 +17,13 @@
            • dzaima/BQN (bqn.bqn), mainly for testing.
            • Erlang, intended for embedding. Too slow to be practical yet: minutes to compile short programs.
            -

            Javascript

            +

            Javascript

            The online REPL is here. The file docs/bqn.js is zero-dependency Javascript, and can be loaded from HTML or Node.js. For command line use, call the Node.js script bqn.js, passing a file and •args, or -e to execute all remaining arguments directly and print the results. This notebook shows how to run it in an Observable notebook.

            -

            CBQN

            +

            CBQN

            C sources are kept in the CBQN repository, but it also depends on bytecode from the BQN sources here. Running make gets a working copy right away with saved bytecode. Then to use the latest bytecode, call $ ./BQN genRuntime /BQN, where /BQN points to this repository, and run make again.

            genRuntime can also be run with another BQN implementation (the Node.js one works but takes up to a minute), and plain ./genRuntime uses your system's bqn executable. I symlink /CBQN/BQN to ~/bin/bqn so I can easily use CBQN for scripting.

            CBQN uses the self-hosted runtime to achieve full primitive coverage, and implements specific primitives or parts of primitives natively to speed them up. This means primitives with native support—including everything used by the compiler—are fairly fast while others are much slower.

            -

            dzaima/BQN

            +

            dzaima/BQN

            dzaima/BQN is an implementation in Java created by modifying the existing dzaima/APL, and should be easy to run on desktop Linux and Android. It may be abandoned as dzaima is now working on CBQN. It has almost complete syntax support but incomplete primitive support: major missing functionality is dyadic Depth (), Windows (), and many cases of set functions (⊐⊒∊⍷, mostly with rank >1).

            In this repository and elsewhere, dzaima/BQN scripts are called with #! /usr/bin/env dbqn. This requires an executable file dbqn somewhere in your path with the following contents:

            #! /bin/bash
            @@ -31,7 +31,7 @@
             java -jar /path/to/dzaima/BQN/BQN.jar "$@"
             

            If compiled with Native Image, nBQN can be used directly instead.

            -

            dzaima+reference BQN

            +

            dzaima+reference BQN

            This repository contains a dzaima/BQN script dzref that fills in gaps in primitive support with BQN implementations. dzaima/BQN has good enough primitive support that I now almost never use this, but it's still needed for the website generator md.bqn. The command-line arguments are a script to run and followed by the •args to supply to it.

            -

            BQN2NGN

            +

            BQN2NGN

            BQN2NGN is a prototype implementation in Javascript build to experiment with the langauge, which is now abandoned.

            diff --git a/docs/spec/evaluate.html b/docs/spec/evaluate.html index 044f534c..2d1e5ce9 100644 --- a/docs/spec/evaluate.html +++ b/docs/spec/evaluate.html @@ -4,22 +4,22 @@ Specification: BQN evaluation -

            Specification: BQN evaluation

            +

            Specification: BQN evaluation

            This page describes the semantics of the code constructs whose grammar is given in grammar.md. The formation rules there are not named, and here they are identified by either the name of the term or by copying the rule entirely if there are several alternative productions.

            Here we assume that the referent of each identifier, or equivalently the connections between identifiers, have been identified according to the scoping rules.

            Evaluation is an ordered process, and any actions required to evaluate a node always have a specified order unless performing them in any order would have the same effect. Side effects that are relevant to ordering are setting and getting the value of a variable, causing an error, and returning (with ) from a block. Errors described in this page are "evaluation errors" and can be caught by the Catch () modifier. For caught errors and returns, evaluation halts without attempting to complete any in-progress node, and is restarted by Catch (for errors) or at the end of the appropriate block evaluation (for returns).

            -

            Programs and blocks

            +

            Programs and blocks

            The result of parsing a valid BQN program is a PROGRAM, and the program is run by evaluating this term.

            A PROGRAM or BODY is a list of STMTs, which are evaluated in program order. A result is always required for BODY nodes, and sometimes for PROGRAM nodes (for example, when loaded with •Import). If any identifiers in the node's scope are exported, or any of its statements is an EXPORT, then the result is the namespace created in order to evaluate the node. If a result is required but the namespace case doesn't apply, then the last STMT node must be an EXPR and its result is used. The statement EXPR evaluates some APL code and possibly assigns the results, while nothing evaluates any subject or Derv terms it contains but discards the results. An EXPORT statement performs no action.

            A block consists of several BODY terms, some of which may have an accompanying header describing accepted inputs and how they are processed. An immediate block brImm can only have one BODY, and is evaluated by evaluating the code in it. Other types of blocks do not evaluate any BODY immediately, but instead return a function or modifier that obtains its result by evaluating a particular BODY. The BODY is identified and evaluated once the block has received enough inputs (operands or arguments), which for modifiers can take one or two calls: if two calls are required, then on the first call the operands are simply stored and no code is evaluated yet. Two calls are required if there is more than one BODY term, if the BODY contains the special names 𝕨𝕩𝕤𝕎𝕏𝕊, or if its header specifies arguments (the header-body combination is a _mCase or _cCase_). Otherwise only one is required.

            To evaluate a block when enough inputs have been received, first the correct case must be identified. To do this, first each special case (FCase, _mCase, or _cCase_), excluding FCase nodes containing UndoHead, is checked in order to see if its arguments are strucurally compatible with the given arguments. That is, is headW is a subject, there must be a left argument matching that structure, and if headX is a subject, the right argument must match that structure. This means that 𝕨 not only matches any left argument but also no argument. The test for compatibility is the same as for multiple assignment described below, except that the header may contain constants, which must match the corresponding part of the given argument. If no special case matches, then an appropriate general case (FMain, _mMain, or _cMain_) is used: if there are two, the first is used with no left argument and the second with a left argument; if there are one, it is always used, and if there are none, an error results.

            The only remaining step before evaluating the BODY is to bind the inputs and other names. Special names are always bound when applicable: 𝕨𝕩𝕤 if arguments are used, 𝕨 if there is a left argument, 𝕗𝕘 if operands are used, and _𝕣 and _𝕣_ for modifiers and combinators, respectively. Any names in the header are also bound, allowing multiple assignment for arguments.

            If there is no left argument, but the BODY contains 𝕨 or 𝕎 at the top level, then it is conceptually re-parsed with 𝕨 replaced by · to give a monadic version before application; this modifies the syntax tree by replacing some instances of subject, arg, or Operand with nothing. The token 𝕎 is not allowed in this case and causes an error. Re-parsing 𝕨 can also cause an error if it's used as an operand or list element, where nothing is not allowed by the grammar. Note that these errors must not appear if the block is always called with two arguments. True re-parsing is not required, as the same effect can also be achieved dynamically by treating · as a value and checking for it during execution. If it's used as a left argument, then the function should instead be called with no left argument (and similarly in trains); if it's used as a right argument, then the function and its left argument are evaluated but rather than calling the function · is "returned" immediately; and if it's used in another context then it causes an error.

            -

            Assignment

            +

            Assignment

            An assignment is one of the four rules containing ASGN. It is evaluated by first evaluating the right-hand-side subExpr, FuncExpr, _m1Expr, or _m2Exp_ expression, and then storing the result in the left-hand-side identifier or identifiers. The result of the assignment expression is the result of its right-hand side. Except for subjects, only a lone identifier is allowed on the left-hand side and storage sets it equal to the result. For subjects, destructuring assignment is performed when an lhs is lhsList or lhsStr. Destructuring assignment is performed recursively by assigning right-hand-side values to the left-hand-side targets, with single-identifier assignment as the base case.

            The right-hand-side value, here called v, in destructuring assignment must be a list (rank 1 array) or namespace. If it's a list, then each LHS_ENTRY node must be an LHS_ELT. The left-hand side is treated as a list of lhs targets, and matched to v element-wise, with an error if the two lists differ in length. If v is a namespace, then the left-hand side must be an lhsStr where every LHS_ATOM is an LHS_NAME, or an lhsList where every LHS_ENTRY is an LHS_NAME or lhs "⇐" LHS_NAME, so that it can be considered a list of LHS_NAME nodes some of which are also associated with lhs nodes. To perform the assignment, the value of each name is obtained from the namespace v, giving an error if v does not define that name. The value is assigned to the lhs node if present (which may be a destructuring assignment or simple subject assignment), and otherwise assigned to the same LHS_NAME node used to get it from v.

            Modified assignment is the subject assignment rule lhs Derv "↩" subExpr. In this case, lhs should be evaluated as if it were a subExpr (the syntax is a subset of subExpr), and the result of the function application lhs Derv subExpr should be assigned to lhs, and is also the result of the modified assignment expression.

            -

            Expressions

            +

            Expressions

            We now give rules for evaluating an atom, Func, _mod1 or _mod2_ expression (the possible options for ANY). A literal or primitive sl, Fl, _ml, or _cl_ has a fixed value defined by the specification (literals and built-ins). An identifier s, F, _m, or _c_, if not preceded by atom ".", must have an associated variable due to the scoping rules, and returns this variable's value, or causes an error if it has not yet been set. If it is preceded by atom ".", then the atom node is evaluated first; its value must be a namespace, and the result is the value of the identifier's name in the namespace, or an error if the name is undefined. A parenthesized expression such as "(" _modExpr ")" simply returns the result of the interior expression. A braced construct such as BraceFunc is defined by the evaluation of the statements it contains after all parameters are accepted. Finally, a list "⟨" ? ( ( EXPR )* EXPR ? )? "⟩" or ANY ( "‿" ANY )+ consists grammatically of a list of expressions. To evaluate it, each expression is evaluated in source order and their results are placed as elements of a rank-1 array. The two forms have identical semantics but different punctuation.

            A Return node creates a return function. As discussed in the scoping rules, its identifier indicates a namespace from a particular block evaluation. When called, the function causes an error if that block has finished execution, or if the call includes a left argument 𝕨. Otherwise, evaluation stops immediately, and resumes at the end of the block where it returns the right argument 𝕩 from that block.

            Rules in the table below are function and modifier evaluation.

            diff --git a/docs/spec/grammar.html b/docs/spec/grammar.html index 3ad5bd25..da83d87c 100644 --- a/docs/spec/grammar.html +++ b/docs/spec/grammar.html @@ -4,7 +4,7 @@ Specification: BQN grammar -

            Specification: BQN grammar

            +

            Specification: BQN grammar

            BQN's grammar is given below. Terms are defined in a BNF variant. However, handling special names properly is possible but difficult in BNF, so they are explained in text along with the braced block grammar.

            The symbols s, F, _m, and _c_ are identifier tokens with subject, function, 1-modifier, and 2-modifier classes respectively. Similarly, sl, Fl, _ml, and _cl_ refer to literals and primitives of those classes. While names in the BNF here follow the identifier naming scheme, this is informative only: syntactic roles are no longer used after parsing and cannot be inspected in a running program.

            A program is a list of statements. Almost all statements are expressions. Namespace export statements, and valueless results stemming from ·, or 𝕨 in a monadic brace function, can be used as statements but not expressions.

            diff --git a/docs/spec/index.html b/docs/spec/index.html index b8f1d80e..fa8ad8c9 100644 --- a/docs/spec/index.html +++ b/docs/spec/index.html @@ -4,7 +4,7 @@ BQN specification -

            BQN specification

            +

            BQN specification

            This document, and the others in this directory (linked in the list below) make up the pre-versioning BQN specification. The specification differs from the documentation in that its purpose is only to describe the exact details of BQN's operation in the most quickly accessible way, rather than to explain the central ideas of BQN functionality and how it might be used. The core of BQN, which excludes system-provided values, is now almost completely specified. One planned feature—an extension to allow low-rank elements in the argument to Join—has not yet been added, and the spec will continue to be edited further to improve clarity and cover any edge cases that have been missed.

            Under this specification, a language implementation is a BQN pre-version implementation if it behaves as specified for all input programs. It is a BQN pre-version implementation with extensions if it behaves as specified in all cases where the specification does not require an error, but behaves differently in at least one case where it requires an error. It is a partial version of either of these if it doesn't conform to the description but differs from a conforming implementation only by rejecting with an error some programs that the conforming implementation accepts. As the specification is not yet versioned, other instances of the specification define these terms in different ways. An implementation can use one of these terms if it conforms to any instance of the pre-versioning BQN specifications that defines them. When versioning is begun, there will be only one specification for each version.

            The following documents are included in the BQN specification. A BQN program is a sequence of Unicode code points: to evaluate it, it is converted into a sequence of tokens using the token formation rules, then these tokens are arranged in a syntax tree according to the grammar, and then this tree is evaluated according to the evaluation semantics. The program may be evaluated in the presence of additional context such as a filesystem or command-line arguments; this context is presented to the program and manipulated through the system-provided values.

            diff --git a/docs/spec/inferred.html b/docs/spec/inferred.html index c25ad29c..c0deb923 100644 --- a/docs/spec/inferred.html +++ b/docs/spec/inferred.html @@ -4,11 +4,11 @@ Specification: BQN inferred properties -

            Specification: BQN inferred properties

            +

            Specification: BQN inferred properties

            BQN includes some simple deductive capabilities: detecting the type of empty array elements, the result of an empty reduction, and the Undo () and Under () modifiers. These tasks are a kind of proof-based or constraint programming, and can never be solved completely (some instances will be undecidable) but can be solved in more instances by ever-more sophisticated algorithms. To allow implementers to develop more advanced implementations while offering some stability and portability to programmers, two kinds of specification are given here. First, constraints are given on the behavior of inferred properties. These are not exact and require some judgment on the part of the implementer. Second, behavior for common or useful cases is specified more precisely. Non-normative suggestions are also given as a reference for implementers.

            For the specified cases, the given functions and modifiers refer to those particular representations. It is not necessary to detect equivalent representations, for example to reduce (+-×) to . However, it is necessary to identify computed functions and modifiers: for example F when the value of F in the expression is , or (1⊑∧).

            Failing to compute an inferred property for a function or array as it's created cannot cause an error. An error can only be caused when the missing inferred property is needed for a computation.

            -

            Identities

            +

            Identities

            When monadic Fold (´) or Insert (˝) is called on an array of length 0, BQN attempts to infer a right identity value for the function in order to determine the result. A right identity value for a dyadic function 𝔽 is a value r such that ee𝔽r for any element e in the domain. For such a value r, the fold r 𝔽´ l is equivalent to 𝔽´ l for a non-empty list l, because the first application (¯1l) 𝔽 r gives ¯1l, which is the starting point when no initial value is given. It's thus reasonable to define 𝔽´ l to be r 𝔽´ l for an empty list l as well, giving a result r.

            For Fold, the result of 𝔽´ on an empty list is defined to be a right identity value for the range of 𝔽, if exactly one such value exists. If an identity can't be proven to uniquely exist, then an error results.

            For Insert, 𝔽˝ on an array of length 0 is defined similarly, but also depends on the cell shape 1↓≢𝕩. The required domain is the arrays of that shape that also lie in the range of 𝔽 (over arbitrary arguments, not shape-restricted ones). Furthermore, an identity may be unique among all possible arguments as in the case of Fold, or it may be an array with shape 1↓≢𝕩 and be unique among arrays with that shape. For example, with cell shape 32, all of 0, 20, and 320 are identities for +, but 320 can be used because it is the only indentity with shape 32, while the other identities aren't unique and can't be used.

            @@ -68,11 +68,11 @@

            Additionally, the identity of ˝ must be recognized: if 0=≠𝕩 and 1<=𝕩, then ˝𝕩 is (02↓≢𝕩)𝕩. If 1==𝕩, then there is no identity element, as the result of always has rank at least 1, but the cell rank is 0.

            -

            Fill elements

            +

            Fill elements

            Any BQN array can have a fill element, which is a sort of "default" value for the array. The reference implementations use Fill to access this element, and it is used primarily for Take (), First (), and Nudge («, »). One way to extract the fill element of an array a in BQN is 0a.

            A fill element can be either 0, ' ', or an array of valid fill elements. If the fill element is an array, then it may also have a fill element (since it is an ordinary BQN array). The fill element is meant to describe the shared structure of the elements of an array: for example, the fill element of an array of numbers should be 0, while the fill element for an array of variable-length lists should probably be ⟨⟩. However, the fill element, unlike other inferred properties, does not satisfy any particular constraints that relate it to its array. The fill element of a primitive's result, including functions derived from primitive modifiers, must depend only on its inputs.

            In addition to the requirements below, the fill element for the value of a string literal is ' '.

            -

            Required functions

            +

            Required functions

            Combinators ⊣⊢!˙˜´˝∘○⊸⟜⊘◶⍟ do not affect fill element computation: if the combinator calls a function that computes a fill element, then that fill element must be retained if the result is passed to other functions or returned. constructs arrays if its right operand is or contains arrays, and the fill elements of these arrays are not specified; converting 𝕩 to a fill element is a reasonable choice in some cases but not others.

            Arithmetic primitives—all valences of +-×÷⋆√⌊⌈|¬ and dyadic ∧∨<>≠=≤≥—obtain their fill elements by applying to the fill elements of the arguments. If this is an error, there is no fill element; otherwise, the fill element is the result, with all numbers in it changed to 0 and all characters changed to ' '.

            Fill elements for many primitives are given in the table below. The "Fill" column indicates the strategy used to compute the result's fill. Fields 0, 𝕩, 0𝕩, and 00𝕩 indicate the fill directly, while and indicate that the fill is to be computed from the argument fills (if not all arguments have fills, then the fill element is unspecified). For , the fill element of the result is the fill element of 𝕩. For , the fill is equal to the fill values for multiple arrays, provided that they are all equal (it's unspecified if they are not all equal). In the two argument case, these arrays are 𝕨 and 𝕩. In the one-argument case, they are the elements of 𝕩; however, if 𝕩 is empty, then the result's fill is the fill of the fill of 𝕩.

            @@ -120,11 +120,11 @@

            For Group and Group Indices (), the fill element of the result and its elements are both specified: the fill element of each element of the result is the same as that of 𝕩 for Group, and is 0 for Group Indices. The fill element of the result is (01𝕨)𝕩 for Group, and <01𝕩 for Group Indices.

            Fill elements of iteration modifiers such as ¨⌜ are not specified. It is reasonable to define the fill element of 𝔽 or 𝔽¨ to be 𝔽 applied to the fill elements of the arguments. Regardless of definition, computing the fill element cannot cause side effects or an error.

            -

            Undo

            +

            Undo

            The Undo 1-modifier , given an operand 𝔽 and argument 𝕩, and possibly a left argument 𝕨, finds a value y such that 𝕩𝕨𝔽y, that is, an element of the pre-image of 𝕩 under 𝔽 or 𝕨𝔽⊢. Thus it satisfies the constraint 𝕩 𝕨𝔽𝕨𝔽𝕩 (𝕨𝔽 is a right inverse of 𝕨𝔽⊢) provided 𝔽 and 𝔽 both complete without error. 𝔽 should of course give an error if no inverse element exists, and can also fail if no inverse can be found. It is also preferred for 𝔽 to give an error if there are many choices of inverse with no clear way to choose one of them: for example, 00m returns the diagonal of matrix m; 0023 requires values to be chosen for the off-diagonal elements in its result. It is better to give an error, encouraging the programmer to use a fully-specified approach like 23(00) applied to a matrix of initial elements, than to return a result that could be very different from other implementations.

            When working with limited-precision numbers, it may be difficult or impossible to exactly invert the operand function. Instead, it is generally acceptable to perform a computation that, if done with unlimited precision, would exactly invert 𝔽 computed with unlimited precision. This principle is the basis for the numeric inverses specified below. It is also acceptable to find an inverse by numeric methods, provided that the error in the inverse value found relative to an unlimited-precision inverse can be kept close to the inherent error in the implementation's number format.

            Regardless of which cases for Undo are supported, the result of a call, and whether it is an error, must depend only on the values of the inputs 𝔽, 𝕩, and (if present) 𝕨.

            -

            Required functions

            +

            Required functions

            Function inverses are given for one or two arguments, with cases where inverse support is not required left blank.

            For arithmetic functions the implementations below may in some cases not give the closest inverse (that is, there may be some other y so that F y is closer to x than F Fx). Even in these cases the exact functions given below must be used.

            @@ -247,7 +247,7 @@
            -

            Optional functions

            +

            Optional functions

            Several primitives are easily and uniquely undone, but doing so is not important for BQN programming. These primitives are listed below along with suggested algorithms to undo them. Unlike the implementations above, these functions are not valid in all cases, and the inputs must be validated or the results checked in order to use them.

            @@ -300,7 +300,7 @@
            -

            Required modifiers

            +

            Required modifiers

            The following cases of Self/Swap must be supported.

            @@ -464,9 +464,9 @@
            -

            Undo headers

            +

            Undo headers

            An UndoHead header specifies how a block function acts when undone. Like ordinary headers, undo headers are searched for a match when a block function F is undone, or when F˜ is undone with two arguments (including the two modifier cases 𝔽k and 𝔽𝔾k from the previous section). An UndoHead without "˜" matches the F case while one with "˜" matches the F˜⁼ case. The left and right arguments are matched to headW and headX as with ordinary headers, and the first matching case is evaluated to give the result of the Undo-derived function.

            -

            Under

            +

            Under

            The Under 2-modifier conceptually applies its left operand under the action of its right operand. Setting z𝕨𝔽𝔾𝕩, it satisfies (𝕨𝔽𝔾𝕩) 𝔾z. We might say that 𝔾 transforms values to a new domain, and 𝔾 lifts actions 𝔽 performed in this domain to the original domain of values. For example, addition in the logarithmic domain corresponds to multiplication in the linear domain: +() is × (but less precise if computed in floating point).

            Let v𝕨𝔽𝔾𝕩, so that v≡𝔾z. v is of course well-defined, so the inference step is to find z based on v and possibly the original inputs. We distinguish three cases for Under:

              @@ -475,7 +475,7 @@
            • Computational Under: If 𝔾 is provably not a structural function, then the result is 𝔾v if it is defined.

            When implementing, there is no need to implement invertable Under specially: it can be handled as part of the structural and computation cases.

            -

            Mathematical definition of structural Under

            +

            Mathematical definition of structural Under

            In general, structural Under requires information from the original right argument to be computed. Here we will define the structural inverse of structural function 𝔾 on v into 𝕩, where 𝕩 gives this information. The value 𝕨𝔽𝔾𝕩 is then the structural inverse of 𝔾 on 𝕨𝔽𝔾𝕩 into 𝕩.

            We define a structure to be either the value · or an array of structures (substitute 0 or any other specific value for · if you'd like structures to be a subset of BQN arrays; the value is irrelevant). A given structure s captures a BQN value or structure 𝕩 if it is ·, or if s and 𝕩 are arrays of the same shape, and each element of s captures the corresponding element of 𝕩. Thus a structure shares some or all of the structural information in arrays it captures, but none of the data.

            A structure transformation consists of an initial structure s and a result structure t, as well as a relation between the two: each instance of · in t is assigned the location of an instance of · in s. If s captures a value 𝕩, we say that the structural transformation captures 𝕩 as well. Given such a value 𝕩, the transformation is applied to 𝕩 by replacing each · in t with the corresponding value from 𝕩, found by taking the same location in 𝕩 as the one in s given by the transformation.

            @@ -488,10 +488,10 @@

            Following this analysis, z can be constructed by replacing each instance of · in s with the component of 𝕩 or v indicated, and it follows that z is well-defined if it exists—and it exists if and only if t captures v and values in v that correspond to the same position in s have the same value.

            A structural function decomposition is a possibly infinite family of structure transformations such that any possible BQN value is captured by at most one of these transformations. It can be applied to any value: if some transformation captures the value, then apply that transformation, and otherwise give an error. A function is a structural function if there is a structural function decomposition that matches it: that is, for any input either both functions give an error or the results match.

            For a structural function 𝔾, the structural inverse of 𝔾 on v into 𝕩 is the inverse of G on v into 𝕩, where G is the structure transformation that captures 𝕩 from some structural function decomposition Gd matching 𝔾. If no decomposition has an initial structural matching 𝕩 then the structural inverse does not exist.

            -

            Well-definedness

            +

            Well-definedness

            In order to show that the structural inverse of a structural function is well-defined, we must show that it does not depend on the choice of structural function decomposition. That is, for a given 𝕩, if G and H are structure transformations from different decompositions of 𝔾 both capturing 𝕩, then the structural inverse of G on v into 𝕩 matches that of H on v into 𝕩. Call these inverses y and z. Now begin by supposing that H captures y and G captures z; we will show this later. From the definition of a structural inverse, v≡G y, so that v≡𝔾 y, and because H captures y we know that 𝔾 y is H y, so we have v≡H y as well. Let S w indicate the set of all structure transformations F such that w F 𝕩 (this is not a BQN value, both because it is a set and because it's usually infinite): from the definition of z we know that S z is a strict superset of S w for any w other than z with v≡H w. It follows that either yz or S y is a strict subset of S z. By symmetry the same relation holds exchanging y and z, but it's not possible for S y to be a strict subset of S z and vice-versa. The only remaining possibility is that yz.

            We now need to show that H captures y (the proof that G captures z is of course the same as H and G are symmetric). To do this we must show that any array in the initial structure of H corresponds to a matching array in y. For convenience, we will call the initial structures of the two transformations iG and iH, and the final structures fG and fH, and use the notation pa to indicate the value of array a at position p. Choose the position of an array in H, and assume by induction that each array containing it already has the desired property; this implies that this position exists in y as well although we know nothing about its contents. G captures y, so iG is · at this position or some parent position; call this position in iG p. There are now two cases: either G makes use of this p—at least one position in fG corresponds to it—or it doesn't. If it doesn't, then the contents of y at p are the same as those of 𝕩. Since H captures 𝕩, iH matches 𝕩 and hence y as well at p. If it does, then let s be a position in fG that corresponds to p (if there are multiple possibilities, choose one). From v≡G y, we know that sv matches py. We know that fH captures v, so that sfH captures sv, or py. But we can show that the value of sfH is the same as piH, which would prove that H captures y at p. To show this, construct an array xp by replacing the value of 𝕩 at p with piH (to be more careful in our handling of types, we might replace every · with some value that never appears in 𝕩). Both H and G capture xp: clearly they capture it outside p, while at p itself, iG is · and iH is equal to pxp. Now (H xp)(G xp) because both functions match 𝔾 on their domains. Therefore s⊑H xp matches s⊑G xp, which by the definition of s matches pxp, which matches piH. But s⊑H xp comes from replacing each atom in sfH with an atom in xp that's captured by a · in iH. Because it matches piH, every atom in s⊑H xp is ·, but the only instances of · in xp come from our inserted copy of piH and each is immediately captured by the corresponding · in iH. It follows that s⊑H xp, and consequently sfH, is exactly piH, completing the proof.

            -

            Required structural inverses

            +

            Required structural inverses

            The following primitive functions be fully supported by structural Under. Each manipulates its right argument structurally.

            @@ -570,7 +570,7 @@
            -

            A structural Under algorithm

            +

            A structural Under algorithm

            This section offers the outline for a procedure that computes most structural inverses that a programmer would typically use. The concept is to build a special result array whose elements are not BQN values but instead indicate positions within the initial argument. This structural array is applied to the initial argument by replacing its elements with the values at those positions, and inverted by placing elements back in the original array at these indices, checking for any conflicts. If operations like dyadic are allowed, then a structural array might have some indices that are prefixes or parents of others, making it slightly different from a structural transformation as defined above (although it could be represented as a structural transformation by expanding some of these). This requires additional checking to ensure that elements of previously inserted elements can't be modified.

            Structural functions can be applied to structural arrays directly, after ensuring that they have the necessary depth as given below. An array's depth can be increased by expanding each position in it into an array of child positions, or, if that position contains an atom and the structural function in question would tolerate an atom, enclosing it.

            @@ -610,7 +610,7 @@

            Not all primitives in the table above are required. Of note are =≠≢, which accept a structural array but return an ordinary value; this might be used as a left argument later. If the final result is not structural, then the function in question can't be structural, and the attempt to find a structural inverse can be aborted.

            -

            Non-structural case

            +

            Non-structural case

            The behavior of invertible and computational Under is fully dependent on that of Undo, and does not need to be repeated here. However, it is important to discuss when this definition can be applied: specifically, either

            • When 𝔾 is exactly invertible, or
            • diff --git a/docs/spec/literal.html b/docs/spec/literal.html index 3b2b248a..35a280cc 100644 --- a/docs/spec/literal.html +++ b/docs/spec/literal.html @@ -4,7 +4,7 @@ Specification: BQN literal notation -

              Specification: BQN literal notation

              +

              Specification: BQN literal notation

              A literal is a single token that indicates a fixed character, number, or array. While literals indicate values of a data type, primitives indicate values of an operation type: function, 1-modifier, or 2-modifier.

              Two types of literal deal with text. As the source code is considered to be a sequence of unicode code points ("characters"), and these code points are also used for BQN's character data type, the representation of a text literal is very similar to its value. In a text literal, the newline character is always represented using the ASCII line feed character, code point 10. A character literal is enclosed with single quotes ' and its value is identical to the single character between them. A string literal is enclosed in double quotes ", and any double quotes between them must come in pairs, as a lone double quote marks the end of the literal. The value of a string literal is a rank-1 array whose elements are the characters in between the enclosing quotes, after replacing each pair of double quotes with only one such quote. The null literal is the token @ and represents the null character, code point 0.

              The format of a numeric literal is more complicated. From the tokenization rules, a numeric literal consists of a numeric character (one of ¯∞π.0123456789) followed by any number of numeric or alphabetic characters. Some numeric literals are valid and indicate a number, while others are invalid and cause an error. The grammar for valid numbers is given below in a BNF variant. Only four alphabetic characters are allowed: "i", which separates the real and imaginary components of a complex number, "e", which functions as in scientific notation, and the uppercase versions of these letters. Not included in this grammar are underscores—they can be placed anywhere in a number, including after the last non-underscore character, and are ignored entirely.

              diff --git a/docs/spec/primitive.html b/docs/spec/primitive.html index c3f4ab2f..128fd24f 100644 --- a/docs/spec/primitive.html +++ b/docs/spec/primitive.html @@ -4,11 +4,11 @@ Specification: BQN primitives -

              Specification: BQN primitives

              +

              Specification: BQN primitives

              Most primitives are specified by the BQN-based implementation in reference.bqn. This document specifies the basic functionality required by those definitions. Descriptions of other primitives are for informational purposes only.

              -

              Pervasive primitives

              +

              Pervasive primitives

              Functions in this section are defined for atoms only; the reference implementations extend them to arrays.

              -

              Arithmetic

              +

              Arithmetic

              BQN uses five arithmetic functions that are standard in mathematics. The precision of these operations should be specified by the number type.

              • Add +
              • @@ -25,7 +25,7 @@

              In the first two cases, if the result would not be a valid Unicode code point, then an error results. The remaining cases of + and - (adding two characters; negating a character or subtracting it from a number) are not allowed.

              Additionally, the Floor function returns the largest integer smaller than or equal to the argument, or the argument itself if it is ¯∞ or . It's needed because the arithmetic operations give no fixed-time way to determine if a value is an integer. Floor gives an error if the argument is an atom other than a number.

              -

              Comparison

              +

              Comparison

              Two kinds of comparison are needed to define BQN's primitives: equality comparison and ordered comparison.

              Ordered comparison is simpler and is provided by the dyadic Less than or Equal to () function. This function gives an error if either argument is an operation, so it needs to be defined only for numbers and characters. For numbers it is defined by the number system, and for characters it returns 1 if the left argument's code point is less than that of the right argument. Characters are considered greater than numbers, so that nc is 1 and cn is 0 if c is a character and n is a number.

              The dyadic function =, representing equality comparison, can be applied to any two atoms without an error. Roughly speaking, it returns 1 if they are indistinguishable within the language and 0 otherwise. If the two arguments have different types, the result is 0; if they have the same type, the comparison depends on type:

              @@ -40,7 +40,7 @@
            • Block instances are equal if they are the same instance.

            This means that block instance equality indicates identity in the context of mutability: two block instances are equal if any change of state in one would be reflected in the other as well. The concept of identity holds even if the blocks in question have no way of changing or accessing state. For example, ={𝕩{𝕩}}˜@ is 0 while =˜{𝕩{𝕩}}@ is 1.

            -

            Array functionality

            +

            Array functionality

            Several subsets of primitives, or dedicated operations, are used to manipulate arrays in the reference implementation.

            • IsArray returns 1 if the argument is an array and 0 if it's an atom.
            • @@ -57,23 +57,23 @@
            • Pick () selects the element at index 𝕨 from list 𝕩.
            • _amend returns an array identical to list 𝕩 except that the element at index 𝕗 is changed to 𝕨.
            -

            Inferred functionality

            +

            Inferred functionality

            Inferred properties are specified in their own document, not in the reference implementation.

            • Identity gives the identity value for reduction by function 𝕏.
            • Undo () gives a partial right inverse for function 𝔽.
            • Fill gives the enclose of the fill value for array 𝕩.
            -

            Other provided functionality

            +

            Other provided functionality

            • Assert (!) causes an error if the argument is not 1. If 𝕨 is provided, it gives a message to be associated with this error (which can be any value, not necessarily a string).
            • Catch () evaluates 𝔽 on the arguments 𝕨 (if present) and 𝕩. If 𝔽 completes without error it returns the result, but if evaluation of 𝔽 results in an error then the error is suppressed, and Catch evaluates 𝔾 on the arguments and returns the result. Errors in 𝔾 are not caught. Catch only prevents evaluation errors, and not syntax errors: these are considered errors in the program as a whole rather than any particular part of it.
            -

            Commentary on other primitives

            +

            Commentary on other primitives

            As noted above, see reference.bqn for the authoritative definitions. Commentary here gives an overall description and highlights implementation subtleties and edge cases.

            -

            Combinators

            +

            Combinators

            There's little to say about BQN's true combinators, since each is simply a pattern of function application. All primitive combinators use their operands as functions, and thus treat a data operand as a constant function.

            • Choose () is later redefined to use the complete rather than the simple version assumed (using this primitive means it's not a true combinator).
            • @@ -88,7 +88,7 @@
            • After/Bind ()

            The somewhat complicated definition of Valences could be replaced with {𝔽𝕩;𝕨𝔾𝕩} using headers. However, reference.bqn uses a simple subset of BQN's syntax that doesn't include headers. Instead, the definition relies on the fact that 𝕨 works like · if no left argument is given: (1˙𝕨)-0 is 1-0 or 1 if 𝕨 is present and (1˙·)-0 otherwise: this reduces to ·-0 or 0.

            -

            Array properties

            +

            Array properties

            The reference implementations extend Shape () to atoms as well as arrays, in addition to implementing other properties. In all cases, an atom behaves as if it has shape ⟨⟩. The functions in this section never cause an error.

            • Shape () gives the shape of an array or atom.
            • @@ -96,7 +96,7 @@
            • Length () gives the number of major cells, or 1 for an argument of rank 0.
            • Depth () gives the nesting depth. It ignores the shapes of arrays, and considering only the depths of their elements.
            -

            Arithmetic

            +

            Arithmetic

            Arithmetic functions not already provided are defined in layer 1. These definitions, like the provided functions, apply to atoms only; they should be extended to arrays using the _perv modifier from layer 2.

            • Sign (×)
            • @@ -109,7 +109,7 @@
            • Modulus (|) is an extension of modular division to real numbers. As it uses floor instead of truncation, it's not the same as the % operator from C or other languages when 𝕨<0.
            • Comparisons Less Than (<), Greater Than (>), Greater Than or Equal to (), and Not Equals () are defined in terms of the two provided comparisons.
            -

            Iteration modifiers

            +

            Iteration modifiers

            Modifiers for iteration are defined in layers 1, 2, and 4. Two 2-modifiers, and , use a list of numbers obtained by applying the right operand to the arguments in order to control application. This list has one to three elements: if all three are given then they correspond to the monadic, left, and right arguments; if one is given then it controls all three; and if two are given then they control the left argument, and the right and monadic arguments.

            The iteration modifiers ⌜¨˘ process elements or cells in index order, that is, according to lexicographic ordering of indices or according to simple numeric ordering of the indices in the Deshaped () arguments. When both arguments are mapped over independently, the left argument is mapped over "first", or as an outer loop: one part of the left argument is paired with each part of the right in turn, then the next part of the left argument, and so on.

            Table () and Each (¨) map over the elements of arrays to produce result elements. They convert atom arguments to unit arrays. With one argument, the two modifiers are the same; with two, they differ in how they pair elements. Table pairs every element of the left argument with every element of the right, giving a result shape 𝕨𝕩. Each uses leading axis agreement: it requires one argument's shape to be a prefix of the other's (if the arguments have the same rank, then the shapes must match and therefore be mutual prefixes). This causes each element of the lower-rank argument to correspond to a cell of the higher-rank one; it's repeated to pair it with each element of that cell. The result shape is the shape of the higher-rank argument.

            @@ -118,13 +118,13 @@

            Fold (´), Insert (˝), and Scan (`) repeatedly apply a function between parts of an array. Fold requires the argument to have rank 1 and applies the operand between its elements, while Insert requires it to have rank 1 or more and applies it between the cells. For each of these two functions, the operand is applied beginning at the end of the array, and an identity value is returned if the array is empty. While these functions reduce multiple values to a single result, Scan returns many results and preserves the shape of its argument. It requires the argument to have rank at least 1, and applies the function between elements along columns—that is, from one element in a major cell to the one in the same position of the next major cell. This application begins at the first major cell of the array. Scan never uses the identity element of its operand because if the argument is empty then the result, which has the same shape, will be empty as well.

            A left argument for any of the three reduction-based modifiers indicates an initial value to be used, so that the first application of the operand function applies not to two values from 𝕩 but instead to a value from 𝕨 and a value from 𝕩. In Fold and Insert, the entire value 𝕨 is the initial value, while in Scan, 𝕨 is an array of initial values, which must have shape 1↓≢𝕩.

            Repeat () applies the operand function, or its inverse, several times in sequence. The right operand must consist only of integer atoms (arranged in arrays of any depth), and each number there is replaced with the application of the left operand that many times to the arguments. If a left argument is present, then it's reused each time, as if it were bound to the operand function. For a negative number -n, the function is "applied" -n times by undoing it n times. In both directions, the total number of times the function is applied is the maximum of all numbers present: results must be saved if intermediate values are needed.

            -

            Restructuring

            +

            Restructuring

            Enclose (<) forms a unit array that contains its argument.

            Merge (>) combines the outer axes of an array of arrays with inner axes: it requires that all elements of its argument have the same shape, and creates an array such that (ij)⊑>𝕩 is ij𝕩. It also accepts atom elements of 𝕩, converting them to unit arrays, or an atom argument, which is returned unchanged. Solo and Couple () turn one or two arguments into major cells of the result and can be defined easily in terms of Merge.

            Join To () combines its two arguments along an existing initial axis, unless both arguments are units, in which case it creates an axis and is identical to Couple (). The arguments must differ in rank by at most 1, and the result rank is equal to the maximum of 1 and the higher argument rank. Each argument with rank less than the result, and each major cell of an argument with rank equal to it, becomes a major cell of the result, with cells from the left argument placed before those from the right. Join () generalizes the equal-rank subset of this behavior to an array of values instead of just two. The argument must be an array (unlike Merge), and its elements must all the same rank, which is at least the argument rank. Atom elements are treated as unit arrays. Then "outer" argument axes are matched up with leading "inner" element axes, and elements are joined along these axes. In order to allow this, the length of an element along a particular axis must depend only on the position along the corresponding axis in the argument. An empty argument to Join is return unchanged, as though the element rank is equal to the argument rank.

            Deshape () differs from the provided function (which returns the element list of an array) only in that it accepts an atom, returning a one-element list containing it. Reshape () is extended in numerous ways. It accepts any list of natural numbers (including as a unit array or atom) for the left argument and any right argument; 𝕩 is deshaped first so that it is treated as a list of elements. These elements are repeated cyclically to fill the result array in ravel order. If 𝕩 is empty then a non-empty requested result shape causes an error. Furthermore, at most one element of 𝕨 can be a "length code": one of the primitives ⌊⌽↑. In this case, a target length is computed from the number of elements in 𝕩 divided by the product of the other elements of 𝕨 (which must not be zero). If the target length is an integer then it is used directly for the length code. Otherwise, an error is given if the length code is , and the target length is rounded down if the code is and up if it's or . With code , elements are repeated cyclically as usual, but with code , the extra elements after each argument element is used are fill values for 𝕩.

            Transpose () reorders axes of its argument to place the first axis last; if the argument has one or fewer axes then it's enclosed if it's an atom and otherwise returned unchanged. Reorder Axes () requires the left argument to be a list or unit of natural numbers, with length at most the rank of the right argument. This list is extended to match the right argument rank exactly by repeatedly appending the least unused natural number (for example, given 1300, 2 is appended). After extension, it specifies a result axis for each axis of the right argument. There must be no gaps in the list: that is, with the result rank equal to one plus the greatest value present, every result axis must appear at least once. Now each argument axis is "sent to" the specified result axis: in terms of indices, i𝕨𝕩 is (𝕨i)𝕩 if 𝕨 is complete. If multiple argument axes correspond to the same result axis, then a diagonal is taken, and it's as long as the shortest of those argument axes. Like Transpose, Reorder Axes encloses 𝕩 if it's an atom, so that its result is always an array.

            -

            Indices and selection

            +

            Indices and selection

            Each element in an array se is associated with an index, which is a list of natural numbers i such that ´i<s. The list of all indices, which corresponds to the element list e, contains all such lists i in lexicographic order. That is, index i comes before j exactly when the two indices are not the same, and i has the smaller value at the first position where they are unequal. The index of an element along a particular axis a is the value ai.

            Range () is extended to apply to a list of natural numbers, in addition to the provided case of a single natural number (an enclosed natural number 𝕩 should still result in an error). For a list 𝕩, the result is an array of shape 𝕩 in which the value at a given index is that index, as a list of natural numbers. That is, ii⊑↕𝕩 for any list of natural numbers i with ´i<𝕩.

            Pick () is extended to array left arguments. In this case, it requires every depth-1 array in the nested structure of 𝕨 to be a valid index list for 𝕩, and every atom to be contained in one of these lists. The result is 𝕨 with each index list replaced by the element of 𝕩 at that index. In the simple case where 𝕨 itself is an index list, the result is the element of 𝕩 at index 𝕨.

            @@ -133,14 +133,14 @@

            First Cell () selects the initial major cell of 𝕩, giving an error if 𝕩 has rank 0 or length 0.

            Group () performs an opposite operation to Select, so that 𝕨 specifies not the argument index that result values come from, but the result index that argument values go to. The general case is that 𝕨 is a list of arrays of numbers; if it has depth less than 2 it's converted to this form by first enclosing it if it's an atom, then placing it in a length-1 list. After this transformation, the result rank is 𝕨, and each result element has rank (𝕨)+(=𝕩)-+´=¨𝕨, with the initial 𝕨 axes corresponding to elements of 𝕨 and the remainder to trailing axes of 𝕩. Each atom in 𝕨 can be either a natural number or ¯1 (which indicates the corresponding position in 𝕩 will be omitted). If ¯1 doesn't appear, the result has the property that each cell of 𝕩 appears in the corresponding element of 𝕨𝕨𝕩. More concretely, the length of the result along axis a is the maximum value in a𝕨 plus one, or zero if a𝕨 is empty. Axis a corresponds to =a𝕨 axes in 𝕩, and an element of the result at position i along this axis contains all positions in 𝕩 where i=a𝕨. There may be multiple such positions, and they're arranged along axis a of that result element according to their index order in 𝕩. The shapes of components of 𝕨 must match the corresponding axes of 𝕩, except for rank-1 components of 𝕨, which can match or have an extra element. This element, which like the others is either a natural number or ¯1, gives the minimum length of the result axis corresponding to the component of 𝕨 in question, but otherwise does not affect the result. Group Indices treats its argument 𝕩 as a left argument for Group and uses a right argument made up of indices, which is ↕≠𝕩 if 𝕩 has depth 1 and ↕∾≢¨𝕩 if it has depth 2. Because the depth-1 case uses atomic indices, 𝕩 is required to be a list (and it can't be an atom). Much like Range, the result has depth one higher than the argument.

            Indices (/) applies to a list of natural numbers, and returns a list of natural numbers. The result contains i𝕩 copies of each natural number index i for 𝕩, in increasing order.

            -

            Structural manipulation

            +

            Structural manipulation

            Monadic structural functions work on the first axis of the argument, so they require it to have rank at least 1. Reverse () reverses the ordering of the major cells of 𝕩. Nudge (») shifts them forward, removing the last and placing a major cell made up of fill elements at the beginning, while Nudge Back («) does the same in the reverse direction, so it removes the first cell and places fills at the end. Prefixes () and Suffixes () each return lists with length one higher than 𝕩, whose elements are arrays with the same rank as 𝕩. For Prefixes, the element of the result at index i contains the first i major cells of 𝕩 in order, and for Suffixes, it contains all but these major cells.

            The remainder of the functions discussed in this section are dyadic. For all of these, an atom value for 𝕩 is treated as an array by enclosing it before acting, so that the result is never an atom. There are four functions for which 𝕨 is a list of whole numbers—but an atomic number or enclosed number is also permitted, and treated as a 1-element list—and its elements are matched with leading axes of 𝕩. These functions independently manipulate each axis: one way to define such a process is to consider lists running along the axis, where every element of the index is fixed except one. A change to this axis retains the fixed indices, but can move elements from one location to another along the variable index, add fill elements, or split the axis into two axes. A change to a different axis can rearrange these lists along the original axis, but can't affect the placement of elements within them. In the reference implementations, working on leading axes is accomplished using the Cells (˘) modifier recursively, so that action on the first axes doesn't use Cells, on the next is affected by Cells once, then twice, and so on.

            Rotate () is the simplest of these four functions: each element of 𝕨 gives an amount to rotate the corresponding axis, where a rotation of r moves the element at index i+r to i when all indices are taken modulo the length of the axis. Windows () splits each axis of 𝕩 that corresponds to an element of 𝕨 in two, so that the result has one set of axes corresponding to elements of 𝕨, then another, then the unchanged trailing axes. The second set of axes has lengths given by 𝕨 (which must consist of natural numbers), while the first has lengths s¬𝕨, where s contains the lengths of leading axes of 𝕩. Position i in the first set of axes and j in the second corresponds to i+j in the argument, so that fixing one of these positions and varying the other gives a slice of the argument. In both Rotate and Windows, the length of 𝕨 is at most the rank of 𝕩.

            Take () offers several possibilities. The absolute value of 𝕨 gives the final lengths of the axes in the result. It may be positive to indicate that the axis aligns with 𝕩 at the beginning, or negative to indicate it aligns at the end. A zero value gives no result elements, so there is no need to consider alignment. If the absolute value of an element of 𝕨 is smaller than or equal to the corresponding length in 𝕩, then the first or last few elements are taken along that axis. If it is larger, then instead fill elements are added to the end (if positive) or beginning (if negative) to make up the difference in length. Drop () gives 𝕨 a similar meaning, but excludes all elements that Take includes (maintaining the order of the retained ones). The result of Drop never uses fill elements. In a case where Take would use fill elements, it would include all positions from 𝕩, so Drop should include none of them, and the result will have length 0 for that axis. Take and Drop are extended to allow an argument with length greater than the rank of 𝕩. In this case leading length-1 axes are added to 𝕩 so that its rank matches 𝕨 before taking or dropping.

            Replicate (/) is similar to the four dyadic structural functions above, but 𝕨 gives a list of containing lists of natural numbers, or plain or enclosed natural numbers, instead of a simple list. If 𝕨 has depth less than 2, it's considered to be a single value corresponding to one axis of 𝕩, while if it has depth 2 then it's a list of values. If 𝕨 is the empty list ⟨⟩ then it is defined to be in the second case despite having a depth of 1. On a single axis of 𝕩 the corresponding value r from 𝕨 is either a list or a unit: if it's a unit then it is repeated to match the length of that axis of 𝕩, and if it's a list it must already have the same length as that axis. Each number in r now specifies the number of times to repeat the corresponding position in 𝕩. This is equivalent to calling Indices on r and using the result for selection.

            Shift Before (») and Shift After («) are derived from Join To and share most of its behavior. The difference is that only a portion of the result of Join To is returned, matching the length of 𝕩. This portion comes from the beginning for Shift Before and the end for Shift After. The only difference in conditions between the shift functions and Join To is that Join To allows the result to have higher rank than 𝕩. Shifts do not, so the rank of 𝕩 be at least 1 and at least as high as 𝕨.

            -

            Searching

            +

            Searching

            Match () indicates whether two values are considered equivalent. It always returns 0 or 1, and never causes an error. If both arguments are atoms then it is identical to =, and if one is an atom and the other an array then it returns 0. If both arguments are arrays then it returns 1 only if they have the same shape and all pairs of corresponding elements match. Fill elements aren't taken into account, so that arrays that match might still differ in behavior. Not Match simply returns the complement of Match, ¬≡.

            Monadic search functions compare the major cells of 𝕩 to each other. 𝕩 must have rank at least 1. Except for Deduplicate (), the result is a list of numbers with the same length as 𝕩.

              @@ -156,7 +156,7 @@
            • Progressive Index of () processes non-principal cells in ravel order, and gives the smallest index of a principal argument cell that matches the cell that hasn't already been included in the result. Again 𝕨 is returned for a given cell if there is no valid cell.

            Find () indicates positions where 𝕨 appears as a contiguous subarray of a =𝕨-cell of 𝕩. It has one result element for each such subarray of 𝕩, whose value is 1 if that subarray matches 𝕩 and 0 otherwise. Find cannot result in an error unless the rank of 𝕨 is higher than that of 𝕩. If 𝕨 is longer along one axis than the corresponding trailing axis of 𝕩, then the result has length 0 along that axis. Any atom argument to Find is automatically enclosed.

            -

            Sorting

            +

            Sorting

            Sorting functions are those that depend on BQN's array ordering. There are three kinds of sorting function, with two functions of each kind: one with an upward-pointing glyph that uses an ascending ordering (these function names are suffixed with "Up"), and one with a downward-pointing glyph and the reverse, descending, ordering ("Down"). Below, these three kinds of function are described, then the ordering rules. Except for the right argument of Bins, all arguments must have rank at least 1.

            Sort (∧∨) reorders the major cells of its argument so that a major cell with a lower index comes earlier in the ordering than a major cell with a higher index, or matches it. If it's possible for matching arrays to differ in behavior because of different (including undefined versus defined) fill elements, then these arrays must maintain their ordering (a stable sort is required).

            Grade (⍋⍒) returns a permutation describing the way the argument array would be sorted. For this reason the reference implementations simply define Sort to be selection by the grade. One way to define Grade is as a sorted version of the index list ↕≠𝕩. An index i is ordered according to the corresponding major cell i𝕩. However, ties in the ordering are broken by ordering the index values themselves, so that no two indices are ever considered equal, and the result of sorting is well-defined (for Sort this is not an issue—matching cells are truly interchangeable). This property means that a stable sorting algorithm must be used to implement Grade functions. While cells might be ordered ascending or descending, indices are always ordered ascending, so that for example index i is placed before index j if either i𝕩 comes earlier in the ordering than j𝕩, or if they match and i<j.

            diff --git a/docs/spec/scope.html b/docs/spec/scope.html index be592157..5576f6c8 100644 --- a/docs/spec/scope.html +++ b/docs/spec/scope.html @@ -4,10 +4,10 @@ Specification: BQN variable scoping -

            Specification: BQN variable scoping

            +

            Specification: BQN variable scoping

            BQN uses lexical scoping for variables, where scopes correspond roughly to blocks, or pairs of curly braces separated by semicolons. At the top level in a scope, new variables are visible only after they are defined, but in the scopes it contains, all variables defined in that scope are visible. This system is specified more precisely below.

            A running BQN program manipulates variables during its execution, but it is important to distinguish these variables from the identifiers that refer to them. As defined in the tokenization rules, an identifier is a particular kind of token found in a program's source code. The lexical scoping rules in this page define which identifiers are considered the same; these identifiers will refer to the same variables when the program is run. While each variable has only one identifier, an identifier can refer to any number of variables because a new variable is created for that identifier each time its containing scope is instantiated (that is, each time the contents of the block are evaluated).

            -

            Identifier equivalence with lexical scoping

            +

            Identifier equivalence with lexical scoping

            In this section the concept of an identifier's definition, a possibly different instance of that identifier, is specified. The definition determines when identifiers refer to the "same thing". In concrete terms, identifiers with the same definition all manipulate the same variable in a particular instance of the definition's containing scope.

            A scope is a PROGRAM, brSub, FCase, FMain, _mCase, _mMain, _cCase_, _cMain_, or brNS node as defined by the BQN grammar. An identifier instance is an s, F, _m, or _c_ node; its containing scope is the "smallest" scope that contains it—the scope that contains the identifier but not any other scopes containing the identifier. An identifier instance is defined when it is contained in the left hand side of an assignment expression, that is, the leftmost component of one of the five grammatical rules with ASGN, provided that the ASGN node is "←" or "⇐", or in a scope header, that is, a component immediately preceding ":". Each identifier instance in a valid BQN program corresponds to exactly one such defined identifier, called its definition, and two instances are considered to refer to the same identifier if they have the same definition.

            Two identifier instances have the same name if their tokens, as strings, match after removing all underscores _ and ignoring case (so that the letters a to z are equal to their uppercase equivalents A to Z for this comparison). However, instances with the same name are not necessarily the same identifier, as they must also have the same definition. A defined identifier is a potential definition of another identifier instance if the two have the same name, and either:

            @@ -19,16 +19,16 @@

            The definition for an identifier is chosen from the potential definitions based on their containing scopes: it is the one whose containing scope does not contain or match the containing scope of any other potential definition. If for any identifier there is no definition, then the program is not valid and results in an error. This can occur if the identifier has no potential definition, and also if two potential definitions appear in the same scope. In fact, under this scheme it is never valid to make two definitions with the same name at the top level of a single scope, because both definitions would be potential definitions for the one that comes second in program order. Both definitions have the same containing scope, and any potential definition must contain or match this scope, so no potential definition can be selected.

            The definition of program order for identifier tokens follows the order of BQN execution. It corresponds to the order of a particular traversal of the abstract syntax tree for a program. To find the relative ordering of two identifiers in a program, we consider the highest-depth node that they both belong to; in this node they must occur in different components, or that component would be a higher-depth node containing both of them. In most nodes, the program order goes from right to left: components further to the right come earlier in program order. The exceptions are PROGRAM, BODY, NS_BODY, list, subject (for stranding), and body case (FCase, _mCase, _cCase_, FMain, _mMain, _cMain_, brSub, BrFunc, _brMod1, and _brMod2_) nodes, in which program order goes in the opposite order, from left to right (some assignment target nodes also contain lists or strands, but their ordering is irrelevant because if two identifiers with the same name appear in such a list, then it can't be a definition).

            A subject label is the s term in a brSub node. As part of a header, it can serve as the definition for an identifier. However, it's defined to be a syntax error if another instance of this identifier appears, except in a Return node (which cannot access its value).

            -

            Special names

            +

            Special names

            Special names such as 𝕩 or 𝕣 refer to variables, but have no definition and do not use scoping. Instead, they always refer to the immediately enclosing scope, and are defined automatically when the block is evaluated.

            The six special names are 𝕨𝕩𝕗𝕘𝕤𝕣, and the tokens 𝕎𝕏𝔽𝔾𝕊, _𝕣, and _𝕣_ are alternate spellings of these names as described in the tokenization rules. Special names may be modified with assignment but cannot appear as the target of other kinds of assignment. Two special names represent the same identifier if they are the same name and appear in the same body. The initial value these names have is defined by the evaluation rules; the grammar for blocks ensures that all special names used in a block will be defined (possibly as the special value · in the case of 𝕨).

            -

            Imports and exports

            +

            Imports and exports

            Names that are preceded by an atom "." term, or that appear as LHS_NAME terms in an NS_VAR or lhsNs, are variable references in a namespace: in the first case, the result of the atom node, and in the second, of the overall assignments subExpr right hand side. These names do not follow lexical scoping; in general they must be stored in order to perform a name lookup when the namespace is available. Such a name in lhsNs, or in NS_VAR with no accompanying lhs "⇐" term, additionally serves as an identifier within the actual enclosing scope, which works like any other assignment.

            An identifier is exported if the ASGN node in its definition is "⇐", or if it appears anywhere in an EXPORT term. An identifier can only be exported in the scope where it is defined, and not in a containing scope. An EXPORT term that includes an identifier from such a scope causes an error.

            -

            Variables

            +

            Variables

            A variable is an entity that permits two operations: it can be set to a particular value, and its value can be obtained, resulting in the last value it was set to. When either operation is performed it is referred to as accessing the variable.

            When a body in a block is evaluated, it creates a namespace, which contains a variable for each definition (that is, defined identifier instance) the body contains. Whenever another block—the block itself, not its contents—is evaluated during the execution of the block, it is linked to the currently-evaluating block, so that it will use the variables defined in this instance. By following these links repeatedly, an instance of a block is always linked to exactly one instance of each block that contains it. These links form a tree that is not necessarily related to the call stack of functions and modifiers. Using the links, the variable an identifier refers to is the one corresponding to that variable's definition in the linked instance of the containing scope for the definition.

            The first access to a variable must be made by its definition (this also means it sets the variable). If a different instance of its identifier accesses it first, then an error results. This can happen because every scope contained in a particular scope sees all the definitions it uses, and such a scope could be called before the definition is run. Because of conditional execution, this property must be checked at run time in general; however, in cases where it is possible to statically determine that a program will always violate it, a BQN instance can give an error at compile time rather than run time.

            A namespace defines a mapping from names to variables: if the given name is shared by an exported identifier in the body used to create that namespace, then that name maps to the variable corresponding to that identifier. The mapping is undefined for other names.

            -

            Returns

            +

            Returns

            The name NAME | "𝕊" | "𝕣" in a Return node is resolved exactly like any other identifier. Following resolution, the block that defines the identifier must not be a namespace block (export variables or contain an EXPORT statement). Furthermore, if it is a NAME, then its definition must be an internal name for a containing block: s in brSub, F in FuncHead or FMain, _m in Mod1H1 or _mMain, or _c_ in Mod2H1 or _cMain_. When reached, the Return node's identifier is not accessed; instead, it is used to indicate the namespace that contains it, and through this the block evaluation that created that namespace.

            diff --git a/docs/spec/system.html b/docs/spec/system.html index 53cde76c..76d36a2f 100644 --- a/docs/spec/system.html +++ b/docs/spec/system.html @@ -4,11 +4,11 @@ Specification: BQN system-provided values -

            Specification: BQN system-provided values

            +

            Specification: BQN system-provided values

            This portion of the spec is still potentially subject to major changes.

            The symbol is used to access values other than primitives provided by BQN.

            All system values described in the BQN specification are optional: an implementation does not have to include any of them. However, if a system value with one of the names given below is included, then it must have the specified behavior. For namespaces this rule applies to individual fields as well: a namespace may be provided with only some of the fields, but a field with one of the given names must behave as specified.

            -

            Execution and scope manipulation

            +

            Execution and scope manipulation

            @@ -37,7 +37,7 @@

            The effect of •Eval should be the same as if its argument were written as source code in the scope where •Eval appears. It can define variables, and modify those in the current scope or a parent.

            •ScopedEval creates as new scope for evaluation as it is loaded. Other than its syntactic role, it is effectively equivalent to {•Eval}. Parent scopes are visible from the created scope; to make a scope without this property use •BQN"•Eval" or •BQN"•ScopedEval".

            -

            Scripts

            +

            Scripts

            @@ -74,11 +74,11 @@

            •path simply gives the path of the file in which it appears. It includes a trailing slash but not the name of the file itself.

            •name gives the name, including the extension, of the file in which it appears. It doesn't include the path.

            •Exit immediately terminates the running BQN process. If the argument is a valid return code (on Unix, an integer), it is returned; otherwise, the default return code (the one returned when the end of the program is reached) is used.

            -

            Files

            +

            Files

            The system namespace value •file deals with file operations. For the purposes of •file, paths in the filesystem are always strings. As with •Import, file paths may be relative or absolute, and relative paths are relative to •path, except in •file.At which allows 𝕨 to specify an alternate base directory. The value •path used for a particular instance of •file is determined by the file that contains that instance.

            When a •file function returns a file path or portion of a path, the path is always absolute and canonical, with . and .. components removed.

            Possible fields of •file are given in the subsections below.

            -

            File paths

            +

            File paths

            The following functions manipulate paths and don't access files. Each takes a relative or absolute path 𝕩, and At may also take a base directory 𝕨.

            @@ -118,7 +118,7 @@
            -

            File metadata

            +

            File metadata

            Metadata functions may query information about a file or directory but do not read to or write from it. Each takes a path 𝕩, and some functions also allow new data in 𝕨. The returned data in any case is the specified property.

            @@ -172,7 +172,7 @@
          • 'b': Block device
          • 'c': Character device
          • -

            File access

            +

            File access

            File access functions read or write files, either by manipulating files as a whole or interacting with the contents. Whole-file functions cannot overwrite target files: that is, Rename and Copy must give an error if a file exists at 𝕨, and CreateDir if a file exists at 𝕩, while Chars, Lines, and Bytes can overwrite the contents of an existing file 𝕨. However, these three functions must give an error if 𝕨 exists and is a directory.

            @@ -255,8 +255,8 @@
            -

            Open file object

            -

            Input and output

            +

            Open file object

            +

            Input and output

            @@ -285,7 +285,7 @@

            •Out prints a string to stdout, with a trailing newline. •Show displays a BQN value to the programmer (the representation is not specified, and does not need to be plain text). •Fmt returns a string (not a character table: lines are separated by linefeeds) indicating how 𝕩 would be printed by the interactive environment. Both •Show and •Fmt may take a left argument configuring how the value should be formatted.

            •Repr attempts to return a string so that •BQN •Repr 𝕩 matches 𝕩. If 𝕩 contains any mutable values (operations or namespaces), this is not possible. However, if such a values is stateless, in the sense that they don't access variables outside of their own scopes, it is permissible for •Repr to return source code that would create a value with identical behavior.

            -

            Operation properties

            +

            Operation properties

            @@ -411,7 +411,7 @@
            -

            Time

            +

            Time

            diff --git a/docs/spec/token.html b/docs/spec/token.html index e2b851a3..11e661f9 100644 --- a/docs/spec/token.html +++ b/docs/spec/token.html @@ -4,7 +4,7 @@ Specification: BQN token formation -

            Specification: BQN token formation

            +

            Specification: BQN token formation

            This page describes BQN's token formation rules (token formation is also called scanning). Most tokens in BQN are a single character long, but quoted characters and strings, identifiers, and numbers can consist of multiple characters, and comments, spaces, and tabs are discarded during token formation.

            BQN source code should be considered as a series of unicode code points, which we refer to as "characters". The separator between lines in a file is considered to be a single character, newline, even though some operating systems such as Windows typically represent it with a two-character CRLF sequence. Implementers should note that not all languages treat unicode code points as atomic, as exposing the UTF-8 or UTF-16 representation instead is common. For a language such as JavaScript that uses UTF-16, the double-struck characters 𝕨𝕎𝕩𝕏𝕗𝔽𝕘𝔾 are represented as two 16-bit surrogate characters, but BQN treats them as a single unit.

            A BQN character literal consists of a single character between single quotes, such as 'a', and a string literal consists of any number of characters between double quotes, such as "" or "abc". Character and string literals take precedence with comments over other tokenization rules, so that # between quotes does not start a comment and whitespace between quotes is not removed, but a quote within a comment does not start a character literal. Almost any character can be included directly in a character or string literal without escaping. The only exception is the double quote character ", which must be written twice to include it in a string, as otherwise it would end the string instead. Character literals require no escaping at all, as the length is fixed. In particular, literals for the double and single quote characters are written ''' and '"', while length-1 strings containing these characters are "'" and """".

            diff --git a/docs/spec/types.html b/docs/spec/types.html index ea44bb76..ce432dbd 100644 --- a/docs/spec/types.html +++ b/docs/spec/types.html @@ -4,7 +4,7 @@ Specification: BQN types -

            Specification: BQN types

            +

            Specification: BQN types

            BQN programs manipulate data of seven types:

            • Character
            • diff --git a/docs/style.css b/docs/style.css index 3264e776..4c8eb51b 100644 --- a/docs/style.css +++ b/docs/style.css @@ -21,6 +21,19 @@ h1, h2, h3, h4 { padding-bottom: 0.1em; border-bottom: 0.01em solid #9995 } +a:link.header, a:visited.header { + text-decoration:none; color:inherit; +} +.header { position:relative; } +.header:before { + position: absolute; + left: -0.9em; + content: "§"; + opacity: 0; +} +.header:hover:before { + opacity: 0.3; +} table { border-spacing: 0; diff --git a/docs/tutorial/combinator.html b/docs/tutorial/combinator.html index 5052aad1..a9cd7627 100644 --- a/docs/tutorial/combinator.html +++ b/docs/tutorial/combinator.html @@ -4,11 +4,11 @@ BQN Tutorial: Combinators -

              Tutorial: Combinators

              +

              Tutorial: Combinators

              BQN has a normal enough curly-brace syntax for functions and so on. I don't want to talk about it just yet. Before you get to thinking about how to write FORTRAN in any language in BQN, let's see if we can acquire some instincts about idiomatic BQN the way that only being stuck in a tightly restricted and horribly confining programming environment can accomplish.

              There are benefits to being tightly restricted and horribly confined! In programming. I don't just mean that it forces you to learn new techniques like I said, I mean that using the restricted style we will learn is actually a better way to write portions of a program. This is because a restriction you apply in one part of a program is a promise you can rely on in the rest of the program. So what are we giving up, and what can we rely on in return?

              Tacit programming does not use variables during the execution of a function (but you might use them for convenience in order to construct a tacit program). Variables allow you to use any accessible value in the program with the same level of ease. Tacit code doesn't. In fact it becomes pretty unusable when more than about three values are active at once. One consequence is that tacit code won't cause confusion by modifying far-away variables. But something unique to the tacit paradigm is that when only a small number of values are active—which is always true in a small enough portion of a program!—it has more powerful ways to describe the way these values flow through the program. The main way it achieves this is with combinators.

              -

              What's a combinator?

              +

              What's a combinator?

              (It's when you can't stop adding suffixes to "combine"). In the first tutorial, we briefly presented three combinators, , ˜, and ˙. These are functions or modifiers that produce a result from their inputs (arguments or operands) only by applying functions to arguments. For example, let's look at a composition:

              ↗️
                  |- 6
               6
              @@ -156,7 +156,7 @@
                 
               
               
              -

              Comparisons

              +

              Comparisons

            @@ -205,7 +205,7 @@ <@ 1 -

            Booleans

            +

            Booleans

            The return values 0 and 1 are natural choices because BQN has no dedicated boolean type: instead, in BQN, the word boolean indicates a number that's 0 or 1, much like "natural number" selects a subset of the possible numbers. This is a choice that might be at odds with your own programming experience, and especially clashes with the world of typed functional programming, where even using the boolean type rather than a dedicated type for an option might be considered a code smell! The design principle guiding this decision, and most of BQN's type system, is that there should only be one type that accomplishes any given programming task. Any distinctions that it has are there because they are really necessary: conflating numbers and characters would make working with strings too difficult, and functions can't really be combined with modifiers because one-argument functions and 1-modifiers take their inputs on different sides.

            The advantage of this strategy is that you will spend much less time thinking about types when writing programs. The decisions are already made: if there are a few things, they go in a list; if there a few possible values in a qualitative category then they should be labelled with numbers. And if some value has multiple interpretations then BQN is less likely to require an explicit conversion between these. For example, while the result of = might be taking to say whether two atoms have equal values, maybe it also says how many times the atom on the left appears in the right argument—which is at most one, because there's only one right argument. A silly distinction, or is it? An important property of counts is that we can add them together, for instance, to find how many times the letter "e" appears in a string.

            ↗️
                'e' = "George Boole"
            @@ -218,7 +218,7 @@
             3
             

            This, a well-typed and well-spoken programmer should declare, is an outrage! The purpose of types is to protect us from applying functions to types they're not intended for, because the most likely result is that such an application doesn't make sense. Static types provide a valuable service by catching these dangerous actions at compile time and allowing a programmer to prove that they never happen. Well… this is all true. BQN chooses not to pay the type price of this service and so doesn't get the type protection. And it helps that it's consistent about this choice, so you know that BQN's types are never the answer to these sorts of safety concerns. You will have to find a different way to avoid type errors: perhaps just programming carefully in a small or one-off program, and testing and code review or even external tools in a large one. All that said, I think programmers from outside the array programming world (and even many inside!) drastically underestimate how often a boolean value is really just a narrow-minded take on a counting number.

            -

            Comparing whole arrays

            +

            Comparing whole arrays

            <
            @@ -250,7 +250,7 @@ 23423322 1 -

            Length, rank, and depth

            +

            Length, rank, and depth

            I said above that the comparison functions might have a different meaning when called with only one argument, so let's cover a few of these.

            Length () gives the number of elements in a list. For atoms it returns 1; it can't result in an error.

            ↗️
                 "testing"
            @@ -281,9 +281,9 @@
             3
             

            You probably won't end up using Depth too much. The data in a typical program has a fixed, known depth, so there's no point in asking BQN what it is. But it might be useful if you want to write a utility function that's flexible about its input. 0<≡a is the idiomatic way to test whether a is an array.

            -

            Composition

            +

            Composition

            We've discussed Atop (), but hopefully you've intuited that it's not the end of the story as far as compositions go. In fact BQN has three more modifiers that could reasonably be interpreted as varieties of composition.

            -

            Over

            +

            Over

            @@ -375,7 +375,7 @@ -

            Before and After

            +

            Before and After

            Atop () and Over () are both symmetric in some sense: with two arguments, (FG)˜ is F(G˜), and (FG)˜ is (F˜)G. Put another way, reversing the order of arguments to Atop or Over as a whole is the same as reversing the order of every two-argument function inside—G for FG and F for FG. If it's not obvious why this is the case, work it out for yourself by walking through how these functions would apply to their arguments! This causes their diagrams to be symmetric as well. Swap (˜) also has a symmetric diagram, and it's very easy to show that it's symmetric: take a look at (F˜)˜ and (F˜)˜. In both cases I started with F˜, but in one case I applied ˜ to the entire function and in the other I applied it on the inside, to F only. And I won't tell you which is which.

            @@ -497,7 +497,7 @@ ⟨ 0 0.109375 0.1875 0.234375 0.25 0.234375 0.1875 0.109375 ⟩

            Our list of arguments stops before reaching 1, because 8 doesn't include 8. If we wanted a list from 0 to 1 inclusive, we'd need to divide by 7 (that is, 8-1) instead of 8. We can do this as well! But first we need to understand some other ways to apply Before and After.

            -

            Bind

            +

            Bind

            We showed in the first tutorial that a modifier's operand doesn't have to be a function, but can also be a data value. That hasn't come up yet, except for a cryptic use of "Bind" (?) in the function (2⋆↕8)ר from the last tutorial. How does that work? Some kind of secret identity for Before and After?

            ↗️
                1+ 5
             6
            @@ -550,7 +550,7 @@
             ↗️
                (↕÷-1) 8
             ⟨ 0 0.14285714285714285 0.2857142857142857 0.42857142857142855 0.5714285714285714 0.7142857142857143 0.8571428571428571 1 ⟩
             
            -

            Base decoding continued

            +

            Base decoding continued

            We're speeding up a bit now, so in the examples below it might take some time for you to break down what I did and why. Remember that you can open any expression in the REPL in order to change parts of it or view the syntax. And don't get discouraged just because of how long it takes to understand a line of code! First, you'll surely get faster in fitting the pieces together. Second, a line of BQN often has more code in it than a line in other languages, because primitives have such short names. Think about how much functionality you can read and understand rather than how few lines you get through.

            In the last tutorial I went over a way to decode a list of strings containing binary codes for ASCII characters:

            ↗️
                @ + +´¨ (2⋆↕8)ר '0' -˜ "01000010""01010001""01001110"
            @@ -597,7 +597,7 @@
             ↗️
                (@+ ·+(2×)´¨ -'0') "01000010""01010001""01001110"
             "BQN"
             
            -

            Summary

            +

            Summary

            BQN has a full complement of comparison functions, which are pervasive (work on atoms only) like arithmetic functions. The non-pervasive functions Match () and Not Match () compare entire arrays. Comparison functions return 1 if the comparison holds and 0 if it doesn't; these two numbers make up the "booleans".

            diff --git a/docs/tutorial/expression.html b/docs/tutorial/expression.html index 840ec4fd..89a43380 100644 --- a/docs/tutorial/expression.html +++ b/docs/tutorial/expression.html @@ -4,8 +4,8 @@ Tutorial: BQN expressions -

            Tutorial: BQN expressions

            -

            Arithmetic

            +

            Tutorial: BQN expressions

            +

            Arithmetic

            All right, let's get started! Since you can run BQN online from the REPL there aren't any real technical preliminaries, but if you'd like to look at non-web-based options head over to running.md.

            In the code blocks shown here, input is highlighted and indented, while output is not colored or indented. To experiment with the code, you can click the ↗️ arrow in the top right corner to open it in the REPL.

            ↗️
                2 + 3
            @@ -102,7 +102,7 @@
                 # \ shift
             

            In case you're wondering, Logarithm—the other inverse of Power—is written . We'll see how that works when we introduce in the section on 1-modifiers.

            -

            Compound expressions

            +

            Compound expressions

            It's sometimes useful to write programs with more than one function in them. Here is where BQN and any sort of normality part ways.

            ↗️
                2×3 - 5
             ¯4
            @@ -157,7 +157,7 @@
             
             

            The online REPL includes a tool to create diagrams like the one shown above. To enable it, click the "explain" button. Then a diagram of your source code will be shown above the result each time you run an expression.

            The following rule might help you to internalize this system in addition to identifying when parentheses are needed: an expression never needs to end with a parenthesis, or have two closing parentheses in a row. If it does, at least one set of parentheses can be removed without changing the meaning.

            -

            One or two arguments?

            +

            One or two arguments?

            What about functions without a left argument? Let's find an equation with lots of square roots in it… looks good.

            ↗️
                 3 + 2 × 2
             2.414213562373095
            @@ -227,7 +227,7 @@
             
          • A set of parentheses has the same role as whatever's inside it.
          • Perhaps more than you thought! To really get roles, it's important to understand that a role is properly a property of an expression, and not its value. In language implementation terms, roles are used only to parse expressions, giving a syntax tree, but don't dictate what values are possible when the tree is evaluated. So it's possible to have a function with a number role or a number with a function role. The reason this doesn't happen with the numeric literals and primitives we've introduced is that these tokens have a constant value. × or have the same value in any possible program, and so it makes sense that their types and roles should correspond. When we introduce identifiers, we'll see this correspondence break down—and why that's good!

            -

            Character arithmetic

            +

            Character arithmetic

            Gosh, that's a lot of arithmetic up there. Maybe adding characters will mix things up a bit? Hang on, you can't add characters, only subtract them… let's back up.

            ↗️
                'c'
             'c'
            @@ -279,7 +279,7 @@
             
             

            It's a convenient way to write non-printing characters without having to include them in your source code: for example @+10 is the newline character.

            Addition and subtraction with affine characters have all the same algebraic properties that they do with numbers. One way to see this is to think of values as a combination of "characterness" (0 for numbers and 1 for characters) and either numeric value or code point. Addition and subtraction are done element-wise on these pairs of numbers, and are allowed if the result is a valid value, that is, its characterness is 0 or 1 and its value is a valid code point if the characterness is 1. However, because the space of values is no longer closed under addition and subtraction, certain rearrangements of valid computations might not work, if one of the values produced in the middle isn't legal.

            -

            Modifiers

            +

            Modifiers

            Functions are nice and all, but to really bring us into the space age BQN has a second level of function called modifiers (the space age in this case is when operators were introduced to APL in the early 60s—hey, did you know the second APL conference was held at Goddard Space Flight Center?). While functions apply to subjects, modifiers can apply to functions or subjects, and return functions. For example, the 1-modifier ˜ modifies one function by swapping the arguments before calling it (Swap), or copying the right argument to the left if there's only one (Self).

            ↗️
                2 -˜ 'd'  # Subtract from
             'b'
            @@ -332,7 +332,7 @@
             

            Well, I guess it's not pedagogically useless, as it does demonstrate that a modifier can be applied to subjects as well as functions. Even though 3 is a subject, 3˙ is a function, and can be applied to and ignore the two arguments 2 and 4.

            With three examples you may have noticed that 1-modifiers tend to cluster at the top of the screen. In fact, every primitive 1-modifer is a superscript character: we've covered ˜⁼˙, and the remaining array-based modifiers ˘¨⌜´˝` will show up later.

            -

            2-modifiers

            +

            2-modifiers

            Made it to the last role, the 2-modifier (if you think something's been skipped, you're free to call subjects 0-modifiers. They don't modify anything. Just not when other people can hear you). To introduce them we'll use Atop , which works a lot like mathematical composition, except that it's extended to use one or two arguments. These arguments are passed to the function on the right, and the result is passed to the function on the left. So the function on the left is only ever called with one argument.

            ↗️
                3 ט+ 4  # Square of 3 plus 4
             49
            @@ -384,7 +384,7 @@
             

            This ordering is more consistent with the rule that a 1-modifier's operand should go to its left. If we tried going from right to left we'd end up with ×(˜+), which uses ˜ as an operand to . But a modifier can't be used as an operand. To make it work we'd have to give 1-modifiers a higher precedence than 2-modifiers.

            In fact, the rules for modifiers are exactly the same as those for functions, but reversed. So why is there a distinction between 1- and 2-modifiers, when for functions we can look to the left to see whether there is a left argument? The reason is that it's natural to follow a 1-modifier by a subject or function that isn't supposed to be its operand. Using an example from the last section, +˜ 3 has a subject to the right of the 1-modifier ˜. Even worse, +˜ ÷ 3 looks just like + ÷ 3, but it's two functions +˜ and ÷ applied to 3 while the version with Atop is a single function +÷ applied to 3. So the two-layer system of functions and modifiers forces modifiers to have a fixed number of operands even though every function (including those derived by applying modifiers) can be called with one or two arguments.

            Remember that 1-modifiers are all superscripts? The characters for 2-modifiers use a different rule: each contains an unbroken circle (that is, lines might touch it but not go through it). The 2-modifiers in BQN are the combinators ∘○⊸⟜⊘, the sort-of-combinators ⌾◶⍟, and the not-at-all-combinators ⎉⚇⎊. And the functions that make that unbroken circle rule necessary are written ⌽⍉. Since every primitive is a function, 1-modifier, or 2-modifier, you can always tell what type (and role) it has: a superscript is a 1-modifier, an unbroken circle makes it a 2-modifier, and otherwise it's a function.

            -

            Summary

            +

            Summary

            The objects we've seen so far are:

            diff --git a/docs/tutorial/index.html b/docs/tutorial/index.html index 396aabe4..85a2431f 100644 --- a/docs/tutorial/index.html +++ b/docs/tutorial/index.html @@ -4,7 +4,7 @@ BQN tutorials -

            BQN tutorials

            +

            BQN tutorials

            BQN tutorials explain how to approach and use the language as a newcomer (or they try; please get in touch if you find that they skip ahead!). Tutorials are meant to explain key ideas and may ignore details that would be included in the documentation; also unlike the documentation they're meant to be read in order, as each tutorial will build on ideas from the previous ones.

            Tutorials assume (pretty presumptively, really. Disgusting.) that you are already motivated to learn BQN and use simple rather than flashy examples. Documents to induce motivation beyond the README are not yet available. Do feel free to skim or jump around if you find you are reading a lot of things that are already obvious.

            The tutorials available so far:

            diff --git a/docs/tutorial/list.html b/docs/tutorial/list.html index a44c30b4..0e032b98 100644 --- a/docs/tutorial/list.html +++ b/docs/tutorial/list.html @@ -4,7 +4,7 @@ BQN Tutorial: Working with lists -

            Tutorial: Working with lists

            +

            Tutorial: Working with lists

            Enough with all these preliminaries like learning how to read basic expressions. Let's get into what makes BQN special.

            ↗️
                1, 2, 3
             ⟨ 1 2 3 ⟩
            @@ -14,7 +14,7 @@
             ⟨ 2 3 4 ⟩
             

            There we go. Now in BQN arrays are not just lists, which are a 1-dimensional data structure, but can have any number of dimensions. In this tutorial we're going to discuss lists only, leaving the 5-dimensional stuff for later. So we're really only seeing the power of K, an APL-family language that only uses lists (and dictionaries, which BQN doesn't have). K was powerful enough for Arthur Whitney to found two companies and make millions and millions of dollars, and BQN's compiler also runs almost entirely on lists, so this is probably enough power for one webpage.

            -

            List notation

            +

            List notation

            There are three kinds of list notation in BQN. Each of them has a subject role overall, even if expressions used inside it might have other roles. First, a string is a list of characters, and is written by placing those characters in double quotes.

            "Text!"
             
            @@ -86,7 +86,7 @@ 0(12) ⟨ 0 ⟨ 1 2 ⟩ ⟩ -

            BQN types

            +

            BQN types

            Now that all six BQN types have been introduced, let's make a table:

            @@ -111,7 +111,7 @@

            Lists are just one-dimensional arrays. Types are divided into data types, which tend to have a subject role, and operation types, which tend to have a role matching their type. Also, any value that's not an array, such as everything we used in the last tutorial, is called an atom.

            -

            Arithmetic on lists

            +

            Arithmetic on lists

            Arithmetic functions automatically apply to each element of a list argument. If both arguments are lists, they have to have the same length, and they're matched up one element at a time.

            ↗️
                ÷ 2,3,4
             ⟨ 0.5 0.3333333333333333 0.25 ⟩
            @@ -132,7 +132,7 @@
                  10, 2030  +  12, 3 
             ⟨ ⟨ 11 12 ⟩ ⟨ 23 33 ⟩ ⟩
             
            -

            Some list functions

            +

            Some list functions

            @@ -183,7 +183,7 @@ ¯1"bcdea" "abcde" -

            …and modifiers

            +

            …and modifiers

            @@ -235,7 +235,7 @@ ↗️
                  "con", "cat", "enat", "e" 
             "concatenate"
             
            -

            Example: base decoding

            +

            Example: base decoding

            Some people like to imagine that robots or other techno-beings speak entirely in binary-encoded ASCII, like for instance "01001110 01100101 01110010 01100100 00100001". This is dumb for a lot of reasons, and the encoded text probably just says something inane, but you're a slave to curiosity and can't ignore it. Are one and a half tutorials of BQN enough to clear your conscience?

            ¨
            @@ -359,7 +359,7 @@ ERROR +(+˜)´"1001"-'0' 9 -

            Summary

            +

            Summary

            There are three types of syntax that create lists: the "string literal" for lists of characters and either enclosing angle brackets ⟨⟩ with , or or newline characters or connecting ligatures for lists with arbitrary elements. The ligature has a higher precedence than functions or modifiers, so we should add it to our precedence table:

            diff --git a/docs/tutorial/variable.html b/docs/tutorial/variable.html index 33f1e48c..e8a1f06e 100644 --- a/docs/tutorial/variable.html +++ b/docs/tutorial/variable.html @@ -4,7 +4,7 @@ BQN Tutorial: Variables -

            Tutorial: Variables

            +

            Tutorial: Variables

            To take a proud denizen of the eternal cosmos of values, held for a fleeting instant by the course of code, and bind it. Tie it down with a name, failing always to alter its inner nature but allowing context to reform its outer appearance. So labelled, perhaps through the progress of time it will know escape, or else find itself passed through one bond to another, ever tethered. It's a task to be approached only with respect.

            ↗️
                hey  "Hi there"
             
            @@ -12,7 +12,7 @@
             "Hi there, World!"
             

            Like that.

            -

            Defining variables

            +

            Defining variables

            BQN uses the left-pointing arrow to define variables, as shown above. Most of the time it's best to use it in a plain way, with just the name and its definition, but it's also possible to define multiple variables using list notation, or to define a variable as part of a larger expression that continues to the left (in terms of precedence, behaves like a function, but it isn't one—it's a part of syntax).

            ↗️
                pieten   π, 1, 10 
             ⟨ 3.141592653589793 2.718281828459045 10 ⟩
            @@ -40,7 +40,7 @@ ERROR
             ERROR
             

            It's an odd distinction to have when your program is just one long sequence of statements, because there's only ever one arrow you can use: it just changes annoyingly after you define the variable for the first time. With multiple scopes this isn't the case: if you start a new scope inside another, then you'll still be able to use variables from the outside scope. Then lets you change the value of one of these variables while allows you to define your own. If you're coming from a typical curly-brace language, you'd say that both declares and assigns a variable, while only assigns it.

            -

            Variable roles

            +

            Variable roles

            ↗️
                BQN  "[many pages of specification]"
             ERROR
             
            @@ -104,7 +104,7 @@ ERROR ↗️
                1_000_000
             1000000
             
            -

            Function assignment

            +

            Function assignment

            While you could build up a script by computing values and assigning them names, the main way to use assignment in tacit programming is to give names to functions, not data. For example, we might name the base-2 conversion function from our last tutorial:

            ↗️
                Base2  +(2×)´
             
            @@ -130,7 +130,7 @@ ERROR
                 Base2 6
             16
             
            -

            Modifying part of an array

            +

            Modifying part of an array

            You cannot modify part of an array. You can't modify an array: an array that differs a little bit from another array is a different array. And this isn't just a terminology choice: it has real effects on how BQN arrays behave and even which arrays are representable, as we'll discuss later.

            But say I have a list, and I want to subtract one from one of the elements. With the understanding that the resulting list is different from the first one, BQN allows this!

            ↗️
                "BQN"            # A list of characters
            @@ -252,7 +252,7 @@ ERROR
             "abcdEFgh"
             

            (Here I've snuck in a train 2 4 to combine the two functions. As an exercise, you might try to write that function using combinators instead, and as an extra hard exercise you might then ponder why someone would want to add trains to a language).

            -

            Identity functions

            +

            Identity functions

            @@ -282,7 +282,7 @@ ERROR "left"

            They are not complicated functions: if you're confused it's because you don't understand why anyone would ever use them. Indeed, it's harder to see why these functions are useful than to see what they do. That is a fact.

            -

            Modified assignment

            +

            Modified assignment

            Let's revisit our question about modifying an array. As we said, the answer to "how do I modify part of an array?" is simply that you can't, and that the question doesn't make sense. But there's a seemingly similar question with a very different answer: "how do I modify part of a variable whose value is an array?" This is because unlike an array, a variable isn't defined by the value it has, but by the name used to refer to it (and the scope it resides in). Here's how we would modify the variable a:

            ↗️
                a  4            # First it's a number
                 a
            -- 
            cgit v1.2.3