Programming guide

Building tools on Relix

An editor, a notebook, a command-line shell and an assistant all need more from the engine than a query's rows. They read scripts from somewhere of their own, colour text as the user types it, and learn the joins a user keeps writing. This page covers the three methods that serve them. The relix command-line tool and the interactive shell are built on these methods and nothing else; they are separate programs, released apart from the engine jar (see Getting the library).

Parsing a script

Relix.parse turns text into a Script, the tree define(Statement...) takes, without analysing it. Nothing is resolved, so a script may name relations nothing declares:

Java

import com.darkcollective.relix.lang.ast.Script;
import com.darkcollective.relix.lang.ast.Statement;

Script script = Relix.parse("""
        Large := { σ amount > 100 (Orders) };
        query Large;
        """);
for (Statement statement : script.statements()) {
    System.out.println(statement.getClass().getSimpleName());
}

Result

AssignmentStatement
QueryStatement

validate and script take a parsed Script as well as text, so a tool can read a script, leave out some of its statements and check or run the rest. A statement keeps the line and column it was parsed at, so a diagnostic still points into the user's text:

import com.darkcollective.relix.lang.ast.AssignmentStatement;
import com.darkcollective.relix.lang.ast.ScriptBuilders;

Relix relix = Relix.open();
relix.define("""
        Large := [| amount |
                  | 250    |];
        """);

// Run the queries against the session's own Large, not the one the script declares.
Script kept = ScriptBuilders.script(script.statements().stream()
        .filter(st -> !(st instanceof AssignmentStatement a && a.name().equals("Large")))
        .toArray(Statement[]::new));
List<Relation> results = relix.script(kept);

Text that does not parse raises a ScriptParseException, which carries the line and column of the failure:

Java

import com.darkcollective.relix.lang.ast.ScriptParseException;

try {
    Relix.parse("query { σ amount > (Orders) };");
} catch (ScriptParseException e) {
    System.out.println("line " + e.line() + ", column " + e.column());
}

Result

line 1, column 29

Printing a script

ScriptPrinter.print turns a Script back into text that parses to the same script. The layout is normalised — one statement per line, the canonical spelling of each keyword — but the comments of a parsed script come back where they were, so a formatter can rewrite someone's file without losing a word of it. Spelling.KEYWORDS writes each operator in ASCII, the form to generate code in:

Java

import com.darkcollective.relix.ast.Spelling;
import com.darkcollective.relix.lang.ast.ScriptPrinter;

Script commented = Relix.parse("""
        -- Large orders only.
        Large := {   σ amount>100 (Orders) };   -- keep these
        query Large;
        """);
System.out.print(ScriptPrinter.print(commented));
System.out.print(ScriptPrinter.print(commented, Spelling.KEYWORDS));

Result

-- Large orders only.
Large := { σ amount > 100 (Orders) }; -- keep these
query Large;
-- Large orders only.
Large := { SELECT amount > 100 (Orders) }; -- keep these
query Large;

A comment inside a statement is printed immediately before the part of the expression it preceded. A script built from statements rather than parsed has no comments to print.

A piece of an expression prints the same way. Operand.prettyPrint writes a scalar expression as text that parses back to an equal one, and SortSpecification.render writes a sort key the way a τ reads it, so a tool can show what a result is sorted by even when the key is an expression rather than a column:

Java

import com.darkcollective.relix.ast.SortNode;
import com.darkcollective.relix.ast.SortSpecification;
import com.darkcollective.relix.lang.ast.ExpressionQueryTarget;
import com.darkcollective.relix.lang.ast.QueryStatement;

Script sorted = Relix.parse("query { τ price * qty DESC, name (Orders) };");
QueryStatement query = (QueryStatement) sorted.statements().get(0);
SortNode sort = (SortNode) ((ExpressionQueryTarget) query.target()).expression();
for (SortSpecification key : sort.sortSpecs()) {
    System.out.println(key.render() + "  (column: " + key.columnName().orElse("none") + ")");
}

Result

price * qty DESC  (column: none)
name  (column: name)

A ScriptLoader is where parsing usually happens. A session resolves each import through its loader, and a loader reads the file and parses it, naming the file so a failure inside it points there. This one serves files from memory:

Java

import java.util.Map;

Map<String, String> files = Map.of("orders.relix", """
        Orders := [| id | amount |
                   | 1  | 120    |
                   | 2  | 80     |];
        """);

try (Relix withImports = Relix.builder()
        .scriptLoader(path -> Relix.parse(files.get(path), path))
        .build()) {
    withImports.define("import \"orders.relix\";");
    System.out.println(withImports.relation("Orders").count());
}

Result

Highlighting as the user types

Relix.tokens splits text into classified tokens for syntax highlighting. It is lenient, because an editor asks on every keystroke: the well-formed part is tokenized as usual and the rest becomes one INCOMPLETE token. Comments are tokens and whitespace is not. Offsets count chars, as String.charAt does.

Java

import com.darkcollective.relix.embed.Token;

String line = "σ amount > 100 (Orders) -- the large ones";
for (Token token : Relix.tokens(line)) {
    System.out.println(token.kind() + "  " + line.substring(token.start(), token.end()));
}

Result

OPERATOR  σ
IDENTIFIER  amount
COMPARISON  >
NUMBER  100
BRACKET  (
IDENTIFIER  Orders
BRACKET  )
COMMENT  -- the large ones

Half-typed text still colours up to the point it stops making sense:

Java

String typing = "π name (\"unfinished";
for (Token token : Relix.tokens(typing)) {
    System.out.println(token.kind() + "  " + typing.substring(token.start(), token.end()));
}

Result

OPERATOR  π
IDENTIFIER  name
BRACKET  (
INCOMPLETE  "unfinished

Learning the joins a user writes

A session's schema graph records how relations join: a relate statement declares an edge, and Builder.relationships supplies more. A tool can also learn edges from the queries its user writes. demonstratedRelationships() on a SemanticModel lists the column equalities written across a join, in a query or a view, that the graph does not already record:

Java

import com.darkcollective.relix.symbol.graph.Relationship;

String data = """
        Users := [| id | name |
                  | 1  | Ada  |
                  | 2  | Bo   |];
        Issues := [| id | title | assignee | reporter |
                   | 10 | Crash | 1        | 2        |];
        """;

List<Relationship> learned;
try (Relix first = Relix.open()) {
    first.define(data + "Assigned := { Issues ⨝ Issues.assignee = Users.id Users };");
    learned = first.model().demonstratedRelationships();
}
for (Relationship edge : learned) {
    System.out.println(edge.name() + "  " + edge.origin());
}

Result

Issues_assignee  LEARNED

Nothing ran to learn that: an analysis alone says which columns a join equated. resolveJoins goes the other way, assembling the join between relations from the edges the graph records. Given the learned edge, a later session can write the join without its condition:

Java

import com.darkcollective.relix.symbol.graph.JoinResolution;
import com.darkcollective.relix.symbol.graph.SchemaGraph;

try (Relix later = Relix.builder()
        .relationships(SchemaGraph.EMPTY.with(learned.getFirst()))
        .build()) {
    later.define(data);
    JoinResolution resolution =
            later.model().resolveJoins(later.relation("Issues ⋈ Users").node());
    if (resolution instanceof JoinResolution.Resolved resolved) {
        System.out.println(resolved.program());
    }
}

Result

(Issues) ⨝ Issues.assignee = Users.id (Users)

When more than one edge could join the relations, the answer is Ambiguous and lists the edges by name, which is a question a user can answer. With both of an issue's links to a user declared:

Java

try (Relix tracker = Relix.open()) {
    tracker.define(data + """
            relate "Assignee" Issues.assignee -> Users.id;
            relate "Reporter" Issues.reporter -> Users.id;
            """);
    JoinResolution resolution =
            tracker.model().resolveJoins(tracker.relation("Issues ⋈ Users").node());
    if (resolution instanceof JoinResolution.Ambiguous ambiguous) {
        for (JoinResolution.Alternative alternative : ambiguous.alternatives()) {
            System.out.println(alternative.label() + ": " + alternative.description());
        }
    }
}

Result

Assignee: Issues.assignee = Users.id
Reporter: Issues.reporter = Users.id

Relationship.edgeIdentity() says when two edges join the same columns whatever they are called. A tool that lets its user reject a learned edge keeps the identities it rejected and skips any candidate that matches one.

The language reference

The language reference is published as an artifact of its own, com.darkcollective.relix:relix-docs, so a tool can show a user the page for what they typed while an application that shows no documentation does not download it. RelixDocs.referencePages() lists every page with the words it is found by, and RelixDocs.referencePage(path) returns its markdown:

Java

import com.darkcollective.relix.docs.ReferencePage;
import com.darkcollective.relix.docs.RelixDocs;

ReferencePage selection = RelixDocs.referencePages().stream()
        .filter(page -> page.keys().contains("σ"))
        .findFirst().orElseThrow();
System.out.println(selection.title() + " (" + selection.symbol() + "): " + selection.summary());
System.out.println(RelixDocs.referencePage(selection.path()).orElseThrow().lines().findFirst().orElseThrow());

Result

Selection (σ): Filter rows by a condition
# Name: Selection (σ / SELECT)

A function's page is not among them: it belongs to the library that offers the function, and FunctionCatalog.documentation(docKey) serves it, so a function library installed later is documented the same way the shipped one is.

A tool that shows a page for whatever its user typed wants both kinds in one place, and ReferenceLookup is that index. Built over a function catalogue, it finds a page by glyph, keyword or name, the language's or a function's:

Java

import com.darkcollective.relix.docs.ReferenceLookup;
import com.darkcollective.relix.function.FunctionCatalog;

ReferenceLookup docs = ReferenceLookup.of(FunctionCatalog.discover());
System.out.println(docs.lookup("select").orElseThrow().lines().findFirst().orElseThrow());
System.out.println(docs.lookup("fix").orElseThrow().lines().findFirst().orElseThrow());
System.out.println(docs.function("fix").orElseThrow().summary());
System.out.println(docs.functionsByCategory().keySet());

Result

# Name: Selection (σ / SELECT)
# Name: General Recursion (FIX)
Truncate toward zero
[string, json, similarity, math, datetime, conditional, typecheck, conversion, nested, ordering]

A word finds the first page that claims it: the language's pages first, then any a tool adds of its own with ReferenceLookup.of(functions, pages, markdown), then the functions'. So fix is the recursion operator, and function("fix") asks among functions only, where it is the Fix that truncates a number.

Connectors and drivers

A tool that manages plugins asks where they live and which are installed, and a command-line tool registers the JDBC drivers an earlier run downloaded before it opens a session:

Java

import com.darkcollective.relix.connectors.std.DriverProvisioner;
import com.darkcollective.relix.processor.connector.ConnectorProvisioner;

DriverProvisioner.loadInstalled();
System.out.println(new java.util.TreeSet<>(ConnectorProvisioner.installedTypes()));

Result

[clf, csv, gedcom, log]

ConnectorProvisioner.defaultDirectory() is where a plugin is installed, and loadInstalled() returns one description per driver it registered, for a startup log.