Skip to the content
Diaspar

Decision D033 · accepted

Data read at compile time: embed and package data

A record of the language's design, as it stood on 2026-11-18: written for the people who build Diaspar, and kept as it was written. Its code is a sketch, not a program of the site, and an older record may show syntax that a later one changed. It names files of the language's source, which is not public yet.

Status: accepted (2026-11-16)

Decision

Macros may read data files of their package at compile time and turn them into values with ordinary Diaspar code. The result is a literal: in a compiled program, static data in read-only memory. This is Zig's @embedFile with comptime, and F#'s type providers, in the language's own terms: no separate generator, no generated sources in the repository.

;; package.dias
#{:name "diaspar/std"
  :data #["unicode/*.txt"]}                     ; the files macros may read

;; std/unicode.dias
(defmacro unicode-table [path prop]
  (-> (embed path)                              ; at compile time: the file's text
      (ucd.parse-ranges prop)                   ; ordinary Diaspar functions
      ucd.compact))                             ; → a literal

(def grapheme-break (unicode-table "unicode/GraphemeBreakProperty.txt" :all))

;; the same for any data: CSV, JSON, templates, schemas
(def cities (csv.parse (embed "data/cities.csv")))

The rules

  1. embed reads a file at compile time, as a string ((embed path)) or bytes ((embed path as: :bytes)), relative to the package root. Only files declared in the manifest's :data may be read; anything else is a compile error naming the manifest key. This keeps builds hermetic: the compiler knows every input, and a macro cannot read a developer's machine. It refines D015's rule that macros perform no effects beyond reading their own sources: declared data files are sources.
  2. Expansion results are cached by the macro (its version), its arguments, and the hashes of the files it read, so a file is parsed once, not at every call or every build, until it changes.
  3. Literals are static data. A literal, including one computed by a macro, is an immortal object (D028) placed in read-only memory; reference counting does nothing for it. A def whose value is such a literal has its cell filled when the program is linked (D026): nothing is built, allocated or copied when the program starts. def with a literal is the language's constant, as Rust's static and Zig's const; no separate form is needed. Identical literals are merged into one object (within a module first; across modules by a content hash later), which no program can observe, since values are immutable and = compares contents (D013). The interpreter builds the value once, when the module loads.

The first use: Unicode tables

The standard library's Unicode tables (grapheme clusters, display width, case, white space) are built this way from the Unicode Character Database files committed under the standard library's data. They are checked with the UCD's own test files (GraphemeBreakTest.txt and others), and compared with the interpreter's Rust crates (unicode-segmentation, unicode-width) on the same inputs. Updating Unicode means replacing the data files.

Reading and writing data: std.data (accepted, 2026-11-17)

Diaspar data written as text (numbers, strings, keywords, vectors, maps, sets, structs printed as they are constructed) is read and written by std.data, the format of the language itself, as Clojure's EDN:

(use std.data)
(data.write config)                    ; a string that data.read gives back
(data.read text)                       ; literals only: names and calls are refused
(data.read text file: "theme.dias")    ; the file name for the ReadError's position
  • Literals only. data.read never evaluates and never resolves a name: a symbol or a call in the text is a ReadError at its place. So a file from outside (a theme, a downloaded manifest) cannot run code, which separates it from reading code, Clojure's read-string against clojure.edn/read-string.
  • Its users: package manifests and lockfiles (D025), the :data files of packages, configuration and themes, the compiler's caches, saved property-test failures and fuzz verdicts, the messages of the live protocol (D020), and the messages of actors in another OS process (D035). The toolchain itself needs it, so it is in std, not in a package.
  • Errors are the reader's one ReadError (D014): data.read parses with the same reader as code.
  • Code is read by std.lang.reader, as syntax with positions; the line editor of the REPL asks it whether a form is complete.
  • The prelude has neither. read-string is removed: data goes through data.read, code through reader.read; reading text from outside shows at the import. read-data and pr-data become data.read and data.write.

Why

  • One language: the code that turns data into tables is Diaspar, run by the compiler; no Python generator and no ten-thousand-line generated file.
  • A tool for everyone: embed serves CSV, JSON, templates and schemas in user programs the same way.
  • It exercises the compiler where it matters: compile-time evaluation, the expansion cache, static data.

Rejected alternatives

  • A generator script with its output committed (Rust's std, Go's go generate, Swift): fast builds and reviewable diffs, but a second language and generated sources; the expansion cache gives the speed, and the data files' diffs are the review.
  • Reading any file from a macro: builds would depend on the machine, and caches could not know their inputs.