Decision D033 · accepted
Data read at compile time: embed and package data
A record of the language's design, as it stood on 2026-11-18: written for the people who build Diaspar, and kept as it was written. Its code is a sketch, not a program of the site, and an older record may show syntax that a later one changed. It names files of the language's source, which is not public yet.
Status: accepted (2026-11-16)
Decision
Macros may read data files of their package at compile time and turn them into values with ordinary Diaspar code. The result is a literal: in a compiled program, static data in read-only memory. This is Zig's @embedFile with comptime, and F#'s type providers, in the language's own terms: no separate generator, no generated sources in the repository.
;; package.dias
#{:name "diaspar/std"
:data #["unicode/*.txt"]} ; the files macros may read
;; std/unicode.dias
(defmacro unicode-table [path prop]
(-> (embed path) ; at compile time: the file's text
(ucd.parse-ranges prop) ; ordinary Diaspar functions
ucd.compact)) ; → a literal
(def grapheme-break (unicode-table "unicode/GraphemeBreakProperty.txt" :all))
;; the same for any data: CSV, JSON, templates, schemas
(def cities (csv.parse (embed "data/cities.csv")))
The rules
embedreads a file at compile time, as a string ((embed path)) or bytes ((embed path as: :bytes)), relative to the package root. Only files declared in the manifest's:datamay be read; anything else is a compile error naming the manifest key. This keeps builds hermetic: the compiler knows every input, and a macro cannot read a developer's machine. It refines D015's rule that macros perform no effects beyond reading their own sources: declared data files are sources.- Expansion results are cached by the macro (its version), its arguments, and the hashes of the files it read, so a file is parsed once, not at every call or every build, until it changes.
- Literals are static data. A literal, including one computed by a macro, is an immortal object (D028) placed in read-only memory; reference counting does nothing for it. A
defwhose value is such a literal has its cell filled when the program is linked (D026): nothing is built, allocated or copied when the program starts.defwith a literal is the language's constant, as Rust'sstaticand Zig'sconst; no separate form is needed. Identical literals are merged into one object (within a module first; across modules by a content hash later), which no program can observe, since values are immutable and=compares contents (D013). The interpreter builds the value once, when the module loads.
The first use: Unicode tables
The standard library's Unicode tables (grapheme clusters, display width, case, white space) are built this way from the Unicode Character Database files committed under the standard library's data. They are checked with the UCD's own test files (GraphemeBreakTest.txt and others), and compared with the interpreter's Rust crates (unicode-segmentation, unicode-width) on the same inputs. Updating Unicode means replacing the data files.
Reading and writing data: std.data (accepted, 2026-11-17)
Diaspar data written as text (numbers, strings, keywords, vectors, maps, sets, structs printed as they are constructed) is read and written by std.data, the format of the language itself, as Clojure's EDN:
(use std.data)
(data.write config) ; a string that data.read gives back
(data.read text) ; literals only: names and calls are refused
(data.read text file: "theme.dias") ; the file name for the ReadError's position
- Literals only.
data.readnever evaluates and never resolves a name: a symbol or a call in the text is aReadErrorat its place. So a file from outside (a theme, a downloaded manifest) cannot run code, which separates it from reading code, Clojure'sread-stringagainstclojure.edn/read-string. - Its users: package manifests and lockfiles (D025), the
:datafiles of packages, configuration and themes, the compiler's caches, saved property-test failures and fuzz verdicts, the messages of the live protocol (D020), and the messages of actors in another OS process (D035). The toolchain itself needs it, so it is instd, not in a package. - Errors are the reader's one
ReadError(D014):data.readparses with the same reader as code. - Code is read by
std.lang.reader, as syntax with positions; the line editor of the REPL asks it whether a form is complete. - The prelude has neither.
read-stringis removed: data goes throughdata.read, code throughreader.read; reading text from outside shows at the import.read-dataandpr-databecomedata.readanddata.write.
Why
- One language: the code that turns data into tables is Diaspar, run by the compiler; no Python generator and no ten-thousand-line generated file.
- A tool for everyone:
embedserves CSV, JSON, templates and schemas in user programs the same way. - It exercises the compiler where it matters: compile-time evaluation, the expansion cache, static data.
Rejected alternatives
- A generator script with its output committed (Rust's std, Go's
go generate, Swift): fast builds and reviewable diffs, but a second language and generated sources; the expansion cache gives the speed, and the data files' diffs are the review. - Reading any file from a macro: builds would depend on the machine, and caches could not know their inputs.