Dune DSL

A mini-language (DSL) for Dune files embedded in OCaml that produces readable, valid Dune include files. The mini-language closely matches the structure of a regular Dune file, and because it is embedded in OCaml you get OCaml type-safety and the power of your favorite OCaml IDE as you write your Dune files.

The Dune DSL uses a tagless final design so your Dune DSL can be interpreted in a variety of ways that are independent of Dune itself. The standard dkml-dune-dsl-show interpreter allows all of the Dune string values (executable names, rule targets, etc.) to be parameterized from external JSON files. That lets you produce many similar executables, libraries or assets using the same Dune logic without repeating yourself (DRY).

Status

This is an early release. The author jonahbeckford@ put in enough Dune syntax for what he needed; if you want more, this project accepts PRs!

There is at least one sharp edge:

  1. TLDR: If your build is not updating, erase everything in your "dune.inc" and then rerun Dune twice. What is happening? If the version of Dune DSL you are using has a bug where it generates a syntactically incorrect Dune file, Dune will stop the build (which is expected). However even if you upgrade to a bugfix version of the Dune DSL, Dune still will not be able to generated a corrected Dune file. You will need to truncate the "dune.inc" file so that the file is completely empty; then Dune will be able to generate the correct "dune.inc", and if you run Dune once more then the new "dune.inc" logic will run. The situation is a classic chicken-and-egg because we let Dune generate its own include files.

Usage

If you already know how to use Dune DSL, you can skip to the API documentation at DkmlDuneDsl.Dune.SYM.

Step One - Install

Install the DSL and a DSL interpreter with Opam using:

opam install dkml-dune-dsl dkml-dune-dsl-show

Step Two - dune-project

If you use dune-project files to manage your project, add the DSL to it with the build flag:

(package
 ...
 (depends
  ...
  (dkml-dune-dsl (and (>= 0.1.0) :build))
  (dkml-dune-dsl-show (and (>= 0.1.0) :build))
 )
)

Step Three - Parameters

An interpreter takes your Dune DSL expression and any command line arguments you supply to it, and performs some activity on it. In particular:

The format of the parameters file for dkml-dune-dsl-show is:

{
  "param-sets": [ ... we'll describe this soon ... ]
}

You can add your own JSON fields as long as they start with an underscore. So the following is a valid parameters file:

{
  "_documentation": "Some documentation of this file",
  "param-sets": [ ... we'll describe this soon ... ]
}

In what follows we will use the dkml-dune-dsl-show interpreter and give it a parameters file called "dune-parameters.json":

{
  "param-sets": [
    {"name": "batman", "age": 39},
    {"name": "robin", "age": 24}
  ]
}

The example above has two parameter sets, which the dkml-dune-dsl-show interpreter will use to print your Dune DSL expression twice into the same file:

  1. The first print will replace all "{{{ name }}}" and "{{{ age }}}" strings with "batman" and "39", respectively
  2. The second print will replace all "{{{ name }}}" and "{{{ age }}}" strings with "robin" and "24", respectively

Let's complete the parameter set story:

  1. All of those "{{{ ... }}}" expressions are known as Mustache expressions. Mustache is great at operating on JSON documents, which is exactly what the parameters file is. Most but not all of the Mustache expression language can be used; see OCaml Mustache for the exact specification.
  2. Our examples use three braces "{{{ ... }}}" rather than two braces "{{ ... }}" because the three braces is not HTML escaped.

  3. Any "{{{ xyz }}}" parameter where xyz is not in your parameter set will raise an error:

    Fatal error: exception Mustache_types.Missing_variable("xyz")
  4. If you don't supply a parameters file to the dkml-dune-dsl-show interpreter it will print your Dune DSL expression once.
  5. Rather than use a single parameter set to print (ex. {"name": "batman", "age": 39}), you can use the entire parameters file if you use the pragma once mode. We'll describe that pragma in the next section, but for now remember that pragma once gives you different parameters.

Step Four - Writing

Write your Dune configuration file in the OCaml DSL language. For what follows, we'll assume it is called "my_dune.ml":

open DkmlDuneDsl
module Build (I: Dune.SYM) = struct
  open I
  let res = [
    pragma "once"
      (rule
          [
            deps [ glob_files "*_data.ml" ];
            target "common.ml";
            action
              (run
                [ `S "something.exe"; `S "-o"; `S "%{target}"; `S "%{deps}" ]);
          ]);
    pragma "once" (library [ name "common"; modules (set_of [ "common" ]) ]);
    (*
        Being part of a DSL gives us whatever power the interpreter "I" lets us have.
        The dkml-dune-dsl-show interpreter "I" supports Mustache template expressions
        like {{name}} that are consumed from a parameters file.
        .
        We didn't want to repeat the previous stanzas multiple times, so we wrapped
        them in a (pragma "once" ...).
    *)
    library
      [
        name "{{name}}";
        modules (set_of [ "{{name}}" ]);
        libraries [ `S "common" ];
        preprocess (pps [ `S "ppx_deriving.ord" ]);
      ];
  ]
end

Let's peek ahead for a second. Once we complete all the steps, the Dune DSL tooling should give us a valid Dune include file that looks roughly like:

    (rule [
      (deps [glob_files "*_asset.png"]);
      (target "common.ml");
      (action
        (run ["something.exe"; "--output"; "%{target}"; "%{deps}"]))]);
    (library [
      (name "common");
      (modules [ "common" ]);
    ]);

    (library [
      (name "batman");
      (modules [ "batman" ]);
      (libraries ["common"]);
      (preprocess
        (pps ["ppx_deriving.ord"]))])
    (library [
      (name "robin");
      (modules [ "robin" ]);
      (libraries ["common"]);
      (preprocess
        (pps ["ppx_deriving.ord"]))])

The full set of expressions is available in DkmlDuneDsl.Dune.SYM. The res result variable you write must be a list of type [`Stanza] repr.

If you have a stanza that does not use a parameter "{{{ ... }}}" that is a strong signal you should wrap it in a DkmlDuneDsl.Dune.SYM.pragma "once".

Step Five - Glue

Write some small Dune glue in your "dune" file to execute your Dune configuration file with an interpreter. The only standard interpreter is the dkml-dune-dsl-show interpreter. The following snippet goes into the normal "dune" file, not the DSL:

(rule
 (target dune_inc.ml)
 (action
  (with-stdout-to
   %{target}
   (echo
    ; YOU: Change "My_dune" to the name of your Dune DSL expression module
    "module M = My_dune.Build (DkmlDuneDslShow.I)
     let () = print_string @@ DkmlDuneDslShow.plain_hum M.res"))))

(executable
 (name dune_inc)
 ; YOU: Change "my_dune" to the name of your Dune DSL file
 (modules dune_inc my_dune)
 (libraries dkml-dune-dsl-show))

(rule
 (alias gendune)
 (action
  (progn
   (with-stdout-to
    dune.gen.inc
    (run ./dune_inc.exe %{dep:dune-parameters.json}))
   (diff? dune.inc dune.gen.inc))))

(include dune.inc)

You will need to change "my_dune" and the corresponding name of the module ("My_dune") in the above snippet.

Then write an empty "dune.inc" file in the same directory as your "dune" file.

Finished!

Run "dune build '@gendune'". Dune will create a "dune.inc" in your "_build/default" directory (not your source directory) containing the information from your DSL expression. If you run it again, Dune will run all the build instructions in the "dune.inc".

It will also tell you when your source directory is out of sync with your DSL expression. If you are comfortable with the changes it shows you, run dune promote and commit the changes to your source code repository.

You are done!

Tagless Final Configuration

If you are a user, you don't need to read this section. It is for those interested in creating their own configuration format using tagless final DSL languages.

The author jonahbeckford@ uses DSLs for configuration files because:

  1. as a creator of configuration file formats, I don't need to write a lexer or a parser. I instead re-use the OCaml language and just lean on OCaml's type-safety to create a rich DSL
  2. as an author of configuration files, I get syntax checking, documentation and perhaps auto-completion with a modern editor with some setup
  3. as a user of configuration files, I just need to compile it to see if the syntax is correct
  4. at some point in the future, as a user of configuration files I can inspect the compiled bytecode to see if it is "safe" and avoids any system calls (all system calls, like printing to the standard output, should be done by the interpreter). For some configuration files calling out to the system is okay, but it is easy to conceive of a tool that validates safety for the majority of configuration files that don't need it.

The sore spot today is the setup required to author a configuration file. You need:

  1. an Opam switch (or a configured findlib)
  2. the DSL package installed (ex. dkml-dune-dsl)
  3. the OCaml language server protocol and/or Merlin installed depending on your editor

The setup should ideally be zero. It would be great that anybody could open up a DSL configuration file in their favorite editor and start configuring.

To support this, there is a convention that has been followed:

open TheCamelCasedNamedOfTheDslOpamPackage
module NamedFromTheDslSpec (I: NamedFromTheDslSpec.SYM) = struct
  open I
  let res = failwith "TODO: Replace this with your DSL expression here"
end

There should be nothing else in a configuration file except OCaml odoc comments. Just the open statement at the top followed by a module definition!

For the Dune DSL:

The convention for using I as the interpreter, using SYM as the module type, and using res as the interpreted result comes directly from Oleg Kiselyov's Tagless-final primer. The tagless final primer is a good read if you want to implement your own DSL!

You can lightly parse a configuration and know that the Opam package for DkmlDuneDsl is "dkml-dune-dsl" with a library with the same "dkml-dune-dsl" name.

Introspecting that library with a OCaml toplevel REPL would show:

utop # #require "dkml-dune-dsl";;
utop # DkmlDuneDsl.spec_version;;
- : int = 1

utop # DkmlDuneDsl.spec_modules;;
- : (string * string) list =
[("Build", "Dune"); ("Project", "DuneProject"); ("Workspace", "DuneWorkspace")]

Low Priority Tasks