Console Packager
The goal of this chapter is to demonstrate how archives are created, and specifically how self-extracting archives work on Windows.
Generating the installer starts with an Opam switch
We’ll just mimic an Opam switch by creating a directory structure and some files.
We want to model an Opam “installer” package that has two components: * dkml-component-offline-test-a * dkml-component-offline-test-b
The files will just be empty files except dkml-package-entry.exe is
a real executable that prints “Yoda”.
If this were not a demonstration, we would let the dkml-install-api framework generate those two files for us.
$ install -d _opam/bin
$ install -d _opam/lib/dkml-component-offline-test-a
$ install -d _opam/lib/dkml-component-offline-test-b
$ install -d _opam/lib/dkml-component-staging-ocamlrun
$ install -d _opam/share/dkml-component-offline-test-a/static-files
$ install -d _opam/share/dkml-component-offline-test-b/staging-files/generic
$ install -d _opam/share/dkml-component-staging-ocamlrun/staging-files/windows_x86_64/bin
$ install -d _opam/share/dkml-component-staging-ocamlrun/staging-files/windows_x86_64/lib/ocaml/stublibs
$ install -d _opam/share/dkml-component-offline-test-b/staging-files/darwin_arm64
$ install -d _opam/share/dkml-component-offline-test-b/staging-files/darwin_x86_64
$ diskuvbox touch _opam/bin/example-admin-runner.exe
$ diskuvbox touch _opam/bin/example-user-runner.exe
$ diskuvbox touch _opam/share/dkml-component-offline-test-a/static-files/README.txt
$ diskuvbox touch _opam/share/dkml-component-offline-test-a/static-files/icon.png
$ diskuvbox touch _opam/share/dkml-component-offline-test-b/staging-files/generic/somecode-offline-test-b.bc
$ diskuvbox touch _opam/share/dkml-component-offline-test-b/staging-files/darwin_arm64/libpng.dylib
$ diskuvbox touch _opam/share/dkml-component-offline-test-b/staging-files/darwin_x86_64/libpng.dylib
$ diskuvbox touch _opam/share/dkml-component-staging-ocamlrun/staging-files/windows_x86_64/bin/ocamlrun.exe
$ diskuvbox touch _opam/share/dkml-component-staging-ocamlrun/staging-files/windows_x86_64/lib/ocaml/stublibs/dllthreads.dll
$ diskuvbox tree --encoding UTF-8 -d 5 _opam
_opam
├── bin/
│ ├── example-admin-runner.exe
│ └── example-user-runner.exe
├── lib/
│ ├── dkml-component-offline-test-a/
│ ├── dkml-component-offline-test-b/
│ └── dkml-component-staging-ocamlrun/
└── share/
├── dkml-component-offline-test-a/
│ └── static-files/
│ ├── README.txt
│ └── icon.png
├── dkml-component-offline-test-b/
│ └── staging-files/
│ ├── darwin_arm64/
│ │ └── libpng.dylib
│ ├── darwin_x86_64/
│ │ └── libpng.dylib
│ └── generic/
│ └── somecode-offline-test-b.bc
└── dkml-component-staging-ocamlrun/
└── staging-files/
└── windows_x86_64/
├── bin/
└── lib/
What are these components?
In a typical graphical desktop installer, you are able to select which pieces of an application are installed on your machine. For example, a Git installer could ask whether you wanted to install the “Git LFS” extension for large file support. These pieces of an application are called components.
For now, we’ll define two do-nothing test components:
offline-test-aoffline-test-b
Using the Console installers will automatically bring in two other components:
* staging-ocamlrun
* xx-console
The installation of offline-test-b depends on offline-test-a.
That means during the installation of offline-test-b, offline-test-a
will also be installed.
Note
We chose to make the uninstallation of offline-test-b _not_ depend on
offline-test-a.
That means during the uninstallation of offline-test-b, no files from
offline-test-a will be involved in the uninstallation. And it also
means the uninstaller for offline-test-b is smaller because it does
not include offline-test-a files.
Often it is simpler to uninstall than install. In fact, uninstalling may simply be removing a directory, regardless of how many components were installed.
We will also use a library to generate an executable
called create_installers.exe:
$ cat test_windows_create_installers.ml
module Term = Cmdliner.Term
(* Create some demonstration components that are immediately registered *)
let () =
let reg = Dkml_install_register.Component_registry.get () in
Dkml_install_register.Component_registry.add_component ~raise_on_error:true
reg
(module struct
include Dkml_install_api.Default_component_config
let component_name = "offline-test-a"
(* During installation test-a needs ocamlrun.exe. staging-ocamlrun
is a pre-existing component that gives you ocamlrun.exe. *)
let install_depends_on = [ "staging-ocamlrun" ]
(* During uninstallation test-a doesn't need ocamlrun.exe.
Often uninstallers just need to delete a directory and other
small tasks that can be done directly using the install API
and/or the install API's standard libraries (ex. Bos).
Currently the console installer and console uninstaller always force a
dependency on staging-ocamlrun; this may change and other types of
uninstallers may not have the same behavior.
*)
let uninstall_depends_on = []
end);
Dkml_install_register.Component_registry.add_component ~raise_on_error:true
reg
(module struct
include Dkml_install_api.Default_component_config
let component_name = "offline-test-b"
(* During installation test-b needs test-a *)
let install_depends_on = [ "staging-ocamlrun"; "offline-test-a" ]
let uninstall_depends_on = []
end)
(* Let's also create an entry point for `create_installers.exe` *)
let () =
exit
(Dkml_package_console_create.create_installers
{
legal_name = "Legal Name";
common_name_full = "Common Name";
common_name_camel_case_nospaces = "CommonName";
common_name_kebab_lower_case = "common-name";
}
{
name_full = "Full Name";
name_camel_case_nospaces = "FullName";
name_kebab_lower_case = "full-name";
installation_prefix_camel_case_nospaces_opt = None;
installation_prefix_kebab_lower_case_opt = None;
}
{
url_info_about_opt = None;
url_update_info_opt = None;
help_link_opt = None;
estimated_byte_size_opt = None;
windows_language_code_id_opt = None;
embeds_32bit_uninstaller = true;
embeds_64bit_uninstaller = true;
})
You can see a real “curl” component at https://github.com/diskuv/dkml-component-curl/tree/40a6484a3fe3636d02b3c1ead41ad8c6d97dc449
In particular:
dkml-component-staging-curl.opamwill download acurlexecutable for Windows and stage it for installation on the end-user machine.src/buildtime_installer/dkml_component_staging_curl.mldefines the component. It tells DkML Install API that it needs to run code during installation for Unix machines only.src/installtime_enduser/unix/unix_install.mlis code that runs on the end-user machine during installation, if and only if it is Unix. It creates a symlink in a well-known location pointing to whichevercurlis found in the PATH during installation.
Use the generated create_installers.exe
Important
create_installers.exe will create installers for you based on the components
you registered earlier.
Create the temporary work directory and the target installer directory:
$ install -d work
$ install -d target
We will need to supply two important files generated with a “packager”. Today the only packager is the Console packager, which runs installation/uninstallation on the end-user’s machine as a Console program (as opposed to a GUI program traditional on Windows machines).
If this were not a demonstration focused only on how the installer is made, we would let the dkml-install-api framework generate those two files for us. Instead we use two test executables:
$ cat ./setup_print_hello.ml
let () = print_endline "Hello"
$ cat ./uninstaller_print_bye.ml
let () = print_endline "Bye"
$ ./setup_print_hello.exe
Hello
$ ./uninstaller_print_bye.exe
Bye
We’ll directly run the create_installers.exe executable. But if this were not a demonstration, you would be doing the same steps in your installer .opam file with something like:
[
"%{bin}%/dkml-install-create-installers.exe"
"--program-version"
version
"--work-dir"
"%{_:share}%/w"
"--target-dir"
"%{_:share}%/t"
"--packager-setup-bytecode"
"%{bin}%/setup.exe"
"--packager-uninstaller-bytecode"
"%{bin}%/uninstaller.exe"
]
Running the create_installers.exe gives:
$ ./test_windows_create_installers.exe --program-version 0.1.0 --component=offline-test-b --opam-context=_opam/ --target-dir=target/ --work-dir=work/ --abi=linux_x86_64 --abi=windows_x86_64 --packager-install-exe ./entry_print_salut.exe --packager-uninstall-exe ./entry_print_salut.exe --packager-setup-bytecode ./setup_print_hello.exe --packager-uninstaller-bytecode ./uninstaller_print_bye.exe --runner-admin-exe ./runner_admin_print_hi.exe --runner-user-exe ./runner_user_print_zoo.exe --verbose
test_windows_create_installers.exe: [INFO] Installers will be created that include the components:
[offline-test-a; offline-test-b;
staging-ocamlrun; xx-console]
test_windows_create_installers.exe: [INFO] Uninstallers will be created that include the components:
[offline-test-b; staging-ocamlrun;
xx-console]
test_windows_create_installers.exe: [INFO] Installers and uninstallers will be created for the ABIs:
[generic; linux_x86_64;
windows_x86_64]
test_windows_create_installers.exe: [INFO] Generating script target\bundle-full-name-generic-u.sh that can produce full-name-generic-u-0.1.0.tar.gz (etc.) archives
test_windows_create_installers.exe: [INFO] Generating script target\bundle-full-name-generic-i.sh that can produce full-name-generic-i-0.1.0.tar.gz (etc.) archives
test_windows_create_installers.exe: [INFO] Generating script target\bundle-full-name-linux_x86_64-u.sh that can produce full-name-linux_x86_64-u-0.1.0.tar.gz (etc.) archives
test_windows_create_installers.exe: [INFO] Generating script target\bundle-full-name-linux_x86_64-i.sh that can produce full-name-linux_x86_64-i-0.1.0.tar.gz (etc.) archives
test_windows_create_installers.exe: [INFO] Generating target\unsigned-full-name-windows_x86_64-u-0.1.0.exe
Parsing of manifest successful.
test_windows_create_installers.exe: [INFO] Generating script target\bundle-full-name-windows_x86_64-u.sh that can produce full-name-windows_x86_64-u-0.1.0.tar.gz (etc.) archives
test_windows_create_installers.exe: [INFO] Generating target\unsigned-full-name-windows_x86_64-i-0.1.0.exe
Parsing of manifest successful.
test_windows_create_installers.exe: [INFO] Generating script target\bundle-full-name-windows_x86_64-i.sh that can produce full-name-windows_x86_64-i-0.1.0.tar.gz (etc.) archives
test_windows_create_installers.exe: [INFO] Installers and uninstallers created successfully.
The --work-dir will have ABI-specific archive trees in its “a” folder.
The archive tree is the content that is packed into the installer file (ex. setup.exe, .msi, .rpm, etc.) and which gets unpacked on the end-user’s machine.
Each archive tree contains a “sg” folder for the staging files … these are files that are used during the installation but disappear when the installation is finished.
Each archive tree also contains a “st” folder for the static files … these are files that are directly copied to the end-user’s installation directory.
Each archive tree also contains the packager executables named
bin/dkml-package.bc.
$ diskuvbox tree --encoding UTF-8 -d 6 work
work
├── a/
│ ├── i/
│ │ ├── generic/
│ │ │ ├── sg/
│ │ │ │ └── offline-test-b/
│ │ │ │ └── generic/
│ │ │ └── st/
│ │ │ └── offline-test-a/
│ │ │ ├── README.txt
│ │ │ └── icon.png
│ │ ├── linux_x86_64/
│ │ │ ├── bin/
│ │ │ │ ├── dkml-install-admin-runner.exe
│ │ │ │ ├── dkml-install-user-runner.exe
│ │ │ │ ├── dkml-package-entry.exe
│ │ │ │ └── dkml-package.bc
│ │ │ ├── sg/
│ │ │ │ └── offline-test-b/
│ │ │ │ └── generic/
│ │ │ └── st/
│ │ │ └── offline-test-a/
│ │ │ ├── README.txt
│ │ │ └── icon.png
│ │ └── windows_x86_64/
│ │ ├── bin/
│ │ │ ├── dkml-install-admin-runner.exe
│ │ │ ├── dkml-install-user-runner.exe
│ │ │ ├── dkml-package-entry.exe
│ │ │ ├── dkml-package-uninstall.exe
│ │ │ └── dkml-package.bc
│ │ ├── sg/
│ │ │ ├── offline-test-b/
│ │ │ │ └── generic/
│ │ │ └── staging-ocamlrun/
│ │ │ └── windows_x86_64/
│ │ └── st/
│ │ └── offline-test-a/
│ │ ├── README.txt
│ │ └── icon.png
│ └── u/
│ ├── generic/
│ │ └── sg/
│ │ └── offline-test-b/
│ │ └── generic/
│ ├── linux_x86_64/
│ │ ├── bin/
│ │ │ ├── dkml-install-admin-runner.exe
│ │ │ ├── dkml-install-user-runner.exe
│ │ │ ├── dkml-package-entry.exe
│ │ │ └── dkml-package.bc
│ │ └── sg/
│ │ └── offline-test-b/
│ │ └── generic/
│ └── windows_x86_64/
│ ├── bin/
│ │ ├── dkml-install-admin-runner.exe
│ │ ├── dkml-install-user-runner.exe
│ │ ├── dkml-package-entry.exe
│ │ └── dkml-package.bc
│ └── sg/
│ ├── offline-test-b/
│ │ └── generic/
│ └── staging-ocamlrun/
│ └── windows_x86_64/
├── setup.exe.manifest
└── sfx/
└── 7zr.exe
Bring-your-own-archiver archives
Currently there is only one “supported” archiver: tar.
You could use your own tar archiver so you can distribute software for *nix machines like Linux and macOS in the common .tar.gz or .tar.bz2 formats.
There could be others in the future:
a zip archiver so you can use builtin zip file support on modern Windows machines. (But the setup.exe installers are probably better; see the next section)
a RPM/APK/DEB packager on Linux
We create “bundle” scripts that let you generate ‘tar’ archives specific to the target operating systems. You can add tar options like ‘–gzip’ to the end of the bundle script to customize the archive.
Note
The reason we use scripts rather than create the archives directly is to lessen the OCaml dependencies of dkml-install-api. You usually can install or use an archiver (ex. tar.exe + gzip.exe) on a build system, which will be more performant, maintainable and customizable than doing tar (or RPM, etc.) inside of OCaml.
$ diskuvbox tree --encoding UTF-8 -d 6 work
work
├── a/
│ ├── i/
│ │ ├── generic/
│ │ │ ├── sg/
│ │ │ │ └── offline-test-b/
│ │ │ │ └── generic/
│ │ │ └── st/
│ │ │ └── offline-test-a/
│ │ │ ├── README.txt
│ │ │ └── icon.png
│ │ ├── linux_x86_64/
│ │ │ ├── bin/
│ │ │ │ ├── dkml-install-admin-runner.exe
│ │ │ │ ├── dkml-install-user-runner.exe
│ │ │ │ ├── dkml-package-entry.exe
│ │ │ │ └── dkml-package.bc
│ │ │ ├── sg/
│ │ │ │ └── offline-test-b/
│ │ │ │ └── generic/
│ │ │ └── st/
│ │ │ └── offline-test-a/
│ │ │ ├── README.txt
│ │ │ └── icon.png
│ │ └── windows_x86_64/
│ │ ├── bin/
│ │ │ ├── dkml-install-admin-runner.exe
│ │ │ ├── dkml-install-user-runner.exe
│ │ │ ├── dkml-package-entry.exe
│ │ │ ├── dkml-package-uninstall.exe
│ │ │ └── dkml-package.bc
│ │ ├── sg/
│ │ │ ├── offline-test-b/
│ │ │ │ └── generic/
│ │ │ └── staging-ocamlrun/
│ │ │ └── windows_x86_64/
│ │ └── st/
│ │ └── offline-test-a/
│ │ ├── README.txt
│ │ └── icon.png
│ └── u/
│ ├── generic/
│ │ └── sg/
│ │ └── offline-test-b/
│ │ └── generic/
│ ├── linux_x86_64/
│ │ ├── bin/
│ │ │ ├── dkml-install-admin-runner.exe
│ │ │ ├── dkml-install-user-runner.exe
│ │ │ ├── dkml-package-entry.exe
│ │ │ └── dkml-package.bc
│ │ └── sg/
│ │ └── offline-test-b/
│ │ └── generic/
│ └── windows_x86_64/
│ ├── bin/
│ │ ├── dkml-install-admin-runner.exe
│ │ ├── dkml-install-user-runner.exe
│ │ ├── dkml-package-entry.exe
│ │ └── dkml-package.bc
│ └── sg/
│ ├── offline-test-b/
│ │ └── generic/
│ └── staging-ocamlrun/
│ └── windows_x86_64/
├── setup.exe.manifest
└── sfx/
└── 7zr.exe
$ diskuvbox tree --encoding UTF-8 -d 2 target
target
├── bundle-full-name-generic-i.sh
├── bundle-full-name-generic-u.sh
├── bundle-full-name-linux_x86_64-i.sh
├── bundle-full-name-linux_x86_64-u.sh
├── bundle-full-name-windows_x86_64-i.sh
├── bundle-full-name-windows_x86_64-u.sh
├── full-name-windows_x86_64-i-0.1.0.7z
├── full-name-windows_x86_64-i-0.1.0.sfx
├── full-name-windows_x86_64-u-0.1.0.7z
├── full-name-windows_x86_64-u-0.1.0.sfx
├── unsigned-full-name-windows_x86_64-i-0.1.0.exe
└── unsigned-full-name-windows_x86_64-u-0.1.0.exe
$ target/bundle-full-name-linux_x86_64-i.sh -o target/i tar
$ tar tvf target/i/full-name-linux_x86_64-i-0.1.0.tar | head -n5 | awk '{print $NF}' | sort
./
full-name-linux_x86_64-i-0.1.0/.archivetree
full-name-linux_x86_64-i-0.1.0/bin/
full-name-linux_x86_64-i-0.1.0/bin/dkml-install-admin-runner.exe
full-name-linux_x86_64-i-0.1.0/bin/dkml-install-user-runner.exe
$ target/bundle-full-name-linux_x86_64-i.sh -o target/i -e .tar.gz tar --gzip
$ tar tvfz target/i/full-name-linux_x86_64-i-0.1.0.tar.gz | tail -n5 | awk '{print $NF}' | sort
full-name-linux_x86_64-i-0.1.0/sg/offline-test-b/generic/somecode-offline-test-b.bc
full-name-linux_x86_64-i-0.1.0/st/
full-name-linux_x86_64-i-0.1.0/st/offline-test-a/
full-name-linux_x86_64-i-0.1.0/st/offline-test-a/README.txt
full-name-linux_x86_64-i-0.1.0/st/offline-test-a/icon.png
setup.exe installers
There are also fully built setup.exe installers available. The setup.exe is just a special version of the decompressor 7z.exe called an “SFX” module, with a 7zip archive appended.
Let’s start with the 7zip archive that we generate. You will see that its
contents is exactly the same as the archive tree, except that
bin/dkml-package-entry.exe (the packager proxy setup) has been renamed to
setup.exe.
$ ../assets/lzma2107/bin/7zr.exe l target/full-name-windows_x86_64-i-0.1.0.7z | awk '$1=="Date"{mode=1} mode==1{print $NF}'
Name
------------------------
bin
sg
sg\offline-test-b
sg\offline-test-b\generic
sg\staging-ocamlrun
sg\staging-ocamlrun\windows_x86_64
sg\staging-ocamlrun\windows_x86_64\bin
sg\staging-ocamlrun\windows_x86_64\lib
sg\staging-ocamlrun\windows_x86_64\lib\ocaml
sg\staging-ocamlrun\windows_x86_64\lib\ocaml\stublibs
st
st\offline-test-a
.archivetree
sg\offline-test-b\generic\somecode-offline-test-b.bc
sg\staging-ocamlrun\windows_x86_64\bin\ocamlrun.exe
sg\staging-ocamlrun\windows_x86_64\lib\ocaml\stublibs\dllthreads.dll
st\offline-test-a\icon.png
st\offline-test-a\README.txt
bin\dkml-package.bc
bin\dkml-install-admin-runner.exe
bin\dkml-install-user-runner.exe
setup.exe
bin\dkml-package-uninstall.exe
vcruntime140.dll
vcruntime140_1.dll
vc_redist.dkml-target-abi.exe
------------------------
folders
$ ../assets/lzma2107/bin/7zr.exe l target/full-name-windows_x86_64-u-0.1.0.7z | awk '$1=="Date"{mode=1} mode==1{print $NF}'
Name
------------------------
bin
sg
sg\offline-test-b
sg\offline-test-b\generic
sg\staging-ocamlrun
sg\staging-ocamlrun\windows_x86_64
sg\staging-ocamlrun\windows_x86_64\bin
sg\staging-ocamlrun\windows_x86_64\lib
sg\staging-ocamlrun\windows_x86_64\lib\ocaml
sg\staging-ocamlrun\windows_x86_64\lib\ocaml\stublibs
.archivetree
sg\offline-test-b\generic\somecode-offline-test-b.bc
sg\staging-ocamlrun\windows_x86_64\bin\ocamlrun.exe
sg\staging-ocamlrun\windows_x86_64\lib\ocaml\stublibs\dllthreads.dll
bin\dkml-package.bc
bin\dkml-install-admin-runner.exe
bin\dkml-install-user-runner.exe
uninstall.exe
vcruntime140.dll
vcruntime140_1.dll
vc_redist.dkml-target-abi.exe
------------------------
folders
We would see the same thing if we looked inside the installer
unsigned-NAME-VER.exe (which is just the SFX module and the .7z archive above):
$ ../assets/lzma2107/bin/7zr.exe l target/unsigned-full-name-windows_x86_64-i-0.1.0.exe | awk '$1=="Date"{mode=1} mode==1{print $NF}' | head -n10
Name
------------------------
bin
sg
sg\offline-test-b
sg\offline-test-b\generic
sg\staging-ocamlrun
sg\staging-ocamlrun\windows_x86_64
sg\staging-ocamlrun\windows_x86_64\bin
sg\staging-ocamlrun\windows_x86_64\lib
When the installer setup.exe is run, the SFX module knows how to find the 7zip archive stored at the end of the installer setup.exe (which you see above), decompress it to a temporary directory, and then run an executable inside the temporary directory.
To make keep things confusing, the temporary executable that 7zip runs is the member “setup.exe” (the packager setup.exe) found in the .7z root directory.
Since the installer unsigned-NAME-VER.exe will decompress the .7z archive and
run the packager proxy setup.exe it found in the .7z root directory, we expect to
see “Hello” printed. Which is what we see:
$ target/unsigned-full-name-windows_x86_64-i-0.1.0.exe
Salut
To recap:
Opam directory structure is used to build a directory structure for the archive tree.
You can create .tar.gz or .tar.bz2 binary distributions from the archive tree.
You can also use the installer unsigned-NAME-VER.exe which has been designed to automatically run the packager proxy setup.exe.
Whether manually uncompressing a .tar.gz binary distribution, or letting
installer unsigned-NAME-VER.exe do it automatically, the
packager proxy setup.exe will have full access to the archive
tree.
The only thing that remains is digitally signing the unsigned-NAME-VER.exe;
typically you would name the signed version setup-NAME-VER.exe. You would
typically distribute both the signed and unsigned executables because:
Any signed executable is unreproducible for the public, but the signed executable will trigger far less anti-virus warnings.
Everyone should be able to reproduce the unsigned executable, especially if you use locked Opam files with no pinned packages.
That’s it for how archives and setup.exe work!
Go through the remaining documentation to see what a real
packager setup.bc does, and what goes into a real component.