Measuring startup and shutdown overhead of several code interpreters

I used the hyperfine tool to measure the overhead of startup and shutdown of several code interpreters. Each interpreter was invoked with an empty string for evaluation. In few cases it was not possible, an empty file was passed as argument.
This is more of a interesting experiment and presentation of hyperfine utility, and its plotting capabilities rather than a serious benchmark. The results should not be taken as an indication of "X is faster than Y".

<p dir="auto">Here's the link to the <a href="https://github.com/serpent7776/startup-shutdown-bench" target="_blank" rel="nofollow noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">repo with the code: <h2>Hyperfine <p dir="auto">Hyperfine is command-line benchmarking tool. It reruns the given commands multiple times and does statistical analysis. It can export measurements to various file formats like csv, JSON, markdown, etc. Those results can be then used as an input for a bunch of included scripts (e.g. to draw a plot). <p dir="auto">Basically, it does all the hard part, so that you don't have to. <p dir="auto">Check out <a href="https://github.com/sharkdp/hyperfine" target="_blank" rel="nofollow noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">the official hyperfine Github repo <p dir="auto">Here's the <a href="https://github.com/serpent7776/startup-shutdown-bench/blob/master/startup-test.sh#L5" target="_blank" rel="nofollow noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">command I used to do measurements: <pre><code>hyperfine -N --warmup 1 --export-csv 1.csv --export-json 1.json <CODE_INTERPRETERS...> <ul> <li><code>-N tells <code>hyperfine to not use shell for launching commands. It's not needed in my case, because I'm launching binaries. <li><code>--warmup 1 performs one run of each command before doing actual measurements. <li><code>--export-csv 1.csv exports summary to csv file. <li><code>--export-json 1.json expoorts measurements data to JSON file. This is needed to plot the results. <p dir="auto">Other useful options include: <ul> <li><code>--min-runs performs at least n runs for each command. <li><code>--runs performs exactly n runs for each command. <li><code>--parameter-list lets you execute parametric runs. <h2>Plotting the results <p dir="auto">Hyperfine comes with a <a href="https://github.com/sharkdp/hyperfine/blob/master/scripts/plot_whisker.py" target="_blank" rel="nofollow noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Python script that generates a box plot of the measurements.<br /> It requires the measurement data to be exported as JSON, which is used as the input file.<br /> I made some minor <a href="https://github.com/sharkdp/hyperfine/pull/727" target="_blank" rel="nofollow noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">improvements to the script to better suit my needs.<br /> If you're not familiar with box plot, you can read e.g. <a href="https://stats.stackexchange.com/questions/549751/how-to-interpret-the-result-of-this-box-plot" target="_blank" rel="nofollow noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">How to interpret the result of a box plot? <h2>Interpreters <p dir="auto">I used the following code interpreters: <pre><code>Lua 5.4.6 LuaJIT 2.1.1702233742 GNU bash, version 5.2.26(1)-release zsh 5.9 dash 0.5.12-1 fish, version 3.7.0 GNU Awk 5.3.0 perl 5, version 38, subversion 2 (v5.38.2) ruby 3.0.6p216 Python 3.11.8 PHP 8.3.3 JavaScript-C60.9.0 JavaScript-C78.15.0 guile (GNU Guile) 3.0.9 CHICKEN Version 5.3.0 (rev e31bbee5) Erlang/OTP 26 [erts-14.2.2] julia version 1.10.2 The OCaml toplevel, version 5.1.0 SWI-Prolog version 9.0.4 (GNU Prolog) 1.5.0 scryer-prolog "c8bf0be" Poly/ML 5.9.1 Release R version 4.3.3 tcl 8.6.14-1 cbqn r1646.49c0d9a-1 <h2>Results <p dir="auto">The results of the box plot are sorted by median time. <p dir="auto"><img src="https://images.hive.blog/DQmNrTPchMXaXd6hWq5cTiLNKc1yT9mmKX4FWjH5xFGwZFg/full.png" alt="Results" /> <p dir="auto"><a href="https://github.com/serpent7776/startup-shutdown-bench/blob/master/full.png" target="_blank" rel="nofollow noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Higher res image <p dir="auto">Here's the results without the slowest ones: <p dir="auto"><img src="https://images.hive.blog/DQmQTnffsZRXv9yRvF1T5Gwir7Skeb1Y9jrQFRJheuPGPrV/zoom.png" alt="Results without slowest ones" /> <p dir="auto"><a href="https://github.com/serpent7776/startup-shutdown-bench/blob/master/zoom.png" target="_blank" rel="nofollow noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Higher res image <p dir="auto">Some insights: <ul> <li><code>dash, <code>luajit and <code>lua have the smallest overhead. <li><code>perl is right next. It's actually fascinating that its overhead is smaller than that of shells considering their usage for scripting. <li>The next group are <code>tclsh, <code>awk, <code>sh, <code>bash, <code>zsh and <code>gprolog. Two surprising things here. First, I did not expect <code>awk to have bigger overhead than <code>perl, considering it's much smaller and more frequently used for small scripts. Second, I did not expect language like prolog to even be here (Especially looking at results of other prolog implementation). I'd think that such higher level, more complex languages often have bigger startup/shutdown overhead. <li>Past this point, the results are much more varied. <li>The next comes chicken scheme, polyml and <code>fish. Fish has somewhat bigger overhead than other shells. That shouldn't matter though, because it's more focused on interactive use. You should still write your shell scripts in <code>/bin/sh (or <code>/usr/bin/env bash). <li>The next is guile scheme and <code>bqn, which has the biggest span of the measurements so far. <li>After that there's <code>php and <code>python. <li>Then there's a small gap and there's spider monkey (<code>js78 and <code>js60). <li>After that <code>ocaml and then <code>node. A bit surprising that SpiderMonkey gets better results here than <code>node. <li>Then there's a big jump for <code>ruby. It's quite surprising, because I'd expect it to be closer to <code>python/<code>php. <li>Then there's even bigger gap and we get <code>julia, <code>R, erlang and <code>scryer-prolog <li>We finish with another big jump and last result, which is swi-prolog <p dir="auto">I tried to add clojure, but its overhead was twice that of swi-prolog, which is probably expected. Java is known to not be fastest one when it comes to startup. <p dir="auto">Note that I was running this on my personal machine. Some interpreters read configuration file, while others do not. This adds to the overhead. For more reliable results, it should be run in a separate, clean namespace. <h2>Conclusions <p dir="auto"><code>hyperfine is cool. It does all the hard work and comes with a script to visualise the results. You should use it for your projects. <p dir="auto">No definitive conclusions should be made from the measurements themselves, because in most cases the overhead of startup and shutdown doesn't matter. It only applies if you have really tiny script that is run huge number of times. If you really do then maybe dash, lua or perl would be a good choice, or maybe not, please measure.