Fix self-hosted test failures

[kanaka]

It's been a while since I've done a comprehensive self-host test run. I've add a input variable to workflow so that self-hosted mode can be added for manually triggered workflows. The CI run is at: https://github.com/kanaka/mal/actions/runs/10309343464/job/28538806985

Some of these implementations might just need a NO_SELF_HOST setting (such as xslt), but a lot of the failures look like bugs rather than timeouts or memory exhaustion.

Once all these have been fixed or NO_SELF_HOST is added, we should make self-hosted test step enabled by default (it only adds a few minutes overall to the full CI workflow).

UPDATE: self-host mode has been made the default for workflows and the remaining broken implementations already had or have been updated with NO_SELF_HOST (erlang, jq, nasm, xslt, vbs).

Here is the the full list of self-hosted failures with very brief statement of the issue (see comments for more details):

• awk
• c.2
• cpp
• common-lisp
• dart
• erlang - memory leak (e.g. (sumdown 40000))
• io
• java
• jq - empty lists inside functions/closures are failing
• kotlin
• nasm - Needs more heap. With increased heap second quasiquote call fails
• ps
• powershell
• python.2
• sml
• vala
• vbs (vbscript) - startup/invocation error
• xslt - memory/recursion limit
• swift5

[kanaka]

awk fixed by e62567f and d1afe8d

[kanaka]

python.2 fixed in 0d32585

[kanaka]

Added common-lisp. Now that the underlying common-lisp issue with hash-maps is fixed, it reveals a new self-hosted failure (step4 issue with how nil is handled).

[kanaka]

More recent CI run (with awk and python.2 fixed), is here: https://github.com/kanaka/mal/actions/runs/10393189805/job/28780104927

Some of the failures have some common patterns:

• c.2, cpp, dart, swift5 all fail in step4 with the same type of error. Appears that the result of the list command is being evaluated again when it shouldn't be. For example:

  $ make DOCKERIZE=1 MAL_IMPL=cpp repl^step4
  mal-user> (list 1)
  Uncaught exception: "1" is not applicable
  

• java and kotlin appear to have a similar symptom:

  $ make DOCKERIZE=1 MAL_IMPL=kotlin repl^step4
  mal-user> (do (prn 101) 7)
  101
  101
  101
  101
  7
  

  And important to note that using Ctrl-D results in multiple exits from the prompt (more levels after running the broken do call).

Other failure summaries:

• common-lisp: in self-hosted mode, nil and () are being conflated (which is interesting because in traditional Lisps, those are often represented the same way):

  make DOCKERIZE=1 MAL_IMPL=common-lisp repl^mal^step4
  mal-user> nil
  ()
  

• jq: immediately exits (with 0 exit code) when trying to launch step2
• nasm: complains about Error: Run out of memory for Cons objects. Increase heap_cons_limit. Doubling the cons and array memory array results steps 0 - 6 working. step7 breaks like this:

  $ make DOCKERIZE=1 MAL_IMPL=nasm repl^mal^step7
  mal-user> (quasiquote ())
  ()
  mal-user> (quasiquote ())
  Error: list does not begin with a function
  Uncaught exception: (#<unknown value>)
    in mal EVAL: (quasiquote ())
  ``
  

• ps has problems starting at step8:

  $ make DOCKERIZE=1 MAL_IMPL=ps repl^mal^step8
  mal-user> (defmacro! unless (fn* (pred a b) `(if ~pred ~b ~a)))
  <(fn* [& args] (EVAL (nth ast 2) (new-env env (nth ast 1) args)))>
  mal-user> (unless 7 8 9)
  Uncaught exception: typecheck: get at 8638
    in mal EVAL: (unless 7 8 9)
  

• powershell also has an application error in step4 but it manifests differently than symptom effecting cpp, c.2, etc:

  $ make DOCKERIZE=1 MAL_IMPL=ps repl^mal^step4
  mal-user> (list? (list))
  Uncaught exception: can only apply functions
    in mal EVAL: (list? (list))
  

• sml in mlton mode appears to have reader/parsing errors that aren't specific enough (just "SyntaxError").
• vala: macros don't appear to be evaluating their results in self-hosted mode
• xslt: fails with Error: Too many nested template or function calls. The stylesheet may be looping. . Unless there is a way to increase the call stack size, this one will probably just need to be marked NO_SELF_HOST

[kanaka]

@asarhaddon @wasamasa @dubek Anybody want to help me track down some fun self-hosted failures?

[kanaka]

c.2, cpp, dart, and swift5 fixed in 7b23c7e

[kanaka]

java and kotlin fixed in b31180c (last position of do was being EVAL'd twice).

[kanaka]

common-lisp fixed in 40aca13 list? was defined to return true for nil. Added a test to catch that too.

[kanaka]

Powershell fixed in 78000d6. However, we probably want to mark powershell as NO_SELF_HOST because the self-hosted tests take 26 minutes to run in GHA. The next longest pole in the tent is rpython which takes less than 10 minutes for everything (most of it is build time).

[dubek]

I looked at debug logs from jq running repl^mal^step2. It starts processing the expressions from step2_eval.mal, and in the middle of the big (def! EVAL ...) expression I see a halt command which tells it to stop. Not sure what's going on yet.

[dubek]

Okay, found it: jq chokes on (); the following patch circumvents the jq bug:

diff --git a/impls/mal/step2_eval.mal b/impls/mal/step2_eval.mal
index 4d40f942..f5242a8c 100644
--- a/impls/mal/step2_eval.mal
+++ b/impls/mal/step2_eval.mal
@@ -24,7 +24,7 @@

       (list? ast)
       (if (empty? ast)
-        ()
+        (list)
         (let* [a0  (first ast)
                f    (EVAL a0 env)
                args (rest ast)]

Need to add a tests for () somewhere, and then fix it for jq. Not tonight...

[dubek]

But then I run into a new issue in mal step3:

mal-user> (+ 1 2)
Uncaught exception: 'env-find-str' not found

more fun with jq awaits...

[kanaka]

ps/postscript fixed in a160866 Missing a "pop" in the macro? function 😧

[kanaka]

I have a fix for the sml mlton self-host issue: 4719e58

However, I haven't merged it yet because the test I added to catch the issue in non-self-hosted mode revealed a problem in xslt:

$ make DOCKERIZE=1 repl^xslt^step9
user=> (read-string "(+ 1 2")      ;; note the missing paren in the string
(+ 1)
user=> (+ 1 2
Error: EOF while reading list

For some reason, the read-string core function is returning the partial parsed result rather than conveying the parsing error. I suspect the fix isn't too difficult, but it will requiring groking the xslt implementation at least somewhat. @asarhaddon since you've been in the xslt code before, any chance you could look at this?

[asarhaddon]

Do not expect much from me in the xslt field. A few years ago, I have given up merging eval and evalast.