Parse standard operators

[liamhuber]

Currently, parseable workflow python is restricted to

• Assigning variables to names
• Assigning a list to a variable / appending to it in a for-scope
• Flow control operations (for/while/if-elif-else/try-except)

Such that we cannot currently parse a workflow like this:

def my_wf_to_parse(a, b, c):
    d = a + b
    e = c + d
    return e

because of the + operators.

However, this is strictly a parsing limitation, and not a workflow limitation. One can get the recipe I imaging we would want from parsing such a definition by manually constructing a node for the operator wrapping the standard library:

import operator

from pyiron_snippets import versions

from flowrep.api import schemas as frs

add_recipe = frs.AtomicNode.model_validate(
    {
        "inputs": ["a", "b"],
        "outputs": ["result"],
        "reference": {
            "info": versions.VersionInfo.of(operator.add),
            "restricted_input_kinds": {
                "a": "POSITIONAL_ONLY",
                "b": "POSITIONAL_ONLY",
            },
        },
    }
)

Since we can use recipes in workflow definitions (#199) and basic atomic recipes are callable (#197), we can write a parseable, recipe-based definition that behaves like our target

import flowrep as fr

@fr.workflow
def my_wf(a, b, c):
    d = add_recipe(a, b)
    e = add_recipe(c, d)
    return e

my_wf(1,2,3)
>>> 6

And is a valid, runnable recipe

from flowrep.api import tools as frt

frt.run_recipe(my_wf.flowrep_recipe, a=1, b=2, c=3).output_ports["e"].value
>>> 6

I.e., getting my_wf_to_parse to produce a valid recipe is purely a parser extension, and based around recognizing that we have an expression using an operator that ought to produce a new node, which we can dynamically produce based on the underlying builtin operator function. The example here is for +, but this pattern holds for all the standard operators.

We'd need to be careful to cross our t's and dot our i's, but from a technical perspective there is nothing particularly difficult here. But, ought we do this? In an off-grid conversation with @samwaseda, he expressed concern that in the current version of the parser, he's not clear what will parse and what won't (I suspect largely because for-loops have introduced this [] list operations). So making this sort of change would be positive in that users could write "more natural" python and have it parse, but negative in that the list of "this is what is parseable" becomes more complex and the boundaries harder to remember. Now, as long as failed parsing raises clean, clear, and helpful error messages when the parsing fails, I lean towards embracing the extra flexibility -- but there is a trade-off.

[samwaseda]

I strongly oppose it for the following reasons:

• Domestic aspect: There are simply too many tasks to accomplish elsewhere to create a full fledged WfMS, while without this one it would be also feature-complete
• Ontological aspect: It is impossible to give a semantic meaning to simple operators
  • Conceptual aspect: We do not write workflows to make writing functions easier, but in order to properly define each step, so that the operations can be fully stored, made submittable and documented
• Practical aspect: It becomes even more difficult to define what's possible and what's not, as you mentioned already above

The most important reason for me is the ontological one

[liamhuber]

• Domestic aspect: There are simply too many tasks to accomplish elsewhere to create a full fledged WfMS, while without this one it would be also feature-complete

I'm confused by this. What I'm talking about here is unrelated to a WfMS. This is only related to parsing recipe objects from python function definitions, the underlying recipes produced are already possible as demonstrated above.

• Ontological aspect: It is impossible to give a semantic meaning to simple operators

Yes, this is a big concern for me too. I think it might be possible to propagate some ontological information through some operations, but I don't think it's possible to do this ubiquitously. To the best of my current understanding, the only way to robustly and universally propagate ontological information is for each step to pass through an ontologically-annotated function.

* Conceptual aspect: We do not write workflows to make writing functions easier, but in order to properly define each step, so that the operations can be fully stored, made submittable and documented

As a point of clarity, this is all still generally true of the workflows here whether or not we adopt operator parsing. From a technical standpoint the only thing such a change would impact is annotation propagation. Since annotations don't appear in the flowrep recipe model to start with, these changes would neither help nor hinder the steps being "properly defined".

• Practical aspect: It becomes even more difficult to define what's possible and what's not, as you mentioned already above

That's a fair concern. I am happy to hold off implementing this indefinitely, as I don't feel nearly as strongly about having it as you feel about not having it. However, I do think we're likely to encounter a diversity of opinions here, and I'd like to leave the door open to parser choices that relax syntax constraints.

The most important reason for me is the ontological one

Yes, for me too. I think it will be impossible to robustly propagate ontological constraints through arbitrary operators. But, in my mind, flowrep is not exclusively for use with ontologies. So for me it leaves open the question of whether it's better to provide such syntax convenience for non-ontology users, or outlaw it for everyone since one user group can't use it.

Presently, flowrep doesn't know anything about ontologies, only that in the live data format there is room for annotations to be stored. I could thus imagine a compromise solution where the underlying flowrep recipe parser (which produces recipe models that don't even know about annotations) allows this, while recipe validation at the semantikon level detects that ontological information gets lost to the void (e.g. because the _operator.add recipe above has no annotations) and raises a clean error.