Implicit Concurrency from Sources, Filters, and Sinks
One of my areas of investigation is how to allow a language to exploit implied concurrency. One way is by expressing algorithms in terms of source, filters, and sinks.
If shell scripting has taught me anything, it is that many sophisticated algorithms can be expressed in terms of a source, filter, and a sink. The source produces data, the filter transforms the data, and the sink consumes the data.
Expressing these algorithms in shell languages is dirt easy, but requires a ton of boilerplate in many languages (e.g. C++, C#, Java). In Heron I not only want to make it easy to write these kinds of programs, but I also want to allow the compiler to take advantage of the implicit concurrency automatically.
I am planning to introduce an associative operator "|>" which means take the data from a source and pump it into a filter or sink. The expression "source |> filter" would generate a new source, while "source |> sink" would generate a new sink.
Consider then the example of "A |> B |> C |> D". Rather than waiting for each item to pump through the entire pipeline, in theory we can have multiple items in the pipeline at a single time. I just haven't worked out an efficient implementation yet.
Right now, I am looking into an implementation that assigns a thread to each stage of the pipeline, and shares results with the next stage via a queue. The big problem I have to solve, is how to do it safely but as efficiently as possible.
I'll keep you posted as my solution develops, but in the meantime I'd love to hear your suggestions.