It's definitely not easy to keep queues shallow end-to-end. In my experiences with Erlang systems, queue management is an area where the BEAM VM's runtime is definitely not actively helpful enough. One anecdote: within the last few months, the weak scheme that the BEAM uses for runtime back-pressure was removed because it isn't effective enough to justify the complexity of its code.
When a system does have the ability to keep queues shallow end-to-end, then I think it's a good base to build on, adding additional features to allow deeper queues where and when we want them.
Regarding Wallaroo: Today, you can use Kafka as a data source, allowing Kafka to be your deep-as-you-wish buffer upstream and/or downstream of Wallaroo. Tomorrow (a.k.a. vapor, though we've discussed the feature internally) it's certainly feasible to add an option to (for example) `TCPSource` that would add a large buffer at the entry to a Wallaroo cluster ... with flexibility to queue in RAM, disk, or elsewhere. It would also make failure recovery more complex, and it's one reason why we've deferred implementing it.
Howdy. I'm the author of both parts of this 2-page series on overload mitigation. Let me know if I've overlooked an important technique for handling too-big workloads or if you have questions (e.g., how we've stitched together TCP's sliding window protocol flow control with Wallaroo's internal flow control).
I, too, have a bit of experience with backpressure implementations in both Erlang and Pony. Erlang's "penalize the sender" doesn't work in many cases, so I'm not surprised that it's being removed. Erlang's remote distribution implementation & messaging semantics are Mostly Great but is definitely not Perfect.
1. Head-of-line blocking caused by congestion on the single TCP connection used for message transmission between any two Erlang nodes. This can cause major problems for apps control vs. data plane design, such as Riak and Lasp. Work on Partisan (https://github.com/lasp-lang/partisan) appears to be a substantial improvement.
2. If the single remote distribution TCP connection between two nodes is broken, then the first Erlang process to send a message to the remote node is, hrrm, well, borrowed/co-opted by the BEAM VM to connect the new TCP session. IIRC that process is marked unrunnable by the scheduler until the connection is set up or perhaps there's an error. If that process is really important to the functioning of your app, for example, an important system manager in the control plane of the app, then you have a very difficult-to-diagnose latency Heisenbug to cope with.
'penalize the sender' has the benefit of being very clean; it doesn't work in >1 machine distributed cases, but then neither do go or pony yet. I'm surprised to discover that it's being removed. I wonder what the thinking is for the local case.
Hi, everyone. I'm the author of this article. My apologies in advance for not covering non-DTrace tools available to Linux. The intent was "long article" and not "comprehensive book of everything about tracing". Please take a look at the list of references at the end of the article for lots of eBPF, uprobes, and SystemTap good stuff.
Howdy. Erlang's modules are involved in code namespace for the compiler & runtime (i.e., the 'M' of the 'MFA' triple of Module + Function name + Arity that names a function) as well being the unit/scope/bound for hot code loading (i.e., you must load or unload an entire module at a time).
I've worked in Erlang-land far longer than I've lived in any OOP-land, so I'm not sure what you mean by enterprise'y OOP. Coincidentally, Pony's rules for "Packages" are something I just smacked my ignorant head against a few hours ago. The subsections of https://tutorial.ponylang.org/packages/ in the tutorial can probably answer at least some of your question: specifically "Package System" and "Use Statement".
I'll take the liberty to speculate and assume your parent comment was asking if Pony classes have the same features as Java classes -- namely class-level attributes, instance-level attributes, different access levels to attributes and methods. Those things.
It's definitely not easy to keep queues shallow end-to-end. In my experiences with Erlang systems, queue management is an area where the BEAM VM's runtime is definitely not actively helpful enough. One anecdote: within the last few months, the weak scheme that the BEAM uses for runtime back-pressure was removed because it isn't effective enough to justify the complexity of its code.
When a system does have the ability to keep queues shallow end-to-end, then I think it's a good base to build on, adding additional features to allow deeper queues where and when we want them.
Regarding Wallaroo: Today, you can use Kafka as a data source, allowing Kafka to be your deep-as-you-wish buffer upstream and/or downstream of Wallaroo. Tomorrow (a.k.a. vapor, though we've discussed the feature internally) it's certainly feasible to add an option to (for example) `TCPSource` that would add a large buffer at the entry to a Wallaroo cluster ... with flexibility to queue in RAM, disk, or elsewhere. It would also make failure recovery more complex, and it's one reason why we've deferred implementing it.