I really wonder what should be the difference between those Link and Smart Home projects and something like OpenHAB, besides that is newly started by Mozilla. I think the current adaption of DIY or build-something-through existing frameworks shows that most people don't really care about such solutions, they want something that works out of the box - even if all parts come from a single manufacturer.
From an engineering perspective implementing lots of proprietary protocols in a single gateway device is lots of work and will never work 100% fine. Most people that worked on such solutions will confirm you that, and I can also do that (in a somewhat different domain). You can sink lots of money is these projects, and they still might not work for some customers at all - just like universal remotes - only more complex.
One way forward could be pushing the creation of devices (gateways, sensors, actors) that are based solely on standardized and accessible protocols. I'm not the biggest fan of CoAP (because I think it's hardly possible to implement it correctly), but I would still prefer this to reimplementing Z-Wave or Hue protocols. HTTP/2 could also be a good fit. But even if you have at least the protocol layer standardized, there possibilities for API design are endless. Standardizing IoT APIs throughout the whole stack in a backwards compatible fashion could really be something worthwhile, but the effort will be big, and it's questionable if other parties are actually interested in standardization.
And just a side node: I don't think Rust is the best choice for building a prototype for a mostly event based system. EventLoop semantics are hard to implement because of ownership/borrowing issues, the necessary dependencies (websocket libraries, etc.) are all in their in infancy, and it might also not be something that allows to move as much forward in speed as you want for a prototype (where you as a tradeoff can live with lower performance). Yes - I'm sure the Rust commmunity will disagree with me on the last point ;)
I would disagree with you on the first point more; Rust is actually perfect for event loop based semantics (we use this in Servo, it's very clean). Ownership fits perfectly with message passing.
You should avoid sharing state when you're using a message passing-based system. The cross-thread communication is handled by the messaging, sharing the messages itself just muddles things further. So the borrow checker disallowing that isn't a bad thing.
> the necessary dependencies (websocket libraries, etc.)
I agree; though the websocket library Servo uses is pretty good (still, not battle-tested, so it's a very valid point).
> it might also not be something that allows to move as much forward in speed as you want for a prototype
Again, with the prototyping thing, sure, it's harder to prototype in Rust, but that's not the whole story -- you spend less time writing tests and debugging.
(Also, I'm not sure how much harder it is; I've never had trouble with doing it and before Rust I was mostly programming in Python/JS)
> Yes - I'm sure the Rust commmunity will disagree with me on the last point
Which sort of indicates that it's probably inaccurate? :)
A lot of the perceived problems with Rust (e.g. fighting the borrow checker) go away after a month or two of actively using the language.
Just for the reference: I'm building similar systems professionally (and have done that with most mainstream programming languages on the market), and I was probably the first one who explored async IO and eventloops in Rust (https://github.com/Matthias247/revbio - but I'm not really proud of it) - so I think I'm at least halfway qualified to talk about this.
And unfortunatly by experimenting with these things I really got the fealing that these problems don't expose the best side of Rust. Wrapping lots of types into stuff like Rc<RefCell<T>> wasn't the greatest experience, and the problem that this didn't work with trait objects back then did not increase that either (might have changed).
Message passing is a low level primitive, and it's probably a decent solution for communication between threads. However there is also a need asynchronous communication inside a thread (the eventloop thing), for which they are not really useful. Futures/Promises/Observables are great for both use cases (and imho even greater in a singlethreaded environment). These are definitly harder to implement in Rust - and if you don't believe me just check out how many good implementations are currently available in Rust vs. other languages.
How does the eventloop design in Servo look like? Wasn't that some integration with Spidermonkey?
I agree that in Rust you will spend less time on writing tests and debugging - but this applies also for a lot of other statically typed languages which bring in a better ecosystem for that task.
Yes, that was missing back then. But also including the ability to do checked upcast/downcasts in the results, do change between Rc<RefCell<Base>> and Rc<RefCell<Derived>>. Is this now possible?
Rust in 2014 was a very different language. It looked the same at the surface, but a lot of the innards (including the fact that it shipped its own async I/O solution!) have changed since then.
> might have changed
It has. Though the explicitness of Rc<RefCell<T>> is still there, since Rust prefers ownership and mutability implications to be explicit. Not much extra typing, and you can always typedef it.
> a need asynchronous communication inside a thread
Agreed. There are some coroutine libraries in Rust which are pretty nice. There is active effort towards having coroutines inside the language itself (see https://github.com/erickt/stateful for a POC plugin, which should get RfCd at some point when it's complete).
> How does the eventloop design in Servo look like? Wasn't that some integration with Spidermonkey?
Like I mentioned, I meant the event loop model of communication. We have a lot of message passing between threads, many of which are event loops.
The script thread does feed events into its own event loop (after all, JS is event looped, we need to mirror that), and it works pretty well. We do some gymnastics for thread safety (especially because the Javascript GC is involved) but the interface is safe and clean.
> but this applies also for a lot of other statically typed languages which bring in a better ecosystem for that task.
Sure, but these languages don't have quick prototyping either. Except perhaps Go.
Rust doesn't really end up with an additional burden on prototyping over any other similar statically typed language; the borrow checker is something you rarely tussle with once you get used to it.
> I really wonder what should be the difference between those Link and Smart Home projects and something like OpenHAB, besides that is newly started by Mozilla. I think the current adaption of DIY or build-something-through existing frameworks shows that most people don't really care about such solutions, they want something that works out of the box - even if all parts come from a single manufacturer.
Very good question. There is definitely a large intersection between OpenHAB and Project Link. Both projects share one of their objectives: letting DIY users rig together and script their devices. However, beyond this point, our objective with Project Link diverge. We aim to explore ways to put the user in control of their data, including anonymity, authentication, storage, cloud access, web access, and certainly many others that we have yet to discover. It is our impression that OpenHAB's current architecture doesn't match our experimentation objectives.
A second difference is, of course, the architecture. We use Rust, betting that it will let us execute Project Link on devices with little memory, with no supervision, and with very long uptime. In particular, in the case of memory usage, while there are no decisions on this, there are projects that could help us run Rust and Project Link on almost bare metal, should we decide to head in this direction. We felt that OpenHAB's architecture was not adapted to such explorations.
> I think the current adaption of DIY or build-something-through existing frameworks shows that most people don't really care about such solutions, they want something that works out of the box - even if all parts come from a single manufacturer.
We are also exploring out-of-the-box, with other projects. I'm not involved, though, so there isn't much I could say on that topic.
[...]
> One way forward could be pushing the creation of devices (gateways, sensors, actors) that are based solely on standardized and accessible protocols.
Yes, we would very much like to do that. But before we can do that, we first need to be actors on the field, with well-used projects and a large community.
[...]
> Yes - I'm sure the Rust commmunity will disagree with me on the last point ;)
> [...]I'm not the biggest fan of CoAP (because I think it's hardly possible to implement it correctly)[...]
Out of curiosity, do you have any more details on what you think is impossible to implement correctly?
I've been doing a lot with CoAP and I'd agree that the RFC isn't as clear as it could be about some behavior, but I haven't come across anything that I'd say is impossible to implement or implement correctly.
I thought the base protocol was quite OK, but the complexity arrives as soon as you try to integrate blockwise access and notifications. From my look on it there are quite a lot of possibilites for race conditions there (e.g. bigger data chunks changing while transfers to clients are in progress) for which you might need to implement additional functionality (e.g. making copies of that data). But then you also need to exactly track how many clients are subscribed to that data und how long you need to keep that it (you want timers, and probably more timers than in any other protocol).
I don't recall the exact thing, but there was also something that I didn't like about the message IDs which you somehow need to keep ordered and check which ID might be used at the current point of time by which peer. A correct implementation may not send something while no ID is available, but to track all of those you needed lots of memory (much more than TCP buffers), and the open source implementations I looked at all elided that detail.
From an engineering perspective implementing lots of proprietary protocols in a single gateway device is lots of work and will never work 100% fine. Most people that worked on such solutions will confirm you that, and I can also do that (in a somewhat different domain). You can sink lots of money is these projects, and they still might not work for some customers at all - just like universal remotes - only more complex.
One way forward could be pushing the creation of devices (gateways, sensors, actors) that are based solely on standardized and accessible protocols. I'm not the biggest fan of CoAP (because I think it's hardly possible to implement it correctly), but I would still prefer this to reimplementing Z-Wave or Hue protocols. HTTP/2 could also be a good fit. But even if you have at least the protocol layer standardized, there possibilities for API design are endless. Standardizing IoT APIs throughout the whole stack in a backwards compatible fashion could really be something worthwhile, but the effort will be big, and it's questionable if other parties are actually interested in standardization.
And just a side node: I don't think Rust is the best choice for building a prototype for a mostly event based system. EventLoop semantics are hard to implement because of ownership/borrowing issues, the necessary dependencies (websocket libraries, etc.) are all in their in infancy, and it might also not be something that allows to move as much forward in speed as you want for a prototype (where you as a tradeoff can live with lower performance). Yes - I'm sure the Rust commmunity will disagree with me on the last point ;)