Thank you for great article. I've been writing my own search engine during last 3 years and it's funny how similar my setup is to yours with K8S/Kafka/Streams/Go/RocksDB.
Actually about RocksDB - are you using it from Go(via gorocksdb?). Now and then I have hard time optimising RocksDB and still have very loose understanding how much RAM it will consume
Funny that your stack exactly matches what I'm doing currently on my own. I wonder - how do you build Clojure(Script) using Buck? Mind share a bit of details?
I've been working on the CLJ(S) build system at Ladder. We use Buck to AOT compile clojure & clojurescript. The approach varies a little for the two.
For clojure, we set up the proper classpath based on the deps & run clojure's compile primitive. This part isn't too hard. We use a genrule macro to get the classpath of a java_* rule ($(classpath //:foo)), and then we symlink the sources into $TMP/srcs in the right layout. For better performance we actually AOT the third party dependencies as separate rules. That way when we AOT our code, we only get .class files only for our code.
For CLJS the story is a bit more complicated. The CLJS compiler doesn't natively support partial compilation. We ended up hacking the clojurescript compiler to dump the intermediate state as part of the build output. This isn't novel though - the compiler already dumps .cache.edn files with the same information. Once we have that, to compile dependent rules, we inject that state into the compiler before compilation.
Other than the hack, CLJS compilation is like CLJ compilation. We compile third party deps as separate rules and use their namespace caches to compile dependent rules. The hard part is getting this to be fast - the cljs compiler isn't fast to boot. To help with start time, we developed a little clojure class that lets us inject a ClassLoader into a running clojure process. It properly manages both loading and unloading of clojure namespaces. We then use that as part of a worker_tool rule (https://buckbuild.com/rule/worker_tool.html) to keep a warm compiler for the entire build (in a similar spirit to Boot's "pods"). We're hoping to open-source what we're calling PersistentClj separately.
As with all things novel, this is a bit experimental right now. We only recently started using buck for development. Moving production over is the next big step. There's still a long way to replacing figwheel usage, but we're going to try to do it. The biggest roadblock there is time - our priority is always on our product.
We are definitely planning on opening this up once we've proven it out, so stay tuned. Or even better, come help us build it :)
I'm now learning Clojure and simplicity and power of lisp just blows my mind, really happy to write on it. But I wonder what of these books do you recommend to read to those people who are using Clojure?
If you ever wondered why Lispers were considered condescending, arrogant, self-absorbed dorks back in the old ways reading LoL might give you a few hints.
I can't recommend that Clojure people read Hoyte's masterpiece, because Clojure is the first "larger" Lisp without a completely horrible community and I'd like to keep it that way.
As a relatively recent user I can only add that the community around common lisp, while being fairly distributed, are a great bunch. I have found on average no more douches than occupy any community.
