Nice language with many built in features! Do you gain much in comparison to languages, where these concepts are implemented as frameworks? Not clear, how the memory management is implemented? Is it manual?
I tried to keep the ideas conceptual for and not bring too many frameworky features to the language itself but rather a the standard library.
However, there are features that I intentionally brought it to the language:
- DI and IoC features:
this language is not made for writing hello world programms, rather some microservices and the code suppose to have abstraction layer and impl layer that connected via DI
- Different Function Types: the reason for having differnet function types is rather memory management, basically give Compiler more information; therefore, at the compile time we know how to manage the memory, It also brings some limitations (for example, a mapper function is not allowed to make any mutation or actions calls)
- Memory management: it's still not perfect as the compiler is still not in the fully self-hosted stage and currently it complies to C99 then to binary, but the idea is to avoid GC while not making hard-to-learn syntax (e.g. Rust borrowing)
see more at : https://code-by-sia.github.io/xi/memory/
The comparison with Cython is indeed very relevant as it is currently the standard for making Python modules and it has greatly improved support for the standard Python syntax over the last few years, e.g. the type hints. However, it still can not produce an executable independent from the Python runtime. Besides the manual memory management available in Cython, Nimic also provides deterministic memory management (ARC/ORC) rather than the Python's reference counting for the compiled code. Nim intermediate code is easier to debug than Cython produced C and the code can also compile to LLVM IR. I'm not an expert on Cython, does it have the modern systems language functionality at level it is present in Nim?
True, nimic is a statically typed Python subset without much of its dynamism with the aim to be foremost an efficient systems language. Though in nimic, without using isinstance, the instance-based dispatch is realised via variant types. Yes, emulating the Nim constructions in Python was the hardest part, while making the transpiler was straightforward.
Indeed, the AOT compilation leads to great speed-ups for heavy custom numeric calculations that cannot be easily vectorized in numpy, such as the raytracing logic. In cases when most of calculations are performed in an external module (written in e.g. C, Rust or Cython etc) the performance gain might be much less, but the added value here is that the high performance module itself can be written in nimic, keeping the codebase purely in Python and consolidating the codebase.
When starting with a "pythonic" code, rewrites in nimic can be substantial but so would be a rewrite in a systems language like C or Rust. Besides allowing to optimise the fast path, nimic provides low level functionality, such as pointers to pointers and bitwise operations that actually executes within CPython, for example, as in mp4 muxer implementation in dima-quant/ndsl_raytracer/src/nraytracer/minimp4.py
the variant-types-for-dispatch thing is a nice tradeoff, makes sense. honestly the strongest pitch in there is the "write the hot module in nimic instead of dropping to c/rust/cython" angle, which is basically the cython niche. so the question is what nimic adds over cython for that. my guess is dual-mode: a .pyx isn't valid python and won't run under plain cpython, but nimic stays importable and debuggable in cpython and you compile only when you want speed. if that's the edge i'd put it front and center, it's a way sharper sell than "python but fast".
one thing i'd worry about with "runs unmodified in cpython" though: python ints are arbitrary precision and nim's aren't. so the same code can give you a bignum under cpython and a wraparound under the compiled path. how do you handle that, or is matching cpython semantics explicitly a non-goal for the typed subset?
So I'd say in Nimic Python you get Nim-style integer emulation when run from Python, keeping both paths consistent with each other - but breaking consistency with the rest of Python. Which is OK, I think, given it's explicitly a subset(s) of the language(s). It would be possible to make `int` transpile to some BigInt Nim implementation, but you'd need an external dependency for this, as they are not in Nim's stdlib. However, in the speed-focused context, I'm not sure if defaulting to BigInt every time the compiler sees an `: int` annotation would work well. It's a hard decision to make. Curious what's the OP opinion here?
Your explanation is completely correct. There are Nim implementations of BigInt, but the foremost focus of Nimic is enabling as many standard Nim features as possible. In general, there is another parallel stream, which aims at implementing many Python features in Nim (both approaches can be combined, extending the Python subset covered by Nim implementation): https://github.com/nimpylib/nimpylib/wiki
indeed, as far as I know, at this moment nimic is the only package that provides systems language functionality running in CPython, while the corresponding code also compiles AOT to an efficient native binary.
nimic is a lightweight pure Python package that emulates Nim types and constructions, making it straightforward to transpile to Nim and compile AOT. Key principle: nimic code is valid Python that runs unmodified in CPython and also transpiles to equivalent Nim code.
Because nimic code is just standard Python with type hints and ctypes shims, it is a fully valid CPython script, so you can use the Python REPL during development, drop a breakpoint in the middle of a heavy algorithmic loop and inspect the variables natively.
Zero Lock-In: You don't need a special runtime engine. If the Nim compiler is not available, your script still runs (albeit slower than standard Python due to some emulation overhead) on any machine with Python installed.
Seamless Distribution: You can use this to develop high-performance logic natively in Python, debug it with Python tooling, and then compile to a native executable or C-extension via Nim.
Why Nim? Its syntax maps well to Python, it is rather clear how to emulate its constructions in Python, and its performance is comparable to C (as it compiles to C). Port of the "trace-of-radiance" Nim project to nimic can be found in "ndsl_raytracer" in my GitHub repo (dima-quant). With the compiled executable the render time for a single 512x288 scene dropped from many hours in Python to just 10 minutes on a single M1 CPU core. The repo also includes nimic ppm to mp4 converter.
Similar projects:
- Pyccel (https://github.com/pyccel/pyccel): Python extension language using accelerators
- SPy (https://github.com/spylang/spy) is a variant of Python specifically designed to be statically compilable while retaining a lot of the "useful" dynamic parts of Python.
- Codon (https://github.com/exaloop/codon) is a high-performance Python implementation that compiles to native machine code without any runtime overhead.
It is still work in progress, e.g. there is no JIT and multiprocessing support yet, but now I'm not sure what functionality would be best to implement next. Any suggestions?
If it's possible and you intended for it to be a long-term project, I would suggest looking at designing an MIR for the Python to lower to and then lowering it to Nim, and that is probably the most valuable feature you can add to Nimic. That's what Rust had to learn the hard way, and that's how LLVM works, and having an intermediate representation means your compiler is going run into a lot less edge cases during development.
Currently, Nim fits quite well with Python but there is a project to directly emit LLVM IR from Nim and MIR might be relevant there: https://github.com/arnetheduck/nlvm
Oh, I wasn't referring to emitting LLVM IR directly from Nim, but that you should consider adding an MIR layer between your Nimic subset of Python and Nim instead of emitting Nim from Python directly. Otherwise, compilation would get difficult and adding new supported Python will get painful.
so it is what https://github.com/mypyc/mypyc was supposed to be ... i can just use write my python code add some type hints and it should run faster ... like cython but without learning a new dsl?
Yeah, indeed, I actually read about mypyc a few years ago before starting nimic. Though the goal is very similar, mypyc does not support compilation to a native executable and the performance gains were somehow limited to 2x-4x, as reported in https://sichard.ca/blog/2022/05/compiling-black-with-mypyc-p...
Not quite "all valid Python." Just to be clear: nimic is a strict, statically typed subset of Python. At this stage, the downside of having Nim as an intermediate step is not really a slower compilation, as Nim compiler is very fast, but more like:
debugging executable, might involve line numbers of the intermediate Nim or C source code;
the need to install Nim compiler for development;
memory management (currently it is ORC/ARC or manual in Nim, no borrow checker yet)
Yeh, indeed, the README was mostly generated, though the introduction is largely human-written :-) Is some specific description missing? I can provide more practical code examples directly in README. The benchmarks are mostly on runtime performance I assume?
No, the problems are a) you get the same exact AI "voice" that is tedious to keep reading, b) it's verbose and focuses on the wrong things (the whole Module Architecture section doesn't belong there), and c) it's a sign that it's slop and not well tested.
> The benchmarks are mostly on runtime performance I assume?
You tell me! You (or Claude) make performance claims - "aiming to get C-level performance without leaving Python". Does it actually get anywhere near that claim?
Fully fair, I'd need to spend more time on the text. The nimic was mostly tested on the raytracer project dima-quant/ndsl_raytracer, and the performance was indeed quite near the C-level.
I enjoyed a lot reading the book and also learned a few things there: The Pragmatic Programmer, 20th Anniversary Edition. The main guiding principle: Good design is easier to change than bad design
AI is very good at standard and well defined problems but much less so with complex and vague. For example, currently available models cannot write or even port a full-feature compiler of a mature language without very custom tweaking and direct supervision. Even with multiple agents, it still remains just a tool, though quite powerful. As you already know physics and programming, data science can be a natural alternative that requires both skills. From my experience, many problems there are vague and most of the job is to actually make a proper representative dataset and translate vague requirements to something well defined. AI helps with the actual implementation, the other part remains mostly human.
