It depends on what model you’re running, and for what workload. For personal use (one or two convos at a time) with models that fit in gpu memory, pcie bandwidth doesn't really matter. Just try and be on gen 3 x8 or higher.
Llama is decent at auto optimizing it if you let it use both gpus. It’ll split the workload so the contiguous layers are all on one gpu. Once the model is loaded, you only transfer weights between gpus once (per token?), at the layer boundary.
I run on an 8gb 3070 and 12 gb 3060, and the only weird thing is that the weaker card gets more layers (and therefore work) because it has more ram.
Oh, if you’re barely fitting the models into your vram, you may need to explicitly adjust the layer balance between cards — sometimes it fails to realize it should have put certain things (like draft models) on the other card so you can fit one more layer in.
> Given the lack of forecasting signal, the obvious next step then is to seek out external (exogenous) features in the real world that can help prediction models. After all many real world time series are event-driven (FX, bitcoin) and hence they are exposed to shocks and drifts which can also be measured or accounted for in the data generating process.
This is the approach I'm using at my job, which is incident detection with customer metrics. We're tagging our time series data with common features -- such as country, customer type, etc -- with the idea that we can do a graph-like search to find exogenous variables. We can also use this to identify time series that have a similar "data generating process" and are simply different "realizations" of each other.
We don't need great time series forecasts, just something that detects large deviations quickly. We can then add in an existing dataset of _known_ incidents, indexed by the same common features, as a training/validation set.
> We can then add in an existing dataset of _known_ incidents, indexed by the same common features, as a training/validation set.
Beware of the Anna Karenina principle. Well behaved data might be explicable by the same common features, but often the anomalies all have unique characteristics.
To be fair, the article calls that out specifically at the end:
> What makes this especially frustrating is the lack of transparency. Every other ADS company in the NHTSA database, Waymo, Zoox, Aurora, Nuro, provides detailed narratives explaining what happened in each crash. Tesla redacts everything. We cannot independently assess whether Tesla’s system was at fault, whether the safety monitor failed to intervene in time, or *whether these were unavoidable situations caused by other road users*. Tesla wants us to trust its safety record while making it impossible to verify.
“Lacking” implies it’s an unintentional deficiency.
Rust has deliberately taken a “batteries not included” approach due to learned experiences in python, where many std-lib packages are basically abandoned/not recommended anymore.
This gives the ecosystem time to develop better approaches to things, rather than ossifying around the initial implementation.
Yes, it does lead to a larger set of dependencies, but some of the most common packages (including anyhow) were written and are maintained by members of the rust lang organization/team itself.
> But it is a problem, because somewhere along the line one of those indirect dependencies is going to start conflicting with something else in the graph; the bigger the graph, the more conflicts.
Rust is statically compiled, so it’s possible to have different versions of the same dependency in your dependency graph without build issues. Yes, that can lead to binary bloat and slower build times and some related problems, but in practice the compiler is good at dealing with those issues.
Why do you say shuffle is “SIMD’s killer app”? I’ve only dabbled in vector instructions from a learning perspective, and seen others mention it’s important too, but have yet to understand why.
Because you can do things using bitmasks and single instructions instead of brute forcing using multiple instructions.
Let's say you have a whole heap of 8-bit numbers you want to multiply by 2 and you have a set of 256-bit registers and a nice SIMD multiply command. If you don't have a shuffle you need to assemble your series of 2s for the second operand for each lane before you can even start. This is going to take hundreds of instructions and hundreds of clocks. Shuffle means you load up lane 0 with the "2" and then splat the contents of lane 0 across the other 31 lanes in two instructions and a few clocks using the shuffle unit.
N.B. Shuffle isn't just about splatting. There's a whole heap of different operations it can do that are useful. I just picked a simple example with an obvious massive performance increase for illustrative purposes.
You're not talking about shuffle, you're talking about broadcast. Shuffle instructions is where you take one or two vectors, and output a third with elements from any index of the input. So for example `out = [in[2], in[1]]` is a shuffle of a vector of length 2.
It's useful for example if you have say RGB color data stored contiguously in memory as say RGBRGBRGBRGB..., and you want to vectorize operations on R, B and G separately. You can load a few registers like [RGBR][GBRG][BRGB], and then shuffle them to [RRRR][BBBB][GGGG]. In fact it's not entirely trivial how to shuffle optimally, it takes a few shuffles to get there.
More generally, if you have an array of structs, you often need to go to struct of arrays to do vectorized operations on the array, before returning to an array of struct again.
Another example is fast matrix transpose (in fact you can think of the RGB example a 3 by N matrix transpose to N by 3, where N is the vector width -- AoS -> SoA is a transpose too, in a sense). Suppose you have a matrix of size N by N where N is the vector width, you need N lg N shuffles to transpose the matrix.
I think that example is too simple to show the benefit of shuffle. It's like explaining the benefit of an adder by showing how you can move a value with X = Y + 0. Especially since there's also a (much simpler) piece of hardware dedicated to ultra-fast splat/broadcast (under the right conditions).
It's basically several moves for the price of one. Given that you operate on multiple values at once, being able to shuffle or duplicate values comes up all the time.
For example if you're filtering four image lines at a time using a 1D filter kernel, you'll want to replicate the filter coefficient to each SIMD element, so that you can multiply each of the four pixel values with the same coefficient. Shuffle lets you replicate a single coefficient value into all the elements of a register in one instruction.
This Rust issue [0] was the best short summary of what an SIMD Shuffle is I could find:
„A "shuffle", in SIMD terms, takes a SIMD vector (or possibly two vectors) and a pattern of source lane indexes (usually as an immediate), and then produces a new SIMD vector where the output is the source lane values in the pattern given.“
Those numbers aren’t directly comparable — caches usually trade off size for performance, and Apple’s L1 sits in performance somewhere between Ryzen’s L1 and L2. Ditto for Apple’s L2 versus Ryzen’s L2 and L3.
I can’t remember the exact article I read that in, but it’s pretty obvious when looking at Anandtech’s latency charts.
Road/car dominance wasn’t exactly established in a free market either.
There’s a whole host of subsidies for building/maintaining roads paid for by everyone [1]. There’s the regulatory advantages driving has (ex: the common zoning requirement that new construction ensure enough parking space [2]). There’s the negative externalities of highways/roadways. (Ex: the health impacts of pollution[3], and the health impacts of sitting in a car rather than walking/biking (part of the way) to their destination [4])
Jane street had a puzzle related to this, called eldrow [1], a few months ago.
The gist is you’re trying to find the worst sequence of (hard mode) guesses possible for any word in the wordle set.
Their site shows submissions of up to 16 words and claims 17 is impossible [2]. I’ve been trying to do the same, but have yet to cut the search space enough[3].
Ex: calculating the worst chain for jazzy (with a 1000 word subset of the full word list) took an m1 core 1 hour and resulted in the ten-word chain:
> I think apple (maybe as just an arm feature?) can do encrypted pointers, with a per application key tracked by the kernel.
ARM has added both pointer authentication codes and memory tagging extensions to their ISA, from (I think) ARMv8.5-A. Apple are the only ones to implement silicon that supports PAC that anyone can buy today but a) this will change fast and b) I might've missed something else.
I wouldn't say "encrypt". You can still read what the pointer is, you just can't forge it. It is more like a message authentication code, or keyed hash. Confusingly, the ARM ISA suggests you use the block cipher QARMA, a lightweight block cipher of their design. This is not a mistake - from a crypto academic's perspective the distinction between block cipher and hash is not so easy to draw as it tends to be communicated: you can get a block cipher out of the SHA family of hash functions (called SHACAL) and so on. There was a line of work on finding a suitable family of permutations (e.g. Gimli) that could be hardware-accelerated and appropriate constructions built from that.
tl;dr it is kinda a "signed pointer", and if you forge the pointer but get it wrong you don't get another chance because the CPU triggers a fault, so you can use lower security lightweight ciphers and still make exploitation very unreliable.
Now Fuchsia:
As to how Fuchsia does it, we can just look at the source:
For example. I don't think there's any magic here: the kernel maintains some structure to reference jobs, tasks or whatever we are calling our isolated privilege boundaries, in which the capabilities are described. On the face of it this "looks" a lot like the traditional permission-style system, but the key difference is that the root job will hand out capabilities to processes it has created. An example is probably the best way to proceed: on a linux system, you might decide your process needs to read `/system/sensitive_file`. To allow this to happen, you might grant access to the user using some other command: chown .../chmod ..., or you might set an selinux policy, semanage fcontext -a -t myprocess_t /system/sensitive_file. In a capability-based system, chmod/chown and even running the process as a user don't exist as concepts. You would need to use a process authorized to transfer a capability to your target process to access this file. In some ways, this is closer to selinux, in the sense that semanage fcontext updates policy, except that the policy is somewhat dynamic here: if your controlling process provides the capability, then the child process gets it.
In a microkernel-based architecture, the idea is that "servers" (jobs that are arguably part of the system, but don't need to be in kernel) are similarly userspace processes subject to said capabilities also.
Of course, some part of the code still needs to be privileged here, let's call it the "executive". This code is doing the capability enforcement, and if that code contains bugs, all bets are off, as always. You can't defend such code from itself, even with capabilities.
The real disappointment is that Zircon is written in C++. Granted it was probably started before Rust was a thing, but while Ada/SPARK might be considered a bit unfashionable, it has some formal verification available, and if you're going for a software solution this should help reduce exploitable bugs. Personally I also wonder what about seL4 made it unsuitable for this: granted, it is not complete as a system, but neither is Zircon alone.
> ARM has added both pointer authentication codes and memory tagging extensions to their ISA, from (I think) ARMv8.5-A. Apple are the only ones to implement silicon that supports PAC that anyone can buy today but a) this will change fast and b) I might've missed something else.
I'm not aware of other generally available chips with pointer authentication. The Qualcomm Snapdragon 8cx Gen 3 supposedly support PA. The Lenovo ThinkPad X13s supposedly contains it, but when will it actually be commercially available is anyone's guess.
True. I was mainly responding to the ARM encrypted pointers part with that bit. ARM also didn't invent memory tagging, which might also have come from SPARC ADI (I'm not sure), but they also plan to include that. QARMA itself looks to be designed by Qualcomm engineers: https://eprint.iacr.org/2016/444.pdf
Yeah, ISA says you can. I wouldn't be surprised if other people pick other implementations, I'm not entirely sure why ARM felt the need for QARMA in the first place given there are already extensive choices for lightweight block ciphers, but... there you are.
Llama is decent at auto optimizing it if you let it use both gpus. It’ll split the workload so the contiguous layers are all on one gpu. Once the model is loaded, you only transfer weights between gpus once (per token?), at the layer boundary.
I run on an 8gb 3070 and 12 gb 3060, and the only weird thing is that the weaker card gets more layers (and therefore work) because it has more ram.
Oh, if you’re barely fitting the models into your vram, you may need to explicitly adjust the layer balance between cards — sometimes it fails to realize it should have put certain things (like draft models) on the other card so you can fit one more layer in.