A big part of the presentation so far was an engineer describing the huge difficulty of stitching together the multiple cameras into one vector space that can be the input to the network, instead of treating each camera individually.
Seems like the biggest problem was each pixel from a camera does not tell you how far away it is, so even if you know the camera is X feet off the grounding pointing at Y degrees, you don't know if there's a wall in front of the camera or not. So you can easily reconstruct a 2D space from all the cameras just by knowing their positions, but you can't simply translate that back to 3D.
I could barely follow the solution to this problem, but it seems to approximate the distance somewhat okay... but it required yet another universe of features in the neural net pipeline just to achieve it.
The funniest part is... isn't this exactly what LIDAR is amazing at?? Or what about having each "camera" actually be two cameras that can achieve depth via parallax?
While taking two sequential images from the same camera in can provide this depth information also. But simple use of stereo cameras can solve a magnitude of problems (standing still and low parallax motions). Traditional stereo and even machine-learning based methods have have great success and accuracy for many years and could easily be an alternative to LIDAR also.
I really don't know why this isn't leveraged more (maybe it is and I am unaware?).
Tesla has multiple cameras looking out the front, with some offset. I believe that in past talks they did talk about using that parallax to achieve better depth.
That was basically what I was thinking as well. There's a reason that Mobileye's camera only solutions tend to have a lot more cameras than Tesla does. Stereoscopic vision would be a big help, and tesla only has it in some directions and with relatively low resolution.
Yes. This is exactly what LIDAR is for. This is the reason every single team in the 2007 DARPA Urban challenge that finished the race was equipped with a Velodyne laser. We knew it was a critical enabling technology 14 years ago, and it baffles me Tesla is eschewing it. Tesla's lack of this technology is the main reason I feel they will never achieve what they claim to without a huge breakthrough in AI.
If Tesla switches to stereoscopic vision with fully articulated cameras that can move independently of the vehicle (not fully independent, but limited 3 degrees of motion), and then they manage to integrate the output into something meaningful then maybe it would be comparable.
Animals tend to use a combination of four tools to detect distance. Stereo vision, parallax, focus detection and scene context. Independent movement is not required for any of these (though it can help generate parallax.) Humans (and other animals) can learn to do pretty well with one eye and no head movement given the appropriate training regime.
It makes the software problem harder but it certainly is possible.
They already have multiple cameras; it's not clear why articulation of cameras would be required as they do not have foveas that require such articulation; they already do integrate the output of each camera into a view of the vehicle's surroundings.
Animals also have (non)-artificial general intelligence. That doesn't mean Tesla's making a reasonable design choice if they build something in a way that relies on having that kind of capability.
The way I imagine it, Elon made Andrej put in that slide about "We are effectively building a 'synthetic animal' from the ground up." It vaguely hints at a small-scale AGI. A little bit of Andrej's soul dies when he has to says those ridiculous words, but he views this as the price he has to pay to have his dream job. Anyhow, that's how it would have played out if i had Andrej's job.
Stick an animal, even a human, into a room of non-realistic objects, misleading proportions etc., and see how well it orients itself: it will be very bad. That is because sight and orientation rely heavily on intelligence: we can tell how far something is because we recognize the object and can tell whether it is small and close or large and far away. Even if one object is misleading, we look at other objects around and can usually tell if we're looking at a toy car on a real road etc.
But betting your FSD on doing object reconition and physical intuition seems like a bad bet.
When we're talking about the brain and its processing, planning, and decision-making abilities exhibited during driving, no. We are distinct from animals. We have the technology to build robots that mimic the capabilities of animals, from birds to fish to snakes, and now approaching dogs. Trying to do what humans do while driving with binocular cameras is far beyond our capabilities. Like I said, if you want to drive with just binocular cameras like a human, you'll need a human-level AI brain to go with it, and that's not coming any time soon.
edit: I don't understand how this is a controversial statement...
Animals do just fine with more complex navigation challenges than driving a car. While we can 'mimic' various animal capabilities, I don't believe we are anywhere close to AGI with either a dog's level of intelligence, motor control or sensory integration.
> Animals do just fine with more complex navigation challenges than driving a car.
Do you have any scenarios in mind? Driving is not only a navigation challenge, but a social challenge as well. Have you ever tried to navigate a 4 way stop when multiple cars get there at the same time?
The theory of mind problems involved in driving are huge and are primarily why I don't think you could teach an non-human animal to drive on streets / in traffic.
But depth perception isn't really involved there so those theory of mind issues don't really have any bearing on the need for lidar for self driving cars. Clearly creating world models of sufficient accuracy for driving scale navigation challenges is possible (but potentially pretty hard) with just vision data.
I think originally it was because of the cost of the equipment. Elon and team saw how expensive lidar was and thought they needed to be able to solve the problem without it. However lidar has dropped drastically in price, so at this point I think they aren't using it just for ego reasons (i.e., we claimed it was possible to get self driving without lidar in the past, so we have to continue down that path)
Lidar might have dropped in price, but please show me the Lidar that can produce that quality output, is incredibly cheap, easy to manufacture and integrate into the car and available millions of time by next year.
And it would be great if it something that almost never broke as things that move a lot tend to. As far as I know static lidar are nowhere near as good.
Then if the difference in quality is not actually that great, doing the development effort once will scale to millions of cars the next few years. So we are literally talking many, many years of millions of lidars produced and integrated (and not replacing other sensors) to maybe have a slightly better outcome.
However that depends on your sensor fusion and how the lidar handles weather conditions and other complexities.
Just because not needing lidar would be great for Tesla, doesn't make it real. That's wishful thinking. Tesla is not entitled to a well working lidar-free solution just because they need to sell cars right now.
Can you point me to literally any evidence what so ever that the results fake? Is the FSD Beta regularly crashing into things it put at the wrong range?
Tesla has cars with Lidar on the road to check the models as well.
And who says anything about entitled, they have spent a huge amount of effort and therefore money into a solution that they think will be better.
Nobody with Lidar has delivered either so what is better is still an open question. Tesla has picked their strategy and working on it. Others do the same.
My point is simply, not that Tesla has the perfect solution, rather the idea that Tesla is operating based on 'ego' is idiotic. Elon Musk and his companies have repeatably shown that they are very willing, comically so, to throw away work and go back to the beginning if they have encountered a problem. E
lon is well known for not falling into the sunk cost fallacy. Consider the manufacturing of Model 3. Consider Starship being planned to be built from carbon fiber, and switching to stainless steel.
Do you really believe they know lidar would be better and they not changing because of what Musk said 4 years ago? That is literally the opposite of how his company have operated.
That's exactly a range thing: LIDAR could tell you that there is an obstacle in front of you, it couldn't mistake a 3D object for a 2D picture on the road.
Just in case you missed it, the person you're replying to asked a focused question about the new FSD Beta software.
The investigation you reference stems from previous versions of the softawre.
The broader context can't be ignored when discussing the new software, but it's reasonable to point to progress as demonstration that the chosen technology path is a viable one.
Like you say, FSD beta is new, so, no. The autopilot software has killed people though.
The sunk cost fallacy is stronger when it's not just R&D but thousands of people who paid 10k$ on the promise that their car had all the hardware required... Cars that shipped without a lidar.
And I believe waymo uses lidar and has, as far as I know, killed nobody. They also take a more cautious approach where bystanders are not turned into beta testers, of course :-)
> The autopilot software has killed people though.
Sorry, what’s your point? You need to prove that the autopilot software has killed more people than if the cars were driven by humans.
> And I believe waymo uses lidar and has, as far as I know, killed nobody.
With a footprint probably orders of magnitude smaller than Tesla. What do you want to bet is the difference in terms of miles driven by Waymo vs. miles driven by Tesla autopilot?
> And it would be great if it something that almost never broke as things that move a lot tend to. As far as I know static lidar are nowhere near as good.
Keep in mind you're saying this about something that attaches to a car, which already has dozens or hundreds of parts that rotate at hundreds or thousands of RPM.
> Then if the difference in quality is not actually that great, doing the development effort once will scale to millions of cars the next few years
The adage "do things that don't scale" comes to mind. If your options are "have tens of thousands of autonomous vehicles" or "have none", I'm not sure how the second is better. And we know that hundreds of thousands or even millions of lidar units are being produced (cruise and waymo and co have multiple units per vehicle and collectively hundreds of thousands of vehicles).
Adding more rotating parts is exactly what you are trying to avoid. For example, go look at how Tesla designed the heating and cooling system. The integrated all the different streams of hot and cold in one system duplicate these parts that can break easily.
In some cases it can simply not be avoided of course but you better make sure that you really need it if you are gone add it to millions of cars.
> cruise and waymo and co have multiple units per vehicle and collectively hundreds of thousands of vehicles
Waymo has vehicle count of a few 1000s as far as I know. I don't know about cruise but I don't think the have even 100k vehicles. Please show me a source.
And even if that were the case, the build those fleets of quite a long time and the cost per vehicle they currently have is nowhere near close to what would be possible for a car like the Model 3 or future cheaper models.
Margins in car industry are incredibly tight, car companies invest gigantic amount of capital to remove a few $ per car. These vehicles need to be able to be produced at a run rate of millions per year. If you are proposing to add something that cost 100-1000$ that is a absolute killer.
In addition its an additional part that can hold up your production. This current chip shortage is a perfect example where having a Lidar would just introduce a whole host of new chips that might be in a shortage.
> Margins in car industry are incredibly tight, car companies invest gigantic amount of capital to remove a few $ per car. These vehicles need to be able to be produced at a run rate of millions per year. If you are proposing to add something that cost 100-1000$ that is a absolute killer.
I don't get this. You have to pay ~8000$ as an extra to get the FDA in a Tesla. Can't they charge the cost of the lidar in there? Now you have to pay 9000 dollars to get it.
You didn't really address the meat of the comment: The adage "do things that don't scale" comes to mind. If your options are "have tens of thousands of autonomous vehicles" or "have none", I'm not sure how the second is better.
Waymo and cruise especially, but even a number of others, have demonstrated better autonomy than Tesla. Musk has been claiming full self driving is 6 months away for 5 years now, and he hasn't gotten that much closer.
> Waymo and cruise especially, but even a number of others, have demonstrated better autonomy than Tesla.
Have they? They have a different approach where they focus huge effort on individual Geo-fenced locations and they both are losing 100M of $ every year. Neither has given any timeline for general availability in all locations.
The race, as far as I am concerned is still open. It is not at all clear to me that Waymo is ahead at what actually matters.
> Musk has been claiming full self driving is 6 months away for 5 years now, and he hasn't gotten that much closer.
The claim they made no progress is objectively false.
> They have a different approach where they focus huge effort on individual Geo-fenced locations
Where they actually have driverless vehicles. That's the key difference. Waymo and cruise have demonstrated driverless vehicles. People get driverless taxi rides today from waymo, and I think cruise as well. Tesla doesn't. It's still sitting in driver assist land and isn't particularly better than other luxury vehicle driver assist.
> The claim they made no progress is objectively false.
Wasn't another factor the physical size of the lidar units? Part of Tesla's schtick is making normal or even attractive looking cars (as opposed to the "alien bug" aesthetic EVs were synonymous with at the time) and that's a lot harder to do with a big lidar unit on top.
Also they marketed autonomous driving features as future software updates (including a discount/price hike for early/later adopters). Early adopters would be unhappy if future features required additional hardware.
They upgraded hardware in the past to support FSD customers, so I’m not sure they’re completely against it. If it was, say, just a $500 decision to make it completely viable, I don’t think Tesla would hesitate. But the $500 lidars can’t do what needs to be done.
> so at this point I think they aren't using it just for ego reasons
If their engineering is driven by ego (most of us think - it is) then - are they really engineers? We all know it’s coming from Top - Elon, in this case.
See, for example, Andrej's talk starting at the 6-minute mark. Lidar requires a pre-rendered detailed map of the lanes, traffic lights and obstacles. Vision can operate in any novel environment the car is not preprogrammed for and is thus more scalable.
I for the life of me cannot find the quote, but at some point I believe he said that if humans can judge distance/drive with nothing but vision, cars/computers will be able to as well.
It was around the time he made the comment that anyone using lidar is doomed, which is much easier to dig up as it was a headline everywhere:
> Your eyes already do the distance estimation and they are basically cameras so we can do this just as well or better.
This is not really accurate. Our brains do the distance estimation, and they use all kinds of tricks and contextual clues to do it, not just parallax (~2.5 inch parallax for objects more than ~100ft away isn't all that helpful). And this is the whole problem with vision-only autonomous driving - ML capabilities are nowhere near the capabilities of a human brain.
Monocular depth estimation has gotten really good recently though[0]. Not saying this one paper/method is 100% sufficient, but we're closing the gap in this one capability (depth estimation from pure vision) quite rapidly.
That’s exactly what Tesla described tonight. 8 video cameras on a moving platform provide enough eyes, parallax, context, etc to build accurate 4D (!) models in real time.
> Our brains do the distance estimation, and they use all kinds of tricks and contextual clues to do it
So, first, this presentation we're talking about is literally about problems like that.
Second: your brain isn't nearly as good as you think it is, it's just constructing a coherent story to fool you into thinking it is. Try this on the highway sometime as a passenger: close your eyes and recite the distances to the vehicle in front of you and the one to either side. I bet you anything a Tesla is going to do that better.
Third: they pretty much cracked this already. They stopped shipping radar on US Model 3's and Y's in the spring, have shipped hundreds of thousands of them now, and there's not a hint of signal that something is off with distance measurements with the cars. My car doesn't get this perfectly (you can actually watch the animations on screen bounce around a bit as the estimates change) but I think it objectively does better than I do.
Distance/Lidar framing is old news, basically. Vision works fine. The worst bugs remaining with all the FSD Beta footage on Youtube are almost entirely pathing and planning issues. The car sees its environment just fine.
Yeah but some people suck at throwing darts. So some part of their mechanical system is bad at making judgements. Sure they can learn. But I want a 2 ton cyber truck hurtling down the freeway to KNOW how far things are away.
Telsa should pivot and innovate to make lidar cheaper.
It doesn't work well in rain, snow, fog, dust, etc. It's great in good conditions, but you either have a car that can drive only in good conditions, or you need a car that can drive safely without LIDAR. (Or someone needs to invent a better LIDAR.)
I think Elon basically says that long term they will only need vision so they will spend all their time focusing on vision from the start. Maybe lidar will succeed first, but it will be a worse success than vision: redundant / expensive / intrusive.
Seems like the biggest problem was each pixel from a camera does not tell you how far away it is, so even if you know the camera is X feet off the grounding pointing at Y degrees, you don't know if there's a wall in front of the camera or not. So you can easily reconstruct a 2D space from all the cameras just by knowing their positions, but you can't simply translate that back to 3D.
I could barely follow the solution to this problem, but it seems to approximate the distance somewhat okay... but it required yet another universe of features in the neural net pipeline just to achieve it.
The funniest part is... isn't this exactly what LIDAR is amazing at?? Or what about having each "camera" actually be two cameras that can achieve depth via parallax?