With version 9 of the FSD Beta Tesla has rolled out an updated view of the car visualizations, known as “Mind of the car.”
Elon Musk is calling the visualizations, 'mind of the car' because Tesla is attempting to show you what the car is recognizing and it's confidence level in a visual way.
The new visualizations are combining what the FSD beta used to recognize in beta 8 and earlier into a more production ready visualization. The visualization is similar to the production version in Teslas today, but it's much more advanced in what it shows. The new visualizations are all 3D objects rendered on the screen instead of the dots and rectangles we saw in earlier betas. The updated visualizations are able to render complicated intersections, curves and turns. Similar to earlier betas they paint road boundaries in red and divide on-coming traffic with same-way traffic with a purple line. The yellow lines represent their real life counterparts, lines that you should not cross on the road. In addition lane markings are more consistent and smooth.
Tesla FSD beta 9 visualization is amazing. Hope this will be incorporated in upcoming V11 software pic.twitter.com/gbUKjSXiyl
— Blue Sky Kites (@blueskykites) July 11, 2021
With this beta, Tesla also switched over to vision only and stopped relying on radar. Due to improvements with their vision detection the car is now able to detect and display objects much more accurately. The car detects and displays more vehicles in the visualizations and more detailed surroundings.
Tesla found a clever way to show how confident the vehicle is in each object it's displaying. The brightness and translucency of each object will vary, depending on the confidence level of the neural net for that object.
Mind of car visualization will improve a lot in coming releases.
— Elon Musk (@elonmusk) July 10, 2021
Brightness/translucency indicate confidence of neural net in predicting any given object.
Acid test is being able to assess with quick glance at screen if car fully understands environment.
Although previous betas showed a rectangular prism for each object detected, even if the car didn't recognize what the object was, this beta requires knowing what the object is before it will render it on the screen.
I expect this to change and Tesla to not only add many more 3D assets, but also to come up with a way to show an unknown object. Maybe they go back to a 3D rectangle for these situations. Elon Musk has said that car visualizations will “improve a lot in coming releases.” He recently commented that turn signals, hand gestures and more are going to be added soon.
In order for Tesla to achieve a high level of autonomy, they will essentially need to be able to build a highly detailed video game that represents the real world, in real-time. We are getting closer and the visualizations help us understand what the car sees and understands. With the ‘Mind of the car' visualizations we will now be able to tell how confident the car is in determining certain situations and that will let us better understand when we should take over.
