The evolution of Tesla’s FSD suite has been consistently defined by a singular engineering philosophy. The physical world can be navigated and interacted with best through pure vision and generalized artificial intelligence.
In previous installments of our How FSD Works Series, we explored how FSD’s underlying neural networks construct three-dimensional spaces out of two-dimensional pixels and track moving objects through time. While that’s all very complex, one of the most persistent hurdles in autonomy doesn’t involve dynamic hazards; rather, it involves understanding the invisible, shifting, and human-oriented logic of road architecture.
When an autonomous vehicle approaches a complex, unmapped, multi-lane intersection, it must determine which lane connects to which exit vector across the intersection. Competitors like Waymo and Zoox attempt to solve this by anchoring themselves to brittle, high-definition (HD) localized maps. Tesla, conversely, treats the problem as a living puzzle.
According to a recent patent application, US 2026/0170852 A1, titled "Vision-Based Machine Learning Model for Lane Connectivity in Autonomous or Semi-Autonomous Driving," Tesla has unlocked this capability by repurposing an architecture typically reserved for generative AI and LLMs: the autoregressive transformer.
Tokenizing Asphalt
To understand how Tesla’s lane connectivity engine operates, it helps to look at how Large Language Models (LLMs) compose sentences. When a system like Grok generates text, it does not produce a complete paragraph at once. Instead, it breaks language down into discrete fragments called tokens, which are syllables or words, and predicts the most statistically logical next token based on the context of the tokens that preceded it.
Tesla is applying this exact logic to road geometry. As the vehicle traverses a real-world environment, the vision suite streams raw imagery into FSD’s neural networks. These raw pixels are filtered through backbone networks and fused via a cross-attention transformer into a three-dimensional, top-down vector space known as a bird's-eye view (BEV).
The full function loop looks like this:
[Camera Pixels] ➔ [Backbone Networks] ➔ [Multicam Fusion] ➔ [Autoregressive Transformers] ➔ [Lane Graph]
Once this spatial representation is established, Tesla’s AI stops treating the environment as a simple static image. Instead, it begins to "read" the intersection. The network tokenizes coordinates, translating physical positions on the road into discrete tokens.
The autoregressive blocks within the transformer network select a foundational coordinate, such as the starting point of a lane, and then predict subsequent spatial points (X, Y coordinates) further along the driving path.
The Autoregressive Loop
The true magic of this architecture lies in its iterative execution loop. The system operates via a continuous feedback mechanism:
Initial Prediction: The network identifies an entry token at the mouth of an intersection.
Contextual Feeding: This token is immediately fed back into the input layers of the autoregressive blocks alongside the spatial feature maps.
Sequential Tracking: Based on this prior context, the block predicts the next logical point in the lane, stringing tokens together to form a highly accurate path across the intersection.
Structural Attribute Labeling: Parallel layers simultaneously assign attributes to each coordinate token, classifying whether a specific point represents a standard travel path, a merge vector, a fork deviation, or an intersection interior lacking visible paint lines.
This process repeats inside a loop many times (typically ranging from 64 to 108 inferences per cycle). Once a specific lane is completed and terminated, the loop resets to characterize the adjacent lanes.
By describing lanes as a structured sequence of tokens, the network effectively writes a "sentence" that maps the entire intersection. If a five-way junction splits unpredictably or curves sharply, the transformer calculates the trajectory based on the context of the lanes behind it, ensuring the vehicle never switches path mid-intersection.
Overcoming Amnesia Across Time and Space
The patent explicitly acknowledges that real-world driving is rife with sensory disruptions. Structural lane lines can be degraded, completely covered by construction debris, or temporarily occluded by large lead vehicles like commercial box trucks. To prevent the network from suffering from immediate spatial amnesia, Tesla integrates a specialized video queue module.
This video queue acts as a short-term spatial and temporal memory bank. When a vehicle moves forward, features derived from previous time stamps are retained within the queue. To ensure consistency, the network executes a frame alignment sequence. If the vehicle moves forward 20 meters, the historical feature maps stored in the video queue are mathematically shifted to correct for the vehicle's movement.
This spatial alignment ensures that even if a lane marker disappears beneath a neighboring vehicle's chassis, the autoregressive blocks can cross-reference the historical features to maintain an uninterrupted, accurate prediction of the lane's connectivity.
Maps are a Hint, Not the Truth
For years, the autonomous vehicle industry has argued that safe, driverless navigation is impossible without pre-mapped, centimeter-accurate HD data. Tesla Vision has already proven this to be false with its generalized approach to autonomy quickly spreading across the world, but this specific patent proves it.
While Tesla does feed standard map data into the neural networks as an additional input, it is treated merely as a hint. In fact, the architecture is engineered to inject a "don't know" signal into the network if the localized map data is determined to be unreliable or outdated for a specific geographic zone.
By forcing FSD to rely fundamentally on real-time vision, Tesla ensures that the vehicle behaves like an intelligent human operator. It looks at the layout of the land, processes the context of the intersection, and infers the correct path based on visual reasoning rather than obsolete, cached database listings.
As Tesla moves closer to commercializing its unsupervised ride-hailing network, this patent proves that the computing framework under the hood is no longer just processing images; it is actively reading the world.
If you enjoyed this article, we recommend reading our full series on Tesla patents related to FSD:
