Google built a way for any Android phone to sense depth without extra hardware, and it works. We spent two weeks building on it, and found out why nobody does anymore.
What AR is, and what it can do
AR (augmented reality) simply means putting digital things on top of the real world through a camera. You point your phone at a table and a character is standing on it. No headset, no special room, just the phone already in your pocket.
It’s used more than people realise. Furniture apps that let you see a sofa in your own living room before buying it, face filters on Instagram and Snapchat, try-on for glasses and makeup, navigation arrows painted onto the street in Google Maps, and of course games — Pokémon GO is still the one everyone remembers.
For brands it’s one of the few things that still makes people stop and look. A printed card that turns into a 3D character, or a product box that comes alive when you scan it — people show that to each other, which is exactly what a brand wants.
We built a small demo along those lines ourselves, where you build a little city by scanning simple cardboard cutouts and placing other cutouts on top of them. The placement logic isn’t much different from a regular game; it just happens on real cardboard instead of only inside your phone.
Like that demo, most of that AR is fairly flat. The digital part sits on top of the camera image, or on a flat floor the phone has spotted. It doesn’t really know what’s actually in front of it. For that you need depth — the phone understanding how far away every part of the scene is, so a character can walk behind your coffee mug instead of through it.
Google did build exactly that. ARCore, Google’s AR system for Android, has a proper Depth API, and the clever part is that it needs no extra hardware at all. In Google’s own words when they launched it in June 2020, it “uses our depth-from-motion algorithms to generate a depth map with a single RGB camera”, and “depth sensors, such as time-of-flight (ToF) sensors, are not required for the Depth API to work.”
That’s a genuinely impressive piece of engineering, and it’s still supported today — Google’s device list marks brand-new phones like the Pixel 10 and the Galaxy S25 Ultra as supporting the Depth API. So we thought we’d build something real on top of it.
What we tried to build
We spent about two weeks on one idea: a character that walks and climbs over real objects on a real table. Not a scan of the table made earlier — the actual table, live. Slide a real book under a ledge and the character climbs onto it. Move a mug and the path changes.
We searched around quite a bit and couldn’t find anyone who had done it. That’s either a very good sign or a very bad one 😅. This time it turned out to be the bad one.
We built it twice. First in the browser, since a link that just opens is the easiest thing to hand to someone — TypeScript, three.js and the WebXR depth API on Android Chrome. After a couple of tries we were able to have it detect a live moving object (a hand, for example) while moving the phone, and move the virtual box based on that:
What's on screen
Top: the live depth readout while the demo runs.
Middle: a virtual box sitting on a real desk, shifting as a real hand passes near it.
Bottom: the controls for this run — object size, floor mode, and how the box resolves against what the camera thinks is there.
But the web version just wasn’t that accurate at detecting things. Once the browser version ran out of road, we rebuilt it natively in Unity with AR Foundation and ARCore, where you can get the raw depth data the browser never gives you.
Where it fell apart
The trouble started with the floor. Objects being slightly off was fine, but the ground under the character has to be steady, and it never was.
We rewrote that floor detection four times — different fitting methods, different thresholds, different tricks. The error came out at the same number every single time, in every scene.
In Google’s own SDK issue tracker, a Google engineer explains that “the 160x90 depth (and similar resolutions) image is interpolated and stretched to cover the camera image”, and other developers in that thread report the same tiny depth image.
That one line explains the floor. The surface our character had to stand on was being described by a depth image a fraction of the size of the picture we were looking at, then stretched to fit — so no amount of better fitting on our side was going to sharpen something that was never sharp to begin with.
Google’s raw depth documentation is refreshingly honest about the trade too. The raw version “provides depth estimates with higher accuracy, but raw depth images may not include depth estimates for all pixels”, while the everyday version “provides estimated depth for every pixel, but per-pixel depth data may be less accurate due to smoothing and interpolation of depth estimates.” In other words, the nice complete-looking depth map you get by default is partly filled in rather than measured. The same page also notes that textured areas get higher confidence than plain ones “such as a blank wall” — which is exactly the problem when your game is set on a clean tabletop.
There was one more thing we hit that we hadn’t expected at all: the depth quality gets worse while you hold the phone still. It makes sense once you say it out loud — the phone works out distance by comparing camera frames as you move it, so if you stop moving, it stops learning. A small object we could see clearly at the start had half faded away a minute later, and one small sideways step brought it right back. That’s a real problem for us, because holding still is something people do naturally.
Making sure it wasn’t our fault
“You’re probably using it wrong” is always the most likely explanation, and you can’t rule that out from inside your own code. So we stopped building and ran a few checks.
We checked arfoundation-samples and ran them on the same phone. Same resolution, same noise, same everything. Not our code, then. Honestly this should be the first thing anyone does rather than the fifth — it’s one build, and it shows you your ceiling straight away.
We also tried a completely different system — Niantic’s SDK, which uses an AI model to estimate depth instead of comparing frames. On our phone it was a bit better, but not by much. Different approach, same ceiling.
Google also made a demo app for all of this, called ARCore Depth Lab. It’s a proper showcase — snow settling on your furniture, paint splatters, virtual objects hiding behind real ones — and it’s still on the Play Store.
It just doesn’t work anymore. The listing says it was last updated on 11 May 2022, and on most regular phones the store shows “This app is not available for your device” under the install button. An app that hasn’t kept up with Play’s target API requirement “will only be available on devices running Android OS that are the same or lower than your app’s target API level”. Depth Lab’s last build targets Android 12. So anyone on a newer phone simply can’t install Google’s own depth demo.
And the source repository now opens with this:
The public Android app and this repository is no longer under active maintenance, we encourage contributors to look at XR Blocks where we migrated real-time depth-based interaction like Ballpit and Rain to laptops and Android XR devices.
That one sentence explains the whole situation better than any of our measurements. The depth work didn’t stop — it moved off the phone. XR Blocks is Google’s newer framework, and the depth demos they carried across from Depth Lab now run on laptops and Android XR headsets instead.
None of this means the Depth API is switched off. It still ships, it’s still listed as supported on the newest phones, and it still does a decent job for things like putting a virtual sofa behind a real chair.
The future of AR on phones
So, is AR dead on phones? Not at all. Face filters, product previews, printed-card games and everything else that lives on a flat plane all work well today. It’s only the “the phone understands the shape of your room, live and accurately” part that isn’t really there — and that’s a hardware decision made years ago, not something a developer can fix.
There are alternatives, though — not on Android specifically, but on iOS, where the LiDAR camera does capture depth far more accurately. And there’s Android XR (headsets and glasses), which is basically where Google has moved this whole project. Those are still working and improving day by day. It’s just a pain to see this tech missing from our usual phones.
In the end, we still learned a lot about AR and where it’s going. We’re also putting our WebXR depth experiments up as a public repo — not as a library, just a working reference, so the next person doesn’t need two weeks to find the same wall we did.
In future blogs we’ll cover other tech like VR and ML. So stay tuned, and see you in the next one 👋