With the release of Canvas in the Gemini web app, our Android XR team began to explore how we might use it to make immersive computing accessible to developers and users at large. We noticed that Gemini is particularly adept at generating interactive 3D web graphics, and asked a simple question: What if you could experience Gemini’s 3D web creations in extended reality?
To test this, our team collaborated to build an immersive biology simulation. Gemini first modeled the complex interactions between blood cells and rendered them visually using WebGL and Three.js. It looked like a living textbook illustration, but the shift occurred when we converted that content to XR using WebXR APIs. A button appeared on the screen: “Enter XR”. Exploring car-sized blood cells at a sub-cellular scale demonstrated Gemini’s capability to build fully realized, interactive XR experiences.
This led to the development of tools that allow you to build complex 3D prototypes and interactive models significantly faster than traditional methods. Here’s how you can give this a try yourself and turn your ideas into functional 3D experiences with ease.
Get started building 3D experiences with Gemini
While high-fidelity XR development traditionally requires a broad range of skills, Gemini offers a new way to rapidly prototype and explore interactive 3D concepts in real-time. This approach allows you to move quickly from a creative spark to a functional experience you can test immediately. To begin building your own interactive environments, you will need a Samsung Galaxy XR headset and access to Gemini 3 Pro.
- Set up your XR Gem: You can start with the pre-built XR Blocks Gem, or build your own version by downloading the XR Blocks ultra-prompt to give Gemini advanced perception capabilities for realistic AR physics. Upload it to a new Gem in the Gemini web interface and add a description like “you are a creative and resourceful WebXR developer with superb aesthetic taste and technical execution,” and any other information to capture your vision. This persona allows Gemini to interpret your visual scene and use image-generation tools to create polished 3D textures.
- Launch the experience: Open Gemini in Chrome on your Galaxy XR, start a new chat with your XR Gem, and select Canvas. This setup allows you to build interactive components, like a dandelion that reacts to your touch. Try prompting Gemini to embed Gemini Live within the experience. This allows you to talk to Gemini directly, or even continue creating without leaving the immersive session.
Left: Senior XR Engineer – 1 day, Right: Canvas, < 1 min
3. Build and iterate: Think of an idea and ask Gemini to build it. You can say “create a volcanic environment” to change your surroundings instantly, or “add a butterfly there” to place specific objects. You can even give characters backstories and agency within your scene. If the result isn’t perfect, try again or layer simple ideas together to create complexity. When you’re finished, your creation becomes a web link that you can share with anyone.
Screen capture of a user creating a volcanic world and adding dinosaurs.
Ideas to get you building today
We have a few recommended starter prompts if you want to try something simple that you can build on:
- “Make a pen that draws rainbows in 3D”
If you are feeling adventurous, we have more advanced challenges for you to try.
- Create a smart planetarium with real star data, ex: NASA exoplanets, Smithsonian star catalog, European Space Agency’s 3D map of the galaxy.
- Create a clone of your favorite 80s arcade game, and re-skin it to be cat-themed.
- Create a tool that turns your 3D doodles into lifelike sculptures.
Create something great and share your Canvas so other people can try it and build on it as well. Browse XR Blocks demos to get inspired with live demos designed for Android XR. Head over to gemini.google.com to get started today using the steps above!
