--- title: "New ChatGPT AI Watches Through Your Camera, Offers Advice on What It Sees" description: "OpenAI demoed its new ChatGPT AI's ability to adapt to what it \"sees\" through the user's camera, allowing it to help a solve math problem." date: "2024-05-13" modified: "2024-05-13" authors: - name: "Victor Tangermann" job_title: "Senior Editor" link: "https://futurism.com/authors/victor" url: "https://futurism.com/new-chatgpt-ai-camera-video" categories: - "Artificial Intelligence" - "OpenAI" tags: - "chatgpt" - "gpt-4" - "gpt-4o" - "OpenAI" --- # New ChatGPT AI Watches Through Your Camera, Offers Advice on What It Sees ![OpenAI demoed its new ChatGPT AI's ability to adapt to what it "sees" through the user's camera, allowing it to help a solve math problem.]() *\Image: OpenAI\* Today, OpenAI [showed off]() its latest large language model (LLM) called GPT-4o — that's a lower-case "o," for "omni" — that the company promises can "reason across audio, vision, and text in real time." During its brief announcement, the company demonstrated the AI's uncanny ability to assess what it "sees" through the user's smartphone camera, allowing it to help solve math problems and even assist with coding. OpenAI is making the new model "available to all ChatGPT users, including on the free plan" [per OpenAI CEO Sam Altman](). "So far, GPT-4 class models have only been available to people who pay a monthly subscription." It's arguably a natural evolution of the popular AI chatbot; by harnessing a live video stream, the assistant could likely be more helpful by benefiting from far more context. It's also unsurprising, considering we've seen very similar demos by AI hardware companies [Humane]() and [Rabbit](), who both attempted to bring an AI chatbot-based gadget with a built-in camera to market this year — albeit with [catastrophic results](). Given its positioning at the forefront of the tech, however, OpenAI is leveraging the computing power of the modern smartphone instead, and from what we've seen, that approach makes for a far more seamless experience, with barely any delays between a user's question and GPT-4o's answer. https://twitter.com/sama/status/1790068956557881504 OpenAI [claims]() GPT-40 can respond to audio inputs in as little as 232 milliseconds, which is "similar to human response time in a conversation." That's in part because it doesn't have to transcribe the text, and has "all inputs and outputs" be "processed by the same neural network." In other words, OpenAI may have just given both Humane and Rabbit, whose products can take an [eternity to respond]() to the user's inputs, a run for their money. The new model also sounds considerably more natural and "emotional," with a life-like female voice seemingly picking up on tone and emotions of the user in real-time. Put differently, it's a [lot closer]() to Scarlett Johansson's voice in the 2013 sci-fi blockbuster "Her." "I'm all ears," ChatGPT told OpenAI research lead Barret Zoph with a notably cheerful voice during a demo today. "What math problem can I help you tackle today?" The demo, which relied on ChatGPT's new ability to see the world around it, didn't go by without a hitch. "OK, I see it," ChatGPT said after Zoph asked her to help with a calculus problem without revealing the answer right away. "No, I didn't show you yet!" Zoph answered, perplexed. "Whoops, I got too excited," ChatGPT replied sheepishly while Zoph hurriedly wrote out the math problem with a sharpie on a paper in front of him. "I'm ready when you are." Of course, we should take what OpenAI showed off today with a healthy grain of salt. Tech demos are tech demos — and it would be [far from the first time]() we've seen big tech companies fudge the presentation with carefully rehearsed and conveniently curated demonstrations. Late last month, for instance, news broke that the creators of a two-minute video titled "Air Head" — allegedly generated with OpenAI's [new text-to-video AI Sora]() — had augmented the footage with more traditional filmmaking techniques. In short, it remains to be seen how well the new ChatGPT will be able to respond to questions that involve a live smartphone camera feed in the real world, which tends to be far messier than a simple math problem written out in a perfectly lit studio environment. Besides, OpenAI likely hasn't been able to solve far stickier problems, like having its AIs "[hallucinate]()" facts or perpetuate harmful biases. Nonetheless, based on what we've seen today, it's still a considerable step forward for the tech that could make it even more useful than it is today. **More on ChatGPT:** *[Stack Overflow Bans Users for Protesting Against It Selling Their Answers to OpenAI as Training Data]()* ## Author I've been at Futurism since 2017, where my role has evolved to encompass design, writing, and increasingly editing. I've always been fascinated by space exploration and advanced transportation, which I've leaned into by interviewing luminaries in those fields while closely following the dimensions of policy and regulation that allow next-generation projects to succeed -- or, sometimes, to fail. I'm also keenly interested in the effects of generative AI on society, policies, and democratic institutions, as well as clean energy, physics and biology, and the vagaries of tech leadership. My work for Futurism has been cited by publications including Ars Technica, Gizmodo, PC Magazine, Jalopnik, Fox News, and the New York Post. I spent my childhood living in locations including Manila, the Philippines, and Geneva, Switzerland, attended McGill University, and now live in Toronto, Canada. Before Futurism I worked at AskMen and a small photography studio. In my free time, I'm an avid gardener, foodie, and craft beer lover, as well as a maker of artisanal hot pepper sauces. I have a magnificent dog named Freida. ### Author social links [Bluesky]()