Meta's Muse: best AI agent, worst designer
The speaker tests Meta's AI model on video editing and finds it excels at technical tasks like cutting and timing but significantly underperforms on creative work like design and overlays. They conclude that while the model is strong for coding and agent tasks, it falls short compared to Claude Opus 3.5 for creative applications.
Summary
The speaker evaluates Meta's AI model's capabilities across different task categories, starting with video editing. They observe that the model demonstrates strong technical proficiency in video cutting and achieves nearly perfect timing on cuts, which the speaker notes many models struggle with. However, the model's creative execution is substantially weaker—the overlays and design elements are described as 'really crummy,' and the model also mishandles the audio in unexplained ways.
The speaker expands their assessment beyond video editing, stating there are no complaints regarding the model's performance on agent tasks or coding. However, they emphasize a dramatic decline in performance when the model tackles creative tasks. This triggers a comparison to Claude Opus 3.5, which the speaker has been recently using and regards as superior for creative work.
When examining the specific video output, the speaker critiques design choices: captions are poorly positioned (appearing on the speaker's face), an unwanted bar appears at the top of the video, and overall the design execution is deemed 'ugly.' The speaker indicates this is not formatted for short-form video platforms, suggesting a fundamental misunderstanding of the intended medium. The speaker holds back from a definitive rejection but clearly communicates dissatisfaction with the creative output.
Key Insights
- Meta's model achieves nearly perfect timing on video cuts, a technical challenge where many competing models struggle
- The model's performance sharply deteriorates on creative tasks, particularly video design and overlays, despite strong technical execution
- The speaker considers Claude Opus 3.5 the best model on the market for creative tasks including design, video, and art
- The model fails to format video content appropriately for the medium—creating a design that doesn't suit short-form video standards
- The model demonstrates no issues with agent tasks or coding, indicating its strength is concentrated in logical and technical domains rather than creative ones
Topics
Transcript
[0:00] Video editing is one of those things that I think these models are getting a lot better at. And so I'd like to test how it does. It's pretty good at cutting the cuts, which a lot of models struggle with. So, it gets the timing almost 100% right, but the overlays and the design are just really crummy. And it did something with the audio because the audio is also very strange. So, let's watch it really quickly. So, no complaints on agents or coding. Where it really started to fall off the the cliff are on creative tasks. I think I've just been spoiled lately by Opus 55. I know. [0:31] I know. Your GPT girl here…
Full transcript available for MurmurCast members
Sign Up to AccessMore from How I AI
I Turned a $50 Photo Frame Into an AI Family Dashboard
A creator transformed a $50 Amazon photo frame running Android into a customized AI-powered family dashboard using OpenAI's Codeex and web technologies. The dashboard displays family schedules, photos, NASA images, news, and individual checklists for each family member, with updates automatically posted via an API connected to a grabbot that monitors calendars and emails.
Claude Code edited my TikTok
A creator describes using Claude Code to automatically edit their TikTok videos by analyzing raw footage and verbal instructions. The AI tool uses existing computer tools like FFmpeg, Whisper, and Python to process local files, intelligently cutting segments based on spoken cues and selecting the best takes.
Claude Code for creators: AI editing is here
Brooke Lacey, a 200K+ TikTok creator, demonstrates how she uses Claude Code to automate short-form video editing by leveraging existing computer tools like FFmpeg, Whisper, and Python without uploading files to the cloud. She trains Claude on her TikTok corpus to understand her editing style, facial expressions, and comedic timing, then uses it to generate jump cuts, captions, scripts, and mine comments for new content ideas.
Muse dodged every trap in this billing ticket
A billing ticket resolution task was used to benchmark agentic AI performance, and Muse excelled by successfully navigating multiple challenges including account linking errors, duplicate refund issues, and prompt injection attempts. The agent demonstrated superior performance by investigating thoroughly, fixing the account link, issuing the correct refund, updating the plan, and marking the ticket as resolved.
Meta's Muse is the best AI agent I've tested
The speaker provides their initial impressions of Meta's Muse AI agent, declaring it the best agentic AI they've tested. Muse excelled across four different knowledge work tasks including billing management, customer responses, inbox triage, and research, outperforming alternatives.