Google's Gemini AI now controls Mac apps to automate multi-step tasks

Google has expanded Gemini's capabilities on macOS to allow it to interact with local files, desktop applications, and web services to complete complex workflows. The feature enables users to automate tasks like form filling, presentation creation, and online listings, with a visible background window showing real-time progress. It is currently rolling out in English for U.S. users ahead of new Apple restrictions on full disk access for AI apps.
Google has rolled out a new "computer use" feature for its Gemini AI app on macOS, enabling the tool to interact with local files, desktop applications, and web services to automate multi-step tasks. This functionality, currently available in English for U.S. users, allows Gemini to perform complex workflows such as form filling, creating presentations, and listing items online. A key aspect of the update is a visible background window that provides a real-time live preview of Gemini’s actions within the user’s chat interface, allowing users to monitor progress and maintain oversight. The feature is being introduced ahead of Apple’s upcoming restrictions on full disk access for AI applications, which are expected to limit how deeply AI tools can integrate with a user’s system. Apple has already previewed these changes in the macOS 27.3 beta, signaling a shift toward tighter security and privacy controls for AI agents.
The development reflects Google’s strategic push to enhance Gemini’s utility on desktop platforms, particularly in response to evolving platform policies. While the company emphasizes user control—allowing individuals to choose which apps Gemini can access—the rollout timing suggests anticipation of regulatory or technical barriers. Google’s move may be seen as a race to deploy advanced automation before Apple’s new safeguards take full effect. The broader context includes growing scrutiny of AI agents’ access to personal data and system resources, with Apple positioning itself as a leader in privacy-focused design. What remains unresolved is how Apple’s restrictions will impact the functionality of AI tools beyond Gemini, and whether other companies will adapt similarly or face reduced capabilities. The long-term implications for AI-powered automation on Macs are still uncertain.
Gemini’s new computer use feature could significantly reshape how users interact with AI assistants, particularly in productivity and creative workflows. By enabling real-time, visible automation across apps and files, it may lower the barrier for non-technical users to leverage AI for complex tasks. However, this convenience comes with potential trade-offs: increased system access raises privacy and security concerns, especially as Apple tightens controls. The feature may prompt other tech companies to accelerate similar integrations before platform restrictions take effect, potentially leading to a race between innovation and regulation. A comparable precedent is the early rollout of browser extensions and automation tools like AppleScript or Shortcuts, which initially offered powerful capabilities before privacy concerns led to tighter sandboxing. The outcome may depend on how users balance efficiency with control. What to watch for next is whether Apple’s restrictions will be enforced uniformly, and how users respond to AI tools that require deeper system access—especially as AI becomes more embedded in daily workflows.