What it does
An MCP server exposing Midscene's vision-driven UI automation for Android devices and emulators. Rather than relying on selectors or accessibility trees—which break on UI changes and fail on custom controls—it works from screenshots alone. It pairs Claude with multimodal AI models (Qwen, Doubao, GLM-4.6V, Gemini, or self-hosted UI-TARS) to localize and interact with UI elements. Claude describes actions in natural language and the server translates them into device interactions: taps, text input, assertions on visual state. Handles both native and cross-platform Android apps.
Who it's for
Android QA engineers and test automation teams moving away from brittle, selector-based testing. Developers integrating AI-driven test flows for native apps. Teams testing applications that DOM-based tools can't reach.
Common use cases
- Automate Android app testing without maintaining selectors across UI refactors
- Test native Android apps and cross-platform applications
- Verify visual properties, colors, and layout on Android screens
- Build AI-driven end-to-end test flows and scenarios
- Automate complex user journeys requiring human-like interaction (form filling, navigation)
Setup pitfalls
- Requires Android device or emulator with
ADBconnectivity and appropriate permissions - Requires API key for a compatible multimodal model; Midscene supports several (Qwen, Doubao, GLM-4.6V, Gemini) but configuration varies by provider
- secrets_found: 1 — repository contains credentials; handle and secure carefully before use
- Model choice affects performance; select models with strong UI localization capability for reliable interactions
- Network access required to reach model APIs; offline or air-gapped environments won't work