The utility of Unmanned Aerial Vehicles (UAVs) for routine pavement distresses inspection has been increasingly recognized due to their efficiency, flexibility, safety, and low-cost automation. However, UAV-acquired high-altitude images present unique challenges for deep learning-based semantic segmentation models, such as minute crack details, blurred boundaries, and high levels of environmental noise. We propose GLoU-MiT, a lightweight segmentation model designed to address the difficulties of UAV-based pavement crack segmentation. Our model integrates a U-shaped Mix Transformer architecture for efficient hierarchical feature extraction, a Global-Local Mamba-Guided Skip Connection for improved feature alignment and computational efficiency, and a Boundary / Semantic Deep Supervision Refinement Module to enhance segmentation precision in complex scenarios. Extensive experiments on UAV-Crack500, CrackSC and Crack500 datasets demonstrate that GLoU-MiT effectively improves segmentation accuracy, particularly in low-contrast and complex background environments, making it a robust solution for UAV-based pavement crack inspection tasks. Furthermore, inference speed and energy consumption evaluations conducted on the Jetson Orin Nano (8 GB) show that our model achieves an excellent balance between accuracy, energy efficiency, and speed. The code will be released at: https://github.com/SHAN-JH/GLoU-MiT。