We headed straight out to the Sultan Qaboos Grand Mosque, which is open to non-Muslims very limited hours. The mosque was beautiful - one of the most beautiful mosques we've ever seen. Although the exterior is rather austere, once you get inside, the carvings, the chandelier and the carpets are amazing. We were allowed to mostly roam freely around, also checking out the gardens and other buildings. This was a real highlight of Muscat.
Getting a taxi back to the hotel from the mosque could have been very expensive, as the taxis that queue up outside the mosque wanted a very high price. However, walking out to the main road and hailing a cab got us a rate about half of what was offered 5 minutes away, inside the mosque grounds themselves.