The ethnically diverse high-altitude region of Gilgit-Baltistan, with its complex political history, remains relatively free from the controlling gaze of the central state apparatus of Pakistan. In these extraordinary terrain, where local communities rule the region as the "State by proxy", informal gemstone mining provides an important supplement to livelihoods. This paper shows that gemstone mining in Gilgit-Baltistan is characterised by customary rules and regulations that are based on collective responsibility, and members follow customary authorities because they are not external to the community. It argues that the very idea of centralised "governance" of mineral resources, used widely in contexts of resource extraction as the panacea, needs to be reconsidered when dealing with the particularities of the context. It draws on the concept of ungovernability to underline the fact that sometimes it may simply be impossible to administer informal mining in the conventional sense of the term. It suggests in conclusion that informal mining, which has a long history in Pakistan and has played an increasing role as a source of rural employment and revenue, needs to be accommodated within the legal framework of mining. More importantly, to govern informal mining of gemstones, the first task would be to consider how things are currently done, understanding and respecting customary laws, and build upon them to incorporate their elements into systems that acknowledge community rights and a more equitable sharing of benefits.