Considerable research has focused on understanding how upland farmers adjust land-based livelihoods to the influences of agrarian change in Southeast Asia. In the process, an 'upland bias' has emerged where researchers focus narrowly on the uplands as localities with distinct, coherent features, neglecting how families engage place, social relations and ethnicity as they access opportunities in proximate spaces. This paper considers how the Tagbanua - long considered an upland swidden people - have 'stepped back' from swidden agriculture due to declining yields and debt to harvest the lucrative grouper (e.g. Plectropomus leopardus). We show how Tagbanua families on Palawan Island have adjusted swidden as they negotiate social relations, ethnic cleavages and economic barriers to effectively engage the grouper industry. Rather than cast such farmers and fishers as ideal types in place, we argue that how they negotiate social relations creates new livelihood opportunities in varied environments, reinforcing the dynamic, recursive context of agrarian change.