Despite the importance of rare earth elements in modern technologies, the establishment of rare earth plants often faces strong opposition from the community due to a fear of environmental impacts. This study examines information about actual and perceived pollution produced by a rare earth plant in Malaysia, Lynas Advanced Material Plant (LAMP), and the community's response to it. A survey of 570 respondents reveals that LAMP is perceived to have a high environmental impact in the long run, although its pollution levels are mostly below the limit set by the authorities. About 18% of respondents have undertaken averting behaviours (AB) such as protest, reduction in outdoor activities, and increased health expenditure. The estimated averting expenditure, excluding migration cost, is about 0.06% of household monthly income. Logit estimations show that perception is a significant determinant of AB, while Tobit estimations indicate that young, male, non-Malays and those exposed to poor surrounding air quality, low levels of trust and perceived high levels of environmental impact will incur additional cost. Rare earth companies and the authorities should therefore find ways to communicate with and gain trust from the locals to reduce their fear and to avoid unnecessary protective actions.