In theory, investors choosing to invest only in socially responsible entities restrict their investment universe and should thus be penalized in a mean-variance framework. When computed, this penalty is usually viewed as valid for all socially responsible investors. This paper shows however that the additional cost for responsible investing depends essentially on the investors’ risk aversion. Social ratings are introduced in mean-variance optimization through linear constraints to explore the implications of considering a social responsibility (SR) threshold in the traditional Markowitz (1952) portfolio selection setting. We consider optimal portfolios both with and without a risk-free asset. The SR-efficient frontier may take four different forms depending on the level of the SR threshold: a) identical to the non-SR frontier (i.e. no cost), b) only the left portion is penalized (i.e. a cost for high-risk-aversion investors only), c) only the right portion is penalized (i.e. a cost for low-risk aversion investors only) and d) the whole frontier is penalized (i.e. a positive cost for all the investors). By precisely delineating under which circumstances SRI is costly, those results help elucidate the apparent contradiction found in the literature about whether or not SRI harms diversification.