Inside the world of ESG ratings: Academic paper looks at multiple issues with process and outcomes

‘Recently, ESG ratings providers have come under scrutiny over concerns about the reliability of their assessments,’ write Stanford University academics David Larcker and Brian Tayan in a paper written with Edward Watts of the Yale School of Management and Lukasz Pomorski of AQR Capital Management.

In their publication – a ‘closer look’ at numerous studies around ESG ratings – they examine these concerns, though naming the paper ESG ratings: A compass without direction provides a pretty clear view of their findings.

‘We find that while ESG ratings providers may convey important insights into the non-financial impact of companies, significant shortcomings exist in their objectives, methodologies and incentives, which detract from the informativeness of their assessments,’ write the authors.

‘Unfortunately, it is rare for ratings providers to offer concrete, systematic evidence to back up claims about their ratings.’

The researchers say they were seeking answers to the following:

• Why do ESG ratings often fail to meet their stated objectives? Is it due to methodological choices these firms make, or the sheer challenge of measuring a concept as broad and all-encompassing as ESG?
• Are fund managers properly motivated to ensure ESG ratings are reliable in predicting risk and performance? What steps do they take to validate ratings before using them?
• Despite their weaknesses, do ESG ratings have a role to play in reporting and compliance purposes?
• Would more expansive corporate disclosure improve the quality of ESG ratings? Is it even possible for companies to disclosure the vast number of stakeholder-related metrics that feed into ESG ratings?
• Should the SEC establish policies, procedures and protections to reduce conflicts of interest and improve market confidence in the quality of ESG ratings?

The 16-page paper is a comprehensive look at past research highlighting numerous failings around the ESG ratings process, from investor understanding of what the ratings actually mean to conflicts of interest that see ‘sister firms’ of a provider rated higher. Here, IR Magazine pulls out a selection of findings.

The large-cap effect

The paper authors write that ‘systemic patterns are observed in ESG ratings’ and one such pattern is related to company size: large companies receive higher average ratings than smaller companies.

‘This might be due to the more significant resources large firms are able to invest in ESG initiatives, or it might be due to the fact that large companies have greater disclosure of ESG data,’ write the authors. ‘A second pattern is industry-related: while some ESG ratings are industry-adjusted, those that are not may have higher average scores for certain industries (such as banks and wireless communications) than for others (such as tobacco and gaming).

‘It is not clear [whether] these patterns are due to fundamental differences in ESG quality across industries, or a result of the methodological choices and input variables that underpin ESG ratings models.’

Geography also has an impact. The researchers find that European companies have higher average ESG scores than US companies, which they say could be due to political and regulatory differences across countries. ‘Firms in emerging markets also have lower ratings than firms in more developed economies,’ they add.

An upward drift

Research demonstrates an ‘upward drift’ in ESG ratings over time. One study cited by the researchers, which analyzes the aggregate ESG scores for all Russell 1000 companies as calculated by MSCI between January 2015 and December 2021, finds an 18 percent aggregate improvement over the measurement period.

A number of factors – such as changes in index composition as higher-rated firms (the researchers use Microsoft as an example) grow to represent a larger percentage of the total index, changes to MSCI weightings or increased company disclosure regardless of whether a company’s underlying performance has actually changed – could all account for some part of this upward drift.

Adjusting for these structural changes, however, MSCI ratings were still found to be subject to an aggregate 12 percent adjusted improvement that could not be explained.

A separate review of MSCI ratings conducted by Bloomberg finds that most upgrades occur for what Bloomberg calls ‘rudimentary business practices’ rather than substantive improvements, note the paper authors. In justifying 155 upgrades, MSCI cites governance improvements almost half (42 percent) of the time – significantly more than social (32 percent) or environmental (26 percent) improvements.

‘Upgrades were often driven by check-the-box practices, such as conducting an employee survey that might reduce turnover, and rarely for substantial practices, such as an actual reduction in carbon emissions,’ note the researchers. ‘Half of companies were upgraded for doing nothing’, simply as ‘the result of methodological changes.’

ESG ratings providers don’t agree

‘Studies find low correlations across ESG ratings providers,’ write the paper authors, adding that ‘this is perhaps surprising if ESG ratings are supposed to measure the same construct.’

Looking at a number of research papers that have examined the issue of low correlations, the paper authors note:

• Differences in measurement (56 percent) and scope (38 percent) account for most of the divergence, with weighting differences accounting for just 6 percent of the variance. ‘This illustrates how fundamental the methodological differences are across firms,’ the authors write
• Corporate disclosure does not reduce the divergence of ESG ratings but instead increases it – something that is put down to the subjective nature of ESG information and the fact that disclosure expands opportunities for different interpretations of information. This suggests that greater corporate disclosure requirements of environmental and social data might not lead to more consistent ESG ratings, say the paper authors. In fact, ESG ratings might be similar to equity analyst ratings, where the rating is ultimately dependent on the interpretation of information rather than its availability.

Ratings do not predict returns – or even ESG performance

What about the purpose of ESG ratings? Whether designed to show investors the impact the world will have on company performance, the impact of the company on the world or to be used as an indicator of future financial performance for those that believe better ESG practices equate to better returns, the paper authors uncover a number of studies pointing to flaws in the ratings process:

• One study finds that companies in ESG portfolios (those with high Sustainalytics ratings) have worse records for compliance with labor and environmental laws than companies in non-ESG portfolios. Companies added to ESG portfolios also do not subsequently improve compliance with labor or environmental regulations
• Another finds that US firms that join the UN Principles for Responsible Investment, a commitment to incorporating ESG factors into their decision-making processes, earn worse ESG ratings (as assigned by MSCI, Refinitiv and Sustainalytics) than US firms that do not make this commitment
• Yet another study, looking at the relation between fund sustainability and performance (using Sustainability fund ratings) finds that funds with low sustainability ratings perform better than those with high ratings
• Elsewhere it is found that companies with high ESG ratings from MSCI perform better during good economic times but worse during bad economic times
• In a ‘substantial literature review of more than 1,100 primary peer-reviewed papers and 27 meta-analyses on ESG and sustainable investing published between 2015 and 2020’, other researchers conclude that ‘the financial performance of ESG investing has on average been indistinguishable from conventional investing’.

The writers of ‘A compass with no direction’ also consider the possibility that, while the ratings published by any single ratings provider are not predictive of performance, the assessments of multiple providers might be informative when considered in aggregate. They look at a study that does in fact find a stronger relationship between ESG and performance by combining the ratings of multiple providers to reduce the ‘noise’ from conflicting assessments.