The public economist: learning from our Citizens’ Panels about the UK economy

Overview

The Bank of England, like other central banks, has traditionally been better at talking to the public than listening to it. However, communication is a two-way street, and our Citizens’ Panels and the survey discussed in this article show how communication can become more meaningful for both us and the public we serve.

Since 2018, the Bank of England has hosted Citizens’ Panels across the UK. These provide a platform for broad engagement with the public. Listening to our panellists helps us to understand how major events, such as Covid, affect people financially, and how they think the future will unfold.

Citizens’ Panels play an important part in the Bank of England’s engagement with the public. Over the past year, they have provided valuable insights into the day-to-day lives of people during one of the most challenging periods in recent times. These human stories complement and inform the Bank of England’s economic analysis, intelligence gathering by the Agents, and the policymaking process. Citizens’ Panels also give members of the public a chance to learn more about how the Bank maintains monetary policy and financial stability in the UK. It is crucial that transparent and open dialogue continues between the public and the Bank.^{footnote [2]}

As well as hosting panel discussions, we have also trialled a survey of our panel members to gain more insights into their views on the health of the economy and their lived experience of it. Our Citizens’ Panel survey and forecasting competition ran between September 2019 and August 2020. Participants had to evaluate the economic health of their community and their personal financial situation (both the present situation and prospects going forward were separately considered), as well as identify possible threats to the economy. Participants were also invited to predict key macroeconomic indicators, such as consumer price inflation and changes in unemployment, as part of a forecasting competition.

In this article, we explore the results of this survey and forecasting exercise. We consider differences by region, how the Covid pandemic affected people’s views and what concerned them most. We find that panellists provided real-time and diverse information, much of which was mirrored in economists’ questions around reactions of unemployment and inflation when the pandemic hit.

A two-way channel of central bank communication

Nowadays, communication is a key tool for achieving central banks’ missions, such as those of the Bank of England. Central banks, by explaining present economic conditions, their decisions and willingness to act, facilitate more effective transmission of their policy by fostering trust with markets and the public they ultimately serve. For example, former European Central Bank president Mario Draghi’s ‘Whatever it takes’ statement in 2012^{footnote [3]} is widely credited as a decisive moment during the European sovereign debt crisis. The statement increased trust in the stability of euro and alleviated pressure on troubled governments’ financing costs.

However, communication was not always so central to central bankers. Montagu Norman, Governor of the Bank of England 1920–44, reportedly said ‘never apologise, never explain.’ Half a century later, in 1987, the then chairman of the Federal Reserve Board Alan Greenspan argued ‘Since I’ve become a central banker, I’ve learned to mumble with great incoherence. If I seem unduly clear to you, you must have misunderstood what I said.’^{footnote [4]}

Towards the end of the 20th century, as central banks gained their independence and many of them adopted explicit inflation targets, their approach to communication shifted. The Bank of England was given operational independence to conduct monetary policy in 1997, and became accountable for maintaining price stability with an inflation target of 2%.^{footnote [5]} Communication is key to achieving this goal, eg to ‘anchor inflation expectations’ at target.^{footnote [6]} Communication has mostly been one-way despite an increased number of publications to a broader base of stakeholders.^{footnote [7]} However, as well as explaining our actions it is important that we also listen to the views and concerns of a wide set of audiences, not least those of the public we serve.

Our Citizens’ Panels, a platform for public engagement launched in late 2018, play a central role in developing this two-way communication with the public. The Citizens’ Panels have enabled us to engage in a constructive dialogue with the public about the economy, the financial system and our work.^{footnote [8]} This complements other sources like our Agency network^{footnote [9]} or survey of businesses’ decision-makers.^{footnote [10]}

In the summer of 2019, we started an online community to bring the discussion from our Citizens’ Panels to an even wider audience. We launched a simple form of social network to host discussions and events. Unintentionally, this inoculated much of the Panels’ activities against the impact of the pandemic, allowing us to collect useful real-time information during this very difficult time. Our Citizens’ Panel events ran virtually in 2020 due to Covid-related restrictions.

Between September 2019 and August 2020, we ran a monthly survey within the Citizens’ Panels online community. The goal of this survey was to allow people to engage with us, contribute to discussions about the UK economy, and for us to assess the information we collected alongside other more traditional sources. We asked members for their views on local economic conditions and personal financial situations, both at that point in time and looking to the future.

Our survey also included the first public forecasting competition run by a central bank, which we saw as both novel and important. Panellists assessed recent changes in regional unemployment and consumer price inflation, and considered how those may evolve in the following six months. The competition was between panellists, with points given for participation and for accurate predictions.^{footnote [11]}

Research has shown that the aggregate opinion of engaged non-professionals can be useful in addressing complex problems with uncertain outcomes.^{footnote [12]} Predicting macroeconomic developments arguably classifies as such a problem. While individual predictions will be noisy, the principle of the ‘wisdom of the crowd’ suggests that their average opinion may be accurate.^{footnote [13]}

The outbreak of the Covid pandemic and the subsequent lockdown in March 2020 can be seen as a natural experiment within the survey setting. Consequently, much of the content of this article will focus on this event. We will see that the diverse voices of panellists offer a rich source of information with potentially helpful insights alongside conventional economic analysis. Early insights from our survey informed the May 2020 Monetary Policy Report, providing timely assistance to our work at the start of the pandemic.

In this article, we consider survey responses across time and demographics, and on the impact of the pandemic. Then we present the forecasting competition and assesses how its outcomes may be useful alongside conventional analysis. We conclude with a short discussion on the merits and challenges of collective forecasts and how this sort of material could be used alongside conventional economic analysis in the future.

We would also like to let some of our participants speak for themselves. Therefore, please click below for quotes from participating panellists, and information on how they approached the questions and tasks put in front of them in the survey and forecasting competition.

• View quotes from our participating panellists

  Julie Donaghey (South West): I have seen how a curve ball event can cause you to lose the details amid the headlines. A chance illness left me with a disability that cost me my job, my identity and my role in the economy. My perspective changed and as such my interaction with the world around me also changed. One is often limited by their own perspective, so this challenge to harness a variety of perspectives, enables a universal approach to understanding. My challenge approach was informed by my local council’s goals for grant funding for any big-ticket items: a ‘get the money, worry about how you’ll spend it afterwards’ approach that has generated positivity despite a beleaguered nationwide outlook.

  David Jones (Scotland): I am 71 years old and a retired maths teacher who became a secondary head teacher in Sunderland before moving to Scotland as a Director of Education, Social Services and Housing. I finished my working life as Chief Executive of a Scottish council. I thought I could put a little of my experience into being involved in looking at the crucial issues facing the country in this time of global pandemic and Brexit. My views, expressed in my predictions, come from taking a very close interest in the affairs of the UK and Scottish governments and in the local situation affecting the area in which I live.

  Kel Nwanuforo (East Midlands): I am 30 and work as an investment consultant in Leicestershire. I took part in the Citizens’ Panel forecasting competition as I have always taken a keen interest in economic developments and the very real impacts that they have on people’s lives. In approaching the challenge, I felt that it was important to place both recent and possible future events into a broader context. In particular, I sought to consider the continuing effects of those longer-term forces, which have in recent years been driving structural change in the UK economy.

  Andrew Nevill (East Midlands): I am an engineer working in electronics and software but with a keen interest in economics, statistics, money (how it works) and natural justice. I’ve spent most of the past 20 years working for start-ups and in the semiconductor industry. I took part because I wanted to measure my understanding of economics against others. My approach involved extrapolating recent interest rate history combined with what I expected of key indicators such as fuel prices and then taking into account the unprecedented effect of Covid at the end of the measured period.

  Andrew Mills (North West): I have a small business and have always had a keen interest in the economy, especially regarding inflation, the base rate, and employment statistics. I enjoyed the Citizen’s Panel and wanted to be involved with the forecasting competition. I looked holistically at the economy and our society, historically and in the present, and tried to predict what would happen with spending and how this would affect inflation and the base rate.

  John Piddock (South East): I am qualified as a physicist and originally worked in the energy sector. Later, I moved into the health sector and qualified as a Chartered Company Secretary. I retired from the NHS as a Director of Corporate Affairs. Being interested in the economy and current affairs, I joined the Bank of England Citizens’ Panel in Kent. The forecasting competition presented a further opportunity to contribute. I approached the challenge using my previous experience of periods of different inflation and interest rates, recognising the impact these have on the economy. I adopted a risk aware/averse approach, framing my forecast around what I felt would happen in our local community. Living close to the Channel Tunnel, I considered Brexit would impact strongly on local employment and trading with Europe; Covid added a further layer of uncertainty.

  Sharon Raj (London): I am 48, married with two teenage children and I live in South West London. I am a part-time mature student studying behavioural science and I also volunteer for a few local charities. When I first left university I worked as an emerging markets economist, but over the years I shifted into operational roles, so I have not worked as an economist for more than 15 years. I took part in the forecasting competition to challenge myself and see how much of my former training I could remember!

  Tom McDonnel (West Midlands): I have been a member of the Citizens’ Panel since its formation, driven, as was the case with the forecasting challenge, by my lifelong interest in the mechanics of the UK economy. A career spent in commercial banking and the not-for-profit sector gave me significant skills in risk assessment, business operations and analysis and I deployed these together with a longstanding passion for history and current affairs in approaching the challenge.

  Bowen Song (London): I am an immigrant who first arrived in the UK in 2014. I dropped out of school when I was 16 years old, and self-taught computer programming. As I was looking into investing, I started to read and learn more about related subjects, such as the economy and the role of central banks. I joined the Bank of England Citizens’ Panel London event in 2019 by chance and I have completed the online surveys since then. To come up with the numbers for the forecasting competition, I extrapolated relevant data available online, such as the data published by the ONS and Bank of England, and then adjusted the forecasted numbers based on the sentiments about them from various sources, such as news articles, online forums, the forecasts produced by the OBR and Bank and some other sources. If the sentiments disagree with the extrapolated numbers by a large degree, I tend to error on side of the extrapolated numbers. I doubt the method I used is any superior than a sophisticated mathematical model.

  Harry Cross (North East): I am a PhD student at Durham University. I took part in the challenge to learn more about the Bank of England and how the Bank and the government respond to changes in the UK economy. I have been concerned about the distributional impacts of Covid and lockdowns, both nationally and in the North East. This informed my responses to the challenge.

Our survey results: through the public’s eyes

The survey sample

Our respondents reflect the UK population well by region and income, but less so by age.

A total of 487 people, more than half of registered panellists at the time, participated in the survey between September 2019 and August 2020. By contrast, the earlier months in 2019 had only a modest number of responses reflecting the gradual growth of the online community. Our analysis reflects this pattern by mostly focusing on responses in 2020, and particularly on the impact of the first lockdown.

While participation in Citizens’ Panels events is actively controlled to, as far as possible, represent the UK population by region and demographic characteristics, the online community and survey are open to everyone interested.

As such, the first step of our analysis is to determine whether our survey sample is representative of the UK population. Chart 1 shows a comparison of our survey sample with the UK population by region, income and age. We see that our survey sample represents the UK population reasonably well by region (all within 3 percentage points of their population shares) and income (all within 5 percentage points of their population shares, though lower income households are slightly under-represented), but is older than the overall population. In particular, younger people below age 25 are underrepresented in our sample.^{footnote [14]}

Chart 1: The Citizens’ Panel survey sample is fairly representative of the UK population by region and income, less so by age

The representation of demographics in the UK population and among the Citizens’ Panel survey sample

Footnotes

• Sources: Bank of England Citizens’ Panel Survey and Office for National Statistics (ONS).

Next we look at the responses and what we can learn from them.

Feelings about the present and future

People are on average content with the present situation, but are gloomy about the future.

The results of our survey from September 2019 to February 2020 (ie the pre-Covid phase of our survey) pointed to a public who were largely positive when asked about present conditions, but less optimistic about the future.

Across the board, whether looking at participants’ personal financial conditions, their perception of the economic health of their local community, which may be one’s neighbourhood, council, town, city or region (the interpretation was left to participants), or their views on changes in unemployment or prices, people were generally positive when asked about present conditions.^{footnote [15]}

For example, respectively 36% and 52% of respondents said that they were rather positive about their present economic community and personal financial conditions between September 2019 and February 2020. That compares with only 31% and 24% of respondents saying that they were negative over the same period.

However, respondents were less positive in their outlook for the future. Only respectively 23% and 32% of respondents thought that their economic community or personal financial conditions would be positive in six months’ time, compared to 47% and 25% of respondents who thought they would be negative. These results are summarised in Table A, which shows the net balances between positive and negative responses for the present and future questions. The difference between these balances is substantially negative in both cases indicating that respondents had a generally pessimistic outlook.

The impact of the pandemic on expectations

The pandemic both shifted and polarised expectations about unemployment and inflation.

When looking at the expectations for future unemployment, defined as expected changes in the regional unemployment rate over the next six months, respondents were much gloomier from March 2020 onwards. The distribution of responses for this question and each survey round is shown in the upper panel of Chart 2. Around half of respondents expected a rise in unemployment before the pandemic hit. With the start of the first lockdown in March 2020, around 70% expected a large increase in unemployment over the coming six months. This response mirrored well the Covid-19 Stringency Index for the UK, which tracks the state of restrictions, eg school closures and travel bans.

Chart 2: Covid shifted and polarised future unemployment and inflation expectations

Share of respondents by future unemployment outlook compared with the Covid Stringency Index

Share of respondents by future inflation outlook compared with the price dispersion among items of the consumer prices index (CPI)

Footnotes

• Sources: Bank of England Citizens’ Panel Survey, Covid-19 Government Response Tracker (Blavatnik School of Government, University of Oxford) and ONS.

An interesting observation, apart from the large, and partly expected, shift to a much more negative outlook, is the increased polarisation. Contrary to the pre-lockdown period, very few had an ‘about the same’ view on future unemployment. An increased fraction even thought that unemployment may strongly decrease after lockdown. This suggests that some people saw opportunities for either themselves or certain businesses. This is what we observed. While for most sectors, Covid’s impact was negative, some businesses – for example, the providers of video conferencing software, home delivery companies or streaming services – were positively affected by changing patterns of work and consumption.

We made a similar observation of an increased dispersion of views for future prices, or inflation expectations, when asking about what consumer price inflation would be in six months. Inflation expectations are a key indicator for macroeconomists and those employed at central banks in particular. This dispersion is shown in the lower panel of Chart 2. We see more extremes at both ends of the spectrum after the first lockdown in March 2020.

Some respondents even saw the possibility of falling prices, a rare event in the UK’s recent economic history. Consumer price inflation has only briefly dipped below zero in the past 30 years, falling to minus 0.1% year on year for three months in 2015. However, such views were not outlandish, as inflation actually did come down quite a bit in the second half of 2020 and early 2021.

More importantly, economists everywhere, including at the Bank of England, struggled with the assessment of the impact of the pandemic, because it represented a hit to both demand (desired spending by consumers and businesses) and supply (the ability to produce goods and services) at the same time. On the one hand, consumers stayed away from certain activities to protect themselves from Covid. On the other hand, many businesses were not allowed to operate due to the imposed restrictions. It was not at all clear what this would actually mean for aggregate prices.

The consumption basket that is used to assess changes in overall consumer prices is made up of hundreds of items, such as bread, haircuts and insurance. The prices of many of those items fluctuated markedly more following the first lockdown. This is shown by the item price dispersion in the lower panel of Chart 2.^{footnote [16]}

This means that information provided by our Citizens’ Panel members reflected actual economic developments, and also reflected the same uncertainties that professional economists were wrestling with at the time. This finding indicates that the responses obtained from our Citizens’ Panels could be useful for economic forecasting – a point we will revisit later.

The geographic impact of the pandemic

The pandemic levelled perceptions of economic health between the north and the south of the UK.

One of the major motivations behind the Citizens’ Panels is to reach out and listen to people across the whole of the UK. People in different parts of the country may have different views, or may evaluate events differently, depending on their particular circumstances. Panellists identified themselves as being a resident of one of the areas covered by our 12 Agencies, which cover nine regions of England and the three devolved nations.^{footnote [17]}

This allowed us to break results down geographically, as shown in Figure 1 for the perception of local economic conditions. The left-hand side shows the average level within each region in the first half of the survey (September 2019 to February 2020) with negative and positive values representing negative and positive views, respectively.^{footnote [18]}

We observed a clear north-south divide. Those in the south, particularly London, were more positive than those in the north. With the exception of Northern Ireland and Scotland, these results roughly reflect differences in economic conditions across the country as measured by income or economic output.^{footnote [19]} This again demonstrates how the Citizens’ Panels can help us to bridge the gap between personal experience and macroeconomic outcomes.

Next, we consider the pandemic’s impact on the perceptions of local economic conditions across the country. This is shown on the right-hand side of Figure 1, which shows differences of views on average local conditions between the first and second half of our survey, with the sample split being the imposition of the first lockdown in March 2020.^{footnote [20]}

Figure 1: Economic perceptions in southern UK regions were more negatively impacted by Covid

Footnotes

• Notes: Left-hand side – perception of local economic conditions by region September 2019 to February 2020. Right-hand side – The change in perception of local economic conditions by region between September 2019 to February 2020 and March 2020 to August 2020.
• Source: Bank of England Citizens’ Panel Survey.

We see a general reversal on pre-lockdown perceptions. The previously more ‘upbeat’ south judged the impact of the pandemic larger than the north. Overall, the pandemic seems to have acted as a ‘great leveller’, at least in perceptions.^{footnote [21]} Scores on local economic conditions where closer to zero (neutral view) across most of the country in the second half of our survey period. Notable exceptions to this are the North of England and Scotland, where perceptions remained rather negative, while they somewhat improved in the latter.

It is hard to pin down how to interpret these findings, as the impact of the pandemic was felt differently across the regions depending on the precise concern, eg worries about work, health and wellbeing, access to care and essential shopping.^{footnote [22]} However, these findings provide us with clues about where to look if we want to understand the impact of the pandemic on people’s lives.

Horizon scanning

Panellists saw the pandemic coming early.

An innovative feature of our Citizens’ Panel survey was a free-text field where participants could state and describe worries that they had about the economic health of their community. Such text data is a so-called unstructured data source, meaning it does not come in the common form of a pre-defined table of columns and rows. Instead, it comes in letters, words and punctuation making up sentences and paragraphs.

This format poses challenges (the lack of format, and complexity of the raw information) and opportunities (the richness of information). One key advantage is that this type of data can be used to analyse hard to pin down concepts like topics and sentiment. These would be more difficult to analyse with conventional data sources as specific survey questions would have to be very precise and plentiful, making them impractical.

The richness of this raw data is show in Figure 2 in the form of a word cloud. It shows non-fill words, ie those carrying some meaning on their own, with more frequent words placed more centrally and bigger. From the word cloud, we see that our participants had three main concerns: Brexit, Covid and jobs. The shares of responses containing words associated with the three top topics are shown in Chart 3.^{footnote [23]} Initially concerns about leaving the European Union dominated, but later in our survey period, concerns about Covid and its impact on jobs became more prevalent.

Figure 2: Brexit, Covid and jobs were the main topics of concern panellists had of the economy

Footnotes

• Source: Bank of England Citizens’ Panel Survey.

Chart 3: Brexit, the pandemic and jobs concerned participants the most across time

The share of responses citing Brexit, Covid and jobs as concerns over time

Footnotes

• Source: Bank of England Citizens’ Panel Survey.

A wholly unanticipated topic like the pandemic can be used to evaluate the usefulness of our free-text ‘concerns box’ for horizon scanning, which was a key motivation for including this question in the survey.

We see in Chart 3 that the spread of Covid and the subsequent pandemic was well documented as a major cause of concern in our February round (which concluded in mid-March), and had become the dominant topic by that point. Looking at weekly shares of the topics from Chart 3 within survey rounds as responses arrived, Covid clearly dominated people’s concerns in the week before the first lockdown, and overtook Brexit in the 2–3 weeks before lockdown, when both topics featured in slightly below 40% of responses.

This means collecting detailed real-time information from an engaged public can help to identify potentially fast-moving threads early on.^{footnote [24]} At the same time, modern data analytics techniques – such as natural language processing – can be used to analyse such data efficiently.

The forecasting competition

We test an innovative way to harness public knowledge to better understand the UK economy.

Another novel component of our survey was a forecasting competition to predict macroeconomic outcomes, particularly changes in unemployment and consumer prices. These are hard to predict, as they depend on a myriad of factors, including the behaviour of customers and businesses, global markets and international developments, such as political conflict or the spread of the pandemic. As a result, trusted and accurate official statistics for macroeconomic indicators are often only available with several months delay, and some may even be revised multiple times affecting measured outcomes over years. At the same time, economic policy makers face a daunting task, as they need to rely on current assessments of economic conditions and estimates of what they will be in the future.

Given that macroeconomic outcomes ultimately are the result of everybody’s actions in the economy, there is some logic in asking individuals who make up the economy about what macroeconomic indicators are and will be in the future.

Our panellists were asked to predict recent changes in unemployment and consumer prices, and what those changes will be in six months’ time. Employment conditions often are local and participants will likely be more knowledgeable about local conditions than national ones. We therefore asked for changes in the unemployment rate within a participant’s region.

Prices can change on a local, regional and national level, and people’s consumption baskets mix those. This makes it harder to differentiate between regional and national factors consistently. We therefore asked people to predict changes in the consumer prices index, a national indicator.

The assessments of individuals – based on a large variety of sources, different mental processes and scattered across the country – likely come with a considerable level of noise. However, averaging such predictions forms the idea behind the ‘wisdom of the crowd’: noise will cancel out when combining signals from many and largely unrelated sources each carrying some piece of relevant information. This can provide us with viable and possibly highly accurate predictions. Our analysis will focus on what we can learn from these aggregate views of our participants.

Individual performance

High-hitting participants follow the news closely and have a keen interest in economics.

Chart 4 shows ‘hit-rates’: shares of correct responses among participants as a function of the number of surveys someone participated in.^{footnote [25]} Each dot is the average hit rate of participants who participate that number of months in the survey.

The hit-rate of all participants (red line in Chart 4) was above the chance rate (horizontally dashed line), ie the hit-rate obtained by picking one response out of five at random (0.2). This means that, on average, participants’ responses provided information. However, the average rate often was not much above the chance rate, pointing to the difficulty of the task.

Chart 4: Tracking responses across time helped to separate skill from luck

The share of correct predictions about the economy made by Citizens’ Panels and the Bank of England

Footnotes

• Sources: Bank of England Citizens’ Panel Survey and ONS.

The average hit-rate increased slightly with the number of surveys people took part in. This can be interpreted as an engagement effect. People who were more interested and put more effort into providing answers were more likely to answer correctly.

The highest hit-rate achieved by one or more participants (light pink line) rapidly decreased with the number of surveys panellists participated in, and roughly levelled off after about three rounds. This is a so-called ‘reversion to the mean’ effect. If we had many participants only taking part in a single round or two rounds, it is highly likely that we would see high hit-rates just by chance. This is akin to the observation that people regularly do win in lotteries despite the low chances of winning, but people do not do so regularly.

It requires skill, or extremely good luck, to maintain a high hit-rate. The level of this skill is represented by the hit-rate achieved over many survey rounds, as pure luck cannot sustain a high rate for long. We use this observation to select a subset of ‘high-hitters’ in the upper right quadrant demarcated by the cross of dashed lines in Chart 4.^{footnote [26]} These were 16 participants who achieved a hit rate of at least 40% (double the chance rate) and participated in at least four surveys.^{footnote [27]} While being a small group compared to all participants, we will see that high-hitters made some predictions that were qualitatively different from other participants.

It is interesting to look at the demographics of our high-hitting group and compare it to our other participants. About 70% of high-hitters have studied economics at some point and they also keep up with the news on a daily basis. While the high-hitters group is too small to draw specific conclusions, it may be fair to say that they show a keen interest in economics, central banking and overall developments beyond these fields, which helps to connect the dots. They also have diverse backgrounds, which is important when aggregating responses as this avoids ‘groupthink’.

The question arises what the hit rate of the Bank of England has been over the same period. This is referenced by the yellow horizontal line in Chart 2. We see that this surpassed even the best individual Citizens’ forecasters when comparing the survey results to similar internal forecasts.^{footnote [28]} This comparison is of course not fair, because our analysis is a collective effort of a group of multiple professional economists who closely work together with decision-making committees. This means that Bank of England predictions are more akin to collective predictions from the Citizens’ Panels. It also is not the goal of this analysis, because a more interesting question is: to what extent does the Citizens’ forecasts add value to our own forecasts?

Collective predictions

Citizens’ Panel forecasts can offer a second opinion, especially during difficult and uncertain times like during the Covid pandemic when the collection of contemporary and reliable data was challenging.

Chart 5 shows aggregate Citizens’ Panel predictions for our four macroeconomic forecasting questions: changes in inflation and unemployment in the present and the future (six months from now). We differentiate between all participants (red) and the subgroup with a high hit-rate (light pink line) defined in Chart 4. Aggregate Citizens’ Panel predictions are formed by averaging the midpoints of the predefined buckets from the survey across all responses. Regional unemployment predictions are converted to national ones via population-weighted averages. The reference measures published by the ONS, against which predictions are evaluated, are shown in blue. The corresponding Bank of England predictions are shown in yellow.^{footnote [29]} We make a couple of key observations.

First, the overall Citizens’ Panels predictions (red line, Chart 5) tracked employment measures closer than inflation. Inflation expectations were generally somewhat upwards biased with more than half of respondents thinking that prices would rise above the Bank of England’s target in the six months after the survey. This observation is in line with previous findings of persistent overestimation of actual and future price rises in surveys.^{footnote [30]} A possible explanation for the discrepancy in performance between measures is that changes in the unemployment rate were small compared to those of consumer prices throughout the survey period. That is, it was potentially ‘easier’ to make statements about unemployment than prices.

Chart 5: Citizen Panel predictions tracked official data often well

Citizen’s Panels and Bank of England predictions about economic indicators compared with the official ONS data

Footnotes

• Sources: Bank of England Citizens’ Panel Survey and ONS.

Second, the observed bias is much smaller for our top hitters (light pink line), where inflation predictions are generally closer to realised values, often as close as our own. This suggests that the two groups of participants made qualitatively different assessments, and that those may offer value alongside our own analyses. A point we will revisit in more detail below.

Third, absolute prediction errors, as a measure for forecast performance, were mostly lower for the Bank of England forecasts across time, especially for inflation. However, the error for future unemployment was lower for Citizens’ Panel forecasts. This result is driven by performance during the second pandemic half of our sample period, a period where there were questions about measurement within the labour market.^{footnote [31]} At the same time, panellists could not make extreme forecasts meaning that large divergences from target are potentially supressed when the first lockdown was imposed in the UK.

As we mentioned before, the two sets of predictions are not directly comparable and there are other important measures of performance of complementarity, which we will evaluate next.

Information content

Joint forecasts from the Bank of England and Citizens’ Panels can perform strongly in terms of both accuracy and signal strength.

Comparing errors alone is a crude way to evaluate forecast performance. While low errors are preferred to high ones, another equally important measure is signal value. That is, how reliable a prediction is as an indication about where a targeted quantity is moving.

A simple and principled way to do this is to use linear regression models. We say that a source carries information content if the corresponding regression coefficient can be said to be different from zero with a certain level of confidence.^{footnote [32]}

With this in hand, regression analyses can also be used to compare the signal strengths of different inputs. This gives us two dimensions along which to compare predictions: prediction error and signal strength.

The results for this analysis are shown in Chart 6, which comprises a comparison of the aggregate foresting performance of each of the main forecasting groups: overall Citizens’ Panel, high-hit rate Citizens’ Panel, the Bank of England, and we also show two ‘optimal forecast combinations’ (dark and light teal bars) for the Bank of England’s forecast with either of the two aggregate Citizens’ Panel predictions.^{footnote [33]} Results are shown for each of the four economic variables across the two distinct dimensions; error performance is displayed in the bars and signal strength is captured by the stars and numbers in each bar.

The bars show the normalised relative error. Bar length indicates the error relative to all other predictions for this question with the highest error being set to one, so that values below one indicate a lower error. The significance level, or the confidence we have into a predicted series in a model, is indicated by the stars. Three stars indicate a high level of confidence, while fewer stars indicate reduced significance and no stars indicates a low level.^{footnote [34]} The number in each bar is the rank by signal strength of a measure among the five prediction series for each questions (the lower the better).

Chart 6: Citizen Panel predictions provide a useful signal along our own analysis

The aggregate foresting performance of the Bank of England and Citizens’ Panel groups as well as optimal forecast combinations

Footnotes

• Sources: Bank of England Citizens’ Panel Survey and ONS.

For example, the upper-most red bar in Chart 6 represents the average prediction error for the present level of consumer price inflation made from averaging all survey responses. Despite providing confidence (three stars), it had the largest error among all series (bar length of one) and the weakest signal (rank 5).

Overall, we see that the great majority of forecasts carried a high level of confidence meaning that they were well aligned with the realised targets in a statistical sense (mostly many stars). Bank of England forecasts mostly were considerably more accurate than those derived from the Citizens’ Panels (shorter bars) and were highly significant (always many stars).

However, Bank of England forecasts did not always provide the strongest signal. For instance, despite the bias in the levels (high error), Citizens’ Panels inflation forecasts often provided a signal of higher confidence. The reason for this is that a regression model accounts for differences in the level (bias) and the response magnitude of an input variable relative to the predicted target (variance). The former is captured by the intercept and the latter by the regression coefficient in a model. In this way, co-movement of series with the predicted target can be detected and scaled appropriately. As such, even a numerically weak forecast can contain useful information.

More importantly to comparing individual forecasts, combinations of Bank of England and Citizens’ Panels’ signals had clear benefits. They always provided a stronger signal than our own forecasts (higher rank), and led to more accuracy at times. This was especially the case for the assessment of unemployment. In both cases, a forecast combination offered an attractive combination of accuracy and signal.

Arguably the sample period is rather short, especially for the forward-looking questions, and rather unique given that it includes the onset of the pandemic. This means that our statistical exercises needs to be treated with care. Nevertheless, this demonstrates that adding information extracted from Citizens’ Panel forecasts can potentially add value alongside our own analysis.

Discussion and the way forward

We presented the results of the first Citizens’ Panel survey and forecasting competition in this article. We saw that information from panellists provided interesting and useful insights about the UK economy. Citizens’ Panels generated an early signal for the negative impact of the Covid pandemic, while actual outcomes chimed well with panellists’ responses.

One example is the uncertainty about inflation among professional economists when the pandemic hit. This was mirrored in the increased polarisation recorded in the survey after the first lockdown in March 2020.

Collective forecasts for inflation and unemployment from our panellists were found to provide strong and meaningful signals alongside our own assessments. Forecast combinations of Bank of England and average Citizens’ Panel predictions often performed best in terms of both accuracy and confidence.

However, these results also come with some caveats. For instance, the group of panellists with a high hit-rate, and qualitatively different and promising forecasts compared to those of the majority, could only be identified after the fact. There is a risk of ‘overfitting’ here, meaning we may be overconfident about our results.

We looked for this post-hoc selection bias by selecting an alternative high hitting group based on the first half of responses people gave if they answered at least four surveys in total. That is, somebody was a high hitter if their first half of survey responses (at least two rounds) resulted in an average hit rate of at least 40%. We then looked at the second half of responses to test this newly selected group of now 18 high-hitters. In this more stringent test we indeed saw a larger reversion effect with the hit-rate dropping to 30% indicating that our previous high-hitter results may be somewhat overconfident.

This issue and the one of the particular survey period including the pandemic (during which the alternative group of high-hitters achieved a rate of 35%) can be tested more comprehensively by repeating such exercises in similar settings, likely over several rounds, and incorporating the lessons we have learned from this proof of concept. Looking forward, we are expanding the online community in our Citizens’ Forum in collaboration with Economy. This offers the opportunity to revisit the questions addressed and raised by this first survey and forecasting challenge. A new survey and forecasting competition will run from January 2022 onwards.

Additional resources, either to get in touch with the economics community or to learn more about what the Bank of England does, can be found at the Economics Observatory, Discover Economics or our KnowledgeBank.

Annex

• Citizens’ Panel Survey questions (September 2019 to August 2020) and Covid questions (April 2020 to August 2020)

  Economic health of their local community

  1. How do you judge the economic health of your local community? This can be the council area, city or town you live in.

  Answers:

  • Very good

  • Good

  • Neither good nor bad

  • Bad

  • Very bad

  2. Looking forward, how do you think that the economic situation of your community will develop over the next six months, eg judged by the number of jobs and businesses?

  Answers:

  • Will improve

  • Likely to improve

  • Not sure

  • Likely to deteriorate

  • Will deteriorate

  3. What do you think poses the biggest threat to the economic health of your community? This can be economic, political, social, environmental or something else. Please feel free to give as much detail as you like on your thinking.

  Open text field

  Personal financial situation

  4. How satisfied are you with your personal financial situation, judged by your income, cash at hand and financial security?

  Answers:

  • Very good

  • Good

  • Neither good nor bad

  • Bad

  • Very bad

  5. How do you think that your personal financial situation will develop over the next six months?

  Answers:

  • Very satisfied

  • Rather satisfied

  • Neither satisfied nor dissatisfied

  • Rather dissatisfied

  • Very dissatisfied

  6. Do you think that any changes to your personal financial situation are more likely to be related to local factors in your community or factors beyond it, eg those impacting the UK as a whole?

  Answers:

  • Local

  • UK wide

  • Outside UK

  • None of the above

  Unemployment

  7. Unemployment in the recent past: How do you think the rate of unemployment has changed in your region in the past six months?

  Answers:

  • Large decrease [drop of at least -0.5pp, ie more people in work]

  • Small decrease [drop of -0.2pp to -0.4pp]

  • About the same [drop of -0.1pp to increase of +0.1pp]

  • Small increase [increase of 0.2pp to 0.4pp]

  • Large increase [increase of 0.5pp or more, ie fewer people in work]

  8. Unemployment in the near future: How do you think the rate of unemployment will change in your region over the next six months?

  Answers:

  • Large decrease [drop of at least -0.5pp, ie more people in work]

  • Small decrease [drop of -0.2pp to -0.4pp]

  • About the same [drop of -0.1pp to increase of +0.1pp]

  • Small increase [increase of 0.2pp to 0.4pp]

  • Large increase [increase of 0.5pp or more, ie fewer people in work]

  Inflation (consumer prices index)

  9. Current price changes: What do you judge the current rate of inflation to be, relative to the Bank of England’s 2% target? (Measured by the consumer prices index (CPI))

  Answers:

  • Much lower [CPI of 1% or less, ie items cost about the same or are getting cheaper]

  • Somewhat lower [CPI of 1.1% to 1.7%]

  • About the same [CPI of 1.8% to 2.2%]

  • Somewhat higher [CPI of 2.3% to 2.9%]

  • Much higher [CPI of 3% or above, ie items cost are getting more expensive]

  10. Future prices: What do you think the rate of inflation will be in six months’ time, relative to the Bank of England’s 2% target? (Measured by the consumer prices index (CPI))

  Answers:

  • Much lower [CPI of 1% or less, ie items cost about the same or are getting cheaper]

  • Somewhat lower [CPI of 1.1% to 1.7%]

  • About the same [CPI of 1.8% to 2.2%]

  • Somewhat higher [CPI of 2.3% to 2.9%]

  • Much higher [CPI of 3% or above, ie items cost are getting more expensive]

  Region

  11. Please select the region you are forecasting for. These are the same as used by the ONS.

  Answers:

  • London

  • North East

  • North West

  • South East

  • South West

  • East

  • East Midlands

  • West Midlands

  • Yorkshire and the Humber

  • Scotland

  • Northern Ireland

  • Wales

  Covid questions

  12. To what extent has your financial situation been affected by Covid-19 in the past month?

  Answers:

  • Significantly negatively

  • A little negatively

  • Not much change

  • Improved a little

  • Improved a lot

  13. Have your work patterns changed as a result of Covid-19? (Tick all that apply)

  Answers:

  • No longer working

  • Working fewer hours

  • Increased working from home

  • Going to work but taking special measures

  • No change to my work patterns

  • Not applicable (eg student or retired)