IMPROVEMEnTS In ASSESSInG THE FORECASTS ACCURACY - A CASE STUDY FOR ROMAnIAn MACROECOnOMIC FORECASTS

The objective of this study is to introduce new forecasts’ accuracy measures for two types of predictions: point forecasts (radical of order n of the mean of squared errors, mean for the difference between each predicted value and the mean of the effective values, ratio of radicals of sum of squared errors (RRSSEfor forecasts comparisons), different versions of U2 Theil’s statistic)) and for forecast intervals (number of intervals including the realization, difference between the realization and the lower limit, the upper one, respectively the interval centre). Comparisons are made to present the differences in results determined by the application of the classical measures of predictions accuracy for the inflation and unemployment rate forecasts provided for Romania by Institute for Economic Forecasting (IEF) and National Commission of Prognosis (NCP) on the horizon 20102012 and the values of new point forecasts accuracy measures. The hierarchy of predictions provided by the classical indicators and by the new ones are different. A novelty in literature is also brought by the methods of building the forecasts intervals. In addition to the classical interval based on historical error method, some new techniques of building forecasts are used: intervals based on the standard deviation and those constructed using bootstrap technique bias-corrected-accelerated (BCA) bootstrap method.


National Commission of Prognosis (NCP).
Our approach consists in reaching objectives like: proposal of some indicators to assess the point forecasts accuracy, the computing of modified versions of U2 Theil's statistic and the proposal of measures of accuracy for interval forecasts, the last contribution bringing a perspective that were not taken in consideration in literature till now.The real reasons for choosing more accuracy measures are the need to assess the predictions' precisionfrom different perspectives and the necessity to have an accuracy indicator for interval forecasts.
The new ways of assessing the predictions' accuracy for comparisons bring different results from those based on classical accuracy indicators.Only the most accurate forecast is indicated by the usual measures of accuracy and also by our proposed indicators.Actually, our objective is to analyse the problem of accuracy from different points of view.Each indicator communicates something regarding the precision of the provided forecasts.It is possible that one indicator suggests one particular forecast as being the best, while other indicator recommends another.More indicators having the lowest value for a certain type of forecasts, more the chances are to be in the case of the most accurate forecast.

LITERATURE REVIEW
Many international institutions are specialized in providing their own macroeconomic appreciations.Some researchers were interested in evaluating the accuracy of those predictions (Timmermann for IMF Melander for European Commission, Vogel for OECD), neglecting the comparison with government's expectations.
Genrea, Kenny, Meylera and Timmermann (2013) made forecasts combinations starting from SPF predictions for ECB and using performance-based weighting, trimmed averages, principal components analysis, Bayesian shrinkage, least squares estimates of optimal weights.Only for the inflation rate there was a strong evidence of improving the forecasts accuracy with respect to the equally weighted average prediction.
Hess and Orbe (2013) studied the association between analyst characteristics and the macroeconomic forecasts accuracy, noticing that the experience and the abilities of the analyst generate a better accuracy.
Clarck and McCracken (2013) brought recent and important contribution in this domain: the assessment of point and density forecasts using the Vector Autoregression, direct and iterative forecasts with more steps, the application of accuracy tests on different samples of forecasts.
Bratu (2012) asessed the accuracy of some macroeconomic predictions for Romania made by the Institute of Economic Forecasting and the National Commission of Prognosis, the last institution outperforming the forecasts for: inflation, unemployment, GDP deflator, export rate and exchange rate on the horizon 2004-2011. Novotny and Rakova (2012) assessed the accuracy of macroeconomic forecasts made by Consensus for the Czech Republic, observing an improvement in accuracy from a year to another on the horizon 1994-2009.
The authors also proposed a regression for comparing the predictions.
Abreu (2011) was interested in assessing the performance of macroeconomic predictions of IMF, European Commission and OECD and two private institutions (Consensus Economics and The Economist).The directional accuracy and the ability of predicting an eventual economic crisis were studied.
Dovern and Weisser (2011) used a broad set of individual forecasts to analyze four macroeconomic variables in G7 countries.Analyzing accuracy, bias and forecasts efficiency, resulted large discrepancies between countries and also in the same country for different variables.In general, the forecasts are biased and only a fraction of GDP forecasts are closer to the results registered in reality.
Gorr (2009) showed that the univariate method of prediction is suitable for normal conditions of forecasting while using conventional measures for accuracy, but multivariate models are recommended for predicting exceptional conditions when ROC curve is used to measure accuracy.Ruth (2008), using the empirical studies, obtained forecasts with a higher degree of accuracy for European macroeconomic variables by combining specific sub-groups predictions in comparison with forecasts based on a single model for the whole Union.
Heilemann and Stekler (2007) explain why macroeconomic forecast accuracy in the last 50 years in G7 has not improved.The first explanation refers to the critic brought to macroeconometrics models and to forecasting models, and the second one is related to the unrealistic expectations of forecast accuracy.Problems related to the forecasts bias, data quality, the forecast process, predicted indicators, the relationship between forecast accuracy and forecast horizon are analyzed.
In order to evaluate the forecast performance, and also to order the predictions, statisticians have developed several measures of accuracy.Fildes R. and Steckler (2000)  Terốsvirta, van Dijkand Medeiros (2005) examine the accuracy of forecasts based on linear autoregressive models, autoregressive with smooth transition (STAR) and neural networks (neural network-NN) time series for 47 months of the macroeconomic variables of G7 economies.For each model is used a dynamic specification and it is showed that STAR models generate better forecasts than linear autoregressive ones.Neural networks over long horizon forecast generate better predictions than the models using an approach from private to general.Armstrong and Fildes (1995) shows that the purpose of measuring forecast error is the provision of information about the shape of errors distribution and proposed a loss function for measuring the forecast error.Armstrong and Fildes show that it is not sufficient to use a single measure of accuracy.
If X is the predicted quantitative variable, the error of forecast is computed as the difference between the registered and the predicted value: e x.The frequently used indicators for evaluating the forecasts accuracy, according to Fildes and Steckler (2000), are computed as: Root Mean Squared Error (RMSE): (1) Mean error (ME): (2) and Mean absolute error (MAE): (3) These measures of accuracy have some disadvantages.For example, RMSE is affected by outliers.These measures are not independent of the unit of measurement, unless if they are expressed as percentage.If we have two forecasts with the same mean absolute error, RMSE penalizes the one with the biggest errors.U Theil's statistic, utilized in comparing the forecasts, can be calculated in two variants, specified also by the Australian Tresorery.U1 and U2.U1 statistic values are compared and closer to zero is a prediction, higher accuracy has: (4) A U2 less than one supposes a superior forecast comparedto the naţve one that is based on random walk: (5) The following notations were utilized: aactual values, p-predicted values, treference time, e-error (e=a-p), n-number of time periods.We propose the introduction of measures of accuracy as: -Radical of order n of the mean of squared errors: (6) -The mean for the difference between each predicted value and the mean of the effective values on the forecasting horizon: (7) -For comparisons with the naive forecasts a new indicator is computed: ratio of radicals of sum of squared errors: (8) In order to compare two forecasts even for different variables, the values of this indicator are compared, a value closer to zero indicating a better accuracy.

THE ASSESSMENT OF POINT FORECASTS ACCURACY
The new accuracy measures are computed for the inflation and unemployment rate provided by Institute for Economic Forecasting (IEF) and National Commission of Prognosis (NCP) on the forecasting horizon 2010-2012.The data are published by the two institutions in their papers of forecasting and they are available on the websites (www.ipe.ro and www.cnp.ro).Firstly, some usual accuracy indicators are computed.Amid a severe economic contraction in 2009 in Romania, a relatively .2.
According to the new accuracy measures, the best forecasts are provided by IEF for the inflation rate, the difference between the mean of registered values and that of the predictions being only of -0.2043.The same value in absolute terms for the mean of the deviation of each predicted value from the average of the effective values supports the persistence of the overestimation of the average.The hierarchy of the predictions regarding the accuracy is evaluated using the RRSSE indicator and it is different as that provided by the evaluation of U1 statistic of Theil: forecasts of IEF for inflation rate, IEF predictions for unemployment rate, NCP forecasts for inflation, respectively for unemployment.
The U2 statistic can be modified in order to make the comparisons with other   U2* statistic values show that, excepting the NCP inflation rate, all the other predictions are better than the filtered naţve forecasts based on Hodrick-Prescott filter.The indicators forecasted by IEF are more accurate than the smoothed naţve ones in Holt-Winters variant and even than the forecasts resulted from the proposed model.For NCP predictions the situation is exactly the opposite.

THE ASSESSMENT OF FORECAST INTERVALS ACCURACY
In this study some methods of building the forecast intervals are proposed.
Firstly, to construct the forecasts intervals we take into account the predictions provided by NCP in the pessimistic and optimistic versions and those of IEF for main, respectively desirable scenario.The samples of forecasts have low volume, so the t-Student distribution is used. (10) The average of all the predictions that were proposed by forecasters is computed and the standard deviation coorected with the number of forecasts will be utilized in constructing new forecast intervals: (11) Other forecasts intervals are constructed using the RMSE of the previous year, where the RMSE is computed differently as the root 58 M.Bratu / SJM 8 (1) (2013) 53 -65 mean squared of differences between each prediction of a year made by a certain institution and the real value of the indicator. (12) A resampling technique is applied to build forecasts intervals and it consists in replicating the sample of predictions a huge number of times.Basically, a proxy population is made starting only from a sample.This is actually an artificial population.
The bias-corrected-accelerated interval (BCA) is a complex bootstrap technique used to construct confidence intervals.In this case, Davison and Hinkley (1997) showed that estimates for bias and acceleration are provided using the initial sample and the bootstrap samples.
One accuracy measure for forecast intervals could be the number of intervals in which the real value is placed.The results are presented in Table 4 between the realization for a specific year and the lower limit of each interval or the upper one or the interval centre could be considered as suitable measures of predictions accuracy.A lower difference implies a better forecast interval.The differences are computed as: d1= realization -lower limit, d2= realization -upper limit, d3= realization -centre.Starting from these deviations we can compute their average or their absolute average on the forecasting horizon, the results being presented in Table 5.
Only 5 of the forecast intervals based on bootstrap method include the real values of inflation rate and 3 of those based on BCA bootstrap technique are suitable.7 out of 11 forecast intervals based on standard deviation contain the inflation rate values.All the intervals constructed using the inflation rate of the previous year include the realizations of inflation.Excepting the interval from 2002, all the others built using the previous RMSE include the real values.The new accuracy measures are shown in Appendix 1 for the inflation rate predictions and in Appendix 2 for unemployment rate.
For the forecast intervals of inflation rate based on bootstrap technique, the lowest value is registered by d2, but when absolute values of deviation are taken into account d2 is the highest.
BCA bootstrap method gave the best results for d1 (0.069), the lowest value for all methods.In average, the deviation between the realization and the inferior limit is 0.069% while the one between the realization and the intervals' centres is around 1.23%.
d3 registered the lowest value compared to d1 and d2 when the method applied is based on the previous registered value.
If the forecast intervals are based on standard deviation, d1 registers again the best value.It maintains to be the minimum even if the absolute values of the deviations are computed.
According to the values of d1, d2 and d3 and the corresponding values for absolute deviations, the BCA bootstrap technique provided the best intervals for inflation rate.
A negative value but better than d1 and d2 is registered when the historical RMSE is utilized.
For unemployment rate forecast intervals the better value is registered for d3 when BCA bootstrap method is applied.d1 is a good measure of accuracy for this method.d2 has the lowest value for intervals based on bootstrap method.
So, the new accuracy measures recommend the forecast intervals based on BCA bootstrap technique for inflation (d1) and for unemployment rate (d3).

CONCLUSION
This research enlarges the perspective of measuring the forecasts accuracy, by proposing some new measures for point forecasts and also for forecast intervals.The proposed measures draw attention about different results that may be registered when more predictions are compared.U1 Theil's statistic and the new indicator (ratio of
Кључне речи: Предвиђање, Тачност, "U Theil's" статистика, Интервали предвиђања, Техника убрзања корелације radicals of sum of squared errors) gave the different results regarding the hierarchy of forecasts.Our indicator reduces the too large weight assigned to large errors.Therefore, the new measure could be used to identify the most accurate forecast.According to RRSSE, the hierarchy of the predictions regarding the accuracy on the horizon 2010-2012 is: forecasts of IEF for inflation rate, IEF predictions for unemployment rate, NCP forecasts for inflation, respectively for unemployment.
The BCA bootstrap techniques gave the best results for the same accuracy measures of the prediction intervals for Romanian inflation and unemployment.The measures of accuracy proposed for forecast intervals are a novelty in this field.time series: A re-examination.International Journal of Forecasting, 21 (4): 755-774.

APPEnDIX 1
The new accuracy measures for forecast intervals of inflation

Table 1 .
Classical measures of accuracy for the forecasts made by IEF and NCP for inflation and unemployment rate Source: Author's computations

Table 2 .
The new measures of accuracy for the forecasts made by IEF and NCP for inflation and unemployment rate

Table 3 .
The U2 transformed statistic for the forecasts made by IEF and NCP for inflation and unemployment rate