by abelardo.moralejo » Mon Nov 23, 2015 7:24 pm
Hi,
I already sent to Adiv the Fig. 2 (SED) with upper limits. I put back the log-parabola as the function shown in the plot, we do not want to highlight it as a good 5-parameter "empirical fit" to the observed spectrum - since a simple power-law times the Domínguez nomimal EBL (i.e. 2 parameters in total) is better.
Below are comments to the current version, with very specific suggestions for whole paragraphs, in order to make things faster.
I have read the whole paper again, so some of the comments do not refer to the recent changes - but they are important (the referee will not be the last person to read the paper, I hope).
Abstract:
"including a scale factor" => "including a normalization factor" (for consistency with the next section of the abstract).
"from which the EBL imprint could be measured with a significance of 4.6 sigma for an opacity normalization factor…" =>
" … with a significance of 4.6 sigma, corresponding to an opacity normalization factor $\alpha_0$ = 1.07…. with respect to the nominal one in the assumed EBL template (Domínguez et al 2011)"
Results
"After background suppression cuts, 3219 gamma-like ON events above 60 GeV" => you do not provide the estimated background, so this is not very informative. Write either the excess events or both ON and OFF (the former is preferred). MORE IMPORTANT: is the number a typo?? I checked by chance your flute outputs and I see many more events! Post-cuts we have ~6200 EXCESS events after all cuts (including hadronness & theta2), i.e. those which go into the spectrum! PLEASE CHECK! We should put the number of excess events which go into the spectral calculation, especially because you say "above 60 GeV", I can guess your number comes from some Odie "FR" or something like that, but that is obviously wrong!
All the paragraph starting "In order to find an acceptable fit..." and ending in "in the literature has been used to fit the observed data from an HBL at VHE (Aharonian & et al. 2007)" has to be much reduced and simplified. And as we discussed already in this thread, it makes no sense to argue about fits to observed spectra to justify that the intrinsic spectrum must be this or that... (if anything, we should argue from BL Lac spectra in the optically-thin regime, like those of Fermi-LAT).
Here is my proposed version to replace that part:
"The average observed spectral energy distribution is shown in fig. 2. The estimated {\it intrinsic} spectrum, assuming the EBL model by Domínguez et al. (2011), can be fitted with a simple power-law function (PWL) with a probability of 0.23 (Chi2/d.o.f.= 16.3/13), a photon index Gamma = 2.0 +/- 0.1 and a normalization factor at 250 GeV f0 = (5.4 +/- 0.1) * 10^-11 cm-2 s-1 TeV-1. The {\it observed} spectrum is clearly curved. Several functions were tried to parametrize it: power-law with an exponential cut-off (EPWL), log-parabola (LP), log-parabola with exponential cut-off (ELP), power-law with a sub/super-exponential cutoff (SEPWL) and a smoothly-broken power-law (SBPWL). Of these, only the SBPWL (with 5 parameters), achieves an acceptable fit (P = 0.16), though with a sharp change of photon index by Delta_Gamma = 1.35 within less than a factor 2 in energy. Among the other, smoother functions, the next-best fit is provided by the LP (shown in fig. 2), with P = 2.2*10^-3. This non-trivial shape of the observed spectrum, and its simplification when the expected effect of the EBL is corrected, strongly suggests this observation has high potential for setting EBL constraints."
BUT PLEASE CHECK: You state 12 d.o.f. for the LP, but we have 14 points so it should be 11 for the log-parabola. I am afraid you are still including the next point (>4 TeV, with only 3 events, which we are not considering since the beginning). Please fix this, this inconsistency was also pointed out by the referee. Put the right P & Chi2 values for the 11 d.o.f. in the new paragraph above.
The last paragraph of the Results section should also be simplified:
"The night-wise estimated intrinsic spectra could all be fitted with power-laws, and the evolution of the resulting photon indices is shown in figure 3. In the latter part of the observed period, the activity of the source was lower, resulting in larger uncertainties for the fits. There is no evidence for significant spectral variability in the period covered by MAGIC observations, despite the large variations in absolute flux."
EBL measurement
"These arguments put serious constrains to the photon index of the energy spectrum of VHE photons" => "constraints"
In the paragraph "For the modeling of the intrinsic..." I would replace the part right after "...nor has it been observed in any BL Lac in the optically-thin regime."
by the following:
"... in the optically-thin regime. In particular, the un-absorbed part of BL Lac spectra measured by Fermi-LAT are well fitted by log-parabolas (Ackermann et al 2102).
\par
The PWL and the LP are functions that are linear in their parameters in the log flux - log E representation (hence well-behaved in the fitting process), and both can model pretty well the de-absorbed spectrum found in Sect. 3. The EPWL, ELP and SEPWL have additional (non-linear) parameters which are physically motivated, e.g. to account for possible internal absorption at the source. Note that these functions (except the PWL) can also mimic the {\it overall} spectral curvature induced by the EBL over a wide range of redshifts, but will be unable to fit the inflection point (in the optical depth vs. log E curve) that state-of-the-art EBL models predict around 1 TeV. We therefore expect an improvement of the fit quality as we approach the true value of the scaling factor $\alpha$, hence providing a measurement of the actual EBL density. The chosen spectral functions, however, do not exhaust {\it all possible} concave shapes. Therefore the EBL constraints we will obtain are valid under the assumption that the true intrinsic spectrum can be well described (within the uncertainties of the recorded fluxes) by one of those functions. As we saw in section 3, the 5-parameter smoothly-broken power-law (not included among the possible spectral models) provides an acceptable fit to the {\it observed} spectrum; if considered a plausible model for the intrinsic spectrum, it would severely weaken the lower EBL density constraint. On the contrary, the upper constraint (arguably the most interesting one VHE observations can contribute) from this work would be unaffected, as we will see below."
"To apply the TS we decided to use an average spectrum" => "For the calculation of the TS we decided to use the average flare spectrum"
The changes also require to modify this part of the same paragraph:
"Despite changing the flux level, the EBL determination method should work properly as long as the average intrinsic spectrum in the observation period can be described with one of the tested parameterizations. This would of course be the case if the spectral shape is stable, or changes moderately.
A varying spectral shape would in any case need quite some fine tuning to reproduce, in the average spectrum, a feature like the one expected to be induced by the EBL. A simple way to check the stability of the spectral shape is fitting the points on the Fig. 3 to a constant value. The Chi2/d.o.f. of this fit is 23.5/16 and the probability is 10%."
Instead, I suggest:
"Despite changing the flux level, the EBL determination method should work properly as long as the average intrinsic spectrum in the observation period can be described with one of the tested parameterizations. Assuming that is the case for the different states of the source, it will also hold for the average spectrum if the spectral {\it shape} is stable through the flare. A simple way to check the stability of the spectral shape is fitting the points on Fig. 3 to a constant value. The Chi2/d.o.f. of this fit is 23.5/16 and the probability is 10%, so there is no clear signature of spectral variability - beyond a weak hint of harder spectra in the second half of the observation period. A varying spectral shape would in any case need quite some fine tuning to reproduce, in the average spectrum, a feature like the one expected to be induced by the EBL."
Right after this:
"Then the Poissonian likelihood of the actual observation (the post-cuts number of recorded events vs Eest, in both the ON and OFF regions) was computed, after maximizing it in a parameter space which includes, besides the intrinsic spectral parameters, the Poisson parameters of the background in each bin of Eest."
Please include the following footnote, which addresses one of the referee's concerns:
"Note that in the Poissonian likelihood approach we have included the point at E ~ 55 GeV which was shown just as an upper limit in fig. 2, since it has an excess of just around 1 standard deviation above the background. The fits performed with the Poissonian likelihood approach have therefore one more degree of freedom than the Chi2 fits reported in section 3, and the 55 GeV point is included in fig. 7"
"Fig. 4 shows the -2 logL probabilities..." : although I think I suggested this wording myself, I now think that the proper way of saying it is "Chi2 probabilities". Although it is a poissonian likelihood, in the end in order to calculate probabilities we have to trust that -2log(L/Lmax) is Chi2-distributed in the null hypothesis (and in the asymptotic limit). PLEASE NOTE that this has to be changed also in Fig.4: y-axis should be "Fit Chi2 probability".
Actually, fig. 5 y-axis label is also wrong. What is plotted is not the "raw" poisonian L, but actually -2 log (L/Lmax), where Lmax is the "absolute maximum" likelihood, i.e. that of a model which would get exactly the number of Non and Noff events in all bins. This is asymptotically, and again thanks to the Wilks theorem, a Chi2 with the number of d.o.f. of the fit (points-parameters) because the alternative hypothesis (=perfect prediction) has 0 degrees of freedom - as many parameters as points, obviously. Either we explain it properly, or we just put on the y-axis "fit Chi2"
Then, replace the (still outdated) part starting "Following the approach by Abramowski et al. (2013) would lead us...", until the end of the paragraph, by:
"Following the approach in Abramowski et al. (2013) would lead us to choose the PWL as model for the intrinsic spectrum, as the next models in complexity (LP and EPWL) are not preferred at the 2-$\sigma$ level. However, choosing the PWL is rather questionable, since it does not allow for any intrinsic spectral curvature, meaning that all curvature in the observed spectrum will be attributed to the EBL absorption. If this procedure is applied to a large number of spectra, as in in Biteau & Williams (2015), individual < 2-$\sigma$ hints of intrinsic (concave) curvature might be overlooked and accumulate to produce a bias in the EBL estimation. In our case, the assumption of power-law intrinsic spectrum would result in an EBL “detection” at the 13-$\sigma$ level. We prefer to adopt a more conservative approach, choosing the next-best function, the LP. Note however that at the best-fit $\alpha$, all the tested functions become simple power-laws, hence the fit probabilities at the peak depend only on the number of free parameters.
At the end of section 4, add this paragraph:
"We again remark that allowing for other concave spectral shapes, like the SBPWL, would severely affect our lower EBL bound. This would also be the case for earlier published EBL lower constraints based on gamma-ray data - especially those in which the PWL is among the allowed models for the intrinsic spectrum. For the observations discussed in the present paper, the SBPWL would achieve an acceptable fit even in the no-EBL assumption. This and earlier "detections" of the EBL through its imprint on gamma-ray spectra hence rely on somewhat tentative assumptions on the intrinsic spectra - but assumptions which, as far as we know, are not falsified by any observational data available on BL Lacs. On the other hand, the upper bounds we have obtained are robust, since they are driven by the fact that convex spectral shapes (completely unexpected for BL Lacs at VHE) would be needed to fit our observations if EBL densities above the best-fit value are assumed."
Systematic uncertainty
I would remove:
"with the difference that in our case the scaling factors were applied to the data, whereas in Aleksi´c et al. (2015b), the light scale was artificially shifted in the MC simulations."
(the difference is not so relevant and it is actually not easy to understand)
Remove also:
"With this procedure, the accuracy of the light scale is determined, and since the number of photoelectrons is related to the energy of the shower, then a shift in the light scale is equivalent to a shift in the energy scale."
Sorry I did not catch the above earlier, but it is not correct. We do not determine here the accuracy of the light scale, we use the +/-15% to see how much the EBL results vary. And it is not equivalent to a shift in the energy scale.
For the part starting "Comparing the profiles, we looked..." until the end of the paragraph I suggest:
"From the 1-$\sigma$ uncertainty ranges in $\alpha$ obtained for the different shifts in the light scale, we determine the largest departures from our best-fit value $\alpha_0$, arriving to the final result $alpha_0 = 1.07 (-0.20, +0.24)_{stat+sys}$"
Conclusions
"The intrinsic spectrum of 1ES 1011+496, apart of being quite hard, apparently does not have intrinsic curvature, which makes it a good candidate for observations, unlike other sources at similar redshifts (z=0.212)."
Change to
"The spectrum of 1ES 1011+496 during this flare displays little intrinsic curvature over > 1 order of magnitude in energy, which makes this an ideal observation for constraining the EBL."
"the EBL was detected" => I would put "detected" in quotation marks, given the discussion with the referee.
"About the most constraining measurement of the EBL..." => how about HESS' PKS2155? at least the TS of one of the HESS samples in their EBL paper was larger. Did we discuss that statement? - I can't remember now.
PLEASE add in the acknowledgements the standard sentence thanking the anonymous referee for his fruitful suggestions etc.
Hi,
I already sent to Adiv the Fig. 2 (SED) with upper limits. I put back the log-parabola as the function shown in the plot, we do not want to highlight it as a good 5-parameter "empirical fit" to the observed spectrum - since a simple power-law times the Domínguez [i]nomimal[/i] EBL (i.e. 2 parameters in total) is better.
Below are comments to the current version, with very specific suggestions for whole paragraphs, in order to make things faster.
I have read the whole paper again, so some of the comments do not refer to the recent changes - but they are important (the referee will not be the last person to read the paper, I hope).
[b]Abstract:[/b]
"including a scale factor" => "including a normalization factor" (for consistency with the next section of the abstract).
"from which the EBL imprint could be measured with a significance of 4.6 sigma for an opacity normalization factor…" =>
" … with a significance of 4.6 sigma, corresponding to an opacity normalization factor $\alpha_0$ = 1.07…. with respect to the nominal one in the assumed EBL template (Domínguez et al 2011)"
[b]Results[/b]
"After background suppression cuts, 3219 gamma-like ON events above 60 GeV" => you do not provide the estimated background, so this is not very informative. Write either the excess events or both ON and OFF (the former is preferred). MORE IMPORTANT: is the number a typo?? I checked by chance your flute outputs and I see many more events! Post-cuts we have ~6200 EXCESS events after all cuts (including hadronness & theta2), i.e. those which go into the spectrum! PLEASE CHECK! We should put the number of excess events which go into the spectral calculation, especially because you say "above 60 GeV", I can guess your number comes from some Odie "FR" or something like that, but that is obviously wrong!
All the paragraph starting "In order to find an acceptable fit..." and ending in "in the literature has been used to fit the observed data from an HBL at VHE (Aharonian & et al. 2007)" has to be much reduced and simplified. And as we discussed already in this thread, it makes no sense to argue about fits to [b]observed[/b] spectra to justify that the intrinsic spectrum must be this or that... (if anything, we should argue from BL Lac spectra in the optically-thin regime, like those of Fermi-LAT).
Here is my proposed version to replace that part:
"The average observed spectral energy distribution is shown in fig. 2. The estimated {\it intrinsic} spectrum, assuming the EBL model by Domínguez et al. (2011), can be fitted with a simple power-law function (PWL) with a probability of 0.23 (Chi2/d.o.f.= 16.3/13), a photon index Gamma = 2.0 +/- 0.1 and a normalization factor at 250 GeV f0 = (5.4 +/- 0.1) * 10^-11 cm-2 s-1 TeV-1. The {\it observed} spectrum is clearly curved. Several functions were tried to parametrize it: power-law with an exponential cut-off (EPWL), log-parabola (LP), log-parabola with exponential cut-off (ELP), power-law with a sub/super-exponential cutoff (SEPWL) and a smoothly-broken power-law (SBPWL). Of these, only the SBPWL (with 5 parameters), achieves an acceptable fit (P = 0.16), though with a sharp change of photon index by Delta_Gamma = 1.35 within less than a factor 2 in energy. Among the other, smoother functions, the next-best fit is provided by the LP (shown in fig. 2), with P = 2.2*10^-3. This non-trivial shape of the observed spectrum, and its simplification when the expected effect of the EBL is corrected, strongly suggests this observation has high potential for setting EBL constraints."
BUT PLEASE CHECK: You state 12 d.o.f. for the LP, but we have 14 points so it should be 11 for the log-parabola. I am afraid you are still including the next point (>4 TeV, with only 3 events, which we are not considering since the beginning). Please fix this, this inconsistency was also pointed out by the referee. Put the right P & Chi2 values for the 11 d.o.f. in the new paragraph above.
The last paragraph of the Results section should also be simplified:
"The night-wise estimated intrinsic spectra could all be fitted with power-laws, and the evolution of the resulting photon indices is shown in figure 3. In the latter part of the observed period, the activity of the source was lower, resulting in larger uncertainties for the fits. There is no evidence for significant spectral variability in the period covered by MAGIC observations, despite the large variations in absolute flux."
[b] EBL measurement [/b]
"These arguments put serious constrains to the photon index of the energy spectrum of VHE photons" => "constraints"
In the paragraph "For the modeling of the intrinsic..." I would replace the part right after "...nor has it been observed in any BL Lac in the optically-thin regime."
by the following:
"... in the optically-thin regime. In particular, the un-absorbed part of BL Lac spectra measured by Fermi-LAT are well fitted by log-parabolas (Ackermann et al 2102).
\par
The PWL and the LP are functions that are linear in their parameters in the log flux - log E representation (hence well-behaved in the fitting process), and both can model pretty well the de-absorbed spectrum found in Sect. 3. The EPWL, ELP and SEPWL have additional (non-linear) parameters which are physically motivated, e.g. to account for possible internal absorption at the source. Note that these functions (except the PWL) can also mimic the {\it overall} spectral curvature induced by the EBL over a wide range of redshifts, but will be unable to fit the inflection point (in the optical depth vs. log E curve) that state-of-the-art EBL models predict around 1 TeV. We therefore expect an improvement of the fit quality as we approach the true value of the scaling factor $\alpha$, hence providing a measurement of the actual EBL density. The chosen spectral functions, however, do not exhaust {\it all possible} concave shapes. Therefore the EBL constraints we will obtain are valid under the assumption that the true intrinsic spectrum can be well described (within the uncertainties of the recorded fluxes) by one of those functions. As we saw in section 3, the 5-parameter smoothly-broken power-law (not included among the possible spectral models) provides an acceptable fit to the {\it observed} spectrum; if considered a plausible model for the intrinsic spectrum, it would severely weaken the lower EBL density constraint. On the contrary, the upper constraint (arguably the most interesting one VHE observations can contribute) from this work would be unaffected, as we will see below."
"To apply the TS we decided to use an average spectrum" => "For the calculation of the TS we decided to use the average flare spectrum"
The changes also require to modify this part of the same paragraph:
"Despite changing the flux level, the EBL determination method should work properly as long as the average intrinsic spectrum in the observation period can be described with one of the tested parameterizations. This would of course be the case if the spectral shape is stable, or changes moderately.
A varying spectral shape would in any case need quite some fine tuning to reproduce, in the average spectrum, a feature like the one expected to be induced by the EBL. A simple way to check the stability of the spectral shape is fitting the points on the Fig. 3 to a constant value. The Chi2/d.o.f. of this fit is 23.5/16 and the probability is 10%."
Instead, I suggest:
"Despite changing the flux level, the EBL determination method should work properly as long as the average intrinsic spectrum in the observation period can be described with one of the tested parameterizations. Assuming that is the case for the different states of the source, it will also hold for the average spectrum if the spectral {\it shape} is stable through the flare. A simple way to check the stability of the spectral shape is fitting the points on Fig. 3 to a constant value. The Chi2/d.o.f. of this fit is 23.5/16 and the probability is 10%, so there is no clear signature of spectral variability - beyond a weak hint of harder spectra in the second half of the observation period. A varying spectral shape would in any case need quite some fine tuning to reproduce, in the average spectrum, a feature like the one expected to be induced by the EBL."
Right after this:
"Then the Poissonian likelihood of the actual observation (the post-cuts number of recorded events vs Eest, in both the ON and OFF regions) was computed, after maximizing it in a parameter space which includes, besides the intrinsic spectral parameters, the Poisson parameters of the background in each bin of Eest."
Please include the following footnote, which addresses one of the referee's concerns:
"Note that in the Poissonian likelihood approach we have included the point at E ~ 55 GeV which was shown just as an upper limit in fig. 2, since it has an excess of just around 1 standard deviation above the background. The fits performed with the Poissonian likelihood approach have therefore one more degree of freedom than the Chi2 fits reported in section 3, and the 55 GeV point is included in fig. 7"
"Fig. 4 shows the -2 logL probabilities..." : although I think I suggested this wording myself, I now think that the proper way of saying it is "Chi2 probabilities". Although it is a poissonian likelihood, in the end in order to calculate probabilities we have to trust that -2log(L/Lmax) is Chi2-distributed in the null hypothesis (and in the asymptotic limit). PLEASE NOTE that this has to be changed also in Fig.4: y-axis should be "Fit Chi2 probability".
Actually, fig. 5 y-axis label is also wrong. What is plotted is not the "raw" poisonian L, but actually -2 log (L/Lmax), where Lmax is the "absolute maximum" likelihood, i.e. that of a model which would get exactly the number of Non and Noff events in all bins. This is asymptotically, and again thanks to the Wilks theorem, a Chi2 with the number of d.o.f. of the fit (points-parameters) because the alternative hypothesis (=perfect prediction) has 0 degrees of freedom - as many parameters as points, obviously. Either we explain it properly, or we just put on the y-axis "fit Chi2"
Then, replace the (still outdated) part starting "Following the approach by Abramowski et al. (2013) would lead us...", until the end of the paragraph, by:
"Following the approach in Abramowski et al. (2013) would lead us to choose the PWL as model for the intrinsic spectrum, as the next models in complexity (LP and EPWL) are not preferred at the 2-$\sigma$ level. However, choosing the PWL is rather questionable, since it does not allow for any intrinsic spectral curvature, meaning that all curvature in the observed spectrum will be attributed to the EBL absorption. If this procedure is applied to a large number of spectra, as in in Biteau & Williams (2015), individual < 2-$\sigma$ hints of intrinsic (concave) curvature might be overlooked and accumulate to produce a bias in the EBL estimation. In our case, the assumption of power-law intrinsic spectrum would result in an EBL “detection” at the 13-$\sigma$ level. We prefer to adopt a more conservative approach, choosing the next-best function, the LP. Note however that at the best-fit $\alpha$, all the tested functions become simple power-laws, hence the fit probabilities at the peak depend only on the number of free parameters.
At the end of section 4, add this paragraph:
"We again remark that allowing for other concave spectral shapes, like the SBPWL, would severely affect our lower EBL bound. This would also be the case for earlier published EBL lower constraints based on gamma-ray data - especially those in which the PWL is among the allowed models for the intrinsic spectrum. For the observations discussed in the present paper, the SBPWL would achieve an acceptable fit even in the no-EBL assumption. This and earlier "detections" of the EBL through its imprint on gamma-ray spectra hence rely on somewhat tentative assumptions on the intrinsic spectra - but assumptions which, as far as we know, are not falsified by any observational data available on BL Lacs. On the other hand, the upper bounds we have obtained are robust, since they are driven by the fact that convex spectral shapes (completely unexpected for BL Lacs at VHE) would be needed to fit our observations if EBL densities above the best-fit value are assumed."
[b] Systematic uncertainty [/b]
I would remove:
"with the difference that in our case the scaling factors were applied to the data, whereas in Aleksi´c et al. (2015b), the light scale was artificially shifted in the MC simulations."
(the difference is not so relevant and it is actually not easy to understand)
Remove also:
"With this procedure, the accuracy of the light scale is determined, and since the number of photoelectrons is related to the energy of the shower, then a shift in the light scale is equivalent to a shift in the energy scale."
Sorry I did not catch the above earlier, but it is not correct. We do not determine here the accuracy of the light scale, we use the +/-15% to see how much the EBL results vary. And it is not equivalent to a shift in the energy scale.
For the part starting "Comparing the profiles, we looked..." until the end of the paragraph I suggest:
"From the 1-$\sigma$ uncertainty ranges in $\alpha$ obtained for the different shifts in the light scale, we determine the largest departures from our best-fit value $\alpha_0$, arriving to the final result $alpha_0 = 1.07 (-0.20, +0.24)_{stat+sys}$"
[b]Conclusions[/b]
"The intrinsic spectrum of 1ES 1011+496, apart of being quite hard, apparently does not have intrinsic curvature, which makes it a good candidate for observations, unlike other sources at similar redshifts (z=0.212)."
Change to
"The spectrum of 1ES 1011+496 during this flare displays little intrinsic curvature over > 1 order of magnitude in energy, which makes this an ideal observation for constraining the EBL."
"the EBL was detected" => I would put "detected" in quotation marks, given the discussion with the referee.
"About the most constraining measurement of the EBL..." => how about HESS' PKS2155? at least the TS of one of the HESS samples in their EBL paper was larger. Did we discuss that statement? - I can't remember now.
PLEASE add in the acknowledgements the standard sentence thanking the anonymous referee for his fruitful suggestions etc.
