add low NHI HCDs

[londumas]

@fjaviersanchez, @TEtourneau tested adding the low NHI HCDs and he does not recover the proper number density of NHI~20 HCDs. do you know what is happening? It is good at ~17 and ~22. Maybe you have a power-low instead of the pygm model.

using /global/cscratch1/sd/tetourne/DesiMocks/v4.3.0/mock_0/output/master_DLA_3.fits

[jmarclegoff]

@londumas, this must be for the last version produced by Thomas with N_HI_min = 17.2. Thomas saw that setting N_HI_min = 17.2 reduces the number of DLA with N_HI> 20 by a factor 5. This seems compatible with having a correct distribution when N_HI_min = 20. If this is the case we can focus on understanding what is happening when we reduce N_HI_min.
May be you could make the same plot with Thomas's previous version which still had N_HI_min = 20.

[fjaviersanchez]

@londumas, in the code we directly call pyigm to get f(N) (see here; also note that fN.evaluate returns log(f(N)) see here for more details and that's why there's an exponential in the function to get the probability of getting a given f(N,z)).

I am confused on what's going on 😕, though because if we are using pyigm I would expect this to be automatically right. Could you please add a comparison with pyigm integrating f(N,z)dz from z=2.1 until the edge of the box (is that z=4??)? I think that would be the fairer comparison. In any case, from that plot, it looks like going as low as N=17.2 is too much and we may want to dial it back to N=20 or oversample the number of DLAs in general and then apply some sort of selection function. Thoughts?

[jmarclegoff]

@fjaviersanchez, It looks like if you just use --nmin parameter for DLA_Saclay.py it goes only to dNdz(). I believe it should also go into add_DLA_table_to_object_Saclay(0 in order to go to get_N().
So we need to know how @TEtourneau did to go down to N_HI_min = 17.2.

[londumas]

using the catalog that goes above NHI>20, we get the same as before.
/global/cscratch1/sd/tetourne/DesiMocks/v4.3.0/mock_0/output/master_DLA_3.fits

[jmarclegoff]

Ok, so it's likely that there was some problem in the way N_HI_min was reduced.
Let's wait for @TEtourneau to tell us what he did exactly.

[TEtourneau]

I did use --nmin parameter in both dNdz() and get_N() functions (I added the use of --nmin in add_DLA_table_to_object_Saclay)

[jmarclegoff]

I did some tests of pyigm and DLA_Saclay.py

z = 2.37 *np.ones(1000000)
nHi = get_N(z,Nmin=17.2)
plt.hist(nHi,bins=100)
Nmin = 17.2
Nmax = 22.5
nsamp = 100
nn = np.linspace(Nmin,Nmax,nsamp)
z = 2.37 *np.ones(1)
auxfN = fN_default.evaluate(nn,z)
fN  = np.exp( auxfN.reshape(nsamp) )
plt.plot(nn,5.64E12*fN)

This code gives the plot below and shows that get_N() produces the proper distribution of N_HI up to an ad hoc normalisation (the orange curve is 5.64E12 * exp(f(N_HI)) ).

Then I tested dNdz()

def dNdz(z, Nmin=20.0, Nmax=22.5):
    """ Get the column density distribution as a function of z,
    for a given range in N"""
    if use_pyigm:
        # get incidence rate per path length dX (in comoving coordinates)
        dNdX = fN_default.calculate_lox(z,Nmin,Nmax)
        # convert dX to dz
        dXdz = fN_cosmo.abs_distance_integrand(z)
        return dNdX * dXdz
    else:
        return dnHD_dz_cumlgN(z,Nmax)-dnHD_dz_cumlgN(z,Nmin)

In [42]: dNdz(z,Nmin=20)/ dNdz(z,Nmin=17.2)
Out[42]: array([0.21780969])

This does not seem consistent with the plot above. Indeed
In [32]: 1.*len(nHi[nHi>20]) / len(nHi)
Out[32]: 0.039272

Is it that fN_default.calculate_lox()is not doing what we believe ? I went to github/pyigm and did not find much documentation but may be I did not look t the proper place.
On the other hand, this seems just a problem of normalisation and should not explain the distribution of f(n_HI) obtained by @londumas for the mocks.
I do not know the details of DLA_Saclay.py, @fjaviersanchez, do you have any idea ?

[jmarclegoff]

By the way, @fjaviersanchez from igmhub/LyaCoLoRe#32
I understand that you set Nmin = 17.2 in LyaCoLoRe. Are you still waiting for James to produce the DLA catalogue ? It would be interesting to see if this results in the same N_HI distribution as measured by @londumas for Saclay mocks.

[fjaviersanchez]

@jmarclegoff thanks a lot for the plots and for following up on this. I think that you are right and fN_default.calculate_lox is not giving what we expected. I modified dNdz as follows:

def dNdz(z, Nmin=20.0, Nmax=22.5, nsamp=100):
    """ Get the column density distribution as a function of z,
    for a given range in N""" 
    # get incidence rate per path length dX (in comoving coordinates)
    nn = np.linspace(Nmin,Nmax,nsamp)
    aux = fN_default.evaluate(nn, z)
    dNdz = np.sum(np.exp(aux)*(nn[1]-nn[0]))
    return dNdz

I tested this with the data that you provided above and it seems to be working. What I did is the following:

dNdz(2.7, Nmin=20.0)/dNdz(2.7, Nmin=17.2) ~ 0.038

1-np.count_nonzero((nHi>=17.2) & (nHi <=20))*1.0/len(nHi) ~ 0.039 (which is equivalent to np.count_nonzero(nHi>20)*1.0/len(nHi) as you did).

As for the London mocks: yes, I am waiting for James to produce the new mocks but I don't know exactly @andreufont and @jfarr03's plans. In any case, I suspect that they'd need to change dNdz as well.

Please, feel free to test this fix and let me know if there are any other issues. Thanks again!

[TEtourneau]

@fjaviersanchez Thanks for the suggestion. I tested what you posted (I just copied/pasted the new dNdz function into the DLA_Saclay.py code).
I first did 2 runs, with rmin = 20 and rmin = 17.2
For rmin = 20, I only found 1 DLA (for about 550 000 QSO)
for rmin = 17.2 I found 14 DLA
These catalogs are here:
/global/cscratch1/sd/tetourne/DesiMocks/v4.3.0/mock_0/Chunk_ra0189.982649735_dec020/dla5.fits : rmin=20
/global/cscratch1/sd/tetourne/DesiMocks/v4.3.0/mock_0/Chunk_ra0189.982649735_dec020/dla6.fits : rmin=17.2

I then did 2 other runs, with a factor 20000 in front of dNdz:
I found 11000 DLA for rmin=20 (here: /global/cscratch1/sd/tetourne/DesiMocks/v4.3.0/mock_0/Chunk_ra0189.982649735_dec020/dla7.fits)
I found 285765 DLA for rmin=17.2, with 11158 DLA with n_HI > 20 (here: /global/cscratch1/sd/tetourne/DesiMocks/v4.3.0/mock_0/Chunk_ra0189.982649735_dec020/dla8.fits)

Here are the histograms for these 2 last files:

[TEtourneau]

I forgot:
The number of QSO I indicated is all the QSO, so 1.8 < z < 2.1
There are 359362 QSO at z > 2.1
and 11158 DLA with n_HI > 20 (including the 20000 factor on dN/dz)
So we have ~ 0.03 DLA per QSO.
So, if I'm right, we are still missing a factor 0.2 / 0.03 ~ 6.4 on dN/dz

[fjaviersanchez]

thanks for testing this so quickly @TEtourneau. Maybe I understood it incorrectly but it sounds like there's a normalization problem. Do you think that using a factor of 6.4*20000 will solve the discrepancies?