Load fits problem

[eduardojsbarroso]

I see that there is a problem with the load_fits function and the new metadata reading from newer versions of sacc.

When we generate the file, we create an empty metadata column. When running the load_fits method, it calls numpy_to_vanilla method to read all columns. If I am not wrong, recent updated made this function also read a metadata column. If there is nothing there, we go trough all the ifs:

def numpy_to_vanilla(x):
    """
    Convert a NumPy scalar type to its corresponding Python built-in type.

    Parameters
    ----------
    x : numpy scalar
        A NumPy scalar value (e.g., np.str_, np.int64, np.float64, np.bool).

    Returns
    -------
    object
        The equivalent Python built-in type (e.g., str, int, float, bool).
    """
    if type(x) == np.str_:
        x = str(x)
    elif type(x) == np.int64:
        x = int(x)
    elif type(x) == np.float64:
        x = float(x)
    elif type(x) == np.bool:
        x = bool(x)
    return x

np.bool is deprecated and so it gives an error. I propose we remove

    elif type(x) == np.bool:
        x = bool(x)

from the function. However IDK if there is other things to be checked since before there was no metadata column in the generated SACC files

[eduardojsbarroso]

@joezuntz @marcpaterno I don't know if I am the only one having these problems with the new release. I added a simple python file here to show the problem if you have SACC 2.1

[joezuntz]

Thanks for this! It looks like the tests are failing because a data file is missing - does it need to call the create function first?

We might actually want to use np.bool_ instead of just np.bool, that should also fix it. The error will depend on the numpy version used.

[eduardojsbarroso]

Thanks for this! It looks like the tests are failing because a data file is missing - does it need to call the create function first?

We might actually want to use np.bool_ instead of just np.bool, that should also fix it. The error will depend on the numpy version used.

By data file you mean the sacc file? if I just run this test with python I get:

python test_load_fits.py 
<class 'sacc.tracers.clusters.BinZTracer'>
SACC file saved with 4 data points: simple_mock_clusters.sacc
Filename: simple_mock_clusters.sacc
No.    Name      Ver    Type      Cards   Dimensions   Format
  0  PRIMARY       1 PrimaryHDU       5   ()      
  1  tracer:survey:my_mock_survey    1 BinTableHDU     18   1R x 3C   [14A, 7A, D]   
  2  tracer:bin_z:zbin_0    1 BinTableHDU     20   1R x 4C   [6A, 7A, D, D]   
  3  tracer:bin_z:zbin_1    1 BinTableHDU     20   1R x 4C   [6A, 7A, D, D]   
  4  tracer:bin_richness:rich_0    1 BinTableHDU     20   1R x 4C   [6A, 7A, K, K]   
  5  tracer:bin_richness:rich_1    1 BinTableHDU     20   1R x 4C   [6A, 7A, K, K]   
  6  covariance:full:cov    1 BinTableHDU     16   4R x 1C   [4D]   
  7                1 BinTableHDU     21   4R x 5C   [14A, 6A, 6A, K, K]   
  8  metadata:metadata:metadata    1 BinTableHDU     12   0R x 0C   []   
/sps/lsst/groups/clusters/cl_pipeline_project/conda_envs/txpipe_clp/lib/python3.10/site-packages/sacc/utils.py:210: FutureWarning: In the future `np.bool` will be defined as the corresponding NumPy scalar.
  elif type(x) == np.bool:
Traceback (most recent call last):
  File "/sps/lsst/users/ebarroso/sacc/test/test_load_fits.py", line 50, in <module>
    t2 = sacc.Sacc.load_fits("simple_mock_clusters.sacc")
  File "/sps/lsst/groups/clusters/cl_pipeline_project/conda_envs/txpipe_clp/lib/python3.10/site-packages/sacc/sacc.py", line 997, in load_fits
    return cls.from_tables(tables, cov=cov)
  File "/sps/lsst/groups/clusters/cl_pipeline_project/conda_envs/txpipe_clp/lib/python3.10/site-packages/sacc/sacc.py", line 1134, in from_tables
    objs =  io.from_tables(tables)
  File "/sps/lsst/groups/clusters/cl_pipeline_project/conda_envs/txpipe_clp/lib/python3.10/site-packages/sacc/io.py", line 316, in from_tables
    dps = DataPoint.from_table(table, lookups=lookups)
  File "/sps/lsst/groups/clusters/cl_pipeline_project/conda_envs/txpipe_clp/lib/python3.10/site-packages/sacc/data_types.py", line 487, in from_table
    tags = {numpy_to_vanilla(name): row[name] for name in tag_names}
  File "/sps/lsst/groups/clusters/cl_pipeline_project/conda_envs/txpipe_clp/lib/python3.10/site-packages/sacc/data_types.py", line 487, in <dictcomp>
    tags = {numpy_to_vanilla(name): row[name] for name in tag_names}
  File "/sps/lsst/groups/clusters/cl_pipeline_project/conda_envs/txpipe_clp/lib/python3.10/site-packages/sacc/utils.py", line 210, in numpy_to_vanilla
    elif type(x) == np.bool:
  File "/sps/lsst/groups/clusters/cl_pipeline_project/conda_envs/txpipe_clp/lib/python3.10/site-packages/numpy/__init__.py", line 305, in __getattr__
    raise AttributeError(__former_attrs__[attr])
AttributeError: module 'numpy' has no attribute 'bool'.
`np.bool` was a deprecated alias for the builtin `bool`. To avoid this error in existing code, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
    https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations. Did you mean: 'bool_'?

So it seems that it is generating the file normally but it is not able to read it. And if I add the lines for np.bool, it works

[joezuntz]

You can see the error message here:
https://github.com/LSSTDESC/sacc/actions/runs/18748537529/job/53481737961?pr=134

click the expand button next to the red marker

[eduardojsbarroso]

You can see the error message here: https://github.com/LSSTDESC/sacc/actions/runs/18748537529/job/53481737961?pr=134

click the expand button next to the red marker

I put this file in tests but the problem is not with pytest, I just putted it here to show the problem. To get the error I am talking about, just do
python test_load_fits.py

[github-actions]

☂️ Python Coverage

current status: ✅

Overall Coverage

Lines	Covered	Coverage	Threshold	Status
1667	1479	89%	0%	🟢

New Files

No new covered files...

Modified Files

File	Coverage	Status
sacc/utils.py	87%	🟢
TOTAL	87%	🟢

updated for commit: a6ac659 by action🐍

[joezuntz]

The error doesn't appear when testing on Github actions. Could you tell me what version of numpy version you have so I can check against it?

[eduardojsbarroso]

I have numpy=1.24.4, which I chose as the one we require in TXPipe, even though I know the np.bool was depracated, but before the new sacc release It worked

[eduardojsbarroso]

I have numpy=1.24.4, which I chose as the one we require in TXPipe, even though I know the np.bool was depracated, but before the new sacc release It worked

Thank you Joe. Sorry if it was me using it wrong somehow.

[joezuntz]

No worries, this was a real issue! I'm just confused about why it shows up sometimes but not others!