Makefile incorrectly searches numpy

[VictorEijkhout]

I get an error on numpy headers not found, so I edit line 518 in common.mk to

            $(error Required ${NUMPY_INCL_PATH} $(ccred)numpy headers$(ccreset) are missing...stopp\ing the compilation. You might be able to fix this by installing $(ccblue)numpy-devel$(ccreset))

which outputs

../common.mk:518: *** Required /work2/00434/eijkhout/stampede3/python/installation-3.11.0/numpy/core/include/numpy/ numpy headers are missing...stopping the compilation. You might be able to fix this by installing numpy-devel.  Stop.
make[2]: Leaving directory '/work2/00434/eijkhout/corrfunc/corrfunc-2.5.2/theory'
make[1]: *** [Makefile:9: theory] Error 2
make[1]: Leaving directory '/work2/00434/eijkhout/corrfunc/corrfunc-2.5.2'

so I go searching for that directory:

[staff corrfunc:96] ls /work2/00434/eijkhout/stampede3/python/installation-3.11.0/numpy/core/include/numpy/
ls: cannot access '/work2/00434/eijkhout/stampede3/python/installation-3.11.0/numpy/core/include/numpy/': No such file or directory
[staff corrfunc:97] ls /work2/00434/eijkhout/stampede3/python/installation-3.11.0/numpy/core/include/
ls: cannot access '/work2/00434/eijkhout/stampede3/python/installation-3.11.0/numpy/core/include/': No such file or directory
[staff corrfunc:98] ls /work2/00434/eijkhout/stampede3/python/installation-3.11.0/numpy/core/
arrayprint.py    _dtype_ctypes.py  einsumfunc.py   function_base.py  __init__.py   _internal.py   _multiarray_umath.py  numerictypes.py  __pycache__/  shape_base.py  _utils.py
defchararray.py  _dtype.py         fromnumeric.py  getlimits.py      __init__.pyi  multiarray.py  numeric.py            overrides.py     records.py    umath.py

Are you sure that path should exist?

[VictorEijkhout]

[staff corrfunc:100] find $TACC_PYTHON_DIR -name arrayobject.h
/work2/00434/eijkhout/stampede3/python/installation-3.11.0/numpy/_core/include/numpy/arrayobject.h

[lgarrison]

I think that's a Numpy 2 path. Can you try with Numpy < 2?

[VictorEijkhout]

      In file included from /tmp/pip-build-env-2zutweuw/normal/lib/python3.11/site-packages/numpy/_core/include/numpy/ndarrayobject.h:22,
                       from /tmp/pip-build-env-2zutweuw/normal/lib/python3.11/site-packages/numpy/_core/include/numpy/arrayobject.h:5,
                       from _countpairs.c:18:
      /tmp/pip-build-env-2zutweuw/normal/lib/python3.11/site-packages/numpy/_core/include/numpy/__multiarray_api.h:1475:10: fatal error: numpy/_public_dtype_api_table.h: No such file or directory
       1475 | #include "numpy/_public_dtype_api_table.h"
            |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

but

/work2/00434/eijkhout/stampede3/python/installation-3.11.0-gcc/numpy/_core/include/numpy/_public_dtype_api_table.h

so I think I need to adjust an include path.

[VictorEijkhout]

Here is the compile line:

gcc -DVERSION=\"2.5.2\" -DUSE_UNICODE -std=c99 -g -Wsign-compare -Wall -Wextra -Wshadow -Wunused -fPIC -D_POSIX_SOURCE=200809L -D_GNU_SOURCE -O3 -ftree-vectorize -funroll-loops -fprefetch-loop-arrays --param simultaneous-prefetches=4 -fopenmp -funroll-loops -fno-strict-aliasing -Wformat=2 -Wpacked -Wnested-externs -Wpointer-arith -Wredundant-decls -Wfloat-equal -Wcast-qual -Wcast-align -Wmissing-declarations -Wmissing-prototypes -Wnested-externs -Wstrict-prototypes -Wno-unused-local-typedefs -I../../theory/DD -I../../theory/DDrppi -I../../theory/wp -I../../theory/xi -I../../theory/DDsmu -I../../theory/vpf -isystem/work2/00434/eijkhout/stampede3/python/installation-3.11.0-gcc/include/python3.11 -isystem /work2/00434/eijkhout/stampede3/python/installation-3.11.0-gcc/numpy/_core/include/numpy/ -I../../io -I../../utils _countpairs.c -c -o _countpairs.o
In file included from /work2/00434/eijkhout/stampede3/python/installation-3.11.0-gcc/numpy/_core/include/numpy/ndarrayobject.h:22,
                 from /work2/00434/eijkhout/stampede3/python/installation-3.11.0-gcc/numpy/_core/include/numpy/arrayobject.h:5,
                 from _countpairs.c:18:
/work2/00434/eijkhout/stampede3/python/installation-3.11.0-gcc/numpy/_core/include/numpy/__multiarray_api.h:1475:10: fatal error: numpy/_public_dtype_api_table.h: No such file or directory
 1475 | #include "numpy/_public_dtype_api_table.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I don't understand who constructs this.

[manodeep]

Hmm - this definitely looks like a numpy issue - similar to this one. May be this suggestion could solve the issue?

That should be #include <numpy/arrayobject.h>, not #include <arrayobject.h>.

Will you please confirm that that fix works?

[manodeep]

@VictorEijkhout What version of numpy are you using?

[VictorEijkhout]

Numpy 2.0.1

And that fix makes things worse:

make[3]: Entering directory '/work2/00434/eijkhout/corrfunc/corrfunc-2.5.2/theory/python_bindings'
gcc -DVERSION=\"2.5.2\" -DUSE_UNICODE -std=c99 -g -Wsign-compare -Wall -Wextra -Wshadow -Wunused -fPIC -D_POSIX_SOURCE=200809L -D_GNU_SOURCE -O3 -ftree-vectorize -funroll-loops -fprefetch-loop-arrays --param simultaneous-prefetches=4 -fopenmp -funroll-loops -fno-strict-aliasing -Wformat=2 -Wpacked -Wnested-externs -Wpointer-arith -Wredundant-decls -Wfloat-equal -Wcast-qual -Wcast-align -Wmissing-declarations -Wmissing-prototypes -Wnested-externs -Wstrict-prototypes -Wno-unused-local-typedefs -I../../theory/DD -I../../theory/DDrppi -I../../theory/wp -I../../theory/xi -I../../theory/DDsmu -I../../theory/vpf -isystem/work2/00434/eijkhout/stampede3/python/installation-3.11.0-gcc/include/python3.11 -isystem /work2/00434/eijkhout/stampede3/python/installation-3.11.0-gcc/lib/python3.11/site-packages/numpy/_core/include/numpy/ -I../../io -I../../utils _countpairs.c -c -o _countpairs.o
_countpairs.c:18:10: fatal error: numpy/arrayobject.h: No such file or directory
   18 | #include <numpy/arrayobject.h>
      |          ^~~~~~~~~~~~~~~~~~~~~

I think I need to add an include path, but I can't find where you set those.

[VictorEijkhout]

My fix:

          export NUMPY_INCL_FLAG := $(shell $(PYTHON) -c "from __future__ import print_function; im\port numpy; print('-isystem ' + numpy.__path__[0] + '/_core/include/numpy/' )" )
          export NUMPY_INCL_FLAG2 := $(shell $(PYTHON) -c "from __future__ import print_function; i\mport numpy; print('-isystem ' + numpy.__path__[0] + '/_core/include/' )" )
[ ... stuff .... ]
          endif
          export NUMPY_CHECKED:=1
        endif

        export PYTHON_CFLAGS := $(PYTHON_INCL) $(NUMPY_INCL_FLAG) $(NUMPY_INCL_FLAG2)

[lgarrison]

Corrfunc hasn't yet been updated to support NumPy 2, but I took a quick stab at it and it seems to work (it builds and passes tests). I put it here if you'd like to try it: https://github.com/manodeep/Corrfunc/tree/numpy-2

It breaks NumPy 1 build support, which I'm not sure we want to do on main yet, so I'm going to leave it in a separate branch for now.

Note that in that branch NumPy 2 is just a build dependency, and it should run with a recent NumPy 1 as well.

[manodeep]

Numpy 2.0.1

And that fix makes things worse:

make[3]: Entering directory '/work2/00434/eijkhout/corrfunc/corrfunc-2.5.2/theory/python_bindings'
gcc -DVERSION=\"2.5.2\" -DUSE_UNICODE -std=c99 -g -Wsign-compare -Wall -Wextra -Wshadow -Wunused -fPIC -D_POSIX_SOURCE=200809L -D_GNU_SOURCE -O3 -ftree-vectorize -funroll-loops -fprefetch-loop-arrays --param simultaneous-prefetches=4 -fopenmp -funroll-loops -fno-strict-aliasing -Wformat=2 -Wpacked -Wnested-externs -Wpointer-arith -Wredundant-decls -Wfloat-equal -Wcast-qual -Wcast-align -Wmissing-declarations -Wmissing-prototypes -Wnested-externs -Wstrict-prototypes -Wno-unused-local-typedefs -I../../theory/DD -I../../theory/DDrppi -I../../theory/wp -I../../theory/xi -I../../theory/DDsmu -I../../theory/vpf -isystem/work2/00434/eijkhout/stampede3/python/installation-3.11.0-gcc/include/python3.11 -isystem /work2/00434/eijkhout/stampede3/python/installation-3.11.0-gcc/lib/python3.11/site-packages/numpy/_core/include/numpy/ -I../../io -I../../utils _countpairs.c -c -o _countpairs.o
_countpairs.c:18:10: fatal error: numpy/arrayobject.h: No such file or directory
   18 | #include <numpy/arrayobject.h>
      |          ^~~~~~~~~~~~~~~~~~~~~

I think I need to add an include path, but I can't find where you set those.

Ahh never mind then. Looks like Lehman has a potential fix for numpy2

@lgarrison I made a suggestion on the commit - will you be able to check whether my suggestion works for both numpy1 and numpy2?

[manodeep]

@VictorEijkhout I merged in the numpy2 capability and made a new release. Do you mind checking whether the code now compiles with numpy2 (either from source or from the PyPI repo with Corrfunc v2.5.3)?