Adding Own Environment Install Error

[CameronStockton]

I have followed all the steps to add my own packages to an environment set out on this page: https://github.com/LSSTDESC/desc-python/wiki/Add-Your-Own-Packages-to-the-desc-python-Environment

I just want to install the cesium package from here: http://cesium-ml.org but the error I am getting is that I need NumPy installed in my environment and so I've installed NumPy with pip install NumPy --user and that said it was successfully installed but then I get the same error when trying to install cesium, that I need NumPy. So I did Conda install NumPy --user and that also said it was installed successfully but yet I get the same error when installing cesium. The same error also occurs when trying to install scikit-learn.

[johannct]

hi, can you provide the exact sequence of command lines you executed? In case you tried with jupyter, can you please try first with a python/ipython session?

[CameronStockton]

Here are my commands, I also attached the errors that come up (requirements.txt has all the dependencies I need to install, I can provide that if you'd like as well):
`conda create --clone desc -p /global/homes/c/cstock/mydesc
conda activate /global/homes/c/cstock/mydesc
pip install -r requirements.txt --user

ERROR: Command errored out with exit status 1:
command: /global/common/software/lsst/common/miniconda/py3.7-4.7.12.1-v2/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-l7gdy7yd/cesium/setup.py'"'"'; file='"'"'/tmp/pip-install-l7gdy7yd/cesium/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-xt470mxx --python-tag cp37
cwd: /tmp/pip-install-l7gdy7yd/cesium/
Complete output (6 lines):
To install cesium from source, you will need numpy.
Install numpy with pip:
pip install numpy
Or using conda:
conda install numpy
or use your operating system package manager.

Building wheel for scikit-learn (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: /global/common/software/lsst/common/miniconda/py3.7-4.7.12.1-v2/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-g499o_ch/scikit-learn/setup.py'"'"'; file='"'"'/tmp/pip-install-g499o_ch/scikit-learn/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-aj7pvk0s --python-tag cp37
cwd: /tmp/pip-install-g499o_ch/scikit-learn/
Complete output (19 lines):
Partial import of sklearn during the build process.
Traceback (most recent call last):
File "/tmp/pip-install-g499o_ch/scikit-learn/setup.py", line 168, in get_numpy_status
import numpy
ModuleNotFoundError: No module named 'numpy'
Traceback (most recent call last):
File "/tmp/pip-install-g499o_ch/scikit-learn/setup.py", line 148, in get_scipy_status
import scipy
ModuleNotFoundError: No module named 'scipy'
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-install-g499o_ch/scikit-learn/setup.py", line 269, in
setup_package()
File "/tmp/pip-install-g499o_ch/scikit-learn/setup.py", line 249, in setup_package
.format(numpy_req_str, instructions))
ImportError: Numerical Python (NumPy) is not installed.
scikit-learn requires NumPy >= 1.8.2.
Installation instructions are available on the scikit-learn website: http://scikit-learn.org/stable/install.html

pip install numpy --user

Installing collected packages: numpy
WARNING: The scripts f2py, f2py3 and f2py3.7 are installed in '/global/homes/c/cstock/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed numpy-1.19.5

pip install -r requirements.txt --user

(Same error as above)
conda install numpy
pip install -r requirements.txt --user

(Same error as above)
`

I'm not familiar with the NERSC environment, can you elaborate on how to try this with a python/ipython session?

[heather999]

Let's start from the beginning - we really wouldn't recommend doing pip install --user for anything as it is a bit error prone. We might need to clean that up by looking your $HOME/.local/lib/ area and removing those files, but let's try to proceed without worrying about that yet.
I would remove anything that was put into your $HOME area such as the /global/homes/c/cstock/mydesc directory and start over. after cleaning up, and starting a fresh bash shell or even logging into cori again to start a new session:

source /global/common/software/lsst/common/miniconda/setup_current_python.sh
conda create --clone desc -p /global/homes/c/cstock/mydesc
conda activate /global/homes/c/cstock/mydesc

If that is successful, then you can add additional packages - I'll edit this further, but post this now so you can get started.

At that point, you have an environment that already has numpy installed, as is scikit-learn, so you don't need to install them again. You can really just install cesium, which is available on conda-forge, by doing:
conda install -c conda-forge cesium
That will install cesium in your cloned environment that is in /global/homes/c/cstock/mydesc.
Then to make sure you continue to use your cloned environment going forward, let's say when you log back into Cori in the future, you would need to add an environment variable that points to your cloned environment. That way, every time you go to activate desc-python, it would automatically load your cloned environment, whether you do that at the Cori command line or in jupyter, making your installed packages like cesium available. So I would add to your .bashrc (assuming bash is your shell!):
export DESCUSERENV=/global/homes/c/cstock/mydesc
if you later decide you want to ignore or get rid of your /global/homes/c/cstock/mydesc you would remove that environment variable from your .basrhc and it will no longer be used when you do a source /global/common/software/lsst/common/miniconda/setup_current_python.sh and you will be using the original DESC desc-python.

[CameronStockton]

Where would I find my .bashrc? Also, if I wanted to use cesium, would the file that uses it need to be within the new folder that this environment created in my $HOME?

[heather999]

Look inside your $HOME, so do:
ls -a $HOME/.bashrc
To make sure bash is your shell, you can do: echo $SHELL and it should say something like: /bin/bash

I'm not sure about the 2nd question - are you referring to script or notebooks that use cesium? Those scripts or notebooks could be anywhere on Cori - they don't have to be located in or near your new environment.

[CameronStockton]

I ran into this problem earlier with Renee... I dont seem to have a .bashrc file. I confirmed that when I run echo $SHELL it returns bin/bash.

(/global/homes/c/cstock/mydesc_new) cstock@cori02:~> ls -a $HOME/.bashrc
ls: cannot access '/global/homes/c/cstock/.bashrc': No such file or directory

I only ask my second question because when I run the notebook that contains the cesium import it still does not run. I get "no module named 'cesium'" as my error, which has been the error the entire time.

[heather999]

I think you can go ahead and create $HOME/.bashrc and then just add this one line to it:

export DESCPYTHONPATH=/global/homes/c/cstock/mydesc
I apologize..this wasn't what I should have written.. it should have been
export DESCUSERENV=/global/homes/c/cstock/mydesc

basically, point it at the directory where you cloned desc-python.
To test that it is working - I would do a fresh login into Cori and then do:
source /global/common/software/lsst/common/miniconda/setup_current_python.sh
and part of the output from that should mention that it is pointing to your $HOME/mydesc directory.
Then I would restart all kernels in jupyter and try to access cesium again.

[CameronStockton]

I have done all these steps and it has said I have installed cesium correctly. I created a .bashrc file and added that line to it (it has since disappeared) and yet my file is still not able to import cesium correctly. I'm not getting any errors except when I run my notebook which tries to import cesium.

[cwwalter]

As a check: You will need to make sure you are using a Jupyter kernel for your custom environment.

[heather999]

Hi @CameronStockton Let's start by making sure things are working at the command line on Cori and then get things working on jupyter.
You mentioned that $HOME/.bashrc disappeared. From the Cori command line can you do:
cat $HOME/.bashrc and show the output here?
If that file does not exist, or if it doesn't contain the line:
export DESCUSERENV=/global/homes/c/cstock/mydesc
then please try doing this at the Cori command line:
echo "export DESCUSERENV=/global/homes/c/cstock/mydesc" > $HOME/.bashrc
Please note this is different than what I wrote last night (I just edited my comment from last night).. I wasn't thinking.. you want to create an environment variable named DESCUSERENV
After you do this, please open up a new Cori command line..and then do
source /global/common/software/lsst/common/miniconda/setup_current_python.sh
and can you show us here what that outputs?
From there you should be able to start up python from the Cori command line:
python
and then do an import cesium and that should work. If not, then we still have to fix something. If it's not working, I'd probably want to see the output from which python to make sure the right python installation is in use.

If after all that, it does work.. then I would open up a fresh jupyter.nersc.gov, restart any running kernels, make sure to connect to the desc-python kernel and see if you can do import cesium in a notebook.

[CameronStockton]

@heather999 Here is my output from your steps:

cstock@cori08:~> cat $HOME/.bashrc export DESCPYTHONPATH=/global/homes/c/cstock/mydesccstock@cori08:~> echo "export/.bashrcRENV=/global/homes/c/cstock/mydesc" > $HOME/
cstock@cori08:~> cat $HOME/.bashrc export DESCUSERENV=/global/homes/c/cstock/mydesc
cstock@cori08:~> source /global/common/software/lsst/common/miniconda/setup_current_python.sh Including user python path: /global/homes/c/cstock/mydesc Now using /global/common/software/lsst/common/miniconda/py3.7-4.7.12.1-v2/envs/desc/bin/python
(desc) cstock@cori08:~> python
Python 3.7.6 | packaged by conda-forge | (default, Jun 1 2020, 18:57:50)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import cesium
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'cesium'

I ran which python and here is my output:

/global/common/software/lsst/common/miniconda/py3.7-4.7.12.1-v2/envs/desc/bin/python

I also just tried to install cesium again from your command above and try to import cesium again and it did not work either.

[heather999]

It's difficult to read the above due the formatting. If you can put code in code blocks it will format correctly, see the <> button in the GitHub Issue editor. When you click that, it will insert two back-ticks and you can put code in between those characters.

After you updated your $HOME\.bashrc by doing echo export DESCUSERENV=/global/homes/c/cstock/mydesc" > $HOME/.bashrc
Did you logout and back into Cori? Please do that and then redo source /global/common/software/lsst/common/miniconda/setup_current_python.sh that output should look a little different. What you posted suggests that your current session on Cori has not been updated for the changes you made in $HOME\.bashrc

We might want to try scheduling 15 minutes on zoom to get this cleared up. I'm in US Eastern Time - so maybe you can suggest a couple times and we can see what works out.

[johannct]

@CameronStockton It would help if you could turn your working area readable.

When basic things fail as here, it is always good to go back to basic. Can you try :

source /global/common/software/lsst/common/miniconda/setup_current_python.sh
mkdir cesium_test
export PYTHONPATH=~/cesium_test/lib/python3.7/site-packages
python -c 'import cesium;print(cesium.file)'
pytest cesium_test/lib/python3.7/site-packages/cesium/tests/

and report results?

[CameronStockton]

Hi everyone, thanks so much for your help.

After restarting Cori I ran an import cesium in python in the terminal with no error! I just need help figuring out how to access this new kernel in JupyterLab. This should be the final step... I hope!

[heather999]

@CameronStockton That sounds like good news.. so to recap, it sounds like you have a $HOME/.bashrc with the contents:
export DESCUSERENV=/global/homes/c/cstock/mydesc
and when at the Cori command line, you can do:
source /global/common/software/lsst/common/miniconda/setup_current_python.sh
and if you start up python, you can successfully do import cesium

To use this in jupyter at NERSC, do you already have the DESC jupyter kernels set up? Meaning, you already have access to desc-python? Have you already followed the steps here to set up the DESC kernels? Specifically, if you haven't already, you need to run this script once:
source /global/common/software/lsst/common/miniconda/kernels/setup.sh
if you have done that already, there is no need to do it again.
Once that is done, you will have access to DESC jupyter kernels when you visit jupyter.nersc.gov and when you start up jupyter, you can choose desc-python as your kernel and then you should have access to both the DESC python environment, plus cesium.