Error when loading Python .so (Linux)

[RoyVII]

Hi everybody,

First of all, thank you very much for this plugin, is really useful!

I'm having some trouble when using the PythonPlugin to load a compiled Python code .so file. I get the following output

Python Filter updating settings.
Python Filter setting num outputs to 0
Python event channel created
init function is: PyInit_test
loaded pluginStartup 
 
 
 Traceback (most recent call last):
  File "__init__.pxd", line 164, in init test
ModuleNotFoundError: No module named 'numpy'

The above exception was the direct cause of the following exception:

SystemError: <built-in method write of _io.TextIOWrapper object at 0x7febdc3e8630> returned a result with an error set
Exception ignored in: 'test.pluginStartup'
SystemError: <built-in method write of _io.TextIOWrapper object at 0x7febdc3e8630> returned a result with an error set
NameError: name 'pluginOp' is not defined
Exception ignored in: 'test.getParamNum'
NameError: name 'pluginOp' is not defined
NameError: name 'pluginOp' is not defined
Exception ignored in: 'test.getParamConfig'
NameError: name 'pluginOp' is not defined

I'm using the code from the new_cmake_build branch and had no problem compiling the plugin. I'm also using an Anaconda environment with Python 3.6, Numpy 1.18.1 and Cython 0.28.2 and setting export CONDA_HOME=/home/anaconda3/envs/oeEnv. This is part of the CMake output:

-- Found PythonInterp: /home/anaconda3/envs/oeEnv/bin/python (found version "3.6.10") 
-- Found PythonLibs: /home/anaconda3/envs/oeEnv/lib/libpython3.6m.so (found version "3.6.10") 
-- Python home: /home/anaconda3/envs/oeEnv
-- Python executable: /home/anaconda3/envs/oeEnv/bin/python
-- Python library: /home/anaconda3/envs/oeEnv/lib/libpython3.6m.so
-- Python lib dir: /home/anaconda3/envs/oeEnv/lib
-- Python include dir: /home/anaconda3/envs/oeEnv/include/python3.6m

The Python module that I'm trying to load is just the template. I get the deprecated numpy warning when compiling but that's all.

I understand that the problem is related to the plugin not using the right environment so it does not find numpy, but I have oeEnv activated when launching Open Ephys and have even used conda run -n oeEnv ./open-ephys to be sure. If I add the following lines to the PythonPlugin::SetFile function

std::cout << "Python version " << PY_MAJOR_VERSION << "." << PY_MINOR_VERSION << " ";
system("python -c \"import numpy; print('Numpy version', numpy.version.version)\"");

the output is

Python Filter updating settings.
Python Filter setting num outputs to 0
Python event channel created
init function is: PyInit_test
Numpy version 1.18.1
Python version 3.6 loaded pluginStartup

Traceback (most recent call last):
  File "__init__.pxd", line 164, in init test
ModuleNotFoundError: No module named 'numpy'

so I don't really get what is going on.

Do you have any idea of what might be causing this?
Thank you very much!

Rodrigo

[Claybarn]

Sorry for the very late response. I have had some trouble myself getting the python plugin to work with the new version of open ephys. Your error is strange, I will ask around and see if there is any reason for it. Also it has been awhile since you posted, so let me know if you have made any progress!

[samuelgarcia]

Hi.
I have the same error.
I am trying to install on ubuntu 20.4 with python3.8 and I am not able to do it anymore.

I suspect that PYTHON_PATH is not set correctly at some steps.

Note that I do not use conda but virtualenv+pip but I have the exact problem with the same branch.

Also note that you have to change the hidden file .config.py to be able to compile some python modules.

[samuelgarcia]

@RoyVII : did you find a solution ?

[samuelgarcia]

Does someone known why here https://github.com/NeuroNetMem/PythonPlugin/blob/master/PythonPlugin/PythonPlugin.cpp#L694
thePYTHONPATH is set only for windows and not for linux ?
Maybe this is the reason. Not sure at all.

[RoyVII]

Sorry, I could not find a solution to this and I ran out of ideas...

[samuelgarcia]

@Claybarn @fpbattaglia
Hi here.

I am still fighting to install the pluguin with branch new_cmake_build branch.
I basically have the same error as describe here.
I think there is a problem the PYTHON_PATH / Cython / C++ integration.

I am able to compile C++.
I am able to compile ythe cython code
I am NOT able to run it due to strange python path that do not reconize "numpy". Meaning that environement is not loaded.

Note that in the branch *new_cmake_build the .config.py have to be changed.

Does anyone is able to compile some python and run it ?

Any help from some dev would appreciated.
Does a move toward cython 0.29 would help ?
What is missing for cython 0.29. I can help.

[Claybarn]

Hey @samuelgarcia. I have been busy recently and have not been able to dedicate the time to get this new version of the plugin off the ground. I ran into the same problem you are experiencing. I too thought there was something wrong with the python path, I am not sure what happened. I can try to make some time in the near future. What have you tried thus far? As for cython 0.29, I do not remember what the issue was. That was awhile ago. Can give it a shot! But imagine you will run into the same problem.

[samuelgarcia]

Hi @Claybarn

I finally manage to run the python plugin.

I am trying to run it with:

• python3.8
• on ubuntu 20.04
• with pip installed pacake (no conda)
• in a virtual envorvement in my home.
• plugin-GUI version 0.5 release 3 day ago/

Here some important details:

• PYTHON_PATH have to be set before cmake .. (there is a trick for win installation when not set but not for linux one)
• pip have to be very up to date with pip install --upgrade pip before installing any package in the env.
• pacakge that do a source installation (with .egg in theXXX/lib/python3.8/site-packages folder) do not work.
  This is is old fashion of installing packages. Now the folder is copied.
• package that are installed with "setuptools" instead of of "distuils" are NOT importable. I needed to modify the setup.py of
  pyacq for instance.

It took me several hours to discover this. So I share it.

Note that I have started a PR here to fix the cmake installation.
#31
Please have a look.

I will also put some important details in the README if you are OK.

Do you plan to merger th branch "new_cmake_build" into master soon ?

[Claybarn]

Hey @samuelgarcia thank you for this! I will see if I can get it running with your instructions. Feel free to add to the README, and we certainly want to merge new_cmake_build into master soon.

[Claybarn]

It seems like the file structure of OE has changed considerably, I am having to change a good amount of paths. @samuelgarcia did you also encounter this? Also, we will have to get the plugin working with GitHub workflows and should adopt the file structure of the plugins on open-ephys-plugins.

[samuelgarcia]

No I did change path manually. With the cmake, all path have been set automagically for me.
Note that in my case the PythonPlugin and plugin-Gui are in the same folder. I guess this is the same for you.

The branch new_cmake_build have already more or less the good folder structure no ?