Lightsheet Processing

[jesusdpa1]

Hi Eric,

I manage to established the pipeline using flowdec to deconvolved the lighthsheet images, but I am observing that for each channel and block the tensorflow enviorment is reload. Is there a way to avoid this?

Script:

data = []
total_data = []

algo = fd_restoration.RichardsonLucyDeconvolver(3).initialize()

for n in range(4):
for k in range(4):
datao = dapi[:, n*480:(n+1)480, k480:(k+1)*480]
res = algo.run(fd_data.Acquisition(data=datao, kernel=psf_single), niter=70)
data_deconvolved = res.data
data.append(data_deconvolved)
total_data.append(data)
data = []
combined_image = da.block(total_data)
combined_image.shape

Verbose:

2020-04-14 11:51:31.772567: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-04-14 11:51:31.796488: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2599990000 Hz
2020-04-14 11:51:31.796918: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55806f8d8a70 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-04-14 11:51:31.796938: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-04-14 11:51:31.863707: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-04-14 11:51:31.864106: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55806fa87b00 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-04-14 11:51:31.864120: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce RTX 2070 with Max-Q Design, Compute Capability 7.5
2020-04-14 11:51:31.864332: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-04-14 11:51:31.864642: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 2070 with Max-Q Design computeCapability: 7.5
coreClock: 1.185GHz coreCount: 36 deviceMemorySize: 7.79GiB deviceMemoryBandwidth: 357.69GiB/s
2020-04-14 11:51:31.864701: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-04-14 11:51:31.864727: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-04-14 11:51:31.864748: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-04-14 11:51:31.864769: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-04-14 11:51:31.864790: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-04-14 11:51:31.864811: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-04-14 11:51:31.864832: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-04-14 11:51:31.864879: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-04-14 11:51:31.865210: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-04-14 11:51:31.865520: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-04-14 11:51:31.865559: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-04-14 11:51:31.866593: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-04-14 11:51:31.866603: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] 0
2020-04-14 11:51:31.866607: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0: N
2020-04-14 11:51:31.866686: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-04-14 11:51:31.867018: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-04-14 11:51:31.867325: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7010 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2070 with Max-Q Design, pci bus id: 0000:01:00.0, compute capability: 7.5)
2020-04-14 11:51:32.577218: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-04-14 11:51:32.813305: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-04-14 11:51:37.893802: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-04-14 11:51:37.894299: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 2070 with Max-Q Design computeCapability: 7.5
coreClock: 1.185GHz coreCount: 36 deviceMemorySize: 7.79GiB deviceMemoryBandwidth: 357.69GiB/s
2020-04-14 11:51:37.894369: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-04-14 11:51:37.894390: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-04-14 11:51:37.894407: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-04-14 11:51:37.894424: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-04-14 11:51:37.894442: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-04-14 11:51:37.894460: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-04-14 11:51:37.894478: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-04-14 11:51:37.894538: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-04-14 11:51:37.894779: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-04-14 11:51:37.894979: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-04-14 11:51:37.894998: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-04-14 11:51:37.895003: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] 0
2020-04-14 11:51:37.895007: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0: N
2020-04-14 11:51:37.895075: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-04-14 11:51:37.895318: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-04-14 11:51:37.895524: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7010 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2070 with Max-Q Design, pci bus id: 0000:01:00.0, compute capability: 7.5)
2020-04-14 11:51:43.133267: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-04-14 11:51:43.133516: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 2070 with Max-Q Design computeCapability: 7.5
coreClock: 1.185GHz coreCount: 36 deviceMemorySize: 7.79GiB deviceMemoryBandwidth: 357.69GiB/s

[eric-czech]

Hi @jesusdpa1 , it is definitely not normal to see that, but it's hard to say for sure without the whole script. Is dapi a dask or numpy array?

[jesusdpa1]

Hi Eric, I loaded the image using the lazy dask arrays but before each loop I compute each indexed image (move to memory dask to numpy array). Here is the whole script,

lightsheet-flodwec.txt

[eric-czech]

hmm what version of TF do you have installed?

[jesusdpa1]

I have the 2.1, should I test with an older one?

[eric-czech]

Ah, well I can't say that's definitely the issue but support for TF 1.x was added in #28 though it's really only tested against 1.14.

I would try installing tensorflow-gpu==1.14.0 first and see if it goes away.

[joaomamede]

Tensorflow 2.2.0 runs on my linux machine with flowdec without a problem, or so I thought until reading this (the windows one somehow can't and I need to use an older version).

I do something very similar to your example (well, visit points loop, time points loop, and channel loop). I just realized that I have the same issue as you, I have all the cuda libraries being loaded in every cycle. However it doesn't seem to affect speed.

**Edit: tensorflow 1.14 in the windows machine does the same **