karel911 / tracer Goto Github PK

TRACER: Extreme Attention Guided Salient Object Tracing Network (AAAI 2022) implementation in PyTorch

License: Apache License 2.0

Python 99.76% Shell 0.24%

pytorch aaai2022 aaai-2022 pytorch-implementation image-segmentation salient-object-detection attention attention-mechanism background-removal

tracer's Introduction

TRACER: Extreme Attention Guided Salient Object Tracing Network

This paper was accepted at AAAI 2022 SA poster session. [pdf]

Updates

[09/06/2022] Demo has been released on Try it now!

[06/17/2022] Now, fast inference mode offers a salient object result with the mask.
We have improved a result quality of salient object as follows.
You can get the more clear salient object by tuning the threshold. We will release initializing TRACER with a version of pre-trained TE-x.

[04/20/2022] We update a pipeline for custom dataset inference w/o measuring.

Run main.py scripts.


TRACER
├── data
│   ├── custom_dataset
│   │   ├── sample_image1.png
│   │   ├── sample_image2.png
      .
      .
      .

# For testing TRACER with pre-trained model (e.g.)  
python main.py inference --dataset custom_dataset/ --arch 7 --img_size 640 --save_map True

Datasets

All datasets are available in public.

Download the DUTS-TR and DUTS-TE from Here
Download the DUT-OMRON from Here
Download the HKU-IS from Here
Download the ECSSD from Here
Download the PASCAL-S from Here
Download the edge GT from Here.

Data structure


TRACER
├── data
│   ├── DUTS
│   │   ├── Train
│   │   │   ├── images
│   │   │   ├── masks
│   │   │   ├── edges
│   │   ├── Test
│   │   │   ├── images
│   │   │   ├── masks
│   ├── DUT-O
│   │   ├── Test
│   │   │   ├── images
│   │   │   ├── masks
│   ├── HKU-IS
│   │   ├── Test
│   │   │   ├── images
│   │   │   ├── masks
      .
      .
      .

Requirements

Python >= 3.7.x
Pytorch >= 1.8.0
albumentations >= 0.5.1
tqdm >=4.54.0
scikit-learn >= 0.23.2

Run

Run main.py scripts.


# For training TRACER-TE0 (e.g.)
python main.py train --arch 0 --img_size 320

# For testing TRACER with pre-trained model (e.g.)  
python main.py test --exp_num 0 --arch 0 --img_size 320

Pre-trained models of TRACER are available at here
Change the model name as 'best_model.pth' and put the weights to the path 'results/DUTS/TEx_0/best_model.pth'
(here, the x means the model scale e.g., 0 to 7).
Input image sizes for each model are listed belows.

Configurations

--arch: EfficientNet backbone scale: TE0 to TE7.
--frequency_radius: High-pass filter radius in the MEAM.
--gamma: channel confidence ratio \gamma in the UAM.
--denoise: Denoising ratio d in the OAM.
--RFB_aggregated_channel: # of channels in receptive field blocks.
--multi_gpu: Multi-GPU learning options.
--img_size: Input image resolution.
--save_map: Options saving predicted mask.

Model	Img size
TRACER-Efficient-0 ~ 1	320
TRACER-Efficient-2	352
TRACER-Efficient-3	384
TRACER-Efficient-4	448
TRACER-Efficient-5	512
TRACER-Efficient-6	576
TRACER-Efficient-7	640

Citation


@article{lee2021tracer,
  title={TRACER: Extreme Attention Guided Salient Object Tracing Network},
  author={Lee, Min Seok and Shin, WooSeok and Han, Sung Won},
  journal={arXiv preprint arXiv:2112.07380},
  year={2021}
}

tracer's People

Contributors

Stargazers

Watchers

tracer's Issues

Adaptive pixel intensity loss

hi,thank you for your excellent work，I don't understand the following formula. Can you explain it for me？

1、Why does equation 8 need to be multiplied by yij? This will cause all background weights to be 0。
2、What is the meaning of adding 1.5 to the denominator in formula 9?
Thank you very much and hope to get your reply！！

Error after few iteration on training

I want to train the network --arch 7 with my custom 62k dataset that is similar to DUTS. I am using 48GB CUDA and batch size 8. After a few iteration, I am getting the following error
Traceback (most recent call last):
File "main.py", line 55, in
main(args)
File "main.py", line 35, in main
Trainer(args, save_path)
File "/root/TRACER/trainer.py", line 58, in init
train_loss, train_mae = self.training(args)
File "/root/TRACER/trainer.py", line 117, in training
loss.backward()
File "/root/TRACER/venv/lib/python3.6/site-packages/torch/_tensor.py", line 307, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "/root/TRACER/venv/lib/python3.6/site-packages/torch/autograd/init.py", line 156, in backward
allow_unreachable=True, accumulate_grad=True) # allow_unreachable flag
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

Mask Generation?

Hello! And thank you for this work. I was curious if you would be releasing or amending (or if I missed it in your paper) the mask generation part of your testing code? To try against more complicated imagery in visual form?

Pretrained weight

Thank you for your great work. Can I know what dataset you used to train the weight that you are public?

does TRACER pretrained model used for training?

Hi,
I downloaded pre-trained models of TRACER as you mentioned in README, but I'm confused about the step below
Change the model name as 'best_model.pth' and put the weights to the path 'results/DUTS/TEx_0/best_model.pth'

question is, this path is model save path, not pretrained model load path, and I could not find you load "best_model.pth" at start of training anywhere in your code, did I miss something?

Custom Data Train Error

Hi, when i try to train on a custom dataset I got this error. @Karel911

C:/cb/pytorch_1000000000000/work/aten/src/ATen/native/cuda/Loss.cu:102: block: [0,0,0], thread: [0,0,0] Assertion input_val >= zero && input_val <= one failed.
C:/cb/pytorch_1000000000000/work/aten/src/ATen/native/cuda/Loss.cu:102: block: [0,0,0], thread: [1,0,0] Assertion input_val >= zero && input_val <= one failed.
C:/cb/pytorch_1000000000000/work/aten/src/ATen/native/cuda/Loss.cu:102: block: [0,0,0], thread: [2,0,0] Assertion input_val >= zero && input_val <= one failed.
C:/cb/pytorch_1000000000000/work/aten/src/ATen/native/cuda/Loss.cu:102: block: [0,0,0], thread: [3,0,0] Assertion input_val >= zero && input_val <= one failed.
C:/cb/pytorch_1000000000000/work/aten/src/ATen/native/cuda/Loss.cu:102: block: [0,0,0], thread: [4,0,0] Assertion input_val >= zero && input_val <= one failed.
C:/cb/pytorch_1000000000000/work/aten/src/ATen/native/cuda/Loss.cu:102: block: [0,0,0], thread: [5,0,0] Assertion input_val >= zero && input_val <= one failed.
C:/cb/pytorch_1000000000000/work/aten/src/ATen/native/cuda/Loss.cu:102: block: [0,0,0], thread: [6,0,0] Assertion input_val >= zero && input_val <= one failed.
C:/cb/pytorch_1000000000000/work/aten/src/ATen/native/cuda/Loss.cu:102: block: [0,0,0], thread: [7,0,0] Assertion input_val >= zero && input_val <= one failed.
C:/cb/pytorch_1000000000000/work/aten/src/ATen/native/cuda/Loss.cu:102: block: [0,0,0], thread: [8,0,0] Assertion input_val >= zero && input_val <= one failed.
C:/cb/pytorch_1000000000000/work/aten/src/ATen/native/cuda/Loss.cu:102: block: [0,0,0], thread: [9,0,0] Assertion input_val >= zero && input_val <= one failed.
Traceback (most recent call last):
line 2762, in binary_cross_entropy
return torch._C._nn.binary_cross_entropy(input, target, weight, reduction_enum)
RuntimeError: CUDA error: device-side assert triggered

How can i fix this? thanks

Testing on test images without mask

I have few test images without mask. I just want to pass it to the model and get the output. How can I do it?

System specifications

Hey can you share us the system specifications? I tried training it on EFFicient net v3 and found that it can only ru in a batchsize of 2 or 3 on Tesla T4.

Export to ONNX

How would one export this model to ONNX? It keeps hanging for me while trying to export it

Changing black background to transparent

Hello, thanks for the code and paper. Amazing work! I was doing some tests and the model works very great. I wanted to see the results when background is transparent but i realized there is a black hue near edges when i applied the code below to change the background. I know that your project is RGB based but i wonder is there a workaround for me to get the results without black background or is it impossible? Thanks in advance.

img is the result of the model.
color = (0, 0, 0)
mask = np.where((img == color).all(axis=2), 0, 255).astype(np.uint8)
result = img.copy()
result = cv2.cvtColor(result, cv2.COLOR_BGR2BGRA)
result[:, :, 3] = mask

model result:

code result:

original image:

About the edge folder

I notice that there is a edge folder in dataset, however, I didn't find this folder of DUTS or DUTO.
How to generate these edge folders?

Accessing The Actual Predicited Masks

Hi, I tried to replicate your code and managed to replicate the test on the DUTS test set using TE0_0 pretrained model. However I would like to know how to get the actual masks predicted of the images after running the test. Thanks, Malcolm

Test on my data

Thanks for your great work! How can I use the trained model to test when I got the best model after training on my data？ Because it seems that the pre training model is still used in the inference stage?

the API loss

Hi, sorry to bother you. I use my own dataset to train the model and I have faced the same trouble like #9
and I found the "amae" in the TRACER/util/losses.py was so big. Is this normal? 😔

Your code is

def adaptive_pixel_intensity_loss(pred, mask):
    w1 = torch.abs(F.avg_pool2d(mask, kernel_size=3, stride=1, padding=1) - mask)
    w2 = torch.abs(F.avg_pool2d(mask, kernel_size=15, stride=1, padding=7) - mask)
    w3 = torch.abs(F.avg_pool2d(mask, kernel_size=31, stride=1, padding=15) - mask)

    omega = 1 + 0.5 * (w1 + w2 + w3) * mask

    bce = F.binary_cross_entropy(pred, mask, reduce=None)
    abce = (omega * bce).sum(dim=(2, 3)) / (omega + 0.5).sum(dim=(2, 3))

    inter = ((pred * mask) * omega).sum(dim=(2, 3))
    union = ((pred + mask) * omega).sum(dim=(2, 3))
    aiou = 1 - (inter + 1) / (union - inter + 1)

    mae = F.l1_loss(pred, mask, reduce=None)
    amae = (omega * mae).sum(dim=(2, 3)) / (omega - 1).sum(dim=(2, 3))

    return (0.7 * abce + 0.7 * aiou + 0.7 * amae).mean()

In your article:

I want to know in your code, how do you use H*W in the denominator in Eq. 11?
Thank you very much and look forward to your reply!!

Can you upload the result file to test the effect?

Can you upload the result file for evaluating the performance?

Reproduce you result on TE1. The MAE is much worse than yours on DUTS-TE

Hi,
I tried your code to reproduce your result.
I used python main.py train --arch 1 --img_size 320 to train the model. I did not change any setting in your code. I used four GPU to train this model. But the MAE I got on DUTS-TE is around 0.088 (it is 0.033 in paper). In picture, it shows the loss and the MAE of train and val set when training. It early stopped at 82th epoch. It kind of converged.

I would like to know did you do any data augmentation off-line (I train on DUTS-TR and the edge gt you provided. I simply change the folder directory to meet your folder structure mentioned in readme)? Should I change any hyper parameter to achieve the same result?

Licence

Hi, thanks for great job. Can you provide a licence?

API_loss for multi_classes semantic segmentation

Hello,

I would like to apply your API_loss to my 3_classes semantic segmentation task, but i am not sure it will be work or not after some modification?
Do you have any advice? Look forward to your reply~

Thanks, best wish!

Error while trying to Train in a DataParallel setting on 4 GPUs

Hello,
Thank you for your amazing work. I had an issue while trying to train the model on 4 GPUs. The trace is as shown below. I however was able to train the model with one GPU.
Could you please help me out with this

Traceback (most recent call last):
File "/home/ec2-user/SageMaker/segment-anything/TRACER/main.py", line 55, in
main(args)
File "/home/ec2-user/SageMaker/segment-anything/TRACER/main.py", line 35, in main
Trainer(args, save_path)
File "/home/ec2-user/SageMaker/segment-anything/TRACER/trainer.py", line 56, in init
train_loss, train_mae = self.training(args)
File "/home/ec2-user/SageMaker/segment-anything/TRACER/trainer.py", line 105, in training
outputs, edge_mask, ds_map = self.model(images)
File "/home/ec2-user/anaconda3/envs/new_fast_env/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/ec2-user/anaconda3/envs/new_fast_env/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ec2-user/anaconda3/envs/new_fast_env/lib/python3.9/site-packages/torch/nn/parallel/data_parallel.py", line 185, in forward
outputs = self.parallel_apply(replicas, inputs, module_kwargs)
File "/home/ec2-user/anaconda3/envs/new_fast_env/lib/python3.9/site-packages/torch/nn/parallel/data_parallel.py", line 200, in parallel_apply
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
File "/home/ec2-user/anaconda3/envs/new_fast_env/lib/python3.9/site-packages/torch/nn/parallel/parallel_apply.py", line 110, in parallel_apply
output.reraise()
File "/home/ec2-user/anaconda3/envs/new_fast_env/lib/python3.9/site-packages/torch/_utils.py", line 694, in reraise
raise exception
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
File "/home/ec2-user/anaconda3/envs/new_fast_env/lib/python3.9/site-packages/torch/nn/parallel/parallel_apply.py", line 85, in _worker
output = module(*input, **kwargs)
File "/home/ec2-user/anaconda3/envs/new_fast_env/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/ec2-user/anaconda3/envs/new_fast_env/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ec2-user/SageMaker/segment-anything/TRACER/model/TRACER.py", line 38, in forward
features, edge = self.model.get_blocks(x, H, W)
File "/home/ec2-user/SageMaker/segment-anything/TRACER/model/EfficientNet.py", line 245, in get_blocks
x = block(x, drop_connect_rate=drop_connect_rate)
File "/home/ec2-user/anaconda3/envs/new_fast_env/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/ec2-user/anaconda3/envs/new_fast_env/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ec2-user/SageMaker/segment-anything/TRACER/model/EfficientNet.py", line 122, in forward
x_squeezed = self._se_reduce(x_squeezed)
File "/home/ec2-user/anaconda3/envs/new_fast_env/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/ec2-user/anaconda3/envs/new_fast_env/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ec2-user/SageMaker/segment-anything/TRACER/util/effi_utils.py", line 301, in forward
x = F.conv2d(x, self.weight, self.bias, self.stride, self.padding, self.dilation, self.groups)
RuntimeError: GET was unable to find an engine to execute this computation

U2Net benchmark comparison

Hey! Loved the paper.

Have you tried doing a comparison to U2Net? As for now, it's considered the state-of-the-art in SOD. I was wondering if you tested against it.

how to create the edge mask

Can you please share the code for creating the edge data from the alpha layer, so that we could retrain it with our own data
Thanks!

Reproducing paper results of TE7

Tried training using the same hyper params of the paper in order to reproduce results on the DUTS-TE split for example.
Apart from batch size which had to be reduced to 8, all other params were the same.
However, I reach inferior results:

Dataset:DUTS
Test Loss:1.457 | MAX_F:0.903 | AVG_F:0.873 | MAE:0.030 | S_Measure:0.891, time: 68938.211s

This is the training command I used:

python main.py train --exp_num 1 --arch 7 --img_size 640 --model_path /data/outputs/TRACER --data_path /data/datasets/TRACER/ --multi_gpu true --num_workers 12 --batch_size 8

The best model was achieved at epoch 14 as stated in the logs:

Best Val Epoch:14 | Val Loss:38.960 | Val MAE:0.019

And the training concluded after epoch 24 due to early stopping mechanism.

Hardware specs to reproduce the training results

What hardware was used for the training to reproduce the results in the paper?

How do I use my train best_model.pt ?

Hello,
How do I use the best_model.pt after running the train code? Is it going to use it by itself? if not how do I tell it goes to the results/custom_dataset/ My inference keeps loading the pre-train model, which I don't want.

Losses decrease quite slowly with custom dataset

I created a custom dataset and tried training with arch 5 with image size 512 but the loss decreased quite slowly. Is this normal, or is my data problem?

my loss
Epoch:[001/200]
Train Loss:99.721 | MAE:0.189
Valid Loss:82.747 | MAE:0.125
=============================Epoch:[002/200]
Train Loss:91.156 | MAE:0.156
Valid Loss:91.560 | MAE:0.166
=============================Epoch:[003/200]
Train Loss:91.532 | MAE:0.155
Valid Loss:78.634 | MAE:0.106
=============================Epoch:[004/200]
Train Loss:86.963 | MAE:0.135
Valid Loss:76.247 | MAE:0.095
=============================Epoch:[005/200]
Train Loss:85.692 | MAE:0.130
Valid Loss:74.397 | MAE:0.086
=============================Epoch:[006/200]
Train Loss:85.188 | MAE:0.127
Valid Loss:74.988 | MAE:0.086
=============================Epoch:[007/200]
Train Loss:84.972 | MAE:0.125
Valid Loss:73.414 | MAE:0.082
=============================Epoch:[008/200]
Train Loss:83.838 | MAE:0.121
Valid Loss:75.131 | MAE:0.087
=============================Epoch:[009/200]
Train Loss:83.403 | MAE:0.119

Custom Data

Hi, I'm getting this error. @Karel911

<---- Training Params ---->
Namespace(RFB_aggregated_channel=[32, 64, 128], action='train', arch='7', aug_ver=1, batch_size=8, channels=[24, 40, 112, 320], clipping=2, criterion='API', data_path='/home/nagas/TRACER/data', dataset='DUTS', denoise=0.93, epochs=100, exp_num=0, frequency_radius=16, gamma=0.1, img_size=640, lr=5e-05, lr_factor=0.1, model_path='/home/nagas/TRACER/results', multi_gpu=True, num_workers=4, optimizer='Adam', patience=5, save_map=None, scheduler='Reduce', seed=42, weight_decay=0.0001)
train length : 42
val length : 3
Loaded pretrained weights for efficientnet-b7
0%| | 0/5 [00:00<?, ?it/s]ERROR: Unexpected segmentation fault encountered in worker.
ERROR: Unexpected segmentation fault encountered in worker.
ERROR: Unexpected segmentation fault encountered in worker.
ERROR: Unexpected segmentation fault encountered in worker.
0%| | 0/5 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/home/nagas/miniconda3/envs/tracer/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 990, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/home/nagas/miniconda3/envs/tracer/lib/python3.7/multiprocessing/queues.py", line 104, in get
if not self._poll(timeout):
File "/home/nagas/miniconda3/envs/tracer/lib/python3.7/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/home/nagas/miniconda3/envs/tracer/lib/python3.7/multiprocessing/connection.py", line 414, in _poll
r = wait([self], timeout)
File "/home/nagas/miniconda3/envs/tracer/lib/python3.7/multiprocessing/connection.py", line 921, in wait
ready = selector.select(timeout)
File "/home/nagas/miniconda3/envs/tracer/lib/python3.7/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/home/nagas/miniconda3/envs/tracer/lib/python3.7/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 20604) is killed by signal: Segmentation fault.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "main.py", line 56, in
main(args)
File "main.py", line 35, in main
Trainer(args, save_path)
File "/home/nagas/TRACER/trainer.py", line 56, in init
train_loss, train_mae = self.training(args)
File "/home/nagas/TRACER/trainer.py", line 101, in training
for images, masks, edges in tqdm(self.train_loader):
File "/home/nagas/miniconda3/envs/tracer/lib/python3.7/site-packages/tqdm/std.py", line 1185, in iter
for obj in iterable:
File "/home/nagas/miniconda3/envs/tracer/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 521, in next
data = self._next_data()
File "/home/nagas/miniconda3/envs/tracer/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1186, in _next_data
idx, data = self._get_data()
File "/home/nagas/miniconda3/envs/tracer/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1152, in _get_data
success, data = self._try_get_data()
File "/home/nagas/miniconda3/envs/tracer/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1003, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
RuntimeError: DataLoader worker (pid(s) 20604) exited unexpectedly

GPU memory questions

Hello.
Thank you for your wonderful research.

I have a simple question.
I want to know the GPU memory usage for this research.
Thank you.

Does not support negative images in training

I put some images which does not have label in it i.e completely black mask with respect to those images and the output of it is tensors having nan's instead of 0's

outputs----- tensor([[[[nan, nan, nan, ..., nan, nan, nan], [nan, nan, nan, ..., nan, nan, nan], [nan, nan, nan, ..., nan, nan, nan], ..., ..., [nan, nan, nan, ..., nan, nan, nan], [nan, nan, nan, ..., nan, nan, nan], [nan, nan, nan, ..., nan, nan, nan]]]], device='cuda:0',

due to which there is an error coming up in loss.backward()

Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass RuntimeError: CUDA error: device-side assert triggered

My idea was to put some negative images into training so that model understands a bit clearly about the busy background, and as we remove these negative images and corresponding masks the code is working fine.

Please confirm , how to resolve this so that i can consider those negative images as well ???

Runtime Speeds

Hi, thanks for putting out an interesting paper and the code. I was testing it with the various pretrained weights on DUT-TE, and I'm getting an end-to-end speed of ~11 FPS on an RTX 2070 Super. Is that an expected speed?

Are training hyperparameters the same for all types of pretrained models with different backbones?

Hi,

Thanks for your incredible research.
I tried train network from scratch on dataset with objects and tried multiple pretrained backbones with removed edge generation.
Model with backbone 0 or 1 came to very high accuracy during training with your model-training params, such channels, RFB_aggregated_channel, frequency_radius, denoise and other values in the config.py.

I tried move to heavy 7 backbone, but accuracy couldn't went to desireble values with same model parameters.

So, i got question, are model parameters such channels, RFB_aggregated_channel, frequency_radius, denoise and gamma used the same as in config.py for training your model with 7 backbone?

After 75th step It generated an error:

RuntimeError: Function 'UpsampleBilinear2DBackward1' returned nan values in its 0th output.

The model trained successfully when using BCE loss.

We even checked for NaN values using torch.autograd.set_detect_anamoly(True) But it returned False stating that no NaN values were found