jina-ai/examples

Jina examples and demos to help you get started

PLEASE CHECK OUR DOC WEBSITE FOR MORE COMPREHENSIVE EXAMPLES.

Examples for Jina

Table of Contents

• Adding Tests for Examples
• Community
• License

These examples showcase Jina in action and provide sample code for you to work from.

We suggest you read the following to get an overview of what Jina is and how it works:

• What is neural search?
• Get started - especially the cookbooks for Document, Executor and Flow

🐣 Simple Examples

📄	Semantic Wikipedia Search with Transformers and DistilBERT Brand new to neural search? See a simple text-search example to understand how Jina works
📄	Search Lyrics with Transformers and PyTorch Get a better understanding of chunks by searching a lyrics database. Now with shiny front-end!
📄	Find Similar Audio Clips A simple example to show how to find similar audio clips using Jina

🚀 Advanced Examples

📄	Querying While Indexing in the Wikipedia Search Example Support both querying and indexing simultaneously in our Wikipedia Search Example
🖼️📄	Cross Modal: Search images from captions and vice-versa Use one modality (text) to search another (images)

Community Examples

Want to add your own example? Please check our guidelines!

🖼️📄	Meme Search - Text search / Image search / Front end Search memes by caption or similar image.
📄	App Store Search Use Transformers to search through a rich app store dataset with a responsive front-end
📄	Star Wars Question Answering System Generate answers based on star wars descriptions from wookieepedia.

Legacy examples

📄	Financial Question Answering Search Opinionated QA passage retrieval with BERT-based reranker

Adding Tests for Examples

You are highly encouraged to add a test for your example so that we will be alerted if it breaks in the future:

1. Put your test data in the tests folder. The test data can be a few text sentences, images or audio samples
2. Create test_[your_example].py in the tests folder. Add your test cases to the tests file with meaningful asserts depending on example input and output
3. Run the test locally to confirm before pushing with pytest
4. Add your example folder name to the path variable in matrix of .github/worflows/ci.yml. This will trigger your example test on creating a pull request.

Testing Tips

• For reference, check out the tests folder from any of the examples in this repo.
• Try using the original example function by importing them to the test. Avoid any modifications to original Flow or logic.
• Use the pytest fixture tmpdir for temporary directory

Community

• Slack channel - a communication platform for developers to discuss Jina
• LinkedIn - get to know Jina AI as a company and find job opportunities
• - follow us and interact with us using hashtag #JinaSearch
• Company - know more about our company, we are fully committed to open-source!

License

Copyright (c) 2021 Jina AI Limited. All rights reserved.

Jina is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.

examples's Issues

the requirements is missing in examples

reduce the replicas to make examples less memory hungry

reduce the replicas to make examples less memory hungry

'TransformerTorchEncoder' object has no attribute '_device' when running Lyrics Search

Following up on #97, I decided to run the Lyrics Search code on my local machine (no Docker) as that would allow me to copy the toy dataset into the ./data folder (with this docker image I am unable to log inside using bash, it automatically calls app.py).

The problem is when I run app.py index. I get the following error:

join_all@18945[I]:collected 2/2 parts of IndexRequest
        gateway@18763[E]:callback() may not work properly due to the bad response: encoder-1(TransformerTorchEncoder) throws AttributeError("'TransformerTorchEncoder' object has no attribute '_device'")
        gateway@18763[E]:[pod: "encoder-1"
pod_id: "0ceea959f9"
executor: "TransformerTorchEncoder"
exception: "AttributeError(\"\'TransformerTorchEncoder\' object has no attribute \'_device\'\")"
traceback: "Traceback (most recent call last):\n  File \"/Users/cesc/opt/anaconda3/envs/CescPython37/lib/python3.7/site-packages/jina/peapods/pea.py\", line 275, in msg_callback\n    self.zmqlet.send_message(self._callback(msg))\n  File \"/Users/cesc/opt/anaconda3/envs/CescPython37/lib/python3.7/site-packages/jina/peapods/pea.py\", line 264, in _callback\n    self.pre_hook(msg).handle(msg).post_hook(msg)\n  File \"/Users/cesc/opt/anaconda3/envs/CescPython37/lib/python3.7/site-packages/jina/peapods/pea.py\", line 161, in handle\n    self.executor(self.request_type)\n  File \"/Users/cesc/opt/anaconda3/envs/CescPython37/lib/python3.7/site-packages/jina/executors/__init__.py\", line 553, in __call__\n    d()\n  File \"/Users/cesc/opt/anaconda3/envs/CescPython37/lib/python3.7/site-packages/jina/drivers/encode.py\", line 31, in __call__\n    embeds = self.exec_fn(contents)\n  File \"/Users/cesc/opt/anaconda3/envs/CescPython37/lib/python3.7/site-packages/jina/executors/decorators.py\", line 158, in arg_wrapper\n    return func(self, data, *args, **kwargs)\n  File \"/Users/cesc/opt/anaconda3/envs/CescPython37/lib/python3.7/site-packages/jina/executors/decorators.py\", line 58, in arg_wrapper\n    r = func(self, *args, **kwargs)\n  File \"/Users/cesc/opt/anaconda3/envs/CescPython37/lib/python3.7/site-packages/jina/executors/encoders/nlp/transformer.py\", line 71, in encode\n    seq_output, *extra_output = self.model(token_ids_batch, attention_mask=mask_ids_batch)\n  File \"/Users/cesc/opt/anaconda3/envs/CescPython37/lib/python3.7/site-packages/jina/executors/encoders/nlp/transformer.py\", line 122, in model\n    self._model = self.get_model()\n  File \"/Users/cesc/opt/anaconda3/envs/CescPython37/lib/python3.7/site-packages/jina/executors/encoders/nlp/transformer.py\", line 250, in get_model\n    self.to_device(_model)\n  File \"/Users/cesc/opt/anaconda3/envs/CescPython37/lib/python3.7/site-packages/jina/executors/frameworks.py\", line 95, in to_device\n    model.to(self.device)\n  File \"/Users/cesc/opt/anaconda3/envs/CescPython37/lib/python3.7/site-packages/jina/executors/frameworks.py\", line 41, in device\n    if self._device is not None:\nAttributeError: \'TransformerTorchEncoder\' object has no attribute \'_device\'\n"
time {
  seconds: 1594802831
  nanos: 597826000
}

(By the way, I learned the hard way that the number before the seconds is the training example:
index [==== ] 📃 832 ⏱️ 148.7s 🐎 5.6/s 104 batch

in the case of the toy dataset there are just 1,000 training examples and it takes just a few minutes, so it is definitely impossible to process the entire lyrics dataset on a desktop. It would have helped me a lot to get the total number of training examples next to the number eg: 832/167,499 to understand that the code would basically "never end" on a a macbook. )

Eventually, indexing ends, but I anyway got the same error when running the search:

curl --request POST -d '{"top_k": 10, "mode": "search",  "data": ["text:hey, dude"]}' -H 'Content-Type: application/json' 'http://0.0.0.0:65481/api/search'`

"status": {
    "code": "ERROR",
    "description": "tf_encode(TransformerTorchEncoder) throws AttributeError(\"'TransformerTorchEncoder' object has no attribute '_device'\")",
    "details": [
      {
...

I installed all the requirements and upgraded to the last version of jina, transformers, torch, etc. What am I doing wrong?

Lyrics Search indexing is very slow

On the Lyrics search example, I run indexing via Docker but it takes a long time (it has been running for 3 hours). It is unclear to me what exactly is indexing (the toy data or the full data) because when running search for the first time it initially downloads a 263MB file which I suspect is the full dataset.

Just wanted to know from your end:

1. How long does it take (approximately) to you to run the index on a laptop machine (both for the toy dataset and the full dataset)?

2. How can I know during indexing if it is indexing the full dataset or the toy dataset?

3. Can indexing time be improved significantly by using a GPU machine?

4. While indexing, files are generated in the ./workspace folder. However, errors are returned if I try to perform a search before indexing ends. Particularly:
   you can not query from NumpyIndexer as its "query_handler" is not set. If you are indexing data from scratch then it is fine. If you are querying data then the index file must be empty or broken and 'NoneType' object has no attribute 'shape'. Is it so that indexing has to complete before search can be performed upon, meaning it is not possible to run searches on a partial index?

5. Is indexing necessary when running it via Docker, or does the Docker image already contain indices?

Issue on Windows 10 . Shows os error

Ran the code code unchanged. Used the same pods files (.yml files). Shows an error of os .

compound indexer example

I just update the demo (only indexing part) according to jina's latest API. The indexing can run without problem.

@policeme please follow this https://github.com/jina-ai/examples/blob/master/tumblr-gif-search/index/chunk.yml

unknown exception: ImageNotFound(HTTPError('404 Client Error: Not Found for url: http+docker://localhost/v1.35/containers/create?name=encoder'))

Traceback (most recent call last):
File "/home/jinqiu/anaconda3/envs/jina/lib/python3.7/site-packages/docker/api/client.py", line 261, in _raise_for_status
response.raise_for_status()
File "/home/jinqiu/anaconda3/envs/jina/lib/python3.7/site-packages/requests/models.py", line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.35/containers/create?name=encoder

Traceback (most recent call last):
File "/home/jinqiu/anaconda3/envs/jina/lib/python3.7/site-packages/jina/peapods/pea.py", line 330, in run
self.post_init()
File "/home/jinqiu/anaconda3/envs/jina/lib/python3.7/site-packages/jina/peapods/container.py", line 70, in post_init
entrypoint=self.args.entrypoint,
File "/home/jinqiu/anaconda3/envs/jina/lib/python3.7/site-packages/docker/models/containers.py", line 807, in run
detach=detach, **kwargs)
File "/home/jinqiu/anaconda3/envs/jina/lib/python3.7/site-packages/docker/models/containers.py", line 861, in create
resp = self.client.api.create_container(**create_kwargs)
File "/home/jinqiu/anaconda3/envs/jina/lib/python3.7/site-packages/docker/api/container.py", line 430, in create_container
return self.create_container_from_config(config, name)
File "/home/jinqiu/anaconda3/envs/jina/lib/python3.7/site-packages/docker/api/container.py", line 441, in create_container_from_config
return self._result(res, True)
File "/home/jinqiu/anaconda3/envs/jina/lib/python3.7/site-packages/docker/api/client.py", line 267, in _result
self._raise_for_status(response)
File "/home/jinqiu/anaconda3/envs/jina/lib/python3.7/site-packages/docker/api/client.py", line 263, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/home/jinqiu/anaconda3/envs/jina/lib/python3.7/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error: Not Found ("No such image: jinaai/hub.executors.encoders.image.torchvision-mobilenet_v2:latest")

fix broken links in README

flower-search and urbandict-search examples not working

I met problems running flower-search and urbandict-search. Error messages are as follow:

The problem seems related to the use of buffer. I haven't tested other buffer-used examples. But when I change all buffer back to raw_bytes, both can work well.

problem on execute the news searcah example

I execute the news search example as following
(1)pip install jina[all]
(2)download the example data
(3)extract the data to the /tmp dir
(4)execute: python prepare.py
(5)execute: python app.py -t index
and got error as following:
encoder driver throws an exception, the sequel pipeline may not work properly
🐳 encoder@ 8[I]:received "index" from gateway▸extractor-head▸extractor-1▸extractor-tail▸encoder-head▸⚐
🐳 encoder@ 8[I]:batching enabled for BaseTransformerEncoder.encode(). batch_size=32 num_batch=None axis=0
encoder@21899[I]:received "control" from gateway▸extractor-head▸extractor-1▸extractor-tail▸encoder-head▸encoder-0▸⚐
encoder@21902[I]:received "index" from gateway▸extractor-head▸extractor-1▸extractor-tail▸encoder-head▸encoder-0▸⚐
chunk_indexer@22043[E]:unknown exception: vectors shape 790 is not valid, expecting "vectors" to have rank of 2
Traceback (most recent call last):
File "/usr/anaconda3/lib/python3.7/site-packages/jina/peapods/pea.py", line 352, in run
self.loop_body()
File "/usr/anaconda3/lib/python3.7/site-packages/jina/peapods/pea.py", line 317, in loop_body
msg = self.zmqlet.recv_message(callback=self.msg_callback)
File "/usr/anaconda3/lib/python3.7/site-packages/jina/peapods/zmq.py", line 222, in recv_message
return callback(msg)
File "/usr/anaconda3/lib/python3.7/site-packages/jina/peapods/pea.py", line 301, in msg_callback
return self._callback(msg)
File "/usr/anaconda3/lib/python3.7/site-packages/jina/peapods/pea.py", line 291, in _callback
self.pre_hook(msg).handle(msg).post_hook(msg)
File "/usr/anaconda3/lib/python3.7/site-packages/jina/peapods/pea.py", line 179, in handle
self.executor(self.request_type)
File "/usr/anaconda3/lib/python3.7/site-packages/jina/executors/init.py", line 534, in call
d()
File "/usr/anaconda3/lib/python3.7/site-packages/jina/drivers/index.py", line 32, in call
self.exec_fn(np.array([c.chunk_id for c in chunk_pts]), np.stack(embed_vecs))
File "/usr/anaconda3/lib/python3.7/site-packages/jina/executors/decorators.py", line 22, in arg_wrapper
f = func(self, *args, **kwargs)
File "/usr/anaconda3/lib/python3.7/site-packages/jina/executors/indexers/vector/numpy.py", line 87, in add
raise ValueError('vectors shape %s is not valid, expecting "vectors" to have rank of 2' % vectors.shape)
ValueError: vectors shape 790 is not valid, expecting "vectors" to have rank of 2
chunk_indexer@22046[E]:unknown exception: vectors shape 1162 is not valid, expecting "vectors" to have rank of 2
Traceback (most recent call last):
File "/usr/anaconda3/lib/python3.7/site-packages/jina/peapods/pea.py", line 352, in run
self.loop_body()
File "/usr/anaconda3/lib/python3.7/site-packages/jina/peapods/pea.py", line 317, in loop_body
msg = self.zmqlet.recv_message(callback=self.msg_callback)
File "/usr/anaconda3/lib/python3.7/site-packages/jina/peapods/zmq.py", line 222, in recv_message
return callback(msg)
File "/usr/anaconda3/lib/python3.7/site-packages/jina/peapods/pea.py", line 301, in msg_callback
return self._callback(msg)
File "/usr/anaconda3/lib/python3.7/site-packages/jina/peapods/pea.py", line 291, in _callback
self.pre_hook(msg).handle(msg).post_hook(msg)
File "/usr/anaconda3/lib/python3.7/site-packages/jina/peapods/pea.py", line 179, in handle
self.executor(self.request_type)
File "/usr/anaconda3/lib/python3.7/site-packages/jina/executors/init.py", line 534, in call
d()
File "/usr/anaconda3/lib/python3.7/site-packages/jina/drivers/index.py", line 32, in call
self.exec_fn(np.array([c.chunk_id for c in chunk_pts]), np.stack(embed_vecs))
File "/usr/anaconda3/lib/python3.7/site-packages/jina/executors/decorators.py", line 22, in arg_wrapper
f = func(self, *args, **kwargs)
File "/usr/anaconda3/lib/python3.7/site-packages/jina/executors/indexers/vector/numpy.py", line 87, in add
raise ValueError('vectors shape %s is not valid, expecting "vectors" to have rank of 2' % vectors.shape)
ValueError: vectors shape 1162 is not valid, expecting "vectors" to have rank of 2
chunk_indexer@22043[I]:no update since 2020-05-19 07:55:48, will not save. If you really want to save it, call "touch()" before "save()" to force saving
chunk_indexer@22043[I]:no update since 2020-05-19 07:55:48, will not save. If you really want to save it, call "touch()" before "save()" to force saving
chunk_indexer@22043[I]:no update since 2020-05-19 07:55:48, will not save. If you really want to save it, call "touch()" before "save()" to force saving
chunk_indexer@22043[I]:dumped changes to the executor, 4s since last the save
chunk_indexer@22043[I]:#sent: 1 #recv: 1 sent_size: 295 Bytes recv_size: 287.5 KB
chunk_indexer@22046[I]:no update since 2020-05-19 07:55:48, will not save. If you really want to save it, call "touch()" before "save()" to force saving
chunk_indexer@22046[I]:no update since 2020-05-19 07:55:48, will not save. If you really want to save it, call "touch()" before "save()" to force saving
chunk_indexer@22046[I]:no update since 2020-05-19 07:55:48, will not save. If you really want to save it, call "touch()" before "save()" to force saving
chunk_indexer@22046[I]:dumped changes to the executor, 4s since last the save
chunk_indexer@22046[I]:#sent: 1 #recv: 1 sent_size: 295 Bytes recv_size: 288.6 KB
chunk_indexer@22046[I]:#sent: 1 #recv: 1 sent_size: 295 Bytes recv_size: 288.6 KB
chunk_indexer@22046[S]:terminated
chunk_indexer@22043[I]:#sent: 1 #recv: 1 sent_size: 295 Bytes recv_size: 287.5 KB
chunk_indexer@22043[S]:terminated
🐳 encoder@ 8[E]:
🐳 Found no NVIDIA driver on your system. Please check that you
🐳 have an NVIDIA GPU and installed a driver from
🐳 http://www.nvidia.com/Download/index.aspx
🐳 Traceback (most recent call last):
🐳 File "/opt/conda/lib/python3.7/site-packages/jina/drivers/encode.py", line 30, in call
🐳 embeds = self.exec_fn(contents)
🐳 File "/opt/conda/lib/python3.7/site-packages/jina/executors/decorators.py", line 177, in arg_wrapper
🐳 r = func(self, b, *args, **kwargs)
🐳 File "/opt/conda/lib/python3.7/site-packages/jina/executors/decorators.py", line 53, in arg_wrapper
🐳 r = func(self, *args, **kwargs)
🐳 File "/opt/conda/lib/python3.7/site-packages/jina/executors/encoders/nlp/transformer.py", line 61, in encode
🐳 token_ids_batch = self.array2tensor(token_ids_batch)
🐳 File "/opt/conda/lib/python3.7/site-packages/jina/executors/encoders/nlp/transformer.py", line 250, in array2tensor
🐳 return tensor.cuda() if self.on_gpu else tensor
🐳 File "/opt/conda/lib/python3.7/site-packages/torch/cuda/init.py", line 149, in _lazy_init
🐳 _check_driver()
🐳 File "/opt/conda/lib/python3.7/site-packages/torch/cuda/init.py", line 54, in _check_driver
🐳 http://www.nvidia.com/Download/index.aspx""")
🐳 AssertionError:
🐳 Found no NVIDIA driver on your system. Please check that you
🐳 have an NVIDIA GPU and installed a driver from
🐳 http://www.nvidia.com/Download/index.aspx
🐳 encoder@ 8[W]:encoder driver throws an exception, the sequel pipeline may not work properly
encoder@21899[I]:received "control" from gateway▸extractor-head▸extractor-1▸extractor-tail▸encoder-head▸encoder-0▸⚐
encoder@21902[I]:received "index" from gateway▸extractor-head▸extractor-1▸extractor-tail▸encoder-head▸encoder-0▸⚐

could you pleased to tell me what happen or how to overcome it, thank you.

fix links in news-search

fix some links after merge to master

Southpark-search indexing process got stuck

I have been trying to run southpark-search following the instruction and the process seemed to stuck somewhere every time. (Before I have already tried jina hello-world and it can be executed without a problem.)

At first I thought it's still in progress just very slowly. But then I checked the system monitor they all seem kinda "idle". Nothing changed even after 20+ minutes.

Environment

Desktop: Fedora 31 (kernel 5.6)
Python version: 3.7.7
Jina version: 0.1.10 (installed via PyPi)

Steps

• Cloned the project to my local folder
• pip install -r requirements.txt > Success
• bash ./get_data.sh > Success
• python app.py -t index -n 10000 > Process Stuck

Logs

Below is the logs that I reproduced today:
https://gist.github.com/pswu11/0724531689b5f700a40ccd4384c3fb15

Screenshot

[zh/webqa-search] 'jina.executors.indexers.vector.milvusdb.milvusdbhandler' can't be imported

Follow the introduction of webqa-search exmaple, everything seems fine, but still get an Error as below:

JINA@5762[E]:theses modules or classes can not be imported ['jina.executors.indexers.vector.milvusdb.milvusdbhandler']

Any idea what's wrong and how to fix this?

btw, I tried other example, all of them has this problem.

The python version is 3.8.2 with ubuntu 20.04, and jina version is 0.4.0.

use distilbert docker image encoders in examples

adapt faiss to the latest indexer api

the yaml files might needs to change.

https://github.com/jina-ai/examples/blob/master/faiss-search/yaml/index-chunk.yml#L5

Reference:

jina-ai/jina#592

adapt the flower search to the latest api

x-as-service: Fix automatic download of the transformer image

The README states
In the example above, we use a prebuilt image from Jina Hub. It automatically pulls the image to local when you run this flow.

However, I just cloned the repository and ran app.py without success. I needed to pull the image manually. Then app.py run without issues.

Perhaps the problem is that the image to download is too large to be readily downloaded in 20s depending the connection (working from home here).

Ubuntu Focal Fossa: python: command not found

Testing Jina on a fresh Ubuntu 20.04 LTS install.

Notes

• I made the fresh install of Ubuntu yesterday - there's likely no weird custom settings that should cause this issue
• Python 3 is installed, but there is no /usr/bin/python, only /usr/bin/python3
• I know how to work around this issue (by creating symbolic link), but thought I'd share in case of confused users

Reproduce

cd examples/southpark-search
bash ./get_data.sh

Result

remote: Enumerating objects: 3852, done.
remote: Total 3852 (delta 0), reused 0 (delta 0), pack-reused 3852
Receiving objects: 100% (3852/3852), 5.11 MiB | 3.05 MiB/s, done.
Resolving deltas: 100% (40/40), done.
./get_data.sh: line 5: python: command not found

add an example for using tf serving

Based on the official tf serving tutorial at https://www.tensorflow.org/tfx/serving/serving_basic, we build an encoder for encoding mnist images.

Here is the plan,

• serve the mnist classification examples based on the tutorial at https://www.tensorflow.org/tfx/serving/serving_basic. One need to tweak the model export a little bit and use the second last layer for output instead of the classification results.
• implement MnistTFServingClientEncoder inherited from BaseTFServingClientEncoder. We've made a simple one in the unit test, https://github.com/jina-ai/jina/blob/master/tests/unit/executors/encoders/clients.py#L10.
• build an example that load the MNIST data set and encode with the MnistTFServingClientEncoder. One can refer to the hello-world example in the jina core, https://github.com/jina-ai/jina/blob/master/jina/helloworld/__init__.py
• write down a tutorial

Timeout error

hello， when I run a demo of jina, i got some questions

As you can see, when I ran one of jina's demos called "x-as-service," the following error occurred

I wonder if this is just my network problem or something else

thanks

fix requirements in southpark

examples/Jina Command-Line Interface url error

The invalid url is: https://docs.jina.ai/chapters/cli/main.html , it should be replaced with https://docs.jina.ai/chapters/cli/index.html.

The error of the link has led to a Page Not Found error in all examples, Documentation Section, behind text Jina command line interface arguments explained.

examples/jina-0.4.1 migration

As was discussed, I'll start working on Jina examples migrations for Jina 0.4.1 release following the migration plan here.

This issue is about to set up 0.4.1 migration for all the examples

Example	Migrate-by	Peer-tested-by	Status	Pull Request
face-db-search	@bwanglzu	@YueLiu-jina	Merged	#117
faiss-search	@bwanglzu		Merged	#130
flower-search	@YueLiu-jina		Merged	#119
helloworld-in-cs	@YueLiu-jina		Merged	#140
in-mock-app	@YueLiu-jina		Merged	#125
multilingual-search	@bwanglzu		Merged	#127
multires-lyrics-search	@bwanglzu		Merged	#128
pokedex-with-bit	@bwanglzu	@YueLiu-jina	Merged	#120
southpark-search	@YueLiu-jina	@bwanglzu	Merged	#119
tumblr-gif-search	@bwanglzu		Merged	#129
urbandict-search	@YueLiu-jina		Merged	#119
x-as-service	@bwanglzu	@YueLiu-jina	Merged	#122
zh/news-search	@YueLiu-jina		Problem	#143
zh/webqa-search	@YueLiu-jina		Merged	#124

topk should be change to top_k

In the example for south-park semantic search the parameter topk for search task should be replaced with top_k.

examples/southpark-search/app.py

Line 57 in b4266a5

f.search_lines(lines=[text, ], output_fn=ppr, topk=top_k)

No requirements.txt for Southpark Search example

Missing requirements.txt file

examples/southpark-search throws exception

        gateway@3810[E]:callback() may not work properly due to the bad response: encoder-1(TransformerTorchEncoder) throws IndexError('index 2 is out of bounds for axis 0 with size 1')
        gateway@3810[E]:[pod: "encoder-1"
pod_id: "7d854a9657"
executor: "TransformerTorchEncoder"
exception: "IndexError(\'index 2 is out of bounds for axis 0 with size 1\')"
traceback: "Traceback (most recent call last):\n  File \"/media/kavan/Important/Job/jina/venv/lib/python3.7/site-packages/jina/peapods/pea.py\", line 275, in msg_callback\n    return self._callback(msg)\n  File \"/media/kavan/Important/Job/jina/venv/lib/python3.7/site-packages/jina/peapods/pea.py\", line 264, in _callback\n    self.pre_hook(msg).handle(msg).post_hook(msg)\n  File \"/media/kavan/Important/Job/jina/venv/lib/python3.7/site-packages/jina/peapods/pea.py\", line 161, in handle\n    self.executor(self.request_type)\n  File \"/media/kavan/Important/Job/jina/venv/lib/python3.7/site-packages/jina/executors/__init__.py\", line 553, in __call__\n    d()\n  File \"/media/kavan/Important/Job/jina/venv/lib/python3.7/site-packages/jina/drivers/encode.py\", line 31, in __call__\n    embeds = self.exec_fn(contents)\n  File \"/media/kavan/Important/Job/jina/venv/lib/python3.7/site-packages/jina/executors/decorators.py\", line 158, in arg_wrapper\n    return func(self, data, *args, **kwargs)\n  File \"/media/kavan/Important/Job/jina/venv/lib/python3.7/site-packages/jina/executors/decorators.py\", line 58, in arg_wrapper\n    r = func(self, *args, **kwargs)\n  File \"/media/kavan/Important/Job/jina/venv/lib/python3.7/site-packages/jina/executors/encoders/nlp/transformer.py\", line 71, in encode\n    output = reduce_cls(_seq_output, _mask_ids_batch, self.cls_pos)\n  File \"/media/kavan/Important/Job/jina/venv/lib/python3.7/site-packages/jina/executors/encoders/helper.py\", line 40, in reduce_cls\n    mask_pruned = prune_mask(mask_2d, cls_pos)\n  File \"/media/kavan/Important/Job/jina/venv/lib/python3.7/site-packages/jina/executors/encoders/helper.py\", line 52, in prune_mask\n    result[num_tokens - 1] = 1\nIndexError: index 2 is out of bounds for axis 0 with size 1\n"
time {
  seconds: 1593986002
  nanos: 871291000
}
, pod: "chunk_indexer-1"
pod_id: "bef3ebcf12"
executor: "ChunkIndexer"
exception: "ValueError(\'not enough values to unpack (expected 2, got 1)\')"
traceback: "Traceback (most recent call last):\n  File \"/media/kavan/Important/Job/jina/venv/lib/python3.7/site-packages/jina/peapods/pea.py\", line 275, in msg_callback\n    return self._callback(msg)\n  File \"/media/kavan/Important/Job/jina/venv/lib/python3.7/site-packages/jina/peapods/pea.py\", line 264, in _callback\n    self.pre_hook(msg).handle(msg).post_hook(msg)\n  File \"/media/kavan/Important/Job/jina/venv/lib/python3.7/site-packages/jina/peapods/pea.py\", line 161, in handle\n    self.executor(self.request_type)\n  File \"/media/kavan/Important/Job/jina/venv/lib/python3.7/site-packages/jina/executors/__init__.py\", line 553, in __call__\n    d()\n  File \"/media/kavan/Important/Job/jina/venv/lib/python3.7/site-packages/jina/drivers/search.py\", line 114, in __call__\n    idx, dist = self.exec_fn(embed_vecs, top_k=self.req.top_k)\n  File \"/media/kavan/Important/Job/jina/venv/lib/python3.7/site-packages/jina/executors/indexers/vector/numpy.py\", line 52, in query\n    dist = _euclidean(keys, self.query_handler)\n  File \"/media/kavan/Important/Job/jina/venv/lib/python3.7/site-packages/jina/executors/indexers/vector/numpy.py\", line 82, in _euclidean\n    A_ext, B_ext = _ext_arrs(A, B)\n  File \"/media/kavan/Important/Job/jina/venv/lib/python3.7/site-packages/jina/executors/indexers/vector/numpy.py\", line 69, in _ext_arrs\n    nA, dim = A.shape\nValueError: not enough values to unpack (expected 2, got 1)\n"
time {
  seconds: 1593986002
  nanos: 884475000
}
]

add badges at both headers and footers

fix the typo in separated_workspace

Timestamps on output?

Hi, how about using dashboard for that?

Try for example JINA_LOG_PROFILING=1 jina hello-world --logserver and goto http://dashboard.jina.ai/ Hit refresh button in the browser for couple of times to make sure it is connected to your localhost

I'll look into it tomorrow!

add footer to the README

add more contents to the README

examples/southpark-search throws exception

The southpark-search throws an exception in the query stage for me. Following error:
unknown exception: 'NoneType' object has no attribute 'shape'
in jina/executors/indexers/vector/numpy.py", line 148, in _ext_arrs`

I can also see this in the logs:
you can not query from <jina.executors.indexers.vector.numpy.NumpyIndexer object at 0x7fb6f03d4250> as its "query_handler" is not set. If you are indexing data from scratch then it is fine. If you are querying data then the index file must be empty or broken.

Can someone reproduce this?

examples/southpark-search: encoder driver throws an exception, the sequel pipeline may not work properly

Environment

• Fresh install of Ubuntu 20.04
• /usr/bin/python points to /usr/bin/python3

Reproduce

Note: Already installed all prerequisite packages via pip, and downloaded data via get_data.sh

python app.py -t index -n 10000

Result

See attached error.log

Notes

I encountered the exact (I think) same error on my Manjaro install, so it's likely not an Ubuntu thing

[UrbanDict] ruamel.yaml.constructor.ConstructorError: could not determine a constructor for the tag '!DocKVIndexDriver'

Hi guys,

I am trying to setup the UrbanDict example.
Instead using index 10000, I just used 100 (since I just want to test a bit, and check out the process only).
But I faced this when running python app.py -t index -n 100.
Env: python3.8, jina-0.2.8

chunk_indexer@6939[I]:input tcp://0.0.0.0:51377 (PULL_BIND) 	 output tcp://0.0.0.0:51385 (PUSH_CONNECT)	 control over tcp://0.0.0.0:51379 (PAIR_BIND)
  chunk_indexer@6939[S]:ready and listening
    doc_indexer@6940[C]:unknown exception: ConstructorError(None, None, "could not determine a constructor for the tag '!DocKVIndexDriver'", <_ruamel_yaml.Mark object at 0x10f9ea770>)
Traceback (most recent call last):
  File "/Users/binhnguyen/env/jina-python3-env/lib/python3.8/site-packages/jina/peapods/pea.py", line 343, in run
    self.loop_body()
  File "/Users/binhnguyen/env/jina-python3-env/lib/python3.8/site-packages/jina/peapods/pea.py", line 305, in loop_body
    self.load_executor()
  File "/Users/binhnguyen/env/jina-python3-env/lib/python3.8/site-packages/jina/peapods/pea.py", line 198, in load_executor
    self.executor = BaseExecutor.load_config(self.args.yaml_path,
  File "/Users/binhnguyen/env/jina-python3-env/lib/python3.8/site-packages/jina/executors/__init__.py", line 404, in load_config
    return yaml.load(tmp_s)
  File "/Users/binhnguyen/env/jina-python3-env/lib/python3.8/site-packages/ruamel/yaml/main.py", line 343, in load
    return constructor.get_single_data()
  File "/Users/binhnguyen/env/jina-python3-env/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 113, in get_single_data
    return self.construct_document(node)
  File "/Users/binhnguyen/env/jina-python3-env/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 118, in construct_document
    data = self.construct_object(node)
  File "/Users/binhnguyen/env/jina-python3-env/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 146, in construct_object
    data = self.construct_non_recursive_object(node)
  File "/Users/binhnguyen/env/jina-python3-env/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 181, in construct_non_recursive_object
    data = constructor(self, node)
  File "/Users/binhnguyen/env/jina-python3-env/lib/python3.8/site-packages/jina/executors/__init__.py", line 445, in from_yaml
    return cls._get_instance_from_yaml(constructor, node)[0]
  File "/Users/binhnguyen/env/jina-python3-env/lib/python3.8/site-packages/jina/executors/__init__.py", line 449, in _get_instance_from_yaml
    data = ruamel.yaml.constructor.SafeConstructor.construct_mapping(
  File "/Users/binhnguyen/env/jina-python3-env/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 440, in construct_mapping
    return BaseConstructor.construct_mapping(self, node, deep=deep)
  File "/Users/binhnguyen/env/jina-python3-env/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 255, in construct_mapping
    value = self.construct_object(value_node, deep=deep)
  File "/Users/binhnguyen/env/jina-python3-env/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 146, in construct_object
    data = self.construct_non_recursive_object(node)
  File "/Users/binhnguyen/env/jina-python3-env/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 188, in construct_non_recursive_object
    for _dummy in generator:
  File "/Users/binhnguyen/env/jina-python3-env/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 723, in construct_yaml_map
    value = self.construct_mapping(node)
  File "/Users/binhnguyen/env/jina-python3-env/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 440, in construct_mapping
    return BaseConstructor.construct_mapping(self, node, deep=deep)
  File "/Users/binhnguyen/env/jina-python3-env/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 255, in construct_mapping
    value = self.construct_object(value_node, deep=deep)
  File "/Users/binhnguyen/env/jina-python3-env/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 146, in construct_object
    data = self.construct_non_recursive_object(node)
  File "/Users/binhnguyen/env/jina-python3-env/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 188, in construct_non_recursive_object
    for _dummy in generator:
  File "/Users/binhnguyen/env/jina-python3-env/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 723, in construct_yaml_map
    value = self.construct_mapping(node)
  File "/Users/binhnguyen/env/jina-python3-env/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 440, in construct_mapping
    return BaseConstructor.construct_mapping(self, node, deep=deep)
  File "/Users/binhnguyen/env/jina-python3-env/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 255, in construct_mapping
    value = self.construct_object(value_node, deep=deep)
  File "/Users/binhnguyen/env/jina-python3-env/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 146, in construct_object
    data = self.construct_non_recursive_object(node)
  File "/Users/binhnguyen/env/jina-python3-env/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 188, in construct_non_recursive_object
    for _dummy in generator:
  File "/Users/binhnguyen/env/jina-python3-env/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 717, in construct_yaml_seq
    data.extend(self.construct_sequence(node))
  File "/Users/binhnguyen/env/jina-python3-env/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 211, in construct_sequence
    return [self.construct_object(child, deep=deep) for child in node.value]
  File "/Users/binhnguyen/env/jina-python3-env/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 211, in <listcomp>
    return [self.construct_object(child, deep=deep) for child in node.value]
  File "/Users/binhnguyen/env/jina-python3-env/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 146, in construct_object
    data = self.construct_non_recursive_object(node)
  File "/Users/binhnguyen/env/jina-python3-env/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 181, in construct_non_recursive_object
    data = constructor(self, node)
  File "/Users/binhnguyen/env/jina-python3-env/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 739, in construct_undefined
    raise ConstructorError(
ruamel.yaml.constructor.ConstructorError: could not determine a constructor for the tag '!DocKVIndexDriver'
  in "<unicode string>", line 22, column 7
    doc_indexer@6940[S]:terminated
    doc_indexer@6932[C]:fail to start <class 'jina.peapods.pea.BasePea'> with name doc_indexer
       join_all@6941[I]:post initiating, this may take some time...```

add docker images for the southpark example

Build docker image for flower search

Build docker image for flower search

make the input of flower demo to raw image data rather than file path

to make it more realistic in practice, e.g. user can upload an image to search for similar flower.

update examples according to latest ingest API

mac直接执行 python3 app.py -t index 出错

objc[3871]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called.
objc[3871]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.

fix url in tutorial table

@YueLiu-jina

Make sure you fix all table inside every README

Query function is not working for urbandict-search example

I am having the below problem while trying to run query for urbandict-search (python app.py -t query
).

TypeError: py_client() got multiple values for keyword argument 'port_grpc'

The cause is in _get_client function of flow; port_grpc is already defined but it is also coming from kwargs from above.
The reason is flow-query.yml of urbandict-search has "port_grpc: 56798" defined in it. This line should be removed.

southpark-search: Exception iterating requests

Environment

• Clean Focal Fossa install
• Tested on two branches:
  • master
  • fix-broken-flow-in-linux-49

Reproduce

python app.py -t index -n 100000

Output

           Flow@32924[S]:�[32msuccessfully built Flow from a yaml config�[0m
     dispatcher@32933[I]:�[37mpost initiating, this may take some time...�[0m
     dispatcher@32933[I]:�[37mpost initiating, this may take some time takes 0.000 secs�[0m
     dispatcher@32933[S]:�[32msuccessfully built BaseExecutor from a yaml config�[0m
     dispatcher@32933[I]:�[37msetting up sockets...�[0m
     dispatcher@32933[I]:�[37minput �[33mtcp://0.0.0.0:53089�[0m (PULL_BIND) 	 output �[33mtcp://0.0.0.0:54667�[0m (PUB_BIND)	 control over �[33mtcp://0.0.0.0:52269�[0m (PAIR_BIND)�[0m
     dispatcher@32933[S]:�[32mready and listening�[0m
       splittor@32936[W]:�[40m�[33mthe min_sent_len (=2) should be smaller or equal to the max_sent_len (=64)�[0m
       splittor@32936[I]:�[37mpost initiating, this may take some time...�[0m
       splittor@32936[I]:�[37mpost initiating, this may take some time takes 0.000 secs�[0m
       splittor@32936[S]:�[32msuccessfully built Sentencizer from a yaml config�[0m
       splittor@32936[I]:�[37msetting up sockets...�[0m
       splittor@32936[I]:�[37minput �[33mtcp://0.0.0.0:54667�[0m (SUB_CONNECT) 	 output �[33mtcp://0.0.0.0:47697�[0m (PUSH_CONNECT)	 control over �[33mtcp://0.0.0.0:54221�[0m (PAIR_BIND)�[0m
       splittor@32936[S]:�[32mready and listening�[0m
        encoder@32939[I]:�[37mpost initiating, this may take some time...�[0m
        encoder@32939[I]:�[37mpost initiating, this may take some time takes 1.769 secs�[0m
        encoder@32939[S]:�[32msuccessfully built TransformerTorchEncoder from a yaml config�[0m
        encoder@32939[I]:�[37msetting up sockets...�[0m
        encoder@32939[I]:�[37minput �[33mtcp://0.0.0.0:47697�[0m (PULL_BIND) 	 output �[33mtcp://0.0.0.0:41123�[0m (PUSH_CONNECT)	 control over �[33mtcp://0.0.0.0:43255�[0m (PAIR_BIND)�[0m
        encoder@32939[S]:�[32mready and listening�[0m
  chunk_indexer@32942[I]:�[37mpost initiating, this may take some time...�[0m
  chunk_indexer@32942[I]:�[37mpost initiating, this may take some time takes 0.000 secs�[0m
  chunk_indexer@32942[S]:�[32msuccessfully built NumpyIndexer from a yaml config�[0m
  chunk_indexer@32942[I]:�[37mpost initiating, this may take some time...�[0m
  chunk_indexer@32942[I]:�[37mpost initiating, this may take some time takes 0.000 secs�[0m
  chunk_indexer@32942[S]:�[32msuccessfully built BasePbIndexer from a yaml config�[0m
  chunk_indexer@32942[I]:�[37mpost initiating, this may take some time...�[0m
  chunk_indexer@32942[I]:�[37mpost initiating, this may take some time takes 0.000 secs�[0m
  chunk_indexer@32942[S]:�[32msuccessfully built ChunkIndexer from a yaml config�[0m
  chunk_indexer@32942[W]:�[40m�[33myou can not query from <jina.executors.indexers.vector.numpy.NumpyIndexer object at 0x7f9e657a65b0> as its "query_handler" is not set. If you are indexing data from scratch then it is fine. If you are querying data then the index file must be empty or broken.�[0m
  chunk_indexer@32942[I]:�[37msetting up sockets...�[0m
  chunk_indexer@32942[I]:�[37minput �[33mtcp://0.0.0.0:41123�[0m (PULL_BIND) 	 output �[33mtcp://0.0.0.0:53169�[0m (PUSH_CONNECT)	 control over �[33mtcp://0.0.0.0:34031�[0m (PAIR_BIND)�[0m
  chunk_indexer@32942[S]:�[32mready and listening�[0m
    doc_indexer@32945[I]:�[37mpost initiating, this may take some time...�[0m
    doc_indexer@32945[I]:�[37mpost initiating, this may take some time takes 0.000 secs�[0m
    doc_indexer@32945[S]:�[32msuccessfully built DocPbIndexer from a yaml config�[0m
    doc_indexer@32945[I]:�[37msetting up sockets...�[0m
    doc_indexer@32945[I]:�[37minput �[33mtcp://0.0.0.0:54667�[0m (SUB_CONNECT) 	 output �[33mtcp://0.0.0.0:53169�[0m (PUSH_CONNECT)	 control over �[33mtcp://0.0.0.0:57293�[0m (PAIR_BIND)�[0m
    doc_indexer@32945[S]:�[32mready and listening�[0m
       join_all@32948[I]:�[37mpost initiating, this may take some time...�[0m
       join_all@32948[I]:�[37mpost initiating, this may take some time takes 0.000 secs�[0m
       join_all@32948[S]:�[32msuccessfully built BaseExecutor from a yaml config�[0m
       join_all@32948[I]:�[37msetting up sockets...�[0m
       join_all@32948[I]:�[37minput �[33mtcp://0.0.0.0:53169�[0m (PULL_BIND) 	 output �[33mtcp://0.0.0.0:34151�[0m (PUSH_BIND)	 control over �[33mtcp://0.0.0.0:42943�[0m (PAIR_BIND)�[0m
       join_all@32948[S]:�[32mready and listening�[0m
       join_all@32924[I]:�[37mpost initiating, this may take some time...�[0m
       join_all@32924[I]:�[37mpost initiating, this may take some time takes 0.001 secs�[0m
       join_all@32924[S]:�[32mgateway is listening at: 0.0.0.0:55767�[0m
           Flow@32924[I]:�[37m7 Pods (i.e. 7 Peas) are running in this Flow�[0m
           Flow@32924[S]:�[32mflow is now ready for use, current build_level is GRAPH�[0m
           JINA@32924[W]:�[40m�[33m"raw_bytes" is deprecated in "index()" and will be removed in the next version; please use "input_fn" instead�[0m
       join_all@32924[S]:�[32mconnected to the gateway at 0.0.0.0:55767!�[0m

�[36mindex�[0m [�[32m=�[0m                   ] 📃      0 ⏱️ 0.0s 🐎 0.0/s      0 batchindex ...	file not found: /tmp/jina/southpark/character-lines.csv
�[32m    [0.001 secs]�[0m
	�[32m✅ done in ⏱ 0.0s 🐎 0.0/s�[0m
       join_all@32924[I]:�[37msetting up sockets...�[0m
       join_all@32924[I]:�[37minput �[33mtcp://0.0.0.0:34151�[0m (PULL_CONNECT) 	 output �[33mtcp://0.0.0.0:53089�[0m (PUSH_CONNECT)	 control over �[33mipc:///tmp/tmpzl6dj5e7�[0m (PAIR_BIND)�[0m
       join_all@32924[I]:�[37mprefetching 50 requests...�[0m
       join_all@32924[W]:�[40m�[33mif this takes too long, you may want to take smaller "--prefetch" or ask client to reduce "--batch-size"�[0m
       join_all@32924[I]:�[37mprefetching 50 requests takes 0.000 secs�[0m
       join_all@32924[I]:�[37m#sent: 0 #recv: 0 sent_size: 0 Bytes recv_size: 0 Bytes�[0m
       join_all@32924[S]:�[32mterminated�[0m
       join_all@32924[S]:�[32mterminated�[0m
       join_all@32948[I]:�[37mreceived "control" from ctl�[32m▸�[0m⚐�[0m
       join_all@32948[I]:�[37mbreak from the event loop�[0m
       join_all@32948[I]:�[37m#sent: 0 #recv: 1 sent_size: 0 Bytes recv_size: 249 Bytes�[0m
       join_all@32948[I]:�[37m#sent: 1 #recv: 1 sent_size: 289 Bytes recv_size: 249 Bytes�[0m
       join_all@32948[S]:�[32mterminated�[0m
    doc_indexer@32945[I]:�[37mreceived "control" from ctl�[32m▸�[0m⚐�[0m
    doc_indexer@32945[I]:�[37mbreak from the event loop�[0m
    doc_indexer@32945[I]:�[37mno update since 2020-05-17 14:29:50, will not save. If you really want to save it, call "touch()" before "save()" to force saving�[0m
    doc_indexer@32945[I]:�[37mexecutor says there is nothing to save�[0m
    doc_indexer@32945[I]:�[37m#sent: 0 #recv: 1 sent_size: 0 Bytes recv_size: 249 Bytes�[0m
    doc_indexer@32945[I]:�[37m#sent: 1 #recv: 1 sent_size: 292 Bytes recv_size: 249 Bytes�[0m
    doc_indexer@32945[S]:�[32mterminated�[0m
  chunk_indexer@32942[I]:�[37mreceived "control" from ctl�[32m▸�[0m⚐�[0m
  chunk_indexer@32942[I]:�[37mbreak from the event loop�[0m
  chunk_indexer@32942[I]:�[37mno update since 2020-05-17 14:29:50, will not save. If you really want to save it, call "touch()" before "save()" to force saving�[0m
  chunk_indexer@32942[I]:�[37mno update since 2020-05-17 14:29:50, will not save. If you really want to save it, call "touch()" before "save()" to force saving�[0m
  chunk_indexer@32942[I]:�[37mno update since 2020-05-17 14:29:50, will not save. If you really want to save it, call "touch()" before "save()" to force saving�[0m
  chunk_indexer@32942[I]:�[37mdumped changes to the executor,   0s since last the save�[0m
  chunk_indexer@32942[I]:�[37m#sent: 0 #recv: 1 sent_size: 0 Bytes recv_size: 249 Bytes�[0m
  chunk_indexer@32942[I]:�[37m#sent: 1 #recv: 1 sent_size: 294 Bytes recv_size: 249 Bytes�[0m
  chunk_indexer@32942[S]:�[32mterminated�[0m
        encoder@32939[I]:�[37mreceived "control" from ctl�[32m▸�[0m⚐�[0m
        encoder@32939[I]:�[37mbreak from the event loop�[0m
        encoder@32939[I]:�[37m#sent: 0 #recv: 1 sent_size: 0 Bytes recv_size: 249 Bytes�[0m
        encoder@32939[I]:�[37m#sent: 1 #recv: 1 sent_size: 288 Bytes recv_size: 249 Bytes�[0m
        encoder@32939[S]:�[32mterminated�[0m
       splittor@32936[I]:�[37mreceived "control" from ctl�[32m▸�[0m⚐�[0m
       splittor@32936[I]:�[37mbreak from the event loop�[0m
       splittor@32936[I]:�[37m#sent: 0 #recv: 1 sent_size: 0 Bytes recv_size: 249 Bytes�[0m
       splittor@32936[I]:�[37m#sent: 1 #recv: 1 sent_size: 289 Bytes recv_size: 249 Bytes�[0m
       splittor@32936[S]:�[32mterminated�[0m
     dispatcher@32933[I]:�[37mreceived "control" from ctl�[32m▸�[0m⚐�[0m
     dispatcher@32933[I]:�[37mbreak from the event loop�[0m
     dispatcher@32933[I]:�[37m#sent: 0 #recv: 1 sent_size: 0 Bytes recv_size: 249 Bytes�[0m
     dispatcher@32933[I]:�[37m#sent: 1 #recv: 1 sent_size: 291 Bytes recv_size: 249 Bytes�[0m
     dispatcher@32933[S]:�[32mterminated�[0m
           Flow@32924[S]:�[32mflow is closed and all resources should be released already, current build level is EMPTY�[0m
Traceback (most recent call last):
  File "/home/alexcg/.local/lib/python3.8/site-packages/jina/clients/python/grpc.py", line 71, in start
    self.call(*args, **kwargs)
  File "/home/alexcg/.local/lib/python3.8/site-packages/jina/clients/python/__init__.py", line 121, in call
    for resp in self._stub.Call(req_iter):
  File "/home/alexcg/.local/lib/python3.8/site-packages/grpc/_channel.py", line 416, in __next__
    return self._next()
  File "/home/alexcg/.local/lib/python3.8/site-packages/grpc/_channel.py", line 689, in _next
    raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.UNKNOWN
	details = "Exception iterating requests!"
	debug_error_string = "None"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "app.py", line 82, in <module>
    main()
  File "/home/alexcg/.local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/alexcg/.local/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/alexcg/.local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/alexcg/.local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "app.py", line 65, in main
    fl.index(raw_bytes=read_data(data_path, num_docs), batch_size=8)
  File "/home/alexcg/.local/lib/python3.8/site-packages/jina/helper.py", line 31, in wrapper
    return f(*args, **kwargs)
  File "/home/alexcg/.local/lib/python3.8/site-packages/jina/flow/__init__.py", line 606, in index
    self._get_client(**kwargs).index(input_fn, output_fn)
  File "/home/alexcg/.local/lib/python3.8/site-packages/jina/clients/python/__init__.py", line 184, in index
    self.start(output_fn, **kwargs)
  File "/home/alexcg/.local/lib/python3.8/site-packages/jina/clients/python/grpc.py", line 82, in start
    raise BadClient('%s error in grpc: %s '
jina.excepts.BadClient: StatusCode.UNKNOWN error in grpc: Exception iterating requests! often the case is that you define/send a bad input iterator to jina, please double check your input iterator

fail to start <class 'jina.peapods.container.ContainerPea'> with name encode-1

Hi team, when I execute the X-as-Service example from here, service could not start. Could you help?

Full log as below:

examples/southpark-search index error

• all config file are default
• index function

def index():
    data_path = os.path.join(os.environ['DATA_DIR'], os.environ['DATA_FILE'])
    f = Flow().load_config('flow-index.yml')
    data_path_df = pd.read_csv(data_path)
    with f:
        f.index_lines(lines=list(data_path_df['file_data']), batch_size=32)

• num docs = ~130000
• full error (https://pastebin.com/2tAAyc2H)

add doc links for shards

add doc links for shards

[zh/news-search] No dockerimage “jinaai/examples.hub.encoder.nlp.transformers-hit-scir”, and code running error

For https://github.com/jina-ai/examples/tree/master/zh/news-search, its README, flow-index.yml and flow-query.yml contain an image as below, which is not existed in docker hub.

!Flow
encoder:
    image: jinaai/examples.hub.encoder.nlp.transformers-hit-scir

Assume, it should be "jinaai/hub.executors.encoders.nlp.transformers-hitscir" (https://hub.docker.com/r/jinaai/hub.executors.encoders.nlp.transformers-hitscir)

Hope someone could fix it.

fix the shards usage in examples

fix the shards usage in examples

• southpark
• flower
• urbandict

adapt the southpark example to the latest api

fix the jina version for examples

As discussed at https://jina-ai.slack.com/archives/C016C7RCTT8/p1594745300076000, we'd like to freeze the jina version fo r the examples so that we don't need to worry about the version and can update the examples afterwards.

fix links in webqa