Fail to pull in MediaPipe's external dependencies due to unstable internet connection

[google]

Fail to pull in MediaPipe's external dependencies due to unstable internet connection

[chenloveheimei]

@chenloveheimei and I tested the current version of MediaPipe. We verified that MediaPipe demo apps can be successfully built in China now. However, due to the unstable/slow internet connection, we prefer to ask Bazel to cache the external repositories locally and search for the local archives before accessing the network.

For the users with unstable/slow internet connection, you may need to try the following build command several times to download all the needed external repositories:

$ bazel build -c opt --config=android_arm64 \ 
  --distdir=<your cache directory> \
  --repository_cache=<your cache directory> \
  --keep_going \
  mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu

I am now able to build object and face detection demos from Shenzhen, China. The problem is solved.
Thank you , you are good guy.

[jiuqiant]

Yes, it's possible to cache the external dependencies and use the cached version later. We are testing the following workflow:

# On the workstation that has the full access to the Internet, fetch all the dependencies to a cache directory by doing:
$ git clone https://github.com/google/mediapipe.git
$ cd mediapipe
$ bazel fetch //...  --keep_going --repository_cache=<your cache directory> 

# Then, copy the cache directory to the machine that has restricted Internet access and do:
$  bazel build -c opt --config=android_arm64  mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu --distdir=<your cache directory> 

If the workflow works, we may host an archive that contains all the MediaPipe external dependencies for the users in China.

Stay tuned.

[chenloveheimei]

Yes, it's possible to cache the external dependencies and use the cached version later. We are testing the following workflow:

# On the workstation that has the full access to the Internet, fetch all the dependencies to a cache directory by doing:
$ git clone https://github.com/google/mediapipe.git
$ cd mediapipe
$ bazel fetch //...  --keep_going --repository_cache=<your cache directory> 

# Then, copy the cache directory to the machine that has restricted Internet access and do:
$  bazel build -c opt --config=android_arm64  mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu --distdir=<your cache directory> 

If the workflow works, we may host an archive that contains all the MediaPipe external dependencies for the users in China.

Stay tuned.

thanks.I hope have a good news and can you @me when you update the new version?

[jiuqiant]

Hi Long, the error you pasted above is more like a network issue. More specifically, your machine is not able to download the pre-compiled OpenCV android SDK from https://sourceforge.net/projects/opencvlibrary/files/4.0.1/opencv-4.0.1-android-sdk.zip/download.
Unfortunately, we use the pre-compiled OpenCV Android SDK directly, rather than building the libraries from source.

Could you verify that you can access https://sourceforge.net/projects/opencvlibrary/files/4.0.1/opencv-4.0.1-android-sdk.zip/download? If so, can you retry the build command?

[chenloveheimei]

Hi Long, the error you pasted above is more like a network issue. More specifically, your machine is not able to download the pre-compiled OpenCV android SDK from https://sourceforge.net/projects/opencvlibrary/files/4.0.1/opencv-4.0.1-android-sdk.zip/download.
Unfortunately, we use the pre-compiled OpenCV Android SDK directly, rather than building the libraries from source.

Could you verify that you can access https://sourceforge.net/projects/opencvlibrary/files/4.0.1/opencv-4.0.1-android-sdk.zip/download? If so, can you retry the build command?

i can acces your website.
now i try it again. and generate a new Error:
INFO: Call stack for the definition of repository 'FP16' which is a third_party_http_archive (rule definition at /home/long/.cache/bazel/_bazel_long/2855d1951dc2684b853376a37e34b96b/external/org_tensorflow/third_party/repo.bzl:206:28):

• /home/long/.cache/bazel/_bazel_long/2855d1951dc2684b853376a37e34b96b/external/org_tensorflow/third_party/FP16/workspace.bzl:6:5
• /home/long/.cache/bazel/_bazel_long/2855d1951dc2684b853376a37e34b96b/external/org_tensorflow/tensorflow/workspace.bzl:37:5
• /home/long/.cache/bazel/_bazel_long/2855d1951dc2684b853376a37e34b96b/external/org_tensorflow/tensorflow/workspace.bzl:73:5
• /media/long/data/android/mediapipe/WORKSPACE:96:1
  ERROR: An error occurred during the fetch of repository 'FP16':
  java.io.IOException: Error downloading [https://mirror.bazel.build/github.com/Maratyszcza/FP16/archive/febbb1c163726b5db24bed55cc9dc42529068997.tar.gz, https://github.com/Maratyszcza/FP16/archive/febbb1c163726b5db24bed55cc9dc42529068
  997.tar.gz] to /home/long/.cache/bazel/_bazel_long/2855d1951dc2684b853376a37e34b96b/external/FP16/febbb1c163726b5db24bed55cc9dc42529068997.tar.gz: All mirrors are down: [GET returned 404 Not Found, connect timed out]
  INFO: Call stack for the definition of repository 'android_opencv' which is a http_archive (rule definition at /home/long/.cache/bazel/_bazel_long/2855d1951dc2684b853376a37e34b96b/external/bazel_tools/tools/build_defs/repo/http.bzl:237:
  16):
• /media/long/data/android/mediapipe/WORKSPACE:112:1
  INFO: Call stack for the definition of repository 'eigen_archive' which is a tf_http_archive (rule definition at /home/long/.cache/bazel/_bazel_long/2855d1951dc2684b853376a37e34b96b/external/org_tensorflow/third_party/repo.bzl:126:19):
• /home/long/.cache/bazel/_bazel_long/2855d1951dc2684b853376a37e34b96b/external/org_tensorflow/tensorflow/workspace.bzl:148:5
• /media/long/data/android/mediapipe/WORKSPACE:96:1
  ERROR: /home/long/.cache/bazel/_bazel_long/2855d1951dc2684b853376a37e34b96b/external/org_tensorflow/tensorflow/lite/delegates/gpu/common/BUILD:149:1: no such package '@FP16//': java.io.IOException: Error downloading [https://mirror.baze
  l.build/github.com/Maratyszcza/FP16/archive/febbb1c163726b5db24bed55cc9dc42529068997.tar.gz, https://github.com/Maratyszcza/FP16/archive/febbb1c163726b5db24bed55cc9dc42529068997.tar.gz] to /home/long/.cache/bazel/_bazel_long/2855d1951dc
  2684b853376a37e34b96b/external/FP16/febbb1c163726b5db24bed55cc9dc42529068997.tar.gz: All mirrors are down: [GET returned 404 Not Found, connect timed out] and referenced by '@org_tensorflow//tensorflow/lite/delegates/gpu/common:types'
  ERROR: Analysis of target '//mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu:facedetectiongpu' failed; build aborted: no such package '@FP16//': java.io.IOException: Error downloading [https://mirror.bazel
  .build/github.com/Maratyszcza/FP16/archive/febbb1c163726b5db24bed55cc9dc42529068997.tar.gz, https://github.com/Maratyszcza/FP16/archive/febbb1c163726b5db24bed55cc9dc42529068997.tar.gz] to /home/long/.cache/bazel/_bazel_long/2855d1951dc2
  684b853376a37e34b96b/external/FP16/febbb1c163726b5db24bed55cc9dc42529068997.tar.gz: All mirrors are down: [GET returned 404 Not Found, connect timed out]
  INFO: Elapsed time: 10.975s
  INFO: 0 processes.
  FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)
  Fetching @com_google_common_flogger; fetching 10s
  Fetching @com_google_common_flogger_system_backend; fetching 10s

I am in China. I can't open the VPN recent.

[jiuqiant]

Hi Long, sorry about the inconvenience.
https://mirror.bazel.build/ is a google owned website, and it is likely to be blocked by the Great Firewall. But, https://github.com/ should not be blacklisted. If you don't have VPN, the best solution might be retry the Bazel build command later. Bazel caches the downloaded libraries, so you don't need to worry about losing them by retrying the same build command.
Hope this helps.

[chenloveheimei]

Hi Long, sorry about the inconvenience.
https://mirror.bazel.build/ is a google owned website, and it is likely to be blocked by the Great Firewall. But, https://github.com/ should not be blacklisted. If you don't have VPN, the best solution might be retry the Bazel build command later. Bazel caches the downloaded libraries, so you don't need to worry about losing them by retrying the same build command.
Hope this helps.

My windows PC can opens VPN, but the Ubuntu can't.so I think I'll download the libraries which project needs from Windows and put them into the Ubuntu to setup?

[camillol]

@jiuqiant it seems like this would be a common issue for projects using Bazel. There are similar issues on the TensorFlow and Bazel trackers; we should check in with them and see if there's an existing workaround or plan.

[jiuqiant]

@chenloveheimei and I tested the current version of MediaPipe. We verified that MediaPipe demo apps can be successfully built in China now. However, due to the unstable/slow internet connection, we prefer to ask Bazel to cache the external repositories locally and search for the local archives before accessing the network.

For the users with unstable/slow internet connection, you may need to try the following build command several times to download all the needed external repositories:

$ bazel build -c opt --config=android_arm64 \ 
  --distdir=<your cache directory> \
  --repository_cache=<your cache directory> \
  --keep_going \
  mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu

[hippyk]

Yes, it's possible to cache the external dependencies and use the cached version later. We are testing the following workflow:

# On the workstation that has the full access to the Internet, fetch all the dependencies to a cache directory by doing:
$ git clone https://github.com/google/mediapipe.git
$ cd mediapipe
$ bazel fetch //...  --keep_going --repository_cache=<your cache directory> 

# Then, copy the cache directory to the machine that has restricted Internet access and do:
$  bazel build -c opt --config=android_arm64  mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu --distdir=<your cache directory> 

If the workflow works, we may host an archive that contains all the MediaPipe external dependencies for the users in China.

Stay tuned.

Any others work but the 'opencv-3.4.3-android-sdk.zip'.

[Papageno2]

Yes, it's possible to cache the external dependencies and use the cached version later. We are testing the following workflow:

# On the workstation that has the full access to the Internet, fetch all the dependencies to a cache directory by doing:
$ git clone https://github.com/google/mediapipe.git
$ cd mediapipe
$ bazel fetch //...  --keep_going --repository_cache=<your cache directory> 

# Then, copy the cache directory to the machine that has restricted Internet access and do:
$  bazel build -c opt --config=android_arm64  mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu --distdir=<your cache directory> 

If the workflow works, we may host an archive that contains all the MediaPipe external dependencies for the users in China.

Stay tuned.

hi, thanks for the above hints.

I want to install MediaPipe on CentOS7, but the internet connection error occurs (Beijing now), so I try to fetch the cache dependencies on my PC(macOS), but the above workflow didn't work on my PC, anywhere I can download all the MediaPipe externel dependencies directly ?

the following is part of the warnings and errors on PC fetching(*** is my username):

WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/unicode-org/icu/archive/release-64-2.zip failed: class java.io.IOException connect timed out
WARNING: Download from https://github.com/unicode-org/icu/archive/release-64-2.zip failed: class javax.net.ssl.SSLProtocolException Read timed out
ERROR: An error occurred during the fetch of repository 'icu':
   java.io.IOException: Error downloading [https://storage.googleapis.com/mirror.tensorflow.org/github.com/unicode-org/icu/archive/release-64-2.zip, https://github.com/unicode-org/icu/archive/release-64-2.zip] to /private/var/tmp/_bazel_***/392f0b43f8832c7859e2f502bc85241d/external/icu/release-64-2.zip: Read timed out
ERROR: /private/var/tmp/_bazel_***/392f0b43f8832c7859e2f502bc85241d/external/org_tensorflow/third_party/icu/data/BUILD.bazel:39:1: no such package '@icu//': java.io.IOException: Error downloading [https://storage.googleapis.com/mirror.tensorflow.org/github.com/unicode-org/icu/archive/release-64-2.zip, https://github.com/unicode-org/icu/archive/release-64-2.zip] to /private/var/tmp/_bazel_***/392f0b43f8832c7859e2f502bc85241d/external/icu/release-64-2.zip: Read timed out and referenced by '@org_tensorflow//third_party/icu/data:conversion_data'
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/open-source-parsers/jsoncpp/archive/1.8.4.tar.gz failed: class java.io.IOException connect timed out
WARNING: Download from https://github.com/open-source-parsers/jsoncpp/archive/1.8.4.tar.gz failed: class java.io.IOException connect timed out
ERROR: An error occurred during the fetch of repository 'jsoncpp_git':
   java.io.IOException: Error downloading [https://storage.googleapis.com/mirror.tensorflow.org/github.com/open-source-parsers/jsoncpp/archive/1.8.4

[Samuel-wei]

@chenloveheimei and I tested the current version of MediaPipe. We verified that MediaPipe demo apps can be successfully built in China now. However, due to the unstable/slow internet connection, we prefer to ask Bazel to cache the external repositories locally and search for the local archives before accessing the network.
For the users with unstable/slow internet connection, you may need to try the following build command several times to download all the needed external repositories:

$ bazel build -c opt --config=android_arm64 \ 
  --distdir=<your cache directory> \
  --repository_cache=<your cache directory> \
  --keep_going \
  mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu

I am now able to build object and face detection demos from Shenzhen, China. The problem is solved.
Thank you , you are good guy.

I had a similar issues like as you encounter issues what about above.
I had workflow above, but I am unsuccessful.
Would you give me some advises?
thank you

[Lr-2002]

HI,could i download the dependencies via firefox and the copy it into the fire storage ?
things doesn't work well,It seems bazel making different dictionary every time?

[11676670]

if you network blocked by the Great Firewall. you can change your repositories in WORKSPACE file like this.
repositories =
[
"https://maven.aliyun.com/repository/google",
"https://maven.aliyun.com/repository/central",
"https://maven.aliyun.com/repository/public",
],
Good luck.

[DLGL99GL]

如果你的网络被防火长城。 您可以更改您的存储库工作区中的文件。
存储库=
(
“https://maven.aliyun.com/repository/google”,
“https://maven.aliyun.com/repository/central”,
“https://maven.aliyun.com/repository/public”,
),
祝你好运。

May I ask which file path is modified specifically？

[11676670]

#git clone https://github.com/google/mediapipe.git #cd mediapipe #vi  WORKSPACE       //this WORKSPACE  file

…

------------------ 原始邮件 ------------------ 发件人: ***@***.***>; 发送时间: 2021年9月16日(星期四) 下午4:29 收件人: ***@***.***>; 抄送: ***@***.***>; ***@***.***>; 主题: Re: [google/mediapipe] Fail to pull in MediaPipe's external dependencies due to unstable internet connection (#3) 如果你的网络被防火长城。 您可以更改您的存储库工作区中的文件。 存储库= ( “https://maven.aliyun.com/repository/google”, “https://maven.aliyun.com/repository/central”, “https://maven.aliyun.com/repository/public”, ), 祝你好运。 May I ask which file path is modified specifically？ — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.