tensorflow/cloud

The TensorFlow Cloud repository provides APIs that will allow to easily go from debugging and training your Keras and TensorFlow code in a local environment to distributed training in the cloud.

Documentation

TensorFlow is an end-to-end open source platform for machine learning. It has a comprehensive, flexible ecosystem of tools, libraries, and community resources that lets researchers push the state-of-the-art in ML and developers easily build and deploy ML-powered applications.

TensorFlow was originally developed by researchers and engineers working within the Machine Intelligence team at Google Brain to conduct research in machine learning and neural networks. However, the framework is versatile enough to be used in other areas as well.

TensorFlow provides stable Python and C++ APIs, as well as a non-guaranteed backward compatible API for other languages.

Keep up-to-date with release announcements and security updates by subscribing to [email protected]. See all the mailing lists.

Install

See the TensorFlow install guide for the pip package, to enable GPU support, use a Docker container, and build from source.

To install the current release, which includes support for CUDA-enabled GPU cards (Ubuntu and Windows):

$ pip install tensorflow

Other devices (DirectX and MacOS-metal) are supported using Device plugins.

A smaller CPU-only package is also available:

$ pip install tensorflow-cpu

To update TensorFlow to the latest version, add --upgrade flag to the above commands.

Nightly binaries are available for testing using the tf-nightly and tf-nightly-cpu packages on PyPi.

Try your first TensorFlow program

$ python

>>> import tensorflow as tf
>>> tf.add(1, 2).numpy()
3
>>> hello = tf.constant('Hello, TensorFlow!')
>>> hello.numpy()
b'Hello, TensorFlow!'

For more examples, see the TensorFlow tutorials.

Contribution guidelines

If you want to contribute to TensorFlow, be sure to review the contribution guidelines. This project adheres to TensorFlow's code of conduct. By participating, you are expected to uphold this code.

We use GitHub issues for tracking requests and bugs, please see TensorFlow Forum for general questions and discussion, and please direct specific questions to Stack Overflow.

The TensorFlow project strives to abide by generally accepted best practices in open-source software development.

Patching guidelines

Follow these steps to patch a specific version of TensorFlow, for example, to apply fixes to bugs or security vulnerabilities:

• Clone the TensorFlow repo and switch to the corresponding branch for your desired TensorFlow version, for example, branch r2.8 for version 2.8.
• Apply (that is, cherry-pick) the desired changes and resolve any code conflicts.
• Run TensorFlow tests and ensure they pass.
• Build the TensorFlow pip package from source.

Continuous build status

You can find more community-supported platforms and configurations in the TensorFlow SIG Build community builds table.

Official Builds

Build Type	Status	Artifacts
Linux CPU		PyPI
Linux GPU		PyPI
Linux XLA		TBA
macOS		PyPI
Windows CPU		PyPI
Windows GPU		PyPI
Android		Download
Raspberry Pi 0 and 1		Py3
Raspberry Pi 2 and 3		Py3
Libtensorflow MacOS CPU	Status Temporarily Unavailable	Nightly Binary Official GCS
Libtensorflow Linux CPU	Status Temporarily Unavailable	Nightly Binary Official GCS
Libtensorflow Linux GPU	Status Temporarily Unavailable	Nightly Binary Official GCS
Libtensorflow Windows CPU	Status Temporarily Unavailable	Nightly Binary Official GCS
Libtensorflow Windows GPU	Status Temporarily Unavailable	Nightly Binary Official GCS

Resources

• TensorFlow.org
• TensorFlow Tutorials
• TensorFlow Official Models
• TensorFlow Examples
• TensorFlow Codelabs
• TensorFlow Blog
• Learn ML with TensorFlow
• TensorFlow Twitter
• TensorFlow YouTube
• TensorFlow model optimization roadmap
• TensorFlow White Papers
• TensorBoard Visualization Toolkit
• TensorFlow Code Search

Learn more about the TensorFlow community and how to contribute.

Courses

• Coursera
• Udacity
• Edx

License

Apache License 2.0

cloud's Issues

Add license check github action

This is a tracking task to add a license checker to Github Actions CI to ensure files without license are not checked in accidentally. This should cover .cpp & .py files as well notebook files. There are a few GH actions that may be a good match here.

A test case fails when testing under environment with TF2.2

Test case test_request_dict_with_TPU_worker fails when TF2.2 (or TF version) is installed. This is caused by tpuTfVersion in request body set to TF2.2, while the test case require it to be TF2.1. Should we raise an error when tpuTfVersion is trying to be set to TF>2.1, and change this test case?

version is specified in both __init__.py and setup.py

This is a quick clean up filing this for tracking ... should preferably be only maintained in one location.

Allow caller to specify a custom job_id

This is particularly useful in case of automation and rapid iterations. It allows for easy tracking of the execution result, without needing to keep track of generated IDs and map them to external keys.

deploy_test.py - Ambiguous import: "call" from package "mock"

tensorflow_cloud/core/tests/unit/deploy_test.py

line 32: from mock import call, patch: Ambiguous import: "call" from package "mock": cannot determine whether "call" is an attribute on the package, or a module .
line 32: from mock import call, patch: Ambiguous import: "patch" from package "mock": cannot determine whether "patch" is an attribute on the package, or a module.

containerize.py - Ambiguous import: APIClient

In /src/python/tensorflow_cloud/core/containerize.py, no direct deps found for imports:

line 30: from docker import APIClient: Ambiguous import: "APIClient" from package "docker": cannot determine whether "APIClient" is an attribute on the package, or a module.

How to load a dataset from google cloud storage with tensorflow cloud?

Tensorflow cloud configuration:

  GCP_BUCKET = "stereo-train"

  tfc.run(
    requirements_txt="requirements.txt",
    chief_config=tfc.MachineConfig(
      cpu_cores=8,
      memory=30,
      accelerator_type=tfc.AcceleratorType.NVIDIA_TESLA_T4,
      accelerator_count=1,
    ),
    docker_image_bucket_name=GCP_BUCKET,
  )

And I have a bucket called gs://stereo-train that contains the dataset. The exact location of the dataset is:

gs://stereo-train/data_scene_flow/training/dat

However, when using this location like so:

tf.keras.preprocessing.image_dataset_from_directory("gs://stereo-train/data_scene_flow/training/dat", image_size=(375,1242),\
                                                         batch_size=6, shuffle=False, label_mode=None)

Behavior:

getting the error that "gs://stereo-train/data_scene_flow/training/dat" doesn't exist

Expected behavior:
tf.keras.preprocessing.image_dataset_from_directory should know that there's a gs bucket associated with the account and the dataset should be loaded.

Add multi file example

Add an integration test example with python script on multiple files.

Docker error while running setup

Hi, I have followed the "High level overview" steps on the readme to train the models on GCP. I have created both mnist_example.py and scale_mnist.py and followed all the setup instructions. However, when I run python scale_mnist.py, I encounter the following error:

docker.errors.DockerException: Error while fetching server API version: (2, 'CreateFile', 'El sistema no puede encontrar el archivo especificado.'), which translated into English says that it cannot find a file. I am using a Cloud Storage bucket in Europe-west1

My configuration:

• Windows 10
• Python 3.7.7
• Tensorflow 2.3.0
• Tensorflow Cloud 0.1.4

I have also tried on macOS Catalina with the same result.

Let me know if you need any further information. Thank you!

Update docs to recommend cloud build over local build.

Feedback from Yixing Fu:

Maybe it would be better for the tutorial to suggest the bucket approach at first over local build, as cloud-build is more consistent?

Possible to extend to other clouds?

Is there any opportunity to extend this to other clouds? It appears as though GCP is pretty deeply wired in.

dogs_classification.ipynb missing license

file src/python/tensorflow_cloud/core/tests/examples/dogs_classification.ipynb is missing a license header.

Error while creating container

Environment:

 Mac, 
 Python 3.7, 
 gcloud 
     Google Cloud SDK 300.0.0
     bq 2.0.58
     core 2020.07.06
     gsutil 4.51
     kubectl 2020.05.01

Steps:

• create the sample mnist.py, run it locally - OK
• create a script to submit it to GCP

   import tensorflow_cloud as tfc
   tfc.run(entry_point='mnist_train.py')
  

• run the script

Expected:

The script build a container and runs the job in GCP

Observed:

An exception while building container:

(tfcloud) :tf_cloud$ python run_on_gcp.py 
INFO:tensorflow_cloud.containerize:Building docker image: gcr.io/my-project/tf_cloud_train:c74be2d8_b6cc_4832_9522_b1c69ded7f95
INFO:tensorflow_cloud.containerize:{"stream":"Step 1/4 : FROM tensorflow/tensorflow:2.2.0-gpu"} {"stream":"\n"}
   Traceback (most recent call last):
      File "/Users/dournov/tf_cloud/tfcloud/lib/python3.7/site-packages/tensorflow_cloud/containerize.py", line 287, in _get_logs
    line = json.loads(unicode_line)
  File "/Users/dournov/miniconda3/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/Users/dournov/miniconda3/lib/python3.7/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 62)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "run_on_gcp.py", line 2, in <module>
    tfc.run(entry_point='mnist_train.py')
  File "/Users/dournov/tf_cloud/tfcloud/lib/python3.7/site-packages/tensorflow_cloud/run.py", line 188, in run
    docker_img_uri = container_builder.get_docker_image()
  File "/Users/dournov/tf_cloud/tfcloud/lib/python3.7/site-packages/tensorflow_cloud/containerize.py", line 235, in get_docker_image
    image_uri = self._build_docker_image()
  File "/Users/dournov/tf_cloud/tfcloud/lib/python3.7/site-packages/tensorflow_cloud/containerize.py", line 255, in _build_docker_image
    self._get_logs(bld_logs_generator, 'build')
  File "/Users/dournov/tf_cloud/tfcloud/lib/python3.7/site-packages/tensorflow_cloud/containerize.py", line 294, in _get_logs
    'There was an error decoding the Docker logs')
RuntimeError: There was an error decoding the Docker logs

containerize_test.py - Ambiguous import: "call" from package "mock"

In src/python/tensorflow_cloud/core/tests/unit/containerize_test.py

Import statement should be changed to import module not attribute.

line 24: from mock import call, patch: Ambiguous import: "call" from package "mock": cannot determine whether "call" is an attribute on the package, or a module 
line 24: from mock import call, patch: Ambiguous import: "patch" from package "mock": cannot determine whether "patch" is an attribute on the package, or a module not provided by a direct dependency.

core/tests/unit/* references testdata files that aren't used

reported by @ucdmkt

performance improvement on integration tests.

This is to decrease the cost and time of running integration tests. We can lower the number of epochs / steps to 1-2. Additionally decrease the dataset size used for integration tests.

Avoid using private APIs in implementation

In src/python/tensorflow_cloud/core/deploy.py, there should not be a call to protected API under `SubProcessError.

Plus, I don't believe _get_reason() is even implemented for SubProcessError, according to source code

Distributed training for Keras-Tuner

After some experimenting, using keras-tuner with chief-worker distribution can (only) be done by setting KERASTUNER_ORACLE_IP to 0.0.0.0 on chief, but the actual chief IP obtained from TFCONFIG on worker replica.
Distributed training is certainly useful when we support keras-tuner. Should we implement this special case here, or just create an example showing how to distribute keras-tuner clearly?

Alternatively this could be solved from the keras-tuner side, but from what I see this is only a problem putting KT on AI platform using tensorflow-cloud; but not an issue for KT distribution in general.

Instructions missing

Looks like you need to define:

1. GCP Project
2. GCP authentication
3. Google Cloud SDK
4. Docker client

I can submit a PR with the requirements.

Instruct users to keep images small by using dedicated project directories.

Calling run() on a directory which includes a virtual environment or other large files will either time-out or take hours to build. Any virtual environment used for Cloud will contain tensorflow, so we should by default recommend that users create a dedicated subdirectory for each project, containing only the essentials.

Allow passing in Google Cloud Credentials (instead of application default credentials)

Kaggle Notebooks supports integrations with several Google Cloud services by authenticating the user via OAuth and providing a credentials object (for example https://www.kaggle.com/product-feedback/163416).

We'd like to provide support for tensorflow_cloud on Kaggle using user's existing auth via this mechanism. One way to do that would be allowing the library to have a credential object passed in on initialization (like other Google Cloud client libraries).

Another approach, which I've sort of hacked together below[1] is to take the credential object and write it to the filesystem as the GOOGLE_APPLICATION_CREDENTIALS file. This actually works and is implemented by the google_auth library but it always returns a None project id (https://github.com/googleapis/google-auth-library-python/blob/9058f1fea680613d9717a62ee37dc294c11b9c8a/google/auth/_default.py#L126) so tensorflow_cloud throws an error:

cloud/tensorflow_cloud/gcp.py

Line 27 in dd5c188

if project_id is None:

. Would it be possible to support this approach, but have another mechanism to pass in the project id, in a constructor for example?

Is there any constraint that disallows using user credentials (and requires service accounts to be used)?

[1]

import json
import os
from kaggle_secrets import UserSecretsClient
user_secrets = UserSecretsClient()
user_credential = user_secrets.get_gcloud_credential()
with open('/tmp/key.json', 'w', encoding='utf-8') as f:
    f.write(str(user_credential))
os.environ['GOOGLE_APPLICATION_CREDENTIALS']="/tmp/key.json"
!gcloud config set project vimota-project2

Bug while pulling 2.3.0rc0 docker image

If the user have tf 2.3.0rc0 installed locally.
TF cloud would try to pull docker image tensorflow/tensorflow:2.3.0-rc0-gpu which doesn't exist. The correct tag is tensorflow/tensorflow:2.3.0rc0-gpu, which doesn't have the - between 2.3.0 and rc0.

switch testing framework for pytest to abseil

https://abseil.io/docs/python/guides/testing, this will allow us to benefit from some of the test automated infrastructure namely be able to share nicely formatted test results back on GH for integration tests.

Support Cloud Build

Since images are built locally then pushed to GCR, for limited local resources will be nice to add support to build in GCR and then Train model. With this method the Docker image is built in Google Cloud infrastructure.
Google Cloud Build: https://cloud.google.com/cloud-build/docs/running-builds/start-build-manually#api

Move python/tensorflow_cloud/core to python/core

This is tracking to restructure the repo once integration tests are up and running.

validate_test.py - Ambiguous import: "patch" from package "mock"

src/python/tensorflow_cloud/core/tests/unit/validate_test.py, no direct deps found for imports:
line 20: from mock import patch: Ambiguous import: "patch" from package "mock": cannot determine whether "patch" is an attribute on the package, or a module not provided by a direct dependency.

Setting stream_logs=False, does not seem to be working

sample

import tensorflow_cloud as tfc

# Automated MirroredStrategy: chief config with multiple GPUs
tfc.run(
    entry_point="../../tests/testdata/mnist_example_using_fit_no_reqs.py",
    distribution_strategy="auto",
    chief_config=tfc.MachineConfig(
        cpu_cores=8,
        memory=30,
        accelerator_type=tfc.AcceleratorType.NVIDIA_TESLA_P100,
        accelerator_count=2,
    ),
    worker_count=0,
    stream_logs=False,
    docker_image_bucket_name="<some_bucket>",
)

output

Job submitted successfully.
Your job ID is:  tf_cloud_train_..._67a20a7a68d1
Please access your job logs at the following URL:
https://console.cloud.google.com/mlengine/jobs/tf_cloud_train_..._67a20a7a68d1?project=...
Streaming job logs: 

expected job logs not to stream.

Add magic phrase support to rerun tests

Currently to re run tests need to submit an empty commit

git commit -m "retest" --allow-empty and then 
git push <branch-name>

This is a request for a nice to have feature to enable rerun of tests with a magic phrase such as "/retest" or "/rerun tests".

Authentication error when running the run command on Colab

When running the Colab example I am able tu run the code and successfully save the model inside the created bucket, but the run command keeps trying multiple GETs returning:

"Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project."

The run command I am using is

f = open('requirements.txt', 'w')
f.write('tensorflow-datasets\n')
f.write('pandas')
f.close()

tfc.run(
    entry_point=None,
    distribution_strategy="auto",
    requirements_txt="requirements.txt",
    chief_config=tfc.MachineConfig(
        cpu_cores=8,
        memory=30,
        accelerator_type=tfc.AcceleratorType.NVIDIA_TESLA_P100,
        accelerator_count=2,
    ),
    docker_image_bucket_name=BUCKET_NAME,
    docker_base_image='tensorflow/tensorflow:2.2.0-gpu',
    worker_count=0
)

I had to add docker_base_image='tensorflow/tensorflow:2.2.0-gpu' since I was getting a message saying that there is no docker for the TF version 2.2.0.

End to end example with prediction via REST API

Requesting a feature:

Could you develop the library further for deployment management, or is it already in the roadmap,
nevertheless can an example workflow be added for deployment and support REST API via gcloud SDK management
as best practice ?

Price callback

It would be nice to have a price callback to monitor job costs.

setup.py execution depends on current working directory

This is a small issue that impacts users if they build the package from any folder other than python. For example if users runs the build package command as

python python/setup.py bdist_wheel

The build will complete with an empty package, source will not include any of the modules.
to mitigate this issue setup.py should set current working directory to setup.py location.

import os
setup_file_directory = os.path.dirname(os.path.abspath(__file__)
os.chdir(setup_file_directory)

TensorFlow Cloud on Colab

Is there any way of using this tensorflow-cloud repository on Google Colab?

Improving cloud build timeout

Feedback from Yixing Fu:

Runtime Error: There was an error executing the cloud build job. Job status: WORKING.

This error description is not very helpful. Also from GCP console the build is actually successful. This error did not come up the second time I tried it. I am guessing it is caused by creating a docker image that took a long time and triggered some time out?
Update: Got this again today. Seems build time > 5 produces this problem, and build time <3 are fine.
Update: Now that I looked a bit into the code, this error is clearly raised by the 10 trial, each 30 second limit. Maybe there should be a check to see if the job status is still “WORKING”, raise a warning that building time is long, and only kill it for much longer time or when job status is something else?

Bug cannot import tensorflow when running on AI platform

If I run tfc with this base image docker_base_image="tensorflow/tensorflow:2.2.0-custom-op-gpu-ubuntu16,
it would successfully submit the job, but got
ImportError: No module named tensorflow in the cloud log.
Log Link

tfc.run() should return job_id

Either tfc.run() should return the job_id or have a method that allows to retrieve the job id.

Also a follow up ask ( nice to have) here would be to directly return the job status (running, failed, succeeded ) as a Enum for automation.

log streaming does not terminate if a job is terminated while streaming logs

I started a job with log streaming enabled. While the job was running I manually terminated the job in AI Platform training by clicking on stop job in the UI. The log streaming continued to run after 5 minutes until manual termination with the following:

  File ".../cloud/tensorflow_cloud/deploy.py", line 211, in _stream_logs
    output = process.stdout.readline()

I expected when the job is terminated on AI Platform training the log streaming should also terminate / stop.

Add cloudtuner and cloud_fit to github actions

github actions currently only runs on core tests ... this is to add github actions for cloud_fit and cloudtuner

Avoid using "from . import" in import statement

This is a tracking bug to update the import statements for core to avoid using import from "." This can create long term maintenance issue with copybara.

cloud build fail when tf-nightly installed locally

In an environment with tf-nightly, trying to run tfc with cloud build fails.

The failure is caused by dockerfile use version to determine base image. In the case of tf-nightly, it gives something like 2.x.0-dev2020mmdd.

It should be good to check if corresponding image exist before starting build, and throw relevant error message. Also, for the case of nightly build, can we just use nightly for all versions labeled with 'dev'?

Refactor TFCloudHttpRequest to a common utils

To be able to use it form core, tuner, cloud_fit etc.

validate.py import error 'tensorflow.python.framework.versions'

validate.py",

line 27, in : Can't find module 'tensorflow.python.framework.versions'. [import-error]

recommended method is to check the loaded module versions

import pkg_resources

version = "2.2.0"
for pkg in pkg_resources.working_set:
    if "tensorflow" == pkg.key:
        version = pkg.version

Permission Error in Windows

Tensoflow cloud version: (0.1.5)
Python version : 3.7.6
OS: Windows 10 64-bit

When running from a notebook:

import tensorflow_cloud as tfc
tfc.run(entry_point='mnist.py',
        docker_image_bucket_name="a-test",
)

mnist.py file is as in example :

import tensorflow as tf

(x_train, y_train), (_, _) = tf.keras.datasets.mnist.load_data()

x_train = x_train.reshape((60000, 28 * 28))
x_train = x_train.astype('float32') / 255

model = tf.keras.Sequential([
  tf.keras.layers.Dense(512, activation='relu', input_shape=(28 * 28,)),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10, activation='softmax')
])

model.compile(loss='sparse_categorical_crossentropy',
              optimizer=tf.keras.optimizers.Adam(),
              metrics=['accuracy'])

model.fit(x_train,  y_train, epochs=10, batch_size=128)

Result:

INFO:tensorflow_cloud.core.containerize:Uploading files to GCS.

INFO:tensorflow_cloud.core.containerize:Building and publishing docker image using Google Cloud Build: gcr.io/dydra- 282320/tf_cloud_train:371ef57f_1db8_4c96_aae6_09e373926732

---

PermissionError Traceback (most recent call last)
in
1 import tensorflow_cloud as tfc
2 tfc.run(entry_point='mnist.py',
----> 3 docker_image_bucket_name="a-test",
4 )

~\anaconda3\lib\site-packages\tensorflow_cloud\core\run.py in run(entry_point, requirements_txt, distribution_strategy, docker_base_image, chief_config, worker_config, worker_count, entry_point_args, stream_logs, docker_image_bucket_name, job_labels, **kwargs)
224 # Delete all the temporary files we created.
225 if preprocessed_entry_point is not None:
--> 226 os.remove(preprocessed_entry_point)
227 for f in container_builder.get_generated_files():
228 os.remove(f)

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\Users\semih\AppData\Local\Temp\tmpkjul06z0.py'

Make the file path in integration tests relative

Currently integration tests assuming they are being ran from root, this is to make their path relative so that regardless of where they are being executed they can be run

current format

tfc.run(
    entry_point="tests/testdata/mnist_example_using_ctl.py", ...

suggested path look up

# Path to the source code in test environment
TEST_DATA_PATH = os.path.join(
  os.path.dirname(os.path.abspath(__file__)), '../testdata/')

...
tfc.run(
    entry_point=os.path.join(TEST_DATA_PATH, 'mnist_example_using_fit_no_reqs.py'), ...

Decouple core/tests/unit/* from strcit pytest dependency

reported by: @ucdmkt

Bug tfc uses python2 in the cloud container

When I have tf version 2.1.0 installed locally with python3 interpreter, not providing the docker_base_image arg to tfc.
tfc would use python2.7 in the cloud container.
File "/usr/local/lib/python2.7/dist-packages/kerastuner/engine/hypermodel.py
log link

Bucket name ending with '/' may error out.

From: @yixingfu

Path ending with / makes a problem. For example, when setting path as ‘gs://[BUCKET_NAME]/saves’ works fine, but ‘gs://[BUCKET_NAME]/saves/’ breaks down when trying to reload in remote. For example, running “call_run_on_script_with_keras_save_and_load.py” in the integration test, the log stops updating after the following one:
But the job keeps running. I manually killed the job after a while. I think it is because the extra ‘/’ preventing it from correctly loading. Maybe an error should be raised? Or remove any trailing ‘/’ when parsing the path?

Re-use images (only changing parameters)?

My current workflow is to use the gcloud CLI to submit jobs with a common base image and differ the parameters (used in a CI setting). I like that in tensorflow cloud that the machine types and accelerator types are expressible as code, but I don't want to rebuild the image (and re-upload) the image each time I train.

Is there anyway to re-use the same image (but just supply a different 'trainer' file)? What I'd like to do is to push the trainer file + hyper parameters (as YAML or JSON) and keep the image the same across runs (except when my dependencies change).

I see that you can specify a custom base image, but AFAICT a new image would still be built / pushed.

Add user agent instrumentation for run API.

Exclude tests from setup.py

Test files are currently included in the pypi package, this is to update the setup.py file to exclude tests.

find_packages("core", exclude=["tests"])

Alternatively we can move tests folder up to /python.