azure/gen-cv

Vision AI Solution Accelerator

Vision AI Solution Accelerator

This repository serves as a rich resource offering numerous examples of synthetic image generation, manipulation, and reasoning. Utilizing Azure Machine Learning, Computer Vision, OpenAI, and widely acclaimed open-source frameworks like Stable Diffusion, it equips users with practical insights into the application of these powerful tools in the realm of image processing.

Content

• 🆕 Guided Image Generation
• 🆕 Find and analyze Videos using GPT4-Vision with Video Enhancements
• Create engagaing Avatar Videos
• Create interactive Avatar Experiences
• Stable Diffusion XL with Azure Machine Learning
• Azure Computer Vision in a Day Workshop
• Explore the OpenAI DALL E-2 API
• Create images with the Azure OpenAI DALL E-2 API
• Remove background from images using the Florence foundation model
• Precise Inpainting with Segment Anything, DALLE-2 and Stable Diffusion
• Add custom objects and styles to image generation models with Dreambooth
• Create and find images with Stable Diffusion and Florence Vector Search
• Manage image embeddings with the Cognitive Search Vector Store

Getting Started

The code within this repository has been tested on both Github Codespaces compute and an Azure Machine Learning Compute Instance. Although the use of a GPU is not a requirement, it is highly recommended if you aim to generate a large number of sample images using Stable Diffusion.

Follow these steps to get started:

1. Clone this repository on your preferred compute using the following command:

git clone https://github.com/Azure/gen-cv.git

2. Create your Python environment and install the necessary dependencies. For our development, we utilized Conda. You can do the same with these commands:

conda create -n gen-cv python=3.10
conda activate gen-cv
pip install -r requirements.txt

3. From the list provided above, select a sample notebook. After making your selection, configure the Jupyter notebook to use the kernel associated with the environment you set up in Step 2.
4. Copy the .env.template file to .env to store your parameters:

cp .env.template .env

5. Add the required parameters and keys for your services to the .env file.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

gen-cv's Issues

Value AzureWebJobsStorage

Hi Team,

I am trying to build the Avatar Interactive and I found a few issues on of them is AzureWebJobsStorage I don't understand from where I can get this value please?

Thanks
Tony

crop the video generated from Text to Speech Avatar

Hi,

Need to crop the video generated from Text to Speech Avatar part of our solution. Any references or which way we can crop the video either by using configuration or programmatically. using Speech SDK for JavaScript.

thank you.

Question: about ICE_CONNECTION_STRING

In ./avatar/interactive./README.md, It've noted that you'll need to fill in the credentials in local.settings.json
However, I'm curious about how can I retrieve the last input ICE_CONNECTION_STRING ?
Which Azure resouce credentials is this in? Thanks.

Cuz tihs error occurs while I'm trying to run the solution locally, and I assume it's caused by this.
[api] Exception: ValueError: Invalid connection string. You can get the connection string from your resource page in the Azure Portal. The format should be as follows: endpoint=https://<ResourceUrl>/;accesskey=<KeyValue>

Resize Text to Speech Avatar video

Hi,

How to resize & remove background color for the video generated from Text to Speech Avatar part of our solution. Any references using Speech SDK for JavaScript.

thank you.

Interactive avatar & login page not working

Issue is resolved.

Interactive avatar is not coming after Speech and Ice token fetch succeeds

I am trying to run interactive avatar locally on a newly created azure free account. I have created all the resources mentioned in the readme file. When I run the app locally, both the APIs getSpeechToken and getIceServerToken succeeds, creates PeerConnection as well but it's throwing error in the below line:
speechSynthesizer.setupTalkingAvatarAsync(JSON.stringify(clientRequest), complete_cb, error_cb)

Login and Password unknown

Hello,

I have deployed the full project and I have got a webpage asking for username and password and I am not able to move forward. What is the username and password please?

Thanks
Tony

Interactive avatar not working

Hi @harmke

Hope you are doing good, Happy Thanksgiving
We tried deploying interactive avatar,

But we are getting login page with username and password, if we remove the page we are just getting text and no avatar
web App code doesn't seem to work,

Can you please share latest web app code
also is the ICE connection string is communication service connection string ?

Thanks
Radhakrishnan Guhan

SyntaxError: Unexpected end of JSON input at main.js:56:32

I've already tested this soultion successfully locally, which everything works fine.
However, while deploying it to the Azure Static Web, after I clicked on the login button, the following error occurs and it won't jump to the main page.

Again, it works great while running locally. This only happened after I deploy this apllication to Azure Static Web Apps.