Before the summer OpenAI had its developer day where they presented the GPT-4o model and its capabilities. They showed a demo of an app that helps visually impaired/blind people to “see” by processing photos of what’s in front of them and playing of an audio response of the description of the photos. This inspired me to see if it was possible to recreate by using Microsoft technology. Turnes out it is! I have created a canvas app in Power Apps where you can take a photo and send it to Power Automate. Power Automate does a HTTP request with the photo to Azure OpenAI GPT-4o, and sends the description of the image over to Azure AI Speech through another HTTP request. You then get an audiofile that is being uploaded to SharePoint. When the file is created, the app plays the audiofile for you.

Here are two demos of the app, the first one shows the capabilities to describe a room and the other one shows how it can read an ingredient label even though its not in the native language of the app.
Bellow them you will find a complete step by step guide on how to set this up yourself.

Go to the Azure Portal and search for OpenAI

Click Create.

Fill out and click Next, Next, Next.

Click Create.

Copy one of the keys and the endpoint url.

Go to Azure OpenAI Studio.

Open the new studio.

Click the upper right corner and select the correct resource.

Go to deployments and click Deploy base model.

Select get-4o and click Confirm.

Set a name and select as in the picture. You can adjust the token limit a bit down probably. Click Deploy.

Copy the deployment name.

Go to the Azure Portal again and search for Speech services.

Click Create.

Fill out and click Review + Create.

Click Create.

Click Go to resource.

Copy the endpoint url.

Copy one of the keys.

Go to make.powerapps.com and click to create a blank app.

Click create under Blank canvas app.

Give it a name and select Phone. Click Create.

Add a Camera

Add an image.

Add a timer.

Add an audio component.

Make sure the components have the same name as below.

It should look like this.

Click Create new flow.

Click Create from blank.

Add an input.

Click Text.

Call it MyText

Click New step.

Select Compose.

Add dynamic content and click on MyText

Rename the flow to VoiceVisionFlow

Save the flow and go back to the app. Copy the commands below and paste under OnSelect in the Camera1 component.

Type Camera1.Photo in the image property of the Image1 component.

Create a SharePoint site and copy the URL to the document library, and replace it in the commands bellow. Copy the commands and paste in the OnTimerEnd property of the Timer1 component. Set the timer duration to 5000 and type varStartTimer in the Start property.

Set Media to varAudioFile and Start to PlayAudio for the Audio1 component.

Create an HTTP action.

Change the resource and deployment part of the URL, and enter your key/secret. Copy the JSON below and remember to add the Compose to the request.

Create a Parse JSON action.

Add the Body of the HTTP action as the content, and copy the Schema below.

Create a HTTP action. It’s important that all actions has the same name as in the screenshots. Add your Azure AI Speech resource to the URL and the key to Ocp-ApimSubscription-Key property.

Create a Compose action.

Add the body of HTTP 2 as input.

Create a HTTP action. Add your resource to the URL and set configuration as below.

Add the expression into the request.

Create an Initialize variable action.

Create an Set variable action.

Add the expression from bellow.

Create an Compose action and add the expression.

Create a SharePoint – Create file action.

Select your site/library, add a filename and Compose 3 as file content.

Create an Respond to Power App or flow action.

call it returntext and add test as the text.

That’s it, you should now have a functioning POC for VoiceVision!