The AI War Is On! Every Microsoft Build AI Announcement

The AI War Is On! Every Microsoft Build AI Announcement

This is the moment for each and every one of us. We now possess the capabilities in our hands, thanks to these new tools in the early stages of this fresh platform, to accomplish truly remarkable feats. Our challenge, without a doubt, is to embark on a journey that will leave a lasting impression on others, evoking awe and admiration. Just last November, we experienced a significant upgrade, transitioning from the bicycle era to the steam engine era with the introduction of chat GPT. In fact, every layer of the software stack is undergoing a permanent transformation. We anticipate over 50 announcements, and I would like to emphasize five of them.

the first

is we're bringing search grounding and

Bane to chat GPT chat GPT is the most

fast growing consumer app we've ever

seen and such grounding is very key

feature right so that all the

information is current and grounded by

what you have from the crawl on the

index next we're bringing the co-pilot

to the biggest canvas of all windows

this is going to make every user a power

user of Windows the other thing that

we're also very excited to launch is the

co-pilot stack we want to make that

available so that everyone here can

build their own co-pilot for their

applications so to be able to show you

everything in action from both the

plugin extensibility to all of the

co-pilots we announced let me invite up

on stage yusufmady to come show you all

of this Yusuf let me try it

hi everybody

we're making Fast Pro

G liberating our vision of your co-pilot

for the web and for business and today

as Satya said we're excited to announce

that we're going to bring chat GPT and

Bing together as with the default search

experience to give you higher quality

answers and more timely answers let's

take a look

here I am in chat GPT and as you can see

now Bing is the default and when I come

in and select it I can now ask sort of

real-time queries for example let's ask

what I should expect to hear from about

build and Dot net and what you can see

is the results now are more up to date

they include fresh content and they

include citations in fact if you can see

the links on that page there you can

click those those will take you straight

to a web page that's sourced by big

we're also excited to announce that

we're going to bring interoperability

between chat GPT and Bing for plugins so

you write them once and they're going to

run everywhere so as you can see here in

chat GPT I've got Zillow and instacart

enabled but I want to show them to you

here in Bing chat so we'll flip over and

you can see again I've got the same

plugins now in both Bing chat and in

chat GPT

and what we're going to show you now is

I'll do a search here for houses in

Chicago and I can ask for a set of

criteria learn a little bit about the

neighborhoods and now I can

automatically call Zillow by saying hey

give me three houses in a certain price

range that meet my criteria now what you

can see is now I get these great options

and I'm also going to get all of the

other great things you get with Bing

like helpful City Guides and maps and

prompts

I'm going to tell you how we're going to

further add value to the plugins that

you write they're going to work not just

in chat and chatgpt they're going to

work across the entire web courtesy of

the edge browser so here's an example

I'm on a web page here checking out a

recipe for a cake and now I can call

Bing chat and ask it to tell me hey give

me the ingredients from this webpage and

notice Bing can read the context of the

web page understand those ingredients

put them into chat and then I can say

hey give me a shopping list for this and

it'll automatically call the instacart

plugin take those ingredients right off

the page and put them into an instacart

shopping and with one click I can get

those now delivered to my house this is

incredible productivity benefit for

people

as Satya shared we're excited to

announce the windows co-pilot I think

it's going to change how you use your PC

forever let me show it to you here I am

in the coding project on my PC but I

want to configure my PC to help me be

more creative and more productive all I

have to do now is invoke the windows

co-pilot and now just come down here to

the taskbar I click on that and now

we'll pop up the windows copilot on the

right the side pane here will be

consistent across every app that you use

on your PC and just like with Bing chat

I can now ask it questions like how can

I adjust my system to get work done and

not only will I get a bunch of great

suggestions but watch this I can now

with one click take action on those

suggestions for example I can put into

Focus mode

I also know that as developers we like

dark mode there's a suggestion here for

dark mode so with one click I'm now here

on the dark side and and to really get

going I want to get that coding playlist

going so I'll pull the plugin from

Spotify and say give me a great coding

playlist in this case Chill Vibes will

come up and now I'll have it ready to go

and finally there's a suggestion here

that says hey to organize your PC let's

take advantage of snap so with one click

it snaps all the windows right in the

place I need them so I can be super

productive what do you think

[Applause]

incredible array of powerful AI powered

co-pilots we've got over 50 plugins

already available for customers and

thousands more coming I can't wait to

see what you're all going to build thank

you very much

as Yusuf said fantastic momentum already

building and this is about really

creating that opportunity for developers

to reach all users across all of these

surface areas and of course all AI

applications start with data

and

we're really thrilled to be announcing

Microsoft fabric it's probably perhaps

the biggest launch of a data product

from Microsoft since the launch of SQL

Server it unifies compute and storage it

unifies all of the full analytics stack

product experiences it brings together

governance so it unifies governance with

Analytics

and most interestingly it unifies the

business model right across all the

different type of analytics workloads

whether they're SQL machine learning

whatever job you want and this

unification at the end of the day is

what I think will fuel the next

generation of AI applications let's roll

the video introducing Microsoft fabric a

unified data analytics platform one

product one Experience One architecture

one business model

unified data is stored in one link a SAS

data Lake for the entire organization

data is integrated and stored in an open

format allowing one copy to be used to

train machine learning models visualize

data and run SQL queries on the lake and

data warehouse

a unified experience brings together all

the tools data professionals need

pipelines for orchestrating data

movement experiments for training

machine learning models semantic models

for defining key metrics and much more

and for business users fabric brings

together data for collaborating and

doing ad hoc analysis in Microsoft 365.

unified governance security and

compliance is built in for all your data

and with copilot the Microsoft fabric AI

helps everyone be more productive

whether it's writing SQL statements

building reports or setting up

automations based on triggers

all your data all your teams all in one

place this is Microsoft fabric

[Applause]

you know what the AI supercomputer did

for the infrastructure layer

Microsoft fabric will do to the data

layer for this next generation of AI

applications now

one of the things that I think we should

ask ourselves as developers is why do we

build

why do we build technology

this relationship between

economic progress and economic growth

and technology has been there for a long

time in fact this graph when you see it

it's pretty stunning that for most of

human history

we didn't have much economic growth

nor did we have much technology

and then something happened 250 years

ago

right which was long in building by the

way from perhaps the enlightenment to

Scientific Revolution to the Industrial

Revolution was close to 400 years

but then there was real progress you see

that slope going upwards

and then of course over the last 70

years information technology has played

a role across all of those sort of

seminal moments on the March towards

that dream machine and of course we now

enter the age of AI and we get to Define

what that slope looks like going forward

for economic growth but it's not even

just economic growth on its own right we

don't build just because we want

economic growth we want economic growth

so that we can have human development

index growth we want the lifespans to go

up we want education and prosperity and

standard of living to go up everywhere

that's why we build that's why we

innovate that's why technology exists

it's not for Technology's sake but it is

for that broad impact it really is

inspiring to see this technology getting

diffused so quickly and having a real

positive impact across the globe I guess

it goes without saying that a lot has

changed in the world of technology over

these past four years

an incredible amount of attention being

paid right now to what's happening with

the rapid progress with these AI models

these Foundation models that we're

calling them now and in particular like

the rapid pace of innovation that's

being driven by open AI in in their

partnership with Microsoft so let's talk

about co-pilots we sort of think of chat

GPT is fitting this copilot pattern Bing

chat certainly GitHub co-pilot the

Microsoft security co-pilot Microsoft

365 co-pilot and designer so the thing

that we noticed is we were building

these co-pilots starting with GitHub

co-pilot several years ago is that the

idea of a co-pilot is actually pretty

General so this notion that you're going

to have a multi-turned conversational

agent-like interface on your software

that helps you do cognitively complex

things applies to more than just helping

someone do software development like we

have search co-pilots now we're going to

have security co-pilots we have

productivity copilots and we're going to

have all of the co-pilots that you all

build and the thing that we noticed for

us at Microsoft is that we needed to

look at what is common across all of

these things so that we can understand

how to design great user experiences and

what the technology stack is that is

going to empower us to deliver these

things safely responsibly cost

effectively at scale so there are three

boxes you can think of these as roughly

corresponding to the three tiers of a

normal application so you've got a front

end you've got a mid-tier you've got a

back end the thing that's a little bit

different in a co-pilot is you're going

to spend less time thinking about like

what your user interface widgets are and

like trying to second guess the user

about what it is they want because they

have this really natural mechanism to

express what it is they want natural

language and so like what you have to

think about in the design of these

co-pilots is like what it is you want

the co-pilot to actually be capable of

so like what are the things a model

can't do that you need to augment with

for instance with GitHub co-pilot like a

bunch of the work that we did is to like

keep the model on task which is help

helping you solve your development

problems like you're not trying to

figure out what the best menu item is on

Taco Bell when you're sitting in GitHub

co-pilot trying to write a piece of code

so now let's talk about orchestration so

orchestration is like the business logic

of your co-pilot one of the things that

we did that greatly affected our ability

to get these co-pilots out to Market at

scale and to do more ambitious things

was to decide that inside of Microsoft

we are going to have one orchestration

mechanism that we will use to help build

our apps so inside of the orchestration

layer the the fundamental thing that

you're going to be manipulating is a

prompt and a big part of handling those

prompts at the beginning stages of

orchestration is prompt and response

filtering so basically saying I'm not

going to allow these prompts through

because maybe they will cause the model

to respond in a way that doesn't meet

the needs of your application or do

something unsafe I mean you also filter

responses on the way back up so after

the model has produced a response to the

prompt like you may decide that you want

to filter some or all of the prompt out

you also have this unit of prompt code

called metaprompt and the metaprompt is

the standing set of instructions that

you give to your co-pilot that get

passed down to the model on every turn

of conversation that tells it how to

accommodate itself to the co-pilot that

you're trying to build it's where a

bunch of your safety tuning is going to

happen it's where you sort of tell the

model like what personality you want it

to have so like for instance we use the

metaprompt to do things like

telling Bing chat to be more balanced

the verse is more precise it is also

like how you sort of teach the model new

capabilities so like you can even think

of metaprompt design as a form of

fine-tuning this is the co-pilot sack

top to bottom and what I want to do now

is make this maybe a little bit less

abstract by talking about a co-pilot

that I wrote

so I host a podcast called behind the

tech and every month when the podcast

airs my team comes and bugs me to write

a social media post to advertise the

podcast and I suck at this like I forget

to read my emails like they have to bug

me over and over again and really want a

Kevin's social media co-pilot so they

don't have to go through the irritation

of dealing with me and so I had the

honor recently of interviewing Neil

deGrasse Tyson on the podcast and so

like I'm just going to walk you through

this co-pilot that we built that we're

actually just ran and it did the social

media post for the Neil syson podcast

that just went live

so

here's what it looks like so just

end-to-end picture like the co-pilot

runs on a Windows PC it uses a mixture

of Open Source models and hosted models

it like does retrieval augmented

generation and it calls a plug-in to

finish doing its work so let's walk

through these step by step

so the first step of this process is we

have an audio file and we need a

transcript so on our Windows PC we take

the open AI open source whisper model

and run the audio through the model to

get a transcript does a really amazing

job so once we have the transcript like

the next stage in the orchestration is

we have the databricks dolly 2.0 12

billion parameter large language model

running on our Windows PC and we ask it

some things about the transcript like

for instance who was the guest this

episode because again we want to do this

like lights out like not have to have

Kevin answering a bunch of questions

because he's slow and annoying so the

next thing we do once we have the

transcript and we have all of this

information that we've extracted from

from the transcript like we want to send

a chunk of that to the Bing API or like

we want to send Neil's name to the Bing

API to get a bio and then we're going to

combine all of this stuff together into

a single packet of information like a

big prompt that has some stuff about the

transcript some and some stuff about

Neil and we get our social media blur

like this is a pretty good blurb so

we're going to go to the next step here

which is like we need a thumbnail so we

call our hosted open AI API to get an

image from the dolly model this looks

pretty good it's Cosmic it's sort of

podcasty like plenty good enough for

this this post and so the last step is

we want to invoke a plug-in for LinkedIn

that will take the thumbnail and the

post and the link to the podcast and

just post it to my LinkedIn feed one of

the super important things about this is

we are like before we take an action on

the user behalf we want to present to

them like what it is that's going to

happen because if for some reason or

another the model went haywire and

produced something that we didn't want

to post like once I hit yes this is

going to 800 000 people on LinkedIn so

we review we click yes and we post

so uh and this is the live post that's

like on LinkedIn right now you should go

check out this episode with Neil it's

awesome

[Music]

and so this is the moment for all of us

now we have capabilities in our hands

with these new tools in the early days

of this new platform to absolutely do

amazing things where literally the

challenge for you all is to go do some

legendary

that someone will be in awe of you for

one day

and so with that I would like to bring

to Stage my colleague Executive Vice

President of cloud and AI the legend

himself Scott Guthrie

[Applause]

well good morning everyone AI is going

to profoundly change how we work and how

every organization operates

and every existing app is going to be

reinvented with AI and we're going to

see new apps be built with AI that

weren't possible before

and Kevin just walked us through the

co-pilot stack and explained the

different layers involved in Building

Solutions that take deep advantage of

large AI models I'm going to build on

his talk and walk you through how we're

going to make it easy to develop these

AI Solutions using Azure GitHub and

visual studio now as Kevin explained

earlier AI orchestration involves

grounding prompt design and Engineering

evaluation and AI safety and these are

kind of the core fundamentals for how

you create great co-pilot experiences

and the reason we've been able to move

at this fast pace is because we built

all of these co-pilots on top of one

platform which is azure Ai and this is

the exact same platform that all of you

can also leverage as you build your own

copilot Nai experiences as well

Azure AI includes several categories of

AI capabilities including our Advanced

Azure AI infrastructure an Azure open AI

service is our newest capability and

became generally available in January of

this year

the Azure open AI service enables

developers and organizations to access

and use the most advanced AI models in

the world including both chat GPT and

gpd4 and all of these Microsoft co-pilot

experiences that you're seeing here at

build are built using these Azure open

AI models now what makes the Azure

opening AI service so powerful is that

it enables you to ground and fine-tune

AI models with your data and this

enables get much more intelligent and

tailored outputs from the AI models

and that's why more than 4 500 customers

are already trusting and using the Azure

open AI service today it's the fastest

growing service in azure's history and

let's now dive deep on to our Azure AI

infrastructure

we've invested heavily over many years

now to make Azure the place to do

Cutting Edge AI innovation Azure is very

much now the world's AI supercomputer

and it's the infrastructure that's

powering not just the chat CPT

experience but all and all of the AI

products and services coming from

Microsoft but it's also the AI super

computer that was used to train the

large Foundation models that openai

produced

and to do this we've built purpose-built

AI infrastructure to deliver reliability

and performance at scale we built the

largest AI model training system in the

world and one with tens of thousands of

interconnected gpus and fast networking

a picture here is one of our new Nvidia

Hopper based GPU systems Azure was the

first cloud provider in the world to

deploy and offer hopper-based GPU

systems to customers

and the picture here you can see a lot

of cables these These are infiniband

cables and they're really important for

our GPU clusters and it's one of the

ways that we really differentiate

ourselves from others I mean either AWS

nor Google support infiniband today and

infiniband gives Azure the highest

Network bandwidth and lowest Network

latency of any cloud hyperscaler in the

world today

that picture shows you what the servers

are and I'll switch gears now and take a

look at what one of these data center

facilities looks like the picture on the

top right you can see one of our new

data centers that we're building to add

to our Azure Cloud region in Dublin and

those little things are tractors and

cars so it gives you a sense of the

scale and that building is just one data

center that we're commissioning and

adding into that region

and to give you a sense of the scale

that Azure is now operating at we'll now

bring live more than 120 data centers

this calendar year alone that's one new

data center every three days

in most of these data centers like this

one are ones that we're designing and

building ourselves and multiple data

centers that make up what we call an

Azure region that you can deploy across

now our Azure region in Dublin is made

up of many data centers like this one

and it's total size is already over a

mile and a half long and half a mile

wide an area is 2 million square feet of

data center floor space that's already

live that you can deploy your workloads

in today and to put two million square

feet of in perspective that's the

equivalent of 35 football stadiums in

size all just in one region inside

Dublin

and Dublin is less than three percent of

our total data center footprint around

the world today we now have more than 60

Azure regions live Dublin is just one of

those dots there on the map it's more

regions and more countries than any

other cloud provider and we're bringing

all of the scale and all this power to

you so you can build amazing Solutions

on top of it

now the Azure AI infrastructure is being

used for many different AI models and

scenarios we're talking at this

conference and today a lot about the

great work we're doing with openai and

llms on Azure but we're also using this

exact same infrastructure for many other

AI models and experiences as well and

another unique AI partnership we have is

with Nvidia

Nvidia is bringing its AI software to

Azure by integrating nvidia's AI

Enterprise with Azure AI organizations

are going to be able to have a secure

Enterprise ready platform to quickly

build and deploy custom machine learning

models and nvidia's on Universe Cloud

provides a full stack environment to

design develop deploy and manage

industrial metaverse applications and

this gives Enterprises access to the

full stack Suite of Omniverse software

applications and Nvidia OBX

infrastructure to scale and combine it

with a scale and security of azure Azure

is the only cloud provider offering

Omniverse Cloud today

and with underverse cloud and Azure

organizations like BMW are now able to

build truly virtual factors the BMW team

can aggregate data into massive high

performance models connect their domain

specific software tools and enable

multi-user live collaboration across

Factory locations they can run real-time

twins digital twin simulations virtually

optimizing layouts Robotics and

Logistics systems even years before a

factory opens and it's all possible from

any location and on any device

now data all the AI Innovation that

we've seen today is built on top of a

foundation of data data is really the

fuel that powers Ai and AI is only as

good as your data and so it's never been

more important to have a great data

analytics and data management foundation

in place

and this week it build we're excited to

announce Microsoft fabric a unified

platform for data analytics really

designed for the era of AI

with Microsoft fabric we're bringing

together power bi data Factory and the

next generation of synapse and fabric

unifies all these analytics tools at

every layer into one seamless product

providing a single source of Truth for

everyone in the organization and a

single way to secure it and a unified

business model will help ensure that all

your resources are used in the most cost

effective way and this complete platform

really delivers data analytics as a

software as a service model just like

Microsoft 365 making it much easier to

stand up and manage with frictionless

onboarding automatic optimization and

integration built-in security governance

and compliance it's Lake Centric and has

an open data architecture and it has

deep integration with Microsoft 365. uh

and what's great is it's designed in

particular for the era of AI with

built-in co-pilot support and this

copilot support enables you to use AI to

understand and reason over data in

extraordinary new ways and while I do

show a video of some of these new

capabilities in action

let's take a look at how copilot and

power bi is empowering everyone to

quickly find and share insights with

copilot and power bi I can simply

describe the report I want to build and

get insights in seconds

maybe we need a breakdown of some of our

employee data to understand demographics

and hiring Trends I just describe what I

need and copilot will automatically

analyze my data and create a new report

based on my needs immediately I have a

beautiful power bi report that has been

automatically created and is fully

interactive I can start slicing and

dicing my data to explore deeper

co-pilot added the charts and slicers I

asked for but also conveniently gives me

options to adjust the output to get just

what I need for example let's switch to

ask for metrics and Trends in a slightly

different layout automatically copilot

updates the report

let's go ahead and keep this report page

it goes much further I can also ask

questions such as why is our attrition

rate going up copilot responds by adding

a new page to my report and tapping into

Power bi's built-in Advanced analysis

capabilities for finding key influencers

for variables in my data in addition to

giving me a summary of an insight about

different employee types having

different nutrition rates I can easily

explore power bi's key influencer visual

to see what's driving the biggest impact

in our attrition rate I can even

automatically see the most significant

segments in my data just by clicking

now let's go back to the first page and

finish up our new report I'm going to

ask copilot to make it look like our

existing exec dashboard and instantly

it's applied the same formatting in

style and adjusted the layout to match

best of all I can still interact

directly with the powerbear report that

copilot has created so for example if I

wanted to manually change this bar chart

to a tree map on my own it's easy to do

with just a few clicks

let's add employee functional area and

now I can filter by this new field

finally to make the report even easier

for my team to understand I'm going to

ask copilot to add a rich text

description of my data right inside the

report this narrative summary is fully

Dynamic and not only does co-pilot

highlight interesting insights for my

data it will update the summary every

time the data is refreshed or people

filter the report

and just like that in seconds I've

created a report that would have taken

hours or days to do manually

copilot built on top of azure open AI is

truly revolutionary for how we Empower

everyone to find and share insights

and what you saw there is how we're able

to organize data in Microsoft Fabric and

in turn enable this amazing natural

language experience in power bi on top

of it Microsoft fabric is designed for

the era of AI and its serverless data

management engine is tuned for advanced

AI scenarios like that and with

Microsoft fabric we have a deep

commitment as well to open formats and

apis fabric comes with a SAS multi-cloud

data Lake which we call one Lake that's

built in and automatically available to

every Microsoft tenant one lake is built

on open formats including Delta and

parquet meaning it's compatible with

Solutions like databricks and other open

source tools and since we know that many

customers want to build AI Solutions

using data that might already be stored

in other clouds we've also built

Microsoft fabric to work across clouds

including AWS and in the future Google

and I'm incredibly pleased to announce

that Microsoft fabric is now available

in public preview for everyone to use so

we have an exciting future ahead of us

um you know Azure and the Microsoft

cloud you know is the place for every

developer and every organization on the

planet to be able to innovate using AI

uh we're really looking forward to

building some great AI Solutions with

you together and I hope you have a great

rest of the build uh and a great rest of

the day thank you very much