Microsoft Certified: Azure AI Engineer Associate Study Guide

Hugo Barona included in Azure Microsoft Artificial Intelligence Certification AI AI100 AI102

2021-02-07 4238 words 20 minutes

/ai-100-study-guidedesigning-and-implementing-an-azure-ai-solution/featured-image.png

Contents

Note: In case you are planning to study to this exam and certification, please keep in mind that this exam will be replaced by the AI-102 exam coming on February 23, 2021 - (AI-102)

The Microsoft Certified: Azure AI Engineer Associate certification enables you to get the knowledge to be subject matter expert using Cognitive Services, Machine Learning (ML), and knowledge mining to architect and implement AI solutions on Azure involving natural language processing, speech, computer vision, conversational AI and more.

In this study guide, I will share with you some of the useful resources you can use to guide you during your learning path to get this certification.

Certification Path

Exam Name	Link
Exam AI-100: Designing and Implementing an Azure AI Solution	Exam Details

Exam AI-100: Designing and Implementing an Azure AI Solution

The AI-100 has a length of three hours. There are 50+ questions and you need a minimum of 700 of 1000 points to pass the exam.

First place to go is the Microsoft Learn platform where a dedicated learning path is available, for free. Also, you should have a look to the Resources section in this study guide where you have useful resources to help you consolidate the knowledge that will help you get the exam and certification. If you prefer to watch videos, instead of read, explaining these core concepts and showing how to get started using Power Platform, then I invite you to have a look to the Microsoft Azure AI Engineer (AI-100) path, available on Pluralsight.

Skills measured

Analyze solution requirements (25-30%)

Recommend Azure Cognitive Services APIs to meet business requirements

select the processing architecture for a solution
select the appropriate data processing technologies
select the appropriate AI models and services
identify components and technologies required to connect service endpoints
identify automation requirements

Map security requirements to tools, technologies, and processes

identify processes and regulations needed to conform with data privacy, protection, and regulatory requirements
identify which users and groups have access to information and interfaces
identify appropriate tools for a solution
identify auditing requirements

Select the software, services, and storage required to support a solution

identify appropriate services and tools for a solution
identify integration points with other Microsoft services
identify storage required to store logging, bot state data, and Azure Cognitive Services output

Design AI solutions (40-45%)

Design solutions that include one or more pipelines

define an AI application workflow process
design a strategy for ingest and egress data
design the integration point between multiple workflows and pipelines
design pipelines that use AI apps
design pipelines that call Azure Machine Learning models
select an AI solution that meet cost constraints

Design solutions that uses Cognitive Services

design solutions that use vision, speech, language, knowledge, search, and anomaly detection APIs

Design solutions that implement the Microsoft Bot Framework

integrate bots and AI solutions
design bot services that use Language Understanding (LUIS)
design bots that integrate with channels
integrate bots with Azure app services and Azure Application Insights

Design the compute infrastructure to support a solution

identify whether to create a GPU, FPGA, or CPU-based solution
identify whether to use a cloud-based, on-premises, or hybrid compute infrastructure
select a compute solution that meets cost constraints

Design for data governance, compliance, integrity, and security

define how users and applications will authenticate to AI services
design a content moderation strategy for data usage within an AI solution
ensure that data adheres to compliance requirements defined by your organization
ensure appropriate governance of data
design strategies to ensure that the solution meets data privacy regulations and industry standards

Implement and monitor AI solutions (25-30%)

Implement an AI workflow

develop AI pipelines
manage the flow of data through the solution components
implement data logging processes
define and construct interfaces for custom AI services
create solution endpoints
develop streaming solutions

Integrate AI services and solution components

configure prerequisite components and input datasets to allow the consumption of Azure Cognitive Services APIs
configure integration with Azure Cognitive Services
configure prerequisite components to allow connectivity to the Microsoft Bot Framework
implement Azure Cognitive Search in a solution

Monitor and evaluate the AI environment

identify the differences between KPIs, reported metrics, and root causes of the differences
identify the differences between expected and actual workflow throughput
maintain an AI solution for continuous improvement
monitor AI components for availability
recommend changes to an AI solution based on performance data

Additional Notes

During the process of studying to this certification and the AI-100 exam, I took some personal notes. So I am sharing below the notes of some important concepts I studied to the exam.

Cognitive Services
- Scenarios
  - Little to no customization needed
  - Team with limited ML and Data Science capabilities
    - Existance of solution to solve your ML problem
  - Categories
    - Decision
      - Anomaly Detector
      - Embed anomaly detection capabilities to apps
        
        Identify quickly problems in your apps
        
        Ingest timeseries data and select the best fitting detection model for your data
        
        Provides Container Support to run your service in the Edge using a container
      - Content Moderator
        
        Images, Text and Video moderation
        
        Moderate content
        
        Identify adult, offensive or inappropriate content
      - Metrics Advisor
      - Personalizer
        
        Deliver rich and personalized experience to users, by personalizing content according to user preferences and historic
    - Language
      - Immersive reader
      - Language Understanding (LUIS)
        
        Applies ML intelligence to conversational natural language text to predict and extract relevant detailed information
        
        Used commonly in chat bots to identify requests from users and provide expected outcomes to users.
        
        Provides support to be exported as IoT Edge module
      - QnA maker
        
        Extract questions and answers from your content
      - Text Analytics
        
        Detect sentiment in text (value 0 (negative sentiment) to 1 (positive sentiment)). Uses classifier model.
        
        Provides support to be exported as IoT Edge module
        
        Key Phrease extraction
        
        Provides support to be exported as IoT Edge module
        
        Language Detection
        
        Provides support to be exported as IoT Edge module
        
        Named Entity Recognition
        
        Provides Container Support to run your service in the Edge using a container
      - Translator
        
        Detect Languages
        
        Translate text into different languages
        
        Online and offline translate support
    - Speech
      - Speech to Text: Real-time transcription to text
        
        Provides Container Support to run your service in the Edge using a container
      - Text to Speech: Speak to users naturally, convert text to audio near real-time
        
        Provides Container Support to run your service in the Edge using a container
      - Speech Translation: Real-time speech translation
      - Speaker Recognition: Verify voice to identify the person talking
      - Custom Models
        
        Acoustic Model: e.g. Use for particular environment, such as objects, instruments or specific population
        
        Language Model: e.g. Tailoring to the vocabulary and speaking style of users
        
        Pronunciation Model: e.g. Specific phonetic form and display of a word or term
        
        Voice Font: Create custom voices
    - Vision
      - Computer Vision
        
        Analyze Image: Tagging, Object Detection, Adult Content
        
        Analyze Text
        
        Identify Celebrities
      - Face
        
        Detect Faces in Images, Face Identification, identify characteristics in a face (age, gender, smile, etc.)
        
        Provides Container Support to run your service in the Edge using a container
        
        Provides support to be exported as IoT Edge module
      - Video Indexer
        
        Facial, Object recognition, OCR, transcription
        
        Topic inference, brands and emotion detection
      - Custom Vision
        
        Upload images, label images and train your custom models to label new images
        
        Model based Edge support - Export your custom trained model and use it with different frameworks (tensorflow, etc.)
      - Form Recognizer
        
        Extract text, key-value pairs and tables from documents, forms and receipts with low or no manual intervention
        
        Leverages pre-built components to understand layout and relationship between fields and entries in your documents
        
        Provides Container Support to run your service in the Edge using a container
        
        Provides support to be exported as IoT Edge module
  - Search

Uses prebuilt models

Capability	Input type	Key benefit
Text Analytics API	Text	Evaluate sentiment and topics to understand what users want.
Entity Linking API	Text	Power your app’s data links with named entity recognition and disambiguation.
Language Understanding Intelligent Service (LUIS)	Text	Teach your apps to understand commands from your users.
QnA Maker Service	Text	Distill FAQ formatted information into conversational, easy-to-navigate answers.
Linguistic Analysis API	Text	Simplify complex language concepts and parse text.
Knowledge Exploration Service	Text	Enable interactive search experiences over structured data via natural language inputs.
Web Language Model API	Text	Use predictive language models trained on web-scale data.
Academic Knowledge API	Text	Tap into the wealth of academic content in the Microsoft Academic Graph populated by Bing.
Bing Autosuggest API	Text	Give your app intelligent autosuggest options for searches.
Bing Spell Check API	Text	Detect and correct spelling mistakes in your app.
Translator Text API	Text	Machine translation.
Recommendations API	Text	Predict and recommend items your customers want.
Bing Entity Search API	Text (web search query)	Identify and augment entity information from the web.
Bing Image Search API	Text (web search query)	Search for images.
Bing News Search API	Text (web search query)	Search for news.
Bing Video Search API	Text (web search query)	Search for videos.
Bing Web Search API	Text (web search query)	Get enhanced search details from billions of web documents.
Bing Speech API	Text or Speech	Convert speech to text and back again.
Speaker Recognition API	Speech	Use speech to identify and authenticate individual speakers.
Translator Speech API	Speech	Perform real-time speech translation.
Computer Vision API	Images (or frames from video)	Distill actionable information from images, automatically create description of photos, derive tags, recognize celebrities, extract text, and create accurate thumbnails.
Content Moderator	Text, Images or Video	Automated image, text, and video moderation.
Emotion API	Images (photos with human subjects)	Identify the range emotions of human subjects.
Face API	Images (photos with human subjects)	Detect, identify, analyze, organize, and tag faces in photos.
Video Indexer	Video	Video insights such as sentiment, transcript speech, translate speech, recognize faces and emotions, and extract keywords.

Trained with custom data you provide

Capability	Input type	Key benefit
Custom Vision Service	Images (or frames from video)	Customize your own computer vision models.
Custom Speech Service	Speech	Overcome speech recognition barriers like speaking style, background noise, and vocabulary.
Custom Decision Service	Web content (for example, RSS feed)	Use machine learning to automatically select the appropriate content for your home page
Bing Custom Search API	Text (web search query)	Commercial-grade search tool.

What are your options when choosing an NLP service?

In Azure, the following services provide natural language processing (NLP) capabilities:

Key selection criteria

To narrow the choices, start by answering these questions:

Do you want to use prebuilt models? If yes, consider using the APIs offered by Microsoft Cognitive Services.
Do you need to train custom models against a large corpus of text data? If yes, consider using Azure HDInsight with Spark MLlib and Spark NLP.
Do you need low-level NLP capabilities like tokenization, stemming, lemmatization, and term frequency/inverse document frequency (TF/IDF)? If yes, consider using Azure HDInsight with Spark MLlib and Spark NLP.
Do you need simple, high-level NLP capabilities like entity and intent identification, topic detection, spell check, or sentiment analysis? If yes, consider using the APIs offered by Microsoft Cognitive Services.

ContentModerator

SDK
Use the Content Moderator client library for .NET to:
- Moderate text
- Moderate images
- Create a review
Image Moderation
- Create and initialize an instance of the Content Moderator API wrapper. ContentModeratorClient client = new ContentModeratorClient(new ApiKeyServiceClientCredentials(CMSubscriptionKey));
- Evaluate for adult and racy content. client.ImageModeration.EvaluateUrlInput(“application/json”, url, true);
- Detect and extract text. client.ImageModeration.OCRUrlInput(“eng”, “application/json”, url, true);
- Detect faces. client.ImageModeration.FindFacesUrlInput(“application/json”, url, true);
Image Reviews
- Team Name - This must be the team name you used to create your Content Moderator account.
- Create Review client.Reviews.CreateReviewsWithHttpMessagesAsync(“application/json”, TeamName, requestInfo);
- Get Review client.Reviews.GetReviewWithHttpMessagesAsync(TeamName, item.ReviewId);
Text Moderation
- Screen the input text: check for profanity, classify the text into three categories, do autocorrect text, and check for personally identifying information (PII) client.TextModeration.ScreenText(“text/plain”, new MemoryStream(Encoding.UTF8.GetBytes(text)), language: “eng”, autocorrect: true, pII: true, listId: null, classify: true);
Video Moderation

Language Understanding Intelligent Service (LUIS)

LUIS makes use of three key aspects for understanding language:
Utterances: An utterance is input from the user that your app needs to interpret.
Intents: An intent represents a task or action the user wants to do. It’s a purpose or goal expressed in a user’s utterance.
Entities: An entity represents a word or phrase inside the utterance that you want to extract.
LUIS config (JSON format)
LUIS Pizza App Example (JSON)
LUIS API
LUIS URL Format: https://<region>.api.cognitive.microsoft.com

LUIS Response Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


{
  "query": "Book me a flight to Cairo",
  "prediction": {
      "topIntent": "BookFlight",
      "intents": {
          "BookFlight": {
              "score": 0.9887482
          },
          "None": {
              "score": 0.04272597
          },
          "LocationFinder": {
              "score": 0.0125702191
          },
          "Reminder": {
              "score": 0.00375502417
          },
          "FoodOrder": {
              "score": 3.765154e-7
          }
      },
      "entities": {
          "Location": [
              "Cairo"
          ]
      }
  }
}

SDK Examples

Create Application

1
2
3
4
5
6
7
8


          var app_info = new ApplicationCreateObject()
          {
              Name = app_name,
              InitialVersionId = app_version,
              Description = app_description,
              Culture = app_culture
          };
          var app_id = await client.Apps.AddAsync(app_info);

Create Intents

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


  List<String> intents = new List<String>();
          intents.Add("Greeting");
          intents.Add("SearchPic");
          intents.Add("OrderPic");
          intents.Add("SharePic");
          foreach (string intent in intents)
          {
              await client.Model.AddIntentAsync(app_info.ID, app_info.Version, new ModelCreateObject()
              {
                  Name = intent
              });
              Console.WriteLine("Created intent {0}", intent);
          }

Create Entity

1
2
3
4


  var facetEntityId = await client.Model.AddEntityAsync(app_info.ID, app_info.Version, new ModelCreateObject()
           {
               Name = "facet"
           });

Create Utterances

1

  var resultsList = await client.Examples.BatchAsync(app_info.ID, app_info.Version, utterances);

Train Model

1

  var response = await client.Train.TrainVersionAsync(app.ID, app.Version);

Publish App

1
2


  ApplicationPublishObject obj = new ApplicationPublishObject { VersionId = app.Version, IsStaging = true };
  var info = await client.Apps.PublishAsync(app.ID, obj);

Check Training Status

1
2


  var response = await client.Train.GetStatusAsync(app.ID, app.Version); 
  Console.WriteLine(response[0].Details.Status.ToString());

Speech API

Some Scenarios Speech to Text
- Translation of live presentations
- In-person or remote translated communications
- Customer support
- Business intelligence
- Media subtitling
- Multilingual AI interactions

SDK

Create configuration

1

var config = SpeechConfig.FromSubscription("KEY", "LOCATION");

Create SpeechRecognizer

1

var recognizer = new SpeechRecognizer(config, audioInput);

Recognize Async

1

var result = await recognizer.RecognizeOnceAsync();

Text to Speech

1
2
3


var fileOutput = AudioConfig.FromWavFileOutput(fileName);
var synthesizer = new SpeechSynthesizer(config, fileOutput);
var result = await synthesizer.SpeakTextAsync(text);

Creates a speech synthesizer using the SpeechSynthesisVoiceName for the audio output.
1

var speech_synthesizer = new SpeechSynthesizer(config);

Use the synthesized voice to speak the translation

1

await speech_synthesizer.SpeakTextAsync(result.Translations[toLanguage]);

Text Analytics API

Key Phrases Input’s JSON Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


{
    "documents": [
        {
            "language": "en",
            "id": "1",
            "text": "Document 1 text here"
        },
        {
            "language": "fr",
            "id": "2",
            "text": "Document 2 text here"
        }
    ]
}

Key Phrases Output’s JSON Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


{
  "documents": [
    {
      "id": "1",
      "keyPhrases": [
        "world",
        "input text"
      ]
    },
    {
      "id": "2",
      "keyPhrases": [
        "monde"
      ]
    },
    {
      "id": "3",
      "keyPhrases": [
        "carretera",
        "tráfico",
        "día"
      ]
    }
  ],
  "errors": []
}

SDK - Key Phrases

Instantiate the client

1
2
3
4
5


var credentials = new ApiKeyServiceClientCredentials(key);
var client = new TextAnalyticsClient(credentials)
{
    Endpoint = endpoint
};

Obtain Key Phrases

1

  var response = await client.KeyPhrasesAsync(doc);

Process Key Phrases

1
2
3
4


foreach (string keyphrase in response.KeyPhrases)
{
  Console.WriteLine($"\t{keyphrase}");
}

Language Detection

JSON Example Input

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


	{
	     "documents": [
	         {
	             "id": "1",
	             "text": "This document is in English."
	         },
	         {
	             "id": "2",
	             "text": "Este documento está en inglés."
	         },
	         {
	             "id": "3",
	             "text": "Ce document est en anglais."
	         },
	         {
	             "id": "4",
	             "text": "本文件为英文"
	         },
	         {
	             "id": "5",
	             "text": "Этот документ на английском языке."
	         }
	     ]
	 }

JSON Example Output

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


{
    "documents": [
        {
            "id": "1",
            "detectedLanguages": [
                {
                    "name": "English",
                    "iso6391Name": "en",
                    "score": 1
                }
            ]
        },
        {
            "id": "2",
            "detectedLanguages": [
                {
                    "name": "Spanish",
                    "iso6391Name": "es",
                    "score": 1
                }
            ]
        }
     ]
}

NOTE: In case of a multi-language text, the service will identify the predominant language and return that single language
The last condition to consider is when there is ambiguity as to the language content. The scenario might happen if you submit textual content that the analyzer is not able to parse. As a result, the response for the language name and ISO code will indicate (unknown) and the score value will be returned as NaN, or Not a Number. The following example shows how the response would look.

SDK Example

Instantiate client

1

  var client = new TextAnalyticsClient(endpoint, credentials);

Detect Language

1

    DetectedLanguage detectedLanguage = client.DetectLanguage("Ce document est rédigé en Français.");

Vision Services

Face API, Emotion API, Vision API
Face API
The Face API tasks fall into five categories:
- Verification: Check the likelihood that two faces belong to the same person.
- Detection: Detect human faces in an image.
- Identification: Search and identify faces.
- Similarity: Find similar faces.
- Grouping: Organize unidentified faces into groups, based on their visual similarity.
Face API Attributes
- Age
- Gender
- Smile intensity
- Facial hair
- Head pose (3D)
- Emotion
Vision API

Urls

Analyze images: https://<region>.api.cognitive.microsoft.com/vision/v2.0/analyze?visualFeatures=<…>&details=<…>&language=<…> The parameters that are sent to the service are visualFeatures, details, and languages. Set the details parameter to Landmarks or Celebrities to help you identify landmarks or celebrities. visualFeatures identify what kind of information you want the service to return you. The Categories option will categorize the content of the images like trees, buildings, and more. Faces will identify people’s faces and give you their gender and age.

Recognize Text: https://<region>.api.cognitive.microsoft.com/vision/v2.0/recognizeText?mode=<…> If present, the mode parameter must be set to Handwritten or Printed and is case-sensitive. If the parameter is set to Handwritten or is not specified, handwriting recognition is performed. If the parameter is set to Printed, then printed text recognition is performed. The time it takes to get a result from this call depends on the amount of writing in the image.

Response’s Example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


{
  "status":"Succeeded",
  "recognitionResult":{
      "lines":[
        {
            "boundingBox":[
              123,
              131,
              404,
              138,
              403,
              169,
              123,
              162
            ],
            "text":"It does not matter",
            "words":[
              {
                  "boundingBox":[
                    128,
                    132,
                    163,
                    133,
                    164,
                    163,
                    129,
                    163
                  ],
                  "text":"It"
              }, (... - other words)
  ]
        }, (... - other lines)
                ]
  }
}

Get an image’s thumbnail: https://<region>.api.cognitive.microsoft.com/vision/v2.0/generateThumbnail?width=<…>&height=<…>&smartCropping=<…> Different parameters can be provided to the API to generate the proper thumbnail for your needs. The width and height parameters are required. They tell the API which size you need for a specific image. The smartCropping parameter generates smarter cropping by analyzing the region of interest in your image to keep it within the thumbnail. As an example, with smart cropping enabled, a cropped profile picture would keep someone’s face within the picture frame even when the picture has a different aspect ratio.
Extract Printed Text in Image: https://<region>.api.cognitive.microsoft.com/vision/v2.0/ocr?language=<…>&detectOrientation=<…> As usual, all calls must be made to the region where the account was created. The call accepts two optional parameters: • language: The language code of the text to be detected in the image. The default value is unk,or unknown. This let’s the service auto detect the language of the text in the image. • detectOrientation: When true, the service tries to detect the image orientation and correct it before further processing, for example, whether the image is upside-down.

Custom AI Solutions

Scenarios:
- Highly customized, custom algorithms
- Use existing custom workflows/solutions
- Team with ML capabilities

Azure Machine Learning

Compute: Local & Remote
- Options
  - Azure ML Compute Cluster
  - Remote VM
  - Azure Databricks
  - Azure Data Lake Analytics
  - Azure HDInsight
  - Azure Batch

Compute Targets for inference

Compute target	Used for	GPU support	FPGA support	Description
Local web service	Testing/debugging			Use for limited testing and troubleshooting. Hardware acceleration depends on use of libraries in the local system.
Azure Kubernetes Service (AKS)	Real-time inference	Yes (web service deployment)	Yes	Use for high-scale production deployments. Provides fast response time and autoscaling of the deployed service. Cluster autoscaling isn’t supported through the Azure Machine Learning SDK. To change the nodes in the AKS cluster, use the UI for your AKS cluster in the Azure portal. Supported in the designer.
Azure Container Instances	Testing or development			Use for low-scale CPU-based workloads that require less than 48 GB of RAM. Supported in the designer.
Azure Machine Learning compute clusters	Batch inference	Yes (machine learning pipeline)		Run batch scoring on serverless compute. Supports normal and low-priority VMs. No support for realtime inference.

Training Compute Targets

Training targets	Automated machine learning	Machine learning pipelines	Azure Machine Learning designer
Local computer	Yes
Azure Machine Learning compute cluster	Yes	Yes	Yes
Azure Machine Learning compute instance	Yes (through SDK)	Yes
Remote VM	Yes	Yes
Azure Databricks	Yes (SDK local mode only)	Yes
Azure Data Lake Analytics		Yes
Azure HDInsight		Yes
Azure Batch		Yes

Azure Databricks

Analytics Spark Platform
Integrated with Azure
Build streamlined workflows and interactive workspace to enable collaboration between data science teams
Provides Spark SQL to perform analysis and DataFrames to store your data in distributed collections composed by named columns
Real-time data processing and analysis for analytical and interactive applications
Built-in integration with Hadoop FS, Flume and Kafka
Provides MLib (ML library with common ML algorithms and utilities including classification and regression, dimensionality reduction, optimization primitives and more.
Has GraphX which graphs and graph computation for a broad scope of use cases from Cognitive Analytics to Data Exploration
Supports Spark Core API and multiple languages including R, SQL, Python, Scala & Java
Advantages
- Fully Managed clusters
- Interactive workspace for exploration and visualization
- Platform for powering your spark-based apps
- Serverless - Abstracts the users from the infrastructure being used by your data-driven solutions
Data Exploration, Notebooks, Familiar Tools to visualize data

Azure Data Factory

Integration Runtimes (Public & Private Networks)
- Azure: Enables Data Flow, Data Movement and Activity Dispatch
- Self-hosted: Enables Data movement and Activity dispatch
- Azure-SSIS: Enables SSIS package execution Note: Use self-hosted integration runtime to support data stores that requires bring-your-own driver such as SAP Hana, MySQL, etc. For more information, see supported data stores.

Big Data

Spark:
- Process streaming data in real-time or in batches (uses RAM to process dataset - RDD)
- Uses Hadoop MapReduce but with some improvements
- Spark uses for a real-time stream, batch process and ETL also.
Kafka:
- Kafka is a Message broker.
- Kafka provides real-time streaming, window process (record by record)
- Kafka has Producer, Consumer, Topic to work with data
- In Kafka, we cannot perform a transformation
- Kafka does not support any programming language to transform the data
- So Kafka is used for real-time streaming as Channel or mediator between source and target
Storm
- Distributed real-time processing of large volumes of high-velocity data.
- Supports micro-batching processing
- Data is mostly dynamic and continuously streamed.
- Relatively slow.
- Architecture consists of sprouts and bolts.
- Scalable and fault-tolerant.
- Implemented in Clojure.
- Simple and can be used with any programming language.
- Easy to set up and operate.
- Used in business intelligence and Big Data analytics.
- Open source (Apache license).
Hadoop
- Distributed batch processing of large volumes of high-velocity data.
- Persists data to disk after completing operations, so slower than Spark
- Data is mostly static and stored in persistent storage.
- Relatively fast.
- Architecture consists of HDFS and MapReduce.
- Scalable and fault-tolerant.
- Implemented in Java.
- Complex and can be used with any programming language.
- Easy to set up but difficult to operate.
- Used in business intelligence and Big Data analytics.
- Open source (Apache license).

VM Series

A-Series: Entry-level economical VMs for dev/test
Bs-Series: Economical burstable VMs
D-Series: General purpose compute
DC-series: Protect data in use
E-Series: Optimised for in-memory hyper-threaded applications
F-Series: Compute optimised virtual machines
G-Series: Memory and storage optimised virtual machines
H-Series: High Performance Computing virtual machines
Ls-Series: Storage optimised virtual machines
M-Series: Memory optimised virtual machines
Mv2-Series: Largest memory optimised virtual machines
N-Series: GPU enabled virtual machines
Versions of VMs for ML
- Pre-configured and pre-installed ML tools in these VMs
  - Data Science VM
  - Deep Learning VM
- Geo AI Data Science VM

Azure CLI

Cognitive Services

Command	Description
az cognitiveservices list	List available Azure Cognitive Services accounts.
az cognitiveservices account show	Get the details of an Azure Cognitive Services account.
az cognitiveservices account create	Create an Azure Cognitive Services account.
az cognitiveservices account delete	Delete an Azure Cognitive Services account.
az cognitiveservices keys list	List the keys of an Azure Cognitive Services account.

Command	Description
az ml computetarget create aks	Create an AKS compute target.
az ml computetarget create amlcompute	Create an AzureML compute target.
az ml computetarget create computeinstance	Create an AzureML compute instance target.
az ml computetarget create datafactory	Create a data factory compute target.
az ml run submit-script	Submit a script for execution.
az ml run submit-pipeline	Submit a pipeline for execution, from a published pipeline ID or pipeline YAML file.
az ml model register -n sklearn_mnist –asset-path outputs/sklearn_mnist_model.pkl –experiment-name myexperiment –run-id myrunid –tag area=mnist	Register a ML model

Resources

Resource
Microsoft Learning Path - Evaluate text with Azure Cognitive Language Services
Microsoft Learning Path - Process and Translate Speech with Azure Cognitive Speech Services
Microsoft Learning Path - Create Intelligent Bots with the Azure Bot Service
Microsoft Learning Path - Process and classify images with the Azure cognitive vision services
Microsoft Certified: Azure AI Engineer Associate
Microsoft Azure AI Engineer (AI-100) Pluralsight Certification Path