Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create Routing_Instructions.md and routing references #42

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ Follow the instructions in the [Prerequisites](#magic_wand-prerequisites) to con
| Scenario | Description | dotnet | C | python |
| -------- | ------------|--------|---|------- |
| [Getting Started](./scenarios/getting_started/) | This quick start scenario simulates basic MQTT tasks.| :white_check_mark:|:white_check_mark:|:white_check_mark:|
| [Telemetry](./scenarios/telemetry/) | This scenario simulates multiple clients (the producers) sending data to a different set of topics to be consumed by a single application (the consumer). | :white_check_mark:|:white_check_mark:|:soon:|
| [Telemetry](./scenarios/telemetry/) | This scenario simulates multiple clients (the producers) sending data to a different set of topics to be consumed by a single application (the consumer). This scenario also showcases routing the data to an Azure service. | :white_check_mark:|:white_check_mark:|:soon:|
| [Command](./scenarios/command/) | This scenario simulates the request-response messaging pattern using MQTT v5. | :white_check_mark:|:soon:|:soon:|
| [Alert](./scenarios/alert/) | This scenario simulates a fan-out use case where multiple clients receive a singlemessage from the same topic. | :soon:|:soon:|:soon:|

Expand Down
5 changes: 4 additions & 1 deletion scenarios/telemetry/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

| [Create Client Certificates](#lock-create-client-certificates) | [Configure Event Grid Namespaces](#triangular_ruler-configure-event-grid-namespaces) | [Configure Mosquitto](#fly-configure-mosquitto) | [Run the Sample](#game_die-run-the-sample) |

This scenario shows how multiple clients send data (the producers) to different topics that can be consumed by a single application (the consumer). This scenario also showcases routing the data to an Azure service.
This scenario shows how multiple clients send data (the producers) to different topics that can be consumed by a single application (the consumer). This scenario also showcases routing the data to Azure Event Hubs.

Consider a use case where a backend solution needs to identify the location of vehicles on a map. Vehicles should be prohibited from listening to other vehicles location on their behalf. Finally, the location data need to be routed to a storage queue.

Expand Down Expand Up @@ -152,6 +152,9 @@ echo "MQTT_CERT_FILE=map-app.pem" >> map-app.env
echo "MQTT_KEY_FILE=map-app.key" >> map-app.env
echo "MQTT_CA_PATH=/etc/ssl/certs" >> map-app.env # required by mosquitto_lib to validate EG Tls cert
```
### Configure routing

Follow the [Routing Instructions](./Routing_Instructions.md) to route the telemetry to an Azure Event Hubs instance.

## :fly: Configure Mosquitto

Expand Down
68 changes: 68 additions & 0 deletions scenarios/telemetry/Routing_Instructions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# Routing Telemetry
Event Grid allows you to route your MQTT messages to Azure services or webhooks for further processing. Accordingly, you can build end-to-end solutions by leveraging your IoT data for data analysis, storage, and visualizations, among other use cases.

The routing configuration enables you to send all your messages from your clients to an [Event Grid custom topic](https://learn.microsoft.com/en-us/azure/event-grid/custom-topics), and configuring [Event Grid event subscriptions](https://learn.microsoft.com/en-us/azure/event-grid/subscribe-through-portal) to route the messages from that custom topic to the [supported event handlers](https://review.learn.microsoft.com/en-us/azure/event-grid/event-handlers).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The routing configuration enables you to send all your messages from your clients to an [Event Grid custom topic](https://learn.microsoft.com/en-us/azure/event-grid/custom-topics), and configuring [Event Grid event subscriptions](https://learn.microsoft.com/en-us/azure/event-grid/subscribe-through-portal) to route the messages from that custom topic to the [supported event handlers](https://review.learn.microsoft.com/en-us/azure/event-grid/event-handlers).
The routing configuration enables you to send all your messages from your clients to an [Event Grid custom topic](https://learn.microsoft.com/azure/event-grid/custom-topics), and configuring [Event Grid event subscriptions](https://learn.microsoft.com/azure/event-grid/subscribe-through-portal) to route the messages from that custom topic to the [supported event handlers](https://review.learn.microsoft.com/azure/event-grid/event-handlers).


These instructions guide you to route your filtered MQTT messages from your Event Grid namespace to an [Azure Event Hubs](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-about) for further processing. Consider a use case where you want to route the telemetry from only vehicle1.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
These instructions guide you to route your filtered MQTT messages from your Event Grid namespace to an [Azure Event Hubs](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-about) for further processing. Consider a use case where you want to route the telemetry from only vehicle1.
These instructions guide you to route your filtered MQTT messages from your Event Grid namespace to an [Azure Event Hubs](https://learn.microsoft.com/azure/event-hubs/event-hubs-about) for further processing. Consider a use case where you want to route the telemetry from only vehicle1.


## Configure resources
Configure these resources after the Event Grid namespace configuration to enable the routing flow.

### Create an Event Hubs instance
Use [these instructions](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-quickstart-cli) to create an Event Hubs instance.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Use [these instructions](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-quickstart-cli) to create an Event Hubs instance.
Use [these instructions](https://learn.microsoft.com/azure/event-hubs/event-hubs-quickstart-cli) to create an Event Hubs instance.


### Configure the Event Grid topic
- Configure your Event Grid Topic where your messages will be routed. The region of the topic needs to match the region of your namespace.
```bash
az eventgrid topic create
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
az eventgrid topic create
az eventgrid topic create \

--name {EG custom topic name} \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some of these are defined in the create an event hub instance instructions - we should either use the same vars or if these are from az.env, include source az.env

-l {region name} \
-g $rg \
--input-schema cloudeventschemav1_0
```
- Assign EventGrid Data Sender role on the Event Grid topic to your principal ID
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is my principal ID here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The output I get back from the previous command has it as null

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, I see there's a note on this further down - I'd put it at the top here

```bash
az role assignment create
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
az role assignment create
az role assignment create \

--assignee "{Your Principal ID}" \
--role "EventGrid Data Sender" \
--scope "/subscriptions/$sub_id/resourcegroups/$rg/providers/Microsoft.EventGrid/topics/{EG Custom Topic Name}"
```
> Note:
> You can find your principal ID using the command: az ad signed-in-user show

### Configure your Event Grid event subcription

Create an Event Grid event subscription to route these messages from the Event Grid topic to your Event Hubs instance. The subscription uses the "Subject Begins With" filter that filters on the MQTT topic used in the MQTT messages.

```bash
az eventgrid event-subscription create --name contosoEventSubscription \
--source-resource-id "/subscriptions/$sub_id/resourceGroups/$rg/providers/Microsoft.EventGrid/topics/{Your Event Grid Topic Name}" \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we save the event grid topic name so the customers don't have to keep copying and pasting these commands to put their topic name in?

--endpoint-type eventhub \
--endpoint /subscriptions/$sub_id/resourceGroups/$rg/providers/Microsoft.EventHub/namespaces/{Event Hub Namespace Name}/eventhubs/{Event Hub Name}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
--endpoint /subscriptions/$sub_id/resourceGroups/$rg/providers/Microsoft.EventHub/namespaces/{Event Hub Namespace Name}/eventhubs/{Event Hub Name}
--endpoint /subscriptions/$sub_id/resourceGroups/$rg/providers/Microsoft.EventHub/namespaces/{Event Hub Namespace Name}/eventhubs/{Event Hub Name} \

--event-delivery-schema cloudeventschemav1_0 \
--subject-begins-with vehicles/vehicle1
```

### Configure routing in the Event Grid Namespace
Set the routing configuration on the Event Grid referring to the Event Grid topic.

```bash
az resource update --id $res_id --is-full-object --properties '{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

which resource id should this be? I was assuming the one in my az.env, but I get this error:

unrecognized arguments: --is-full-object --properties {
  "properties": {
    "isZoneRedundant": true,
    "topicsConfiguration": {
      "inputSchema": "CloudEventSchemaV1_0"
    },
    "topicSpacesConfiguration": {
      "state": "Enabled"
      "routeTopicResourceId": "/subscriptions/$sub_id/resourceGroups/$rg/providers/Microsoft.EventGrid/topics/val-eg-custom-topic"
    }
  },
  "location": "eastus2euap"
}

Examples from AI knowledge base:
az resource update --ids /subscriptions/{SubID}/resourceGroups/{ResourceGroup}/providers/Microsoft.Web/sites/{WebApp} --set tags.key=value
Update a webapp. (autogenerated)

az resource update --ids /subscriptions/{SubID}/resourceGroups/{ResourceGroup}/providers/Microsoft.Web/sites/{WebApp} --set tags.key=value --latest-include-preview
Update a webapp by using the latest api-version whether this version is a preview version.

https://docs.microsoft.com/en-US/cli/azure/resource#az_resource_update
Read more about the command in reference docs

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe this should be

Suggested change
az resource update --id $res_id --is-full-object --properties '{
az resource patch --ids $res_id --is-full-object --properties '{

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After changing to the patch command, I'm still getting an error
(InvalidRequest) Route topic val-eg-custom-topic not found for namespace mqtt-samples-eg Code: InvalidRequest Message: Route topic val-eg-custom-topic not found for namespace mqtt-samples-eg

Not sure if this is because I had the wrong resource id or should have somehow linked the event hub steps to my event grid namespace

"properties": {
"isZoneRedundant": true,
"topicsConfiguration": {
"inputSchema": "CloudEventSchemaV1_0"
},
"topicSpacesConfiguration": {
"state": "Enabled"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"state": "Enabled"
"state": "Enabled",

"routeTopicResourceId": "/subscriptions/{Subscription ID}/resourceGroups/{Resource Group ID}/providers/Microsoft.EventGrid/topics/{EG Custom Topic Name}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"routeTopicResourceId": "/subscriptions/{Subscription ID}/resourceGroups/{Resource Group ID}/providers/Microsoft.EventGrid/topics/{EG Custom Topic Name}"
"routeTopicResourceId": "/subscriptions/$sub_id/resourceGroups/$rg/providers/Microsoft.EventGrid/topics/{EG Custom Topic Name}"

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nevermind, I see this has to be typed out :(

}
},
"location": "eastus2euap"
}'
```
### View the routed MQTT messages in Azure Event Hubs
After you run the samples to send the MQTT messages, follow these steps to view the routed MQTT messages in Azure Event Hubs using Azure Stream Analytics query
- Navigate to the Event Hubs instance on the Azure portal.
- Go to [**Process data**](https://learn.microsoft.com/en-us/azure/event-hubs/process-data-azure-stream-analytics) using Azure Stream Analytics.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- Go to [**Process data**](https://learn.microsoft.com/en-us/azure/event-hubs/process-data-azure-stream-analytics) using Azure Stream Analytics.
- Go to [**Process data**](https://learn.microsoft.com/azure/event-hubs/process-data-azure-stream-analytics) using Azure Stream Analytics.

- Observe the routed MQTT messages based on the default query.