VNet Integration

[musteepha]

Hi, I have an interesting scenario, we have deployed Azure IPAM into our sandbox and realised we could not use VNet integration on the App service because it is a multi-container App.

What would be the best way to decouple the containers so that they run on multiple App services, and without the nginx loadbalancer

[DCMattyG]

Hi @musteepha, we've been working on some alternative architecture guidance for customers in your same position. Do you have a preference around one of the following options?

1. Azure Application Gateway w/ single container App Services
2. Azure Kubernetes Service (still uses NGINX as the Ingress)
3. None of the above

Let me know which aligns more to your preferences and I can share some of the preliminary work we've done. Additionally, I'm checking internally as to when the vNET integration will be supported for multi-container (Docker Compose) App Services, and will let you know the response I get.

[musteepha]

I'm using the debian files to build the images, so it should be port 80. I am hosting in an ACR, I've got the VNet pull working but the engine is still not starting up, do I need to specify a startup command ?

Also how does the UI app know which domain name to contact the engine app on ?

[DCMattyG]

No startup command required. Here's essentially how we build the containers from the Deployment script:

Debian:
az acr build -r <ACR URL>.azurecr.io -t ipam-engine:latest -f Dockerfile.deb --build-arg PORT=80
az acr build -r <ACR URL>.azurecr.io -t ipam-ui:latest -f Dockerfile.deb --build-arg PORT=80

RHEL:
az acr build -r <ACR URL>.azurecr.io -t ipam-engine:latest -f Dockerfile.rhel --build-arg PORT=8080
az acr build -r <ACR URL>.azurecr.io -t ipam-ui:latest -f Dockerfile.rhel --build-arg PORT=8080

[DCMattyG]

Technically the "PORT" argument isn't required, because 80 & 8080 are the defaults, respectively.

[musteepha]

Also how does the UI app know which domain name to contact the engine app on ?

[DCMattyG]

It pulls it from your browser session. So it's assuming the UI is hosted at "/" and the API (Engine) is hosted at "/api" paths.

[musteepha]

Ok, hence the need for path routing, engine now displays after removing the ports configuration

[DCMattyG]

Good morning @musteepha, just checking in on your progress and to see if there's anything I can do to help you.

[musteepha]

Hi @DCMattyG I didn't, make any progress with the error,. I have started deploying application gateway for the custom routing. I'll let you know the outcome shortly. Thanks for following up :)

[musteepha]

I have now got it working, the issue was the managed identity, the solution works as described with the application gateway, thanks alot for your help @DCMattyG

[DCMattyG]

Great news @musteepha, let me know if you need anything else!

[musteepha]

Hi @DCMattyG I have implemented the solution behind an Azure Frontdoor, the main pointers are,

• deploy separate app service plans in different resource groups
• deploy the ui app and engine app to separate app service plans - you can't use the same custom domain on app services with the the same webspaceName
• then add and validate the custom domain on each app service before redirecting to the frontdoor instance
• The frorntdoor will need separate routing rules, 1 to the paths on the ui /admin, /discover, /analyze, /*
• The other on the api, /api /api/*

In my case the frontdoor is a shared platform so it works out cheaper than a new appgateway and public IP, both the frontdoor and appgateway will require a WAF policy

[musteepha]

Hi @DCMattyG hope you are well, seem to be having an issue deploying containers to separate app services:

App services can pull image from private ACR over VNET, cosmos is deployed abut keep getting this error on engineApp

23-12-08T01:19:11.739594933Z INFO: Will watch for changes in these directories: ['/ipam']
2023-12-08T01:19:11.740278494Z INFO: Uvicorn running on http://0.0.0.0:80 (Press CTRL+C to quit)
2023-12-08T01:19:11.740739335Z INFO: Started reloader process [7] using WatchFiles
2023-12-08T01:19:17.042585343Z Process SpawnProcess-1:
2023-12-08T01:19:17.059447452Z Traceback (most recent call last):
2023-12-08T01:19:17.059511158Z File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-12-08T01:19:17.059518359Z self.run()
2023-12-08T01:19:17.059523159Z File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-12-08T01:19:17.059528060Z self._target(*self._args, **self._kwargs)
2023-12-08T01:19:17.059532860Z File "/usr/local/lib/python3.9/site-packages/uvicorn/_subprocess.py", line 76, in subprocess_started
2023-12-08T01:19:17.059537761Z target(sockets=sockets)
2023-12-08T01:19:17.059542561Z File "/usr/local/lib/python3.9/site-packages/uvicorn/server.py", line 61, in run
2023-12-08T01:19:17.059547161Z return asyncio.run(self.serve(sockets=sockets))
2023-12-08T01:19:17.059558262Z File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
2023-12-08T01:19:17.059562963Z return loop.run_until_complete(main)
2023-12-08T01:19:17.059567363Z File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
2023-12-08T01:19:17.059572064Z File "/usr/local/lib/python3.9/site-packages/uvicorn/server.py", line 68, in serve
2023-12-08T01:19:17.059576564Z config.load()
2023-12-08T01:19:17.059580964Z File "/usr/local/lib/python3.9/site-packages/uvicorn/config.py", line 467, in load
2023-12-08T01:19:17.059585465Z self.loaded_app = import_from_string(self.app)
2023-12-08T01:19:17.059590365Z File "/usr/local/lib/python3.9/site-packages/uvicorn/importer.py", line 21, in import_from_string
2023-12-08T01:19:17.059595066Z module = importlib.import_module(module_str)
2023-12-08T01:19:17.059599466Z File "/usr/local/lib/python3.9/importlib/init.py", line 127, in import_module
2023-12-08T01:19:17.059604066Z return _bootstrap._gcd_import(name[level:], package, level)
2023-12-08T01:19:17.059608667Z File "", line 1030, in _gcd_import
2023-12-08T01:19:17.059613767Z File "", line 1007, in _find_and_load
2023-12-08T01:19:17.059618368Z File "", line 986, in _find_and_load_unlocked
2023-12-08T01:19:17.059622968Z File "", line 680, in _load_unlocked
2023-12-08T01:19:17.059627569Z File "", line 850, in exec_module
2023-12-08T01:19:17.059632369Z File "", line 228, in _call_with_frames_removed
2023-12-08T01:19:17.059637169Z File "/ipam/app/main.py", line 14, in
2023-12-08T01:19:17.059653471Z from app.routers import (
2023-12-08T01:19:17.059658171Z File "/ipam/app/routers/azure.py", line 31, in
2023-12-08T01:19:17.059662872Z from app.models import *
2023-12-08T01:19:17.059667372Z File "/ipam/app/models.py", line 81, in
2023-12-08T01:19:17.059671873Z class BlockReq(BaseModel):
2023-12-08T01:19:17.059676373Z File "/usr/local/lib/python3.9/site-packages/pydantic/_internal/_model_construction.py", line 182, in new
2023-12-08T01:19:17.059682574Z complete_model_class(
2023-12-08T01:19:17.059687074Z File "/usr/local/lib/python3.9/site-packages/pydantic/_internal/_model_construction.py", line 491, in complete_model_class
2023-12-08T01:19:17.059691774Z schema = cls.get_pydantic_core_schema(cls, handler)
2023-12-08T01:19:17.059696275Z File "/usr/local/lib/python3.9/site-packages/pydantic/main.py", line 578, in get_pydantic_core_schema
2023-12-08T01:19:17.059700875Z return __handler(__source)
2023-12-08T01:19:17.059705276Z File "/usr/local/lib/python3.9/site-packages/pydantic/_internal/_schema_generation_shared.py", line 82, in call
2023-12-08T01:19:17.059710076Z schema = self._handler(__source_type)
2023-12-08T01:19:17.059714476Z File "/usr/local/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py", line 468, in generate_schema
2023-12-08T01:19:17.059719177Z schema = self._generate_schema(obj)
2023-12-08T01:19:17.059723577Z File "/usr/local/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py", line 700, in _generate_schema
2023-12-08T01:19:17.059728278Z schema = self._post_process_generated_schema(self._generate_schema_inner(obj))
2023-12-08T01:19:17.059732778Z File "/usr/local/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py", line 722, in _generate_schema_inner
2023-12-08T01:19:17.059737478Z return self._model_schema(obj)
2023-12-08T01:19:17.059741779Z File "/usr/local/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py", line 541, in _model_schema
2023-12-08T01:19:17.059746579Z {k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
2023-12-08T01:19:17.059751080Z File "/usr/local/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py", line 541, in
2023-12-08T01:19:17.059755980Z {k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
2023-12-08T01:19:17.059760480Z File "/usr/local/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py", line 886, in _generate_md_field_schema
2023-12-08T01:19:17.059765081Z common_field = self._common_field_schema(name, field_info, decorators)
2023-12-08T01:19:17.059769681Z File "/usr/local/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py", line 951, in _common_field_schema
2023-12-08T01:19:17.059774282Z schema = self._apply_annotations(
2023-12-08T01:19:17.059783283Z File "/usr/local/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py", line 1654, in _apply_annotations
2023-12-08T01:19:17.059788183Z schema = get_inner_schema(source_type)
2023-12-08T01:19:17.059793083Z File "/usr/local/lib/python3.9/site-packages/pydantic/_internal/_schema_generation_shared.py", line 82, in call
2023-12-08T01:19:17.059797784Z schema = self._handler(__source_type)
2023-12-08T01:19:17.059802184Z File "/usr/local/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py", line 1638, in inner_handler
2023-12-08T01:19:17.059806985Z metadata_js_function = _extract_get_pydantic_json_schema(obj, schema)
2023-12-08T01:19:17.059811885Z File "/usr/local/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py", line 1990, in _extract_get_pydantic_json_schema
2023-12-08T01:19:17.059816485Z raise PydanticUserError(
2023-12-08T01:19:17.059820986Z pydantic.errors.PydanticUserError: The __modify_schema__ method is not supported in Pydantic v2. Use __get_pydantic_json_schema__ instead.
2023-12-08T01:19:17.059825686Z
2023-12-08T01:19:17.059830087Z For further information visit https://errors.pydantic.dev/2.5/u/custom-json-schema
2023-12-08T01:23:01.718Z ERROR - Container ava-eus-prod-platform-apppsvc-001_1_bfbdfc18 for site ava-eus-prod-platform-apppsvc-001 did not start within expected time limit. Elapsed time = 230.1406747 sec
2023-12-08T01:23:01.724Z ERROR - Container ava-eus-prod-platform-apppsvc-001_1_bfbdfc18 didn't respond to HTTP pings on port: 80, failing site start. See container logs for debugging.
2023-12-08T01:23:01.755Z INFO - Stopping site ava-eus-prod-platform-apppsvc-001 because it failed during startup.
2023-12-08T01:24:23 No new trace in the past 1 min(s).
2023-12-08T01:25:23 No new trace in the past 2 min(s).
2023-12-08T01:26:23 No new trace in the past 3 min(s).
2023-12-08T01:27:23 No new trace in the past 4 min(s).
2023-12-08T01:28:23 No new trace in the past 5 min(s).
2023-12-08T01:28:45.210Z INFO - Pulling image: mcr.microsoft.com/appsvc/msitokenservice:stage3
2023-12-08T01:28:46.086Z INFO - stage3 Pulling from appsvc/msitokenservice
2023-12-08T01:28:46.088Z INFO - Digest: sha256:20b55dc78976812168c74397e0376f5a45ac3e4c810cce0786c5d4adc32d1ef0
2023-12-08T01:28:46.089Z INFO - Status: Image is up to date for mcr.microsoft.com/appsvc/msitokenservice:stage3
2023-12-08T01:28:46.100Z INFO - Pull Image successful, Time taken: 0 Seconds
2023-12-08T01:28:46.182Z INFO - Starting container for site
2023-12-08T01:28:46.183Z INFO - docker run -d --expose=8081 --name ava-eus-prod-platform-apppsvc-001_1_5240f3be_msiProxy -e PORT=8081 -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=ava-eus-prod-platform-apppsvc-001.azurewebsites.net -e WEBSITE_INSTANCE_ID=8cf0902abe489870c880990eb74e1dc9ddde8dbd2accf3d856c20ea2709e4cdf -e HTTP_LOGGING_ENABLED=1 mcr.microsoft.com/appsvc/msitokenservice:stage3
ava-eus-prod-platform-apppsvc-001
2023-12-08T01:29:11.851Z INFO - Container ava-eus-prod-platform-apppsvc-001_1_5240f3be_msiProxy for site ava-eus-prod-platform-apppsvc-001 initialized successfully and is ready to serve requests.
2023-12-08T01:29:11.852Z INFO - Initiating warmup request to container ava-eus-prod-platform-apppsvc-001_1_5240f3be for site ava-eus-prod-platform-apppsvc-001
2023-12-08T01:29:42.376Z INFO - Waiting for response to warmup request for container ava-eus-prod-platform-apppsvc-001_1_5240f3be. Elapsed time = 30.5543451 sec
2023-12-08T01:29:57.498Z INFO - Waiting for response to warmup request for container ava-eus-prod-platform-apppsvc-001_1_5240f3be. Elapsed time = 45.6770675 sec
2023-12-08T01:30:12.609Z INFO - Waiting for response to warmup request for container ava-eus-prod-platform-apppsvc-001_1_5240f3be. Elapsed time = 60.7874542 sec
2023-12-08T01:30:28.255Z INFO - Waiting for response to warmup request for container ava-eus-prod-platform-apppsvc-001_1_5240f3be. Elapsed time = 76.4337124 sec
2023-12-08T01:30:43.359Z INFO - Waiting for response to warmup request for container ava-eus-prod-platform-apppsvc-001_1_5240f3be. Elapsed time = 91.5379533 sec
2023-12-08T01:30:58.497Z INFO - Waiting for response to warmup request for container ava-eus-prod-platform-apppsvc-001_1_5240f3be. Elapsed time = 106.6755866 sec
2023-12-08T01:31:14.209Z INFO - Waiting for response to warmup request for container ava-eus-prod-platform-apppsvc-001_1_5240f3be. Elapsed time = 122.3877178 sec
2023-12-08T01:31:31.784Z INFO - Waiting for response to warmup request for container ava-eus-prod-platform-apppsvc-001_1_5240f3be. Elapsed time = 139.9629117 sec
2023-12-08T01:31:47.229Z INFO - Waiting for response to warmup request for container ava-eus-prod-platform-apppsvc-001_1_5240f3be. Elapsed time = 155.4079864 sec

[musteepha]

Hey @DCMattyG any ideas on the above ?, what i've noticed is the engine App did not create the cosmos db or container automatically, is this normal behaviour ?

[DCMattyG]

Hey @musteepha, can you verify that you're using the latest version of the code from the main branch? This issue looks like something that was remediated a while back when the Pydantic library moved to V2 and some code needed be updated as there ere breaking changes.

Please make sure your containers are built from the latest code and let me know if that helps.

[musteepha]

My code version is quite old, let m rebuild and come back to you

[musteepha]

Rebuilt and working with new code, thanks

[DCMattyG]

Wonderful to hear, so happy everything is working well for you now!