object_score: Support Azure Fabric OAuth Provider

[RobinLin666]

Which issue does this PR close?

Closes #.

Rationale for this change

In Azure Fabric, we use token service to get user access token, for supporting long reading and writing operation and auto refresh access token, we implement this.

What changes are included in this PR?

This pull request introduces significant enhancements to the Azure integration within the object_store module, including the implementation of a new FabricTokenOAuthProvider. These changes aim to improve the authentication mechanism and add support for fabric token services.

Azure Builder Enhancements:

• Added new fields to MicrosoftAzureBuilder to support fabric token services, including fabric_token_service_url, fabric_workload_host, fabric_session_token, and fabric_cluster_identifier. (object_store/src/azure/builder.rs object_store/src/azure/builder.rsR175-R182)
• Updated AzureConfigKey to include new configuration keys related to fabric token services. (object_store/src/azure/builder.rs object_store/src/azure/builder.rsR347-R374)
• Modified impl AsRef<str> and impl FromStr for AzureConfigKey to handle new fabric token service keys. (object_store/src/azure/builder.rs [1] [2]
• Enhanced MicrosoftAzureBuilder to set and get the new fabric token service-related fields. (object_store/src/azure/builder.rs [1] [2]
• Added logic to MicrosoftAzureBuilder to create a FabricTokenOAuthProvider if fabric token service fields are provided. (object_store/src/azure/builder.rs object_store/src/azure/builder.rsR919-R942)

Credential Enhancements:

• Introduced FabricTokenOAuthProvider to handle OAuth token challenges for fabric token services, including methods for token validation and fetching. (object_store/src/azure/credential.rs object_store/src/azure/credential.rsR943-R1049)
• Added new headers and constants related to fabric token services. (object_store/src/azure/credential.rs object_store/src/azure/credential.rsR55-R63)

These changes collectively enhance the Azure integration by supporting more complex authentication mechanisms, particularly for environments utilizing fabric token services.

Are there any user-facing changes?

After that, we can set some environment variables to make it auto refresh access token in Fabric Notebook.

# For Fabric Spark Notebook
import os
import urllib.parse
workload_endpoint = urllib.parse.urlparse(f"{spark.conf.get('trident.lakehouse.tokenservice.endpoint')}/access")
os.environ['azure_fabric_token_service_url'.upper()] = f"https://{spark.conf.get('spark.tokenServiceEndpoint')}/api/v1/proxy{workload_endpoint.path}"
os.environ['azure_fabric_workload_host'.upper()] = f"{workload_endpoint.scheme}://{workload_endpoint.hostname}"
os.environ['azure_fabric_session_token'.upper()] = spark.conf.get("trident.session.token")
os.environ['azure_fabric_cluster_identifier'.upper()] = spark.conf.get("spark.synapse.clusteridentifier")
os.environ['azure_storage_token'.upper()] = notebookutils.credentials.getToken("storage")

# For Fabric Python Notebook
import os
from notebookutils.common import configs
import urllib.parse
workload_endpoint = urllib.parse.urlparse(f"{configs.workload_endpoint()}/access")
os.environ['azure_fabric_token_service_url'.upper()] = f"{configs.ts_endpoint()}/api/v1/proxy{workload_endpoint.path}"
os.environ['azure_fabric_workload_host'.upper()] = f"{workload_endpoint.scheme}://{workload_endpoint.hostname}"
os.environ['azure_fabric_session_token'.upper()] = configs.session_token()
os.environ['azure_fabric_cluster_identifier'.upper()] = configs.cluster_identifier()
os.environ['azure_storage_token'.upper()] = notebookutils.credentials.getToken("storage")

Then, user can read/write delta table without storage_option.

from deltalake import DeltaTable
dt = DeltaTable('abfss://[email protected]/LH.Lakehouse/Tables/dbo/test')
df = dt.to_pyarrow_dataset().head(10).to_pandas()

[tustvold]

The JWT logic seems a little odd to me, are we just using it to decode the expiry? If so could we avoid the additional dependency?

[RobinLin666]

The JWT logic seems a little odd to me, are we just using it to decode the expiry? If so could we avoid the additional dependency?

Hi @tustvold Thank you for you review. Yes, because Token Service only returns a JWT token, so I need to decode the expiry. Any advice without dependency?

[tustvold]

https://jwt.io/introduction you should be able to simply split the string, base64 decode the middle chunk and parse the JSON

[alamb]

I am depressed about the large review backlog in this crate. We are looking for more help from the community reviewing PRs -- see #6418 for more

[tustvold]

I don't have a way to test this, but it looks plausible to me. Thank you.

Perhaps @roeap you might be able to give this one a once over as well?

[RobinLin666]

Thanks all, please help to merge the PR if no question.

[alamb]

Let's wait a day or two before merging to see if @roeap has some time to review. This is getting very close.

Thanks for your patience @RobinLin666 and the help @tustvold

[roeap]

LGTM! Thanks for adding this @RobinLin666.

Unfortunately I also don't have access to a fabric workspace to validate, but I assume @RobinLin666 can see this work live :).

[RobinLin666]

Thanks @roeap , yes, I have validated in Fabric Notebook.

(For testing, I printed something, and deleted it in the PR)

[alamb]

I double checked and this enum is marked #[non_exhaustive] and thus it is ok to add new variants without breaking the API

[alamb]

Thank you very much @tustvold @RobinLin666 and @roeap 🙏