Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature/migrate client #4281

Merged
merged 18 commits into from
Mar 6, 2024
Merged

feature/migrate client #4281

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
c470e0d
- adds migrate command handler
baywet Feb 29, 2024
0bd339c
- draft implementation of the migration logic
baywet Feb 29, 2024
12c3f94
- adds the description copy upon migration
baywet Mar 1, 2024
5c8da95
- updates specification regarding requests behaviour
baywet Mar 1, 2024
da0ccf5
makes the client name optional for the migrate command
baywet Mar 1, 2024
28a7a3e
- defaults the manifest on migration
baywet Mar 1, 2024
0d55f1e
- code linting
baywet Mar 1, 2024
f27b8dc
- code linting
baywet Mar 1, 2024
e3c0ab6
- moves api url logic outside of builder
baywet Mar 1, 2024
9e73e70
- sets the base url for migration
baywet Mar 1, 2024
beca262
- fixes lock file deletion
baywet Mar 1, 2024
2a09fdb
- updates message display for migrate command
baywet Mar 1, 2024
a9006bc
- adds launch configuration for migrate
baywet Mar 1, 2024
317cd2d
- code linting
baywet Mar 1, 2024
852ad09
- adds tests for lock delete
baywet Mar 1, 2024
3f3437b
- adds unit tests for migration orchestration
baywet Mar 1, 2024
de3bc2a
Merge branch 'main' into feature/migrate-client
baywet Mar 5, 2024
06828db
- removes unnecessary field
baywet Mar 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .vscode/launch.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -265,6 +265,20 @@
"console": "internalConsole",
"stopAtEntry": false
},
{
"name": "Launch Migrate",
"type": "coreclr",
"request": "launch",
"preLaunchTask": "build",
"program": "${workspaceFolder}/src/kiota/bin/Debug/net8.0/kiota.dll",
"args": ["config", "migrate"],
"cwd": "${workspaceFolder}/samples/msgraph-mail/dotnet",
"console": "internalConsole",
"stopAtEntry": false,
"env": {
"KIOTA_CONFIG_PREVIEW": "true"
}
},
{
"name": "Launch Login (github - device)",
"type": "coreclr",
Expand Down
62 changes: 7 additions & 55 deletions specs/cli/config-migrate.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,16 @@

This command is valuable in cases where a code base was created with Kiota v1.0 and needs to be migrated to the latest version of Kiota. The `kiota config migrate` command will identify and locate the closest `kiota-config.json` file available. If a file can't be found, it would initialize a new `kiota-config.json` file. Then, it would identify all `kiota-lock.json` files that are within this folder structure and add each of them to the `kiota-config.json` file. Adding the clients to the `kiota-config.json` file would not trigger the generation as it only affects the `kiota-config.json` file. The `kiota client generate` command would need to be executed to generate the code for the clients.

The API manifest won't contain any request after the migration since it could lead to misalignments between the generated client and the reported requests if the description has changed between the initial generation of the client and the migration. To get the requests populated, the user will need to use the generate command.

In the case where conflicting API client names would be migrated, the command will error out and invite the user to re-run the command providing more context for the `--client-name` parameter.

## Parameters

| Parameters | Required | Example | Description | Telemetry |
| -- | -- | -- | -- | -- |
| `--lock-location` | No | ./output/pythonClient/kiota-lock.json | Location of the `kiota-lock.json` file. If not specified, all `kiota-lock.json` files within in the current directory tree will be used. | Yes, without its value |
| `--client-name \| --cn` | No | graphDelegated | Used with `--lock-location`, it would allow to specify a name for the API client. Else, name is auto-generated as a concatenation of the `language` and `clientClassName`. | Yes, without its value |
| `--lock-directory \| --ld` | No | ./output/pythonClient/ | Relative path to the directory containing the `kiota-lock.json` file. If not specified, all `kiota-lock.json` files within in the current directory tree will be used. | Yes, without its value |
| `--client-name \| --cn` | No | graphDelegated | Used with `--lock-directory`, it would allow to specify a name for the API client. Else, name is auto-generated as a concatenation of the `language` and `clientClassName`. | Yes, without its value |

## Using `kiota config migrate`

Expand Down Expand Up @@ -80,64 +82,14 @@ _The resulting `apimanifest.json` file will look like this:_
"apiDescriptionUrl": "https://aka.ms/graph/v1.0/openapi.yaml",
"apiDeploymentBaseUrl": "https://graph.microsoft.com",
"apiDescriptionVersion": "v1.0",
"requests": [
{
"method": "GET",
"uriTemplate": "/users"
},
{
"method": "POST",
"uriTemplate": "/users"
},
{
"method": "GET",
"uriTemplate": "/users/$count"
},
{
"method": "GET",
"uriTemplate": "/users/{user-id}"
},
{
"method": "PATCH",
"uriTemplate": "/users/{user-id}"
},
{
"method": "DELETE",
"uriTemplate": "/users/{user-id}"
}
]
"requests": []
},
"pythonGraphServiceClient": {
"x-ms-kiotaHash": "9EDF8506CB74FE44...",
"apiDescriptionUrl": "https://aka.ms/graph/v1.0/openapi.yaml",
"apiDeploymentBaseUrl": "https://graph.microsoft.com",
"apiDescriptionVersion": "v1.0",
"requests": [
{
"method": "GET",
"uriTemplate": "/users"
},
{
"method": "POST",
"uriTemplate": "/users"
},
{
"method": "GET",
"uriTemplate": "/users/$count"
},
{
"method": "GET",
"uriTemplate": "/users/{user-id}"
},
{
"method": "PATCH",
"uriTemplate": "/users/{user-id}"
},
{
"method": "DELETE",
"uriTemplate": "/users/{user-id}"
}
]
"requests": []
}
}
}
Expand Down Expand Up @@ -181,7 +133,7 @@ Assuming the following folder structure:
```

```bash
kiota config migrate --lock-location ./generated/graph/csharp/kiota-lock.json --client-name GraphClient
kiota config migrate --lock-directory ./generated/graph/csharp --client-name GraphClient
```

_The resulting `kiota-config.json` file will look like this:_
Expand Down
21 changes: 21 additions & 0 deletions src/Kiota.Builder/Extensions/OpenApiDocumentExtensions.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
using System.Collections.Concurrent;
using System.Linq;
using System.Threading.Tasks;
using Kiota.Builder.EqualityComparers;
using Microsoft.OpenApi.Models;

namespace Kiota.Builder.Extensions;
Expand All @@ -26,4 +27,24 @@ internal static void InitializeInheritanceIndex(this OpenApiDocument openApiDocu
});
}
}
internal static string? GetAPIRootUrl(this OpenApiDocument openApiDocument, string openAPIFilePath)
{
ArgumentNullException.ThrowIfNull(openApiDocument);
var candidateUrl = openApiDocument.Servers
.GroupBy(static x => x, new OpenApiServerComparer()) //group by protocol relative urls
.FirstOrDefault()
?.OrderByDescending(static x => x.Url, StringComparer.OrdinalIgnoreCase) // prefer https over http
?.FirstOrDefault()
?.Url;
if (string.IsNullOrEmpty(candidateUrl))
return null;
else if (!candidateUrl.StartsWith("http", StringComparison.OrdinalIgnoreCase) &&
openAPIFilePath.StartsWith("http", StringComparison.OrdinalIgnoreCase) &&
Uri.TryCreate(openAPIFilePath, new(), out var filePathUri) &&
Uri.TryCreate(filePathUri, candidateUrl, out var candidateUri))
{
candidateUrl = candidateUri.ToString();
}
return candidateUrl.TrimEnd(KiotaBuilder.ForwardSlash);
}
}
159 changes: 11 additions & 148 deletions src/Kiota.Builder/KiotaBuilder.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,9 @@
using System.Linq;
using System.Net.Http;
using System.Runtime.CompilerServices;
using System.Security;
using System.Text.RegularExpressions;
using System.Threading;
using System.Threading.Tasks;
using AsyncKeyedLock;
using DotNet.Globbing;
using Kiota.Builder.Caching;
using Kiota.Builder.CodeDOM;
Expand All @@ -25,18 +23,14 @@
using Kiota.Builder.Manifest;
using Kiota.Builder.OpenApiExtensions;
using Kiota.Builder.Refiners;
using Kiota.Builder.SearchProviders.APIsGuru;
using Kiota.Builder.Validation;
using Kiota.Builder.WorkspaceManagement;
using Kiota.Builder.Writers;
using Microsoft.Extensions.Logging;
using Microsoft.OpenApi.Any;
using Microsoft.OpenApi.ApiManifest;
using Microsoft.OpenApi.MicrosoftExtensions;
using Microsoft.OpenApi.Models;
using Microsoft.OpenApi.Readers;
using Microsoft.OpenApi.Services;
using Microsoft.OpenApi.Validations;
using HttpMethod = Kiota.Builder.CodeDOM.HttpMethod;
[assembly: InternalsVisibleTo("Kiota.Builder.Tests, PublicKey=0024000004800000940000000602000000240000525341310004000001000100957cb48387b2a5f54f5ce39255f18f26d32a39990db27cf48737afc6bc62759ba996b8a2bfb675d4e39f3d06ecb55a178b1b4031dcb2a767e29977d88cce864a0d16bfc1b3bebb0edf9fe285f10fffc0a85f93d664fa05af07faa3aad2e545182dbf787e3fd32b56aca95df1a3c4e75dec164a3f1a4c653d971b01ffc39eb3c4")]

Expand Down Expand Up @@ -64,9 +58,11 @@
MaxDegreeOfParallelism = config.MaxDegreeOfParallelism,
};
var workingDirectory = Directory.GetCurrentDirectory();
workspaceManagementService = new WorkspaceManagementService(logger, useKiotaConfig, workingDirectory);
workspaceManagementService = new WorkspaceManagementService(logger, client, useKiotaConfig, workingDirectory);
this.useKiotaConfig = useKiotaConfig;
openApiDocumentDownloadService = new OpenApiDocumentDownloadService(client, logger);
}
private readonly OpenApiDocumentDownloadService openApiDocumentDownloadService;
private readonly bool useKiotaConfig;
private async Task CleanOutputDirectory(CancellationToken cancellationToken)
{
Expand Down Expand Up @@ -359,162 +355,29 @@
}
internal void SetApiRootUrl()
{
if (openApiDocument == null) return;
var candidateUrl = openApiDocument.Servers
.GroupBy(static x => x, new OpenApiServerComparer()) //group by protocol relative urls
.FirstOrDefault()
?.OrderByDescending(static x => x?.Url, StringComparer.OrdinalIgnoreCase) // prefer https over http
?.FirstOrDefault()
?.Url;
if (string.IsNullOrEmpty(candidateUrl))
{
if (openApiDocument is not null && openApiDocument.GetAPIRootUrl(config.OpenAPIFilePath) is string candidateUrl)
config.ApiRootUrl = candidateUrl;
else
logger.LogWarning("No server url found in the OpenAPI document. The base url will need to be set when using the client.");
return;
}
else if (!candidateUrl.StartsWith("http", StringComparison.OrdinalIgnoreCase) && config.OpenAPIFilePath.StartsWith("http", StringComparison.OrdinalIgnoreCase))
{
try
{
candidateUrl = new Uri(new Uri(config.OpenAPIFilePath), candidateUrl).ToString();
}
#pragma warning disable CA1031
catch (Exception ex)
#pragma warning restore CA1031
{
logger.LogWarning(ex, "Could not resolve the server url from the OpenAPI document. The base url will need to be set when using the client.");
return;
}
}
config.ApiRootUrl = candidateUrl.TrimEnd(ForwardSlash);
}
private void StopLogAndReset(Stopwatch sw, string prefix)
{
sw.Stop();
logger.LogDebug("{Prefix} {SwElapsed}", prefix, sw.Elapsed);
sw.Reset();
}

private static readonly AsyncKeyedLocker<string> localFilesLock = new(o =>
{
o.PoolSize = 20;
o.PoolInitialFill = 1;
});
private bool isDescriptionFromWorkspaceCopy;
private async Task<Stream> LoadStream(string inputPath, CancellationToken cancellationToken)
{
var stopwatch = new Stopwatch();
stopwatch.Start();

inputPath = inputPath.Trim();

Stream input;
if (useKiotaConfig &&
config.Operation is ClientOperation.Edit or ClientOperation.Add &&
await workspaceManagementService.GetDescriptionCopyAsync(config.ClientClassName, inputPath, cancellationToken).ConfigureAwait(false) is { } descriptionStream)
{
logger.LogInformation("loaded description from the workspace copy");
input = descriptionStream;
isDescriptionFromWorkspaceCopy = true;
}
else if (inputPath.StartsWith("http", StringComparison.OrdinalIgnoreCase))
try
{
var cachingProvider = new DocumentCachingProvider(httpClient, logger)
{
ClearCache = config.ClearCache,
};
var targetUri = APIsGuruSearchProvider.ChangeSourceUrlToGitHub(new Uri(inputPath)); // so updating existing clients doesn't break
var fileName = targetUri.GetFileName() is string name && !string.IsNullOrEmpty(name) ? name : "description.yml";
input = await cachingProvider.GetDocumentAsync(targetUri, "generation", fileName, cancellationToken: cancellationToken).ConfigureAwait(false);
logger.LogInformation("loaded description from remote source");
}
catch (HttpRequestException ex)
{
throw new InvalidOperationException($"Could not download the file at {inputPath}, reason: {ex.Message}", ex);
}
else
try
{
#pragma warning disable CA2000 // disposed by caller
var inMemoryStream = new MemoryStream();
using (await localFilesLock.LockAsync(inputPath, cancellationToken).ConfigureAwait(false))
{// To avoid deadlocking on update with multiple clients for the same local description
using var fileStream = new FileStream(inputPath, FileMode.Open);
await fileStream.CopyToAsync(inMemoryStream, cancellationToken).ConfigureAwait(false);
}
inMemoryStream.Position = 0;
input = inMemoryStream;
logger.LogInformation("loaded description from local source");
#pragma warning restore CA2000
}
catch (Exception ex) when (ex is FileNotFoundException ||
ex is PathTooLongException ||
ex is DirectoryNotFoundException ||
ex is IOException ||
ex is UnauthorizedAccessException ||
ex is SecurityException ||
ex is NotSupportedException)
{
throw new InvalidOperationException($"Could not open the file at {inputPath}, reason: {ex.Message}", ex);
}
stopwatch.Stop();
logger.LogTrace("{Timestamp}ms: Read OpenAPI file {File}", stopwatch.ElapsedMilliseconds, inputPath);
var (input, isCopy) = await openApiDocumentDownloadService.LoadStreamAsync(inputPath, config, workspaceManagementService, useKiotaConfig, cancellationToken).ConfigureAwait(false);
isDescriptionFromWorkspaceCopy = isCopy;
return input;
}

private const char ForwardSlash = '/';
public async Task<OpenApiDocument?> CreateOpenApiDocumentAsync(Stream input, bool generating = false, CancellationToken cancellationToken = default)
internal const char ForwardSlash = '/';
internal Task<OpenApiDocument?> CreateOpenApiDocumentAsync(Stream input, bool generating = false, CancellationToken cancellationToken = default)
{
var stopwatch = new Stopwatch();
stopwatch.Start();
logger.LogTrace("Parsing OpenAPI file");
var ruleSet = config.DisabledValidationRules.Contains(ValidationRuleSetExtensions.AllValidationRule) ?
ValidationRuleSet.GetEmptyRuleSet() :
ValidationRuleSet.GetDefaultRuleSet(); //workaround since validation rule set doesn't support clearing rules
if (generating)
ruleSet.AddKiotaValidationRules(config);
var settings = new OpenApiReaderSettings
{
RuleSet = ruleSet,
};
settings.AddMicrosoftExtensionParsers();
settings.ExtensionParsers.TryAdd(OpenApiKiotaExtension.Name, static (i, _) => OpenApiKiotaExtension.Parse(i));
try
{
var rawUri = config.OpenAPIFilePath.TrimEnd(ForwardSlash);
var lastSlashIndex = rawUri.LastIndexOf(ForwardSlash);
if (lastSlashIndex < 0)
lastSlashIndex = rawUri.Length - 1;
var documentUri = new Uri(rawUri[..lastSlashIndex]);
settings.BaseUrl = documentUri;
settings.LoadExternalRefs = true;
}
#pragma warning disable CA1031
catch
#pragma warning restore CA1031
{
// couldn't parse the URL, it's probably a local file
}
var reader = new OpenApiStreamReader(settings);
var readResult = await reader.ReadAsync(input, cancellationToken).ConfigureAwait(false);
stopwatch.Stop();
if (generating)
foreach (var warning in readResult.OpenApiDiagnostic.Warnings)
logger.LogWarning("OpenAPI warning: {Pointer} - {Warning}", warning.Pointer, warning.Message);
if (readResult.OpenApiDiagnostic.Errors.Any())
{
logger.LogTrace("{Timestamp}ms: Parsed OpenAPI with errors. {Count} paths found.", stopwatch.ElapsedMilliseconds, readResult.OpenApiDocument?.Paths?.Count ?? 0);
foreach (var parsingError in readResult.OpenApiDiagnostic.Errors)
{
logger.LogError("OpenAPI error: {Pointer} - {Message}", parsingError.Pointer, parsingError.Message);
}
}
else
{
logger.LogTrace("{Timestamp}ms: Parsed OpenAPI successfully. {Count} paths found.", stopwatch.ElapsedMilliseconds, readResult.OpenApiDocument?.Paths?.Count ?? 0);
}

return readResult.OpenApiDocument;
return openApiDocumentDownloadService.GetDocumentFromStreamAsync(input, config, generating, cancellationToken);
}
public static string GetDeeperMostCommonNamespaceNameForModels(OpenApiDocument document)
{
Expand Down Expand Up @@ -1107,7 +970,7 @@

if (!"string".Equals(parameter.Type.Name, StringComparison.OrdinalIgnoreCase) && config.IncludeBackwardCompatible)
{ // adding a second indexer for the string version of the parameter so we keep backward compatibility
//TODO remove for v2

Check warning on line 973 in src/Kiota.Builder/KiotaBuilder.cs

View workflow job for this annotation

GitHub Actions / Build 

Complete the task associated to this 'TODO' comment. (https://rules.sonarsource.com/csharp/RSPEC-1135)
var backCompatibleValue = (CodeIndexer)result[0].Clone();
backCompatibleValue.Name += "-string";
backCompatibleValue.IndexParameter.Type = DefaultIndexerParameterType;
Expand Down Expand Up @@ -1261,7 +1124,7 @@
var suffix = $"{operationType}Response";
var modelType = CreateModelDeclarations(currentNode, schema, operation, parentClass, suffix);
if (modelType is not null && config.IncludeBackwardCompatible && config.Language is GenerationLanguage.CSharp or GenerationLanguage.Go && modelType.Name.EndsWith(suffix, StringComparison.Ordinal))
{ //TODO remove for v2

Check warning on line 1127 in src/Kiota.Builder/KiotaBuilder.cs

View workflow job for this annotation

GitHub Actions / Build 

Complete the task associated to this 'TODO' comment. (https://rules.sonarsource.com/csharp/RSPEC-1135)
var obsoleteTypeName = modelType.Name[..^suffix.Length] + "Response";
if (modelType is CodeType codeType &&
codeType.TypeDefinition is CodeClass codeClass)
Expand Down Expand Up @@ -1394,7 +1257,7 @@
executorMethod.AddParameter(cancellationParam);// Add cancellation token parameter

if (returnTypes.Item2 is not null && config.IncludeBackwardCompatible)
{ //TODO remove for v2

Check warning on line 1260 in src/Kiota.Builder/KiotaBuilder.cs

View workflow job for this annotation

GitHub Actions / Build 

Complete the task associated to this 'TODO' comment. (https://rules.sonarsource.com/csharp/RSPEC-1135)
var additionalExecutorMethod = (CodeMethod)executorMethod.Clone();
additionalExecutorMethod.ReturnType = returnTypes.Item2;
additionalExecutorMethod.OriginalMethod = executorMethod;
Expand Down Expand Up @@ -2401,7 +2264,7 @@
if (!parameterClass.ContainsPropertyWithWireName(prop.WireName))
{
if (addBackwardCompatibleParameter && config.IncludeBackwardCompatible && config.Language is GenerationLanguage.CSharp or GenerationLanguage.Go)
{ //TODO remove for v2

Check warning on line 2267 in src/Kiota.Builder/KiotaBuilder.cs

View workflow job for this annotation

GitHub Actions / Build 

Complete the task associated to this 'TODO' comment. (https://rules.sonarsource.com/csharp/RSPEC-1135)
var modernProp = (CodeProperty)prop.Clone();
modernProp.Name = $"{prop.Name}As{modernProp.Type.Name.ToFirstCharacterUpperCase()}";
modernProp.SerializationName = prop.WireName;
Expand Down
9 changes: 8 additions & 1 deletion src/Kiota.Builder/Lock/LockManagementService.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ namespace Kiota.Builder.Lock;
/// </summary>
public class LockManagementService : ILockManagementService
{
private const string LockFileName = "kiota-lock.json";
internal const string LockFileName = "kiota-lock.json";
/// <inheritdoc/>
public IEnumerable<string> GetDirectoriesContainingLockFile(string searchDirectory)
{
Expand Down Expand Up @@ -132,4 +132,11 @@ private static string GetBackupFilePath(string outputPath)
var hashedPath = BitConverter.ToString((HashAlgorithm.Value ?? throw new InvalidOperationException("unable to get hash algorithm")).ComputeHash(Encoding.UTF8.GetBytes(outputPath))).Replace("-", string.Empty, StringComparison.OrdinalIgnoreCase);
return Path.Combine(Path.GetTempPath(), Constants.TempDirectoryName, "backup", hashedPath, LockFileName);
}
public void DeleteLockFile(string directoryPath)
{
ArgumentException.ThrowIfNullOrEmpty(directoryPath);
var lockFilePath = Path.Combine(directoryPath, LockFileName);
if (File.Exists(lockFilePath))
File.Delete(lockFilePath);
}
}
Loading
Loading