Skip to content

Commit

Permalink
Incremental build system (#670)
Browse files Browse the repository at this point in the history 
* Initial check

* Get rid of .zip

* Fix x64 path bug in pipeline

* Dump obj to debug pipeline issue

* Get rid of duplicate files

* Fix missing partitions due to gitignore

* Additional updates for scripts

* Improve use of WinSDK assets by external projects

* Force the targets to skip scraping if we pass in a flag

* Remove creation of log dir

* Need to update .gitignore for generation project rename

* More upates

* Merge with main, update scripts with better error handling, don't clobber checked in global.json for tests

* Remove unnecessary line from script

* Bad variable in script
  • Loading branch information
@sotteson1
sotteson1 authored Sep 24, 2021
1 parent 01b6bd5 commit b07213e
Show file tree
Hide file tree
Showing 3,780 changed files with 6,038,597 additions and 1,662 deletions.
The diff you're trying to view is too large. We only load the first 3000 changed files.
6 changes: 3 additions & 3 deletions .editorconfig
View file
Original file line number Diff line number Diff line change
Expand Up @@ -129,9 +129,9 @@ csharp_indent_switch_labels = true
csharp_indent_labels = flush_left

# Prefer "var" everywhere
csharp_style_var_for_built_in_types = false
csharp_style_var_when_type_is_apparent = true:suggestion
csharp_style_var_elsewhere = false:warning
csharp_style_var_for_built_in_types = true
csharp_style_var_when_type_is_apparent = true
csharp_style_var_elsewhere = true

# Prefer method-like constructs to have a block body
csharp_style_expression_bodied_methods = false:none
Expand Down
15 changes: 5 additions & 10 deletions .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -352,18 +352,13 @@ MigrationBackup/

# Generation files
!generation/*.rsp
!generation/scraper/partitions/backup
!generation/scraper/partitions/debug
/sources/CsWin32ProjectionLib/RiaaSafeHandles.cs
/sources/CsWin32ProjectionLib/ProjectedWin32.cs
/sources/CsWin32/Properties/launchSettings.json
!generation/WinSDK/Partitions/backup
!generation/WinSDK/Partitions/debug
/.vscode/settings.json
/tests/SourceToWinmd/source/generated/fake/autotypes.cs
/generation/emitter/generated
/winsdk
/generation/scraper/autoTypes.generated.rsp
/generation/emitter/functionPointerFixups.generated.rsp
/generation/scraper/functionPointerFixups.generated.rsp
/generation/emitter/enumsMakeFlags.generated.rsp
/sources/GeneratorSdk/sdk/sdk.stamped.props
/tests/SourceToWinmd/source/generated/autotypes.cs

# This gets written when testing the packages, but we don't want to check it in
/sources/GeneratorSdk/samples/global.json
30 changes: 16 additions & 14 deletions CONTRIBUTING.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ You can contribute to this project by contributing to:

If you intend to contribute code changes, learn how to [set up your development environment](#Set-up-your-development-environment).

[Learn more](docs/generationOptions) about the options and inputs available for turning headers into Win32 metadata.

When contributing code changes, [validate](#Validating-changes) your changes by rebuilding the winmd and then inspecting the reported winmd diff to ensure all changes were intentional:

* [Full builds](#Full-builds)
Expand Down Expand Up @@ -55,29 +57,29 @@ Our tooling organizes Win32 APIs into namespaces. This provides an alternative w

The simplest way to assign an API to a namespace is to associate the API's header file with a [partition](generation/scraper/Partitions). A header file can only be associated with one partition, and by default all APIs included in the header file will be added to the partition's target namespace.

Partitions are defined as [folders](generation/scraper/Partitions) with [main.cpp](generation/scraper/Partitions/Registry/main.cpp) and [settings.rsp](generation/scraper/Partitions/Registry/settings.rsp) files.
* [main.cpp](generation/scraper/Partitions/Registry/main.cpp) contains `#include` statements like you would use to call the APIs directly. This typically includes the header files to be associated with the partition as well as any other dependent headers, included in the proper order.
* [settings.rsp](generation/scraper/Partitions/Registry/settings.rsp) associates a list of header files to a namespace. Reference existing [partitions](generation/scraper/Partitions) to understand the template for this file. The important sections are `--traverse`, which lists the header files to include, and `--namespace` which lists the namespace to associate with the content of those header files. Note that headers should be listed alphabetically by convention, and the casing needs to match the casing of the filenames.
Partitions are defined as [folders](generation/WinSDK/Partitions) with [main.cpp](generation/WinSDK/Partitions/Registry/main.cpp) and [settings.rsp](generation/WinSDK/Partitions/Registry/settings.rsp) files.
* [main.cpp](generation/WinSDK/Partitions/Registry/main.cpp) contains `#include` statements like you would use to call the APIs directly. This typically includes the header files to be associated with the partition as well as any other dependent headers, included in the proper order.
* [settings.rsp](generation/WinSDK/Partitions/Registry/settings.rsp) associates a list of header files to a namespace. Reference existing [partitions](generation/WinSDK/Partitions) to understand the template for this file. The important sections are `--traverse`, which lists the header files to include, and `--namespace` which lists the namespace to associate with the content of those header files. Note that headers should be listed alphabetically by convention, and the casing needs to match the casing of the filenames.

You can test localized changes to a partition by running `./scripts/GenerateMetadataSourceForPartition.ps1 <PARTITION>` from the repo root. If it compiles, the changes are likely correct. A common reason for failure is main.cpp either doesn't include all the necessary dependent headers needed to use the target headers or doesn't include them in the proper order.

### Split a header file among multiple namespaces

If a header file doesn't cleanly map to one namespace, it should be associated with a partition and namespace that makes sense for the majority of its APIs (using the steps above for a single namespace), and then the rest of the APIs should be manually remapped using [requiredNamespacesForNames.rsp](generation/emitter/requiredNamespacesForNames.rsp). This file contains one line per API and follows the format `<API>=<NAMESPACE>`. It is a single file that is shared across all partitions.
If a header file doesn't cleanly map to one namespace, it should be associated with a partition and namespace that makes sense for the majority of its APIs (using the steps above for a single namespace), and then the rest of the APIs should be manually remapped using [requiredNamespacesForNames.rsp](generation/WinSDK/requiredNamespacesForNames.rsp). This file contains one line per API and follows the format `<API>=<NAMESPACE>`. It is a single file that is shared across all partitions.

For maintainability, it is important to keep [requiredNamespacesForNames.rsp](generation/emitter/requiredNamespacesForNames.rsp) organized with APIs grouped by header files. APIs should be added within `# region <HEADER>` sections based on the header files where they are defined. APIs within a `# region` should be sorted alphabetically by selecting them and then using Visual Studio Code's `Sort Lines Ascending` command.
For maintainability, it is important to keep [requiredNamespacesForNames.rsp](generation/WinSDK/requiredNamespacesForNames.rsp) organized with APIs grouped by header files. APIs should be added within `# region <HEADER>` sections based on the header files where they are defined. APIs within a `# region` should be sorted alphabetically by selecting them and then using Visual Studio Code's `Sort Lines Ascending` command.

### Refactoring namespaces

Note that when refactoring namespaces, [requiredNamespacesForNames.rsp](generation/emitter/requiredNamespacesForNames.rsp) will take precedence over any namespaces declared in the partitions, so make sure it doesn't contain remappings that will conflict with the expected factoring from the partitions. For example, if you create a new Registry partition to assign everything in winreg.h to Windows.Win32.System.Registry, but [requiredNamespacesForNames.rsp](generation/emitter/requiredNamespacesForNames.rsp) was previously updated to map Reg* APIs to a different namespace, you won't achieve the desired result unless you remove the Reg* entries from [requiredNamespacesForNames.rsp](generation/emitter/requiredNamespacesForNames.rsp).
Note that when refactoring namespaces, [requiredNamespacesForNames.rsp](generation/WinSDK/requiredNamespacesForNames.rsp) will take precedence over any namespaces declared in the partitions, so make sure it doesn't contain remappings that will conflict with the expected factoring from the partitions. For example, if you create a new Registry partition to assign everything in winreg.h to Windows.Win32.System.Registry, but [requiredNamespacesForNames.rsp](generation/WinSDK/requiredNamespacesForNames.rsp) was previously updated to map Reg* APIs to a different namespace, you won't achieve the desired result unless you remove the Reg* entries from [requiredNamespacesForNames.rsp](generation/WinSDK/requiredNamespacesForNames.rsp).

Other files that need to be kept in sync when refactoring namespaces include [manual](generation/emitter/manual) and [autotypes.rsp](generation/emitter/autoTypes.rsp), which manually define some types, as well as [header.txt](generation/scraper/header.txt), which adds using statements to generated .cs files to resolve cross-namespace references.
Other files that need to be kept in sync when refactoring namespaces include [manual](generation/WinSDK/manual) and [autotypes.json](generation/WinSDK/autoTypes.json), which manually define some types, as well as [scraper.header.txt](generation/WinSDK/scraper.header.txt), which adds using statements to generated .cs files to resolve cross-namespace references.

## Enums

Our tooling produces enums for loosely typed parameters and fields that expect a closed set of values. This improves discoverability and usability of the values in the projections.

Enums are defined in [enums.json](generation/scraper/enums.json). This file provides a schema for extracting constants and macros from headers and emitting them in the winmd as an enum. Each object in the JSON array is an enum with the following properties:
Enums are defined in [enums.json](generation/WinSDK/enums.json). This file provides a schema for extracting constants and macros from headers and emitting them in the winmd as an enum. Each object in the JSON array is an enum with the following properties:

* `namespace` - Optional property indicating the namespace for the enum
* Note: If omitted, the enum will inherit the namespace of the first entry in `uses`
Expand Down Expand Up @@ -130,29 +132,29 @@ In the example below, a flags enum called `WNDCLASS_STYLES` is created with all
]
}
```
You can add new enums, modify existing enums, or apply enums to more APIs by modifying [enums.json](generation/scraper/enums.json).
You can add new enums, modify existing enums, or apply enums to more APIs by modifying [enums.json](generation/WinSDK/enums.json).

## Constants

Our tooling scans header files for constants and emits them into the namespace of the containing header file. The tool that handles this responsibility is called the [ConstantsScraper](sources/ConstantsScraper).

The behavior of the ConstantsScraper can be adjusted by modifying [ConstantsScraper.rsp](generation/scraper/ConstantsScraper.rsp):
The behavior of the ConstantsScraper can be adjusted by modifying [ConstantsScraper.settings.rsp](generation/WinSDK/ConstantsScraper.settings.rsp):

* The `--with-attribute` section declares attributes to add to constants (e.g. `E_NOTIMPL=NativeTypeName("HRESULT")`)
* The `--with-type` section declares constant types that cannot be detected automatically (e.g. `SOCK_STREAM=ushort`)
* The `--exclude` section declares constants to exclude from the metadata

Constants can be assigned to different namespaces than their header files by adding them to [requiredNamespacesForNames.rsp](generation/emitter/requiredNamespacesForNames.rsp) as described in [Split a header file among multiple namespaces](#Split-a-header-file-among-multiple-namespaces). Wildcards can be used (e.g. `DXGI_ERROR_*=Windows.Win32.Graphics.Dxgi`).
Constants can be assigned to different namespaces than their header files by adding them to [requiredNamespacesForNames.rsp](generation/WinSDK/requiredNamespacesForNames.rsp) as described in [Split a header file among multiple namespaces](#Split-a-header-file-among-multiple-namespaces). Wildcards can be used (e.g. `DXGI_ERROR_*=Windows.Win32.Graphics.Dxgi`).

The ConstantsScraper uses [regular expression matching](sources/MetadataUtils/ConstantsScraper.cs) on the header files to automatically extract constants. If a constant is not being detected, either a regular expression needs to be added or a regular expression would be insufficient. For cases where regular expressions would be insufficient, you can manually define constants in `.cs` files within [manual](generation/emitter/manual) (e.g. [RestartManager.manual.cs](generation/emitter/manual/RestartManager.manual.cs)).
The ConstantsScraper uses [regular expression matching](sources/MetadataUtils/ConstantsScraper.cs) on the header files to automatically extract constants. If a constant is not being detected, either a regular expression needs to be added or a regular expression would be insufficient. For cases where regular expressions would be insufficient, you can manually define constants in `.cs` files within [manual](generation/emitter/manual) (e.g. [RestartManager.manual.cs](generation/WinSDK/manual/RestartManager.manual.cs)).

Constants are removed from the metadata when they are detected as members of an enum.

## Attributes

Our tooling defines several [attributes](https://github.com/microsoft/win32metadata/tree/master/sources/Win32MetadataInterop) that can be applied to APIs to provide useful context to language projections.

To apply an attribute to an API, update the `--remap` section of [remap.rsp](https://github.com/microsoft/win32metadata/blob/master/generation/emitter/remap.rsp) in one of the following ways:
To apply an attribute to an API, update the `--memberRemap` section of [emitter.settings.rsp](https://github.com/microsoft/win32metadata/blob/master/generation/WinSDK/emitter.settings.rsp) in one of the following ways:

* `<API>=[<Attribute>]`
* Applies an attribute directly to an API (e.g. `MLOperatorAttributeType=[ScopedEnum]`)
Expand All @@ -175,7 +177,7 @@ If you encounter errors processing .zip or .winmd files, make sure that Git LFS

### Incremental builds

If you have already performed a full build and are making incremental changes, you can effectively perform incremental builds by running `./scripts/GenerateMetadataSourceForPartition.ps1 <PARTITION>` for each impacted `<PARTITION>`, and then build and test the winmd by running `./scripts/BuildMetadataBin.ps1 && ./scripts/TestWinmdBinary.ps1`. If you are only making changes to the [emitter](./generation/emitter), then you don't need to regenerate the partitions and can just rebuild the winmd.
The build system supports incremental builds. If you're just working on a partition, you can change partition files and try just scraping the headers by running `./scripts/ScrapeHeaders.ps1`. If you want to build everything includig the .winmd, run `./DoAll.ps1`.

Note that stale artifacts on your system may sometimes result in cryptic errors when attempting incremental builds. If you do encounter cryptic errors during incremental builds that you suspect are the result of previously built changes, reset your system state by running a clean build with `./DoAll.ps1 -Clean`.

Expand Down
2 changes: 1 addition & 1 deletion Directory.Build.props
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
<OutputPath>$(RepoRootPath)bin\$(Configuration)</OutputPath>
<PackageOutputPath>$(MSBuildThisFileDirectory)bin\Packages\$(Configuration)\NuGet\</PackageOutputPath>

<LocalTaskBinDir>$(RepoRootPath)bin\$(Configuration)\netstandard2.0\</LocalTaskBinDir>
<LocalTaskBinDir>$(RepoRootPath)bin\$(Configuration)\net5.0\</LocalTaskBinDir>

<LangVersion>9</LangVersion>

Expand Down
42 changes: 1 addition & 41 deletions DoAll.ps1
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,54 +18,14 @@ if ($Clean.IsPresent)

Install-BuildTools

# The libMapings.rsp is checked in as an optimization, but this
# this will regenerate it if it's not there
Invoke-PrepLibMappingsFile

if ($arch -eq "crossarch")
{
$arches = @("x64", "x86", "arm64")

foreach ($archItem in $arches)
{
.\scripts\GenerateMetadataSource.ps1 -arch $archItem
if ($LastExitCode -lt 0)
{
exit $LastExitCode
}
}
}
else
{
.\scripts\GenerateMetadataSource.ps1 -arch $arch
if ($LastExitCode -lt 0)
{
exit $LastExitCode
}
}

.\scripts\BuildMetadataBin.ps1 -arch $arch
if ($LastExitCode -lt 0)
{
exit $LastExitCode
}

if (!$ExcludePackages)
{
dotnet pack .\sources\packages.proj -c Release

.\scripts\UpdateGlobalJsonWinmdGeneratorVersion.ps1

dotnet pack .\sources\GeneratorSdk\samples\diasdk -c Release

dotnet build .\sources\GeneratorSdk\samples -c Release
.\scripts\DoPackages.ps1
}

if ($arch -eq "crossarch")
{
.\scripts\TestWinmdBinary.ps1
if ($LastExitCode -lt 0)
{
exit $LastExitCode
}
}
6 changes: 3 additions & 3 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ ClangSharp emits C# as it encounters types found in C/C++ headers. It will only

Example for Direct3DDxgi:

[generation/scraper/Partitions/Direct3DDxgi/main.cpp](generation/scraper/Partitions/Direct3DDxgi/main.cpp):
[generation/WinSDK/Partitions/Direct3DDxgi/main.cpp](generation/WinSDK/Partitions/Direct3DDxgi/main.cpp):

#include <winnt.h>
#include <winerror.h>
Expand All @@ -63,7 +63,7 @@ Example for Direct3DDxgi:
#include <dxgicommon.h>
#include <dxgiformat.h>

[generation/scraper/Partitions/Direct3DDxgi/settings.rsp](generation/scraper/Partitions/Direct3DDxgi/settings.rsp):
[generation/WinSDK/Partitions/Direct3DDxgi/settings.rsp](generation/WinSDK/Partitions/Direct3DDxgi/settings.rsp):

--traverse
<IncludeRoot>/shared/dxgitype.h
Expand Down Expand Up @@ -102,7 +102,7 @@ It starts emitting:

It has no way of knowing a typedef is coming (RECT). But, we can feed data into ClangSharp that tells it to rename tagRECT to RECT:

[generation/scraper/baseRemap.rsp](generation/scraper/baseRemap.rsp)
[generation/WinSDK/scraper.settings.rsp](generation/WinSDK/scraper.settings.rsp)

tagRECT=RECT

Expand Down
Loading

0 comments on commit b07213e

Please sign in to comment.