[RFC FS-1023 Discussion] - Allow type providers to generate types from types #125
Replies: 64 comments 2 replies
-
Would another type providers type be valid for input? |
Beta Was this translation helpful? Give feedback.
-
Would really like this to be able to know about existing F# types and not just work strictly in terms of CLR level types. Also Run-time type and IL generation seems like a poor alternative if unsafe code is not allowed for cross platform .NET Core libraries. I may be mistaken about this. |
Beta Was this translation helpful? Give feedback.
-
@7sharp9 asks:
I suppose so. Tricky to test thoroughly though. Would you see the full form or the erased form?
I think it would be difficult to get generative type providers working with this. Generative results are provided as an assembly. That assembly would have to include references to the assembly being compiled (without actually having that assembly available on-disk yet). |
Beta Was this translation helpful? Give feedback.
-
Erasing type providers only would severely limit the potential, but it seems maybe a good target as a first step forward. It seems to me that they could still be used to generate lenses or serialization but not more complex things like type proxies or future reflectable artifacts. Maybe for the sake of moving this forward it would be best to limit the scope as much as possible and allow more complex use cases later. |
Beta Was this translation helpful? Give feedback.
-
All of the use cases I'd personally have for this feature would pretty much require generative type providers - erasing really wouldn't add much benefit. Most of my "real world" use cases that I can think of where I'd use this in the near term would require non F# code to be able to use the generated types - and expect them to implement specific interfaces, etc. |
Beta Was this translation helpful? Give feedback.
-
Im struggling to invent many actual real world uses for this feature. For me this look like it just provides a typed input to be a parameter to be queried, but that does give much more than what you can do now with tom. yaml, xml, json as input parameter. I could see a use in using a record as an input parameter to a type provider, where the record only had primitive types. At compile time the record could be erased and the contents were available as input parameters as normal. But thats mainly as a means for efficiently inputting data inot a type provider. Maybe Im missing some bigger picture here, anyone want to enlighten me? |
Beta Was this translation helpful? Give feedback.
-
That would require the input to be instances of a type, not just the type, right? That seems even more difficult...
The main one I've tried are doing in the past was auto-implementation of INPC (I had a type provider that did this using strings to refer to the type in a separate assembly). However, I could see this being very useful for generating proxy types to work around existing types (think wrappers to automatically null->to->option an API or similar), and things along those lines. |
Beta Was this translation helpful? Give feedback.
-
I mostly want this for compile time code generation. That is, fast lenses and serialization (no runtime reflection). This is particularly important now with how limited Core CLR reflection and code gen is. Generics support is only a nice to have, not all that important out of the gate for me. |
Beta Was this translation helpful? Give feedback.
-
How does having a type as input make it easier to generate a lens, and what would such a generated lens look like? What about extrapolating json or xsd schema from a record then using that as input to a type provider? |
Beta Was this translation helpful? Give feedback.
-
Im just worried whats been discussed doesn't amount to any real use-case, just an interesting extension to type providers. |
Beta Was this translation helpful? Give feedback.
-
The great thing about a language feature like this is we don't need to decide exactly what they would look like now, it can be iteratively improved.
I don't really understand what you are suggesting here. You can't pass a runtime string into a type provider and keeping all of your records as xml schemas is pretty nuts.
One area where F# is extremely lacking compared to Haskell or OCaml is compile time code generation. It seems natural to me to extend type providers to fill this gap rather than having a whole new mechanism which would require new and separate tooling. |
Beta Was this translation helpful? Give feedback.
-
By changing the activation context of the type provider mechanism a single type could be provided to the type provider as input. I implemented some of that as a breaking change to the compiler but never progressed it far enough to know if it had merit. Im still not convinced this feature warrants the increased complexity and effort required. |
Beta Was this translation helpful? Give feedback.
-
(if the semantics are wrong, just imagine the closest thing that would be right 😜 ) CmdletProvider<...>The first example I can think of involves Argu and posh (powershell). This is a basic implementation of CLI args as a DU with Argu that gets you an argument parser with relative ease. // CLI argument DUs
type AddReferenceArgs =
| [<CLIAlt "-n">] Name of string
| [<CLIAlt "-p">] Project of string
interface IArgParserTemplate with
member this.Usage =
match this with
| Name _-> "Reference name"
| Project _ -> "Project to which reference will be added"
type AddProjectArgs =
| [<CLIAlt "-n">] Name of string
| [<CLIAlt "-p">] Project of string
interface IArgParserTemplate with
member this.Usage =
match this with
| Name _-> "Project reference path"
| Project _ -> "Project to which reference will be added"
// command handlers
let addReference cont (results : ParseResults<AddReferenceArgs>) =
maybe {
let! name = results.TryGetResult <@ AddReferenceArgs.Name @>
let! project = results.TryGetResult <@ AddReferenceArgs.Project @>
Furnace.loadFsProject project
|> Furnace.addReference (name, None, None, None, None, None)
|> ignore
return cont
}
let addProject cont (results : ParseResults<AddProjectArgs>) =
maybe {
let! path = results.TryGetResult <@ AddProjectArgs.Name @>
let! project = results.TryGetResult <@ AddProjectArgs.Project @>
let name = Path.GetFileName path
let newProject = Furnace.loadFsProject path
Furnace.loadFsProject project
|> Furnace.addProjectReference(path, Some name, None, newProject.ProjectData.Settings.ProjectGuid.Data, None)
|> ignore
return cont
} These CLI arg types and the handlers that process the argument values could be fed into a This is what an F# type that compiles into a Cmdlet looks like : [<Cmdlet("Search", "File")>]
type SearchFileCmdlet() =
inherit PSCmdlet()
/// Regex pattern used in the search.
[<Parameter(Mandatory = true, Position = 0)>]
[<ValidateNotNullOrEmpty>]
member val Pattern : string = null with get, set
/// Array of filename wildcards.
[<Parameter(Position = 1)>]
[<ValidateNotNull>]
member val Include = [|"*"|] with get,set
/// Whether or not to recurse from the current directory.
[<Parameter>]
member val Recurse : SwitchParameter = SwitchParameter(false) with get, set
/// Encoding to use when reading the files.
[<Parameter>]
member val Encoding = Encoding.ASCII with get, set
/// Toggle for case-sensitive search.
[<Parameter>]
member val CaseSensitive : SwitchParameter = SwitchParameter(false) with get, set
/// Do not use regex, just do a verbatim string search.
[<Parameter>]
member val SimpleMatch : SwitchParameter = SwitchParameter(false) with get, set
... With this kind of TypeProvider anyone who implemented a console CLI parser could get the equivalent posh Cmdlet for free ArgParserProvider<...>To push it even further a TypeProvider could be implemented to generate the Argu DUs that implement ArgParser in the first place Instead of type NewCommand =
| [<First>][<CLIAlt "project">] Project
| [<First>][<CLIAlt("file","-f","fl")>] File
| Nada
interface IArgParserTemplate with
member this.Usage =
match this with
| Project -> "Creates new project"
| File -> "Creates new file"
| Nada -> "doesn’t do much" An // library implementations
type ArgData = {
Case : string
Alts : string
Attrs : string list
Msg : string
Handler : string -> unit
}
let argData case alts attrs msg hdl =
{ Case = case; Alts= alts; Attrs=attrs, Msg = msg; Handler = hdl }
// user code
type NewCommand =
ArgParserProvider<
[ argData "Project" ["project"] ["First"] "Creates new project" projFn
; argData "File" ["project"] ["First"] "Creates new file" fileFn
; argData "Nada" [] [] "doesn’t do much" nadaFn
]
> And then if this generated type is fed into the UnionUtilityProvider<...>A TypeProvider used to enhance DUs that takes a type that satisfies type UnionUtility =
| ToString = 0
| Parse = 1
| ParseCaseInsensitive = 2
| TryParse = 3
| TryParseCaseInsensitive = 4
type MyDU = UnionUtilityProvider< typeof<MyDUBase>, ToString|||Parse|||TryParse > Another variant of this kind of provider could take an enum and transform it into a DU where it has an additional transformation method (e.g. FromInt ) that matches against the backing value type of its cases. StaticMemberProvider<...>A TypeProvider that takes a type and generates corresponding curried BehaviorProvider<...>Working with WPF entails a lot of boilerplate - type BlockShift() as self =
inherit Behavior<ScrollBar>()
let subscriptions = ResizeArray<IDisposable>()
static let depReg<'a> name = DependencyProperty.Register(name,typeof<'a>,typeof<BlockShift>)
[<DefaultValue(false)>]
static val mutable private SizeSmallProperty : DependencyProperty
[<DefaultValue(false)>]
static val mutable private SizeBigProperty : DependencyProperty
[<DefaultValue(false)>]
static val mutable private AttachedToProperty : DependencyProperty
[<DefaultValue(false)>]
static val mutable private BlockZoneProperty : DependencyProperty
static do
BlockShift.SizeSmallProperty <- depReg<float> "SizeSmall"
BlockShift.SizeBigProperty <- depReg<float> "SizeBig"
BlockShift.AttachedToProperty <- depReg<FrameworkElement> "AttachedTo"
BlockShift.BlockZoneProperty <- depReg<float> "BlockZone"
member __.DisposeOnDetach disposable = subscriptions.Add disposable
member __.SizeSmall
with get() = self.GetValue BlockShift.SizeSmallProperty :?> float
and set (v:float) = self.SetValue ( BlockShift.SizeSmallProperty, v )
member __.SizeBig
with get() = self.GetValue BlockShift.SizeBigProperty :?> float
and set (v:float) = self.SetValue ( BlockShift.SizeBigProperty, v )
member __.BlockZone
with get() = self.GetValue BlockShift.BlockZoneProperty :?> float
and set (v:float) = self.SetValue ( BlockShift.BlockZoneProperty, v )
member __.AttachedTo
with get() = self.GetValue BlockShift.AttachedToProperty :?> FrameworkElement
and set (v:FrameworkElement) = self.SetValue ( BlockShift.AttachedToProperty, v ) There's barely any distinct content in this type definition, we could probably cut this down to type BlockShift =
BehaviorProvider<
[ depReg<float> "SizeSmall"
; depReg<float> "SizeBig"
; depReg<FrameworkElement> "AttachedTo
; depReg<float> "BlockZone"
]
> ViewModelProvider<...>Defining a ViewModels type also involves far too much boilerplate type QueryBoxConfig = {
TextColor : Color
BlockColor : Color
ScrollVisibility : ScrollBarVisibility
HighlightColor : SolidColorBrush
ResultWidth : float
ScrollOpacity : float
BlockWidth : float
}
type QueryBoxViewModel<'DataType>(config:QueryBoxConfig, filterfn:string->'DataType->'DataType) =
inherit ViewModelBase()
let ( =+= ) arg1 arg2 = self.Factory.Backing(arg1,arg2)
// I've defined plenty of viewmodel types with 3X this number of properties
let textColor = <@ self.TextColor @> =+= config.TextColor
let blockColor = <@ self.BlockColor @> =+= config.BlockColor
let scrollVisible = <@ self.ScrollVisibility @> =+= config.ScrollVisibility
let highlightColor = <@ self.HighlightColor @> =+= config.HighlightColor
let resultWidth = <@ self.ResultWidth @> =+= config.ResultWidth
let scrollOpacity = <@ self.ScrollOpacity @> =+= config.Opacity
let blockWidth = <@ self.BlockWidth @> =+= config.BlockWidth
member __.TextColor
with get() = textColor.Value and set v = textColor.Value <- v
member __.BlockColor
with get() = blockColor.Value and set v = blockColor.Value <- v
member __.ScrollVisibility
with get() = scrollVisible.Value and set v = scrollVisible.Value <- v
member __.HighlightColor
with get() = highlightColor.Value and set v = highlightColor.Value <- v
member __.ResultWidth
with get() = resultWidth.Value and set v = resultWidth.Value <- v
member __.ScrollOpacity
with get() = scrollOpacity.Value and set v = scrollOpacity.Value <- v
member __.BlockWidth
with get() = blockWidth.Value and set v = blockWidth.Value <- v
// with a concrete form to back a specific GUI control
type CommandPaletteControlVM() =
inherit QueryListBoxViewModel<CommandData>(paletteConfig, filterPalettefn) Unfortunately a Concrete viewmodel type is needed because parametrically polymorphic types This could probably be cut down to type CommandPaletteControlVM =
ViewModelProvider<
{ TextColor = Brushes.LightGray.Color
BlockColor = Brushes.DarkCyan.Color
ScrollVisibility = ScrollBarVisibility.Hidden
HighlightColor = Brushes.Black
ResultWidth = 4.5
ScrollOpacity = 0.40
BlockWidth = 5.0
} : QueryBoxConfig
, typeof<CommandData
, filterPalettefn
> PhantomProvider<...>I'm not sure this can work, or what the proper syntax would be, but the idea type Message<'Tag> = PhantomProvider<string>
type DocumentBlock<'Tag> = PhantomProvider<char []>
type Letter = class end
type Email = class end
type SmokeSignal = class end
type Header = class end
type Footer = class end
let fn1 (block:DocumentBlock<Header>) : DocumentBlock<Header> =
block.GetLowerBound() |> ...
let fn2 (block:DocumentBlock<Footer>) : DocumentBlock<Footer> =
block.[0..5] |> ...
let fn3 (dispatch:Message<SmokeSignal>) : Message<Email> =
dispatch.Split [|';' ; '|' |]... Further Potential (the crazy stuff)Perhaps we could define TypeProviders that require the input type to implement an interface or It might also be possible to implement a TypeProvider similar to Nitra
The |
Beta Was this translation helpful? Give feedback.
-
I think all the use cases above can be divided into two categories:
The first category isn't worth to implement as you can do exactly the same stuff passing, say, JSON or XML or some sort of micro DSL to a TP, which may be more readable, than a F# record. As it's compile time checked (it is), I see little difference between this and passing a one-shot-use record. What do you guys think about implementing fsharp/fslang-suggestions#450 instead? However, it would result with ugly code like type TP = MyTP<<@ typeof<T> @>> I.e. you cannot pass other type easily, like Honesty, all these attempts to use TPs in cases where macros should be used result with restricted solutions and clunky client code. Considering how much time it would certainly take to implement, I'm not sure anymore it's worth it. Real macros should have access to everything available on a certain compilation stage, like current method, type, module, namespace, project. And it should be able to amend some parts of the typed AST, like adding/removing members, types, attributes and the like. I think the best thing we can do to add real metaprogramming into F# is learning how macros are implemented in, say, Nemerle, refine the idea, then make a plan to create something similar in F#. This is certainly deserve a lot of time and resources to implement. (somewhat) better TPs? I'm not sure. |
Beta Was this translation helpful? Give feedback.
-
type UC = {a:int; b:string}
let QT<'a> = <@ typeof<'a> @>
type PT = MyTP<QT<UC>> |
Beta Was this translation helpful? Give feedback.
-
Just confirming that what Colin said is accurate. |
Beta Was this translation helpful? Give feedback.
-
@colinbull, @TobyShaw : I was thinking of introducing/using F# for unit testing, being able to do so with some kind of type safety for private methods would be great. So, I'd naturally want them to be invokable. |
Beta Was this translation helpful? Give feedback.
-
It depends what you mean by invokable. If you mean: The type provider is able to invoke the method at compile time, and inspect the result. Then this will not be possible. If instead you mean: The type provider is able to provide a method which invokes a private method, despite being in a context where that method is not visible. Then this is another story, I think the answer is yes. I haven't really touched this aspect of the PR, so @colinbull could give a more convincing answer. |
Beta Was this translation helpful? Give feedback.
-
I am interested in this feature because I think it could be useful as a way of metaprogramming somewhat similar to what Haskell has with generics via I have a code base of a few thousand lines of F# code that used to be a Suave only backend application. After some back and forth, we arrived at FSharp.Json as a nice way to deal with json serialization that works via reflection and allows customization of the serialization process via Attributes. This worked nicely and required no additional code to actually serialize/deserialize all the fields in all the records/DUs. We are now moving some parts of this application to the SAFE stack and want to share the many type definitions between server and client (F# transpiled via Fable to JS). We are now switching to the Thoth library for Json serialization as it exists for both F# on the .NET runtime and as a Fable library and allows us to control serialization in the same way on both runtimes. However, this also means writing by hand a nontrivial amount of Json Encoders/Decoders. Reflection is of course not an option as it does not exist in full form in Fable atm. A code generator is being worked on by the Thoth author but a metaprogramming facility a la Haskells Am I right to hope that an F# type provider capable of generating types from types could be built if this feature were to land that would automate the creation of the Encoders/Decoders for the various types? To express it in code, could something like this: type Api =
{
Title: string
Version: string
Description: string
RecordName: string option
PluralRecordName: string option
}
static member Decoder token =
result {
let! title = Decode.field "title" Decode.string token
let! version = Decode.field "version" Decode.string token
let! description = Decode.field "description" Decode.string token
let! recordName = Decode.option (Decode.field "recordName" Decode.string) token
let! pluralRecordName = Decode.option (Decode.field "pluralRecordName" Decode.string) token
return {
Title = title
Version = version
Description = description
RecordName = recordName
PluralRecordName = pluralRecordName
}
}
static member Encoder (api : Api) =
Encode.object
[ "title", Encode.string api.Title
"version", Encode.string api.Version
"description", Encode.string api.Description
"recordName", Encode.option Encode.string api.RecordName
"pluralRecordName", Encode.option Encode.string api.PluralRecordName
]
let decoded = Thoth.Json.Net.Decode.decodeString Api.Decoder jsonStringValue be replaced with something like this? type Api =
{
Title: string
Version: string
Description: string
RecordName: string option
PluralRecordName: string option
}
type ApiJsonCoders = ThothTypeProvider<Api>()
let decoded = Thoth.Json.Net.Decode.decodeString ApiJsonCoders.Decoder jsonStringValue |
Beta Was this translation helpful? Give feedback.
-
This will definitely be enabled by this language feature. |
Beta Was this translation helpful? Give feedback.
-
I just wanted to reiterate again |
Beta Was this translation helpful? Give feedback.
-
I notice theres no comments on my above comment, is there enough content in this RFC for it to be useful even in the advent that the Type Provider cannot output non F# types? |
Beta Was this translation helpful? Give feedback.
-
Doesn't seem like an important intersection to me. The main use of this RFC is to avoid runtime reflection/codegen. This shouldn't require generating new F# types beyond the ones that are fed in. E.g. a |
Beta Was this translation helpful? Give feedback.
-
@7sharp9 But wouldn't it be possible to create the F# constructs when following the output of the F# compiler itself as described here? https://fsharpforfunandprofit.com/posts/fsharp-decompiled/ |
Beta Was this translation helpful? Give feedback.
-
When this RFC is implemented we will have primitive literals and types allowed as type provider arguments. I would assume that these are manageable because primitive literals and types are known statically. Whereas there isn't a way to mark non-primitive objects (including quotations) as constants, which makes those suggestions much harder? |
Beta Was this translation helpful? Give feedback.
-
It's more because TPs work internally via some name mangling, and mangling quotations and other structured data is going to lead to Very Very Long Internal Strings. I'm not sure there's any specific way around this (though perhaps we could use a hash as we do for the names of anonymous record types) |
Beta Was this translation helpful? Give feedback.
-
The thing that struck me was the scope of the following could be interpreted as F# types can be output after this implementation:
If the type argument was a F# union or record then the scope enlarges, possibly line this should be: This allows the provider to generate types based on analysis of the type argument(s), this would not allow the Type Provider to output F# specific types. Unless of course that work would be done along side this RFC. |
Beta Was this translation helpful? Give feedback.
-
Not sure this has been discussed before. type User =
{ Id: Guid
FirstName: string
LastName: string
Preference: Preference
Email: string
Friends: User [] }
// Post Request, Id, Preference, Friends fields are omitted
type NewUser = Omit<User, "Id|Preference|Friends">
type UserSummary = Omit<User, "Preference">
// type safe automapper?
type mapper = CreateMapper<UserDto, DbUserDto> This is borrowed from typescript but using TypeProvider Also would enable making constrained type easier. type PositiveInt = Constrained<int, <<@fun i -> i >= 0@>>
// generates the following code
module rec PositiveInt =
let validate = fun i -> i >= 0
let create str =
if validate str
then Some <| unsafeCreate str
else None
let unsafeCreate str = PositiveInt str
let inner = function PositiveInt x -> x
[<Struct>]
type T = private PositiveInt of int
type PositiveInt = PositiveInt.T And finally who doesn't want code first like database ORM/schema management, without some sort of IDE plugins? type UserTable =
{ Id: Guid
UserName: String
Email: String option
Password: String }
and TodoTable =
{ Id: Guid
Msg: String
UserId: Guid }
// Food for thought only
type DB = PostgresOrm<UserTable [] * TodoTable[], config> |
Beta Was this translation helpful? Give feedback.
-
What's the current status of this please? 😃 I am in agreement with @Rickasaurus, this will be a huge feature for F#. My use case: I'm generating serializers, by inspecting a Type using the reflection API. So far that is nothing unusual, and often done using runtime reflection. But I want to be able to impose constraints on the encoding of the types, and to have these verified at compile time. For example, if I need the encoding of a type Would this sort of thing be supported by the current proposal? (In the real use case, the constraints are a bit more flexible than "number of bytes", but I think that gives a good analogy.) |
Beta Was this translation helpful? Give feedback.
-
In addition to taking types as input, I'd like to pass in an assembly and be able to call This would allow one to generate serializers for all DTOs defined in a separate project with a single line. |
Beta Was this translation helpful? Give feedback.
-
This is the discussion thread for RFC FS-1023.
Beta Was this translation helpful? Give feedback.
All reactions