-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[move-compiler-v2] add parser code for lambda types #14792
base: main
Are you sure you want to change the base?
Conversation
⏱️ 2h 37m total CI duration on this PR
|
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## 09-27-extend_parser_for_lambda_types #14792 +/- ##
=======================================================================
Coverage ? 59.9%
=======================================================================
Files ? 852
Lines ? 207826
Branches ? 0
=======================================================================
Hits ? 124528
Misses ? 83298
Partials ? 0
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
7f59256
to
ae73f37
Compare
dbd482d
to
270214e
Compare
ae73f37
to
243ea0f
Compare
270214e
to
afbad9b
Compare
243ea0f
to
f3d30eb
Compare
a40a3c9
to
71ef91f
Compare
f3d30eb
to
0cc6890
Compare
71ef91f
to
3a1d362
Compare
0cc6890
to
1b54f30
Compare
c46a244
to
b3c1106
Compare
ad600f0
to
017f471
Compare
b3c1106
to
a65ee0f
Compare
017f471
to
b7d429a
Compare
a65ee0f
to
cbb77b0
Compare
b7d429a
to
c99de22
Compare
cbb77b0
to
b2185c2
Compare
c99de22
to
d3254ad
Compare
b2185c2
to
3f98e56
Compare
d3254ad
to
7e21319
Compare
3f98e56
to
e24bf4e
Compare
7e21319
to
7cfe2c4
Compare
e24bf4e
to
2a740e8
Compare
2a740e8
to
dabc427
Compare
4745b14
to
9b3a2af
Compare
third_party/move/move-compiler-v2/src/env_pipeline/function_checker.rs
Outdated
Show resolved
Hide resolved
@@ -1739,6 +1814,12 @@ impl ExpRewriterFunctions for LoopNestRewriter { | |||
#[derive(Debug, Clone, PartialEq, Eq, Hash)] | |||
pub enum Operation { | |||
MoveFunction(ModuleId, FunId), | |||
/// Build a closure by binding 1 or more leading arguments to a function value. | |||
/// First argument to the operation must be a function; the reamining |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: reaming -> remaining
9395a8f
to
cead217
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Submitting what I have as I saw some updates to the PR last night. Will continue reviewing.
/// fields, | ||
/// } = self; | ||
/// ... | ||
/// impl AstDebug for StructDefinition { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: unnecessary whitespace change?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Undone.
@@ -1901,9 +1902,35 @@ fn at_start_of_exp(context: &mut Context) -> bool { | |||
) | |||
} | |||
|
|||
// Parse the rest of a lambda expression, after already processing any capture designator (move/copy). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would be nice to have the (pseudo) grammar documented, similar to other parse_*
functions around here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
} else { | ||
// token is Tok::PipePipe, i.e., empty bind list in this context. | ||
consume_token(context.tokens, Tok::PipePipe)?; | ||
spanned(context.tokens.file_hash(), start_loc, start_loc + 1, vec![]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is start_loc + 1
the correct value for end
(because "||" is 2 chars)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess so. This code was just moved up from parse_exp
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@brmataptos I don't think this is correct. If you see other usages, end
is context.tokens.previous_end_loc()
(and this is start_loc + len of token).
So we should either make this start_loc + 2
or use context.tokens.previous_end_loc()
like elsewhere.
}; | ||
let body = Box::new(parse_exp(context)?); | ||
let abilities_start = context.tokens.start_loc(); | ||
let abilities = parse_with_abilities(context)?; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you please add a test where we use has
instead of with
for declaring abilities of a lambda (which I expect would be a common mistake to do, given all our previous ability declarations were with has
)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, added lambda/storable/parse_errors.move
to put some parse errors like this.
if !abilities.is_empty() { | ||
let abilities_end = context.tokens.previous_end_loc(); | ||
let loc = make_loc(context.tokens.file_hash(), abilities_start, abilities_end); | ||
require_move_2(context, loc, "Abilities on function expressions"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We would want to gate this with a higher language version (not just 2.0), say 2.2 at least (2.1 is the current stable language version).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changed to 2.2
@@ -2209,6 +2250,10 @@ fn parse_dot_or_index_chain(context: &mut Context) -> Result<Exp, Box<Diagnostic | |||
consume_token(context.tokens, Tok::RBracket)?; | |||
exp | |||
}, | |||
Tok::LParen => { | |||
let args = parse_call_args(context)?; | |||
Exp_::ExpCall(Box::new(lhs), args) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you please add some tests where we mix dot/index chains with call expressions? func_vector[0]()
etc.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added some tests to registry_ok.move to illustrate.
These will not run due to reentrancy, but are ok for now.
@@ -2443,11 +2488,18 @@ fn parse_type(context: &mut Context) -> Result<Type, Box<Diagnostic>> { | |||
Type_::Unit, | |||
) | |||
}; | |||
let abilities_start = context.tokens.start_loc(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Update doc string.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
if !abilities.is_empty() { | ||
let abilities_end = context.tokens.previous_end_loc(); | ||
let loc = make_loc(context.tokens.file_hash(), abilities_start, abilities_end); | ||
require_move_2(context, loc, "Ability constraints on function types"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lang version at least 2.2 instead of 2.0.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
// Parse an optional "with" type constraint: | ||
// WithAbilities = | ||
// ( "with" <Ability> (+ <Ability>)* )? | ||
fn parse_with_abilities(context: &mut Context) -> Result<Vec<Ability>, Box<Diagnostic>> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In the PR description (at the very beginning), it says syntax is extended with the following: move |args| body with copy, store
. But it looks like we use +
(not ,
) as the separator for abilities in both lambda expressions and types. Just double checking that this is what is intended and the PR description was a typo.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed the PR description.
@@ -689,7 +717,7 @@ pub enum Exp_ { | |||
// { seq } | |||
Block(Sequence), | |||
// | x1 [: t1], ..., xn [: tn] | e | |||
Lambda(TypedBindList, Box<Exp>), | |||
Lambda(TypedBindList, Box<Exp>, LambdaCaptureKind, Vec<Ability>), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
update comment above
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
} | ||
|
||
public fun test_functions() { | ||
// let sum = vector[]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
stray comment?
@@ -978,7 +978,8 @@ impl<'env> ExpRewriterFunctions for SimplifierRewriter<'env> { | |||
let ability_set = self | |||
.env() | |||
.type_abilities(&ty, self.func_env.get_type_parameters_ref()); | |||
ability_set.has_ability(Ability::Drop) | |||
// Don't drop a function-valued expression so we don't lose errors. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a test case corresponding to this?
@@ -14,14 +14,16 @@ | |||
//! | |||
//! Lambda lifting rewrites lambda expressions into construction | |||
//! of *closures*. A closure refers to a function and contains a partial list | |||
//! of arguments for that function, essentially currying it. Example: | |||
//! of arguments for that function, essentially currying it. We use the | |||
//! `EarlyBind` operation to construct a closure from a function and set of arguemnts, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
//! `EarlyBind` operation to construct a closure from a function and set of arguemnts, | |
//! `EarlyBind` operation to construct a closure from a function and set of arguments, |
// If final `args` match `lambda_params`, and all other args are simple, then returns | ||
// the simple prefix of `args`. | ||
fn get_args_if_simple<'b>( | ||
lambda_params: &Vec<Parameter>, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lambda_params: &Vec<Parameter>, | |
lambda_params: &[Parameter], |
@@ -96,7 +96,7 @@ fn generate_output(target: &FunctionTarget, test_output: &mut String) -> Option< | |||
}; | |||
*test_output += &format!( | |||
"--- Raw Generated AST\n{}\n\n", | |||
exp.display_for_fun(target.func_env.clone()) | |||
exp.display_for_fun(target.func_env) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can do the same (removing .clone()
) for two more instances below (assign-transformed and var-bound).
@@ -48,7 +48,7 @@ error: expression construct not supported in specifications | |||
69 │ ensures RET = {let y = x; y + 1}; | |||
│ ^^^^^^^^^^^^^^^^^^^^^^^^ | |||
|
|||
error: cannot pass `|(num, num)|num` to a function which expects argument of type `|(num, num)|bool` | |||
error: cannot pass `|(num, num)|num with copy+drop+store` to a function which expects argument of type `|(num, num)|bool` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder if we could remove abilities (with copy+drop+store) from the type error message if the error is not due to abilities mismatch.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is harder than it looks, in general at least. This will require changing the type unification algorithm to build an explanation when a difference is found.
I filed #15414.
│ Expected ';' | ||
│ ^ | ||
|
||
error: expected `|(integer, integer)|_` but found a value of type `integer` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the description of expected is likely not intuitive to the user: should it not be |integer, integer|
instead of |(integer, integer)|_
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The function doesn't exist. We don't know the return type here, so it is written as _
.
We can improve this, but it shouldn't be in this PR, as it will change all test outputs.
Filed #15415
name.display(env.symbol_pool()) | ||
), | ||
// TODO(LAMBDA) | ||
"Currently, lambda expressions must explicitly declare `move` capture of free variables, except when appearing as an argument to an inline functioncall." |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"Currently, lambda expressions must explicitly declare `move` capture of free variables, except when appearing as an argument to an inline functioncall." | |
"Currently, lambda expressions must explicitly declare `move` capture of free variables, except when appearing as an argument to an inline function call." |
} | ||
|
||
// Only allow simple expressions which cannot vary or abort | ||
fn exp_is_simple(fn_exp: &Exp) -> bool { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: fn_exp
-> exp
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We are using "simple" in two different contexts: arg being simple, and expression being simple. Can we use different terms here to avoid confusion?
None | ||
} | ||
|
||
// Only allow simple expressions which cannot vary or abort |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is an expression that can "vary"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A function call. I also don't want things to be too expensive, in case pulling them out of the lambda changes semantics. I changed the comment to indicate that.
If we weren't in rust, then I might be worried about a value that has references taken, but reference semantics should take care of that:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i've just pushed a new commit a moment ago to this PR.
I think I've addressed all reported issues, but I see 3 outstanding issues, seen by looking at lambda/storable/*_ok.lambda..exp
:
- some confusion between supported/implemented which is fixed in a stacked PR
- function values not allowed as field types (also for later PR)
- function types not allowed as type arguments (also for later PR)
I originally thought I could allow function calls to intermingle with .
and [...]
as in Rust (see Rust expression precedence allowing registry.functions[i].f(x)
where f
is a field of function type, but the simple approach leads to this being interpreted as calling a method through receiver syntax.
Instead, I'm requiring that a function-value used as a function be in parentheses for MVP:
f(x)
- (registry.functions[i].f)(x)
Rust does better, but perhaps this be useful enough for now.
So, PTAL. And review and "resolve" any comments you made which are ok.
third_party/move/move-compiler-v2/src/env_pipeline/function_checker.rs
Outdated
Show resolved
Hide resolved
@@ -689,7 +717,7 @@ pub enum Exp_ { | |||
// { seq } | |||
Block(Sequence), | |||
// | x1 [: t1], ..., xn [: tn] | e | |||
Lambda(TypedBindList, Box<Exp>), | |||
Lambda(TypedBindList, Box<Exp>, LambdaCaptureKind, Vec<Ability>), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
@@ -1901,9 +1902,35 @@ fn at_start_of_exp(context: &mut Context) -> bool { | |||
) | |||
} | |||
|
|||
// Parse the rest of a lambda expression, after already processing any capture designator (move/copy). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
Lambda(NodeId, Pattern, Exp), | ||
Lambda(NodeId, Pattern, Exp, LambdaCaptureKind, AbilitySet), | ||
/// Represents a reference to a Move Function as a function value. | ||
MoveFunctionExp(NodeId, ModuleId, FunId), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Renamed as EarlyBind
.
/// fields, | ||
/// } = self; | ||
/// ... | ||
/// impl AstDebug for StructDefinition { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Undone.
if !abilities.is_empty() { | ||
let abilities_end = context.tokens.previous_end_loc(); | ||
let loc = make_loc(context.tokens.file_hash(), abilities_start, abilities_end); | ||
require_move_2(context, loc, "Ability constraints on function types"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
@@ -2209,6 +2250,10 @@ fn parse_dot_or_index_chain(context: &mut Context) -> Result<Exp, Box<Diagnostic | |||
consume_token(context.tokens, Tok::RBracket)?; | |||
exp | |||
}, | |||
Tok::LParen => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, renamed it to parse_dot_or_index_or_call_chain
. Also fixed the bug in the syntax comment.
@@ -2209,6 +2250,10 @@ fn parse_dot_or_index_chain(context: &mut Context) -> Result<Exp, Box<Diagnostic | |||
consume_token(context.tokens, Tok::RBracket)?; | |||
exp | |||
}, | |||
Tok::LParen => { | |||
let args = parse_call_args(context)?; | |||
Exp_::ExpCall(Box::new(lhs), args) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added some tests to registry_ok.move to illustrate.
These will not run due to reentrancy, but are ok for now.
…le of function type
…unnecessarily; avoid generating errors/warnings about default Loc
…ionExpr to Value::Function, got things mostly back in shape
…for abilities in lambda_lifter.rs
…constraints to avoid reducing set of function type abilities when unifying types
…to bind leading arguments
cead217
to
44baaf0
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is still one parser fix I have requested (setting of span location for ||
when parsing lambdas), but approving assuming that is fixed.
Description
Extend syntax
move |args| body with copy+store
. Note that:has copy, store
because commas will confuse parsing in many situationhas copy+store
because that leads to nonuniform syntax withhas
with
, so that's what we're using.|T1, T2| T3 with copy+store
|a, b| f(x, 3, a, b)
EarlyBind
expression now takes a function-valued expression and a set of argument expressions, which correspond to a prefix of the function parameters.(|x, y| x + y)(2, 3)
Modify exp_builder and lambda_lifter to generate function values.
Modify model to track "used" functions in addition to "called" functions to be able to catch all dependencies when function values are created but not called.
Attaches an
AbilitySet
toType::Fun
andExp::Lambda
based on source. Adds a newExpCall
operation in parser/expansion ASTs to be able to carry more generalized function calls through to move-model, which already can support this throughInvoke
, which previously was underutilized. Added basic type checking for function abilities.Current Gaps
I was targeting the test cases
return_func.move
anddoable_func.move
, but this PR is still missingstore
for function values built by curry with only storable parametersI will try to patch those in while this is being reviewed, or add them to a later PR.
How Has This Been Tested?
Added more lambda tests under
move-compiler-v2/tests/lambda/
which are run "with" and "without" lambda features enabled. Currently, many things pass through to hit "not yet implemented" errors in bytecode gen, etc.Ran and carefully checked all tests under third_party and aptos.
Key Areas to Review
Key features are illustrated in test
move-compiler-v2/tests/lambda/storable/doable_func.move
and the two corresponding output files, along with the testreturn_func.move
in the same directory. Most other test outputs should be largely unchanged, although error messages related to lambda use in default configuration have changed.Tricky features are:
ty.rs
: unification checksFun
abilities as well as other things.lambda_lifter.rs
, we (1) reject lambdas withoutmove
free-var handling, (2) try to reduce lambda to curry, by checking for a simple function call with simple args, the last of which are identical to the lambda parameters.Type of Change
Which Components or Systems Does This Change Impact?
Checklist