RFC: Your PGV Wishlist

[elliotmjackson]

The team at @bufbuild are excited to begin maintaining PGV.

This project fills a huge gap in the protobuf ecosystem and deserves careful stewardship. As a company, our incentives are well-aligned with the PGV community's. We want protobuf to be feature-complete and easy to use for personal projects, tiny startups, and giant megacorporations alike, because wider protobuf adoption increases the market for our paid schema registry product. We will take care not to break existing PGV users, and we can devote enough time to the project to clear out the issue backlog and possibly support additional languages. We will, of course, keep everything under an Apache 2 license.

We appreciate the PGV community's continued paitence whilst we iron out a detailed transition plan and thank the @envoyproxy team for all that hard work they've put in to date.

Right now, we want to hear from you - What do you want to see from PGV? What major bugs have you run into and would like to see fixed? Or better yet, what have you already fixed that you'd like merged?

[UPDATE]: Announcing protoc-gen-validate v1.0 and our plans for v2.0

[sarahegler]

would love to see ruby support!

[shine-garg-chime]

Some more color on the Ruby ask by Sarah. We use protobuf schemas heavily at Chime in general, and have lots of Ruby on Rails services that we actively want to move to Protobuf + Twirp. Autogeneration of parameter validation code will really DRY-up our APIs and help with better documentation too by making the semantic rules explicit. We'd greatly appreciate you prioritizing building support for Ruby in this tool. Thank you for taking ownership of PGV and really excited about its evolution!

[mmrath]

I would love to see issue #74 resolved.

[kindermoumoute]

IMO the most upvoted issues is a good start.

[Fleshgrinder]

Not having all valid issues closed without any resolution, since those were wishes. 😝

[pquerna]

Should or could we re-open specific issues that had good context and we're good wish list items, for example: #405

[elliotmjackson]

Hi everyone! This is my fault, after recently taking over maintenance of PGV with 100 issues and 30 open PRs, trying to get a grasp of the current state of the project has been tricky and after a month of fixing various bugs, some tickets I thought were stale or resolved may have been caught up in the mix.

I'll go back over the issues I closed and reopen any that stand out. If I miss any, please don't hesitate to reopen any of particular interest yourself.

But this is a good thing, whilst jagged, all of this helps me and the team develop a deeper understanding of where the project should go and where we should focus our attention immediately.

[utamori]

Node, Deno Support #71

If connect-node and PGV can be combined, great!

[Maragues]

Swift support 🙏🏻

[ysomad]

I would love to see official plugin for buf and more string rules such as alphanumeric etc. Thank you for your hard work! Love the project 💙

[ccakes]

Better structure for validation errors. When validation fails on a deeply-nested message instead of a massive string it'd be great to see JSONSchema-style errors with structure that can be handled by client code.

[trietsch]

Combined validations. E.g. if name = John, then age must > 50. Silly example, but it explains the idea.

message Person {
  string name = 1;
  int32 age = 2;
}

Real world example , we have an enum that determines whether this repeated should have size 1 or not.

[m430]

Custom error message!!!

[shuqingzai]

Custom error message!!!

[birdayz]

• make it possible to have a validation rule on both map keys and values. currently, only one of them is honored.
• typescript support, ideally so it can be used in combination with connect-web and/or ts-proto.

[aucampia]

I would like to be able to generate a JSON schema that includes validation rules specified using validate.rules.

Some options I can thing of:

• Generate some language neutral output that can be used from protoc-gen-jsonschema
• Change protoc-gen-jsonschema to understand the rules specified with validate.rules.
• Change protoc-gen-validate to generate JSON schema.

None of these options are great though. Would like to know what others think.

Closed issue relating to this request:

• Export schema #615

[aucampia]

I would like to be able to validate a stream of protobuf messages using protoc-gen-validate annotations without writing some custom code.

[dimuska139]

Export of validation rules to JSON-schema is the most needed feature in my opinion. I would like to use this so that front-end developers get the validation rules from the proto, and not duplicate them themselves.

[elb3k]

@elliotmjackson any timeline for the new release? The last release was on December 2022.

[chrusty]

@aucampia I'd be open to supporting this in protoc-gen-jsonschema for sure

[elliotmjackson]

Hello everyone!

I am excited to announce the release of protoc-gen-validate (PGV) v1.0! After half a decade of being labeled “alpha,” we’ve decided it’s time to promote PGV to v1.0, ensuring stability for all current workflows. 🎉

But that’s not all! We’re also announcing the imminent release of the future of validation: v2.0 of PGV, soon to be known as protovalidate. This new version, with its first release candidate coming in early June, will be based on Common Expression Language (CEL) and provide a much better foundation for validation going forward. 🚀

In our latest blog post, we reflect on PGV’s history, discuss its limitations, and give you a sneak peek into the exciting features we’re working on for protovalidate. Check it out here: Announcing protoc-gen-validate v1.0 and our plans for v2.0

As always, your feedback is invaluable to us. So please, take a moment to read the blog post and let us know your thoughts. Feel free to share your ideas or feature requests in this wishlist GitHub issue.

Thank you for your ongoing support, and we look forward to hearing from you as we continue to shape the future of Protobuf validation together!

[rofrankel]

In terms of feature requests/wishlist, have you given any thought to message-level rather than field-level validation? For example, google.type.Date says:

// This can represent one of the following:
//
// * A full date, with non-zero year, month, and day values
// * A month and day value, with a zero year, such as an anniversary
// * A year on its own, with zero month and day values
// * A year and month value, with a zero day, such as a credit card expiration
// date

It would be nice to be able to write a message option to express/enforce this, e.g. (year > 0 && month > 0 && day > 0) || (month > 0 && day > 0 && year == 0) || (year > 0 && month == 0 && day == 0) || (year > 0 && month > 0 && day ==0).

[anzboi]

Validation rules on methods!
Without this, it makes it really hard to define validation rules on messages that are intended to be shared between more than one endpoint, even if said messages only form part of a request.

Validation transferral
Really just throwing this idea out there. I would like the ability to say "Copy the validation rules from {name}", where name is the symbol of a proto message, eg: foo.bar.v1.Message.field.

[rofrankel]

A way to provide sample values would be great. These are useful for e.g. generating sample code snippets for APIs.

Arguably this is beyond the scope of validation, but on the other hand it does feel somewhat related, in the sense that the sample values should certainly pass the validation rules (and an API linter might want to validate such a thing).

[FrancoisHub]

Our team would love to have support for the C#/.Net ecosystem

gRPC and proto files are cross-platform and supported in many languages but, so far, PGV does not support C# despite being one of the most popular languages.
Having PGV support C# would greatly help in making gRPC a modern alternative to SOAP when it comes to interoperability across technical stacks for services that needs to be defined with rich data validation rules like it is possible in XSD for SOAP.

[jschaf]

This is great news! Some ideas I've been chewing on:

• Some sort of ternary logic similar to SQL check constraints where evaluating a null (missing field) is ignored.

• Define validation on methods, messages, and fields. Defining validation on methods and messages supports cross-field validation.

• An optimizing expression compiler. A validation like val > 10 should compile into an if statement. Alternately, use reflection to keep generated code size down.

• Support custom CEL methods.

[FarmerChillax]

• Support Customized error messages on validation failures #74

[so1n]

python support pydantic

[rofrankel]

We'd benefit from C#/.NET support at Roblox as well.

[nlachfr]

That looks promising !

I had a similar approach in my protoc-gen-cel-validate validation plugin built around CEL. Here are some features that I was able to implement and that might be useful here:

• Add the possibility to validate nested messages from CEL directly
• Support partial validation functions, with something like fieldmasks
• Support function overloads (registering functions in the CEL runtime from code)
• Having a way to reuse CEL constants / variables / functions to avoid duplication

[loeffel-io]

dart + typescript support 🙏

[Yuto2000]

Required validation for scalar types
#745

[elliotmjackson]

Hi Everyone! 🎉

Exciting times! Protovalidate's beta release for Go is now out! We’re making strides with C++, Java, TypeScript, and Python implementations too.

As we’re in beta, your feedback is critical to shape protovalidate. Please, give it a whirl and share your insights and suggestions. Your support is invaluable in this phase.

Thank you for your ongoing support, and we look forward to hearing from you as we continue to shape the future of Protobuf validation together!

[glerchundi]

This is fantastic! Congrats for this milestone! 💪🏻💪🏻

Just one question, where should I raise issues about the way of defining expressions/validation that are not tied to specific language? I have a question about why not infer the id in cel expressions.

Keep up the good work!

[rodaine]

Hey @glerchundi! Design discussions around protovalidate will get the most coverage in their issues. Would love to chat about decisions and explore other options there!

[felschr]

Rust / prost support 🦀

[rofrankel]

For the .NET users out there, I just provided a starting point C# implementation of CEL. I don't expect to be able to maintain this, but I'm happy for someone else to pick up where I left off. The README describes the known gaps (and there may be unknown ones). @Alfus FYI

[elliotmjackson]

Now that the following runtime implementations of protovalidate can be found in their own repositories, I'm going to close this issue.

1. Go: protovalidate-go (beta release)
2. C++: protovalidate-cc (beta release)
3. Java: protovalidate-java (beta release)
4. Python: protovalidate-python (beta release)

I'd like to thank everyone here for getting involved and letting us know what your wanting out of protobuf validation. Its because of you that we made some of the decisions that shaped protovalidate into what it is today. For that, we are grateful.

I've gone through the comments here and tried to create somewhat of a summary. If i have missed your suggestion, this was not intentional please feel free to open a feature request over at protovalidate.

Features we support in protovalidate today:

1. Better structure for validation errors
2. Combined validations
3. Custom error messages
4. Validation rules on both map keys and values
5. Ternary logic for null evaluation in validation
6. Define validation on messages and fields
7. Custom CEL methods
8. Validate nested messages from CEL directly

Features we will consider support for in the future:

1. Reuse CEL constants, variables, and functions #51
2. TypeScript support #67
3. Ruby support #68
4. C# support #74
5. Swift support #71
6. Validation rules on methods #72

Features we don't currently plan to support or don't have a clear path to support:

1. Generate JSON schema with validation rules
2. Export validation rules to JSON-schema
3. Sample values for validation rules
4. Optimizing expression compiler
5. Python support with pydantic

Thank you again and hope to see your handles over at the new project.

[rayokota]

For the .NET users out there, I just provided a starting point C# implementation of CEL.

Another C# implementation for CEL is here.