[RFC] Dashboard/CRUD framework

[Janpot]

Which problem do we want to solve?

• MUI components are centred around UI, they concern themselves with the visual output of data, and the user interaction. Many of our internal apps and dashboards revolve around tabular data. Or data that can be represented as such. The connection between a stateless backend and a stateful component requires lots of boilerplate to set up.
• We have to repeat logic to do value formatting between the data grid, the charts. X Grid and Charts even have slightly different API for this. Similarly, we have to repeat logic for:
  • column validation
  • column filterability, and filter operators
  • synchronize colors belonging to a data point across charts
  • access control for crud operations
• Where as MUI X data sources]([data grid] Improve Server-side support with new data source mui-x#8179) focus on advanced use cases like infinite lazy loading, tree data, grouping,… much is left on the table for the very common CRUD use case.
• Lots of boilerplate required for creating deep linkable CRUD pages for our data backends.

How will we solve it?

• Data descriptors in data providers. The dashboard framework starts from the data instead of the UI.
• Smart grid/chart/… components that understand data fetching through those data providers, with loading and error states, and sensible defaults for things like labels/formatters/validation/colors/… in columns/series/forms/legends/…
• Data contexts that:
  • apply filtering to all usage of a data provider in its children
  • that synchronize hover state between charts and datagrids when hovering a data point
• Hooks that provide persisted state based on:
  • url parameters
  • local storage
  • session storage
• a CrudPage component that encapsulates routing, filtering, editing, deep linking.

The DataProvider interface

To create a data provider, the user has to implement a minimal interface

interface DataProvider<R> {
  getMany({ filter, pagination, sort }): { data: R[] };
  getOne?(id: Id);
  updateOne?(id: Id, values: Partial<R>);
  createOne?(values: Omit<R, 'id'>);
  deleteOne?(id: Id);
  fields: {
    [key in keyof R]: {
      label?: string;
      type?: 'string' | 'number' | '...';
      valueFormatter?({ value }): string;
      color?: string;
      // ...
    };
  };
}

• Data providers need to at least implement a getMany and a fields property.
• Data providers need to be initialized with the createDataProvider function and are always top-level values. Components in the page will assume they are referentially stable.
• Over time we can provide higher level methods to create data providers based on REST endpoints, graphql endpoints,…

Data components

We will provide versions of the X DataGrid, LineChart, BarChart,… components that take a dataProvider as a prop and that initialize the component with loading and error handling, and sensible defaults for columns, series, axes, legends,… All these defaults can be overwritten with the corresponding props in the original X component.

e.g.

import { DataGrid } from '@toolpad/core';

const myDataProvider = createDataProvider({
  getMany: () => ({ data: [] }),
  fields: {
    firstName: { label: 'First Name' },
    lastName: { label: 'Last Name' },
    age: { label: 'Age', valueFormatter: ({ value }) => `${value} years` },
  },
});

// this will render a grid with columns First Name, Last Name and Age
<DataGrid dataProvider={myDataProvider} />;

// this will render a grid with only the columns First Name and Age
// even though not provided, Age will be formatted according to the data provider
<DataGrid dataProvider={myDataProvider} columns={[{ field: 'firstName' }, { field: 'age' }]} />;

// this will render a grid with only the Age column formatted as a plain number
<DataGrid
  dataProvider={myDataProvider}
  columns={[{ field: 'age', valueFormatter: ({ value }) => String(value) }]}
/>;

When CRUD methods are provided we will make the grid editable and automatically call those methods after the user made the CRUD interaction.

DataContext

This binds a data provider to a filter for its subtree:

<DataContext dataProvider={myDataProvider} filter={{ age: { gte: 18, lte: 50 } }}>
  {/* This grid will now be filtered by age */}
  <Datagrid dataProvider={myDataProvider} />
  {/* This chart as well */}
  <LineChart dataProvider={myDataProvider} />
</DataContext>;

This allows for sharing filters between data display components.

Persistent state hooks

To make it more straightforward to store filter state of the dashboard in the url, We will provide state hooks that automatically sync with a query parameter:

import { useSearchParameterState, Codecs } from '@toolpad/core';
import { myDataProvider } from './data';

export function MyDashboard() {
  const [minAge, setMinAge] = useSearchParameterState<number>('min-age', {
    defaultValue: 18,
    codec: Codecs.Number,
  });

  const [maxAge, setMaxAge] = useSearchParameterState<number>('max-age', {
    defaultValue: 50,
    codec: Codecs.Number,
  });

  return (
    <React.Fragment>
      <NumberField value={minAge} onChange={(_, value) => setMinAge(value)} />
      <NumberField value={maxAge} onChange={(_, value) => setMaxAge(value)} />
      <DataContext dataProvider={myDataProvider} filter={{ age: { gte: minAge, lte: maxAge } }}>
        ...
      </DataContext>
    </React.Fragment>
  );
}

We will provide:

• useSearchParameterState: for syncing with a query parameter in the url
• useLocalStorageState: for syncing with a value in local storage
• useSessionsStorageState: for syncing with a value in session storage

The hook will take a codec parameter which is responsible for parsing/stringifying the values before persisting. We will provide a set of codecs for common data types.

It will also be possible to support more complex use-cases by providing your own custom codec. Or to create a codec based on a schema (e.g. zod)

We will make sure to guarantee in typescript a strong relationship between the state, setState types and the codec.

Demo: npm downloads

Here's an example of an app generated using the aforementioned abstractions, through simple markup:

npmStats.mov

<LineChart
  title="Material UI (@mui/material)"
  dataProvider={dataProvider}
  xAxis={X_AXIS}
  series={[
    {
      dataKey: 'muiMaterialDownloadsCount',
      showMark: false,
    },
    {
      dataKey: 'materialUiCoreDownloadsCount',
      showMark: false,
    },
  ]}
/>;

source: https://github.com/Janpot/dashboard-framework-prototype/tree/master/src/app/npm-stats
live: https://dashboard-framework-prototype.vercel.app/npm-stats

Demo: weather dashboard

Here's a simple dashboard that crunches some weather forecast data, and all the code for it:

weather.mov

"use client";

import * as React from "react";
import {
  Dashboard,
  DataGrid,
  LineChart,
  BarChart,
  Metric,
  useUrlQueryParameterState,
} from "@toolpad/dashboard";
import {
  Box,
  Card,
  CardContent,
  Container,
  Stack,
  TextField,
  Toolbar,
  Typography,
} from "@mui/material";
import Grid from "@mui/material/Unstable_Grid2/Grid2";
import { CITIES, forecast } from "./data";
import { MenuItem } from "@mui/material";

const FORECAST_X_AXIS = [{ dataKey: "time" }];

const DEFAULT_CITY = [...CITIES.keys()][0];

export default function DashboardContent() {
  const [city, setCity] = useUrlQueryParameterState("city", {
    defaultValue: DEFAULT_CITY,
  });

  return (
    <Dashboard bindings={[[forecast, { city: { eq: city } }]]}>
      <Container sx={{ mt: 5 }}>
        <Stack direction="column" spacing={4}>
          <Toolbar disableGutters>
            <Typography variant="h4">Weather Forecast</Typography>
            <Box sx={{ flexGrow: 1 }} />
            <TextField
              select
              label="City"
              value={city}
              onChange={(event) => setCity(event.target.value)}
            >
              {Array.from(CITIES.keys(), (city) => (
                <MenuItem key={city} value={city}>
                  {city}
                </MenuItem>
              ))}
            </TextField>
          </Toolbar>
          <DataGrid dataProvider={forecast} pagination autoPageSize />
          <Box>
            <Grid container spacing={4}>
              <Grid xs={12} md={4}>
                <Metric dataProvider={forecast} field="temperature" />
              </Grid>
              <Grid xs={12} md={4}>
                <Metric dataProvider={forecast} field="wind" />
              </Grid>
              <Grid xs={12} md={4}>
                <Metric dataProvider={forecast} field="precipitation" />
              </Grid>
            </Grid>
          </Box>
          <Card>
            <CardContent>
              <LineChart
                dataProvider={forecast}
                xAxis={FORECAST_X_AXIS}
                series={[{ dataKey: "temperature" }]}
              />
            </CardContent>
          </Card>
          <Card>
            <CardContent>
              <LineChart
                dataProvider={forecast}
                xAxis={FORECAST_X_AXIS}
                series={[{ dataKey: "wind", area: true }]}
              />
            </CardContent>
          </Card>
          <Card>
            <CardContent>
              <BarChart
                dataProvider={forecast}
                xAxis={FORECAST_X_AXIS}
                series={[{ dataKey: "precipitation" }]}
              />
            </CardContent>
          </Card>
        </Stack>
      </Container>
    </Dashboard>
  );
}

CRUD pages

The concept of data providers starts from the data, and can therefore be reused, not only to augment the existing X components, but also to create rich data interaction components.

// app/myData/[[...path]]/page.tsx
'use client';

import { CrudPage } from '@toolpad/core';
import { myDataProvider } from './data';

export default function MyDataPage() {
  // Full featured crud page with deep linkable overview, create, edit, show
  return <CrudPage dataProvider={myDataProvider} />;
}

This component, when rendered in a Next.js dynamic route, will provide pages

• /myData: lists the content of the data provider
• /myData/new: form to create a new record
• /myData/show/:id: shows record with id. This page has edit and delete buttons
• /myData/edit/:id: edit form for record with id

We will centralize validation for the records in the data provider and have that reused for both the create and edit flow.

Demo

Here's a demo of all the functionality that this component would enable:

crudPage.mov

Note: we use the /show/ and /edit/ prefixes to avoid :id clashing with the /new route

Customization

We will provide a set of hooks to support interacting with the data providers:

const { data, pending, error } = useMany(dataProvider);
const { data, pending, error } = useOne(dataProvider, id);
const { createOne, pending, error } = useCreateOne(dataProvider);
// createOne({ ...values })
const { updateOne, pending, error } = useUpdateOne(dataProvider);
// updateOne(id, { ...values })
const { deleteOne, pending, error } = useDeleteOne(dataProvider);
// deleteOne(id)

These hooks will also be aware of the data context and automatically apply the filter

License

As we're wrapping MUI X components, we'll have to consider licensing. We'll provide two packages:

• @toolpad/core: contains all the free code. it'll export all the MIT licensed APIs and components
• @toolpad/enterprise: contains all the commercial code. It'll export wrapped DataGridPro and DataGridPremium. To use these exports, the user will require a corresponding MUI X license.

Prototypes

We have prototyped examples for some of the components:

• a weather dashboard that shares a filter across data components:
  source: https://github.com/Janpot/dashboard-framework-prototype/tree/master/src/app/forecast
  live: https://dashboard-framework-prototype.vercel.app/forecast
• a KPI dashboard that uses a date filter and switches between two dataProviders based on a query parameter:
  source: https://github.com/Janpot/dashboard-framework-prototype/tree/master/src/app/npm-stats
  live: https://dashboard-framework-prototype.vercel.app/npm-stats
• a CRUD page for a dummy in-memory data provider:
  source: https://github.com/Janpot/dashboard-framework-prototype/tree/master/src/app/crud
  live: https://dashboard-framework-prototype.vercel.app/crud

Our Vision

The mission at MUI is to enable developers to build beautiful apps fast. To contribute to that, our vision in the Toolpad team is to create a product that is code-first, with pluggable low-code features. Think: developers building dashboards and internal tools super-fast with code, and non-devs being able to use UIs for configuration and access management. All in code that you own!

[buremba]

Hey! This looks great, the CRUD reminds me of Refine.dev which can be a good influence for this use-case.