doc(dev-guide): reorg design docs (#17649)

Signed-off-by: xxchan <[email protected]>
risingwavelabs · Jul 10, 2024 · 16b358a · 16b358a
1 parent 0987e27
commit 16b358a
Show file tree

Hide file tree

Showing 10 changed files with 28 additions and 75 deletions.
diff --git a/docs/dev/src/SUMMARY.md b/docs/dev/src/SUMMARY.md
@@ -26,29 +26,27 @@
 # Specialized topics
 
 - [Develop Connectors](./connector/intro.md)
+    - [Source](./connector/source.md)
 - [Continuous Integration](./ci.md)
 
 ---
 
 # Design docs
 
-<!-- TODO: perhapts we need to reorder/group these docs? -->
-
 - [Architecture Design](./design/architecture-design.md)
-- [An Overview of RisingWave Streaming Engine](./design/streaming-overview.md)
-- [An Overview of RisingWave State Store](./design/state-store-overview.md)
+- [Streaming Engine](./design/streaming-overview.md)
+    - [Checkpoint](./design/checkpoint.md)
+    - [Aggregation](./design/aggregation.md)
+    - [MView on Top of MView](./design/mv-on-mv.md)
+    - [Backfill](./design/backfill.md)
+- [State Store](./design/state-store-overview.md)
+    - [Shared Buffer](./design/shared-buffer.md)
+    - [Relational Table](./design/relational-table.md)
+    - [Multiple Object Storage Backends](./design/multi-object-store.md)
 - [Meta Service](./design/meta-service.md)
-- [Create MView on Top of MView](./design/mv-on-mv.md)
-- [Checkpoint](./design/checkpoint.md)
-- [Design of Data Source](./design/data-source.md)
 - [Data Model and Encoding](./design/data-model-and-encoding.md)
-- [Design of Batch Local Execution Mode](./design/batch-local-execution-mode.md)
+- [Batch Local Execution Mode](./design/batch-local-execution-mode.md)
 - [Consistent Hash](./design/consistent-hash.md)
-- [Build RisingWave with Multiple Object Storage Backends](./design/multi-object-store.md)
-- [Backfill](./design/backfill.md)
-- [Aggregation](./design/aggregation.md)
-- [Shared Buffer](./design/shared-buffer.md)
-- [Relational Table](./design/relational-table.md)
 - [Keys](./design/keys.md)
 <!--
 

diff --git a/docs/dev/src/connector/intro.md b/docs/dev/src/connector/intro.md
@@ -1,5 +1,8 @@
 # Develop Connectors
 
+This page describes the development workflow to develop connectors. For design docs, see
+- [Source](./source.md)
+
 RisingWave supports a lot of connectors (sources and sinks).
 However, developing connectors is tricky because it involves external systems:
 

diff --git a/docs/dev/src/design/data-source.md → docs/dev/src/connector/source.md b/docs/dev/src/design/data-source.md → docs/dev/src/connector/source.md
@@ -1,15 +1,11 @@
-# Data Source
-
-- [Data Source](#data-source)
-  - [Components](#components)
-    - [Connectors](#connectors)
-    - [Enumerators](#enumerators)
-    - [ConnectorSource](#connectorsource)
-    - [SourceExecutor](#sourceexecutor)
-  - [How It Works](#how-it-works)
+# Source
 
 This page describes RisingWave's Data Source API and the architecture behind it. This may help if you are interested in how data sources work, or if you want to implement a new Data Source.
 
+For the workflow of developing connectors, see [Develop Connectors](../connector/intro.md).
+
+<!-- toc -->
+
 ## Components
 
 RisingWave's data source covers four parts: connectors, enumerators, ConnectorSource and SourceExecutor.
@@ -41,7 +37,7 @@ pub trait SplitReader: Sized {
 
 ### Enumerators
 
-`Enumerator` periodically requests upstream to discover changes in splits, and in most cases the number of splits only increases. The enumerator is a separate task that runs on the [meta](./meta-service.md). If the upstream split changes, the enumerator notifies the connector by means of config change to change the subscription relationship.
+`Enumerator` periodically requests upstream to discover changes in splits, and in most cases the number of splits only increases. The enumerator is a separate task that runs on the [meta](../design/meta-service.md). If the upstream split changes, the enumerator notifies the connector by means of config change to change the subscription relationship.
 
 All enumerators need to implement the following trait.
 

diff --git a/docs/dev/src/design/data-model-and-encoding.md b/docs/dev/src/design/data-model-and-encoding.md
@@ -1,11 +1,6 @@
 # Data Model and Encoding
 
-- [Data Model and Encoding](#data-model-and-encoding)
-  - [Data Model](#data-model)
-  - [In-Memory Encoding](#in-memory-encoding)
-  - [On-Disk Encoding](#on-disk-encoding)
-
-<!-- Created by https://github.com/ekalinin/github-markdown-toc -->
+<!-- toc -->
 
 ## Data Model
 
@@ -26,7 +21,7 @@ Primitive data types:
 - Strings: `VARCHAR`
 - Temporals: `DATE`, `TIMESTAMP`, `TIMESTAMP WITH TIME ZONE`, `TIME`, `INTERVAL`
 
-Composite data types (WIP):
+Composite data types:
 
 - `Struct`: A structure with a list of named, strong-typed fields.
 - `List`: A variable-length list of values with same data type.

diff --git a/docs/dev/src/design/meta-service.md b/docs/dev/src/design/meta-service.md
@@ -1,14 +1,6 @@
 # Meta Service
 
-- [Meta Service](#meta-service)
-  - [Background](#background)
-  - [Meta Store](#meta-store)
-  - [Types of Metadata](#types-of-metadata)
-    - [Catalog](#catalog)
-    - [Storage](#storage)
-  - [Push on Updates](#push-on-updates)
-
-<!-- Created by https://github.com/ekalinin/github-markdown-toc -->
+<!-- toc -->
 
 ## Background
 

diff --git a/docs/dev/src/design/multi-object-store.md b/docs/dev/src/design/multi-object-store.md
@@ -1,7 +1,6 @@
 # Build RisingWave with Multiple Object Storage Backends
 
-
-<!-- Created by https://github.com/ekalinin/github-markdown-toc -->
+<!-- toc -->
 
 ## Overview
 As a cloud-neutral database, RisingWave supports running on different (object) storage backends. Currently, these storage products include

diff --git a/docs/dev/src/design/mv-on-mv.md b/docs/dev/src/design/mv-on-mv.md
@@ -1,4 +1,4 @@
-# Create MView on Top of MView
+# MView on Top of MView
 
 ## Background
 

diff --git a/docs/dev/src/design/shared-buffer.md b/docs/dev/src/design/shared-buffer.md
@@ -1,14 +1,6 @@
 # The Hummock Shared Buffer
 
-Table of contents:
-
-- [Introduction](#introduction)
-- [Part 1: Async Checkpoint](#part-1-async-checkpoint)
-  - [Write Path](#write-path)
-  - [Read Path](#read-path)
-- [Part 2: Write Anytime / Async Flush](#part-2-write-anytime--async-flush)
-  - [A New Merge Iterator](#a-new-merge-iterator)
-    - [Considerations](#considerations)
+<!-- toc -->
 
 ## Introduction
 

diff --git a/docs/dev/src/design/state-store-overview.md b/docs/dev/src/design/state-store-overview.md
@@ -1,18 +1,6 @@
 # An Overview of RisingWave State Store
 
-- [An Overview of RisingWave State Store](#an-overview-of-risingwave-state-store)
-  - [Overview](#overview)
-  - [Architecture](#architecture)
-  - [The Hummock User API](#the-hummock-user-api)
-  - [Hummock Internals](#hummock-internals)
-    - [Storage Format](#storage-format)
-    - [Write Path](#write-path)
-    - [Read Path](#read-path)
-    - [Compaction](#compaction)
-    - [Transaction Management with Hummock Manager](#transaction-management-with-hummock-manager)
-    - [Checkpointing in Streaming](#checkpointing-in-streaming)
-
-<!-- Created by https://github.com/ekalinin/github-markdown-toc -->
+<!-- toc -->
 
 ## Overview
 

diff --git a/docs/dev/src/design/streaming-overview.md b/docs/dev/src/design/streaming-overview.md
@@ -1,16 +1,6 @@
 # An Overview of the RisingWave Streaming Engine
 
-- [An Overview of the RisingWave Streaming Engine](#an-overview-of-risingwave-streaming-engine)
-  - [Overview](#overview)
-  - [Architecture](#architecture)
-  - [Actors, executors, and states](#actors-executors-and-states)
-    - [Actors](#actors)
-    - [Executors](#executors)
-  - [Checkpoint, Consistency, and Fault tolerance](#checkpoint-consistency-and-fault-tolerance)
-    - [Barrier based checkpoint](#barrier-based-checkpoint)
-    - [Fault tolerance](#fault-tolerance)
-
-<!-- Created by https://github.com/ekalinin/github-markdown-toc -->
+<!-- toc -->
 
 ## Overview