Switch Language: 简体中文 | Latest API Docs | Roadmap | Join Wechat Group
Colab Examples: LLM: Data Synthesis | LLM: Off-Table Inference | Billion-Level-Data supported CTGAN
The Synthetic Data Generator (SDG) is a specialized framework designed to generate high-quality structured tabular data.
Synthetic data does not contain any sensitive information, yet it retains the essential characteristics of the original data, making it exempt from privacy regulations such as GDPR and ADPPA.
High-quality synthetic data can be safely utilized across various domains including data sharing, model training and debugging, system development and testing, etc.
We are excited to have you here and look forward to your contributions, get started with the project through this Contributing Overview Guide!
Our current key achievements and timelines are as follows:
🔥 Nov 21, 2024: 1) Model Integration - We've integrated the GaussianCopula model into our Data Processor System. Check out the code example in this PR; 2) Synthetic Quality - We implemented automatic detection of data column relationships and allowed for relationship specification, improved the quality of synthetic data(Code Example); 3) Performance Enhancement - We significantly reduced the memory usage of GaussianCopula when handling discrete data, enabling training on thousands of categorical data entries with a 2C4G setup!
🔥 May 30, 2024: The Data Processor module was officially merged. This module will: 1) help SDG convert the format of some data columns (such as Datetime columns) before feeded into the model (so as to avoid being treated as discrete types), and reversely convert the model-generated data into the original format; 2) perform more customized pre-processing and post-processing on various data types; 3) easily deal with problems such as null values in the original data; 4) support the plug-in system.
🔥 Feb 20, 2024: a single-table data synthesis model based on LLM is included, view colab example: LLM: Data Synthesis and LLM: Off-table Feature Inference.
🔧 Feb 7, 2024: We improved sdgx.data_models.metadata to support metadata information describing for single tables and multiple tables, support multiple data types, support automatic data type inference. view colab example: SDG Single-Table Metadata。
🔶 Dec 20, 2023: v0.1.0 released, a CTGAN model that supports billions of data processing capabilities is included, view our benchmark against SDV, where SDG achieved less memory consumption and avoided crashing during training. For specific use, view colab example: Billion-Level-Data supported CTGAN.
🔆 Aug 10, 2023: First line of SDG code committed.
For a long time, LLM has been used to understand and generate various types of data. In fact, LLM also has certain capabilities in tabular data generation. Also, it has some abilities that cannot be achieved by traditional (based on GAN methods or statistical methods) .
Our sdgx.models.LLM.single_table.gpt.SingleTableGPTModel implements two new features:
No training data is required, synthetic data can be generated based on metadata data, view in our colab example.
Infer new column data based on the existing data in the table and the knowledge mastered by LLM, view in our colab example.
- Technological advancements:
- Supports a wide range of statistical data synthesis algorithms, LLM-based synthetic data generation model is also integrated;
- Optimized for big data, effectively reducing memory consumption;
- Continuously tracking the latest advances in academia and industry, and introducing support for excellent algorithms and models in a timely manner.
- Privacy enhancements:
- SDG supports differential privacy, anonymization and other methods to enhance the security of synthetic data.
- Easy to extend:
- Supports expansion of models, data processing, data connectors, etc. in the form of plug-in packages.
You can use pre-built images to quickly experience the latest features.
docker pull idsteam/sdgx:latestpip install sdgxUse SDG by installing it through the source code.
git clone [email protected]:hitsz-ids/synthetic-data-generator.git
pip install .
# Or install from git
pip install git+https://github.com/hitsz-ids/synthetic-data-generator.gitfrom sdgx.data_connectors.csv_connector import CsvConnector
from sdgx.models.ml.single_table.ctgan import CTGANSynthesizerModel
from sdgx.synthesizer import Synthesizer
from sdgx.utils import download_demo_data
# This will download demo data to ./dataset
dataset_csv = download_demo_data()
# Create data connector for csv file
data_connector = CsvConnector(path=dataset_csv)
# Initialize synthesizer, use CTGAN model
synthesizer = Synthesizer(
model=CTGANSynthesizerModel(epochs=1), # For quick demo
data_connector=data_connector,
)
# Fit the model
synthesizer.fit()
# Sample
sampled_data = synthesizer.sample(1000)
print(sampled_data)Real data are as follows：
>>> data_connector.read()
age workclass fnlwgt education ... capitalloss hoursperweek native-country class
0 2 State-gov 77516 Bachelors ... 0 2 United-States <=50K
1 3 Self-emp-not-inc 83311 Bachelors ... 0 0 United-States <=50K
2 2 Private 215646 HS-grad ... 0 2 United-States <=50K
3 3 Private 234721 11th ... 0 2 United-States <=50K
4 1 Private 338409 Bachelors ... 0 2 Cuba <=50K
... ... ... ... ... ... ... ... ... ...
48837 2 Private 215419 Bachelors ... 0 2 United-States <=50K
48838 4 NaN 321403 HS-grad ... 0 2 United-States <=50K
48839 2 Private 374983 Bachelors ... 0 3 United-States <=50K
48840 2 Private 83891 Bachelors ... 0 2 United-States <=50K
48841 1 Self-emp-inc 182148 Bachelors ... 0 3 United-States >50K
[48842 rows x 15 columns]Synthetic data are as follows：
>>> sampled_data
age workclass fnlwgt education ... capitalloss hoursperweek native-country class
0 1 NaN 28219 Some-college ... 0 2 Puerto-Rico <=50K
1 2 Private 250166 HS-grad ... 0 2 United-States >50K
2 2 Private 50304 HS-grad ... 0 2 United-States <=50K
3 4 Private 89318 Bachelors ... 0 2 Puerto-Rico >50K
4 1 Private 172149 Bachelors ... 0 3 United-States <=50K
.. ... ... ... ... ... ... ... ... ...
995 2 NaN 208938 Bachelors ... 0 1 United-States <=50K
996 2 Private 166416 Bachelors ... 2 2 United-States <=50K
997 2 NaN 336022 HS-grad ... 0 1 United-States <=50K
998 3 Private 198051 Masters ... 0 2 United-States >50K
999 1 NaN 41973 HS-grad ... 0 2 United-States <=50K
[1000 rows x 15 columns]- CTGAN：Modeling Tabular Data using Conditional GAN
- C3-TGAN: C3-TGAN- Controllable Tabular Data Synthesis with Explicit Correlations and Property Constraints
- TVAE：Modeling Tabular Data using Conditional GAN
- table-GAN：Data Synthesis based on Generative Adversarial Networks
- CTAB-GAN:CTAB-GAN: Effective Table Data Synthesizing
- OCT-GAN: OCT-GAN: Neural ODE-based Conditional Tabular GANs
The SDG project was initiated by Institute of Data Security, Harbin Institute of Technology. If you are interested in out project, welcome to join our community. We welcome organizations, teams, and individuals who share our commitment to data protection and security through open source:
- Read CONTRIBUTING before draft a pull request.
- Submit an issue by viewing View Good First Issue or submit a Pull Request.
- Join our Wechat Group through QR code.
The SDG open source project uses Apache-2.0 license, please refer to the LICENSE.
![@pre-commit-ci[bot]](https://avatars.githubusercontent.com/in/68672?s=64&v=4){kind=small}
![@allcontributors[bot]](https://avatars.githubusercontent.com/in/23186?s=64&v=4){kind=small}
![@sweep-ai[bot]](https://avatars.githubusercontent.com/in/307814?s=64&v=4){kind=small}
