pip install Faker<\/code><\/pre>\nTo use the Faker package to generate synthetic data, we need to initiate the Faker<\/code> class.<\/p>\nfrom faker import Faker\nfake = Faker()<\/code><\/pre>\nWith the class initiated, we could generate various synthetic data. For example, we would create a synthetic data name.<\/p>\n
fake.name()<\/code><\/pre>\n![\"Image](\"https:\/\/towardsdatascience.com\/wp-content\/uploads\/2022\/01\/1vS-M7b8WEsdkS19bnrtGpA.png\")
Image by Author<\/figcaption><\/figure>\nThe result is a person’s name when we use the .name<\/code> attribute from the Faker class. Faker synthetic data would produce randomly each time we run the attribute. Let’s run the name one more time.<\/p>\n![\"Image](\"https:\/\/towardsdatascience.com\/wp-content\/uploads\/2022\/01\/1BWx_NUjFqlG4d7s9EL3nlQ.png\")
Image by Author<\/figcaption><\/figure>\nThe result is a different name than our previous iteration. The randomization process is important in generating synthetic data because we want a variation in our dataset.<\/p>\n
There are many more variables we could generate using the Faker package. It is not limited to the name variable – the other example are address, bank, job, credit score, and many more. In the Faker package, this generator is called Provider<\/strong>. If you want to check the whole standard provider<\/a>, community provider<\/a>, and localized provider<\/a>, you could check it out in their documentation.<\/p>\n
\n2. SDV<\/h2>\n
SDV<\/a> or Synthetic Data Vault is a Python package to generate synthetic data based on the dataset provided. The generated data could be single-table, multi-table, or time-series, depending on the scheme you provided in the environment. Also, the generated would have the same format properties and statistics as the provided dataset.<\/p>\nSDV generates synthetic data by applying mathematical techniques and machine learning models such as the deep learning model. Even if the data contain multiple data types and missing data, SDV will handle it, so we only need to provide the data (and the metadata when required).<\/p>\n
Let’s try to generate our synthetic data with SDV. First, we need to install the package.<\/p>\n
pip install sdv<\/code><\/pre>\nFor our sample, I would use the Horse Survival Datase<\/a>t from Kaggle because they contain various datatype and missing data.<\/p>\nimport pandas as pd\ndata = pd.read_csv('horse.csv')\ndata.head()<\/code><\/pre>\n![\"Image](\"https:\/\/towardsdatascience.com\/wp-content\/uploads\/2022\/01\/1VOIA8kcyp4F94XwQuBn7rg.png\")
Image by Author<\/figcaption><\/figure>\nOur dataset is ready, and we want to generate synthetic data based on the dataset. Let’s use one of the available Singular Table SDV models<\/a>, GaussianCopula<\/code>.<\/p>\nfrom sdv.tabular import GaussianCopula\nmodel = GaussianCopula()\nmodel.fit(data)<\/code><\/pre>\nThe training process is easy; we need only to initiate the class and fit the data. Let’s use the model to produce synthetic data.<\/p>\n
sample = model.sample(200)\nsample.head()<\/code><\/pre>\n![\"Image](\"https:\/\/towardsdatascience.com\/wp-content\/uploads\/2022\/01\/1xALdmEYSGSLsrpNmN_HRyQ.png\")
Image by Author<\/figcaption><\/figure>\nWith the .sample<\/code> attribute from the model, we obtain the randomized synthetic data. How much data you want depends on the number you pass into the .sample<\/code> attribute.<\/p>\nYou might realize that the data sometimes contains a unique identifier. For example, I could assign the identifier in the above dataset as the ‘hospital_number.’ The data here above have multiple instances of ‘hospital_nunber,’ which is something we don’t want if it is unique data. In this case, we could pass the primary_key<\/code> parameter to the model.<\/p>\nmodel = GaussianCopula(primary_key='hospital_number')\nmodel.fit(data)<\/code><\/pre>\nThe sample result would be a unique primary key for each sample generated from the model.<\/p>\n
Another question we might ask is, how good is the generated synthetic data? In this case, we could use the evaluate<\/code> function from SDV. This evaluation would compare the real dataset with the sample dataset. Many tests are available, but we would only focus on the Kolmogorov\u2013Smirnov (KS) and Chi-Squared (CS) tests.<\/p>\nfrom sdv.evaluation import evaluate\nevaluate(sample, data, metrics=['CSTest', 'KSTest'], aggregate=False)<\/code><\/pre>\n![\"Image](\"https:\/\/towardsdatascience.com\/wp-content\/uploads\/2022\/01\/1gJ4tYGl586oRbzDVBllcoA.png\")
Image by Author<\/figcaption><\/figure>\nKSTest is used to compare the continuous columns, and CSTest compares the discrete columns. Both tests result in a normalized score between 0 to 1, with the target is to maximize the score. From the result above, we can assess that the discrete sample columns are good (almost similar to the real data). In contrast, continuous columns might have a deviation in distribution. If you want to know all the evaluation methods available in SDV, refer to the documentation page<\/a>.<\/p>\n
\n3. Gretel<\/h2>\n
Gretel<\/a> or Gretel Synthetics is an open-source Python package based on Recurrent Neural Network (RNN) to generate structured and unstructured data. The python package approach treats the dataset as text data and trains the model based on this text data. The model would then produce synthetic data with text data (we need to transform the data to our intended result).<\/p>\nGretel<\/a> required a little bit of heavy computational power because it is based on the RNN, so I recommend using free google colab notebook or Kaggle notebook if your computer is not powerful enough. For accessibility purposes, this article would also refer to the tutorial provided by Gretel.<\/p>\nThe first thing we need to do is install the package using the following code.<\/p>\n
pip install gretel-synthetics<\/code><\/pre>\nWe would then use the following code to generate a config code as a parameter to train the RNN model. The following parameter is based on the training on GPU, and the dataset used is the scooter journey coordinates dataset<\/a> available from Gretel.<\/p>\nfrom pathlib import Path<\/code><\/pre>\nfrom gretel_synthetics.config import LocalConfig<\/code><\/pre>\n# Create a config for both training and generating data<\/code><\/pre>\nconfig = LocalConfig(<\/code><\/pre>\n# the max line length for input training\nmax_line_len=2048, <\/code><\/pre>\n# tokenizer model vocabulary\ndatavocab_size=20000, <\/code><\/pre>\n# specify if the training text is structured, else ``None``\nsizefield_delimiter=",", <\/code><\/pre>\n# overwrite previously trained model checkpoints\noverwrite=True,<\/code><\/pre>\n# Checkpoint location\ncheckpoint_dir=(Path.cwd() \/ 'checkpoints').as_posix(),<\/code><\/pre>\n#The dataset used for RNN training\ninput_data_path="https:\/\/gretel-public-website.s3-us-west-2.amazonaws.com\/datasets\/uber_scooter_rides_1day.csv" # filepath or S3)<\/code><\/pre>\nWe will train our RNN model using the following code when the config is ready.<\/p>\n
from gretel_synthetics.train import train_rnn<\/code><\/pre>\ntrain_rnn(config)<\/code><\/pre>\nDepending on your processing power, the config, and the dataset, the process might take some time. When it’s done, the config would automatically save your best model and be ready to generate synthetic data. Let’s try to generate the data using generate_text<\/code>.<\/p>\nfrom gretel_synthetics.generate import generate_text<\/code><\/pre>\n#Simple validation function of the data is containing 6 data parts or not. You could always free to tweak it.\ndef validate_record(line):<\/code><\/pre>\n rec = line.split(", ")\n if len(rec) == 6:\n float(rec[5])\n float(rec[4])\n float(rec[3])\n float(rec[2])\n int(rec[0])\n else:\n raise Exception('record not 6 parts')<\/code><\/pre>\n#Generate 1000 synthetic data\ndata = generate_text(config, line_validator=validate_record, num_lines=1000)<\/code><\/pre>\nprint(data)<\/code><\/pre>\n![\"Image](\"https:\/\/towardsdatascience.com\/wp-content\/uploads\/2022\/01\/17iw2D03-2WUc386LjojCVA.png\")
Image by Author<\/figcaption><\/figure>\nThe generated data, by default, is a generator object that contains all the synthetic data. We could try to iterate the object values and print the result to access the data.<\/p>\n
for line in data:\n print(line)<\/code><\/pre>\nLet’s take a look closely at the generated synthetic data.<\/p>\n
line<\/code><\/pre>\n![\"Image](\"https:\/\/towardsdatascience.com\/wp-content\/uploads\/2022\/01\/16y_2Ky82qbbpZYxOTA_peg.png\")
Image by Author<\/figcaption><\/figure>\nThe above image is the synthetic data example from the Gretel RNN model. The text parameter is our data is in the form of text data. If we want to access it, we could use the .text<\/code> attribute.<\/p>\nprint(line.text)<\/code><\/pre>\n![\"Image](\"https:\/\/towardsdatascience.com\/wp-content\/uploads\/2022\/01\/1H7m2dZfrBFr15QaoZnXLPQ.png\")
Image by Author<\/figcaption><\/figure>\nHence we only need to split the data by the delimiter (‘, ‘) and process it to the tabular form. Although, not all the data might be valid (depending on your evaluation) and need to be cleaned thoroughly before using it.<\/p>\n![\"Image](\"https:\/\/towardsdatascience.com\/wp-content\/uploads\/2022\/01\/1Je4KzcBZDpfRcgI793bqRg.png\")
Image by Author<\/figcaption><\/figure>\nThe data above are not valid because they produce more than six parts of data while we only need 6. This is why we need to be careful of the data produced by the RNN model. However, the useability outweighs the cons, so a little work should be fine if you use Gretel.<\/p>\n
Get ready to learn data science from all the experts with discounted prices on 365 Data Science!<\/strong><\/p>\nGet ready for a Data Science Summer – 65% Off | 365 Data Science<\/strong><\/a><\/p><\/blockquote>\n
\nConclusion<\/strong><\/h2>\nData is the backbone of any data project, but sometimes the data we want is not available or hard to meet our requirements. That is why we could use the Python package to generate synthetic data. This article explains 3 top Python packages for generating data, they are:<\/p>\n
\n- Faker<\/li>\n
- SDV<\/li>\n
- Gretel<\/li>\n<\/ol>\n
I hope it helps!<\/p>\n
\nVisit me on my Social Media.<\/a><\/strong><\/p>\n