Data cleaning

Real-world files arrive messy — inconsistent labels, duplicate rows, cryptic columns. Data cleaning adds a Standardisation step to the upload wizard that fixes those problems before the dataset exists, with AI doing the drafting and you approving every change.

Turn it on#

Cleaning is a per-project choice: switch on Enable Data Cleaning when creating a project, or later in the project's settings. The toggle notes the trade-off — "Enables data cleaning and standardisation during dataset upload. Maximum upload size: 800 MB."

With cleaning on, the wizard's steps become Upload Dataset → Standardisation → AI Readiness → Confirmation.

Review the inferred schema#

When your file lands, GoPie profiles it in a sandbox and streams its progress: classifying columns, searching dictionaries, inferring enum values, assigning semantic groups. The result is an inferred schema table — each column with its database type, a semantic type and role, and any tagged dictionary.

Two ways to shape it:

• Edit in place — adjust what the inference got wrong.
• Regenerate Schema — describe changes in plain language: "treat state_name as geographical", "mark population as a measure", "remove the id column". The AI applies your instructions and produces a new schema version.

When it looks right, select Accept Schema — that locks the schema the cleaning plan will be built against.

Note

Dictionaries — reference tables of coded values and their canonical labels, managed on the project's Dictionaries tab — are what the inference matches columns against. A matched dictionary standardises that column's values and can add enrichment columns.

Answer the quick questionnaire#

Before planning, two yes/no questions:

• Remove duplicate columns? — columns with identical names are deduplicated, keeping the first.
• Remove rows where all values are null? — entirely-empty rows are dropped.

Approve the plan#

GoPie then validates the data against the accepted schema and returns a plan — every proposed change, grouped and reviewable:

• Automatic fixes — safe corrections, pre-approved.
• Duplicate rows — with a strategy choice: keep first, keep last, or drop all duplicates.
• Assisted fixes — value standardisations the AI is less sure about ("Karntaka" → "Karnataka"), each with a confidence score and per-value Approve / Reject / Edit controls.
• Column order — a suggested ordering you can drag to adjust.
• Enrichment — optional derived columns from a matched dictionary, with the coverage it would achieve.
• Blocked fixes — items waiting on one of your decisions.

Changing decisions? Select Update plan to re-validate. When the plan reads "Plan converged — ready to apply.", select Apply fixes. The fixes run in the sandbox and a before/after diff shows exactly what changed per column; Confirm & Continue carries the cleaned data forward.

From there the wizard continues as normal — AI Readiness, then Confirmation. The dataset is created from the cleaned data; your original file isn't modified.

Get the cleaning report#

On the success screen, Download Metadata produces a report (PDF or JSON) of the finished dataset — overview (rows, columns, inferred time period, units, granularity), per-column details, and the dictionaries used during cleaning. It doubles as a data-dictionary you can hand to consumers of the dataset.

Mind the limits#

• 800 MB per file in cleaning-enabled projects (the standard cap applies elsewhere).
• A cleaning session stays live for 72 hours — leave the wizard for longer and you'll need to start the upload again.
• Cleaning runs in an isolated sandbox on the server; if a session fails mid-plan, retry from the wizard — your file and decisions up to the accepted schema are preserved in the session.