Documentation

Integration and pipeline execution

Where Axolotl runs

Axolotl is a no-code batch data-quality product. You run it in the application at https://app.adequata.ai: connect storage you already use, choose CSV or Parquet paths, and select which stages run. After each stage you review a sample, and only output you accept is written to a destination path you control. There is no customer API or command-line interface for submitting jobs.

  • Amazon S3
  • Google Cloud Storage
  • Azure Blob Storage
  • Snowflake, through files on the external stage’s underlying bucket
  • Databricks, through the S3, Azure Blob, or Google Cloud Storage location where the files already live
  • Not supported: SQL warehouses (Redshift, BigQuery, Synapse SQL, PostgreSQL), spreadsheets, JSON, Avro, and ORC

How a job is placed

Axolotl’s application orchestrates the job. Short-lived workers read from your storage and write accepted output back there. Your dataset is not kept as an ADEQUATA table.

How Axolotl sits between your storage and a short-lived processing job.

Supported file inputs

A run accepts one file type: a folder or a single file of CSV, or a folder or a single file of Parquet. Mixed formats in one folder are not accepted.

InputSupportedNotes
Folder of .csvYesDirectory prefix with or without trailing slash
Single .csv fileYesIn-place output mode not available for single files
Folder of .parquetYesSame path rules as CSV
Single .parquet fileYes
Mixed CSV and Parquet in one folderNoOne file type per run
Excel, JSON, Avro, ORCNo

Schema preflight (before billing)

Before a run is allowed to start, Axolotl reads file headers and Parquet metadata, normalizes column names, and builds one schema for every file in the path you selected. If columns disagree on type, or if several columns appear to describe the same thing (for example more than one date field), you resolve that in the interface before continuing. This step also measures physical size, estimated logical size, and estimated memory. Those figures are required both to start the run and to calculate the Data-Qualifying Units (DQUs) you will be asked to confirm.

Storage connectors

Each connector reads and writes objects in storage you own. Snowflake and Databricks are reached through those files, not through a SQL warehouse or the Databricks Jobs API.

ConnectorConfigurationData plane
AWS S3Access keys or IAM role (CloudFormation template in UI)Objects in your bucket
GCSService account JSONObjects in your bucket
Azure BlobAccount key or service principal (ARM template optional)Blobs in your container
SnowflakeService user (PAT or key pair) + stage bucket credentialsFiles on stage storage; long jobs do not hold a Snowflake session
DatabricksUnderlying S3, Azure, or GCS as aboveFiles in object storage—not Databricks SQL/Jobs API

Batch stages, in order

The pipeline always follows the same order. You choose where it should start and where it should stop, so you can run a smaller portion of the work without rearranging the stages themselves.

The ordered batch pipeline. You may run a consecutive subset of these stages.
  • Stage 0 is schema validation. Axolotl aligns the columns and measures the dataset. This step is required before anything else can run, but it is not one of the billed pipeline stage indexes.
  • Stage 1 covers the basic checks: values that are not valid for their column, rows that are empty or too sparse to keep, and optional deduplication. Deduplication can match rows exactly or with a fuzzy comparison.
  • Stage 2 looks for anomalies. You can use column methods such as IQR and Z-score, dataset methods such as K-Means, DBSCAN, One-Class SVM, and Isolation Forest, or a combination. Each method can be set to low or high sensitivity.
  • Stage 3 fills gaps with synthetic values. You can leave the default settings in place, or allow Axolotl to tune hyperparameters before it imputes.
  • After every stage that actually runs, you review a sample in the application. You accept the result to continue, or you decline it before anything is written back.
  • Your original files stay where they are. Axolotl never edits the input path. Output you accept is written only to the destination path you configured.

The clean dataset comes back in a new folder

You tell Axolotl one output folder. Treat that folder as a drop box. Axolotl does not leave the clean dataset as loose files sitting directly in it. Each run adds one new folder inside the drop box. The clean dataset is inside that new folder. The new folder’s name is a long id that Axolotl makes up. You never choose that name, and you do not look it up in the app.

Suppose the output folder you saved is s3://acme-data/cache/orders/. Monday’s run comes back under s3://acme-data/cache/orders/1f2e3d4c-aaaa-bbbb-cccc-111111111111/. Thursday’s run comes back under s3://acme-data/cache/orders/9a8b7c6d-dddd-eeee-ffff-222222222222/. Monday’s folder stays where it is. Thursday’s folder sits next to it, not inside it. A schedule uses this same drop box every time and still makes a new id folder on every run. A one-off run from the app does the same. If you pointed the output at one single .csv or .parquet file instead of a folder, there is no extra id folder.

Inside the new id folder, each stage that ran writes its own dataset folder, in a directory named for that stage. Your next job uses the dataset folder for the stage you care about. Configure that job for that stage directory. A folder’s write time is not a reliable way to choose among them, because more than one dataset folder can finish under the same id.

How you pick up that dataset without knowing the folder name

The next system in your company needs a path to the clean files. A path saved from an earlier run will miss the next one, because the id in the middle is new every time. The handoff is a rule on the output folder you already chose. Your cloud delivers the new path when the files arrive. No one copies an id out of Axolotl.

Put a rule on the output folder: when a new file appears under cache/orders/, start the next job and hand it the full path of that file. On Amazon S3 the rule is EventBridge, or an S3 event notification, with the prefix cache/orders/. On Google Cloud Storage it is an object-finalize notification on that prefix. On Azure Blob it is an Event Grid subscription for blob created on that prefix. Point the rule at the function, queue, or workflow that should run next.

When the files land, your cloud sends one message. The message already contains the whole path, id included. A Thursday arrival looks like cache/orders/9a8b7c6d-dddd-eeee-ffff-222222222222/ and then the stage directory and dataset folder under it, for example cache/orders/9a8b7c6d-dddd-eeee-ffff-222222222222/synthetic_data_generation_done/dataset/part-000.parquet. Your job reads that path from the message and opens the stage directory it is configured for. A minute earlier it did not know the id. The message delivered it. Friday’s run sends a new message with a new id, and the same job follows it. If several stage directories appear under that id, the job keeps using the stage you selected.

A warehouse task or other scheduled job can do the same thing without an event. It lists the id folders directly under cache/orders/, takes the one with the latest write time, and reads the dataset folder for the stage that job is set up to use. The newest id folder under the output path is the run that just finished. The stage inside it is the one you chose.

Every run stays in the output folder. Axolotl does not remove older id folders. Leave the full set in place when you want each run available for audit or for a later comparison. If you would rather keep only recent runs, configure a lifecycle rule on that output folder in your storage account and let it expire objects on the schedule you choose. Removing older id folders and keeping the latest is also available when that matches your own policy. None of this is required for the next job to find the clean dataset. The arrival rule, or the newest id folder, still does that.

Give each schedule its own output folder, such as cache/orders/ and cache/invoices/. A new folder under cache/orders/ is then always that schedule’s dataset. Axolotl does not call your next job. There is no API to ask for the path. The new file in that output folder is the signal.

Schedules

A schedule is saved in the app on the same screen where you configure a run. It remembers the input folder, its own output folder, which stages to run, and the options. It does not remember an id. Every time it runs, a new id folder appears in that output folder, with the clean dataset inside it. The rule on that folder, or the latest-folder check from the previous section, picks that folder up. You set this up once. After that, every later run follows the same path.

You can start the schedule on a clock: every day, every week, or every month, at a time and timezone you choose. Or you can start it when the input folder changes, because a file was created, modified, or deleted. Those are two different rules from the one on the output folder. The rule on the input folder starts Axolotl. The rule on the output folder starts whatever you run after the clean data is back.

On Amazon S3, the input rule fires as soon as a file changes only if that input bucket has “Send notifications to Amazon EventBridge” turned on. In the AWS console that switch is under S3, your bucket, Properties, Amazon EventBridge. If it is off, an S3 input schedule does not start. Google Cloud Storage and Azure Blob inputs are checked on a timer instead. If there is not enough DQU balance when a run is due, that run is skipped and you are not charged. The schedule stays on for the next time. If the columns in the input change in a way the saved setup cannot resolve, the schedule waits until you resolve that in the app.

Run limits

A run does not start, and DQUs are not consumed, when estimated memory is above about 80 GB. There is no separate cap on row count or file count.

TopicBehavior
Estimated RAM capRuns blocked above ~80 GB estimated RAM (no tasks, no DQU charge). UI also warns above 96 GB—treat 80 GB as the hard stop.
Max rows / file countNo separate cap; gated by RAM estimate
Minimum billed logical size0.01 GB floor per run
Long jobsAWS assume-role paths copy input into memory on the job host and refresh credentials for write-back. A job runs for up to 12 hours.
Session idleApp login may idle out (~1 hour); in-flight batch jobs continue
Parallel runsStart separate runs at the same time. There is no limit on how many run in parallel.
MaskingHandled by the customer, according to their own policy and preference.
Developer toolsNo public job submission API or CLI