Documentation

Nodes reference.

Every Odara ETL node: files, databases, data warehouses, object storage, network and APIs, notifications, transforms, control flow and local system nodes.

4. Nodes Reference

Odara ETL 0.4 ships with 70 node types: 66 in the editor's Nodes palette, the Mapper (added from the canvas) and 3 Maestro steps, grouped into four categories: ingest (sources), transforms, deliver (targets), and orchestrate (control flow). Each node has a dedicated configuration schema shown in the Properties panel.

4.1 Files

4.1.1 CSV Source SOURCE

CSV Source properties panel
CSV Source configuration — file path, header, delimiter, row limit, columns

Reads data from a CSV (Comma-Separated Values) file into RecordBatches. Supports custom delimiters, headers, row skipping, and column selection.

PropertyDescriptionDefault
File PathAbsolute or relative path to the CSV file. Supports variable interpolation (@date).—
First row is headerWhether the first row contains column names.true
DelimiterColumn separator: comma, semicolon, tab, pipe.,
Row LimitMaximum number of rows to read. Empty = unlimited.—
ColumnsSpecific columns to select with per-column type hints. Empty = all.All
Use File SchemaOptional — apply a saved File Schema resource for consistent column types.—
Tip Variable interpolation works in file paths. Use @date, @env, or any context variable to read dated / per-environment files: C:\data\[email protected]

4.1.2 Excel Source SOURCE

Reads data from a .xlsx / .xls file. Supports sheet selection, header row position, row skipping, and column filtering.

PropertyDescriptionDefault
File PathPath to the workbook.—
Sheet NameName of the sheet to read. Empty = first sheet.First sheet
Header Row1-based row number containing column headers.1
Skip RowsExtra rows to skip after the header.0

4.1.3 XML Source SOURCE

Flattens hierarchical XML into tabular rows by specifying the repeating row element path.

PropertyDescriptionDefault
File PathPath to the XML file.—
Row ElementName of the repeating element (e.g., order for <orders><order>…).—
NamespacesOptional XML namespace declarations.—

4.1.4 Magic File Source SOURCE

Reads a file by auto-detecting its format from the content (magic bytes + a sampled head), not the extension — so it works even when the file has no extension at all. Supports CSV, JSON, NDJSON, Parquet, and Excel. Use this when the format may vary or when you want minimal configuration. (Tip: drop an @file.path variable here inside an Iterate loop to read every file in a folder, whatever its format.)

PropertyDescriptionDefault
File PathPath to the file.—
Sample SizeRows to sample for schema inference.1000

4.1.5 CSV Target TARGET

Writes RecordBatches to a CSV file. See Properties in the editor — mirrors the source options plus Write mode (overwrite / append) and Include BOM.

4.1.6 Excel Target TARGET

Writes to a .xlsx file with configurable sheet name and header row.

4.1.7 XML Target TARGET

Writes rows as XML elements under a specified root and row element name.

4.1.8 Magic File Target TARGET

Picks the output format from the file extension (or set a Format Override). Supports CSV, JSON, NDJSON, Parquet and Excel, plus an Append option. Note: writing to a .json path produces JSON Lines (NDJSON) — one object per line — not a single JSON array; that's the format BigQuery, DuckDB and jq ingest natively. If you need a standalone array, post-process with jq -s '.' out.json or use a .ndjson extension to make the intent explicit.

4.2 Databases

All database source nodes share a common pattern: configure the connection inline or pick a saved Data Store (Section 5.2), then write the SQL query (or set collection + filter for MongoDB). Database targets write RecordBatches back to a table.

4.2.1 PostgreSQL Source SOURCE

PostgreSQL Source properties
PostgreSQL Source — connection + SQL query

Connects to a PostgreSQL database and executes a SQL query. All PostgreSQL data types map automatically to Arrow equivalents. SSL modes supported: disable, prefer, require, verify-ca, verify-full.

4.2.2 MySQL Source SOURCE

MySQL Source properties
MySQL Source — supports MySQL 5.7+ and MariaDB

Connects to a MySQL or MariaDB database. Port defaults to 3306. All MySQL data types are mapped to their Arrow equivalents.

4.2.3 Oracle Source SOURCE

Connects to Oracle Database via the sibyl driver. Requires the Oracle Instant Client installed on the machine. Supports service names, SIDs, and TNS names.

4.2.4 SQL Server Source SOURCE

Connects to Microsoft SQL Server via the TDS protocol (tiberius crate). Supports named instances (e.g., SQLEXPRESS), Windows integrated auth (Trusted Connection), and certificate trust override for dev environments.

4.2.5 MongoDB Source SOURCE

Reads from a MongoDB collection. Supports aggregation pipelines, filters, and field projections. Documents are flattened into rows — nested objects can be extracted via dot notation or kept as JSON strings.

4.2.6 ODBC Source SOURCE

Generic ODBC connector — use when you need a database not covered by the dedicated connectors. Requires the appropriate ODBC driver installed (SQL Server, DB2, Informix, Sybase, etc.).

4.2.7 SQLite Source SOURCE

Reads from a local SQLite database file. No server connection — just a path to the .db / .sqlite file.

4.2.8 IBM DB2 VIA ODBC

There is no dedicated DB2 node in the palette. Reach IBM DB2 with the ODBC Read / ODBC Write nodes and IBM's DB2 ODBC driver installed on the machine that runs the pipeline.

Database targets PostgreSQL, MySQL, Oracle, SQL Server, MongoDB, ODBC and SQLite each have a corresponding Write node. All accept a connection + target table with a Table Operation (none / truncate / create / drop & create) and a Data Operation (insert / upsert / replace).

4.3 Data Warehouses

Data warehouse connectors are tuned for analytical workloads: bulk reads, column projection pushdown, and batch inserts.

4.3.1 Snowflake Source SOURCE

Connects to Snowflake with account + warehouse + database + schema + user/password. Supports role impersonation via the role field. SQL queries run on the specified warehouse.

4.3.2 ClickHouse Source SOURCE

Reads from a ClickHouse cluster via the HTTP interface. Provide the URL (including port 8123 or 8443 for TLS), database, user, and password.

4.3.3 BigQuery Source SOURCE

Reads from Google BigQuery. Two modes: Table (direct read via Storage Read API) and Query (SQL query, optionally with a temp dataset for results). Authentication uses a service-account key: Credentials holds either the path of the key file on the machine that runs the pipeline, or the key JSON itself — normally a secret variable such as @{GCP_KEY}, which also works in isolated runs (15.5).

4.3.4 Redshift Source SOURCE

Amazon Redshift via the PostgreSQL wire protocol + TLS. Works with classic Redshift clusters and Redshift Serverless endpoints. DirectWire reads small/medium result sets directly; UnloadS3 streams large reads via an S3 UNLOAD. Known limitation: the DirectWire reader returns NULL for DECIMAL/NUMERIC columns (including SUM(decimal)); cast to a text type — CAST(col AS VARCHAR) — or use UnloadS3 if you need exact decimals. Integers, doubles, dates and timestamps read correctly.

4.3.5 DuckDB Source SOURCE

Embedded analytical database — DuckDB. Just provide a path to the .duckdb file and a SQL query. Zero-configuration for local analytics.

4.3.6 Snowflake Target TARGET

Writes to Snowflake. Table Operation (None / Truncate / Create / Drop & Create) controls DDL; Data Operation chooses Insert vs Copy Into (stage + COPY INTO, dramatically faster on real volumes).

4.3.7 ClickHouse Target TARGET

Writes to ClickHouse over HTTP(S). Configurable table operation, batched inserts (bulk_size), optional async_insert, and an optional cluster for distributed tables.

4.3.8 BigQuery Target TARGET

Writes to BigQuery with a load job whose rows are uploaded with the request: no Cloud Storage bucket is needed, load jobs are free, and a table just recreated by Drop & Create receives its rows at once. The Storage Write API option is not available yet: it uses a load job for a new table and streaming inserts into an existing one. Supports partition/cluster fields and Insert / Replace; Upsert is refused (BigQuery has no key columns in this node). Credentials as for BigQuery Source.

4.3.9 Redshift Target TARGET

Writes to Redshift two ways: DirectWire (multi-row INSERT, fine for dev/small loads) or CopyS3 (stage Parquet in S3, then COPY — recommended for production volumes). Set DISTKEY/DISTSTYLE/SORTKEY, a table operation, and Insert / Upsert / Replace, plus optional pre/post SQL (e.g. ANALYZE). DirectWire Upsert stages into a temp table and merges by your sort keys.

4.3.10 DuckDB Target TARGET

Writes to a local .duckdb file. Configurable table operation and data operation, optional use_appender (DuckDB's fast bulk path) and threads. Great for shipping a result mart as a single portable file.

4.8 Transforms

4.8.1 SQL Transform TRANSFORM

SQL Transform with Monaco editor
SQL Transform — DataFusion-compatible SQL with syntax highlighting and autocomplete

Executes a SQL query against the input RecordBatch(es) using Apache DataFusion. Reference input tables by their node name or use input for a single upstream.

PropertyDescriptionDefault
QueryAny DataFusion-compatible SQL statement.SELECT * FROM input

Example — filter and enrich:

SELECT order_id, customer_name, amount,
       CASE WHEN amount > 1000 THEN 'high' ELSE 'standard' END AS tier
FROM input
WHERE status = 'completed'
DataFusion dialect DataFusion supports most of ANSI SQL: CTEs, window functions, aggregates, joins (inner/left/right/full/semi/anti), string / date / numeric / regex functions. See Appendix 15.3 for the function reference.

4.8.2 Python Transform TRANSFORM

Python Transform with Monaco editor
Python Transform — runs in an isolated subprocess, data exchanged via PyArrow

Runs Python code in an isolated subprocess. Data is passed in and out as pandas.DataFrame via PyArrow IPC (Arrow format, no row or text conversion).

PropertyDescriptionDefault
CodePython function transform(input_df) returning a DataFrame.—
Python DependenciesOptional pip package list; installed on first run.—
import pandas as pd

def transform(input_df):
    df = input_df.copy()
    df['risk_score'] = df['amount'].apply(
        lambda x: 'high' if x > 5000 else 'low'
    )
    df['processed_at'] = pd.Timestamp.utcnow()
    return df
Subprocess isolation Each execution spawns a fresh Python process, so state does not persist between runs. Use Context Variables (Section 5.7) if you need to share values between nodes.

4.8.3 Mapper TRANSFORM

A Talend-tMap-style transform with N inputs and M outputs: joins between the inputs, an expression for every output column, and a filter per output. Add it from the canvas (type on the canvas, or use the + button of a selected node); double-click it to open the Mapper editor.

4.4 Object Storage

Odara treats S3, Azure Blob, and Google Drive as first-class object stores with a consistent pattern: List enumerates objects, Read/Download fetches them, Write/Upload uploads, Copy duplicates within or across containers, Delete removes them.

4.4.1 S3 List SOURCE

Lists objects under a bucket + prefix. Optionally fetches metadata (size, last modified, ETag). Output is one row per object, with columns bucket, key, and optional metadata fields.

4.4.2 S3 Read SOURCE

Downloads a single S3 object to local_path. Can be used in batch mode by setting key_source to From column — then reads one object per input row (typical pipeline: S3 List → S3 Read).

4.4.3 S3 Write TARGET

Uploads a local file to S3. Supports multipart uploads for large files. Same batch mode as Read.

4.4.4 S3 Copy TARGET

Server-side copy between S3 keys (no download/re-upload). Efficient for large objects.

4.4.5 S3 Delete TARGET

Removes an S3 object. Batch mode iterates input rows.

4.4.6 Azure Blob List SOURCE

Enumerates blobs in an Azure Storage container. Authentication modes: account key, SAS token, or connection string.

4.4.7 Azure Blob Read SOURCE

Downloads a blob to local disk. Same batch pattern as S3.

4.4.8 Azure Blob Write TARGET

Uploads to Azure Blob Storage.

4.4.9 Azure Blob Copy TARGET

Server-side blob copy within or across containers.

4.4.10 Azure Blob Delete TARGET

Removes a blob.

4.5 Productivity

Google Drive nodes authenticate in one of two ways:

  • OAuth (a person's account): client_id, client_secret and refresh_token — Odara refreshes the access token automatically. Keep the secret and the refresh token in secret variables.
  • Service account: Credentials holds the path of the key file, or the key JSON itself through a secret variable such as @{GCP_KEY} (works in isolated runs).
Service accounts and personal Drives A service account has no Drive storage of its own: Google refuses to let it create files (Write, Copy) in a folder of a personal Drive ("Service Accounts do not have storage quota"). It can list and read what is shared with it. To create files, use OAuth credentials, or a Shared Drive (Google Workspace) with the service account as a member.

4.5.1 Google Drive List SOURCE

Lists files in a Drive folder by folder_id.

4.5.2 Google Drive Read SOURCE

Downloads a file by file_id. Batch mode supported.

4.5.3 Google Drive Write TARGET

Uploads a local file to a Drive folder.

4.5.4 Google Drive Copy TARGET

Copies a file within Drive (no re-download).

4.5.5 Google Drive Delete TARGET

Moves a file to trash.

Getting a refresh token Open any Google Drive resource in Metadata → click Authorize with Google → complete the OAuth flow → Odara stores the refresh token encrypted.

4.6 Network & APIs

4.6.1 FTP List SOURCE

Enumerates files on an FTP / FTPS / SFTP server. Supports password and SSH key authentication. Recursive listing optional.

4.6.2 FTP Read SOURCE

Downloads a file. Batch mode: one input row = one file fetched.

4.6.3 FTP Write TARGET

Uploads a local file. Can create intermediate directories on the remote.

4.6.4 FTP Rename TARGET

Renames (or moves) a remote file.

4.6.5 FTP Delete TARGET

Removes a remote file.

4.6.6 REST API (GET) SOURCE

REST Source properties
REST Source — auth, pagination, headers, query params

Fetches JSON from a REST endpoint. Built-in pagination strategies: page_number, offset, cursor, link_header, next_url. Authentication: none, basic, bearer, API key (header / query), OAuth2.

4.6.7 REST API (POST) TARGET

Sends rows to a REST endpoint. Batching: per row (one request per row) or array (N rows in one request body). Template-based JSON body with variable interpolation.

4.7 Notifications

4.7.1 Email TARGET

Email Target properties
Email Target — SMTP configuration with provider presets

Sends email via SMTP. Provider presets: Gmail, Outlook, SendGrid, AWS SES, or custom. Security modes: none, STARTTLS, TLS. Supports per-row recipients (To column), attachments from file paths or input columns, and variable interpolation in subject/body.

Batch modes:

  • per_row — one personalized email per input row (bulk marketing, per-user alerts).
  • summary — one email with all rows aggregated (daily digest, reports).

4.9 Control Flow

4.9.1 Sleep CONTROL

Pauses the pipeline for a configurable number of seconds. Useful for rate-limiting API calls or waiting for external processes.

4.9.2 Run After CONTROL

Synchronization point — all upstream branches must finish before downstream nodes start. Use to join parallel paths.

4.9.3 Iterate CONTROL

Runs the downstream subgraph once per input row. Connect a source to Iterate with a dashed iterate-edge (a distinct edge kind that feeds one row at a time); everything after Iterate is a normal data graph that re-runs per row. Each row's columns become @<prefix>.<column> variables (set the Variable Prefix, e.g. file → @file.path, @file.name, @file.stem) that downstream node configs and SQL/paths can interpolate. Choose Sequential or Parallel execution and an On Error policy (skip failed and continue loop vs. abort). Ideal for looping over a folder of files, a list of IDs, or API endpoints. (Don't combine Parallel with an appending target.)

4.9.4 Fail CONTROL

Stops the run with an error message and exit code (Talend's tDie). It runs only when reached, so it usually sits behind a trigger with a condition; with Only if rows it fails only when rows reach it.

4.9.5 Run External Job CONTROL

Runs an exported Talend job (<Job>_run.sh / .bat) or any script from a folder allowed by the server's ODARAX_EXTERNAL_JOB_DIRS setting. For Talend it adds --context and --context_param arguments. Options: extra arguments, working folder, environment variables, a timeout (the whole process tree is stopped) and the exit codes that count as success. It outputs one row with the exit code, status, timing and the last log lines. It needs the server's disk, so it can't be part of an isolated run.

4.10 Local System

4.10.1 File List SOURCE

Lists files in a local directory. The optional Pattern is a real glob — *, ?, […] and prefix*suffix all work, and matching is case-insensitive (so *.CSV matches data.csv). Pattern filters files only, not directories; toggle Include files / Include directories separately, and Recursive to descend. Returns one row per entry with columns name, path, size, is_dir, extension, modified_time, and stem (the filename without extension — handy as @file.stem in an Iterate loop).

4.10.2 Folder Manage TARGET

Creates, deletes, or moves a folder. Can be recursive.

4.10.3 File Manage TARGET

Creates, deletes, moves, or copies a file. Batch mode iterates input rows.

4.11 Debug

4.11.1 Console TARGET

Prints the first N rows to the Console log panel. Use during development to inspect intermediate results without writing to a real target.

PropertyDescriptionDefault
LimitMaximum rows to print.100