The route

Python transform with pandas

Download the exercise files

SQL covers most transforms. Some are simply easier in Python — statistics, text wrangling, anything with a library behind it. The Python node hands you the incoming rows as a pandas DataFrame and takes a DataFrame back. This level uses it to find suspicious readings in 50,000 sensor measurements.

Goal

A Python node that flags every temperature reading more than 3 standard deviations from its own sensor's mean — 89 of them — and passes only those on.

You'll need

  • Python 3 with pip on the machine where Odara runs. Odara uses that Python; it doesn't ship its own. Check with python3 --version (Linux) or py --version (Windows).
  • The exercise files for this level: sensor_readings.csv — 20 sensors × 2,500 readings, every 10 minutes, with a daily temperature cycle and a few spikes planted in it.
  • About 25 minutes.

Steps

1. Source and Python node

Create a pipeline w3-5-python. Add a CSV Source readings for the file, then a Python node from TRANSFORM ▸ Data Processing, named sensor_outliers, and connect them.

Click it. Under the code editor there's a Python dependencies field: packages listed there, one per line, are installed with pip the first time the node runs. Put these two in it:

pandas
pyarrow

pyarrow is how Odara hands the data to Python and back; pandas is what you'll write in. The first run takes a little longer while pip installs them; later runs don't.

The Python node: code editor and dependencies

2. The contract

  • The incoming rows arrive as input_df, a pandas DataFrame, with the types the source inferred: ts is already a datetime, temperature_c a float.
  • Whatever you put in output_df is what the node passes on.

Replace the code with:

import pandas as pd

df = input_df.copy()

# Each sensor has its own "normal", so compare every reading to its own sensor.
by_sensor = df.groupby("sensor_id")["temperature_c"]
df["mean_c"] = by_sensor.transform("mean")
df["std_c"] = by_sensor.transform("std")
df["z"] = (df["temperature_c"] - df["mean_c"]) / df["std_c"]

# Suspicious: more than 3 standard deviations from the sensor's mean.
outliers = df[df["z"].abs() > 3].sort_values(["sensor_id", "ts"])

output_df = (
    outliers[["sensor_id", "ts", "temperature_c", "mean_c", "z"]]
    .round({"mean_c": 2, "z": 2})
    .reset_index(drop=True)
)

The z-score is how many standard deviations away from normal a value is. Computing it per sensor matters: S-20 averages 30.0 °C and S-01 18.6 °C, so against one global average a spike on a cool sensor still looks normal. Computed globally, the same rule finds only 51 of the 89 spikes.

3. Execute

Add a Console after the Python node with Row Limit 20, Execute, and expand the output.

The flagged readings, five clean columns

Expected result

  • The log reads readings: 50000 rows, then sensor_outliers: 89 rows.
  • Five columns: sensor_id, ts, temperature_c, mean_c, z — no index column.
  • The first two are both on S-01: 5.44 °C (z = −5.87) and 35.35 °C (z = 7.45), against a sensor mean of 18.63 °C.

Common pitfalls

Python not available at 'python3'. Install python3 and pyarrow to use Python transforms. There's no Python on the machine running Odara, or it isn't on the PATH of the process that started Odara. Install it, then restart Odara.

ModuleNotFoundError: No module named 'pandas'. The package isn't installed for that Python. List it under Python dependencies, or install it yourself with python3 -m pip install pandas pyarrow.

A column you didn't expect, named after your groupby key. Odara drops a DataFrame's unnamed index — the old row numbers left over from filtering — but keeps a named one as a column: after groupby("sensor_id").mean(), sensor_id lives in the index and comes out as a column. That's usually what you want; reset_index(drop=True) drops it.

KeyError: 'sensor' (or any column name). Pandas errors come back as the full Python traceback in the Console — read the last line; it names the column you misspelled.

Everything is flagged, or nothing is. The statistics are global instead of per sensor: groupby("sensor_id") is missing, or transform was replaced with agg, which collapses the rows instead of broadcasting the value back to each one.

What you built

A transform with a whole ecosystem behind it. The same node can call scikit-learn, parse awkward text with regular expressions, or hit a library you already use — list it under dependencies and import it.

Check yourself

One question, drawn from a larger bank. Get it right to pass.

  1. `transform('mean')` vs `agg('mean')` inside the groupby — why transform?