The route

Write any format with Magic File

Download the exercise files

Magic File also works as a target: give it an output path and the extension decides the format. You'll write the same 20,000 events to four formats in one run, then compare what landed on disk — not just the sizes, but what each format keeps and what it throws away.

Goal

One pipeline that reads events.ndjson once and writes it as CSV, JSON, Parquet and Excel, and a short comparison of the four files.

You'll need

  • The mixed/ folder from level 2.4 — this level reuses events.ndjson.
  • A folder Odara can write to. The screenshots use /tmp/odara-academy/out/.
  • About 15 minutes.

Steps

1. One source, four targets

Create a pipeline w2-5-magic-write. Drop a Magic File source (INGEST ▸ Files) named events, pointing at mixed/events.ndjson.

Now drop four Magic File targets from DELIVER ▸ Files and connect the source to each of them. Name them and set their Output Path:

Node Output Path
events_csv …/out/events.csv
events_json …/out/events.json
events_parquet …/out/events.parquet
events_xlsx …/out/events.xlsx

One source fanning out to four Magic File targets

A source can feed any number of nodes; each gets the full set of rows.

2. Let the extension choose

Click events_parquet. Under the path, Output Format reads PARQUET (from extension), and Format Override is Auto from extension. Change the extension in the path and the format follows. Override it only when the file name has to lie — a .dat that's really CSV, for a system that insists on .dat.

Output Format inferred from the extension

3. Execute and look at the files

Execute. All five nodes report 20,000 rows. Then list the output folder with sizes.

Expected result

File Size What it is
events.ndjson (the input) 2.49 MB text, one JSON object per line
events.json 2.27 MB text, one JSON array of 20,000 objects
events.csv 1.03 MB text, header + 20,000 lines
events.xlsx 0.72 MB a zipped workbook, one sheet
events.parquet 0.38 MB binary, columnar, typed, compressed

Your sizes may differ by a few bytes; the ranking won't.

Now open them and look at what each one kept:

  • events.json is a JSON array — [ {…}, {…}, … ], what most tools expect from that extension. The input was NDJSON, one object per line. Want that shape out? Name the file events.ndjson (or .jsonl). Arrays are easy to hand to an API or a script; NDJSON streams and appends line by line, so it's the one for logs and big feeds.
  • CSV kept the values and dropped the types. In the file, ok is the text true and event_id is a run of digits. Whoever reads the CSV next re-infers the types — and you saw in 2.4 how inference can go wrong.
  • Parquet kept the types — and it's the smallest. The schema is stored inside the file, so a reader doesn't guess: event_id and duration_ms come back as integers and ok as a boolean, every time. Odara writes it Snappy-compressed, the usual default, which is why it's a sixth of the JSON's size.
  • Excel is compact too, because .xlsx is a ZIP archive — but it's the slowest to write (check the run log) and has a hard limit of about a million rows per sheet.

One thing you might expect and won't see:

  • ts is text in the Parquet file, because the NDJSON source read it as text and Parquet stores what it's given. Read the CSV back with a Magic File source, though, and inference turns the same column into a timestamp. Neither is wrong — but only one of them is guaranteed to come out the same next time. Casting it on purpose is World 3.

Common pitfalls

A file comes out in the wrong format. The extension is the switch. events.jsn or a missing extension won't do what you meant; check Output Format in Properties before running.

The consumer wants one object per line, not an array. Use the extension: .ndjson or .jsonl writes one object per line; .json writes an array.

Appending to a .json file that another tool wrote. With Append on, Odara reads the existing array and adds the new rows to it. If the file isn't a JSON array — one object per line, say — the run stops with Can't append; write .ndjson instead.

The Excel target is slow or fails on a big dataset. Excel is for people. Past a few hundred thousand rows, write Parquet or CSV instead.

What you built

The mirror image of level 2.4: one node that writes whatever format the next system needs. And the comparison that goes with it — CSV for humans and old systems, NDJSON for streams and logs, Parquet for anything that will be read by another data tool, Excel for the person who asked for a spreadsheet.

Check yourself

One question, drawn from a larger bank. Get it right to pass.

  1. The data will be read next by another data tool. Which format fits best?