The route

Mini-project: process and archive an inbox

Download the exercise files

A real inbox pipeline doesn't just read the files — it clears the inbox, so tomorrow's run only sees tomorrow's files. That means deleting files, and deleting is where order matters: a file must be gone only after it's safely somewhere else. This level combines everything in World 5 into that pipeline.

Goal

One pipeline that, for every file in project/inbox/:

  1. appends its rows to processed/all_sales.csv,
  2. writes a copy to archive/,
  3. and — only after both — deletes the original.

At the end the inbox is empty, the archive has 12 files, and all_sales.csv has 288 rows.

You'll need

  • A copy of the 5.2 inbox/ folder in a new project/ folder. This pipeline deletes files; don't point it at your only copy.
  • About 30 minutes.

Steps

1. The loop, as in 5.2

Create w5-5-inbox-project: File List inbox on project/inbox (pattern sales_*) → Iterate (Stop loop on first failure) → Magic File read_file on @{path}.

2. Two writes per round

From read_file, two branches:

  • SQL tag_source (SELECT *, '@{name}' AS source_file FROM input) → CSV Write all_sales, processed/all_sales.csv, Append on.
  • Magic File target (DELIVER ▸ Files) archive_copy, Output Path …/project/archive/@{name}.

The archive copy keeps each file's name and its extension, so Magic File writes CSV back as CSV, JSON as JSON, Excel as Excel.

3. Delete — last

Add a Run After both_written with both writes connected into it. After it, a File Management node (DELIVER ▸ File System) named remove_original:

Field Value
Mode Delete File
File Path Static · @{path}

Read once, write twice, delete only after both

File Management deleting the current round's file

Run After is the whole point of this step. Without it, the delete is just another branch after read_file, and nothing stops it from running before the writes finish — or while one of them is failing.

4. Execute

Execute, then look at the three folders.

Expected result

  • project/inbox/ is empty.
  • project/archive/ has the 12 files, same names.
  • project/processed/all_sales.csv has 288 rows from 12 source files.
  • Run it again: the File List finds nothing, the loop doesn't run, and the run is green. An empty inbox is a normal day.

The archived files hold the same data but aren't byte-for-byte copies: Magic File rewrites each one. JSON loses its original spacing and line breaks, and 149.0 may be written as 149. If you need exact copies — for an audit trail, say — Odara has no copy or move node today; archive with a tool outside the pipeline.

Common pitfalls

The delete isn't behind the Run After. Then it's just another branch after read_file and may run before the writes. If a write then fails — a full disk, a typo in the archive path — the original is already gone. Check the edges: both writes → Run After → delete, and nothing else into the delete.

The loop stops halfway and the inbox is half empty. That's Stop working as intended: the rounds before the failure finished (and deleted their files); the failing file and the ones not reached yet are still in the inbox. Fix the file and run again — the pipeline picks up exactly where it stopped, and ends with the same 288 rows.

Stop or Skip here? Both are safe with this design, because a round that fails never reaches its delete. With Skip, the 11 good files are processed and the broken one simply stays in the inbox. That makes the inbox itself the alarm: anything still in it after a run is something that failed. Check it, or add a File List on the inbox after the loop, behind a Run After, to report what's left.

Nothing is deleted. File Path is on From Column mode, which expects a column name like path in its input, not @{path}. Inside a loop, use Static with the variable.

What you built

A pipeline that can run every day without anyone tidying up after it. World 7 schedules it; until then, you've finished World 5.

Check yourself

One question, drawn from a larger bank. Get it right to pass.

  1. The inbox is already empty. What does a run do?