The route

When a round fails — and Sleep

Download the exercise files

Sooner or later a file in the folder is broken — a half-finished upload, an export that crashed. What the loop does next is a setting, and both choices have a cost you need to know about. This level runs the 5.2 pipeline over a folder where one file is corrupt, both ways, and then adds a deliberate pause with Sleep.

Goal

Know exactly what Stop and Skip do with a broken file, and how to tell from the output that something was skipped.

You'll need

  • The exercise files for this level: a flaky/ folder with the same 12 files as 5.2 — except sales_2025-03-07, which is a truncated Excel file.
  • About 20 minutes.

Steps

1. Point the loop at the flaky folder

Duplicate your 5.2 pipeline (or rebuild it) as w5-4-flaky: set the File List's Directory to flaky/ and the CSV Write's Filename to flaky_sales.csv. Keep On Error on Stop loop on first failure. Delete any old flaky_sales.csv, then Execute.

Stop: the Iterate fails on the broken file

Iterate node failed at row 8: Connector error: Magic File detection failed:
Excel error: Zip error: invalid Zip archive: Could not find EOCD

"Row 8" is the eighth file the File List returned — not the eighth day. The message tells you which reader gave up and why: this .xlsx isn't a complete zip archive.

Now look at flaky_sales.csv. It exists, with the rows of the files processed before the broken one. Stop stops; it doesn't undo what earlier rounds already wrote.

2. Skip and continue

Click the Iterate and set On Error to Skip failed and continue loop.

On Error: skip failed rounds and keep going

Delete flaky_sales.csv again and Execute. The run is green.

Count what you got: 259 rows from 11 distinct source_file values — one file short of the 12 in the folder, and 29 rows missing. The Console says which round was skipped and why:

⚠ 1 of 12 rounds failed and were skipped (on error: continue):
  • row 8: Connector error: Magic File detection failed: Excel error: Zip error: invalid Zip archive: …

3. Make the skip visible when nobody is watching

The Console warning is there when you run from the editor. A scheduled run has no Console, and with Skip its status is still green — so the pipeline should check itself. The simplest check is a comparison you can do by eye — files listed versus files in the output:

SELECT COUNT(DISTINCT source_file) AS files_loaded, COUNT(*) AS rows_loaded
FROM input

Point a CSV Source at flaky_sales.csv, add that SQL and a Console, and compare files_loaded with what the File List reported. 11 against 12 is your alarm.

4. Pace the loop with Sleep

Some loops call a service that limits how often you can hit it. From ORCHESTRATE ▸ Flow Control, drop a Sleep between the Iterate and read_file, and set it to 2 seconds.

Execute and time it: every round now waits 2 seconds before reading. Three files take about 6 seconds; the whole folder, about 24. Sleep pauses every round because it sits after the Iterate, inside the loop.

Expected result

  • Stop: the run fails on the corrupt sales_2025-03-07.xlsx; the output keeps the rows written before it.
  • Skip: the run succeeds with 259 rows from 11 files.
  • With Sleep at 2 seconds, the run takes about 2 seconds per file.

Common pitfalls

Choosing Skip because it's green. Green with Skip means "some rounds may have failed". The editor's Console lists them; a scheduled run only has its status and row count. Use Skip together with a check like the one in step 3.

Re-running after a Stop and getting duplicates. The first run appended some rows before stopping; the second appends everything again. Delete the output first, or write each run to its own dated file.

What you built

An informed choice between failing loud and failing quiet — and a check that makes the quiet one safe. Next level puts it all together and starts deleting files, where these choices matter most.

Check yourself

One question, drawn from a larger bank. Get it right to pass.

  1. How many rows does the Skip run load from the flaky folder?