A Python script collects new posts from more than a hundred tech blogs every morning, ranks them with an LLM, and sends me one digest email. It ran for months as a cron job on a mini PC, and it ran well: retries with backoff, conditional GET so unchanged sources cost nothing, atomic state writes, even a short operations manual. There was no burning reason to move it to Kestra.
Because it worked, it makes a useful case study. Most workflow orchestration content starts from a greenfield pipeline or from something visibly broken. The more common enterprise situation is the opposite: a script that works, owned by someone with no time for a rewrite, quietly doing its job for months. Teams stall on orchestration migrations because they assume migration means rewriting.
This post shows that it does not. It also shows something I did not expect: the visibility an orchestration layer adds catches real bugs. Within one day it surfaced a bug that had cost about forty-five minutes every morning for weeks. Nobody was looking for it, because the mail kept arriving. A later incident showed the limit of that visibility, and what it takes to turn it into an alarm.
Kestra is the open source platform built for unified orchestration: declarative, event-driven, language-agnostic, and licensed under Apache 2.0. Flows are written in YAML, and each task can run any language in a container, which is why a working Python script can move over unchanged. Everything below ran on a small home server. The flows, the diagnostic tools, and the script itself are in a public GitHub repository, so you can follow along in the code.
How to wrap a Python script in Kestra without changing the code
The migration followed the strangler fig pattern. The pattern replaces a running system step by step: the new setup grows around the old one until the old one can be turned off. Here that meant four phases: wrap the script unchanged, run old and new in parallel, cut over, and only then, if ever, refactor.
The wrapper is short. A Kestra flow starts a Python container, mounts the script’s directory into it, injects the secrets as environment variables, and runs the script. The contract between Kestra and the script is an exit code and a directory. Kestra neither knows nor cares what happens inside the container.
id: blog_monitor
namespace: kaiwaehner.blog
concurrency:
limit: 1
behavior: QUEUE
inputs:
- id: mode
type: SELECT
values: [dry-run, send]
defaults: send
- id: work_dir
type: STRING
defaults: /home/kaiwaehner/blog_monitor
tasks:
- id: run_monitor
type: io.kestra.plugin.scripts.python.Commands
containerImage: python:3.13-slim
timeout: PT15M
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
volumes:
- "{{ inputs.work_dir }}:/work"
beforeCommands:
- pip install --quiet --no-cache-dir feedparser
env:
ANTHROPIC_API_KEY: "{{ secret('ANTHROPIC_API_KEY') }}"
BLOG_MONITOR_PASSWORD: "{{ secret('BLOG_MONITOR_PASSWORD') }}"
commands:
- "python -u /work/blog_monitor.py {{ inputs.mode == 'dry-run' ? '--dry-run' : '' }}"
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "0 8 * * *"
timezone: Europe/BerlinTwo properties of the script made this clean, and both are worth checking in any script you plan to wrap. First, its environment loader used os.environ.setdefault, so real environment variables win over the local .env file. Kestra’s secret store could take over credential delivery with no code change, and the .env file on disk eventually disappeared. Second, the script derives every path from its own location rather than from the working directory. Pointing the flow at a parallel copy of the directory was therefore enough to give the migration its own configuration and its own state, fully isolated from production.
What the script gained on day one, all of it configuration rather than code: a schedule with a proper timezone, a timeout, a concurrency limit so two executions of the flow can never race on the state file, encrypted secrets with role-based access instead of file permissions, and a run history with logs and durations.
Script owner and platform owner
The team that owns the script does not need to learn Kestra. They keep writing Python and deploying however they like. The platform team owns the flow, and the interface between the two is an exit code. This separation makes the pattern realistic in a company, where the script owner and the platform owner are rarely the same person.
Why the morning run took ten times longer
A healthy run takes about four minutes. On the second morning of parallel operation, the cron mail arrived nearly an hour late, and the Kestra dry run hung the same way. Its runtime display showed why nobody had ever noticed: nothing had failed. The cron run just took more than ten times longer than it should and then delivered normally. It had been doing this for weeks, because a digest that arrives late still arrives.
I diagnosed it wrong three times. Each wrong turn is a trap other teams will hit as well.
First theory: rate limiting, because cron and Kestra had briefly overlapped and hammered the same sources in parallel. Disproven by a manual run with nothing else active, which hung just the same. Second theory: runtime simply varies, so raise the timeout. This was exactly backwards: the long run was the defect. Third theory: it hangs at the last source visible in the log. Also wrong: a source that managed to write a log line has finished. The culprit writes nothing at all. The absence of evidence was the evidence.
Silent sources block the whole run
What actually found it was looking at open TCP connections inside the still-running container. A dozen established connections to one IP address, against a worker pool of ten. Retries were opening new sockets while the old ones never closed. Two sources turned out to open a TCP connection and then send nothing at all. They returned neither an error nor a timeout.
The main lesson of the incident is the difference between failing and hanging. A source that fails fast, say with a 403, costs a fraction of a second and is operationally harmless. A source that goes silent blocks everything behind it. The script waited on Python’s as_completed() with no timeout, so one silent source held the entire run hostage. A socket timeout does not protect you here, because it restarts on every received byte; a server that trickles one byte at a time resets it forever. And even after fixing that, two more layers appeared: leaving a with ThreadPoolExecutor block joins the hung threads, and Python joins every non-daemon thread again at interpreter exit. Under an orchestrator that last one is particularly nasty. The work succeeds, the mail goes out, the state is written, and the process still never exits. The run is reported as a timeout failure.
The ten-line fix
The complete fix is roughly ten lines: a total budget on as_completed, abandoned sources reported as failures instead of silently dropped, a non-blocking pool shutdown, and a hard exit after the final flush. Before the fix, the test scenario with two silent sources hung indefinitely. After it, the run completes, names both sources in the alert section of the mail, and exits cleanly.
This bug lived in production for weeks. The mail always arrived, so no human noticed. Cron reports neither runtime nor failure, so no system noticed either. What found the bug was not better code review. It was a runtime number on a screen and a timeout that turned “slow” into “failed loudly”.
Parked sources instead of deleted sources
Deleting a broken source from the configuration destroys the knowledge of why it was removed; three months later nobody remembers whether it is worth retrying. Instead, broken sources move to a parked_sources block that the digest ignores but the weekly health check still measures, together with the reason, the date, and what a fix would need. One of the two silent sources started responding again a few days later. I did not have to remember to check. The health check told me. What it could not tell me was whether to restore it: a 200 with a few kilobytes of HTML is a reachable server, not a working source, and this one still renders its content client-side. Reachability is what the check measures. Whether a source is worth having back remains a decision, and the parked entry holds exactly the context needed to make it.
Kestra Cases for failures nobody acts on
The digest already reported failing sources, once and then again every seven days. After the eighteenth identical line, nobody acts on the nineteenth. A line in an email has no owner, no status, no deadline, and no history. It is information, not an incident.
Kestra Cases bring incident management into the platform, and the weekly health check now opens a case automatically when sources fail. Three details make this more than a prettier alert. Deduplication is keyed on the flow and task, not on the title, so every later run attaches its evidence to the same case instead of opening a new one each week. One case accumulates the history until somebody resolves it. The case also carries a severity and an SLA: Medium, three days to acknowledge, thirty to resolve. Medium fits a slow digest that annoys but breaks nothing. And the health check flow attaches itself to the case as a case action, so “has it recovered?” is one click, and the answer lands in the case timeline where the next person can see it.
The pattern generalizes far beyond a blog digest. Any recurring check that people have learned to ignore in email is a candidate for exactly this shape: a scheduled flow, a case with deduplication, and a re-check action on the case itself.
Cutover from cron with a single state file
The riskiest moment of most migrations is state transfer. Here it was a non-event, by design: there is exactly one state file, and at any moment exactly one writer.
The concurrency limit enforces that within the flow: one execution at a time, the next one queued. It has two blind spots. It knows nothing about a cron job outside Kestra, which is why cron had to go first. And it knows nothing about a second flow touching the same directory, a hypothetical today and a certainty once the fetch loop is decomposed into tasks. The file is the real shared resource, an asset in Kestra’s vocabulary, so the flow now takes a lock on the asset itself. An execution lock serializes writes across flows and across workers, not only across executions of one flow. It carries a TTL, so a run that hangs the way the silent-source bug hung this one gives the resource back when the lease expires. And it is visible, with holder and expiry on the asset page and lock and unlock as separate permissions. The flow-level limit stays. The asset lock turns exactly one writer from a property of this flow into a property of the file.
- id: acquire_state_lock
type: io.kestra.plugin.kestra.ee.locks.Acquire
assetId: blog-monitor-seen-posts
ttl: PT30M
retry:
type: constant
interval: PT1M
maxAttempts: 20
finally:
- id: release_state_lock
type: io.kestra.plugin.kestra.ee.locks.Release
assetId: blog-monitor-seen-posts
# auth lines omitted, see the repositoryRelease sits in the flow’s finally block, so success, failure, and timeout all give the file back. The TTL covers the case where nothing runs at all, such as a dead worker. If another execution holds the lock, Acquire fails with a 423 and the retry waits.
Switching the flow to production
The cutover was three small changes. Comment out the cron entry, so rollback stays a single command. Point the flow’s working directory at the production directory instead of the parallel copy. Switch the mode from dry-run to send. Nothing was copied, so there is no gap and no overlap: the new process reads and writes the same state the old one maintained. The cutover itself is exactly-once, because no second copy exists that would need reconciling.
I deliberately skipped one modernization. Moving the state into the platform’s key-value store would have been the obvious platform-native step, and it would have ended the parallel phase before it began. A script that reads and writes its state through the orchestrator’s API cannot run outside the orchestrator, and running outside it was exactly what cron still had to do. The file is not legacy here. It is the interface that made the migration possible.
Minutes after the final cron run had collected the morning’s posts, a manual Kestra run against the same directory found nothing new and sent no email. Both systems had worked on the same state. The next morning the digest arrived on schedule, sent by the flow, while cron stayed silent.
At-least-once semantics and observability after cutover
Day-to-day semantics stay at-least-once, and that predates the migration: the script writes state only after the mail server accepts the message. If the send fails, the same posts appear again tomorrow. Duplicates are better than loss here. It is the same trade-off as committing Kafka consumer offsets after processing rather than before. Kestra changes nothing about that semantics, but it changes its visibility completely: a failed send is now a red execution with logs and a duration, not a mail that quietly never arrived.
The bigger win after cutover is operational. Every run is in the history with its logs. A failure is a red execution with the exact error, not a missing mail. The audit log records who changed which flow revision and when. Incidents live in cases with owners and SLAs rather than in inbox archaeology. Dashboards chart durations and failure counts over time, which is precisely the signal that would have caught the silent-source bug weeks earlier. The script itself did not change, but its operations are now observable end to end.
The morning the digest did not arrive
Three weeks after the cutover, the digest was missing. The execution list showed why within seconds: a red run of three minutes, and one log line from the script. Gmail had rejected the login with “Username and Password not accepted”. The password had not changed. The username had. A configuration deploy the day before had copied the repository’s config.json over production, and that copy carries placeholder addresses on purpose. The script had tried to log in as you@gmail.com. The error message pointed at the password, the cause was the username, and the fix was one restored file.
Nothing was lost. The script saves state only after the mail server accepts the message, so one manual execution in send mode delivered the whole backlog an hour later, more than a hundred posts with the ranking on top. The at-least-once design from the cutover did exactly what it was there for. Two things were still wrong. Nobody had told me: a failed execution is a red row in a list that nobody opens at eight in the morning, which is the ignored email line from the Cases section one layer up. And the root cause was a piece of debt I already knew about, mail addresses living in the configuration file. They now come from the secret store, and the script refuses placeholder values before it fetches a single source. Before the fix it fetched more than a hundred sources and paid for the ranking, then failed at the very last step. The repository configuration deploys as-is now.
Alerting as part of the flow
Orchestration made the failure visible in seconds. Turning visibility into an alarm is configuration, exactly like the schedule and the timeout on day one. The additions that close the gap cover one failure class each.
- A failed run. The flow’s errors block opens a Case and sends an alert mail. The alert task uses its own credentials from the secret store, so a configuration mistake in the digest cannot silence the alert about it.
- A run that succeeds but degrades. When the Anthropic credits run out, the digest still arrives, without the ranked Top News and with a note that nobody reads. This class becomes a low-severity Case, deduplicated like the health check, and no mail.
- A run that never happens. An errors block cannot fire for an execution that was never created, and nothing on the mini PC can report that the mini PC is off. An external heartbeat closes that gap: the flow pings it after every successful run, and the service complains when the ping is late.
The rule behind all three: an alert channel must not share a failure mode with the thing it watches. The same Gmail credentials, the same configuration file, or the same machine would have failed together with the digest.
What mission-critical workflows need on top
For a newsletter, losing a morning is annoying and losing a post is acceptable. For a payment batch or a factory schedule it is neither, and the same platform gets used differently.
- Idempotent delivery. Commit state only after confirmed delivery, and give every message an idempotency key so a retry cannot produce a duplicate at the receiver.
- Retries and a dead-letter path. Retry the send with backoff, and park what still fails where a person can see it instead of dropping it.
- Escalation. A Case with an acknowledgement SLA and an on-call rotation behind it, so an unacknowledged alert reaches a second person.
- Redundant state and workers. State in a replicated store instead of a file on one machine, and more than one worker able to run the flow.
- Guaranteed delivery in the messaging layer. A message that must not be lost belongs on a durable log such as Apache Kafka. The orchestrator coordinates producers and consumers and recovers the workflow around them. It does not replace the log.
The script in this post has none of these and does not need them. The value of the migration is that each one has a place to land when the requirement arrives.
Writing Kestra flows with AI agent skills and MCP
Every flow in this post was written with Claude, and how that went explains part of why wrapping was cheap enough to attempt on a weekday evening.
Kestra has conventions a general-purpose model does not know. A boolean input is BOOL, not BOOLEAN, retries are configured as maxAttempts, defaults in Pebble templates use ?? because the ?: shorthand does not exist, and a case action fires without inputs. None of these is hard. Each one costs a failed deployment, a search through the documentation, and a second attempt. Together they are the reason a first flow usually takes an evening rather than twenty minutes.
A skill exists for exactly this class of problem. It packages the flow syntax, the plugin conventions, and the known pitfalls as instructions the model loads when a task touches Kestra, so the knowledge arrives before the first mistake rather than after it. The list above is what such a skill is there to prevent. With a skill loaded, that knowledge is in the model before the first line of YAML.
MCP for live system state
Where no skill exists for a tool, the Model Context Protocol covers the gap from the other side. An MCP server gives the model a live connection to the running system: it can read the current flow definition, pull the logs of a specific execution, or list installed plugins, instead of reasoning from a training snapshot that may be a version behind. A skill carries conventions; MCP carries state. They compose well, one to write the flow, the other to read what happened when it ran.
The rule I have settled on is simple. Use a skill where the vendor publishes one, because it encodes mistakes someone has already made. Use MCP everywhere else, because live access beats a stale snapshot. Kestra offers both: Agent Skills for flow authoring and operations, and MCP servers for current documentation and for the running instance. The pattern is not specific to Kestra, though. Whichever platform you wrap a script into, check for a skill first and an MCP server second before you start reading documentation by hand.
Extending the migrated script through configuration
The most useful consequence of the migration is that the platform now owns scheduling, visibility, and incident handling, so extensions shrink to configuration size.
Adding a new blog to monitor is one JSON entry, and the health check reports whether the new source is reachable before the digest depends on it. I tested this with 25 new sources in one go. The health check ran before the first digest touched them. Twenty-two passed. The other three were a dead feed URL, a site that blocks non-browser clients, and a gateway timeout on the day, and each of them got a decision before it could cost a single morning. A later batch showed the limit of that check: eleven sources were reachable but had no feed URL configured, and only the digest noticed. The check measures reachability, and extending it to configuration errors is the next small fix.
Next increments without a rewrite
Sources that only publish by newsletter could enter through an email trigger. The HTTP-only script could never see them. The fetch loop can gradually decompose into per-source tasks, at which point the execution timeline shows exactly which source is slow, and the guessing games with TCP tables end for good. Flow deployment can move from a script to synchronization straight from the git repository, so a change to production is a merge, with review and history included. Since the mail addresses moved into the secret store, the script’s configuration file can travel the same path, which would have prevented the incident above.
None of these is a rewrite. Each is an increment on a platform that is already carrying the operational load. This is the main argument for wrapping a working script instead of rewriting it: the day-one cost is close to zero, and every improvement after that has a place to land.
To follow this work across data integration, workflow orchestration, process intelligence, and trusted agentic AI, subscribe to the newsletter and connect on LinkedIn.
