Scheduling Trading Jobs in Python

Engineering and education only. This article covers the mechanics of scheduling software jobs. It is not investment advice, a trading signal, or a recommendation. Connecting any scheduled job to live execution is your responsibility and risk.

A "trading job" here means any Python task that needs to run on a clock: pulling end-of-day data, rebalancing a paper portfolio, recomputing features, or rotating logs. The engineering challenge is not the business logic — it is making the job run exactly when it should, exactly once, and cleanly stopping when you redeploy.

Three scheduling primitives

You have three common options, each with a different operational footprint.

• cron — The OS-level scheduler. Dead simple, battle-tested, no Python process to keep alive. The job is a fresh process each run. Downsides: minute-granularity only, no built-in overlap protection, and timezone handling depends on the system clock.
• systemd timers — Like cron but with dependency ordering, logging through journalctl, randomized delays, and Persistent=true to catch missed runs after downtime. Best when you already run on a systemd Linux host.
• APScheduler — An in-process Python scheduler. You keep one long-lived process that owns the schedule. Gives you sub-minute intervals, programmatic control, job stores, and easy access to your existing config and clients.

A reasonable rule of thumb: use cron or systemd timers for coarse, independent batch jobs, and APScheduler when jobs share state, need fine timing, or must coordinate with a running service.

Timezone and market-hours awareness

Never schedule against the server's local time. Pin an explicit timezone and check market calendars in code.

from apscheduler.schedulers.blocking import BlockingScheduler
from apscheduler.triggers.cron import CronTrigger
from zoneinfo import ZoneInfo
 
NY = ZoneInfo("America/New_York")
scheduler = BlockingScheduler(timezone=NY)
 
@scheduler.scheduled_job(CronTrigger(day_of_week="mon-fri", hour=16, minute=5, timezone=NY))
def end_of_day_job():
    if not is_trading_day():   # consult an exchange calendar, e.g. pandas_market_calendars
        return
    run_end_of_day()

Holidays and half-days are not weekends — encode them with a real exchange calendar rather than a hardcoded weekday rule.

Idempotent jobs

Schedulers retry, machines reboot, and Persistent=true replays missed runs. Assume every job may execute more than once and design so a second run is a no-op.

def run_end_of_day():
    run_id = f"eod:{today_ny():%Y-%m-%d}"
    if already_done(run_id):       # check a DB row or marker file
        return
    process_and_store()
    mark_done(run_id)              # write the marker only after success

Key the marker on the logical date the job represents, not on wall-clock time, so a delayed run still maps to the right slot.

Avoiding overlapping runs

A slow run that bleeds into the next trigger can corrupt shared state. With APScheduler, cap concurrency per job:

scheduler.add_job(
    sync_data,
    CronTrigger(minute="*/5", timezone=NY),
    max_instances=1,        # never two copies of this job at once
    coalesce=True,          # collapse missed runs into a single fire
)

For cron or multi-host setups where max_instances cannot help, take an explicit lock. A file lock guards a single host; a database advisory lock or a Redis key with a TTL guards a fleet.

import fcntl, os
 
class JobLock:
    def __init__(self, path): self.path = path
    def __enter__(self):
        self.fd = os.open(self.path, os.O_CREAT | os.O_RDWR)
        fcntl.flock(self.fd, fcntl.LOCK_EX | fcntl.LOCK_NB)  # raises if held
        return self
    def __exit__(self, *_):
        fcntl.flock(self.fd, fcntl.LOCK_UN)
        os.close(self.fd)

Graceful shutdown and logging

A long-lived scheduler must stop cleanly so in-flight jobs are not killed mid-write. Trap signals and call shutdown(wait=True).

import signal, logging
 
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s %(levelname)s %(name)s %(message)s',
)
log = logging.getLogger("scheduler")
 
def shutdown(signum, _frame):
    log.info("signal %s received, draining jobs", signum)
    scheduler.shutdown(wait=True)   # let running jobs finish
 
signal.signal(signal.SIGTERM, shutdown)
signal.signal(signal.SIGINT, shutdown)
scheduler.start()

Log every fire with a correlation id, the logical date, and the outcome. Structured, timestamped logs are what let you answer "did the 16:05 job actually run yesterday?" without guessing.

Where to go next

Once jobs are scheduled and idempotent, the next concern is knowing they stayed healthy in production. Continue with monitoring — heartbeats, error rates, and alerting on a job that silently stopped firing.