Dual-Channel Alerts: Telegram + SMTP Failover | Toss Auto-Trading Dev Journal #21

🌐 한국어로 보기

One notification channel is one point of failure

Toss Securities (토스증권) is a Korean brokerage that exposes a REST API for retail algo trading — most of this series runs against its order/quote endpoints. In the last entry, I added an automatic pause when cumulative loss crosses a threshold. That raised an obvious question: what happens if the alert telling me about that pause never arrives?

During local testing, I saw Telegram API calls occasionally stall for several seconds under network jitter. If that lag lines up with a real -10% loss-limit breach, the alert could silently vanish. That’s the gap this entry closes: a second, independent notification channel.

Consolidating alert logic into notifier.py

Notification code used to be scattered across auto_trader.py — console logging in one place, Telegram calls in another, DB writes somewhere else. I pulled all of it into a dedicated notifier.py module with a clear four-step pipeline: console log → SQLite alerts table → Telegram → (conditionally) email.

class Notifier:
    def send_notification(self, message: str, level: str = "INFO"):
        self._log_console(message, level)
        self._save_to_db(message, level)
        self._send_telegram(message, level)

        if level in ("WARNING", "ERROR", "CRITICAL"):
            self._send_email(message, level)

Best-effort delivery: alerts should never crash the loop

The core design constraint here: a notification failure must never propagate up and stop the monitoring loop or the background scheduler. Every external call — Telegram, SMTP — is individually wrapped:

def _send_email(self, message: str, level: str):
    try:
        with smtplib.SMTP_SSL("smtp.example.com", 465, timeout=5) as server:
            server.login(EMAIL_ACCOUNT, EMAIL_APP_PASSWORD)
            server.sendmail(EMAIL_ACCOUNT, ALERT_RECEIVER, self._build_mime(message, level))
    except (smtplib.SMTPException, socket.timeout) as e:
        self._log_console(f"[EMAIL FAIL] {e}", "WARNING")

Whether the SMTP connection is refused or Telegram’s API times out, the exception dies inside that function. The trading loop stays completely decoupled from notification channel health.

Severity-based routing

• INFO (routine price checks, simulated fills): console + DB + Telegram only
• WARNING / ERROR / CRITICAL (loss limit breach, order limit breach, repeated API failures): Telegram + email, both

Sending every INFO event to email would just create noise and bury the alerts that actually matter — hence the severity split.

Wiring into risk_control.py

This connects directly to the loss-limit logic from the previous entry. Using a placeholder position (mock ticker 005930 / Samsung Electronics, for illustration only — never real account data) with a mock valuation of 1,000,000 KRW dropping to 900,000 KRW, breaching the default -10% threshold triggers:

def check_loss_limit(self, current_pnl_rate: float):
    if current_pnl_rate <= self.LOSS_LIMIT_THRESHOLD:  # default -10%
        engine_control.pause()
        notifier.send_notification(
            f"Cumulative loss {current_pnl_rate}% detected, pausing engine",
            level="ERROR"
        )

That single level="ERROR" call fans out to both channels. Three safety guards remain unchanged throughout this project: DRY_RUN = True, a fixed 1 KRW target buy price, and the live order API call physically commented out in code.

Verified with 134 pytest cases

The new test_notifier.py checks three things: that email is skipped for INFO-level alerts, that both Telegram and email fire for ERROR-level alerts, and that a raised exception inside the email path never propagates past send_notification. All 134 tests pass, including the 3 new ones.

To remove the single point of failure in a Telegram-only alert setup, add an SMTP email channel and route by severity: routine events stay on Telegram, while WARNING/ERROR/CRITICAL events go to both. Wrap every send in best-effort exception handling so alert failures never stop the trading loop.

FAQ

Isn’t Telegram alone good enough?

For routine monitoring, yes. The risk is specifically at the moment that matters most — a loss-limit breach — where even a rare API delay or outage means the alert never lands.

Does sending email slow down the trading loop?

The SMTP timeout is capped at 5 seconds, and any failure is absorbed immediately, so worst case impact is a few seconds — never a hang.

Why not send everything to email too?

Because that turns email into noise, which defeats the purpose — you’d start ignoring it, including the alerts that actually matter.

What’s next in this series?

With notifications and risk management in place, the next entry covers a pre-launch checklist and monitoring dashboard improvements.