Get phone and SMS alerts fromLangSmith
LangSmith traces, evaluates and monitors LLM applications, and its project alerts fire when error rates, latency or feedback scores drift. Spike turns every alert into an incident and pages the on-call engineer — by phone, SMS, Slack or email — escalating until someone picks it up.
Set up with AI
I use LangSmith for monitoring and Spike for on-call and incident management. Walk me through connecting them: create a LangSmith integration in Spike, copy its webhook URL, and add that webhook to LangSmith alert settings. Then help me send a test alert to confirm incidents show up in Spike.
What is LangSmith?
LangSmith is LangChain’s platform for building reliable LLM applications. It records every trace, runs evaluations against datasets, and watches production projects with alert rules on error rate, latency and feedback scores — the signals that tell you an agent has started failing before users report it.
Connected to Spike, a firing alert stops being a notification nobody reads and becomes an incident: the on-call engineer is called and texted within seconds, and an alert nobody acknowledges climbs the escalation policy until someone answers. The incident title is the description you wrote on the LangSmith alert rule, so give the rule a description that says what broke — it is what appears on the lock screen.
Every alert carries its rule’s alert_rule_id, and Spike uses it to keep
repeats together: the same rule firing again is added to the open incident as
an event rather than paging the team a second time, while two different rules
stay two separate incidents. Alert routing rules read the
rest of the payload, so you decide who is paged and how hard.
LangSmith sends nothing when an alert recovers, so incidents do not resolve themselves. Resolve them from Spike once the fix is confirmed, or use Resolve by Timer to close them automatically after a quiet period.
How to set up LangSmith alerts
Every Spike integration is a unique webhook URL. Paste it into LangSmith and alerts start flowing — no agents, no code.
Step 1: Create a LangSmith integration in Spike
In your Spike dashboard, open Integrations and choose LangSmith. Spike generates a unique webhook URL for this integration — copy it, and attach it to the service and escalation policy that should answer these alerts.
Step 2: Add a webhook to your project's alert
In LangSmith, open the project, go to Alerts, create or edit an alert rule, and in the notification section select Webhook. Paste the Spike URL as the webhook URL.
Step 3: Set the header and body
Add the header {"Content-Type":"application/json"} and set the request body to {}. LangSmith adds the alert details to the body itself, so the empty object is all Spike needs.
Step 4: Resolve incidents with a timer
LangSmith does not send a notification when an alert recovers, so Spike cannot close the incident for you. Resolve it by hand, or turn on Resolve by Timer for the integration so incidents close after the period you set.
That's it — LangSmith alerts now page whoever is on call.
