<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>Lab notes · Kamol Nazarov</title>
    <link>https://kamol.dev/blog/</link>
    <atom:link href="https://kamol.dev/blog/rss.xml" rel="self" type="application/rss+xml"/>
    <description>Build write-ups and lab notes from a working repair shop in Brooklyn: what shipped, how it was built, what broke.</description>
    <language>en-us</language>
    <lastBuildDate>Mon, 02 Nov 2026 17:00:00 GMT</lastBuildDate>
    <item>
      <title>Fixing SSE Streaming That Freezes Behind Nginx for LLM Apps</title>
      <link>https://kamol.dev/blog/nginx-sse-streaming-freezes-llm/</link>
      <guid isPermaLink="true">https://kamol.dev/blog/nginx-sse-streaming-freezes-llm/</guid>
      <pubDate>Mon, 02 Nov 2026 17:00:00 GMT</pubDate>
      <description>LLM tokens trickle direct to Node but arrive in one lump behind nginx: one scoped location block with proxy_buffering off, gzip off and a longer read timeout.</description>
      <category>nginx</category>
      <category>sse</category>
      <category>streaming</category>
      <content:encoded><![CDATA[<p>The shop&#39;s parts-lookup assistant streams its answer token by token � you watch it type, the same as any chat app. That&#39;s how it behaved when I hit the Node port directly. The moment it went behind nginx on the VPS, the &quot;typing&quot; vanished: spinner for forty seconds, then the entire answer appearing in one lump. Nothing errored, nothing timed out. Search &quot;nginx sse streaming not working&quot; and you&#39;ll find a hundred people describing exactly this non-failure.</p>
<p>The stream isn&#39;t broken. It&#39;s being <em>buffered</em> � nginx is helpfully holding the bytes so it can forward them efficiently, and for server-sent events that helpfulness is the bug. The fix is one <code>location</code> block, scoped so the rest of the site keeps its compression and caching. (The Node/Express side of streaming � compression middleware, flush headers � is a separate post; this one is strictly the nginx layer.)</p>
<h2 id="confirm-it-s-buffering-not-breakage">Confirm it&#39;s buffering, not breakage</h2>
<p>Two curls tell the whole story:</p>
<pre><code class="language-bash"># direct to the app � tokens trickle line by line:
curl -N http://127.0.0.1:3000/api/chat

# through nginx � silence... then the entire answer at once:
curl -N https://parts.example.com/api/chat</code></pre>
<p>The <code>-N</code> disables curl&#39;s own buffering, so what you see is what the wire delivers. Same app, same request � the difference appears exactly at the proxy. If the direct curl trickles and the proxied one dumps, you have your diagnosis: nginx is standing between your users and the tokens, collecting them first.</p>
<h2 id="the-one-location-block">The one location block</h2>
<pre><code class="language-nginx">location /api/chat {
    proxy_pass http://127.0.0.1:3000;

    proxy_buffering off;
    gzip off;
    proxy_cache off;

    proxy_http_version 1.1;
    proxy_set_header Connection &quot;&quot;;
    proxy_set_header Host $host;

    proxy_read_timeout 1h;
}</code></pre>
<p>Line by line, because each one kills a different way this freezes:</p>
<ul>
<li><strong><code>proxy_buffering off</code></strong> � the main fix. By default nginx reads the upstream response into buffers and forwards it in efficient chunks. For a page, invisible; for a stream, it&#39;s a dam. Off means each event leaves the moment it arrives.</li>
<li><strong><code>gzip off</code></strong> � gzip cannot compress what it hasn&#39;t collected, so a gzipping proxy wants batches of bytes. This disables gzip for this route only. That&#39;s the part most copy-paste configs miss: you want to <em>disable gzip for the SSE location</em> in nginx, not for the whole site � everywhere else keeps compression.</li>
<li><strong><code>proxy_cache off</code></strong> � if a <code>proxy_cache_path</code> is in scope, this location must not try to store a never-ending response. Even if you think no cache applies here, an inherited zone from the surrounding <code>server</code> block is a classic surprise.</li>
<li><strong><code>proxy_http_version 1.1</code> + <code>Connection &quot;&quot;</code></strong> � nginx talks to upstreams over HTTP/1.0 by default; 1.1 with a cleaned connection header is what a persistent event stream expects.</li>
<li><strong><code>proxy_read_timeout 1h</code></strong> � the quiet one. The default is 60 seconds <em>between reads</em>. A model that thinks before its first token, or pauses mid-generation, crosses that gap and gets cut mid-stream � the &quot;nginx server-sent events timeout&quot; people chase for days. Raise it for this location; an idle stream is not a stuck request here.</li>
</ul>
<h2 id="if-you-can-t-touch-nginx-the-response-header-escape-hatch">If you can&#39;t touch nginx: the response-header escape hatch</h2>
<p>Docs sometimes mention <code>X-Accel-Buffering: no</code> � the app sends that header and nginx skips buffering for that response. It works, but it lives in the application, which is exactly the layer this post isn&#39;t about. Worth knowing it exists, because your framework&#39;s SSE helper may already send it � and if tokens still freeze with it set, the cause is usually gzip or the read timeout above.</p>
<h2 id="verify-it-the-way-you-diagnosed-it">Verify it the way you diagnosed it</h2>
<pre><code class="language-bash">sudo nginx -t &amp;&amp; sudo systemctl reload nginx
curl -N https://parts.example.com/api/chat</code></pre>
<p><code>nginx -t</code> first, always � a broken config reloaded at 9 PM is a worse evening than a frozen stream. Then the same proxied <code>curl -N</code> from before: tokens should now trickle exactly like the direct one did. In the browser, the typing animation just comes back.</p>
<p>For the record, the failure-to-knob mapping:</p>
<div class="table-scroll" role="region" aria-label="Table, scrolls sideways" tabindex="0"><table>
<thead>
<tr>
<th>Symptom</th>
<th>The line that fixes it</th>
</tr>
</thead>
<tbody><tr>
<td>Whole answer arrives at the end</td>
<td><code>proxy_buffering off</code></td>
</tr>
<tr>
<td>First tokens stall, then burst</td>
<td><code>gzip off</code> (batching to compress)</td>
</tr>
<tr>
<td>Stream dies ~60s in, mid-generation</td>
<td><code>proxy_read_timeout 1h</code></td>
</tr>
<tr>
<td>Weird keepalive resets</td>
<td><code>proxy_http_version 1.1</code> + <code>Connection &quot;&quot;</code></td>
</tr>
</tbody></table></div>
<p>One sentence to keep: <strong>nginx buffers because it&#39;s trying to be a good proxy � for one streaming route, a scoped <code>location</code> block that turns that off is the whole fix, and the rest of the site never notices.</strong></p>
]]></content:encoded>
    </item>
    <item>
      <title>Fixing zodResolver type errors after upgrading to zod 4 in React Hook Form</title>
      <link>https://kamol.dev/blog/zodresolver-type-error-zod-4-react-hook-form/</link>
      <guid isPermaLink="true">https://kamol.dev/blog/zodresolver-type-error-zod-4-react-hook-form/</guid>
      <pubDate>Thu, 29 Oct 2026 16:00:00 GMT</pubDate>
      <description>Upgrading to zod 4 turns every zodResolver() red in React Hook Form. Three fixes: resolvers v5, dropping explicit useForm generics, and the z.coerce input trap.</description>
      <category>zod</category>
      <category>react</category>
      <category>typescript</category>
      <content:encoded><![CDATA[<p>The intake form for shop drop-offs runs on React Hook Form with a zod schema, so when zod 4 shipped I bumped the version in one line and let the agent upgrade the rest of the app. It came back with every <code>zodResolver(</code> call in the codebase underlined red, and TypeScript errors like this one:</p>
<pre><code>TS2345: Argument of type '{ resolver: Resolver&lt;FormValues, any, FormValues&gt;; }'
is not assignable to parameter of type 'UseFormProps&lt;FormValues, any&gt;'.
  Types of property 'resolver' are incompatible.
    Type 'Resolver&lt;{ title: string; slots: number }, any, { title: string; slots: number }&gt;'
    is not assignable to type 'Resolver&lt;FormValues, any, FormValues&gt;'.</code></pre>
<p>If you searched &quot;zodResolver type error zod 4&quot; and landed here, the good news is the react hook form zod 4 type mismatch almost always comes down to three causes, and you can check them in order. No generic RHF+zod tutorial below � just the version pain.</p>
<div class="table-scroll" role="region" aria-label="Table, scrolls sideways" tabindex="0"><table>
<thead>
<tr>
<th>Your error mentions</th>
<th>The actual cause</th>
<th>Section</th>
</tr>
</thead>
<tbody><tr>
<td><code>Resolver&lt;...&gt;</code> not assignable, or <code>parse</code> missing</td>
<td><code>@hookform/resolvers</code> too old for zod 4&#39;s types</td>
<td>Fix 1</td>
</tr>
<tr>
<td><code>useForm&lt;FormValues&gt;</code> + generic complaints</td>
<td>Explicit generics fighting the resolver&#39;s inference</td>
<td>Fix 2</td>
</tr>
<tr>
<td>Input <code>unknown</code> / <code>string</code> where you wanted <code>number</code></td>
<td><code>z.coerce</code> making input ? output</td>
<td>Fix 3</td>
</tr>
</tbody></table></div>
<h2 id="fix-1-put-the-versions-in-the-only-rows-that-line-up">Fix 1: put the versions in the only rows that line up</h2>
<p>Zod 4 rewrote its TypeScript surface, and the resolver package was re-typed to match. <code>@hookform/resolvers</code> v3/v4 were built against zod 3 � against zod 4 they produce exactly this family of hookform resolvers zod 4 errors, even when runtime behavior looks fine on small schemas.</p>
<div class="table-scroll" role="region" aria-label="Table, scrolls sideways" tabindex="0"><table>
<thead>
<tr>
<th>zod</th>
<th>@hookform/resolvers</th>
<th>react-hook-form</th>
</tr>
</thead>
<tbody><tr>
<td>3.x</td>
<td>3.x � 4.x</td>
<td>7.x</td>
</tr>
<tr>
<td>4.x</td>
<td><strong>5.x+</strong></td>
<td><strong>7.55+</strong></td>
</tr>
</tbody></table></div>
<pre><code class="language-bash">npm i @hookform/resolvers@^5 react-hook-form@^7.55
npm ls zod @hookform/resolvers react-hook-form</code></pre>
<p>Read the <code>npm ls</code> output, don&#39;t skim it: a second copy of zod 3 hiding under some other dependency keeps its old types in scope and re-creates the mismatch after you&#39;ve &quot;fixed&quot; everything. If you see two zod entries, that dedupe is the fix, not your code.</p>
<h2 id="fix-2-drop-the-explicit-useform-generics">Fix 2: drop the explicit <code>useForm</code> generics</h2>
<p>The pattern everyone learned on zod 3:</p>
<pre><code class="language-ts">type FormValues = z.infer&lt;typeof schema&gt;;

const form = useForm&lt;FormValues&gt;({   // ? the explicit generic is now the problem
  resolver: zodResolver(schema),
});</code></pre>
<p><code>zodResolver(schema)</code> already knows the exact field values it will produce, and resolvers v5 + recent RHF infer <code>TFieldValues</code> straight from the resolver. When you pass your own generic on top, TypeScript tries to reconcile the two � the classic zodResolver useForm generic error � and any microscopic difference (an optional made <code>undefined</code>-able, a default added) turns into a wall of arrows. Let the schema drive:</p>
<pre><code class="language-ts">const schema = z.object({
  title: z.string().min(1),
  slots: z.number().int().min(1),
});

const form = useForm({
  resolver: zodResolver(schema), // field types inferred from the schema
});

type FormValues = z.infer&lt;typeof schema&gt;; // for handlers, still fine</code></pre>
<p>Deleting code as the fix feels like a trick, but the explicit generic was duplicating information the resolver now owns. This one change clears the majority of the red after the version bump.</p>
<h2 id="fix-3-the-z-coerce-trap-input-output">Fix 3: the <code>z.coerce</code> trap � input ? output</h2>
<p>HTML forms hand you strings. Coercion was the standard answer:</p>
<pre><code class="language-ts">const schema = z.object({
  slots: z.coerce.number().int().min(1),
});</code></pre>
<p>Here&#39;s the trap: <code>z.infer</code> returns the <strong>output</strong> type (<code>number</code>), but React Hook Form registers and reads <strong>input</strong> values (what the DOM actually holds � <code>string</code>/<code>unknown</code> for coerced fields). Type the form as the output and the resolver&#39;s input side no longer fits:</p>
<pre><code>TS2322: Type 'unknown' is not assignable to type 'number'.</code></pre>
<p>For any schema where input and output differ (coerce, <code>preprocess</code>, transforms), use all three generics <code>useForm</code> accepts � input, context, output:</p>
<pre><code class="language-ts">const form = useForm&lt;z.input&lt;typeof schema&gt;, any, z.output&lt;typeof schema&gt;&gt;({
  resolver: zodResolver(schema),
});</code></pre>
<p>Alternatively, side-step coercion at the field level with <code>register</code>&#39;s built-in: <code>{...form.register(&#39;slots&#39;, { valueAsNumber: true })}</code> and keep the schema on plain <code>z.number()</code>. Both are legitimate; the three-generic form is the one that keeps validation and typing in one place, <code>valueAsNumber</code> is the one that keeps the schema honest about what a text input contains.</p>
<h2 id="the-30-second-checklist">The 30-second checklist</h2>
<ol>
<li><code>npm ls zod @hookform/resolvers react-hook-form</code> � one zod, resolvers 5+, RHF 7.55+.</li>
<li>Remove explicit <code>useForm&lt;...&gt;</code> generics; let <code>zodResolver</code> infer.</li>
<li>Still red? Check for <code>z.coerce</code>/<code>preprocess</code> and switch to the three-generic <code>useForm</code> form.</li>
</ol>
<p>One sentence to keep: <strong>zod 4 didn&#39;t break your form � it made the resolver the source of truth, so match its version and let it do the typing.</strong> The rest of the zod 4 migration (the <code>z.email()</code> rename, one-argument <code>z.record</code>) is mechanical; this wiring is the only part that fights back.</p>
]]></content:encoded>
    </item>
    <item>
      <title>Fixing ModuleNotFoundError When a Python Cron Job Works From the Shell</title>
      <link>https://kamol.dev/blog/python-cron-modulenotfounderror-venv/</link>
      <guid isPermaLink="true">https://kamol.dev/blog/python-cron-modulenotfounderror-venv/</guid>
      <pubDate>Thu, 22 Oct 2026 16:00:00 GMT</pubDate>
      <description>Same script, same machine: works over ssh, ModuleNotFoundError from cron. Cron never activates your venv - point the entry at the venv's own python.</description>
      <category>python</category>
      <category>cron</category>
      <category>venv</category>
      <content:encoded><![CDATA[<p>The nightly job pulls the day&#39;s repair orders into a spreadsheet for the front desk. Tested it over ssh before bed: clean run, rows in the sheet. At 2:30 the next morning cron emailed me this instead:</p>
<pre><code>Traceback (most recent call last):
  File &quot;/home/shop/order-sync/sync.py&quot;, line 3, in &lt;module&gt;
    import requests
ModuleNotFoundError: No module named 'requests'</code></pre>
<p>If you&#39;re searching &quot;python cron job module not found&quot; or &quot;python script works in terminal but not cron&quot;, this is that situation exactly: same machine, same script, same user � but only cron can&#39;t find the modules. There&#39;s really one cause, one fix, and two neighboring traps that ride along with it.</p>
<h2 id="what-cron-actually-runs">What cron actually runs</h2>
<p>Cron is not your login shell. It launches each entry with a nearly empty environment: <code>SHELL=/bin/sh</code>, a minimal <code>PATH</code> (typically <code>/usr/bin:/bin</code>), your <code>HOME</code>, and that&#39;s about it. It never reads <code>.bashrc</code>, <code>.profile</code>, or anything you&#39;ve exported by hand.</p>
<p>Two consequences do the damage together:</p>
<ol>
<li>Whatever you installed into a virtualenv is invisible, because nothing ever ran <code>source venv/bin/activate</code>.</li>
<li>The bare <code>python3</code> in your crontab resolves through that minimal PATH to the <strong>system</strong> interpreter, whose site-packages don&#39;t contain your packages.</li>
</ol>
<p>System Python plus missing venv equals <code>ModuleNotFoundError</code>, every time.</p>
<h2 id="prove-it-in-one-line">Prove it in one line</h2>
<p>Before fixing anything, confirm which interpreter cron is really using. Drop this at the top of the script temporarily:</p>
<pre><code class="language-python">import sys
print(sys.executable)
print(&quot;\n&quot;.join(sys.path))</code></pre>
<p>Cron mails command output to you (or your entry redirects to a log � it should). You&#39;ll see <code>/usr/bin/python3</code> where you expected <code>/home/shop/order-sync/venv/bin/python3</code>. That mismatch is the entire bug.</p>
<h2 id="the-fix-nobody-tells-you-skip-activation-entirely">The fix nobody tells you: skip activation entirely</h2>
<p>Most threads tell you to source <code>activate</code> in a wrapper script. You can � but activation is optional. A venv is just a directory: a copy of the interpreter and its own <code>site-packages</code>, with a <code>pyvenv.cfg</code> beside the binary. The interpreter locates its venv from <strong>its own path</strong>, not from environment variables. Run the venv&#39;s python directly and you get the venv, no activation, no PATH setup, from any shell, service, or container.</p>
<p>So this is how to run a python venv in crontab without the ceremony � point the entry at the venv&#39;s own interpreter:</p>
<pre><code>30 2 * * * /home/shop/order-sync/venv/bin/python3 /home/shop/order-sync/sync.py &gt;&gt; /home/shop/order-sync/cron.log 2&gt;&amp;1</code></pre>
<p>One line, absolute paths, done. That version survives a server rebuild and a migration to systemd later, because it depends on nothing outside the entry itself.</p>
<h2 id="the-two-traps-that-ride-along">The two traps that ride along</h2>
<p>The wrong interpreter explains the import error, but a bare crontab entry tends to collect two more failures on the same night:</p>
<div class="table-scroll" role="region" aria-label="Table, scrolls sideways" tabindex="0"><table>
<thead>
<tr>
<th>Symptom</th>
<th>Cause</th>
<th>Fix</th>
</tr>
</thead>
<tbody><tr>
<td><code>ModuleNotFoundError</code></td>
<td>System interpreter ran instead of the venv</td>
<td>Full path to <code>venv/bin/python3</code></td>
</tr>
<tr>
<td><code>FileNotFoundError</code> on a config or data file</td>
<td>Cron starts in <code>$HOME</code>, not your project dir; relative paths break</td>
<td><code>cd /project &amp;&amp;</code> before the command, or absolute paths</td>
</tr>
<tr>
<td><code>sh: pg_dump: command not found</code> from a subprocess call</td>
<td>Minimal PATH misses <code>/usr/local</code> or custom bins</td>
<td><code>PATH=</code> line at the top of the crontab</td>
</tr>
</tbody></table></div>
<p>The hardened version handles all three:</p>
<pre><code>SHELL=/bin/bash
PATH=/usr/local/bin:/usr/bin:/bin
30 2 * * * cd /home/shop/order-sync &amp;&amp; ./venv/bin/python3 sync.py &gt;&gt; cron.log 2&gt;&amp;1</code></pre>
<p>The <code>PATH=</code> and <code>SHELL=</code> lines apply to every entry below them in the same crontab. The <code>cd ... &amp;&amp;</code> guard is what keeps <code>open(&quot;orders.csv&quot;)</code> resolving to the file you meant.</p>
<h2 id="test-it-the-way-cron-will-run-it">Test it the way cron will run it</h2>
<p>You can reproduce cron&#39;s empty environment without waiting for 2:30 AM:</p>
<pre><code class="language-bash">env -i HOME=&quot;$HOME&quot; SHELL=/bin/sh /bin/sh -c \
  '/home/shop/order-sync/venv/bin/python3 /home/shop/order-sync/sync.py'</code></pre>
<p><code>env -i</code> wipes your shell&#39;s variables, so the command sees exactly the stripped world cron provides � same PATH emptiness, same working directory rules. If it works there, it works in the crontab; if it doesn&#39;t, the error you get is the real one, not one polluted by your interactive session.</p>
<p>One sentence to keep: <strong>cron never activates anything � it runs <code>/bin/sh</code> with a scrubbed environment, so point the entry at the venv&#39;s own interpreter and use absolute paths.</strong> After that, &quot;crontab python no module named&quot; stops being a mystery and your 2:30 AM email goes back to being a receipt.</p>
]]></content:encoded>
    </item>
    <item>
      <title>One night, five launches</title>
      <link>https://kamol.dev/blog/one-night-five-launches/</link>
      <guid isPermaLink="true">https://kamol.dev/blog/one-night-five-launches/</guid>
      <pubDate>Wed, 21 Oct 2026 16:00:00 GMT</pubDate>
      <description>On Oct 19, five names under kamol.dev were switched on in 71 minutes, with AI agents working in parallel. The log, the runbook, and what the count leaves out.</description>
      <category>agents</category>
      <category>deploys</category>
      <category>nginx</category>
      <content:encoded><![CDATA[<p>On the evening of Oct 19, five names under kamol.dev were switched on in 71 minutes, the first at 10:53 PM and the last at 12:04 AM. I run a repair shop in Brooklyn and build its software with AI coding agents, several at a time, each in its own thread. This is the log of that night, the runbook that made it quick, and what the count leaves out.</p>
<h2 id="the-five">The five</h2>
<div class="table-scroll" role="region" aria-label="Table, scrolls sideways" tabindex="0"><table>
<thead>
<tr>
<th>Time (ET)</th>
<th>Name</th>
<th>What it is</th>
</tr>
</thead>
<tbody><tr>
<td>10:53 PM</td>
<td>kamol.dev</td>
<td>A placeholder page. The portfolio replaced it at 1:19 AM.</td>
</tr>
<tr>
<td>10:55 PM</td>
<td>hello.kamol.dev</td>
<td>A test page for the static-site steps</td>
</tr>
<tr>
<td>11:24 PM</td>
<td>ScaledMail</td>
<td>My outreach dashboard, behind my sign-in</td>
</tr>
<tr>
<td>11:36 PM</td>
<td>WhatsMyCost</td>
<td>My AI-spend tracker, reachable only from my private network</td>
</tr>
<tr>
<td>12:04 AM</td>
<td>modernwars.kamol.dev</td>
<td>Frontline, a browser strategy game</td>
</tr>
</tbody></table></div>
<p>Other threads kept running around them. I opened the hosting-pipeline thread at 10:42 PM, on Grok 4.7. The server&#39;s nginx configuration was backed up at 10:53 PM, seconds before the first name went in. I opened a thread to move WhatsMyCost onto the server at 11:31 PM and one to build the portfolio at 11:48 PM, both on Claude Opus 5.5. In another project, BurnRate tagged two betas that evening: v0.3.0-beta.13 at 10:30 PM and v0.3.0-beta.14 at 11:13 PM.</p>
<h2 id="why-it-went-quickly">Why it went quickly</h2>
<p>kamol.dev and every name under it already pointed at one server, and nginx turns a name into a site. What the server didn&#39;t have was a wildcard certificate. That needs DNS changes made through an API, and the server has no credential for it. So every name gets its own certificate, issued over plain HTTP, which the server already answers for any name under kamol.dev.</p>
<p>That makes a new name the same four steps every time: put the files in place, issue the certificate, write one server block, test and reload. The runbook lists them in that order and says not to reload after a failed test. A short, fixed list is something an agent can follow and I can check: the address loads or it doesn&#39;t, and the certificate carries the name or it doesn&#39;t.</p>
<p>The private names use a variant. Public DNS still sends them to the server, so their certificates renew the normal way, but there is no public HTTPS server for them. From outside my private network the connection is simply closed. WhatsMyCost runs like that because it holds admin keys for the services it tracks.</p>
<h2 id="what-the-count-leaves-out">What the count leaves out</h2>
<ul>
<li>hello.kamol.dev is a test page. It is the runbook&#39;s worked example for the static-site steps, not a product.</li>
<li>kamol.dev went in as a placeholder. The portfolio this note lives on went live at 1:19 AM, outside the 71 minutes.</li>
<li>For four of the names, the time is when the server block was created. For WhatsMyCost, it is when its service started. I have no log of the first request to each one.</li>
<li>There is no hours-saved figure here, because I didn&#39;t measure one.</li>
</ul>
<h2 id="sources">Sources</h2>
<p>Server: creation times of the nginx files, the configuration backup and the site folders, and service start times from systemd. BurnRate: its git tags. Threads: creation times in T3 Code, the app I run the agents in.</p>
]]></content:encoded>
    </item>
    <item>
      <title>Fixing Stripe Webhook Signature Verification Failures in Express</title>
      <link>https://kamol.dev/blog/stripe-webhook-signature-verification-express/</link>
      <guid isPermaLink="true">https://kamol.dev/blog/stripe-webhook-signature-verification-express/</guid>
      <pubDate>Tue, 20 Oct 2026 16:00:00 GMT</pubDate>
      <description>express.json() eats the body before constructEvent can verify it: the usual Stripe webhook failure in Express, the raw-body fix, and the rest of the checklist.</description>
      <category>stripe</category>
      <category>express</category>
      <category>webhooks</category>
      <content:encoded><![CDATA[<p>The first time I wired Stripe webhooks into one of the shop&#39;s tools, every test event came back 400 with this:</p>
<pre><code>StripeSignatureVerificationError: No signatures found matching the expected signature for payload.
Are you passing the raw request body you received from Stripe?</code></pre>
<p>Search &quot;stripe webhook signature verification failed express&quot; and you&#39;ll find a hundred possible causes. Nine times out of ten it&#39;s exactly one: something parsed <code>req.body</code> as JSON before <code>stripe.webhooks.constructEvent()</code> ever saw the raw bytes � and in an Express app, that something is almost always <code>express.json()</code>. Here&#39;s why it happens, the two-line fix, and the much shorter list of suspects when the fix isn&#39;t it.</p>
<h2 id="the-code-that-produces-the-error">The code that produces the error</h2>
<pre><code class="language-js">// This version fails:
app.post('/webhooks/stripe', express.json(), (req, res) =&gt; {
  const sig = req.headers['stripe-signature'];

  let event;
  try {
    event = stripe.webhooks.constructEvent(
      req.body, // parsed object, not raw bytes
      sig,
      process.env.STRIPE_WEBHOOK_SECRET
    );
  } catch (err) {
    return res.status(400).send(`Webhook Error: ${err.message}`);
  }

  handleEvent(event);
  res.sendStatus(200);
});</code></pre>
<h2 id="why-the-signature-check-is-so-picky">Why the signature check is so picky</h2>
<p>When Stripe delivers a webhook, it computes an HMAC-SHA256 of <code>timestamp + &#39;.&#39; + raw_body</code> using the endpoint&#39;s signing secret (<code>whsec_...</code>) and ships the result in the <code>stripe-signature</code> header. To verify, the SDK recomputes the same HMAC on your side and compares the two. Identical bytes in, identical hash out � but if the bytes differ by a single character, the signatures don&#39;t match, and the SDK refuses to hand you the event.</p>
<p>The catch: an HTTP request body is a stream, and a stream can be read exactly once. <code>express.json()</code> reads it, parses it, and hands your route a convenient object. The raw bytes are gone by then � the SDK can&#39;t have them.</p>
<p>The trap that keeps people stuck for an afternoon: <code>JSON.stringify(req.body)</code> <em>sometimes</em> reproduces the original payload byte-for-byte (V8 preserves key insertion order, and Stripe sends compact JSON), so the old Stack Overflow answer of &quot;just re-stringify it&quot; appears to work � right up until a payload contains a number that round-trips differently (<code>1e21</code> becomes <code>1e+21</code>), a string that was escaped differently on the wire, or a proxy that normalizes the body in transit. You can&#39;t reconstruct the exact bytes from a parsed object. You have to keep them.</p>
<h2 id="the-fix-raw-body-on-the-webhook-route-json-everywhere-else">The fix: raw body on the webhook route, JSON everywhere else</h2>
<p>Express runs middleware and routes in registration order, so give the webhook its own raw-body parser and register it before the global JSON parser:</p>
<pre><code class="language-js">import express from 'express';

const app = express();

// 1. Webhook route first, with its own raw parser:
app.post(
  '/webhooks/stripe',
  express.raw({ type: 'application/json' }),
  (req, res) =&gt; {
    const sig = req.headers['stripe-signature'];

    let event;
    try {
      event = stripe.webhooks.constructEvent(
        req.body, // now a Buffer of the exact bytes Stripe sent
        sig,
        process.env.STRIPE_WEBHOOK_SECRET
      );
    } catch (err) {
      return res.status(400).send(`Webhook Error: ${err.message}`);
    }

    handleEvent(event);
    res.sendStatus(200);
  }
);

// 2. Everything else still gets normal JSON parsing:
app.use(express.json());</code></pre>
<p><code>express.raw({ type: &#39;application/json&#39; })</code> only fires for JSON content-types, only on that route, and leaves <code>req.body</code> as the untouched Buffer � exactly what <code>constructEvent</code> wants. If the webhook route lives on a router, the same principle applies: <code>router.post(&#39;/stripe&#39;, express.raw(...), handler)</code>.</p>
<h2 id="if-you-can-t-reorder-routes-capture-the-raw-body-during-parsing">If you can&#39;t reorder routes: capture the raw body during parsing</h2>
<p>Some apps have a global <code>express.json()</code> mounted before routes that are hard to move. The parser lets you hook the buffer before it&#39;s parsed:</p>
<pre><code class="language-js">app.use(express.json({
  verify: (req, res, buf) =&gt; {
    // keep a reference to the raw bytes for this route only
    if (req.originalUrl.startsWith('/webhooks/stripe')) {
      req.rawBody = buf;
    }
  }
}));

app.post('/webhooks/stripe', (req, res) =&gt; {
  let event;
  try {
    event = stripe.webhooks.constructEvent(
      req.rawBody,
      req.headers['stripe-signature'],
      process.env.STRIPE_WEBHOOK_SECRET
    );
  } catch (err) {
    return res.status(400).send(`Webhook Error: ${err.message}`);
  }

  handleEvent(event);
  res.sendStatus(200);
});</code></pre>
<p>Same effect: the SDK gets the original bytes, and the rest of the app never notices.</p>
<h2 id="when-the-raw-body-fix-isn-t-it">When the raw-body fix isn&#39;t it</h2>
<p>The error message tells you which failure you actually have:</p>
<div class="table-scroll" role="region" aria-label="Table, scrolls sideways" tabindex="0"><table>
<thead>
<tr>
<th><code>constructEvent</code> throws</th>
<th>What it means</th>
</tr>
</thead>
<tbody><tr>
<td><code>No signatures found matching the expected signature for payload</code></td>
<td>Bytes or secret are wrong � raw-body problem, or the wrong <code>whsec</code></td>
</tr>
<tr>
<td><code>Unable to extract timestamp and signatures from header</code></td>
<td>The <code>stripe-signature</code> header never arrived � a proxy stripped or renamed it</td>
</tr>
<tr>
<td><code>Timestamp outside the tolerance zone</code></td>
<td>The event is older than the tolerance window (default 300s) � check the server clock and any long-running queues</td>
</tr>
</tbody></table></div>
<p>Three checks that catch the remaining cases:</p>
<ol>
<li><strong>Wrong signing secret.</strong> Every webhook endpoint has its own secret. Test-mode and live-mode endpoints differ, and if you deleted and re-created the endpoint, the new one got a new <code>whsec_...</code> � your old env var silently stopped being right. Re-copy it from the Dashboard (Developers ? Webhooks ? your endpoint).</li>
<li><strong>A proxy mangling things.</strong> nginx, Cloudflare or a WAF in front of the app can rewrite the body or drop unusual headers. Compare what actually arrives with what Stripe sent.</li>
<li><strong>Testing locally with the wrong secret.</strong> <code>stripe listen</code> prints its own temporary <code>whsec_...</code> � that&#39;s the one your local server needs while testing, not the Dashboard&#39;s.</li>
</ol>
<h2 id="the-two-minute-loop-to-confirm-the-fix">The two-minute loop to confirm the fix</h2>
<pre><code class="language-bash">stripe listen --forward-to localhost:3000/webhooks/stripe
# in another terminal:
stripe trigger payment_intent.succeeded</code></pre>
<p>The CLI prints its local secret on startup, forwards the event, and shows the response code your server returned. A <code>200</code> means you&#39;re done; a <code>400</code> plus the table above points at the next suspect.</p>
<p>One sentence to keep: <strong><code>constructEvent</code> hashes the exact bytes Stripe sent � anything that touches the body first changes the math.</strong> Give that one route the raw parser and the error stops being mysterious.</p>
]]></content:encoded>
    </item>
  </channel>
</rss>
