Performance at Scale with Ryan Townsend

🎧 New Señors @ Scale Episode

This week, I spoke with Ryan Townsend, Principal Product Manager for Performance at Cloudflare, where he took over the real user monitoring product in March and, after a reorg six weeks in, now covers performance measurement across the company. Ryan started building websites in 2002 as a teenager, spent 23 or 24 years in full stack web development and e-commerce, co-founded Shift Commerce and ran its product and technology arm for nine or ten years with around 40 engineers at peak, and was CTO of Matalan Direct after the Shopify store he built was acquired by Matalan.

In this episode, we dig into why shipping to strangers the same day hooked him, how Rails earned its place against an inherited .NET team, how he learned to let go after an outage during Dreamforce, the blocking script that wrecked P90, and how Shift ran over 99% cache hit ratios on pages with baskets and inventory on them.

⚙️ Main Takeaways

1. The web hooked him because nothing stands between you and the user

The reason he stayed has not changed since 2002.

• The origin: "I was that cousin everyone has who does websites and everything." The web was small enough then that one person could learn it start to finish. "There was only so many HTML tags."
• What still excites him: You build something on your machine, deploy it, and your friends and someone on the other side of the world use it immediately. "It's not gate kept." A mobile app still puts an app store between you and those same people.
• The loop that made the career: "That feedback loop of being able to build something and then put it into someone's hands was what drove me to be interested in this."
• The souvenir: He found his first USB stick from school on his desk during the recording. "It's 16 megabytes in size and it has some of my original PHP on there. And yeah, it's full of security problems and other holes."
• How performance entered: Through e-commerce, where speed decides conversion. His stance on stacks follows from that: "Ultimately, if something's working for a user, then that's what matters."

2. Second place on Shopify with half the trading window

This is the business Matalan acquired, and it entered the competition with half the trading window.

• The competition: Shopify's Build a Business competition scored your best three months of trading out of a six month window.
• The handicap: "Most businesses, your best three months are going to be four, five, and six. For us, we joined halfway through, so we could only use our first few months of trading and we still came second."
• The exit: Matalan acquired it in the UK and it became Matalan Direct, with Ryan as CTO.
• What it exposed: Shopify at the time could not serve the larger end of the market well, which is why he co-founded Shift Commerce as the exit point from Shopify and small self hosted tools, a headless SaaS platform built for enterprise.
• The scale: Billions in revenue each year ran through the platform. A couple of years ago they liquidated the company.

3. He bet a company against "Rails doesn't scale"

He picked up Rails at version 0.9 in late 2005. Years later he bet a company on it.

• Why it clicked: He watched DHH's blog demo after scrambling together PHP and Perl, and what sold him beyond the productivity gains was the rigor. "Rails made that accessible to everyone, and presented it in a way that's like, this is how you build software properly." Most people at the time did some QA and at best ran Selenium against staging.
• The Shift decision: He inherited a .NET team, had a little investment and no long runway, and they could not deliver fast enough. "This is proven to work at scale. It's proven to be productive. We're going to have to use this technology to move forward."
• The verdict: "The whole Rails doesn't scale thing was always a myth anyway. Annoyingly, it stuck as this marketing phrase." Shopify proved it at one scale, Shift at a smaller one.
• The caveat: Rails has concurrency challenges, but the problems are well known and the queuing story is mature: delayed job out of the database, Resque on Redis, Sidekiq for years, and now Good Job and the Solid libraries bringing it back to Postgres. "You can go so far with it without having to concern yourself too much with making these upfront decisions about what the architecture should look like. And that probably exceeds where most people need to go."
• Where it stops fitting: If you are Walmart with an enormous employee base tying together many services, maybe not. For a new company with specific needs, he still calls it a great place to start, and notes Shopify has been helping AI providers with Rails training data.
• The comparison I did not expect: "I feel like the experience I had picking up Rails is what everyone's having now with AI." The feeling of "I can do all of this that I couldn't do previously" is what he had two decades ago.

4. Resume driven development is what a fractional CTO gets hired to stop

As a fractional CTO he kept walking into the same situation in friends' businesses.

• The pattern: A company hires a former developer as full time CTO, pays them a lot, and that person finally has decision making power. They use it to "build out that dream LinkedIn CV that they've always wanted to do, the whole resume driven development thing."
• His advice: "What you need to do is invest in more developers, more engineers to build the stuff, less so in your CTO, but have someone there to guide the ship." One person splitting hands-on work and strategy does neither well.
• His own confession: "My thing has always been trying to avoid over-complicating things as much as possible. Don't jump on the fancy technology. I know I did that with Rails when it first came out."
• The framework marketing problem: For decades the performance industry has watched frameworks promise instant, and "when you dig into it, it's very much not instant." The hello world demo is four lines. "Whereas in reality, most businesses have to scale beyond that first 30 minutes of coding."
• Who to hire instead: People from adjacent industries who stay open. "You've got to avoid learned helplessness," the person who says this will never work because they watched it fail once under different circumstances.

5. Experience plus AI, and who fills the funnel behind it

He explains what LLMs do for senior people and what that leaves for everyone else.

• The trap: An LLM has a reasonable answer for anyone. "The tricky thing is that you don't know to ask certain questions if you've not had that experience." It answers with confidence and skips the clarifying questions a senior would have asked.
• The multiplier: "Having people who have deep experience with something and then giving them AI is a superpower." They know how to cut the noise and how to correct the output, so the loop runs faster than convincing someone else to build it.
• The industry question: "How do we fill that funnel behind? We're not going to have this forever. People are going to retire and things move on."
• The hopeful data point: He had sat through a cohort of Cloudflare intern presentations in the days before we recorded, and what they shipped with mentors and managers around them impressed him. The new talent also challenges the seniors.

6. Letting go started with an outage during Dreamforce

I asked this one because I failed at it myself, and he said he had too.

• The admission: "One, I failed. It wasn't perfect. I made mistakes. I was too involved in the beginning."
• The moment: He was in San Francisco speaking at Dreamforce, shipping code while it was nighttime back home, and something broke. People got woken up at the wrong hour for UK clients. That was the point he accepted it was not sustainable.
• The mentor: His business partner was older, had founded a digital marketing agency and scaled it to around 150 people. "Without that things would have gone a lot worse." As a fractional CTO he later offered new founders an hour a month of free mentorship for the same reason. "Expecting everyone to be perfect on their first go is not feasible."
• The mechanism: Guardrails you are comfortable with. "If you as a team want to go and use some whole new technology, a new programming language, that's a big thing for a company. So that's a decision that needs to involve me. If you want to use some new Ruby library, as long as it falls within this criteria, crack on and do it."
• The cadence: One week sprints, with meeting overhead held to about three hours a week. "Comparing that to most organizations, it sounds like a lot, but it's not." Every week he sat in one team's demo and watched the recordings of the others, async, including demos of technical decisions.
• Why it works: "You can't be in four places at the same time."

7. The woman who only did CSS

A side quest from university shaped how he builds teams.

• The setting: A big consulting firm invited the top students from each university to a competition. At dinner they spaced their employees between the students. He asked the woman next to him what she worked on.
• The answer: "I do CSS." He asked what else. "No, no, I just do CSS." She did not touch HTML or JavaScript, and the fastest anyone moved roles was every six months.
• His reaction: "CSS is like 2% of my day. I was like, this would not work for me." He did not pursue that career.
• What he pushed instead: T-shaped people, broad skills with deeper expertise in a few areas, long before the team adopted Scrum. If someone is off sick, somebody else can cover the work, even if they need reviews.
• How Cloudflare does it: Open a pull request on something outside your area and ask that team for reviews, or put out an RFC saying you want to build a thing you are not an expert in and want a group together. "You chuck things over the fence" is the model he wanted to avoid.
• Why it helps the leader: More people you can trust on more areas, sharing context among themselves instead of routing it all through you.

8. Resist process until someone names the reason

Shift ran lightweight Kanban for years before it needed anything more.

• The default: Lightweight Kanban for a long time, with process added only when they got to the point of needing it.
• The incident reflex: "If you've seen an incident, it's easy to be like, okay, let's bubble wrap everything with all this process so it never happens again." The result is frustration and a business that fails because it cannot move forward.
• The rule: "Pushing and pushing and pushing until someone says, no, we have to have this, and here's the key reason why."
• Standups: Kept short, no waffling about what everyone did. Today you can automate the end of day notes with AI and keep the morning focused on what is slowing you down.
• The leader's job: "You need that feedback loop and you need to care. You can't look at the numbers and be like, okay, this is all fine. You've got to interrogate them." Get into the weeds occasionally so you can speak the same language as everyone else.

9. The whole company trained with the founder of Scrum

When Shift did adopt Scrum, the sales team trained alongside engineering.

• Where it came from: A team lead brought a tiny pocket book on Scrum, around 50 pages, and said read this. Ryan read it, then decided that if they were adopting it they would learn it from the source.
• The trip: They went to Stuttgart and trained with Jeff Sutherland, "so we could hear from the horse's mouth exactly what his mindset was in speccing this up." The whole company did the official training, sales team included.
• The resistance: People had been at companies where Scrum failed. He framed it as an experiment that might not work, and a starting point they needed anyway.
• What leadership usually wants: Standing desks and agile coaches. "They don't want the things that Scrum brings you. They just want to go faster." The staff come back with a list of impediments and leadership says that was not part of the deal: "I'm here to buy the tables and bring in the agile coaches. I'm not here to deal with your problems."
• The surprise: "The sales team were the best adopters of it. They had a whole wall of Post-it notes and all sorts of stuff going on."
• The first impediment: CI took 40 minutes to run the full suite. They switched platform and got better infrastructure, which brought it to about 15 minutes, then improved the tests themselves and got the whole suite to around three minutes. "It's not complex stuff to make things work."
• The thing you avoid: "What you don't want is to wait three months to hear that everything's behind."

10. How his team at Cloudflare runs now

A reorg six weeks in, and a team he mostly sees in the afternoon.

• The shape: He is a product manager who still does some hands-on engineering. His engineers are mostly on the US East Coast, so he gets quiet mornings and meetings from the afternoon into the evening, with customers across the globe.
• The reorg: It landed six weeks into his job and threw "this is how we work, it's worked like this for seven years" into the air. He sees that as good for the company.
• Knowledge sharing: One of his team runs a monthly web performance brown bag focused on what is upcoming or in flight. Standups run about 10 minutes a day. Weekly company-wide meetings let anyone present, which is how he keeps up with a product range he says you cannot keep up with easily.
• The goal: As many people as possible working on performance product work rather than maintenance and firefighting, balanced against capacity planning and the rigor around security and paging. "There's a healthy tension between those things and it's a group effort to figure out how we do that."
• Negotiable scope: Scrum recommends the INVEST criteria for any piece of work, "and that's the N that stands for negotiable. And that's a big point." His question is always how to cut scope and ship part sooner: "how can we get 80% of there in 20% of the time." Product cannot say "this is everything I want, figure out how to deliver it."
• Prototypes as communication: AI lets them put an internal proof of concept together quickly instead of an A4 sheet of requirements. Visualizing it removes some of the problems before anyone argues about them, and it helps people who read diagrams better than documents.

11. You can't optimize what you're not measuring

Asked what Cloudflare can do for your site, he names his own product first and admits that is selfish.

• The principle: Without data you will "reduce something from one millisecond down to half a millisecond, whereas there's something that's like 100 milliseconds sat over here. So you don't move the needle."
• The product: Real user monitoring, also called web analytics. "It's got two names in the dashboard. That's something I still need to sort out." A script runs in people's browsers and reports back, collecting the Core Web Vitals, which measure perceived performance from real people.
• The price: "You can use it completely for free. Whether you're a Cloudflare customer or not, you can set up a free account and you can copy the script and put it on your non-Cloudflare website." On Cloudflare, you turn the feature on and it gets injected.
• Synthetic tests: Also his remit, browser tests and single URL tests, with the caveat that they may not represent real users. "Your laptop might be a brand new M5 Max MacBook Pro or whatever with 128 gigabytes of RAM," while real users are on a 10 year old Android phone in low power mode and low data mode.
• What the data points you at: Slow image loading, which the image compression services address, or a cache hit ratio lower than it should be, which tiered cache addresses by rerouting a miss to another data center before going to origin.
• The rest: Traffic shaping tools and early hints that tell the browser ahead of time what it will need, with many features available on free accounts or around $5 a month. Six months in, "I'm still discovering new things on a daily basis."

12. The personalization script that promised no performance impact

A vendor told one of Shift's customers their script had no performance impact.

• The claim: A customer wanted a personalization tool. "This personalization tool said that there was no performance impact to running their script on the site." He questioned it before it was installed.
• The install: A blocking JavaScript file in the head, so nothing on the page could render until it downloaded and executed.
• Why nobody noticed: For the P50 visitor on a fast connection with a powerful new iPhone, the overhead was 50 to 100 milliseconds. "But beyond that, it was trashing performance."
• The absurd part: All of the personalization happened below the fold, things like previously viewed products. "So we were holding back rendering for no reason." For a first time visitor with no history, "that script is going to load basically nothing, but it was blocking render."
• The fix: Analysis, then convincing the company to load it async further down the page. "It was only say a hundred milliseconds or something at the P50, but you go to P90 and it was like seconds of time."

13. Cache the anonymous page, fetch the basket

Shift ran over 99% cache hit ratios on pages that had logins and baskets on them, and this is the split that made it work.

• The split: A product detail page in its anonymous form can be cached at the edge for a long time, until the product changes. Login state and basket come in through a lightweight bit of JavaScript. "Fire off a fetch request for some HTML, dump it into the mini bag, simple as that."
• Cache the fragment too: The mini bag HTML is cached as well, keyed on a cookie, so it only re-downloads when someone's basket changes. Put a protection mechanism in front of it so nobody is served someone else's basket.
• Why the document comes first: "TTFB, nothing happened before you've got the HTML document, right?" Early hints aside, the browser discovers most of the page from the HTML, so the fastest win is making that document as cacheable as possible, even if a few bits load client side.
• The edge version: Workers can take a cached page and apply customization at the edge, "combining two cached pages together rather than going back to origin all the time," which gives you fully server rendered pages, mini bag and login status included.
• The number: "Over 99% hit ratios when we were running Shift."
• Inventory tiers: Stock dropping from 999 to 998 does not change the page, since most sites do not render the number. Shift regenerated a page only when stock crossed a tier, like dropping below 10 or going from one to zero. He picks the tiers by asking what the user needs to see.

14. Black Friday on Heroku, and why caching came second

I spent five months preparing for Black Friday at an e-commerce company once, so I had to ask.

• The early years: Before they had made all the mistakes they needed to make, Shift threw infrastructure at it, since the whole platform ran on Heroku. "We could call them up and be like, right, we need a cap of a hundred powerful servers so that we can scale it all the way up for the day or two." Paying for what you use meant the overall bill barely moved.
• The correction: "Caching is a point, but that's almost like a plaster over the top." The real work is making the process efficient.
• Where to look: Where your application spends time, measured as "your slowest page is multiplied by their throughputs effectively." Get those down and a cache revalidation returns quickly anyway, which makes caching matter less, though it stays important at large scale.
• The combination: "Doing the groundwork to optimize things and then also doing the work to protect as well."

🧠 What I Learned

• Rails earned its place at Shift because a .NET team could not deliver fast enough, and billions in annual throughput later he still calls the "Rails doesn't scale" line a myth.
• A former developer promoted to CTO will often spend the role on their dream CV, so put the money into engineers and keep one person steering.
• Deep experience plus an LLM compounds, because experienced people know which questions the model forgot to ask.
• Letting go means guardrails you can live with, where a new language needs his sign-off and a new library does not, plus a weekly demo you can watch async and meetings held to about three hours a week.
• Hire and grow T-shaped people, since someone who only does CSS cannot cover for anyone, and a leader cannot step back from a team where every piece of context routes through them.
• Resist process until someone can name the reason for it, and never bubble wrap the company after one incident.
• If you adopt Scrum, train everyone from the source, sales included, and budget for the impediments list that comes back. Shift's first item took CI from 40 minutes to around three.
• Scope is negotiable by design. The N in INVEST exists so product and engineering can ship 80% of the value in 20% of the time.
• Synthetic tests on an M5 Max laptop miss what a 10 year old Android on low power mode experiences, so measure real users first. Cloudflare's RUM is free whether or not you are a customer.
• A "no performance impact" script blocking in the head cost 50 to 100 milliseconds at P50 and seconds at P90, for below the fold content that first time visitors never saw.
• Split the anonymous page from the mini bag and cache both, with the fragment keyed on a cookie. Regenerate only when inventory crosses a tier and 99% hit ratios are reachable on a personalized store.
• Caching is a plaster for Black Friday. Multiply the slowest page's time by its throughput, fix the generation there, then protect what is left.

💬 Favorite Quotes

"I was that cousin everyone has who does websites and everything."
Ryan Townsend

"The whole Rails doesn't scale thing was always a myth anyway. Annoyingly, it stuck as this marketing phrase."
Ryan Townsend

"I feel like the experience I had picking up Rails is what everyone's having now with AI."
Ryan Townsend

"The tricky thing is that you don't know to ask certain questions if you've not had that experience."
Ryan Townsend

"CSS is like 2% of my day. I was like, this would not work for me."
Ryan Townsend

"They don't want the things that Scrum brings you. They just want to go faster."
Ryan Townsend

"You can't optimize what you're not measuring."
Ryan Townsend

"Caching is a point, but that's almost like a plaster over the top."
Ryan Townsend

🎯 Also in this Episode

• The coffee with an ex-employee now senior at Shopify who told him "you literally built a competitor to us" and tried to recruit him, before remote hiring policy outside North America got in the way
• Eighteen months of fractional CTO and web performance consulting between liquidating Shift and joining Cloudflare in March
• Why he describes performance as the hobby he took seriously for two decades without ever holding a job with performance in the title, until now
• Mentoring new founders for an hour a month, free, as a way of giving back to the industry that gave him a mentor when he needed one
• How Shift later folded Shape Up from 37signals into its Scrum process once the framework existed
• His open offer to point anyone with a performance question at the right person or team inside Cloudflare
• His book recommendation: The Phoenix Project, which he calls one of his favorite books of all time because it frames blown timelines and factory floor bottlenecks as stories

Resources

More from Ryan:

• Ryan Townsend on LinkedIn
• Cloudflare Web Analytics and real user monitoring: free to use, on or off Cloudflare

Tools & concepts mentioned:

• Cloudflare Web Analytics / RUM: collects Core Web Vitals from real users' browsers
• Cloudflare synthetic browser and URL tests
• Tiered cache: reroutes an edge miss to another data center before origin
• Early hints: tells the browser what it will need before the HTML arrives
• Cloudflare Workers for combining cached pages at the edge
• Core Web Vitals and TTFB
• Ruby on Rails, with Delayed Job, Resque, Sidekiq, Good Job and the Solid libraries for background work
• Shopify Build a Business competition
• Heroku, the platform Shift ran on
• Scrum and the INVEST criteria, Kanban, and Shape Up from 37signals
• Jeff Sutherland's Scrum training

Books mentioned:

• The Phoenix Project: his number one recommendation for any engineer or leader

🎧 Listen Now

🎧 Spotify
📺 YouTube
🍏 Apple Podcasts

Episode Length: 52 minutes on edge caching, cache hit ratios, the script that wrecked P90, measuring before optimizing, letting go as a CTO, and getting through Black Friday without leaning on the cache.

Whether you have a traffic spike on the calendar or a vendor telling you their script has no performance impact, this one has the numbers and the stories behind them.

Happy caching,
Dan