CI/CD at Scale with Marko Gacesa

🎧 New Señors @ Scale Episode

This week, I spoke with Marko Gaćeša, Head of Product at Semaphore, the agent-native CI/CD platform built for the entire software delivery workflow. Marko is a serial entrepreneur and technologist whose career spans developer tools, IoT, industrial automation and enterprise SaaS, with stops that include founding Dry Tools and serving as founder and Chief Product Officer at Alchemy Cloud. Most of his career was spent in domains — medical equipment, industrial software — where quality isn't a preference. He's also the first product guest I've had on the show.

In this episode, we dig into why testing has overtaken coding as the bottleneck, what "agent native" means once developers stop switching tools and live inside a coding agent, how Semaphore attacks flaky tests by feeding the agent enough data to find the pattern, how they run product experiments on infrastructure, why pricing a CI platform is harder than the cost-per-minute number everyone compares, and whether agents should be allowed to trigger deployments.

⚙️ Main Takeaways

1. Coding got amplified; testing didn't, and that's where the bottleneck moved

The thesis of the whole conversation, and it matches what I keep hearing from teams in Barcelona.

• What changed: "With the coding agents, this part of software development that is developing functionality was significantly amplified."
• What didn't: "As an industry we are not so good at testing. Testing is extremely hard, and it is still done mostly in a non-automated way. So I see the biggest bottleneck being testing."
• Where agents genuinely shine: Prototyping. "Now we can develop prototypes that are really clickable demos. You don't need Figma or something like that." But a clickable demo isn't production.
• The thing that actually matters: "You really want to have fast iterations in production with your customers and learn." That's what testing gates.
• His prediction: More people focusing on testing practice, in two dimensions — the body of knowledge, which is genuinely hard, and the shape of the pipelines themselves.

2. Pipelines were designed for a human's coffee break, and that's over

A small observation with a large consequence for tooling.

• The old rhythm: "Before, if you were developing something for two hours or three, and then your pipeline lasts for 20 or 30 minutes, then you're fine. You need a break anyhow."
• The new one: "If you're instructing agents to do changes, and you're probably managing multiple agents at the same time — three to five, depending on how much you can keep in your head — these agents will make changes in a couple of minutes and they will also like to have feedback in a couple of minutes."
• What follows: "I think pipelines are going to look significantly different." And not just the tooling — "multiple changes need to occur, not in the tool set itself, but in people's behavior."

3. Agent native means the agent is your user, and the developer is still your customer

The clearest definition of a term that gets thrown around loosely.

• What changed about the workspace: A year ago everyone described their setup as an IDE plus a few plugins. "Now most people's core tool is either Claude Code, Codex, Cursor." Developers used to switch between many tools to get work done; now "everybody is in their coding agent, and this is where they live."
• The consequence for a tool vendor: "We as a tool in the toolchain should be natively interfacing with an agent, enabling the agent to do its part in the best possible way."
• Why it's harder than a UI: "There's your tool, there's an agent, and there's a developer. Your developer is still your customer, but there is an agent in between."
• How testing changes: "You really need to run a lot of evaluations to see how does this all perform with actual Claude Code, with actual Codex. So the way we do testing has changed."

4. Flaky tests are the enemy of the trust the whole system runs on

The most practical section of the episode.

• What you're actually building: "You want to trust your test automation. You want to develop a high level of trust so that you're making a change, your pipeline is green, you push it. But the flaky tests are the enemy."
• The honest framing: "Test automation is still code, and there are bugs there. I could say that flaky tests are one kind of bug in test automation."
• What Semaphore does with it: Collect the data across runs, build a profile, identify which tests are flaky, and then push all of that context to the agent so it can look for a pattern.
• Why the agent needs the data: "The flaky test is — there is a bug, but you don't know what it is. It's typically some kind of race condition or a driver, or it really depends on the amount of memory you're using at the time. It is very hard to catch."

5. Speed comes from matching hardware to workload, not from buying the fastest machine

The two layers behind a faster build.

• Layer one: The application layer and how you set up your pipeline — a developer experience problem.
• Layer two: The infrastructure it all runs on. The trick is knowing what kind of loads developers actually run and what hardware characteristics those need.
• The constraint: "It's not only about finding the fastest machine out there, but finding the right balance between the cost and the speed." Faster machines cost more, and customers are buying a ratio, not a peak number.

6. The migration off GitHub Actions is now a skill, not a project

The answer to the switching-cost problem that used to protect incumbents.

• What used to block it: "Before the coding agents, that was a challenge." Nobody rewrites a working pipeline for fun.
• What changed: Semaphore's CI/CD expertise is packaged into agent skills, distributed as Codex and Claude Code plugins that update automatically. "One key component of SemAI is actually migrating from GitHub Actions to Semaphore."
• Why it matters more than the marketing: "You can put a lot of nice things on your website and they can be credible... but the most important thing for people is that they experience it themselves." If a 30-minute pipeline runs in 15, they need to see that on their own repo.
• What it replaces: "Before that you had to go click, create an account, learn a little bit about how do I configure, or maybe interact with our team. That is very hard for developers." The onboarding is now automated end to end.

7. Strongly opinionated users are the best users

His philosophy on building for developers, borrowed from a book he makes new teams read.

• The required reading: Hug Your Haters by Jay Baer. "For the teams that I work with, especially if I'm creating the team from scratch, this is a must read before we start to work very closely."
• Why developers are a good audience: "People who are strongly opinionated are your best users because they will tell you where you are exactly." They're candid, involved, and they tell you what they like and dislike unprompted.
• The product job that remains: "It's your job to compile that. To really see what is under the hood, what is the actual meaning of what they are saying."
• The catch: The bar is brutally high. "But for me, I love hard problems. I have a bias for hard."
• The one real difficulty: Developers hate getting on calls. Coming from chemical engineering, where "you could spend days speaking with customers one after another," he had to adjust — and leaned instead on usage data and support tickets, which reveal not just the pain but the environment the person is working in.

8. Run experiments on infrastructure with feature flags and volunteers

How you validate a hypothesis when the product is a build system.

• Start small: "You start with feature flags."
• Use the customer history: Semaphore is an established company with a long-standing customer base, not a startup, "so we know exactly which customers are inclined to participate in which kind of experiments, and if that's a pain that is relevant for them."
• The approach: Reach out to those customers, say you're solving for a pain you already know they have, flip the flag, and iterate with them directly.
• The epistemics: "I know that I'm not right. I know that I'm wrong, I just don't know how wrong I am. So you need to run experiments in order to prove your hypothesis and see if it works or not."
• On watching competitors: He gets a weekly digest, but he isn't copying features. "I'm actually looking into what kind of developer experience they were trying to achieve with this. And does that align with my beliefs as well?"

9. A minute of CI is not the same minute everywhere

The pricing section, and the sharpest insight in it.

• The comparison everyone makes: Cost per minute. "But the minute of CI is not the same for each and every platform."
• The number that matters: "It's actually about the cost of the pipeline — how much it costs to run a pipeline on that particular platform." That's what Semaphore publishes in its benchmarks alongside duration.
• Why it's hard to compute: "It's not only about that machine that is running, but you have the whole application, you have networking, storage." Tricky, but possible.
• The method: Calculate the raw cost, optimize it, then add a margin that funds the work.
• What the margin funds: Semaphore is fully open source, so you can self-host it the way you would Jenkins. "The reason we are running our hosting environment is to make it more convenient to developers, but also to finance that open source development."
• The go-to-market difference: Semaphore is product-led and self-service, unlike the B2B motion he ran at Alchemy Cloud, where a head of product sits in sales calls designing relevant demos.

10. Whether agents should trigger deployments is a question about your test suite

A refreshingly non-ideological answer.

• Their strategy: "We are the tool for the agent, and we provide all the tool set that the agent needs. And then it's really on the developer to say what kind of access rights or what kind of capabilities of a tool are going to be leveraged." Access rights are granular by design.
• The deciding factor: "If you have proof points that you have test automation in place that is not flaky, that has very high coverage — I don't see a reason why not do it."
• And the inverse: "If you have a high percentage of flakiness, if your coverage is maybe not that high, well then it makes no sense."
• The reframe: "At the end it's just another layer of automation. And with every automation, it depends on how confident you are in your automation."
• Which lands back on the same point: "That's the biggest bottleneck at the moment. How confident you are in your test automation suite."

11. In dev tools, your engineers are the domain experts — so the product team stays small

The organizational insight I hadn't heard framed this way before.

• How it works elsewhere: In chemical or financial software you need a strong product team with deep domain expertise, technical enough to talk to engineering, translating pain points into PRDs and software requirements. "The product team is more of a layer that is validating what's being done."
• Why dev tools are different: "You already have domain expertise and domain experts as your engineering team, because they are dogfooding and they are using the product on a daily basis."
• What the product org becomes: "Your product organization can be very lean and very focused on the experiences, how do we market that, how do we think about pricing, what kind of margins are needed for us to be sustainable. The product team is much more acting as a glue."
• Where features come from: "Most of our features are coming as proposals" from engineers. He sets high-level strategy and the experiences to deliver; the team owns the requirements and the implementation. SemAI's entire shape came from the engineers.
• Why he likes it: "Developers are the ones that have access to enabling technology, and they know exactly what they need to do and what the right developer experience is. In other domains, that's typically a problem you need to tackle inside the product organization."
• How features scale across personas: Progressive disclosure. Compliance and security are designed in from the start because enterprise customers need them, "but that doesn't mean that the customer that is not there should be bothered with it."

🧠 What I Learned

• Coding agents amplified the writing of functionality and left testing untouched, which is why testing is now the constraint on fast iteration.
• Pipelines were implicitly designed around a human taking a break; agents want feedback in minutes, so pipeline design has to change.
• Agent native means the coding agent is now the interface, so a tool has to serve the agent while the developer is still the customer — and you validate it with evals against real agents.
• Flaky tests destroy the trust the whole delivery model depends on, and they're just bugs in test automation: race conditions, drivers, memory-dependent behavior.
• Fixing flakiness at scale means collecting enough run data to profile it and handing that context to an agent to find the pattern.
• Build speed is a hardware-to-workload matching problem balanced against cost, not a race to the fastest machine.
• Packaging migration expertise as an agent skill removes the switching cost that used to protect incumbent CI platforms.
• Opinionated developers are the most valuable users; they just won't get on a call, so usage data and support tickets carry the load.
• You can run real product experiments on infrastructure with feature flags plus customers you already know have the pain.
• Cost per minute is a misleading comparison — cost per pipeline is the honest one.
• Semaphore is fully open source and self-hostable; the hosted product funds the open source work.
• Whether you let agents trigger deployments is entirely a question of how much you trust your test suite.
• In developer tools your engineers already are the domain experts, so the product org can stay lean and act as glue.

💬 Favorite Quotes

"As an industry we are not so good at testing. So I see the biggest bottleneck being testing."

"Before, if you were developing something for two or three hours and your pipeline lasts 20 or 30 minutes, then you're fine. You need a break anyhow."

"Everybody is in their coding agent, and this is where they live."

"Your developer is still your customer, but there is an agent in between."

"You want to trust your test automation... but the flaky tests are the enemy."

"It's not only about finding the fastest machine out there, but finding the right balance between the cost and the speed."

"People who are strongly opinionated are your best users because they will tell you where you are exactly."

"I know that I'm not right. I know that I'm wrong, I just don't know how wrong I am."

"The minute of CI is not the same for each and every platform."

"At the end it's just another layer of automation. And with every automation, it depends on how confident you are in your automation."

🎯 Also in this Episode

• My own year of GitHub Actions outages, including stretches where we couldn't deploy for days, and why a competitive CI market matters
• Why Jenkins is still the largest CI tool in the world, and why only a fraction of all software being written touches CI/CD at all — so there's room for everyone
• The cycle he describes: Jenkins and Travis, then GitHub Actions taking the market, then developer experience shifting again and opening the door back up
• Semaphore dogfooding itself — every feature ships to the team as its first customers
• Why he moved from chemical engineering and industrial automation into developer tools, and the similarities between chemists and developers as candid, technical audiences
• Placing bets in a fast-moving space: knowing some of what you build will be obsolete, without knowing which part
• His read on where the load goes next — more software built per person means more pipelines, so scalability and turnaround time are the infrastructure priorities
• Book recommendations: Hug Your Haters by Jay Baer as required reading for his teams, plus everything from Marty Cagan and the Silicon Valley Product Group

Resources

More from Marko and Semaphore:

• Semaphore — The agent-native CI/CD platform
• Marko on LinkedIn
• Semaphore on GitHub — Fully open source and self-hostable
• Semaphore on X
• Semaphore on LinkedIn
• Semaphore CI/CD benchmark — Duration and cost per pipeline, not per minute
• SemAI — The skills, plugins and agent interface discussed throughout, including the GitHub Actions migration

Books mentioned:

• Hug Your Haters by Jay Baer — Required reading for his teams
• Inspired, Empowered and Transformed by Marty Cagan (Silicon Valley Product Group)

🎧 Listen Now

🎧 Spotify
📺 YouTube
🍏 Apple Podcasts

Episode Length: 45 minutes on testing as the bottleneck of the agent era, what agent-native CI/CD means in practice, flaky test analysis, pricing a platform where a minute isn't a minute, and why dev tools product teams can stay tiny.

Whether you're fighting a flaky suite, weighing a move off GitHub Actions, or trying to work out what product management looks like when your engineers already know the domain better than you do, this one is worth your time.

Happy shipping,
Dan