We launched the Metacast podcast player for iOS and Android in open beta last week. What a couple of weeks those were… In this newsletter, we document the whole process.
🎙️ You can also listen to Metacast: Behind the scenes ep. 53 to hear our discussion of these topics on Metacast, Apple Podcasts, Spotify, or watch on YouTube.
Pre-launch
We wanted to launch on Tuesday, February 6, 2024. Episode 52 of the Metacast podcast was coming out on Wednesday, so we thought it’d be great to tie the launch to the episode.
The reality had different plans.
Audio player broke on Android
We knew app had some intermittent audio failures on iOS, which we deemed acceptable for the beta. Just before the launch, Jennie, our only full-time Android user, discovered that audio playback was completely broken on Android. It was pointless to launch a podcast app that can’t play audio, so we had to put the launch on hold until we fixed it.
The breakage of audio was caused by an upgrade of dependencies and a refactor of audio playing. A bunch of things happened at the same time, and we didn’t thoroughly test for regressions.
AppCheck broke the app
We enabled AppCheck to protect our Firebase backend from unauthorized access. Arnab and I did this on a call together. Click-click in the Firebase UI… and the app broke on iOS, like really broke. It lost access to the backend and became non-functional.
The issue was caused by the AppCheck defaulting to the verification method that we didn’t configure. We disabled AppCheck, reconfigured it, updated the code, and rebuilt the app. The whole ordeal resulted in a few minutes of downtime and added a few hours to the launch timeline.
Note for our future selves
If one-off configuration changes can break the app, it’s cheaper to do them all before the launch (so we break fewer users or none at all).
Not having a dev environment is okay at an early stage. Don’t be afraid of testing in production.
Don’t forget to test on Android.
Launch
We had the app ready to go on Friday, February 9, 2024. We submitted Metacast v0.56 to Google and Apple, and it became available to end users.
Let’s look at how Apple and Google open beta testing programs differ.
Reviews
Apple reviews every single build we publish to beta. They first review it for our closed beta group, then for the open beta group. Reviews can take anywhere between a few hours to a couple of days. When we have urgent bug fixes, long review times are really annoying.
Google appears to do only programmatic reviews. We do see some automated activities — a bunch of fake-looking Gmail accounts log into the app and tap around. Without a human in the loop, our beta releases are near-instantaneous.
I’d prefer if Apple followed a fast track approach for beta releases too. On the other hand, I hope having Apple regularly look at the app during beta will make it easier for them to approve our public launch. We’ll verify this soon!
Installation
To install the app on iOS, users must have a link (we publish it on metacast.app) and open it via TestFlight, an Apple-owned app testing application. It’s a bit of a hassle for users, but it gives developers more control over the distribution of beta.
On Android, Google simply published our app to Google Play Store. Anyone can download it. We have no control over the distribution except limiting the number of users who can install the beta. We added BETA to the app title, so that users know they’re getting an app with rough edges.
Marketing
We knew about the rough edges, so we didn’t go all-in on marketing just yet.
On Friday (the day of launch), we posted on LinkedIn and Reddit. On Monday, we sent a newsletter. We got about 7k impressions on LinkedIn and just over 400 reads on the newsletter. This resulted in some downloads and usage of the app!
We still have a list of 300 emails of people who signed up on our landing page. Since it’s only 300 emails, I’m going to spend a couple of hours and send emails to them individually rather than use an email automation tool.
Companies like to send seemingly personalized “emails from CEO” that are sent by automations. They pretend to be personal while not being personal at all. I don’t like that fakeness and would rather spend a bit of time to make emails truly personal. We can afford that at our scale.
Metrics
We set up our business metrics pipeline a long time ago. It’s very simple — Firebase sends events to Google Analytics, which sends events to BigQuery. We can write SQL queries against BigQuery and get whatever reports we want.
We tried to use the free version of Looker as our dashboarding tool, but I found it to be too limiting and buggy. It feels like an abandoned product, so we ditched it in favor of Google Sheets. Sheets can use BigQuery as a data source for pivot tables. That’s all we need really.
Avoiding vanity metrics
Vanity metrics were first defined by Eric Ries in The Lean Startup. Vanity metrics are metrics that make you feel good, but don’t necessarily reflect the state of the business.
For us, defining active users as “someone who opened an app” would be a vanity metric. They opened an app, so what?
We define a daily active user as a user who played at least one episode. We also track how many users completed at least one episode that day and how many episodes were played and completed in total. We’re yet to define a north star engagement metric like “if they played X episodes within Y days, they’re a customer for life.” That will come later.
Feedback
We’ve created an ingenious approach to collecting user feedback. It’s rudimentary but clever. I’m quite proud of it.
Feedback comes from a variety of sources — Reddit, DMs in messengers (WhatsApp, Telegram, LinkedIn), and email. While you can easily share a link to a Reddit comment or forward an email, sharing the feedback received via DMs is a pain in the butt.
Process for capturing feedback
We have a Google Sheet where each row is “feedback” — it has a very brief description (5-7 words, like “Apple phones heat up”), how many users reported the issue, and links to those reports (in Reddit or Google Groups).
For any non-Reddit reports, we’re forwarding messages, emails, screenshots, etc. to a specially designated Google Group. Everyone gets those reports in their emails, but more importantly, the Google Groups has become the repository where all feedback is stored and is linkable.
This requires a bit of manual work but at our scale we can afford it. We don’t want to be that company that forces users into a company-preferred tool, or the one that never responds to messages. We truly listen and act on feedback.
Diagnostics is pain in the ass
Shipping a mobile app is akin to shipping a CD with software in the good ol’ days. Once it’s installed on the user’s device, we get some alerts for crashes and exceptions, but the information is not nearly enough to troubleshoot the issue.
Over time, there will be weird device-specific issues that we won’t be able to reproduce. To help with that, the first feature we built after the launch is a diagnostic mode. Users can enable the diagnostic mode to make the app save logs on the device, which they can send to us for troubleshooting.
It’s still day 1
With this launch, the shit got real. It’s just the beginning of a long journey.
Until we launch publicly, our priorities are (in order):
Stability of core functionality.
Finish features that we started (playlists, episode search).
Monetization.
Get Metacast podcast player for iOS and Android at metacast.app.
#13: How we launched the beta of Metacast