A Learning a Day

Notes on LLM RecSys Product – Edition 4 of a newsletter focused on building LLM powered products.

---

Quick recap: We’ve covered the central thesis that LLM recsys is the core primitive of AI-native products, how teacher models enable painful self-awareness, and how the eval loop (not just evals) drives systematic improvement.

The eval loop only works if your teacher model knows what “good” looks like. That requires rigorous product policy.

What Product Policy Is

Product policy is the crystallization of the product team’s intuition. It’s the best understanding of what a great user experience looks like.

PRDs describe deterministic features we want to build. Product Policy, on the other hand, defines behavior to exhibit – and gets encoded into the teacher model, the production models, and the eval loop.

You won’t know if your definition of quality matches what users actually value until you test. If your true north metrics – typically laddering into user retention or end outcomes – improve, your intuition was right. If not, you refine. The Product Policy evolves with user signal.

Policy Encodes Judgment

For small products, the Policy could be written entirely by one author.

However, for large products, you typically need multiple contributors who will need to debate every gray area (of which there will be many). These debates matter because policy decisions cascade through the entire system. They define what you measure, what counts as “good,” what users see.

This is where product leaders must be hands-on. You can’t delegate the constitution. Your understanding of the user and your judgment must show.

One note – even in cases of complex policies, I would recommend having one author so your Policy reads as coherent. It forces the entire team to align to one point of view vs. “you take that section and I’ll write this.”

The Rubric

Policy lives in a rubric. This could be a binary 0/1 or a more sophisticated graded rubric. Here’s an example – let’s imagine you’re an e-commerce product team and are building out the policy for a product query.

You look at the most common product queries and pick a popular one – “toaster”

Let’s explore what the rubric might look like. Here’s an example –

Next, assuming the rubric feels right, we might make a policy decision – show only results rated 3-4. Filter out 0s and 1s and show 2/Fair with a different Ux (e.g., “Related”).

This in turn means toaster ovens are out – even though they toast. This is judgment made operational. And it cascades through millions of queries.

The Golden Dataset

The golden dataset brings the rubric to life with examples. A complex policy might need a golden set of 500 or so examples to bring the various items in the judgment to life

Not just “this is a 3” but “this is a 3 because…” – high-quality examples with detailed chain of thought.

The golden dataset evolves over time as you learn from user signal. What you thought was a 3 might become a 2. What you filtered out might need to be included.

The author of the policy should drive the golden dataset process – ensuring consistency, adjudicating disagreements, maintaining the chain of thought quality.

The Two Gates

Policy quality is measured through two gates:

Gate 1: Is the policy clear?

Have multiple raters score the same examples using your rubric. Measure inter-rater agreement using Cohen’s kappa.

Cohen’s kappa measures agreement beyond chance – because even random guessing produces some agreement. (Here’s the Wikipedia article and the Scikit implementation.)

The interpretation:

• κ > 0.8 = strong agreement (policy is clear)
• κ 0.6-0.8 = moderate (policy needs refinement)
• κ < 0.6 = weak (policy is ambiguous)

If your raters can’t agree whether a result is a 2 or a 3, the policy isn’t clear enough. Sharpen definitions, add examples, debate more.

Gate 2: Does the model encode it?

Once Gate 1 passes, measure how well the model’s judgments match your golden dataset. Same metric: Cohen’s kappa between model output and human-labeled examples.

Gate 1 tells you if humans understand the policy. Gate 2 tells you if the model does.

One Policy, Not Many

A common mistake: architecting your system with multiple policies for different stages – e.g., one for ranking, another for filtering, another for personalization.

The problem: every model is imperfect. If each stage operates at 80% quality, your end-to-end experience is 0.8 × 0.8 × 0.8 = 51%.

Three “pretty good” stages compound into a mediocre experience.

The better approach is to build one unified policy. Define what great looks like end-to-end. Train one teacher on that complete experience. Let the production stack learn from that unified signal.

This is harder to build. But it’s the only way to avoid compounding errors through your system.

---

Next up: The eval loop in practice. How teams actually run it, what metrics matter, and when to invest in policy versus when to keep it simple.

One of the things that’s fascinating about the recent conversation about AI’s impact – in the media and across the internet – is that I think we’re showing a certain proclivity for what a friend hilariously termed “doomporn.”

This is similar to the love for “hustleporn” that used to be so prevalent – where everybody was talking about how hard they were hustling and working.

Doomporn has similar characteristics. There’s a lot of love for articles and points of view that forecast doom. And these seem to not just capture imagination but also ripple into the markets in ways that are fascinating to watch. We can name any number of these articles in the past weeks.

Who’s right? Nobody knows.

And that’s one of four truths worth reminding ourselves of –

First, AI is going to bring negative impact along with positive impact – like every major technology. The internet connected all of us – but it also polarized all of us. Television and video gave us access to internet – but it also isolated us. Name the technology and you’ll see accompanying negative impact that is proportional to the positives. AI will be no different. It definitely behooves us to be thoughtful about the impact (e.g., Noah’s post).

Second, nobody knows the future. Everybody is guessing and placing bets. It’s possible some people’s bets are going to be better than others. But it’s certainly hard to call right now.

Third, everybody’s talking their book. This is such a massive financial story that there’s a lot of financial interest no matter what the point of view is. It’s natural for everybody to be biased toward their financial interest and their bets – because they want their bets to succeed.

Finally, while there’s a lot of talk of doom, and who knows, there might be a non-zero chance the threat is existential in the next decades – the truth is this is a moment of change. And the best way to deal with change is to embrace it.

If you’re in the workforce, the message is as clear as ever. Use the AI tools and make them part of your workflow.

The doomporn conversations might be annoying. But let that not bury the lede – the change is real. And we’re better off embracing it.

We were at a somewhat remote place for a few days recently and had rented a car. We ended up buying enough groceries to last us a few days and had enough food throughout – so right at the end of the trip, we took the rental car and drove straight back to the airport.

Our kids pointed out that we didn’t end up using the car at all. Maybe we shouldn’t have rented one. Maybe we could have taken a taxi for example.

The conversation we had then was about the fact that the car had brought us optionality. If we hadn’t gotten the right amount of food or groceries, we knew we could always drive out of the remote location, get to a grocery store, and make it back.

The meta lesson – optionality always costs something. This is obvious when we buy refundable tickets or accommodation anywhere we go. However, it is less obvious when the costs are hidden. For example, if we’re choosing a path that gives us optionality in our career, that’s totally fine. But there’s a cost to doing so vs. say specializing.

It isn’t that optionality is right or wrong. It’s just important to be thoughtful about when we choose to pay that cost – and when we choose not to.

It felt fitting to wrap up my career transition series and share the update I’d shared on my LinkedIn. After a month of catching up on my backlog – across National Parks and career transitions, we’ll be back to regular service tomorrow.

---

Update – I’m joining PayZen to lead the product team.

I’ve always been drawn to acute problems – those that are massive in scale and deeply personal in impact. After a decade working on the job search problem, I was drawn to healthcare affordability.

We’ve all experienced this: when there’s a health problem, everything stops. And yet affording the care they need is top of mind for millions of Americans. It’s so large, so complex… and yet so important.

What drew me in was what the team – led by Itzik and Tobias – has already built. Starting with payment plans and financial tools for patients and health systems, they’ve had real impact on patients and care providers. We all know healthcare affordability is a much larger problem – I believe we’re just getting started on what’s possible.

I’m grateful to Itzik for the opportunity – and it is all the more meaningful that this next chapter starts with Sanjeev, the person who took a bet on an eager grad student who dreamed of making it to the Bay Area and hired me into LinkedIn. Some things come full circle in the most wonderful ways.

PS: Healthcare is complex and I have a lot to learn. Alongside my notes on building LLM-powered products, do expect to see a lot more notes about the US healthcare system and the lessons learned along the way. :-)

#careertransition – #8 of 8 | I recently went through a process of career exploration and made a significant career switch. This series of 8 posts is a synthesis of the lessons I’ve learnt.

---

As you navigate your transition, you’re going to meet a lot of people with a lot of strong opinions on what you should do. What’s right for you. What you’d be crazy to pass up. What you’d be crazy to take.

For what it’s worth, these opinions won’t be limited to the choice you make – they’ll even go down to something as seemingly benign as how much time off you take between roles. :-)

And many of these opinions will not make sense for you in your situation.

One of my go-to examples of really horrible career advice came from someone incredibly successful more than a decade ago. We spoke when I was living outside the US and really wanted to work in the Bay Area.

This person’s advice was simple – get to the Bay Area, spend a few months meeting as many people as possible, and convince somebody to give me a job. He said he knew a couple of people who did it as well.

This was high conviction advice from a person who had seen a lot of success in their career. And yet for somebody like me who didn’t have a US passport and who had just got married, it made little sense. What are the odds that this even works out? What happens to my wife and her career?

Since making the decision to leave my last role, I’ve spoken to many people – some of whom were upfront about wanting to leave too. A few of them, also immigrants, were upfront that it might be the right time from a career standpoint but not from an immigration standpoint. And I completely understand.

A couple of good friends went back to India for a routine visa stamping. At the last minute due to changes in guidance from the state department, embassies canceled appointments and rescheduled them many months later. For some, jobs were at risk – because you can’t be away from the office that long. A couple of friends had to find schooling for their kids for three or four months while they waited.

In another case, just as a wife found her way back to the US after months of waiting, her father-in-law passed away. Her husband, who was in the US all this time, made the gut-wrenching decision not to attend his father’s funeral – because who knows, he could be stuck there for months as well.

Can you imagine making such a decision? To not say goodbye to a parent because you’re worried about what it might do to your life? And yet I know of many who’ve made these decisions. During COVID. And right now.

I tell these stories because people who haven’t gone through this have no idea what it’s like to face such situations. And it’s so easy for them to give advice with high conviction without this understanding.

During times of transition, you’ll find yourself in situations where you feel like you need to explain yourself. You don’t. Nobody fully understands your circumstances. Only you do.

If you’ve given it thought, you’re making a bet. Make the bet. It may or may not work out. But it’s your life to mess up and your call to make. You have to do the work to reason from first principles and make a choice that you will be at peace living with.

Because ultimately, you are the one who’s going to have to live with that reality.

#careertransition – #7 of 8 | I recently went through a process of career exploration and made a significant career switch. This series of 8 posts is a synthesis of the lessons I’ve learnt.

---

Take the time to say goodbye. Meet anyone who wants to meet you. And make sure to take the time to meet people you want to thank. A simple thank-you note takes effort but goes a long way.

It helps to remember that the most important question in the mind of the person you’re meeting is “what does this mean for me?” That tends to be the central theme of such conversations. Keeping this in mind helps you be there for them.

Sometimes people react in ways that might seem unsettling or unkind. There’s no need to read too much into it. When you make a change, it gets people to reflect on where they are. Their response is frequently more about their headspace than about you.

When you have these conversations, focus on the good times. It’s easy to leave feeling bitter or sounding bitter. You might get the opportunity to share feedback. Do it as authentically as possible. But the goal is not to prove a point to anybody.

Multiple things can be true. You might have had a good run. You might have made meaningful connections. And it might be time for a new adventure. For you. That’s the key.

This is where internal clarity helps. When you’ve made a decision, be clear in the truth of why you made it and why it’s the right one for you. That clarity will help you shake away any unpleasantness and stay focused on doing the right thing. The goal is keep these relationships. It’s a small world and you never know when your paths will cross again.

Perhaps more importantly, in the final analysis, the memories and the people are all we have.

#careertransition – #6 of 8 | I recently went through a process of career exploration and made a significant career switch. This series of 8 posts is a synthesis of the lessons I’ve learnt.

---

This applies on both sides – your current employer and your new one.

Let’s say you’ve run a process. You have a couple of offers. Or you’ve decided where you want to go and have gotten an offer. But your existing employer has also opened up an option or two that could be compelling.

The key here is to go back to that first principle of knowing what you want. Don’t negotiate if you know you’re not going to take it eventually. This is one of those things you really want to play well. If there is indeed something that would keep you, be upfront about it – but know that if it’s given to you, then you have to stay.

And on the flip side, if there’s nothing that’ll make you stay, don’t make asks. They’ll just end up wasting a bunch of people’s time and goodwill.

Your most helpful ally is being clear about what you want and what you’re solving for. And assuming that’s not something you’re going to get in your current place, you can just be upfront and treat any negotation or persuasion as a sign of love.

Some of these conversations can be hard – especially when you make an unexpected move. But it’s important to remember that you don’t need to justify it to anybody. It’s a call that makes sense to you, and that’s what matters. The key is having the conversation as respectfully as possible.

Similarly, at your new place – the best negotiations are simple and focused on one or two things that matter. Not attempting to throw the kitchen sink at the problem. Doing it respectfully matters because you want to start on a great foot.

And, in addition to speaking to friends who’ve been through similar situations, I can’t recommend using Claude or ChatGPT enough to prepare extensively for these conversations. It really goes a long way in preparing for sensitive conversations.

#careertransition – #5 of 8 | I recently went through a process of career exploration and made a significant career switch. This series of 8 posts is a synthesis of the lessons I’ve learnt.

---

We spend a lot of time trying to get beginnings right. It makes a ton of sense to figure out how to get the ending right too.

Imagine you’ve made the decision to leave. That’s a big decision. An important decision. It’s important to time it well if you can.

This matters more the longer you’ve been somewhere. You want to leave on a high because it should feel like you’re running towards something, which is hopefully what you’re doing, rather than running away from something.

You don’t always have the chance to do this. Sometimes circumstances play the roles that they do. But if you have the chance, do your best to set things up such that you leave well. People remember peaks and ends. You want to make sure the end they remember is a good one.

When you have a manager you trust, you have an opportunity to give three kinds of notice. The first is when you’ve decided to look outside. The second is when things get serious. And the third is before you sign an offer.

Which of these is most relevant depends on the relationship, the company, and where you are in your career. I know somebody whose manager really appreciated the heads up before he signed. In my case, I was lucky to be close enough that I could give the heads up the moment I first decided to explore. So I was able to give months of notice.

If you have a team you’re responsible for, it helps to time it right – after a win, when there’s planned work ahead, where there’s a successor ready or a clear plan for one, and with well planned communications so it is all done professionally. All of this helps set the transition up in an orderly fashion.

Make the effort to get the timing right. Then sweat the details of the transition to ensure every detail is taken care of.

It matters.

#careertransition – #4 of 8 | I recently went through a process of career exploration and made a significant career switch. This series of 8 posts is a synthesis of the lessons I’ve learnt.

---

As you explore, you’re going to come across jobs/projects/opportunities that look exciting. The question is how to evaluate them.

I think it comes down to four questions:

1. Do you believe in the space? Is this a space where the work matters to you, and where the trajectory makes sense for where the world is headed?
   
2. Do you believe in the company/organization? Is this a company that has a real shot at winning in that space? Is there something about how they operate, their strategy, their position, that gives you conviction?
   
3. Do you believe in the people? Are these people with that combination of intelligence, integrity and energy that you believe in?
   
4. Do you believe you’ll be set up for success? This one is easy to overlook. You might believe in the space, the company, and the people – but if the role, the scope, or the circumstances don’t set you up to do your best work, it’s going to be hard.

Of course, if you’re starting up yourself, some of these questions get modified. But they largely hold.

It’s worth being honest with yourself about where you have conviction and where you don’t. In my case, after many weeks of exploring starting up myself, I realized I didn’t believe in my ability to win. Founding a company requires some earned insight or unfair advantage that typically comes from deep understanding of a problem space – and I was drawn to problem spaces where I didn’t have any advantage.

That led me to realize that I wanted to join a small company in a problem space I was drawn to – a different path than what I thought I’d pursue but the right one at this point of time.

#careertransition – #3 of 8 | I recently went through a process of career exploration and made a significant career switch. This series of 8 posts is a synthesis of the lessons I’ve learnt.

---

One of the things you realize through a career transition is that you meet a lot of wonderful strangers. There are just a lot of kind people who take time out to help someone trying to figure out their next step. In my case, this included people whose companies I was interested in and many others who I spoke to as a prospective start-up founder trying to validate an idea.

To be clear, you do meet the occasional unkind person. But for the most part, it’s incredible how much the kind people outnumber the unkind ones. I’m so grateful to such people. Many of them inspired me to become a kinder version of myself.

There are 3 things to keep in mind as you reach out to people –

1. It’s a lot easier to get help when you know where you’re going. When you’re clear about what you want, people can help you. When you’re not clear, it becomes harder and harder for anyone to fully understand how they can help.
   
2. Show up well and follow up. Make it easy for people to connect you with others. Be polite. Be thoughtful. Prepare. Say thank you.
   
3. Start by speaking to one person per week. You can start with people you know. Ask people who they would recommend you speak to. It’s amazing how quickly this snowballs and how much perspective it gives you.