Phenomenon window

Your Diabetes Model Might Be Too Optimistic—County Splits Reveal the Truth

You've just trained a model to predict diabetes from health survey data, and the accuracy looks amazing. But wait—did you split your data randomly? If so, your results might be a mirage. When data comes from clusters like counties, random splits can leak information, making your model look better than it really is. In this project, you'll explore how splitting data by county versus randomly changes your model's performance. You'll build a simple prediction model, try both split methods, and see the difference for yourself. You'll also learn about a key concept called intraclass correlation (ICC) that tells you when to worry. By the end, you'll know why the way you split data matters just as much as the model you choose—and you'll have a practical skill for any data science project. Ready to see if your model is fooling you?

• “

  Have you ever had a test that felt easy because you'd seen the questions before, but then the real exam was much harder? That's like random splitting—your model 'sees' the same people in training and testing.

• “

  When you study with friends from the same class, do you sometimes all make the same mistakes? That's similar to data from the same county—it's not independent, and it can trick your model.

• “

  Think of a time you predicted something about a friend based on knowing their hometown. Did that help or mislead you? That's the kind of bias county-level grouping can introduce.