Thanks for such a thorough write-up of an interesting paper on a timely topic! I've been really enjoying your newsletter. There are a couple of aspects of this research that would make me hesitate to generalize too much from it. First, this is just looking at Bard. It's possible there are better or worse agents for these kinds of questions. But primarily, the drill-down from randomly selected invites to participants is massive: from 1,400 invites to participate, to 220 responses, to 76 participants leaves a lot of room for all kinds of selection effects. Not saying the conclusions here or in the paper are wrong, of course, but it's hard to judge from this evidence.
Hey John, thanks for the great thoughts here. I totally agree - in general, it can be hard to generalize behaviors when the study only looks at engineers from a single company (ie Google), and we talk about this internally a lot. We try to use the application section to turn early signals into meaningful conversations or actions that engineering leaders can take, knowing that correlation is not the same as causation. We're excited to keep building on the results from this paper and the few others coming out in the space too.
Thanks for such a thorough write-up of an interesting paper on a timely topic! I've been really enjoying your newsletter. There are a couple of aspects of this research that would make me hesitate to generalize too much from it. First, this is just looking at Bard. It's possible there are better or worse agents for these kinds of questions. But primarily, the drill-down from randomly selected invites to participants is massive: from 1,400 invites to participate, to 220 responses, to 76 participants leaves a lot of room for all kinds of selection effects. Not saying the conclusions here or in the paper are wrong, of course, but it's hard to judge from this evidence.
Hey John, thanks for the great thoughts here. I totally agree - in general, it can be hard to generalize behaviors when the study only looks at engineers from a single company (ie Google), and we talk about this internally a lot. We try to use the application section to turn early signals into meaningful conversations or actions that engineering leaders can take, knowing that correlation is not the same as causation. We're excited to keep building on the results from this paper and the few others coming out in the space too.