Dedicated Crossword Practice

Joe has always talked of getting good at crosswords. When my model provided him with a single numeric skill estimate, Joe decided the time had come for some sustained practice. So, over one weekend, he completed roughly four hundred crosswords. Note that his average crossword time is about a minute, so that's something like six hours straight of crosswords!

Since the crossword bot tracks submission time, I isolated his period of dedicated practice and wanted to know whether he got better over that period. To do so, I looked at trends in his residual: the prediction errors my model gets for each submission. Since my model does not assume improvement over time, the prediction errors should trend negative if Joe gets better.

In this plot, the blue points represent residuals, the red line is a smoothing of the blue points, and the x-axis just records all the crosswords Joe has ever done, in order. The period of sustained practice starts at 85 and ends at 495.

You can see that there is no overall trend; Joe finishes the practice period roughly where he started. There is a period from roughly 225 to 275 where Joe is noticeably worse. That corresponds to a practice session on the morning of 5 August (from 9:00 to 14:00). Perhaps Joe was tired that morning,² [² The data report he'd stayed up until 3am solving crosswords the prior night, so he'd had about six hours of sleep.] and it affected his crossword times. It's a good sign that that's something we can see out of the data.

Looking just at the distribution of blue points, it also seems that Joe's performance got more variable over time. Toward the left hand side of the plot, the blue dots are roughly evenly distributed around 0, with a standard deviation of around 1—basically normal, in other words. Toward the right, the points mostly cluster below 0 (suggesting improvement), plus many extreme points as high as 2 or 3. So Joe seems to have gotten better at the median puzzle, but gotten worse at the rare extreme puzzle.

I also wanted to check whether Joe gets better or worse over the course of a practice session. To find out, I looked through the data by hand and picked out nine periods of sustained practice (of 10 to 112 crosswords long) and replotted the points counting from the start of the practice session:

Here the points have different opacity because there can be multiple points for one x value; in fact, up to nine, one for each practice session. As the smoothing curve shows, there's not a clear trend over the course of a session.³ [³ With plots like these, there's a chance of a "composition effect", because different x values have different sessions contributing to them (since sessions have different lengths). I repeated the analysis once per session, but still didn't see much of a coherent trend.]

This might be a good time, by the way, to bring in Joe's thoughts. He feels he did get significantly better thanks to his dedicated practice,⁴ [⁴ This fits with the fact that he over-performed more often, but his under-performances stayed bad enough for no average movement.] but that that occurred between, not during, practice sessions. So not seeing any movement in the plot above is not crazy.

It seems that on average, Joe did not get better by doing many crosswords over the course of a weekend. Maybe you won't, either. However, it does seem that Joe got faster at the crosswords that he solved quickly, and it seems that getting a full night's sleep can have a big impact (30% faster!).

I speculate that to get better at crosswords, it's not enough to just do a lot of them (beyond the beginner period). It helps to sleep well and do them when you're well awake, and it might help to regularly read the news, consume more classic American art, and expand one's vocabulary.