## Sudoku Goggles

A couple of my interests overlapped this morning when I found out that Google Goggles solves Sudoku.

(As an almost-related aside, I’m still tweaking my Go Sudoku solver. More on that later.)

So, after a bit of discussion, we decided to try to mess with it by feeding it invalid puzzles. Here’s one of the tests I tried – note the 4 in the upper-left block, which should be a 3.

What Goggles detected is below. Note that the 4 in question is there, but it’s missed a few numbers elsewhere in the puzzle.

It solves this version quite happily. At first I thought it was cheating by overwriting some of the original cells, until I noticed the gaps. I tried it again a few times, keeping the camera as steady as possible and cropping out other distractions; it detected a different subset of the numbers each time, so I started to think that the OCR was a bit flaky. But it always seemed to detect a valid puzzle… and when I tried it with the *original* valid puzzle, it detected everything.

It seems like Goggles doesn’t just recognise any old 9×9 grid of numbers as a Sudoku puzzle; it will only recognise a *valid Sudoku puzzle* as a Sudoku puzzle. My theory is that when it looks at the invalid Sudoku, its best match for it is a valid Sudoku with fewer numbers, plus some noise where its OCR thought there was *probably* a number but wasn’t quite as confident – as though, for example, there was some dust on the lens in front of those squares. (Most of the time it seemed to drop 1s, which have fewer obvious features than other numbers and are hard to distinguish from 2s.)

The interesting thing about this is that a naïve implementation (i.e. what I’d probably do) would OCR the grid *first*, set that in stone, then feed it into a Sudoku solver/validity-checker. Google’s implementation seems to tie more organically into however its recognition engine works (from this very limited set of tests anyway) – i.e. the space of objects it recognises includes all valid Sudoku puzzles. Somehow.