How content-aware fill decides what goes in the hole

You have a photograph with a region marked as unknown. Every pixel outside that region is known. The job is to choose values for the unknown pixels such that somebody looking at the result cannot tell where the boundary was. Note what the job is not: it is not to recover what was there. Those values are absent from the data, and no quantity of arithmetic conjures them back. Every method below is a different answer to the question what would be consistent with the surroundings, and consistency is a much weaker claim than truth.

The methods divide on where they get their answer. One copies from elsewhere in the same photograph. One propagates inward from the boundary. One has read several million other photographs and produces something statistically typical. Each is better than the others at something, and the third is the only one that can lie convincingly.

Cut the photograph into overlapping squares — seven pixels a side is a good default. For every square that overlaps the unknown region, find the square of fully-known picture that it most resembles, and use that one's contents as a vote for what belongs there. Each unknown pixel is covered by forty-nine squares, so it receives forty-nine votes; average them, weighted by how good each match was, and you have a fill.

The expensive part is finding the matches. Comparing every square to every other square in a twelve-megapixel photograph is not something anybody is going to wait for. The randomised search that makes this tractable is beautifully simple and rests on one observation: good matches are contagious. Start with a random guess for every square. Then sweep across the image; at each square, try the match your neighbour found, shifted by one pixel. If your neighbour found a good piece of sand, the sand next to it is probably good for you. Most of the improvement in the whole algorithm comes from that one step. Then, to stop everything settling into one lucky corner, try a handful of random offsets at exponentially shrinking distances. Five sweeps of propagate-and-jump get within a few per cent of the true nearest neighbour for a rounding error of the cost.

Two refinements matter in practice. The first is the pyramid: run the whole thing at a sixteenth resolution, carry the answer up, run it again at an eighth, and so on. Without it, the centre of a wide region is matching against its own invented pixels from the very first iteration. The second is determinism: the random number generator is seeded from a fixed value, so painting the same region twice gives the same result twice. A tool that produced a different answer each time would be impossible to reason about and impossible to complain about usefully.

• Excellent on texture with no long-range structure: sand, gravel, grass, water, foliage, plaster, knitwear, tarmac.
• Poor wherever a line, an edge or a repeating pattern in perspective crosses the region, because a square of railing resembles another square of railing regardless of where the railing is going.
• Fails visibly rather than convincingly. A bad exemplar fill looks like smeared or duplicated texture, which anyone can see.

The second method treats the region as a hole in a sheet and freezes it shut from the edges. Pixels are settled in order of their distance from known picture — nearest first — and each is given a weighted mean of the already-settled pixels lying within a short radius. The weights carry the whole method. A neighbour in the direction the boundary is moving is trusted further than one off to the side, and that preference is what drags a gradient across the gap rather than levelling it. A neighbour that settled at about the same moment counts for more than one that settled much earlier, which keeps an edge crossing a narrow gap joined up.

On a region two or three pixels wide this is not merely adequate, it is the better answer: every pixel has real information within a couple of steps, the average is drawn from genuine picture, and it costs a few milliseconds. On a region a hundred pixels wide the same arithmetic produces the soft grey-brown bruise everyone recognises as a failed removal, because the middle is averaging averages of averages. Knowing which of those two situations you are in is the entire trick, and the tool measures it: the half-width of the widest part of the region decides which engine gets the work.

A convolutional inpainting network is trained on a few million photographs with rectangles cut out of them, and learns a mapping from the surroundings to the missing contents. The architectures that work well at this use a Fourier-domain convolution, which gives every layer a view of the whole image rather than a local window — and that, rather than raw capacity, is what lets them continue a railing or a horizon across a gap. They know what a line is, in the only sense that matters here: they have seen enough of them to continue one.

The one on this site is MI-GAN, which was built for phones, and the costs are specific rather than approximate. The weight file is 26,639,447 bytes at full precision — 25.4 MB — fetched once from our model file server, and the WebAssembly runtime that executes it is another 13.3 MB served from this domain: about 38.7 MB the first time and nothing afterwards, stated in the panel before a byte moves rather than discovered from a progress bar. It accepts one input size and only one, 512 by 512, so it is never run over a whole photograph and never tiled either — a square is cut around whatever was painted, padded to two and a half times the stroke for context, resampled to 512, run once, and composited back at the picture's own resolution. One removal is one inference, about two and a half seconds on a desktop on single-threaded WebAssembly, longer on a phone. It runs in a background worker, so the page keeps drawing while it works.

Two consequences of that 512 are worth knowing, because they explain the results you will get. A small mark on a large photograph is filled from a window barely larger than itself, at close to native resolution, and comes out well. A large region on the same photograph forces a large window, which is then squeezed into 512 before the model ever sees it — so the model is working from a reduced copy, and the fill comes back softer than its surroundings. The tool does not hide this by upsampling the result; it puts back exactly what the network produced, only inside the painted region.

The last difference is the one worth thinking about longest, and adding the engine to this site has not softened it. A learned fill can be wrong confidently. It will produce a crisp, plausible, entirely fictional continuation, and there is nothing in the result that announces itself as invented. The patch search cannot do that; when it fails, it smears. If the photograph is going to be printed, filed, or shown to somebody as a record of something, a failure you can see is worth a great deal more than one you cannot — which is why the engine list defaults to choosing between the two classical fills and leaves this one to be picked deliberately.

Take a photograph of somebody standing in front of an iron railing, with the railing running behind them at an angle, and paint them out. The patch search returns railing — genuine railing, copied from twenty pixels away, at the wrong angle, so the bars do not meet. The marching fill returns a smooth brown-grey band where the railing was, because it is averaging bars and gaps together. The network returns a railing that continues correctly for most of the width and then drifts, because the window it was working in could not see both ends — and it drifts crisply, which on a railing somebody can measure against is worse than an obvious smear.

The lesson is not that the tools are bad. It is that a removal whose region crosses structure is a hard problem for everything, and the practical move is to change the problem: paint a smaller region that stops at the railing, do it in two passes, or accept a crop. That is the subject of the guide on straight lines.