Why straight lines break

Texture is a surface whose statistics are the same everywhere and whose details are interchangeable. One square foot of sand is as good as any other square foot of sand: move it, rotate it, put it somewhere else, and nobody can tell. Filling a region in texture is therefore a search problem with a guaranteed answer, because the answer is lying around elsewhere in the same photograph.

Structure is a surface whose details are positioned. A railing has bars at specific intervals; a tiled floor has grout lines meeting at specific angles; a horizon is at a specific height and is dead straight for the width of the frame. The information that makes structure look right is not in the appearance of any small piece of it — it is in the relationship between pieces. A patch search, which by construction compares small pieces, has no access to that relationship at all. It is not that it does it badly. It cannot see the question.

The bend. The patch search finds genuine railing and places it at the wrong angle. The bars are real, the texture is right, the metal has the correct highlight — and the line arrives at the far side of the region two degrees off and a few pixels high. This is the most common outcome and the most annoying, because everything about it looks correct except the one thing that matters.

The smear. The marching fill carries a gradient inward from the boundary and averages across the gap. On a narrow gap that is exactly right. Across a wide one it produces a smooth band with no detail at all, because the middle of the region is averaging values that were themselves averages. On a railing the bars and the gaps between them average to a uniform grey.

The drift. A network continues the line correctly for most of the width and then loses it, usually where one processing tile ends and the next begins. The result is a line that is convincingly a line and is nevertheless in the wrong place, which is the hardest of the three to notice and the worst to discover after printing.

1. 1Stop the region at the line. Paint two regions instead of one, with the boundary running along the structure rather than across it. Each fill then draws only from picture on its own side, and the line survives because nothing was ever asked to invent it.
2. 2Work in passes, from the edges inward. Take the parts of the object that sit over plain texture first, run the fill, then paint what is left over the structure. The second region is smaller and has more intact picture around it, which is the single biggest lever on quality.
3. 3Move the boundary to where the eye is not. A seam inside a shadow, behind a branch or in the darkest part of a texture is invisible; the same seam in the middle of a lit wall is not. Extending a region by fifty pixels to end it somewhere forgiving is nearly always worth the extra area.
4. 4Consider the crop. If the structure runs the whole width of the frame and the object sits squarely on it, there may be no removal that holds up at full size. Losing a strip of the picture is a real answer and often the honest one.

Drag the divider slowly across the region and watch the boundary, not the middle. Almost every failure announces itself at the join: a line that steps, a texture that changes frequency, a brightness that shifts. The middle of a fill is usually the most convincing part of it, which is precisely why it is not worth staring at.

Then hold space to snap the whole frame back to how it arrived, and let go. What you are testing is whether anything moves — a line that jumps, a pattern that shifts phase. The eye is far better at detecting a change between two states than at judging a single state, which is the entire argument for building this page around a wipe instead of two thumbnails.

• If the result looks worse the longer you look at it, the problem is structure and no amount of re-running will fix it. Change the region.
• If the result looks fine at fit-to-screen and wrong at full size, the region was large enough that the coarse pyramid levels decided most of it. Try a smaller region in two passes.
• If a faint outline of the old object survives, the region did not cover its edge. Grow by two pixels and run again from the original rather than filling the fill.

Why does a small region on a railing work and a big one not?

Because the patch search is matching squares of about seven pixels. If the region is narrow enough that a single square can see the railing entering on one side and leaving on the other, the match it finds is constrained by both ends and the bars line up. Once the region is wider than that, the square sees only one end, and any piece of railing anywhere in the picture fits equally well.

Would a neural fill handle it?

Better, and not perfectly. A network with a whole-image view has seen enough railings to continue one, which no patch search can do. It still works tile by tile on a large photograph, so a line crossing a tile boundary can drift; and where it errs it errs sharply and with conviction, which on a structure the viewer can measure against is worse than an obvious smear. There is no learned engine on this site yet and the page says so plainly where the engines are listed.

Is there a trick with perspective?

Not one this tool does. The methods that help — rectifying the plane, filling in the rectified space, and warping the result back — need the vanishing points, which means either finding them automatically or asking somebody to draw them. Both are real work and neither is in this build. What you can do by hand is aim the region so that the structure it crosses is close to parallel with the frame, which is the case those methods are approximating anyway.

The companion piece on how the fill decides what goes there covers the search itself in detail, including why the pyramid exists and what the seed is for.