The Challenge of Vision

Charniak Ch. 5 companion – based on Ch. 1 of Visual Intelligence (Torralba, Isola & Freeman, visionbook.mit.edu)

HON 201 - Week 9

A Note on These Slides

All images in this deck are reproduced from Visual Intelligence by Antonio Torralba, Phillip Isola, and William Freeman (visionbook.mit.edu), an open-access textbook published under a CC BY-NC-ND license.

They are used here strictly for in-class instructional purposes – not for redistribution, publication, or any commercial use. Full credit belongs to the authors.

State this plainly before showing any figures: these are the authors’ images, used under fair-use/educational terms for this one class session, not assets to be reused elsewhere. The source is the open MIT “Visual Intelligence” textbook, Chapter 1 (“The Challenge of Vision”), freely readable at visionbook.mit.edu.

Vision Feels Effortless. It Isn’t.

• Your eye receives only the light arriving from each direction – nothing more.
• The brain turns millions of photoreceptor signals into an interpretation of the world.
• Visual illusions prove the point: what you see is not the same as the light that actually reached your eye.

Hans Moravec, roboticist – namesake of Moravec’s Paradox: the things vision makes look effortless (recognizing a face, finding a path across a room) are the hardest things to automate, while the things that feel effortful to us (arithmetic, chess) are comparatively easy for a computer.

Section 1.1, Introduction. Core idea of the whole chapter: vision feels like the easiest thing we do, but it is solving an extraordinarily hard inverse problem, one no artificial system fully matches yet. The eye is a thin sensor; the brain does the heavy lifting of turning raw light measurements into a usable model of the world. Illusions are the chapter’s recurring proof technique – they expose the gap between the actual input (light) and the output (our experience), which is exactly where the interesting computation is happening. Also worth noting: the book’s techniques generalize beyond images to other signal types, and “vision science” has two goals – understanding biological perception, and building artificial systems that interpret images or image sequences. The Moravec photo above is worth a beat of its own: Moravec’s Paradox (the framing hook for today’s whole lesson plan) is the sharpest one-line statement of this slide’s point – evolution spent hundreds of millions of years tuning sensorimotor and perceptual skills, and only a few thousand years on abstract reasoning, which is exactly why the “easy” stuff for us (seeing, walking) is the hard stuff for machines, and vice versa.

What Is Vision, Actually?

Vision is “to know what is where by looking.” – David Marr

Two sources of difficulty:

1. The structure of the input – what light actually carries.
2. The structure of the desired output – what we actually want to recover from it.

Computer vision is usually studied disembodied – no agent, no task, no body. That omission makes the problem harder, not easier.

Section 1.2, “Vision.” Marr’s definition is the chapter’s anchor quote and worth writing on the board. The “disembodied” point matters pedagogically: human and animal vision always comes attached to a body with goals (find food, avoid obstacles); most computer vision research strips that away and studies image interpretation in isolation, which removes a lot of the context that would otherwise constrain the problem. Flag this now – it resurfaces explicitly with Gibson’s ecological approach later in the deck.

The Input: Ambient Light

Section 1.2.1. Light bounces between surfaces repeatedly before any of it reaches an eye, so even a single ray’s journey through a scene is complex – the laser-pointer demonstration in Figure 1.1 makes that visible. The term “ambient light” comes from James J. Gibson. Adelson and Bergen’s plenoptic function formalizes this: light intensity as a function of viewing direction, wavelength, time, and position – in principle it contains all the light information present in a space. Crucially, an observer only ever samples a tiny slice of this function, and the function itself doesn’t include the observer. Occlusion is the single biggest obstacle: most surfaces in a scene are hidden from any one vantage point, and recovering the entire plenoptic function was never the goal of vision in the first place – recovering the meaningful structure behind a small sample of it is.

The Output: Light vs. Scene Properties

Vision is not a fixed function of the image. Prior knowledge, goals, and expectations all shape what you see.

The goal isn’t to measure light like a photometer – it’s to recover the surfaces, objects, and materials that actually matter to the observer.

The same point in the same scene, probed with a laser pointer under three different lighting conditions – the measured light swings wildly, even though the underlying surface never changes.

Section 1.2.2. This demo is the output-side companion to the Figure 1.1 laser-pointer demo: the same physical point on the same surface returns three very different raw light measurements depending on illumination alone – proof that raw pixel values radically underdetermine the scene. The book also walks through Figure 1.2’s three illusions to make the complementary point, that vision reconstructs 3D scene structure rather than reporting flat image values: (a) occluding figures – a blue square is perceived as continuing behind a red square rather than as an odd L-shape, because the visual system infers occlusion rather than reading the image literally; (b) Shepard’s turning tables – two tabletops with identical on-page lengths look different because the brain reads the image as a 3D scene and “corrects” for the implied perspective; (c) Adelson’s checkershadow – two squares with the exact same gray pixel value look different shades because the brain discounts what it infers to be a cast shadow. Close this slide with the chapter’s list of what vision needs to actually do: detect change, estimate motion, recognize objects and materials, read text, build 3D models, find free space, and even interpret animal behavior – some of these need outside knowledge, others may be solvable from the image alone.

Gestalt Psychology: The Whole, Not the Parts

• Began ~1912 with Max Wertheimer’s phi phenomenon – two flashed bars appear to move continuously, even though nothing physically moved between them.
• Central claim: perception is about wholes, not an assembly of independent parts.
• This becomes perceptual organization – how scattered features get grouped into objects at all.

Section 1.3.3 (skipping 1.3.1-1.3.2’s ancient extramission theories and jumping straight to the 20th century). The phi phenomenon is the founding demonstration of Gestalt psychology: if perception just assembled static frames one at a time, two sequential flashes wouldn’t look like motion. Since they do, something beyond simple registration of individual stimuli is clearly happening in perception – grouping, inference, organization. This section matters for computer vision because its core question – “how do individual pixels/edges/regions get grouped into one object?” – is literally the problem of image segmentation.

Gestalt’s Grouping Laws

Figure 1.6 – examples of Gestalt grouping rules (after Palmer and Koffka).

Walk through the named grouping principles – they’re “laws” only loosely, and vary in strength: proximity (nearby items group together), similarity (similar items group), closure (a closed outline is seen as a shape, even with gaps), good continuation (smooth, continuing edges group over sharp turns), parallelism and symmetry, common fate (elements that move together are grouped, even if visually dissimilar), and past experience (learned, familiar groupings persist even against other cues). These rules are exactly what a hand-engineered segmentation algorithm from the 1970s-90s tried to formalize computationally, decades before learned segmentation took over.

Lightness, Shadows, and Illusory Contours

Figure 1.7 – the Koffka ring: identical gray on both halves, perceived as different.

Figure 1.8 – amodal completion (the “B or 13” figure, and a bar seen through overlapping squares) and modal completion (the Kanizsa triangle, bottom right).

Lightness (how light or dark a surface looks) depends on both perceived reflectance and perceived illumination – it’s contextual, not purely local pixel brightness, and the Koffka ring (Figure 1.7) makes this vivid: both halves of the ring are the literal same gray, but the surrounding spatial arrangement makes them look different. Then two completion phenomena, both in Figure 1.8: amodal completion is perceiving a whole object when only part of it is visible (you know the rest is there, even unseen); modal completion is seeing contours that are not physically present at all – the Kanizsa triangle is the classic case, where illusory white edges appear to connect three partial circles into a triangle nobody drew. Land this section with its payoff: these Gestalt principles directly shaped early computer vision, especially the entire subfield of image segmentation.

Gibson’s Ecological Approach

James J. Gibson’s argument: study vision in the context of a body moving through an environment – not isolated lab stimuli.

• The environment: medium, substances, surfaces, change and persistence.
• The ground plane matters enormously for any land animal.
• Ecological optics: study the ambient optic array actually converging on an observer, not idealized physics.

Section 1.3.4. Gibson’s framing is explicitly a reaction against lab-based, disembodied vision research (the very gap flagged back on the “What Is Vision, Actually?” slide). His ecological categories matter because the same physical substance plays different roles for different animals – water is a medium you swim through if you’re a fish, but a substance you’d drown in if you’re a land animal; the environment only makes sense relative to an organism’s own capacities. Figure 1.9’s bird optic-flow example shows how motion through an environment generates a rich, structured pattern of changing light that itself carries usable information, without any further “inference” step required.

Affordances and Direct Perception

• Affordances: what the environment offers an animal to do – a path affords walking, an obstacle affords collision, a staircase affords climbing.
• Direct perception: the ambient optic array is already rich enough that perceiving is closer to picking up information than to inferring it.
• Contrast: the indirect view holds that the input is impoverished, and the brain must supplement it with extra computation.

This is Gibson’s most provocative and most contested claim. If perception really is “direct” – picking up already-present structure in the optic array rather than solving an inference problem – then vision needs far less internal computation and representation-building than researchers like Marr would go on to argue. The book’s own verdict (previewed here, confirmed on the Marr slides): Gibson asked excellent questions about embodiment and the richness of natural stimuli, but likely underestimated just how much computation vision genuinely requires.

How the Eye Itself Works

(Section 1.3.5, neurophysiology – skipped from today’s main throughline, included here as an optional aside.)

Helmholtz’s ophthalmoscope (1850): the first instrument letting a doctor look directly at a living retina.

Cajal’s drawing of the retina’s own layered circuitry: photoreceptors feeding bipolar and horizontal cells, feeding ganglion cells, whose axons leave as the optic nerve.

Young-Helmholtz trichromatic theory: three overlapping primaries (red, green, blue) explain the full range of perceived color.

Optional aside, section 1.3.5 (skipped from the chapter’s main throughline today, but worth a beat since these images are on hand). Hermann von Helmholtz is the connecting figure across all three images: in 1850 he invented the ophthalmoscope, the first instrument that let a physician look directly at a living retina rather than studying it only in cadavers – the fundus photograph shown here (the blood vessels radiating from the optic disc) is exactly the kind of view it opened up. Helmholtz is also, with Thomas Young, the namesake of the trichromatic theory of color vision: human color perception can be explained by just three types of cone photoreceptor, each tuned to a different range of wavelengths, whose combined signals span the full range of perceived color – the classic three-circle color-mixing diagram illustrates this directly. Cajal’s drawing (decades later, based on his own silver-staining neuroanatomy) shows what’s actually happening physically inside the retina Helmholtz could now see: light-sensitive rods and cones feed horizontal and bipolar cells, which feed ganglion cells, whose long axons bundle together and leave the eye as the optic nerve – the retina is not a passive sensor but a small piece of brain tissue doing real computation before a signal ever reaches the brain proper. This is the neurophysiological complement to everything else in this deck: Gestalt, Gibson, and Marr all describe vision from the outside (behavior, environment, computation); this is the hardware those theories are ultimately theories of.

Marr’s Computational Theory of Vision

Published posthumously, 1982 (Marr died of leukemia in 1980, age 35). Framed vision as an information-processing task, analyzable at three independent but loosely coupled levels:

1. Computational theory – what is the task, and why does it need solving?
2. Representation and algorithm – how are inputs/outputs encoded, and what procedure transforms one into the other?
3. Hardware implementation – what physical substrate (silicon, or neurons) actually carries it out?

Section 1.3.6. This three-level separation is one of the most durable ideas in all of cognitive science, not just vision – the same cash-register example from the Ch. 5 lesson plan’s Shannon/chess material applies here too (addition is the computational level; how digits are represented and carried is the algorithmic level; the actual gears or circuits are the hardware level). Marr explicitly credits Gibson with asking the right questions, but argues Gibson underestimated how hard those questions actually are to answer computationally – direct pickup of information turns out to require a great deal of hidden machinery once you try to formalize it.

Marr’s Sequence of Representations

Image (raw pixel intensities) -> Primal sketch (edges, junctions, zero-crossings) -> 2.5D sketch (depth map + surface orientation, viewer-centered) -> 3D model (objects as shapes, arranged in space)

Figure 1.14 – a cube and its 2.5D sketch: red arrows are “gauge figures,” a visualization convention showing estimated surface orientation at each point.

Walk the class through the pipeline left to right – it’s the direct ancestor of the “bottom-up feature pipeline” language already used in the lesson plan for this chapter. The book also discusses the Necker cube (Figure 1.13) here: a bistable figure whose same 2D line drawing flips between two different 3D interpretations, a vivid demonstration that the brain is actively constructing a 3D model rather than just reading one off the image – worth sketching on the board even without a slide image. The gauge-figure arrows pictured above are the visualization convention for the 2.5D sketch itself: each arrow shows which way a patch of surface is tilted relative to the viewer, at every point across the cube’s visible faces. Close with the book’s own verdict on Marr: his three-level separation was hugely productive, but his biggest blind spot, in hindsight, was underestimating how much of vision’s representation-building would end up being learned from data rather than hand-specified – which is exactly where this story is headed.

Computer Vision, 1960-2000: The Geometry Era

1960-1990 – dominated by geometry: stereo, shape-from-shading, multiview geometry, motion estimation, model-based recognition. Representations: edges, image pyramids, generalized cylinders.

1990-2000 – shift toward measuring real ecological image properties: segmentation, texture modeling, early learning-based face detection. Digital cameras become common.

(Larry Roberts earned the first computer-vision PhD at MIT in 1963 – the same Larry Roberts from the blocks-world material two chapters ago.)

Section 1.3.7. This is the book’s own periodization of the field’s history, and it should feel familiar after today’s earlier Ch. 5 material on Roberts and the Summer Vision Project – this slide is effectively zooming back out to the decades-long view. The throughline across both 30-year windows: everything is still hand-engineered. Nobody is learning representations from data yet; researchers are hand-designing geometric models, hand-designing edge detectors, hand-designing texture descriptors. That’s the setup for the next two slides.

Computer Vision, 2000-2020: Recognition, Then Learning

2000-2010 – focus shifts to classification and recognition. Benchmarks appear (Caltech 101, PASCAL, LabelMe, CIFAR). Hand-designed descriptors (SIFT, HOG, GIST) feed classifiers (SVMs, boosting). Large datasets like ImageNet appear – but architectures and features are still mostly hand-designed.

2010-2020 – deep learning comes to dominate every area of computer vision.

An early-2000s learned face detector locating a face in a real photo (Viola-Jones-era) – the decade’s standout non-neural recognition success.

The 2000-2010 decade is worth lingering on, because it’s easy to misread: ImageNet (the dataset) existed years before deep learning (the method) arrived to exploit it properly. Having a large labeled dataset and having a learning algorithm capable of using it well turned out to be two separate breakthroughs that didn’t happen at the same time – the dataset came first, the method came roughly a decade later, which sets up the “ImageNet Moment” slide directly ahead. The pictured face detector is the decade’s best-known success story for recognition without deep learning – Viola and Jones’s 2001 boosting-based detector (referenced again two slides ahead), which shipped in real cameras years before convolutional networks were practical.

Learning-Based Vision and the Rise and Fall of Neural Nets

Figure 1.16 – enthusiasm for neural networks has risen and fallen on a cycle of roughly 25 years. What happens next?

Section 1.3.8 (the figure and history worth covering here, even without the full “three ways to program a vision algorithm” framing around it). The hype cycle Figure 1.16 plots: Rosenblatt’s perceptron (1958) could learn simple visual patterns and generated real excitement; Minsky and Papert (1969) proved perceptrons’ sharp mathematical limits, which cooled the field for roughly a decade; the 1980s saw a revival once researchers started stacking perceptrons into deeper networks – Fukushima’s neocognitron anticipated convolutional networks, and LeCun et al. (1989) showed deep nets could actually be trained end-to-end, on handwritten digit recognition; then the 1990s-early 2000s brought another “AI winter” for neural nets specifically (even as other vision techniques advanced – Viola and Jones’s 2001 boosting-based face detector is the era’s standout success, and it was not a neural network). The chapter’s own punchline, worth posing to the class rather than answering: given this ~25-year rise-and-fall rhythm, where are we now, and is another winter coming? (This is also, worth noting explicitly, the same question underlying this week’s Kuhn material on crisis and resolution.)

The ImageNet Moment (2012)

AlexNet dramatically outperformed every prior method at the ImageNet competition – trained end-to-end, straight from raw pixels to output predictions, with no hand-designed features in between.

After 2012, learning became central to nearly all computer vision research.

This is the chapter’s single biggest turning point, and it directly answers the hype-cycle question just posed. ImageNet itself (the dataset, organized under the WordNet hierarchy of noun categories) had existed for several years already – what changed in 2012 was that a deep convolutional network, AlexNet, finally had enough data and enough compute to make end-to-end learning decisively beat every hand-engineered pipeline (SIFT/HOG/GIST + SVM) that the field had spent the 2000s refining. The book’s own framing, worth stating directly: learning does not change the basic structure of the vision problem – input, output, and a mapping between them. What changes is how the mapping is found. Hand-designed and learned algorithms often end up looking similar in what they compute; the learned ones now simply tend to do it better. This is the exact moment Ch. 5’s “normal science vs. crisis” question from earlier in the lesson plan gets its clearest real answer.

What’s Next? (Section 1.4)

Vision theory has been under construction for over 2,000 years – from Greek extramission theories to modern neuroscience and deep learning. Each stage revealed something the last one missed.

Named gaps still open today:

• Visual common sense – now being explored with large language models.
• Integration with the other senses.
• A theory of embodied perception – the chapter calls this the most important missing piece.
• (And, almost certainly, gaps we can’t see yet.)

Close on this note deliberately: the chapter’s own self-assessment is that every past theory of vision – extramission, Gestalt, Gibson’s ecological approach, Marr’s computational theory, modern deep learning – got something importantly right while still missing something the next theory would catch. That is a strikingly Kuhnian way to end a chapter: no single paradigm has yet been the final word, and the authors are explicit that the current deep-learning paradigm likely has its own blind spots not yet visible from the inside. Good last beat for the room: which of today’s five theories (Gestalt, Gibson, Marr, the geometry era, deep learning) do they think will look most dated in twenty years, and why?