XenonStack Blog History of Computer Vision: From 1959 to Vision Agents

Computer vision is the branch of artificial intelligence that lets software interpret images and video. Its history runs from neuroscience in 1959, through rule-based scene analysis in the 1960s and hand-crafted features in the 2000s, to deep learning after 2012 and, since 2020, transformers and foundation models trained on hundreds of millions of images. Each step lowered the cost of teaching a machine to see a new thing.

For enterprise leaders the history matters because it explains today's choices: when a classic rule-based system is still the right answer, when to fine-tune a foundation model, and why vision is now moving from dashboards to governed vision agents that act on what cameras see. This guide sets out the milestones with their original sources, what has changed since 2024, and how to adopt computer vision safely.

What is computer vision?

Computer vision turns pixels into structured information: a class label, a bounding box, a segmentation mask, a reading of text, a track of a moving object or a written description of a scene. That output then feeds a decision, such as rejecting a part, raising an alert or updating a record. Modern systems learn these mappings from data rather than from hand-written rules, which is the main thread of the history below. For how a project runs end to end today, see our computer vision workflow guide.

How did computer vision begin?

The field grew out of two lines of work in the 1950s and 1960s: studies of how biological vision processes images, and early attempts to make computers describe simple scenes.

1959: how the visual cortex processes images

In 1959 David Hubel and Torsten Wiesel published Receptive fields of single neurones in the cat's striate cortex in The Journal of Physiology. Their work on the visual system later shared the 1981 Nobel Prize in Physiology or Medicine "for their discoveries concerning information processing in the visual system". The idea that vision is built up in layers, from simple features to complex shapes, shaped neural network design for decades.

1963 to 1966: the first machine vision programmes

Lawrence G. Roberts' 1963 MIT PhD thesis, Machine perception of three-dimensional solids, is one of the earliest studies of how a computer could recover three-dimensional objects from images. In July 1966 Seymour Papert's MIT memo The Summer Vision Project (AI Memo 100) set summer workers the task of building a significant part of a visual system: figure-ground analysis, region description and finally object identification against a vocabulary of known objects. The project showed how much harder vision was than it first appeared.

1980: the neocognitron

In 1980 Kunihiko Fukushima published the neocognitron in Biological Cybernetics: a multilayer network, modelled loosely on Hubel and Wiesel's hierarchy, that learned without a teacher to recognise shapes regardless of their position. Its alternating feature-extracting and shift-tolerant layers are an ancestor of today's convolutional neural networks.

What are the key milestones in the history of computer vision?

The table lists the milestones that most shaped enterprise computer vision, each linked to its original paper or record, and what each one means for a project today.

How did hand-crafted features shape the 2000s?

Before deep learning, engineers designed the features a system looked for. Two papers show the approach at its best.

• Viola and Jones (2001). Rapid Object Detection using a Boosted Cascade of Simple Features introduced the integral image, AdaBoost feature selection and an attentional cascade that rejects most background quickly. The paper reports face detection at about 15 frames per second on a 700 MHz Pentium III.
• SIFT (2004). David Lowe's Distinctive image features from scale-invariant keypoints (International Journal of Computer Vision, 60(2), 91 to 110) gave a reliable way to match points between images. The University of British Columbia's US patent 6,711,293 expired on 7 March 2020, and open implementations are available in libraries such as OpenCV.

The limit of this era was data. Each new object or defect type needed new expert-designed features. ImageNet, first published in 2009, changed that by organising labelled images according to the WordNet hierarchy, aiming for an average of 1,000 images per concept. Large labelled datasets made it possible to learn features instead of designing them. Labelling is still the slowest step in most projects; AgentLabel combines AI-assisted annotation with human review to label text, images and more at scale.

Why was 2012 a turning point for computer vision?

In 2012 Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton trained a deep convolutional network on the ImageNet competition data. Their paper, ImageNet Classification with Deep Convolutional Neural Networks, describes a network with 60 million parameters and five convolutional layers classifying images into 1,000 classes. In the ILSVRC 2012 results, their SuperVision entry won the classification task with a top-5 error of 0.153, against 0.262 for the best entry from another team.

Progress then came quickly. ResNet (He et al., December 2015) trained residual networks up to 152 layers deep and placed first in the ILSVRC 2015 classification task, with an ensemble reaching 3.57% error on the ImageNet test set. YOLO (Redmon et al., 2015) framed detection as a single regression problem and reported real-time processing at 45 frames per second, which made video use cases practical.

How did transformers and foundation models change computer vision?

Since 2020 the field has moved from one model per task to large pre-trained models that are adapted with little data.

The practical result is that many enterprise projects now start from a foundation model and fine-tune or prompt it, rather than collecting tens of thousands of labelled images first. Check each model's licence before commercial use, because licences differ by model and version.

How does modern computer vision work?

A current enterprise system combines pieces from every era of the history above.

1. Capture. Cameras, scanners, drones or depth sensors produce images or video.
2. Pre-processing. Frames are resized, normalised and, where privacy requires it, cropped or blurred.
3. Perception model. A convolutional network, vision transformer or vision-language model produces labels, boxes, masks, text or a description.
4. Decision logic. Thresholds, business rules or an AI agent decide what the result means and whether to act.
5. Action and record. The system rejects a part, raises an alert or updates a record, and logs what it saw and why it acted.
6. Monitoring. Low-confidence cases and drift are sent back for review and relabelling.

Steps 4 and 5 are where the field is moving now. Rather than only showing detections on a dashboard, vision AI agents read the scene, check policy and take the next step, as described in our guide to agentic AI for real-time video analytics.

Video: what computer vision is and why enterprises use it (XenonStack).

What has changed in computer vision since 2024?

What do governance, security and cost look like for computer vision?

How should enterprises adopt computer vision today?

1. Pick one decision, not one model. Name the action the system will take and the error rate the business can accept.
2. Check the legal position first. Screen the use case against the EU AI Act prohibited list, high-risk categories and GDPR before buying cameras.
3. Start from a foundation model. Prompt or fine-tune an existing model and measure it on your own images before collecting large datasets.
4. Label what the model gets wrong. Use model proposals with human review rather than labelling everything by hand.
5. Decide edge, cloud or hybrid. NexaStack is an agentic operating platform for physical AI that runs across on-premises, cloud and hybrid environments, including vision AI at the edge.
6. Govern the action, not only the model. Record who or what approved each automated step; Audit Agent assembles decision lineage, policy versions, approvals and authority chains for compliance teams.
7. Scale by site and use case. Add cameras and use cases once monitoring shows the first one is stable.

For use cases by industry, see our guide to computer vision applications, and for factory inspection, our machine vision guide.

What are the trade-offs, and when is computer vision the wrong choice?

• Rule-based tools still win in controlled scenes. Fixed lighting and known parts often need only classic measurement, which is cheaper to validate than deep learning.
• Foundation models are large. They may be too slow or costly for small edge devices without distillation or quantisation.
• Accuracy is not uniform. Performance can drop for rare conditions, new sites or under-represented groups of people; test by slice, not only on average.
• Some uses are not allowed. If the use case needs emotion recognition at work or untargeted facial scraping, it is prohibited in the EU.
• Another sensor may be simpler. Barcodes, RFID, weight sensors or a form field can answer the question with less risk.

What will change in computer vision over the next 3 to 5 years?

The history shows a steady pattern: each era made it cheaper to teach a machine a new visual task. The predictions below rest on dated regulation, vendor notices and published research. Where a row is our own view, it says so.

What this means for you: if you are a CTO or CDO, the next step is not a bigger model but a governed one: pick a use case that is legal, measurable and logged, and build from a foundation model.

To map one vision decision your agents should make, and the context and policy it depends on, book a Context Workshop. See what happens in the workshop.

Frequently asked questions about the history of computer vision

Who started computer vision?

No single person did. Key early work includes Hubel and Wiesel's 1959 study of the visual cortex, Lawrence Roberts' 1963 MIT thesis on perceiving three-dimensional solids, and Seymour Papert's 1966 MIT Summer Vision Project.

Why was AlexNet important in the history of computer vision?

In ILSVRC 2012 the AlexNet entry, SuperVision, won image classification with a top-5 error of 0.153 against 0.262 for the best other team. It showed that deep convolutional networks trained on large datasets beat hand-crafted features.

What is the difference between classic and modern computer vision?

Classic systems used features designed by engineers, such as Viola-Jones cascades or SIFT keypoints. Modern systems learn features from data with convolutional networks, vision transformers and foundation models such as CLIP and Segment Anything.

What changed in computer vision regulation recently?

The EU AI Act bans untargeted facial image scraping, emotion recognition at work and in education, and some biometric categorisation from 2 February 2025. High-risk duties for Annex III use cases apply from 2 December 2027.

Is SIFT still patented?

No. The University of British Columbia's US patent 6,711,293 for SIFT expired on 7 March 2020, according to David Lowe's SIFT page.

How will computer vision change over the next 3 to 5 years?

We expect foundation models to become the usual starting point and vision agents to search, summarise and act on video. In the EU, high-risk vision uses need full compliance from 2 December 2027 (Annex III) and 2 August 2028 (Annex I).